text:
Error looking for icon for tutorial for 0.3.0.dev9

This is with Python 3.6.5 in a clean venv:
(.venv) PS C:\Users\_\Desktop\toga_tutorial> python .\helloworld.py
[Winforms] No valid icon format available for C:\Users\brcan\Desktop\toga_tutorial\.venv\lib\site-packages\toga\resources\tiberius; fall back on Tiberius instead

Unhandled Exception: Python.Runtime.PythonException: FileNotFoundException : Could not find file 'C:\Users\brcan\Desktop\toga_tutorial\.venv\lib\site-packages\toga\resources\tiberius.ico'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.Drawing.Icon..ctor(String fileName, Int32 width, Int32 height)
   at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Hrm... which tutorial is this (i.e., tutorial 0? tutorial 1?)
On a related note - if you've got any tips/pointers on how to run automated tests of GUI code on Windows, I'd love to hear about them - at the moment, we're only doing automated testing of internal APIs, so calls on GUI functionality are far too succeptible to breakage.

@freakboy3742 I know as much about Windows GUI programming as Toga has taught me so far. 😁 @d3r3kk might know.

And this was tutorial 0; basically you can't load an app on Windows without that file in place.

I am just now learning how to get up and running with toga, on a Windows machine (also on Linux). If I can, I will have a look and see if I can help out with automated tests and all that.
Not to bum anyone out, but pretty much all my prior experience with GUI tests is a storied tale of sorrow and loss...
But hey!

@d3r3kk Any assistance (or even guidance) you can provide would be most welcome.
And yeah... I've already got the scars from past battles with GUI testing. Nice to know time hasn't improved the situation... :sigh:...

repo: pybee/toga


text:
Caddy HTTP 500 when uploading a file

Hi!, how are you?. I've a system in production. When i try to upload a file with POST Caddy show me error 500. In local deployment it's ok but in production with caddy doesn't.
The log line is:
caddy_1 | 181.31.240.183 - - [07/Aug/2018:05:24:12 +0000] "POST /.../app_configurations/appconfiguration/1/change/ HTTP/2.0" 500 1646
Request doesn't reach Django never. can someone help me please?. I've checked upload file size but the image has 66kb.

Thanks for the report. We've migrated away from Caddy due to its licence (see #1714), so I'm going to close this. If there is still an issue with the new config, let's start a new issue.

source: gharchive/issue

repo: pydanny/cookiecutter-django

license_type: permissive

license_source: bigquery


text:
Simplified environment file

Description
Nowadays the environment files you get after generating the Django skeleton with this cookiecutter-django is sometimes too much complex. Most of the times you end up with a folder per environment with folders with a environment file per Django app and another for the database inside. In some cases is more comfortable to have just an environment file in the root of the project to handle all the environment variables.
Rationale
For small projects and even the ones you don't need to configure a database with environment file it feels too much to handle and you end up deleting all the environment structure dumped by this cookiecutter to switch it to the file mentioned in the Description section to have it simplified.
Use case(s) / visualization(s)
I see this could be handled during the questions asked in the cookiecutter console prompt. A quiestion like this could be added to the cookiecutter.json file:

"simplified_environment_file": "n",

Then have the simple environment file or the current structure is nowadays.
Do you have this in the development road map? Are you open on having PR on this matter?

After reading again the documentation (https://cookiecutter-django.readthedocs.io/en/latest/deployment-with-docker.html) is everything clear. I'm going to close this issue


text:
Prevent panicking when __dict__ changes during iteration

Change Summary
Gather the items of __dict__ into a SmallVec before iterating during serialization, so that modifications to the __dict__ in the process (e.g. from a @cached_property called by a field serializer) don't cause iteration to panic.
Replaces https://github.com/pydantic/pydantic-core/pull/1093
Related issue number
FIxes https://github.com/pydantic/pydantic/issues/7832
Checklist

[x] Unit tests for the changes exist
[x] Documentation reflects the changes where applicable
[x] Pydantic tests pass with this pydantic-core (except for expected changes)
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

I agree panicking isn't good, but I don't like the performance regression, and also worried about hampering future performance improvements.
@davidhewitt could we get this to raise an exception instead of panicking? Would that be a better way forward?

Sorry, I've reviewed the issue now.
Maybe this is the right idea, then we work on FastModel to improve performance significantly.

Oh, also the benchmark that's changed is unrelated.

source: gharchive/pull-request

authors: alexmojaki samuelcolvin

repo: pydantic/pydantic-core

license: MIT

license_source: github-api


text:
Class attributes starting with underscore do not support assignment

Bug
For bugs/questions:

OS: macOS
Python version import sys; print(sys.version): 3.6.6
Pydantic version import pydantic; print(pydantic.VERSION): 0.14

In #184 it was suggested to use variables starting with the underscore, however this does not work.
The last comment in #184 referred to the same problem, but it is technically a separate issue.
from pydantic import BaseModel
m = BaseModel()
m._foo = "bar"

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dmfigol/projects/my/public/simple-smartsheet/.venv/lib/python3.6/site-packages/pydantic/main.py", line 176, in __setattr__
    raise ValueError(f'"{self.__class__.__name__}" object has no field "{name}"')
ValueError: "BaseModel" object has no field "_foo"

(I've updated the issue title to better reflect the issue).
Personally, I don't think it's that important, but if you'd like the feature, I'd be happy to accept a PR.

I'm having this same problem in a non-workaroundable context. Can someone maybe give some context into why there is this restriction and if there's any way to hack around it, even if ugly?

Have you tried using field aliases?
eg,
class MyModel(BaseModel):
    foobar: str
    class Config:
        fields = {'foobar': '_foobar'}

That works in a small example
import json, my_models

with open('test.json') as f: d = json.load(f)
model = my_models.MyModel(**d)

I continue to have problems from the context of FastAPI. That's not the beginning of a new question, it's for the next person to arrive here from Google.
Thanks for your help!

That's not the beginning of a new question, it's for the next person to arrive here from Google.
👋 Oh hi!
@asemic-horizon Did you find a solution or a good workaround yet? I'll post again if I find one

Yeah sorry.
As I had said, field aliases worked in small toys examples but didn't seem to work in a larger app context, which made me suspect the problem was at FastAPI itself and therefore not pertinent to this issue.
After further rounds of research and bug squashing, minor and subtle errors on my end (my code on top of FastAPI) surfaced. By then I had forgotten I had gone to Pydantic support for it.
Thanks everyone!

(On my end I would "vote this issue closed" or its equivalent in the github issue tracker. I suspect I can't because I didn't actually open it.)

I'm pretty user aliases work fine in fastapi.
Closing, but if someone really wants this, let me know and I can reopen it.

@asemic-horizon field aliases work normally in FastAPI and are even documented/recommended in several places.
If you still have issues with them, feel free to open an issue in FastAPI with the minimum sample to reproduce your issue: https://github.com/tiangolo/fastapi/issues/new/choose

I read the above but saw no explanation for this limitation. From brief study of the source code I could not determine any danger in sunder fields.
This check is introduced in the first commit ever a8e844da and a check added in f9cf6b42 but no explanation given in commit messages.
Python itself by design does not enforce any special meaning to sunder and dunder fields - the "we are all consenting adults here" approach. I am used to a convention of marking fields private-like with sunder.
I would  very much like to know the motivation or changing this.

Hello there,  I use pydantic in FastAPI as well, and I use mongodb as database.
Mongo sets Identifiers as the field _id in collections/query outputs and handling those via Pydantic models is quite confusing because of this issue. Let me try to show what I do to ask if I'm doing something wrong; situations I've met with my use case

when sending the id as output I have to create a Model with the alias as

class Foo(BaseModel):
    id: str = Field(..., alias='_id')
    ...

query = db.collection.find(...)
validated = Foo(**query).dict()
# where validated = {'id': MONGO_HASH, ...}
# send validated as response

Note: if I'd want to send the id like mongo does (if my API are used by another service that deals with queries and mongo for example) I need to use
validated = Foo(**query).dict(by_alias=True)
# where validated = {'_id': MONGO_HASH, ...}

when storing a PUT in the database the above require the input to be using id instead of _id like:

data = {'id': EXISTING_MONGO_HASH}
db.collection.update(Foo(**data).dict(by_alias=True))

but if some other service is sending me raw data I would get the _id in input so I need to change the model to
class Foo(BaseModel):
    id: str = Field(..., alias='_id')
    ...
    class Config:
        allow_population_by_field_name = True

otherwise I get a validation error

There are other situations that I can't recll exactly enough to show by code,
but in general I'm currently forced by pydantic since the underscores are not accepted.
I don't feel like using the id translation at all, I'd go with _id always, because I'm risking non deterministic behaviour in models wrapper methods in the codebase and in the responses.
Hope this helps to reason on it, but thanks for this awesome library!

Hello @samuelcolvin 👋
I just started using pydantic along with FastAPI and pymongo. I used alias like @pdonorio mentions above. But I would like to understand why attributes starting with underscore are disallowed in the first place?

I still think most python users would not expect "private" attributes to be exposed as fields.
I know python doesn't formally prevent attributes who's names start with an underscore from being accessed externally, but it's a pretty solid convention that you're hacking if you have to access attributes with leading underscores. (Yes I know there are some exceptions, but in the grand scheme they're rare).
In the example quoted above regarding mongo, I would personally feel it's inelegant design to keep the mongo attribute _id in python with the same name. You should be able to use an alias generator and a shared custom base model to change the field name in all models without repeated code.
Still, if people really want this, I'd accept a PR to allow field names with underscores. I guess the best solution would be to add a new method to Config which replaces this logic, perhaps returning None to fallback to the default logic.
Then in v2 we can completely remove Config.keep_untouched it's a bad name and a confusing attribute.

None of my business, but: @gpakosz, unless i'm missing something, you're using a synchronous library for io (pymongo) while using an async web framework (starlette+fastapi). I would worry that your entire process will hang during every database call, thereby negating the point in asyncio.

I'm agree that, in python, underscore variables are considered private, so I understand your point
But
everyone that uses mongodb (perhaps others) must flow our code with by_alias=True which is not that bad (but talking about elegance seems very weird)
The problem becomes deeper when you use lists of pydantic models
Then you must add a good amount of extra fors to return the information in the correct way (for read and write)
What about a config attribute like always_by_alias = True?

You can just implement your own custom base model and overwrite .dict() to have by_alias=True default.
I don't want to add more config options unless absolutely necessary. Also it would be weird since I think .dict() and .schema() have different defaults.
Perhaps we can change the default to True in V2 once we have load and dump aliases.

Allowing fields with underscores would also conflict with #1139.

I'm very new to pydantic, however i'm really impressed and really like it, however with the current implementation filtering out any field starting with an underscore which is completely unconfigurable main.py 162
except monkey patching is not really acceptable. So I think it is really necessary to add an additional configuration parameter.

Same thing happens in connexion framework. I want to use default mongo _id as the id.
class ConfigurationFileMetadata(BaseModel):
    _id: str
    files: List[FileMetadata]
    type: str
    network_element_name: str
    created_date: str

Set id
data = ConfigurationFileMetadata.parse_obj(info.to_dict())
data._id = my_custom_id

Error
ValueError: "ConfigurationFileMetadata" object has no field "_id"

Better to use an alias for that field.

For the pymongo issue specifically I moved to mongoengine, which exposes _id as id. No more need for aliasing!

The solution for me was using aliases and by_alias when exporting the model. Using the example from @Seshirantha that would look as follows:
class FileMetadata(BaseModel):
    id: str = Field(alias="_id") # can be done using the Config class
    created_date: str

input_data = {"_id": "some id", "created_date": today()}
FileMetadata(**input_data).dict(by_alias=True)

That should return  {"_id": "some id", "created_date": "whatever today is"}
I am using Pydantic with pymongo therefore having access to _id and _meta on the validated model is very important.

+1 on this, would be a nice feature. Ill use alias in the meantime but this for sure caught me off-guard.

I earlier thought I needed that feature, and also found a quite easy way to have it via monkey patching:
`
monkey patch to get underscore fields
def is_valid_field(name:str):
if not name.startswith('__'):
return True
elif name == 'root':
return True
pydantic.main.is_valid_field = is_valid_field
`
However I have to say that using an alias works very well now for me!

#1679 would probably be a good solution

+1, "allowing fields with underscores" is necessary for mongodb issue. Using aliases is just workaround  and not very handsome

+1 – Also fields starting with _ are also useful for ArangoDB, each vertex object there has a _key, _id, and _rev, and edges also have _from and _to.

I am finding that I can't set attributes starting with underscore internally (via self.), not just externally. Is this a separate issue?

Hello,
Should we close this issue now that private model attributes have been implemented?
from pydantic import BaseModel, PrivateAttr

class Model(BaseModel):
    _foo = PrivateAttr('')

m = Model()
print(repr(m._foo))  # ''
m._foo = 'foo'
print(repr(m._foo))  # 'foo'

Sorry, in which version underscore_attrs_are_private are implemented? I use 1.7.2 and still have error while using private attributes:
import pkg_resources
print(pkg_resources.get_distribution("pydantic").version)

from pydantic import BaseModel

class Model(BaseModel):
    _foo: int
    class Config:
        underscore_attrs_are_private = False

m = Model(_foo=2)
print(m, m._foo)

1.7.2
Traceback (most recent call last):
  File "/Users/aovsyannikov/Library/Application Support/JetBrains/PyCharm2020.2/scratches/scratch_139.py", line 14, in <module>
    print(m, m._foo)
AttributeError: 'Model' object has no attribute '_foo'

Apparently with the new private (underscore_attrs_are_private) implementation, you can still not provide the values during initialization of the class. Moreover, the values are not included in the .dict() output.
class Model(BaseModel):
    _id: str = None

    class Config:
        underscore_attrs_are_private = True

x = Model(_id='a/123')
# Model()
x._id
# None
x._id = 'a/123'
x
# Model()
x._id
# 'a/123'
x.dict()
# {}

If I include extra = 'forbid' in the Config it forbids including _id when making the model, it can still be assigned after making the model, but is still not included in the dict output.

Ok, thanks, clear. Just copy there my comment to #2101
Now, been having the Config for such attributes it seems quite artificial limitation to ignore sunder attributes (which is only private in Python world, and widely used outside), so it brings lot of inconvenience when work with models for MongoDB i.g. Suppose more flexible config is required, like sunder_attributes : Literal['normal', 'ignored', 'private']. I will risk opening a new feature request.

@153957, @PrettyWood, just to make it clear: underscore_attrs_are_private and private attributes themselves are not meant to be used as fields. They're meant to be internally implemented and used only inside of model class, not anywhere outside.

Understood, however, for my usage with ArangoDB the issue remains that there are some 'normal/valid' fields which start with a single underscore that should be included in the output of dict(). For now I am using something similar to the monkey patch suggested by mortbauer.

I have just tried a workaround like this to issue the data model to MongoDB using .dict() method:
class A(BaseModel):
    _id = 5

    def dict(self):
        d = super.dict()
        d['_id'] = d['id']
        del d['id']
        return d

When I call A().dict(), it gives an error:  AttributeError: type object 'super' has no attribute 'dict'. But BaseModel has a dict method. I'm probably missing some Python feature. Can someone enlight me why this is not working? Thanks.

Hey @emremrah
Probably because you want to call super() not super ;)

Hey @emremrah
Probably because you want to call super() not super ;)

Ahh you are right @PrettyWood ,:
class A(BaseModel):
    id = 5

    def dict(self):
        d = super().dict()
        d['_id'] = d['id']
        del d['id']
        return d

returned {'_id': 5}. So if anyone is interested. I just have to call .dict() method when inserting to MongoDB. But initiating A with a MongoDB document still won't work as we wanted.

@emremrah Using alias id = Field(5, alias='_id') should work 👍

@PrettyWood Yep, I just have to be careful to use .id and not ._id in code. Thank you!

Thanks @PrettyWood, closed.

I have a use case that is kind of the opposite of what others are doing. Effectively, I have a use case where the raw input needs to be processed or fetched lazily and then cached. In my case, if a value is absent, then I need to make an API call to a remote system to fetch the data.
I need to load the raw value that arrives and keep it in a private member (something prefixed with an underscore _). Users will access a property of the class which will do the needful to ensure that a proper value is returned.
Here is what I am trying to do:
class SomeThing(BaseModel):

    some_field: str = Field(..., alias="someField")
    some_other_field: str = Field(..., alias="someOtherField")

    # Lazy-loading fields if they are absent
    _lazy_field: Optional[Dict[str, Any]] = Field(..., alias="lazyField")

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self._all_loaded = self._lazy_field != None

    def _lazy_load():
        # reach to to external API...
        # self._lazy_field = processed result from API...
        self._all_loaded = True

    @property
    def lazy_field():
        if not self._all_loaded:
            self._lazy_load()

        return self._lazy_field

Unfortunately, this does not work, because fields that are prefixed with an underscore are ignored. In order to make this work, I need to scrap the underscore on the field (make the field named raw_lazy_load or something equivalent). I would prefer to keep with the "expected" Python behavior and have it underscored to indicate that you should not directly access that field.
I tried the other class Config options mentioned in the thread, but don't work for me. If it's easy to fix, I'm happy to submit a PR for it.

Yeah, second all the _attribute support. it's not just Mongo, any Elasticsearch work behind FastAPI misses out on the meta-properties on documents. any in general as a DB developer, I find _attributes an expressive way of working around langue-specific keywords when necessary, like _id and _timestamp
to me, it seems like a by_alias workaround, while sufficient to solve any immediate issues, is not expressing the schema being parse.
It seems that the justification for why it's not supported is because Python relies on underscores for privacy problems, but I would argue that Python metaclasses being a way of having the language rewrite itself throws that issue out the window, particularly in their use in pydantic.
Additionally, I would ask what is the core responsibility or value proposition of pydantic, which I believe is as a model parsing library with applications in messaging applications. (e.g. there's nothing stopping a protobuf loader in the Model Config),a and if that IS the value proposition, then the BaseModel declaration should look as close as possible to the model being parsed.
so really, any restrictions on identifiers that aren't explicitly prohibited by the python language (like @timestamp), should probably be supported natively.
Just my two cents.

Yea we also really need this for use with ArangoDB.

So I've been working on this. What I needed is to easily cast between ObjectId and str to save to MongoDB or return data to frondend.
To keep things simple on the frontend, object ids are strings; not Objects with "$oid" key in it.
So I implement my data models like this:
# to cast strings to object id for incoming data, especially from frontend
class MongoObjectId(ObjectIdStr):
    "Cast string to ObjectId."""
    @classmethod
    def __get_validators__(cls):
        yield cls.validate

    @classmethod
    def validate(cls, v):
        if bson.ObjectId.is_valid(v):
            return bson.ObjectId(v)
        raise ValueError(f'{v} is not a valid ObjectId for type {cls}')

def oidstr(d):
    """Recursively search for values that objectids and convert them to strings."""
    if isinstance(d, dict):
        return {k: oidstr(v) for k, v in d.items()}
    if isinstance(d, list):
        return [oidstr(v) for v in d]
    if isinstance(d, bson.ObjectId):
        return str(d)
    return d

class MyModel(BaseModel):
    person_id: MongoObjectId

    def dict_with_str(self, *args, **kwargs):
        """Dump the model to a dict and convert ObjectIds to str."""
        # use the default dict method
        d = super().dict(*args, **kwargs)

        # recursively convert ObjectIds to str
        d = oidstr(d)

        return d

I don't know if I'm overcomplicating things while trying to keep it simple for front end. Please share your thoughts. Thank you.

As @frndlytm said:

so really, any restrictions on identifiers that aren't explicitly prohibited by the python language (like @timestamp), should probably be supported natively.

Regardless of the underlying database, Pydantic should not enforce what is only a convention. This is not an expected behavior.

I still think most python users would not expect "private" attributes to be exposed as fields.
I know python doesn't formally prevent attributes who's names start with an underscore from being accessed externally, but it's a pretty solid convention that you're hacking if you have to access attributes with leading underscores. (Yes I know there are some exceptions, but in the grand scheme they're rare).
In the example quoted above regarding mongo, I would personally feel it's inelegant design to keep the mongo attribute _id in python with the same name. You should be able to use an alias generator and a shared custom base model to change the field name in all models without repeated code.
Still, if people really want this, I'd accept a PR to allow field names with underscores. I guess the best solution would be to add a new method to Config which replaces this logic, perhaps returning None to fallback to the default logic.
Then in v2 we can completely remove Config.keep_untouched it's a bad name and a confusing attribute.
None of my business, but: @gpakosz, unless i'm missing something, you're using a synchronous library for io (pymongo) while using an async web framework (starlette+fastapi). I would worry that your entire process will hang during every database call, thereby negating the point in asyncio.

I also need a possibility to use underscore attributes.
As @samuelcolvin suggested himself, the easiest way would be probably to implement a simple switch in the Config, which would allow to use "_attribute" style.
Is it something that you would accept in PR?

Maybe this helps:
class PrivateInitBaseModel(BaseModel):
    """
    Workaround for initializing models by specifying their private attributes
    """

    def __init__(self, **data: Any) -> None:
        for private_key in self.__class__.__private_attributes__.keys():
            try:
                value = data.pop(private_key)
            except KeyError:
                ...
            finally:
                setattr(self, private_key, value)
        super().__init__(**data)

Can the original question be solved in any way? I tried using alias, but still model._foo = False raises the same error.

I have just started to use Pydantic in a couple of applications I am working on at the moment. The issue I am facing, which is probably related to the design I am using is the following:

I have created a series of Pydantic BaseModel classes, with a shared MyBaseModel, which are converting and validating data coming from a remote server API. The validation works, I am able to test it with a series of mock data and also live data, they behave as expected. So I was very happy with Pydantic till this point 😉
I need to integrate the data models into application models, I am not using FastAPI but just my own python application. What I mean with this, is that I have for example a Market(MarketBaseModel, AutoLogger) which does a classical mixin between the Pydantic model and the AutoLogger class which adds distributed logging features. The AutoLogger adds a local self.__logger attribute to the Market class, and getter and setter properties for the logger.

The problem I am facing is that no matter how I call the self.__logger, or self._logger or self.logger attribute, even if it is initialized in the init method and it isn't declared as a class attribute, because the MarketBaseModel is a Pydantic Model, extends the validation not only at the attributes defined as Pydantic attributes but also to every other attribute which is declared in init. Now I might be thinking too complicated, and surely I could use composition and pass a logger instance to the constructor of the Market, but that would cause to add a lot of boiler plate code that I want to avoid at least at the high level APIs. Any idea on how I can still inherit from a Pydantic BaseModel while having arbitrary defined instance attributes?

I'm thinking it might be possible to do something with keep_ignored, and setting an attribute on the class to a type in that.
More explicitly, you should be able to do something like:
class MyModel:
    a: int
    b: str  # whatever, random fields

    logger = MyLogger()

    class Config:
        ...
        keep_untouched = (MyLogger,)

Any chance this approach might work for you?

I'm thinking it might be possible to do something with keep_ignored, and setting one of the attributes on the class to have a value that is a type in that setting.
More explicitly, you should be able to do something like:
class MyModel:
    a: int
    b: str  # whatever, random fields

    logger = MyLogger()

    class Config:
        ...  # whatever other values you want to set on the Config
        keep_untouched = (MyLogger,)

Any chance this approach might work for you?

logger is not seems to be public variable. use __logger instead

Thanks for your suggestions, It appears I have managed to solve this problem at least at the moment, by changing the design and going for composition instead of inheritance. So I have now my model classes which have a class attribute called schema_class which is type bound to my schema superclass (a Pydantic BaseModel extension) and they have a from_data and to_data methods, which allows to read the serialized structure, validate the model using the schema_class, and store it as data storage as self.data into the model class. With the composition design I have also managed to create a PersistenceEnabled mixin which allow me to extend the main Model with SQLAlchemy persistence, using the same pattern, by declaring a db_class class attribute, and providing saving, loading and searching features on the model. This of course requires to have 3 classes implemented:

The main domain related Model class (e.g.: MyModel) which now can mixin with all other regular python classes;
The Pydantic schema class (e.g.: MyModelSchema) which does all of the validation and serialization business;
If the domain model needs persistence also a db class (e.g.: MyModelDB) which does all of the CRUD business behind the scenes, and allows also for transaction participation (in case of multiple items operation).

I didn't really solve the problem, but I went around it. Every piece is smaller, has clearer responsibilities and doesn't cause interference between the different frameworks.

I could be misunderstanding something, but I'm completely confused about how to work around this, as it seems to break private attrs, even for private usage. I'm using version pydantic@1.10.6 on python@3.8.16 and find private variables completely unusable.
I think in my case all I want is for a private variable to be perfectly, completely unknown to pydantic. Is there really no way to accomplish this? (Whether this is indicated by virtue of it being underscored or explicitly declared as unhandled by pydantic is pretty irrelevant to me.)
from pydantic import BaseModel

class Foo(BaseModel):
    _flag: bool = True

    def perform_action(self):
        self._flag = False

foo = Foo()
foo.perform_action()

Raises:
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    foo.perform_action()
  File "test.py", line 7, in perform_action
    self._flag = False
  File "pydantic/main.py", line 357, in pydantic.main.BaseModel.__setattr__
ValueError: "Foo" object has no field "_flag"

Sorry if I am being naive here, but after reading this thread, I came across this: https://docs.pydantic.dev/latest/usage/models/#private-model-attributes
This seems to solve this problem?
class Foo(BaseModel):
    class Config:
        extra = Extra.forbid

  bar: date
  __myvar__ = PrivateAttr(...)

@tommyjcarpenter I believe pydantic v2 was just released, and it looks like the diff in that code block is:
# v1.1:
    _secret_value: str = PrivateAttr()

# v2:
    _secret_value: str

so maybe it's that PrivateAttr is the default behavior for underscored attributes so that you don't explicitly need to specify? Not certain if there are differences beyond that.

@rreusser sorry I don't follow - just saying, I used PrivateAttr and it seemed to solve all the commotion in this thread.

Oh, sorry, I misread! Disregard. I've done the same, but I'm on v1.1 still, and when I clicked the above link I noticed that maybe this has changed slightly in v2.

authors: 153957 AntonOvsyannikov Bi0max GDYendell Garito MrMrRobat Pedrexus PrettyWood Seshirantha StephenBrown2 aolefira asemic-horizon bartekbrak dmfigol dmontagu edatpb emremrah fernando-freitas-alves frndlytm gpakosz iyedg jtfidje killswitch-GUI lgvld mikegrima mortbauer pdonorio rreusser samuelcolvin tiangolo tommyjcarpenter tumma72

repo: pydantic/pydantic


text:
Private model attributes handled incorrectly by type checkers

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
While private model attributes are handled correctly by pydantic at runtime as described in https://docs.pydantic.dev/latest/concepts/models/#private-model-attributes, it seems that type checkers still require private model attributes to be provided when instantiating a model.
For the example code provided, both Mypy and Pyright/Pylance show this problem:

The code runs just fine regardless of the type checker complaining.
On the flip side, attempting to actually specify the private attribute when instantiating keeps the type checker satisfied, but causes a runtime error.
thing = MyModel(_private_attr="bar")

# Traceback (most recent call last):
#   File "*/.venv/lib/python3.9/site-packages/pydantic/main.py", line 746, in __getattr__
#     return self.__pydantic_private__[item]  # type: ignore
# KeyError: '_private_attr'
#
# The above exception was the direct cause of the following exception:
#
# Traceback (most recent call last):
#   File "*/test.py", line 9, in <module>
#     print(thing._private_attr)
#   File "*/.venv/lib/python3.9/site-packages/pydantic/main.py", line 748, in __getattr__
#     raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}') from exc
# AttributeError: 'MyModel' object has no attribute '_private_attr'

I'm not familiar with the inner workings of pydantic, but my assumption is that this happens because private attributes are being added to the __init__ function signature of the model, which should not be the case (I think?).
Example Code
from pydantic import BaseModel

class MyModel(BaseModel):
    _private_attr: str

thing = MyModel()
print(thing.model_dump())

Python, Pydantic & OS Version
Python: 3.9.18
pydantic: 2.6.3
OS: macOS 14.2.1 (23C71)

I've done some more digging.
Just to confirm that the issue is not with the VSCode extension I performed the check with mypy directly and getting the same problem.
.venv ❯ mypy test.py
test.py:8: error: Missing named argument "_private_attr" for "MyModel"  [call-arg]
Found 1 error in 1 file (checked 1 source file)

I looked into how the signatures are build built for the models and everything seems to check out.https://github.com/pydantic/pydantic/blob/a3b7214a1d6ac8e32c29e9d4436ea6a9e253ead7/pydantic/_internal/_model_construction.py#L570-L573
Using inspect.signature on the model returns the correct signature without the private attributes.
from inspect import signature

from pydantic import BaseModel

class MyModel(BaseModel):
    _private_attr: str

print(signature(MyModel)) # () -> None

This begs the question of where are the type checkers pulling their signatures from?

@TeeSeal I ran into the same issue. Since https://github.com/pydantic/pydantic/pull/9293 was merged, I was now able to solve the pyright issue by marking the attribute as a PrivateAttr:
from pydantic import BaseModel, PrivateAttr

class MyModel(BaseModel):
    _private_attr: str = PrivateAttr()

thing = MyModel()
print(thing.model_dump())

@mvernacc

I ran into the same issue. Since #9293 was merged, I was now able to solve the pyright issue by marking the attribute as a PrivateAttr:

Thanks. Yeah using PrivateAttr() does make it behave as intended. However, the documentation still suggests that it should be usable without https://docs.pydantic.dev/latest/concepts/models/#private-model-attributes
from datetime import datetime
from random import randint

from pydantic import BaseModel, PrivateAttr

class TimeAwareModel(BaseModel):
    _processed_at: datetime = PrivateAttr(default_factory=datetime.now)
    _secret_value: str

    def __init__(self, **data):
        super().__init__(**data)
        # this could also be done with default_factory
        self._secret_value = randint(1, 5)

m = TimeAwareModel()
print(m._processed_at)
#> 2032-01-02 03:04:05.000006
print(m._secret_value)
#> 3

I'll try to have a look deeper at how PrivateAttr is handled and try to apply the same logic for regular underscore attributes, see if that fixes it.

authors: TeeSeal mvernacc


text:
incomplete ValidationError when validating nested models with validate_default=True

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
The following example uses a model with a field that is also a model, with a field with a default value, and using  validate_default=True. When default values fail validation (e.g. after checking the environment or so), the ValidationError is missing some errors entries and that the ones referring to nested models errors have a wrong loc.
Example Code
from typing import Annotated
import pydantic

class BaseModel(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(validate_default=True)

class S(BaseModel):
    s_id: Annotated[int, pydantic.Field(ge=3)] = 2

class M(BaseModel):
    m_id: Annotated[int, pydantic.Field(ge=2)] = 1
    s: Annotated[S, pydantic.Field(default_factory=S)]  # FAIL
    # s: Annotated[S, pydantic.Field(default_factory=dict)]  # Works!

# Trying to validate an empty object, thus using default values, shows that
# the ValidationError is missing an error (about M.m_id) and that
# the "loc" entry of M.s.s_id is incorrectly set to "s_id":
>>> M.model_validate({})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../.venv/lib/python3.11/site-packages/pydantic/main.py", line 509, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.11/site-packages/pydantic/main.py", line 171, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for S
s_id
  Input should be greater than or equal to 3 [type=greater_than_equal, input_value=2, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/greater_than_equal

# As compared to validating with explicit default values:
>>> M.model_validate(obj)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../.venv/lib/python3.11/site-packages/pydantic/main.py", line 509, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for M
m_id
  Input should be greater than or equal to 2 [type=greater_than_equal, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/greater_than_equal
s.s_id
  Input should be greater than or equal to 3 [type=greater_than_equal, input_value=2, input_type=int]
    For further information visit https://errors.pydantic.dev/2.6/v/greater_than_equal

# Same with pytest

expected_errors = [
    (("m_id",), "Input should be greater than or equal to 2"),
    (("s", "s_id"), "Input should be greater than or equal to 3"),
]

def test_model_defaults() -> None:  # FAIL
    with pytest.raises(pydantic.ValidationError) as cm:
        M.model_validate({})
    assert [(e["loc"], e["msg"]) for e in cm.value.errors()] == expected_errors

def test_model() -> None:  # PASS
    obj = {"m_id": 1, "s": {"s_id": 2}}
    with pytest.raises(pydantic.ValidationError) as cm:
        M.model_validate(obj)
    assert [(e["loc"], e["msg"]) for e in cm.value.errors()] == expected_errors

Python, Pydantic & OS Version
pydantic version: 2.6.4
        pydantic-core version: 2.16.3
          pydantic-core build: profile=release pgo=true
                 install path: /home/denis/src/dalibo/pglift/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
                     platform: Linux-6.1.0-18-amd64-x86_64-with-glibc2.36
             related packages: pydantic-settings-2.2.1 typing_extensions-4.10.0 mypy-1.9.0
                       commit: unknown

@dlax,
Thanks for reporting this. I don't think this is actually a bug - the behavior you're seeing is the same as that which you get if you don't validate the default values, but instead pass in the same invalid values to the constructor:
import json
from typing import Annotated

import pydantic

class BaseModel(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(validate_default=True)

class S(BaseModel):
    s_id: Annotated[int, pydantic.Field(ge=3)] = 2

class M(BaseModel):
    m_id: Annotated[int, pydantic.Field(ge=2)] = 1
    s: Annotated[S, pydantic.Field(default_factory=S)]  # FAIL
    # s: Annotated[S, pydantic.Field(default_factory=dict)]  # Works!

try:
    s = S()
except pydantic.ValidationError as e:
    print(json.dumps(e.errors(include_url=False), indent=2))
"""
[
  {
    "type": "greater_than_equal",
    "loc": [
      "s_id"
    ],
    "msg": "Input should be greater than or equal to 3",
    "input": 2,
    "ctx": {
      "ge": 3
    }
  }
]
"""

try:
    m = M()
except pydantic.ValidationError as e:
    print(json.dumps(e.errors(include_url=False), indent=2))
"""
[
  {
    "type": "greater_than_equal",
    "loc": [
      "s_id"
    ],
    "msg": "Input should be greater than or equal to 3",
    "input": 2,
    "ctx": {
      "ge": 3
    }
  }
]
"""

# let's consider the case without default values (and without validating them)
class S(pydantic.BaseModel):
    s_id: Annotated[int, pydantic.Field(ge=3)]

class M(pydantic.BaseModel):
    m_id: Annotated[int, pydantic.Field(ge=2)]
    s: S

try:
    s = S(s_id=2)
except pydantic.ValidationError as e:
    print(json.dumps(e.errors(include_url=False), indent=2))
"""
[
  {
    "type": "greater_than_equal",
    "loc": [
      "s_id"
    ],
    "msg": "Input should be greater than or equal to 3",
    "input": 2,
    "ctx": {
      "ge": 3
    }
  }
]
"""

try:
    m = M(m_id=1, s=S(s_id=2))
except pydantic.ValidationError as e:
    print(json.dumps(e.errors(include_url=False), indent=2))
"""
[
  {
    "type": "greater_than_equal",
    "loc": [
      "s_id"
    ],
    "msg": "Input should be greater than or equal to 3",
    "input": 2,
    "ctx": {
      "ge": 3
    }
  }
]
"""

What's happening in both of these cases is that validation is failing on the construction of the S model for the s field, and so validation terminates at that point.

authors: dlax sydney-runkle


text:
Fixed #12661: more clarification in the where statement

[x] closes #12661
[ ] tests added / passed
[ ] passes git diff upstream/master | flake8 --diff
[ ] whatsnew entry

Improvement in the documentation w.r.t to issue #12661

Personally, I feel explanation of .where() is inappropriate here. Because the section focuses on df[df.A > 0] and df[df > 0]
Maybe following sentence should be fixed, or pls add an explanation if you add .where() example.

A where operation for getting.

ok sir, i will add brief explanation to where

I agree with @sinhrks, I don't think the explanation of where is needed here. It is a section about boolean indexing (which does a 'where' operation), not about the where method. And I don't think it is needed to mention the where method in an 10min intro (there is already a section on where in the docs: http://pandas.pydata.org/pandas-docs/stable/indexing.html#the-where-method-and-masking).
And sorry that this was not really clear from the issue #12661.
I would just remove the `` around 'where' to make it clear that it is not a method.

I would just remove the `` around 'where' to make it clear that it is not a method.

so, I just remove the `` from where
and also it is given in detail in the indexing section:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#the-where-method-and-masking

so, any further work to be done?

closing, but if updated pls reopen

authors: Prabhnith jorisvandenbossche jreback sinhrks

repo: pydata/pandas


text:
BUG: SparseArray numeric ops misc fixes

[x] no existing issue
[x] tests added / passed
[x] passes git diff upstream/master | flake8 --diff
[x] whatsnew entry

Fixed following 3 issues occurred on the current master.
1. addition ignores rhs fill_value
pd.SparseArray([1., 1.]) + pd.SparseArray([1., 0.], fill_value=0.)
# [2.0, nan]
# Fill: nan
# IntIndex
# Indices: array([0], dtype=int32)

Expected:
# [2.0, 1.0]

2. mod raises AttributeError
pd.SparseArray([1, 1]) % pd.SparseArray([1, np.nan])
# AttributeError: 'module' object has no attribute 'sparse_nanmod'

3. pow outputs incorrect result wiht 1.0 ** np.nan
pd.SparseArray([1., 1.]) ** pd.SparseArray([1., np.nan])
# [1.0, nan]
# Fill: nan
# IntIndex
# Indices: array([0], dtype=int32)

Expected:
# [1.0, 1.0]

# NumPy result
np.array([1., 1.]) ** np.array([1, np.nan])
# array([ 1.,  1.])

whoosh you blew away lots of code. must have been work-arounds built in there for maybe an old cython or something. Just want to be sure (since tests were removed) everything still working (as may not be testing some of the numeric ops as much, though did see you added some tests)

The removed test calls _nan funcs which is no longer exists, and remaining logic is tested with _op_tests below. I'm willing to add more tests if you have anything.

thanks!

authors: jreback sinhrks


text:
To numeric enhance

This is a test PR copy of #13425 to test travis - do not merge

[Current coverage][cc-pull] is 84.34%

Merging [#13512][cc-pull] into [master][cc-base-branch] will increase coverage by <.01%

@@             master     #13512   diff @@
==========================================
  Files           138        138
  Lines         51107      51113     +6
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits          43103      43109     +6
  Misses         8004       8004
  Partials          0          0

Powered by Codecov. Last updated by [bf4786a...ac37bd4][cc-compare]
[cc-base-branch]: https://codecov.io/gh/pydata/pandas/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/pydata/pandas/compare/bf4786a15f51d14ccf1318e44ae080868d8595ee...ac37bd45fd35a5c743d9e76fd4e0b446870ce66b
[cc-pull]: https://codecov.io/gh/pydata/pandas/pull/13512?src=pr

authors: codecov-io nparley


text:
WIP: Factor out a function for checking dimension-related errors

This is a WIP follow-up for #8079 and I think also for #7051. The pattern
missing_dims = set(dims) - set(self.dims)
if missing_dims:
    raise ValueError(f"Dimensions {missing_dims} not found in data dimensions {tuple(self.dims)}")

occurs in many methods, with small variations in the way missing_dims is calculated, the error message, and also if it's ValueError or KeyError. So it would make sense to factor it out. But I'm not familiar enough with the context around  #7051 to know how to deal with sets vs tuples, so this is just a sketch for now.

[ ] Tests added
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst

No worries! It might take me a while before I can finish this up, but I've rebased this now that #8079 is merged.

Is the idea to move many of these checks to use the check_dims function?

Yes. There is a possibility that it will turn out messy or unreadably generic because there are quite a few different variants of the pattern, even though they are similar.

Ah, I see — I think the diff I reviewed two days ago was almost exclusively the prior PR, because there was a conflict and so GH showed too big a diff.
That was a big benefit. My guess is that delegating the message construction is a smaller benefit, and maybe a bit too generic, as you suggest. But very open to it if we can find enough commonality...

Yes, this PR was just one commit on top of the previous one. I'll see if this turns out to be worthwhile.

authors: max-sixty mgunyho

repo: pydata/xarray


text:
Release automation

Things I'd like to automate:

From tag to PyPi -- ideally, to test-pypi on every tagged commit (https://github.com/marketplace/actions/pypi-publish) -- and the actual release would be a special action I just click on. It's not that it takes a lot of time, but I'm always nervous I mess something up with the command line, forget to delete a file, git clean, etc. -- I very diligently work through the release-checklist to avoid that. Literally signing things off.
Auto-formatting -- I tend to clean up the formatting of the lexers every time close to release, at least the worst offenders. I use autopep8 at the moment, would rather apply flake8 or black on the entire codebase.
Auto-check that the new arguments like URL etc. are present on new lexers
Auto-check .. versionadded::  is there -- costs me a lot of time to open up every Lexer close to release and make sure it's present and in the right format (i.e. 2.17.0 vs. 2.17)
Actually get all checks working/passing (i.e. the additional checkers I wrote and possibly PyLint). check_whitespace_tokens and check_repeated_tokens need an expected-fail list so we can whitelist currently existing lexers until we fix those, but new lexers should always pass those tests.
Verify all PR numbers closed/merged since last release are mentioned in the CHANGES file. I'm pretty good at assigning tasks to milestones now, but I still miss things in the CHANGES file, and it's super time consuming to open 100 tabs, go through each item one-by-item, check the PR number/issue number is present, etc. If there was a way to auto-generate the changelog that would be even better, but my experience is that those look pretty ugly and some manual checkup is fine.

I'll get to the release in a moment, thanks for the offer though!
Originally posted by @Anteru in https://github.com/pygments/pygments/pull/2576#discussion_r1398348679

Yes. The problem is that we haven't been doing this consistently though, so I'd like one file which contains all exclusions. I.e. run this one over all lexers, find which checkers work/don't work, then write it out with all exclusions, so we can run it for new ones while reducing the exclusion list. That requires however unifying all the checkers somehow or port them to the test framework, and I haven't had enough time to figure out how to integrate all of this with pytest. I'm sure it's doable.

Yes, Ruff is really good but its linting only completely replaces pyflakes for now. pylint catches way more things (also way more false positives, sigh). A clean Ruff run could be part of CI.
For formatting, if we ever wanted to do that, Ruff is also a good option because it allows selecting single quotes as the preferred style :smile:

Ok, I also prefer single quotes for strings, so I guess Ruff it is. I'll give it a try.

I don't have a strong opinion either. I'm not a huge fan of Black's style, but realistically, what are our options? I'm extensively using flake8 when editing Python code (and try to fix warnings it finds in Pygments as well), and without some automated code formatting, I'll spend the rest of my life telling people about styling issues. Like, I'd really like to enforce 79-wide lines so I can diff two things side-by-side and making this automatic is much less work than arguing with every single committer about the width of their screen and how much more productive they get when writing 95 character long lines.
I've been doing the same at work, you force clang-format on everything, and it just saves time in the end. It also makes other linters etc. more useful as my VS Code will only show "real" issues afterwards, and not mix in the formatting issues.

Lol. I don't have marked opinions on formatting, but I thought you didn't want Black due to its style? (Ruff's formatter is designed to produce output almost identical to Black.)

Yeah I didn't want to suggest to implement it immediately, and possibly there will never be a formatter that keeps the lexer code compact enough for my tastes...

In my opinion we're just delaying the inevitable here. Will it be ugly? Probably. Does it mean we have a few cases of foo|bar|baz which are better off using words(...)? For sure. But what's the alternative -- delay auto-formatting indefinitely, and end up doing some small fixes every once in a while?

I also despise this waste of space:
(
    regex,
    token
),
(
    regex,
    token
)

Black would reformat that to:
(regex, token),
(regex, token)

It would only expand to one argument per line if they don't all fit in the line limit.
Or, if you manually added the "magic trailing comma" after token and it'll assume you actually want that and expand it:
(
    regex,
    token,
),
(
    regex,
    token,
)

But don't add those if you don't want that :)

authors: Anteru birkenfeld hugovk jeanas

repo: pygments/pygments


text:
[INF] Set independent environment for building documentation

PR Description
Please describe the changes proposed in the pull request:
/environment-dev.yml is complex.
It's used for building doc, setting developer local env, and setting ci test env.
This file becomes big and big and hard to understand.
Part of #1133.
PR Checklist

Please ensure that you have done the following:

[x] PR in from a fork off your branch. Do not PR from <your_username>:dev, but rather from <your_username>:<feature-branch_name>.

[x] If you're not on the contributors list, add yourself to AUTHORS.md.

[x] Add a line to CHANGELOG.md under the latest version header (i.e. the one that is "on deck") describing the contribution.

Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

Automatic checks
There will be automatic checks run on the PR. These include:

Building a preview of the docs on Netlify
Automatically linting the code
Making sure the code is documented
Making sure that all tests are passed
Making sure that code coverage doesn't go down.

Relevant Reviewers

Please tag maintainers to review.

@ericmjl

Why this test case (tests/functions/test_conditional_join.py::test_multiple_eqs_extension_array) failed in https://github.com/pyjanitor-devs/pyjanitor/runs/7761821318?check_suite_focus=true but passed now?

Why this test case (tests/functions/test_conditional_join.py::test_multiple_eqs_extension_array) failed in https://github.com/pyjanitor-devs/pyjanitor/runs/7761821318?check_suite_focus=true but passed now?

I'm not sure, @Zeroto521 - some tests may be flaky, I guess. Usually a re-run of the CI might do the trick.
With everything green, I'm happy to merge!

repo: pyjanitor-devs/pyjanitor


text:
Add clv_summary Utility Function

When modeling entire customer populations, the clv models require recency, frequency, and T summary variables for each customer. For the Gamma-Gamma model, average spend beyond the first transaction is also required. Formatting input data in this fashion is non-trivial, so a utility function would be highly convenient for end users.
The summary_data_from_transaction_data() function from Lifetimes can be brought over for this, but will likely require revision because it currently ignores SettingwithCopyWarning in Pandas. The name of the Lifetimes function also seems verbose to me, but if clv_summary() is not descriptive enough, we can discuss alternatives.

clv_summary sounds fine

repo: pymc-labs/pymc-marketing


text:
Slack invitation has expired

I'd like to join the Slack channel, but the invitation link has expired.
Cheers!

It seems invitation links are only valid for 30 days. You should be able to use this one. It would be nice to have a solution that lasts longer.

Thanks for letting us know! I updated the link with one that is supposed to be non-expiring.

authors: FoundationKen flit mbrossard

repo: pyocd/pyOCD


text:
Remove f string with no f

Make flake8 happier by removing the f from the f string with no f'd parts. :)
This should allow CI to pass so future dependabot updates (like #405, #406, #407) will be easier to merge, preventing the project from falling behind security updates.

Use of a hammer: $1.00
Knowing where to tap: $999.00

😂

repo: pyouroboros/ouroboros


text:
Traceback exposed if upload fails

Hello,
I got a traceback in flit while trying to publish a package:
$ flit publish
Found 360 files tracked in git                                                                                I-flit.sdist
Built sdist: dist/MyPackage-0.5.0.tar.gz                                                                                I-flit_core.sdist
Copying package file(s) from /tmp/tmpdw358fz9/MyPackage-0.5.0/mypackage                                                                                I-flit_core.wheel
Writing metadata files                                                                                I-flit_core.wheel
Writing the record of files                                                                                I-flit_core.wheel
Built wheel: dist/mypackage-0.5.0-py3-none-any.whl                                                                                I-flit_core.wheel
Using repository at https://upload.pypi.org/legacy/                                                                                I-flit.upload
Uploading dist/mypackage-0.5.0-py3-none-any.whl...                                                                                I-flit.upload
Traceback (most recent call last):
  File "/home/palao/.virtualenvs/flit-py39/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/home/palao/.virtualenvs/flit-py39/lib/python3.9/site-packages/flit/__init__.py", line 185, in main
    main(args.ini_file, repository, args.pypirc, formats=set(args.format or []),
  File "/home/palao/.virtualenvs/flit-py39/lib/python3.9/site-packages/flit/upload.py", line 268, in main
    do_upload(built.wheel.file, built.wheel.builder.metadata, pypirc_path, repo_name)
  File "/home/palao/.virtualenvs/flit-py39/lib/python3.9/site-packages/flit/upload.py", line 246, in do_upload
    upload_file(file, metadata, repo)
  File "/home/palao/.virtualenvs/flit-py39/lib/python3.9/site-packages/flit/upload.py", line 239, in upload_file
    resp.raise_for_status()
  File "/home/palao/.virtualenvs/flit-py39/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Invalid value for requires_dist. Error: Can't have direct dependency: 'somedep @ git+https://github.com/whoever/somedep.git' for url: https://upload.pypi.org/legacy/

the problem is that PyPI rejects it (seems to be related to their policy, see #555).
However my question is: is the display of the traceback a feature, or a bug? Should I make a PR?

In that case, the traceback clearly isn't helpful - there's a problem with the package we're uploading, not the uploader code. I'm wary of hiding tracebacks in general, though, because they can be useful to understand what's going on.
Some possible ways to address that:

Extend the checks we do when building to cover such cases - the idea is that we catch things like this before you try to upload to PyPI. Checking requirements is done here. This one is tricky, however, because a requirement specified by direct URL may be OK for a package you're using privately, it just can't go on PyPI.
Hide the traceback specifically when we get status code 400, which hopefully means PyPI is telling us about something wrong with the package. :crossed_fingers:
Hide the traceback normally but have some way to turn it on (this is inspired by Rust). The downside is that it's a pain for rare errors where it might be fine when you run it again.

P.S. Sorry this one went unanswered for so long

Thank you, @takluyver for answering and for sharing this clear analysis. I really appreciate your comments: they are very insightful.
No worries: it is not a critical thing... still I think it should be fixed, and I agree, the big problem I see here is that the traceback can be confusing. The error is correctly describing what happens, but I needed a while to understand that flit was doing the right thing and that it was my ignorance about the PyPI policy what caused the problem.
I would be in favour of not doing a very clever checking. I don't see a practical way to address your point 1.
Your reasoning about hiding tracebacks is convincing (your point 3): it can be a painful default. Still it is more appealing than point 1. I use this strategy sometimes.
My winner is point 2: if flit is doing things right and the hosting is legitimately answering "No, you can't do that", I think this is a case that should be properly addressed by flit. Simply preventing the exception to propagate and printing an error similar to what's displayed  in the last line should be enough (maybe adding some more generic explanation).
What do you think? Should I do a PR?

I think that's reasonable, yeah.
What we do for other scenarios like this is that the main CLI code catches a group of errors where we just want to display the message, not the traceback - e.g. here it is for flit build:
https://github.com/pypa/flit/blob/f5704ea31f0fcc579b8518ea85d641651cba4f71/flit/__init__.py#L178-L182
So I would define a new exception like UploadRejectedError, transform HTTP 400 errors into this, and then apply a similar mechanism to flit publish.

I like it. It makes sense.

authors: palao takluyver

repo: pypa/flit


text:
failed install jsonpath with indexes of "https://pypi.doubanio.com/simple/"

pipenv --version
pipenv, version 2018.11.26

>pipenv install jsonpath
Installing dependencies from Pipfile…
An error occurred while installing envelopes! Will try again.
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]: File "/home/testhadoop/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pipenv/core.py", line 1874, in do_install
[pipenv.exceptions.InstallError]: keep_outdated=keep_outdated
[pipenv.exceptions.InstallError]: File "/home/testhadoop/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]: pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]: File "/home/testhadoop/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]: retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]: File "/home/testhadoop/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]: _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]: File "/home/testhadoop/.pyenv/versions/3.6.8/lib/python3.6/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]: raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Looking in indexes: https://pypi.doubanio.com/simple/']
[pipenv.exceptions.InstallError]: ["ERROR: Could not install packages due to an EnvironmentError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check',))"]
ERROR: ERROR: Package installation failed...

I can't reproduce this issue (see below) and it may be an issue with whatever that other site is. Please run pipenv --check and include the output.

Seems an issue with the mirror index, can you check if it happens with pip install?

I met the sime problem  while using pipenv install from my own pypi source. but it works ok when using pip install, I dont know why
[pipenv.exceptions.InstallError]: ["DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support", "ERROR: Could not install packages due to an EnvironmentError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))"]
ERROR: ERROR: Package installation failed..

Is there any one workout it

老铁, 问题解决了吗?
我在 pipenv 安装 scrapy 的时候也遇到这个问题. 莫名其妙, 以前一直OK, 神奇的是使用 pip 能安装scrapy成功;   后来换成阿里的源, 然后就没有报错了
阿里源  "http://mirrors.aliyun.com/pypi/simple/"

老铁, 问题解决了吗?
我在 pipenv 安装 scrapy 的时候也遇到这个问题. 莫名其妙, 以前一直OK, 神奇的是使用 pip 能安装scrapy成功;
执行 pipenv check , 显示所有的依赖都安装了, 执行 pipenv install这个错误就又出来了
后来换成阿里的源, 然后就没有报错了
阿里源 "http://mirrors.aliyun.com/pypi/simple/"

至今没整明白问题怎么出现的， 最后我把pipenv卸载了， 就用virtualenv 和virtualenvwrapper 少了很多问题， 我只想说pipenv真的不好用

This issue is caused by the douban mirror index(pypi.doubanio.com/simple), change to another index(such as http://mirrors.aliyun.com/pypi/simple/ , metioned by @jeeyshe  ) should fix it.

@sc-syf thanks，I got

authors: 527147280 frostming jeeyshe jeyrce mocobk nicktimko sc-syf

repo: pypa/pipenv


text:
up/down arrow history doesn't work on pipenv shell on windows

In the cmd shell, usually you can hit 'up arrow' to get your previous command. Once you run 'pipenv shell', this no longer works. if you run some commands inside the pipenv shell and then hit 'up' it does nothing.

FWIW, it works in 'ConEmu', just not the normal 'Windows Console Host' process that you get by running cmd.exe directly, so whatever they're doing in there to pass along the arrow keys is what you'd need. Not sure it's worth it to implement seperately in pipenv, but at least this is some kind of a workaround.

This is an issue with pew since it spawns the subshell on windows.  Can you open an issue over there?  I'm a maintainer of that project too and I'll make sure to address it on that project!
https://github.com/berdario/pew/issues

(FYI in the next release -- which will be today -- we are no longer using pew for managing subshells).

I tried installing from the repo and after I did pipenv shell, command history still didn't work. This is how I installed:
pip install git+https://github.com/pypa/pipenv
Collecting git+https://github.com/pypa/pipenv
  Cloning https://github.com/pypa/pipenv to c:\<snip>\appdata\local\temp\pip-req-build-yv19jpwg
  Installing build dependencies ... done
Requirement already satisfied: pip>=9.0.1 in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (10.0.1)
Requirement already satisfied: certifi in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (2018.4.16)
Requirement already satisfied: setuptools>=36.2.1 in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (39.2.0)
Requirement already satisfied: virtualenv-clone>=0.2.5 in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (0.3.0)
Requirement already satisfied: virtualenv in c:\<snip>\python36\lib\site-packages (from pipenv==2018.7.1.dev0) (16.0.0)
Installing collected packages: pipenv
  Running setup.py install for pipenv ... done
Successfully installed pipenv-2018.7.1.dev0

I tried it with an existing virtualenv and a newly recreated one. Let me know if I should try another branch.
Thanks!

Right this is a limitation of subshells in windows. There isn’t a fix for that and nobody is actively implementing one (this would effectively mean reimplementing subshells as far as I understand). Just sharing that pew is no longer involved in this aspect

I keep seeing that said that, but with virtualenvwrapper-win workon <env> this works fine and I think its just batch files on top of virtualenv.
Virtualenvwrapper-win workon: https://github.com/davidmarble/virtualenvwrapper-win/blob/master/scripts/workon.bat
Worth noting: virtualenvwrapper-win does not support powershell it is mentioned there are versions which do, but I'm not sure which is the best so I won't link to one.
Maybe a pipenvwrapper-win needs to appear. But that would be sad I think.
If there are suggestions on approaches to fixing, I'm happy to take a swing, I just don't know this area well enough to do without a little guidance.
Thanks!

Just want to mention there is always pipenv run. Subshell-omg is by nature difficult to handle and has many platform restrictions. run is a better model unless you have a very subshell-friendly shell (e.g. fish).

I'm intrigued, but I don't understand.
How would you use pipenv run  to have your dev or run time environment available at the windows CLI or from an IDE launched from same?

I’m afraid I don’t quite understand the use cases you mentioned :( Could you be more specific what you think is the problem with run?

I I see your point now (tried a few more things). When you pipenv run cmd you will have your environment available to you and you can launch your IDE from there as well. (VS Code: code .) and within VS Code in terminal you will also see all your pipenv installed resources as well.
AND your command history work (up arrow).
That seems like how it should work to begin with. I wonder if there are any downsides to this approach.

I can’t speak for everyone but I very rarely use subshells myself and I know @uranusjr doesn’t either. Since we are the main devs on the project following our workflow is most likely a safe bet

I had assumed pipenv shell was the analog to workon <env>. If pipenv run cmd is the better answer then I'm totally stoked!
Thanks!

@W7PEA

When you pipenv run cmd
AND your command history work (up arrow)

up arrow still doesn't work for me in this scenario. I'm using latest version 2018.7.1

Okay, now we purged Pew, we should track this on our own :p
From the discussion in berdario/pew#165, I think the simplest way to do this right now is to perform DLL-injection. Vendor the Clink executable, and when we are about to launch cmd.exe, instead of running it directly, run Clink instead. We can probably reuse Clink’s startup BAT file, or clink.exe inject to achieve the same effect.

Semi check semi todo list:

Clink breaking DOSKEY could be a problem. We should likely need to provide a toggle for Clink injection. Should we offer the injection as an opt-in, or opt-out, however?
Could GPL be a problem? We will probably only vendor the binaries, so maybe not.
Clink in binary form is about 1.5MB—quite significant—but there is not much we can shed. What do we do?
There’s clink inject --pid to inject into an arbitrary PID. Sounds very useful for us. clink --pid <pid> --nolog --quiet would likely behave well.

Seems like there is a fork to make DOSKEY work, but I’m not sure if it’s is a good idea to use it. Cmder also uses the fork (if I read the thread correctly), so that seems like well-proofed work.
https://github.com/mridgers/clink/pull/464#issuecomment-338764961

We need to find a way to be clever about this. I guess we can build platform dependent wheels.

I’m thinking maybe we can perform a one-time download at runtime, to avoid vendoring altogether. Only plain cmd.exe users would need to make that download once (unless the user clears cache).

Note: The above comment is outdated. Clink released 0.4.9 which included the DOSKEY fix, and Cmder is using that instead of the fork.

@mungojam this may no longer be relevant with the new changes down the road, but for me on Windows 10, this worked:
pip install git+https://github.com/pypa/pipenv.git
cd \someemptyfolder
pipenv install requests
pipenv run cmd

Now you're at a new prompt run a command like DIR and then hit the up arrow... bingo!
NOTE: It doesn't access the command history from before you launched the new CMD. I think that is to be expected.

@W7PEA
pip install git+https://github.com/pypa/pipenv.git
Unfortunately I fail at this first hurdle, my computer is just hanging when I try to install this from GitHub. I'm only able to install the latest from pypi which still exhibits the issue.
One thing I have realised is that pipenv run cmd works as a workaround when in a standard command prompt, but not within VS Code terminal using either cmd terminal or powershell.

@mungojam Are you starting VS Code from inside the virtualenv? So if I start a clean CMD and do this...
mkdir test
cd test
pipenv install requsts
pipenv run cmd
code .
CTL+` (Open VS Code Terminal)
pip list

You will see requests is installed because VS Code was started inside your Virtualenv.
Now open a new CMD prompt
cd test
code .
CTRL+` (Open Terminal)
pip list

You only see what you have installed in your OS, not the virtualenv.
This all worked for me from the released PyPi version.
As for the hanging install from the Git Repo. I can't reproduce that sorry.

Thanks for the tips, useful for me to be able to get it working for myself, but I look forward to when I can point my colleagues at generic pipenv guides and things work as they are used to with pipenv shell. Good luck getting it all working, sounds complex!

I I see your point now (tried a few more things). When you pipenv run cmd you will have your environment available to you and you can launch your IDE from there as well. (VS Code: code .) and within VS Code in terminal you will also see all your pipenv installed resources as well.
AND your command history work (up arrow).

I am on Windows 10 Pro (10.0.17134 Build 17134) with Pipenv 2018.11.14 and the pipenv run cmd does not fix the up/down arrow issue for me.

This is still a thing over here on my windows 10 machine running the latest windows 10 and the latest pipenv.

I can access my cmd shell history using pipenv run cmd but the interactive python shell history does not work. I'm on windows 10 running pipenv 2018.11.26

This happens for me with latest pipenv on Windows 10 1903. I don't recall it happening a year or so ago.

Hi guys, having the same issue with pipenv on my machine (Windows 10 1903/pipenv 2018.11.26) but I played around with it today and noticed some things while trying to get command history working for my personal scrapy project for when I use scrapy's shell from within a pipenv (pipenv scrapy shell "[url]"). I also included some snapshots of the process hierarchy during my testing to help explain.
I found that if instead of invoking pipenv through its .exe (pipenv shell) and did it using python's module execution (python -m pipenv shell) I'm able to use my command history correctly inside the pipenv shell, though I don't have command history for python if I invoke it from within the pipenv.
Non-working Before:

Working After:

So I did some more digging and testing and noticed that when the pipenv shell is started it's done so as "C:\Windows\System32\cmd.exe" /c "cmd.exe" with cmd.exe as a child process so I dug through the code and when I edited the function pipenv.shell._handover replacing:
sys.exit(subprocess.call(args, shell=True, universal_newlines=True))
with
sys.exit(subprocess.call(args, universal_newlines=True))
It'd start it as just "cmd.exe" without the parent cmd.exe and when I invoked python from the pipenv shell then my command history worked for python! And across multiple starts and stops from within the same shell instance! Not sure if that's the best way to invoke the shell yet, but it worked.
After code change:

So I'm guessing from my testing that if a process sits between the invoking process and the currently active child process it screws up command history, because I'd also loose my command history if I used scrapy's native scrapy.exe from Python's Scripts folder (like pipenv.exe) inside the pipenv shell so I also have to use python -m scrapy shell in order to test the shell, same would happen for pipenv run so making me have to use pipenv run python -m scrapy shell.
Non-working Scrapy from pipenv shell:

Working Scrapy from pipenv shell:

For the TL;DR:

python -m pipenv shell worked for me (had to not use "pipenv.exe" at all for command history in the shell)
Removing shell=True in pipenv.shell._handover alongside using:
pipenv run python -m [library] [args] worked for me for starting sub-shells provided by python libraries. (bypasses using [pipenv folder]\Scripts\[library].exe when launching)

This just worked for me and I'm not sure if this works for anybody else yet, so if anyone else could double check and test this on their Windows systems that would be much appreciated to know. (Just remember to edit using a program started with admin privileges for editing pipenv.shells._handover if you installed python under "C:\Program Files" or "C:\Program Files (x86)")

Given LanetheGreat's work, would it be possible to introduce an environment variable that would control whether or not the new shell was spawned with sys.exit(subprocess.call(args, shell=True, universal_newlines=True))
i.e. if the env var, say "PIPENV_WIN_SPAWN_WITH_SHELL" was anything other than false, use the above line, otherwise use sys.exit(subprocess.call(args, shell=False, universal_newlines=True))

python -m pipenv shell works for cmd, but doesn't seem to work in cmd in VS Code.

python manage.py shell works better than manage.py shell in cmd in VS Code, for what it's worth... the latter is simply unusable, while the former at least remembers the current session...

Up arrows working on my VS Code in Windows after  python -m pipenv shell

It still doesn't here. Tested in both Win 7 and Win 10.

Writing here to draw attention to the issue as it still not solved. Yes - I'm fasing the same problem. Looking forward for possible solution(s).

python -m pipenv shell works in pure cmd.exe, but doesn't seem to work in cmd in VS Code console (arrows still don't work).

it works for me, thank you

python -m pipenv shell works in pure cmd.exe, but doesn't seem to work in cmd in VS Code console (arrows still don't work).

it works for me, thank you

python -m pipenv shell works in pure cmd.exe, but doesn't seem to work in cmd in VS Code console (arrows still don't work).

worked like a charm. I will use this workaround for now.

Confirmed: it doesn't work in VS Code's built-in console but works on external consoles.
I just got it to work when using cmder with Powershell.

I have a same problem even with python -m pipenv shell or python -m pipenv run powershell.

Interestingly for me, I can't traverse history at all when running pipenv shell in VS Code's console, when using python -m pipenv shell in VS Code's console I can traverse history using up / down, however if I enter python from within this shell, my python console then doesn't let me traverse history.

Not sure if anyone mentioned already, but I tried opening another cmd from the pipenv shell using "start cmd.exe" and it seemed to preserve the environment while enabling up/down arrow history...

I see this issue got closed.
Is it finally fixed?

I just tried this on Win 10 from cmd.exe using pipenv 2022.5.2, doesn't seem to be fixed.
@matteius  - could you share which version it this was fixed/closed in?  It'd be great to start using that.

This stopped working for me even in Git bash and PowerShell.

No Python REPL history is also an issue on Windows 11 / Powershell / pipenv shell. A workaround is to activate the virtual environment created by pipenv using the Activate.ps1 script. Not very convenient but I guess okay if you only need REPL history occasionally but still want to stick with pipenv on Windows.

Doesn't work in Powershell or git bash.

I tested it on my Windows 10 Pro machine in cmd.exe with Python 3.13 and pipenv 2024.4.0 and the up/down history works fine.

python -m pipenv shell works for me for the command line, but I still lose up/down when inside a python console.
pipenv shell works for the python console, but not the command line.
Sad that we can't find a solution to fix both after so many years :(

@paketep, what do you mean by "python console"? Can you describe what is showing up after running pipenv shell? Can you run the where python and `where pipenv commands and post what is showing up? We need to verify that the same version of Python is being run for both.

Why is this thread closed? - They issue is still not resolved. Works with python -m pipenv shell on pure cmd though.

authors: HerbinZYH HowManyOliversAreThere LanetheGreat LucianoBAF MausamGaurav MeViMo PetterS W7PEA Xeizzeth airstrike asweigart bblanchon clragon crioux edonghyun erinxocon fireattack gregcowell jackdewinter joweenflores max-hoffmann-mck mikedrawback mungojam paketep rbrisita rlleshi rocketmonkeys snphan techalchemy uranusjr


text:
[BUG]  An explicitly excluded module is included in built package

setuptools version
67.7.2
Python version
Python 3.11.3
OS
macOS 13.3.1
Additional environment information
No response
Description
My pyproject.toml explicitly specifies a packgaes list for setuptools to include.
However, tests/test_utils.py is still included in the generated .tar.gz archive.
Also, adding the exclude key didn't help.
Expected behavior
Not including the tests/test_utils.py
How to Reproduce
git clone git@github.com:doronz88/pymobiledevice3.git -b refactor/python_package
cd pymobiledevice3
python3 -m pip install -U build
python3 -m build

Open and view the dist/*.tar.gz file.
Output
The tests/test_utils.py is included

Hi @doronz88, the packages configuration will determine which files are installed in the end users machine when they install a package. Consequently, this configuration influences contents of the generated wheel (.whl) file.
However .tar.gz files are a different beast all together... They are what we call source distribution (sdist), which is an intermediate artefact, meant to be platform independent, and work in a different level of optimization than wheel files.
Most of the times, it includes more files than the ones you see in the wheel. When you install a .tar.gz file with pip, pip will first "compile it" into a .whl file (which will not contain the tests/test_utils.py) and then install it. Before everyone started using git, it was a common practice to include all the files that are used to develop your package in the sdist as they serve as a "git checkout on steroids"[^1].
Sdists are still used when installing your package in a platform for which there are no wheels available and in some OS redistributions of Python packages (e.g. apt, dnf, pacman).
There is some level of agreement in the community that believe it is a good practice to include test files into the sdists, that is why setuptools include them by default.
When you install a .tar.gz file with pip, pip will first "compile it" into a .whl file (which will not contain the tests/test_utils.py) and then install it.
If you don't want to include files in the sdist, that is still possible, but the configuration that you need to change is the one in MANIFEST.in: https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html.
[^1]: They are more than a VCS checkout, as they include extra metadata generated by setuptools.

@abravalheri thanks for the amazingly fast and detailed response!
However, both when I remove the packages key and leave it as-is, I still get an unexpected result. The tests module remains, but the it doesn't include the inner modules: tests.cli & tests.services

Hi @doronz88, the default set of files automatically included in MANIFEST.in come from the original distutils implementation: https://github.com/pypa/distutils/blob/4435cec31b8eb5712aa8bf993bea3f07051c24d8/distutils/command/sdist.py#L233-L246, and it includes tests/test*.py and test/test*.py.
To do something different than the default, it is necessary to configure MANIFEST.in.
If you want all the files in git to be part of the sdist, you can also use setuptools-scm instead of configuring MANIFEST.in.

Thanks! I completely understand the rational and behavior now 😃

repo: pypa/setuptools

license: mit


text:
Redirect files &/or display action URLs from legacy

I see some users using legacy URLs like:

https://pypi.python.org/pypi?:action=files&name=limix&version=1.0.14 (on legacy: https://legacy.pypi.org/pypi?:action=files&name=limix&version=1.0.14 )
https://pypi.python.org/pypi?:action=display&name=craft-ai (on legacy: https://legacy.pypi.org/pypi?:action=display&name=craft-ai )
https://pypi.python.org/pypi?:action=display&name=text_progress_bar (on legacy: https://legacy.pypi.org/pypi?:action=display&name=text_progress_bar )

It'd be nice to redirect the display URLs and, potentially, the files URLs to Warehouse.
Followup to #3799.

Working on this.

authors: brainwane ewdurbin

repo: pypa/warehouse


text:
Add user registration functionality

New items:

Recaptcha service (conditionally functional, dependent on env vars)
forms.RegistrationForm
Reusable recaptcha template

Also added the responses lib as a dependency for mocking recaptcha tests
Features:

[x] reCaptcha service
[x] Reusable reCapatcha template
[x] forms.RegistrationForm
[x] views.register

Review items:

[x] Address SSL verification issue
[x] Expose reCaptcha keys through make targets and docker-compose
[x] Move from base requests.post to a thread local session implementation
[x] Handle timeouts & other lower level socket errors in recaptcha.py
[x] Security review on session handling in registration handler
[x] Bottom load recaptcha scripts
[x] Create a threadlocal requests session
[x] Configure sane timeouts for requests.Session instance

Some tests are broken, a few still need to be written, and some finishing touches have to be made yet, but I figured this would be as good a time as any to get some feedback on current progress.

This work is aimed to be partially implement #322 (email verification and Github-based registration still required) and #707 (captcha will still be required for login).

Overall looks pretty good, I made some comments on the code that I saw. As far as the TODO items, we shouldn't use a module scoped session. IIRC requests sessions are not thread safe (and in general, module scoped connection pools are not really great anyways). This should either be scoped to the service (so it'd be created in the service's __init__() method) or it should be created in the includeme() and stashed in the registry and passed into the service when it's created (see warehouse/db.py for an example). Another option is to register a HTTP Client service so that multiple different things can share the requests session (might not be worth it though).
In any case, since the sessions aren't compeltely threadsafe, any use of session that isn't scoped to a specific request will probably need a wrapper around it so that a single session is actually a session per thread.
/cc @sigmavirus24 @lukasa since they know requests behavior here better than me :)

Yeah, the point about moving to a module-scope instance really should have been "more generalized handling of sessions", but good to know about threading issues.
I'll follow the same pattern as warehouse/db.py for sessions. I'd considered an HTTP client service initially, but thought it would be a tad overkill.
Thanks for the review! Will get the comments as soon as I have time.

The thread-safety of Session objects is a tricky one. The things that put their thread-safety at risk are primarily the cookie dictionary and the (hateful) 301 cache (there are some problems with users manipulating Session properties without being careful about synchronising access, but I care less about that). If you aren't using either of those, you can actually force both of them to be no-ops and obtain the thread-safet you need.
However, a Session in thread-local storage is a really good idea. Having a Session live as long as possible is a good idea (the connection pooling works best that way). Thread-local storage can do a very good job of that.

Hmm, maybe requests-toolbelt should get a ThreadLocalSession.

@dstufft Not sure that would be all that useful: how would it work in a way that is more useful than just using the standard thread local storage logic in Python?

I have the same question as @Lukasa but wouldn't be opposed to it if we could figure out how it's an improvement.

Benefits:

It'd have the same interface as a requests Session allowing it to be used transparently anyplace (or most anyplace) something depends on a session. Which means the code doesn't have to be aware it's using a thread local.
Thread local storage doesn't have a creation function, so that means anytime you attempt to use the the thread's session, you have to first check to see if the thread even has a session created yet, and if not create it.

Essentially, I'd think it'd basically be:
import threading
import requests

class ThreadLocalSession:
    def __init__(self):
        self._threadlocal = threading.local()

    def _get_session(self):
        try:
            return self._threadlocal.session
        except AttributeError:
            self._threadlocal.session = requests.Session()

    def __getattr__(self, name):
        return getattr(self._get_session(), name)

You'd need something that would let you have settings on the Session though, so maybe the __init__() would accept keyword arguments that would get setattr()'d onto a new session whenever it was created. Another option would be, in the spirit of attempting to mimic the Session API, add a dict that stores things like ThreadLocalSession().verify = False and propagates that to all of the sessions somehow.
Dunno!

Another option would be, in the spirit of attempting to mimic the Session API, add a dict that stores things like ThreadLocalSession().verify = False and propagates that to all of the sessions somehow.

Perhaps we should discuss this in an issue over on the toolbelt, but I'm not sure there's a great way to do what you're asking for here.

@dstufft: This should be functionally complete now.

Added some comments about changes, this is also failing HTML linting... but one of the changes I mentioned should resolve that issue.

@dstufft: Review issues addressed and tests are passing.

authors: Lukasa demianbrecht dstufft sigmavirus24


text:
Push hierarchical attributes care about task parent changes

Issue

task parent changes are not handled with event handling with hierarchical value pushing from parent to task on attribute value change
it is confusing for users that value set on (e.g.) Shot is not propagated to value on created task which is annoyng especially if task attribute is used in expression

Changes

event handler also cares about change of task parent
in that case is pushing of values more robust and a little bit slower as have to query hierarchical attribute values all the time

How to test

Run event server with this branch
Make sure you have custom attribute available for Shot and Task and is hierarchical at the same time (frameStart or frameEnd)
Make sure settings for this event have proper values (ProjectSettings/Ftrack/Server Actions/Events/Sync Hierarchical and Entity Attributes)

event handler is enabled
entity type of interest contain Shot (used in example)
Attribute to sync contain frameStart

Create a Shot entity in ftrack
Change frameStart to 1001
The value should be propagated to the other one (hierarchical -> nonhierarchical or other way)
Create task under the Shot and check if value of frameStart will change to 1001 (on both hierarchical and non hierarchical attribute)

closes: https://github.com/pypeclub/OpenPype/issues/1748

OpenPype 2 PRs

pype
https://github.com/pypeclub/OpenPype/pull/1765

When changing the frameStart value, the event servers returns this error:

ERR: 2021-06-28 12:53:11,933 >>> { PushFrameValuesToTaskEvent }: [  Event "PushFrameValuesToTaskEvent" FAILED: 'PushFrameValuesToTaskEvent' object has no attribute 'interest_attributes'  ]
==============================
'PushFrameValuesToTaskEvent' object has no attribute 'interest_attributes'
==============================
Traceback (most recent call last):
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\modules\ftrack\lib\ftrack_event_handler.py", line 23, in wrapper_launch
    func(*args, **kwargs)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\modules\ftrack\event_handlers_server\event_push_frame_values_to_task.py", line 47, in launch
    self.process_by_project(session, event, project_id, entities_info)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\modules\ftrack\event_handlers_server\event_push_frame_values_to_task.py", line 127, in process_by_project
    interest_attributes = set(self.interest_attributes)
AttributeERROR: 'PushFrameValuesToTaskEvent' object has no attribute 'interest_attributes'
!!! ERR: 2021-06-28 12:53:17,826 >>> { PushFrameValuesToTaskEvent }: [  Event "PushFrameValuesToTaskEvent" FAILED: 'PushFrameValuesToTaskEvent' object has no attribute 'interest_attributes'  ]
==============================
'PushFrameValuesToTaskEvent' object has no attribute 'interest_attributes'
==============================
Traceback (most recent call last):
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\modules\ftrack\lib\ftrack_event_handler.py", line 23, in wrapper_launch
    func(*args, **kwargs)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\modules\ftrack\event_handlers_server\event_push_frame_values_to_task.py", line 47, in launch
    self.process_by_project(session, event, project_id, entities_info)
  File "C:\Users\qam\pypeclub\pype3\pype\openpype\modules\ftrack\event_handlers_server\event_push_frame_values_to_task.py", line 127, in process_by_project
    interest_attributes = set(self.interest_attributes)
AttributeERROR: 'PushFrameValuesToTaskEvent' object has no attribute 'interest_attributes'

When changing the frameStart value, the event servers returns this error:

Should be fixed now

repo: pypeclub/OpenPype


text:
Ftrack: Action to easily create daily review session

Brief description
Added action to create daily review session with ability to auto-create dailies each day.
Description
Action creates review session with name based on template defined in settings. If there is review session with same name then rest of logic is skipped. If there is not review session with same name and then is review session from "previous day" which does not contain any reviewables then that is just renamed otherwise new review session object is created. Action does daily cycles during which checks if project has enabled "daily creation" of review sessions and create them automatically for all enabled so the action does not have to be executed manually.
Additional info
If template does not contain day formatting key it probably won't be "daily" so it can be used for e.g. weeklies too. Default template is {yy}{mm}{dd} (2 digits year, 2 digits month and 2 digits day).
Testing notes:

Run the action on any project
Enable cycle creation on a project and wait until next day

here is what I can see in Project actions

It's server action, not local.