text:
Bug located in Modeler2D: object has no attribute 'components_3d_names' (in duplicate_and_mirror)

Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

Description of the bug
when using Modeler2D.duplicate_and_mirror() I have the following error.
I guess it's because of the changes #1385 where
def components_3d_names(self) was renamed into def user_defined_component_names(self) in Primitives.py
but not in line 2431 of Modeler.py in def duplicate_and_mirror()
if is_3d_comp:
orig_3d = [i for i in self.components_3d_names]
by the way, comparing files seems to show that the change seems to be done for def duplicate_around_axis()

pyaedt error: Method duplicate_and_mirror Failed:  Attribute Error. Please Check again
Method duplicate_and_mirror Failed:  Attribute Error. Please Check again
pyaedt error: 'Modeler2D' object has no attribute '**components_3d_names**'
'Modeler2D' object has no attribute 'components_3d_names'
pyaedt error: Arguments Provided:
Arguments Provided:
pyaedt error:     objid = [<pyaedt.modeler.Primitives.Polyline object at 0x000002193745E648>, <pyaedt.modeler.Primitives.Polyline object at 0x000002193745E208>]
    objid = [<pyaedt.modeler.Primitives.Polyline object at 0x000002193745E648>, <pyaedt.modeler.Primitives.Polyline object at 0x000002193745E208>]
pyaedt error:     position = [0, 0, 0]
    position = [0, 0, 0]
pyaedt error:     vector = ['cos((360deg/SymmetryFactor/2)+90deg)', 'sin((360deg/SymmetryFactor/2)+90deg)', 0]
    vector = ['cos((360deg/SymmetryFactor/2)+90deg)', 'sin((360deg/SymmetryFactor/2)+90deg)', 0]
pyaedt error:     is_3d_comp = True
    is_3d_comp = True
pyaedt error:     duplicate_assignment = None
    duplicate_assignment = None
pyaedt error: Error in :   File "C:\Users\Sharath\Desktop\Solution-Applications\solutions-app-webapp-poc\venv\lib\site-packages\pyaedt\modeler\Modeler.py", line 2431, in duplicate_and_mirror
Error in :   File "C:\Users\Sharath\Desktop\Solution-Applications\solutions-app-webapp-poc\venv\lib\site-packages\pyaedt\modeler\Modeler.py", line 2431, in duplicate_and_mirror
pyaedt error: Check Online documentation on: https://aedtdocs.pyansys.com/search.html?q=duplicate_and_mirror
Check Online documentation on: https://aedtdocs.pyansys.com/search.html?q=duplicate_and_mirror
pyaedt error: Method duplicate_and_mirror Failed:  Attribute Error. Please Check again
Method duplicate_and_mirror Failed:  Attribute Error. Please Check again
pyaedt error: 'Modeler2D' object has no attribute 'components_3d_names'
'Modeler2D' object has no attribute 'components_3d_names'
pyaedt error: Arguments Provided:
Arguments Provided:
pyaedt error:     objid = [<pyaedt.modeler.Primitives.Polyline object at 0x000002193A852CC8>, <pyaedt.modeler.Primitives.Polyline object at 0x0000021939813708>]
    objid = [<pyaedt.modeler.Primitives.Polyline object at 0x000002193A852CC8>, <pyaedt.modeler.Primitives.Polyline object at 0x0000021939813708>]
pyaedt error:     position = [0, 0, 0]
    position = [0, 0, 0]
pyaedt error:     vector = ['cos((360deg/SymmetryFactor/2)+90deg)', 'sin((360deg/SymmetryFactor/2)+90deg)', 0]
    vector = ['cos((360deg/SymmetryFactor/2)+90deg)', 'sin((360deg/SymmetryFactor/2)+90deg)', 0]
pyaedt error:     is_3d_comp = True
    is_3d_comp = True
pyaedt error:     duplicate_assignment = None
    duplicate_assignment = None
pyaedt error: Error in :   File "C:\Users\Sharath\Desktop\Solution-Applications\solutions-app-webapp-poc\venv\lib\site-packages\pyaedt\modeler\Modeler.py", line 2431, in duplicate_and_mirror
Error in :   File "C:\Users\Sharath\Desktop\Solution-Applications\solutions-app-webapp-poc\venv\lib\site-packages\pyaedt\modeler\Modeler.py", line 2431, in duplicate_and_mirror
pyaedt error: Check Online documentation on: https://aedtdocs.pyansys.com/search.html?q=duplicate_and_mirror
Check Online documentation on: https://aedtdocs.pyansys.com/search.html?q=duplicate_and_mirror
2022-07-27 15:39:43.135 (  39.582s) [                ]vtkWin32OpenGLRenderWin:680    ERR| vtkWin32OpenGLRenderWindow (0000021933393060): failed to get wglChoosePixelFormatARB
ERROR:root:failed to get wglChoosePixelFormatARB
2022-07-27 15:39:43.137 (  39.583s) [                ]vtkWin32OpenGLRenderWin:758    ERR| vtkWin32OpenGLRenderWindow (0000021933393060): failed to get valid pixel format.
ERROR:root:failed to get valid pixel format.
2022-07-27 15:39:43.137 (  39.584s) [                ]vtkOpenGLRenderWindow.c:493    ERR| vtkWin32OpenGLRenderWindow (0000021933393060): GLEW could not be initialized: Missing GL version
ERROR:root:GLEW could not be initialized: Missing GL version

### Steps To Reproduce

You can run the code here: https://aedtdocs.pyansys.com/examples/02-Maxwell/Maxwell2D_NissanLeaf.html#sphx-glr-examples-02-maxwell-maxwell2d-nissanleaf-py

### Which Operating System are you using?

Windows

### Which Python version are you using?

3.9

### Installed packages

appdirs==1.4.4
async-timeout==4.0.2
attrs==21.4.0
Babel==2.10.3
backcall==0.2.0
certifi==2022.6.15
charset-normalizer==2.0.12
colorama==0.4.5
cycler==0.11.0
debugpy==1.6.0
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.15.3
fonttools==4.33.3
frozenlist==1.3.0
idna==3.3
ipython-genutils==0.2.0
json5==0.9.8
jupyterlab-pygments==0.2.2
kiwisolver==1.4.3
MarkupSafe==2.1.1
mistune==0.8.4
multidict==6.0.2
nest-asyncio==1.5.5
numpy==1.23.0
pandocfilters==1.5.0
parso==0.8.3
pickleshare==0.7.5
Pillow==9.1.1
plumbum==1.7.2
prometheus-client==0.14.1
prompt-toolkit==3.0.29
psutil==5.9.1
pure-eval==0.2.2
pycparser==2.21
Pygments==2.12.0
pyparsing==3.0.9
pyrsistent==0.18.1
pytz==2022.1
pywin32==303
pywinpty==2.0.5
pyzmq==23.2.0
requests==2.28.0
rpyc==5.0.1
scooby==0.5.12
Send2Trash==1.8.0
six==1.16.0
sniffio==1.2.0
soupsieve==2.3.2.post1
terminado==0.15.0
tinycss2==1.1.1
tornado==6.1
traitlets==5.3.0
urllib3==1.26.9
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.3
yarl==1.7.2
zipp==3.8.0

Hi @romainlejeune  I've fixed it

Hi @maxcapodi78 , thanks a lot!

repo: pyansys/pyaedt


text:
create_rlc_component() not working for rotated components

Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

Description of the bug
For rotated components the create_rlc_component does not work correctly.
The rotation is not copied from the host component and the pads are placed on the wrong location.
Changing the SetPosition/GetPosition to SetTransform/GetTransform fixes this for my (mentor expedition ODB++) designs.
Steps To Reproduce
Read ODB++ file with rotated components (For example soic 4).
creata rlc on pins 2 and 3.
Check the EDB file with SiWave
Which Operating System are you using?
Windows
Which Python version are you using?
3.8
Installed packages
aiohttp==3.8.3
aiosignal==1.2.0
appdirs==1.4.4
async-timeout==4.0.2
attrs==22.1.0
autopep8==1.7.0
cffi==1.15.1
charset-normalizer==2.1.1
clr-loader==0.2.3
contourpy==1.0.5
cycler==0.11.0
fonttools==4.37.3
frozenlist==1.3.1
idna==3.4
imageio==2.22.0
kiwisolver==1.4.4
matplotlib==3.6.0
multidict==6.0.2
numpy==1.23.3
packaging==21.3
pandas==1.5.0
Pillow==9.2.0
pip-review==1.2.0
plumbum==1.7.2
psutil==5.9.2
pyaedt==0.6.5
pycodestyle==2.9.1
pycparser==2.21
pyparsing==3.0.9
python-dateutil==2.8.2
pythonnet==3.0.0rc6
pytz==2022.2.1
pyvista==0.36.1
pywin32==304
rpyc==5.0.1
scooby==0.6.0
six==1.16.0
toml==0.10.2
vtk==9.1.0
wslink==1.8.2
yarl==1.8.1

fixed currently merged in the main branch.

source: gharchive/issue

authors: ErikZalm svandenb-dev

license: MIT

license_type: permissive

license_source: github-api


text:
Implement ARM Test to verify workflows using the unified installer

Add ARM test which installs the public package and runs a failure analysis with the dpf server of the unified installer

[ ] test standalone installer (local server)

Use model in POST:Core/POST_COMPOSITES_FAILURE/POST_COMPOSITES_FAILURE_010 to test assembly
Test these features

[ ] Test failure plot: for the entire model or just a sub-part of the assembly

[ ] Test sampling point: test on different sub-part of the assembly

Created task 803214 in TFS

@janvonrickenbach , task 803214 in TFS is closed. Can we close this issue as well? René

A test has been added for the inprocess server. We should also create a test for starting the grpc server.

authors: janvonrickenbach roosre

repo: pyansys/pydpf-composites


text:
Bug located in nodal_solution reading

🔍 Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

🐞 Description of the bug
When using pymapdl-reader to read nodal solutions, the extracted values are mostly 0 with a few extreme maximum or minimum values, making it impossible to read normally.
📝 Steps to reproduce
Using Ansys 2023R1 workbench Static Analysis module.
The following code can be used to reproduce the issue：

from ansys.mapdl import reader as pymapdl_reader
rst = pymapdl_reader.read_binary(rstfile)
DISP = rst.nodal_solution(0)
Usis = DISP[1][:]

the values of "Usis" are mostly 0 with a few extreme maximum or minimum values, making it impossible to read normally.
!!!Note that: A project created solely using the Ansys 2023R1 Workbench static analysis module may encounter this issue, but projects saved using the Ansys 2021R2 Workbench static analysis module can be opened in Ansys 2023R1 Workbench and re-saved without encountering this issue. I suspect this to be a compatibility problem with Ansys 2023R1.
💻 Which operating system are you using?
Windows
📀 Which ANSYS version are you using?
ANSYS 2023R1
🐍 Which Python version are you using?
3.8
📦 Installed packages
absl-py==1.4.0
ansys-dpf-core==0.8.0
ansys-dpf-gate==0.3.1
ansys-dpf-gatebin==0.3.1
ansys-dpf-post==0.4.0
ansys-grpc-dpf==0.7.1
ansys-mapdl-reader==0.52.11
appdirs==1.4.4
astunparse==1.6.3
Bottleneck==1.3.5
cachetools==5.3.0
certifi==2022.12.7
charset-normalizer==3.1.0
colorama==0.4.6
contourpy==1.0.7
cycler==0.11.0
et-xmlfile==1.1.0
flatbuffers==2.0.7
fonttools==4.39.3
gast==0.4.0
google-api-core==2.11.0
google-api-python-client==2.83.0
google-auth==2.17.2
google-auth-httplib2==0.1.0
google-auth-oauthlib==1.0.0
google-pasta==0.2.0
googleapis-common-protos==1.59.0
grpcio==1.53.0
h5py==3.8.0
httplib2==0.22.0
idna==3.4
imageio==2.27.0
importlib-metadata==6.1.0
importlib-resources==5.12.0
joblib==1.2.0
keras==2.7.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.4
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.2
matplotlib==3.7.1
mkl-fft==1.3.1
mkl-random==1.2.2
mkl-service==2.4.0
numexpr==2.8.4
numpy==1.24.2
oauthlib==3.2.2
openpyxl==3.1.2
opt-einsum==3.3.0
packaging==23.0
pandas==1.5.3
Pillow==9.5.0
pip==23.0.1
platformdirs==3.2.0
pooch==1.7.0
protobuf==4.22.1
psutil==5.9.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.9
python-dateutil==2.8.2
pytz==2022.7
pyvista==0.38.5
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn==1.2.2
scipy==1.10.1
scooby==0.7.1
setuptools==67.6.1
six==1.16.0
tensorboard==2.12.1
tensorboard-data-server==0.7.0
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.7.0
tensorflow-gpu==2.7.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.2.0
threadpoolctl==3.1.0
tqdm==4.65.0
typing_extensions==4.5.0
uritemplate==4.1.1
urllib3==1.26.15
vtk==9.2.6
Werkzeug==2.2.3
wheel==0.40.0
wincertstore==0.2
wrapt==1.15.0
zipp==3.15.0

This is another example where PyMAPDL Reader won't be able to keep up with MAPDL. There are compression changes occurring "under the hood" that are difficult to incorporate within this reader.
I'd strongly recommend transitioning to PyDPF-Core. If you absolutely need support for this, email me at alexander.kaszynski@ansys.com and I can work something out with the powers that be.

This is another example where PyMAPDL Reader won't be able to keep up with MAPDL. There are compression changes occurring "under the hood" that are difficult to incorporate within this reader.
I'd strongly recommend transitioning to PyDPF-Core. If you absolutely need support for this, email me at alexander.kaszynski@ansys.com and I can work something out with the powers that be.

Because I want to extract the X-component, Y-component, Z-component of displacement and X-component, Y-component, Z-component of rotation for static mechanical simulation, I can only use pymapdl-reader. PyDPF-Core and PyDPF-Post cannot meet my requirements as they can only extract nodal solutions, including the X-component, Y-component, Z-component of displacement.

Please upload a minimum working example result file here as a zip archive and I'll do the best I can to support it. I will also email the developers of DPF and have these features added for the next release.
CC @PProfizi @rlagha

Please upload a minimum working example result file here as a zip archive and I'll do the best I can to support it. I will also email the developers of DPF and have these features added for the next release.
CC @PProfizi @rlagha

Thank you so much for your help. Enclosed is a minimum working example result file that I have compressed into a zip archive. Kindly do your best to address this issue.
nodal_solution.zip

Hello @Tony-ooo ,
please find below an script showing how to extract rotational displacement values from the previously shared result files with PyDPF-Core.
Does this answer your need?
I will be adding this feature to the PyDPF-Post library as soon as possible.
import ansys.dpf.core as dpf
import numpy as np

rstfile = r'D:/nodal_solution/file.rst'
datfile = r'D:/nodal_solution/ds.dat'

substep = 4

# create the DataSource
ds = dpf.DataSources()
ds.set_result_file_path(rstfile, key="rst")
ds.add_file_path(datfile, key="dat")

# instantiate a Model
model = dpf.Model(data_sources=ds)
print(model)

# define a time_scoping on substep 4
time_scoping = dpf.time_freq_scoping_factory.scoping_by_step_and_substep_from_model(
    load_step_id=1,
    subset_id=substep,
    model=model
)

# extract the raw displacements
displacement_fc = dpf.operators.result.raw_displacement(
    time_scoping=time_scoping,
    data_sources=model,
).eval()
print(displacement_fc)

# extract the field from the fields_container
disp_field = displacement_fc[0]
print(disp_field)

# one can access the field's underlying numpy.ndarray using ".data"
print(disp_field.data)

# save to csv
np.savetxt(
    'displacement2_pydpf-core.csv',
    X=disp_field.data,
    delimiter=',',  comments='', fmt='%.18e',
    header="UX, UY, UZ, RX, RY, RZ"
)

# optional: extract the rotational dofs from the field
rotational_disp = dpf.operators.logic.component_selector(
    field=disp_field,
    component_number=[3, 4, 5]
).eval()
print(rotational_disp)

@PProfizi , please could you help @Tony-ooo , to get this use case working with pyDPF core and ensure that this is addressed with the new dpf-post API?

I have tested the new DPF-POST API "post.load_simulation" in ANSYS 2023R1, and it is functioning correctly. However, I have noticed that unlike pymapdl-reader, the new API is unable to extract the X, Y, and Z components of rotation from the rst file. It can only extract the X, Y, and Z components of displacement, which does not meet my requirements. Could you add the functionality of extracting the X, Y, and Z components of rotation in the new API? I am in urgent need of this feature.

Hi @Tony-ooo,
The reason you do not get identical field data for translational displacements between PyDPF-Core and PyDPF-Post is that when requesting them in PyDPF-Core, there is no specification of node IDs to request data for, and PyDPF-Core returns data as it is ordered server-side, which is not by ascending node ID.
When requesting the displacement data with PyDPF-Post however, you do indicate node_ids=nnum, which tells PyDPF-Post to request data for those specific nodes IDs in that specific order.
To obtain the same results, and as I guess you may want them ordered by ID for your .csv file to make sense, what you can do is modify the PyDPF-Core request as below:
nnum = dat.nnum
node_scoping = dpf.mesh_scoping_factory.nodal_scoping(node_ids=nnum)

# extract the raw displacements
displacement_fc = dpf.operators.result.raw_displacement(
    time_scoping=time_scoping,
    mesh_scoping=node_scoping,
    data_sources=model,
).eval()
disp_field = displacement_fc[0]

This will result in disp_field having data ordered by node ID, and your .csv being identical for translational displacement.

Thank you very much for your assistance. You solved my problem perfectly. I'm looking forward to the addition of this feature in the PyDPF-Post API.

Thanks @PProfizi for your rapid follow up here.

authors: PProfizi Tony-ooo akaszynski

repo: pyansys/pymapdl-reader


text:
added ctime() code in time.java

Implementing ctime() from time.py(standard library) in time.java .

ping ping @eliasdorneles  :-p

Thanks @prithvisingh18, merging !! :tada:

source: gharchive/pull-request

authors: eliasdorneles goanpeca prithvisingh18

repo: pybee/voc


text:
英文文本需要潤飾

看了一兩頁，有些英文⋯⋯不太像英文，可能要找人修一下。

嗯嗯，當初 event 下的幾個頁面請一些人幫忙看過，但看起來品質還是不夠。
TP 看到是那幾頁？我覺得不是很好的頁面像是總覽 (Overview), 教學 (Tutorial)。

keynote 那頁目前先簡單介紹，要再與講者確認介紹內容。

同意 overview 不太好，我沒仔細看 tutorial 但好像內容和 talk 那邊不太一致？

authors: chairco uranusjr

license: mit

license_source: bigquery


text:
Switch template to calendar versioning

Description
The template is currently versioned after the version of Django it uses:
https://github.com/pydanny/cookiecutter-django/blob/b48067c03494993933bd2eb0aa2e5afdf367f12f/setup.py#L11-L13
I suggest changing this to calendar versioning instead e.g. 2021.1.24
Rationale
With Github actions, we've been able to automate a few things around the project management:

Drafting Github releases thanks to release drafter

Generating Changelog with a custom script that we run every night.

However, this approach has a few issues:

Changelog file and release changes are not consistent, they 2 different views of the same thing.
The Github release doesn't tell much about when changes were actually usable, one typically uses the latest in master when calling the cookiecutter CLI.
The changelog script might need some maintenance down the road, would be nicer to rely on a community supported Github action.
Changing the version template is not automated yet, and we often miss the step when the version of Django is updated.

Solution
Ideally we would use off the shelf action(s) like:

https://github.com/fregante/daily-version-action
https://github.com/notlmn/release-with-changelog

These are providing some building pieces. Ideally, we also need one to keep the changelog file up to date.
Would be nice to find an all in one solution, an equivalent of https://github.com/relekang/python-semantic-release, but for calendar base releases.
This process would run like the current update changelog script, every night on schedule.

Hm is it possible to simply parse the requirements file?

is it possible to simply parse the requirements file?

Not sure I understand the question... What file are you talking about?

So in the actual cookiecutter, find requirements/base.txt, parse it and find the version Django is currently using. This way, setup.py's version for this project itself is automated.

import pathlib
import pkg_resources

try:
    with pathlib.Path("{{ cookiecutter.project_slug }} / requirements / base.txt").open() as requirements_txt:
        pkg = [x for x in pkg_resources.parse_requirements(requirements_txt) if x.name == "django"][0]
except IndexError:
    quit(1)

VERSION = pkg.specs[1]

On the topic itself, I like the calendar versioning. I think a calendar versioning method is fine too considering the changes that go into this package are not actually that much related to the version of Django. For the Django versioning method, it's ok too since it acts like a barrier for people to know "hey we don't support this yet so don't try...".

Ah yes, we could indeed parse requirements if the only point was to remove the version from setup.py. But you got my main point, which is that the version of the template being tied to the version of Django isn't very meaningful IMO. It might not be bumped for months while the template could be going through significant changes.

Right. So maybe using datetime to get the current UTC date, then release at the end of greenwich time for a day in case a PR was merged? That'd make the most sense to me.

Yes, pretty much. That's been working pretty well for generating our CHANGELOG.md. We do so every day at 2 AM (UTC):
https://github.com/pydanny/cookiecutter-django/blob/34ad90bdd83a9e771c3cca801b00b02ad6b8991d/.github/workflows/update-changelog.yml#L4-L6
The script we run iterate through the pull requests from the day before:
https://github.com/pydanny/cookiecutter-django/blob/34ad90bdd83a9e771c3cca801b00b02ad6b8991d/scripts/update_changelog.py#L11-L12
In case it fails for an external reason and needs to retried (that happened once AFAIR), we can trigger it manually, and we have the whole day to do so:
https://github.com/pydanny/cookiecutter-django/blob/34ad90bdd83a9e771c3cca801b00b02ad6b8991d/.github/workflows/update-changelog.yml#L7-L8

authors: Andrew-Chen-Wang browniebroke

repo: pydanny/cookiecutter-django


text:
Recursive model with discriminated union has exponential time and space complexity for validation

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
When validating the recursive model given in the example, Pydantic has exponential time and space complexity, meaning that validation time and memory usage may grow indefinitely and rapidly depending on model depth, even when using a discriminator field.
Given the presence of a discriminator field, this should not be the case.
A model like this can potentially be abused in a denial-of-service attack, if the data passed to the validator can be controlled by the user.
Example Code
from __future__ import annotations
from typing import Literal, Annotated
from pydantic import Field, TypeAdapter, BaseModel

class NestedState(BaseModel):
    state_type: Literal["nested"]
    substate: AnyState

# If this type is left out, the model behaves normally again
class LoopState(BaseModel):
    state_type: Literal["loop"]
    substate: AnyState

class LeafState(BaseModel):
    state_type: Literal["leaf"]

AnyState = Annotated[NestedState | LoopState | LeafState, Field(..., discriminator="state_type")]

def build_nested_state(n):
    if n <= 0:
        return {"state_type": "leaf"}
    else:
        return {"state_type": "loop", "substate": {"state_type": "nested", "substate": build_nested_state(n-1)}}

adapter = TypeAdapter(AnyState)

# the next statement takes around 0.8s
adapter.validate_python(build_nested_state(9))

# the next statement takes around 3.5s
adapter.validate_python(build_nested_state(10))

# the next statement takes around 12.5s
adapter.validate_python(build_nested_state(11))

Python, Pydantic & OS Version
pydantic version: 2.6.0
        pydantic-core version: 2.16.1
          pydantic-core build: profile=release pgo=true
                 install path: /<redacted>/venv-3.11/lib/python3.11/site-packages/pydantic
               python version: 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]
                     platform: macOS-14.3-arm64-arm-64bit
             related packages: fastapi-0.105.0 typing_extensions-4.8.0 pydantic-settings-2.1.0
                       commit: unknown

@palle-k,
Thanks for reporting this. @davidhewitt, could you please take a look at this performance issue when you have a chance? Thank you!

It looks like there's a potential schema-building bug here.
The core schema looks like this:
{'definitions': [{'cls': <class '__main__.NestedState'>,
                  'config': {'title': 'NestedState'},
                  'custom_init': False,
                  'metadata': {'pydantic.internal.needs_apply_discriminated_union': False,
                               'pydantic_js_annotation_functions': [],
                               'pydantic_js_functions': [functools.partial(<function modify_model_json_schema at 0x7fba5440fce0>, cls=<class '__main__.NestedState'>),
                                                         <bound method BaseModel.__get_pydantic_json_schema__ of <class '__main__.NestedState'>>]},
                  'ref': '__main__.NestedState:94515929304032',
                  'root_model': False,
                  'schema': {'computed_fields': [],
                             'fields': {'state_type': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func at 0x7fba54486fc0>],
                                                                    'pydantic_js_functions': []},
                                                       'schema': {'expected': ['nested'],
                                                                  'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                  'type': 'literal'},
                                                       'type': 'model-field'},
                                        'substate': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func at 0x7fba54487100>],
                                                                  'pydantic_js_functions': []},
                                                     'schema': {'default': Ellipsis,
                                                                'schema': {'choices': [{'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                                'schema_ref': '__main__.NestedState:94515929304032',
                                                                                'type': 'definition-ref'},
                                                                                {'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                                'schema_ref': '__main__.LoopState:94515929369184',
                                                                                'type': 'definition-ref'},
                                                                                {'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                                'schema_ref': '__main__.LeafState:94515929326848',
                                                                                'type': 'definition-ref'}],
                                                                           'metadata': {'pydantic.internal.needs_apply_discriminated_union': True,
                                                                                'pydantic.internal.union_discriminator': 'state_type'},
                                                                           'type': 'union'},
                                                                'type': 'default'},
                                                     'type': 'model-field'}},
                             'model_name': 'NestedState',
                             'type': 'model-fields'},
                  'type': 'model'},
                 {'cls': <class '__main__.LoopState'>,
                  'config': {'title': 'LoopState'},
                  'custom_init': False,
                  'metadata': {'pydantic.internal.needs_apply_discriminated_union': False,
                               'pydantic_js_annotation_functions': [],
                               'pydantic_js_functions': [functools.partial(<function modify_model_json_schema at 0x7fba5440fce0>, cls=<class '__main__.LoopState'>),
                                                         <bound method BaseModel.__get_pydantic_json_schema__ of <class '__main__.LoopState'>>]},
                  'ref': '__main__.LoopState:94515929369184',
                  'root_model': False,
                  'schema': {'computed_fields': [],
                             'fields': {'state_type': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func at 0x7fba544871a0>],
                                                                    'pydantic_js_functions': []},
                                                       'schema': {'expected': ['loop'],
                                                                  'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                  'type': 'literal'},
                                                       'type': 'model-field'},
                                        'substate': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func at 0x7fba54487380>],
                                                                  'pydantic_js_functions': []},
                                                     'schema': {'default': Ellipsis,
                                                                'schema': {'choices': [{'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                                'schema_ref': '__main__.NestedState:94515929304032',
                                                                                'type': 'definition-ref'},
                                                                                {'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                                'schema_ref': '__main__.LoopState:94515929369184',
                                                                                'type': 'definition-ref'},
                                                                                {'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                                'schema_ref': '__main__.LeafState:94515929326848',
                                                                                'type': 'definition-ref'}],
                                                                           'metadata': {'pydantic.internal.needs_apply_discriminated_union': True,
                                                                                'pydantic.internal.union_discriminator': 'state_type'},
                                                                           'type': 'union'},
                                                                'type': 'default'},
                                                     'type': 'model-field'}},
                             'model_name': 'LoopState',
                             'type': 'model-fields'},
                  'type': 'model'},
                 {'cls': <class '__main__.LeafState'>,
                  'config': {'title': 'LeafState'},
                  'custom_init': False,
                  'metadata': {'pydantic.internal.needs_apply_discriminated_union': False,
                               'pydantic_js_annotation_functions': [],
                               'pydantic_js_functions': [functools.partial(<function modify_model_json_schema at 0x7fba5440fce0>, cls=<class '__main__.LeafState'>),
                                                         <bound method BaseModel.__get_pydantic_json_schema__ of <class '__main__.LeafState'>>]},
                  'ref': '__main__.LeafState:94515929326848',
                  'root_model': False,
                  'schema': {'computed_fields': [],
                             'fields': {'state_type': {'metadata': {'pydantic_js_annotation_functions': [<function get_json_schema_update_func.<locals>.json_schema_update_func at 0x7fba54484fe0>],
                                                                    'pydantic_js_functions': []},
                                                       'schema': {'expected': ['leaf'],
                                                                  'metadata': {'pydantic.internal.needs_apply_discriminated_union': False},
                                                                  'type': 'literal'},
                                                       'type': 'model-field'}},
                             'model_name': 'LeafState',
                             'type': 'model-fields'},
                  'type': 'model'}],
 'schema': {'choices': {'leaf': {'schema_ref': '__main__.LeafState:94515929326848',
                                 'type': 'definition-ref'},
                        'loop': {'schema_ref': '__main__.LoopState:94515929369184',
                                 'type': 'definition-ref'},
                        'nested': {'schema_ref': '__main__.NestedState:94515929304032',
                                   'type': 'definition-ref'}},
            'discriminator': 'state_type',
            'from_attributes': True,
            'metadata': {'pydantic.internal.needs_apply_discriminated_union': True,
                         'pydantic.internal.union_discriminator': 'state_type'},
            'strict': False,
            'type': 'tagged-union'},
 'type': 'definitions'}

In particular the interesting bit to me is that we have 3 definitions instead of 4; the AnyState union is inlined in each model. But in the inlined model copies it is 'type': 'union' instead of 'type': 'tagged-union', which would explain where there's linear search rather than tagged dispatch at each level.
Hence the exponential blowup.
cc @adriangb as the schema building expert ;)

The schema simplifies to:
{
    "type": "definitions",
    "schema": {
        "type": "tagged-union",
        "choices": {
            "nested": {
                "type": "definition-ref",
                "schema_ref": "__main__.NestedState:4587657680"
            },
            "loop": {
                "type": "definition-ref",
                "schema_ref": "__main__.LoopState:4587670848"
            },
            "leaf": {
                "type": "definition-ref",
                "schema_ref": "__main__.LeafState:5124597648"
            }
        },
        "discriminator": "state_type"
    },
    "definitions": [
        {
            "type": "model",
            "cls": "__main__.LeafState",
            "schema": {
                "type": "model-fields",
                "fields": {
                    "state_type": {
                        "type": "model-field",
                        "schema": {
                            "type": "literal",
                            "expected": [
                                "leaf"
                            ]
                        }
                    }
                },
                "model_name": "LeafState"
            },
            "ref": "__main__.LeafState:5124597648"
        },
        {
            "type": "model",
            "cls": "__main__.LoopState",
            "schema": {
                "type": "model-fields",
                "fields": {
                    "state_type": {
                        "type": "model-field",
                        "schema": {
                            "type": "literal",
                            "expected": [
                                "loop"
                            ]
                        }
                    },
                    "substate": {
                        "type": "model-field",
                        "schema": {
                            "type": "default",
                            "schema": {
                                "type": "union",
                                "choices": [
                                    {
                                        "type": "definition-ref",
                                        "schema_ref": "__main__.NestedState:4587657680"
                                    },
                                    {
                                        "type": "definition-ref",
                                        "schema_ref": "__main__.LoopState:4587670848"
                                    },
                                    {
                                        "type": "definition-ref",
                                        "schema_ref": "__main__.LeafState:5124597648"
                                    }
                                ]
                            }
                        }
                    }
                }
            },
            "ref": "__main__.LoopState:4587670848"
        },
        {
            "type": "model",
            "cls": "__main__.NestedState",
            "schema": {
                "type": "model-fields",
                "fields": {
                    "state_type": {
                        "type": "model-field",
                        "schema": {
                            "type": "literal",
                            "expected": [
                                "nested"
                            ]
                        }
                    },
                    "substate": {
                        "type": "model-field",
                        "schema": {
                            "type": "default",
                            "schema": {
                                "type": "union",
                                "choices": [
                                    {
                                        "type": "definition-ref",
                                        "schema_ref": "__main__.NestedState:4587657680"
                                    },
                                    {
                                        "type": "definition-ref",
                                        "schema_ref": "__main__.LoopState:4587670848"
                                    },
                                    {
                                        "type": "definition-ref",
                                        "schema_ref": "__main__.LeafState:5124597648"
                                    }
                                ]
                            }
                        }
                    }
                }
            },
            "ref": "__main__.NestedState:4587657680"
        }
    ]
}

The issue is that the tagged union is not carried along so it's loss after the first union and thus performance degrades.
So yes it's a schema generation bug.
@sydney-runkle is that enough to go off of? I'm guessing we're not doing the apply_discrimnators stuff correctly.

Hmm, this looks similar to this issue: https://github.com/pydantic/pydantic/issues/8688, which we fixed here https://github.com/pydantic/pydantic/pull/8702 for the 2.6.1 release. @palle-k, could you please confirm whether or not this issue has been resolved with 2.6.1?

@davidhewitt, ah rats, ok 🐀.

+1 this seems pretty critical 😬, is there any workaround?

This is a regression because I don't see this issue with
pydantic==2.4.2
pydantic_core==2.10.1

@rossmacarthur,
Thanks for the +1. I've added to the list of issues we'll attempt to fix for 2.6.2. Hopefully we can get that release out next week. I'll keep you updated as we progress. I don't currently know of a workaround for this performance issue.

Here's a somewhat clumsy workaround:
from __future__ import annotations
from typing import Any, Literal, Annotated
from pydantic import AfterValidator, Field, TypeAdapter, BaseModel
import time

def validate_any_state(value: Any):
    return _ADAPTER.validate_python(value)

AnyState = Annotated[Any, AfterValidator(validate_any_state)]

class NestedState(BaseModel):
    state_type: Literal["nested"]
    substate: AnyState

# If this type is left out, the model behaves normally again
class LoopState(BaseModel):
    state_type: Literal["loop"]
    substate: AnyState

class LeafState(BaseModel):
    state_type: Literal["leaf"]

_ADAPTER = TypeAdapter(Annotated[NestedState | LoopState | LeafState, Field(..., discriminator="state_type")])

def build_nested_state(n):
    if n <= 0:
        return {"state_type": "leaf"}
    else:
        return {"state_type": "loop", "substate": {"state_type": "nested", "substate": build_nested_state(n-1)}}

for i in range(13):
    start = time.time()
    _ADAPTER.validate_python(build_nested_state(i))
    print(f"Time for {i}: {time.time() - start}")

It works by cutting the recursive model apart a bit crudely with the AfterValidator. This has downsides; it's opaque to pydantic-core so will affect the way that things like model_config are propagated and usually an AfterValidator is not as efficient as builtin design, if we can support it. In this case while we've got the performance bug the AfterValidator performs significantly better 😂
Disclaimer: I haven't checked if doing this creates other unexpected effects. YMMV.

I git bisected pydantic and pydantic-core and the bug was introduced in these commits
pydantic: bump pydantic-core to 2.14.0: https://github.com/pydantic/pydantic/commit/fb8a5946f5e167a7bf0b1cdecbfc9e28592685b8
pydantic-core: replace ultra_strict with new union implementation: https://github.com/pydantic/pydantic-core/commit/f409e0013e4f7b1937c2bd14ca877f7d27541be3

Okay, it looks like using union_mode="left_to_right" doesn't have this problem

Thanks for checking that 👍
Given that those commits changed the way a non-tagged union is evaluated, that's not a huge surprise. I suspect that the truth is the schema building bug illustrated above existed before those commits, but the newer smart union implementation (which tries to be more accurate in which type gets produced) exacerbated the existing bug.

If anyone else needs a workaround until this fixed I'm currently using a Discriminator with a custom function and annotating each type with Tag. The built schema always has a tagged-union in this case:
def get_state_type(v: Any) -> Hashable:
    if isinstance(v, dict):
        return v.get("state_type")
    return getattr(v, "state_type", None)

AnyState = Annotated[
    Annotated[NestedState, Tag("nested")]
    | Annotated[LoopState, Tag("loop")]
    | Annotated[LeafState, Tag("leaf")],
    Discriminator(get_state_type),
]

@sydney-runkle by the way, as I've mentioned previously, this issue can lead to a denial of service attack (and I did find apps which are vulnerable to this issue). Should we publish a security advisory somewhere?

Facing this issue as well at both the 01 and open-interpreter projects. Things were working fine for me yesterday, but since today morning, my FastAPI server does not start due to this Pydantic schema issue. This is also coming up in my OpenAI dependency, for which I've opened an issue here.
I did a memray memory profile to investigate further. Here is the report from open-interpreter, and here is the report from the 01 project. In both, the inflated memory seems to be due to this issue, since both FastAPI and OpenAI use Pydantic.
Could you give a timeline for this issue to be fixed? If this is not expected soon, could you give a patch I can apply in the meantime?
cc @KillianLucas

Hi all,
Given the significance of this bug, we're working on a fix now, and will plan to do a 2.6.3 release with said fix, once we have it! We'll keep this thread updated as we progress.

Hey all 😄,
After some extended debugging, we found a fix, and will release 2.6.3 today with the changes. Please ping me if you're experiencing any other issues. Happy coding! 🚀

authors: adriangb davidhewitt palle-k rossmacarthur shivenmian sydney-runkle

repo: pydantic/pydantic


text:
SettingsConfigDict: extra_forbidden error when no extra="ignore"

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
On pydantic-settings version 2.1.0 I haven't this error, but now I need to add extra="ignore" to avoid this. It seems, that something changes not in this default param, but in part of parsing .env file: I have error for all other env vars with not prefix "db_".
Example Code
from functools import lru_cache

from pydantic_settings import BaseSettings, SettingsConfigDict

class DBConfig(BaseSettings):
    host: str
    port: int
    database: str
    username: str
    password: str

    model_config = SettingsConfigDict(env_file=".env", env_prefix="db_")

class Config(BaseSettings):
    db: DBConfig()

@lru_cache
def get_config() -> Config:

    return Config(
        db=DBConfig(),
    )

config = get_config()

Python, Pydantic & OS Version
pydantic version: 2.6.3
        pydantic-core version: 2.16.3
    pydantic-settings version: 2.2.1
          pydantic-core build: profile=release pgo=true
                 install path: /home/aryadovoy/Documents/code/Appello/FastAPI Boilerplate/output/.venv/lib/python3.12/site-packages/pydantic
               python version: 3.12.1 (main, Jan 14 2024, 17:42:34) [GCC 11.3.1 20220421 (Red Hat 11.3.1-3)]
                     platform: Linux-6.0.12-100.fc35.x86_64-x86_64-with-glibc2.34
             related packages: fastapi-0.110.0 pydantic-settings-2.2.1 typing_extensions-4.10.0 mypy-1.8.0
                       commit: unknown

Thanks @aryadovoy for reporting this.
There is an issue in pydantic-settings related to this.
I close this one.

authors: aryadovoy hramezani


text:
Wrong dtype for empty Series.

Creating an empty Series with dtype as a string-type results in an instance with dtype=object.
import pandas as pd

dtype = 'S1'
ps = pd.Series([], dtype=dtype)
assert ps.dtype == dtype, "Series dtype = %r; requested dtype = %r" % (ps.dtype, dtype)

Traceback (most recent call last):
  File "issue_2.py", line 5, in <module>
    assert ps.dtype == dtype, "Series dtype = %r; requested dtype = %r" % (ps.dtype, dtype)
AssertionError: Series dtype = dtype('O'); requested dtype = 'S1'

all string types are represented as object dtype. You cannot have a fixed string type. Mainly this would make any kind of updating type of operation non-performant, with no real advantages.

authors: jreback santegoeds

repo: pydata/pandas


text:
Plot expressions

It would be nice to have the possibility to plot expressions. Suppose I have a dataframe with three columns: col1, col2, col3. Now I can do:
df.plot.scatter(x='col1', y='col2')

it would be nice to be able to plot expressions directly:
df.plot.scatter(x='col1 ** 2', y = 'col2 / sqrt(col3)')

and also for other plotting methods (e.g. histograms)

This is a duplicate of https://github.com/pydata/pandas/pull/12168, which we passed on at the time.
Since you are the second person to request it though, perhaps it's worth a second look. Feel free to post over there with additional use-cases / why this would be worth implementing 👍

I am coming from ROOT where this feature is very common with TTree::Draw. In physics this is very common. Suppose I have a three columns with the position of a point ('x', 'y', 'z') and one with the the time, I want to plot the distance from the origin:
df.plot.scatter(x='time', y='sqrt(x**2 + y**2 + z**2)')

authors: TomAugspurger wiso


text:
Should xarray allow assigning a masked constant?

Currently, ds['a'] = ((), ma.masked) where ds is an xarray.Dataset gives ValueError: Could not convert tuple of form (dims, data[, attrs, encoding]): ((), masked) to Variable., whereas ds['a'] = (), ma.MaskedArray(0.0, True) works (it sets the indicated value to NaN).  Should assigning ma.masked be equivalent to assigning ma.MaskedArray(0.0, True), or are there good reasons for the difference in behaviour?

One possible issue is that we don't know the dtype for ma.masked. This means we don't know which sentinel missing value to replace it with.
Consider this in the context of our current logic for dealing with masked arrays:
https://github.com/pydata/xarray/blob/17fa64f5314aa898f262a73fdc00d228ec380968/xarray/core/variable.py#L171-L178

Closing as stale but please reopen if sufficiently pressing.

authors: gerritholl max-sixty shoyer

repo: pydata/xarray


text:
Non-aggregating grouped operations on dask arrays are painfully slow to construct

These are both entirely lazy operations:
>>> %time res = ds.groupby('time.month').mean('time')
CPU times: user 142 ms, sys: 20.3 ms, total: 162 ms
Wall time: 159 ms
>>> %time res = ds.groupby('time.month').apply(lambda x: x - x.mean())
CPU times: user 46.1 s, sys: 4.9 s, total: 51 s
Wall time: 50.4 s

I suspect the issue (in part) is that _interleaved_concat_slow indexes out single elements from each dask array along the grouped axis prior to concatenating them together (unit tests for interleaved_concat can be found here). So we end up creating way too many small dask arrays.
Profiling results on slightly smaller data are in this gist.
It would be great if we could figure out a way to make this faster, because these sort of operations are a really nice show case for xray + dask.
CC @mrocklin in case you have any ideas.

Interesting.  Looks like the overhead of managing the graph is becoming non-trivial.
How long do these operations take to execute?  What is your blockdims?

Here's the timing info:
%time res = ds.t2m.groupby('time.month').mean('time').sum()
# CPU times: user 133 ms, sys: 6.39 ms, total: 140 ms
# Wall time: 145 ms
%time res.load_data()
# CPU times: user 2min 47s, sys: 1min, total: 3min 48s
# Wall time: 1min 19s

%time res = ds.t2m.groupby('time.month').apply(lambda x: x - x.mean()).sum()
# CPU times: user 49.1 s, sys: 6.39 s, total: 55.5 s
# Wall time: 55.1 s
%time res.load_data()
# CPU times: user 6min 17s, sys: 2min 20s, total: 8min 38s
# Wall time: 3min 25s

Blocks shape is {'latitude': (256,), 'longitude': (512,), 'time': (124, 124, 112, ..., 124, 120, 124)}

The good news about that timing info is that dask is still much faster for calculating the graph than doing the actual computation. But it's still not ideal from an interactivity perspective.

Hrm, not that much faster.  It's 25% overhead on the apply.  Worth checking out.  I might not get to this before release though.

Yeah, like I said in the other issue I don't think this is a blocker (we can add a disclaimer to the docs).

I reran @shoyer's original benchmark. I think this can be closed. Constructing the graphs for the example shown above (on slightly different data) took 84.7 ms and 241 ms respectively. Much better than before.

authors: jhamman mrocklin shoyer


text:
In a specific case, decode_cf adds encoding dtype that breaks to_netcdf

What happened?
Though the time variable in the two example datasets, ds1 and ds3, looks the same, the encoding is actually different and that makes it so the Dataset cannot be saved to netcdf. The encoding is added by using decode_cf — but only sometimes. In the case of ds3 below, the encoding of dtype '<M8[ns]' is the problem and is added in that case (open_mfdataset with 2 files) with decode_cf but is not added in the case of ds1 (open_mfdataset with 1 file in a list). When the "encoding dtype" is float, everything works ok (shown in case ds2) — again even though the times in both cases are visually appearing as datetime64-interpreted strings of dates and appear to be essentially the same setup.
What did you expect to happen?
These situations should all work and save to netcdf.
Minimal Complete Verifiable Example
import pandas as pd
import xarray as xr
tod = pd.Timestamp.today()
locs = [tod.strftime('https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/%Y/%m/%d/nos.cbofs.regulargrid.n001.%Y%m%d.t00z.nc'),
      tod.strftime('https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/%Y/%m/%d/nos.cbofs.regulargrid.n002.%Y%m%d.t00z.nc')]

# THIS WORKS: using `open_mfdataset` with 1 file
ds1 = xr.open_mfdataset([locs[0]])
print('DATASET1: ', ds1['ocean_time'].attrs, ds1['ocean_time'].encoding)
print(ds1['ocean_time'].dtype)
print(xr.decode_cf(ds1).ocean_time.encoding)
xr.decode_cf(ds1).to_netcdf('test1.nc')

# THIS WORKS: using `open_mfdataset` with 2 files but with times not decoded
ds2 = xr.open_mfdataset(locs, decode_times=False)
print('\nDATASET2: ', ds2['ocean_time'].attrs, ds2['ocean_time'].encoding)
print(ds2['ocean_time'].dtype)
print(xr.decode_cf(ds2).ocean_time.encoding)
xr.decode_cf(ds2).to_netcdf('test2.nc')

# THIS DOES NOT WORK: using `open_mfdataset` with 2 files with times decoded
ds3 = xr.open_mfdataset(locs)
print('\nDATASET3: ', ds3['ocean_time'].attrs, ds3['ocean_time'].encoding)
print(ds3['ocean_time'].dtype)
print(xr.decode_cf(ds3).ocean_time.encoding)
xr.decode_cf(ds3).to_netcdf('test3.nc')

Relevant log output
DATASET1:  {'long_name': 'time since initialization', 'field': 'time, scalar, series'} {'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/07/nos.cbofs.regulargrid.n001.20220407.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}
datetime64[ns]
{'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/07/nos.cbofs.regulargrid.n001.20220407.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}

DATASET2:  {'long_name': 'time since initialization', 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian', 'field': 'time, scalar, series'} {}
float64
{'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian', 'dtype': dtype('float64')}

DATASET3:  {'long_name': 'time since initialization', 'field': 'time, scalar, series'} {}
datetime64[ns]
{'dtype': dtype('<M8[ns]')}
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [9], in <module>
     24 print(ds3['ocean_time'].dtype)
     25 print(xr.decode_cf(ds3).ocean_time.encoding)
---> 26 xr.decode_cf(ds3).to_netcdf('test3.nc')

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/core/dataset.py:1900, in Dataset.to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1897     encoding = {}
   1898 from ..backends.api import to_netcdf
-> 1900 return to_netcdf(
   1901     self,
   1902     path,
   1903     mode,
   1904     format=format,
   1905     group=group,
   1906     engine=engine,
   1907     encoding=encoding,
   1908     unlimited_dims=unlimited_dims,
   1909     compute=compute,
   1910     invalid_netcdf=invalid_netcdf,
   1911 )

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/api.py:1072, in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1067 # TODO: figure out how to refactor this logic (here and in save_mfdataset)
   1068 # to avoid this mess of conditionals
   1069 try:
   1070     # TODO: allow this work (setting up the file for writing array data)
   1071     # to be parallelized with dask
-> 1072     dump_to_store(
   1073         dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
   1074     )
   1075     if autoclose:
   1076         store.close()

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/api.py:1119, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1116 if encoder:
   1117     variables, attrs = encoder(variables, attrs)
-> 1119 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/common.py:265, in AbstractWritableDataStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    263 self.set_attributes(attributes)
    264 self.set_dimensions(variables, unlimited_dims=unlimited_dims)
--> 265 self.set_variables(
    266     variables, check_encoding_set, writer, unlimited_dims=unlimited_dims
    267 )

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/common.py:303, in AbstractWritableDataStore.set_variables(self, variables, check_encoding_set, writer, unlimited_dims)
    301 name = _encode_variable_name(vn)
    302 check = vn in check_encoding_set
--> 303 target, source = self.prepare_variable(
    304     name, v, check, unlimited_dims=unlimited_dims
    305 )
    307 writer.add(source, target)

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:464, in NetCDF4DataStore.prepare_variable(self, name, variable, check_encoding, unlimited_dims)
    461 def prepare_variable(
    462     self, name, variable, check_encoding=False, unlimited_dims=None
    463 ):
--> 464     datatype = _get_datatype(
    465         variable, self.format, raise_on_invalid_encoding=check_encoding
    466     )
    467     attrs = variable.attrs.copy()
    469     fill_value = attrs.pop("_FillValue", None)

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:139, in _get_datatype(var, nc_format, raise_on_invalid_encoding)
    137 def _get_datatype(var, nc_format="NETCDF4", raise_on_invalid_encoding=False):
    138     if nc_format == "NETCDF4":
--> 139         return _nc4_dtype(var)
    140     if "dtype" in var.encoding:
    141         encoded_dtype = var.encoding["dtype"]

File ~/miniconda3/envs/model_catalogs/lib/python3.8/site-packages/xarray/backends/netCDF4_.py:160, in _nc4_dtype(var)
    158     dtype = var.dtype
    159 else:
--> 160     raise ValueError(f"unsupported dtype for netCDF4 variable: {var.dtype}")
    161 return dtype

ValueError: unsupported dtype for netCDF4 variable: datetime64[ns]

Anything else we need to know?
I tried turning off all the keyword-based inputs in decode_cf and the situation is the same as shown above. Currently my workaround is to first check to see if the datetimes in the Dataset are float and need to be decoded (because they were read in with decode_times=False), and only then are they decoded with decode_cf. In that case, the problem doesn't arise. (This is the case of ds2.)
Environment
INSTALLED VERSIONS
commit: None
python: 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 19:11:19)
[Clang 9.0.0 (tags/RELEASE_900/final)]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.1
libnetcdf: 4.8.1
xarray: 0.21.1
pandas: 1.4.1
numpy: 1.22.2
scipy: 1.8.0
netCDF4: 1.5.8
pydap: None
h5netcdf: 0.13.1
h5py: 3.6.0
Nio: None
zarr: 2.11.0
cftime: 1.5.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2022.02.0
distributed: 2022.02.0
matplotlib: 3.5.1
cartopy: 0.20.2
seaborn: None
numbagg: None
fsspec: 2022.01.0
cupy: None
pint: None
sparse: None
setuptools: 59.8.0
pip: 22.0.3
conda: None
pytest: 7.0.1
IPython: 8.0.1
sphinx: 4.4.0

Thanks for the report!  This is indeed odd behavior.  I was able to reproduce this issue with the latest release of xarray, 2022.3.0, but interestingly when I try running the example with the development version of xarray, everything works:
DATASET1:  {'long_name': 'time since initialization', 'field': 'time, scalar, series'} {'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/09/nos.cbofs.regulargrid.n001.20220409.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}
datetime64[ns]
{'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/09/nos.cbofs.regulargrid.n001.20220409.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}

DATASET2:  {'long_name': 'time since initialization', 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian', 'field': 'time, scalar, series'} {'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/09/nos.cbofs.regulargrid.n001.20220409.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64')}
float64
{'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/09/nos.cbofs.regulargrid.n001.20220409.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}

DATASET3:  {'long_name': 'time since initialization', 'field': 'time, scalar, series'} {'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/09/nos.cbofs.regulargrid.n001.20220409.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}
datetime64[ns]
{'source': 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/CBOFS/MODELS/2022/04/09/nos.cbofs.regulargrid.n001.20220409.t00z.nc', 'original_shape': (1,), 'dtype': dtype('float64'), 'units': 'seconds since 2016-01-01 00:00:00', 'calendar': 'gregorian'}

It's good that this appears to be fixed, but it perhaps suggests we have some missing test coverage.  I might see if I can develop a simple test we can add to our test suite.

@spencerkclark Interesting! Good I am glad it is fixed with the dev version. Great if this work up can be used for a test.
Mostly, I wanted to also get this documented since it took forever to track down the issue — maybe it will save someone else some time when they are googling.

After a little more investigating I think I've narrowed this down to the following example.  This particular example happens to be problematic even with the development version of xarray (i.e. it is an existing bug):
In [1]: import pandas as pd
   ...: import xarray as xr

In [2]: times = pd.date_range("2000", periods=2)

In [3]: variable = xr.Variable(["time"], times)

In [4]: variable.encoding
Out[4]: {}

In [5]: xr.conventions.decode_cf_variable(variable).encoding
Out[5]: {'dtype': dtype('<M8[ns]')}

The issue is that calling decode_cf_variable on a variable that already has a datetime64[ns] dtype adds an encoding attribute that leads to problems downstream ('<M8[ns]' is not a valid dtype in a netCDF file).
In this case it seems like we'd prefer not to add or modify any encoding attributes at all -- we are perfectly capable of handling empty encoding dictionaries for datetime-like data when writing it out to disk.

authors: kthyng spencerkclark


text:
CI test suites with pinned minimum dependencies

Second step towards resolving #3222.
Added two suites of CI tests:

Pinned minimum versions for all optional dependencies, except NEP18-dependant ones
Pinned minimum versions for NEP18 optional dependencies - at the moment only sparse; soon also pint (#3238)

All versions are the frozen snapshot of what py36.yml deploys today.
This PR ensures that we won't have accidental breakages from this moment on.
I made no effort  to try downgrading to sensible obsolete versions, as that would require a completely different order of magnitude of work.
I would suggest to proceed  with the downgrades (and consequent bugfixes) over several small, iterative future PRs that build upon this framework.

Ready for review

Thanks all


text:
Fix html repr in untrusted notebooks (plain text fallback)

[x] Closes #4041
[x] Tests added
[x] Passes isort -rc . && black . && mypy . && flake8
[x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is not very elegant (actually plain text repr is already included in the notebook as text/plain mime type but it is ignored when text/html mime type is present), but it seems to work. I haven't found a better workaround.
I don't really know if this can be properly tested (I only added a basic test).
Steps to test this fix:

To "untrust" a notebook: open an existing notebook with a simple editor, manually edit one output cell with a xarray object repr, and save the ipynb file.
Open this notebook with the Notebook app, you should see the plain text repr.

I tested this out exporting a notebook to Github:
https://gist.github.com/shoyer/6ee36acd164116fee4681f6237ca5249
Here's what I see:

Somehow we get both text and malformed HTML? :(

Mmm it looks like Github's notebook renderer removes the hidden attribute of the xr-wrap div in addition to removing CSS. That's unfortunate.
I tested this workaround in jupyterlab, classic notebook and nbviewer. It works as expected. I haven't tested other front-ends (Colab, vscode, etc.).

I sent a support request to GitHub asking for supporting the hidden attribute. We'll see if we get a response!

Thanks @benbovy

just confirmed that the fallback works perfectly. Thanks @benbovy !

authors: benbovy dcherian shoyer


text:
parameter documentation for DataArray.sel

I noticed yesterday that DataArray.sel doesn't document its parameters. Instead, it points to Dataset.sel (using the See Also section).

[x] Passes isort -rc . && black . && mypy . && flake8
[x] Fully documented, including whats-new.rst for all changes and api.rst for new API

LGTM. Thanks @keewis!

authors: dcherian keewis


text:
Rolling exp sum

[ ] Tests added
[x] Passes pre-commit run --all-files
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst

Requires merging and releasing https://github.com/shoyer/numbagg/pull/28

Hello @max-sixty! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file xarray/core/rolling_exp.py:

Line 1:1: F401 'typing.Optional' imported but unused

Should we go forward with this as-is? The 'workaround' is not so complicated...

Yes! We need to merge it on the numbagg side first though. I'll ask @shoyer later today on the call.

Test failure looks like a flaky issue with the docs

authors: mathause max-sixty pep8speaks


text:
Add a CI build for SymPy master

Fixes #94

e new experience. The majority of our users find it easier to fix failing builds.
Opt In
Jobspygaegalgebra sympy_master_ci 3580 (python-3.8-sympy-master)
Rerun workflow
FAILED
Finished:
3 hr ago (04:44)
Previous:
3579
Parallelism:
1x out of 0x
Queued:
05:08 waiting + 00:02 in queue
Resources:
2CPU/4096MB
Workflow:
build
Context:
N/A
Triggered by:
Utensil Song (pushed e03f0a4)
PR:
#289
COMMITS (1)
Utensil SongGitHube03f0a4Add a CI build for SymPy master
Test Summary (1)
Configuration
Timing
Parameters

PYTEST
1 FAILURE
Your job ran 294 tests with 1 failure
Cell 19 - examples.ipython.dop.ipynb
examples/ipython/dop.ipynb
#x1B[91mNotebook cell execution failed#x1B[0m
#x1B[94mCell 19: Cell outputs differ

Input:
#x1B[0mF.norm()

#x1B[94mTraceback:#x1B[0m
#x1B[94m mismatch 'text/latex'#x1B[91m

 assert reference_output == test_output failed:

  '$\\sqrt{\\op...i \\right)}}$' == '$\\sqrt{\\le...i \\right)}}$'
  - $\sqrt{\left(\operatorname{F^{\phi}}\right)^{2}{\left(r,\theta,\phi \right)} + \left(\operatorname{F^{r}}\right)^{2}{\left(r,\theta,\phi \right)} + \left(\operatorname{F^{\theta}}\right)^{2}{\left(r,\theta,\phi \right)}}$
  ?        ------                       -------                                    ------                    -------                                    ------                         -------
  + $\sqrt{\operatorname{F^{\phi}}^{2}{\left(r,\theta,\phi \right)} + \operatorname{F^{r}}^{2}{\left(r,\theta,\phi \right)} + \operatorname{F^{\theta}}^{2}{\left(r,\theta,\phi \right)}}$
#x1B[0m

The expected (old) output is:
\displaystyle \sqrt{\operatorname{F^{\phi}}^{2}{\left(r,\theta,\phi \right)} + \operatorname{F^{r}}^{2}{\left(r,\theta,\phi \right)} + \operatorname{F^{\theta}}^{2}{\left(r,\theta,\phi \right)}}

Seems related to sympy/sympy#18256

Odd, it seems to me that galgebra.printer already has a special-case for this, so I'm surprised we were caught out by that.

The behavior seems to be introduced earlier in sympy/sympy#17288 .

B.T.W. It's obvious to me that it should not parenthesize_super in this case.

Here's the code I'm thinking of:
https://github.com/pygae/galgebra/blob/f1ea1b76e62fe58b9b64a451f198bc93d079f320/galgebra/printer.py#L782-L800

I would like to allow the CI against sympy master to fail, but CircleCI doesn't seem to support so (googled allow_fail, optional etc.)

Unforunately it seems there's no way to conditionally skip a test with nbval either.

For now, can you just add another section to .nbval_sanitize.cfg to allow that discrepancy?

Here's the code I'm thinking of

Some tests show _print_Function is not executed for that case(i.e. F.norm()) and its type is actually sympy.core.power.Pow.

Perhaps exp is no longer passed to print_function by sympy master

authors: eric-wieser utensil

repo: pygae/galgebra


text:
Make Mv._make_x functions private and static

Depends on gh-58 in order to have a readable diff

These don't need to mutate any state, and they way they did before made the iterative approach in make_multivector quite confusing.
The functions now return a raw sympy expression, which is assigned to self.obj in a single place.
This also adds errors to the paths that didn't set anything at all.
The one assignment to self.grade_i was pointless, as it was overwritten later anyway

Looks nice to me.


text:
Weird line rendering with a long line where relatively few points have a high alpha value

For example in this video the length of each of these lines (left to right) is 8410, and only a few points along the line have high alpha values. The artifact is that arrows appear at the edges of the where the colors fade out (could it be an interpolation thing?).
https://user-images.githubusercontent.com/9403332/210306005-19fa7d20-b237-416b-af0e-849ae57a6200.mp4
The lines are instantiated like this:

pygfx.LineMaterial

pygfx.Line(
  geometry=pygfx.Geometry(positions=<[8410, 3]>, colors=<[8410, 4]),
  material=material(thickness=20, vertex_colors=True)
)

I tried adjusting the camera near and far planes but it doesn't make a difference. GPU is an nvidia RTX 3090, OS is Debian 11, vulkaninfo seems fine.
Anyways the purpose is to use color intensity of a very long line that represents timepoints to indicate the magnitude of a quantitative variable, not sure if it's possible to do this with a simple Mesh instead if that's better than a Line?

It looks like an interpolation problem.
Since gfx does not disclose the API for "changing vertex attribute interpolation method" at present, I think you can consider storing colors in the texture and using the nearest mode to sample.
In this case, the code may look like this:
positions = <[8410, 3]>
colors = <[8410, 4]>
texcoords = <[8410]>

tex = gfx.Texture(colors, dim=1)
view = tex.get_view(address_mode="repeat", filter="nearest")

gfx.Line(
    gfx.Geometry(positions=positions, texcoords=texcoords),
    gfx.LineMaterial(thickness=20, map=view),
)

Hopes this can help you.

I did a test.
There is a bug at present.  see: https://github.com/pygfx/pygfx/pull/420
And, there is another limitation: the maximum width and height of the texture is 8192, 😅
The whole test code:
import numpy as np
import random
from wgpu.gui.auto import WgpuCanvas, run
import pygfx as gfx

canvas = WgpuCanvas()
renderer = gfx.WgpuRenderer(canvas)

scene = gfx.Scene()
positions = []
colors = []
texcoords = []

for i in range(8192):
    positions.append([i, 0, 0])
    texcoords.append(i)
    r = random.random()
    if r > 0.9:
        colors.append([0.0, 1.0, 0.0, 1.0])
    else:
        colors.append([0.0, 1.0, 0.0, 0.0])

tex = gfx.Texture(np.array(colors, dtype=np.float32).reshape(8192, 4), dim=1)
view = tex.get_view(address_mode="repeat", filter="nearest")

line = gfx.Line(
    gfx.Geometry(positions=positions, texcoords=texcoords),
    gfx.LineMaterial(thickness=20.0, map=view),
)
scene.add(line)

camera = gfx.OrthographicCamera(600, 500)
camera.position.set(300, 0, 0)

controller = gfx.PanZoomController(camera.position.clone())
controller.add_default_event_handlers(renderer, camera)

def animate():
    controller.update_camera(camera)
    renderer.render(scene, camera)

if __name__ == "__main__":
    canvas.request_draw(animate)
    run()

I'm not sure if this result is what you want.

I can reproduce a similar effect using this minimal example:
import numpy as np
from wgpu.gui.auto import WgpuCanvas, run
import pygfx as gfx

canvas = WgpuCanvas()
renderer = gfx.WgpuRenderer(canvas)

scene = gfx.Scene()

x = np.linspace(0, 100, 8410)

positions = np.zeros((x.size, 3), np.float32)
positions[:, 0] = x

colors = np.zeros((x.size, 4), np.float32)
colors[:, 0] = 1
colors[::10, 3] = 1

line = gfx.Line(
    gfx.Geometry(positions=positions, colors=colors),
    gfx.LineMaterial(vertex_colors=True),
)
scene.add(line)

camera = gfx.OrthographicCamera(110, 10)
camera.position.set(50, 0, 0)
controller = gfx.PanZoomController(camera.position.clone())
controller.add_default_event_handlers(renderer, camera)

def render_scene():
    controller.update_camera(camera)
    renderer.render(scene, camera)

if __name__ == "__main__":
    canvas.request_draw(render_scene)
    run()

I think this is simply an aliasing issue. You have 8410 points, in a window that's about 800 pixels wide. So you have about 10 points on the line at each pixel. The point on the line being sampled is thus subject to a high degree of aliasing.
There is not much we can do from the point of view of Pygfx. You'd get similar aliasing when visualizing a highly detailed mesh model when zoomed out. I'd try and see if you can use a line with (much) less points.
If the effect that you were observing is different from what I think it is, could you provide a minimal example to reproduce it?

import numpy as np
from wgpu.gui.auto import WgpuCanvas, run
import pygfx as gfx

canvas = WgpuCanvas()
renderer = gfx.WgpuRenderer(canvas)

scene = gfx.Scene()

x = np.linspace(0, 100, 8410)

positions = np.zeros((x.size, 3), np.float32)
positions[:, 0] = x

colors = np.zeros((x.size, 4), np.float32)
colors[:, 0] = 1
colors[::10, 3] = 1

line = gfx.Line(
    gfx.Geometry(positions=positions, colors=colors),
    gfx.LineMaterial(vertex_colors=True),
)
scene.add(line)

camera = gfx.OrthographicCamera(110, 10)
camera.position.set(50, 0, 0)
controller = gfx.PanZoomController(camera.position.clone())
controller.add_default_event_handlers(renderer, camera)

def render_scene():
    controller.update_camera(camera)
    renderer.render(scene, camera)

if __name__ == "__main__":
    canvas.request_draw(render_scene)
    run()

If I set thickness=20 I can reproduce it with the arrows:

Is there a way to alter aliasing at the WGPU level? I'll look into how to set aliasing options for nvidia on debian.

Ah, I can see what you mean when I set a higher thickness. This is not antialiasing but a glitch that seems to occur in straight parts of the line. It also occurs when zoomed in a lot (passed where antialiasing could be the cause). I just made the line a sine wave, and then the triangles only occur at the inflection points (where the line is briefly straight). Looking for a fix ...

authors: almarklein kushalkolar panxinmiao

repo: pygfx/pygfx


text:
Exception when using PYGLET_DEBUG_GL_SHADERS

The following exception is raised when running with PYGLET_DEBUG_GL_SHADERS set.
[snip]
File "/Users/richard/Projects/template/venv/lib/python3.10/site-packages/pyglet/graphics/shader.py", line 394, in _introspect_uniforms
    for uniform in self._uniforms.values():
AttributeError: 'ShaderProgram' object has no attribute '_uniforms'. Did you mean: 'uniforms'?

The exception is spot on, I think; it should be uniforms. self._uniforms is not set until _introspect_uniforms returns.
https://github.com/pyglet/pyglet/blob/00c24ca456e2ef9ca9e92e614f26f25ea4039222/pyglet/graphics/shader.py#L383-L395
https://github.com/pyglet/pyglet/blob/00c24ca456e2ef9ca9e92e614f26f25ea4039222/pyglet/graphics/shader.py#L274

Thanks for opening the ticket. That bug slipped in during a recent cleanup.
This is now fixed in master, and will be in the next point release.

repo: pyglet/pyglet


text:
lean: fix the name regex

This corrects it to include the full range of unicode characters that are legal.
This also takes the opportunity to add support for highlighting `name and  ``checked_name literals.

Merged, thanks. My Unicode skills aren't good enough to understand whether the ranges map but I presume you know what you're doing there :)

authors: Anteru eric-wieser

repo: pygments/pygments


text:
[DOC] add examples for move

Add examples for the move function

Please tag maintainers to review.

@ericmjl

🚀 Deployed on https://deploy-preview-1245--pyjanitor.netlify.app

authors: ericmjl samukweku

repo: pyjanitor-devs/pyjanitor


text:
gitignore-style pattern matching

@Yajo Please have a look.

gitignore style matching for exclude and skip
abdicate pattern matching to pathspec
expands tests
updates readme
works with unicode

Closes: #118

@Yajo Do not want to be impolite, but I need to merge this now before you could have a look.

repo: pykong/copier


text:
How to do a monthly or yearly analysis in RFM?

In my company we sell yearly and monthly subscriptions with possibilities of cross-sell and upsell. How can I use the rfm_summary with monthly or yearly periods? When I try to set time_unit="M", I get this error:
ValueError: Unit M is not supported. Only unambiguous timedelta values durations are supported. Allowed units are 'W', 'D', 'h', 'm', 's', 'ms', 'us', 'ns'

Hey @franzoni315,
Use the time_unit and time_scaler parameters for this:
rfm_summary(data,
            customer_id_col,
            datetime_col,
            time_unit="W", # "M" & "Y" are unsupported TimeDelta units in pandas,
            time_scaler=4.33, # so Weeks must be rescaled into Months
            )

It worked, thanks very much, @ColtAllen !

@ColtAllen I actually discovered that this parameter is not available for the expected_customer_lifetime_value function. I am trying to adapt the CLV quickstart notebook for a yearly based business model, so for the rfm_summary I used the parameters time_unit="W", time_scaler=55, but I can't do the same for the expected_customer_lifetime_value function. I tried to set
clv_estimate = gg.expected_customer_lifetime_value(
    transaction_model=bgm,
    data=rfm_data,
    future_t=55,  # year
    discount_rate=0.01,  # monthly discount rate ~ 12.7% annually
    time_unit="W")

but I am not sure if this is correct, and also it takes too long.

@franzoni315 sorry for not seeing this sooner; I get a lot of emails related to pymc-marketing notifications.

I actually discovered that this parameter is not available for the expected_customer_lifetime_value function.

It's not used in this function because rfm_summary aggregates data for model fitting, not predictions.

clv_estimate = gg.expected_customer_lifetime_value(
    transaction_model=bgm,
    data=rfm_data,
    future_t=55,  # year
    discount_rate=0.01,  # monthly discount rate ~ 12.7% annually
    time_unit="W")

but I am not sure if this is correct, and also it takes too long.

Set time_unit='M' and future_t to the number of months in the future you want predicted. "Years" is an unsupported time unit, but equivalent to future_t=12*N_years. This will also require the transaction model to be trained on monthly data.
Also, is this model only being used for cross-sells and upsells? If you're trying to predict if a customer will renew a subscription, ShiftedBetaGeoModelIndividual is the intended model.
However, for large customer populations ShiftedBetaGeoModelIndividual may take too long since it'll generate unique posteriors for each customer, and it can't currently be used with the expected_customer_lifetime_value function. This could be the proper motivation for us to create the population equivalent of this model and add discounted CLV support.

repo: pymc-labs/pymc-marketing


text:
Add support for Python 3.12 by updating pypa/cibuildwheel.

Initial Python 3.12 support added to pypa/cibuildwheel@v2.13.0:
https://github.com/pypa/cibuildwheel/releases/tag/v2.13.0
But final 3.12.0 support added to pypa/cibuildwheel@v2.16.2, which is the latest release:
https://github.com/pypa/cibuildwheel/releases/tag/v2.16.2

I downloaded my fork CI uploaded artefacts and the 3.12 wheels were generated. However, I've not really tested them.
What would be the best way to test them? Installing them in a virtual environemnt and running the test.py file?

Even if PR https://github.com/pyocd/libusb-package/pull/20 supersedes the need to add explicity Python 3.12 support. This PR can still be merged to add the classifiers and update the pypa/cibuildwheel action version.

Hi, any news regarding this issue? Source distribution for the latest version is missing https://pypi.org/project/libusb-package/1.0.26.2/#files and the wheel for Python 3.12 is not available.

repo: pyocd/libusb-package


text:
Pipenv lock ResolutionFailure exception on private package

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.
Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.
Make sure to mention your debugging experience if the documented solution failed.
Issue description
Pipenv lock results in ResolutionFailure exception with the message of ERROR: No matching distribution found for package==0.0.1 when the package is added from a private repository.
Expected result
New lock file gets created.
Actual result
Reporter.starting()
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('ariadne-django-ext'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('ariadne-django-ext'), None)
Reporter.adding_requirement(SpecifierRequirement('simplejson'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('simplejson'), None)
Reporter.adding_requirement(SpecifierRequirement('django-social-oauth-token'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('django-social-oauth-token'), None)
Reporter.adding_requirement(SpecifierRequirement('markdown'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('markdown'), None)
Reporter.adding_requirement(SpecifierRequirement('psycopg2-binary'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('psycopg2-binary'), None)
Reporter.adding_requirement(SpecifierRequirement('pillow'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('pillow'), None)
Reporter.adding_requirement(SpecifierRequirement('ariadne-django'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('ariadne-django'), None)
Reporter.adding_requirement(SpecifierRequirement('django-filter'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('django-filter'), None)
Reporter.adding_requirement(SpecifierRequirement('django-user-email'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('django-user-email'), None)
Reporter.adding_requirement(SpecifierRequirement('celery'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('celery'), None)
Reporter.adding_requirement(SpecifierRequirement('django-cors-headers'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('django-cors-headers'), None)
Reporter.adding_requirement(SpecifierRequirement('django-oauth-toolkit'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('django-oauth-toolkit'), None)
Reporter.adding_requirement(SpecifierRequirement('private_package==7.0.1'), None)
INFO:pipenv.patched.notpip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('private_package==7.0.1'), None)
CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement private_package==7.0.1 (from versions: none)
Traceback (most recent call last):
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
raise RequirementsConflicted(criterion)
pipenv.patched.notpip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('private_package==7.0.1')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/patched/notpip/_vendor/resolvelib/resolvers.py", line 350, in resolve
raise ResolutionImpossible(e.criterion.information)
pipenv.patched.notpip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('private_package==7.0.1'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 661, in resolve
results = resolver.resolve(self.constraints, check_supported_wheels=False)
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/patched/notpip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
raise error from e
pipenv.patched.notpip._internal.exceptions.DistributionNotFound: No matching distribution found for private_package==7.0.1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/resolver.py", line 857, in
main()
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/resolver.py", line 843, in main
_main(
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/resolver.py", line 822, in _main
resolve_packages(
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/resolver.py", line 771, in resolve_packages
results, resolver = resolve(
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/resolver.py", line 751, in resolve
return resolve_deps(
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1068, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 862, in actually_resolve_deps
resolver.resolve()
File "/Users/khasbilegt/.local/pipx/venvs/pipenv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 663, in resolve
raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for private_package==7.0.1
✘ Locking Failed!
Steps to replicate

Add a private source to the Pipfile
Create a new virtualenv, install the package. (First time locking works somehow)
Try to install the different version of the package but fails when locking.

@khasbilegt Please see: https://github.com/pypa/pipenv/issues/5053#issuecomment-1101000045

authors: khasbilegt matteius

repo: pypa/pipenv