text:
Latest unc0ver fails to jailbreak iOS 11

Iphone x
11.12
App 4.3.0

@ali201089 please use 3.8.0 beta 1 for iOS 11

@ali201089 also post your unc0ver diagnostics.

How can i resign it ?

As im out of jailbreak

This issues page is for unc0ver issues itself not personal user problems. @ali201089

@ali201089 also post your unc0ver diagnostics.

[] unc0ver Version: 4.3.0
[] Bundled Resources Version: 1.0~b26
refreshPreferences: HangTracerEnabled: 0
refreshPreferences: HangTracerDuration: 500
refreshPreferences: ActivationLoggingEnabled: 0 ActivationLoggingTaskedOffByDA:0
[] Status: Jailbreak
[] Progress: Ready to jailbreak
[] Status: Jailbreaking
[] KernelUtilities.c(1503): Assertion failure
[] message size for kalloc.4096: 2956
[] got user client: 0xb50b
[] Progress: Exploiting kernel...
[] prepared kqueue
[] our task port is at 0xffffffe005bd4690
[] found target port with suitable allocation page offset: 0xffffffe0096f9d88
[] kCFCoreFoundationVersionNumber: 1445.320000
[] offsets selected for iOS 11.0 to 11.2.6
[] replacer_body_size: 0xb74
[] message_body_offset: 0x448
[] 0
[] e00002c9
[] 0
[] 0
[] 1
[] 2
[] 3
[] 4
[] 5
[] 6
[] 7
[] 8
[] 9
[] 10
[] 11
[] 12
[] 13
[] 14
[] 15
[] 16
[] 17
[] 18
[] 19
[] 20
[] 21
[] 22
[] 23
[] 24
[] 25
[] 26
[] 27
[] 28
[] 29
[] 30
[] 31
[] 32
[] 33
[] 34
[] 35
[] 36
[] 37
[] 38
[] 39
[] 40
[] 41
[] 42
[] 43
[] 44
[] 45
[] 46
[] 47
[] 48
[] 49
[] 50
[] 51
[] 52
[] 53
[] 54
[] 55
[] 56
[] 57
[] 58
[] 59
[] 60
[] 61
[] 62
[] 63
[] 64
[] 65
[] 66
[] 67
[] 68
[] 69
[] 70
[] 71
[] 72
[] 73
[] 74
[] 75
[] 76
[] 77
[] 78
[] 79
[] 80
[] 81
[] 82
[] 83
[] 84
[] 85
[] 86
[] 87
[] 88
[] 89
[] 90
[] 91
[] 92
[] 93
[] 94
[] 95
[] 96
[] 97
[] 98
[] 99
[] 100
[] 101
[] 102
[] 103
[] 104
[] 105
[] 106
[] 107
[] 108
[] 109
[] 110
[] 111
[] 112
[] 113
[] 114
[] 115
[] 116
[] 117
[] 118
[] 119
[] 120
[] 121
[] 122
[] 123
[] 124
[] 125
[] 126
[] 127
[] 128
[] 129
[] 130
[] 131
[] 132
[] 133
[] 134
[] 135
[] 136
[] 137
[] 138
[] 139
[] 140
[] 141
[] 142
[] 143
[] 144
[] 145
[] 146
[] 147
[] 148
[] 149
[] 150
[] 151
[] 152
[] 153
[] 154
[] 155
[] 156
[] 157
[] 158
[] 159
[] 160
[] 161
[] 162
[] 163
[] 164
[] 165
[] 166
[] 167
[] 168
[] 169
[] 170
[] 171
[] 172
[] 173
[] 174
[] 175
[] 176
[] 177
[] 178
[] 179
[] 180
[] 181
[] 182
[] 183
[] 184
[] 185
[] 186
[] 187
[] 188
[] 189
[] 190
[] 191
[] 192
[] 193
[] 194
[] 195
[] 196
[] 197
[] 198
[] 199
[] got replaced with replacer port 72
[] found kernel vm_map: 0xfffffff106b22420
[] second time got replaced with replacer port 0
[] will try to read from second port (fake kernel)
[] kernel read via fake kernel task port worked?
[] 0x0000000000420000
[] 0x0000000000000000
[] 0xfffffff106b4a0a0
[] 0xffffffe0070cd590
[] about to build safer tfp0
[] message buffer: ffffffe008337000
[] fake_kernel_task_kaddr: ffffffe008337000
[] read fake_task_refs: d00d
[] about to test new tfp0
[] kernel read via second tfp0 port worked?
[] 0x0000000000420000
[] 0x0000000000000000
[] 0xfffffff106b4a0a0
[] 0xffffffe0070cd590
[] built safer tfp0
[] about to clear up
[] cleared up
[] tfp0: 0x188d70b
[] Successfully exploited kernel.
[] Finding kernel offsets...
[] Progress: Initializing jailbreak...
[] KernelUtilities.c(197): Assertion failure
[] KernelUtilities.c(2720): Assertion failure
[] Unable to find kernel offsets.
[] Unplatformizing...
[] KernelUtilities.c(349): Assertion failure
[] Unable to make my task a non-platform task.
[] KernelUtilities.c(485): Assertion failure
[] Unable to make my codesign blob a non-platform blob.
[] Sandboxing...
[] Downgrading host port...
[] Deallocating ports...
[] Deallocating memory...
[] Status: Jailbroken

Please help me if yu can fix

@ali201089 no the only solution is to use an older version.

please post the diagnostics file not copy paste @ali201089

Ohh soo im stuck 😩

please post the diagnostics file not copy paste @ali201089

Im not dont knw how to do it?

@ali201089 Open unc0ver app, go to the settings tab, scroll down, hit share diagnostics.

Iv tryed its keeps on crashing

Wat should i do then


text:
🛑 NARLabs is down

In 1c398c1, NARLabs (https://www.narlabs.org.tw/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: NARLabs is back up in 039e251.

source: gharchive/issue

repo: pwtsai/upptime

license: MIT

license_type: permissive

license_source: github-api


text:
Work with 2 models at the same time

Hi all,
I want to perform a coupled analysis of 2 models, let's say model 1 and model 2. From the solution of model 1, I change some parameters values of the model 2, and from the solution of model 2 and change parameters values of model 1.
The models are defined by input files that are quit large. Is it possible to not read them again every time I want to switch of model? For example by creating 2 mapdl instances?
I have tried to create a MAPDL pool but it fails to create instances after the first one, and I obtain an error such as "enable to extract ansys version from path" although I have defined the execution path in the LocalMapdlPool command.
Thank you for your help,
Louis

I have tried to create a MAPDL pool but it fails to create instances after the first one, and I obtain an error such as "enable to extract ansys version from path" although I have defined the execution path in the LocalMapdlPool command.

Could you please post the full traceback here? It looks like there's an issue with the pool creation. Regular testing for the pool occurs on Linux, and it's irregular on Windows.

I see the issue. I'll patch this and have a release out shortly.

thanks!

Before I implement it, could you please run the following?
from ansys.mapdl.core import launch_mapdl
exec_file = '/ul/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211'
mapdl_a = launch_mapdl(exec_file=exec_file)
print(mapdl_a)
mapdl_b = launch_mapdl(exec_file=exec_file)
print(mapdl_b)

Actually, I am not able to launch mapl using launch_mapdl. I have to create a grpc instance on a terminal using:
/ul/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211 -grpc
and then connect to it doing
from ansys.mapdl.core import Mapdl
mapdl=Mapdl()

If I run
from ansys.mapdl.core import Mapdl
exec_file = '/ul/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211'
mapdl_a = Mapdl(exec_file=exec_file)
print(mapdl_a)
mapdl_b = Mapdl(exec_file=exec_file)
print(mapdl_b)

I obtain

Actually, I am not able to launch mapl using launch_mapdl.

That's the issue. LocalMapdlPool creates (and maintains) instances on the fly using launch_mapdl under the hood.
If you can get me the full trackback you get when you run launch_mapdl, I can fix it.

Ok, I see. Here is what I obtain when I run the lines you asked

:

@germa89, please prioritize this.

@germa89 ,

I wonder if you have any env variable configured. Can you please share the PyMAPLD report here? Just copy and paste here the result from executing:

Here is what I obtain. I don't see any environment variables:

PyMAPDL Software and Environment Report
Date: jeu. avril 07 08:50:48 2022 CEST
            OS : Linux
        CPU(s) : 24
       Machine : x86_64
  Architecture : 64bit
           RAM : 251.6 GiB
   Environment : IPython
   File system : ext4

NVIDIA Corporation : GPU Vendor
Quadro P4000/PCIe/SSE2 : GPU Renderer
4.5.0 NVIDIA 460.91.03 : GPU Version
Python 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]
    matplotlib : 3.3.4
         numpy : 1.19.5
       pyvista : 0.33.3
       appdirs : 1.4.4
          tqdm : 4.63.0
        pyiges : 0.2.1
         scipy : 1.6.0
          grpc : 1.44.0

ansys.api.mapdl.v0 : 0.4.1
ansys.mapdl.reader : 0.51.10
google.protobuf : 3.19.4
Ansys Installation

Unable to locate any Ansys installations
Ansys Environment Variables

None

When I do
from ansys.mapdl.core import launch_mapdl
exec_file = '/ul/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211'
mapdl_a = launch_mapdl(exec_file=exec_file, port=50052)
print(mapdl_a)
mapdl_b = launch_mapdl(exec_file=exec_file, port=50053)
print(mapdl_b)

mapdl_a is successfully obtained but not mapdl_b. Maybe it is because it requires 2 ANSYS licenses? I think I only have one license.

Indeed, the second terminal returns an error:

Sorry, I didn't know it wasn't possible to create a mapdl pool with one license only.

No problem! Your feedback will help to improve PyMAPDL so it is much appreciated!

I'm leaving this issue open because it reflects that some changes should done to the license module and error handling.

Hi @lschuler1
I have been having another look at this issue. It seems to me that because you have installed ansys in a "unusual" location, PyMAPDL have some troubles to load the ansys executables.
For example, when PyMAPDL loads (ansys\mapd\core\__init__.py), it also create the variable HAS_ANSYS. This variable is created from calling _version_from_path. If ANSYS is not found in the usual locations (/ansys_inc/and/usr/ansys_inc/`), then it will prompt the user to input the ansys executable location, and it will save that location in a configuration file for future uses.
I believe the first time you tried to use PyMAPDL, it did ask you about the ANSYS path location and you input /u/logiciels/ANSYS/ansys_inc/v211/Framework/bin/Linux64/runwb2, which is the Workbench executable I believe. But PyMAPDL requires the MAPDL executable, very likely in /u/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211.
Am I right? Or at least close to?
If I'm right, to make PyMAPDL work (and being able to launch MAPDL instances) you just need to edit that configuration file. This file path can be obtained using:
>>> import appdirs
>>> print(appdirs.user_data_dir("ansys_mapdl_core"))
C:\Users\gayuso\AppData\Local\ansys_mapdl_core\ansys_mapdl_core

In that folder, you will find a file called config.txt.
You can delete that file, or just replace the path inside it (I presume referred to /u/logiciels/ANSYS/ansys_inc/v211/Framework/bin/Linux64/runwb2) by /u/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211.
If you opt for deleting that file, PyMAPDL will ask you again for a valid executable next time you start PyMAPDL.
Please let me know if this helps or not.

Hi @germa89,
Yes, the first time I tried to use PyMAPDL, I put the wrong path. But then I modified it and put the right one (/u/logiciels/ANSYS/ansys_inc/v211/ansys/bin/ansys211).
I have tried today to delete the file config.txt and the to put the right path when starting again PyMAPDL.
The error I get when I try to create a MAPDL pool is different and related to the insufficient number of licenses:

This message is way more helpful that the one I had the first time, as I understand that I just need more ANSYS licenses.
Thank you very much for your help.
Louis

repo: pyansys/pymapdl


text:
Separate Voltages for events and setting initial SOC

Description
It appears the parameters
Lower voltage cut-off [V]	2.8
Upper voltage cut-off [V]	4.2
are used both to set the initial SOC and for simulation events, as a result if the voltage limits for discharge are changed, this impacts the initial SOC.
Motivation
It would be good to have a sperate variable for the simulation event termination and calculation of SOC/initial stoichiometry.
The present implementation is breaking my use case of comparing the model to different discharge voltages, as the initial SOC changes also.
Possible Implementation
No response
Additional context
No response

Hi Jason :) You can change the values in the experiment specification
pybamm.Experiment(
    [
        "Discharge at C/5 for 10 hours or until 3 V",
        "Rest for 1 hour",
        "Charge at 1 A until 4.2 V",
        "Hold at 4.2 V until 10 mA",
        "Rest for 1 hour",
    ]
)

without affecting the voltage limits for SOC, is this what you need?

Hi Tino, thanks for the rapid response! I'm actually using a recorded
current trace, and feeding in using the old interpolant method. I'll give
the new experiment using drive cycles a go. I just pulled upstream changes
and broke a few things on my branch... so im trying to catch up on
parameter changes.
What I want to do is charge to 110% SOC or over discharge assuming the ocp
curve is well defined and stoichiometries are nonzero (within limits) this
means the voltage events should be outside the min/max voltages used for
calculating SOC... if that makes sense... happy to talk through my idea if
you have time.
Best -J
On Wed, Dec 14, 2022, 9:48 PM Valentin Sulzer @.***>
wrote:

Hi Jason :) You can change the values in the experiment specification
pybamm.Experiment(
[
"Discharge at C/5 for 10 hours or until 3 V",
"Rest for 1 hour",
"Charge at 1 A until 4.2 V",
"Hold at 4.2 V until 10 mA",
"Rest for 1 hour",
]
)
without affecting the voltage limits for SOC, is this what you need?
—
Reply to this email directly, view it on GitHub
https://github.com/pybamm-team/PyBaMM/issues/2562#issuecomment-1352487749,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE2YEPGCSBWZVMNWMEXAQTWNKBHBANCNFSM6AAAAAAS7GYB64
.
You are receiving this because you authored the thread.Message ID:
@.***>

I see, that makes sense. And even if the SOC is still <100% the voltage might go above the cut-off. So we should add parameters "Open circuit potential at 0% SOC" and "Open circuit potential at 100% SOC", which can be different from the voltage cut-offs.
Could you open a PR to do this? It should be pretty straightforward. Basic steps are:

Add these parameters to each parameter set, defaulting to the same value as the cut-offs
Add a two new Parameter object to the lithium_ion_parameters.py files (e.g. self.opc_soc_0, self.ocp_soc_100)
Replace the cut-off voltage in this line (and other places in that file) with the new target ocp voltage https://github.com/pybamm-team/PyBaMM/blob/1f25a1a3bbaf739848b67dec3b81562db5ad5c7a/pybamm/models/full_battery_models/lithium_ion/electrode_soh.py#L446

I'll take a stab at it!
On Thu, Dec 15, 2022, 10:24 AM Valentin Sulzer @.***>
wrote:

I see, that makes sense. And even if the SOC is still <100% the voltage
might go above the cut-off. So we should add parameters "Open circuit
potential at 0% SOC" and "Open circuit potential at 100% SOC", which can be
different from the voltage cut-offs.
Could you open a PR to do this? It should be pretty straightforward. Basic
steps are:

Add these parameters to each parameter set, defaulting to the same
value as the cut-offs
Add a two new Parameter object to the lithium_ion_parameters.py
files (e.g. self.opc_soc_0, self.ocp_soc_100)
Replace the cut-off voltage in this line (and other places in that
file) with the new target ocp voltage
https://github.com/pybamm-team/PyBaMM/blob/1f25a1a3bbaf739848b67dec3b81562db5ad5c7a/pybamm/models/full_battery_models/lithium_ion/electrode_soh.py#L446

—
Reply to this email directly, view it on GitHub
https://github.com/pybamm-team/PyBaMM/issues/2562#issuecomment-1353261958,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE2YEMCQTAO6JUNAHUWNG3WNMZ2TANCNFSM6AAAAAAS7GYB64
.
You are receiving this because you authored the thread.Message ID:
@.***>

I made the changes you suggested in my repo for testing, but it seems that
the voltage is not updated,
is it possible that if the initial SOC is still one then the calculation of
the initial concentrations is being skipped?
https://github.com/js1tr3/PyBaMM/blob/7b943d147f14a51892e29cf1b6e16295ec1358b7/pybamm/simulation.py#L384
On Thu, Dec 15, 2022 at 6:39 PM Jason Siegel @.***> wrote:

I'll take a stab at it!
On Thu, Dec 15, 2022, 10:24 AM Valentin Sulzer @.***>
wrote:

I see, that makes sense. And even if the SOC is still <100% the voltage
might go above the cut-off. So we should add parameters "Open circuit
potential at 0% SOC" and "Open circuit potential at 100% SOC", which can be
different from the voltage cut-offs.
Could you open a PR to do this? It should be pretty straightforward.
Basic steps are:

Add these parameters to each parameter set, defaulting to the same
value as the cut-offs
Add a two new Parameter object to the lithium_ion_parameters.py
files (e.g. self.opc_soc_0, self.ocp_soc_100)
Replace the cut-off voltage in this line (and other places in that
file) with the new target ocp voltage
https://github.com/pybamm-team/PyBaMM/blob/1f25a1a3bbaf739848b67dec3b81562db5ad5c7a/pybamm/models/full_battery_models/lithium_ion/electrode_soh.py#L446

—
Reply to this email directly, view it on GitHub
https://github.com/pybamm-team/PyBaMM/issues/2562#issuecomment-1353261958,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE2YEMCQTAO6JUNAHUWNG3WNMZ2TANCNFSM6AAAAAAS7GYB64
.
You are receiving this because you authored the thread.Message ID:
@.***>

Does test_electrode_soh.py work as expected?

Can you open a draft pull request so I can check it? It shouldn't make a difference what Vmin and Vmax are for this calculation, if it's based entirely on OCP at 0/100% SOC. Make sure you change voltage_low_cut_dimensional to opc_soc_0_dimensional everywhere relevant.

@js1tr3 how is this going?

@js1tr3 how is this going?

Good, got tied up with teaching this semester, but hope to get back on this shortly. Any suggestions for resolving conflicts in jupyter notebooks?

Solved by #2580

authors: js1tr3 rtimms tinosulzer

repo: pybamm-team/PyBaMM


text:
Point parameter estimation methods

Overarching parent issue to cover the family of all methods that give only "point" parameter estimates (rather than distributions) - i.e., maximum likelihood estimation (MLE) including variations such as prediction error minimisation (PEM); maximum a posteriori (MAP) estimation.

in terms of a method to get the point estimate, CMAES is a pretty reliable method (again lots of implementations including PINTS ;)

interesting; we should have a deep dive discussion on all this. We have tended previously just to use variants of BFGS

We have added quite a few frequentistic methods, closing this issue.

authors: BradyPlanden davidhowey martinjrobins

repo: pybop-team/PyBOP


text:
Mypy plugin doesn't consider allow_population_by_field_name

Initial Checks

[X] I have searched GitHub for a duplicate issue and I'm sure this is something new
[X] I have searched Google & StackOverflow for a solution and couldn't find anything
[X] I have read and followed the docs and still think this is a bug
[X] I am confident that the issue is with pydantic (not my code, or another library in the ecosystem like FastAPI or mypy)

Description
Probably an issue with the Pydantic mypy plugin. It happened after the mypy 1.0.0 release.
Consider this code, and run mypy --strict:
from pydantic import BaseModel, Field

class Model(BaseModel):
    field: str = Field(alias="alias")

    class Config:
        allow_population_by_field_name = True

Model(alias="")  # OK.
Model(field="")  # error: Unexpected keyword argument "field" for "Model"  [call-arg]

At runtime, this is fine, because of the allow_population_by_field_name setting, but the mypy plugin only allows instantiating the class through the alias.
Python, Pydantic & OS Version
Python 3.9.13
Pydantic 1.10.5
Ubuntu on ARM, Linux 5.15.0-58

Affected Components

[ ] Compatibility between releases
[ ] Data validation/parsing
[ ] Data serialization - .model_dump() and .model_dump_json()
[ ] JSON Schema
[ ] Dataclasses
[X] Model Config
[ ] Field Types - adding or changing a particular data type
[ ] Function validation decorator
[ ] Generic Models
[ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
[X] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

More interesting, there is a difference between mypy 1.0.1 and the latest from GitHub. They do the opposite. The file test.py is the snippet from above:
$ mypy --version
mypy 1.0.1 (compiled: no)

$ mypy --strict ./test.py
test.py:9: error: Unexpected keyword argument "alias" for "Model"  [call-arg]
Found 1 error in 1 file (checked 1 source file)

$ mypy --version
mypy 1.1.0+dev.ed3a6c1c019a8098f65e768697f1048d1f97a4a1 (compiled: no)

$ mypy --strict ./test.py
test.py:10: error: Unexpected keyword argument "field" for "Model"  [call-arg]
Found 1 error in 1 file (checked 1 source file)

More interesting, there is a difference between mypy 1.0.1 and the latest from GitHub. They do the opposite.

The current mypy master breaks the pydantic mypy plugin. See #5077 for a fix.

Thanks @jonathanslenders for reporting this.
Yes, you are right. I can confirm the problem in 1.10.5 with both mypy 1.0.1 and mypy 1.1.0+dev.
I've tested it on V2 and they return Unexpected keyword argument "field" for "Model"  [call-arg] with both mypy 1.0.1 and mypy 1.1.0+dev
@cdce8p I've tried your fix on mypy with https://github.com/pydantic/pydantic/pull/5077, but it doesn't fix the problem on 1.10.x and V2

@cdce8p I've tried your fix on mypy with #5077, but it doesn't fix the problem on 1.10.x and V2

That fix only restores the state of 1.0.1. I.e. the only error should be
test.py:9: error: Unexpected keyword argument "alias" for "Model"  [call-arg]

--
Technically, the behavior described here is documented that way in docs

If Config.allow_population_by_field_name=True, the generated signature will use the field names, rather than aliases.

https://docs.pydantic.dev/mypy_plugin/#plugin-capabilities
It's also similar to how pyright handles it, although it always uses the alias and ignores allow_population_by_field_name completely.
In theory it might be possible to allow both with custom overloads but that would require a larger change to the mypy plugin and would get complicated pretty quickly. You would need an overload for every single combination of possible args.

You would need an overload for every single combination of possible args.

I think it would be a significant improvement to just do two overloads -- one that uses all aliases, and one that uses none (when both are acceptable). Usually users are consistent about using one of these two approaches. (I personally would want it to always not use aliases when the field names are acceptable, but I understand not everyone feels the same..) That said, we'd probably want to make sure that it didn't result in a significant increase in error message verbosity (and corresponding decrease in clarity) when using invalid keyword arguments, i.e., arguments that aren't a part of either signature.
I haven't worked on the mypy plugin for a while (and am not very familiar with pyright) so I don't know how hard this would be.

That fix only restores the state of 1.0.1. I.e. the only error should be
test.py:9: error: Unexpected keyword argument "alias" for "Model"  [call-arg]

@cdce8p BTW, I've installed mpy dev (that contains the other fix) with your patch but it returns:
Unexpected keyword argument "field" for "Model"  [call-arg]

That fix only restores the state of 1.0.1. I.e. the only error should be
test.py:9: error: Unexpected keyword argument "alias" for "Model"  [call-arg]

@cdce8p BTW, I've installed mpy dev (that contains the other fix) with your patch but it returns: Unexpected keyword argument "field" for "Model" [call-arg]

Are you sure you've enabled the plugin in the mypy config?
Tbh I'm testing the version backported to 1.10.X of my PR as mypy is quite slow with 2.0.dev but that works as explained above.

For what it's worth, if this is not resolved by #5111, I would be happy to accept a PR fixing this (in pydantic v1.10.X or v2).
If no one else gets to this first, I expect we will fix this at some point in the not-distant future in connection with the v2 release, but it will definitely happen faster if someone has a chance to figure it out what's going wrong before then.
For what it's worth, I would be okay with either of these approaches:

Generating only a single signature using no aliases when allow_population_by_field_name is True
Generating two overloads, one with all aliases and one with no aliases, when allow_population_by_field_name is True

I don't think it's a good idea to try to support every combination of keyword argument names as that will bog mypy down massively for any model with a large number of keywords, but the all-aliases and no-aliases approach seems reasonable to me.

@dmontagu : I'd very much appreciate the two overloads. Unfortunately I don't have time right now to contribute.

I'm running into this issue as well and have narrowed it down to pydantic==1.10.3 causing issues with mypy==1.1.1
Downgrading to pydantic=1.10.2 resolves the issue for me.

I currently suspect this is an issue caused by moving to using ConfigDict instead of class Config:. Resolving this will be my top priority to investigate once we put out the alpha release.

I can't seem to reproduce this issue — I get:
9: error: Missing named argument "field" for "Model"  [call-arg]

as the only error when running on the code shared in the start of this issue with every pydantic 1.10.X version I've tried, and with mypy==1.1.1 and mypy==1.0.1 (and older versions of mypy)
If someone can confirm the issue is still happening and confirm to me how to reproduce it, I'll be happy to investigate.
One note: if you haven't, I would try deleting the mypy cache (generally stored in a folder called .mypy_cache). It's possible that this could be the source of some issues (though I thought we had things in place to prevent that).

I can't seem to reproduce this issue — I get:
9: error: Missing named argument "field" for "Model"  [call-arg]

as the only error when running on the code shared in the start of this issue with every pydantic 1.10.X version I've tried, and with mypy==1.1.1 and mypy==1.0.1 (and older versions of mypy)
If someone can confirm the issue is still happening and confirm to me how to reproduce it, I'll be happy to investigate.
One note: if you haven't, I would try deleting the mypy cache (generally stored in a folder called .mypy_cache). It's possible that this could be the source of some issues (though I thought we had things in place to clear the cache on upgrade).

I just created this repo and ran it in codespaces and encountered the issue. I am new to codespaces so if there is a way to grant you direct access to the codespace let me know and I can do that as well.
https://github.com/mdgilene/pydantic-mypy

@mdgilene can you share the result of running:
python -c "from mypy.version import __version__; print(f'                 mypy_version: {__version__}')"
python -c "import pydantic.utils; print(pydantic.utils.version_info())"

I cloned your repository, and with this set of version info:
                 mypy_version: 1.0.1
             pydantic version: 1.10.5
            pydantic compiled: True
                 install path: /Users/davidmontague/Programming/pydantic/pydantic-v1/env/lib/python3.11/site-packages/pydantic
               python version: 3.11.1 (main, Jan 11 2023, 20:15:27) [Clang 13.0.0 (clang-1300.0.27.3)]
                     platform: macOS-12.2.1-arm64-arm-64bit
     optional deps. installed: ['devtools', 'dotenv', 'email-validator', 'typing-extensions']

I just get:
pydantic-mypy on  main is 📦 v0.1.0 via 🐍 v3.11.1 (env)
❯ mypy .
mypy_pydantic/model.py:11: error: Unexpected keyword argument "alias" for "Model"  [call-arg]
Found 1 error in 1 file (checked 1 source file)

Version Info
(mypy-pydantic-py3.10) @mdgilene ➜ /workspaces/codespaces-blank (main) $ python -c "from mypy.version import __version__; print(f'                 mypy_version: {__version__}')" && python -c "import pydantic.utils; print(pydantic.utils.version_info())"
                 mypy_version: 1.1.1
             pydantic version: 1.10.3
            pydantic compiled: True
                 install path: /home/codespace/.cache/pypoetry/virtualenvs/mypy-pydantic-keAEYBDO-py3.10/lib/python3.10/site-packages/pydantic
               python version: 3.10.4 (main, Mar 13 2023, 19:44:25) [GCC 9.4.0]
                     platform: Linux-5.4.0-1105-azure-x86_64-with-glibc2.31
     optional deps. installed: ['typing-extensions']

MyPy Output
(mypy-pydantic-py3.10) @mdgilene ➜ /workspaces/codespaces-blank (main) $ mypy mypy_pydantic/model.py
mypy_pydantic/model.py:10: error: Unexpected keyword argument "field" for "Model"  [call-arg]
Found 1 error in 1 file (checked 1 source file)

I just downgraded mypy to 1.0.1 like you have and it looks like the error actually goes away and I am seeing what you are seeing. I am wondering then if this change in mypy 1.1.1 is causing issues with pydantic.
https://mypy-lang.blogspot.com/2023/03/mypy-111-released.html#:~:text=Support for dataclass_transform,recognized by mypy.

This release adds full support for the dataclass_transform decorator defined in PEP 681. This allows decorators, base classes, and metaclasses that generate a init method or other methods based on the properties of that class (similar to dataclasses) to have those methods recognized by mypy.

@mdgilene okay, I'm able to reproduce it now using mypy 1.1.1 in your repo. I'll see if I can figure out what the issue is

@mdgilene okay, it looks to me like this is fixed with pydantic==1.10.7 and mypy==1.1.1

Okay yes, upgrading to 1.10.7 does resolve the issue.

Okay, given that, I'm going to close this. (There were various issues with the mypy plugin that were resolved in the 1.10.7 release.) But please continue to report any issues with the plugin

Just tested this locally with poetry and python v3.10.10.
poetry init go with defaults
poetry add pydantic@1.10.7
poetry add mypy@1.2.0
create main.py containing:
from pydantic import BaseModel, Field

class Model(BaseModel):
    field: str = Field(alias="alias")

    class Config:
        allow_population_by_field_name = True

Model(alias="")
Model(field="")

Running poetry run mypy . fails with the following:

Adding the pydantic plugin by creating mypy.ini with the following content:
[mypy]
plugins = pydantic.mypy

Running poetry run mypy .fails with the following:

It seems to me that this is still an issue. Could we re-open this?

@andersfylling as far as I can tell, the code in your example is behaving as expected.
Keep in mind that without the use of the mypy plugin, we are largely dependent on the way mypy handles dataclass_transform.
One thing we may be able to do is add another keyword argument besides alias that does the same thing, but would be ignored by @dataclass_transform, and you could use that keyword argument if you don't want the type-checked signature to use the alias. That would be a pretty straightforward change, though we'd need to establish consensus on the name of that keyword argument, etc.
If you want to generate an overloaded signature for __init__ (where one overload uses all aliases and the other uses none), I think that would be more work than I'm willing to put into V1 at this stage, though I would review a PR. Even for v2, I would probably prefer the option above of letting you use a different keyword argument to Field to specify whether you want the alias to affect the __init__ signature or not.
Either way, I think it would be better to create a new issue to keep it focused on the requested feature, rather than continuing in this issue (which was about a different problem that was resolved in 1.10.7).

This is the same issue as described in the thread. While it works at runtime, mypy complains. The message above fails when using the mypy plugin and without the mypy plugin. Exactly the same error.

If Config.allow_population_by_field_name=True, the generated signature will use the field names, rather than aliases.

As this doesn't work,  I'd suggest this bug report to be reconsidered as open. It's currently blocking us from upgrading pydantic and mypy.

from pydantic import BaseModel, Field

class Model(BaseModel):
    field: str = Field(alias="alias")

    class Config:
        allow_population_by_field_name = True

Model(alias="")
Model(field="")

@andersfylling, I think what you're missing is that given the above, only 1 variation can ever be valid at a given time.
Config.allow_population_by_field_name just switches which one is valid for mypy when used with the plugin.

Just FYI for anybody ending here on google search.
[mypy]
plugins = pydantic.mypy, pydantic.v1.mypy

authors: andersfylling cdce8p dmontagu hramezani jonathanslenders mdgilene pkit

repo: pydantic/pydantic


text:
Properties order v1 -> v2

related https://github.com/tiangolo/fastapi/discussions/9709#discussioncomment-6239747
class Payload(BaseModel):
    username: Optional[str] = None
    password: Optional[str] = None

when I generate schema the order of the fields changes:
{
  "properties": {
    "password": {
         ...
    },
    "username": {
         ...
    }
  },
  "title": "Payload",
  "type": "object"
}

(first goes password, instead of username)
this was working fine on v1
Python, Pydantic & OS Version
pydantic version: 2.0b3
        pydantic-core version: 0.39.0 release build profile
                 install path: /private/tmp/fastapitest/.venv/lib/python3.9/site-packages/pydantic
               python version: 3.9.15 (main, Dec  6 2022, 21:16:57)  [Clang 14.0.0 (clang-1400.0.29.202)]
                     platform: macOS-13.4-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

I don't think it's a bug per say, since it was introduced on purpose on #6043, but since we don't provide a way to customize it, I think we can revert it.
What do you think @adriangb ?

I don't think it's a bug per say

Yeah, technically it is not , but from UI perspective it's a nightmare :D

I think that the intent in #6043 was to be deterministic and decouple the attributes order in JSON Schema from internal implementation changes. IMO, this doesn't contradict preserving user-defined attributes order.

authors: Kludex lig vitalik


text:
'ValidatedEmail' object has no attribute 'normalized'

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
For pydantic 2.3.0 and pydantic-core 2.6.3 getting this error:
AttributeError: 'ValidatedEmail' object has no attribute 'normalized'
Example Code
from pydantic import BaseModel, EmailStr

class A(BaseModel):
    m: EmailStr

A.model_validate({"m": "test@example.com"})

Python, Pydantic & OS Version
pydantic version: 2.3.0
        pydantic-core version: 2.6.3
          pydantic-core build: profile=release pgo=true
                 install path: /home/bKash.com_mallik.vivasoft/.cache/pypoetry/virtualenvs/bkash-payroll-disbursement-mw-nrLUCnFp-py3.10/lib/python3.10/site-packages/pydantic
               python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
                     platform: Linux-5.19.0-50-generic-x86_64-with-glibc2.35
     optional deps. installed: ['email-validator', 'typing-extensions']

Selected Assignee: @dmontagu

You most probably have an unsupported email-validator version installed. Try:
pip install -U email-validator

You most probably have an unsupported email-validator version installed. Try:
pip install -U email-validator

It's working. Thanks @Viicos

Thanks @Viicos for helping

authors: Viicos akashmallik247 hramezani


text:
Using a default Field will crash when the type annotation references a class of the same name as the field

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
Traceback (most recent call last):
  File "/home/louis/lab/tubeulator/pascalify/Journey.py", line 10, in <module>
    class X(BaseModel):
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 178, in __new__
    set_model_fields(cls, bases, config_wrapper, types_namespace)
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 452, in set_model_fields
    fields, class_vars = collect_model_fields(cls, bases, config_wrapper, types_namespace, typevars_map=typevars_map)
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/_internal/_fields.py", line 122, in collect_model_fields
    type_hints = get_cls_type_hints_lenient(cls, types_namespace)
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/_internal/_typing_extra.py", line 212, in get_cls_type_hints_lenient
    hints[name] = eval_type_lenient(value, globalns, localns)
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/_internal/_typing_extra.py", line 224, in eval_type_lenient
    return eval_type_backport(value, globalns, localns)
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/_internal/_typing_extra.py", line 240, in eval_type_backport
    return typing._eval_type(  # type: ignore
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/typing.py", line 327, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/typing.py", line 693, in _evaluate
    type_ = _type_check(
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/typing.py", line 176, in _type_check
    raise TypeError(f"{msg} Got {arg!r:.100}.")
TypeError: Forward references must evaluate to types. Got FieldInfo(annotation=NoneType, required=False).

This is a very odd and specific bug, I've tried changing aspects of it and have whittled it down to the reproducible essence, as in this issue's title.

It only occurs when you use Field, even when equivalent to not using Field (i.e. = Field(None) rather than = None)
It only occurs when the field name matches the [forward referenced] field type name
It is not affected by stringifying the type

This isn't some contrived example, it came up when trying to codegen for the TfL API.
I think I can work around this, it's a shame that I can't use aliases in this way though.
Example Code
from __future__ import annotations

from pydantic import BaseModel, Field

class Aaa(BaseModel):
    b: str = None

class X(BaseModel):
    Aaa: Aaa = Field(None)  # This crashes
    # Aaa: "Aaa" = Field(None)  # Likewise
    # Aaa: Aaa = None  # This is fine

Python, Pydantic & OS Version
pydantic version: 2.6.4
        pydantic-core version: 2.16.3
          pydantic-core build: profile=release pgo=true
                 install path: /home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic
               python version: 3.10.14 (main, Mar 21 2024, 16:24:04) [GCC 11.2.0]
                     platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
             related packages: typing_extensions-4.10.0
                       commit: unknown

@lmmx,
Thanks for the detailed bug report. I know that @Viicos worked on a PR recently to disallow / warn for cases similar to yours above. @Viicos, any idea why that wouldn't take effect for this case (when running on main, where your change is active).

Aaa: Aaa = Field(None) should have been caught by https://github.com/pydantic/pydantic/pull/8243, however I don't think it works with the __future__ import. I'll try to take a look at it and see if it can be handled as well.
Aaa: Aaa = None seems to be the same as https://github.com/pydantic/pydantic/pull/8243#issuecomment-1831900199, and can be very misleading as the Aaa annotation will take the value of None.
@lmmx My suggestion is to follow the alternatives mentioned here. I know this isn't perfect but Pydantic is limited by the annotation evaluation of Python (assignments are evaluated before annotations, meaning when Python tries to evaluate Aaa, it uses the assigned value in the class as it is in the same class scope). I'll also note that type checkers won't support the annotation as well (see this screenshot).
To catch this kind of issues, I wrote a Flake8 plugin (rule can be found here). This detection could be implemented in Pydantic but it would require parsing the AST which isn't an ideal solution considering it would slow down model creation.

Thanks for the prompt triage @sydney-runkle and explanation @Viicos ! Best team :muscle:
Due to the specific [imposed] requirements I have in this project:

The code is codegen'd, some keys in the JSON it parses are reserved words in Python so would be illegal [in the AST] to leave as model field names, hence must be aliased from camelCase to PascalCase
There are cyclic inter-model references (so forward refs must be stringified with the future annotations import)

mean that I can't use the alternatives but good to know!
I've opted to go with an AliasGenerator for now which seems to do the trick for my needs, still wanted to report it back to base :-) Plugin looks great too, I use flake8 all the time (even though my pre-commit is in ruff now so cool to see the plans for that!)
I went back and checked your warning that a stringified annotation - one example is the Identifier model I generated here which includes such a field:
class Identifier(BaseModel):
    Crowding: Crowding = None

Indeed you were right :face_exhaling:
>>> from tubeulator.generated.Journey import Identifier
>>> Identifier.model_validate({})
Identifier(Id=None, Name=None, Uri=None, FullName=None, Type=None, Crowding=None, RouteType=None, Status=None)
>>> Identifier.model_validate({"crowding": {}})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/louis/miniconda3/envs/tubeulator/lib/python3.10/site-packages/pydantic/main.py", line 509, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for Identifier
crowding
  Input should be None [type=none_required, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.6/v/none_required

Thanks for the specific warning there, I thought I was good to go ahead !

I use flake8 all the time (even though my pre-commit is in ruff now so cool to see the plans for that!)

My plan is to implement these rules at Ruff at some point (but still need to a bit more familiar with Rust first). But yeah aliases seems like a good option

@Viicos, do you think this still fits within the scope of v2.10?

I'd say we should close it given we can't really do much here as said in the previous comments

authors: Viicos lmmx sydney-runkle


text:
DOC: HDF access limitations and Comparison of SQL/HDF

I think that the documentation for HDFStore could be more explicit about the limitations of accessing data from an HDF file. For example, it is not at all clear from the docs that when selecting several columns of data from a saved table that the entire table needs to be read into memory. Specifically, one might be tempted to do the following to select 3 columns from a very large table:
df = h5.select('test', columns=['col2', 'col7', 'col128'])

However, that process will potentially read a very large portion (if not all) of the table into memory, which is obviously a problem if the table doesn't fit into memory. Instead, one should really do something like this:
columns = ['col2', 'col7', 'col128']
df = pd.concat([tmp for tmp in h5.select('test', columns=columns, chunksize=50000)])

I think it also might be helpful to include some sort of comparison of HDF and SQL solutions for table storage. Maybe we could solicit use-cases from the mailing list for successful applications of one or the other. I've been using HDF files for several years, though never before with tables containing many, many columns, which tripped me up in my example above. I almost turned to SQLite, because the above selection is very simple and has a reasonable memory usage footprint:
engine = sqlalchemy.create_engine('sqlite:///test.db')
df = pd.read_sql_query('select col2,col7,col128 from test', engine)

However, I profiled this and it is several times slower, so I'll probably stick with HDF because I am much more familiar with the format and prefer the access speed for large tables.
Anyway, just a couple of thoughts. I'm happy to try to put some of this into the docs, if people think it is appropriate.

you should have a look here: http://pandas.pydata.org/pandas-docs/stable/io.html#performance-considerations
as to your first point - this is a known limitation of row oriented tables and explicit in the PyTables docs - If it's not in the caveats section you could certainly add it

@jreback I will look into adding something about this limitation in the caveats section and a couple of notes in other sections. However, I looked around in the PyTables documentation, and I couldn't find an explicit reference to this issue with HDF files. It would be nice to include a link if you know of one.
Regarding the comparisons issue... I had looked at that performance results page. However, the difference is not so striking if you only need to select a small subset of rows from a larger-than-memory data set. For example, on my machine I found that HDF outperforms SQL by a factor of ~3 when selecting a subset of rows from a large table:
http://nbviewer.ipython.org/gist/rnelsonchem/e6fa80e71365a0981c59
Also, the SQL selection example is pretty straight forward, but I had to do a bit of tinkering to get the HDF example to work without overloading my memory. Still, a factor of three is a factor of three.
I guess I wasn't thinking of read/write timing performance comparison, exactly, but I thought there might be use cases where one solution has advantages over the other. Pandas has nice support for both options, but from the docs/performance results, I get the impression that HDF is preferred in every case. I haven't worked with SQL much, so I wasn't sure if there were cases that this wasn't true. (For example, it seems like things like merging/joining require a bit of extra work with HDF.) If so, it might be useful for new users to have some guidance on successful use cases for the two solutions. Perhaps as another section in the cookbook alongside the HDF/SQL performance comparison or even a link to a preexisting page. I did see this discussion, which is a start:
https://groups.google.com/forum/#!topic/pydata/S3kLxyrizkI
Just a thought.

see the read method here: http://pytables.github.io/usersguide/libref/structured_storage.html#table-methods-reading
You are dealing with a row-oriented structure so by definition it will read in all columns (exception for a single column using the special method).
You may want to look at using bcolz a columnar store. blaze and into provide easy access to these types of structures (as well as conversion between many other types).
Generally HDF5 is vastly superior to anything SQL-ike when modify/update is not a concern and data is mostly read/append only. You have to be careful about the limitations inherent in the data storage formats though. Pandas provides access to these features, but users are in charge of the data arangements.
So, additional docs are always helpful, as an update to the SQL comparison. However, pandas can't try (or should) provide guidance in all but the common cases.

authors: jreback rnelsonchem

repo: pydata/pandas


text:
BUG: DataFrame.to_string with formatters, header and index False

closes #13032

[x] tests added / passed - added test specific to format bug
[x] passes pep8radius master --diff
[x] whatsnew entry - not needed

Found this bug experimenting with formatters. First pull request to pandas, but I believe guidelines are quite clear. I can explain what was happening in more detail if that is necessary.

is this related to #13032 ?
can you show an example of before / after

Turns out is exactly related to #13032. I did see this issue, but didn't make the connection initially. In my testing, the lines should be removed not modified. To be honest, I didn't understand what these lines accomplished when I was reviewing the code originally.
Minimal code to reproduce (stripped down from #13032, easier to see with formatters):
>>> import pandas as pd
>>> frame = pd.DataFrame(data={0: 0, 1: 0}, index=[0])
>>> formatter = lambda x: '{:10.3f}'.format(x)
>>> print(frame.to_string(index=False, header=False))

Before (adding slashes to make space counting easier):
\ \ \ \ 0.000\ \ \ \ \ \ 0.000
After:
\ \ \ \ \ 0.000\ \ \ \ \ \ 0.000
Output from PR for #13032 (without formatter):
>>> print(df.to_string(index=False))
      one       two     three
 1.722364  0.846757  0.094394
-0.578834  0.836656  0.665414
 0.345460  1.782786  1.760175

Output from PR for #13032 (with formatter 10.3f, slashes added once for column count):
>>> print(df.to_string(index=False,formatters=[formatter,formatter,formatter]))
       one        two      three
\ \ \ \ \ 1.722      0.847      0.094
    -0.579      0.837      0.665
     0.345      1.783      1.760

I will add the PR and Issues numbers plus make corrections shortly.

[Current coverage][cc-pull] is 84.23%

Merging [#13350][cc-pull] into [master][cc-base-branch] will increase coverage by <.01%

@@             master     #13350   diff @@
==========================================
  Files           138        138
  Lines         50713      50721     +8
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits          42715      42723     +8
  Misses         7998       7998
  Partials          0          0

Powered by Codecov. Last updated by [ce56542...73d7b7e][cc-compare]
[cc-base-branch]: https://codecov.io/gh/pydata/pandas/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/pydata/pandas/compare/ce56542d1226adf8b3439c51f0c34b49dd53bb28...73d7b7edd24060e4f59ff2b11a156be36b497d75
[cc-pull]: https://codecov.io/gh/pydata/pandas/pull/13350?src=pr

Additionally, the failed Travis CI checks don't appear to be related to my subtrations/additions. Lastly, the unit test must use assert because of the location in the code. Should it be moved?

@barrymoo can you also read thru the comments on that issue (and the referenced one) and see if covering bases.

@jreback Of course,
From comment https://github.com/pydata/pandas/issues/13032#issuecomment-222835024:
Before:
Name    Value
                                  Short        1
                                 Longer  9374518
Much Longer name to the Max -----------    32432

After (desired output, correct?):
                                    Name    Value
                                   Short        1
                                  Longer  9374518
 Much Longer name to the Max -----------    32432

I think the others comments would be unaffected. To be fair though, I am very new to pandas.

I see what you mean now. From #11833:
Code:
>>> import pandas as pd
>>> df = pd.DataFrame({'a':range(5)})
>>> df.to_string(index=False)
u' a\n 0\n 1\n 2\n 3\n 4'
>>> formatter = lambda x: '{:1d}'.format(x)
>>> df.to_string(formatters=[formatter], index=False)
>>> u'a\n0\n1\n2\n3\n4'

I should be able to fix this issue quickly.

After looking at this for some time. We are adding 2 lines of code to get around a very minor issue and which breaks more realistic use cases. formatter fixes #11833 with existing code (see above). This is a trade-off, let me know what you think.

A different perspective: where are those spaces coming from in the first place? (I will try to track this down)

Lastly, for now, there is checks failing for a test that was made to test the issues linked above. Ideally, I will clean all of this up.

Code:
#!/usr/bin/env python
import pandas as pd
import numpy as np

# Test 1
frame = pd.DataFrame(data={0: 0, 1: 0}, index=[0])
formatter = lambda x: '{:10.3f}'.format(x)
string = frame.to_string(index=False, header=False)
print('--> Begin Test 1 <--')
print(string)
print('-->  End Test 1  <--')

# Test 2
df = pd.DataFrame(np.random.randn(3, 3), columns=['one', 'two', 'three'])
string = df.to_string(index=False)
print('--> Begin Test 2 <--')
print(string)
print('-->  End Test 2  <--')

# Test 3
df = pd.DataFrame({'a':range(5)})
string = df.to_string(index=False)
print('--> Begin Test 3 <--')
print(string)
print('-->  End Test 3  <--')

# Test 4
NAMES = ['Short', 'Longer', 'Much Longer name to the Max -----------']
VALUES = [1, 9374518, 32432]
d = pd.DataFrame({'Name': NAMES, 'Value': VALUES})
string = d.to_string(index=False)
print('--> Begin Test 4 <--')
print(string)
print('-->  End Test 4  <--')

Produces:
$ python quick-test.py
--> Begin Test 1 <--
0 0
-->  End Test 1  <--
--> Begin Test 2 <--
      one       two     three
-0.117275 -0.410192 -2.170441
 0.194766  0.521318  0.936951
-0.923841 -1.829388  1.078478
-->  End Test 2  <--
--> Begin Test 3 <--
a
0
1
2
3
4
-->  End Test 3  <--
--> Begin Test 4 <--
                                    Name   Value
                                   Short       1
                                  Longer 9374518
 Much Longer name to the Max -----------   32432
-->  End Test 4  <--

I believe this is the desired output. Unfortunately, this code fails about 20 tests. Hopefully, it is because the spacing in the expected output has changed slightly.

About 30 failed checks from nosetests pandas/tests/formats/test_format.py. I will checking the expected outputs this weekend. Note, I edited https://github.com/pydata/pandas/pull/13350#issuecomment-223483638 to include the new code edits.

I have created https://github.com/barrymoo/pandas-pr-13350-supplement to document the test failures (and eventually generate new expected strings). This is a work-in-progress.

I have added some more tests to the supplement. There is one test which I am having some difficulty with, starts: https://github.com/barrymoo/pandas-pr-13350-supplement/blob/master/tests.py#L474. I think the frame is supposed to overcome the terminal size, but it doesn't.

Hey @jreback I finished my supplement but I need some opinions about output formatting especially concerning the test_*_east_asian* tests (I don't know what these should look like). Is there anyone else we could pull in to review this? That way I can fix the rest of the formatting concerns and fix everything with one PR.
For the supplement, clone it, activate the dev environment (tested with 2 & 3, but have not examined diffs of the output), run python tests.py, and review formatting. Submit issues to the other repo with concerns of specific tests.

@barrymoo not sure what you mean by supplement. simply update this PR, comments can just be done here.
cc @sinhrks

@barrymoo I live in Japan and am willing to check the output format amd provide test cases:)

@jreback I ripped out the failing tests so one can easily print the results out on the command line. That way I can get some input from the community on how people want things formatted and make additional changes.

Here's a great example for why I need the supplement. For test_datetimelike_frame, my changes lead to the following output
                          dt  x
0  2011-01-01 00:00:00-05:00  1
1  2011-01-01 00:00:00-05:00  2
..                       ... ..
8                        NaT  9
9                        NaT 10

But, do you like:
                          dt  x
 0 2011-01-01 00:00:00-05:00  1
 1 2011-01-01 00:00:00-05:00  2
..                       ... ..
 8                       NaT  9
 9                       NaT 10

or...
                         dt  x
0 2011-01-01 00:00:00-05:00  1
1 2011-01-01 00:00:00-05:00  2
.                       ... ..
8                       NaT  9
9                       NaT 10

I can easily generate all of these outputs. Or, would you rather I pick what I luck and get all of the tests working.

I also worked on this; getting all of the tests to pass afterward is a nightmare. It looks like this change removes the leading space on integers but leaves it on floats. Is that true?

@evanpw it's very time consuming. I am still working through all of the tests, but I don't have a ton of free time. If you're looking at all positive numbers there is an extra space for the nonexistent "-" sign. There is an example in one of the above comments.

After this change there won't be an extra leading space for a column of positive integers, but there will still be one for a column of positive floats, right?

That's correct, but I can likely fix that too. Again the majority of this work is fixing the tests.

can you rebase / update?

I think I did that correctly. I really just don't have time to finish the tests.

source: gharchive/pull-request

authors: barrymoo codecov-io evanpw jreback sinhrks


text:
mypy issues on main

The pre-commit is reporting some mypy issues on the main branch:
xarray/core/formatting.py:193: error: No overload variant of "max" matches argument types "signedinteger[Any]", "int"  [call-overload]
xarray/core/formatting.py:193: note: Possible overload variant:
xarray/core/formatting.py:193: note:     def [SupportsLessThanT <: SupportsLessThan] max(__arg1, SupportsLessThanT, SupportsLessThanT, *_args: SupportsLessThanT, key: None = ...) -> SupportsLessThanT
xarray/core/formatting.py:193: note:     <5 more non-matching overloads not shown>
xarray/core/indexes.py:166: error: Incompatible types in assignment (expression has type "Union[dtype[object_], dtype[Any], dtype[void]]", variable has type "dtype[object_]")  [assignment]
xarray/core/computation.py:607: error: Argument 1 to "append" of "list" has incompatible type "None"; expected "slice"  [arg-type]
xarray/core/accessor_str.py:277: error: <nothing> not callable  [misc]
Found 4 errors in 4 files (checked 143 source files)

does anyone know how to fix those?

That might be unrelated to the change to main. Probably the numpy version changed from 1.20.3 to 1.21.0. However, its difficult to know as the pre-commit step does not show the installed versions...

We could pin the pre-commit if needed... Maybe at least temporarily if no one has capacity to fix these now?

Sorry — @mathause , you're too fast for me!

authors: keewis mathause max-sixty

repo: pydata/xarray


text:
to_zarr raises ValueError: Invalid dtype with mode='a' (but not with mode='w')

What happened?
A dataset in which a data variable has dtype='|S35' can be written to zarr without error as follows
import xarray as xr
import numpy as np

data = np.zeros((2, 3), dtype='|S35')
ds = xr.DataArray(data, name='foo').to_dataset()
ds.to_zarr('test.zarr', mode='w')

Changing the value of mode from 'w' to 'a', raises ValueError: Invalid dtype for data variable:
!rm -rf test.zarr
ds.to_zarr('test.zarr', mode='a')

Full Traceback
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 ds.to_zarr('test.zarr', mode='a')

File ~/miniconda3/envs/pangeo-forge-recipes/lib/python3.9/site-packages/xarray/core/dataset.py:2036, in Dataset.to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
   2033 if encoding is None:
   2034     encoding = {}
-> 2036 return to_zarr(
   2037     self,
   2038     store=store,
   2039     chunk_store=chunk_store,
   2040     storage_options=storage_options,
   2041     mode=mode,
   2042     synchronizer=synchronizer,
   2043     group=group,
   2044     encoding=encoding,
   2045     compute=compute,
   2046     consolidated=consolidated,
   2047     append_dim=append_dim,
   2048     region=region,
   2049     safe_chunks=safe_chunks,
   2050 )

File ~/miniconda3/envs/pangeo-forge-recipes/lib/python3.9/site-packages/xarray/backends/api.py:1406, in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options)
   1391 zstore = backends.ZarrStore.open_group(
   1392     store=mapper,
   1393     mode=mode,
   (...)
   1402     stacklevel=4,  # for Dataset.to_zarr()
   1403 )
   1405 if mode in ["a", "r+"]:
-> 1406     _validate_datatypes_for_zarr_append(dataset)
   1407     if append_dim is not None:
   1408         existing_dims = zstore.get_dimensions()

File ~/miniconda3/envs/pangeo-forge-recipes/lib/python3.9/site-packages/xarray/backends/api.py:1301, in _validate_datatypes_for_zarr_append(dataset)
   1292         raise ValueError(
   1293             "Invalid dtype for data variable: {} "
   1294             "dtype must be a subtype of number, "
   (...)
   1297             "object".format(var)
   1298         )
   1300 for k in dataset.data_vars.values():
-> 1301     check_dtype(k)

File ~/miniconda3/envs/pangeo-forge-recipes/lib/python3.9/site-packages/xarray/backends/api.py:1292, in _validate_datatypes_for_zarr_append.<locals>.check_dtype(var)
   1283 def check_dtype(var):
   1284     if (
   1285         not np.issubdtype(var.dtype, np.number)
   1286         and not np.issubdtype(var.dtype, np.datetime64)
   (...)
   1290     ):
   1291         # and not re.match('^bytes[1-9]+$', var.dtype.name)):
-> 1292         raise ValueError(
   1293             "Invalid dtype for data variable: {} "
   1294             "dtype must be a subtype of number, "
   1295             "datetime, bool, a fixed sized string, "
   1296             "a fixed size unicode string or an "
   1297             "object".format(var)
   1298         )

ValueError: Invalid dtype for data variable: <xarray.DataArray 'foo' (dim_0: 2, dim_1: 3)>
array([[b'', b'', b''],
       [b'', b'', b'']], dtype='|S35')
Dimensions without coordinates: dim_0, dim_1 dtype must be a subtype of number, datetime, bool, a fixed sized string, a fixed size unicode string or an object

What did you expect to happen?
I would expect the behavior of  mode='w' and mode='a' to be consistent as regards dtypes of data variables.
Minimal Complete Verifiable Example
See What Happened? section above
Relevant log output
See What Happened? section above
Anything else we need to know?
No response
Environment
INSTALLED VERSIONS
------------------
commit: None
python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:28:27)
[Clang 11.1.0 ]
python-bits: 64
OS: Darwin
OS-release: 21.0.1
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.1
libnetcdf: 4.8.1

xarray: 2022.3.0
pandas: 1.4.1
numpy: 1.22.2
scipy: 1.8.0
netCDF4: 1.5.8
pydap: installed
h5netcdf: 999
h5py: 3.6.0
Nio: None
zarr: 2.11.0
cftime: 1.6.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.10
cfgrib: 0.9.8.5
iris: None
bottleneck: None
dask: 2022.02.1
distributed: 2022.2.1
matplotlib: 3.5.1
cartopy: None
seaborn: None
numbagg: None
fsspec: 2022.02.0
cupy: None
pint: None
sparse: None
setuptools: 59.8.0
pip: 22.0.4
conda: None
pytest: 6.2.5
IPython: 8.1.1
sphinx: None

cc @rabernat

The relevant code is here
https://github.com/pydata/xarray/blob/d293f50f9590251ce09543319d1f0dc760466f1b/xarray/backends/api.py#L1405-L1406
and here
https://github.com/pydata/xarray/blob/d293f50f9590251ce09543319d1f0dc760466f1b/xarray/backends/api.py#L1280-L1298
What I don't understand is why different validation is needed for the append scenario than for the the write scenario. @shoyer worked on this in #5252, so maybe he has some ideas.

I just ran into this today as well. I am trying to add a dimensionless variable to an existing Zarr store, to help with CF-compliance (if exporting to netCDF), and I ran into this issue. The dtype of my variable is '|S1', and the error message is printed below:
ValueError: Invalid dtype for data variable: <xarray.DataArray 'crs' ()> array(b'', dtype='|S1')

The data type restriction here seems to date back to the original PR adding support for appending. I turned up this comment that seems to summarize the motivation for this check:
https://github.com/pydata/xarray/pull/2706#issuecomment-502481584
I think the original issue was that appending a fixed-width string could be a problem if the fixed-width does not match the width of the existing string dtype stored in Zarr.
This obviously doesn't apply in this case, because you are adding an entirely new variable. So I guess the check could be removed in that case.

It seems like what we really want to do is verify that the datatype of the appended data matches the data type on disk.

So it looks like changing
https://github.com/pydata/xarray/blob/d293f50f9590251ce09543319d1f0dc760466f1b/xarray/backends/api.py#L1280-L1301
to
def _validate_datatypes_for_zarr_append(store, dataset):
    """DataArray.name and Dataset keys must be a string or None"""

    def check_dtype(vname):
        store_dtype = store.get_variables()[vname].dtype
        dataset_dytpe = dataset[vname].dtype
        if not  store_dtype == dataset_dytpe:
            raise ValueError(
                f"Mismatched dtypes for variable {vname} between Zarr store on disk "
                f"and dataset to append. Store has dtype {store_dtype} but dataset to "
                f"append has dtype {dataset_dtype}"
            )

    for vname in dataset.data_vars:
        check_dtype(vname)

could work?

authors: cisaacstern kmsampson rabernat shoyer


text:
occasional segfaults on CI

The upstream-dev CI currently fails sometimes due to a segfault (the normal CI crashes, too, but since we use pytest-xdist we only get a message stating "worker x crashed").
I'm not sure why, and I can't reproduce locally, either. Given that dask's local scheduler is in the traceback and the failing test is test_open_mfdataset_manyfiles, I assume there's some issue with parallel disk access or the temporary file creation.
log of the segfaulting CI job
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/runner/work/xarray/xarray
configfile: setup.cfg
testpaths: xarray/tests, properties
plugins: env-0.8.1, xdist-3.3.1, timeout-2.1.0, cov-4.1.0, reportlog-0.1.2, hypothesis-6.75.6
timeout: 60.0s
timeout method: signal
timeout func_only: False
collected 16723 items / 2 skipped

xarray/tests/test_accessor_dt.py ....................................... [  0%]
........................................................................ [  0%]
........................................................................ [  1%]
........................................................................ [  1%]
...............................                                          [  1%]
xarray/tests/test_accessor_str.py ...................................... [  1%]
........................................................................ [  2%]
........................................................................ [  2%]
.............................................                            [  3%]
xarray/tests/test_array_api.py ...........                               [  3%]
xarray/tests/test_backends.py ........................X........x........ [  3%]
...................................s.........................X........x. [  3%]
.........................................s.........................X.... [  4%]
....x.......................................X........................... [  4%]
....................................X........x.......................... [  5%]
.............X.......................................................... [  5%]
....X........x....................x.x................X.................. [  5%]
x..x..x..x...................................X........x................. [  6%]
...x.x................X..................x..x..x..x..................... [  6%]
..............X........x....................x.x................X........ [  7%]
..........x..x..x..x.......................................X........x... [  7%]
...........................................X........x................... [  8%]
...ss........................X........x................................. [  8%]
.................X........x............................................. [  8%]
..X........x.............................................X........x..... [  9%]
............................................X........x.................. [  9%]
.........................................................X........x..... [ 10%]
......................................................................X. [ 10%]
.......x................................................................ [ 11%]
Fatal Python error: Segmentation fault

Thread 0x00007f9c7b8ff640 (most recent call first):
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00007f9c81f1d640 (most recent call first):
  File "/home/runner/work/xarray/xarray/xarray/backends/file_manager.py", line 216 in _acquire_with_cache_info
  File "/home/runner/work/xarray/xarray/xarray/backends/file_manager.py", line 198 in acquire_context
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 392 in _acquire
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 398 in ds
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 336 in __init__
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 389 in open
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 588 in open_dataset
  File "/home/runner/work/xarray/xarray/xarray/backends/api.py", line 566 in open_dataset
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 121 in _execute_task
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 224 in execute_task
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 238 in <listcomp>
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 238 in batch_execute_tasks
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f9c82f1e640 (most recent call first):
  File "/home/runner/work/xarray/xarray/xarray/backends/file_manager.py", line 216 in _acquire_with_cache_info
  File "/home/runner/work/xarray/xarray/xarray/backends/file_manager.py", line 198 in acquire_context
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 392 in _acquire
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 398 in ds
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 336 in __init__
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 389 in open
  File "/home/runner/work/xarray/xarray/xarray/backends/netCDF4_.py", line 588 in open_dataset
  File "/home/runner/work/xarray/xarray/xarray/backends/api.py", line 566 in open_dataset
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py", line 73 in apply
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py", line 121 in _execute_task
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 224 in execute_task
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 238 in <listcomp>
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 238 in batch_execute_tasks
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 953 in run
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f9ca575e740 (most recent call first):
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/threading.py", line 320 in wait
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/queue.py", line 171 in get
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 137 in queue_get
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py", line 500 in get_async
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/threaded.py", line 89 in get
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/base.py", line 595 in compute
  File "/home/runner/work/xarray/xarray/xarray/backends/api.py", line 1046 in open_mfdataset
  File "/home/runner/work/xarray/xarray/xarray/tests/test_backends.py", line 3295 in test_open_mfdataset_manyfiles
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/python.py", line 1799 in runtest
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 323 in _main
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/config/__init__.py", line 166 in main
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module>
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/runpy.py", line 86 in _run_code
  File "/home/runner/micromamba-root/envs/xarray-tests/lib/python3.10/runpy.py", line 196 in _run_module_as_main

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, numexpr.interpreter, bottleneck.move, bottleneck.nonreduce, bottleneck.nonreduce_axis, bottleneck.reduce, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, cftime._cftime, yaml._yaml, cytoolz.utils, cytoolz.itertoolz, cytoolz.functoolz, cytoolz.dicttoolz, cytoolz.recipes, xxhash._xxhash, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, numpy.linalg.lapack_lite, matplotlib._c_internal_utils, PIL._imaging, matplotlib._path, kiwisolver._cext, scipy._lib._ccallback_c, _cffi_backend, unicodedata2, netCDF4._netCDF4, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.h5r, h5py.utils, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5t, h5py._conv, h5py.h5z, h5py._proxy, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5o, h5py.h5l, h5py._selector, pyproj._compat, pyproj._datadir, pyproj._network, pyproj._geod, pyproj.list, pyproj._crs, pyproj.database, pyproj._transformer, pyproj._sync, matplotlib._image, rasterio._version, rasterio._err, rasterio._filepath, rasterio._env, rasterio._transform, rasterio._base, rasterio.crs, rasterio._features, rasterio._warp, rasterio._io, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, msgpack._cmsgpack, numcodecs.vlen, zstandard.backend_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._flinalg, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize.__nnls, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._statlib, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.cluster._vq, scipy.cluster._hierarchy, scipy.cluster._optimal_leaf_ordering, shapely.lib, shapely._geos, shapely._geometry_helpers, cartopy.trace, scipy.fftpack.convolve, tornado.speedups, cf_units._udunits2, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils (total: 241)
/home/runner/work/_temp/b3f3888c-5349-4d19-80f6-41d140b86db5.sh: line 3:  6114 Segmentation fault      (core dumped) python -m pytest --timeout=60 -rf --report-log output-3.10-log.jsonl

There are similar segfaults in an xncml PR: https://github.com/xarray-contrib/xncml/pull/48
Googling around suggest it is related to netCDF not being thread-safe and recent python-netcdf4 releasing the GIL.

If I'm reading the different issues correctly, that means this is a duplicate of #7079

Defering to #7079

authors: huard keewis max-sixty


text:
Docstring for Dataset.drop needs revision

The docstring for Dataset.drop indicates that the first argument, labels, should be a string indicating names of variables or index labels to drop. I'm not sure, but I'm guessing it either takes in several strings (in which case the docstring should say *labels), or it takes in either a string or an iterable (in which case the argument type should reflect this).

Thanks for pointing this out -- fixed!

authors: mcgibbon shoyer


text:
mfdataset - ds.encoding["source"] to retrieve filename not valid key

What happened?
Looking at the doc https://docs.xarray.dev/en/stable/generated/xarray.open_mfdataset.html

preprocess (callable(), optional) – If provided, call this function on each dataset prior to concatenation. You can find the file-name from which each dataset was loaded in ds.encoding["source"].

I expected to be able to use ds.encoding["source"] in my preprocess function to retrieve the filename. However I get
What did you expect to happen?
I expected the doc to be correct? unless I missed something trivial.
Minimal Complete Verifiable Example
def preprocess_xarray_no_class(ds):
    filename = ds.encoding["source"]
    ds = ds.assign(
        filename=("time"), [filename])
    )  # add new filename variable with time dimension
    return ds

ds = xr.open_mfdataset(
    fileset,
    preprocess=preprocess_xarray_no_class,
    engine='h5netcdf',
    concat_characters=True,
    mask_and_scale=True,
    decode_cf=True,
    decode_times=True,
    use_cftime=True,
    parallel=True,
    decode_coords=True,
    compat="equals",

)

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[ ] Complete example — the example is self-contained, including all data and the text of any traceback.
[ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.
[X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output
...
      1 def preprocess_xarray_no_class(ds):
----> 2     filename = ds.encoding["source"]
      3     ds = ds.assign(
      4         filename=("time",), [filename])
      5     )  # add new filename variable with time dimension

KeyError: 'source'

Anything else we need to know?
No response
Environment

INSTALLED VERSIONS
------------------
commit: None
python: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
python-bits: 64
OS: Linux
OS-release: 6.5.0-1023-oem
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: en_IE.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.3-development
xarray: 2024.6.0
pandas: 2.2.2
numpy: 1.26.4
scipy: 1.13.1
netCDF4: 1.7.1
pydap: None
h5netcdf: 1.3.0
h5py: 3.11.0
zarr: 2.18.2
cftime: 1.6.4
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.3.8
dask: 2024.6.0
distributed: 2024.6.0
matplotlib: 3.9.0
cartopy: None
seaborn: 0.13.2
numbagg: 0.8.1
fsspec: 2024.6.0
cupy: None
pint: None
sparse: None
flox: 0.9.7
numpy_groupies: 0.11.1
setuptools: 70.0.0
pip: 24.0
conda: None
pytest: 8.2.2
mypy: 1.10.0
IPython: 7.34.0
sphinx: None

this depends on what fileset is, unfortunately. If it contains a list of strings (filepaths, urls, or a glob), then yes, it not working might be a bug. If that's fsspec objects, though, we need #8923.

fileset is a list of s3_fs objects:
        s3 = boto3.client("s3")
        s3_fs = s3fs.S3FileSystem(anon=True)

        remote_files = [f"s3://{bucket_name}/{key}" for key in object_keys]

        fileset = [s3_fs.open(file) for file in remote_files]

then #8923 will fix this!

authors: keewis lbesnard