text:
backdoor.distance_matching: Problem with exact matching

I think there is a bug with the matching function.
When trying to insert an argument of exact matching - there are controls who stick to the cases even though there is no similarity in the 'exact' variables.
It can be seen that there is not a full balance in the sample created after the match,
and that in many cases there are also controls from the other groups.

Code:

from dowhy import CausalModel
import dowhy.datasets
import numpy as np
import pandas as pd

data = dowhy.datasets.linear_dataset(
    beta=10,
    num_common_causes=5,
    num_instruments=0,
    num_samples=10000,
    treatment_is_binary=True)

#### change W3 to 1/0 for exact matching:
data["df"] = data["df"].assign(W3 = lambda x: np.where(x.W3>0,1,0))

model = CausalModel(
    data=data["df"],
    treatment=data["treatment_name"],
    outcome=data["outcome_name"],
    graph=data["gml_graph"])

model.view_model()

identified_estimand = model.identify_effect()

estimate = model.estimate_effect(identified_estimand,
                                 target_units = 'att',
                                 method_name="backdoor.distance_matching",
                                 method_params={
                                 'exact_match_cols':['W3'],
                                 'num_matches_per_unit':2
                                 })

print(estimate)

df_m=[]
for i in range(df.shape[0]):
    if data["df"].iloc[i].v0==True:
        df_m.append( data["df"].iloc[[i]+ estimate.estimator.matched_indices_att[i] ] )
df_matched = pd.concat(df_m)

#####%% Compare covariate balancing before and after matching:
balance = (pd.concat([
    df_matched.groupby('v0').mean().T.drop(True,axis =1 ),
    data["df"].groupby('v0').mean().T],axis = 1 )
    #.reset_index()
    .query("index!='y'")
           )

balance.columns = ['matched controls','treatment','unmatched controls']
balance.plot()
plt.show()

res =[]
for i in range(df.shape[0]):
    if data["df"].iloc[i].v0==True:
        res.append(data["df"].iloc[[i]+ estimate.estimator.matched_indices_att[i] ].W3.value_counts().shape[0])

#### if > 1 then we have more than 1 W3 values inside each 'case' group of matches:
print(np.array(res).mean())
1.176697874869966

DoWhy 0.9.1

I opened a pull request with correction:
https://github.com/py-why/dowhy/pull/819


text:
Set minimum python version to 3.7

Close #252

Thanks for updating this! The PR looks ready to merge. I pushed one commit with a minor update to the installation guide, and removing an empty file that seemed unused.

source: gharchive/pull-request

authors: aperezhortal dnerini

repo: pySTEPS/pysteps

license_type: permissive

license_source: github-api


text:
Flip line endings in daqmx test

Fixes one of the travis failures.

I don't anderstand the patch.
What have change ?

authors: campagnola samuelgarcia

repo: pyacq/pyacq

license_source: bigquery


text:
Non-food items in food slots of Dig Site

Mod source
Factorio Mod Portal
Which mod are you having an issue with?

[X] pyalienlife
[ ] pyalternativeenergy
[ ] pycoalprocessing
[ ] pyfusionenergy
[ ] pyhightech
[ ] pyindustry
[ ] pypetroleumhandling
[ ] pypostprocessing
[ ] pyrawores

Operating system

=Windows 10

What kind of issue is this?

[ ] Compatibility
[ ] Locale (names, descriptions, unknown keys)
[ ] Graphical
[ ] Crash
[ ] Progression
[ ] Balance
[ ] Pypostprocessing failure
[X] Other

What is the problem?
A non-food item can be placed by inserter into food slot of Dig Site:

Steps to reproduce

Place Dig Site
Place inserter and chest
Put non-food item into chest

Additional context

Log file
No response

Thought this was already fixed. What version of pyAL are you using?

I believe that the newest one (no updated available via Factorio Mod portal...)

Hmm. Just checked again can confirm your issue, But it seems to be an API limitation. Using the API I can set a filter, but I cannot make a filter for multiple types of food. Not going to fix this. For now, make sure not to use non-food items on the digosaurs food belt.

source: gharchive/issue

authors: mrozpara notnotmelon

repo: pyanodon/pybugreports

license: Unlicense


text:
Focus events getting lost?

Hey, I need something to update when a widget looses focus and I can do that with something simple like
class CustomRangeSlider(QtWidgets.QSlider):
    def focusOutEvent(self, event):
        # Call the base class focusOutEvent method
        super().focusOutEvent(event)
        # Handle the focus loss here
        print("lost focus")

But if I try that with any superqt slider it looks like this signal is never sent? Or is superseded or intercepted by something?
I tried setting the Focus Policy using
rangeSlider.setFocusPolicy(Qt.FocusPolicy.ClickFocus)
and tried looking at events by using
class QLabeledRangeSlider_(QLabeledRangeSlider):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
    def event(self, ev: QEvent) -> bool:
        print("_"*20)
        print(f"{ev.type()}")
        print("_"*20)
        if ev.type() in [QEvent.Type.FocusIn, QEvent.Type.FocusOut]:
            print("Focused in/out")
        if ev.type() == QEvent.Type.WindowActivate:
            self.update()
        elif ev.type() in (QEvent.Type.HoverEnter, QEvent.Type.HoverMove):
            self._updateHoverControl(self._event_position(ev))
        elif ev.type() == QEvent.Type.HoverLeave:
            self._hoverControl = QtWidgets.QStyle.SubControl.SC_None
            lastHoverRect, self._hoverRect = self._hoverRect, QRect()
            self.update(lastHoverRect)
        return super().event(ev)

But I just don't understand what is going on or why the focus event wouldnt be showing up anywhere.
I'm really lost here if anyone could point me in the right direction I would really appreciate it.

hi and thanks for the clear issue.
Interestingly, your first example also doesn't work if you subclass from QAbstractSlider (which is what we subclass for our sliders):
class CustomRangeSlider(QtWidgets.QAbstractSlider):
    def focusOutEvent(self, event):
        # Call the base class focusOutEvent method
        super().focusOutEvent(event)
        # Handle the focus loss here
        print("lost focus")

so I need to look into the C code to see why it's different.

aha, it's this line right here:
https://github.com/qt/qtbase/blob/568ab62185bfa7876db4d8804101a8f946b04898/src/widgets/widgets/qslider.cpp#L46
will add it to our own labeled sliders.

#175 should fix it for you, but in the meantime, and until the next release, you should be able to fix it locally by calling:
slider.setFocusPolicy(QtCore.Qt.FocusPolicy.TabFocus)

on your slider instance

repo: pyapp-kit/superqt


text:
Simulation fails at high current inputs

Describe the bug
I used the Chen2020 chemistry and lithium_ion.DFN model. When I tried to change the current function to 25 A, (5C_rate), it took more than 30 mins to run and the output voltage curve was sharp "V". If I change the model to SPM, the simulation would fail to run. Based on the tutorial, I expected to see the voltage curve ends earlier than the simulated time frame. But it failed to have results if the current is high. Is there a range for the current?
To Reproduce
Steps to reproduce the behaviour:

change parameter_values["Current function [A]"] = 25
run

Expected behaviour
Terminal outputs for DFN model:
At t = 0.0453447, , mxstep steps taken before reaching tout.
At t = 0.0453447, , mxstep steps taken before reaching tout.
At t = 0.0453447, , mxstep steps taken before reaching tout.
Maximum number of decreased steps occurred at t=509.09090909090907. Try solving the model up to this time only or reducing dt_max (currently, dt_max=3.5458164923888056)
Terminal outputs for SPM model:
a\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\numpy\lib\function_base.py", line 488, in asarray_chkfinite
raise ValueError(
ValueError: array must not contain infs or NaNs

Hi @shant4! Note that the Chen2020 cell is an energy cell so in practice the maximum recommended C-rate is 1.5C. From the modelling point of view, at high C-rates you will encounter two main issues:

The discharge time is very short, and you will need to provide a shorter t_eval and shorter dt_max, otherwise the solver won't converge.
The electrolyte will deplete. For the DFN, this means it will stop when the electrolyte gets close to zero (hence it will stop before reaching the 2.5V cut-off). For the SPM, the simulations should run (once the dt_max is reduced) but the results will make no physical sense as they do not include any electrolyte effect.

This is a very interesting question that can be useful to other people, so I will convert it to a discussion.

repo: pybamm-team/PyBaMM


text:
[Bug]: Sign error in Ohmic loss variable?

PyBaMM Version
22.12
Python Version
3.8.10
Describe the bug
I noticed that X-averaged battery solid phase ohmic losses [V] is always smaller in magnitude than the individual electrode Ohmic losses X-averaged {domain} electrode ohmic losses [V], but I expected it to be the sum. I believe this is a sign error in the definition of delta_phi_s_av in base_battery_model, unless I am misunderstanding the purpose of this variable.
Here is an illustrative plot (sorry for not providing a more complete code example):

Steps to Reproduce
plt.plot(sol["Time [s]"].entries, 1e3 * sol["X-averaged positive electrode ohmic losses [V]"].entries / sol["Current [A]"].entries, label="PE")
plt.plot(sol["Time [s]"].entries, 1e3 * sol["X-averaged negative electrode ohmic losses [V]"].entries / sol["Current [A]"].entries, label="NE")
plt.plot(sol["Time [s]"].entries, 1e3 * sol["X-averaged battery solid phase ohmic losses [V]"].entries / sol["Current [A]"].entries, label="tot")
plt.legend()
plt.grid()

Relevant log output
No response

Can you check with the latest version whether this still persists? I made some changes to this recently. Also check https://github.com/pybamm-team/PyBaMM/blob/develop/examples/notebooks/plotting/plot_voltage_components.ipynb for the intended signs

Closing due to 30 days with no reply. Feel free to reopen if needed

authors: DavidMStraub tinosulzer


text:
Refs #361 Fix

Changes

Change exception from Runtime to ClassNotFound in importPythonModule. This is because the method itself throws ClassNotFound exception rather than Runtime

Removed the try catch block, This is because if a module doesn't exist in python too then why are we catching the ClassNotFoundException rather than simply throwing it. Main reason of issue#361

Testing

test_import successfully completed

Expected o/p for issue #361

Import tests are passing with these changes

It produces the expected #361 behavior

We need to throw ClassNotFound Exception rather than throwing runtime exception here

@ASP1234 could you perform a rebase with the latest master?
Cheers

Hi there! It looks like this PR might be dead, so we're closing it for now. Feel free to re-open it if you'd like to continue, or think about directing your efforts to https://github.com/beeware/briefcase or https://github.com/beeware/toga. Both of these have more active development right now. 😄

authors: ASP1234 goanpeca phildini

repo: pybee/voc


text:
Small integer preallocation optimization

This PR implements the preallocation and initialization of the integers [-5, 257) so that they can be reused. In general, this means that when the python code is parsed and an integer object is detected, if it is in this range we will not create a new Integer object but rather use the preallocated one.
Object construction is expensive memory and time-wise. We can see a performance gain even running a simple performance test like this one:
Pre-optimization
Running test_small_integers ...
  Elapsed time:  33.07889499713201  ms
  CPU process time:  289.66  ms

With optimization
Running test_small_integers ...
  Elapsed time:  23.559533001796808  ms
  CPU process time:  240.0899999999998  ms

(Of course there is always some variance between different runs, but from what I've seen there is consistent improvement.)

As a further piece of work I think we should consider restricting access to the Int constructor altogether. This is perhaps a small issue on the transpilation side, for example, when we have an instruction that creates an integer from a value inside the variable so we're not able to check if the integer is small and generate lookup instructions. However, the math operations on Int objects create new Int objects all the time. It should be fairly straightforward to refactor the code to check for small integers, if it's something we want to do.

Some updated numbers:
Running test_small_integers
Elapsed time:  122.01105412701145  sec
CPU process time:  0.0006999999999999784  sec
Running test_small_integers
Elapsed time:  128.66034104398568  sec
CPU process time:  0.0008010000000000517  sec
Running test_small_integers
Elapsed time:  122.50824150099652  sec
CPU process time:  0.00048599999999998644  sec
Running test_small_integers
Elapsed time:  122.89947666198714  sec
CPU process time:  0.0008429999999999827  sec
Running test_small_integers
Elapsed time:  122.97956180298934  sec
CPU process time:  0.00045000000000000595  sec

@patiences Do you have baseline numbers for the benchmark? The ticket only has the "progress" numbers...

@freakboy3742  Yes, of course! Here we go:
Pre-optimization
Running test_small_integers
Elapsed time:  144.74749750000774  sec
CPU process time:  0.00047200000000002795  sec
Running test_small_integers
Elapsed time:  143.55906505999155  sec
CPU process time:  0.0010949999999999571  sec
Running test_small_integers
Elapsed time:  149.2436827890051  sec
CPU process time:  0.0008159999999999834  sec
Running test_small_integers
Elapsed time:  146.150813241984  sec
CPU process time:  0.0008259999999999934  sec
Looks like about a 15% improvement.

python/testfile.class has also been inadvertently added.

That file needs to be stripped out (ideally, rolling back the commit where it was added; if you just >delete the file, it will still be in the commit history, and so every person who clones the repo will >download the file, even though it doesn't exist any more).

Really? That is so weird. Is that the case with .py files too (some of those got added and deleted at other points too, oops)? I actually am not sure how to revert just the specific commit (bb40b5b) -- I tried to rebase it without that commit locally but I'm not sure that's going to work...


text:
connections: Split to multiple files

Split too big and difficult to understand connections.py into multiple files.

@laurensvalk Want to move the line_handler out of PybricksHub since it is reused by the REPLHub as well.
Would you mind if the line_handler would write the files into the temp directory? (the location would be printed out on log creation)
On my Mac, the path the _file_begin_ my_log_file command would produce /var/folders/7s/bnq_fqm90hd6wnxxcfmgckfw0000gn/T/my_log_file-basq29em.log.

@laurensvalk Want to move the line_handler out of PybricksHub since it is reused by the REPLHub as well.
Would you mind if the line_handler would write the files into the temp directory? (the location would be printed out on log creation)
I would use tempfile.NamedTemporaryFile(mode="w+", prefix=line[path_start:].decode(), suffix=".log", delete=False) to generate the fie.
On my Mac, the path the _file_begin_ my_log_file command would produce /var/folders/7s/bnq_fqm90hd6wnxxcfmgckfw0000gn/T/my_log_file-basq29em.log.
See the separate commit I've made.

I haven't had the chance to check any details yet, but that said, I'd like to leave the created files in the same folder as the script that runs, at least for now. We use this extensively, so changing it would break a few things. This might be worth a separate discussion if you find that is getting in the way.

I haven't had the chance to check any details yet, but that said, I'd like to leave the created files in the same folder as the script that runs, at least for now. We use this extensively, so changing it would break a few things. This might be worth a separate discussion if you find that is getting in the way.

Let me think about an elegant way to have the line_handler in a separate function and yet allow a custom file location.

Let me think about an elegant way to have the line_handler in a separate function and yet allow a custom file location.

Is there a compelling reason not to just use the current working directory?

Is there a compelling reason not to just use the current working directory?

Ease of implementation really. With TMP dir the line_handler can just create the file. If we want to store the log file the way it currently is, the function needs to accept one more argument with a file name to create.
However, I understand why it's convenient to have the log stored along with the user program, so I'll make it work the same it is currently.

@dlech Tested the rxpy branch now. Not a single issue observed 🎉

authors: dlech laurensvalk lobodpav

repo: pybricks/pybricksdev

license: MIT


text:
Add tests for no isal installation

fixes #58
This restores the coverage after #57 . Except for the lines after except ImportError when isal is imported, but these are so simple (set two variables to None) that testing is not really needed.
EDIT: https://app.codecov.io/gh/pycompression/xopen/compare/59/changes#D1L42 These lines can probably not be recovered for coverage. Unless we start testing on 32-bit systems that will never support isal. Or architectures other than x86_64 and aarch64.

Nice!

authors: marcelm rhpvorderman

repo: pycompression/xopen


text:
Update deployment-with-docker.rst

Description
I edited the doc,i added a step " supervisorctl update " which is need to add the project to process group.
Rationale
After supervisorctl read i tried to start the project using supervisor start but it said
projectname : ERROR (no such process)
Use case(s) / visualization(s)

Thanks

authors: Gilbishkosma browniebroke

repo: pydanny/cookiecutter-django


text:
Fix for gulp watching on windows

Description
Fix for browsersync and file watchers not working correctly on windows. This uses a legacy polling method, so it registers events if the user is on windows (which I assume the windows? is meant to represent.
Fixes #2642 which I've encountered.
Rationale
There are some windows developers around, and cookiecutter-django is a great start for a project whatever OS you use.
Use case(s) / visualization(s)
Not much to visualize sadly, but I think it's a pretty simple PR to understand what it does :)

@Tusky For the CONTRIBUTOR's doc, RST hyperlinks work by having the `@Tusky`_ with the link to your GitHub

@Tusky For the CONTRIBUTOR's doc, RST hyperlinks work by having the @Tusky_ with the link to your GitHub. You can take a look below all the names in the table to know what I mean.

Thank you, forgot to add it at the bottom :(

authors: Andrew-Chen-Wang Tusky


text:
Basic fix of GenericModel cache to detect order of arguments in Union models

Change Summary
Basic fix of GenericModel cache to include the order of arguments in Union models into the cache key.
This fixes the basic problem described in issue #4474, so that e.g. Model[Union[float, int]] is not used as a
blueprint for Model[Union[float, int]].  Added tests for basic and nested versions of the problem. Fixing the
nested problem depends on the fixing of the almost identical issue described in
https://github.com/python/cpython/issues/86483 in CPython. The test was thus skipped, but kept for
documentation.
Related issue number
fix #4474
Checklist

[ x ] Unit tests for the changes exist
[ x ] Tests pass on CI and coverage remains at 100%
[ x ] Documentation reflects the changes where applicable
[ x ] changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details.
You can skip this check if the change does not need a change file.)
[ x ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

LGTM.
@PrettyWood please review,  you happy with this?

Thanks so much @sveinugu.

Thanks so much @sveinugu.

You're welcome! Happy to get the fix in. And thank you for creating such a useful library! Looking forward to v2.

Hi @sveinugu @samuelcolvin
We recently bumped Pydantic from 1.10.1 to 1.10.2 and our tests started failing. After some lengthy investigation I found this change to be the root cause. I'd like to ask you for your opinions.
We have a Pydantic model with a field such as
f: MyGenericModel[Callable[[Task, Dict[str, Any]], Iterable[Result]]] = Field(...)

Just importing Python module with this code gives me:
cls = <class 'foo.MyGenericModel'>
params = typing.Callable[[foo.Task, typing.Dict[str, typing.Any]], typing.Iterable[foo.Result]]

    def __class_getitem__(cls: Type[GenericModelT], params: Union[Type[Any], Tuple[Type[Any], ...]]) -> Type[Any]:
        """Instantiates a new class from a generic class `cls` and type variables `params`.

        :param params: Tuple of types the class . Given a generic class
            `Model` with 2 type variables and a concrete model `Model[str, int]`,
            the value `(str, int)` would be passed to `params`.
        :return: New model class inheriting from `cls` with instantiated
            types described by `params`. If no parameters are given, `cls` is
            returned as is.

        """

        def _cache_key(_params: Any) -> Tuple[Type[GenericModelT], Any, Tuple[Any, ...]]:
            return cls, _params, get_args(_params)

>       cached = _generic_types_cache.get(_cache_key(params))
E       TypeError: unhashable type: 'list'

The problem seems to be that the function GenericModel._cache_key() now calls get_args() which in turns calls Python's typing.get_args() -> and this function returns a tuple with a list in it. And that makes it unhashable.
def get_args(tp):
    """Get type arguments with all substitutions performed.

    For unions, basic simplifications used by Union constructor are performed.
    Examples::
        get_args(Dict[str, int]) == (str, int)
        get_args(int) == ()
        get_args(Union[int, Union[T, int], str][int]) == (int, str)
        get_args(Union[int, Tuple[T, int]][str]) == (int, Tuple[str, int])
        get_args(Callable[[], T][int]) == ([], int)
    """
    if isinstance(tp, _GenericAlias) and not tp._special:
        res = tp.__args__
        if get_origin(tp) is collections.abc.Callable and res[0] is not Ellipsis:
            res = (list(res[:-1]), res[-1])             # <======== this list is a problem
        return res
    return ()

If I skip the callable typing and only define the field as
f: MyGenericModel[Callable] = Field(...)

then it runs okay.
Thanks in advance!

@pepastach Hmm... Seems we did not think of that. Surprised that typing uses lists and not tuples for Callable arguments, but there is probably a good reason for this (or it might be just that it would be too much a nuisance to switch between brackets and parentheses...). In any case, there might be other examples of unhashable return values from get_params, so a simple solution could be to just catch TypeError and in that case default to the previous _cache_key generation.
I am new as contributor to pydantic, but I assume it would be cleanest if you could create another issue for this.

Yes, new issue which references this PR.
But honestly, generics are one of the things that needs to be refactored properly in V2.
This patches-on-patches-on-patches are getting more and more complex - no one's fault, but I really hope we can get away from some of these anti-patterns in V2.
Anyway, apart from me whinging, this shouldn't be too hard to fix/work around as @sveinugu suggests.

Yes, new issue which references this PR.

I've creatd #4551

authors: pepastach samuelcolvin sveinugu

repo: pydantic/pydantic


text:
encoding not respected on read_msgpack

as discussed on https://groups.google.com/forum/#!topic/pydata/ngROaML_hLI
encoding does not seem to be respected on reading a msgpack, below i am expecting to get back what
I put in as utf8
In [17]: s
Out[17]: u'\u2019'

In [18]: s = pd.Series({'a' : u"\u2019" })

In [19]: s.values[0]
Out[19]: u'\u2019'

In [20]: pd.read_msgpack(s.to_msgpack(encoding='utf8')).values[0]
Out[20]: u'\xe2\x80\x99'

in stepping through, part of the problem seems to be that in the call to unpack on https://github.com/pydata/pandas/blob/master/pandas/io/packers.py#L134 that there is no encoding argument passed and so it defaults to latin1 in https://github.com/pydata/pandas/blob/master/pandas/io/packers.py#L558
changing L134 to :
l = list(unpack(fh, **kwargs))

and passing the encoding like:
pandas.read_msgpack(m, encoding='utf8')

makes it work for me, however i don't have en environment set up to submit this as a pull request via GH, and we're still using 0.14.1 due to compatibility issues.

even after making this change, this surprisingly raises a UnicodeDecodeError in msgpack.cpp:
pandas.read_msgpack(pandas.DataFrame([[401L, u'a']], index=[0], columns=['k', 'v']).to_msgpack(encoding='utf8'), encoding='utf8')

but i'm having trouble stepping through as my PyCharm environment crashes on me when inspecting

repo: pydata/pandas


text:
NaNs in Float64Index are converted to silly integers using index.astype('int')

Code Sample, a copy-pastable example if possible
>>> df = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
>>> df.index = [None, 1]
>>> df
      a  b
NaN   1  3
 1.0  2  4
>>> df.index = df.index.astype('int')
>>> df
                      a  b
-9223372036854775808  1  3
 1                    2  4

output of pd.show_versions()
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.1.13-100.fc21.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.utf8

pandas: 0.18.1
nose: None
pip: 8.1.1
setuptools: 20.2.2
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: None
xarray: 0.7.2
IPython: 4.2.0
sphinx: 1.3.5
patsy: None
dateutil: 2.4.2
pytz: 2015.7
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

This is numpy behaviour:
In [22]: np.array([np.nan, 1.]).astype(int)
Out[22]: array([-2147483648,           1])

But, we should probably check for the occurence of NaNs, just as we do for Series:
In [29]: df.iloc[0,0] = np.nan

In [30]: df.a
Out[30]:
NaN   NaN
 1      2
Name: a, dtype: float64

In [31]: df.a.astype(int)
...

C:\Anaconda\lib\site-packages\pandas\core\common.pyc in _astype_nansafe(arr, dty
pe, copy)
   2726
   2727         if np.isnan(arr).any():
-> 2728             raise ValueError('Cannot convert NA to integer')
   2729     elif arr.dtype == np.object_ and np.issubdtype(dtype.type, np.intege
r):
   2730         # work around NumPy brokenness, #1987

ValueError: Cannot convert NA to integer

I wanted to fix this bug but noticed a similar behaviour of other objects: DatetimeIndex, TimedeltaIndex, Categorical, CategoricalIndex. Namely (all four of them behave identically):
A = pd.DatetimeIndex([1e10,2e10,None])
A
Out[76]: DatetimeIndex(['1970-01-01 00:00:10', '1970-01-01 00:00:20', 'NaT'], dtype='datetime64[ns]', freq=None)
A.astype(int)
Out[77]: array([         10000000000,          20000000000, -9223372036854775808])

However, unlike with Float64Index, this is invertible:
pd.DatetimeIndex(A.astype(int))
Out[78]: DatetimeIndex(['1970-01-01 00:00:10', '1970-01-01 00:00:20', 'NaT'], dtype='datetime64[ns]', freq=None)

My question: is this behaviour also a bug and should be fixed the same way (raising a ValueError)? And if so, should all the fixes be placed into one commit/pull request?
By the way, there might be other objects with the same issue, which call numpy.ndarray.astype(). And numpy is also a bit inconsistent here:
np.array([1,np.nan]).astype(int)
Out[84]: array([                   1, -9223372036854775808])
np.array([1,np.nan], dtype = int)
Traceback...
ValueError: cannot convert float NaN to integer

@ch41rmn these are all as expected. converting to int converts to the underlying integer based representation.
The only issue is that Float64Index.astype(int) should raise (as its effectively non-convertible).

@jreback I actually think we should raise in the datetimeindex case as well (ideally). A NaT cannot be converted to int (just as float nan cannot be converted). There is the asi8 attribute if you want this.
But, of course, that is not really back compat. Internally I think we consequently use asi8? But not sure about external use of course

Raising for CategoricalIndex seems less of a problem (not a common thing to do)

This is excactly what should be returned (and is useful). yes its equivalen to internal .asi8, but I dont' see a good reason to NOT do this.
In [20]: pd.DatetimeIndex([1e10,2e10,None]).astype(int)
Out[20]: array([         10000000000,          20000000000, -9223372036854775808])

authors: ch41rmn jorisvandenbossche jreback pijucha


text:
problem with building pandas - not using gcc as compiler

"""the example is merely
pip install pandas - which fails, so download and run
python ./setup.py build
I fear it is a specific behavior of gcc compared to not gcc/gnu 'features' (aka NGNU for 'Not "GNU Not Unix"')

I will attach the .cpp, as well as a .i (-E) output, as well as verbose include file processing (.info file)

The build stops with the following messages:

root@x064:[/data/prj/python/pipbuilds/pandas-0.19.0]das/msgpack/_unpacker.cpp -o build/temp.aix-5.3-2.7/pandas/msgpack/_unpacker.o                                           <
"pandas/msgpack/../src/msgpack/unpack_template.h", line 173.13: 1540-0063 (S) The text "..." is unexpected.
"pandas/msgpack/../src/msgpack/unpack_template.h", line 75.19: 1540-0700 (I) The previous message was produced while processing "unpack_execute<1>(unpack_context *, const char *, size_t, size_t *)".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 468.45: 1540-0700 (I) The previous message was produced while processing "unpack_construct".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 173.13: 1540-0063 (S) The text "..." is unexpected.
"pandas/msgpack/../src/msgpack/unpack_template.h", line 75.19: 1540-0700 (I) The previous message was produced while processing "unpack_execute<0>(unpack_context *, const char *, size_t, size_t *)".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 469.40: 1540-0700 (I) The previous message was produced while processing "unpack_skip".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 433.29: 1540-0063 (S) The text "..." is unexpected.
"pandas/msgpack/../src/msgpack/unpack_template.h", line 412.19: 1540-0700 (I) The previous message was produced while processing "unpack_container_header<144,220>(unpack_context *, const char *, size_t, size_t *)".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 470.46: 1540-0700 (I) The previous message was produced while processing "read_array_header".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 433.29: 1540-0063 (S) The text "..." is unexpected.
"pandas/msgpack/../src/msgpack/unpack_template.h", line 412.19: 1540-0700 (I) The previous message was produced while processing "unpack_container_header<128,222>(unpack_context *, const char *, size_t, size_t *)".
"pandas/msgpack/../src/msgpack/unpack_template.h", line 471.44: 1540-0700 (I) The previous message was produced while processing "read_map_header".

reading carefully - it comes down to errors while processing these four lines:
  +468  static const execute_fn unpack_construct = &unpack_execute<true>;
  +469  static const execute_fn unpack_skip = &unpack_execute<false>;
  +470  static const execute_fn read_array_header = &unpack_container_header<0x90, 0xdc>;
  +471  static const execute_fn read_map_header = &unpack_container_header<0x80, 0xde>;

These become (from the .i file)
#line 468
static const execute_fn unpack_construct = &unpack_execute<1>;
static const execute_fn unpack_skip = &unpack_execute<0>;
static const execute_fn read_array_header = &unpack_container_header<0x90, 0xdc>;
static const execute_fn read_map_header = &unpack_container_header<0x80, 0xde>;
#line 314 "pandas/msgpack/_unpacker.cpp"

And - whether it is pandas or perhaps cython - my knowledge of C does not understand how:
&unpack_execute<1> and &unpack_execute<0>
are useable against:

static inline int unpack_execute(unpack_context* ctx, const char* data, size_t len, size_t* off)

FYI: _packer.cpp is fine.

Will try again to attach file - getting message "We don't support that file type", even after putting them into a zip file.

please let me know how to attach a text or zip file - if interested!

pls show architecture, version of cython & python. as well as how you got the source.

AIX 5.3, (not gcc, but xlc), 64-bit
root@x064:[/]pip list
Cython (0.24.1)
numpy (1.11.2)
pip (8.1.1)
python-dateutil (2.5.3)
pytz (2016.7)
requests (2.11.1)
setuptools (20.10.1)
six (1.10.0)
pip install pandas
pip download pandas
root@x065:[/data/prj/python/pipbuilds]ls *.gz
numpy-1.11.2.tar.gz     pandas-0.19.0.tar.gz

I repeated the process, on AIX 6.1 - 32-bit mode, python built using gcc rather xlc.
This ends one(?) file earlier - I think the first .cpp (packer.cpp), with different warnings:
gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -D_LARGE_FILES -D__BIG_ENDIAN_=1 -Ipandas/src/msgpack -Ipandas/src/klib -Ipandas/src -I/opt/lib/python2.7/site-packages/numpy/core/include -I/opt/include/python2.7 -c pandas/msgpack/_packer.cpp -o build/temp.aix-6.1-2.7/pandas/msgpack/_packer.o -Wno-unused-function
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]
pandas/msgpack/_packer.cpp: In function 'Py_ssize_t __Pyx_PyIndex_AsSsize_t(PyObject*)':
pandas/msgpack/_packer.cpp:7517:9: error: call of overloaded 'abs(const Py_ssize_t&)' is ambiguous
pandas/msgpack/_packer.cpp:7517:9: note: candidates are:
In file included from /opt/include/python2.7/Python.h:42:0,
from pandas/msgpack/_packer.cpp:4:
/opt/lib/gcc/powerpc-ibm-aix5.3.7.0/4.7.4/include-fixed/stdlib.h:309:14: note: int abs(int)
In file included from /opt/include/python2.7/pyport.h:325:0,
from /opt/include/python2.7/Python.h:58,
from pandas/msgpack/_packer.cpp:4:
/opt/lib/gcc/powerpc-ibm-aix5.3.7.0/4.7.4/include-fixed/math.h:947:14: note: float abs(float)
/opt/lib/gcc/powerpc-ibm-aix5.3.7.0/4.7.4/include-fixed/math.h:977:20: note: long double abs(long double)
error: command 'gcc' failed with exit status 1
The "error" being this line:
pandas/msgpack/_packer.cpp:7517:9: error: call of overloaded 'abs(const Py_ssize_t&)' is ambiguous
FYI: when using xlc - _packer.o completed.
45482770  116 -rw-r-----  1 root      system      117901 Oct  7 10:46 ./pandas-0.19.0/build/temp.aix-5.3-2.7/pandas/msgpack/_packer.o
root@x065:[/data/prj/python/pipbuilds]oslevel
6.1.0.0
Note the directory above is build/temp.aix5.3-2.7 not aix6.1-2.7

we don't support aix directly, meaning that it's not tested at all
so would accept compatible patches
we also make no guarantees about big endian (powerpc)
again would accept patches

Well, I would consider working on a patch. I am just a humble packager with some programming skills.
Perhaps you could give a hint about where to look? As the code seems to be generated I do not think debugging generated code will help much.
In other words - I accept hints and tips :)

@aixtools no real idea what is going on here, maybe @llllllllll has some

And how would I get, rather this question - get any attention from  @llllllllll - as I have never been properly introduced.

@aixtools there is nothing to do, he will answer when/if he can. otherwise others may be able to offer assistance.

It looks like the compiler does not support case ranges like:
switch(x) {
case a ... b:
}

The current CPP branching defines USE_CASE_RANGE when !defined(_MSC_VER). I think we should invert this to be: If you are compiling with something that supports gnu C, or ifdef __GNUC__.
diff --git a/pandas/src/msgpack/unpack_template.h b/pandas/src/msgpack/unpack_template.h
index 95af673..fba372d 100644
--- a/pandas/src/msgpack/unpack_template.h
+++ b/pandas/src/msgpack/unpack_template.h
@@ -17,7 +17,7 @@
  */

 #ifndef USE_CASE_RANGE
-#if !defined(_MSC_VER)
+#ifdef __GNUC__
 #define USE_CASE_RANGE
 #endif
 #endif

In the short term, you can define _MSC_VER which will disable the case range here, hopefully that doesn't cause anything to use a microsoft specific feature.

Many thanks to @llllllllll . As this was closed had fallen low on my radar. However, I hear from a colleague this took care of the problem. Kudos!

@aixtools if you'd like to do a PR with your change would be great.

Have to learn how to PR I guess. :)
I have git working again (with the strict SSL requirements) - so, eventually, it shall come.

I tried doing everything locally (on my server) but could not find the exact syntax for a pull-request. So, tried a couple of paths to get a PR done. Hope this is satisfactory!

authors: aixtools jreback llllllllll


text:
ERR: Index.shift() gives confusing error message when no DatetimeIndex

In [1]: idx = pd.Index(range(5))

In [2]: idx
Out[2]: Int64Index([0, 1, 2, 3, 4], dtype='int64')

In [3]: idx.shift(1)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-d8362983dbf7> in <module>()
----> 1 idx.shift(1)

c:\users\vdbosscj\scipy\pandas-joris\pandas\core\index.pyc in shift(self, period
s, freq)
   1096             return self
   1097
-> 1098         offset = periods * freq
   1099         return Index([idx + offset for idx in self], name=self.name)
   1100

TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'

Should shift be allowed at all when it is no DatetimeIndex?

closed by #11211

authors: jorisvandenbossche jreback


text:
BUG: More followups on to_datetime exceptions, xref #13033

@jorisvandenbossche I believe this will cover your comments.

Looks good, thanks!


text:
round function added to DatetimeIndex

closes #4314

closing pls reopen if/when updated

authors: bbartaia jreback


text:
DOC: StringMethods html doc fixes

I noticed some of the HTML links are broken in text.rst
@sinhrks @jorisvandenbossche this is related to https://github.com/pydata/pandas/pull/9843 but can be independently merged

yes, thanks!

authors: jorisvandenbossche mortada


text:
Update minimum version of dask

Xarray currently states that it supports dask version 0.9 and later. However, 1) I don't think this is true and my quick test shows that some of our tests fail using dask 0.9, and 2) we have a growing number of tests that are being skipped for older dask versions:
$ grep -irn "dask.__version__" xarray/tests/*py
xarray/tests/__init__.py:90:    if LooseVersion(dask.__version__) < '0.18':
xarray/tests/test_computation.py:755:        if LooseVersion(dask.__version__) < LooseVersion('0.17.3'):
xarray/tests/test_computation.py:841:    if not use_dask or LooseVersion(dask.__version__) > LooseVersion('0.17.4'):
xarray/tests/test_dask.py:211:    @pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_dask.py:223:    @pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_dask.py:284:    @pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_dask.py:296:    @pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_dask.py:387:        if LooseVersion(dask.__version__) == LooseVersion('0.15.3'):
xarray/tests/test_dask.py:784:    pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_dask.py:802:    pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_dask.py:818:@pytest.mark.skipif(LooseVersion(dask.__version__) <= '0.15.4',
xarray/tests/test_variable.py:1664:        if LooseVersion(dask.__version__) <= LooseVersion('0.15.1'):
xarray/tests/test_variable.py:1670:        if LooseVersion(dask.__version__) <= LooseVersion('0.15.1'):

I'd like to see xarray bump the minimum version number of dask to something around 0.15.4 (Oct. 2017) or 0.16 (Nov. 2017).
cc @mrocklin, @pydata/xarray

DataArray.dot() is using dask.einsum, and it only woks with dask >= 0.17.5 (#2089).
Does this requirement sound is too strict? If so, we maybe better to add another pass for older dask.

We have traditionally allowed for a fairly large lag in version dependencies so I'd guess that 0.17.5 (May 16, 2018) is too recent. That said, I'm in favor of shrinking that window to facilitate faster integration of new features.

Requiring 0.15.4 seems reasonable to me. There is virtue in supporting a (small) range of versions in dependency. But clearly 0.9 was too out of date here.
We should probably add a CI test against the oldest version of dask that we support, just to make sure things don't sneak through.

How do we consider the minimum version of optional dependency?
If we say "the minimum version of dask is 0.16", then all our dask related functions should work with dask=0.16?
or do we allow functions that only work with newer version of dask if we raise an appropriate error message?

The policy I've tried to keep to is keeping everything working with base
supported version, but we can add new features that rely on new dask
features.
On Wed, May 30, 2018 at 8:20 PM Keisuke Fujii notifications@github.com
wrote:

How do we consider the minimum version of optional dependency?
If we say "the minimum version of dask is 0.16", then all our dask related
functions should work with dask=0.16?
or do we allow functions that only work with newer version of dask if we
raise an appropriate error message?
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/2203#issuecomment-393389052, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1uTxKa_GJxvqVj2H2LvL-IYTOCWPks5t32FugaJpZM4UT_GD
.

authors: fujiisoup jhamman shoyer

repo: pydata/xarray


text:
Recommendations for domain-specific accessor documentation

Hi,
I'm currently working on an ocean domain specific accessor for a machine learning technique (https://github.com/gmaze/pyxpcm/tree/nfeatures).
I implemented an xarray.DataSet accessor (name space pyxpcm) that I'd like to document.
I thus wonder whether the xarray/pangeo community has recommendations about how to do this appropriately ?
Right now, I simply have an API auto-generated page, see:
https://pyxpcm-dev.readthedocs.io/en/latest/api.html#xarray-dataset-accessor-the-pyxpcm-name-space

@gmaze Just as an example, here is what we recently added for MetPy: https://unidata.github.io/MetPy/latest/api/generated/metpy.xarray.html. Previously, we just had a narrative-form tutorial.
If there are other recommendations, it would be great to incorporate those into MetPy as well!

Thanks for the example @jthielen, this looks great !
May be it would be better to not show to readers the accessor class name, since it will never be seen on the API frontend, only the scope name will.
So it would be great to be able to have the documentation to read something like:
xarray.DataSet.<scope_name>.<accessor_method_or_property>
instead of:
<package_name>.xarray.<accessor_class_name>.<accessor_method_or_property>
(in the case where the accessor class is in xarray.py file on the package)
But I don't know how to manage that with sphinx documentation

comment from @benbovy in #3625:

I re-used pandas templates for xarray accessors here: https://github.com/benbovy/xarray-simlab/blob/45359e99cbf6341464b02cb937618c051a58a31c/doc/conf.py#L233
It works pretty well: https://xarray-simlab.readthedocs.io/en/latest/api.html#dataset-xsimlab-xarray-accessor

For reference, here's pandas' version: https://github.com/pandas-dev/pandas/blob/837daf18d480cce18c25844c591c39da19437252/doc/source/conf.py#L441-L591
I think to make this a bit easier for everyone we should add this to the documentation on accessors.
Also, we could create a sphinx extension module that contains all the code in conf.py so defining the custom directives would be as simple as adding something like "xarray.doc.accessors" to extensions. The module doesn't have to be in xarray, though, it could also become a separate package.

I think to make this a bit easier for everyone we should add this to the documentation on accessors.

Agreed, this would definitely be useful.

Also, we could create a sphinx extension module that contains all the code in conf.py

I also agree! Since the code in conf.py is exactly the same for both pandas and xarray, it would be nice to hear thoughts from pandas developers @jorisvandenbossche.

I think a documentation build only dependency would be very easy for both xarray and pandas to use. So a separate package (a sphinx extension?) sounds like a good idea.

I created such a package since I think I'm going to need that for pint-xarray. It currently lives at https://github.com/keewis/accessors

authors: benbovy gmaze jthielen keewis shoyer


text:
Is to_dask_dataframe(set_index=...) correct?

What happened:
Calling ds.to_dask_dataframe(set_index='lat') raises on attempting to create a MultiIndex.
What you expected to happen:
Shouldn't this create a normal index with just lat?
Minimal Complete Verifiable Example:
In [1]: ds = xr.tutorial.load_dataset('air_temperature')

In [2]: ds.to_dask_dataframe(set_index='lat')
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-2-e13a093182d0> in <module>
----> 1 ds.to_dask_dataframe(set_index='lat')

~/workspace/xarray/xarray/core/dataset.py in to_dask_dataframe(self, dim_order, set_index)
   5534                 # triggers an error about multi-indexes, even if only one
   5535                 # dimension is passed
-> 5536                 df = df.set_index(dim_order)
   5537
   5538         return df

~/.asdf/installs/python/3.8.10/lib/python3.8/site-packages/dask/dataframe/core.py in set_index(***failed resolving arguments***)
   4177             from .shuffle import set_index
   4178
-> 4179             return set_index(
   4180                 self,
   4181                 other,

~/.asdf/installs/python/3.8.10/lib/python3.8/site-packages/dask/dataframe/shuffle.py in set_index(df, index, npartitions, shuffle, compute, drop, upsample, divisions, partition_size, **kwargs)
    140             index = index[0]
    141         else:
--> 142             raise NotImplementedError(
    143                 "Dask dataframe does not yet support multi-indexes.\n"
    144                 "You tried to index with this index: %s\n"

NotImplementedError: Dask dataframe does not yet support multi-indexes.
You tried to index with this index: ['lat', 'time', 'lon']
Indexes must be single columns only.

Anything else we need to know?:
Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: 95ba539f660e696fc080f39dd0afc0e29385fabc
python: 3.8.10 (default, May  9 2021, 13:21:55)
[Clang 12.0.5 (clang-1205.0.22.9)]
python-bits: 64
OS: Darwin
OS-release: 20.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: None
libnetcdf: None
xarray: 0.18.2
pandas: 1.2.4
numpy: 1.20.3
scipy: 1.6.3
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.8.3
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.3
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.06.1
distributed: 2021.06.1
matplotlib: 3.4.2
cartopy: None
seaborn: 0.11.1
numbagg: 0.2.1
pint: None
setuptools: 56.0.0
pip: 21.1.2
conda: None
pytest: 6.2.4
IPython: 7.24.0
sphinx: 4.0.1

The docs are very clear!

set_index (bool, optional) – If set_index=True, the dask DataFrame is indexed by this dataset’s coordinate. Since dask DataFrames do not support multi-indexes, set_index only works if the dataset only contains one dimension.

Closing


text:
ENH: Support using opened netCDF4.Dataset (Fixes #1459)

Make the filename argument to NetCDF4DataStore polymorphic so that a
Dataset can be passed in.

[x] Closes #1459
[x] Tests added / passed
[x] Passes git diff upstream/master | flake8 --diff
[x] Fully documented, including whats-new.rst for all changes and api.rst for new API

#1459 discussed adding an alternate constructor (i.e. a class method) to NetCDF4DataStore to allow this, which would be my preferred approach rather than making a filename polymorphic (via isinstance). Unfortunately, alternate constructors only work by taking one set of parameters (or setting defaults) and then passing them to the original constructor. Given that, there's no way to make an alternate constructor without also making the original constructor somehow aware of this functionality--or breaking backwards-compatibility. I'm open to suggestions to the contrary.

I've taken a crack at refactoring the constructor to make that take the dataset instance and have an open class method that opens the dataset. It was a little more effort to break everything apart (owing to opener) but I think what I have is ok. I'd appreciate feedback before I go and update the whats new and rebase to clean up.

Ok, so I think we're closer now. The tests at least pass on my machine now. 😁

I went ahead and rebased on master and squashed down the WIP commits. I suspect this is ready now.

Looks like test failures are those mentioned in #1540

Thanks @dopplershift !

authors: dopplershift shoyer


text:
Fix CRS being WKT instead of PROJ.4

See https://github.com/mapbox/rasterio/blob/master/CHANGES.txt#L7 for the change in rasterio 1.0.14 and my conda-forge issue where I discovered this: https://github.com/conda-forge/gdal-feedstock/issues/262
This was put in a bugfix (micro version number) release of rasterio which made it harder to discover. @sgillies @snowman2 I saw that you brought this up and how it affected xarray (https://rasterio.groups.io/g/dev/message/68). If this pull request is a duplicate of a fix one of you made let me know.
To xarray devs, if I need to make any other changes (whats-new?) also let me know.

[ ] Closes #xxxx
[ ] Tests added
[ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Does this need a whats-new entry?

Does this need a whats-new entry?

Yes, I would recommend that here -- especially for cases where we're changing behavior without a deprecation cycle (which is sometime the best path forward)

This PR isn't changing any functionality. It is making the same functionality available with newer versions of rasterio.

OK, good point. I do still try to lean towards adding notes about bug fixes in "what's new" but it isn't strictly required.

@shoyer Added something to the whats-new. Let me know if anything needs changing.

thanks David, that looks great

@shoyer Any idea when a 0.11.4 or 0.12 will be released? I'm trying to work around some other rasterio bugs and would like to remove the restriction on the rasterio version used in my CI tests, but that requires this PR.

Hopefully quite soon! see https://github.com/pydata/xarray/pull/2777 for
current status
On Tue, Mar 12, 2019 at 10:31 AM David Hoese notifications@github.com
wrote:

@shoyer https://github.com/shoyer Any idea when a 0.11.4 or 0.12 will
be released? I'm trying to work around some other rasterio bugs and would
like to remove the restriction on the rasterio version used in my CI tests,
but that requires this PR.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/2715#issuecomment-472102830, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1hs0lknMUSnOP9A_Qw0124V8HZllks5vV-SGgaJpZM4aUQen
.

authors: dcherian djhoese shoyer


text:
new polyval algo

[x] Closes #6526
[x] Tests added
[ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Several open points still:

Unittests for datetime values are failing, I might need some help with that since I have no idea what this means for polynomials.
Algorithm should work also with Datasets (any combination of DataArray and Dataset for coord and coeffs inputs). Still needs to be checked and tested (How does one define typing for such cases? I.e. DataArray + Dataset -> Dataset but DataArray + DataArray -> DataArray?)
It uses Horners method instead of Vandermonde matrix, should be faster and consume less memory (unless the overhead of sorting index,  isel etc. is too large). Maybe some performance comparisons should be done.
Instead of coord the input should simply be called x or similar, however this would break backwards compatibility, so maybe we just leave it.

I noticed that broadcasting Datasets behaves weird, see https://github.com/pydata/xarray/issues/6549, so I used a "hack" of adding an 0-valued DataArray/Dataset.
Anyone got a better idea?

This looks like excellent code @headtr1ck , especially so for a first PR. Welcome!

so I used a "hack" of adding an 0-valued DataArray/Dataset.

I think it's fine to do the hack and reference that issue; we can clean it up then (though if others have ideas then great).

I had to add a copy(deep=True) since the broadcast returned a read-only DataArray. Any better ideas?

IIRC we generally leave this up to the caller (and generally discourage mutation).

Someone who knows better should review the algo & opine on the datetime issue; I agree I'm not sure whether we need to support them.

Some performance comparison:
With 5th order polynomial and 10 x-values:
old: 1.05 ms ± 15.8 µs per loop
new: 1.41 ms ± 11.6 µs per loop
With 5th order polynomial and 10000 x-values:
old: 1.46 ms ± 10.5 µs per loop
new: 1.41 ms ± 14.5 µs per loop
With 5th order polynomial and 1mio x-values:
old: 65.1 ms ± 332 µs per loop
new: 6.99 ms ± 168 µs per loop
As expected for small arrays the new method creates some overhead, but for larger arrays the speedup is quite nice.
Also, it uses in-place operations with much less memory usage.

I added a rough support for datetime values. Someone with more knowledge of handling them should take a look, the code seems too complicated and I am sure there is a more clever solution (I could not use get_clean_interp_index since it is not an index anymore).

I agree I'm not sure whether we need to support them.

I think keeping support is nice, since they are a commonly occuring coordinates and we do not want to break anything if possible.

Benchmark did not succeed since the inputs are not compatible with the old algorith...
Do we change it such that it is compatible?

Benchmark did not succeed since the inputs are not compatible with the old algorith...
Do we change it such that it is compatible?

If you're confident this is faster and it's not a trivial amount of work to adjust them, I would leave it...

First time working with asv...
It seems that module level variables affect all other peakmem tests (i guess memory usage of the phyton process is measured,).
We should refactor all dataarrays into the setup functions, otherwise O(n) memory algos will show wrong numbers and adding new tests will show regression on other tests.

Nice  this looks  like an improvement for everything other than dask arrays with only 100 elements, which is not a good use-case  for dask.
I was slightly concerned that the recrusive algorithm wouldn't work well with dask but it does seem to work better.
def other_polyval(coord, coeffs, degree_dim="degree"):
    x = coord.data

    deg_coord = coeffs[degree_dim]
    N = int(deg_coord.max()) + 1

    lhs = xr.DataArray(
        np.stack([x ** (N - 1 - i) for i in range(N)], axis=1),
        dims=(coord.name, degree_dim),
        coords={
            coord.name: coord.data,
            degree_dim: np.arange(deg_coord.max() + 1)[::-1],
        },
    )
    return xr.dot(lhs, coeffs, dims=degree_dim)

coeffs = xr.DataArray(np.random.randn(2), dims="degree")
da = xr.DataArray(dask.array.random.random((10**6), chunks=(10000)), dims=("x"))
print(len(da.data.dask))
print(len(xr.polyval(da, coeffs).data.dask))
print(len(other_polyval(da, coeffs).data.dask))

100
502
1005

One minor open point: what to do with a non-integer "degree" index?
Float type could be cast to integer (thats what is happening now). But datetime etc. should raise an error?

Forcing the user to be explicit reduces bugs and user support requests :) so we like to do that.
Thanks again @headtr1ck this is a great PR!

authors: dcherian headtr1ck max-sixty


text:
Add test file with a non byte-aligned segmentation

Describe the changes
Add a new test file that is a binary, natively encoded segmentation (with BitsAllocated=1) whose frames contain a number of pixels that are not a multiple of 8. Such files are valid but tricky to deal with because of the misalignment of the frame and byte boundaries.
Partial fix for https://github.com/pydicom/pydicom/issues/2134
Tasks

[x] In a branch on your fork of pydicom-data, add or modify data files
[x] Create a pull request for pydicom-data
[ ] Update the commit hash for the urls in a fork/branch of the pydicom file [urls.json]https://github.com/pydicom/pydicom/blob/master/pydicom/data/urls.json and create a pydicom pull request
[ ] pydicom unit tests passing
[ ] Documentation updated (if needed)

I should mention that to create this I took the existing liver image (originally 512 x 512) and just cropped it down to a "weird" size (510 x 511) and correctly re-encoded the pixels with bit-packing.

repo: pydicom/pydicom-data


text:
Missing Content-Type In Some API Responses

If the return value from a resource was either a string or a two part tuple the default content type header
was not being set.   The return value from get_content_type() was also incorrect (should be a list so setRawHeaders() can use it).

@guidow, PTAL.  Found this bug while writing tests for tasks and tasklogs.

Coverage increased (+0.06%) to 73.6% when pulling 12caa74a8b92ca9884b2cdb603af7a1c2d32c8eb on bugfix_content_type_header into 67f5bae219c54cc603c16e81d9984dc0ffa96d37 on master.

Coverage increased (+0.06%) to 73.6% when pulling 12caa74a8b92ca9884b2cdb603af7a1c2d32c8eb on bugfix_content_type_header into 67f5bae219c54cc603c16e81d9984dc0ffa96d37 on master.

Coverage increased (+0.09%) to 73.64% when pulling 734cef7a5a10ca13ed4acc276eb33fd43bf5dcdf on bugfix_content_type_header into 67f5bae219c54cc603c16e81d9984dc0ffa96d37 on master.

Looks okay so far.

Have you had a chance to run this yet?  If so and you haven't had any problems we should probably go ahead and merge this.

Yes, I've had a chance to run this in our test farm, and I haven't seen anything amiss.

Cool thanks, I'll go ahead and merge this then.  Just wanted to be sure.

authors: coveralls guidow opalmer

repo: pyfarm/pyfarm-agent


text:
CI: update black version

Currently failing, see https://stackoverflow.com/questions/71673404/importerror-cannot-import-name-unicodefun-from-click

The pytest failures are already being fixed in https://github.com/pygeos/pygeos/pull/450/, so going to merge this as such

repo: pygeos/pygeos


text:
Conda-forge integration with docker build

As part of our docker build we'll need to make sure to integrate properly with conda-forge.
As If we eventually want to distribute on conda-forge then users will likely be using a python binary from conda-forge (https://anaconda.org/conda-forge/python/files?version=3.6.7) rather than defaults. Regardless we can sort this out later, just wanted to chime in.  as @kyleabeauchamp said in #49

Closing for now.

authors: springcoil twiecki


text:
Breakpoint causes unhandled exeception

Setup: ST-Link v2 with an STM32F411CE. Setup works with Mbed Studio (which uses pyOCD). However, trying to use pyOCD without studio -- either the version included or the latest version -- causes a failure when gdb sets a breakpoint.  I have tried both the stripped down pack included with Mbed Studio and the full pack. I've tried remote and extended remote. I've try gdb-multiarch and the arm noabi gdb. All the same results.
The exact same setup works fine with OpenOCD so I think wiring etc isn't the issue.
When I try to set a breakpoint I get:
0012712 E Unhandled exception in handle_message (b'm'): result size (6) != requested size (4) [gdbserver]
Traceback (most recent call last):
  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 423, in handle_message
    reply = handler(msg[msgStart:])

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 844, in get_memory
    mem = self.target_context.read_memory_block8(addr, length)

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/debug/cache.py", line 42, in read_memory_block8
    return self._memcache.read_memory_block8(addr, size)

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/cache/memory.py", line 258, in read_memory_block8
    assert len(result) == size, "result size ({}) != requested size ({})".format(len(result), size)

AssertionError: result size (6) != requested size (4)

0012712 E Unexpected exception: a bytes-like object is required, not 'tuple' [gdbserver]

Traceback (most recent call last):

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 390, in _run_connection
    self.packet_io.send(resp)

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/packet_io.py", line 83, in send
    self._write_packet(packet)

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/packet_io.py", line 146, in _write_packet
    written = self._abstract_socket.write(packet)

  File "/home/alw/.local/lib/python3.8/site-packages/pyocd/utility/sockets.py", line 58, in write
    return self.conn.send(data)

TypeError: a bytes-like object is required, not 'tuple'

I feel like I must be doing something wrong. I've tried simple invocations and trying to mimic the Mbed Studio invocation. So:
pyocd gdbserver -t stm32f411ce
Along with explicit pack definitions. Also
python3 -mpyocd gdbserver --target STM32F411CE --erase=chip --frequency 1800000 --pack '/home/alw/.config/Mbed Studio/mbed-studio-tools/cmsis-packs/Keil.STM32F4xx_DFP.2.15.0-small.pack' -O connect_mode=under-reset   -O enable_swv=1 -O swv_system_clock=96000000 -O swv_raw_port=3443 -O swv_clock=2000000
And many variations (e.g., no swv, etc.).

I am facing this issue too within gdbserver. but during load with custom target.

@Hoohaha Do you mean when executing load? Are you using remote or extended-remote in the target specifier? It does change the error depending on which way I try, but in my case seems to always be when a breakpoint is set or when you try to run (and, presumably, it tries to write the breakpoint).

Can you elaborate on how your flash was misconvigured? I also get the tuple message after the initial message.

Thanks for the report! It's pretty clearly a bug in pyocd. Not sure why it didn't show up in CI, since breakpoints in gdb are tested. I'll take a look at it this weekend.

@wd5gnr The "AssertionError: result size (6) != requested size (4)" seems to be the root cause. This spills down and causes a further exception because the gdbserver somehow handles it wrong. You're not doing anything wrong! 😄
The main question is why the data cache sees an incorrect size. It might be caused by attempting to read at the boundary of a memory region, such that 2 bytes are clipped off.
I'll need a gdbserver debug log, and ideally a copy of the firmware .elf if you can provide it (or an equivalent reproducer .elf). To get the debug log, run with -vv -Lpyocd.gdbserver.*.trace*=debug on the pyocd command line. This can produce a lot of data, so you might want to redirect to a file with 2> pyocd.log. (Only stderr is needed.)
If you can provide the ELF file, you can send to me via email (please see my GitHub profile for my email address) or attach to a comment here.

Will get that together. Thanks
On Fri, Jun 10, 2022, 4:05 PM Chris Reed @.***> wrote:

@wd5gnr https://github.com/wd5gnr The "AssertionError: result size (6)
!= requested size (4)" seems to be the root cause. This spills down and
causes a further exception because the gdbserver somehow handles it wrong.
You're not doing anything wrong! 😄
The main question is why the data cache sees an incorrect size. It might
be caused by attempting to read at the boundary of a memory region, such
that 2 bytes are clipped off.
I'll need a gdbserver debug log, and ideally a copy of the firmware .elf
if you can provide it (or an equivalent reproducer .elf). To get the debug
log, run with -vv -Lpyocd.gdbserver..trace=debug on the pyocd command
line. This can produce a lot of data, so you might want to redirect to a
file with 2> pyocd.log. (Only stderr is needed.)
If you can provide the ELF file, you can send to me via email (please see
my GitHub profile for my email address) or attach to a comment here.
—
Reply to this email directly, view it on GitHub
https://github.com/pyocd/pyOCD/issues/1409#issuecomment-1152738177, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AARTDE2XMD6GJVPH5QWMDKLVOOU27ANCNFSM5YBIO7UA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I'm going to zip together some full log files and also the elf file for you, but here's a quick excerpt:
026624 D GDB getMem: addr=8004398 len=2 [gdbserver]
0026625 D --<<<< GDB send 8 bytes: b'$2846#d4' [packet_io]
0026625 D -->>>> GDB read 16 bytes: b'$Z1,8004398,2#85' [packet_io]
0026625 D GDB breakpoint b'Z'1 @ 8004398 [gdbserver]
0026625 D set bkpt type HW at 0x8004398 [manager]
0026625 D using hw bp instead because addr is flash [manager]
0026625 D selected bkpt type HW for addr 0x8004398 [manager]
0026625 D using hw bp instead because addr is flash [manager]
0026625 D selected bkpt type HW for addr 0x8004378 [manager]
0026625 D --<<<< GDB send 6 bytes: b'$OK#9a' [packet_io]
0026625 D -->>>> GDB read 10 bytes: b'$vCont?[#49](https://github.com/pyocd/pyOCD/pull/49)' [packet_io]
0026625 D --<<<< GDB send 21 bytes: b'$vCont;c;C;s;S;r;t#be' [packet_io]
0026625 D -->>>> GDB read 11 bytes: b'$vCont;c#a8' [packet_io]
0026627 E Unhandled exception in handle_message (b'v'): result size (4) != requested size (1) [gdbserver]
Traceback (most recent call last):
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 423, in handle_message
reply = handler(msg[msgStart:])
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 710, in v_command
return self.v_cont(cmd)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 728, in v_cont
if self.is_threading_enabled():
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 1220, in is_threading_enabled
return (self.thread_provider is not None) and self.thread_provider.is_enabled \
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/rtos/rtx5.py", line 409, in is_enabled
return self.get_kernel_state() != 0 and not self._target.in_thread_mode_on_main_stack()
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/rtos/rtx5.py", line 447, in get_kernel_state
return self._target_context.read8(self._os_rtx_info + RTX5ThreadProvider.KERNEL_STATE_OFFSET)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/core/memory_interface.py", line 156, in read8
return self.read_memory(addr, 8, now)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/debug/cache.py", line 33, in read_memory
return self._memcache.read_memory(addr, transfer_size, now)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/cache/memory.py", line 230, in read_memory
data = self.read_memory_block8(addr, 1)[0]
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/cache/memory.py", line 258, in read_memory_block8
assert len(result) == size, "result size ({}) != requested size ({})".format(len(result), size)
AssertionError: result size (4) != requested size (1)
0026628 D --<<<< GDB send 2 bytes: (b'$E01#a6', 0) [packet_io]
0026628 E Unexpected exception: a bytes-like object is required, not 'tuple' [gdbserver]
Traceback (most recent call last):
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/gdbserver.py", line 390, in _run_connection
self.packet_io.send(resp)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/packet_io.py", line 83, in send
self._write_packet(packet)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/gdbserver/packet_io.py", line 146, in _write_packet
written = self._abstract_socket.write(packet)
File "/home/alw/.local/lib/python3.8/site-packages/pyocd/utility/sockets.py", line 58, in write
return self.conn.send(data)
TypeError: a bytes-like object is required, not 'tuple'
0029613 D -->>>> GDB read 1 bytes: b'\x03' [packet_io]
0033612 D GDB packet thread: other side closed connection [packet_io]

Here's a full package. It is hard to say, but I think maybe pyocd doesn't do extended-remote gdbserver? I ran it both ways in case. Also, in some of the runs the chip was already flashed. In some I flashed with gdb and some I flashed with monitor load. The flashes appear successful since the device will run when reset.
package.zip
The zip has a very simple ELF along with full.txt which has a gdb transcript, command line, and the error log. The other files have a bit less info but are also logs.

Thanks very much!
Pyocd does support extended-remote. This makes the server persistent, and adds support for the restart command.

@maxgerhardt Thanks for the additional info!

authors: Hoohaha flit wd5gnr

repo: pyocd/pyOCD


text:
ATSAMD21G18A TransferError: No ACK received

Problem
I'm trying to use pyOCD with an ATSAMD21G18A connected to an nEDBG CMSIS-DAP, but any command I try results in the following error:
pyocd.probe.pydapaccess.dap_access_api.DAPAccessIntf.TransferError: No ACK received
Possibly useful info

When I start pyocd commander I get Connected to ATSAMD21G18A [Lockup]: MCHP3313031800000599.  I'm not sure what [Lockup] means in this context.
I note that the ATSAMD21G17D is provided by Microchip.SAMD21_DFP.3.6.144 but the ATSAMD21G18A is provided by Keil.SAMD21_DFP.1.3.1, which is deprecated.  Perhaps this is a pack issue, but I don't know how to change it.

Background
The Curiosity Nano board has a built-in nEDBG adaptor and an ATSAMD21G17D target MCU on-board.  I was able to connect to this chip fine using pyOCD, however I now want to use the nEDBG adaptor to program an ATSAMD21G18A chip on a different board.  I cut the SWD traces on the development board, as described in the user guide, and connected it to my target processor, but pyOCD refuses to play.
Atmel (Microchip) Studio running on the same machine with no change to the hardware setup works perfectly, so I'm confident it isn't a hardware issue.
I'm using a MacBook Pro (Ventura), running pyOCD natively.  I run Atmel Studio using Parallels.

Ok, I manually hacked the files to remove the Keil PDSC from the list, pyOCD found and installed the Microchip PDSC and everything works now.
There seems to be two related issues:

the cmsis-pack-manager cache is keyed by part number (eg. ATSAMD21G18A), so it can't handle more than one PDSC option for a given part.
there is no way to exclude PDSC entries marked as 'deprecated' (which would imply a second PDSC entry for the same part).

I'm going to raise this with cmsis-pack-manager, but if it is fixed there then presumably a change will be needed to support it here also.

Any update on this.?
I'm trying to use ATSAMV71Q21 board, when I give the command pyocd list, I see that target is n/a, even after installing the pack,
I tried installing both Microchip specific pack and also Keil.SAM-V_DFP, but doesn't help.
Probe/Board                  Unique ID                  Target

0   Atmel Corp. EDBG CMSIS-DAP   ATMLxxxxxxxxxxxxxxxx       n/a
Is this EDBG supported by pyocd, or do I need to convert it to st-link.?
PS, I also tried by setting the option 'cmsis_dap.prefer_v1' to True

authors: Oozlum tdasika


text:
Custom building package with different path

Hello, folks
Currently I have an unconventional folder structure that looks like this:
...
project/
    └───src/
        └───backend/
            └───project/
                 │   file1.py
                 │   ...
        └───frontend/
            └───project/
                 │   file.js
                 │   ...
...

Was wondering if there is a way to specify the src/backend/project as project when building the package as it happens when its just src/project? I was able to build the thing by specifying project.name = "backend.project", but it ends up creating a package named backend.project which is fine, but the imports of course will be messed up. So I assumed that specifying tool.flit.module.name = "project" would sort of fix the problem. Package would be "misnamed", but everything else would remain the same, but it doesn't as it breaks with:
Traceback (most recent call last):
  File "path_to_module", line 5, in <module>
    from project.manage import main
ModuleNotFoundError: No module named 'project.manage'

Meaning that that tool.flit.module.name does not work as intended (at least for this case). Am I missing something? Apparently flit only accepts 2 layouts src/project or project with no workarounds?

Sorry, there's no way to configure it for that precise situation. Flit expects your Python package (project/ containing .py files) to be either in the same directory as pyproject.toml, or in a src/ subdirectory. Those are the only possibilities it checks.
It doesn't insist that pyproject.toml is in the repository root directory, though - so you could have something like:
project/
    └───src/
        └───backend/
            │   pyproject.toml
            └───project/
                 │   file1.py
                 │   ...

Then you'd run flit from inside the backend directory.
I'm going to close this because I don't plan to implement more config around this - Flit aims for simplicity over supporting all possible cases.

authors: caligotori takluyver

repo: pypa/flit


text:
Pipenv install fails when pysqlite3-binary and any other package are both required

Issue description
I have a Pipfile which has two requirements in it, pysqlite3-binary and pytz. Either requirement installs successfully on its own, but pipenv fails when asked to install them together. In testing, I've discovered that it appears that ANY package paired with pysqlite3-binary will cause the install to fail.
Expected result
I expected the install to work. This same setup worked last year, but when I tried to do it again, it failed this time.
Actual result
Build log
 > [python-deps 5/5] RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose:
#10 4.731 Using python: 3.10
#10 4.731 Path to python: /var/lang/bin/python3
#10 4.731 Creating a virtualenv for this project...
#10 4.731 Pipfile: /var/task/Pipfile
#10 4.917 Using /var/lang/bin/python3 (3.10.11) to create virtualenv...
#10 8.962 created virtual environment CPython3.10.11.final.0-64 in 2898ms
#10 8.962   creator CPython3Posix(dest=/var/task/.venv, clear=False, no_vcs_ignore=False, global=False)
#10 8.962   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
#10 8.962     added seed packages: pip==23.1.2, setuptools==67.7.2, wheel==0.40.0
#10 8.962   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
#10 8.962
#10 8.994 ✔ Successfully created virtual environment!
#10 9.396 Virtualenv location: /var/task/.venv
#10 9.734 Installing dependencies from Pipfile.lock (36200f)...
#10 9.799 Writing supplied requirement line to temporary file: "pysqlite3-binary ; platform_machine == 'x86_64'"
#10 9.823 Writing supplied requirement line to temporary file: 'pytz==2023.3 --hash=sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588 --hash=sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb'
#10 9.827 Install Phase: Standard Requirements
#10 9.827 Preparing Installation of 'pysqlite3-binary'
#10 9.827 Preparing Installation of 'pytz'
#10 9.828 $ /var/task/.venv/bin/python /var/lang/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-deps --exists-action=i -r /tmp/pipenv-_poa8qn4-requirements/pipenv-manv2i0z-hashed-reqs.txt
#10 9.828 Using source directory: '/var/task/.venv/src'
#10 13.90 Collecting pysqlite3-binary (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-manv2i0z-hashed-reqs.txt (line 1))
#10 13.90
#10 14.17 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#10 14.17     pysqlite3-binary from https://files.pythonhosted.org/packages/e5/af/750aa272a8282f6518a8532f31c51cc9e49815c626cf0df8cdf181367b52/pysqlite3_binary-0.5.0-cp310-cp310-manylinux_2_24_x86_64.whl (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-manv2i0z-hashed-reqs.txt (line 1))
#10 14.17 An error occurred while installing pysqlite3-binary ; platform_machine == 'x86_64'! Will try again.
#10 14.17 An error occurred while installing pytz==2023.3 --hash=sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588 --hash=sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb! Will try again.
#10 14.18 Installing initially failed dependencies...
#10 14.20 Writing supplied requirement line to temporary file: "pysqlite3-binary ; platform_machine == 'x86_64'"
#10 14.20 Writing supplied requirement line to temporary file: 'pytz==2023.3 --hash=sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588 --hash=sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb'
#10 14.20 Install Phase: Standard Requirements
#10 14.20 Preparing Installation of 'pysqlite3-binary'
#10 14.20 Preparing Installation of 'pytz'
#10 14.20 $ /var/task/.venv/bin/python /var/lang/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-use-pep517 --no-deps --exists-action=i -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt
#10 14.20 Using source directory: '/var/task/.venv/src'
#10 18.22 Collecting pysqlite3-binary (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt (line 1))
#10 18.22
#10 18.49 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#10 18.49     pysqlite3-binary from https://files.pythonhosted.org/packages/e5/af/750aa272a8282f6518a8532f31c51cc9e49815c626cf0df8cdf181367b52/pysqlite3_binary-0.5.0-cp310-cp310-manylinux_2_24_x86_64.whl (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt (line 1))
#10 18.50 [pipenv.exceptions.InstallError]: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#10 18.50 [pipenv.exceptions.InstallError]:     pysqlite3-binary from https://files.pythonhosted.org/packages/e5/af/750aa272a8282f6518a8532f31c51cc9e49815c626cf0df8cdf181367b52/pysqlite3_binary-0.5.0-cp310-cp310-manylinux_2_24_x86_64.whl (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt (line 1))
#10 18.50 ERROR: Couldn't install package: [Requirement(_name='pysqlite3-binary', vcs=None, req=NamedRequirement(name='pysqlite3-binary', version='', req=Requirement.parse('pysqlite3-binary; platform_machine == "x86_64"'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=pysqlite3-binary, path=None, uri=None, extras=(), markers=platform_machine == 'x86_64', vcs=None, specifier=None, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pysqlite3-binary)>), markers='platform_machine == "x86_64"', _specifiers='', index=None, editable=False, hashes=frozenset(), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=pysqlite3-binary, path=None, uri=None, extras=(), markers=platform_machine == 'x86_64', vcs=None, specifier=None, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pysqlite3-binary)>, _ireq=None), Requirement(_name='pytz', vcs=None, req=NamedRequirement(name='pytz', version='==2023.3', req=Requirement.parse('pytz==2023.3'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=pytz, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2023.3, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pytz==2023.3)>), markers=None, _specifiers='==2023.3', index='pypi', editable=False, hashes=frozenset({'sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb', 'sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588'}), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=pytz, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2023.3, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pytz==2023.3)>, _ireq=None)]
#10 18.50  Package installation failed...
------
executor failed running [/bin/sh -c PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose]: exit code: 1

Steps to replicate
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).
Take this Dockerfile:
FROM public.ecr.aws/lambda/python:3.10 AS base

# Set up environment
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1

FROM base AS python-deps

RUN pip install --upgrade pip
RUN pip install pipenv

COPY Pipfile* ./

RUN pipenv --support >> /tmp/pipenv-support.txt
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose

And this Pipfile:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pytz = "*"
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }

[requires]
python_version = "3.10"

And then attempt to build the Dockerfile:
$ pipenv update # To create the lockfile
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build .

NB: the pysqlite3-binary package only has Linux wheels, hence the platform specifics. Also, I have tried both version = "*" and pinning the version to 0.5.0, but get the same failure each time.
I'm on an M1 Mac, although this same error is happening on GitHub Actions when it tries to build the container there.

$ pipenv --support
Pipenv version: '2023.6.2'
Pipenv location: '/var/lang/lib/python3.10/site-packages/pipenv'
Python location: '/var/lang/bin/python3.10'
OS Name: 'posix'
User pip version: '23.1.2'
user Python installations found:
PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.10.11',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.10.76-linuxkit',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021',
 'python_full_version': '3.10.11',
 'python_version': '3.10',
 'sys_platform': 'linux'}

System environment variables:

_
HOME
SHLVL
PYTHONDONTWRITEBYTECODE
TZ
LAMBDA_RUNTIME_DIR
LANG
PYTHONFAULTHANDLER
PWD
PATH
LAMBDA_TASK_ROOT
LD_LIBRARY_PATH
LC_ALL
PIP_DISABLE_PIP_VERSION_CHECK
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:
Debug–specific environment variables:

PATH: /var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin
LANG: C.UTF-8
PWD: /var/task

Contents of Pipfile ('/var/task/Pipfile'):
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
# requests = "*"
# boto3 = "*"
# jinja2 = "*"
pytz = "*"
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }

# [dev-packages]
# isort = "*"
# black = "*"
# flake8 = "*"

[requires]
python_version = "3.10"

Contents of Pipfile.lock ('/var/task/Pipfile.lock'):
{
    "_meta": {
        "hash": {
            "sha256": "2b1a32fdca8e7cac9c7b8bc7db8cd7219f6ff51cd0ed4002293e5ff66d36200f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pysqlite3-binary": {
            "platform_machine": "== 'x86_64'",
            "version": "*"
        },
        "pytz": {
            "hashes": [
                "sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588",
                "sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb"
            ],
            "index": "pypi",
            "version": "==2023.3"
        }
    },
    "develop": {}
}

This works, and seems like the least bad option. But please let me know if I'm missing something.

Yeah, this is probably your least bad option for now.   The fundamental problem of building an sdist and getting the hashes for a different system than the one you are locking on is a well known problem.

Alright, thanks for confirming. Would be great to get a cleaner solution, but I'll use the multiple categories for now.

repo: pypa/pipenv


text:
[Docs]  Please add changelog link to Github Releases

Summary
Release notes pushed to Github with almost no description:
https://github.com/pypa/setuptools/releases

Please add at least a link to changelog of specific version, like https://setuptools.pypa.io/en/stable/history.html#v69-0-1. Or even add the same changelog items to release description, converting them from .rst syntax to .md, here is some example.
OS / Environment
No response
Additional Information

Code of Conduct

[X] I agree to follow the PSF Code of Conduct

This is probably a duplicate of #2892 and #2807.
Any help implementing that is appreciated. I am happy to review PRs for that topic.
Maybe it is needed to coordinate with https://github.com/jaraco/jaraco.develop, which is the tool we use to create the release via Github API.

So you don't use Github Actions to publish releases to PyPI and Github? All this is just manual?

No, the process is automated. To understand more about the automation, I recommend having a look at:

the linked issues (to which I consider this issue to be a duplicate),
the workflow file
the docs website (in the section about development process and releases)
the tox.ini file.

I've just tried to see logs of workflow which (as I expected) was used to publish the recent release, but got only failed jobs:
https://github.com/pypa/setuptools/actions/runs/7211233162
release job was skipped

The last automated release to PyPI was 69.0.2.

authors: abravalheri dolfinus

repo: pypa/setuptools

license: mit


text:
Creating FreeBSD packages

Originally reported by: alexandro (Bitbucket: alexandro, GitHub: alexandro)

Hi good people.
Does anybody mind that implementing support for creating native FreeBSD packages by one command for setuptools would be a good idea?
Since deploying python projects with high complexity to FreeBSD platform is a strong headache I suppose automatic tools instead of hand operations would be highly appreciated.

Bitbucket: https://bitbucket.org/pypa/setuptools/issue/481

You can check https://github.com/kxepal/setuptools-pkg where I'm solving this problem. There are quite a lot of questions, problems and ideas to apply, so it's good to have it as setuptools extension rather than builtin feature. At least for now.
Any feedback is welcome!

authors: bb-migration kxepal


text:
quote library_dir_option after calling unpatched version

rather than before (#715), which should avoid double-quotes if the unpatched function does its own quoting.
I believe this closes #722

I confirm that works for me, thanks! 👍

authors: anthrotype minrk


text:
PyPI downloads page does not show all wheels available

Describe the bug
Earlier when trying to figure out why we could not get a specific package to install via pip I realized that not all wheels are shown under the "Built Distributions" section of a given package's download page.
Expected behavior
I would expect that the downloads page would show me all available wheels to download should I need to.
To Reproduce
A good example exists here:
https://pypi.org/project/mysqlclient/2.1.0/#files
Only win_amd64.whl files are shown, though linux_x86_64.whl versions do exist.
My Platform
I am running Firefox 101.0.1 on Windows 11.
Additional context
I feel this is an important issue as it would be useful to be able to determine if an install issue lies with pip, pypi, or the package its self. From a lot of trial and error we were able to deduce that the mysqlclient package must just be missing linux distributions for 2.1.1, however, when we looked at 2.1.0 and also saw only windows distributions listed it added a lot of confusion as we know for a fact that they exist (as we were able to install 2.1.0).

I can't confirm this issue yet - inspecting https://pypi.org/simple/mysqlclient/ and https://pypi.org/pypi/mysqlclient/json/ both confirm the same set of built distributions.
How are you confirming that linux_x86_64.whl versions exists?

I just realized that pip was grabbing source and building the wheel, so looks like it might be a different issue entirely.

authors: colcrunch miketheman

repo: pypa/warehouse