text:
Tool calling with Gemini:  AssertionError: Expected ArgsObject, got ArgsJson(args_json='{"foo":"bar"}')

I had my first bot working with OpenAI but wanted to try gemini-1.5-flash; it breaks with:
    |   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    |     return await anext(self.gen)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/martin/proj/foreclosures/env/lib/python3.12/site-packages/pydantic_ai/models/gemini.py", line 182, in _make_request
    |     either_content = self._message_to_gemini(m)
    |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/martin/proj/foreclosures/env/lib/python3.12/site-packages/pydantic_ai/models/gemini.py", line 267, in _message_to_gemini
    |     return _utils.Either(right=_content_function_call(m))
    |                                ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/martin/proj/foreclosures/env/lib/python3.12/site-packages/pydantic_ai/models/gemini.py", line 401, in _content_function_call
    |     parts: list[_GeminiPartUnion] = [_function_call_part_from_call(t) for t in m.calls]
    |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/martin/proj/foreclosures/env/lib/python3.12/site-packages/pydantic_ai/models/gemini.py", line 428, in _function_call_part_from_call
    |     assert isinstance(tool.args, ArgsDict), f'Expected ArgsObject, got {tool.args}'
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | AssertionError: Expected ArgsObject, got ArgsJson(args_json='{"foo":"bar"}')

I'm using
pydantic                                 2.10.3
pydantic-ai                              0.0.12
pydantic-ai-examples                     0.0.12
pydantic-ai-slim                         0.0.12
pydantic_core                            2.27.1
pydantic-settings                        2.6.1

@sydney-runkle yes, everything you need. how can I help?

@runa,
Could you send the code you used to produce this issue? Specifically, the agent definition and tool call definitions? Thanks!

Sure:
agent = Agent(
    'gemini-1.5-flash',
    result_type=str,
    system_prompt=(
        """blabla"""
    ),
)

@agent.tool_plain
async def property_info(parcel_id: str) -> str:
    """return the property information for a given parcel id.

    Args:
        parcel_id (str): The parcel id to search for.

    Returns:
        str: The results of the search.
    """
    return blah.property_info(parcel_id)

@runa,
Perhaps a bit more context - what's a reasonable system prompt to use? blah is undefined... Hoping to be able to run this locally to reproduce your issue.

Sorry, I'm new debugging this kind of things :)
This should work:
please give me more information about parcel 1234

Hmm, not able to repro your issue:
from pydantic_ai import Agent

agent = Agent(
    'gemini-1.5-flash',
    system_prompt='Use the `parcel_info` function to return more information about a parcel.',
)

@agent.tool_plain
async def parcel_info(parcel_id: int) -> str:
    """Return the property information for a given parcel id.

    Args:
        parcel_id (str): The parcel id to search for.

    Returns:
        str: The results of the search.
    """
    return f'info: {parcel_id}'

response = agent.run_sync('Please give me more info about parcel 1234')
print(response.data)
"""
OK. Here's the information I have about parcel 1234: info: 1234.
Is there anything else I can help you with?
"""

What would help us the most here is if you could provide one file with an agent config, tool config, and run that reproduces this issue. Happy to help with that!

I think this fixes the issue:
diff --git a/pydantic_ai_slim/pydantic_ai/models/gemini.py b/pydantic_ai_slim/pydantic_ai/models/gemini.py
index d1739e1..f642fb2 100644
--- a/pydantic_ai_slim/pydantic_ai/models/gemini.py
+++ b/pydantic_ai_slim/pydantic_ai/models/gemini.py
@@ -12,6 +12,7 @@ from typing import Annotated, Any, Literal, Protocol, Union
 import pydantic_core
 from httpx import USE_CLIENT_DEFAULT, AsyncClient as AsyncHTTPClient, Response as HTTPResponse
 from pydantic import Discriminator, Field, Tag
+from pydantic_core import from_json
 from typing_extensions import NotRequired, TypedDict, TypeGuard, assert_never

 from .. import UnexpectedModelBehavior, _pydantic, _utils, exceptions, result
@@ -23,7 +24,7 @@ from ..messages import (
     ModelTextResponse,
     RetryPrompt,
     ToolCall,
-    ToolReturn,
+    ToolReturn, ArgsJson,
 )
 from ..settings import ModelSettings
 from ..tools import ToolDefinition
@@ -463,8 +464,15 @@ class _GeminiFunctionCallPart(TypedDict):

 def _function_call_part_from_call(tool: ToolCall) -> _GeminiFunctionCallPart:
-    assert isinstance(tool.args, ArgsDict), f'Expected ArgsObject, got {tool.args}'
-    return _GeminiFunctionCallPart(function_call=_GeminiFunctionCall(name=tool.tool_name, args=tool.args.args_dict))
+    args = tool.args
+    if isinstance(args, ArgsDict):
+        args_dict = args.args_dict
+    elif isinstance(args, ArgsJson):  # type: ignore[reportUnnecessaryIsInstance]
+        args_dict = from_json(args.args_json)
+    else:
+        assert_never(args)
+
+    return _GeminiFunctionCallPart(function_call=_GeminiFunctionCall(name=tool.tool_name, args=args_dict))

 def _structured_response_from_parts(

But I think it might make sense to make this a method on ToolCall. There are some open PRs that do heavier refactors, once we get those merged (hopefully in the next 24 hours) I think we can add this fix.

authors: dmontagu runa sydney-runkle

repo: pydantic/pydantic-ai


text:
allow typing-extensions-4.7.0+ on pypy3.9+

Change Summary
Remove the PyPy-specific version restriction on typing-extensions when PyPy3.9 or newer is used.  The bug cited
in 12073b84563e3c74da9bbf847f6af88c8c229c12 is specific to PyPy3.7 and PyPy3.8.
Related issue number
(discussed in https://github.com/pydantic/pydantic-core/commit/12073b84563e3c74da9bbf847f6af88c8c229c12#r120315009)
Checklist

[ ] Unit tests for the changes exist
[ ] Documentation reflects the changes where applicable
[x] Pydantic tests pass with this pydantic-core (except for expected changes)
[x] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @davidhewitt

please review

Thanks!

source: gharchive/pull-request

repo: pydantic/pydantic-core

license: MIT

license_type: permissive

license_source: github-api


text:
Fix array of floats bug

This PR resolves the bug documented here: https://github.com/pydata/pandas-gbq/issues/123
Basically, the custom dtype handling we have to convert missing values to NaNs fails when the field in BigQuery is an array (that is, in repeated mode). We simply set the dtype to an object if it is an array.
Added a test to check for this in the future. All previous tests still pass. Can also verify the bug and that this fixes by running:
gbq.read_gbq("select [1.1,2.2,3.3]","project",dialect="standard")

@tswast @MtDersvan

Codecov Report

Merging #134 into master will decrease coverage by 44.84%.
The diff coverage is 25%.

@@             Coverage Diff             @@
##           master     #134       +/-   ##
===========================================
- Coverage   75.83%   30.98%   -44.85%
===========================================
  Files           8        8
  Lines        1626     1630        +4
===========================================
- Hits         1233      505      -728
- Misses        393     1125      +732

Impacted Files
Coverage Δ

pandas_gbq/gbq.py
20.56% <ø> (-56.24%)
:arrow_down:

pandas_gbq/tests/test_gbq.py
26.88% <25%> (-56.77%)
:arrow_down:

pandas_gbq/_load.py
62.5% <0%> (-35%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5666e6d...ec28045. Read the comment docs.

Oh, and mind adding an entry to the changelog? This would be for version 0.3.2 / [TBD]

@tswast Hm, most of the tests were skipped, was there something I was supposed to do? https://travis-ci.org/jasonqng/pandas-gbq/builds/345048014
Pytest runs on my local and everything passes here.

@tswast All tests pass: https://travis-ci.org/jasonqng/pandas-gbq/builds/345048014

authors: codecov-io jasonqng tswast

repo: pydata/pandas-gbq

license_source: bigquery


text:
API: Implement pipe protocol

Today @shoyer and I were talking about a new "protocol" that will let us sidestep the whole macro / method chaining issue.  The basic idea is that pandas objects define a pipe method (ideally other libs will implement this to, assuming this is useful).
The method is just
def pipe(self, func, *args, **kwargs):
    return func(self, *args, **kwargs)

That's it. This lets you write code like:
(sns.load_dataset('iris')
    .query("sepal_length > 5")
    .assign(sepal_ratio = lambda x: x.sepal_length / x.sepal_width)
    .pipe(lambda _: sns.violinplot(x='species', y='sepal_ratio', data=_))
)

seaborn didn't have to do anything! If the DataFrame is the first argument to the function, things are even simpler:
def f(x):
    return x / 2

df[['sepal_length']].pipe(f)

As pointed out by Stephan in the comments, libraries can work around the need for a  lambda _: to handle which argument the data goes into if we make a slight change to pipe.
def pipe(self, func, *args, **kwargs):
    if hasattr(func, 'pipe_arg'):
        kwargs[func.pipe_arg] = self
        return func(*args, **kwargs)
    return func(self, *args, **kwargs)

Libraries could then decorate their functions to denote the kwarg expecting the data.
@pipe_arg('data')
def sns.violinsplot(x=None, y=None, hue=None, data=None):

and users write
df.pipe(sns.violinplot, x='species', y='sepal_ratio')

A relevant thread on python-ideas, started by @mrocklin: https://mail.python.org/pipermail/python-ideas/2015-March/032745.html
This doesn't achieve everything macros could. We still can't do things like df.plot(x=x_col, y=y_col) where x_col and y_col are captured by df's namespace. But it may be good enough.
Going to cc a bunch of people here, who've had interest in the past.
@shoyer
@mrocklin
@datnamer
@dalejung

I really like how simple the implementation is. No PEPs, no hacking on CPython, others should be able to implement the same protocol.
One (maybe) drawback is that we must rely on people to be responsible and not modify the self being passed in. But that's a very pythonic sort of "problem". Anyone, I look forward to the discussion :)

http://toolz.readthedocs.org/en/latest/api.html#toolz.functoolz.pipe

I like the idea of our community establishing and using protocols more heavily.  Will ponder this one for a while.

@jreback I've used that in the past, and this would be very similar. toolz.pipe is less convenient for interactive use though, I think. You need to define each function ahead of time, including simple functions to get the data to pipe into the correct position for functions that don't expect dataas the first argument. We'll still have to do that withdf.pipe`, but I think it's easier to do as you work through the chain.

We might also extend pipe to allow functions to customize which argument is used for chaining. This would be useful for seaborn, which currently takes dataframes in the data keyword argument:
# in pandas
def pipe(self, func, *args, **kwargs):
    if hasattr(func, ‘pipe_arg’):
        kwargs[func.pipe_arg] = self
        return func(*args, **kwargs)
    return func(self, *args, **kwargs)

# in seaborn (personally, I would probably implement this with a decorator)
sns.violinplot.pipe_arg = 'data'

# in user code
df.pipe(sns.violinplot, x='species', y='sepal_ratio')

Alternatively, we could just ask @mwaskom to break Seaborn's API again ;).

I agree that we can probably make something more pandas-y than toolz.pipe which, while convenient in its own way, doesn't jive particularly well with the pandas flow.

Is it possible to get infix syntax using a decorator? https://github.com/JulienPalard/Pipe

That's done by overriding __or__ though, right? I wouldn't feel comfortable doing that in library code.

@datnamer Yes, but not satisfactorily without macros. The problem is that to get df >> sns.violinplot('x', 'y') to work properly, you need to allow sns.violinplot('x', 'y'), even though that would usually raise an error because the required data argument is not provided.
So turning functions into pipe objects means that their non-piped usability is compromised. There are ways around this, e.g., using tuples like dask: df >> (sns.violinplot, 'x', 'y'), but >> is only one less character than .pipe (which is much more explicit).

This isn't a viable candidate for a Pandas solution, but I thought I'd point out toolz.curry which lives in the same space.

Following is a list of pipes which R's magrittr package provides (the package is used by dplyr).
http://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html

pipe: current example (and main usecase). %>% in magrittr.
tee: apply the function to left value, and return the left value as it is. %T>% in magrittr.
pipe and assignment: corresponding to panda's inplace option, %<>%  in magrittr.

If any of them is useful, preparing an option (or method) might be useful.

tee should be quite easy as a separate method. Just apply func and return self.
df.tee(sns.violinplot, x='x', y='y')
    .pipe(sm.OLS.from_formula('y ~ x'))

I do consider this experimental though. My initial thought was to start slow and see how things go so I didn't include that in the initial proposal.

:+1: I like this in general as an alternative to the current df.plot.  It also fits pretty well with where I (speaking only for me) want to push the mpl API.
tee worries me a bit as you lose the ability to get the Artists back which is important if you want to do anything interactive.
Instead of pushing the decorator out to other libraries, I think it would be better to encourage end user to decorate on the fly, ex
pd_violin = pd.pipeify(sns.violin)
artists = df.pipe(pd_violin, x='foo', y='bar')

or to give pipe a reserved kwarg like _pipe_target so
def pipe(self, func, _pipe_target=None, *args, **kwargs):
    if pipe_target is not None:
        kwargs[_pipe_target] = self
        return func(*args, **kwargs)
    else:
        return func(self, *args, **kwargs)

I've been playing with this some today. Another drawback with this API is that the user must be a bit more disciplined about positional arguments vs kwargs. Assuming df.pipe and that seaborn's violin plot has been decorated.
Either of these work for seaborn:

In [11]: sns.violinplot('sepal_length', 'sepal_width', data=iris)
Out[11]: <matplotlib.axes._subplots.AxesSubplot at 0x116d85c88>

In [12]: sns.violinplot(x='sepal_length', y='sepal_width', data=iris)
Out[12]: <matplotlib.axes._subplots.AxesSubplot at 0x116d85c88>

With .pipe,
In [16]: sns.violinplot('sepal_length', 'sepal_width', data=iris)
Out[16]: <matplotlib.axes._subplots.AxesSubplot at 0x116d85c88>

works as expected, but (the clearer)
In [15]: iris.pipe(sns.violinplot, x='sepal_length', y='sepal_width')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-dc30bf133b66> in <module>()
----> 1 iris.pipe(sns.violinplot, x='sepal_length', y='sepal_width')
 ...
TypeError: violinplot() got multiple values for argument 'x'

fails.

Another drawback is that libraries wishing to support pandas <0.17 will need to implement there own pipeable decorator. I've got one in pandas/util/decorators.py
def pipeable(pipe_arg):
    def decorate(func):
        setattr(func, 'pipe_arg', pipe_arg)

        @wraps(func)
        def wrapper(*args, **kwargs):
            return func(*args, **kwargs)
        return wrapper
    return decorate

It's not difficult, but it's essential that everyone agrees on the name for pipe_arg. This is a point in favor of @tacaswell's suggestion, but I'm pretty attached to simply being able to write df.pipe(sns.violinplot, 'x', 'y') :)

@josef-pkt would statsmodels be interested in adding the decorators to support this? I see it working quite nicely with the formula API. I'll send over a PR if you want. I believe you're hoping to have a release in the not too distant future? It'd be nice to lay the groundwork in statsmodels 0.7. Even if that comes out before the next version of pandas, adding the decorators should entirely transparent to pre 0.17 pandas and just work for pandas >= 0.17.

Another drawback with this API is that the user must be a bit more disciplined about positional arguments vs kwargs. Assuming df.pipe and that seaborn's violin plot has been decorated.

Hm, tricky. Though, FacetGrid.map works the same way, so there's precedent.

What's the difference between
model = df.pipe(sm.OLS.from_formula('y ~ x')) and
model = sm.OLS.from_formula('y ~ x', df) ?
Would this be worth the increased complexity and fragility and possibly reduced flexibility?
It might be more useful in standalone function like some plots, or hypothesis tests or data transformations.
Also, I guess more useful than getting a model could be a wrapper for something like prediction, (assuming we are not interested in other results statistics):
predicted = smf.ols('y ~ x', df).fit().predict(df2)    returns currently a numpy array
or
fitted = sm.OLS.from_formula('y ~ x', df).fit().fittedvalues     returns a pandas.Series
The main problem I see is that statsmodels is largely model centric and not data centric like pandas and seaborn. The other design issue is that statsmodels has little functional code, I just spend most of two weeks chaining models and results through multiple inheritance and similar. (We need to modify or chain 3 to 7 methods of a Maximum Likelihood Model.)
Personally, I find magrittr code pretty ugly, since my tradition is in repeated assignments to temporaries a la matlab/numpy.
Nevertheless, the best user interfaces, pandas wrapper and formulas, were included without my involvement. And we do have quite a bit of non-model functions.

@josef-pkt No functional difference between those first two. And readabilty-wise they're pretty much the same.
The (potential) benefit comes in chains, but that case is admittedly weaker for statsmodels since it's typically the last step in the chain, and you'll probably want to have a reference to the DataFrame that went into the model.

@TomAugspurger   We are holding on to the dataframe right now because we use it for some optional methods. I also see more benefits in other functions like in my examples where statsmodels is not necessarily the last step in the chain.
In some cases the main advantage in interactive work of having a method available is that I don't need an explicit import. That's one advantage of a plot method, since most of the time I don't import matplotlib automatically.
Just an idea: instead of a pipe we could add (monkey patch) a method on pandas.DataFrame, like df.sm(...) which automatically uses the function or class in the sm namespace and replaces the data argument with df. I guess we don't have enough consistent usage of a data argument yet to make that work consistently. df.sns('violinplot', ...) ?

@TomAugspurger Hmm -- I'm not seeing issue you've described with seaborn. Are you sure you're running the latest dev version? Here's my example notebook, in which each of iris.pipe(sns.violinplot, 'species', 'sepal_width') and iris.pipe(sns.violinplot, x='species', y='sepal_width') work: http://nbviewer.ipython.org/gist/shoyer/3f73dc7ffa03883d3a43
I suppose we can include a utility function for decorating functions as pipeable, but really it's even easier than you think. For example, you can just add the attribute directly: sns.violinplot.pip_arg = 'data'. Or if you prefer a decorator, you can use something even simpler:
def pipeable(pipe_arg):
    def decorate(func):
        func.pipe_arg = pipe_arg
        return func
    return decorate

@josef-pkt To clarify, your first example should actually be: model = df.pipe(sm.OLS.from_formula, 'y ~ x')
Coming back to the API design discussion, one of my favorite things about the original, simple version of pipe is that it is entirely explicit and there is absolutely no magic:
def pipe(self, func, *args, **kwargs):
    return func(self, *args, **kwargs)

So no, I don't think it's a good idea to encourage monkey patching DataFrame from external libraries. It's not explicit and very much goes against the norms of idiomatic Python.
Similarly, as much as I want sns.violinplot to be pipeable, I'm not convinced it's actually a good idea to encourage library authors to decorate their functions to denote pipe targets. This adds a whole additional level of complexity which effectively needs to be documented as part of the function signature, but which isn't actually part of the function signature and thus will be far less discoverable. We could do it, but I would only recommend that library authors use it with extreme reservations -- it's far better to just accept data as the first argument to any functions that you want to be pipeable.
So something like @tacaswell's suggestion might be the better way to go. A few other options:

Pipe automatically uses the data argument if present, as determined by inspecting the function signature (with inspect.getargspec). Unfortunately, the inspect module is quite fragile.
df.pipe.data(sns.violinplot, 'x', 'y') or df.pipe['data'](sns.violinplot, 'x', 'y'). These aren't very pretty and would have a much more complex implementation.
We allow the first argument of pipe to be a string, in which case it's used to specify the target argument: df.pipe('data', sns.violinplot, 'x', 'y'). This is also messy to implement.
We use some sort of sentinel object to signal the pipe argument: df.pipe(sns.violinplot, 'x', 'y', data=pipe.HERE). But now pipe needs to import a shared module instead of simply being a protocol.

Of these, I think my favorite is @tacaswell's original suggestion df.pipe(sns.violinplot, 'x', 'y', _target='data'). Or sticking with the simple implementation, and encouraging use of lambda: df.pipe(lambda df: sns.violinplot('x', 'y', data=df))
@mwaskom can you remind us why you don't want to keep data as the first argument to sns.violinplot? I guess it's because it's the only way to make sns.FacetGrid.map work reliability without any magic? This is basically the same problem I'm trying to avoid here (to avoid implementing pipe_arg). On the plus side, at least pipe will already pair well with the explicit sns.FacetGrid API.

I just realized that method or functions that take a dataframe as a first argument will work without any changes (even if the data argument is called exog)
with the original proposed pipe
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf

df = pd.DataFrame(np.random.randn(10, 3))
df.columns = ["a", "b", "c"]
result = smf.ols('a ~ b', df).fit()

def pipe(self, func, *args, **kwargs):
    return func(self, *args, **kwargs)

# monkey to get the example to work
pd.DataFrame.pipe = pipe

predicted = df.pipe(result.predict)
print(predicted)
print(df[df['a'] < 0].pipe(result.predict))

>>> df.pipe(result.predict)
array([ 0.77901375,  0.95606063,  0.72349671,  0.96823813,  1.33929812,
        0.42730606,  0.57528495,  0.47756362,  0.52891164,  0.40572964])
>>> df[df['a'] < 0].pipe(result.predict)
array([ 0.77901375,  0.42730606,  0.57528495])

or with a lambda function
>>> pt = lambda exog, formula: smf.ols(formula, df).fit().summary2().tables[1]

>>> df.pipe(pt, 'a ~ b')
              Coef.  Std.Err.         t     P>|t|    [0.025    0.975]
Intercept  0.560831  0.430557  1.302571  0.228965 -0.432035  1.553698
b          0.369255  0.490102  0.753426  0.472782 -0.760921  1.499431

>>> df.pipe(pt, 'a ~ c')
              Coef.  Std.Err.         t     P>|t|    [0.025    0.975]
Intercept  0.780714  0.392976  1.986672  0.082195 -0.125490  1.686917
c          0.375825  0.593915  0.632793  0.544528 -0.993745  1.745395

>>> df.pipe(pt, 'a ~ b + c')
              Coef.  Std.Err.         t     P>|t|    [0.025    0.975]
Intercept  0.561965  0.422812  1.329111  0.225494 -0.437828  1.561757
b          0.657157  0.543692  1.208693  0.266020 -0.628471  1.942784
c          0.742642  0.652395  1.138331  0.292437 -0.800028  2.285312

and
>>> df[df['a'] < 0].pipe(pt, 'a ~ b + c')
              Coef.  Std.Err.         t     P>|t|    [0.025    0.975]
Intercept  0.561965  0.422812  1.329111  0.225494 -0.437828  1.561757
b          0.657157  0.543692  1.208693  0.266020 -0.628471  1.942784
c          0.742642  0.652395  1.138331  0.292437 -0.800028  2.285312

@shoyer I must have had a bug in mine. It works both with named and positional arguments.

@mwaskom can you remind us why you don't want to keep data as the first argument to sns.violinplot? I guess it's because it's the only way to make sns.FacetGrid.map work reliability without any magic? This is basically the same problem I'm trying to avoid here (to avoid implementing pipe_arg). On the plus side, at least pipe will already pair well with the explicit sns.FacetGrid API.

Right, one of the motivations for the change was to allow things to work better with FacetGrid, but also to bring boxplot and violinplot in line with most of the rest of the package. I guess there could be some argument that all seaborn function should take data first (like ggplot), but that would be an even more dramatic change. Also, for almost all the functions, data is optional and the positional arguments are happy to take vectors directly; I think originally I had the mindset of extending the basic matplotlib api and then adding the ability to name variables in a data object later, rather than orienting everything around a common dataframe representation...

@mwaskom OK, fair enough. I agree that for Seaborn, for which DataFrames are optional, it makes sense for data not to be the first argument.
I suppose another option would be to support an alternative API in Seaborn itself, e.g., sns.pipeable.violinplot. This seems nicer than the other explicit options we've seen. Again I'm still torn about whether or not pipe_arg is a good idea.

I suppose another option would be to support an alternative API in Seaborn itself, e.g., sns.pipeable.violinplot. This seems nicer than the other explicit options we've seen. Again I'm still torn about whether or not pipe_arg is a good idea.

This would be reasonably easy to implement on my end, especially for the categorical plots, so that would be fine. Though, for the user, that approach probably ends up involving almost as much extra typing as just making an intermediate dataframe :)

So, attempting a summary.

People are +1 to the idea of a .pipe method in general (modulo details)
There's been no discussion of the name, which I take to mean .pipe is the correct name(?)
The contention is on how to pipe data to the correct position in func. Do we

1: Have users pass in a lambda _: func(a, b, _)
2: Include a special _pipe_target keyword in .pipe indicating where to go
3: Have library authors / users decorate their functions (and rewrite docs / examples).

FWIW, I'm coming around to 1 (maybe 2). Taking things slowly and seeing how the community responds could be good. The nice thing about 1 is it gives us the most flexibility in the future if we see something that needs to change, without breaking anyones code. With 2 and especially 3 we only get one shot.
side note: I'm moving this week so I won't have much time for this other than today. I've written a section in the release notes and basics.rst. I plan to go through the rest of the docs today looking for places where .pipe would be appropriate (there may not be much / any). So I'll open up a PR this afternoon or if someone beats me to it I can do a PR against their branch for the docs.

Another idea I have been noodling with but have not written out anywhere yet is for pandas to provide an 'unpack' decorator to throwing data at non-panadas aware code (note really silly name given because I don't want to think about a good name right now)

def unpackify_decorator(func):
    def wrapped(df, column_map, *args, **kwargs):
        for k, v in column_map.items():
            if k in kwargs:
                raise ValueError()
            kwargs[k] = df[v].values
        return func(*args, **kwargs)
    return wrapped

or something similar so the usage with pipe would look like
awarified = unpackify_decorator(ax.scatter)
df.pipe(awarififed, {'x': 'age', 'y': 'total_commit', 'c': 'number_of_projects', 's': 'net_LoC'}, cmap='gray')

Having written this out it might also make sense to have this logic be part of the pipe method.
def pipe(self, func, _pipe_target=None, *args, **kwargs):
    if is_string(_pipe_target):
        kwargs[_pipe_target] = self
        return func(*args, **kwargs)
    elif is_dict(_pipe_target):
        for k, v in _pipe_target.items():
           if k in kwargs:
                raise ValueError()
            kwargs[k] = self[v].values
        return func(*args, **kwargs)
    elif is_tuple(_pipe_target):
        args = tuple(self[k] for k in _pipe_target) + args
        return func(*args, **kwargs)
    else:
        return func(self, *args, **kwargs)

so the uasge would look like
df.pipe(ax.scatter, {'x': 'age', 'y': 'total_commit', 'c': 'net_LoC', 's': 'number_of_projects'}, cmap='gray')

or
df.pipe(ax.scatter, ('age', 'total_commit', 'net_LoC', 'number_of_projects'), cmap='gray')

It is a bit verbose (but I picked a complicated example mapping 4 columns and used really descriptive column names), but it I think captures all of the simple cases for functions.  If you want to mix the order of user supplied args and data extracted from df you still have to write a lambda, but doing that in a general way would be ugly to write and very difficult to use.

And thinking about this a bit more, having pipe as a stand-alone top-level function broadcast_labeled_data(dict_like, function,  _target, *args, **kwargs) is probably a useful thing to have and I want to have a version of that available in mpl.

I think I agree with @TomAugspurger's suggestion to avoid defining a protocol for now, but I just want to suggest that if we do define a protocol, then instead of looking for a magic pipe_arg attribute it would be more elegant to look for a pipe_fn(pipe_arg, *args, **kwargs) method that can then whatever we want. It's strictly more flexible and seems more in keeping with other protocols in python.
I'd also like to strongly suggest that someone try and get other Python communities that use method chaning involved in the discussion (e.g. SQLAlchemy?), by contacting them directly and/or by posting to python-ideas, b/c more eyes and broader buy-in can only help make de facto standards better.

@njsmith Indeed, those are both great ideas. So the protocol version looks like:
def pipe(self, func, *args, **kwargs):
    pipe_func = getattr(func, '__pipe_func__', func)
    return pipe_func(self, *args, **kwargs)

This nicely opens the door to handling complex use cases like those @tacaswell suggests without putting that complexity in the pipe method itself.
Let's update the first post in this thread (@TomAugspurger is it OK if I do that?); then I'll post to r/Python and potentially Python ideas.

What is self if we would add a __pipe_func__?
I might want self to be a results instance in statsmodels, then this would provide a cheap way for us to use other postestimation functions   result.pipe(sm.graphics.regressionplot, [1]) result.pipe(outlier_test).dataframe() or something like that.

@josef-pkt That's totally up to you as the library author, but the best practice would to be return the standard return value of the function. So indeed that's probably a results object from statsmodels. There's no particular reason why it should be a dataframe, for example. (It's messy enough to allow for different parsing of input arguments when a function is piped.)

@shoyer I updated the original post. Feel free to make edits as you see necessary.
To make sure I understand the intent of pipe_func = getattr(func, '__pipe_func__', func), the idea is to leave it up to the function author to decide how it is called, rather than the author of .pipe? So using seaborn as an example
def violinplot(x=None, y=None, hue=None, data=None):
    code

How would seaborn implement __pipe_func__? As simple as
violinplot.__pipe_func__ = lambda data, x, y, hue: violinplot(x, y, hue, data)

I could be missing something simple.

@shoyer the way I understood your example is that self is the input argument not the return.
Is the discussion now about general pipe patterns across libraries or specifically to support piping of dataframes into downstream functions?
in some case it might be possible to dispatch on types of self in __pipe_func__

@TomAugspurger

To make sure I understand the intent of pipe_func = getattr(func, '__pipe_func__', func), the idea is to leave it up to the function author to decide how it is called, rather than the author of .pipe?

Yes, exactly.
For Seaborn, your example would work, though to handle optional arguments properly you'd want something like this:
violinplot.__pipe_func__ = lambda data, *args, **kwargs: violinplot(*args, data=data, **kwargs)

In practice, I would probably write a decorator to consolidate the logic:
def pipeable(func):
    def pipe_func(data, *args, **kwargs):
        return func(*args, data=data, **kwargs)
    func.__pipe_func__ = pipe_func
    return func

# now we decorate all the Seaborn methods as pipeable

@pipeable
def violinplot(x=None, y=None, hue=None, data=None, ...):
    # ...

@josef-pkt I misread your comment -- indeed, it would also make sense to define .pipe on statsmodels objects. self is just the object on which you are calling the pipe method. I agree, dispatching on the type of self in __pipe_func__ would be entirely appropriate.

Yep, your way is simpler. I'll add it to the original. I posted on our ML. Feel free to post to r/python and / or ideas

Something related but not much practical value:
def recursive(df,fn):
    """
    recursive function application
    on a dataframe, use like

    >>> df.recursive(lambda self: ((
    ...     lambda _df, fn:
    ...         _df if _df.max() > 1 else
    ...         fn(_df*1.01, fn))
    ... )
    """
    return fn(df,fn)

pd.DataFrame.recursive = recursive

CCing @pzwang and @bryevdv who might be interested in working this into blaze and or bokeh.

Please, note that this looks much like the stream library. (maybe stream is a better name?)
Also, how are pipes going to play with the recent  await and yield from tools. This coroutine syntax sugar plays well with composability ("piping") and seems natural to me that pipes should be implemented with it.
Not being picky or making requests, but just trying to bring these discussions up in order to avoid future problems.

@FelipeLema It's best to think of everything we can ahead of time.
I don't think this proposal really interacts with async / await. It's entirely up to the functions being called in pipe whether they want to use parallel execution / asycio.

FWIW, bokeh's charting API all take their data as the first argument, so this will work seamlessly.

The lower-level glyph API doesn't flow quite as smoothly, but I think that's ok.

FYI, I finally got around to posting on Python-Ideas: https://mail.python.org/pipermail/python-ideas/2015-May/033673.html
@TomAugspurger  Nice to see this already works with bokeh!

HI @TomAugspurger and others. The timing here is auspicious. A few of us are convened in Austin this week to hash out a final interface for the charts interface, so it would be good to have any input from this issue. What I got from my brief reading was "data as the first argument" is good. I don't expect that that would change, but it's good to know. If there is any other things that you would like to pass on to make sure things integrate as well as possible, please let me know here (or in an issue on the Bokeh GH). Ping @fpliger @rothnic

I definitely don't see bokeh.charts "data as the first argument" interface changing either. Also think it's a perfect timing for suggestions.. :wink:

FYI, there is a new piping discussion here : https://mail.python.org/pipermail/python-ideas/2015-May/033491.html

@josef-pkt @mwaskom @tacaswell How important to you is the ability of the pipe argument to control what it does (via __pipe_func__)?  As much as I love that flexibility, it's a little un-Pythonic, and I've gotten pushback every time I've introduced it to non-PyData programmers.
@bryevdv @fpliger Seems like bokeh.charts has the right API to use this immediately :). "Data the first argument" is indeed helpful :).

I wonder if that's just a fundamental difference. I spend way more time at the interpreter as a user of pandas than as a developer of it.  And that's when the pipe_func is useful.
Anyway, I'm fine with excluding pipe_func for 0.17. We'd be able to add it later without breaking backwards compatibility if people find the lamdas too annoying.

I am :-1: on pipe looking for attributes on the input function to sort out what to do.  The mess that will result if more than one library adopts using __pipe_func__ seems worse than the benefits.  I am also not convinced that __pipe_func__ makes it more flexible, I think it makes everything far more brittle.
I am (predictably) in favor of my suggestion above to specify how to un-pack data frames into non-data frame aware function as input to pipe, providing a helper function factory which does the aformentioned remapping, or to just insisting that users write explicitly data-frame aware functions (either natively or to wrap whatever they have) to use pipe.
Speaking for my self, I would be against mpl adding __pipe_func__ to our functions if this does get adopted.

@tacaswell I feel pretty strongly that pipe itself needs to be kept to a bare minimum number of lines of code if we are going to encourage its adoption by other libraries. Hence __pipe_ufunc__ -- a way for library authors to control that behavior without needing to put that complexity in pipe itself.
The helper function factory for unpacking dataframes seems like a decent idea, thought I'm not sure it's necessary to put it in pandas.
Indeed, we may want to simply encourage authors that would use __pipe_func__ (e.g., Seaborn) to instead consider providing a separate namespace with "pipeable" functions, e.g., seaborn.pipeable.violinplot, perhaps imported by convention with import seaborn.pipeable as snsp.

Perhaps the concern is that one could consider "piping" in contexts other
than pandas.  Perhaps the protocol should have "DataFrame" in the name.
On Jun 1, 2015 9:14 AM, "Stephan Hoyer" notifications@github.com wrote:

@tacaswell https://github.com/tacaswell I feel pretty strongly that pipe
itself needs to be kept to a bare minimum number of lines of code if we are
going to encourage its adoption by other libraries. Hence pipe_ufunc
-- a way for library authors to control that behavior without needing to
put that complexity in pipe itself.
The helper function factory for unpacking dataframes seems like a decent
idea, thought I'm not sure it's necessary to put it in pandas.
Indeed, we may want to simply encourage authors that would use
pipe_func (e.g., Seaborn) to instead consider providing a separate
namespace with "pipeable" functions, e.g., seaborn.pipeable.violinplot,
perhaps imported by convention with import seaborn.pipeable as snsp.
—
Reply to this email directly or view it on GitHub
https://github.com/pydata/pandas/issues/10129#issuecomment-107336106.

The helper function factory for unpacking dataframes seems like a decent idea, thought I'm not sure it's necessary to put it in pandas.

Where else would it make sense for it to live?

Indeed, we may want to simply encourage authors that would use pipe_func (e.g., Seaborn) to instead consider providing a separate namespace with "pipeable" functions, e.g., seaborn.pipeable.violinplot, perhaps imported by convention with import seaborn.pipeable as snsp.

This strikes me as a recipe for confusing users. How many people will first encounter a particular function in a "piped" context, then import the wrong thing and not understand why other examples don't work? The duplication of functions only makes sense if you have a pretty thorough understanding of pandas implementation details.
What if instead it were possible to do
DataFrame(...).pipe("data", sns.violinplot, "day", "tip")

Where if the first argument is a string and the second is a callable, the first is interpreted as the keyword arg to pipe the dataframe into. I'm not sure if this would require manual inspection of an *args list or if it could be handled in a nice way with multiple dispatch based on argument type.

Could also maybe be more straightforward to do
DataFrame(...).pipe((sns.violinplot, "data"), "day, "tip")

then the code on the pandas side is simpler, and it's clear that the "data arg" is more tightly associated with what to call than what to plot (if that makes sense).

@mrocklin I like that. Instead of __pipe_func__ we would have __pipe_dataframe__. If anyone else wants their own pipe override, it's obvious they should use their own name for the special attribute (e.g., __pipe_formula__ or whatever).
Here's what @mwaskom's proposals look like (in code). Both of these seem pretty reasonable to me:
def pipe1(self, target, *args, **kwargs):
    if callable(target):
        return target(self, *args, **kwargs)
    else:
        kwargs[target] = self
        func = args[0]
        args = args[1:]
        return func(*args, **kwargs)

def pipe2(self, target, *args, **kwargs):
    if isinstance(target, tuple):
        func, data_arg = target
        kwargs[data_arg] = self
        return func(*args, **kwargs)
    else:
        return target(self, *args, **kwargs)

Summary
There wasn't too much feedback from either python-ideas or r/python. The most common response was more or less 'why are you using __?'
We're also pretty concerned with keeping .pipe itself clean and simple. This is our concern with the more complex implementations, e.g. here.
There's some worry that the data being piped may be too flexible to be really useful as a protocol. Perhaps we should make it DataFrame specific.
Tasks
These are all intertwined

Decide whether we really are implementing and promoting an ad-hoc protocol, or just implementing a useful method on DataFrame
Decide whether the benefits of  __pipe_arg__ outweigh its un-pythonicness
If we drop __pipe_arg__ do we have users write lambdas to shuffle args around, or accept tuples of (function, target)

As @TomAugspurger describes in the PR (#10253), in the dev meeting we settled on @mwaskom's tuple proposal:
def pipe(self, target, *args, **kwargs):
    # DataFrame(...).pipe((sns.violinplot, "data"), "day, "tip")
    if isinstance(target, tuple):
        func, data_arg = target
        kwargs[data_arg] = self
        return func(*args, **kwargs)
    else:
        return target(self, *args, **kwargs)

There's no magic, and it leaves the door open to future extensions. Please speak up if you have any further concerns.

I think this is a missed opportunity to make it easy to export data in a DataFrame to non-pandas aware libraries.

@tacaswell I also think that the simplicity of the implementation in #10253 is crucial.
And the boat hasn't entirely sailed on something like what you proposed with a _pipe_target specifying how self should be unpacked. I'd like to see what people do with the simpler implementation first.
Thanks everyone for the feedback and ideas (and civil discourse! <3 PyData)

@TomAugspurger Fair enough, I do see the case for keeping it as simple as possible.
As linked above, I have versions of the more verbose code in a PR against mpl which can also serve as a Guinea pig (and any feed back from the pandas folks on what I did wrong would be helpful).

source: gharchive/issue

authors: FelipeLema TomAugspurger bryevdv datnamer douglas-larocca fpliger josef-pkt jreback mrocklin mwaskom njsmith shoyer sinhrks tacaswell

repo: pydata/pandas


text:
Expose use_cftime option in open_zarr

use_cftime was recently added as an option to decode_cf and open_dataset to give users a little more control over how times are decoded (#2759).  It would be good if it was also available for open_zarr.  This perhaps doesn't have quite the importance, because open_zarr only works for single data stores, so there is no risk of decoding times to different types (e.g. as there was for open_mfdataset, #1263); however, it would still be nice to be able to silence serialization warnings that result from decoding times to cftime objects in some instances, e.g. #2754.

Hey, can I work on this issue? I'm still new to this so a little heads up would be very nice!

Absolutely - I think this should be as simple as adding a use_cftime argument to open_zarr, threading its value through as an argument to the call to conventions.decode_cf here:
https://github.com/pydata/xarray/blob/7b76f163394a35c9cd8013e835e9d0b2050fd9a6/xarray/backends/zarr.py#L542-L545
and adding a test to make sure it works.
For more general tips on contributing to xarray, see the contributing guide here.

Couldn't figure out how to implement the tests

@naomi-henderson, I was working on this. But lately, I have been quite busy so I haven't implemented the tests yet. You can fix it if you wish to.

@spencerkclark @dcherian for transparency I unpinned this to make room for the broken docs -> github link. Feel free to unpin the build warnings if this is still top-of-mind

authors: Geektrovert max-sixty spencerkclark

repo: pydata/xarray


text:
List of projects using xarray

I'm particularly interested in projects that we should highlight in our docs and other publications, either as evidence of the impact of xarray or because other xarray users would find it useful to know about them. We could put together an "Xarray ecosystem" docs page like pandas.
What I have so far falls into three categories:
Tools for analyzing global climate model output with xarray:

xgcm: for analyzing general circulation model output data
oocgcm: analysis of large gridded geophysical dataset
maps_xarray: wrapper to allow input of mpas data into xarray
marc_analysis: Analysis package for CESM/MARC experiments and output

Other weather/climate specific tools:

windspharm: wind spherical harmonics
eofs: empirical orthogonal functions
infinite-diff: xarray-based finite-differencing
aospy: Climate data analysis, database, and visualization

Non-climate:

Datashader: visualization for large data
cesium: machine learning for time series analysis
ptsa: EEG Time Series Analysis
pyGDX: GDX file data access
pycalphad: Computational Thermodynamics in Python

I'm currently developing spyfit, which is based on xarray and which is for handling data related to FTIR spectroscopy of the atmosphere. I hope to release it soon. It may fall into the 2nd category I think.
Depending on my position, I'd like to use xarray in other projects related either to atmospheric chemistry modeling or earth surface dynamics modeling!

@shoyer, we are in the nascent stage of development for an analysis package for MPAS-Ocean (https://github.com/MPAS-Dev/MPAS-Analysis) that will be leveraged for analysis in ACME (http://climatemodeling.science.energy.gov/projects/accelerated-climate-modeling-energy).

@shoyer I work in the NCL team at NCAR and am developing the soon-to-be released wrf-python package, which gives Python users the same functionality found in WRF-NCL.  The metadata is managed using xarray.

does anyone know of projects doing declarative visualization on top of xarray (like seaborn and altair have on pandas)?

@neuromusic we have some basic stuff in our own plotting module (though mostly a matplotlib wrapper), but also take a look at HoloViews which recently added xarray support.

@shoyer ahhh good to know. just stumbled on holoviews recently. thanks!!

(and definitely highlight HoloViews in the docs update)

I wrote regionmask to automatically create mask of regions defined by polygons (#501). Arbitrary longitude/ latitude grids can be passed. Handles xarray input and output.

Salem extends xarray to add geolocalised subsetting, masking, and plotting
(It definitely has some overlap with other tools on this list!)
If anyone wants to put together a PR to add a mention of this list to the docs, that would be appreciated. I'm happy to highlight any packages that make use of or integrate with xarray, but obvious let's highlight those projects that would be relevant to the largest slices of xarray users.

authors: benbovy bladwig1 mathause neuromusic pwolfram shoyer


text:
Migrate treenode module.

Migrate datatree's treenode module the base class representing node of a tree, with methods for accessing nodes and traversal.

[X] completes step 2 datatree/treenode.py #8572
[x] Tests added or updated
[N/A] User visible changes (including notable bug fixes) are documented in whats-new.rst
[X] Internal Changes (including notable bug fixes) are documented in whats-new.rst
[N/A] New functions/methods are listed in api.rst

overall thoughts on the design / implementation here?

So my overall thoughts as I was moving it / grokking it was that the separation of concerns between the movement, linking of nodes and parenting rules vs the finding by names and paths as pretty clear.  The hooks for before and after attaching children seem thoughtful.  I haven't dived deep into the datatree code yet but this all makes sense so far.  If you were to scrap this for a nested dictionary, wouldn't you end up with pretty complicated code to handle moving/adding nodes? My gut says this is a solid choice to represent tree data.

personally, would probably rename test_io.py to something related to
datatree io and move all of the tests into xarray/testsat the top level.

That seems fine to me. Integrate the IO tests because the backends are
integrated. Separate the tests of tree functionality so that it's easy to
test different parts of the codebase independently during development
(that's why having a separate test_treenode.py is useful). But whether
test_treenode.py lives in a subdirectory or not just doesn't really matter
I think.
On Mon, Feb 19, 2024, 11:02 AM Justus Magin @.***>
wrote:

@.**** commented on this pull request.
On xarray/tests/datatree/test_treenode.py
https://github.com/pydata/xarray/pull/8757#discussion_r1494845710:
let's discuss this in the meeting tomorrow, but yeah, I don't have any
strong opinions.
—
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/pull/8757#discussion_r1494845710, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AISNPI366CTJ6KUSLPD4PILYUOAS7AVCNFSM6AAAAABDK6TBVGVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQOBYHA4DONJVGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

So that failure looks like a common one as opposed to something from this PR.  Is there anything specific to address in the PR before it can be merged?

But whether test_treenode.py lives in a subdirectory or not just doesn't really matter
I think.

We made an executive decision in the datatree meeting today to not bother with subdirectories (for tests or code). We can always reorganise this later if we want to.

Is there anything specific to address in the PR before it can be merged?

I don't think so - @keewis what say you?

authors: TomNicholas flamingbear


text:
metal::packed_float3 is being generated (inappropriately?) and used in expressions without a cast, breaks calls to metal::dot

See this issue: https://github.com/gfx-rs/naga/issues/1561
Also this: https://stackoverflow.com/questions/54407927/no-matching-function-for-call-to-dot-and-reflect
I had to change:
        struct IncidentLight {
            color: vec3<f32>,
            direction: vec3<f32>,
            visible: bool,
        };
...
        struct BlinnPhongMaterial {
            diffuse_color: vec3<f32>,
            specular_color: vec3<f32>,
            specular_shininess: f32,
            specular_strength: f32,
        };

to
        struct IncidentLight {
            color: vec3<f32>,
            visible: bool,
            direction: vec3<f32>,
        };

...
        struct BlinnPhongMaterial {
            diffuse_color: vec3<f32>,
            specular_shininess: f32,
            specular_color: vec3<f32>,
            specular_strength: f32,
        };

in pygfx/renderers/wgpu/_shaderlib.py to get the cube example to run with Metal.
pygfx-0.1.12
wgpu-0.9.4
MacOS 10.14

Thanks for reporting this!
It's odd that the naga issue you linked has been fixed April last year, so that fix should have landed in wgpu-native by now. Also, I'm on a Mac too (MacOS 13.2) but my machine does not express the bug. Any thoughts on that?
That said, the fix seems simple enough. Would you mind preparing a PR for that?

Perhaps higher MacOS versions / better hardware support operations like dot() on packed operands? AFAIK the error occurs not because of packed_float3 per se but because dot() cannot operate on this data type. PR: https://github.com/pygfx/pygfx/pull/483

authors: almarklein sadaszewski

repo: pygfx/pygfx


text:
[DOC] Fix notes alignment

This PR fixes the notes alignment in the select_ functions.

🚀 Deployed on https://deploy-preview-1195--pyjanitor.netlify.app

authors: ericmjl samukweku

repo: pyjanitor-devs/pyjanitor


text:
Missing "menu" icon when viewing in narrow browser

The "menu" icon is missing in some places, but works on the main landing page.
Landing page menu

vs
Sponsors page menu

Closed via #125

repo: pyladies/global-conference


text:
slm_image from useq event

Made some progress! Tested with MightexPolygon1000 DMD. Submitting this pull request so we have a place for discussion, also there are some issues with useq-schema for which I'd be grateful for help.
With the current code, one can set slm_image.data to:

True / False single bool: All pixels are set to ON/OFF, looking up the default value which is hard coded in pymmcore-plus for some devices.
0-255 single int: sets all pixels to this value (these are both running setSLMPixelsTo)
[w,h,(rgb)] np.array of u8ints: sets DMD pixels to image (images use setSLMImage)
[w,h,(rgb)] np.array of bool: sets DMD pixels to image, replacing True values with the hardcoded default value.

Example to set all pixels to ON state:
event_slm_on = MDAEvent(slm_image=SLMImage(data=True))
mmc.mda.engine.setup_event(event_slm_on)

Issues:

There is an error in useq._base_model, _non_default_repr_args when comparing np.arrays. I just skipped this test for now, there is also a comment (consider using model_copy directly) - how should we proceed with this?

when using a mda_engine, slm_image is only applied if there is a single event in the list, see example below:

Works:
events = []
for slm_image_data in [0]:
    slm_image = SLMImage(data = slm_image_data,device=slm_dev)
    events.append(useq.MDAEvent(exposure=500,slm_image=slm_image))

thread = mmc.run_mda(events)
thread.join()

events = []
for slm_image_data in [255]:
    slm_image = SLMImage(data = slm_image_data,device=slm_dev)
    events.append(useq.MDAEvent(exposure=500,slm_image=slm_image))

thread = mmc.run_mda(events)
thread.join()

Doesn't work:
events = []
for slm_image_data in [0,255]:
    slm_image = SLMImage(data = slm_image_data,device=slm_dev)
    events.append(useq.MDAEvent(exposure=500,slm_image=slm_image))

thread = mmc.run_mda(events)
thread.join()

Other TODO:

[ ] Decide if/where to hardcode the SLM ON-pixel values.
[ ] I don't have a DMD with a builtin light source, so it's hard to test the exposure time and DMDs that require displaySLMImage. The polygon1000 sets pixels immediately when calling setSLMImage or setSLMPixelsTo. Will check if this is implemented in the genericSLMDevice.
[ ] Test with RGB, other DMDs

woot!  excited to take a look

There is an error in useq._base_model, _non_default_repr_args when comparing np.arrays. I just skipped this test for now, there is also a comment (consider using model_copy directly) - how should we proceed with this?

ah yes... the ever-annoying cannot test equality of numpy arrays :joy: ... let's fix that in useq schema.  there is some prior art for this, but for now just open an issue there

btw, quick tip:  rather than
thread = mmc.run_mda(events)
thread.join()

just do mmc.mda.run(events)   (the unthreaded version)

Tested with #5563fc0 , works!

For now I just disabled sequencing if two slm_images are non-equal (again issues with np.arrays). What's your opinion on proper SLM sequencing? I'm afraid this could lead to a rabbit hole, some DMDs working with DisplaySLMImage while some require CommandTrigger. Can we leave this for another time? Same with exposure, I don't have a good way of testing this.

Sample code for affine transformation calibration, works great! Much cleaner than what we had before.
p0, p1, p2 = ([456,380],[304,760],[608,760])

exposure = 500
events = []
for p in [p0,p1,p2]:
    img_p = np.zeros((slm_height,slm_width)).astype(np.uint8)
    img_p[p[1],p[0]] = 255
    event_p = MDAEvent(slm_image=SLMImage(data=img_p,device=slm_dev),exposure=exposure)
    events.append(event_p)

#disconnect previous callbacks
mmc.mda.events.frameReady.disconnect()

@mmc.mda.events.frameReady.connect
def new_frame(img: np.ndarray, event: MDAEvent):
    #find max value x/y
    max_x = np.argmax(img.max(axis=0))
    max_y = np.argmax(img.max(axis=1))
    plt.imshow(img,cmap='gray_r')
    plt.scatter(max_x,max_y,color='#00000000',marker='o',edgecolor='red')
    plt.show()

mmc.mda.run(events)

couple more things need to be fixed here:

change the dependencies in pyproject.toml to useq-schema >=0.6.2  to indicate that we have a new minimum requirement
please also change it in .pre-commit-config.yaml, so that the typing checks are updated as well.

just took another look and this is just about ready to go.  I added some tests in 6061bbd.  I did it by mocking the important core methods that actually need a device, just to assert that they've actually been called with the values we expect; that's all we need to test for anyway, since it's not our job to test the actual device adapters themselves

hey @hinderling, I had a chance to look at the names in some real device adapters and updated this slightly in https://github.com/pymmcore-plus/pymmcore-plus/pull/406/commits/50e368c849b9786f1475a06255de80e8820ce6a9 will merge once tests pass.  Thanks again for your contribution here!

repo: pymmcore-plus/pymmcore-plus


text:
Better (or more clearly documented) ways to have component states

I've had a great time playing around with pynecone, and it seems very promising.
One thing that trips me up though, i've been trying to create components in a project that are aware of their own states, somewhat similar to how you could simply use the "useState" hook in any isolated react component. So far, I haven't found a way to make this work properly. I find myself passing the app state as a parameter to whatever component function I'm calling, and managing absolutely everything in the central state.
Now, the documentation mentions substates, but there seem to be 2 issues with that;

a lack of clarity about real life applications, and nothing informative in the pynecone examples either
they seem to rely on subclassing the root state, apparently. If i want an isolated component to manage its own state (and define it in a separate module), that doesn't seem very viable. Is there a way for an imported component to manage its own isolated state? If so, it would be worth showing in the documentation, i think

Yes, I am looking for a way to divide my application to lots of components, that I can reuse it or even publish it to PyPI, a Pynecone component library, like the React things.

Thanks for raising this issue. Currently all state has to be managed through the app state (or substates). I agree the documentation is lacking here, we will improve it.
But you're right even with this method, you would have to hook up the state to each instance of the component you create. There's not a simple way for the component to manage it's own isolated state independent from the rest of the app. I think this seems super useful though so we will think about how we can add this.

I agree that it would be a great improvement. as @FHU-yezi mentioned, being able to publish sophisticated frontend components to PyPI would be a major benefit.
I also understand that there's no simple way to manage this given the current way pynecone implements state (that was my impression after all), but I'd suggest that it might be better to mimic react's handling of states a bit more closely. Certainly having something like a useState hook would be a major benefit.

authors: FHU-yezi mwesterhof picklelo

repo: pynecone-io/pynecone


text:
Support abi3 wheels

Hey there! I am working on a Maturin wheel built for abi3 and wasm32-emscripten-unknown, something which appears to work in Pyodide. I'm coming at this as a Rust developer, not really too knowledgable about Python, so most of what I know about abi3 is from the PyO3 documentation: https://pyo3.rs/v0.14.5/features.html#abi3
My understanding is that abi3 wheels are forwards compatible, meaning an abi3-py39 wheel will work with Python 3.9, 3.10, 3.11, 3.12, and so on. Empirically I can say our abi3-py39 wheel works just fine in Pyodide v0.26.2 (with some recent bugfixes I have recently done to perspective's build).
This PR adds support for abi3 wheels so that they can be added to pyodide-lock.json files. Will add a test to test_wheel.py
abi3 is also known as the stable ABI https://docs.python.org/3/c-api/stable.html#stable-abi

This doesn't appear to actually produce a pyodide-lock patched with the package though… investigating.

Certainly! Amended the commit to update the CHANGELOG

repo: pyodide/pyodide-lock


text:
Update URLs in documentation: http:// → https://

Update link to an old MSDN article and point to a newer article.

lgtm

authors: DimitriPapadopoulos lazka

repo: pypa/distutils


text:
Pipenv 2020 error's out not finding the requirement.txt

Issue description
On a empty folder installing -
pipenv install thoth-sourcemanagement
Error's out stating missing requirements.txt
However works as expected with pipenv 2018 version.
Steps to replicate
Environment -
Fedora 32
You can also replicate the error on a katacoda environment - https://www.katacoda.com/courses/ubuntu/playground1804
Try installing the following Pipfile with pipenv 2020
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
thoth-sourcemanagement = "*"

[requires]
python_version = "3.6"

The new version of pipenv upgrades the internal version of pip, which in turn relies on pep517 based building and, if this worked in the past, apparently it handles errors a bit less gracefully.
The new setup doesn't make any promises about putting the package directory on sys.path at build time, so trying to open requirements.txt (as opposed to os.path.join(os.path.dirname(os.path.abspath(__file__)), "requirements.txt") for example) is not guaranteed to work. This is a departure in terms of build process but most projects felt the pain of this some time ago.
That said, this isn't the best practice for telling setuptools about your requirements, and it's not really supported in any of the tooling anymore so I'm not sure we will want to make substantial changes to accommodate this workflow

I've run into this as well.  Is the fix basically to migrate to using pyproject.toml instead of or in addition to setup.py for declaring dependencies?

I've run into this as well. Is the fix basically to migrate to using pyproject.toml instead of or in addition to setup.py for declaring dependencies?

@taybin So pyproject.toml is what will be used to build the eggs for pypi?

@taybin So pyproject.toml is what will be used to build the eggs for pypi?

I don't know.  It's not like there's much documentation around on how to use this stuff.  Most documentation I see like (https://docs.python-guide.org/writing/structure/) doesn't mention pyproject.toml at all.  So I guess I need to use poetry for libraries, and pipenv for applications until I can switch them over to use poetry as well for consistency.  I'm just guessing though.

The fix is to not require implicit path lookups in your setup.py.
The real fix is to declare dependencies directly in your setup.py or setup.cfg, the issue here has absolutely nothing to do with pyproject.toml whatsoever. The problem is that the python packaging ecosystem has relied on quirks of the packaging tooling that are not promises, like attempting to read a naked, relative path (in this case requirements.txt) because of the assumption that, at build time, the actual build utility will execute in the package's directory.
The problem is that this is not the case anymore -- builds don't have to happen in your package directory, they can take place from any directory. So, if I am in /tmp/some-temp-build-dir and I run a build against /home/yourhomedir/yoursourcecode, I'm still in /tmp/some-temp-build-dir.
The original post references thoth-sourcemanagement so I'll use their setup.py as the example here: https://github.com/thoth-station/source-management/blob/e7d044ffa5725bfde8774fb2556f41f2cca36fd4/setup.py#L22-L26
This is a direct excerpt from the package setup.py, which refers to requirements.txt. This is a path that needs to exist relative to the current working directory of the build. That is just not going to be true, because the build is happening in a temporary directory.
You can fix this with an absolute path, but you will still break dependency resolvers. You SHOULD fix this by specifying requirements in setup.py or setup.cfg instead of dynamically calculating them at runtime. If you're already using setuptools there is absolutely no reason to change all of your infrastructure just to specify dependencies statically, since setuptools obviously has that capability already.
This is widely misunderstood which I get, but while it's a little bit nuanced, the fix isn't that complicated. More importantly, these issues wouldn't happen in the first place with static declarations which would be required in pyproject.toml anyway.

The real fix is to declare dependencies directly in your setup.py or setup.cfg, the issue here has absolutely nothing to do with pyproject.toml whatsoever.

Thank you for the explanation.  This makes sense.  I'm confused why it stopped working and why using requirement.txt files has been so common if it's inherently broken.  A mismatch of the workflows for applications vs libraries?

The real fix is to declare dependencies directly in your setup.py or setup.cfg, the issue here has absolutely nothing to do with pyproject.toml whatsoever.

Thank you for the explanation. This makes sense. I'm confused why it stopped working and why using requirement.txt files has been so common if it's inherently broken. A mismatch of the workflows for applications vs libraries?

The reason people make the association between pyproject.toml and the issue described here is because both changes occurred with the same release of pip and setuptools (and coincidentally marked the start of a series of delays in being able to put together a release of pipenv) -- pep517 was finally implemented as a default behavior in pip's build process in december 2018 I think, and by default if projects had a pyproject.toml present, pip would assume they meant to use the new build process.
Firstly, this didn't always work because the mere presence of the file doesn't mean that the requisite keys have been defined ([build-sustem.requires] and [build-system.build-backend]), and secondly because of the issue described above -- the previous build process never left the package directory to perform the build, so many times people would assume that they could use a relative path in setup.py or a relative import (e.g. importing their own library) and this would work just fine.
With the isolated pep517 build process, the package gets cloned to a temporary directory and the build launched inside a subprocess, but the temporary directory is not added to sys.path and the subprocess is not set to chdir to that directory before executing the build.
Basically, this broke a lot of assumptions, and the outcome of a number of discussions about the topic was that setuptools and pip wouldn't be accommodating this use case anymore as it shouldn't have worked to begin with.
I'm not a fan of breaking backward compatibility, but I have to balance that with supporting the modern build processes as well so we are basically stuck with this as the current situation. Statically defined requirements are easier to resolve anyway so it will likely be better in the long run even if it is more painful short term.

FWIW, I can't duplicate this anymore, even without updating my libraries to not use a requirements.txt file.  Still planning on doing that, but not having my builds break is also a relief. 👍

I did a bit of extra work to try parsing setup.py without breaking, and it seems to have paid off in your case. For this case and others like it, pipenv falls back to requirementslib which ultimately tries to build a wheel using the pep517 wheel builder and, if unspecified, the setuptools legacy build backend. If that is successful we will consume metadata from the built wheel -- this is obviously much less efficient than just parsing textual metadata that is declared in a manifest, but it's kind of neat that it works again ^^
I'll close this issue for now as I was able to confirm this myself directly with requirementslib:
>>> from requirementslib.models.requirements import Requirement
>>> r = Requirement.from_line("-e .")
>>> r.run_requires()
{'name': 'thoth-sourcemanagement', 'version': '0.2.9', 'base_dir': '/tmp/reqlib-srczugi9sxi/source-management', 'ireq': <InstallRequirement object: thoth-sourcemanagement==0.2.9@ file:///tmp/source-management from file:///tmp/source-management editable=True>, 'build_backend': 'setuptools.build_meta:__legacy__', 'build_requires': ('wheel', 'setuptools>=40.8'), 'requires': {'ogr': Requirement.parse('ogr'), 'requests': Requirement.parse('requests'), 'delegator.py': Requirement.parse('delegator.py')}, 'python_requires': '>=3.6', 'extra_kwargs': {'build_dir': '/tmp/reqlib-buildyrhlbxd0', 'src_dir': '/tmp/reqlib-srczugi9sxi', 'download_dir': '/home/techalchemy/.cache/pipenv/pkgs', 'wheel_download_dir': '/home/techalchemy/.cache/pipenv/wheels'}, 'setup_cfg': PosixPath('/tmp/reqlib-srczugi9sxi/source-management/setup.cfg'), 'setup_py': PosixPath('/tmp/reqlib-srczugi9sxi/source-management/setup.py'), 'pyproject': PosixPath('/tmp/reqlib-srczugi9sxi/source-management/pyproject.toml')}

authors: saisankargochhayat taybin techalchemy

repo: pypa/pipenv


text:
pipenv install  fails when PIP_EXTRA_INDEX_URL environment variable is set (from version 2021.11.23)

Issue description
Pipenv 2021.11.23 fails when I try to install a package using pipenv install , if the environment variable PIP_EXTRA_INDEX_URL is set. I get the same problem in both windows and the ubuntu-latest docker image.
The following works:

"pipenv install" with the package listed in Pipfile
unsetting PIP_EXTRA_INDEX_URL
rolliong back to 2021.11.15

Expected result
Package installed and added to Pipfile
Actual result
Package not installed and not added to Pipfile. Error message:
´´´
Traceback (most recent call last):
File "C:\Users\IND\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\IND\AppData\Local\Programs\Python\Python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\IND\tmpenv\Scripts\pipenv.exe_main.py", line 7, in
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\core.py", line 1128, in call
return self.main(*args, **kwargs)
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\core.py", line 1053, in main
rv = self.invoke(ctx)
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\vendor\click\core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\cli\command.py", line 194, in install
do_install(
File "c:\users\ind\tmpenv\lib\site-packages\pipenv\core.py", line 2114, in do_install
index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'
´´´
Steps to replicate
> set PIP_EXTRA_INDEX_URL=<extra_index_url>
> python -m venv tmpenv --upgrade-deps
> tmpenv\Scripts\activate
> pip install pipenv
> pipenv install pandas

$ pipenv --support
Pipenv version: '2021.11.23'
Pipenv location: 'c:\\users\\ind\\tmpenv\\lib\\site-packages\\pipenv'
Python location: 'c:\\users\\ind\\tmpenv\\scripts\\python.exe'
Python installations found:

3.9.4: C:\Users\IND\AppData\Local\Programs\Python\Python39\python.exe

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.9.4',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.19041',
 'python_full_version': '3.9.4',
 'python_version': '3.9',
 'sys_platform': 'win32'}

System environment variables:

PIP_EXTRA_INDEX_URL
PROMPT
VIRTUAL_ENV
_OLD_VIRTUAL_PATH
_OLD_VIRTUAL_PROMPT
__PSLOCKDOWNPOLICY
ALLUSERSPROFILE
APPDATA
CHOCOLATEYINSTALL
COMMONPROGRAMFILES
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
COMPUTERNAME
COMSPEC
DRIVERDATA
HOMEDRIVE
HOMEPATH
HOMESHARE
LOCALAPPDATA
LOGONSERVER
NUMBER_OF_PROCESSORS
ONEDRIVE
OS
PATH
PATHEXT
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMDATA
PROGRAMFILES
PROGRAMFILES(X86)
PROGRAMW6432
PSMODULEPATH
PUBLIC
PYENV
SNOW_INVENTORY_HOME
SYSTEMDRIVE
SYSTEMROOT
TEMP
TMP
UATDATA
USERDNSDOMAIN
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
WINDIR
WSLENV
WT_PROFILE_ID
WT_SESSION
ZES_ENABLE_SYSMAN
PIP_SHIMS_BASE_MODULE
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

PipenvÔÇôspecific environment variables:
DebugÔÇôspecific environment variables:

PATH: C:\Users\IND\tmpenv\Scripts;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\ProgramData\chocolatey\bin;C:\Users\IND\ExeFiles;;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\IND\.poetry\bin;C:\Users\IND\AppData\Local\Programs\Python\Python39\Scripts\;C:\Users\IND\AppData\Local\Programs\Python\Python39\;C:\Users\IND\AppData\Local\Programs\Python\Launcher\;C:\Users\IND\AppData\Local\Microsoft\WindowsApps;C:\Users\IND\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\IND\AppData\Local\Programs\Git\cmd;C:\Users\IND\AppData\Roaming\npm;C:\Users\IND\ExeFiles\azcopy_windows_amd64_10.10.0\azcopy_windows_amd64_10.10.0;C:\Users\IND\dapr;C:\Users\IND\.dapr\bin;C:\Users\IND\.dapr\bin;C:\Users\IND\Miniconda3;C:\Users\IND\Miniconda3\Library\mingw-w64\bin;C:\Users\IND\Miniconda3\Library\usr\bin;C:\Users\IND\Miniconda3\Library\bin;C:\Users\IND\Miniconda3\Scripts
VIRTUAL_ENV: C:\Users\IND\tmpenv

Contents of Pipfile ('C:\Users\IND\Pipfile'):
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.9"

@frostming, I still have the issue and I have installed the latest pipenv release.

Same error here.
Environment Mac, pipenv, version 2021.11.23.
After creating a new directory and trying to install any package:
pipenv install celery

Pipfile is created
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"

Packages are not installed.
Traceback:
p@Mac pipenv-test % pipenv install celery
Installing celery...
Adding celery to Pipfile's [packages]...
Traceback (most recent call last):
  File "/Users/pedro/.pyenv/versions/3.10.0/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/cli/command.py", line 194, in install
    do_install(
  File "/Users/pedro/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pipenv/core.py", line 2114, in do_install
    index, verify_ssl=index.startswith("https:")
AttributeError: 'list' object has no attribute 'startswith'

Same goes to other environments with other packages already installed.
@frostming @ngiind

authors: nbraun-wolf ngiind phbernardes