text:
Functions - syntax, scope and limitations

In issue #437 we ended up with quite a lot of discussion around the syntax for functions. For anyone browsing the issue log that might be quite difficult to find so I'm opening a separate issue for it here. The syntax also gets updated in PR #442 .
The current syntax is described in: https://github.com/prql/prql/blob/main/reference/src/functions.md
@aljazerzen also gave some descriptions on Discord which I've reproduced below:
Currently they are limited to "any expression", but there are plans for them to allow:

"late binding variables"

func net gross = gross * (1 - tax)

from finances
select [net gross_salaries, net gross_materials, net gross_something]

where tax is column from the finances, but is not resolved until after the function is materialised

currying

func add a b = a + b
func add_one = add 1

from employees
select [ age_next_year: add_one age ]

transform functions

func take_first col = (sort col | take 1)

from employees
take_first age

The function syntax seems like the least polished aspect of PRQL to me so far. While it has a lot of really elegant ideas from functional languages, my worry is that it's a bit too esoteric for beginners and might confuse them. In my experience, SQL is often something non-programmers pick up coming from Excel and it would be great if we could appeal to those people as well. So while we want to enable concise and logical expression of data pipelines for experienced users, we should also aim to make PRQL easy to pickup for beginners.

I like the maxim from the Zen of Python that "There should be one– and preferably only one –obvious way to do it.".
With currying and pipelining I fear that there may be multiple ways of expressing the same thing which may be confusing to beginners. I think the currying in particular is not very beginner friendly.
Also since we're going with the whole pipeline approach, shouldn't we see if we can double down on that and force that for all function applications?
So instead of
group [title, country_code] (
  aggregate [
    average salary,
  ]
)

have
group [title, country_code] (
  aggregate [
    salary | average,
  ]
)

?

I'm not a big fan of the | symbol for pipelines. While I know it's standard in the unix shell and Jinja templates for example, it's also often used for logical OR and unions in type definitions. Also in regexes | is a separator between alternatives so that's what my mind often jumps to.
Moreover there is no intrinsic indicator to a newcomer in which direction the flow of data is. I think something like >> would make that more obvious, e.g.
group [title, country_code] (
  aggregate [
    salary >> average,
  ]
)

transform functions

func take_first col = (sort col | take 1)

from employees
take_first age

I think this is a bad idea. What happens when I put this in a derive block?
func take_first col = (sort col | take 1)

from employees
derive [
  min_age = take_first age,
  seniority = age - min_age,
]

Will that work? I guess maybe it unrolls it into CTE or a subquery? At this stage I don't understand enough about the compiler and how it works. However I see PRQL mostly as a more convenient way to write SQL so I would prefer to keep it at the level where I can predict the SQL that will come out of it.
I would also prefer it if the top level commands in the left column of data pipelines remain keywords so that one can quickly scan a pipeline to see what transformations take place. take_first in this example breaks that.
Therefore I propose that functions remain expressions that can be used anywhere within an expression.
If there really is a need for naming common pipeline steps then maybe introduce another language construct for that such as macros or subpipelines or something like that.

Coming to the function syntax and examples from #437
func add x::number to::number:1 ::number = x + to

or
func (add x::number to::number=1)::number = x + to

I propose the following:
func add = (x:number, to:number=1) : number => x + to

or
func add = (x:number, to:number=1) => x + to : number

basically using the JS, Typescript () => ... syntax for functions. I think that very clearly and visually demonstrates that some values are used to produce a new value which should be understandable for newcomers.

Let me lay a mental framework of how I think about the functions.
Functions are actually the core of the language, but remain hidden because of subtle syntax of the function call. Pipeline in PRQL means a series of functions and relational means that they operate on tabular data/tables/dataframes.
So when working on compiler I realized that pipeline transforms actually are normal functions, one could define as (using pseudo-Rust):
type Frame = HashMap<String, Vec<T>>

fn from(name: Ident) -> Frame {
	Frame { }
}

fn derive((alias, expr), frame: Frame) -> Frame {

	frame.alias = match expr {
		Ident(i) => frame[i],
		ScalarFunction(f(column)) => {
			frame.column.map(f)
		}
		VectorFunction(f(column)) => {
			f(frame.column)
		}
	}

	frame
}

fn aggregate((alias, expr), frame: Frame) -> Frame {
	new_frame = {}
	new_frame.alias = match expr {
		AggregationFunction(f(column)) => {
			[ f(frame.column) ]
		}
	}

	new_frame
}

fn multiply(factor: float, value: float) -> float {
	factor * value
}

fn sum(column: Vec<float>) -> float {
	column.iter().sum()
}

fn lag(column: Vec<float>) -> Vec<float> {
	column.insert_front(null)
	column.remove_back()
}

Now, pipeline is just a different notation of turning foo(bar(x)) into x | bar | foo.
from employess
derive [
	net_salary: multiply 0.8 gross_salary
]
aggregate [
	total: sum net_salary
]

is equivalent to:
frame = from(employess)
frame = derive((net_salary, multiply(0.8)), frame)
frame = aggregate((total, sum()), frame)

Here derive x is a function call of function derive with one argument. Because derive expects two parameters, this call evaluates to a curry: a function that still expects one argument. This makes it possible to use this expression in a pipeline.

I very strongly agree with "There should be one– and preferably only one –obvious way to do it."
I see how it goes against currying and I do actually think that inline pipeline, currying and transform functions are all advanced features that should remain hidden for average user. To go even further I think that a linter should discourage use of single-function pipelines. This way, simple queries will almost never contain | symbol.
But for power-users, functional programming offers a well-defined and concise language. Such users are likely to be familiar with notion that | represents a pipe.

So applying the previous suggestion to the examples from PR #442 in https://github.com/prql/prql/blob/99898cac610eff157dd65bc43e283f331703e729/reference/src/functions.md:
func (celsius_of_fahrenheit temp) = (temp - 32) * 3

from cities
derive temp_c = (celsius_of_fahrenheit temp_f)

becomes
func celsius_from_fahrenheit = (temp) => (temp - 32) * 3

from cities
derive [
  temp_c1 = temp_f >> celsius_from_fahrenheit,
  temp_c2 = celsius_from_fahrenheit(temp_f),

For functions with multiple parameters, I don't think it's always practical to have the main argument as the last parameter so I propose to use _ as a placeholder to indicate which parameter should be substituted with the previous pipeline step when using the pipeline syntax.
So
func (interp lower=0 higher x) = (x - lower) / (higher - lower)

from students
derive [
  sat_proportion_1 = (interp 1600 sat_score),
  sat_proportion_2 = (interp lower=0 1600 sat_score),
]

becomes
func interp = (x, lower=0, higher) => (x - lower) / (higher - lower)

from students
derive [
  sat_proportion_1 = interp(sat_score, 0, 1600),
  sat_proportion_2 = sat_score >> interp(_, lower=0, 1600),
]

Regarding function transforms: actually my function definition example was not entirely correct:
- func take_first col = (sort col | take 1)
+ func take_first col frame:Frame = (frame | sort col | take 1)

Your example would now produce a type error, because a call of take_first would be missing the frame argument.
Note that this is not meant for average user. It's the hidden tool that advanced used can leverage. It is also very logical production of how PRQL works.
It may seem that this makes PRQL unpredictable in its SQL output, but in essence, our functions are just macros - substitution of ident with an expression. It's just that in this case, use have a macro that substitutes frame transforms.

I propose the following:
func add = (x:number, to:number=1) : number => x + to

Of course! => is very obvious. I'm quite surprised we did not think of it before :dagger:
What do you think about ->? It's used in Java & Rust.
Is there a reason for = after function name? I would prefer for function definition to be as close to function call, because that gives feeling of language consistency.
func add x:number to:number=1 => x + to

https://github.com/prql/prql/issues/444#issuecomment-1115992366
Ok, having the frame parameter does make it clearer and alleviates my concern around that I think.

https://github.com/prql/prql/issues/444#issuecomment-1116002602
No particularly strong feelings on => vs ->. I don't know Rust and JS has => so that's what I went with. Happy with either.
func add x:number to:number=1 => x + to

I find the lack of parentheses and commas confusing to be honest. Like why are there no commas separating parameters because we have those everywhere else like in [ , ] expressions etc...?
To me having the = makes sense because if you think of functions as first-class objects then it's just another assignment to a name which just happens to have a function on the right hand side. I was initially going to propose doing away with the func and just have something like
add = (x:number, to:number=1) => x + to

but I saw a comment from you somewhere that it's nice to have the func in the left column so one can just use that to scan for function definitions quickly so I kept it.
So I find the = makes sense because add and (x:number, to:number=1) => x + to are equal in that you should be able to use each interchangeably, so in my syntax the following should give the same results:
func add = (x, to=1) => x + to

from transactions
derive [
  total_1 = add(cost, markup),
  total_2 = ((x, to=1) => x + to)(cost, markup),
]

Then let's rather use ->, because ⇒ means implication in math.

Oh, that does make sense: everything has the syntax of lambda function.
@max-sixty wanted to have a language accessible to non-programmers, so he made function calls not use any punctuation. And that decision goes well with functional programming aspect of PRQL, because pipeline transforms have the same syntax.
With that in mind, what about:
func add = x to=1 -> x + to

from transactions
derive [
  total_1 = add cost markup,
  total_2 = cost markup | x to=1 -> x + to,
]

I've changed total_2 to inline pipeline, because function calls can only be parsed if they have ident as first term, preventing having a lambda function instead of function name.
But now, cost markup would be parsed as a function call... Ah. This no-punctuation rule is hard to keep consistent.

I sign off for a few hours and the PRQL has completely changed! Jokes aside, this is a good discussion, thank you.

func add = x to=1 -> x + to

This is interesting

The punctuation between add and the arguments is reasonable and consistent with the rest of the language, even if it's not consistent with Ocaml, the original inspiration.

We do end up quite close to just doing a lambda:
derive add = ([x, to=1] -> x + to)

...i.e. transform a list of args to an expression, and there's no need for func.
I think there are reasons to not pursue this — lambda functions are much less helpful in PRQL than most languages, because all operations happen on arrays / columns already. We don't need to map (x, y) => (x + 2y) over a list, because PRQL & SQL already just accept time + 2*overtime, where time and overtime are arrays. So I think it's quite unlikely people will be using lambda functions inline.

I'm not a great fan of using the = for both the add = and the to= — it's difficult to visualize where the break is. I think we should use it once there. And maybe use : for the other, like:
func add = x to:1 -> x + to

This would be consistent with having aliases & assigns use different punctuation from named args

Maaaybe we need parentheses around x to=1 -> x + to to ensure the -> binds tighter than the = in func add =?

cost markup | x to=1 -> x + to,

Pipelines can never pass more than value (at least how I'm thinking about them). They can pass tuples, but I don't think we need that language feature.

Re => vs ->: no strong view — IIRC JS does => and Java does ->. Rust only uses -> in the type definition.
even though it's not how ocaml does it, which was the inspiration for this.

While roundly condemned last time I brought it up, indentation would work:
func add x y
  x + y

func lag_day x
  group sec_id
    sort date
    window
      lag 1 x

(ok ok, I'll try not to bring it up again)

I very strongly agree with "There should be one– and preferably only one –obvious way to do it."
I see how it goes against currying and I do actually think that inline pipeline, currying and transform functions are all advanced features that should remain hidden for average user. To go even further I think that a linter should discourage use of single-function pipelines. This way, simple queries will almost never contain | symbol.
But for power-users, functional programming offers a well-defined and concise language. Such users are likely to be familiar with notion that | represents a pipe.

100% agree with @aljazerzen here + his previous comment

Trying to add things up into some options across this & https://github.com/prql/prql/issues/444 (though good idea to keep them as separate issues originally), some broad clusters of options:

Status quo — don't change anything, use :: for annotationsfunc::number interp lower:0 higher x::number = (x - lower) / (higher - lower)

from s:students
derive [
  sat_proportion_1: (interp 1600 sat_score),
  sat_proportion_2: (interp lower:0 1600 sat_score),
]

Use : for Named args, = for Aliases & Assigns, :: for annotations:
Note that we can dispense with the parentheses in the derive for this & the next couple.func::number interp = lower:0 higher x::number -> (x - lower) / (higher - lower)

from s=students
derive [
  sat_proportion_1 = interp 1600 sat_score,
  sat_proportion_2 = interp lower:0 1600 sat_score,
]

Use = for Named args, = for Aliases & Assigns, :: for annotations:func::number interp = lower:0 higher x::number -> (x - lower) / (higher - lower)

from s=students
derive [
  sat_proportion_1 = interp 1600 sat_score,
  sat_proportion_2 = interp lower:0 1600 sat_score,
]

I'm going to put one more out there — use <> for types. They are used for generic in Rust, but impl Point<i32> is correct and similar to what we're doing, and lets us use : more clearly for other things.
Use : for Named args, = for Aliases & Assigns, <> for annotations:func<frame> interp = lower:0 higher x<number> -> (x - lower) / (higher - lower)

from s=students
derive [
  sat_proportion_1 = interp 1600 sat_score,
  sat_proportion_2 = interp lower:0 1600 sat_score,
]

(We don't need to define => vs -> / argument-as-lists, etc here — this is for the issues where we face tradeoffs between different threads)

Just to follow-up on this:

@max-sixty wanted to have a language accessible to non-programmers, so he made function calls not use any punctuation. And that decision goes well with functional programming aspect of PRQL, because pipeline transforms have the same syntax.

Right — general purpose languages have lots of punctuation, because they need lots of widgets — lists & tuples & sets & dicts. The two types of languages without lots of punctuation:

Functional languages like Ocaml, where the language is very small and most things are implementations
DSLs like SQL, which have limited semantics

So I envisage PRQL as a DSL for analytical queries — centering around the notion of a pipeline, removing SQL's warts, adding functions & variables, and keeping features like bare words (select foo rather than select "foo"). And so appealing both to 1) advanced users who miss basic features, tire of the warts, and have seen languages which don't need to be complicated, and 2) beginners who don't want to learn lots of punctuation[^1].
One piece of feedback I'm open to is: not having as much punctuation makes it more complicated to read or learn. But it's important not to have that as a first reaction only because it's different from Python & Java. So I would want to hear what people like @aljazerzen think, who has spent a lot of time writing the lang.
[^1]: e.g. for a toy example: why do python functions need colons at the end?
Check out Ocaml functions for more (and labeled arguments too!)

2 gets my vote.

I agree that the most readable option is

Use : for Named args, = for Aliases & Assigns, :: for annotation

so Named Args get a different symbol from Aliases and Assigns.
Just one idea (since I'm not shy about throwing wacky ones out there), what about ~ for Named Args, that way we could still keep : for annotations?
So instead of
func::number interp = lower:0 higher x::number -> (x - lower) / (higher - lower)

have
func:number interp = lower~0 higher x:number -> (x - lower) / (higher - lower)

I'm not aware of : being used for named args in (m)any languages so we wouldn't be going against an established convention and this would allow us to keep with the convention of having : for type annotations.
One use of ~ is for "distributed as" in statistics (e.g. X~N(0,1)) so here we would just have the parameter distributed according to a singleton distribution of the default value. Anyway, it's a bit of a stretch I know but it would be nice to keep the : annotations as I find the :: introduces a bit of line noise.

https://github.com/prql/prql/issues/444#issuecomment-1116910865
Thanks. I had a look at OCaml functions because I don't know OCaml at all.
In 2.4.2 he writes

We now have two ways we could write an increment function:
let inc x = x + 1
let inc = fun x -> x + 1

It seems that the function syntax we have arrived at is very close to the second form (apart from moving the fun(c) token to the front and not requiring this be in a let/derive block).
I feel quite strongly that the first form is not very intuitive to beginners and should not be supported in PRQL. There is nothing inherent in the notation to suggest that the first name is the name of the function and the remaining names are the names of parameters.
I agree that lambdas/anonymous functions are probably not going to play a big role in PRQL but I think the syntax quite intuitively communicates that you are dealing with a mapping.
One case where I think lambdas could come up is when trying to apply transformations, so if we have a rule that function transformation always apply to the last or first parameter then you might want to define a lambda just ahead of that to move the parameter you are interested in, into the required position (if you can't specify it otherwise). Not sure if that scenario would really ever come up but it's the main case I could think of.
While on the OCaml syntax, in section 2.4.6 of that link, he says

Here are two equivalent ways of squaring 6:
square (inc 5);;
5 |> inc |> square;;

So OCaml uses |> for the pipeline operator. As mentioned elsewhere, I'm not a fan of the | for the pipeline operator and would prefer something like |> or >> which gives a visual indication of the data flow similar to the %>% in dplyr and R.

(since I'm not shy about throwing wacky ones out there)
Yeah keep 'em coming!

We did talk about ~ , using it for derives and aliases because of it's use to mean approximate, but in the end the : won out.

So OCaml uses |> for the pipeline operator. As mentioned elsewhere, I'm not a fan of the | for the pipeline operator and would prefer something like |> or >>

I'm going to go out on a limb here and say I dont like this method of function calling.
I love having the transformations piped into one another, and it makes logical sense to support them for functions, but I think it's confusing when reading the query.  I can envision half of the people will use this everywhere, and the other half won't use it all, splitting PRQL users into two camps.

Again, just documenting current state of the language.
Currently, operator precedence is as follows:

unary operators
multiplication / division
addition / subtraction
comparisons
logical operators (and, or)
function calls

Function call is an ident, followed by any number of expressions.

Example:
foo -hello + 2 bar find 5 + age / 2 < 10 and is_single

is a function call of foo, with args:

(-hello) + 2
bar
find
( (5 + (age / 2)) < 10 ) and is_single

There is also a rule that a function argument cannot be direct function call:

foo bar sum salary 5
...is same as
foo (bar) (sum) (salary) (5)
...but different from
foo (bar) (sum salary 5)

So your example can be written as:
derive difference = abs (coalesce a.amount 0) - (coalesce b.amount 0)

but without all parenthesis it would mean:
derive difference = abs (coalesce) (a.amount) (0 - coalesce) (b.amount) (0)

This precedence has been adapted from MS SQL and Postgres and is such so that abs length - 5 is interpreted as abs (length - 5).

To be honest I don't like mixing frame columns and function definitions. From:
select [
    null2zero = x -> coalesce x 0,
    absdiff1 = a b -> abs (null2zero a - null2zero b),
    absdiff2 = a b -> null2zero a - null2zero b | abs,
    difference = absdiff2 a.amount b.amount,
]

... one may figure that there are 4 columns in the frame, while in truth there is only one. This may not be obvious, because of currying, which is only a logical consequence in a function language:
select [
  add = a b -> a + b
  add_one = add 1
  age_next_year = add_one age
]

Instead, I am for less-lambda-oriented function definition:
func absdiff1 a b:1 -> abs (null2zero a - null2zero b)

Because we agreed that named arguments use :, we don't need any parenthesis and can even drop ->:
func absdiff1 a b:1 = abs (null2zero a - null2zero b)

I'm fine with either of these. I'm not sure where to put types of the return value though. After the function name?

Agree with @aljazerzen above!

To ensure we balance being open to changes while focusing the discussion on the areas most likely to lead to productive changes — to change the | syntax to >> or |>, we would need cases where it's better, and given it's a simple swap — and for an additional character of such a fundamental language — I think that would be a challenge.
I would weigh cases far above our intuition of what looks better in the abstract — cases give us a shared legible understanding which we can build on. And I'm including my own intuition here when I say it should come below cases.

Working on compiler: it's easier to have -> instead of = in current parser implementation.

Regarding pipes: because this is quite fundamental part of the language I would avoid |> just because is harder to type than alternatives. I'm fine with both | and >>.

Regarding the pipes: I also don't like |> and just suggested it because of the OCaml references.
@max-sixty I agree with your point about weighing cases higher than abstract reasoning. From my examples above and the return index example in the Readme.md, I have to say the | looks fine and I have come around to that side.
Thanks for the function syntax clarifications @aljazerzen . I had read through the syntax and functions pages on https://lang.prql.builders/ but didn't see anything about this before.
Is the low precedence for function calls the common convention for functional languages? I must say I find those examples you posted impossible to read.
Comparing
foo -hello + 2 bar find 5 + age / 2 < 10 and is_single

with the usual parentheses based function invocation
foo(-hello + 2, bar, find, 5 + age / 2 < 10 and is_single)

, at least in the latter a novice can find their way around with high school math.
Not saying that anyone would write code like that but given that someone might be able to seems like a bit of a concern to me because advanced users can write queries that beginners won't be able to understand. SQL kinda forces you to ride your bike with the training wheels on, which while frustrating to us, does keep it accessible. That said the examples in the Readme file read great. Perhaps it's another case of worrying too much about things in the abstract without concrete examples to back it up.
I just think that keeping the language accessible to novice programmers would be quite key. I deal with quite a lot of Data Analysts for example who only really have Excel experience and for SQL is the next step up. I don't see them ever groking those function application rules but maybe I'm wrong.
Anyway, I've been throwing out a lot of comments and suggestions and don't want to take up too much of your bandwidth. IMHO keeping the language simple would probably help with wider adoption but that's just a hunch. I want to get myself set up to start using PRQL for some of my data work to get a feel for it in practice and then comment from that perspective rather than arguing in the abstract.

That is true - the example is not readable at all. But thats an extreme example - most functions have only one or two params that are pretty obvious.
Regarding the clarifications, they are fresh from the compiler :D They should eventually make it to the reference book, but I really don't like writing documentation. Can someone help here?
I also agree that we should postpone this discussion after we generated more examples and used PRQL in practice.
Yesterday I was working on the compiler I was changing assigns to =. During that process I found out that having -> for functions makes parsing quite easier, so I've also changed that. This means that now compiler uses my the last proposed syntax. But that is not a final decision, this issue will remain open.

Regarding the clarifications, they are fresh from the compiler :D They should eventually make it to the reference book, but I really don't like writing documentation. Can someone help here?

Yeah, docs aren't my favorite but happy to take this one given our relative contributions over the past couple of weeks!

Thanks to everyone for the discussion and to @aljazerzen for the implementation. I'm v happy about the place we're in. I'll close this. I'll do some work on the associated docs.

authors: aljazerzen charlie-sanders max-sixty snth

repo: prql/prql


text:
fix(optimize_build): remove travis, add ifs to action

Precommit-Verified: 837ace3492538fd683db3d6806ae4319defa9a7062797a06f31b006426862a15

Codecov Report

Merging #254 (62467f6) into main (802240c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #254   +/-   ##
=======================================
  Coverage   80.29%   80.29%
=======================================
  Files           4        4
  Lines         873      873
=======================================
  Hits          701      701
  Misses        172      172

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 802240c...62467f6. Read the comment docs.

source: gharchive/pull-request

authors: codecov-io pryorda

license_type: permissive

license_source: github-api


text:
Build failed with M1 Mac

Build fails on an M1 MacBook. Looks like ld in the binutils package doesn't like aarch64. I guess a linux arm64 would have the same issue (raspberry pi 4 for example).
bash-3.2$ ./build-all.sh
Installing latest environment status
HEAD is now at 4459dab Merge pull request #102 from uyjulian/buildsystem_improve
Already on 'master'
Your branch is up to date with 'origin/master'.
Installing latest environment status
HEAD is now at 484a8dd Merge pull request #12 from uyjulian/buildsystem_improve
Already on 'main'
Your branch is up to date with 'origin/main'.
HEAD is now at 9cca5c1 Added abi options to dvp-as for toolchain compatibility
Already on 'dvp-v2.14'
Your branch is up to date with 'origin/dvp-v2.14'.
creating cache ./config.cache
*** This configuration is not supported in the following subdirectories:
     ld
    (Any other directories should still work fine.)
rm: conftest.dSYM: is a directory
../configure: line 2713: test: too many arguments
../configure: line 2785: test: too many arguments
../configure: line 2857: test: too many arguments
../configure: line 3001: test: too many arguments
../configure: line 3073: test: too many arguments
../configure: line 3145: test: too many arguments
../configure: line 3217: test: too many arguments
../configure: line 3289: test: too many arguments
../configure: line 3370: test: too many arguments
../configure: line 3442: test: too many arguments
../configure: line 3514: test: too many arguments
../configure: line 3586: test: too many arguments
../configure: line 3658: test: too many arguments
../configure: line 3730: test: too many arguments
../configure: line 3802: test: too many arguments
updating cache ./config.cache
creating ./config.status
creating Makefile
sed: file conftest.s1 line 26: unterminated `s' command
make: *** No targets.  Stop.
../scripts/001-binutils.sh: Failed.
../scripts/001-dvp.sh: Failed.
../scripts/001-toolchain.sh: Failed.

also with noting that gnu sed is being used:
ian@Ians-MBP ps2dev % sed --version
sed (GNU sed) 4.9

TL;DR: ARM64 target for Darwin (macOS) is not really supported by binutils and GCC used in ps2sdk toolchains.

Looks like the main issue preventing binutils-dvp from being built is that configure scripts used in binutils have no idea what arm64 is and detect host as -apple-darwin23.1.0, breaking everything.
You can "fix" this by manually overriding host detection, adding this after line 37 of ps2toolchain-dvp/scripts/001-binutils.sh:
if [ $OSVER == Darwin ] && [ $(uname -p) == arm ]; then
   TARG_XTRA_OPTS="--host=arm-apple-darwin"
fi

ld will still fail to build, but AFAIK ld from GNU binutils has never supported macOS in the first place. That doesn't prevent the binaries from working, anyway.

To build GCC for IOP toolchain, I had to add ./contrib/download_prerequisites to ps2toolchain-iop/scripts/002-gcc-stage1.sh before line 58, otherwise it always complains about not having GMP, MPFR and MPC (despite being already installed via Homebrew):
configure: error: Building GCC requires GMP 4.2+, MPFR 2.3.1+ and MPC 0.8.0+.
You'll also have to disable precompiled headers by commenting out a few lines, see this:
https://github.com/riscv-software-src/homebrew-riscv/issues/47#issuecomment-772306695
Don't forget to remove git fetch/reset/checkout from the script (lines 27-30) to keep your changes from being reset every time you launch it.

To build binutils for EE toolchain, I had to add --with-libgmp-prefix=/opt/homebrew/Cellar/gmp/6.3.0 to configure at line 60 of ps2toolchain-ee/scripts/001-binutils.sh, otherwise it couldn't find GMP to configure and build gdb. It still complains about other libraries, but at least they are not required.
Workarounds for building IOP GCC are also required for building stage1 and stage2 of EE GCC.

After all of that I was able to build PS2SDK and make a working build of OPL from source.
To honest, I'm not sure why GCC and binutils can't find any of the libraries (which causes half of the problems here). pkg-config is working properly on my system and resolves all names except mpc for some reason.

@pcm720 You are truly a wonderful human. I don't know you, but you made my life a little better. Thank you. I hope you understand the extent to which I appreciate what you've done here.

Could be nice to update scripts for allowing M1 machines…

This was already solved, please re-open if it is not working for you

source: gharchive/issue

authors: bigianb fjtrujy pcm720 quisys

license: MIT


text:
Template Eloquent ORM methods where possible

cc @weirdan in case this interest you at all

Maybe later. But should I start using Laravel I'll certainly look into this.

https://github.com/psalm/psalm-plugin-laravel/pull/52

Going to consider this closed!

authors: mr-feek muglug weirdan

repo: psalm/psalm-plugin-laravel


text:
How to make a bluetooth micphone?

Hi,
I'd like to make a bluetooth microphone which can be used with my laptop.
But I find all the examples only works with a bluetooth speaker.
I want the ESP32 can be found by laptop and pair. Is there any demos for this?
Thanks，
Ziv

Here is the Arduino Sketch example: https://github.com/pschatzmann/arduino-audio-tools/tree/main/examples/examples-basic-api/base-i2s-a2dp
Please note that you need to google how to turn your laptop into an A2DP Sink yourself since this is dependent on the OS.

Thanks a lot.
I have tested streams-i2s-webserver_wav, it works well.
And Is it possible for us to make a device that can act as my bluetooth headset?
I can search and pair my bluetooth headset directly with my laptop.
Ziv

That would be any of the Sink examples in this project.
Just read the Readme!
You can try to use the internal DAC - though the quality will not be great and I recommend to use an external DAC. There is plenty of information in the Wiki

Thanks, I would stuby WIKI.


text:
User comments in the code generated

We want to be able to add comments defined in our source files into our output

Closing as duplicate


text:
Add specification for hostname

For Beaker it is possible to choose machine according to hostname.
This is used widely and should be supported.
We should be able to choose hostname via regular expression matching.
Resolves #942
Signed-off-by: Miroslav Vadkerti mvadkert@redhat.com

@happz ptal

@psss @lukaszachy I am a bit confused from regex matching operator:

we have it now =~ but in context we use ~= which might be a bit confusing maybe?

seems it is common to use does not match regex, we need to decide how that should be written =!~ or ! =~ or something else?

I vote for ! =~.

+1 on hostname.
My two cents: =~ + !~ would look best to me, but if you already have an existing implementation that's using ~=, I'd suggest aligning HW requirements with that implementation (on the assumption regex operators in HW requirements are rarely used so far), and then I'd vote for ~= and ~! (both replacing = with ! while keeping ~ to make clear it's regex operator). =!~ uses both = and ! (yeah, I know, != does the same...) and it's one extra character, and ! =~ contains a space (and two extra characters...). Bash uses special ! before the test, Perl has !~, just like Prometheus. ~ seems to be commonly used. I'm biased, sue me ;)
There's also negative lookahead, hostname: =~ (?!kvm-01.*) ;)

I'd prefer ~ and !~ as a new regexp operator because it will stand out more from the existing:
binary_operator ::= '==' | '!=' | '<' | '<=' | '>' | '>=' |
    '~=' | '~!=' | '~<' | '~<=' | '~>' | '~>='

IMO if we have both "~=" and "=~" it will be very hard to stop the typo (or remember which is which).

I'd prefer ~ and !~ as a new regexp operator because it will stand out more from the existing:
+1 from me!

authors: happz jirka-h lukaszachy thrix

repo: psss/tmt


text:
event for sorting

how to get the sorted column to use it to call ajax.

Pass a function to the data attribute. That function will receive 2 arguments, an object containing all the data you will need to process the table rows, and a callback to send your data to the datatable. Here is an example:
<datatable :columns="columns" :data="getRows"></datatable>

new Vue({
    data: {
        //...
        getRows: function(info, callback){
            var filter = info.filter;
            var sort_by = info.sort_by;
            var sort_dir = info.sort_dir;
            var page_size = info.page_length;
            var page_number = info.page_number;

            // your logic here

            callback(the_rows_you_want_to_display, the_total_number_of_filtered_rows);
        }
        //...
    }
});

I hope that makes sense.

Closed because upvoted by the author without response.

Hi, i use this code but obtain this
[Vue warn]: Error in nextTick: "TypeError: callback is not a function"


text:
Local fixes to the rspec name disambiguation test

Updates rspec to use the documentation style and corrects the test for name disambiguation for LDAP changes to given names–lowercase, doesn't include, middle name, etc.

Coverage remained the same at 57.992% when pulling 73b1368cd14a6d6c7878cd006a239ce926050622 on rspec-fixes into 71089113da99b93a4f8c280b838b84a4966421da on develop.

authors: coveralls mtribone

repo: psu-stewardship/scholarsphere


text:
Settings button active by default (Chrome OSX)

Affects v0.2.0 on Chrome OSX.

Looks like a Chrome issue. http://stackoverflow.com/questions/16701082/chrome-extension-first-link-is-auto-focused-in-popup

repo: pswai/forex-time

license: mit

license_source: bigquery


text:
Docker improvements

@pslobo What do you think?

Looks good to me. I'll add my changes once you've merged those too.

@pslobo actually i need to test this more so instead of you creating a pull request that merges to master, you can make it merge here. we only have to make sure we merge this PR first.


text:
TFTP class based rewrite

TFTPD was starting to look a bit brittle, so as part of the cleanup effort I rewrote it using classes rather than dictionaries.
This solves all the bugs in #59 including the one I added today.
It handles both cases of options vs. no options as recently patched. I've also tested it with up to 500 concurrent clients and it handles it fine.

LGTM :+1: Since this contains a good amount of fixes I'm going to merge this and rebase #59 off of development after this merge.

Although not perfect, I don't think that either of the other two modules warrant this extreme of a rewrite at this point, so I'll leave them as is and just add line items to #59

authors: mmattioli psychomario

repo: psychomario/PyPXE


text:
demo_hello_world.py fails due to circular import

Hi,
I am encountering a problem to run python demo_hello_world.py.
it likely fails due to a circular import problem.
➜  palette (.venv) python demo_hello_world.py
Traceback (most recent call last):
  File "/home/max/Development/Python/palette/imgui_bundle.py", line 1, in <module>
    from imgui_bundle import immapp
  File "/home/max/Development/Python/palette/imgui_bundle.py", line 1, in <module>
    from imgui_bundle import immapp
ImportError: cannot import name 'immapp' from partially initialized module 'imgui_bundle' (most likely due to a circular import)

I follow the installation instruction in virtual environment.
Python version is 3.12.3
Is this an usual problem ?

Hellon
No, this is not a usual problem. Did you install from source or did you install using pipy?
I just did an installation from both source or using pipy, and both worked.
Could you try to install in a fresh environment?
If it still fails, please give me a full summary of all the commands you run, + the OS.

Thanks for your feedback. It was installed using pipy.
I run demo again using another fresh environment and it works good.
Actually,  I am stupid. I name the source file imgui_bundle.py . After rename it, the issue gone.

authors: maxmbed pthom


text:
Help Required: API not documented

I use these few APIs for world political data analysis provided by Lowy Institute, Australia

https://power.lowyinstitute.org/data/2020.json
https://power.lowyinstitute.org/countries.json
https://power.lowyinstitute.org/world.json
https://power.lowyinstitute.org/network-power.json

Their reports and data are also used IMF and other UN foundations.
But they don't have proper documentation so i can't add one direct link in repo. Please suggest me what should i do?

(Expecting replies since 1931.)

@0x0is1 As stated in the contribution guide, it's necessary for the API to have official documentation in order to be added to the list. And as this API doesn't meet this requirement, unfortunately, it's not possible to add it.

That's quite sad😓 but okay..!

That's quite sad😓 but okay..!

@0x0is1 I thought of something. 🤔
You can create a repository here on github and create unofficial documentation for this API. So I can accept for listing.
You should document and explain what each of these endpoints is for so others can easily use them.
What do you think?

Some examples you can follow:

https://github.com/btcid/indodax-official-api-docs/blob/master/Public-RestAPI.md
https://github.com/thundercomb/poetrydb#readme
https://github.com/CrossRef/rest-api-doc

Thanks for the amazing suggestion. Working on it. :)

Okay, I'll be waiting. Request my review when you're done, please. ;)

Resolved in #2982,

repo: public-apis/public-apis


text:
Mobile hamburger menu bug

The hamburger menu in the dashboard header has a visualisation problem on mobile devices ( tested on Iphone X, not sure about other devices). This is consistent across different pages ( content pages,  maps, etc) but not on the homepage.

This is already addressed in a different development branch.

authors: martafioni xenatisch


text:
[react-ui] Choix du nombre de décimale pour l'affichage des arrondis par règle

Le component react affiche les valeurs arrondies au centième (ce qui est suffisant dans la majorité des cas)

Mais dans certains cas, comme par exemple pour la règle salarié . cotisations . APEC . salarié, il est important d'afficher l'arrondi au millième, sinon le résultat affiché parait erroné.
Est-il possible de forcer l'affichage avec un arrondi au millième, sinon, quelles modifications sont envisageables ?
Merci d'avance.

Est-il possible de forcer l'affichage avec un arrondi au millième, sinon, quelles modifications sont envisageables ?

Effectivement, c'est impossible dans l'immédiat, mais c'est une fonctionnalité qui aurait tout à fait sa place.
Il est possible de « ruser » en n'apposant pas d'unité mais ce n'est pas satisfaisant :
foobar:
  valeur: 0.0024232
  unité: "%"

On pourrait imaginer plusieurs pistes pour répondre au problème :

Ajouter des méta « affichage » dans la règle publicodes qui contiennent des informations sur la précision à afficher (entre autres)
Utiliser la valeur du mécanisme arrondi si il existe
Avoir un mécanisme de mise en forme directement dans publicodes #403

Une approche discutée jadis mais pas implémenté, c'était de gérer cela automatiquement via le décompte et l'inférence des chiffres significatifs (= du nombre de décimales). Ceci dit pour les valeurs littérales comme dans l'exemple initial le fait de garder la précision du nombre saisi était géré avant mais ça s'est sans doute perdu dans une modification du code.

Je relance le sujet car j'ai également rencontré le problème. Dans les règles de la réduction générale des cotisations patronales, nous travaillons avec des coefficients à 4 décimales.
Pour les calculs, l'arrondi semble respecté car le résultat final est correct.
En revanche la documentation n'affiche le coefficient qu'avec 2 décimales (0,2597 est affiché 0,26 dans la doc)
J'ai contourné le problème en affichant le coefficient sous forme de pourcentage (25,97% dans l'exemple) mais je perds en cohérence avec les autres sources qui utilisent cette valeur et qui affichent 0,2597.
Parmi les solutions proposées, l'usage de la valeur du mécanisme arrondi me semble être la plus pertinente. Imho cela est à la fois le plus simple (pas de champ supplémentaire à renseigner) et le plus transparent (la documentation reprend exactement les valeurs utilisées pour les calculs).

@liliced, j'ai oublié de le noter, mais cette issue a normalement été réglée dans la dernière version de publicodes.
#548

authors: eweap johangirod liliced mquandalle

repo: publicodes/publicodes


text:
Fix inability to search and filter packages at the same time

Is this feature Currently Implemented in any way with the upstream API Server

[ ] Yes
[X] No

Is your feature request related to a problem? Please describe.
There's one endpoint — /package/search for searching against the name of a package. There's another endpoint — /packages — for filtering on packages with fields like service. This means there appears to be no way to search and filter at the same time.
Describe the solution you'd like
I think there should be one endpoint — probably just /packages. If q is present, a search is performed. If other fields are present, other clauses are added to the SQL query.

You are totally right here that it seems weird for them to be broken up.
And this issue gets even worse when you consider that the search endpoint does already support the order parameter.
It'll likely take some time to get the SQL statement right to accommodate all of these different features, but this is a great idea.
I'll add, what we would likely want to do is perfect one of the database functions to be able to handle each use case, then have these two endpoints rely on the same function, maybe even deprecating the special search endpoint, as it's redundant at that point.

authors: confused-Techie savetheclocktower

repo: pulsar-edit/package-backend


text:
Add vnet support

This is draft. I'm waiting for release in TF provider in 2 weeks.
PRs that I'm waiting for:
https://github.com/cloudflare/terraform-provider-cloudflare/pull/1672
https://github.com/cloudflare/terraform-provider-cloudflare/pull/1668
After release go.sum & go.mod will change but everything else will remain the same.
This PR adds support for virtual networks in cloudflare.
Related issues:
https://github.com/cloudflare/terraform-provider-cloudflare/issues/1605
https://github.com/cloudflare/terraform-provider-cloudflare/issues/1602

These resources will be picked up automatically when they are released in the upstream Terraform provider, within a few weeks generally.  If you see that the upstream release has happened, feel free to open an issue here and we can try to speed up the new pulumi-cloudflare version release.

authors: jkisk vavsab

repo: pulumi/pulumi-cloudflare


text:
Remove unnecessary checks for API handles

An extra check is not needed for a zero API handle in functions like the following.

Image
Sampler::validate

Resolved ( commit e91aa78 ).
I decided to remove handle check from Image class and few other places.
I left Sampler class untouched for now.
Checking for VH_NULL_HANDLE in validate() method is currently consistent
for many classes that manage lifetime of Vulkan objects ( like Sampler,
FrameBuffer, ImageView and many others ).
These classes will be reviewed by me in near future and suitable
changes will be made so that the outlook of these classes is consistent.
Thank you for reminding me about these unnecessary checks :)

Thanks for your small source code improvement.
Will any update candidates adjusted a bit later?

Yes, I am planning code review for different aspects of Pumex library in the near future,
because they need improvement in my opinion.
These aspects include:

lifetime of Vulkan objects
validation of scenegraphs
rebuilding of command buffers
secondary command buffers
Vulkan synchronization
render workflow compilation

These are my plans to improve code quality.
Plans for new features are listed at the end of README.md file.
Cheers :)

authors: elfring pumexx

repo: pumexx/pumex


text:
Allowing other extensions in file upload

Hey, I've a snippet with a singleton files field and I wonder how can I allow to upload other extensions beside rtf pdf office txt (which is default) !?
thanks!

Closing this and opening #193 to track progress of adding this into apostrophe-files in the unstable branch.

I wrote a nice HOWTO for this that doesn't require you to restate all the
existing settings:
http://apostrophenow.org/howtos/adding-file-types-in-the-media-library.html
On Fri, Feb 20, 2015 at 5:32 PM, benirose notifications@github.com wrote:

Closing this and opening #193
https://github.com/punkave/apostrophe/issues/193 to track progress of
adding this into apostrophe-files in the unstable branch.
—
Reply to this email directly or view it on GitHub
https://github.com/punkave/apostrophe/issues/145#issuecomment-75331923.

--
*THOMAS BOUTELL, *DEV & OPS
P'UNK AVENUE | (215) 755-1330  |  punkave.com

Thanks Tom! Your note at the bottom of the HOWTO might not be applicable any longer, and definitely isn't future proof: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img
Scroll down to "Supported image formats". However, the ones we support are known to be supported in all browsers.

Yeah, you could tweak the language if you like to indicate it doesn't make
sense to add a format unless it is supported in all browsers, and leave
it at that.
On Mon, Feb 23, 2015 at 3:41 PM, benirose notifications@github.com wrote:

Thanks Tom! Your note at the bottom of the HOWTO might not be applicable
any longer, and definitely isn't future proof:
https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img
Scroll down to "Supported image formats". However, the ones we support are
known to be supported in all browsers.
—
Reply to this email directly or view it on GitHub
https://github.com/punkave/apostrophe/issues/145#issuecomment-75627187.

--
*THOMAS BOUTELL, *DEV & OPS
P'UNK AVENUE | (215) 755-1330  |  punkave.com

Ya that makes sense.

authors: Danny-weebo benirose boutell

repo: punkave/apostrophe


text:
Pupillometry analysis script unreachable

The pupillometry analysis link returns a 404 page: Eye Tracking Research Data Analysis - Python pupillometry analysis scripts to analyze pupil_data.
Thank you for your great work!
Cheers

@qalhata did you move the repository to https://github.com/qalhata/Eye-Tracking-Research-Fixation-EDA-PupilLabs-Data- or is this repository unrelated? If there is no response within 3 days I will remove the dead link from the list.
@tombanfi Based on the description in our README I am guessing that the original was deprecated and not compatible with the current version of Pupil Capture.

@papr  @tombanfi  : My  apologies guys. I have been under a rock trying to complete my Thesis. with any luck I should submit that bad boy this weekend...  Phew!... #labouroflove.
To answer the question. Yes indeed, the link papr refers to is the correct one.  I did move the work there indeed
@tombanfi I just dumped it there not thinking anyone would be interested in playing with it. But please do feel free to tweak to your heart's desire. I was thinking of perhaps building an add-on  for the software - to allow quick visualisations etc from within the software. @papr : do you think this is feasible?
Cheers guys.

Updated with d6e0258ee3f3ed5080485820292799ac2aaaf551

authors: papr qalhata tombanfi willpatera

repo: pupil-labs/pupil-community


text:
Screenshots of Multiple Elements on Page Result in Some Blank Screenshots

Steps to reproduce
Tell us about your environment:

Puppeteer version: 1.2.0
Platform / OS version: macOS 10.13.3
URLs (if applicable): http://graasp.eu/ils/5a9d467e45aca55d33eafce6/?lang=en&printPreview
Node.js version: 8.10.0

What steps will reproduce the problem?
I'm trying to take screenshots of a series of HTML elements. The first couple of elements in the series are captured without a problem. However,  the third through the tenth appear blank. The screenshots have the right dimensions, but they have no content. Interestingly, if I take a screenshot of the full page, the elements appear fine.
Please include code that reproduces the issue.
const puppeteer = require('puppeteer');

const captureScreenshotsOfElements = async (elements) => {
  let i = 0;
  for (const element of elements) {
    await element.screenshot({ path: `tmp/${i}.png` });
    i += 1;
  }
};

//
(async () => {
  const browser = await puppeteer.launch({headless: false});
  try {
    const page = await browser.newPage();
    await page.setViewport({width: 1200, height: 1200});
    await page.goto('http://graasp.eu/ils/5a9d467e45aca55d33eafce6/?lang=en&printPreview', { waitUntil: 'networkidle0' });

    // wait five more seconds just in case
    await page.waitFor(5000);

    // dismiss cookie banner
    const dismissCookiesMessageButton = 'a.cc-dismiss';
    await page.waitForSelector(dismissCookiesMessageButton);
    await page.click(dismissCookiesMessageButton);

    // print screenshot
    await page.screenshot({path: 'tmp/fullPage.png', fullPage: true});

    const gadgets = await page.$$('div.gadget-content');
    await captureScreenshotsOfElements(gadgets, page);

    await browser.close();
  } catch (err) {
    console.error(err);
    browser.close();
  }

})();

In short I do the following:

Load up the page.
Dismiss cookie banner.
Take screenshot of full page.
Select the elements and capture screenshots of each of them.

If have switched steps 3 and 4 around and get the same result.
What is the expected result?
All of the screenshots of the elements show the element content.
What happens instead?
The first three element screenshots show the content. The rest of them are empty.
I believe that there is a small moment after the viewport changes where the element is blank and that's when the screenshot is take. Is there anyway of waiting a bit before taking the screenshot of the element?

I was able to work around this by setting the viewport height to a very large number (e.g. 1000000).

@joeldouglass, I just tried doing that and it results in the browser crashing.

Having the same issue on a longer page. Increasing the viewport height was the only workaround that worked for me (waits didn’t change the behavior).
It does look like a common issue: https://github.com/GoogleChrome/puppeteer/issues/2423#issuecomment-383345790
@juancarlosfarah Have you tried setting the viewport height to the real page height (you shouldn’t need more).

I have also had issues with this, and none of the viewport workarounds that were posted on related issues worked out for me. It seems like the problem has something to do with how puppeteer/chrome resizes the viewport before clipping the element - I notice the page content going completely blank during the half second the screenshot is taken.
However, I observed that when screenshotting the page, the page doesn't go blank. So I came up with workaround based on that: rather than using puppeteer to screenshot the element, I screenshot the whole page, and then outsource the cropping of the element to sharp, an image processing library.
const pathname = path.join(screenshotDir, screenshotFile);
const bb = await element.boundingBox();
if (!bb) {
    throw 'No ad bounding box';
}

// Sometimes the bounding box has non-integer values, in the sites I'm automating. Round the
// values just in case.
let boundingBox = {
    x: Math.floor(abb.x),
    y: Math.floor(abb.y),
    height: Math.ceil(abb.height),
    width: Math.ceil(abb.width)
}

// Take screenshot of page, then crop with sharp
const buf = await page.screenshot();
await sharp(buf).extract({
    left: boundingBox.x,
    top: boundingBox.y,
    width: boundingBox.width,
    height: boundingBox.height
}).toFile(pathname);

authors: eric-zeng joeldouglass juancarlosfarah michaelx

repo: puppeteer/puppeteer


text:
ci: add default timeouts to tests

TestExpectaions may be wrong now that we we may Fail rather expect Mocha Timeout.
So we may need to handle that in the test runner.

We would need a solution for when we set the timeout inside the test themself,
Maybe we can extend the original this.timeout to do this as well.


text:
(QENG-4034) Automatically URL-decode raw input

This commit adds a "pre-processing" step to the Parser that will always
attempt to URL-decode the input before parsing it. This behavior is
necessary to support specifying arbitrary settings in the input string
when the input ends up as part of an HTTP URL, such as almost every
usage we currently have in Jenkins.

@waynr @johnduarte For review
I'm confident these are the changes necessary, but I still want to verify we can use these URL-encoded inputs as test_targets in Jenkins so I'm going to create a simple job with an axis to reproduce the buggy behavior and validate these changes fix it.

:+1:  LGTM. Thanks @nwolfe

:+1:  looks good to me also, let me know when you've had a chance to try it out on a Jenkins instance.

@waynr @johnduarte Jenkins job demonstrating this will work: https://jenkins-qe.delivery.puppetlabs.net/job/beaker-hostgenerator_layout_axis_test/

authors: johnduarte nwolfe waynr

repo: puppetlabs/beaker-hostgenerator


text:
(PUP-7281) Add ability to declare options as an entry in Hiera defaults

This commit makes it possible to declare an options hash under the
hiera.yaml defaults entry. That hash will then be used by all entries
beneath hierarchy that does not declare an options entry of their
own. The options hash is never merged. If an entry declares it, it
replaces the default.

CLA signed by all contributors.

authors: puppetcla thallgren

repo: puppetlabs/puppet


text:
(PUP-7742) Ensure getpwent and getgrent iterators are properly closed

A regression introduced in PUP-7498 commit e45ae7c where an iteration
was introduced that started with setpwent or setgrent and then iterated
each entry with the corresponding getpwent or getgrent until no more
entries were found. A call to endpwent or endgrent was never made.
In essence, this means that the internal iterator is reset to the start
before each iteration so it should be fine. The problem is that on some
platforms (notably on CentOS 7) the lack of the call to endpwent or
endgrent, causes a cached iterator to be reused. It is never refreshed
with new values. Consequently, a call to useradd would succeed, but the
added user would then not be found.
This commit replaces the initial call to setpwent or setgrent with
an ensured call to endpwent or endgrend at the end. The initial
set call was deemed unnecessary since it should be considered a serious
problem if the iterator is left open somewhere else. It's better to make
such problems abundantly apparent than adding attempts to overcome them.
This commit contains no tests. A proper acceptance test must be written
that asserts this functionality on CentOS-7 (the first platform where the
problem was discovered).

CLA signed by all contributors.

Thanks @thallgren

authors: MosesMendoza puppetcla thallgren


text:
(PUP-8653) Use new REST client to download certificates

This commit updates the SSL::Host class to use the new REST client to
download certificates, instead of going through the indirector. It also
updates logic for saving certs.

jenkins please test this

CLA signed by all contributors.

This needs https://github.com/puppetlabs/puppet-runtime/pull/54 to be picked up by puppet-agent before we can properly run acceptance tests.

jenkins please test this

It would be nice to have some logging around failure.

jenkins please test this

jenkins please test this

The acceptance run from that last check was green, so pending comments from @pcarlisle I think this is ready to go.

Updated with logging around failure to load invalid certs, will add logging on 404 responses when I update this to use the streaming API I'm working on as part of PUP-9702.

jenkins please test this

See the above linked PR for other failure logging.

authors: Magisus pcarlisle puppetcla


text:
(PUP-9727) Reduce specificity of gem, pip providers

Due to the recent changes on package providers, when
no default package provider is explicitly set it will
always fallback to gem/puppet_gem/pip/pip3 because of the way
that specificity is calculated for a system which does
not have a default(the match expresion and number of ancestors)
Since those are specific language package managers,
their specificity was lowered to 1 if none of those providers
are set as default provider, otherwise the specficity
is calculated like it was before.

CLA signed by all contributors.

ad hoc build: https://jenkins-master-prod-1.delivery.puppetlabs.net/view/Adhoc/job/platform_puppet-agent-extra_puppet-agent-integration-suite_adhoc-ad_hoc/602/
There are some errors which do not seem related to the package providers changes

The debian puppet failure is due to PUP-9564

authors: gimmyxd joshcooper puppetcla


text:
不算bug，一丢丢文档问题

描述问题 (Describe the problem)
文档地址：https://yiming_chang.gitee.io/pure-admin-doc/pages/routerMenu/#%E5%A6%82%E4%BD%95%E5%8F%AA%E8%A6%81%E9%9D%99%E6%80%81%E8%B7%AF%E7%94%B1
加入使用静态路由只需要贴心按照文档的描述改就行，再修改login页面的时候我发现如果使用wen'dang文档下面的截图写的话会在初次登录的时候无法获取到除首页以外的其他静态路由（仅在首次登录时候会获取不到静态路由 刷新后可以）

修改后发现截图和上面让改写的差了一个setToken先后的区别，如果按照下图修改是可以的

如何复现该问题 (How to reproduce the problem)

操作系统和浏览器信息 (Operating system and browser information)
No response
验证 (Verify)

[ ] 是否仔细阅读过 文档 (Have you read documentation carefully)
[X] 检查是否存在相同或类似的问题 issues (Check for the same or similar issues)

文档等我们开发任务完成后 会统一补充

这个只要静态路由的文档删了？

repo: pure-admin/vue-pure-admin


text:
Introduce monad transformer FT

This allows running FT atop any monad, not just Identity.
History and usecase: purescript-corefn uses purescript-foreign to parse the corefn produced by PureScript. With non-trivial programs the corefn quickly grows large enough to blow the stack while parsing. By recovering the ability to change out the underlying monad, we can instead run on top of Trampoline and evaluate using runTrampoline, thereby recovering stack safety.

This looks like something we should try to do with the 0.14 compiler release, is there any objection?

Any updates on this PR?

Any updates on this PR?

authors: JordanMartinez felixSchl kl0tl

repo: purescript/purescript-foreign


text:
More efficient toUnfoldable without the Ord constraint

Keep a List of Maps as the unfold state instead of a single Map.
Appending Maps together was linear time (making the entire function quadratic?).
Prepending to a List is constant time and doesn't require an Ord k instance.

Here's benchmark results for smaller maps, showing the quadratic behavior:

(json)
Here's the code. It's using {1: 1, ..., n: n} as test cases.
It gets really slow for big maps. The stack blows with maps sized around 10,000 because List's append is not stack-safe. The new version doesn't have this issue because it's not using <>.

Wow, that's pretty significant! Ok then, LGTM :+1:

Awesome, thanks!

authors: garyb hdgarrood sammthomson

repo: purescript/purescript-maps


text:
updated question.model.js

Related Issue
closes #163, now the user is able to add questions link as well
Checklist:

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.

Screenshots

Original
Updated

original screenshot
updated screenshot

@codewithvk pls review


text:
api: split graphql schemas

Description
< Some description of what changes are included in this PR >
Tasks
Major Changes

:rocket: < Name of the task > WT-XXX

Bug Fixes

:ambulance: < Name of the task > WT-XXX

Minor Tweaks

:lipstick: < Name of the task > WT-XXX

Pull request checklist
For feature development:

[x] Have you tested your code?
[ ] Did you commit only code related to your feature?
[x] Are you sure you did not break any other feature?
[x] Have you checked your PR before you assigned it to reviewer(s)?
[x] Are you sure your feature does not contain any hacks?

So the issue with appsync simulator is resolved @bauer01 ?

So the issue with appsync simulator is resolved @bauer01 ?

So the issue with appsync simulator is resolved @bauer01 ?

@FilipPyrek yes, watchman reinstall and then simulator upgrade helped 😉

repo: purple-technology/purple-stack


text:
Use AsRef generic type parameter instead of &str or &Path

For filename operations, it might be better to accept anything that is AsRef<Path> instead of limiting input to only &stror &Path.
As a bonus directory argument of find() function is not mut anymore.

This makes a lot of sense to me. However, it implies a change in the public API, so I'd love to hear some feedback from others on whether it's worth it. @casey @daboross thoughts?

Sure. Please note that it changes public API in a backward compatible fashion - since AsRef<Path> is implemented for str and Path

I'm in support of accepting AsRef<Path>, but could we do it without allocating a PathBuf? Storing &'a Path should work just as well as &'a str did before.
If there's a concrete advantage to the PathBuf I'd be OK with that, but right now it looks like the only advantage is not needing 'a.
This isn't technically a breaking change, but it can be one in a lot of cases anyways. This will break any places where the compiler relied on concrete &str or &Path arguments to infer some other type. Still in support of changing this, just wanted to make that note.

PathBuf was already in use by Finder.directory. I was considering replacing it with Cow but decided to experiment with it later.

This seems like a good change to me. There are some cases where it won't be backwards compatible, but I think this is worth the version bump.
I agree with @daboross that if filename can be a &Path it probably should be.

Converting filename back to &Path becomes surprisingly convoluted and calls for other visible API change. Anyway, long-time keeping &Path which is created from a parameter that is AsRef<Path> is not trivial, since it will outlive the parameter itself (not necessarily the value it represents).

Ah right, I ran into that recently myself. An example which shows the problem:
use std::path::{Path, PathBuf};

struct Foo<'a> {
  path: &'a Path,
}

impl<'a> Foo<'a> {
  fn new<P: AsRef<Path> + 'a>(path: P) -> Foo<'a> {
    Foo {
      path: path.as_ref(),
    }
  }
}

fn main() {
  let p = PathBuf::new();

  // `p` is passed by value, and so no longer exists when the call to new returns
  // which prevents Foo from retaining a reference to the path created by `as_ref`
  let foo = Foo::new(p);
}

It should be possible to have Finder::filename take filename: &'a P instead of filename: P to resolve any lifetime problems with AsRef. It'll force the actual value to be borrowed from outside, so it will live long enough. Not an ideal API, but it's internal-only, so it seems fine?
Here's a patch to do this which should compile and pass tests: https://gist.github.com/daboross/346bc99b070d61d08ec638cd38215bc7. Feel free to incorporate and/or use as a reference if needed.
Glad to see this API change - I just don't want it to be at the cost of efficiency.

@daboross Here we go. Appreciate your persistence!

Thanks @imp! I'm good with this PR now if @purpliminal and @casey are.

Looks good to me! Finder.directory was there from a previous iteration where you could set the directory to begin searching from, but it was unnecessary so I pushed a commit removing it.

@casey Good. I was thinking about removing Finder.directory, but wasn't sure whether it is here for a future expansion or just a leftover from the previous iteration.

Maybe it'll be used eventually, if someone ever wants a feature to set the directory that dotenv searches from, but for now it's just cruft :P

So, is there any outstanding issues with this PR?

I don't think so! Merging. 💯

Thanks!

authors: casey daboross imp purpliminal

repo: purpliminal/rust-dotenv


text:
How to get the userinfo

Hi,
I'm wondering is it possible to expose the user profile endpoint? So I can pass the information to my app.
This PR(https://github.com/bitly/oauth2_proxy/pull/388) seems do the trick!?
thanks

Perhaps the code from that old PR could be ported across to this repo?

authors: JoelSpeed WakeupTsai


text:
REST adapter client logging not configurable

The log4j2.xml is compiled into the JAR. It would be nice if alternative logging configuration could be provided.

Alternative logging can be provided by setting the log4j.configurationFile system property.

Note added to README in d853d4c60c31677f40f078af2e6f19ac9a9f4f33

repo: pushtechnology/diffusion-rest-adapter


text:
sprig with form upload file attachments

Hey Ben,
I use the sprig plugin to load the form and bypass the cache. But I need to extend the form with upload file attachments. For that I use the enctype tag in the form.
     <form class="kontakt--form" sprig="sprig" s-method="post" s-action="contact-form/send" enctype="multipart/form-data">
But I get this message
TypeError Argument 1 passed to craft\web\View::renderTemplate() must be of the type string, null given, called in /app/vendor/putyourlightson/craft-sprig-core/src/controllers/ComponentsController.php on line 65.
Am I missing something?

It looks like you should be using the s-encoding attribute, see https://putyourlightson.com/plugins/sprig#s-encoding

Hey Ben,
I did already test it with s-encoding  . The Form works but no attachment arrives. If I test the same code, without sprig the attachment will arrive.
without sprig:

with sprig:

the input file looks like this:
<input type="file" name="attachment[]" multiple="multiple">
Do you have any idea what needs to be adjusted?
Thanks for you help,
Pascal

I can't replicate this locally. Please send me a scaled down version of your form, that does not work.

Looks like you have a typo, it should be s-encoding="multipart/form-data".

wow, you are right. I'm sorry about your time.
I took the enctype="multipart/form-data" from the craft contact form plugin Docs and just put an s in front of it. how embarrassing.  I think it's time for vacation. thank you very much

Sure thing!

authors: bencroker pacoNew

repo: putyourlightson/craft-sprig


text:
improve speed of Location.get_airmass

[x] Closes #502
[x] I am familiar with the contributing guidelines.
[ ] Fully tested. Added and/or modified tests to ensure correct behavior for all reasonable inputs. Tests (usually) must pass on the TravisCI and Appveyor testing services.
[x] Updates entries to docs/sphinx/source/api.rst for API changes.
[x] Adds description and name entries in the appropriate docs/sphinx/source/whatsnew file for all changes.
[x] Code quality and style is sufficient. Passes git diff upstream/master -u -- "*.py" | flake8 --diff
[x] New code is fully documented. Includes sphinx/numpydoc compliant docstrings and comments in the code where necessary.
[x] Pull request is nearly complete and ready for detailed review.

@mikofski see the linked issue for timings. I slightly prefer this approach for style reasons. Happy to change if most people prefer the other approach.

repo: pvlib/pvlib-python


text:
Allow prefix <CONFIG_DIR> in env variables

I have a few environment variables that are a directory, and it would be nice to make this relative to the directory containing mprocs.yaml , like is possible for cwd

Can you elaborate please?

I can do procs: {baz: {cwd: <CONFIG_DIR>/foo/bar }}.
I would like to also do procs: {baz: {env: { FOO_DIR: <CONFIG_DIR>/foo/bar }}}

I understand now. Would be a good feature.

authors: jayvdb pvolok

repo: pvolok/mprocs


text:
Bolt Action Compatibility

Is your feature request related to a problem? Please describe.
Would love to use your app to create Army lists for Bolt Action v2.0
Describe the solution you'd like
Can you please make Battlescribe2TTS compatible with Bolt Action v2.0?

Just to set expectations, I do not play Bolt Action and I'm unlikely to get to this in the near future, sorry. I'll leave this open on the off chance the pandemic leaves me with more free time than I expect though.
Also, the code is 100% open source - if anyone in the Bolt Action community that you can find writes Haskell and wants to do the work, I'll 100% be helpful!

authors: UpperSideBand pwestling

repo: pwestling/battlescribe-roster-parser


text:
NetworkError

No matter what roster I try to upload (I tried 3), I am not getting a code to imput into TTS, only a message saying Network Error. My internet seems to be working fine and I rebooted both my PC and the router as well as attempting it again while connected to a VPN.

I had the same problem... until I realized I was trying to upload the wrong file. If you upload a PDF or an HTML file instead of the .rosz, then you get a "Network Error". Perhaps this could help!


text:
Drop Device API

It is no longer needed as we will focus only on bulldog library.

Fixed in https://github.com/px3/silverspoon/commit/ca886832f25b637fc510c0c36682e1f2003f4f55 and https://github.com/px3/silverspoon/commit/e963a1474a8c87a7e4e9e8391bc455842336a362


text:
Custom Drawer Header with no Menu Icon not Aligned with Nav Items

Describe the bug
Header content is shifted over whenever there is no icon present.
What is the expected behavior?
Whenever there is no icon in the menu header, the header text should align with the nav items.
What are the steps to reproduce?
Go to the custom drawer header storybook example; remove the menu icon from the code and look at the alignment
Screenshots or links to minimum reproduction example

Environment

Windows Chrome
Anything else to add?

pxb-drawer-header-icon-wrapper is still rendered when no icon is provided and this takes up space, shifting header content over.

repo: pxblue/angular-component-library


text:
Enable components to be customized with classes

Describe the desired behavior
Components should accept a classes prop that will allow a user to add additional styles to elements in the component via classes (refer to how material-ui does this)
Describe the current behavior
No such customization
Is this request related to a current issue?
Requested by AO team working on WAS to be able to use theme values in component props

This functionality was added to all components as a series of PRs.

repo: pxblue/react-component-library


text:
connect a processedVariable to faraday experimental database

Summary

need a standard data format.
standard protocol format
standard parameter format

Motivation
for comparing a simulation to experimental data

Is this still something we are planning on doing?

I haven't heard anything about the Faraday experimental database, so going to close this

authors: martinjrobins tinosulzer

repo: pybamm-team/PyBaMM


text:
Py_Initialize: can't initialize sys in XCode 8

I just download the latest release for iOS platform, drag into project and run the hello world.
xcode output:
Could not find platform independent libraries
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to [:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
code in ViewController.m:
#import <Python/Python.h>
- (void)viewDidLoad {
    Py_Initialize();
    PyRun_SimpleString("print(\"Hello world!\")");
    Py_Finalize();
}

Frameworks:
...
libsqlite.tbz
libz.tbz
Enviorment:
XCode 8
iOS 9.3

I have try Py_SetPythonHome() with NSHomeDirectory() etc. and it just didn't work. Could you give me some advice or sample code.
thx.

I don't know for sure what's up, but it may help to cross-check your code with main.m that cookiecutter template generates, you can check the result e.g. here https://github.com/dimaqq/playing-with-python-on-ios/blob/master/dimark/main.m
You'll see there's a bunch more settings calls before Py_Initialize

@dimaqq Thanks for your answer. I'll try it.

@chinsyo Any luck resolving this? I'm having a very similar issue. App builds and starts, but calling Py_Initialize crashes it.

Hi there. I have tried the same code in the link above and on Xcode 8.x but Py_Initialize() still results in:
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {

    NSString *tmp_path;
    NSString *python_home;
    wchar_t *wpython_home;

    NSString * resourcePath = [[NSBundle mainBundle] resourcePath];

    // Special environment to prefer .pyo; also, don't write bytecode
    // because the process will not have write permissions on the device.
    putenv("PYTHONOPTIMIZE=");
    putenv("PYTHONDONTWRITEBYTECODE=1");

    python_home = [NSString stringWithFormat:@"%@/Library/Python.framework/Resources", resourcePath, nil];
    NSLog(@"PythonHome is: %@", python_home);
    wpython_home = Py_DecodeLocale([python_home UTF8String], NULL);
    Py_SetPythonHome(wpython_home);

    // iOS provides a specific directory for temp files.
    tmp_path = [NSString stringWithFormat:@"TMP=%@/tmp", resourcePath, nil];
    putenv((char *)[tmp_path UTF8String]);

    NSLog(@"Initializing Python runtime");
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    // Override point for customization after application launch.
    return YES;
}

Any help would be appreciated

It's very important that you get the right template depending on your Python version. Python 3.4 and 3.5 require very different startup code, and if you try to run a 3.4 template using Python 3.5, you're going to get errors like the one you describe.

I was able to make this work by doing the following:
    NSString *tmp_path;
    NSString *python_home;
    wchar_t *wpython_home;

    NSString * resourcePath = [[NSBundle mainBundle] resourcePath];

    // Special environment to prefer .pyo; also, don't write bytecode
    // because the process will not have write permissions on the device.
    putenv("PYTHONOPTIMIZE=");
    putenv("PYTHONDONTWRITEBYTECODE=1");

    python_home = [NSString stringWithFormat:@"%@/Python.framework/Resources", resourcePath, nil];
    NSLog(@"PythonHome is: %@", python_home);
    wpython_home = Py_DecodeLocale([python_home UTF8String], NULL);
    Py_SetPythonHome(wpython_home);

    // iOS provides a specific directory for temp files.
    tmp_path = [NSString stringWithFormat:@"TMP=%@/tmp", resourcePath, nil];
    putenv((char *)[tmp_path UTF8String]);

    NSLog(@"Initializing Python runtime");
    Py_Initialize();

And I had to copy the Python.framework as part of build settings

Closing on the basis the problem seems to be resolved; please reopen if that's not the case.

authors: AnuiShark chinsyo dimaqq freakboy3742 settwi

repo: pybee/Python-Apple-support


text:
Update tutorial-0.rst

I'm new at contributing.
UPDATED FOR LINUX UBUNTU 16.04
If I can do anything better please let me know I just want to be helpful!
Cheers,
-Ryan

Created a Pull request for the added tutorial

repo: pybee/briefcase


text:
Make PyObject subclass Generic & expose it to users

Discussed in https://github.com/samuelcolvin/pydantic/discussions/3833

Originally posted by rsayn February 19, 2022
Hello,
I was thinking that we could add type hinting to PyObject in order to enable editor suggestions based on the actual runtime object that is passed or imported.
I experimented a bit locally and I came up with the following code:
# inside types.py
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ PYOBJECT TYPE ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

class _PyObject(Generic[T]):
    validate_always = True

    @classmethod
    def __get_validators__(cls) -> 'CallableGenerator':
        yield cls.validate

    @classmethod
    def validate(cls, value: Any) -> T:
        if callable(value):
            return cast(T, value)

        try:
            value = str_validator(value)
        except errors.StrError:
            raise errors.PyObjectError(error_message='value is neither a valid import path not a valid callable')

        try:
            return cast(T, import_string(value))
        except ImportError as e:
            raise errors.PyObjectError(error_message=str(e))

if TYPE_CHECKING:
    PyObject = Type # instead of `Callable`
else:
    class PyObject(Generic[T], _PyObject[T]):
        pass

This is fully backwards compatible with the current PyObject signature, the only adjustment would be using PyObject[Any] instead of the plain PyObject.
Attaching screenshot of working editor suggestions in VS Code:

This also suggests constructor arguments properly:

The other change I made here was to move the PyObject definition from the "else" branch to the actual codebase. The reason for this is that users could then be able to subclass _PyObject, should they need to customize the validation internals.
Currently, when trying to subclass PyObject, static checkers would complain because the actual implementation is not available at type checking time.
EDIT
Now that I'm opening the issue, I notice that in the example I used the type annotation should read PyObject[Type[MyClass]] instead of PyObject[MyClass]... My bad 😄
Also, the "type check" time PyObject could be set to Union[str, Callable] rather than Type.

As commented in https://github.com/pydantic/pydantic/discussions/3833#discussioncomment-5759263, If I understand correctly, I think you can get the behavior you want via cls: typing.Type[MyClass] = MyClass. Does PyObject have some different behavior to this that is important to preserve? (PyObject has been removed in v2.)
Since that seems to be the right solution, this issue is somewhat stale, and PyObject has been removed in v2, I am going to close this issue. But please comment here if the answer is not sufficient and you would like the issue re-opened.

authors: dmontagu rsayn

repo: pydantic/pydantic


text:
Cannot use proper subclass of str as input to a strict str field

from pydantic import BaseModel

class Model(BaseModel):
    model_config = {'strict': True}
    x: str

class MyString(str):
    pass

Model(x=MyString('foo'))
"""
pydantic_core._pydantic_core.ValidationError: 1 validation error for Model
x
  Input should be a string, not an instance of a subclass of str [type=string_sub_type, input_value='foo', input_type=MyString]
    For further information visit https://errors.pydantic.dev/0.41.0/v/string_sub_type
"""

In general, I don't think strict mode should prevent the use of strict subclasses as inputs to the, as this is at odds with the type system used by python type checkers.
I think the above example is also a legitimate scenario where you might want to support subclasses of str — otherwise, it's impossible to use strict mode if you might want to pass subclasses of str.
Also, as far as I can tell, no other field type works like this (e.g., int), which makes me question the value of having this failure mode for strict str — it seems like an inconsistency, and I don't see it adding much/any value.
It feels to me that in the rare scenario where this behavior is desirable, it would be better to use a custom validator to reject subclasses; this could be baked into a reusable type via:
from typing import Any, Annotated

from pydantic import BaseModel, BeforeValidator

def is_exact_str(v: Any) -> str:
    if isinstance(v, str) and type(v) != str:
        raise ValueError('Must not provide a subclass of `str`')
    return v

ExactStr = Annotated[str, BeforeValidator(is_exact_str)]

class Model(BaseModel):
    x: ExactStr

class MyString(str):
    pass

Model(x=MyString('foo'))
"""
pydantic_core._pydantic_core.ValidationError: 1 validation error for Model
x
  Value error, Must not provide a subclass of `str` [type=value_error, input_value='foo', input_type=MyString]
    For further information visit https://errors.pydantic.dev/0.41.0/v/value_error
"""

I feel somewhat strongly that this behavior should be changed, but am open to disagreement.
(Also, I guess this will need to be implemented in pydantic-core.)

I hadn't realised about this, I think it's caused strange behaviour:
from typing import Annotated

from pydantic import BaseModel, StringConstraints

class MyStr(str):
    pass

class TestModel(BaseModel):
    test: str

m1 = TestModel.model_validate(dict(test=MyStr("test")))
print('m1 MyStr:', type(m1.test))

m2 = TestModel.model_validate(dict(test=MyStr("test")), strict=True)
print('m2 str:', type(m2.test))

class TestModel2(BaseModel):
    test: Annotated[str, StringConstraints(strip_whitespace=True)]

m3 = TestModel2.model_validate(dict(test=MyStr("test")))
print('m3 MyStr:', type(m3.test))

m4 = TestModel2.model_validate(dict(test=MyStr("testtest123")), strict=True)
print('m4 str:', type(m4.test))

Also different behaviour for strict=True means that str | int will retain subtype information while str will not.

The cases with StringConstraints I expect. We always loose information when checking constraints. One could argue that if we're just checking length or something we should preserve the original input object but that is not possible here given that you are using strip_whitespace.
The only one that surprised me is that the lax version without constraints doesn't retain the type. I presume we can change that. Then again we've made the promise in v2 that we give you the right output type, not that we preserve the input type (which was kind of an implicit promise in v1). So if we can fix that to make it more consistent with low effort I think we should. But I wouldn't consider it a huge issue.

conversation moved to https://github.com/pydantic/pydantic-core/issues/786.

authors: adriangb dmontagu samuelcolvin


text:
Incorret generated JSON for a model containing a 'set' -> error on validation model

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
Hello,
the JSON generated from a model containing a 'set' is not correct.  It is impossible to validate it.
It seems that the generated JSON is not correct:
{'my_set': ['bbb', 'aaa']}

instead of
{'my_set': {'bbb', 'aaa'}}

When I validate the model (with strict mode), I have the following error
Input should be a valid set [type=set_type, input_value=['bbb', 'aaa'], input_type=list]
  For further information visit https://errors.pydantic.dev/2.4/v/set_type

Example Code
from pydantic import BaseModel

class MyObject(BaseModel):
    my_set: set[str]

me = MyObject(my_set={"aaa", "bbb"})

model_as_json = me.model_dump(mode='json')

# validate the model, an error is thrown
model = MyObject.model_validate(model_as_json, strict=True)

Python, Pydantic & OS Version
pydantic version: 2.4.2
        pydantic-core version: 2.10.1
          pydantic-core build: profile=release pgo=true
                 install path: /home/yoann/.local/lib/python3.9/site-packages/pydantic
               python version: 3.9.16 (main, Apr  3 2023, 12:39:37)  [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
                     platform: Linux-4.18.0-477.10.1.el8_8.x86_64-x86_64-with-glibc2.28
             related packages: email-validator-2.0.0.post2 typing_extensions-4.7.1 fastapi-0.103.1

{'my_set': {'bbb', 'aaa'}} isn't valid JSON (there's no such things as sets, so pydantic uses arrays instead).
So yes, trying to use model_validate with strict=True will raise a ValidationError because the set was "converted" to an array. Instead, you can do:
model_as_json = me.model_dump_json(mode='json')

model = MyObject.model_validate(model_as_json, strict=True)

authors: Viicos koikoube


text:
Add how to skip the change file check to pull request template

Change Summary
skip change file check

Related issue number

Checklist

[ ] Unit tests for the changes exist
[ ] Tests pass on CI and coverage remains at 100%
[ ] Documentation reflects the changes where applicable
[ ] changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

But if you add this to the pull request template, it'll be included in the pull request body by default. 🙃
Options:

Special regex to chat catch this in hooky
Keep it like this, but check manually (I'm not keen)
Have a link to instructions in the template

Ah :smile: :man_facepalming: .
I've changed it and added a link to hooky.
Feel free to close the PR if you think it doesn't make sense at all

Looks good, thanks.
It would be amazing if there was a way to render the PR advice in markdown, before creating a PR, but there doesn't seem to be a way.

authors: hramezani samuelcolvin


text:
Pivot_table with categorical data gives ValueError

This errors for me on master as well: http://stackoverflow.com/questions/29945342/pandas-error-with-pivot-table-and-categorical-data
  category1   category2  numeric
0      grey    complete        1
1      grey    complete        2
2       NaN  incomplete        3
3      blue    complete        4

In [13]: df = pd.read_clipboard()

# with plain strings it works

In [16]: pd.pivot_table(df, index = 'category1', columns = 'category2', values = 'numeric', aggfunc = 'count')
Out[16]:
category2  complete
category1
blue              1
grey              2

In [17]: df['category1'] = df['category1'].astype('category')

In [18]: df['category2'] = df['category2'].astype('category')

In [19]: pd.pivot_table(df, index = 'category1', columns = 'category2', values =
 'numeric', aggfunc = 'count')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

...

c:\users\vdbosscj\scipy\pandas-joris\pandas\core\common.py in _astype_nansafe(ar
r, dtype, copy)
   2726
   2727         if np.isnan(arr).any():
-> 2728             raise ValueError('Cannot convert NA to integer')
   2729     elif arr.dtype == np.object_ and np.issubdtype(dtype.type, np.intege
r):
   2730         # work around NumPy brokenness, #1987

ValueError: Cannot convert NA to integer

dupe of #9534

authors: jorisvandenbossche jreback

repo: pydata/pandas


text:
strftime and/or format support for DatetimeAccessor

There's no easy way currently to control the conversion to string of time values/series. For this purpose, Panda's own .dt attribute has an implementation of strftime.
Is there interest in adding similar functionality to xarray?

Sure, I think we would be happy to copy almost any method from .dt in pandas to xarray.

authors: dopplershift shoyer

repo: pydata/xarray