text:
OpenAPI spec JSON download URL is not available with @fastify/swagger and @scalar/fastify-api-reference

What happens?
I am using @fastify/swagger and am investigating swapping from @fastify/swagger-ui to @scalar/fastify-api-reference.
Scalar is much nicer, thank you for making it, but I've hit a roadblock:
Some of my tooling fetches the OpenAPI spec JSON file from a URL provided by @fastify/swagger-ui.
With @scalar/fastify-api-reference, it seems no such URL exists, despite https://github.com/scalar/scalar/pull/2947
Is there no endpoint I can fetch that would result in the spec JSON file?
What did you expect to happen?
Be able to fetch the spec JSON for use in external tooling, but no endpoint exists to grab it from
OpenAPI Document
No response

Ah, @scalar/fastify-api-reference does the rendering only. You can use @fastify/fastify-swagger to generate your OpenAPI document, and expose and an endpoint for it and pass that endpoint to @scalar/fastify-api-reference.
Does that help? I should make this more clear in the README.

Cheers for the fast response, I will look into the suggestion.
I've mostly followed this documentation: https://github.com/scalar/scalar/tree/main/packages/fastify-api-reference
It does say

Actually, we’re picking it up automatically, so this would be enough:
await fastify.register(require('@scalar/fastify-api-reference'), {
  routePrefix: '/reference',
})

That made me assume it would be a drop-in replacement of @fastify/fastify-swagger, and it was, except for the download link issue.
I scrolled through https://www.npmjs.com/package/@scalar/fastify-api-reference for extra hints, to no avail.
Now though, I do see the NPM doc linking to this: https://github.com/scalar/scalar/blob/main/documentation/integrations/fastify.md#set-up-fastify-swagger-optional
Which in turn has this in its depths:
// Serve an OpenAPI file
fastify.get('/openapi.json', async (request, reply) => {
  return fastify.swagger()
})

Coming solely from @fastify/swagger-ui, I assure you this is the first time I saw @fastify/fastify-swagger used like that 😄
I would assume most people who come from @fastify/swagger-ui are in the same boat as me, and would save us digging through a few scattered docs and make the switch to Scalar trivial, if @scalar/fastify-api-reference would expose an endpoint by default.
For reference, @fastify/swagger-ui's download link seems to point to${routePrefix}/json
And for the record, the only other incompatibility is that @fastify/swagger-ui seems to overbearingly have Fastify's ignoreTrailingSlash: true set somewhere, had me scratching my head for 2 minutes as to why Scalar won't load with my old links. Might warrant a mention in the docs?
I will try these changes a bit later and report back.

I exposed the docs, and it does work, but whereas the swagger-ui config was a one-liner:
const routePrefix = `/reference`;

await fastify.register(fastifySwaggerUi, { routePrefix, });

such elegance is now gone:
const routePrefix = `/reference`;
const routePostfix = `/json`;

fastify.get(`${routePrefix}${routePostfix}`, { schema: { hide: true, }, }, async () => {
    return fastify.swagger();
});

await fastify.register(fastifyScalar, {
    routePrefix,
    configuration: {
        spec: {
            url: `${routePrefix}${routePostfix}`,
        },
    },
});

Note the hide: true in schema, without this the exposed endpoint would end up in the spec.
Definitely a step back from the "Actually, we’re picking it up automatically, so this would be enough" minimal example over at: https://github.com/scalar/scalar/tree/main/packages/fastify-api-reference
And as for the trailing slash, that is handled weirdly by @fastify/swagger-ui, it is not a redirect, whereas here I can solve it with one, but I elected not to.
So yes, if the minimal config of
await fastify.register(fastifyScalar, { routePrefix, })

would handle exposing /reference/json, that would be a kind change for folks moving over from @fastify/swagger-ui, keeping it a one-liner! 😄

Nice! That’s awesome feedback. If you don’t find time to make PR leave a short note and I’ll submit one. :)

Hey, this was done in #3098!

authors: danwithabox hanspagel

repo: scalar/scalar


text:
feat: docusaurus integration

WIP 👀
I’m stuck! Looks like we need to somehow fix the setup to make JSX work properly. Maybe @amritk can help here?
Compiled with problems:
×
ERROR in ../../packages/docusaurus/src/ScalarDocu.ts
Module build failed (from ../../node_modules/.pnpm/babel-loader@9.1.3_@babel+core@7.24.0_webpack@5.90.3/node_modules/babel-loader/lib/index.js):
SyntaxError: /Users/hanspagel/Documents/Projects/scalar/packages/docusaurus/src/ScalarDocu.ts: Type parameter list cannot be empty. (25:4)

closing in favour of amrit/docusaurus

source: gharchive/pull-request

authors: hanspagel marclave

license: MIT

license_type: permissive

license_source: github-api


text:
Ensure traverse and sequence return ordered results

We already had tests for the order of elements in traverse.
The real issue was about the order in which effects were being evaluated.
Fix #157

@AL333Z Awesome work! Thank you! 🙏

repo: scalaz/scalaz-zio


text:
delay-job-completion: fix help message

We're currently facing
Notice: --------------------------------------------------
Sleeping for 30 minutes.
To cancel, delete the file '/home/runner/work/ring/ring/timeout_date' or end the debugging session.
To change the delay duration, update the file '/home/runner/work/ring/ring/timeout_date' with:
  date '+%s' --date='now + <X> minutes' > /home/runner/work/ring/ring/timeout_date ."
--------------------------------------------------
/home/runner/work/_temp/ca5a77ac-482f-47fc-83ac-876c2bbf4a8d.sh: line 11: Delete: command not found
Error: Process completed with exit code 127.

@tcarmet what is the process to test/tag/deploy this ?

@oliviergaraud To test it you can target the branch / commit directly and push a branch of the caller with this updated pointer. If it helps.

@oliviergaraud To test it you can target the branch / commit directly and push a branch of the caller with this updated pointer. If it helps.

Test is OK now

authors: liaud oliviergaraud

repo: scality/actions


text:
[FEATURE] include property name when validating response with response_schema_conformance

Is your feature request related to a problem? Please describe
When unsing response_schema_conformance the violation only describes the property type which is violated, but not the actual name of the property, which leads to longer lookups for a manual fix.
Describe the solution you'd like
I think it would be nice to improve this by including the name of the property of which the returned value is violated. Instead of:
- Response violates schema

    None is not of type 'string'

    Schema:

        {
            "type": "string"
        }

    Value:

        null

I would prefer something like this:
- Response violates schema

    None is not of type 'string'

    Schema:

        {
            "type": "string"
        }

    Value:

        null

    Property:

        <propertyName>

Describe alternatives you've considered
I checked the documentation but couldn't find any workarounds :(

Hi!
thank you for opening the issue! Indeed, it will be helpful. However, the schema could be arbitrary, so probably showing the path to the violated subschema would be the way to go. Ie “Failed at /properties/my-property”

Hi @Stranger6667, i agree, that would be even better.

The change will be available in Schemathesis 3.30 :) I hope to finish all the necessary details & publish it by the end of this week

Wow, that was super quick. Thank you 😄

source: gharchive/issue

repo: schemathesis/schemathesis


text:
API call to https://cis.schibsted.com/api/v1/identify is blocked by browser

Provide environment information
Firefox: 112.0.2
Which area(s) of the code does the issue occur in?
No response
To Reproduce
Open the network analysis of firefox browser. the api call is blocked:
Describe the Bug

Expected Behavior
What does this api call? If this api call isn't necessary, could it be removed?

This is related to #105 :)
The blocked request is coming from Firefox built-in tracking protection.
Since we use it internally in Schibsted we wanted some basic tracking to know if people were using it or not.
So we would love for anyone to contribute to #105, but it is also on the radar for the frontend re-write I am trying to do in #215

authors: Johannes-Andersen main-sonn

repo: schibsted/WAAS


text:
Extending a custom view with an initializer?

Hi again!
I'm trying to extend the lottie framework so that I can use it with layout.
The initializer for LOTAnimationView takes a name argument specifying the json filename you are looking to animate.
After reading through the Custom Property Types section - I was hoping to be able to extend the view so that I could declare the image in the xml file via an attribute i.e.
<LOTAnimationView
    name="IMAGE_NAME"
/>

However, it seems as though it must be added at declaration, not after. Is there anyway to use the extension to add an initializer that would allow me to do the above?

It's not documented because it's a bit experimental, but something like this should work:
extension LOTAnimationView {
    open override class func create(with node: LayoutNode) throws -> LOTAnimationView {
        guard let name = node.expressions["name"] else {
            throw LayoutError("name is required")
        }
        return self.init(name: name)
    }

    open override class var expressionTypes: [String: RuntimeType] {
        var types = super.expressionTypes
        types["name"] = RuntimeType(String.self)
        return types
    }

    open override func setValue(_ value: Any, forExpression name: String) throws {
        switch name {
        case "name":
            break // Ignore this - we set it during creation
        default:
            try super.setValue(value, forExpression: name)
        }
    }
}

Note that this will only work with a hard-coded name string in your XML though, it won't let you use an expression. There is a mechanism for that, but it's currently private.

I've added a better API for this in 0.4.18. It's documented in the README, but in short the above can now be simplified to the following, and will now work with any expression for "name", not just a hard-coded value:
extension LOTAnimationView {
    open override class func create(with node: LayoutNode) throws -> LOTAnimationView {
        guard let name = try node.value(forExpression: "name") else {
            throw LayoutError("name is required")
        }
        return self.init(name: name)
    }

    open override class var parameterTypes: [String: RuntimeType] {
        return ["name": RuntimeType(String.self)]
    }
}

authors: nextriot nicklockwood

repo: schibsted/layout

license: mit

license_source: bigquery


text:
Implement base class BaseRandom (c++20)

On branch dev-cpp20. Let's take benefit of code implemented for c++11.

Completed; Tested.

repo: schmouk/CppRandLib


text:
document (sparse) data format

Please document the data format that the tsne model wants with type 'sparse'.
I tried a nested array like [[i_11,...i_1j],...[i_n1,...i_nk]] where i_rs are the indices of the set values but that did not work.

for anyone wondering, it seems to be [ [i1, j1], ... , [in, jn] ] where i,j are the indices of the non-zero values.

repo: scienceai/tsne-js


text:
Expose more TQDM kwargs (specifically 'leave')

Hi! Thank you for making this library.
At this time, the only parameter exposed regarding the tqdm loading bar is verbose.
I'd like to expose the other kwargs, such as 'leave', which is the one that removes the loading bar from the output once it is done running.
I am not too sure regarding how to proceed, as adding all of the tqdm kwargs to all of the classes where 'verbose' appears seems like a poor choice. I think the best solution would be to create an Optional dataclass argument, which by default is None, capturing all of the tqdm kwargs, and make this available everywhere the 'verbose' argument appears.
Do let me know whether you find this agreeable - if so I will proceed with the PR.
Luca

This is a good idea in general. Not exactly conforming to the scikit-learn API, but quite close. But I think that using kwargs dictionary would be simpler for maintenance than a custom class. In short, if non-empty dict is passed in verbose argument, we assume that a) verbose is true (existing conditions will work as-is) b) kwargs will be passed to tqdm.

I would lean towards using a custom type such as the one in my PR #298 since dictionaries would require validation. With a custom type, you get validation built-in.

repo: scikit-fingerprints/scikit-fingerprints


text:
BN curve

This PR implements infrastructure for BN family of curves and instantiate BN256 from Etherem.
Type: Feature
Label: Ready to review
Priority: Medium
Motivation
Support of BN family that is widely used for pairing-based cryptography (Ethereum, libsnark, DIZK, Halo).
Followups

[ ] Instantiate BN254b from DIZK
[ ] Instantiate BN382 from o1-labs for Halo-like recursion

@Pratyush @howardwu @kobigurk That would be great if you can review this!

Thanks for the helpful chronology! Given this info, what do you suggest we
should name the curve? I think bn256 doesn’t make sense if there are two
different curves with that name.
On Sat, Jun 20, 2020 at 11:26 AM Kobi Gurkan notifications@github.com
wrote:

@kobigurk commented on this pull request.
In algebra/Cargo.toml
https://github.com/scipr-lab/zexe/pull/225#discussion_r443150273:

"bn256", "bls12_377", "bls12_381", "cp6_782", "bw6_761", "mnt4_298", "mnt4_753",
"mnt6_298", "mnt6_753", "ed_on_bls12_377", "ed_on_cp6_782",
"ed_on_bw6_761", "ed_on_bls12_381", "ed_on_mnt4_298", "ed_on_mnt4_753"
]

+bn256 = []
If I remember correctly, the history goes like this:

bn256 from cloudflare and google implement a BN curve that's
unrelated to Zcash. When the Ethereum devs ported it to the libsnark curve,
they kept the name.
bn128 is the name that was used by libsnark and then by Zcash,
denoting 128-bit security.
Later on, Zcash and the community mostly decided to switch to the
bn254 naming, since it wasn't 128-bit security anymore and 254-bit is the
size of the curve.

I believe that the pairing-friendly curve draft from IETF actually calls
this curve BN_SNARK1.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scipr-lab/zexe/pull/225#discussion_r443150273, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAYSJ6WBXRWASFVGALDFABDRXT5M5ANCNFSM4OBLVEBA
.

IMO bn254.

Ok sounds good then!

Thank you Kobi for the chronology! If we are to choose the name that makes most sense, then bn254 is the best option. We just need to mention (in mod.rs for instance) that the curve is known also as bn256 in Geth and bn128 in libff.

For naming I suggest BN254_Snarks as in the literature BN254 often refers to the one from Nogami et al with parameter -(2^62 + 2^55 + 1)
MCL also uses similar scheme to differentiate both: https://github.com/herumi/mcl#support-curves

Hmm I feel like bn254 is fine as it is well known in the context of SNARK systems (which this library is largely targeted towards). Furthermore, the curve will probably eventually be deprecated anyway by most systems because it's insecure.

authors: Pratyush kobigurk mratsim yelhousni

repo: scipr-lab/zexe


text:
More vectorized functions

In the last days, I tried using two scipy functions, scipy.linalg.solve_banded (to solve a linear system with tridiagonal matrix) and scipy.signal.residue (to compute a partial fraction decomposition) but ended up implementing them myself because I needed to apply them to multiple (>1000) inputs (i.e. 1000 banded solves with the same band structures, and 1000 partial fraction decompositions with the same polynomial degrees), and doing so in a for loop was terribly slow compared to just doing the bits of arithmetic explicitly myself, in a vectorized fashion.
Is there something I'm missing here, as to (a) why these functions do not simply accept inputs with trailing dimensions (b) how other uses deal with this? Adding numba around the for loop didn't help because using scipy functions makes numba go into object mode.
I can see how vectorization is not straightforward for iterative algorithms. However, I believe even in that case, just running as many iterations as needed for the slowest converging input, or using boolean masks, would be okay?  Was it a conscious design decision not to do that / would PRs in that direction have a chance of approval?

Hi @soerenwolfers Can you elaborate on what you mean by vectorizing? If you have 1000 LHS (1000 A matrix) there is no point in vectorizing since solver will dominate the time, if you mean 1000 RHS then you can stack them in your b array (i.e. Ax = [b1, b2, b3...])
So it depends what you are vectorizing. I am slowly getting a PR together (#12824) such that these will be automated but there is an upper bound on what vectorization can achieve.

I did have 1000 LHS. In my case, they were tridiagonal O(10 x 10) matrices, solving a single system can be done by the simple O(n) Thomas algorithm (which could be numba-ized, but I didn't even do that), and writing that Thomas algorithm in a vectorized fashion for all 1000LHS was much faster than solving the 1000 systems one after another in a for loop.

You would get the same speed if you put a for loop around your Thomas algorithm. The point is that the for loop cost is negligible compared to the solving step with or without numba, cython etc. does not matter.
solve_banded is a generic solver not a tridiagonal solver and also has significant overhead due to f2py passing it to Fortran layer. That's what I am trying to remove in the mentioned PR above.

@ilayn "You would get the same speed if you put a for loop around your Thomas algorithm" -- not true: To solve 1e5 tridiagonal systems of size 10*10, a for loop around my own Thomas takes 4.2 seconds (1.7s with numba), whereas my vectorized Thomas takes  0.75s.
I didn't dispute that solve_banded does more than Thomas, but given the results above, I would still feel that it may equally profit from vectorization.
Independent of the question at hand, I think your PR is very important. Coming from MATLAB 10 years ago I was quite disappointed having to give up on the omnipotent backslash.

I don't think you can vectorize so that you get that much speed. Most probably you solved all systems in one go which is not vectorization quite vectorization but solving a very big system. Or at least I wouldn't call that vectorization.

I feel like we're getting side tracked, but I really don't know what distinction you are trying to make.
Let me be specific by sharing my code:
import numpy as np
import timeit

def solve_tridiagonal(A, y):
    """
    Solve Ax=y for x, where A is tridiagonal.

    :param A: Upper diagonal, diagonal, lower diagonal, in same format as solve_banded
    :param y: right hand side
    :return: x=A^{-1}y
    """
    y_ = y.copy()
    x = np.zeros_like(y)
    n = y.shape[0]
    a0, a1, a2 = A.copy()
    for i in range(1, n):
        w = a2[i-1] / a1[i-1]
        a1[i] -= w * a0[i]
        y_[i] -= w * y_[i-1]
    x[-1] = y_[-1] / a1[-1]
    for i in range(n-1)[::-1]:
        x[i] = (y_[i] - a0[i+1] * x[i + 1]) / a1[i]
    return x

if __name__ == '__main__':
    m = 10000
    n = 10
    bands = np.tile(
        np.array([
            np.zeros(n),
            np.ones(n),
            np.zeros(n),
        ])[..., None],
        [1, 1, m]
    )
    rhs = np.ones((n, m))

    tic0 = timeit.default_timer()
    x = solve_tridiagonal(bands, rhs)
    tic1 = timeit.default_timer()
    print(tic1 - tic0)

    x = np.zeros_like(rhs)
    for i in range(m):
        x[:, i] = solve_tridiagonal(bands[..., i], rhs[:, i])
    tic2 = timeit.default_timer()
    print(tic2 - tic1)

The function solve_diagonal is the type of function that I wish was more ubiquitous around scipy: It semantically operates over the first two axes of the input array A and on the first axis of the input array y. However, it allows arbitrarily many extra dimensions, and when it is given those, it applies the very same operations to each slice, thus solving multiple systems in a vectorized fashion, which Wikipedia defines as "a style of computer programming where operations are applied to whole arrays instead of individual elements".
The snippet above runs what I called the vectorized versions and non-vectorized versions before, where the former passes a 3 and 2 dimensional inputs to solve_diagonal (which is not possible for scipy's solve_banded) and the latter passes 2 and 1 dimensional inputs within a loop over the third axis (which I would have to do if I were to use scipy's solve_banded).
If you run this yourself you should be able to see the speedup I mentioned before (a bigger speedup even, because I stripped away some plumbing).

Would be nice to turn this into an example somehere. Linalg tutorial (a new DIY section) or a cookbook repository spring to mind.

Here is another thread for small determinants from NumPy to make my point a bit more obvious https://github.com/numpy/numpy/issues/17166 Same trick with simultaneous computations again.
Imagine you are solving generic 10x10 problems, 1000 of full 10x10 matrices. If there were no pivoting involved that would again be possible to "vectorize" the computations.
For example here another trick that performs almost as good as yours and involves nothing more than scipy.linalg, add this part to your set;
tic3 = timeit.default_timer()
la.solve_banded((1,1), bands.reshape(3,100000), rhs.reshape(100000, 1))
tic4 = timeit.default_timer()
print(tic4-tic3)

I get the following timings
0.004259000000047308
0.3971710000000712
0.004860500000177126

Stacked array solutions are very tricky to strike a nice balance between safety and speed. For example you need to check for all a1s in your case for zeros before division.

Well just realized that you have to arrange your sub- and sup- diags in the bands so there is no overlap but I hope that's an acceptable cost.

Were you surprised about the timings of my snippet then or not? I am still confused about your claim that "You would get the same speed if you put a for loop around your Thomas algorithm".
Your addition to my snippet, while fun and interesting, is based on exploiting a peculiarity of the particular problem that solve_banded solves and the way its arguments work, and doesn't address my question/request for vectorized functions because I don't think scipy's API is designed around magic tricks.
Your second point about ill conditioning and reporting is fair, though it doesn't seem to be undoable, and it certainly could be done in a way that leaves the non-vectorized experience untouched.

I wasn't that surprised because I was expecting something of that sort, though couldn't explain it well above, I guess.
If the problem allows, I am all in for tricks and treats. I just wanted to show another problem-specific way of doing it, what you are doing is also aligning the data such that all operations are "safe" and simultaneous hence essentially the same class as the numpy trick I linked to. That was what I tried to mean by the 10x10 full matrix case.
So if you feel like it and know how to do it with all fences about ill-conditionedness etc., please help us out. Because I don't see it right away. We would be more than happy to include it.

No further follow-up and it doesn't look like this is actionable, so let's close it. Thanks for the discussion all.

authors: ev-br ilayn rgommers soerenwolfers

repo: scipy/scipy


text:
Raspberry Pi 4 aarch64: ModuleNotFoundError: No module named 'numpy.random.bit_generator'

I'm using the NLU framework Rasa using TensorFlow which is relying on scipy. I have no issues running the code on a amd64 platform, but I'm using a Raspberry Pi 4 aarch64 architecture, where I hit the problem:

ModuleNotFoundError: No module named 'numpy.random.bit_generator'

I thought recompiling everything would fix this issue with:
pip3 install scipy==1.7.1 numpy==1.18.5 scikit-learn==0.24.2 --no-cache-dir --no-binary :all:

but I get the same error.
I cannot update at least numpy, because version 1.19 is introducing some breaking changes and TensorFlow does not work anymore.
Reproducing code example:
The problem can be also reproduced by executing in a Python 3 shell:
mkdir test
cd test
python3 -m venv venv
source venv/bin/activate
pip3 install scipy==1.7.1 numpy==1.18.5 scikit-learn==0.24.2
python3

Within the Python 3 shell execute:
import scipy.stats

Error message:

>>> import scipy.stats
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/accessibility/a/venv/lib/python3.8/site-packages/scipy/stats/__init__.py", line 441, in <module>
    from .stats import *
  File "/home/ubuntu/accessibility/a/venv/lib/python3.8/site-packages/scipy/stats/stats.py", line 43, in <module>
    from . import distributions
  File "/home/ubuntu/accessibility/a/venv/lib/python3.8/site-packages/scipy/stats/distributions.py", line 11, in <module>
    from . import _discrete_distns
  File "/home/ubuntu/accessibility/a/venv/lib/python3.8/site-packages/scipy/stats/_discrete_distns.py", line 19, in <module>
    from .biasedurn import (_PyFishersNCHypergeometric,
  File "biasedurn.pyx", line 1, in init scipy.stats.biasedurn
ModuleNotFoundError: No module named 'numpy.random.bit_generator'

Scipy/Numpy/Python version information:

1.7.1 1.18.5 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)

Unfortunately, numpy 1.18 renamed bit_generator to _bit_generator as part of an overall privatization of the modules in numpy.random. However, that particular module was intended to be public, so it was renamed back to bit_generator in 1.19. You can try downgrading to 1.17.

This helps at this end, but unfortunately brings up:
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

You may need to rebuild thinc with the version of numpy that you have installed.

Currently trying this (again), I think I tried this already, but then TensorFlow complaint, maybe I have to rebuilt this also (which is taking a day), but I'm wondering why on amd64 platforms these issues do not exist, there must be something special with the aarch64 wheels.

This does look like a bug in how we build the biasedurn module, it shouldn't try to use numpy.random.bit_generator for numpy < 1.1.9, see https://github.com/scipy/scipy/blob/master/scipy/stats/setup.py#L63.
Also, IIRC TensorFlow misdiagnosed the issue here, and should have changed their constraint. Indeed, downloading a TensorFlow 2.5.0 wheel I see:
Requires-Dist: numpy (~=1.19.2)

So any other NumPy version should work.

Also, IIRC TensorFlow misdiagnosed the issue here, and should have changed their constraint. Indeed, downloading a TensorFlow 2.5.0 wheel I see:
Requires-Dist: numpy (~=1.19.2)

So any other NumPy version should work.

Yeah, they even refuse to support 1.20 for now because they don't want to have >1.20 in their CI...

@tupui not sure what you mean, do you have a link? The Requires-Dist above says any numpy version is fine except 1.19.2. I don't see a <1.19 constraint anywhere.

@tupui not sure what you mean, do you have a link? The Requires-Dist above says any numpy version is fine except 1.19.2. I don't see a <1.19 constraint anywhere.

There are multiple threads:
https://github.com/tensorflow/tensorflow/issues/47691
https://github.com/tensorflow/tensorflow/pull/48918#issuecomment-868734334

What a mess. The attitude of the TensorFlow maintainers is that it's just not important for them. Both the bug and the unwillingness to fix it are 100% on their side.

What a mess. The attitude of the TensorFlow maintainers is that it's just not important for them. Both the bug and the unwillingness to fix it are 100% on their side.

Totally... Glad to read it's not just me 😅

I could create a workaround with a lot of compile time. All numpy modules have to be built from scratch. Important is --no-build-isolation, otherwise the buggy 1.18.5  version is used when building the wheel.
sudo apt-get install python3-dev python3-pip python3-venv python3-wheel libssl-dev
sudo apt-get install gcc gfortran python3-dev libopenblas-dev liblapack-dev
pip3 install wheel Cython
pip3 uninstall scipy spacy numpy blis thinc scikit-learn
pip3 install numpy==1.17.5 pybind11 pythran=0.9.11
pip3 install scipy==1.7.1 blis==0.7.4 scikit-learn==0.24.2 thinc==8.0.8 spacy==3.0.7 --no-build-isolation --no-binary :all:

I've looked into this, and the reason is this (from pyproject.toml):
    # numpy 1.19 was the first minor release to provide aarch64 wheels, but
    # wheels require fixes contained in numpy 1.19.2
    "numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64'",

This is what caused the original problem with numpy 1.18.5: the aarch64 wheel is built with 1.19.2, so not usable with older NumPy versions.
The from-source build is fine, I verified the generated Cython code. The second reported problem here is a misunderstanding by the user.
The things that could be done here are:

declare this a non-issue and close it
have a clearer error message, so that import scipy says something like "installed numpy version is too old"
have install_requires entries for numpy which are different in each wheel build. so that if we build against 1.19.2, it would say install_requires>=1.19.2,<1.24.0 rather than using the global numpy minimum version (= 1.17.5 here).

Either (2) or (3) would be more correct, but annoying in terms of build infra / code generation. There is no way to run into this problem unless one mixed a wheel build with a from-source build, so it's not high prio. I think it'd be nice to fix, but it's not blocking for 1.17.3.

Thinking about this again, the way to fix this is to add installed_numpy_version = x.y.z to _distributor_init at wheel build time and use that from the main __init__.py. Not too hard to implement, and should be robust.
Moving to the 1.8.0 milestone.

We got another report about this (gh-15209), so should prioritize this and try to backport the fix.

TF should now support numpy > 1.20 :) (TF maintainer here :P)

Thanks @mihaimaruseac!

@rgommers Should we consider closing this now? Now that it's fixed on TF side, I don't know if it's worth doing (2) or (3).

Good point Tirth. If one waits long enough, some problems fix themselves:)
In this case, let's close it when we drop numpy 1.18.x support, which we can do for the next release. No need to backport to 1.8.1 anymore I'd say.

Actually we stuck with numpy 1.18.5 for the 1.9.0 release, so bumping the milestone to 1.10.0.

Upgraded to 1.19.5 as minimum version now in main, so this will be fixed in 1.10.0. Nothing further to do, so I'll close it now.
If anyone still runs into this for 1.9.0: use numpy >=1.19.5 to build (use pip install scipy --no-build-isolation to avoid picking up numpy 1.18.x) if needed, or build from main.

authors: kaoh mihaimaruseac rgommers rkern tirthasheshpatel tupui


text:
BUG: minimize - COBYQA is very inefficient w.r.t time.

Describe your issue.
CI tests revealed that various COBYQA tests were taking a long time. Local testing reveals that there is a lot of overhead in COBYQA internals compared to the time taken to call the objective function. I believe that there are inefficiencies in the implementation that need to be discovered and reduced.
The example below shows that only 0.2% of time is spent in the objective function for COBYQA, whereas for nelder-mead it's 33%. COBYQA gets to the solution in a smaller number of nfev (nelder-mead doesn't actually get to a good minimum), but it takes two orders of magnitude more time.
@ragonneau
Reproducing Code Example
import numpy as np
from scipy.optimize import rosen, differential_evolution, minimize
bounds = [(0, 10)] * 10
rng = np.random.default_rng(1)
x0 = rng.random(size=10)*10

start = time.time()
res =  minimize(rosen, x0, method='cobyqa')
finish = time.time()
sftime = time.time()
_ = [rosen(res.x) for i in range(res.nfev)]
fftime = time.time()

print((fftime - sftime)/(finish - start), res.nfev, finish-start)

start = time.time()
res =  minimize(rosen, x0, method='nelder-mead')
finish = time.time()
sftime = time.time()
_ = [rosen(res.x) for i in range(res.nfev)]
fftime = time.time()

print((fftime - sftime)/(finish - start), res.nfev, finish-start)

### Error message

```shell
0.0019886230044629835 1087 2.6466002464294434
0.30101082386617767 1898 0.02931809425354004

SciPy/NumPy/Python version and system information
N/A

e.g. A long time is spent in Framework.get_best_index. There is a loop in that method that calls Framework.merit repeatedly. As part of merit the constraint violations are calculated.
In a subsequent line in the loop the max constraint value is calculated. This maxcv calculation reevaluates the constraint violations again. i.e. duplication of calculation.
Furthermore, it's not clear why those constraint violations even need to be calculated for a system where there are no constraints or bounds.

Most of this loop could be vectorised.

Vectorize this line

Norm accepts axis keyword. No need to loop there

Hi, I'm interested in helping out on this issue. Are you interested in PRs from anyone on this topic, or just the original author?

I've started a branch at https://github.com/andyfaff/cobyqa/tree/efficiency that's looking at various efficiency things. This is based on a jupyter notebook where I was line profiling things, https://gist.github.com/andyfaff/4cfdc554f9f520fb1a6a6f983025996f.
I'm yet to find the big prize.
It would be good to have the authors on board here, as the originators of the code they're familiar with it, and it's going to be easier for them. You're definitely more than welcome to contribute here, but it might be a slow slog through profiling country.

There are a lot of expensive scipy.linalg.eigh calls.
It's annoying that the code wraps optimize.LinearConstraint, optimize.NonLinearConstraint with problem.LinearConstraints and problem.NonLinearConstraints instead of using the original classes directly.
I suspect that there's a lot of linear algebra going on that doesn't need to be. e.g. if there are no constraints or bounds, then the algorithm should be designed to quickly bypass all calls to evaluation of those constraints, and the linear algebra involved with processing them to direct the minimisation.

There are a lot of expensive scipy.linalg.eigh calls.

I found it was possible to reduce the time spent in eigh() by 30% by passing drive="evd" to eigh().
E.g.
eig_values, eig_vectors = eigh(a, driver=driver, overwrite_a=overwrite, check_finite=False)

Unfortunately, it also increases nfev by 3%, which may not be worth it.
I also tried passing overwrite_a=True, but I did not find any speed-up from this.

This looks very helpful, I had a similar impression when I had a glance over COBYQA's source code. For the sake of repo hygiene, could we move the discussion to the COBYQA repo though? It must be fixed there anyway.

Please reassess the fail_slow marks (gh-20699, gh-20741) when this is resolved.

Hello @andyfaff and all,
First of all, thank you very much for bringing this issue up and for proposing improvements (see also https://github.com/cobyqa/cobyqa/pull/152). They are extremely important to us, and we cherish your suggestions and inputs.
Meanwhile, I would like to mention that COBYQA (PRIMA as well) is primarily designed for "derivative-free optimization" (DFO), aka "black-box optimization" or "simulation-based optimization". We assume that function evaluations predominate the cost. In practice, each function evaluation can take seconds, minutes, hours, or days, because it often involves sophisticated simulations.  In contrast, the costs incurred inside the solver (e.g., floating point operations, memory allocations) are assumed to be negligible.  That is why we commonly use the number of function evaluations to measure the performance of DFO solvers.
However, we (Tom @ragonneau and I) always keep in mind that our assumptions can be wrong, and users may use our DFO solvers to solve non-DFO problems. Therefore, we should not ignore the internal costs of our solvers. Prioritization is key:
Reducing function evaluations >> enhancing code simplicity & clarity >> economizing solver's internal costs.
Any improvement of the code that does not increase the number of function evaluations should be integrated, just like what we have done at https://github.com/cobyqa/cobyqa/pull/152 (credit goes to @andyfaff . Thank you!).
If you are interested in this topic, the following notes on DFO may be interesting to you.  You are invited to make comments there:
https://github.com/orgs/libprima/discussions/145
Many thanks again!
Cheers,
Zaikun

authors: andyfaff dschmitz89 ilayn mdhaber nickodell zaikunzhang


text:
BUG: ss2zpk goes through ill-conditioned transfer function; avoid polynomial coefficients for numerical stability

ss2zpk is implemented as
return tf2zpk(*ss2tf(A, B, C, D, input=input))

Please don't go through transfer-function form as it throws away numerical accuracy.
I am not sure of the best way to implement ss2zpk. The poles are easy: get the eigenvalues. Not sure about the zeros.
Related to the inverse function #5912.

Oops, duplicate of #3560

sample:
import scipy.signal
import numpy as np

N = 20
A = np.diag(1.0/np.array(range(1,N+1)))
B = np.array([[1] + [0]*(N-1)]).T
C = np.ones((1,N))
sys1 = scipy.signal.lti(A,B,C,0)
scipy.signal.ss2zpk(sys1.A, sys1.B, sys1.C, sys1.D)

which produces this on scipy 0.14 (alas, I can't upgrade at home until I get a new backup hard drive):
(array([ 0.50000000+0.j        ,  0.33333332+0.j        ,
         0.25000072+0.j        ,  0.19997778+0.j        ,
         0.16701636+0.j        ,  0.13738928+0.00391595j,
         0.13738928-0.00391595j,  0.11056321+0.01596029j,
         0.11056321-0.01596029j,  0.08817085+0.01811839j,
         0.08817085-0.01811839j,  0.07093331+0.01481982j,
         0.07093331-0.01481982j,  0.06509391+0.j        ,
         0.05857294+0.00949702j,  0.05857294-0.00949702j,
         0.05124722+0.00439428j,  0.05124722-0.00439428j,  0.04856395+0.j        ]),
 array([ 1.00000000+0.j        ,  0.50000000+0.j        ,
         0.33333334+0.j        ,  0.24999937+0.j        ,
         0.20001204+0.j        ,  0.16663189+0.j        ,
         0.14071380+0.j        ,  0.13422601+0.j        ,
         0.11325882+0.01707588j,  0.11325882-0.01707588j,
         0.09049564+0.02175423j,  0.09049564-0.02175423j,
         0.07234222+0.01944259j,  0.07234222-0.01944259j,
         0.05984464+0.01394214j,  0.05984464-0.01394214j,
         0.05217800+0.00783644j,  0.05217800-0.00783644j,
         0.04829228+0.00237204j,  0.04829228-0.00237204j]),
 0.99999999999999956)

The system should have all real poles, which is easily determined from np.linalg.eig(sys1.A)

hmmm... when in doubt, turn to the Dahlehs; they give some comments on determining poles and zeros from state-space.


text:
Disable deprecated Numpy API in __odrpack.c

This PR resolves DeprecationWarnings caused by usage of deprecated array members.

Thanks @Dormouse759. This looks good to me aside from the one define I commented on.

@Dormouse759 would you mind addressing the comment? Then I'll merge this PR.

@rgommers This fell way too deep in my todo list. Sorry for that. On it.

@rgommers Is there any way to reschedule the checks?
It seems like they are caused by a CI issue rather than the patch itself.

close/reopen triggers a CI restart. there's also buttons on the CI itself, but only for people with commit access. close/reopen will work for you on any repo

scipy/fft/_pocketfft/pypocketfft.cxx:15:31: fatal error: pybind11/pybind11.h: No such file or directory
#include <pybind11/pybind11.h>
This seems like there are some development files not present on the machine.

@Dormouse759 the CI configs have been updated since this PR was opened. Rebasing on master should fix the issue.

CI passed, so merging.


text:
Small test fixes for pypy + win + mmap

Came up in the context of https://github.com/conda-forge/scipy-feedstock/pull/180:
=========================== short test summary info ===========================
FAILED io/tests/test_netcdf.py::test_read_write_files - PermissionError: [Win...
FAILED io/tests/test_wavfile.py::test_write_roundtrip - OSError: [Errno 22] I...
= 2 failed, 32019 passed, 2572 skipped, 104 xfailed, 11 xpassed, 41 warnings in 2113.66s (0:35:13) =

@mattip suggested to follow the approach he chose in https://github.com/numpy/numpy/pull/16974, and now the test suite passes.
While I was at it, I decided to parametrize test_wavfile.test_write_roundtrip, because a) it's perfect for it b) I wanted it to be easier to reason about where to put the break_cycles().

Commit https://github.com/scipy/scipy/pull/14549/commits/18c521b9eb42dcaaa727b4c2be3e5ede2c2e0e2f is actually about half the size when ignoring whitespace changes.

Thanks. Looks good to me. At some point it might be nice to add a corresponding CI job for PyPy, but they seem to take twice the time as the CPython ones on windows, so I wouldn't want to slow things down even more than they already are.

A little tricky to follow the testing logic, but reviewers seem happy and CI is green.

Thanks @h-vetinari et al.

authors: h-vetinari mattip tylerjereddy


text:
Mark fitpack sources as recursive

Reference issue

Hopefully closes gh-11828, closes gh-14162
What does this implement/fix?

Mark FITPACK functions/subroutines recursive, as suggested in https://github.com/scipy/scipy/issues/11828#issuecomment-612529411
This change Fortran 90 only, so  it makes SciPy not F77 compliant. It probably was not already anyway, as we dropped F77 a while ago.
I'm also adding implicit none to all sources in the same patch. It's likely a bad form to lump two changes into the same patch, but since I've already had 80-odd files open, decided to do it anyway. The only non-trivial fix due to implicit none is in fpcons.f, which might need checking --- the first thing to check is if we actually use this file: not all of FITPACK is wrapped. But this is for some other day.
Additional information

Threading issues were reported on Windows in gh-11828 and gh-14162, which I cannot test at the moment, so testing on Windows would be great. The OP reports have reproducible code examples, so it's a matter of somebody having access to a windows machine. @thomasaarholt  @unknownUser123456 would be great of you could rerun your code samples.

@melissawm since you mentioned a sprint, I wonder if building scipy on windows might be some sprint material

Hmm, I'm not able to reproduce the issues on Windows in my development environment.
With either main or this feature branch checked out, I can run the example code in gh-14162 with Nthreads 1-10 and no output is generated. Running the example code in gh-11828, I get this plot in both branches (with no visible difference):

The interpolate test suite passes with -m full in both branches.

Wait, but in SciPy 1.7.1 in my base conda environment, I can reproduce both problems.

Thanks for checking Matt! So, something else changed between 1.7.1 and main. Is your dev env meson-based?

No I'm still not using meson. Could it be OpenBLAS vs MKL? I will try checking out 1.7.1 in my dev environment later to see if it's something in the environment rather than the code.

Could it be OpenBLAS vs MKL?

Yup, most likely.
TBH, I'm not sure how much effort needs to be spent digging. Various fortran compilers may do various things and in effect the current main branch is relying on implementation-defined behavior IIUC. Marking things as recursive is probably the language-layer thing to do. I'll ping @sturlamolden for he was discussing these things in the past and @dimpase for his interest in uncommon Fortran compilers

OK. I was unable to build maintenance/1.7.1 today due to difficult with the boost submodule. I tried
git checkout maintenance/1.7.x
git clean -dxf
git submodule update --init

but I think there was trouble with submodules. So I removed all the submodule directories that git status showed in red and did:
git submodule deinit -f .
rm -rf scipy/sparse/linalg/_propack/PROPACK/
rm -rf .git/modules/scipy/sparse/linalg/_propack/PROPACK/
git submodule update --init --recursive

but still:
(scipydev) C:\Users\mhaberla\Desktop\scipy>git submodule update --init
fatal: 'upstream' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: Fetched in submodule path 'scipy/_lib/boost', but it did not contain 2110ce20e59917b85b12059b3ddd8b133549f662. Direct fetching of that commit failed.
fatal:

Could it be OpenBLAS vs MKL?

Yup, most likely. TBH, I'm not sure how much effort needs to be spent digging. Various fortran compilers may do various things and in effect the current main branch is relying on implementation-defined behavior IIUC. Marking things as recursive is probably the language-layer thing to do. I'll ping @sturlamolden for he was discussing these things in the past and @dimpase for his interest in uncommon Fortran compilers

Fortran compilers are free to make local variables static, even when they are not declared with SAVE. They will do this depending on compiler, compiler version and compiler settings. The RECURSIVE qualifier prevents this from happening. In general, any Fortran function or subroutine not decalred RECURSIVE cannot be considered reentrant and thread-safe. RECURSIVE does not imply that the function or subroutine will be recursive, however, it just ensures that it can be reentrant.
Unfortunately, if a Fortran compiler can infere that a function is not recursive, it is also free to ignore the RECURSIVE qualifier. For example, if it sees that the function is only making calls to PURE, ELEMENTAL or INTRINSIC functions, or no calls at all, it can drop the RECURSIVE attribute. The only way to really ensure that a fortran function/subroutine is reentrant, is to confuse the compiler, e.g. make a call to an empty dummy function in another compile unit. That way it cannot know that the function will not recurse.
To make things even worse, many Fortran routines still in use were written 30-40-50 years ago, when hardware was a scarse resource. Many Fortran programmers did not trust the compiler to make the right choice regarding intrinsic SAVE on local variables. Thus they were in the habit of putting it in explicitly. We must therefore also read the code and decide which SAVE attributes can be deleted and which must be left inplace.
The long-term solution to this nightmare is of course to write C and never put more Fortran into existence.

To make things even worse, many Fortran routines still in use were written 30-40-50 years ago, when hardware was a scarse resource. Many Fortran programmers did not trust the compiler to make the right choice regarding intrinsic SAVE on local variables.

No - in fact, before Fortran 77 (or even before Fortran 90, where recursion became "officially" possible) recursion was just not allowed by the compiler, not that programmers didn't trust things. This shows  old Fortran code in a different light.

The long-term solution to this nightmare is of course to write C and never put more Fortran into existence.

yeah, let's rewrite stuff with C pointers, and spend another 30-50 years debugging memory leaks etc...

To make things even worse, many Fortran routines still in use were written 30-40-50 years ago, when hardware was a scarse resource. Many Fortran programmers did not trust the compiler to make the right choice regarding intrinsic SAVE on local variables.

No - in fact, before Fortran 77 (or even before Fortran 90, where recursion became "officially" possible) recursion was just not allowed by the compiler, not that programmers didn't trust things. This shows old Fortran code in a different light.

What we are discussing has nothing to do with recursion. It has to do with reentracy and thread-safety, due to what a Fortran compiler can do with local variables. The keyword is calles RECURSIVE, but this PR is using it for something else.

It has to do with reentracy and thread-safety

in 1966 (Fortran IV) none of these had any meaning, and there was no SAVE in Fortran prior to Fortran 77.

@langou Apologies for summoning you once again but you are de facto our Netlib repo contact :-)
It would be a great help to us if you know anython about the status is of FITPACK or DIERCKX from a maintenance perspective. We have no maintainer for legacy Fortran code and we are thinking about asking folks in fortran-lang for this. Hence our question whether we should indeed look for alternatives in case FITPACK is moving towards abandonware.

It has to do with reentracy and thread-safety

in 1966 (Fortran IV) none of these had any meaning, and there was no SAVE in Fortran prior to Fortran 77.

The legacy Fortran code we are using are Fortran 77 and lots of it have SAVE everywhere, both explicitely and implicitely. It causes tremendous headache for getting multithreading to work correctly, which is why we often just have to prevent it completely with the GIL.

The legacy Fortran code we are using are Fortran 77 and lots of it have SAVE everywhere

There is no explicit SAVE anywhere in fitpack, AFAICS

To repeat the request:
start every subroutine with the prefix "recursive" for https://www.netlib.org/fitpack/all and for http://www.netlib.org/dierckx/
Would that do the trick?

start every subroutine with the prefix "recursive" for https://www.netlib.org/fitpack/all and for http://www.netlib.org/dierckx/

While I cannot speak for Ilhan, I don't think this would be relevant to SciPy.
SciPy has been shipping essentially a fork of http://www.netlib.org/dierckx/ for a rather long time, in  https://github.com/scipy/scipy/tree/main/scipy/interpolate/fitpack (https://www.netlib.org/fitpack/all is something else ISTM)
The SciPy's version is different in that it has

real -> real*8 everywhere --- this was done some time in 1999-2003, I think
several bug fixes and enhancements on top (Here's the github history from 2007 onwards:   https://github.com/scipy/scipy/commits/main/scipy/interpolate/fitpack)
it may have some fortran 90 features, and may not build with g77 (did not check)
this PR proposes to add the recursive suffix and implicit none
there are several other tweaks in the pipeline (most notably, gh-10129)

So for SciPy, it's the version. Whether it makes sense to replace the netlib version by the scipy version and what's the status of the latter, I don't know and would be interested in hearing your opinion @langou.
Also your opinion on this patch itself --- adding recursive everywhere to avoid threading issues ---  would be much appreciated, if you would weigh in!

This will also autmatically deal with the 32-bit vs. 64-bit issue with Fortran.

Let's take a pragmatic view on this PR:

A possible use of  iso_c_binding is out of scope of this PR;
We will not be rewriting Dierckx / Fitpack code with real(kind=...), not in this PR anyway;
single-precision literals (1.0 instead of 1d0) should indeed be changed if found. Either push to this branch, or drop a commit somewhere where I can cherry-pick it or just open a PR;
A discussion about the future of Fortran 77 based code in general is bigger than this PR, so will start a separate thread later; can carry on discussing here until then of course;

So, let's focus on whether this PR fixes gh-11828, closes gh-14162 and on whether this PR brings any additional problems or complications. If the answer is "yes" and "no", then I suggest we merge it as is as a bugfix and look at a bigger picture afterwards. Thoughts?

I am fine with merging as is since there wasn't any important breaking cases brought up.

The mailing list is silent, and I rather strongly suspect this patch will only get tested in the beta/rc phase. I'm thus adding this PR to a 1.9.0 milestone with an intention to merge next week.  @tylerjereddy  please let me know if you're not OK with this testing-in-rc strategy (I'll be around for a quick revert if someone reports problems).

Locally Tested https://github.com/scipy/scipy/issues/11828 and https://github.com/scipy/scipy/issues/14162 on Windows 10: OK now with a freshly checked-out scipy build with msys2 ucrt64 GCC toolchain

authors: carlkl dimpase ev-br ilayn langou mdhaber sturlamolden


text:
ENH: added analytical formulas for entropies

Reference issue

What does this implement/fix?

Additional information

Adds analytical formulas for entropies (#17748)

About the implementation itself: many of these formulas first use log(exp(something)). I would suggest (see one of the suggestions) to simplify these expressions symbolically and then perform the calculation directly in logspace.

Good idea, done.

About tests: for a first test that the implementation is probably right, i would compare with the current results from the main branch using the numerical integration (see for example the plots in https://github.com/scipy/scipy/pull/17742 ).

Is this necessary? test_continuous_basic checks entropies, so between that and the high-precision testing, I think there's enough test coverage.

Then, for unit testing some comparison with a multiprecision library is needed. In https://github.com/scipy/scipy/pull/17758 , an example was given to compute the entropy integral using https://github.com/scipy/scipy/pull/17758#discussion_r1068346336.

Done. This is only tangential to the PR, but I noticed that the dgamma distribution doesn't have any tests. I'm beginning to add tests for a separate PR, but I just want to put that out there in case anybody was confused why the dgamma test suite only has the entropy test.

One last tip regarding the lint errors: they can be found locally by running python dev.py lint.

I ran that, and it said just about all of the lint failures were from code other than what I wrote. The scipy development workflow guide says to only focus on your own linting problems, but that wouldn't solve the other linting problems that are causing the lint check to fail.

About tests: for a first test that the implementation is probably right, i would compare with the current results from the main branch using the numerical integration (see for example the plots in #17742 ).

Is this necessary? test_continuous_basic checks entropies, so between that and the high-precision testing, I think there's enough test coverage.

The rationales behind high-precision testing based on numerical integration from the discussion were:

sometimes the analytical integral evaluations are questionable. In the case of the F distribution, the Wikipedia formula was plain wrong. If we just use the mpmath versions of the scipy implementation, we do not test this.
test_continuous_basic does not actually test the computed entropy values against a reference from what I see.

I think the code snippet for high precision integration could serve as a reference here too, just the pdf needs to be replaced.

could you maybe post screenshots of the formulas from the book you used also for comparison? Currently, the math can actually only be reviewed for truncnorm :)

Yes, but with the caveat that the definitions have errors, albeit ones that can be easily corrected. I didn't mention it, but the book shows how the entropy formulas are derived, which might make the need for numerical integration redundant.
https://imgur.com/a/t4S4y3z

which might make the need for numerical integration redundant

Scratch that. I've looked over the math for the entropy derivations a bit more closely, and I think the author messed it up. I'm going to add the numerical integration, as well as review the derivations, but that'll take me a bit of time (probably no longer than a week).

@MatteoRaso : would you consider opening separate PRs for the individual distributions when you have time to get back to this?
I don't intend to scratch your work here, but that would make review a lot easier. Some of these expressions can become quite involved, and as you also noticed, references are not always reliable. In case one of the distributions you are working on is difficult, it would hold up the rest. Not talking about the general review backlog ;).

@MatteoRaso : would you consider opening separate PRs for the individual distributions when you have time to get back to this?

Good idea. As you mentioned, I have been having problems with a few of the distributions, so I'll open PRs for the ones that I have done.


text:
BUG: Fix dtype for sum of linear operators

A mutable argument in scipy.sparse.interface._get_dtypes
causes the inferred dtype of a sum or product of operators
to be inferred incorrectly sometimes.
This fixes it by removing the mutable argument.
Fixes #6078

@@            master   #6083   diff @@
======================================
  Files          238     238
  Stmts        43863   43865     +2
  Branches      8218    8219     +1
  Methods          0       0
======================================
+ Hit          34293   34295     +2
  Partial       2603    2603
  Missed        6967    6967

Review entire Coverage Diff as of 18f643e

Powered by Codecov. Updated on successful CI builds.

authors: codecov-io simongibbons


text:
TST: skip a problematic linalg test

Cherry-pick the commit from @rgommers used in previous maintenance branches.
Fixes #8064.

In it goes. Thanks @tylerjereddy

authors: rgommers tylerjereddy


text:
BUG: qr_updates fails if u is exactly in span Q

This change (from < to <=) is in Riechel's paper and was missed when the code was first written.
closes gh-9733

I am also having similar problems with Jenkins at work. Overall google search tells me that people use these pipeline based tools a lot so clearly there is value in them but I have yet to get a straightforward function out of them similar to Travis or Circle CI. I am a bit loss at how to fix these failures.

Rebasing on latest master will fix Azure stuff

rebased.

Approving review from core dev, CI is green, and unit test appears to match the original issue report, so in it goes.

Thanks @ewmoore

authors: ewmoore ilayn tylerjereddy


text:
fix AnalysisHandler.download when ticket is None

Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

👍

Merging since we need the fix.

authors: ambrussimon-invenshure nagem

repo: scitran/core


text:
opening directories recursively

I added a -r option to open files inside a directory (recursively). This is inspired by the similar feature in the BASH version of rmate. It also contains the -n flag to open a new Sublime window.
More explanation here: https://github.com/randy3k/RemoteSubl/issues/19

I don't think I will merge this. Neither the description here nor the linked issue indicate what value this provides, and apparently it doesn't work well anyway. Why not just do something like find . -type f | xargs rmate (or some more specific find to avoid files that shouldn't be opened)?

Though if you wanted to make a PR specifically for adding support for opening a new window in Sublime I would be open to that. I just don't see value in the recursive option

My apologies for not providing more explanation. The idea was to open a complete project directory (with Sublime) as you would do locally, in order to keep working on the code for longer than just editing a single file. As I mentioned, this still falls short due to a few issues like preserving the folder structure. However, I think these problems are solvable so I wanted to leave my code here as a starting point. No need to merge them if you consider the feature incomplete.
I will create a separate PR for opening a new window.

My apologies for not providing more explanation. The idea was to open a complete project directory (with Sublime) as you would do locally, in order to keep working on the code for longer than just editing a single file. As I mentioned, this still falls short due to a few issues like preserving the folder structure. However, I think these problems are solvable so I wanted to leave my code here as a starting point. No need to merge them if you consider the feature incomplete.
I will create a separate PR for opening a new window.

authors: janwilamowski sclukey

repo: sclukey/rmate-python


text:
train on own dataset with fm_joint.cfg

I would like to train the mono_fm_joint model on my own dataset.
Should I revise the folder_dataset.py or the kitti_dataset.py (mono_dataset.py)?

you may feel more convenient to modify folder_dataset.py.

repo: sconlyshootery/FeatDepth


text:
snapshot: Create bundles without object deltas.

The bundles we create may not be ingestible by shallow clones, depending on the changes involved. This is an attempt to generate bundles that don't reference the history of included files and can therefore be successfully ingested by repos with limited history.

Codecov Report

Merging #285 into master will increase coverage by 0.03%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master     #285      +/-   ##
==========================================
+ Coverage   59.43%   59.46%   +0.03%
==========================================
  Files          90       90
  Lines        6971     6977       +6
==========================================
+ Hits         4143     4149       +6
  Misses       2523     2523
  Partials      305      305

Impacted Files
Coverage Δ

snapshot/git/gitdb/bundlestore.go
55.61% <85.71%> (+1.47%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca2f270...c67ed86. Read the comment docs.

repo: scootdev/scoot


text:
Properly indent, format and remove unused variables from server.js

Before you describe your issue, idea or feature, please consider using this issue template.
Type of issue
Improvement detail
Short description on the issue
The server.js in stackle_api has some unused variables. Also the code is not properly formatted and indented.
Possible fix
I have opened a PR #77 which fixes this issue. I have reformatted and indented the code. I have also removed the unused variables.

PR #77 has been merged

repo: scorelab/Stackle


text:
Change in Login UI

Before

Now

Below the Google login button the option for creating a new account was written wrongly. Now it is correct.

@Raj2367 can you complete changes on this PR ASAP ? Add a space between question mark and Create one link. Then squash all commits.

repo: scorelab/fact-Bounty


text:
Removing allowBackup attribute.

allowBackup attribute shouldn't be used in libraries due to build problems.

In my project I'm using allowBackup=true and I cannot build because the values are different.
Please check my pull request.
Cheers.

@scottyab hi, when can you update the library on maven? I need to implement it in my app by today :(

Not going to have chance to push to maven at the moment. But you could use
jitpack.io to reference the current commit in github.
On Fri, 2 Oct 2015 05:58 cre8ivejp notifications@github.com wrote:

@scottyab https://github.com/scottyab hi, when can you update the
library on maven? I need to implement it in my app by today :(
—
Reply to this email directly or view it on GitHub
https://github.com/scottyab/rootbeer/pull/4#issuecomment-144922054.

That's too bad.
But the way, the link on maven is wrong.
The correct is "compile('com.scottyab:rootbeer-lib:0.0.2')". Please update your README when you have time.

repo: scottyab/rootbeer


text:
Added H2 randomizer

Resolves #7

These are awesome! Can you blend the random text you have into the splashText variable in the updated scripts/js file?

repo: scrabill/how-many-days-until-halloween


text:
tried to add tests for validation.py

Is it ok that function _validate_pertain_list uses 'skip' field of language info and not 'pertain' field while making list of pertain tokens?

Codecov Report

Merging #318 into master will increase coverage by 1.83%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #318      +/-   ##
==========================================
+ Coverage   78.94%   80.78%   +1.83%
==========================================
  Files          34       34
  Lines        4018     4059      +41
==========================================
+ Hits         3172     3279     +107
+ Misses        846      780      -66

Impacted Files
Coverage Δ

tests/test_languages.py
99.47% <100%> (+0.14%)
:arrow_up:

dateparser/utils/__init__.py
89.38% <0%> (+3.53%)
:arrow_up:

dateparser/languages/validation.py
97.84% <0%> (+33.33%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f18ad29...3170fe7. Read the comment docs.

authors: codecov-io eszakharova

repo: scrapinghub/dateparser


text:
multiple regex

I analyse the \portia-master\slybot\slybot\extractors.py file, is it possible to use more than one regex pattern for a field

It should be. In the UI just drag another extractor to the field that you want to add additional extractors to.

authors: agramesh ruairif

repo: scrapinghub/portia


text:
How to install portia in Ubuntu 14.04

I follow the Documentation , and choose to install Docker. And I can run Portia using
docker run -i -t --rm -v <PROJECTS_FOLDER>:/app/data/projects:rw -p 9001:9001 scrapinghub/portia
But I can use it to design a spider but can't run it and can't find the Log to solve the problem . Also I am sure that the spider is OK because I ever run a spider in Scrapinghub.
And then I use Developing Portia using Docker to build a Portia project but when it goes to
docker build . -t portia
it shows

root@ubuntu:/home/wuhaonan/portia/portiaui# ember build
Built project successfully. Stored in "dist/".
root@ubuntu:/home/wuhaonan/portia/portiaui# docker build . -t portia
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/wuhaonan/portia/portiaui/Dockerfile: no such file or directory

And I can't find any solution in Stackoverflow, so how can I deal with it?

Please follow the steps given at the bottom in #736
This is for the docker approach, not using the develop using docker.

@hackrush01 Thank you for your help, but it doesn't work.

And the console shows

Internal Server Error: /api/projects/test/spiders/blog.csdn.net/schedule
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/rest_framework/viewsets.py", line 87, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/utils/decorators.py", line 185, in inner
return func(*args, **kwargs)
File "/app/portia_server/portia_api/resources/route.py", line 74, in dispatch
return super(JsonApiRoute, self).dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 466, in dispatch
response = self.handle_exception(exc)
File "/app/portia_server/portia_api/resources/route.py", line 77, in handle_exception
response = super(JsonApiRoute, self).handle_exception(exc)
File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 463, in dispatch
response = handler(request, *args, **kwargs)
File "/app/portia_server/portia_api/resources/spiders.py", line 82, in schedule
request = requests.post(settings.SCHEDULE_URL, data=schedule_data)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded with url: /schedule.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff264517a90>: Failed to establish a new connection: [Errno 111] Connection refused',))
[05/Mar/2017 13:19:22] "POST /api/projects/test/spiders/blog.csdn.net/schedule HTTP/1.0" 500 14774

Can you please give more detail as to which site you are trying to scrape and the exact step you followed up until now. Because the error log shows that connection to the site was refused. Maybe then we can zero in on the problem.

@hackrush01 I scrape my blog's article lists.
And this site is be accessable in portia, while I use
docker run -i -t --rm -v ~:/app/data/projects:rw -p 9001:9001 scrapinghub/portia
to run it.
I can create a project, spiders and examples in 127.0.0.1:9001. I can edit everything in the browser, but I can run the spider through it.

This is Portia, my spider and items.

Is the possible reason that I try to scrape the site in China?

I think it's a more general problem, I too am not able to scrape websites designed in languages other that English(such as French, Hindi etc).

@hackrush01 But I can scrape the websites with Chinese characters in Scrapinghub. And the console shows that there are something wrong about the connection.

@Mr-Phoebe For that to work you need to have scrapyd running so it can receive your spider and load it.

@ruairif Thank you for your help.
I installed scrapyd, but failed to run it.  It shows
Failed to load application: 'module' object has no attribute 'OP_NO_TLSv1_1'
So I try to update pyopenssl with the advice from here, but I can't do it.
Does it need a virtual environment?
Or how to deal with it.
I appologize for my naive and poor behavior. But I really need help.

You can install it in  a venv and install a more up to date openssl

Same problem
docker run --rm -it -v ~/data/portia:/app/data/projects:rw -p 9001:9001 scrapinghub/portia

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/viewsets.py", line 87, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/utils/decorators.py", line 185, in inner
    return func(*args, **kwargs)
  File "/app/portia_server/portia_api/resources/route.py", line 74, in dispatch
    return super(JsonApiRoute, self).dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 466, in dispatch
    response = self.handle_exception(exc)
  File "/app/portia_server/portia_api/resources/route.py", line 77, in handle_exception
    response = super(JsonApiRoute, self).handle_exception(exc)
  File "/usr/local/lib/python2.7/dist-packages/rest_framework/views.py", line 463, in dispatch
    response = handler(request, *args, **kwargs)
  File "/app/portia_server/portia_api/resources/spiders.py", line 82, in schedule
    request = requests.post(settings.SCHEDULE_URL, data=schedule_data)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 110, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded with url: /schedule.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f577c3ec850>: Failed to establish a new connection: [Errno 111] Connection refused',))

The issue still occurs on both Ubuntu 16 and Windows 10 x64 when running using docker:
docker run -i -t --rm -v /home/marcin/portia:/app/data/projects:rw -p 9001:9001 scrapinghub/portia

authors: Mr-Phoebe hackrush01 marrrcin ruairif wenerme


text:
Mirror the tree that the file was open from

Hi,
In vim I always have many different trees and when I open another file from a the tree I always have to choose which tree to mirror, but all I really need is to mirror the tree that I am opening the file from.
Is there a way of doing this?
If someone could guide me in the right direction I might take a stab at implementing it.
Thanks

Im not sure exactly what you mean, but this sounds like what you are after https://github.com/jistr/vim-nerdtree-tabs

authors: TomasM scrooloose

repo: scrooloose/nerdtree

license: WTFPL


text:
Cannot open file or directory whose name include [ or ]

MacOS 10.12.3
Vim 8.0
scrooloose/nerdtree installed via vundle

Confirmed this bug on my installations.

I believe this is only a problem when the file contains both brackets, with [ followed by ]. I'm not even seeing the file in the NERDTree. #391 may be the same issue.

Hey, @PhilRunninger! I've isolated the bug to the s:UI._stripMarkup() in ui.vim. It was pretty clear that files like this must be treated like flags, so I looked there. I'm fairly confident that line 389 in ui.vim is the source of our problem.
I'm not a regular expression wizard yet. Maybe you can take a look.

It looks like line 389 is used to remove the flag that git-nerdtree-plugin puts on, but if foo[bar].txt is not a git-controlled file, it removes the wrong thing. If we force substitute() to look at the beginning of the word only, then we run into problems with [foo]bar.txt. My initial thought was to use filereadable(), and skip the substitute if it's not needed.

That actually sounds like a good decision for now. It seems more like a temporary fix, because other flags can exist that cause filereadable() to fail. We might need a more "heavyweight" solution for a long-term fix.

I've been hit with this bug and even though I agree that the problem requires a significant change to be rock-solid, but I've found a minimal change that works under the (weak) assumption that flags generally use special characters.
Here's the patch, for those interested:
diff --git a/lib/nerdtree/ui.vim b/lib/nerdtree/ui.vim
index 196d745..07a87e6 100644
--- a/lib/nerdtree/ui.vim
+++ b/lib/nerdtree/ui.vim
@@ -381,7 +381,7 @@ function! s:UI._stripMarkup(line)
     let line = substitute (line, '*\ze\($\| \)', "","")

     " strip off any generic flags
-    let line = substitute (line, '\[[^]]*\]', "","")
+    let line = substitute (line, '\[\W*\]', "","")

     let line = substitute (line,' -> .*',"","") " remove link to

Should you be interested in a PR, I'm just a ping away :)

@lifecrisis, I'm thinking about a fix for this issue. What if we used a non-breaking space (ASCII code 160) to delimit the filename from the flags and bookmark label? That character is very unlikely to be used in a filename, and parsing the node's text would be a lot easier. I'll work on it, and throw a PR out for review.

@faaaar @xcambar , would you be willing to test the pull request #868 I've put together for this issue? Details are in the PR. Thanks.

I confirm #868 fixes the issue. Congrats and thanks!

#868 fixes the issue. thanks!

Thanks for confirming this for me. I'll close the issue again when I do the merge.

authors: PhilRunninger faaaar lifecrisis xcambar


text:
Update exit codes for drafter

drafter may return underlying errors from snowcrash it's internal library for parsing for various reasons.
I've updated the list in syntastic to include any blueprint error.
List of exit codes can be found at https://github.com/apiaryio/snowcrash/blob/master/src/SourceAnnotation.h#L100-L106. I've intentionally left out ApplicationError (1) as this would be an application error and not a blueprint error.
Thanks :octocat:.

Leaving out anything but 0 and 2 is intentional.  Syntastic doesn't produce any useful output for ModelError and MSONError, and you want to know there's a problem.

I don't think that's correct @lcd047.
The syntastic checker already prints all warnings and errors, that would include model errors and mson errors.
For example, see my attachment:

With these changes, syntastic is capable of showing these errors:

Without them it will error incorrectly:

Then let's put it this way: at the time I looked at the sources to make sure the errorformat works, I only looked at return code 2.  IIRC return codes 3 and 4 can produce errors in other formats, and they would be silently discarded if syntastic accepted 3 and 4 as valid.  Back then, the checker wasn't even named drafter.
Now, if you look at the sources and guarantee drafter returns errors in the same format for exit codes 3 and 4 as it does for 2, I'll take your word for it.  But somebody has to check that, and I'm afraid I don't have time to do it myself right now. shrug

Now, if you look at the sources and guarantee drafter returns errors in the same format for exit codes 3 and 4 as it does for 2, I'll take your word for it.

I am Pavan. I work at Apiary which involves developing the API Blueprint parser. I am the guy who added those error codes. Yes, they are the same format as the error code 2. @kylef is my colleague.

Ok then, merged, thank you.

authors: kylef lcd047 pksunkara

repo: scrooloose/syntastic


text:
Test no_std for schnorrkel

Should be fixed now, but needs testing

@brenzi can this be closed?

repo: scs/substraTEE-worker


text:
Different picture folders visbile in app (not only camera roll)

Lespas offers a nice way of organizing/selecting and backup your photos.
The current problem for me is that I can only choose photos from camera roll, but not photos which were send via another app or image I saved. The reason seems to be that those pictures are in a different folder than camera roll.
In the app itself only camera roll seems easily visible, other folders not.
It would be nice if it was possible to have all folders that include pictures/videos at the homescreen of the app. Then it can be used as gallery and also backup app.
Introducing 2 apps is always difficult to explain to the wife and adds a complexity factor ;)

Les Pas started as a mobile display and a sharing vehicle for my own photographic works, I didn't pay much attention to management of phone's images, since I'm not a fan of mobile photography and don't like the idea of rebuilding the wheel of a file manager. However I start to realize that it would be better if Les Pas acts more like a general purpose picture manager. I will put this in my todo list. But it won't happen soon.

My current work around is folder sync Pro. Here I set up a local to local folder synchronization and then I use the "move instead of copy" function to move all pictures from the source folder to the camera roll folder. Thus, I basically misuse this app to simply move all pictures from other folders to the camera roll folder. My family members are quite happy with this functionality.
Consequently,  there workflow is to simply sort all photos from the camera roll album to the new desired albums via Les Pas since all photos on the phone are always in the camera roll folder.
The folder sync app works in the background. You can also enable instant sync to immediately move files to the camera roll folder. So no interaction from the user required here.
But I would also appreciate the requested feature ones there is time to implement it :)

A revamp of device media managment is brewing in the lab, will replace the current half-heartedly camera roll feature. It will be more Google Photo feel plus folder sync type of backup setting for each folder you chose. Still need another week or two.

authors: flutschepfeil frederikb96 scubajeff

repo: scubajeff/lespas


text:
"ValueError: value.index does not match parent’s axis 0 names" error when trying to read h5ad that was processed through scVI

Hi !
I am new to the Scanpy/Anndata ecosystem, and trying to use scVI for data integration.
I trained a scVI model on an Anndata object, saved the object as h5ad, and now I get an error when trying to read the file.
The error occurs when using scanpy.read_h5ad or scvi.data.read_h5ad, and I found someone else reporting the error in this repo before , so I am posting the issue here. Kindly let me know if I should post this somewhere else.
adipo_all = scvi.data.read_h5ad("/home/yyyyy/analysis/anndata_working/adipo_sn_01112021_trained_v1.h5ad")

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/anndata.py:120: ImplicitModificationWarning: Transforming to str index.
  warnings.warn("Transforming to str index.", ImplicitModificationWarning)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/scratch/33545574/ipykernel_5065/2292619582.py in <module>
----> 1 adipo_all = scvi.data.read_h5ad("/home/yyyyy/analysis/anndata_working/adipo_sn_01112021_trained_v1.h5ad")

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_io/h5ad.py in read_h5ad(filename, backed, as_sparse, as_sparse_fmt, chunk_size)
    435     _clean_uns(d)  # backwards compat
    436
--> 437     return AnnData(**d)
    438
    439

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/anndata.py in __init__(self, X, obs, var, uns, obsm, varm, layers, raw, dtype, shape, filename, filemode, asview, obsp, varp, oidx, vidx)
    320                 varp=varp,
    321                 filename=filename,
--> 322                 filemode=filemode,
    323             )
    324

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/anndata.py in _init_as_actual(self, X, obs, var, uns, obsm, varm, varp, obsp, raw, layers, dtype, shape, filename, filemode)
    508
    509         # TODO: Think about consequences of making obsm a group in hdf
--> 510         self._obsm = AxisArrays(self, 0, vals=convert_to_dict(obsm))
    511         self._varm = AxisArrays(self, 1, vals=convert_to_dict(varm))
    512

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/aligned_mapping.py in __init__(self, parent, axis, vals)
    233         self._data = dict()
    234         if vals is not None:
--> 235             self.update(vals)
    236
    237

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/_collections_abc.py in update(*args, **kwds)
   Supraclavicular 839             if isinstance(other, Mapping):
    840                 for key in other:
--> 841                     self[key] = other[key]
    842             elif hasattr(other, "keys"):
    843                 for key in other.keys():

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/aligned_mapping.py in __setitem__(self, key, value)
    149
    150     def __setitem__(self, key: str, value: V):
--> 151         value = self._validate_value(value, key)
    152         self._data[key] = value
    153

/home/xxxxx/pyenv_custom/py_scanalysis_env/lib/python3.7/site-packages/anndata/_core/aligned_mapping.py in _validate_value(self, val, key)
    211             # Could probably also re-order index if it’s contained
    212             raise ValueError(
--> 213                 f"value.index does not match parent’s axis {self.axes[0]} names"
    214             )
    215         return super()._validate_value(val, key)

ValueError: value.index does not match parent’s axis 0 names

This is the offending anndata object.
AnnData object with n_obs × n_vars = 123472 × 36795
    obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_spliced', 'nFeature_spliced', 'nCount_unspliced', 'nFeature_unspliced', 'nCount_HTO', 'nFeature_HTO', 'HTO_maxID', 'HTO_secondID', 'HTO_margin', 'HTO_classification', 'HTO_classification.global', 'hash.ID', 'nCount_SCT', 'nFeature_SCT', 'batch', 'n_genes', 'annot', 'sample_origin', 'day', 'depot', 'tissue', 'timepoint', 'n_genes_by_counts', 'total_counts', 'total_counts_percent_ribo', 'pct_counts_percent_ribo', '_scvi_batch', '_scvi_labels'
    var: 'features', 'spliced_features', 'unspliced_features', 'n_cells', 'highly_variable', 'highly_variable_rank', 'means', 'variances', 'variances_norm', 'highly_variable_nbatches', 'percent_ribo', 'n_cells_by_counts', 'mean_counts', 'pct_dropout_by_counts', 'total_counts'
    uns: 'hvg', 'sample_origin_colors', 'depot_colors', '_scvi'
    obsm: '_scvi_extra_categoricals', '_scvi_extra_continuous', 'X_scVI'
    layers: 'counts', 'spliced', 'unspliced'

Whereas the previous version below can be read just fine. scanpy.pp.highly_variable_genes and sc.pp.calculate_qc_metrics were run followed by model training in scVI to get the above anndata.
AnnData object with n_obs × n_vars = 123472 × 36795
    obs: 'orig.ident', 'nCount_RNA', 'nFeature_RNA', 'nCount_spliced', 'nFeature_spliced', 'nCount_unspliced', 'nFeature_unspliced', 'nCount_HTO', 'nFeature_HTO', 'HTO_maxID', 'HTO_secondID', 'HTO_margin', 'HTO_classification', 'HTO_classification.global', 'hash.ID', 'nCount_SCT', 'nFeature_SCT', 'batch', 'n_genes', 'annot', 'sample_origin', 'day', 'depot', 'tissue', 'timepoint'
    var: 'features', 'spliced_features', 'unspliced_features', 'n_cells'
    layers: 'counts', 'spliced', 'unspliced'

Libraries: scanpy==1.8.1 anndata==0.7.6 umap==0.5.1 numpy==1.20.3 scipy==1.7.1 pandas==1.3.3 scikit-learn==0.24.2 statsmodels==0.13.0rc0 python-igraph==0.9.6 pynndescent==0.5.4

I had a similar error and doing the following resolved the error. Make sure all 'obsm' objects that are dataframes have the same index as the 'obs' dataframe.

Seems like this is solved, and the discussion happens in #311
Please tell us if you need anything

authors: cinaljess flying-sheep nagendraKU

repo: scverse/anndata


text:
anndata write or anndata read seem to be losing uns["log1p"]["base"] when value for key base is None

I have noticed that on Scanpy, when setting andata.uns["log1p"]["base"] = None and then the object is written to disk and then read again, then base is no longer a key in andata.uns["log1p"]. This has implications in a number of downstream Scanpy methods when writing to disk in the middle and then reading back again, as maybe parts of scanpy seek to do:
if 'log1p' in adata.uns_keys() and adata.uns['log1p']['base'] is not None:

in various places. Maybe the underlying problem is that sc.pp.log1p(adata) is not marking base as math.e in uns. I wonder if the write/read process might be prunning other keys that have None values?

Notice this problem also and agree to have this feature to avoid one more explicit step to avoid errors of detecting log1p but no base in it.

I'm still affected by this issue. Without a setting for base, rank_genes_groups will not work.
Steps to reproduce:
import scanpy as sc

adata = sc.datasets.blobs()
sc.pp.log1p(adata)
adata.uns['log1p']

Output:
{'base': None}

Now, writing adata to file and reading from file:
adata.write('adata.h5ad')
adata = sc.read('adata.h5ad')
adata.uns['log1p']

Output:
{}

So, the entry for base is gone. Now,
sc.tl.rank_genes_groups(adata, 'blobs')

throws this error:
WARNING: Default of the method has been changed to 't-test' from 't-test_overestim_var'

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[16], line 1
----> 1 sc.tl.rank_genes_groups(adata, 'blobs')

File /opt/conda/lib/python3.9/site-packages/scanpy/tools/_rank_genes_groups.py:590, in rank_genes_groups(adata, groupby, use_raw, groups, reference, n_genes, rankby_abs, pts, key_added, copy, method, corr_method, tie_correct, layer, **kwds)
    580 adata.uns[key_added] = {}
    581 adata.uns[key_added]['params'] = dict(
    582     groupby=groupby,
    583     reference=reference,
   (...)
    587     corr_method=corr_method,
    588 )
--> 590 test_obj = _RankGenes(adata, groups_order, groupby, reference, use_raw, layer, pts)
    592 if check_nonnegative_integers(test_obj.X) and method != 'logreg':
    593     logg.warning(
    594         "It seems you use rank_genes_groups on the raw count data. "
    595         "Please logarithmize your data before calling rank_genes_groups."
    596     )

File /opt/conda/lib/python3.9/site-packages/scanpy/tools/_rank_genes_groups.py:93, in _RankGenes.__init__(self, adata, groups, groupby, reference, use_raw, layer, comp_pts)
     82 def __init__(
     83     self,
     84     adata,
   (...)
     90     comp_pts=False,
     91 ):
---> 93     if 'log1p' in adata.uns_keys() and adata.uns['log1p']['base'] is not None:
     94         self.expm1_func = lambda x: np.expm1(x * np.log(adata.uns['log1p']['base']))
     95     else:

KeyError: 'base'

Output of sc.logging.print_versions():
-----
anndata     0.9.1
scanpy      1.9.3
-----
PIL                 9.2.0
anyio               NA
arrow               1.2.3
asciitree           NA
asttokens           NA
astunparse          1.6.3
attr                23.1.0
babel               2.12.1
backcall            0.2.0
bottleneck          1.3.7
brotli              NA
certifi             2023.05.07
cffi                1.15.1
chardet             5.1.0
charset_normalizer  2.1.1
cloudpickle         2.2.1
colorama            0.4.6
comm                0.1.3
cycler              0.10.0
cython_runtime      NA
cytoolz             0.12.0
dask                2023.4.1
dateutil            2.8.2
debugpy             1.6.7
decorator           5.1.1
defusedxml          0.7.1
dill                0.3.6
entrypoints         0.4
executing           1.2.0
fasteners           0.17.3
fastjsonschema      NA
fqdn                NA
gmpy2               2.1.2
google              NA
h5py                3.8.0
idna                3.4
igraph              0.10.4
importlib_resources NA
ipykernel           6.23.0
ipython_genutils    0.2.0
isoduration         NA
jedi                0.18.2
jinja2              3.0.3
joblib              1.2.0
json5               NA
jsonpointer         2.0
jsonschema          4.17.3
jupyter_events      0.6.3
jupyter_server      2.5.0
jupyterlab_server   2.22.1
kiwisolver          1.4.4
leidenalg           0.9.1
llvmlite            0.39.1
louvain             0.8.0
lz4                 4.3.2
markupsafe          2.1.2
matplotlib          3.7.1
mpl_toolkits        NA
mpmath              1.3.0
msgpack             1.0.5
natsort             8.3.1
nbformat            5.8.0
numba               0.56.4
numcodecs           0.11.0
numexpr             2.7.3
numpy               1.23.5
opt_einsum          v3.3.0
packaging           23.1
pandas              1.5.0
parso               0.8.3
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
platformdirs        3.5.0
plotly              5.14.1
prometheus_client   NA
prompt_toolkit      3.0.38
psutil              5.9.5
ptyprocess          0.7.0
pure_eval           0.2.2
pvectorc            NA
pyarrow             10.0.1
pydev_ipython       NA
pydevconsole        NA
pydevd              2.9.5
pydevd_file_utils   NA
pydevd_plugins      NA
pydevd_tracing      NA
pygments            2.15.1
pyparsing           3.0.9
pyrsistent          NA
pythonjsonlogger    NA
pytz                2023.3
requests            2.29.0
rfc3339_validator   0.1.4
rfc3986_validator   0.1.1
scipy               1.10.1
send2trash          NA
session_info        1.0.0
setuptools          67.7.2
six                 1.16.0
sklearn             1.2.2
sniffio             1.3.0
socks               1.7.1
sparse              0.14.0
sphinxcontrib       NA
stack_data          0.6.2
sympy               1.11.1
tblib               1.7.0
texttable           1.6.7
threadpoolctl       3.1.0
tlz                 0.12.0
toolz               0.12.0
torch               2.0.0
tornado             6.3
tqdm                4.65.0
traitlets           5.9.0
typing_extensions   NA
unicodedata2        NA
uri_template        NA
urllib3             1.26.15
wcwidth             0.2.6
webcolors           1.13
websocket           1.5.1
yaml                5.4.1
zarr                2.14.2
zipp                NA
zmq                 25.0.2
zoneinfo            NA
zope                NA
-----
IPython             8.13.2
jupyter_client      8.2.0
jupyter_core        5.3.0
jupyterlab          3.6.3
notebook            6.5.4
-----
Python 3.9.15 | packaged by conda-forge | (main, Nov 22 2022, 08:45:29) [GCC 10.4.0]
Linux-5.15.0-71-generic-x86_64-with-glibc2.31
-----
Session information updated at 2023-06-14 10:31

Duplicate of #673
PR exists: #999

Wait, so why won't this be fixed?

That’s not what that means. Hover your mouse over “not planned” and you’ll see:

authors: alam-shahul brainfo fbnrst flying-sheep pcm32


text:
Cannot plot continuous var values to circles

Unfortunately it seems that also circles have a problem with var (probably fixing this would also solve the problems with labels).
I used this code to prepare and plot a variation of the blobs dataset so that the table annotates the circles.
import spatialdata as sd
from spatialdata.models import TableModel
from spatialdata.datasets import blobs
import spatialdata_plot
from napari_spatialdata import Interactive
import numpy as np

sdata = blobs()

# let's fetch the table metadata
region = sdata.table.uns[TableModel.ATTRS_KEY]['region']
region_key = sdata.table.uns[TableModel.ATTRS_KEY]['region_key']
instance_key = sdata.table.uns[TableModel.ATTRS_KEY]['instance_key']

# let's see what the table annotates
print(region)

# it annotates the labels; let's make the table annotate the circles instead
new_table = sdata.table.copy()
sdata.table.obs[region_key] = 'blobs_circles'

# there were 21 labels, but now there are only 5 circles, we need to subset
new_table = sdata.table[:5]
new_table.uns[TableModel.ATTRS_KEY]['region'] = 'blobs_circles'

# double checking that the indices match
print(new_table.obs[instance_key].to_numpy())
print(sdata['blobs_circles'].index.to_numpy())

# they don't, let's fix this
new_table.obs[instance_key] = np.array(range(5))

# the new table is ready, let's replace the old one
del sdata.table
sdata.table = new_table

# ready for viz
sdata.pl.render_shapes('blobs_circles', key='channel_0_sum').pl.show()

# Interactive(sdata)

I get this result

While the one produced by napari_spaitaldata is the following (note, the sizes of the circles are wrong in napari_spatialdata, we are waiting for a fix upstream in napari).

My mistake, I had to use color instead of key. The code correctly plots the data.

repo: scverse/spatialdata-plot


text:
packer: share AMIs with the entire aws scylla organization

With this change, AMIs wil be shared with the entire aws scylla org (instead of specific accounts) to allow all accounts to use the AMI before it is promoted + avoid the need to maintain a list of accounts that require AMI access,
This is about the AMI, on another change on pkg we need to adjust the change_snapshot_permissions() function to loop through all the org account
Ref https://github.com/scylladb/scylla-pkg/issues/3529

Verified with https://jenkins.scylladb.com/job/scylla-master/job/releng-testing/job/next-machine-image/293/

AFAIK the cloud needs those images on the enterprise branch for testing, I don't see a reason to backport this.

Yes, and they probably need it for future releases only but since it's a minor trivial change we'll backport it

AFAIK the cloud needs those images on the enterprise branch for testing, I don't see a reason to backport this.

Yes, and they probably need it for future releases only but since it's a minor trivial change we'll backport it

We should backport if it's needed or if it's improved something (even if it's a minor and there is no risk).  Let's verify before if it's needed for releases. Thanks

AFAIK the cloud needs those images on the enterprise branch for testing, I don't see a reason to backport this.

Yes, and they probably need it for future releases only but since it's a minor trivial change we'll backport it

We should backport if it's needed or if it's improved something (even if it's a minor and there is no risk). Let's verify before if it's needed for releases. Thanks

Yes it's an improvement and I'm not sure what verification you're looking for here,
In any case, let's please avoid a long discussion about something so minor that at the end will effect/help nothing

AFAIK the cloud needs those images on the enterprise branch for testing, I don't see a reason to backport this.

Yes, and they probably need it for future releases only but since it's a minor trivial change we'll backport it

We should backport if it's needed or if it's improved something (even if it's a minor and there is no risk). Let's verify before if it's needed for releases. Thanks

Yes it's an improvement and I'm not sure what verification you're looking for here, In any case, let's please avoid a long discussion about something so minor that at the end will effect/help nothing

My point exactly to will help nothing :-)

authors: benipeled yaronkaikov

repo: scylladb/scylla-machine-image