text:
Using scikit-fem on pip: solver remark

Although this is tangentially related to scikit-fem since the primary objective of scikit-fem is assembly and not solving the resulting systems, it maybe important to note that when working with a default pip installation on Windows, there is a significant performance degradation in solving, since the default sparse solver is SuperLU which is super slow. I think the conda installation may not suffer from this since the default solver is UMFPACK (?) which can be faster. There maybe further improvements possible when numpy is compiled against an optimized BLAS but that is something most users wouldn't care for.
I managed to get Pardiso working on Windows and with pypardiso things are significantly faster. See the timings on example 36 with nthreads=8 (and Mesh.refined(3)) and a slight addition at https://github.com/kinnala/scikit-fem/blob/c07ff9f6d6a51e62eac2777351b20b4549027ada/skfem/utils.py#L102
namely simply calling pypardiso.spsolve
def solver_direct_pypardiso(**kwargs) -> LinearSolver:
    """Linear solver provided by Pardiso."""

    def solver(A, b):
        return pypardiso.spsolve(A, b)

    return solver

Timings
Pardiso
Solve time: 0.3992440700531006
1, norm_du: 45.481410024640475, norm_dp: 14.934126744945504
Solve time: 0.31940722465515137
2, norm_du: 16.15926762709032, norm_dp: 20.93859584370273
Solve time: 0.3229987621307373
3, norm_du: 3.2471798205287667, norm_dp: 15.053533831714226
Solve time: 0.35599684715270996
4, norm_du: 0.4538801110428914, norm_dp: 2.6708267761548887
Solve time: 0.3192098140716553
5, norm_du: 0.027865264212750027, norm_dp: 0.1630714233226417
Solve time: 0.328704833984375
6, norm_du: 0.002041324778791658, norm_dp: 0.016283691403898768
Solve time: 0.3122262954711914
7, norm_du: 1.6033937733037654e-05, norm_dp: 0.0003318753774227752
Solve time: 0.33296942710876465
8, norm_du: 1.6887613490335165e-09, norm_dp: 4.315368227582554e-08
Solve time: 0.35301685333251953
9, norm_du: 4.4086266413504256e-15, norm_dp: 1.7099154835473912e-14
Total time: 98.29319977760315

SuperLU
Solve time: 19.864436626434326
1, norm_du: 45.48141002464045, norm_dp: 14.934126744945653
Solve time: 17.160115242004395
2, norm_du: 16.159267627090234, norm_dp: 20.93859584370295
Solve time: 18.916175842285156
3, norm_du: 3.247179820528735, norm_dp: 15.053533831714496
Solve time: 17.143075704574585
4, norm_du: 0.45388011104290815, norm_dp: 2.670826776154936
Solve time: 17.078906536102295
5, norm_du: 0.027865264212752802, norm_dp: 0.16307142332264873
Solve time: 17.101752281188965
6, norm_du: 0.0020413247787918446, norm_dp: 0.016283691403900905
Solve time: 17.19295907020569
7, norm_du: 1.6033937733110564e-05, norm_dp: 0.00033187537742298715
Solve time: 17.220698833465576
8, norm_du: 1.688761318114545e-09, norm_dp: 4.3153681931898745e-08
Solve time: 16.51562809944153
9, norm_du: 4.3255823949057516e-15, norm_dp: 1.5848762808515266e-14
Total time: 251.57028770446777

Just as a sidenote, why does conda-forge still have version 2.4.0?

I believe it's caused by this line: https://github.com/conda-forge/scikit-fem-feedstock/blob/master/recipe/meta.yaml#L39
skfem.importers was renamed to skfem.io.

@gdmcbain weren't you using Windows at some point?

Yes, I use Ubuntu 20.04.2 LTS, Pop!_OS 21.04, and MS-Windows 10 regularly, all with Python installed from Miniconda and scikit-fem then pip-installed from master or other branch of this repo or sometimes PyPI.
I do have the anecdotal finding that when I rewrote the unsteady Navier–Stokes tutorial example from FEniCS in scikit-fem in gdmcbain/fenics-tuto-in-skfem#5, the timings were roughly: FEniCS Linux half an hour, scikit-fem Linux twelve minutes, scikit-fem MS-Windows five minutes, but the Windows and Linux were different machines, so really not a very useful comparison.
I find that I can conda install -c haasad pypardiso and then import spsolve from pypardiso on Pop!_OS too, so I'll be giving that a go.  Thanks, Bhavesh!

Trying to fix conda-forge in a PR: https://github.com/conda-forge/scikit-fem-feedstock/pull/17

Perfect!! Thanks! I have been meaning to switch to conda for quite sometime, but have resisted so far since another package netgen that I use for mesh generation wasn't available with anaconda.

Yes, I use Ubuntu 20.04.2 LTS, Pop!_OS 21.04, and MS-Windows 10 regularly, all with Python installed from Miniconda and scikit-fem then pip-installed from master or other branch of this repo or sometimes PyPI.
I do have the anecdotal finding that when I rewrote the unsteady Navier–Stokes tutorial example from FEniCS in scikit-fem in gdmcbain/fenics-tuto-in-skfem#5, the timings were roughly: FEniCS Linux half an hour, scikit-fem Linux twelve minutes, scikit-fem MS-Windows five minutes, but the Windows and Linux were different machines, so really not a very useful comparison.
I find that I can conda install -c haasad pypardiso and then import spsolve from pypardiso on Pop!_OS too, so I'll be giving that a go. Thanks, Bhavesh!

Thanks, I think conda uses UMFPACK so solving should be just as fast, but yes definitely worthwhile seeing if pypardiso improves things. SuperLU is horribly slow. I recall observing this in #439 but didn't pay much attention at that time. I have been using PyPI for the most part because of netgen, but will try out conda in the future.

Well, I tried this out on my System 76 laptop running Pop!_OS 21.04—not on that unsteady Navier–Stokes solver, that's all preconditioned Krylov, but for the steady Navier–Stokes solver in ex27 which (currently still) needs direct solvers to track the steady solution branch past the supercritical Hopf bifurcation, and got the opposite result!  The default SuperLU was significantly quicker than pypardiso! (The latter, changing only ex27, is in https://github.com/gdmcbain/scikit-fem/commit/bd2553b366c58a04ed9a57aa224400893d29b67b.)
Here's how SnakeViz saw it, run as, e.g.
python -m cProfile -o ex27.pardiso.prof -s cumtime docs/examples/ex27.py
snakeviz ex27.pardiso.prof

Master (SuperLU):

pypardiso:

I tried the same comparison, master v. bd2553b, on a Hewlett Packard laptop running MS-Windows 10 and it gave similar pictures, the big blocks again being linsolve.spsolve at 20.5 s v. pardiso_wrapper.factorize at 45.6 s.
Did I muck something up in bd2553b?

linsolve.spsolve at 20.5 s v. pardiso_wrapper.factorize at 45.6 s.

Hmm interesting and opposite of what I've read so far. I can try running ex27 as well just to double check. It is possible that something is peculiar about the problem I'm solving in ex36.

Yes, please do.  Maybe ex27 is too small?  I did reduce the number of elements so that it wouldn't take so long every time tests.test_examples.TestEx27 is run.

To that effect, here is the snakeviz from running ex36 with the two solvers
PyPardiso

Scipy

The assembly time is also large for my case, but the difference in solving is clear. Now onto, ex27.
Code
import numpy as np
from scipy.sparse import bmat
from skfem.helpers import grad, transpose, det, inv, identity
from skfem import *

mu, lmbda = 1., 1.e3

def F1(w):
    u = w["disp"]
    p = w["press"]
    F = grad(u) + identity(u)
    J = det(F)
    Finv = inv(F)
    return p * J * transpose(Finv) + mu * F

def F2(w):
    u = w["disp"]
    p = w["press"].value
    F = grad(u) + identity(u)
    J = det(F)
    Js = .5 * (lmbda + p + 2. * np.sqrt(lmbda * mu + .25 * (lmbda + p) ** 2)) / lmbda
    dJsdp = ((.25 * lmbda + .25 * p + .5 * np.sqrt(lmbda * mu + .25 * (lmbda + p) ** 2))
             / (lmbda * np.sqrt(lmbda * mu + .25 * (lmbda + p) ** 2)))
    return J - (Js + (p + mu / Js - lmbda * (Js - 1)) * dJsdp)

def A11(w):
    u = w["disp"]
    p = w["press"]
    eye = identity(u)
    F = grad(u) + eye
    J = det(F)
    Finv = inv(F)
    L = (p * J * np.einsum("lk...,ji...->ijkl...", Finv, Finv)
         - p * J * np.einsum("jk...,li...->ijkl...", Finv, Finv)
         + mu * np.einsum("ik...,jl...->ijkl...", eye, eye))
    return L

def A12(w):
    u = w["disp"]
    F = grad(u) + identity(u)
    J = det(F)
    Finv = inv(F)
    return J * transpose(Finv)

def A22(w):
    u = w["disp"]
    p = w["press"].value
    Js = .5 * (lmbda + p + 2. * np.sqrt(lmbda * mu + .25 * (lmbda + p) ** 2)) / lmbda
    dJsdp = ((.25 * lmbda + .25 * p + .5 * np.sqrt(lmbda * mu + .25 * (lmbda + p) ** 2))
             / (lmbda * np.sqrt(lmbda * mu + .25 * (lmbda + p) ** 2)))
    d2Jdp2 = .25 * mu / (lmbda * mu + .25 * (lmbda + p) ** 2) ** (3/2)
    L = (-2. * dJsdp - p * d2Jdp2 + mu / Js ** 2 * dJsdp ** 2 - mu / Js * d2Jdp2
         + lmbda * (Js - 1.) * d2Jdp2 + lmbda * dJsdp ** 2)
    return L

mesh = MeshTet().refined(3)
uelem = ElementVectorH1(ElementTetP2())
pelem = ElementTetP1()
elems = {
    "u": uelem,
    "p": pelem
}
basis = {
    field: Basis(mesh, e, intorder=2)
    for field, e in elems.items()
}

du = basis["u"].zeros()
dp = basis["p"].zeros()
stretch_ = 1.

ddofs = [
    basis["u"].find_dofs(
        {"left": mesh.facets_satisfying(lambda x: x[0] < 1.e-6)},
        skip=["u^2", "u^3"]
    ),
    basis["u"].find_dofs(
        {"bottom": mesh.facets_satisfying(lambda x: x[1] < 1.e-6)},
        skip=["u^1", "u^3"]
    ),
    basis["u"].find_dofs(
        {"back": mesh.facets_satisfying(lambda x: x[2] < 1.e-6)},
        skip=["u^1", "u^2"]
    ),
    basis["u"].find_dofs(
        {"front": mesh.facets_satisfying(lambda x: np.abs(x[2] - 1.) < 1e-6)},
        skip=["u^1", "u^2"]
    )
]

dofs = {}
for dof in ddofs:
    dofs.update(dof)

du[dofs["left"].all()] = 0.
du[dofs["bottom"].all()] = 0.
du[dofs["back"].all()] = 0.
du[dofs["front"].all()] = stretch_

I = np.hstack((
    basis["u"].complement_dofs(dofs),
    basis["u"].N + np.arange(basis["p"].N)
))

@LinearForm(nthreads=8)
def a1(v, w):
    return np.einsum("ij...,ij...", F1(w), grad(v))

@LinearForm(nthreads=8)
def a2(v, w):
    return F2(w) * v

@BilinearForm(nthreads=8)
def b11(u, v, w):
    return np.einsum("ijkl...,ij...,kl...", A11(w), grad(u), grad(v))

@BilinearForm(nthreads=8)
def b12(u, v, w):
    return np.einsum("ij...,ij...", A12(w), grad(v)) * u

@BilinearForm(nthreads=8)
def b22(u, v, w):
    return A22(w) * u * v

from time import time
t0 = time()
for itr in range(12):
    uv = basis["u"].interpolate(du)
    pv = basis["p"].interpolate(dp)

    K11 = asm(b11, basis["u"], basis["u"], disp=uv, press=pv)
    K12 = asm(b12, basis["p"], basis["u"], disp=uv, press=pv)
    K22 = asm(b22, basis["p"], basis["p"], disp=uv, press=pv)
    f = np.concatenate((
        asm(a1, basis["u"], disp=uv, press=pv),
        asm(a2, basis["p"], disp=uv, press=pv)
    ))
    K = bmat(
        [[K11, K12],
         [K12.T, K22]], "csr"
    )
    t1 = time()
    uvp = solve(*condense(K, -f, I=I), solver="pardiso")
    print(f"Solve time: {time() - t1}")
    delu, delp = np.split(uvp, [du.shape[0]])
    du += delu
    dp += delp
    normu = np.linalg.norm(delu)
    normp = np.linalg.norm(delp)
    print(f"{itr+1}, norm_du: {normu}, norm_dp: {normp}")
    if normu < 1.e-8 and normp < 1.e-8:
        break

print(f"Total time: {time() - t0}")

if __name__ == "__main__":
    mesh.save(
        "example36_results.xdmf",
        {"u": du[basis["u"].nodal_dofs].T, "p": dp[basis["p"].nodal_dofs[0]]},
    )

With ex27, indeed SuperLU is faster.
scipy

pardiso

I guess what I said might not be true in general then. It highly depends on the specific characteristics of the problem (associated jacobian and residual). Just out of curiosity, have you tried ex27 with larger meshes?

I have tried now with self.mesh.refined(4) and the results are still in favor of SuperLU.

vs

Just to be definite, this is the code that I am running
from skfem import *
from skfem.helpers import grad, dot
from skfem.models.poisson import vector_laplace, laplace
from skfem.models.general import divergence, rot
from skfem.utils import LinearSolver
import skfem.io.json
from functools import partial

from pathlib import Path
from typing import Tuple, Iterable
from matplotlib.pyplot import subplots
from matplotlib.tri import Triangulation
import numpy as np
from scipy.sparse import bmat, block_diag, csr_matrix

from pacopy import natural
from pypardiso import spsolve

def solver_direct_pypardiso() -> LinearSolver:
    """Linear solver provided by Pardiso."""

    return spsolve

# solve = partial(solve, solver=solver_direct_pypardiso())

@LinearForm(nthreads=8)
def acceleration(v, w):
    """Compute the vector (v, u . grad u) for given velocity u
    passed in via `wind` after having been interpolated onto its
    quadrature points.
    In Cartesian tensorial indicial notation, the integrand is
    .. math::
        u_j u_{i,j} v_i.
    """
    return np.einsum('j...,ij...,i...', w['wind'], grad(w['wind']), v)

@BilinearForm(nthreads=8)
def acceleration_jacobian(u, v, w):
    """Compute (v, w . grad u + u . grad w) for given velocity w
    passed in via w after having been interpolated onto its quadrature
    points.
    In Cartesian tensorial indicial notation, the integrand is
    .. math::
       (w_j du_{i,j} + u_j dw_{i,j}) v_i
    """
    return dot(np.einsum('j...,ij...->i...', w['wind'], grad(u))
               + np.einsum('j...,ij...->i...', u, grad(w['wind'])), v)

class BackwardFacingStep:
    element = {'u': ElementVectorH1(ElementTriP2()),
               'p': ElementTriP1()}
    def __init__(self,
                 length: float = 35.):
        self.mesh = skfem.io.json.from_file(Path(__file__).parent / 'meshes' / 'backward-facing_step.json')
        self.mesh.refined(6)
        self.basis = {variable: Basis(self.mesh, e, intorder=3)
                      for variable, e in self.element.items()}
        self.basis['inlet'] = FacetBasis(self.mesh, self.element['u'],
                                         facets=self.mesh.boundaries['inlet'])
        self.basis['psi'] = self.basis['u'].with_element(ElementTriP2())
        self.D = np.concatenate([b.all() for b in self.basis['u'].find_dofs().values()])
        A = asm(vector_laplace, self.basis['u'])
        B = asm(divergence, self.basis['u'], self.basis['p'])
        self.S = bmat([[A, -B.T],
                       [-B, None]], 'csr')
        self.I = np.setdiff1d(np.arange(self.S.shape[0]), self.D)
    def inlet_dofs(self):
        return self.basis['inlet'].find_dofs()['inlet'].all()

    @staticmethod
    def parabolic(x):
        """return the plane Poiseuille parabolic inlet profile"""
        return np.stack([4 * x[1] * (1. - x[1]), np.zeros_like(x[0])])
    def make_vector(self):
        """prepopulate solution vector with Dirichlet conditions"""
        uvp = np.zeros(self.basis['u'].N + self.basis['p'].N)
        I = self.inlet_dofs()
        uvp[I] = projection(self.parabolic, self.basis['inlet'], I=I)
        return uvp
    def split(self, solution: np.ndarray) -> Tuple[np.ndarray,
                                                   np.ndarray]:
        """return velocity and pressure separately"""
        return np.split(solution, [self.basis['u'].N])

    def streamfunction(self, velocity: np.ndarray) -> np.ndarray:
        A = asm(laplace, self.basis['psi'])
        psi = self.basis['psi'].zeros()
        vorticity = asm(rot, self.basis['psi'],
                        w=self.basis['u'].interpolate(velocity))
        psi = solve(*condense(A, vorticity, D=self.basis['psi'].find_dofs()['floor'].all()))
        return psi

    def mesh_plot(self):
        """return Axes showing boundary of mesh"""
        termini = self.mesh.facets[:, np.concatenate(list(
            self.mesh.boundaries.values()))]
        _, ax = subplots()
        ax.plot(*self.mesh.p[:, termini], color='k')
        return ax

    def triangulation(self):
        return Triangulation(*self.mesh.p, self.mesh.t.T)

    def streamlines(self, psi: np.ndarray, n: int = 11, ax=None):
        if ax is None:
            ax = self.mesh_plot()
        n_streamlines = n
        plot = partial(ax.tricontour,
                       self.triangulation(),
                       psi[self.basis['psi'].nodal_dofs.flatten()],
                       linewidths=.5)
        for levels, color, style in [
            (np.linspace(0, 2/3, n_streamlines),
             'k',
             ['dashed'] + ['solid']*(n_streamlines - 2) + ['dashed']),
            (np.linspace(2/3, max(psi), n_streamlines)[0:],
             'r', 'solid'),
            (np.linspace(min(psi), 0, n_streamlines)[:-1],
             'g', 'solid')]:
            plot(levels=levels, colors=color, linestyles=style)
        ax.set_aspect(1.)
        ax.axis('off')
        return ax

    def inner(self, u: np.ndarray,  v: np.ndarray) -> float:
        """return the inner product of two solutions
        using just the velocity, ignoring the pressure
        """
        return self.split(u)[0] @ self.split(v)[0]

    def norm2_r(self, u: np.ndarray) -> float:
        return self.inner(u, u)

    def N(self, uvp: np.ndarray) -> np.ndarray:
        u = self.basis['u'].interpolate(self.split(uvp)[0])
        return np.hstack([asm(acceleration, self.basis['u'], wind=u),
                          self.basis['p'].zeros()])

    def f(self, uvp: np.ndarray, reynolds: float) -> np.ndarray:
        """Return the residual of a given velocity-pressure solution"""
        out = self.S @ uvp + reynolds * self.N(uvp)
        out[self.D] = uvp[self.D] - self.make_vector()[self.D]
        return out

    def df_dlmbda(self, uvp: np.ndarray, reynolds: float) -> np.ndarray:
        out = self.N(uvp)
        out[self.D] = 0.
        return out

    def jacobian_solver(self,
                        uvp: np.ndarray,
                        reynolds: float,
                        rhs: np.ndarray) -> np.ndarray:
        u = self.basis['u'].interpolate(self.split(uvp)[0])
        duvp = solve(*condense(
            self.S +
            reynolds
            * block_diag([asm(acceleration_jacobian, self.basis['u'], wind=u),
                          csr_matrix((self.basis['p'].N,)*2)]),
            rhs, self.make_vector() - uvp, I=self.I))
        return duvp

bfs = BackwardFacingStep()
psi = {}

def callback(_: int,
             reynolds: float,
             uvp: np.ndarray,
             milestones=Iterable[float]):
    """Echo the Reynolds number and plot streamlines at milestones"""
    print(f'Re = {reynolds}')
    if reynolds in milestones:
        psi[reynolds] = bfs.streamfunction(bfs.split(uvp)[0])
        if __name__ == '__main__':
            ax = bfs.streamlines(psi[reynolds])
            ax.set_title(f'Re = {reynolds}')
            ax.get_figure().savefig(Path(__file__).with_name(
                f'{Path(__file__).stem}-{reynolds}-psi.png'),
                                    bbox_inches="tight", pad_inches=0)

if __name__ == '__main__':
    milestones = [50., 150., 450., 750.]
else:
    milestones = [50.]
natural(bfs, bfs.make_vector(), 0.,
        partial(callback,
                milestones=milestones),
        lambda_stepsize0=50.,
        milestones=milestones)

A weird thing that I notice is that when I try self.mesh.refined(6) instead of self.mesh.refined(4) I get a smaller time for the solve in SuperLU whereas Pardiso is constant. Is this how it is supposed to be? Or am i doing something wrong here? (49.69 s for SuperLU vs 68.75 for Pardiso).

Ah, I have previously, but since pygmsh switched first its licence and then its API, the code referring to it was removed and replaced by a fixed mesh...
https://github.com/kinnala/scikit-fem/blob/c07ff9f6d6a51e62eac2777351b20b4549027ada/docs/examples/ex27.py#L109
Unfortunately appending .refined() fails, raising
  File "/home/gdmcbain/src/scikit-fem/docs/examples/ex27.py", line 124, in __init__
    facets=self.mesh.boundaries['inlet'])
TypeError: 'NoneType' object is not subscriptable

i.e. boundaries are not preserved during refinement.  Let me see if I can dig out the old mesh-generating function and pip install pygmsh\<7. …Nope, that doesn't work any more, too many upstream changes in pygmsh and meshio.

A weird thing that I notice is that when I try self.mesh.refined(6) instead of self.mesh.refined(4) here

Oh, that worked for you?

Ah, hang on: what is your changed line?  Note that the .refined method is functional, it returns the refined mesh rather than mutating self.   There used to be a .refine method that mutated self and returned None.  That is, if you merely called
    self.mesh.refined(n)

it would not have done anything; that would be consistent with

whereas Pardiso is constant

See #536 for discussion of the change from .refine to .refined.

You're right, I seem to have confounded the two. And there is the error you posted.

But then, if I naively do
In [1]: from scipy.sparse import random as rand

In [2]: import numpy as np

In [3]: import pypardiso

In [4]: from scipy.sparse.linalg import spsolve as spl

In [5]: A = rand(10000, 10000, density=0.01, format="csr", dtype= np.float64)

In [6]: A += identity(A.shape[0])

In [7]: b = np.random.randn(A.shape[0])

In [8]: %timeit pypardiso.spsolve(A,b)
45.6 ms ± 8.92 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [9]: %timeit spl(A, b)

scipy.sparse.linalg.spsolve has been running for quite sometime now. I tested on a smaller matrix and found
In [12]: A = rand(1000, 1000, density=0.01, format="csr", dtype= np.float64) + identity(1000)

In [13]: %timeit spl(A, b)
286 ms ± 3.42 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [14]: %timeit pypardiso.spsolve(A,b)
330 µs ± 2.28 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Any thoughts @gdmcbain ?

Well, I'm not an expert but my intuition is that the sparse matrices assembled from finite element problems do have some kind of structure compared to rand and maybe the SuperLU is designed to do better on that kind of thing?  I have no actual basis for believing that though.
How do we go with simple Laplace + Dirichlet, as in ex01?

You could try reordering the rows and the columns, e.g., using skfem.utils.rcm. This kind of reordering is done to reduce the NNZ entries in the LU/Cholesky decomposition. I guess some linear algebra packages may do that automatically.

Ah, yes, reordering #168.

I have been using PyPI for the most part because of netgen, but will try out conda in the future.

I don't use conda too much, but I think you can install pip inside conda and use both tools to install packages.

How do we go with simple Laplace + Dirichlet, as in ex01?

0.41s (pardiso) vs 1.495s (SuperLU) on my laptop. Will check skfem.utils.rcm

I don't use conda too much, but I think you can install pip inside conda and use both tools to install packages.

Thanks! I have been meaning to do this for sometime. Back then I did not have a good understanding of package management and virtual environments.

I don't use conda too much, but I think you can install pip inside conda and use both tools to install packages.

This is correct.  These days a conda env includes its own pip command that just pip-installs inside that conda env.

O. K., with the proposed fix #693 to #691, I can now read MSH 4.1 (or rather VTU recoded & saved from MSH 4.1) in ex27 and so can generate meshes of different fineness to see how pardiso scales compared to the default solver.
Halving the mesh-size with (see #693 for original generation instructions at -clscale 0.2)
gmsh -2 -clscale 0.1 backward-facing_step.geo

and recoding & saving with Mesh.load(...).save gives 8685 vertices and pardiso does now do slightly better than the default: 145 v. 168 s.

Halving again (-clscale 0.05, 33751 vertices),  the default solver used 1.96×10³ s but after doing fine at first (up to Reynolds number about 545), pardiso stalled in the natural-parameter continuation (observe that the continuation step in Reynolds number has dropped to 0.1189) and then failed
Step 20: lambda  5.742e+02 + 1.189e-01  ->  5.743e+02
||F(u)|| = 1.933075e-01
||F(u)|| = 3.569919e-03
Traceback (most recent call last):
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/cProfile.py", line 185, in <module>
    main()
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/cProfile.py", line 178, in main
    runctx(code, globs, None, options.outfile, options.sort)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/cProfile.py", line 19, in runctx
    return _pyprofile._Utils(Profile).runctx(statement, globals, locals,
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/profile.py", line 62, in runctx
    prof.runctx(statement, globals, locals)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/cProfile.py", line 100, in runctx
    exec(cmd, globals, locals)
  File "docs/examples/ex27.py", line 266, in <module>
    natural(bfs, bfs.make_vector(), 0.,
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pacopy/natural.py", line 93, in natural
    u, newton_steps = newton(
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pacopy/newton.py", line 20, in newton
    du = jacobian_solver(u, -fu)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pacopy/natural.py", line 95, in <lambda>
    lambda u, rhs: problem.jacobian_solver(u, lmbda, rhs),
  File "docs/examples/ex27.py", line 229, in jacobian_solver
    duvp = solve(*condense(
  File "/home/gdmcbain/src/scikit-fem/skfem/utils.py", line 226, in solve
    return solve_linear(A, b, x, I, solver, **kwargs)  # type: ignore
  File "/home/gdmcbain/src/scikit-fem/skfem/utils.py", line 193, in solve_linear
    y[I] = solver(A, b, **kwargs)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pypardiso/scipy_aliases.py", line 48, in spsolve
    x = solver.solve(A, b)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pypardiso/pardiso_wrapper.py", line 155, in solve
    x = self._call_pardiso(A, b)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pypardiso/pardiso_wrapper.py", line 262, in _call_pardiso
    raise PyPardisoError(pardiso_error.value)
pypardiso.pardiso_wrapper.PyPardisoError: The Pardiso solver failed with error code -4. See Pardiso documentation for details.

Hitherto I had not noticed any difference in the solutions obtained by pardiso & scipy, but I think this says that there must be.  It's true that this is a fairly challenging nonlinear problem.

Hmm, the failure isn't deterministic either; here it was rerun (without profiling); same PyPardisoError but at a slightly different Reynolds number:
No convergence for lambda=572.6578742265701.
Step 13: lambda  5.584e+02 + 7.127e+00  ->  5.655e+02
||F(u)|| = 1.812719e-03
||F(u)|| = 3.381340e-06
||F(u)|| = 5.901106e-08
||F(u)|| = 2.683568e-11
||F(u)|| = 1.979994e-13
Re = 565.5306950211525
Step 14: lambda  5.655e+02 + 8.018e+00  ->  5.735e+02
||F(u)|| = 2.343903e-03
||F(u)|| = 2.885877e-06
||F(u)|| = 1.470555e-09
||F(u)|| = 2.115316e-09
Traceback (most recent call last):
  File "/home/gdmcbain/src/scikit-fem/docs/examples/ex27.py", line 266, in <module>
    natural(bfs, bfs.make_vector(), 0.,
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pacopy/natural.py", line 93, in natural
    u, newton_steps = newton(
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pacopy/newton.py", line 20, in newton
    du = jacobian_solver(u, -fu)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pacopy/natural.py", line 95, in <lambda>
    lambda u, rhs: problem.jacobian_solver(u, lmbda, rhs),
  File "/home/gdmcbain/src/scikit-fem/docs/examples/ex27.py", line 229, in jacobian_solver
    duvp = solve(*condense(
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/skfem/utils.py", line 226, in solve
    return solve_linear(A, b, x, I, solver, **kwargs)  # type: ignore
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/skfem/utils.py", line 193, in solve_linear
    y[I] = solver(A, b, **kwargs)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pypardiso/scipy_aliases.py", line 48, in spsolve
    x = solver.solve(A, b)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pypardiso/pardiso_wrapper.py", line 155, in solve
    x = self._call_pardiso(A, b)
  File "/home/gdmcbain/miniconda3/envs/skfem/lib/python3.9/site-packages/pypardiso/pardiso_wrapper.py", line 262, in _call_pardiso
    raise PyPardisoError(pardiso_error.value)
pypardiso.pardiso_wrapper.PyPardisoError: The Pardiso solver failed with error code -4. See Pardiso documentation for details.

Another rerun kept going, but the continuation really wasn't getting anywhere:

No convergence for lambda=575.1003060782954.
Step 57: lambda  5.751e+02 + 1.467e-04  ->  5.751e+02

Yes, so far Pardiso looks better on everything except ex27 (lucky me), so I do think it's worth persisting with, particularly as it's so easy to use in scikit-fem via pypardiso.  I'd say it was well worth including in the suite of integrations #474.  I don't know why ex27 was problematic or why Pardiso gave different results to SciPy and different results from run to run, but would be interesting in learning more if this is expected or other evidence is found.

authors: bhaveshshrimali gdmcbain kinnala

repo: kinnala/scikit-fem


text:
kola: Add a filter to run tests based on offering

kola: Add a filter to run tests based on offering
With the introduction of the Pro offering, it is not possible to
test Pro specific features or test just for Pro without them runing
on the basic offering as well. This changes introduces a new flag
that helps building a clear distinction between Pro and the nornal
offering.
The excludeOffering maybe does not make sense now with two offering
but kept it for future feasibility.
Signed-off-by: Sayan Chowdhury sayan.chowdhury2012@gmail.com
How to use
./build kola
./bin/kola run --offering=pro

Testing done
Not done yet.

I've updated tests with the Offering param so that it is easier and cleaner in jenkins to just pass the parameter basic or pro.

I've updated tests with the Offering param so that it is easier and cleaner in jenkins to just pass the parameter basic or pro.

source: gharchive/pull-request

repo: kinvolk/mantle

license_type: permissive

license_source: github-api


text:
Server URL example is unclear

Under the 'Server URL' box it says "Example: demo.mealie.io", which implies it wants a URL without http but that does not work and gives an error. If I enter my server's IP address with "http://" in front of it then it works as expected.
I suggest the example text be changed to "Example: http://demo.mealie.io"

@dogyx Hello!
Right now, if the user doesn't provide the scheme (as in the example) then the app defaults to HTTPS. I believe, it gives an error in your case because your Mealie installation doesn't support HTTPS.
I will try to figure out a way for the app to determine the scheme on its own instead of always assuming HTTPS, which should resolve your issue without having you type more information than you should've.
Thanks for the feedback, it motivates me to improve the app 👍

@kirmanak
I tested PR #124 and it works as expected now. :)

source: gharchive/issue

authors: dogyx kirmanak

repo: kirmanak/Mealient

license: MIT


text:
pre-training for data params

Unsupervised learning for choosing data parameters

Method: K-means clustering
Score: model's inertia (Sum of squared distances of samples to their closest cluster center)
Parameters:

window size: window size of Gaussian kernel for smoothing data
K: the threshold for the length of segments
num_kernels: number of kernels used in ROCKET transformation

Criteria: the smaller the inertia of the clustering model, the better.
working dir: /work/notebooks/3-decoding-model/2-classification/3-time-series-classification/2-pre-training

Work Flow

[x] research on how to use K-means
[x] train script for testing on diff params of data -> train.py

large scale of num_kernels and K
~small scale~

[x] Analyze the training results -> 02-influence-of-data-param.ipynb

visualized the results and it's the opposite of what we expected

~get the best param pair from the large scale~
~examine the small scale results~

[ ] ~Analyze the similarity between the clustering results and y(position) -> 03-similarity-between-clusters-positions.ipynb~

~coverage rate, e.g probability of instances from the same position are in the same cluster~
~if the coverage rate is above 50% ?~

K-means usage
from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=2, random_state=0, n_init="auto")
kmeans.fit(X_train)
kmeans.predict(X_train)

Regarding Data

Use all the data to train a K-means model -> Modify UniformSegmentDataset

[x]  change train_ratio to Optional
[x]  if it's given a float number then split the data
[x]  if not, then only return the train set which used all the available data.
[x] import the modified UniformSegmentDataset

The /data folder was deleted

[x] convert the original xlsx data to csv
:point_right: It takes Wall time of 266 ms to read from csv files of all mice, 45.4 sfor xlsx files

K-means training

the cv was not used in the training
However, If we are to use CV for K-means: k-fold Cross Validation for determining k in k-means?

A ground truth set is needed
Specify a scoring method.

checking the results
The desired result: the inertia would reduce as more kernels are used
Reasons

the small the inertia is, the data is more likely to be separated
the previous supervised method showed the higher the num_kernels is, the more accurate the model is.
👉  However the opposite result was shown

Questions to be answered

is it true that the small the inertia is, the data is more likely to be separated?
how did num_kernels influenced inertia?

The n_clusters parameter was set to a wrong number, retrained the k-means model after modifying.

repo: kisaragiRY/decoding-DG


text:
Demo page is not working

https://taskboard.matthewross.me/demo-live/ does not bring up log in window. Just shows empty space.

Well... that's not good.

Fixed!

authors: kiswa siakc

repo: kiswa/TaskBoard


text:
Switch to external GraphQL data source

https://github.com/kitibyte/ldb-graphql

📝Changed routes:

/stations/[name]

(view full size)

2 deleted routes:

/api/stations/[name]
/api/stations

Commit 3791dd61b7144ddf01557c383317f82c3069a960 (https://ldb-git-graphql.kitibyte.vercel.app).

repo: kitibyte/ldb-web


text:
Wait for db using healthchecks

closes #16

thanks for the PR, I'll test it

authors: JannikStreek squatica

repo: kitsteam/excalidraw-storage-backend


text:
[PRO-2779][v1] - Pre-commit hooks

Similar to https://github.com/kiva/protocol-sdk-verifier/pull/14, this doesn't solve all our problems, but it's a start

For the record I don't see a postinstall script as any more invasive than asking people to install our NPM project, but I don't want to make anyone unnecessarily uncomfortable.
Also, people may have their own hooks set up. Overwriting those hook scripts with ours would definitely be a Bad Thing.

repo: kiva/protocol-sdk-issuer


text:
Crash when MarkerMapLayer without Marker

When adding a MarkerMapLayer with no marker to a MapView, the application crashes.
Even though it may not be relevant to add a MarkerMapLayer without Marker, we also get the issue when adding makers then removing them all.
Sample application:
from kivy.base import runTouchApp
from kivy.garden.mapview import MapView, MarkerMapLayer

mapview = MapView()
# empty MarkerMapLayer, layer causes ValueError exception
# ValueError: max() arg is an empty sequence
mapview.add_layer(MarkerMapLayer())
runTouchApp(mapview)

Stacktrace:
[INFO              ] [Logger      ] Record log in /home/andre/.kivy/logs/kivy_15-07-31_46.txt
[INFO              ] [Kivy        ] v1.9.0
[INFO              ] [Python      ] v2.7.9 (default, Feb  2 2015, 23:47:54)
[GCC 4.8.3]
[INFO              ] [Factory     ] 173 symbols loaded
[INFO              ] [Image       ] Providers: img_tex, img_dds, img_gif, img_pygame, img_pil (img_ffpyplayer ignored)
[INFO              ] [Window      ] Provider: pygame(['window_egl_rpi'] ignored)
libpng warning: iCCP: known incorrect sRGB profile
[INFO              ] [GL          ] OpenGL version <3.0 Mesa 10.2.8>
[INFO              ] [GL          ] OpenGL vendor <Intel Open Source Technology Center>
[INFO              ] [GL          ] OpenGL renderer <Mesa DRI Intel(R) Ivybridge Mobile >
[INFO              ] [GL          ] OpenGL parsed version: 3, 0
[INFO              ] [GL          ] Shading version <1.30>
[INFO              ] [GL          ] Texture max size <8192>
[INFO              ] [GL          ] Texture max units <16>
[INFO              ] [Window      ] virtual keyboard not allowed, single mode, not docked
[INFO              ] [Text        ] Provider: pygame
[INFO              ] [OSC         ] using <multiprocessing> for socket
[INFO              ] [ProbeSysfs  ] device match: /dev/input/event1
[INFO              ] [MTD         ] Read event from </dev/input/event1>
[INFO              ] [Base        ] Start application main loop
[INFO              ] [GL          ] NPOT texture support is available
[INFO              ] [Base        ] Leaving application in progress...
 Traceback (most recent call last):
   File "tests/marker_map_layer_crash.py", line 8, in <module>
     runTouchApp(mapview)
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/base.py", line 487, in runTouchApp
     EventLoop.window.mainloop()
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/core/window/window_pygame.py", line 398, in mainloop
     self._mainloop()
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/core/window/window_pygame.py", line 285, in _mainloop
     EventLoop.idle()
     Exception in thread Thread-1:
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/base.py", line 327, in idle
 Traceback (most recent call last):
     Clock.tick()
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/clock.py", line 483, in tick
     self._process_events()
   File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/clock.py", line 615, in _process_events
     self.run()
     event.tick(self._last_tick, remove)
   File "/usr/lib64/python2.7/threading.py", line 763, in run
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/clock.py", line 374, in tick
     self.__target(*self.__args, **self.__kwargs)
     ret = callback(self._dt)
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/input/providers/mtdev.py", line 197, in _thread_run
   File "/home/andre/.kivy/garden/garden.mapview/mapview/view.py", line 692, in do_update
     _device = Device(_fn)
     layer.reposition()
   File "/home/andre/Progz/guyps/lib/python2.7/site-packages/kivy/lib/mtdev.py", line 131, in __init__
   File "/home/andre/.kivy/garden/garden.mapview/mapview/view.py", line 209, in reposition
     self._fd = os.open(filename, os.O_NONBLOCK | os.O_RDONLY)
     margin = max((max(marker.size) for marker in markers))
 OSError: [Errno 13] Permission denied: '/dev/input/event1'
 ValueError: max() arg is an empty sequence

Additional info:

Python 2.7.9
Kivy==1.9.0
garden.mapview master (currently 97257ac8c7187051a7cd652bc7d2ef04fe849dad) installed with garden install mapview

A trick is to add a marker whose source is a transparent .png. It should be added apart from other markers (stored in a different variable), so that when you remove the visible markers, the transparent one stays and the MarkerMapLayer never gets empty.
I'm curious about other solutions for avoiding this error.

I was just faced with the same problem and adding a transparent .png worked perfectly! Thanks!

authors: AndreMiras gyorgytamara zoltan-fedor

license: mit

license_source: bigquery


text:
Property force_dispatch doesn't work on android.

Hi, I tried to create a Carousel widget with a swiped property, so it would dispatch events only when the carousel is swiped. On this example the property swiped is either "1" or "-1". Since I needed it to dispatch every time a swipe happens, I added the force_dispatch=True to the property.
This works totally fine on my desktop/linux machine, but when I build a pkg and run it on an android the force dispatch doesn't work. But setting the property to "0" and back to the value its supposed to be causes two dispatches as expected.
class XCarousel(Carousel):

    swiped = NumericProperty(0, force_dispatch=True)

    def __init__(self, **kwargs):
        super(XCarousel, self).__init__(**kwargs)

    def on__offset(self, *args):
        old_index = self.index
        super(XCarousel, self).on__offset(*args)
        swiped = self.index-old_index
        if swiped != 0:
            self.swiped = swiped

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/25374170-property-force_dispatch-doesn-t-work-on-android?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F42681&utm_medium=issues&utm_source=github).

Did you check that you were using kivy 1.9.0 or newer on android ? if it works on desktop, it should work on android.

Oh, you are right. It was kivy 1.8.0 in the apk. Thanks for making the world make sense again. It was an rtfm issue on my part. I wrongly assumed that Python for Android would use 1.9.0 as default. For anyone who might have any similar issue:
for the latest version use
./distribute.sh -m "openssl kivy==master"
and for any specific version
./distribute.sh -m "openssl kivy==1.9.0" etc.

authors: tshirtman vasss

repo: kivy/kivy


text:
Font not found error: cour.ttf

Software Versions

Python: 3.6.7
OS: Ubuntu 18.04.5 LTS x64
Kivy: 1.11.1
Kivy installation method: conda

Describe the bug
I run the application in Windows 10 and there is no problem. But I run in Ubuntu 18 python gives this error:

Traceback (most recent call last):
File "main.py", line 591, in
main()
File "main.py", line 587, in main
app.run()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/app.py", line 855, in run
runTouchApp()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/base.py", line 504, in runTouchApp
EventLoop.window.mainloop()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/core/window/window_sdl2.py", line 747, in mainloop
self._mainloop()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/core/window/window_sdl2.py", line 479, in _mainloop
EventLoop.idle()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/base.py", line 348, in idle
Clock.tick_draw()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/clock.py", line 598, in tick_draw
self._process_events_before_frame()
File "kivy/_clock.pyx", line 427, in kivy._clock.CyClockBase._process_events_before_frame
File "kivy/_clock.pyx", line 467, in kivy._clock.CyClockBase._process_events_before_frame
File "kivy/_clock.pyx", line 465, in kivy._clock.CyClockBase._process_events_before_frame
File "kivy/_clock.pyx", line 167, in kivy._clock.ClockEvent.tick
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/uix/label.py", line 413, in texture_update
self._label.refresh()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/core/text/init.py", line 759, in refresh
self.resolve_font_name()
File "/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/core/text/init.py", line 356, in resolve_font_name
raise IOError('Label: File %r not found' % fontname)
OSError: Label: File 'cour.ttf' not found

Expected behavior
There should no error.
To Reproduce
I do not use specifically cour.tff font. I do not know which code use it.
Additional context
I fixed the problem via downloading cour.tff on the internet then I added it to this folder:

/home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/data/fonts

This is code in /home/halil/.conda/envs/chargegen_test/lib/python3.6/site-packages/kivy/core/text/init.py
if filename is None:
    # XXX for compatibility, check directly in the data dir
    filename = pep8_fn = os.path.join(kivy_data_dir, fontname)
    if not os.path.exists(pep8_fn) or not os.path.isfile(pep8_fn):
        raise IOError('Label: File %r not found' % fontname)

I found that I added that font in kv file. My search algorithm not found it, sorry.
            Label:
                font_name: 'cour'


text:
Unnecessary black window appears when using Kivy and Multiprocessing

Software Versions

Python:3.9.10
OS:Windows11
Kivy:2.2.1
Kivy installation method:pip install

Describe the bug
I noticed that when I add some Kivy library Imports, an unwanted black window appears.
This problem seems to occur when using Kivy and Multiprocessing at the same time.
The following code is the normal code.
The black window does not appear.
    from kivy.app import App
    from kivy.uix.widget import Widget
    from multiprocessing import Process, Value
    from kivy.lang import Builder
    """
    from kivy.uix.checkbox import CheckBox
    from kivy.uix.spinner import Spinner
    from kivy.properties import StringProperty,NumericProperty,ObjectProperty, BooleanProperty
    from kivy.core.window import Window
    from kivy.uix.boxlayout import BoxLayout
    from kivy.uix.popup import Popup
    from kivy.uix.treeview import TreeView, TreeViewLabel, TreeViewNode
    from kivy.uix.label import Label
    from kivy.uix.textinput import TextInput
    from kivy.uix.progressbar import ProgressBar
    from kivy.uix.button import Button
    """
    Builder.load_string("""
    <TextWidget>:
    	BoxLayout:
    		orientation: 'vertical'
    		size: root.size

    		Button:
    			id: button1
    			text: "start"
    			font_size: 48
    			on_press: root.buttonClicked()
    		Label:
    			id: lab
    			text: 'result'
    """)

    class TextWidget(Widget):
    	def __init__(self, **kwargs):
    		super(TextWidget, self).__init__(**kwargs)
    		self.process_test = None
    		self.proc = None

    		# shared memory
    		self.count = Value('i', 0)
    		self.proc = start_process(self.count)
    		self.ids.button1.text = "stop"

    	def buttonClicked(self):
    		if self.ids.button1.text == "start":

    			self.ids.button1.text = "stop"
    		else:
    			if self.proc:
    				self.proc.kill()
    			self.ids.button1.text = "start"
    			self.ids.lab.text = str(self.count.value)

    class TestApp(App):
    	def __init__(self, **kwargs):
    		super(TestApp, self).__init__(**kwargs)
    		self.title = 'testApp'

    	def build(self):
    		return TextWidget()

    def start_process(count):
    	process_test = Process(target=p_test, args=[count], daemon=True)
    	process_test.start()
    	return process_test

    def p_test(count):
    	i = 0
    	while True:
    		print(i)
    		i = i + 1
    		with count.get_lock():
    			count.value = i

    if __name__ == '__main__':
    	TestApp().run()

However, when uncommented out, a black screen appears, with the GUI hidden behind it, which must be accessed through the Task Manager.
    from kivy.app import App
    from kivy.uix.widget import Widget
    from multiprocessing import Process, Value
    from kivy.lang import Builder

    from kivy.uix.checkbox import CheckBox
    from kivy.uix.spinner import Spinner
    from kivy.properties import StringProperty,NumericProperty,ObjectProperty, BooleanProperty
    from kivy.core.window import Window
    from kivy.uix.boxlayout import BoxLayout
    from kivy.uix.popup import Popup
    from kivy.uix.treeview import TreeView, TreeViewLabel, TreeViewNode
    from kivy.uix.label import Label
    from kivy.uix.textinput import TextInput
    from kivy.uix.progressbar import ProgressBar
    from kivy.uix.button import Button

    Builder.load_string("""
    <TextWidget>:
    	BoxLayout:
    		orientation: 'vertical'
    		size: root.size

    		Button:
    			id: button1
    			text: "start"
    			font_size: 48
    			on_press: root.buttonClicked()
    		Label:
    			id: lab
    			text: 'result'
    """)

    class TextWidget(Widget):
    	def __init__(self, **kwargs):
    		super(TextWidget, self).__init__(**kwargs)
    		self.process_test = None
    		self.proc = None

    		# shared memory
    		self.count = Value('i', 0)
    		self.proc = start_process(self.count)
    		self.ids.button1.text = "stop"

    	def buttonClicked(self):
    		if self.ids.button1.text == "start":

    			self.ids.button1.text = "stop"
    		else:
    			if self.proc:
    				self.proc.kill()
    			self.ids.button1.text = "start"
    			self.ids.lab.text = str(self.count.value)

    class TestApp(App):
    	def __init__(self, **kwargs):
    		super(TestApp, self).__init__(**kwargs)
    		self.title = 'testApp'

    	def build(self):
    		return TextWidget()

    def start_process(count):
    	process_test = Process(target=p_test, args=[count], daemon=True)
    	process_test.start()
    	return process_test

    def p_test(count):
    	i = 0
    	while True:
    		print(i)
    		i = i + 1
    		with count.get_lock():
    			count.value = i

    if __name__ == '__main__':
    	TestApp().run()

How can I solve this problem?
I am using Windows11 and WinPython.

This is a problem with the way multiprocessing works in python, and the way kivy works.  Here is a multiprocessing example using kivy: https://github.com/ElliotGarbus/MidiClockGenerator
Take a look at main.py, notice that most of the code is under a gate: "if name == 'main':"
This ensures that only the main processes creates a Window.  The import of kivy.app creates a window, so that needs to be under a gate.
This is discussed briefly in the python docs. https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming
Read the section titled: "Safe importing of main module"

Duplicate of #8340 and #4744

authors: ElliotGarbus Julian-O emoto-yasushi


text:
img_dds.py fix for upside down dds files.

All DDS files I tested where loaded upside down, but to be honest I'm not sure if some tools save the data inside dds file in some other arrangement - OpenGL and Direct3D use different conventions for where up and down is in a texture.

It probably stems from the fact that Window uses the top left of the screen as (0, 0), and OpenGL the bottom left. The option is already there to flip, so I don't really see the point of changing the default? It's so easy to change and so obvious. Do you really think it warrants the change?

authors: Zen-CODE wezu


text:
chore: Allow werkzeug >= 1.0.0

Resolves #433

Codecov Report

Merging #434 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #434   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          25     25
  Lines        1843   1843
  Branches      292    292
=====================================
  Hits         1843   1843

Flag
Coverage Δ

#unittests
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f4fd769...3aca7e9. Read the comment docs.

repo: kiwicom/schemathesis


text:
load as extension

Hi,
how can I register and load your nice extension with load_extension ?

It doesn't really behave like an extension of Telescope, but rather it provides nice action functions you can easily map to Telescope.

I know, but is it possible to register it as extension?
Like:
local path_actions = require("telescope_insert_path")
	require("telescope").setup({
		extensions = {
			mappings = {
				n = {
					["["] = path_actions.insert_reltobufpath_visual,
				},
			},
		},
	})

require('telescope').load_extension 'telescope_insert_path'
something like this?


text:
🛑 MON 8 is down

In 8b7d093, MON 8 ($MON_URL/10_0_9_8.html) was down:

HTTP code: 404
Response time: 361 ms

Resolved: MON 8 is back up in 82d7370 after 10 minutes.

repo: kizzna/bovorn-mon


text:
🛑 Sangharaja is down

In 2a44d40, Sangharaja (https://sangharaja.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Sangharaja is back up in 6f97d5c after 9 minutes.

repo: kizzna/uptime


text:
Windows still panics in some cases

Hi all,
Thanks for the fixes to, I think we managed to fix some of our Windows instability with this PR: https://github.com/kkawakam/rustyline/issues/359
Unfortunately, users are still reporting very similar issues. I haven't been able to reproduce it, yet, but I wanted to file an issue in case you are able to reproduce: https://github.com/nushell/nushell/issues/2440
Happy to help wherever we can.

A backtrace would be helpful.

@gwenn - I'll ask and see if I can get a backtrace

I've tracked down the specific issue to here: https://github.com/kkawakam/rustyline/blob/32f87be475234c7ffbdf2ecb603411423c37eeaf/src/tty/windows.rs#L514-L516
The problem is that, when running in a terminal that isn't cmd.exe (like Windows Terminal), set_console_cursor_position returns an error code 87, which matches ErrorKind::InvalidInput. It appears that errno 87 is always InvalidInput (per https://github.com/rust-lang/rust/blob/master/library/std/src/sys/windows/mod.rs#L64), so I think this line needs to be changed. I'll submit a PR making the change.

v6.3.0 released

authors: Lucretiel gwenn jonathandturner

repo: kkawakam/rustyline


text:
ExternalPrinter can't handle ASCII control character

I am using rustyline version 11.0.0
I want use ExternalPrinter to print string that contains some ASCII control character
if I write the following code, it output a single character t. this make since because \x08 is a backspace.
fn main(){
    let mut rl = rustyline::DefaultEditor::new().unwrap();
    let mut printer = rl.create_external_printer().unwrap();
    printer.print("test\x08\x08\x08".to_owned()).unwrap();
}

but if I instead using the following code, ExternalPrintetr output all four character test.
and I expect it output just a t
fn main(){
    let mut rl = rustyline::DefaultEditor::new().unwrap();
    let mut printer = rl.create_external_printer().unwrap();
    thread::spawn(move || {
        printer.print("test\x08\x08\x08".to_owned()).unwrap();
    });
    let line = rl.readline(">>").unwrap();
}

why ASCII control character didn't work in ExternalPrinter that I use it in another thread?
and how can I make ExternalPrinter handle ASCII control character?

Raw mode is activated  only while reading / readline(">>") is called. And deactivated when readline(">>") returns.
And some escape sequences are interpreted only in raw mode.
I guess replxx behaves the same.

Same behavior with linefeed:
diff --git a/examples/demo.rs b/examples/demo.rs
index a7ad87c..4754cac 100644
--- a/examples/demo.rs
+++ b/examples/demo.rs
@@ -90,8 +90,8 @@ fn main() -> io::Result<()> {
                     let mut rng = thread_rng();
                     let mut i = 0usize;
                     loop {
-                        writeln!(iface, "[#{}] Concurrent message #{}",
-                            my_thread_id, i).unwrap();
+                        writeln!(iface, "test\x08\x08\x08",
+                            ).unwrap();
                         let wait_ms = rng.gen_range(1, 500);
                         thread::sleep(Duration::from_millis(wait_ms));
                         i += 1;

linefeed % cargo run --example demo
demo> spawn-log-thread
Spawning log thread #0
test
test
test
test
test
demo>
Goodbye.

But if I add one character like "test\x08\x08\x08!"
I get "t!st" with both rustyline and replxx...

do we have any plan to add the feature that enable ASCII control character in external printer?
or maybe I just misunderstanding the usage of external printer. because it by design to print plain text info...

To print plain or styled text

tested with this bash script:
#!/bin/bash
input="./test.txt"
while IFS= read -r line
do
echo -e "$line"
done < "$input"
test.txt is just one line:
test\b\b\b!
yield (as  rustyline):
t!st

is there something like the external printer that supports ansii?

authors: Outfluencer gwenn michabs yalikes


text:
Change default behavior of .zprintrc/:search-config?

Hey there!
I feel like the current workflow for using a given project's .zprintrc doesn't work well when dealing with multiple open source projects and contributors.
I'm trying to use zprint in a community project and finding it hard to get everyone on board with setting up $HOME/.zprintrc (that says {:search-config? true}). I know a solution is to move the options into a (z/set-options! ...) call, but that then requires either always requiring that namespace everywhere zprint is used or creating some wrapper around zprint and then only using our wrapper (which then loads the options) or setting the options at the top of every file where zprint is used (and then keeping all of those in sync somehow). Another solution I just thought of is to move the file into resources/ and then read it with (clojure.edn/read-string (slurp (io/resource ".zprintrc"))) and then use that in my set-options! call, but that likewise feels like quite the complex set of steps for something intended to be "simple".
The solution folks at my job found (before I joined) is to copy the entirety of our .zprintrc to the user's home directory in our Dockerfile lol: COPY .zprint.edn /home/$USER/.zprint.edn.
Given all of that, I would like it if zprint used :search-config? for the opposite purpose: if :search-config? is false, only use $HOME/.zprintrc; otherwise, use the project's .zprintrc. This would more closely align zprint with other similar formatting tools, such as prettier/black/rustfmt/etc. It would also allow for contributors to open source projects to correctly format the project's code without having to touch files outside of the project directory.

Thanks for asking, I appreciate the opportunity to try to figure out a solution for you.
I believe I understand what you would like to have changed, but I don't entirely understand the way that you are using zprint.  Most people use zprint either from the pre-built binaries, the uberjar, or lein-zprint.  But you mentioned (set-options! ...), which doesn't fit that usage model.  It would help me a lot if I had some additional overall context for how you are using zprint in your environment.  In other words, I'd like to understand your workflow in more detail.
That said, I have two things to offer regarding your current proposal:

The {:search-config? true} capability came as an addition some years after zprint was first released.  Since it didn't exist initially, I wasn't willing to make a behavior change that would affect people already using zprint in a way that they might not expect.  That remains true today, though I have no idea how many people would actually be affected by such a change.  I agree that it would have been better if zprint had first been released with {:search-config? true} as the default behavior.

If you are using the uberjar, or the pre-built binaries (or lein-zprint, though the approach would be different) for your formatting, you can get the `{:search-config? true}' behavior by specifying it on the command line:

./zprintm-1.2.4 '{:search-config? true}' <helloworld.clj

This is equivalent to having {:search-config? true} in your ~/.zprintrc.
That may or may not help you, but I can imagine an alias or shell script that could hide the {:search-config? true} from people and still let them have their own options maps on the command line.
As I recall, I fixed this to work in late 2019 or early 2021, so if you are using some zprint from before 1.0.0, this certainly won't work. I think I've fixed some minor problems with it since then as well, so you really want to be using a current version of zprint.
All that said, I would be very interested in learning more about your particular workflow so that I can perhaps offer some solution to the problem you are encountering which helps and which doesn't change the current default behavior.
Thanks for bringing this up!

I kind of mushed together multiple scenarios, so let me rephrase.
At my job we use zprint as a cli tool "as intended", so this issue doesn't come up a lot. We include in the "New Hire" docs instructions to add the "search-config" stub to the dev's HOME directory, and do some docker magic to make sure the same file is available for CI. This is very slightly annoying but overall doesn't affect anything.
In an open source project I maintain, we occasionally process csvs into individual edn files. We have roughly 2500 edn of varying size. We use zprint as a library to correctly format the edn files at the time of creation using zprint-str. For a couple years I was the only person who contributed so I never noticed that if someone else contributed and they don't have the "search-config" stub in their HOME directory, when they run the "consumer csv" command, it outputs edn that doesn't conform to the zprintrc file in the root of the repo. This is more annoying lol, as it requires modifying a contributor's HOME directory for a project they might only contribute once to.

So, for your open source project, I have two thoughts:

If there is a single entry point (or maybe one or two), you can call (set-options! ...) with whatever configuration you want zprint to have early on in the code, and then every subsequent call to zprint will have that configuration.  Then you might not need the .zprintrc in the project directory.  Even more interesting in this application (I would think), if your first call to zprint is this: (set-options! {:configured? true}), then zprint will not look at any external files.  Then if your second  call to (set-options! ...) has the configuration you want, nobody else's personal zprint configuration will affect what your use of the zprint library will see as configuration, and all of the calls to zprint-str will use that configuration. (Actually, I think that you might be able to put :configured? true in with your options in the first call, and it will still work, but I don't remember for sure and I don't have time to test it just now.  I know that two back-to-back calls will work!)
If there isn't a single code path at entry where you can call (set-options! ...), then you could define a map with the configuration you want zprint to have.  Then in all of your calls to zprint, you just put the name of that (def ...) as the third parameter of the call to zprint.  Again, you don't have to have a project .zprintrc if you do this.

It seems like one of these approaches would solve your problem, and the first one (if you can do it), would also isolate the zprint format from anyone's personal zprint configuration, which would seem like a good thing in general.
I hope that this helps!

Thanks for the reply and ideas.
The first one doesn't work quite as well because there are multiple Enter points. Sometimes we generate the edn files from a single csv, sometimes we use another namespace that can process some json files, sometimes we use the existing edn to build new edn, lol. It's a mess honestly, but so far it works well.
What this means is that I can't trust that a given namespace will call set-options! unless, like you suggest in your second idea, I dedicate a namespace to zprint config and pull that in in every namespace where zprint is used.
And to follow up on that, if I use such a namespace, I cut myself off from being able to seamlessly use the same configuration at the command line as well as in the code. I like to sanity check the edn files by running zprint from the command line sometimes (after manually editing the files or pulling in someone else's changes), as we don't have CI set up yet. If we did set it up, we'd deal with this same issue.
All that sounds doom and gloom but it's overall pretty small annoyance lol. Would just be nice to have this change (or something similar) to unify everything.
Thanks so much for the in-depth back and forth.

One more thought.  If you use leiningen, you can put :search-config? true is the :zprint key in your project.clj.  Then if you were to somehow create commands that were invoked with or through lein ..., then I think you might get what you want.  Maybe.

authors: NoahTheDuke kkinnear

repo: kkinnear/zprint


text:
Persistent values after reboot for sensor_total_energy and sensor_total_daily_energy

Changes to pulse_meter, sensor_total_energy, and sensor_total_daily_energy to restore their previous values after a reboot
Also adds on_boot flag to esphome which calls pulse_meter.set_total_pulses to manually restore the value as pulse_meter total doesn't have a restore flag.
Finally, added preferences to restrict flash writes to once every 5 minutes, to reduce wear on flash memory.
Tested on ESP32 nodemcu-32s only.

PS: apologies for the excessive number of commits, I haven't done this before.

I don't have any ESP8266 devices available for testing - are you able to make the necessary adjustments?

repo: klaasnicolaas/home-assistant-glow


text:
jitpack.io distribution

It seems to be pretty straightforward to push the artifact to the github based maven repository - https://jitpack.io/#klausbetz/apple-identity-provider-keycloak/1.4.0.
First build failed https://jitpack.io/com/github/klausbetz/apple-identity-provider-keycloak/1.4.0/build.log:
Build starting...
Start: Tue Jan 17 18:54:24 UTC 2023 f52ae56bcf69
Git:
1.4.0-0-g09a2721
commit 09a2721e425bcfe54eba298cb2925c857608fd1c
Author: Klaus Betz
Date:   Fri Dec 9 12:35:52 2022 +0100

    feat(#22): support id-token exchange (#23)

Init SDKMan
Found gradle
Gradle build script
Found gradle version: 7.4.2.
Using gradle wrapper
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Downloading https://services.gradle.org/distributions/gradle-7.4.2-bin.zip
.10%.20%.30%.40%.50%.60%.70%.80%.90%.100%

------------------------------------------------------------
Gradle 7.4.2
------------------------------------------------------------

Build time:   2022-03-31 15:25:29 UTC
Revision:     540473b8118064efcc264694cbcaa4b677f61041

Kotlin:       1.5.31
Groovy:       3.0.9
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          1.8.0_292 (Private Build 25.292-b10)
OS:           Linux 4.14.63-xxxx-std-ipv6-64 amd64

0m3.129s
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~16.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
Getting tasks: ./gradlew tasks --all
Tasks:
Found javadoc task

 ⚠   WARNING:
 Gradle 'publishToMavenLocal' task not found. Please add the 'maven-publish' or 'maven' plugin.
 See the documentation and examples: https://docs.jitpack.io

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
/home/jitpack/build/build.gradle has component: java
Running: ./gradlew clean -Pgroup=com.github.klausbetz -Pversion=1.4.0 -xtest publishToMavenLocal
> Task :clean UP-TO-DATE
> Task :compileJava FAILED
2 actionable tasks: 1 executed, 1 up-to-date
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> invalid source release: 11

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 3s
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Archive: at.klausbetz:apple-identity-provider:1.4.0
Publication: at.klausbetz:apple-identity-provider:1.4.0
Build tool exit code: 0
Looking for artifacts...
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2
Looking for pom.xml in build directory and ~/.m2
Found artifact: at.klausbetz:apple-identity-provider:1.4.0
2023-01-17T18:54:39.969509875Z
Exit code: 0

⚠ ERROR: No build artifacts found
Expected artifacts in: $HOME/.m2/repository/at/klausbetz/apple-identity-provider/1.4.0

For our corporate usage we import the project to internal gitlab and deploy it to our internal repository by tweaking build.gradle:

plugins {
    id 'java'
    id 'org.hibernate.build.maven-repo-auth' version '3.0.3' // use mvn settings.xml for nexus auth
    id 'maven-publish'
}

group 'at.klausbetz'

test {
    useJUnitPlatform()
}

java {
    withSourcesJar()
}

publishing {
    publications {
        myJava(MavenPublication) {
            from components.java
        }
    }

    repositories {
        maven {
            name 'nexus-releases'
            url "https://example.com/repository/maven-releases/"
        }
    }
}

This could be a great improvement!
Would you like to create a pull-request? I'd be more than happy to see you contribute to this extension 😄

@klausbetz PR is ready. The dependency in Maven will look like
	<dependency>
	    <groupId>com.github.klausbetz</groupId>
	    <artifactId>apple-identity-provider-keycloak</artifactId>
	    <version>1.5.0</version>
	</dependency>

For custom group id see https://jitpack.io/docs/#custom-domain-name.
I added a job for tags only - as soon as you create a release this job is proactively initiating a jitpack build.

Closed with #36

authors: EvgeniGordeev klausbetz


text:
Feature: Delete all checked tasks

Added the options "clear" and "clean" to delete all tasks that are checked.
Based on the feature request #40.

#43 is a better solution because it has no redundant code.

repo: klauscfhq/taskbook


text:
Does this have a URL?

Hello I am kind of new to all this.
So does this application have an URL and if it does where can I find it?
So far I have a great time with this app!
Thanks! :)

https://github.com/pythonInRelay/ao/releases
and follow instructions

repo: klaussinani/ao


text:
Feature/monaco

🎉 New feature
Closes #17

Had to recreate the pull request, because the forc sync bot screwed up my master branch after your last commit 😅

repo: klawr/mqtt-inspector


text:
Soramitsu X Klaytn Testnet

So far I haven't found any problems or bugs. maybe you need to add night mode feature.
the night mode feature is very important in my opinion. because the web or other applications now use this feature

@degencrypto27 Thank you for your suggestions, we've taken your request and noted it within our potential enhancement document list!

repo: klaytn/klaytn-dex-frontend


text:
dynamic window parameters

this problem can be solved both for scrumfile/scrumrc and scrumfile.json approach by setting and using $SCRUM_ARGUMENT_NAME / $SCRUM_ARGS_ARGUMENT_NAME environment variable in shell.
e.g.

node executes scrum start services -w asdqwe
sets $SCRUM_W = asdqwe
screen executes -c in given shell with "-w $SCRUM_W" in -c
service starts with -w asdqwe

this is an established pattern given use of $USER in -c string already.

implemented as of c3cde9ccd586ecf60e781d2246d4add6b258905f using handlebars and portions of process and process.env as well as unclaimed arguments to scrum (all scrum arguments used by a command are deleted, those that remain are forwarded to handlebars as a property on the arg object)

repo: kmyr/scrum


text:
Unsolicited response error due to lost session ID

saml2 complains that it cannot find the outstanding query which was saved to cache by djangoSaml2. Is there a configuration option I am missing? Below are relevant log messages:

DEBUG /usr/local/lib/python3.5/dist-packages/djangosaml2/views.py:250 views: Saving the session_id in the OutstandingQueries cache: id-ZEV53y7Cx2PoA3Pfe
...
/ERROR /usr/local/lib/python3.5/dist-packages/saml2/response.py:540 response: Unsolicited response id-ZEV53y7Cx2PoA3Pfe not found in {}

I altered saml2 error message to print the self.outstanding_queries being searched which turns up empty.
Am I missing some configuration, or is this a bug?
IDP is Microsoft azure I have no access to however is returning a successful login before this error.

Hi,
Please do not use py35 if possible, at least py36.
Try to configure allow_unsolicited_response in pysaml2 configuration and then let them know

Looking into the logs more I believe djangoSaml2 is posting two requests, i assume the OutstandingQuery is removed from the cache once the first response is recieved and its the second response with the ID which has since been cleared from the OutstandingQueries cache which causes the error. Could this be an issue caused by DjangoSaml2 sending two requests rather than just the one?
DEBUG /usr/local/lib/python3.5/dist-packages/djangosaml2/views.py:99 views: Login process started
DEBUG /usr/local/lib/python3.5/dist-packages/saml2/mdstore.py:623 mdstore: service => {'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST': [{'location': 'https://login.microsoftonline.com/[redacted]/saml2', '__class__': 'urn:oasis:names:tc:SAML:2.0:metadata&SingleSignOnService', 'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST'}], 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect': [{'location': 'https://login.microsoftonline.com/[redacted]/saml2', '__class__': 'urn:oasis:names:tc:SAML:2.0:metadata&SingleSignOnService', 'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect'}]}
DEBUG /usr/local/lib/python3.5/dist-packages/djangosaml2/views.py:164 views: Trying binding urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect for IDP https://sts.windows.net/[redacted]/
DEBUG /usr/local/lib/python3.5/dist-packages/saml2/mdstore.py:1075 mdstore: service(https://sts.windows.net/[redacted]/, idpsso_descriptor, single_sign_on_service, None)
DEBUG /usr/local/lib/python3.5/dist-packages/saml2/mdstore.py:623 mdstore: service => {'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST': [{'location': 'https://login.microsoftonline.com/[redacted]/saml2', '__class__': 'urn:oasis:names:tc:SAML:2.0:metadata&SingleSignOnService', 'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST'}], 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect': [{'location': 'https://login.microsoftonline.com/[redacted]/saml2', '__class__': 'urn:oasis:names:tc:SAML:2.0:metadata&SingleSignOnService', 'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect'}]}
WARNING /usr/local/lib/python3.5/dist-packages/saml2/client_base.py:192 client_base: The SAML service provider accepts unsigned SAML Responses and Assertions. This configuration is insecure.
DEBUG /usr/local/lib/python3.5/dist-packages/djangosaml2/views.py:187 views: Redirecting user to the IdP via urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect binding.
DEBUG /usr/local/lib/python3.5/dist-packages/saml2/mdstore.py:1075 mdstore: service(https://sts.windows.net/[redacted]/, idpsso_descriptor, single_sign_on_service, urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect)
DEBUG /usr/local/lib/python3.5/dist-packages/saml2/mdstore.py:623 mdstore: service => [{'location': 'https://login.microsoftonline.com/[redacted]/saml2', '__class__': 'urn:oasis:names:tc:SAML:2.0:metadata&SingleSignOnService', 'binding': 'urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect'}]
INFO /usr/local/lib/python3.5/dist-packages/saml2/client.py:111 client: destination to provider: https://login.microsoftonline.com/[redacted]/saml2
INFO /usr/local/lib/python3.5/dist-packages/saml2/entity.py:486 entity: REQUEST: <samlp:AuthnRequest xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" AssertionConsumerServiceURL="https://[redacted].net/saml2/acs/" Destination="https://login.microsoftonline.com/[redacted]/saml2" ID="id-ZEV53y7Cx2PoA3Pfe" IssueInstant="2020-07-21T11:04:29Z" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Version="2.0"><saml:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">https://[redacted].net/saml2/metadata/</saml:Issuer></samlp:AuthnRequest>
INFO /usr/local/lib/python3.5/dist-packages/saml2/client.py:120 client: AuthNReq: <samlp:AuthnRequest xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" AssertionConsumerServiceURL="https://[redacted].net/saml2/acs/" Destination="https://login.microsoftonline.com/[redacted]/saml2" ID="id-ZEV53y7Cx2PoA3Pfe" IssueInstant="2020-07-21T11:04:29Z" ProtocolBinding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Version="2.0"><saml:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">https://[redacted].net/saml2/metadata/</saml:Issuer></samlp:AuthnRequest>
INFO /usr/local/lib/python3.5/dist-packages/saml2/entity.py:226 entity: HTTP REDIRECT
DEBUG /usr/local/lib/python3.5/dist-packages/djangosaml2/views.py:250 views: Saving the session_id in the OutstandingQueries cache: id-ZEV53y7Cx2PoA3Pfe
WARNING /usr/local/lib/python3.5/dist-packages/saml2/client_base.py:192 client_base: The SAML service provider accepts unsigned SAML Responses and Assertions. This configuration is insecure.

DEBUG /usr/local/lib/python3.5/dist-packages/saml2/response.py:325 response: xmlstr: <samlp:Response ID="_d4561ed6-9bf6-4c11-af1c-5e3841fc792a" Version="2.0" IssueInstant="2020-07-21T11:04:30.116Z" Destination="https://[redacted].net/saml2/acs/" InResponseTo="id-ZEV53y7Cx2PoA3Pfe"  ...</samlp:Response>
DEBUG /usr/local/lib/python3.5/dist-packages/saml2/response.py:325 response: xmlstr: <samlp:Response ID="_d4561ed6-9bf6-4c11-af1c-5e3841fc792a" Version="2.0" IssueInstant="2020-07-21T11:04:30.116Z" Destination="https://[redacted].net/saml2/acs/" InResponseTo="id-ZEV53y7Cx2PoA3Pfe" ...</samlp:Response>

DEBUG /usr/local/lib/python3.5/dist-packages/saml2/response.py:291 response: response: <samlp:Response xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Destination="https://[redacted].net/saml2/acs/" ID="_d4561ed6-9bf6-4c11-af1c-5e3841fc792a" InResponseTo="id-ZEV53y7Cx2PoA3Pfe" ...</samlp:Response>
ERROR /usr/local/lib/python3.5/dist-packages/saml2/response.py:540 response: Unsolicited response id-ZEV53y7Cx2PoA3Pfe not found in {}

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/saml2/entity.py", line 1143, in _parse_response
    response = response.loads(xmlstr, False, origxml=xmlstr)
  File "/usr/local/lib/python3.5/dist-packages/saml2/response.py", line 520, in loads
    self._loads(xmldata, decode, origxml)
  File "/usr/local/lib/python3.5/dist-packages/saml2/response.py", line 340, in _loads
    **args)
  File "/usr/local/lib/python3.5/dist-packages/saml2/sigver.py", line 1650, in correctly_signed_response
    raise SignatureError('Signature missing for response')
saml2.sigver.SignatureError: Signature missing for response
ERROR /usr/local/lib/python3.5/dist-packages/saml2/client_base.py:718 client_base: XML parse error: Unsolicited response: id-ZEV53y7Cx2PoA3Pfe
ERROR /usr/local/lib/python3.5/dist-packages/djangosaml2/views.py:319 views: Received SAMLResponse when no request has been made.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/saml2/entity.py", line 1143, in _parse_response
    response = response.loads(xmlstr, False, origxml=xmlstr)
  File "/usr/local/lib/python3.5/dist-packages/saml2/response.py", line 520, in loads
    self._loads(xmldata, decode, origxml)
  File "/usr/local/lib/python3.5/dist-packages/saml2/response.py", line 340, in _loads
    **args)
  File "/usr/local/lib/python3.5/dist-packages/saml2/sigver.py", line 1650, in correctly_signed_response
    raise SignatureError('Signature missing for response')
saml2.sigver.SignatureError: Signature missing for response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/djangosaml2/views.py", line 296, in post
    response = client.parse_authn_request_response(xmlstr, BINDING_HTTP_POST, outstanding_queries)
  File "/usr/local/lib/python3.5/dist-packages/saml2/client_base.py", line 711, in parse_authn_request_response
    binding, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/saml2/entity.py", line 1153, in _parse_response
    response = response.loads(xmlstr, False, origxml=xmlstr)
  File "/usr/local/lib/python3.5/dist-packages/saml2/response.py", line 542, in loads
    "Unsolicited response: %s" % self.in_response_to)
saml2.response.UnsolicitedResponse: Unsolicited response: id-ZEV53y7Cx2PoA3Pfe
ERROR /usr/local/lib/python3.5/dist-packages/saml2/response.py:540 response: Unsolicited response id-ZEV53y7Cx2PoA3Pfe not found in {}

Hi,
Please do not use py35 if possible, at least py36.
Try to configure allow_unsolicited_response in pysaml2 configuration and then let them know

Will upgrade py and try, thank you

I see a single authnRequest there, you can even audit for that using samltracer, on firefox or its equivalent for chrome.
I read that warning about unsigned response, add authn_response_signed as follows
https://github.com/peppelinux/Django-Identity/blob/master/djangosaml2_sp/djangosaml2_sp/djangosaml2_sp/sp_pysaml2_satosa.py#L74
Put saml2.client_base in debug (in settings LOGGING).
Just play with it a bit more, then you'll be able to understand what's happening.

I have the same issue with Python 3.7.6.
DEBUG Trying binding urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect for IDP https://sso.****.***/idp/shibboleth
DEBUG Redirecting user to the IdP via urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect binding.
DEBUG Saving the session_id in the OutstandingQueries cache
[pid: 19882|app: 0|req: 23/27] 10.222.90.79 () {58 vars in 1566 bytes} [Wed Jul 22 13:02:24 2020] GET /saml2/login/ => generated 39 bytes in 30 msecs (HTTP/1.1 302) 7 headers in 830 bytes (2 switches on core 0)
DEBUG Login process started
DEBUG Trying binding urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect for IDP https://sso.****.***/idp/shibboleth
DEBUG Redirecting user to the IdP via urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect binding.
DEBUG Saving the session_id in the OutstandingQueries cache
[pid: 19882|app: 0|req: 24/28] 10.222.90.79 () {56 vars in 1542 bytes} [Wed Jul 22 13:02:25 2020] GET /saml2/login/ => generated 39 bytes in 18 msecs (HTTP/1.1 302) 7 headers in 830 bytes (2 switches on core 0)
Unsolicited response id-TEqwj2KhDl88czDDq
NoneType: None
Unsolicited response
XML parse error: Unsolicited response: id-TEqwj2KhDl88czDDq
ERROR Received SAMLResponse when no request has been made.
Traceback (most recent call last):
  File "/opt/nld/var/pyenv/lib/python3.7/site-packages/djangosaml2/views.py", line 296, in post
    response = client.parse_authn_request_response(xmlstr, BINDING_HTTP_POST, outstanding_queries)
  File "/opt/nld/var/pyenv/lib/python3.7/site-packages/saml2/client_base.py", line 711, in parse_authn_request_response
    binding, **kwargs)
  File "/opt/nld/var/pyenv/lib/python3.7/site-packages/saml2/entity.py", line 1143, in _parse_response
    response = response.loads(xmlstr, False, origxml=xmlstr)
  File "/opt/nld/var/pyenv/lib/python3.7/site-packages/saml2/response.py", line 542, in loads
    "Unsolicited response: %s" % self.in_response_to)
saml2.response.UnsolicitedResponse: Unsolicited response: id-TEqwj2KhDl88czDDq

Not for me, @rotherfuchs are you dealing with ADFS?
I'd investigate in the cache system you're using and in the way these information would be extracted from there.
put a pdb there and follow the stream if you can do that in a debugging environment

The problem is most likely the "SameSite" and HttpOnly cookie restrictions. Basically if you're using the HTTP POST binding, the POST is initiated via JavaScript on the IdP-supplied page which will not include the Django session cookie, thus the outstanding queries being empty because there is no session. You can check that by doing the flow while having the browser's developer tools "network" tab open and then checking the cookies on the /saml2/acs POST request. You should see your session cookie in there for the "outstanding sessions" logic to work but you most likely won't.
Potential solutions:

Set SameSite=None (and the Secure flag since it's required in that case) on the Django session cookie. This might have detrimental security-related side-effects so I wouldn't recommend.

Amending the library to use a different, separate session (with SameSite=None) only for the SAML outstanding queries cache, and keep the default Django session untouched (and secure; with default SameSite settings).

Allowing unsolicited responses in your SAML config. This might have security implications (which I'm trying to determine at the moment as I am facing the exact same issue).

The problem is most likely the "SameSite" and HttpOnly cookie restrictions. Basically if you're using the HTTP POST binding, the POST is initiated via JavaScript on the IdP-supplied page which will not include the Django session cookie, thus the outstanding queries being empty because there is no session. You can check that by doing the flow while having the browser's developer tools "network" tab open and then checking the cookies on the /saml2/acs POST request. You should see your session cookie in there for the "outstanding sessions" logic to work but you most likely won't.
Potential solutions:
1. Set SameSite=None (and the Secure flag since it's required in that case) on the Django session cookie. This might have detrimental security-related side-effects so I wouldn't recommend.

2. Amending the library to use a different, separate session (with SameSite=None) _only_ for the SAML outstanding queries cache, and keep the default Django session untouched (and secure; with default SameSite settings).

3. Allowing unsolicited responses in your SAML config. This might have security implications (which I'm trying to determine at the moment as I am facing the exact same issue).

I found solution 1 ineffective and solution 3 also ineffective.
related: #143

I close this issue and invite all of you to continue it here:
https://github.com/knaperek/djangosaml2/issues/146#issuecomment-666007053
Thanks everyone for the clarifications

Implemented SameSite workaround in v0.30.0

authors: LVMuser Rjevski peppelinux rotherfuchs


text:
[main] Upgrade to latest dependencies

Cron -knative-prow-robot
/cc knative-sandbox/eventing-writers
/assign knative-sandbox/eventing-writers
Produced by: knative-sandbox/knobots/actions/update-deps
Details:
bumping knative.dev/hack 0e0784b...6ffd841:
  > 6ffd841 Update community files (# 168)
  > 02c525c Update community files (# 167)
bumping knative.dev/eventing 58865af...2222324:
  > 2222324 Update actions (# 6323)
  > e7459cb upgrade to latest dependencies (# 6324)
  > e874b89 upgrade to latest dependencies (# 6321)
  > fd10179 Update community files (# 6320)
  > bb76b30 Update community files (# 6310)
  > 5accb7a Update actions (# 6303)
  > 2ff606c upgrade to latest dependencies (# 6307)
bumping knative.dev/pkg 4d62e1d...00c122e:
  > 00c122e Add genreconcile for ConfigMap (# 2489)
  > 6bb6518 Update actions (# 2488)
  > 5b0e728 drop deprecated eventing repos (# 2463)
  > 75629c8 Update community files (# 2487)
  > ca82d2b Add `NewProxyAutoTLSTransport` and `DialTLSWithBackOff` to support TLS proxy (# 2479)
  > e2b4d74 Update community files (# 2486)

/lgtm
/approve

authors: knative-automation lionelvillard

repo: knative-sandbox/eventing-couchdb