text:
WIP: add Chirp Z-transform, zoom FFT

Merging public domain functions written by Paul Kienzle, Nadav Horesh (same as @pkienzle and
@Nadav-Horesh?)
Feature request: https://github.com/scipy/scipy/issues/4288
To do

[x] Should ScaledFFT be removed?
[x] Decide whether to include scale parameter in CZT (Not the same thing as ScaledFFT)
[x] Should it really return a view on a much larger array, or do ascontiguousarray internally?

Docstring was wrong

[x] Returns sections should be removed from class docstrings? How to document that it returns a callable function though?
[x] Should Fs= be capitalized?
[x] The following note is misleading: "However, the chirp z-transform is considerably less precise than the equivalent zero-padded FFT, with differences on the order of 1e-7 from the direct transform rather than 1e-15 as seen with zero-padding."

"I would suggest to simply state that the czt is less precise without giving numbers."

[x] for a, we could calculate the angle first and then project to the complex plane #4607 (comment).
[x] Replacing k**2 by (k**2)%(2*m) should help a lot. #4607 (comment)
[x] from __future__ import division, absolute_import, print_function
[ ] If a test, belongs in the test file. If intended more as a demo/debugging tool, might include it in the documentation somewhere. https://github.com/scipy/scipy/pull/4607#discussion_r25993304
[ ] The tests and demos need to be moved out of the function implementation file. https://github.com/scipy/scipy/pull/4607#issuecomment-77673555
[ ] The demos might be best in a demo tutorial in the documentation. https://github.com/scipy/scipy/pull/4607#issuecomment-77673555
[x] Use assert_(...) for tests.
[x] Probably stick to test_ for the prefix.

These aren't test functions, they are called by test functions

[x] Spaces around +, - operators. Might want to run pep8 on this for style.
[x] Comments instead of docstrings in tests.
[x] Best to have a fixed seed for reproducibility.
[x] Imports need fixing.
[x] How about moving _next_regular and its tests to a new file
[ ] The CZT and ZoomFFT __init__() methods are now duplicating some code, could maybe be deduplicated. https://github.com/scipy/scipy/pull/4607#issuecomment-167426818

Not sure how to do this cleanly or if it's worthwhile

[ ] make sure monkeypatching with pyfftw works https://github.com/scipy/scipy/pull/4607#issuecomment-147018053
[ ] but replacing the calculation of wk with this seems to help https://github.com/scipy/scipy/pull/4607#discussion_r63880528
[x] The calculation in the w is None branch definitely can be replaced by wk2 = np.exp(-1j * pi * ((kk) % (2m)) / m) to increase accuracy.
[x] That also has averaging and windowing etc that aren't included here.
[x] I believe adding the word callable makes this crystal clear. Same for ZoomFFT
[ ] Since we have an explicit expression for a, we could calculate the angle first and then project to the complex plane, i.e. something like https://github.com/scipy/scipy/pull/4607#discussion_r506251925
[x] ERROR: objects in scipy.fft.all but not in refguide::
[x] When accessing scipy.fft.doc, you will see these descriptions.
[x] should I change them to use assert_allclose?
[ ] For every "full circle" made in the exponent, the error of the result grows by roughly 1e-16. This can only be fixed by perfect range reduction https://github.com/scipy/scipy/pull/4607#issuecomment-712741842
[ ] Just to illustrate my point about the complex exponentials: https://github.com/scipy/scipy/pull/4607#issuecomment-712747249
[ ] add an rtol parameter to check_czt and check_zoomfft https://github.com/scipy/scipy/pull/4607#issuecomment-712772808
[ ] use check_czt and check_zoomfft exclusively instead of manually calling fft and czt https://github.com/scipy/scipy/pull/4607#issuecomment-712772808
[ ] set the tolerance dynamically, based on the length of the transform https://github.com/scipy/scipy/pull/4607#issuecomment-712772808
[x] a view on a much larger array is because it literally says so in the docstring:
[x] could be confusing to users to have the class-based and function-based entities differ only by capitalization.
[x] docstrings could be more consistent in language. I.e., zoomfft describes x as "The input signal" while czt uses "the signal to transform."
[x] axis argument is described in terms of "dimension," but in places like fft.fft axis is described as "Axis over which to compute the FFT. If not given, the last axis is used."
[x] czt_points should get "See Also" links to czt and CZT
[x] The functions and classes should get "See Also" links to their counterparts.
[x] There are some formatting tricks to get arguments mentioned in the "Notes" section to show up like x, this would help the ZoomFFT "Notes" section in particular
[x] It's not clear to me what is implied by referring to ZoomFFT as "friendlier" in the CZT "See Also" section. More efficient?
[x] fftshift isn't used anywhere.
[x] I would just call it a "callable class instance". See for example interp1d which operates this way
[x] How about having a parameter endpoint=False like numpy.linspace?
[ ] Decision: endpoint= defaults to true or false?? https://github.com/scipy/scipy/pull/4607#issuecomment-739552809
[ ] Decision: Zoom FFT would fit better in scipy.signal than in scipy.fft / are these transforms better suited in signal alongside other spectral transforms e.g. cwt and stft? https://github.com/scipy/scipy/pull/4607#issuecomment-739552444
[x] Make sure this is compatible with future more accurate PR
[ ] PEP8: snake_case for zoom_fft() would technically be more correct? https://github.com/scipy/scipy/pull/4607#issuecomment-739552575
[x] DOC: make sure bluestein implementation vs CZT transform is clear in the documentation I think it would be correct to say that Bluestein's algorithm is an implementation of CZT.
[x] DOC: This paragraph is obsolete.
[ ] DOC: Any references we can add to ZoomFFT for this implementation? 	reference for Zoom FFT using CZT  clarify steps in comment https://github.com/scipy/scipy/pull/4607#issuecomment-715671479
[x] DOC: CZT is the only one with reference information
[x] DOC: The function versions don't have examples; the ones from the classes can be copied over.
[ ] DOC: The czt vs fft example is probably no longer true with pocketfft: https://github.com/scipy/scipy/pull/4607#issuecomment-711037717
[ ] DOC: ZoomFFT is more accurate because it involves fewer operations.  https://github.com/scipy/scipy/pull/4607#issuecomment-737444787
[ ] DOC: So the "pre-compiled" version is actually a bit faster than pocketfft? https://github.com/scipy/scipy/pull/4607#issuecomment-716056616
[ ] DOC: So the "pre-compiled" version is faster than truncating an oversampled FFT, so that's good. https://github.com/scipy/scipy/pull/4607#issuecomment-716056616
[ ] TST: I can't see the w parameter tested anywhere here even though it's being looped over. https://github.com/scipy/scipy/pull/4607#discussion_r523778355

Possible future PRs

as well as w=(rad, nom, denom) or w=(rad, Fraction(nom, denom)) [different PR]

However, I think the parameter a should be split into modulus and phase, too, for consistency.

ZoomFFT object should have a .fftfreq() equivalent method. [different PR]
Add ScaledFFT?

The tests and demos need to be moved out of the function implementation file. The demos might be best in a demo tutorial in the documentation.

Nice to see this.

Yes, I was just posting what I've done so far if someone else wants to work on it.   Do you agree it should be in its own file like this?

I think being its own file is fine.

I'm not sure about this, but I think there's a bug.  All the tests only compare to FFT along the unit circle, they don't go off into the other parts of the Z plane, so I was trying to write tests for that.
Z-transform of an impulse (δ[n]) is 1 everywhere, so we should expect 1s no matter where we sample the Z plane.  This works as expected:
czt([1])
Out[269]: array([ 1.+0.j])

czt([1], m=10)
Out[271]:
array([ 1.+0.j,  1.+0.j,  1.+0.j,  1.-0.j,  1.-0.j,  1.-0.j,  1.+0.j,
        1.+0.j,  1.+0.j,  1.+0.j])

czt([1], m=10, a=2)
Out[272]:
array([ 1.+0.j,  1.+0.j,  1.+0.j,  1.-0.j,  1.-0.j,  1.-0.j,  1.+0.j,
        1.+0.j,  1.+0.j,  1.+0.j])

czt([1], m=10, a=.0000001)
Out[274]:
array([ 1.+0.j,  1.+0.j,  1.+0.j,  1.-0.j,  1.-0.j,  1.-0.j,  1.+0.j,
        1.+0.j,  1.+0.j,  1.+0.j])

czt([1, 0, 0, 0, 0], m=10, a=.0000001, w=1+1j)
Out[277]:
array([ 1.+0.j,  1.+0.j,  1.+0.j,  1.+0.j,  1.+0.j,  1.+0.j,  1.-0.j,
        1.+0.j,  1.+0.j,  1.-0.j])

The Z-transform of an impulse delayed by 1 sample (δ[n-1] = [0, 1, 0, 0, ...]) is z⁻¹, so the output should just be the sample coordinate inverted:
czt_points(m=10, a=2.3)**-1
Out[283]:
array([ 0.4348-0.j    ,  0.3517-0.2556j,  0.1344-0.4135j, -0.1344-0.4135j,
       -0.3517-0.2556j, -0.4348+0.j    , -0.3517+0.2556j, -0.1344+0.4135j,
        0.1344+0.4135j,  0.3517+0.2556j])

czt([0, 1], m=10, a=2.3)
Out[284]:
array([ 0.4348+0.j    ,  0.3517-0.2556j,  0.1344-0.4135j, -0.1344-0.4135j,
       -0.3517-0.2556j, -0.4348-0.j    , -0.3517+0.2556j, -0.1344+0.4135j,
        0.1344+0.4135j,  0.3517+0.2556j])

but that doesn't happen at integer distances:
czt_points(m=10, a=2)**-1
Out[285]:
array([ 0.5000-0.j    ,  0.4045-0.2939j,  0.1545-0.4755j, -0.1545-0.4755j,
       -0.4045-0.2939j, -0.5000+0.j    , -0.4045+0.2939j, -0.1545+0.4755j,
        0.1545+0.4755j,  0.4045+0.2939j])

czt([0, 1], m=10, a=2)
Out[286]:
array([ 0.+0.j,  0.+0.j,  0.+0.j,  0.-0.j,  0.+0.j,  0.+0.j,  0.+0.j,
        0.-0.j,  0.+0.j,  0.+0.j])

I don't know if my understanding is wrong or the code is wrong.

Ok so

originally they had

     If w is complex, it is the ratio between points in each step.
     If w is float, it serves as a frequency scaling factor.

I thought this was weird to behave differently based on type, so I split the float w off to another parameter, which I called factor
Then found I could make those (most common) FFT-like transforms more accurate by generating their points directly instead of iteratively.
Then converted ZoomFFT and ScaledFFT to use the factor method instead, for more accuracy.
Then after working through all this, I've realized this factor parameter does exactly the same thing as the scale parameter in ScaledFFT, and renamed it to scale.

But now CZT(n, m, scale=something) does the same thing as ScaledFFT(n, m, something).  Should ScaledFFT be kept?  I never really understood the point of it.  It says its intended use is "in a convolution of two signals, each has its own sampling step."  But it seems that even in the original code, this could have been done just by setting w to a float.

Rebased and pushed what I have so far.

Tests are failing, I don't understand why.

It would seem the build error is from the line
from scipy.signal.signaltools import _next_regular

Because signaltools is not a public module?  I wasn't sure if I was doing that correctly.  But I don't understand why that would cause from scipy import fftpack to fail.

Because signaltools is not a public module?

The wonders of imports:) Without looking at it in detail I'll point out that it's undesirable anyway to import anything from signal in fftpack. fftpack is lower-level, so this is not good for import time, circular dependencies, etc. How about moving _next_regular and its tests to a new file scipy/_lib/_numutils.py and using it from there in both fftpack and signal. That decouples things nicely.

Actually, using composites of {2, 3, 5} is specific to the FFTPACK library, so maybe it belongs in scipy.fftpack?
Looking at pyfftw (which I have not used) it gives an example of speeding up fftconvolve by overwriting scipy's fft with fftw's.
# Monkey patch in fftn and ifftn from pyfftw.interfaces.scipy_fftpack
scipy.signal.signaltools.fftn = pyfftw.interfaces.scipy_fftpack.fftn
scipy.signal.signaltools.ifftn = pyfftw.interfaces.scipy_fftpack.ifftn

In this case, you would get better performance by padding your arrays to a composite of {2, 3, 5, 7, 11, 13}.
Similarly, reikna.fft.FFT wants power of 2 sizes only, though they don't mention monkey-patching.
So it would make sense to have a scipy.fftpack.next_nice_fft_size() or something, which could also be patched along with the fft() functions when using alternative fft implementations?  Any opinion on this, @hgomersall?

Hello everybody, and thank you for your work. I would really like to see the czt merged in the next scipy release as I often use it in my workflow and now I'm currently forced to use octave. Is there something I can do to help getting it merged?

Apologies, I missed this thread entirely. I've not actually profiled FFTW for varying FFT lengths.
Are you suggesting incorporating alternative padding sizes into e.g. scipy.signal.fftconvolve?
There's an interesting question here regarding where the optimisation should lie. The interfaces function itself is fairly conservative, doing array twiddling and copying more than it might in order to maintain robust adherence to the output from numpy/scipy (subject to a small caveat or two). By far the best option though is to write a proper algorithm around pyFFTW that minimises problems. Even though, however, things aren't clear cut...
In the case of fftconvolve, this may be of interest: http://stackoverflow.com/a/15165621/709852
It's a short fftconvolve routine that tests and plots a few timings for my pyfftw implementation and the scipy implementation. In cases where N does not have useful factors, it seems the padding to powers of two can win substantially.
In the broader case, the question is, where should the optimisation lie? Does it suggest padding is the right thing to do and use FFTW? But there are some odd cases in above code that fly compared to scipy and others that don't and it's really hard to know in advance which those are.
The correct answer is probably to leave it up to the user and explain that optimisation is about understanding, trade-offs and testing.

@goriccardo  Can you read through the API and this post https://github.com/scipy/scipy/pull/4607#issuecomment-123551005 and comment on it?
@hgomersall fftconvolve already pads to an optimal size for FFTPACK (multiples of 2 3 5), but the optimal sizes for other implementations like FFTW are different, so my question is how to create a best_pad_size function that can be monkey patched along with the fft itself, that functions like fftconvolve can then automatically pad to a good length for the underlying implementation.

from your SO post:

(it doesn't help that fftconvolve forces the transform size to powers of two, otherwise it could be monkey patched).

It doesn't anymore.  It calls next_regular and gets a multiple of 2, 3, 5.

@endolith my bad - I think I was right when I wrote it - I've added an edit.
Maybe the best strategy would be to have a simple tuple lookup of acceptable prime factors (if it's empty, leave as is).

well fftw only likes certain prime factors, so a function is better than a list, I think:

FFTW is best at handling sizes of the form 2a 3b 5c 7d 11e 13f, where e+f is either 0 or 1, and the other exponents are arbitrary.  Other sizes are computed by means of a slow, general-purpose routine (which nevertheless retains  O(n lg n) performance, even for prime sizes).  (It is possible to customize FFTW for different array sizes. See Section Installation and Customization, for more information.)  Transforms whose sizes are powers of 2 are especially fast.

Yes, I see - a function makes sense in that case.

@endolith that reference is with respect to real FFTs. I can't see anything equivalent for the complex case. The best I can find in the general case is:
 For example, the standard FFTW distribution works most efficiently for arrays whose size can be factored into small primes (2, 3, 5, and 7)

So currently it will implement it sort of like this:
from scipy.fftpack.helper import next_pad_size

Does that work for your patching?
(Actually, fftconvolve calls numpy's fftpack for real data and scipy's fftpack for complex, so they could conceivably diverge in what sizes work best.  Hmm...
https://github.com/scipy/scipy/issues/2487 would help, but maybe there should still be separate functions for real and complex?)

I'm sure it would work fine. It would just default to using the included next_pad_size by default, which would likely be pretty sensible anyway.

Hi @endolith, the API is fine for me, I really appreciate the ZoomFFT and CZT classes as I often need several transforms of similar signals. Regarding ScaledFFT I think that it does not provide any practical advantage compared to ZoomFFT or CZT but I do not have any strong opinion against it either.

Actually I'm leaning toward keeping ScaledFFT for FFT-like transforms along the unit circle and removing the scale= parameter from CZT, since it's a bit awkward for scale and w to be mutually exclusive parameters (originally that was one parameter that behaved completely differently if it was real vs complex, which I thought was even more awkward).  Then each way to specify the transform has its own class and function.  ScaledFFT might want a better name though?

I completely agree that scale parameter on CZT may be removed. When a
FFT-like transform is needed ZoomFFT and ScaledFFT offer the best API.
However I was wandering if ScaledFFT really offers a significant advantage
over ZoomFFT. Why defining a scale coefficient would be easier than
defining a scaled frequency range? Am I missing something?
2015-11-25 16:03 GMT+01:00 endolith notifications@github.com:

Actually I'm leaning toward keeping ScaledFFT for FFT-like transforms
along the unit circle and removing the scale= parameter from CZT, since
it's a bit awkward for scale and w to be mutually exclusive parameters
(originally that was one parameter that behaved completely differently if
it was real vs complex, which I thought was even more awkward). Then each
way to specify the transform has its own class and function. ScaledFFT
might want a better name though?
—
Reply to this email directly or view it on GitHub
https://github.com/scipy/scipy/pull/4607#issuecomment-159634272.

Well CZT() uses a complex step to define the parameter, which accumulates error as you go around the spiral.  ZoomFFT and ScaledFFT() both only do transforms along the unit circle, so they have less numerical error because the transform is defined by its endpoints instead of a delta.
Do you mean removing ScaledFFT and using only ZoomFFT for that purpose?

Well CZT() uses a complex step to define the parameter, which accumulates
error as you go around the spiral. ZoomFFT and ScaledFFT() both only do
transforms along the unit circle, so they have less numerical error because
the transform is defined by its endpoints instead of a delta.

Yes, this is clear: when I need to stay on the unit circle I should use
ZoomFFT or ScaledFFT because they calculate the points on the unit circle
more accurately.

Do you mean removing ScaledFFT and using only ZoomFFT for that purpose?

Yes.

Actually, they don't do the same thing and I'm confused.  This matches an rfft:
myczt = CZT(n = len(a), m=len(a)//2+1, scale=(len(a)//2+1) / len(a))

It measures at these points:

ScaledFFT with the same arguments does not match:
myscaled = ScaledFFT(n = len(a), m=len(a)//2+1, scale=(len(a)//2+1) / len(a))

By the way, @goriccardo may I ask what you use CZT for in your workflow?  The only application I really know of is prime-length FFTs.  I was hoping to also use it to detect exponentially-increasing sinusoids, but it seems it doesn't really work as I expected.

@endolith, Mosly I use the ZoomFFT for system identification.
2015-11-26 16:34 GMT+01:00 endolith notifications@github.com:

By the way, @goriccardo https://github.com/goriccardo may I ask what
you use CZT for in your workflow? The only application I really know of is
prime-length FFTs. I was hoping to also use it to detect
exponentially-increasing sinusoids, but it seems it doesn't really work as
I expected.
—
Reply to this email directly or view it on GitHub
https://github.com/scipy/scipy/pull/4607#issuecomment-159941062.

Ok, so there are 4 different functionalities here:

CZT(w=complex) where w is a delta between points along arbitrary logarithmic spirals
CZT(w=float) (which I changed to CZT(scale=)) which goes along a fraction of the unit circle starting at DC.
ZoomFFT(fn=[f1, f2]), which goes along an arbitrary fraction of the unit circle.
ScaledFFT(), which goes along an FFT-symmetrical fraction of the unit circle and output is fftshifted.

Functionalities 2 and 4 can both be done by 3, though not so straightforwardly:
czt(a, 8, scale=0.5) == zoomfft(a, [0, 7/8], 8)
czt(a, 7, scale=0.5) == zoomfft(a, [0, 6/7], 7)
scaledfft(x, 8, scale) == fftshift(zoomfft(x, [-scale, scale*(8-2.)/8], 8))
scaledfft(x, 7, scale) == fftshift(zoomfft(x, [-scale*(7+1)/7, scale*(7-3.)/7], 7))

ScaledFFT seems more like a "truncated FFT", and is mainly useful for convolution of signals with different sampling rates, which isn't otherwise in this PR, so I'd like to leave it for a different PR that focuses on that (and includes improved versions of fconvolve_1d and fconvolve_2d from convolve_by_czt_demo.py.)  Also I haven't checked it thoroughly, whether it works correctly for odd lengths, etc.
Functionality 2 is the last parameter to CZT, so could also be left for another PR if there's a good reason to include it, as either a parameter or a completely different function.  I don't really like it as a parameter to CZT anyway, since it's specifically for partial FFT-like transforms.  So I'm going to remove that and leave it for later, too.
So this PR will just be 1 and 3, which makes it simpler.
Also, should Fs= be capitalized?  It isn't in welch or spectrogram.

I don't understand these errors.  I rebased to Master and squashed some commits, and now it says
  File "/home/travis/build/scipy/scipy/build/testenv/lib/python2.7/site-packages/scipy/fftpack/czt.py", line 34, in <module>
    from scipy import fftpack
ImportError: cannot import name fftpack

But from scipy import fftpack works fine?  I don't get it.

Ok I removed the scale parameter and got it to pass tests.
The CZT and ZoomFFT init() methods are now duplicating some code, could maybe be consolidated.
I'm not sure about pyfftw integration, since this is inside fftpack.  I guess the fft itself can be monkeypatched instead of fftpack altogether?

I'm not sure about pyfftw integration, since this is inside fftpack. I guess the fft itself can be monkeypatched instead of fftpack altogether?

Indeed, that's what PyFFTW does. This PR looks fine to me.
@hgomersall I noticed that https://github.com/hgomersall/pyFFTW/blob/master/pyfftw/interfaces/scipy_fftpack.py hardcodes all scipy.fftpack function names not provided by pyfftw. So you will hide new functions when doing your monkeypatching right? Wouldn't it be better to just use:
from scipy.fftpack import *
__all__ == scipy.fftpack.__all__

Well I was trying to mimic the change in f0e77c1dc933d0a9ca3ae803ded55604153ea496 but it doesn't work here.

There's also this version by @cournape that I haven't looked through: https://github.com/numpy/numpy/compare/master...cournape:bluestein

I would really like to see this implemented so my opinions on the remaining issues are:

Should it really return a view on a much larger array, or do ascontiguousarray internally?

Do ascontiguousarray by default so inexperienced users don't have dimensions changing and add a boolean option to disable this behaviour.

Returns sections should be removed from class docstrings? How to document that it returns a callable function though?

https://github.com/scipy/scipy/blob/04991c623e969745359106cd95b18d0d95fcc57c/scipy/fftpack/czt.py#L113-L116
I believe adding the word callable makes this crystal clear. Same for ZoomFFT

@tsangiotis
What features are you looking for from this? Bluestein's algorithm will be included in the normal FFT functions when scipy switches to pocketfft

I am more interested in the ZoomFFT implemetation. Bluestein's algorithm is just a replacement for Chirp Z.
I want to replicate the behavior of this labview component: http://zone.ni.com/reference/en-XX/help/372416L-01/sndvibtk/zm_fft_sptrm_mag_phase/
Is an official Zoom FFT method in the roadmap whit pocketfft?

Zoom FFT would fit better in scipy.signal than in scipy.fft, since it's a signal analysis technique involving low-pass filtering in addition to fft transforms.

That makes sense. There is even a spectral analysis section where this could fit.

@tsangiotis

I am more interested in the ZoomFFT implemetation.

Ok.

Bluestein's algorithm is just a replacement for Chirp Z.

I think it would be correct to say that Bluestein's algorithm is an implementation of CZT.

I want to replicate the behavior of this labview component: http://zone.ni.com/reference/en-XX/help/372416L-01/sndvibtk/zm_fft_sptrm_mag_phase/

That also has averaging and windowing etc that aren't included here.

Is an official Zoom FFT method in the roadmap whit pocketfft?

No, that would just change the backend that this uses to compute the zoomfft.
@rgommers

Zoom FFT would fit better in scipy.signal than in scipy.fft, since it's a signal analysis technique involving low-pass filtering in addition to fft transforms.

This doesn't have a low-pass filter, right?  CZT is a sampling of the Z transform along a logarithmic curve, and the zoomFFT is the special case where that logarithmic curve is along a segment of the unit circle.

I went by the reference I get from Google when typing in "zoom FFT": https://www.arc.id.au/ZoomFFT.html. No CZT in sight there.

Is there a way to help getting this merged?
The CZT, in particular the ZoomFFT, is incredibly useful for optical calculations. Think of Fraunhofer Diffraction, but also Fresnel Diffraction (see for example https://www.nature.com/articles/s41377-020-00362-z).
More generally, problems involving scattering from regular lattices could be calculated more flexibly and efficiently with this zoomfft.

@llohse Do you have an application for the general CZT, or just the ZoomFFT?  Does the ZoomFFT interface in this PR work for you?  zoomfft(x, fn, m=None, Fs=2, axis=-1)   There may be accuracy issues that could be improved with a dedicated ZoomFFT implementation.
The main reason to add this was the prime-length FFT algorithm, but that's no longer necessary because scipy switched to pocketfft, which uses Bluestein's algorithm.

@llohse Do you have an application for the general CZT, or just the ZoomFFT? Does the ZoomFFT interface in this PR work for you? zoomfft(x, fn, m=None, Fs=2, axis=-1) There may be accuracy issues that could be improved with a dedicated ZoomFFT implementation.

I have not encountered a problem that would require the general CZT, yet. The interface is fine, although the separate Fs is kind of redundant for my use case -- having f1/Fs and f2/Fs as parameters would suffice.
A dedicated implementation of zoomfft with higher accuracy is certainly nice to have, too. How big of an accuracy issue do you think there is?

The main reason to add this was the prime-length FFT algorithm, but that's no longer necessary because scipy switched to pocketfft, which uses Bluestein's algorithm.

Yes, that is what I figured. The point I am trying to make, though, is that zoomfft (and likely the general czt, too) are really useful for simulating problems in physics, too.

Well I rebased it, at least.
I moved it to scipy.fft instead of scipy.fftpack
Interesting that the accuracy test only fails on  Arm64 Test Group 1.
I don't understand this:
ERROR: objects in scipy.fft.__all__ but not in refguide::
    CZT
    ZoomFFT
    czt
    czt_points
    ...
    zoomfft

Didn't I put them in the refguide? https://21762-1460385-gh.circle-artifacts.com/0/html-scipyorg/fft.html#module-scipy.fft

The czt vs fft example is probably no longer true with pocketfft:
    `czt` can quickly compute a prime-length FFT:

    >>> from scipy.fft import fft, czt
    >>> n = 30011  # prime
    >>> a = np.ones(n)  # DC (spectrum is [n, 0, 0, 0, ...])
    >>> f = fft(a)  # takes 1.4 seconds
    >>> c = czt(a)  # takes 21 milliseconds
    >>> np.allclose(f, c, atol=1e-6)
    True

So all of the tests use 2-norm to compare fft output with czt output, for instance.  Is that a good idea for some reason or should I change them to use assert_allclose?

So all of the tests use 2-norm to compare fft output with czt output, for instance. Is that a good idea for some reason or should I change them to use assert_allclose?

We can not expect the two outputs to be perfectly identical, since they are calculated by floating point arithmetic (in slightly different ways), right? In fact I am a little confused that not more tests are failing because of this.
However this might be the cause of the one failing test.

We can not expect the two outputs to be perfectly identical, since they are calculated by floating point arithmetic (in slightly different ways), right?

Yes.  assert_allclose tests them using a tolerance: abs(a - b) <= tol * abs(b) by default.
The current tests use the 2-norm, which works out to sqrt(sum(abs(a-b)**2)) < tol.  I'm not sure if there's a good reason for the extra complexity.

However this might be the cause of the one failing test.

Yeah the failing test is from inaccuracy, but I'm not sure why it only exists on one platform.

The current tests use the 2-norm, which works out to sqrt(sum(abs(a-b)**2)) < tol. I'm not sure if there's a good reason for the extra complexity.

My bad. Next time I'll read the code before commenting. I can see why one would be compelled to use the 2-norm -- it is the "natural" norm for Fourier Transforms due to the Plancherel Theorem -- but the test cases don't seem to depend on that. Therefore I think the much higher number of required operations as compared to the 1-norm outweighs the merit of the 2-norm here.

Yeah the failing test is from inaccuracy, but I'm not sure why it only exists on one platform.

My guess is that it is related to the particular behaviour of 'multadd' on the arm64 platform that somehow amplifies the discrepancy.
Anyway your latest commits seem to have fixed it. Great!

Well, all my commit did is sweep the dirt under the rug by choosing a more easily-passable test.  But allclose(rtol=1e-13) is still pretty good for "they do the same thing"?

Well, all my commit did is sweep the dirt under the rug by choosing a more easily-passable test. But allclose(rtol=1e-13) is still pretty good for "they do the same thing"?

I think it is perfectly fine. For N = 168 points and eps = 1e-16 we have N * eps > 1e-14, so we cannot really expect to obtain 1e-14 relative accuracy.
Being pedantic, I see just a few minor issues remaining:

The following note is misleading: "However, the chirp z-transform is considerably less precise than the equivalent zero-padded FFT, with differences on the order of 1e-7 from the direct transform rather than 1e-15 as seen with zero-padding."
The relative deviation depends on the input length. So giving a fixed difference here has little meaning in my opinion. I would suggest to simply state that the czt is less precise without giving numbers.
https://github.com/scipy/scipy/pull/4607#discussion_r506251925.
https://github.com/scipy/scipy/pull/4607#discussion_r314938187
maybe not capitalize Fs to be more consistent with f1 and f2?

I think it is perfectly fine. For N = 168 points and eps = 1e-16 we have N * eps > 1e-14, so we cannot really expect to obtain 1e-14 relative accuracy.

The FFT error grows only very slowly with the length of the transform; if everything is done perfectly, it should grow with O(sqrt(log(N))). So something around 1e-13 for a length-168 transform is pretty poor.
Concerning 2-norm for error measurement: I think it is the more convenient way to determine deviations between two vectors of numbers, because using allclose with a only relative tolerance specified can always go wrong when an element of the input array is very close to zero. I'd recommend sticking to the 2-norm unless there are important reasons not to.

The FFT error grows only very slowly with the length of the transform; if everything is done perfectly, it should grow with O(sqrt(log(N))). So something around 1e-13 for a length-168 transform is pretty poor.

@mreineck: But is that also the case for Bluestein's method (if done perfectly) on arm64/aarch64? Did you try that with pocketfft? Do you have any idea what might be going wrong?

Concerning 2-norm for error measurement: I think it is the more convenient way to determine deviations between two vectors of numbers, because using allclose with a only relative tolerance specified can always go wrong when an element of the input array is very close to zero. I'd recommend sticking to the 2-norm unless there are important reasons not to.

Good point. I thought that allclose actually computed the absolute-value norm and not the maximal-value norm.

But is that also the case for Bluestein's method (if done perfectly) on arm64/aarch64?

Bluestein's method should give you an additional factor (around 2) in the error, but nothing that scales with transform length.
The underlying hardware should not have a big influence, as long as it implements IEEE754 arithmetics.

Did you try that with pocketfft?

Yes. It's not getting the full O(sqrt(log(N))), because I'm making some compromises when computing the twiddle factors (for performance reaons), but it's certainly not worse than O(log(N)) .

Do you have any idea what might be going wrong?

I suspect that the problems come from evaluating expressions like exp(i*2*pi*x) with large x. For every "full circle" made in the exponent, the error of the result grows by roughly 1e-16. This can only be fixed by perfect range reduction, but it seems that for the general form of the Zoom FFT this is not possible. I'd love to have a closer look at this at some point, but at the moment I'm too busy with other things.
I'd say the PR is fine (except perhaps going back to L2-norm in the test), but I'm not a maintainer :)

Just to illustrate my point about the complex exponentials:
In [1]: import numpy as np

In [2]: np.exp(1j*2*np.pi*np.arange(10))-1.
Out[2]:
array([0.+0.00000000e+00j, 0.-2.44929360e-16j, 0.-4.89858720e-16j,
       0.-7.34788079e-16j, 0.-9.79717439e-16j, 0.-1.22464680e-15j,
       0.-1.46957616e-15j, 0.-1.71450552e-15j, 0.-1.95943488e-15j,
       0.-2.20436424e-15j])

The underlying hardware should not have a big influence, as long as it implements IEEE754 arithmetics.

But the test only fails on ARM64.

I suspect that the problems come from evaluating expressions like exp(i*2*pi*x) with large x. For every "full circle" made in the exponent, the error of the result grows by roughly 1e-16.

This does not explain why it only fails on ARM64, though.
Since @mreineck brought this up.
The test test_czt_vs_fft calls assert_allclose(czt(a), fft(a), rtol=1e-7, atol=1e-8) independent of the length. A relative precision of 1e-7 might be acceptable for extremely long transforms but seems a little to relaxed in general.
There are more uses of assert_allclose all over the place, probably because the hardcoded tolerance of 1e-10 in check_czt is to sharp.
I would suggest the following:

add an rtol parameter to check_czt and check_zoomfft
use check_czt and check_zoomfft exclusively instead of manually calling fft and czt
set the tolerance dynamically, based on the length of the transform

This does not explain why it only fails on ARM64, though.

It could ... depends a bit on how exactly the trigonometric functions in the ARM64 math library perform range reduction. The language standards typically don't prescribe the accuracy of math library functions, so it might just be that little bit less accurate on ARM64, and for large angles this error accumulates.
(That's just one possible explanation for the failure ARM64 failure ... I'm sure there are other plausible resaons.)

OK, I have read through the code some more.
The root of all the potential accuracy problems is the variable w. If this was not specified as a complex value but rather as three values rad (float), nom (int or float) and denom (int or float), such that
w:= rad * exp(i*2*pi*nom/denom)

the code would have all the information it needs to compute the results with the best possible accuracy. If nom and denom are both integers, the involved angles can be calculated accurately.
Of course I understand that such an interface looks pretty ugly ... I'm not sure if it is worth the trouble.

OK, I have read through the code some more.
The root of all the potential accuracy problems is the variable w. If this was not specified as a complex value but rather as three values rad (float), nom (int or float) and denom (int or float), such that
w:= rad * exp(i*2*pi*nom/denom)

the code would have all the information it needs to compute the results with the best possible accuracy. If nom and denom are both integers, the involved angles can be calculated accurately.

This could even be done with nom and denom given as a single parameter of type fraction.Fraction or a float.

Of course I understand that such an interface looks pretty ugly ... I'm not sure if it is worth the trouble.

It's not that bad actually. However, I think the parameter a should be split into modulus and phase, too, for consistency.
If we were to follow that idea, we could cover both CZT and ZoomFFT with a single class.
We could still provide wrappers for convenience.

This could even be done with nom and denom given as a single parameter of type fraction.Fraction or float.

That sounds really good. The only difficulty here would be to make it unmistakably clear that this value is not an angle in radians, but rather a fraction of 2pi.
Having such an implementation, and convenience wrappers for situations where accuracy is not crucial would be great.

although the separate Fs is kind of redundant for my use case -- having f1/Fs and f2/Fs as parameters would suffice.

Having fs= is convenient and consistent with a bunch of other functions, and using zoomfft(x, [f1, f2], fs=fs) is equivalent to zoomfft(x, [f1/fs, f2/fs]), no?

The czt vs fft example is probably no longer true with pocketfft:

Nope:
timeit fft(a)
3.31 ms ± 77.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

timeit czt(a)
8.85 ms ± 200 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Therefore I think the much higher number of required operations as compared to the 1-norm outweighs the conceptual merit of the 2-norm here.

assert_allclose actually takes longer, for whatever reason:
assert_(norm(y-y1)/norm(y) < 1e-10... 15 μs
assert_allclose(y, y1) 71 μs

I'd recommend sticking to the 2-norm unless there are important reasons not to.

Ok, do you know what acceptable tolerances would be?

Of course I understand that such an interface looks pretty ugly ... I'm not sure if it is worth the trouble.

Matlab has a y = czt(x,m,w,a) also, so probably good to support the same interface https://www.mathworks.com/help/signal/ref/czt.html
But we could change it to accept w=complex_number as well as w=(rad, nom, denom) or w=(rad, Fraction(nom, denom)).  It's still "the w parameter", just expressed differently, and the format is unambiguous.

Ok, do you know what acceptable tolerances would be?

Not really for the present form of the algorithm. If it's only to make sure that the code is not doing anything wrong, then 1e-13 or 1e-12 are absolutely fine. If the more accurate representation of w is in place, the error should probably go below 1e-14.

assert_allclose actually takes longer, for whatever reason:

Just a minor comment about this one point -- I don't think we need to optimize the test performance to the microsecond level. assert_allclose is used all over the place in SciPy, which is a good reason to keep using it. Deviating from standard testing practices by creating non-standard assert functions increases maintenance burden -- everyone reading the code has to spend time understanding the function and when it fails, understand a new way to fix or adjust it, the error message will be deficient and nonstandard compared to what assert_allcose produces, etc.
If there is a compelling comparison-validity-based reason to use some other function then we can do it, but let's not switch because assert_allclose is microseconds slower. If this builds up to a meaningful bottleneck in the tests -- i.e., costs us on the order of seconds in the tests being written -- then it might also be worth optimizing or changing.

My recommendation for using the L2 error is not because of performance, but because I feel this is the "right" way to measure closeness of results in this context.
Imagine an extreme example: if you run a forward and then an inverse FFT on the array [1., 0., 0., 0., 0., 0., 0.], the result will have terrible relative errors for all array entries except the first, even though the transforms work perfectly fine. So the expected relative errors for every array entry depend on the distribution of values in the input array, which is quite unsatisfactory. In contrast, using L2 error here just works.

Imagine an extreme example: if you run a forward and then an inverse FFT on the array [1., 0., 0., 0., 0., 0., 0.]

For this personally I would use assert_allclose(..., rtol=1e-12, atol=1e-12) or so.

But if the first array value was 1000., then that test would break again...

Yes but you know your inputs and you can set atol and rtol appropriately, and anyone who uses standard NumPy testing functions will immediately know what you're doing.
Ultimately these are the tradeoffs, I am just advocating for not over-engineering (and especially over-optimizing) the testing parts. If the 2-norm does end up making more sense for the tests that's okay with me.

I agree that it's fine if you have control of your inputs. It's (in my opinion) just more work for no real benefit. The L2 test isn't any less strict than the allclose, and doesn't require this kind of fiddling, so I'd say using allclose is actually the over-engineered way :)
However, using the L2 error admittedly has the disadvantage that if things go wrong, it doesn't tell you where exactly in the array the problem is.

assert_allclose also produces more helpful error messages, as we've seen with this ARM64 failure, so I would prefer that (with rtol and atol) unless there's some good reason not to.

I think using assert_allclose is fine, as long as atol is set to a meaningful value (for example the same as rtol) the way larsoner suggested. We can always benchmark the functions independently of the test framework, if we feel the need.

For the error analysis we don't use L2 for bunch of reasons, the missing entry is one that is already mentioned and the other one is the difficulty of setting up a meaningful number for the tolerance bounds. The typical way to do it is setting atol=M*<max amplitude in array>*np.spacing(<float or complex dtype>(1.)) and rtol=0 via allclose. Here M is a constant depending either on the array size or some generic value 1, 1000 or a million depending on how strict you want to test the values.
Otherwise we have to guess the cumulative error size per entry and derive these values for each test which is tedious and also doesn't say much. Was the error concentrated on a single item or all entries contributed a little ? This is quite laborious for maintainers when debugging since then they have to debug the whole test suite.

@endolith:
Now that the tests are in place, how would you like to proceed?
Personally, I would prefer having the accurate code merged, even if it takes a few days longer, instead of having two separate PRs.
If you like, I could fork your branch and implement it there.

Any references we can add to ZoomFFT for this implementation?  A quick google showed others using frequency shifting, bandpass filtering, and resampling, which doesn't seem like the same thing, even if it produces the same output

Large prime length:
N = 10007
x = np.random.rand(N)
c = CZT(N)
z = ZoomFFT(N, 2*(N-1)/N)

timeit fft(x)
942 µs ± 25.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

timeit zoomfft(x, 2*(N-1)/N)
2.04 ms ± 97.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

timeit czt(x)
2.02 ms ± 49.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

timeit c(x)
696 µs ± 19.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

timeit z(x)
704 µs ± 13.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

So the "pre-compiled" version is actually a bit faster than pocketfft?
Nearby composite length:
N = 10125
...

timeit fft(x)
176 µs ± 3.43 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

timeit czt(x)
2.07 ms ± 91.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

timeit zoomfft(x, 2*(N-1)/N)
2.19 ms ± 85.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

timeit c(x)
765 µs ± 36.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

timeit z(x)
749 µs ± 10.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

zoomfft vs zooming into fft:
N = 10000
x = np.random.rand(N)
f = fft(x, 40000)[:1000]
z = zoomfft(x, fn=999, m=1000, fs=40000)

allclose(f, z)
Out[252]: True

Z = ZoomFFT(N, fn=999, m=1000, fs=40000)

timeit fft(x, 40000)[:1000]
1.18 ms ± 15.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

timeit zoomfft(x, fn=999, m=1000, fs=40000)
1.56 ms ± 32.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

timeit Z(x)
402 µs ± 17.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

So the "pre-compiled" version is faster than truncating an oversampled FFT, so that's good.
Does everything think that this zoomfft interface is good?  Or should f2 be the point after the end of the output?  Then the above examples would simplify to z = ZoomFFT(N, 2) and zoomfft(x, fn=1000, m=1000, fs=40000) for instance.  Any precedent set by other software?

I'm not sure what I meant by

Should it really return a view on a much larger array, or do ascontiguousarray internally?

I remember thinking it kept a large array in memory and only output a small view of it, but I don't see that now. Maybe I was thinking of y = y[..., self._yidx] but ignoring the  * self._wk2 part that turns it into a copy?  Or maybe I'm missing something now.

Or should f2 be the point after the end of the output?

That feels better to me.Specifying the "one after last" index seems to be the preferred choice in most contexts.

Or should f2 be the point after the end of the output?

How about having a parameter endpoint=False like numpy.linspace? Both cases have their applications.

Ohhh, the reason I thought it was a view on a much larger array is because it literally says so in the docstring:

Note that this is a view on a much
larger array.  To save space, you may want to call it as
y = ascontiguousarray(czt(x))

But is this actually true?  Doesn't the line y = y[..., self._yidx] * self._wk2 create a new array of length m?  So I should remove these descriptions?

But is this actually true? Doesn't the line y = y[..., self._yidx] * self._wk2 create a new array of length m? So I should remove these descriptions?

You can easily check if the result has the contiguous flag set.

@llohse Yes, they do, but I just want to make sure I'm not missing something, since the code has always been this way and it doesn't seem like the docstring is correct

@llohse Yes, they do, but I just want to make sure I'm not missing something, since the code has always been this way and it doesn't seem like the docstring is correct

@endolith: I thought about that. Actually there is no reason why A = A[...,i] * B should return a view of A, since it clearly is an out-of-place operation.
The in-place variation,
A = A[...,i]
A *= B,

returns a view of (the original) A.
However, the extra copy here should actually be the more efficient way, because as the memory is C-contiguous, the slice A[...,i] is not contiguous and costly to work with.

It seems to me that it could be confusing to users to have the class-based and function-based entities differ only by capitalization. Are there other instances like this in the codebase?

KroghInterpolator vs krogh_interpolate, BarycentricInterpolator vs barycentric_interpolate, etc?

I guess snake_case for zoom_fft() would technically be more correct?  czt_points is already correct

Anything needed to help push this over the edge?

@mjmdavis @llohse could you look and see if you're happy with the code?
@endolith did you go back through the 100+ comments above and make sure the original comments from @charris, @ewmoore, @hgomersall and @e-q (apologies if I missed anyone) are addressed? And to the long list of people I just spammed, feel free to look again or ask for more time -- if I don't hear anything I'll move forward when things look good.
After @mjmdavis and/or @llohse give an approval / +1 I'll take a deeper look. It seems plausible to get this in within the next week for 1.6.0 if we keep moving.

This is some real nitpicking, but the docstrings could be more consistent in language. I.e., zoomfft describes x as "The input signal" while czt uses "the signal to transform." Also the axis argument is described in terms of "dimension," but in places like fft.fft axis is described as "Axis over which to compute the FFT. If not given, the last axis is used."
Altogether, I think we should try to match existing FFT language for the arguments and output where possible and appropriate.

Other doc notes:

The function versions don't have examples; the ones from the classes can be copied over.
czt_points should get "See Also" links to czt and CZT
The functions and classes should get "See Also" links to their counterparts.
There are some formatting tricks to get arguments mentioned in the "Notes" section to show up like x, this would help the ZoomFFT "Notes" section in particular
CZT is the only one with reference information
It's not clear to me what is implied by referring to ZoomFFT as "friendlier" in the CZT "See Also" section. More efficient?

@larsoner
The code looks fine as is in my opinion although the accurate code path with w=(rad, Fraction(nom, denom)) (which would also obsolete the redundant code in ZoomFFT) is still missing. @endolith prefers to defer that in a later PR (possibly by someone else) though. This seems to be a good idea in light of the approaching deadline.

did you go back through the 100+ comments above and make sure the original comments from

If someone wants to make a checklist of each of those issues to make sure they're addressed, that would help
I think I got them all on my checklist at the top but I'm not sure

Is there a name for a class whose instances are callables?  Should I call it a "function factory" or "callable generator" or something?

I would just call it a "callable class instance". See for example interp1d which operates this way

What's the status of this?

CZT is the only one with reference information

Does anyone know of a reference for this ZoomFFT implementation?  (I didn't write this PR...)

It's not clear to me what is implied by referring to ZoomFFT as "friendlier" in the CZT "See Also" section. More efficient?

I think it means the interface is easier if you want a Zoom FFT

Does anyone know of a reference for this ZoomFFT implementation? (I didn't write this PR...)

The Rabiner et al. article could be cited, as the ZoomFFT really is just a special case of the CZT.

I think it means the interface is easier if you want a Zoom FFT

That, and ZoomFFT is more accurate because it involves fewer operations. This will no longer be the case once the CZT will also accept w and a as a pair of amplitude and (possibly rational) phase - which will not be implemented with this PR.

Ok I made a checklist of everything that anyone has mentioned and will go through and try to check things off.

The Rabiner et al. article could be cited, as the ZoomFFT really is just a special case of the CZT.

We should have a reference that says "this is a legitimate way to calculate the Zoom FFT", though.  Most implementations use modulation and filtering, etc.  I think the multiplication with the chirp accomplishes the same thing?

We should have a reference that says "this is a legitimate way to calculate the Zoom FFT", though. Most implementations use modulation and filtering, etc. I think the multiplication with the chirp accomplishes the same thing?

I disagree. What we do is calculate the CZT with amplitude one. That's what we do and I don't think it needs a separate reference. The term "ZoomFFT" is used ambiguously and other people mean something completely unrelated.

Zoom FFT would fit better in scipy.signal than in scipy.fft

are these transforms better suited in signal alongside other spectral transforms e.g. cwt and stft?

How about voting on this comment?
🚀 = scipy.fft
👀 = scipy.signal

Should zoomfft be renamed to zoom_fft for snake_case?
👍 = zoom_fft
👎 = zoomfft

Should Zoom FFT's endpoint= default to True or False?
👍 endpoint=True
👎 endpoint=False

There are some formatting tricks to get arguments mentioned in the "Notes" section to show up like x, this would help the ZoomFFT "Notes" section in particular

numpydoc spec says

Enclose variables in single backticks.

which is already done for x in ZoomFFT Notes

If this was not specified as a complex value but rather as three values rad (float), nom (int or float) and denom (int or float)

However, I think the parameter a should be split into modulus and phase, too, for consistency.

@mreineck  and @llohse, are you ok with keeping the current API and using some notation like w=(rad, nom, denom) or w=(rad, Fraction(nom, denom)) and a=(modulus, phase) for the future PR that allows for more accuracy?  In other words, it would use the same parameters, with the optional increased accuracy format determined by the parameter being a tuple instead of a complex.  What that tuple actually contains can be determined later, as long as you're ok with this API.

I personaly think the API you are poposing is perfectly fine, but I don't know whether it matches scipy tradition and customs. That's for the maintainers to decide.

@e-q

The function versions don't have examples; the ones from the classes can be copied over.

The CZT example doesn't make sense for the czt function though.  Anyone want to write a better czt example?

I added a reference for using Chirp Z to look at a specific part of the spectrum, but it doesn't actually call it "zoom FFT".
I added an example for czt, but it's not generating images in the artifacts output, and there's an error which I don't understand:
build finished with problems, 1 warning.
Makefile:127: recipe for target 'html-scipyorg' failed
make: *** [html-scipyorg] Error 1
make: Leaving directory '/home/circleci/repo/doc'
Traceback (most recent call last):
  File "runtests.py", line 565, in <module>
    main(argv=sys.argv[1:])
  File "runtests.py", line 214, in main
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.7/subprocess.py", line 468, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['make', '-Cdoc', 'PYTHON="/home/circleci/repo/venv/bin/python"', 'html-scipyorg', 'latex', 'SPHINXOPTS="-j2"']' returned non-zero exit status 2.

Gotta look higher up in the log -- CircleCI treats (most) warnings as errors so searching the log for "warning" you see where the problem is:
/home/circleci/repo/build/testenv/lib/python3.7/site-packages/scipy/fft/czt.py:docstring of scipy.fft.czt:64: WARNING: Exception occurred in plotting scipy-fft-czt-1
 from /home/circleci/repo/doc/source/generated/scipy.fft.czt.rst:
Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/matplotlib/sphinxext/plot_directive.py", line 472, in run_code
    exec(code, ns)
  File "<string>", line 2
    A similar process can be used for system identification purposes,

@larsoner
Ok, I still don't understand what I did wrong:
waiting for workers...
/home/circleci/repo/build/testenv/lib/python3.7/site-packages/scipy/fft/czt.py:docstring of scipy.fft.czt:64: WARNING: Exception occurred in plotting scipy-fft-czt-1
 from /home/circleci/repo/doc/source/generated/scipy.fft.czt.rst:
Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/matplotlib/sphinxext/plot_directive.py", line 472, in run_code
    exec(code, ns)
  File "<string>", line 2
    A similar process can be used for system identification purposes,
            ^
SyntaxError: invalid syntax

The syntax error is pointing to plain text in a docstring?

Looks like a bug with matplotlib/sphinxext/plot_directive.py, maybe we should update the CircleCI image to something newer than the python 3.7.0 image and/or upgrade the matplotlib version that is installed there.

Never mind, we just pip install matplotlib so it should get 3.3.3, they must have some error here:
https://github.com/matplotlib/matplotlib/blob/v3.3.x/lib/matplotlib/sphinxext/plot_directive.py#L472
Want to try making it a minimal example and open a bug report with them? In the meantime you could maybe use fewer sections by taking advantage of subplots, or make some other tweak to fix it. Locally I'd expect you to be able to reproduce if you pip install matplotlib==3.3.3.

Want to try making it a minimal example and open a bug report with them?

I don't really know how to do that, with all the CI side of things.  Can we just link to this and I'll keep a copy of the branch that can be referenced?

In the meantime you could maybe use fewer sections by taking advantage of subplots,

So do you mean that the problem is too many plots in one docstring?  I can combine the time and frequency into subplots

is there a tag like "ci skip" that just builds the docs and skips the rest of the tests?

Ok, so the problem was in having additional text after the last plot command?  I don't understand why.

I don't really know how to do that, with all the CI side of things. Can we just link to this and I'll keep a copy of the branch that can be referenced?

You shouldn't need to deal with CIs. Locally if you set up your env (or a venv) with the same matplotlib version and build the SciPy docs in theory you will see the same problem locally, and then from there you could iterate to make it simpler. Or even take the numpydoc and make an example of it. I did this and opened an issue:
https://github.com/matplotlib/matplotlib/issues/19111

Ok, so the problem was in having additional text after the last plot command? I don't understand why.

Yep, don't have any text after the last plot -- looks like it works:
https://23717-1460385-gh.circle-artifacts.com/0/html-scipyorg/generated/scipy.fft.czt.html#scipy.fft.czt

is there a tag like "ci skip" that just builds the docs and skips the rest of the tests?

You can try [skip github] and someday maybe [skip azp] will work (it doesn't but there is an issue about it somewhere). I'm not sure if skip github actually works anymore though...

We manually added skip github to GH Actions very recently
https://github.com/scipy/scipy/blob/master/.github/workflows/linux.yml#L16
hence it will work after a rebase.

Can someone go through the To Do list in the first comment and decide which of the unfinished things really need to be included in this PR, and which can be deferred to later?
Remember that I didn't write this code; I'm just trying to clean it up enough to get it in, and it's been in limbo for 12 years now, and every suggested change to this PR makes it take longer to get merged.    It's ok to merge something that works, and then make it faster and more accurate and better-documented in future PRs, no?
I've added some of the changes that people suggested (https://github.com/scipy/scipy/pull/4607#discussion_r506251925 https://github.com/scipy/scipy/pull/4607#discussion_r314938187) but without really understanding them completely, if they have subtle bugs, etc.  I would prefer that we get this PR into a mergable state with minimal changes, and suggested changes like those (even the documentation ones) be done by the people who suggested them, in their own future PRs.  Smaller PRs are easier to check for bugs/correctness anyway.

FWIW, I think it makes a lot of sense to merge this now and maybe polish a few more things later on if needed.

Hi all,
I'm really interested in CZT for scipy. I just noticed that there is this branch and that you are already working on it. As of Scipy 1.6 there is scipy.linalg.basic.matmul_toeplitz included, which uses the same method as czt.py in line 227+257. I browsed through the comments here and could not find any reference to this. Do you think it would make sense to use matmul_toeplitz to avoid code duplication within scipy?

@LarsHuebner-LHNW  I'm not sure what you mean.  What would be replaced by what?

(Sorry for asking other people to hurry up, and then not doing so myself.) 😬

Looks like there are a few remaining issues @endolith (you can see them in the "Files" view), 1.7.0 is going to be in a few weeks. I think if we can make the last few changes, this can go in!

Re-milestoning for 1.8.0 but I think there is still time to get this in if you can work on it today and/or tomorrow @endolith . It seems very close!

Hi,
As @LarsHuebner-LHNW pointed out, Scipy v1.6.0 added the scipy.linalg.matmul_toeplitz function for "Efficient Toeplitz Matrix-Matrix Multiplication using FFT". This means that the entire CZT algorithm can be written as follows:
def czt(x, M=None, W=None, A=1.0):

    # Unpack arguments
    N = len(x)
    if M is None:
        M = N
    if W is None:
        W = np.exp(-2j * np.pi / M)
    A = np.complex128(A)
    W = np.complex128(W)

    # CZT algorithm
    k = np.arange(max(M, N))
    Wk22 = W ** (-(k ** 2) / 2)
    r = Wk22[:N]
    c = Wk22[:M]
    X = A ** -k[:N] * x / r
    X = scipy.linalg.matmul_toeplitz((c, r), X)
    X /= c

    return X

Using this function may help to reduce duplication in the Scipy code base. (For more information, this is "Algorithm 1" in: V. Sukhoy and A. Stoytchev, “Generalizing the inverse FFT off the unit circle,” Sci. Rep., vol. 9, no. 1, p. 14443, Dec. 2019, doi: 10.1038/s41598-019-50234-9.)
FYI, I have my own CZT repo here. In this repo, I compare the speed of several different CZT algorithms. I found that the circulant embedding approach is the fastest, but scipy.linalg.matmul_toeplitz isn't far off.

Are there any news when this might get merged?

This line:
np.concatenate(([0, 0, 1], np.zeros(100)))
works correctly, but is producing this error:
scipy/fft/tests/test_czt.py:145: error: Argument 1 to "concatenate" has incompatible type "Tuple[
List[int], ndarray[Any, dtype[floating[_64Bit]]]
]"; expected "Union[
_SupportsArray[dtype[<nothing>]],
Sequence[_SupportsArray[dtype[<nothing>]]],
Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]],
Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]],
Sequence[Sequence[Sequence[Sequence[_SupportsArray[dtype[<nothing>]]]]]]
]"  [arg-type]

Yes, type annotations aren't infinitely flexible. Changing it to np.concatenate((np.array([0, 0, 1]), np.zeros(100))) should fix it.

OK, the tests are fixed.  So now someone needs to decide:

Should it go in scipy.signal or scipy.fft?
Should endpoint= be True or False by default?

Should endpoint= be True or False by default?

Just a personal opinion, but I'd say if it's not blindingly obvious which choice should be the default, then there should be no default at all.
I've seen too many users (myself included) been bitten by unexpected defaults ... it's unrealistic to expect everyone to have completely read and understood the documentation, so I personally prefer an error message due to a missing parameter to a result based on not-totally-obvious assumptions.

I think it should be endpoint=False so that zoomfft(x) is equivalent to fft(x), but that's not the default behavior of the original authors.  I'm not sure if there's a reason for that.

This implementation uses (x, A, W, M) for CZT while we have (x, m, w, a, axis). Any reason to prefer that order?
And for Zoom FFT it has (x, f0, f1, fs, M, axis) while we have (x, (f1, f2), m, fs, endpoint, axis).  Combining the frequencies is better, but any reason to swap M and fs?
We should also add zoomfftfreq(), like they do, but that can be another PR.

Agreed the endpoint=False making the two functions equivalent is nice, and a good justification for having it as the default.

This [gist] implementation uses (x, A, W, M) for CZT while we have (x, m, w, a, axis). Any reason to prefer that order?

In terms of priority for argument ordering and naming, I think we should prioritize 1) consistency within SciPy, 2) what seems most reasonable/cleanest, 3) consistency with MATLAB/Octave/other languages, and then 4) whatever was done in the gist. I think this is more or less our preferred/standard order, at least based on what I've seen in scipy.signal. @endolith can you try to reorder/rename arguments according to this?

@endolith @larsoner no activity here in a few months--is this likely to be ready/decided-on before we branch on Monday?

API

In terms of priority for argument ordering and naming, I think we should prioritize 1) consistency within SciPy,

No prior use in SciPy.

what seems most reasonable/cleanest,

Don't think it makes any difference.

consistency with MATLAB/Octave/other languages,

Matlab uses y = czt(x,m,w,a), matching this PR.

and then 4) whatever was done in the gist.

Note that "the gist" that uses (x, A, W, M) is an unrelated implementation.  Another unrelated implementation uses czt(x, M, W, A).
So I'm not changing this; consistency with two other implementations is good.
scipy.signal vs scipy.fft
4 mild votes for signal:

https://github.com/scipy/scipy/pull/4607#issuecomment-498157401
https://github.com/scipy/scipy/pull/4607#discussion_r574482611
https://github.com/scipy/scipy/pull/4607#discussion_r574897594
Me, now, because CZT is superset of FFT but special case of Z-transform, which is primarily DSP topic

1 vote for fft:

https://github.com/scipy/scipy/pull/4607#issuecomment-739552320

So I moved it to scipy.signal and put an underscore in front of it.
endpoint
Changed to False for equivalence with fft().  Three votes in favor, zero against:

https://github.com/scipy/scipy/pull/4607#issuecomment-716124250
https://github.com/scipy/scipy/pull/4607#issuecomment-896706063
https://github.com/scipy/scipy/pull/4607#issuecomment-901207245

Should zoomfft be renamed to zoom_fft?

In terms of priority for argument ordering and naming, I think we should prioritize 1) consistency within SciPy,

No prior use in SciPy.

Just to clarify for the future since I think it wasn't clear what I meant by this, I don't mean consistency of this function's use within SciPy -- indeed it's new -- I mean consistency with any other similar/related functions already in SciPy. For a hypothetical example, if we already have a lfilter(b, a, x, axis=-1, zi=None), a PR proposing to adding a new filter_polyphase(b, a, x, ..., zi=None, axis=-1) is less good than filter_polyphase(b, a, x, ..., axis=-1, zi=None) because the last two args are switched.
Coming back to this PR, though, I don't see anything else that's really similar, I agree there is no good order based on first principles, and the third priority is satisfied, so I agree we're good with the current order.

Should zoomfft be renamed to zoom_fft?

+1 for the more Pythonic name zoom_fft from me. @endolith do you want to do the rename (assuming you agree) and I can do a final round of review and merge?

Just a reminder that, although the documentation PR has not been merge so I can't say that we've officially adopted the policy, there seemed to be agreement with this part:
Required keyword names
~~~~~~~~~~~~~~~~~~~~~~
For new functions or methods with more than a few arguments, all parameters
after the first few "obvious" ones should *require* the use of the keyword
when given.  This is implemented by including ``*`` at the appropriate point
in the signature.

@WarrenWeckesser Sorry, that's not a convention I've seen before. You mean the definitions should be something like this?
def czt_points(m, *, w=None, a=1+0j):

class CZT:
    def __init__(self, n, *, m=None, w=None, a=1+0j):

class ZoomFFT(CZT):
    def __init__(self, n, fn, *, m=None, fs=2, endpoint=False):

def czt(x, *, m=None, w=None, a=1+0j, axis=-1):

def zoom_fft(x, fn, *, m=None, fs=2, endpoint=False, axis=-1):

(I'd just like to reiterate that I want to make as minimal changes to this PR as possible for it to be merged, and leave everything else for future PRs, since this one has been dragging on for almost 7 years now and little changes keep being suggested and I'm not good at following up on all of them.)

Having said that,  I won't block the PR if the "keyword required" syntax doesn't get in.  We've gotten by without it for a long time, so a couple more functions without it is not a disaster.  It is, however, a nice feature that we really should be adopting in new code.  It encourages a more readable style of using the code, and it also make the API a little less rigid.  With the new syntax, new keywords can be added anywhere after the * without breaking backwards compatibility.  Without it, new keywords can only be added at the end of the signature.

I didn't answer your question...

You mean the definitions should be something like this?

Yes, exactly.

OK, the tests pass without the keyword-only arguments.
I'm not sure where to put the asterisk, though, since what I showed above would break my doctest examples:
>>> points = czt_points(m, w, a)
…
>>> transform = ZoomFFT(len(x), [f1, f2], len(x), 1021)
…
>>> points = czt_points(M + 1, w, a)  # M + 1 to include Nyquist
…
>>> z_vals = czt(x, M + 1, w, a)  # Include Nyquist for comparison to rfft
…
>>> X = zoom_fft(x, [f1, f2], len(x), 1021)

So maybe it should actually be
def czt_points(m, w=None, a=1+0j):

class CZT:
    def __init__(self, n, m=None, w=None, a=1+0j):

class ZoomFFT(CZT):
    def __init__(self, n, fn, m=None, *, fs=2, endpoint=False):

def czt(x, m=None, w=None, a=1+0j, *, axis=-1):

def zoom_fft(x, fn, m=None, *, fs=2, endpoint=False, axis=-1):

Or maybe
class ZoomFFT(CZT):
    def __init__(self, n, fn, m=None, fs=2, *, endpoint=False):

def zoom_fft(x, fn, m=None, fs=2, *, endpoint=False, axis=-1):

...

I'm not sure where to put the asterisk, though, since what I showed above would break my doctest examples:

A point of the new syntax is that you have to use the keyword when you call the function.  Update your examples to use the required keywords, e.g. change
>>> points = czt_points(m, w, a)

to
>>> points = czt_points(m, w=w, a=a)

(Belated bikeshedding comment that you can ignore: if w is "The ratio between points in each step", wouldn't a better name for that parameter be ratio?)

A point of the new syntax is that you have to use the keyword when you call the function.

Yes, but do we want m to be keyword-only (for instance)?

if w is "The ratio between points in each step", wouldn't a better name for that parameter be ratio?)

Every other implementation uses w so I think this is fine; it's from the Rabiner paper (where it is capitalized)

The other thing I forgot about in https://github.com/scipy/scipy/pull/4607#issuecomment-898942816:

And for Zoom FFT it has (x, f0, f1, fs, M, axis) while we have (x, (f1, f2), m, fs, endpoint, axis).  Combining the frequencies is better, but any reason to swap M and fs?

So using the same list:

consistency within SciPy,

There's nothing similar?  Maybe things like butter(N, Wn, btype='low', analog=False, output='ba', fs=None) which put fs last, but only because it was added after the API was heavily-used.

what seems most reasonable/cleanest,

Maybe putting fs next to fn is cleaner?

consistency with MATLAB/Octave/other languages, and then

I don't see a zoom FFT function in other languages.

whatever was done in the gist.

The gist (of unrelated origin) by @aluchies defines zfft(x, f0, f1, fs, M, …) with fs and M swapped.

@endolith I would say use your best judgment here on where (if at all) to put a * to force everything thereafter to be kwarg-only. Same with the order. If that's the current state that's fine, but if you can find places that * could be used well or fs could be moved (for example) now is the best time to do it.

OK, I think putting the asterisk before fs solves both of those issues.  It feels more like that kind of argument to me, at least because it was added late to the filter design functions and is typically used with a keyword in that context.

The false positive test results are annoying.  Is there a way to get just those particular tests to be redone without force-pushing and restarting the whole thing?

Thanks @endolith !

There's an inverse CZT under MIT license here if anyone wants to adapt it: https://github.com/garrettj403/CZT

On 04.12.21 04:34, endolith wrote:

There's an inverse CZT under MIT license here if anyone wants to adapt
it: https://github.com/garrettj403/CZT https://github.com/garrettj403/CZT

I'm not sure if it was this implementation, but at least one of the
inverse CZT implementations is patent-encumbered, so this might need
careful checking.

Closest I can find is https://patents.google.com/patent/US20170149590A1/en which is abandoned.

On 04.12.21 23:21, Stefan van der Walt wrote:

Closest I can find is
https://patents.google.com/patent/US20170149590A1/en
https://patents.google.com/patent/US20170149590A1/en which is abandoned.

I meant this one:
https://patentscope.wipo.int/search/en/detail.jsf?docId=WO2019023220
(see also this thread:
@.***/message/C5DGLMJYKEV72RD7J2FWNS4CLWBXYFNA/)
Cheers,
Martin

authors: LarsHuebner-LHNW WarrenWeckesser charris e-q endolith ewmoore garrettj403 goriccardo hgomersall ilayn larsoner llohse mjmdavis mreineck rgommers stefanv tsangiotis tylerjereddy


text:
ENH: Minor user-friendliness cleanup in LowLevelCallable

Fix some issues in docstring + provide a more friendly error message on failure.

TravisCI is unhappy

The test failures do look unrelated, it's stuff like
FAIL: test_base.Test64Bit.test_no_64(<class 'test_base.TestBSR'>,
'test_elementwise_divide')
FAIL: test_base.TestLIL.test_elementwise_divide

Maybe they go away on restart?

Maybe the failures are due to using Numpy prerelease?

Indeed, gh-6950 has the same failures.

I think the problem is https://github.com/numpy/numpy/pull/8452

Ah, good we ran into that before 1.12.0 final....
There are also some compiler warnings that may be useful to clean up?
In file included from ../scipy/_lib/src/_test_ccallback.c:40:0:
../scipy/_lib/src/ccallback.h: In function ‘ccallback_prepare’:
../scipy/_lib/src/ccallback.h:370:13: warning: passing argument 2 of ‘ccallback__err_invalid_signature’ discards ‘const’ qualifier from pointer target type [enabled by default]
             ccallback__err_invalid_signature(signatures, name);
             ^
../scipy/_lib/src/ccallback.h:201:13: note: expected ‘char *’ but argument is of type ‘const char *’
 static void ccallback__err_invalid_signature(ccallback_signature_t *signatures,
             ^
In file included from /home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1788:0,
                 from /home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/noprefix.h:9,
                 from ../scipy/ndimage/src/nd_image.h:41,
                 from ../scipy/ndimage/src/nd_image.c:32:
/home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
In file included from ../scipy/ndimage/src/nd_image.c:41:0:
/home/rgommers/Code/scipy/scipy/_lib/src/ccallback.h: In function ‘ccallback_prepare’:
/home/rgommers/Code/scipy/scipy/_lib/src/ccallback.h:370:13: warning: passing argument 2 of ‘ccallback__err_invalid_signature’ discards ‘const’ qualifier from pointer target type [enabled by default]
             ccallback__err_invalid_signature(signatures, name);
             ^
/home/rgommers/Code/scipy/scipy/_lib/src/ccallback.h:201:13: note: expected ‘char *’ but argument is of type ‘const char *’
 static void ccallback__err_invalid_signature(ccallback_signature_t *signatures,

In file included from ../scipy/integrate/__quadpack.h:26:0,
                 from ../scipy/integrate/_quadpackmodule.c:4:
/home/rgommers/Code/scipy/scipy/_lib/src/ccallback.h: In function ‘ccallback_prepare’:
/home/rgommers/Code/scipy/scipy/_lib/src/ccallback.h:370:13: warning: passing argument 2 of ‘ccallback__err_invalid_signature’ discards ‘const’ qualifier from pointer target type [enabled by default]
             ccallback__err_invalid_signature(signatures, name);
             ^
/home/rgommers/Code/scipy/scipy/_lib/src/ccallback.h:201:13: note: expected ‘char *’ but argument is of type ‘const char *’
 static void ccallback__err_invalid_signature(ccallback_signature_t *signatures,
             ^
In file included from /home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1788:0,
                 from /home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
                 from /home/rgommers/.local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from ../scipy/integrate/__quadpack.h:28,
                 from ../scipy/integrate/_quadpackmodule.c:4:

Tested on 32-bit to be sure - all is well there.

@rgommers Was this comment meant for this PR or #6950 ?

This PR. But no worries, I'll test yours on 32-bit as well:)

Travis rebuild passes now that the numpy package was updated.

In it goes then, thanks!

source: gharchive/pull-request

authors: jotasi pv rgommers

repo: scipy/scipy

license_type: permissive

license_source: github-api


text:
Fix Jaccard distance between two arrays of all zeros

.../scipy/spatial/distance.py:437: RuntimeWarning: invalid value
encountered in double_scalars
/ np.double(np.bitwise_or(u != 0, v != 0).sum()))

Just in case, this is a duplicate of #7373, originally reported in #7317.

Closing this as a duplicate, gh-7373 looks more complete. Thanks though @SerenoGong !

authors: ElDeveloper SerenoGong rgommers


text:
warning! unknown event group with latest sdk

Not big issue, but annoying.

And what is it exactly? What number does it have?

[sciter] warning! unknown event group 0020
[sciter] warning! unknown event group 0020

BTW, could ypu help answer https://sciter.com/forums/topic/menu-not-be-hidden-when-open-dialog-on-linux/#post-75593?

Huh. How is 020 unknown if it's a size event 🤔

Confirmed that I'm seeing the same thing.

It as indeed annoying. Thanks for the catch!

It as indeed annoying. Thanks for the catch!

source: gharchive/issue

authors: pravic rustdesk snluu

repo: sciter-sdk/rust-sciter

license: MIT


text:
Add user_permissions to leaf nodes

Closes #699
Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

@rentzso would you mind taking a quick look at this? Dan requested there be a reliable way to see the user permissions on leaf nodes like files.

LGTM!

authors: nagem rentzso

repo: scitran/core


text:
🛑 SCNU Online Judge is down

In c678a23, SCNU Online Judge (https://oj.socoding.cn/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SCNU Online Judge is back up in 7509057.

repo: scnu-socoding/status


text:
Application::share() not found in OctoberCMS L5.5 (edge = true)

When updating to the latest edge build the debug bar throws an exception.
Call to undefined method October\Rain\Foundation\Application::share()
C:\xampp\htdocs\www\plugins\bedard\debugbar\vendor\barryvdh\laravel-debugbar\src\ServiceProvider.php line 42
This is the affected line.
$this->app['command.debugbar.clear'] = $this->app->share(
function ($app) {
return new Console\ClearCommand($app['debugbar']);
}
);

Thanks for the heads up, I've just upgraded to be compatible with 5.5!
https://github.com/scottbedard/oc-debugbar-plugin/commit/4fb0b661c2fe59a03c17712dd02b5bb1cb0f481a

authors: Verbat scottbedard

repo: scottbedard/oc-debugbar-plugin

license: mit

license_source: bigquery


text:
Help Request: Using SmartFormat.NET to format complex object properties.

I just found this amazing library today and I have a use case where I currently use reflection to iterate over the properties of a complex object and format the property name and value into strings that then written to a log file for troubleshooting. The method listed below is my current implementation which is a method on my BaseClass object which all other objects in my project inherit from. While I use this function extensively throughout my projects I would much rather have a simple one or two line alternative that would make it easier for the others to read/use when implementing objects created from my base object. Does SmartFormat.NET provide the syntactic sugar for something similar in functionality, either native or via an extension method?
John
/// <summary>
/// Provides a formatted output of all object properties and values using reflection.
/// Example: {"PropertyA" : "PropertyAValue"}, {"PropertyB" : "PropertyBValue"}
/// </summary>
/// <returns>
/// Formatted string of property names/values
/// </returns>
public new string ToString()
{
    const int itemsPerLine = 5;

    var t = GetType();
    var props = t.GetProperties();
    var className = t.Name;

    var returnValue = Environment.NewLine + className + Environment.NewLine;

    var propertiesGroup = from i in Enumerable.Range(0, props.Count())
                            group props[i] by i / itemsPerLine;

    var lastProperty = props.Last();
    foreach (var property in propertiesGroup)
    {
        foreach (var propertyInfo in property)
        {
            var value = propertyInfo.GetValue(this, new object[] { });

            returnValue += "{\"" + propertyInfo.Name + "\" : \"" + value + "\"}";

            if (!propertyInfo.Equals(lastProperty))
                returnValue += ", ";
        }
        returnValue += Environment.NewLine;
    }
    return returnValue;
}

Apologies this is my first post to Github and I was not able to figure out how to submit this item and apply the help tag. Can a moderator/admin please tag this post as a help request?

John,
I'm not sure if SmartFormat is designed to do what you're trying to do.
It sounds like you're trying to serialize your object to your log.  SmartFormat is designed to format objects into strings, but it requires a template, which helps if you know all the property names beforehand.
However, SmartFormat will not help you to iterate through all the properties.  That'll be up to your own code.

Here's one way I could envision SmartFormat helping you:

Gather a list of name/value pairs, using your reflection code above.
Use a SmartFormat template to list out all name/value pairs, using a template.

Here's an attempted example:
public new string ToString()
{
    const int itemsPerLine = 5;

    var t = GetType();
    var className = t.Name;

    var props = t.GetProperties();

	// Get all name/value pairs:
	var properties = propertiesGroup.SelectMany(properties => properties)
		.Select(propertyInfo => new {
			name = propertyInfo.Name,
			value = propertyInfo.GetValue(this, new object[] { })
		});

	// Output the data using this template:
	return Smart.Format("\n{className}\n{properties: {name}: {value}|, }", new { className, properties });
}

The template isn't exactly right, but I didn't want to clutter the code with a bunch of escaped {} braces.

Scott,
Thank you for your quick reply and proposed solution. I had not given any consideration to using Linq to gather a KVP collection as you posted in your solution. Going that route is a much cleaner solution than what I had originally. On a side note this approach got the rusty old gears in my bain turning again on how to improve an issue I have with an MVC ActionFilter on another project as well.
Thank you,
John

Glad to have helped.  Linq's ability to simplify problems sure is awesome.
One other thought that I had; maybe you should consider a JSON serializer, if you're looking for a quick and easy way to dump an object into the log.

authors: johnannis scottrippey


text:
Improvements and fixes for new python client

The PR is created to cover the following minor things found after testing:

[x] fix inconsistencies between project.id - spider.id - job.key (use key or id everywhere)
[x] collection.set() should return None instead of a generator
[x] simplify and unify using HS filter param
[x] activity.add(): add client validation for project id
[x] collections.iter() should iterate through collections
[x] unify errors when apikey is wrong (now it's InvalidUsage or ScrapinghubAPIError)
[x] handle HTTP 401 from JobQ separately with providing a good message instead of html

Other issues added after next step of testing at February 8th:

[x] collection.delete() should return None instead of a generator
[x] add validation for collection.delete input
[x] validate that update_tags method has any input

Each of the points will be implemented in a separate commit(s) to simplify reviewing.

@chekunkov Ok, I think we've done with fixes/improvements, the PR deserves final review and merge.

repo: scrapinghub/python-scrapinghub


text:
🛑 React Round Carousel is down

In 4d02ee1, React Round Carousel (https://react-round-carousel.atanas.info) was down:

HTTP code: 0
Response time: 0 ms

Resolved: React Round Carousel is back up in d32e360.

repo: scriptex/uptime


text:
make particular routes scully static not complete app.

🧩 Feature request
Description
 A clear and concise description of the problem or missing capability...
Describe the solution you'd like
 If you have a solution in mind, please describe it.
Describe alternatives you've considered
 Have you considered any alternative solutions or workarounds?

I'm going to close this because I think this is answered in our FAQ page: Ignore routes without config?.
If that doesn't answer your question, respond in here, and we can reopen.

repo: scullyio/scully


text:
(feat) add local-cache to scully

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/scullyio/scully/blob/main/CONTRIBUTING.md#commit
[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Other... Please describe:

What is the current behavior?

Issue Number: N/A
What is the new behavior?
Does this PR introduce a breaking change?

[ ] Yes
[x] No

Other information

I'm getting the same error

I'm getting the same error

authors: SanderElias jorgeucano


text:
enable sampling along vars axis with AnnLoader

Currently it is only possible to sample batches along obs axis with AnnLoader. Ideally we would be able to do both directions

Hi, @gszep, do you need to iterate through both observations and variables at the same time? Because if not, you can transpose your adata object and pass to AnnLoader.

I need to iterate through both at the same time unfortunately. Constantly transposing obs and var axes is too confusing. I've submitted a PR getting AnnData most of the way there :)

also its not possible to transpose views without copying

authors: Koncopd gszep

repo: scverse/anndata


text:
Set name for each row and panel

The names are invisible for users, but make it easier to maintain and update the dashboard JSON file

Do you mean Grafana 5?
Current Grafana release is 4.6.3
https://github.com/scylladb/scylla-grafana-monitoring/blob/master/start-grafana.sh#L10

yes, I meant grafana 5

@tzach grafana 5 no longer support rows, we can probably close this

Closing it based on all panels have descriptions, rows are no longer supported

authors: amnonh tzach

repo: scylladb/scylla-grafana-monitoring


text:
cloudformation: first instance should be node1

Currently, the first instance is created as node0.

Can someone explain better what is this task?
(I see it's from 2021 and "Large")

Semantic change only - the first node created by the CF stuck, named Node0 and it should be Node1
(the "Large" is a mistake)
But... for now, we're not going to move on with CF tasks unless we'll have a customer request

authors: benipeled roydahan

repo: scylladb/scylla-machine-image


text:
New deserialization API - preparations

This PR consists the first bunch of minor preparatory steps for the main leap of the deserialization refactor, extracted from #1057 as @Lorak-mmk requested.
For explanation and broader context see #1057.
Ref: #462
Pre-review checklist

[x] I have split my patch into logically separate commits.
[x] All commit messages clearly explain what they change and why.
[ ] I added relevant tests for new features and bug fixes.
[x] All commits compile, pass static checks and pass test.
[x] PR description sums up the changes and reasons why they should be introduced.
[x] I have provided docstrings for the public items that I want to introduce.
[ ] I have adjusted the documentation in ./docs/source/.
[ ] I added appropriate Fixes: annotations to PR description.

v1.1: reverted paging_state.clone() to paging_state.take() in iterator.

v1.2: rebased on main.

v1.2.1: fixed compile error on MSRV due to: error[E0277]: the trait bound [u8]: Default is not satisfied

repo: scylladb/scylla-rust-driver


text:
Popup 无法弹出页面

// 按照官方的例子，这样做，但是弹不出页面，怎么做？

Open About Popup

Open Services Popup

About

Close popup

Lorem ipsum dolor ...

Services

Close popup

Lorem ipsum dolor ...

同样遇到这个问题，请问解决了吗

@yuan1995913 后面用了其他方法。

我的生活

<link rel="stylesheet" href="//g.alicdn.com/msui/sm/0.6.2/css/sm.min.css">
<link rel="stylesheet" href="//g.alicdn.com/msui/sm/0.6.2/css/sm-extend.min.css">

Popup(JavaScript)

        ...

Open About Popup
Open Services Popup

About
Close popup
Lorem ipsum dolor ...

Services
Close popup
Lorem ipsum dolor ...

官方那个肯定遗漏什么

`

我的生活

<link rel="stylesheet" href="//g.alicdn.com/msui/sm/0.6.2/css/sm.min.css">
<link rel="stylesheet" href="//g.alicdn.com/msui/sm/0.6.2/css/sm-extend.min.css">

Popup(JavaScript)

        ...

Open About Popup
Open Services Popup

About
Close popup
Lorem ipsum dolor ...

Services
Close popup
Lorem ipsum dolor ...

`

<!DOCTYPE html>

    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>我的生活</title>
    <meta name="viewport" content="initial-scale=1, maximum-scale=1">
    <link rel="shortcut icon" href="/favicon.ico">
    <meta name="apple-mobile-web-app-capable" content="yes">
    <meta name="apple-mobile-web-app-status-bar-style" content="black">

    <link rel="stylesheet" href="//g.alicdn.com/msui/sm/0.6.2/css/sm.min.css">
    <link rel="stylesheet" href="//g.alicdn.com/msui/sm/0.6.2/css/sm-extend.min.css">

Popup(JavaScript)

        ...

Open About Popup

Open Services Popup

About

Close popup

Lorem ipsum dolor ...

Services

Close popup

Lorem ipsum dolor ...

repo: sdc-alibaba/SUI-Mobile


text:
EMA always return 0.00

Hello, i used your code with data and ... nothing works. For example EMA always gives 0
Volume:	870.19 Time:	2021-11-28T07:00:00 -> 2021-11-28T08:00:00
Open:	54377.42
Close:	54475.96
High:	54590.48
Low:	54173.81
Volume:	787.07 Time:	2021-11-28T08:00:00 -> 2021-11-28T09:00:00
Open:	54475.97
Close:	54515.61
High:	54636.65
Low:	54396.33
Volume:	985.20 Time:	2021-11-28T09:00:00 -> 2021-11-28T10:00:00
Open:	54515.60
Close:	54396.20
High:	54682.90
Low:	54231.92
Volume:	823.84 Time:	2021-11-28T10:00:00 -> 2021-11-28T11:00:00
Open:	54393.87
Close:	54335.80
High:	54475.00
Low:	54130.72
Volume:	765.69 Time:	2021-11-28T11:00:00 -> 2021-11-28T12:00:00
Open:	54335.80
Close:	54718.15
High:	54933.80
Low:	54184.60
Volume:	1028.53 Time:	2021-11-28T12:00:00 -> 2021-11-28T13:00:00
Open:	54718.15
Close:	54493.09
High:	54875.00
Low:	54425.27
Volume:	1030.39 Time:	2021-11-28T13:00:00 -> 2021-11-28T14:00:00
Open:	54493.10
Close:	54356.62
High:	54635.52
Low:	54263.16
Volume:	909.93 Time:	2021-11-28T14:00:00 -> 2021-11-28T15:00:00
Open:	54356.62
Close:	54274.20
High:	54396.33
Low:	54203.13
Volume:	777.66 Time:	2021-11-28T15:00:00 -> 2021-11-28T16:00:00
Open:	54274.20
Close:	54365.00
High:	54555.00
Low:	54029.99
Volume:	1130.34 Time:	2021-11-28T16:00:00 -> 2021-11-28T17:00:00
Open:	54365.00
Close:	54296.14
High:	54450.00
Low:	54169.84
Volume:	753.77 Time:	2021-11-28T17:00:00 -> 2021-11-28T18:00:00
Open:	54296.14
Close:	54188.43
High:	54373.00
Low:	53942.01
Volume:	1277.10 Time:	2021-11-28T18:00:00 -> 2021-11-28T19:00:00
Open:	54188.43
Close:	53896.37
High:	54287.25
Low:	53800.00
Volume:	1358.34 Time:	2021-11-28T19:00:00 -> 2021-11-28T20:00:00
Open:	53896.36
Close:	54108.99
High:	54186.17
Low:	53256.64
Volume:	2958.13 Time:	2021-11-28T20:00:00 -> 2021-11-28T21:00:00
Open:	54108.99
Close:	54617.85
High:	54967.50
Low:	54044.98
Volume:	2249.23 Time:	2021-11-28T21:00:00 -> 2021-11-28T22:00:00
Open:	54617.85
Close:	54918.51
High:	55149.99
Low:	54617.84
Volume:	1304.77 Time:	2021-11-28T22:00:00 -> 2021-11-28T23:00:00
Open:	54918.51
Close:	56273.23
High:	56390.00
Low:	54863.01
Volume:	3117.97 Time:	2021-11-28T23:00:00 -> 2021-11-29T00:00:00
Open:	56273.23
Close:	56029.82
High:	56729.72
Low:	56023.01
Volume:	2427.77 Time:	2021-11-29T00:00:00 -> 2021-11-29T01:00:00
Open:	56029.81
Close:	57274.88
High:	57445.05
Low:	56000.00
Volume:	3468.79 Time:	2021-11-29T01:00:00 -> 2021-11-29T02:00:00
Open:	57274.89
Close:	57765.73
High:	58000.15
Low:	57136.56
Volume:	3073.53 Time:	2021-11-29T02:00:00 -> 2021-11-29T03:00:00
Open:	57765.73
Close:	57639.76
High:	58242.09
Low:	57501.99
Volume:	2729.83 Time:	2021-11-29T03:00:00 -> 2021-11-29T04:00:00
Open:	57643.14
Close:	57309.25
High:	57715.42
Low:	57269.72
Volume:	1769.15 Time:	2021-11-29T04:00:00 -> 2021-11-29T05:00:00
Open:	57309.26
Close:	57359.02
High:	57426.95
Low:	57176.00
Volume:	1648.18 Time:	2021-11-29T05:00:00 -> 2021-11-29T06:00:00
Open:	57359.02
Close:	57249.58
High:	57642.93
Low:	57232.00
Volume:	1358.98 Time:	2021-11-29T06:00:00 -> 2021-11-29T07:00:00
Open:	57249.58
Close:	57325.75
High:	57384.99
Low:	57200.00
Volume:	801.75 Time:	2021-11-29T07:00:00 -> 2021-11-29T08:00:00
Open:	57325.76
Close:	57426.91
High:	57500.63
Low:	57309.65
Volume:	1002.85 Time:	2021-11-29T08:00:00 -> 2021-11-29T09:00:00
Open:	57426.92
Close:	57515.13
High:	57688.00
Low:	57390.00
Volume:	1047.27 Time:	2021-11-29T09:00:00 -> 2021-11-29T10:00:00
Open:	57512.17
Close:	57514.33
High:	57642.85
Low:	57455.08
Volume:	174.91]}

EMA output: 0.00

The code :
df := techan.NewTimeSeries()

	for _, k := range *klines {
		// opentime, open, high, low, close, volume
		period := techan.NewTimePeriod(time.Unix(k.OpenTime/1000, 0), *duration)

		candle := techan.NewCandle(period)
		candle.OpenPrice = big.NewFromString(k.Open)
		candle.ClosePrice = big.NewFromString(k.Close)
		candle.MaxPrice = big.NewFromString(k.High)
		candle.MinPrice = big.NewFromString(k.Low)
		candle.Volume = big.NewFromString(k.Volume)

		df.AddCandle(candle)
	}

	fmt.Println(df)
        closePrices := techan.NewClosePriceIndicator(df)
	movingAverage := techan.NewEMAIndicator(closePrices, 10) // Create an exponential moving average with a window of 10

	fmt.Println("EMA output :" + movingAverage.Calculate(0).FormattedString(2))

@Chipsterjulien Hi!
I think the problem is in el Calculate call, as the window size is 10, the Calculate will return 0 from index 0 up to 9, as it has not enough data to fill the window.
Try this:
movingAverage.Calculate(9).FormattedString(2)

If it does not work, take a look if you are adding the candles right. Seems they have to be added time-ordered.

@Chipsterjulien did you ever get this working?

repo: sdcoffey/techan


text:
poetry install fails if specifying full python version in pyproject.toml

[x] I am on the latest Poetry version.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS 10.14
Poetry version: 0.12.4

Issue
If the python version version specified in pyproject.toml includes the patch version poetry install will fail when it attempts to install the current package in development mode:
[tool.poetry.dependencies]
python = "2.7.15"

Produces:
[EnvCommandError]
Command ['/Users/<User>/.pyenv/versions/2.7.15/envs/app/bin/pip', 'install', '-e', '/Users/<User>/Documents/Source/app/py', '--no-deps'] errored with the following output:
Obtaining file:///Users/<User>/Documents/Source/app/py
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
    Complete output from command python setup.py egg_info:
    error in app setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '2.7.15'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /Users/<User>/Documents/Source/app/py/
You are using pip version 18.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Exception trace:
 /Users/<User>/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/<User>/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/<User>/.poetry/lib/poetry/_vendor/py2.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/<User>/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/<User>/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/<User>/.poetry/lib/poetry/_vendor/py2.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/<User>/.poetry/lib/poetry/console/commands/install.py in handle() at line 89
   self.env.run("pip", "install", "-e", str(setup.parent), "--no-deps")
 /Users/<User>/.poetry/lib/poetry/utils/env.py in run() at line 472
   return super(VirtualEnv, self).run(bin, *args, **kwargs)
 /Users/<User>/.poetry/lib/poetry/utils/env.py in run() at line 288
   raise EnvCommandError(e)

[tool.poetry.dependencies]
python = "~2.7"

Works fine.

Commit bdd7909 should fix this.
It will be released in the next bugfix version.

Release 0.12.5 is out!

authors: rainyday sdispater

repo: sdispater/poetry