text:
Added two plotting functions and more support for named dimensions.

This pull-request partially solves issue #576.
I have added some more support for named dimensions in the search-space, as well as two plotting functions. I don't have time to fix your existing plotting functions but I have made TODO comments for what needs to be done in plots.py
Before I start, let me repeat that I really like your library! So please take the following as friendly and constructive feedback. I spent the whole Saturday from morning to evening working on this and half of this Sunday. A large part of yesterday was just trying to understand your code because of the severe lack of comments. I spent this much effort because I think your library is important and I hope you will polish it some more.
Please note that I have to travel tomorrow so it might be a while before I can respond if you have questions / comments to this PR.
How to test this pull-request
The following code uses the additions of this PR by simulating an objective function for tuning the hyper-parameters of a simple neural network. It will save a few plots to disk and print various info. It should work if you copy this to a python-file and run it.
(And yes! My code is also ugly when I'm writing it. I polish it when it is done and works. But this code is not for release :-)
import numpy as np
from math import exp

from skopt import gp_minimize
from skopt.space import Real, Categorical, Integer
from skopt.plots import plot_histogram, plot_contour, plot_objective

dim_learning_rate = Real(name='learning_rate', low=1e-6, high=1e-2, prior='log-uniform')
dim_num_dense_layers = Integer(name='num_dense_layers', low=1, high=5)
dim_num_dense_nodes = Integer(name='num_dense_nodes', low=5, high=512)
dim_activation = Categorical(name='activation', categories=['relu', 'sigmoid'])

dimensions = [dim_learning_rate,
              dim_num_dense_layers,
              dim_num_dense_nodes,
              dim_activation]

default_parameters = [1e-4, 1, 64, 'relu']

def model_fitness(x):
    learning_rate, num_dense_layers, num_dense_nodes, activation = x

    fitness = ((exp(learning_rate) - 1.0) * 1000) ** 2 + \
              (num_dense_layers) ** 2 + \
              (num_dense_nodes/100) ** 2

    fitness *= 1.0 + 0.1 * np.random.rand()

    if activation == 'sigmoid':
        fitness += 10

    return fitness

print(model_fitness(x=default_parameters))

search_result = gp_minimize(func=model_fitness,
                            dimensions=dimensions,
                            n_calls=40,
                            x0=default_parameters)

print(search_result.x)
print(search_result.fun)

for fitness, x in sorted(zip(search_result.func_vals, search_result.x_iters)):
    print(fitness, x)

space = search_result.space

print(search_result.x_iters)

print(space.to_dict(x=default_parameters))

print("Plotting now ...")

# fig = plot_histogram(result=search_result, dimension_id='activation')
fig = plot_histogram(result=search_result, dimension_id='learning_rate', bins=20)
# fig = plot_histogram(result=search_result, dimension_id='num_dense_layers', bins=20)
# fig = plot_histogram(result=search_result, dimension_id='num_dense_nodes', bins=20)
fig.savefig('histogram.png')

fig = plot_contour(result=search_result,
                     dimension_id1='learning_rate',
                     dimension_id2='num_dense_nodes')
fig.savefig('contour_learning_rate_vs_num_dense_nodes.png')

fig = plot_contour(result=search_result,
                     dimension_id1='num_dense_layers',
                     dimension_id2='num_dense_nodes')
fig.savefig('contour_num_dense_layers_vs_num_dense_nodes.png')

print("Done plotting!")

Comments about this PR

I am completely new to your library so it is possible I have misunderstood how something works. Please feel free to make changes to the code I have added. But please keep the good commenting and coding style as it will help people in the future who have to understand and maintain this code.

I am not familiar with the syntax you use for doc-strings. I have tried to imitate it. I normally use PyCharm which has a different syntax for doc-strings. It is one of the things I really don't like about Python, that you don't have to specify data-types in the function-declaration, but then you often have to do it in the doc-strings using a strange syntax. It's a really poor language-design. But anyway, please check if my docstrings are correct.

Some of the functionality I have added such as space.get_dimensions() is not actually used by my code. It is intended to be helpful for when you fix the other plotting functions to support named dimensions and search-spaces with categories, etc.

I added the function space.to_dict(). There is a somewhat related function in utils.point_asdict(). But I prefer the name to_dict() which is also used by Pandas. I realize it is bad to have two naming conventions in one library. Could I convince you to change the name of asdict()? :-)

Comments about your existing code
You REALLY need to make better comments in your code! It is very difficult for others to understand what you want to do and how everything fits together. When there is a bug we have to guess what your intentions are with the code. If you don't have time to maintain this library in the future, it will probably become abandoned because others will find it easier to just start a new project rather than to try and understand and build on yours. That would be a pity, because you have some outstanding ideas in your library!
For example, this is one of the few comments in plots.py:
# plots on the diagonal are special, like Texas. They have
# their own range so do not mess with them.

I had to read this numerous times to finally understand that it was probably a joke
about Texas USA, although I'm still not 100% sure. You really have to be a master at writing comments in order to pull off jokes, otherwise it will be confusing and harmful to the people who try to understand your code. I personally don't write jokes in code and I am very reluctant to make jokes in my video tutorials as well, for the same reason.
Please look at the coding and commenting style in the functions I have added. You may think that I write too many comments, but what I am aiming for is to explain as much as possible in plain English. Every time you have a function call into skopt or matplotlib or some other package, the reader needs to look up the semantics of that function. If that is also poorly documented, reading your code becomes like solving a giant puzzle. It takes immense effort and bugs are allowed to propagate very easily. When everything is commented in plain English, you can read and understand the code very quickly.
Another issue is your actual coding style. I found bugs as well as code that was correct but not very good. For example, these nested loops are typical in plots.py:
for i in range(space.n_dims):  # rows
    for j in range(space.n_dims):  # columns
        # ...
        if i != j:
            # ...
            if j > 0:
                # ...

Firstly, you should have a local variable called n_dims instead of always referring to space.n_dims. Secondly, your nesting depth is sometimes 4 or 5 inside these for-loops, because you haven't been very precise in your loop-conditions. You can rewrite the for-loops to something like the following. It drastically reduces the nesting-depth and also only executes the loops that are necessary:
for i in range(n_dims):
    # Do something for the diagonal case.

    for j in range(i):
        # Do something for the case where j<i.

    for j in range(i+1, n_dims):
        # Do something for the case where j>i.

You also have this code in plots.py which is incredibly difficult to understand because of the multiple nestings and list-comprehension:
diagonal_ylim = (np.min([ax[i, i].get_ylim()[0]
                         for i in range(space.n_dims)]),
                 np.max([ax[i, i].get_ylim()[1]
                         for i in range(space.n_dims)]))

It is better to split code like this into several lines to avoid the complicated nesting and also make it easier to step-debug. I added a revised version to the code, although I've commented it out for now so you can test it properly.
Another thing is that it is quite confusing how the Space and Dimension objects interact and wrap each other, e.g. regarding the transformation. It appears that the transformation and its inverse may be carried out repeatedly to nullify each other, although I'm not sure.

Hello @Hvass-Labs! Thanks for submitting the PR.

In the file skopt/plots.py, following are the PEP8 issues :

Line 135:80: E501 line too long (82 > 79 characters)
Line 173:80: E501 line too long (89 > 79 characters)
Line 308:80: E501 line too long (83 > 79 characters)
Line 528:1: E302 expected 2 blank lines, found 1
Line 643:80: W291 trailing whitespace

In the file skopt/space/space.py, following are the PEP8 issues :

Line 337:80: E501 line too long (89 > 79 characters)
Line 443:80: E501 line too long (93 > 79 characters)
Line 754:80: E501 line too long (111 > 79 characters)
Line 785:80: E501 line too long (84 > 79 characters)
Line 792:80: E501 line too long (83 > 79 characters)
Line 809:1: W293 blank line contains whitespace
Line 829:80: E501 line too long (86 > 79 characters)

I have tested the code in this PR on the actual problem I am trying to solve, which is hyper-parameter optimization of a neural network. It is for one of my tutorials on TensorFlow but it is not yet released as I want to include plots from skopt.
I have a question for you regarding the interpretation of the plots. The plots I have here are similar to the ones in your original version, except that I have normalized the color-gradients.
The parameters for my search-space are:

learning_rate (1e-6 to 1e-2, log-uniform)
num_dense_layers (1 to 5)
num_dense_nodes (5 to 512)
activation (relu or sigmoid)

These are the fitness values and neural network parameters found by skopt. The fitness is the negative classification accuracy on a validation-set.
[(-0.99019999999999997, [0.0010475183322128938, 1, 149, 'relu']),
(-0.98860000000000003, [0.0018261589189723629, 2, 470, 'relu']),
(-0.98839999999999995, [0.00037986474511170302, 2, 512, 'relu']),
(-0.98839999999999995, [0.0010415672190332141, 1, 512, 'relu']),
(-0.98799999999999999, [0.0024978263671842247, 3, 504, 'relu']),
(-0.98760000000000003, [0.0017094485608837695, 1, 512, 'relu']),
(-0.98680000000000001, [0.0017476496180086432, 3, 512, 'relu']),
(-0.98499999999999999, [0.00077047010944170677, 3, 43, 'relu']),
(-0.98480000000000001, [0.0014475695468873279, 3, 220, 'relu']),
(-0.98260000000000003, [0.01, 4, 198, 'relu']),
(-0.97919999999999996, [0.0014732190199105925, 3, 286, 'sigmoid']),
(-0.97499999999999998, [0.0022674676715302932, 3, 481, 'sigmoid']),
(-0.97419999999999995, [4.4040787332101281e-05, 4, 512, 'relu']),
(-0.97419999999999995, [7.1181229112317682e-05, 4, 498, 'relu']),
(-0.9698, [0.01, 2, 5, 'relu']),
(-0.96440000000000003, [0.01, 4, 461, 'relu']),
(-0.96179999999999999, [0.0001, 1, 64, 'relu']),
(-0.96099999999999997, [3.4301200074707198e-05, 3, 512, 'relu']),
(-0.95979999999999999, [0.01, 3, 5, 'relu']),
(-0.91579999999999995, [0.01, 5, 5, 'relu']),
(-0.90759999999999996, [0.00076921095317043851, 3, 5, 'relu']),
(-0.88200000000000001, [1.4842453224480901e-05, 1, 88, 'relu']),
(-0.82620000000000005, [5.7751197554451486e-06, 2, 238, 'relu']),
(-0.64780000000000004, [1.8824150512622488e-06, 5, 259, 'relu']),
(-0.53779999999999994, [0.00049781616995792967, 4, 5, 'relu']),
(-0.47120000000000001, [9.9999999999999995e-07, 1, 512, 'relu']),
(-0.11260000000000001, [1.3010975577257652e-06, 3, 456, 'sigmoid']),
(-0.11260000000000001, [7.553247055497371e-06, 4, 209, 'sigmoid']),
(-0.11260000000000001, [0.0049652350761811535, 1, 5, 'sigmoid']),
(-0.107, [0.0017182189692616892, 5, 49, 'sigmoid'])]
The min and max of the fitness values in result.func_vals are -0.9902 and -0.107.
In the code for plot_contour() that I have submitted in this PR, I have normalized the color-gradients of all these plots to use these min and max values. This is an attempt at giving a consistent interpretation of the color-scales between the different dimensions, because their ranges sometimes vary significantly.
(1) This picture shows the learning_rate vs. num_dense_layers. You will note the red star which shows the best parameters found, and the yellow area which is presumably the minimum area of the Gaussian Process? The zi returned by partial_dependence() ranges between -0.99 and -0.34 but the plot-colors are normalized to range between -0.9902 and -0.107 which are the extreme values from the optimization results.

(2) This next picture shows the learning_rate vs. num_dense_nodes where the zi from partial_dependence() now only ranges between -0.73 and -0.44. Because of the normalization I do using vmin and vmax in my new plot_contour() function, the color-gradients are now almost just completely green.

(3) This next picture shows the num_dense_layers vs. num_dense_nodes, whose zi ranges between -0.78 and -0.47. Again because of the color-normalization in plot_contour() this is almost completely green, with a slight yellow in the center.

I'm a bit confused about how to interpret these plots. Can you perhaps explain that to me? Is it because e.g. plot (3) for num_dense_layers vs. num_dense_nodes "averages" the fitness over the other remaining dimensions for learning_rate and activation? How should I interpret this? Does it mean that the performance is unpredictable from these two variables alone?
But if we look at plot (1) which shows a yellow area in the middle, does this mean that it is "generally" better to have a learning_rate around 0.002 and num_dense_layers of 3. But the best solution had num_dense_layers=1. Is it because the Gaussian Process is incapable of modelling this search-space accurately? Or is it because we are averaging the performance over the remaining variables, so it is possible that for particular choices of the other variables, the best point can fall outside of the yellow area?
Another thing is that these plots change each time I run gp_minimize(). Does that indicate I should run a lot more iteration than the 30 I have done here? The problem is of course that it is very costly to run each of these, as it is a full training of a neural network.
I hope you can enlighten me as to what is going on with these plots :-)

Codecov Report

Merging #579 into master will decrease coverage by 2.73%.
The diff coverage is 14.28%.

@@            Coverage Diff             @@
##           master     #579      +/-   ##
==========================================
- Coverage   86.24%   83.51%   -2.74%
==========================================
  Files          23       23
  Lines        1716     1783      +67
==========================================
+ Hits         1480     1489       +9
- Misses        236      294      +58

Impacted Files
Coverage Δ

skopt/utils.py
97.31% <ø> (ø)
:arrow_up:

skopt/plots.py
8.33% <10.86%> (+0.79%)
:arrow_up:

skopt/space/space.py
87.83% <20.83%> (-5.26%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01ec087...5627c7e. Read the comment docs.

Thanks a lot for your hard work. I think currently everyone is travelling so it will take some time for looking at this seriously.
The doc string format we try to follow https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt this is not entirely successful/uniform across the code base, yet.
The code could (always) be commented better or restructured. It probably also depends on each persons history on how confusing they find different idioms. The way forward is to add comments or propose changes like you did. Leaving comments about work to be done/homework for others (empirically) does not lead to things getting done. So I vote to remove them or turn them into comments about the code.

I get the impression that the plotting functions will not get fixed by anyone else and since I really would like this to work, I'll try and take a look at it. I'll probably have to change the existing plotting functions and we'll have to discuss later if there are API changes etc. E.g. the naming scheme you have now is a bit strange when you already have names in the dimensions, so I'll definitely change that.
@betatim thanks for all the comments, they were quite helpful and I'll try and incorporate your suggestions.
Regarding comments in code, it is the responsibility of the person adding new code to properly document it. He is the only person who really understands what the code is supposed to do. You are hoping for a miracle if you think others will come along and fix your work for you. I will help you out a bit, but to be perfectly honest, it is quite arrogant to add code that isn't properly polished and commented, because if there are 1000 people who will be using the code and they waste just a few hours each, then it is thousands of man-hours that are wasted, because the original author didn't take an hour or two more to document the code. It is not just a problem with your project but a general problem. The solution is simply that you don't accept PR's until they are properly documented so you understand what is going on and you feel the API is sufficiently elegant. Like you are doing with this PR now :-)
PS: I really could use an explanation for the plots that I showed above, so I can understand what is going on and explain it in my tutorial. Perhaps I should have opened a separate thread for that question.

All PRs are reviewed by someone who is not the original author before they are merged. This means there are at least two people who are happy with the state of things. The code base is maintained by people who are experienced with python, numpy, matplotlib and scikit-learn, as a result it relies on common idioms and conventions from the PyData ecosystem. This can result in some pretty dense or hard to read code. The way out is for an outsider (who doesn't view the world through the same biased view) to come along and help improve things.
Some may even disagree on the level of comments/understandability. To avoid bike shedding what the appropriate level of comments, docs, style, etc is we require a concrete proposal via PR. This seems to work quite well.
The miracle is that random people volunteering their time manage to put together a quite well maintained and functioning software project implementing a complex bit of research. There will always be parts that aren't as good as they could be, new things to add, historical legacy that is annoying, etc. Assuming people are already donating as much time and effort as they can, the only way to move things along faster is by donating more effort yourself or donating money to pay for developer time.

As for #590, please remove your inline comments about our coding conventions. This is distracting everyone from the main goal of this PR (i.e., adding two plotting functions). If you want to propose changes to improve how to code looks and is read, we welcome that, but please make self-contained targeted PRs.

I have now pushed a new version. Almost the entire plots.py has been rewritten.
This example tests the new functions using the built-in hart6 benchmark function.
from skopt import forest_minimize
from skopt.space import Real, Categorical, Integer
from skopt.benchmarks import hart6
from skopt.plots import plot_objective_matrix, plot_evaluations_matrix
from skopt.plots import plot_histogram, plot_objective_2D

dim1 = Real(name='foo', low=0.0, high=1.0)
dim2 = Real(name='bar', low=0.0, high=1.0)
dim3 = Real(name='baz', low=0.0, high=1.0)
dim4 = Real(name='rip', low=0.0, high=1.0)
dim5 = Real(name='rap', low=0.0, high=1.0)
dim6 = Real(name='rup', low=0.0, high=1.0)

dimensions = [dim1, dim2, dim3, dim4, dim5, dim6]

print("Optimizing ...")

result = forest_minimize(hart6, dimensions, n_calls=200,
                         base_estimator="ET", random_state=4)

print("Plotting ...")

fig, ax = plot_evaluations_matrix(result=result, bins=20, dimension_names=None)
fig.savefig('plot_evaluations_matrix_all.png')

fig, ax = plot_objective_matrix(result=result, dimension_names=None)
fig.savefig('plot_objective_matrix_all.png')

# Now use a subset of the dimensions identified by their names.
# Note the order is different from the search-space,
# which will also be reflected in the plots.
dim_names = ['bar', 'rip', 'foo']

fig, ax = plot_evaluations_matrix(result=result, bins=20, dimension_names=dim_names)
fig.savefig('plot_evaluations_matrix_some.png')

fig, ax = plot_objective_matrix(result=result, dimension_names=dim_names)
fig.savefig('plot_objective_matrix_some.png')

fig, ax = plot_histogram(result=result, dimension_name='baz', bins=40)
fig.savefig('plot_histogram.png')

fig, ax = plot_objective_2D(result=result,
                            dimension_name1='baz',
                            dimension_name2='rap')
fig.savefig('plot_objective_2D.png')

print("Done plotting!")

These are the resulting plots:

Here is another example using a "simulated" Neural Network for the objective function. This has a Categorical dimension which cannot be plotted.
import numpy as np
from math import exp
from skopt import forest_minimize
from skopt.space import Real, Categorical, Integer
from skopt.plots import plot_objective_matrix, plot_evaluations_matrix
from skopt.plots import plot_histogram, plot_objective_2D

dim_learning_rate = Real(name='learning_rate', low=1e-6, high=1e-2, prior='log-uniform')
dim_num_dense_layers = Integer(name='num_dense_layers', low=1, high=5)
dim_num_dense_nodes = Integer(name='num_dense_nodes', low=5, high=512)
dim_activation = Categorical(name='activation', categories=['relu', 'sigmoid', 'linear'])

dimensions = [dim_learning_rate,
              dim_num_dense_layers,
              dim_num_dense_nodes,
              dim_activation]

default_parameters = [1e-4, 1, 64, 'relu']

def model_fitness(x):
    learning_rate, num_dense_layers, num_dense_nodes, activation = x

    fitness = ((exp(learning_rate - 1e-3) - 1.0) * 5000) ** 2 + \
          (num_dense_layers - 3) ** 2 + \
          ((num_dense_nodes - 200)/100) ** 2

    fitness *= 1.0 + 0.05 * np.random.rand()

    if activation == 'sigmoid':
        fitness += 10
    elif activation == 'linear':
        fitness += 20

    return fitness

print("Optimizing ...")

result = forest_minimize(model_fitness, dimensions, n_calls=200,
                         base_estimator="ET", random_state=4)

print(result.x)
print(result.fun)

for fitness, x in sorted(zip(result.func_vals, result.x_iters)):
    print(fitness, x)

print("Plotting ...")

try:
    fig, ax = plot_evaluations_matrix(result=result, bins=20, dimension_names=None)
    fig.savefig('plot_evaluations_matrix_all.png')

    fig, ax = plot_objective_matrix(result=result, dimension_names=None)
    fig.savefig('plot_objective_matrix_all.png')
except:
    print("Cannot plot Categorical dimensions! Skipping ...")

# Now use a subset of the dimensions identified by their names.
# Note the order is different from the search-space,
# which will also be reflected in the plots.
dim_names = ['num_dense_nodes', 'learning_rate', 'num_dense_layers']

fig, ax = plot_evaluations_matrix(result=result, bins=20, dimension_names=dim_names)
fig.savefig('plot_evaluations_matrix_some.png')

fig, ax = plot_objective_matrix(result=result, dimension_names=dim_names)
fig.savefig('plot_objective_matrix_some.png')

fig, ax = plot_histogram(result=result, dimension_name='learning_rate', bins=40)
fig.savefig('plot_histogram1.png')

fig, ax = plot_histogram(result=result, dimension_name='activation', bins=40)
fig.savefig('plot_histogram2.png')

fig, ax = plot_objective_2D(result=result,
                            dimension_name1='learning_rate',
                            dimension_name2='num_dense_nodes')
fig.savefig('plot_objective_2D.png')

print("Done plotting!")

These are the resulting plots:

Let me once again start out by saying that I really like the idea of your project! My issue is with the implementation and the code "style" of certain parts of the project.
I rewrote almost the entire plots.py except for plot_convergence() which I don't want to spend hours understanding and rewriting. I'm not happy about leaving a file that I've almost completely rewritten, but frankly I've had enough of reading your code and feel my time is better spent elsewhere.
I would also like to add the plotting functions from: https://scikit-optimize.github.io/notebooks/bayesian-optimization.html to plots.py but I'll await your comments on this PR before I spend probably a whole day understanding and rewriting that.
From our previous discussions I get the impression that you don't like comments inside code. You find comments to be redundant or whatever. So I expect that you will either demand that I remove all those comments I added to plots.py, or you will bull-doze them the next time you work on the code yourself. At any rate, I get the impression that you look down on comments in code.
Here is the truth: Code with few comments might be OK if your code is really fantastic and beautiful, and your audience is very familiar with all the API's you are using, the math and methods, etc.
But your code is far from fantastic and I repeatedly got the impression that plots.py was written by someone without formal training in computer science and software engineering. It would certainly not get a passing grade where I come from. And when the code is poorly written AND has almost no comments, then it is really problematic for others to work on. This is what I meant by an arrogant coding style, because it shows complete disregard for whoever is going to work on the code next. Fair enough if the person is not a masterful coder - but everyone can write proper comments in English!
Fact is that the code was broken, and since you didn't want to fix it yourselves, someone from the outside would have to do it for you. If the code had been well-written or at least well-documented and well-commented, then it could have been fixed in a fraction of the time I have now spent. Instead I feel I've wasted almost a week understanding and cleaning up your code, because you didn't even want to add comments that showed what the intention of the code was, so I had to spend hours upon hours trying to deduce it.
And when you talk about "programming idioms" and "complex research" and that is why I might find your code difficult to understand, it is also rather arrogant, because what you're really saying is that you think people who don't understand your code must be ignorant or stupid. But in reality you are just making poor excuses for writing bad code!
So it has been very frustrating working on your code!
Let me ask you two questions:

Do you think someone in the future will find it easier to work on my version of plots.py or your old version? If you ask 100 programmers to fix a bug or extend plots.py with new features, I bet you that most of them would give up trying to understand your old code and they would just hack it until it maybe worked. This would bloat the code and quite possibly introduce new bugs. But even first-year students can now understand my version of plots.py.

Would you rather have someone waste a week trying to understand code you neglected to spend a few hours commenting when you wrote it, or would you rather have that person spend a week contributing something more meaningful to your project?

It is obvious that you are getting quite annoyed with me, but I really hope you take this criticism to heart. You don't have to like me, but at least consider that what I am saying might be reasonable and would significantly improve your project. Scikit-optimize could be a fantastic library! - but you need to improve the code-base so others can understand it and help you extend it, especially when you no longer work on it yourself!
Best wishes and Happy New Years! :-)

authors: Hvass-Labs betatim codecov-io glouppe pep8speaks

repo: scikit-optimize/scikit-optimize


text:
Release notes links point to module rather than section

I would expected the links in the release notes to point to the section in the notes rather than to the module docs.
E.g. to https://docs.scipy.org/doc/scipy/reference/release.1.4.0.html#scipy-stats-improvements
rather than to https://docs.scipy.org/doc/scipy/reference/stats.html#module-scipy.stats
I'm happy to work on a fix if this is not intentional and others would also expect  the behaviour I had.

This confused me for a little while. I now see what you mean:

The sidebar on the right has the links to the doc sections of the release notes, the ones on the left are to the refguide module overview pages. The reason for the latter is that they're the links from the section headers themselves:

Making those links point to themselves makes little sense though. So probably the right thing to do here is to remove the link from the section headers completely, and make the main ToC a normal Sphinx TOC - that will then have the same links as in the right sidebar.

oh, somehow I never noticed the sidebar, so much focused on the middle content 😅

probably the right thing to do here is to remove the link from the section headers completely, and make the main ToC a normal Sphinx TOC - that will then have the same links as in the right sidebar.

@rgommers do you still want something done here? The contents entries still link to the module docs, but we no longer have the links per module in the right-sidebar.

This looks reasonable to me now:

I'd be fine with closing it.

Yeah, I agree with @bsipocz that it seems unintuitive for the links to go to the module ref docs, but probably only worth changing if there's something easy and clearly better. Any thoughts @bsipocz ?

source: gharchive/issue

authors: bsipocz lucascolley rgommers

repo: scipy/scipy

license_type: permissive

license_source: github-api


text:
ENH: Add Distance Correlation to scipy.stats

Is your feature request related to a problem? Please describe.
Multivariate independence testing is a desirable problem to solve in many fields such as connectomics. Looking through the scipy.stats module, there are very few multivariate tests. In fact, there is 1, multiscale_graphcorr, which I had added a few months ago. Distance correlation (Dcorr) is another highly cited independence test that operates very well on multivariate data [1]. In fact, due to the fact the distance and kernel methods are equivalent [2], and that independence and two-sample testing is equivalent [3], we would be able to condense the implementations of Dcorr, Hilbert-Schmidt Independence Criterion (Hsic), Maximal Mean Discrepancy (MMD), and Energy into a single test. Our implementation also includes a number of optimizations, including a test statistic implementation that operates in n log n time instead of n^2 time when the data is univariate and the metric used is euclidean [4], and a chi-square approximation to the p-value that skips the computationally-intense permutation test [5]. Adding this provides another accurate multivariate test to SciPy, and makes the scipy.stats module as a whole far better.
Describe the solution you'd like
We would port our implementation to SciPy and maintain it. Similar to multiscale_graphcorr, a test statistic function would need to be added here, the permutation test can be used with little modification, and the centering of distance matrices needs to be modified here. If fast tests are something that is desired, this code needs to be ported here, and this code needs to be ported here.
Describe alternatives you've considered
N/A
Additional context (e.g. screenshots)

https://projecteuclid.org/journals/annals-of-statistics/volume-35/issue-6/Measuring-and-testing-dependence-by-correlation-of-distances/10.1214/009053607000000505.full
https://link.springer.com/article/10.1007/s10182-020-00378-1
https://arxiv.org/abs/1910.08883
https://www.sciencedirect.com/science/article/abs/pii/S0167947319300313
https://arxiv.org/abs/1912.12150

Hi, thanks for the proposal. I would suggest first that you send a message to the mailing list for such a big change to have more people looking at it.
A quick note about the porting and the licence. This might require a re licensing from the hyppo project as the original licence could be an issue here.

Note that this was already proposed in #10680. Your package, as well as mine (https://github.com/vnmabus/dcor), uses numba for compiling the inner loop of the fast implementation of distance covariance. I do not think that scipy maintainers want to add numba as a dependency. Also I think that adding just the naive implementation of distance covariance/correlation could prevent users to use the fast implementation if it can be used. At least, as a user I would assume that if scipy has it, it is probably the best existing implementation.
I however thing that maintaining two projects that implement the distance correlation in Python is probably a waste of resources, so I would be open to compare our packages in order to pick the parts were your implementation is better, and add you as a collaborator if you wish.

Note that this was already proposed in #10680.

Great, I'll look at this more closely

Your package, as well as mine (https://github.com/vnmabus/dcor), uses numba for compiling the inner loop of the fast implementation of distance covariance. I do not think that scipy maintainers want to add numba as a dependency. Also I think that adding just the naive implementation of distance covariance/correlation could prevent users to use the fast implementation if it can be used. At least, as a user I would assume that if scipy has it, it is probably the best existing implementation.

I agree. The implementation that I plan on adding will be using Cython as a backend instead of numba. The Dcorr implementation that I plan on adding (scipy.stats.multiscale_graphcorr) will likely use many of the same functions that I have already added with a few modifications (i.e. centering scheme for biased and unbiased). I currently have a student transferring our numba code into Cython.

I however think that maintaining two projects that implement the distance correlation in Python is probably a waste of resources, so I would be open to compare our packages in order to pick the parts where your implementation is better, and add you as a collaborator if you wish.

I also agree, and I would love to contribute if I can. I think that your expertise may be beneficial to my package also. My package (hyppo) aims to be a multivariate hypothesis testing package. Currently, we have Dcorr (biased/unbiased stat, fast approximation for the p-value and fast exact test statistic), along with a number of extensions of Dcorr. I think someone of your expertise would be beneficial to collaborate with, so if you have ideas for new tests, or tests that you think should be implemented, please let me know.

As I said, this would benefit from a proper email on the dev mailing list to have more opinion. From the other issue, I see a -0.5 vote from Ralf.
Also, please keep in mind what I said about licensing. We must be really careful here. We often have trouble with PR referencing Matlab, R and other (non) licensed code. Seems to be ok with dcor as MIT, but hyppo being Apache we would need some re-licensing.

As I said, this would benefit from a proper email on the dev mailing list to have more opinions. From the other issue, I see a -0.5 vote from Ralf.

Just sent a submission. I think it's worth noting that I don't plan on adding hyppo as a dependency to SciPy, which was something mentioned in the previous issue. I would just minimally modify existing helper functions I wrote in the past to compute the test statistic and p-value. So maybe that might change Ralf's vote?

Also, please keep in mind what I said about licensing. We must be really careful here. We often have trouble with PR referencing Matlab, R and other (non) licensed code. Seems to be ok with dcor as MIT, but hyppo being Apache we would need some re-licensing.

Is this a problem? I am the original author and maintainer of hyppo, so I have no problem re-licensing the code if necessary. Not exactly sure what that would entail, or if my permission is sufficient.

Is this a problem? I am the original author and maintainer of hyppo, so I have no problem re-licensing the code if necessary. Not exactly sure what that would entail, or if my permission is sufficient.

As long as you have the written consent of all copyright holders to re-licensed it's ok. Of course, for this to be valid, the code would need to be properly licensed in the first place (not derived or copied from a non compatible license.). We often see comments in code such as # found on StackOverflow, or this R code. Then your code would not really be licensed the way you think it is. Anyway, just something to keep in mind.

authors: sampan501 tupui vnmabus


text:
Return vector of indices in scipy.ndimage.minimum_filter1d

Hi,
This is a feature request. I have a use case where it would really helpful to have access to the indices of the input of the min filter that correspond with the minimum values the filter outputs. Some sort of argmin filter. Thank you!
https://github.com/scipy/scipy/blob/de80faf9d3480b9dbb9b888568b64499e0e70c19/scipy/ndimage/_filters.py#L976

I think that adding something like
*idxline++ = minpair->death - k;

to this conditional block
https://github.com/scipy/scipy/blob/e789092f7d5e5956bd26bceeadac3b259b1a9882/scipy/ndimage/src/ni_filters.c#L480
would save the right indices. idxline would be the line buffer for idx, the vector of indices I am looking for.


text:
RFC: Naming convention for generalised ufuncs in special

Hello all! This is a request for comment about a general issue in special. It is related to https://github.com/scipy/scipy/pull/20438, and is part of the overall refactoring in special that is currently happening. Here we go.
In special, there are several functions that return subarrays for each value in the input. The simplest example is the Legendre polynomials lpn(n, z), returning n + 1 values for each z so the output shape is (n + 1,) + z.shape. The reason that they do this is because the values are calculated using a recurrence relation, so it's a waste to recompute each value from scratch. Instead everything is stored and returned.
However, it's also true that sometimes you don't need all the values. Say I wanted the 1000000 value of lpn only, I'd rather compute that and not return the other 999999 values. Or I might want a scattered set of values, something like [3, 1000000, 749]. While lpn cannot do this at present, other functions like lpmv can and do.
So essentially we have two kinds of functions. One that uses the input as values to directly evaluate the functions with, and another that uses the input to define a size or shape and evaluates all values in that shape. And there are several functions in special for which we actually want both available. For instance, the latter kind has been requested for the Bessel functions a couple of times, see https://github.com/scipy/scipy/issues/19079.
In https://github.com/scipy/scipy/pull/20438, we sorted out both kinds for the spherical harmonics. The sph_harm(m, n, theta, phi) function that is currently in SciPy takes arrays m, n, theta, and phi and directly evaluates the spherical harmonics with them. It returns output of shape np.broadcast_shapes(m.shape, n.shape, theta.shape, phi.shape). There is also now a new function _sph_harm_all(m, n, theta, phi) that takes integers m and n as well as arrays theta and phi. It returns output of shape (2 * m + 1, n + 1) + np.broadcast_shapes(theta.shape, phi.shape), efficiently storing all recursive calculations.
The question here is what should the naming convention be for such functions? In the past, it seems someone thought of exactly this for the parabolic cylinder functions. That is why we have pbdv,  pbvv, and pbwa as well as pbdv_seq,  pbvv_seq, and pbwa_seq. I propose we come up with a suffix to append to special functions that return a whole set of values to save computation, rather than directly evaluating their inputs. And, in the future, we will use this suffix and make these functions part of the public API.
Why is _seq not enough? In my opinion, _seq makes sense really for those functions that are defined by 1D recurrence relations, like the parabolic cylinder functions. For something that is 2D or more, a name like sph_harm_seq doesn't read correctly to me. Possible options are _all, _seq, _table, _array, _multi, and _full. Of these, I think _all is the clearest, but I'd very much like to hear from others.
Another question is can we fold these two kinds of functions together? I certainly understand why that is wanted, but I don't think so, not sensibly. Of course, we can always add a keyword that switches behaviour, just like you could for any function. But both kinds of functions map to their own (g)ufuncs now, so it's quite artificial to do that. Also, I'd argue that we should keep the possible special array API extension in mind, see https://github.com/data-apis/array-api/issues/725. My understanding is that those array APIs that support JITs want to be able to predict the output shapes based on the input shapes, which this would violate.
Finally, I just want to say that it's really quite easy to add both kinds of generalised ufuncs now, especially with https://github.com/scipy/scipy/pull/20320. I'm certainly not saying we have to add both kinds for all relevant functions in special, but there are many that I think a lot of people would really appreciate.

I would like to understand why you don't like the shape change. This is quite common in SciPy codebase especially in linalg say lstsq or eig or stats. In fact this makes sure that you always have the same signature and you perform things depending on your kwargs.
The switch does not need to be explicit. We do detect the input to be scalar or array all the time in many public API functions and take different branches.
In fact I am in the opposite side that _seq or other variant names are quite unreadable and very matlab-y. Like I mentioned in the other issue _all does not ring to me anything about broadcasting.

I would like to understand why you don't like the shape change. This is quite common in SciPy codebase especially in linalg say lstsq or eig or stats. In fact this makes sure that you always have the same signature and you perform things depending on your kwargs.
The switch does not need to be explicit. We do detect the input to be scalar or array all the time in many public API functions and take different branches.
In fact I am in the opposite side that _seq or other variant names are quite unreadable and very matlab-y. Like I mentioned in the other issue _all does not ring to me anything about broadcasting.

It's a good question, why not just use keyword arguments to switch behaviour? I'll address this a bit more thoroughly.
Let's stick with sph_harm(m, n, theta, phi) and sph_harm_all(m, n, theta, phi) as examples. I'd just point out that sph_harm(1, 2, theta, phi) is totally valid, returning an array of shape np.broadcast_shapes(theta.shape, phi.shape) that contains the m = 1 and n = 2 spherical harmonic. And so is sph_harm_all(1, 2, theta, phi), returning an array of shape (3, 2) + np.broadcast_shapes(theta.shape, phi.shape) that contains all spherical harmonics for -1 <= m <= 1 and 0 <= n <= 2. So we cannot use whether something is a scalar or not to choose the implementation, the only alternative to having separate functions is a keyword argument like sph_harm(m, n, theta, phi, all=True).
Why have separate functions? Fundamentally, these are separate functions. At the C / C++ level, they are just different. Here is sph_harm and here is sph_harm_all. They then each become their own ufunc or gufunc. At the Python level, sph_harm is a ufunc and can be used directly, whereas sph_harm_all is a Python function that calls the _sph_harm_all gufunc just because it needs to allocate output arrays with the correct shape, see https://github.com/scipy/scipy/blob/main/scipy/special/_basic.py#L3439. Mixing them together in one function would only work at the Python level, not at the ufunc level and not at the C / C++ level. I just think we should be true to their nature and keep them separate each level, rather than have separate C / C++, separate ufuncs, but together Python.
Also, I think it's good to expose NumPy ufuncs directly to the user where possible. If we used a keyword, we'd wrap the ufunc behind a Python function even though we don't need to.
Finally, on the JIT shape stuff. It was just my understanding that the array libraries that support JITs want to have predictable shape outputs. And switching shapes based on a keyword doesn't encourage that. But this is just one reason of a few.
Those are my thoughts!

I think this needs some actual examples, I am still having hard time understanding the underlying difference. If you look at the kernel of both functions. They are cephes.poch for some loop I'm not fully following. Hence I don't see how they are different yet.
However, if you look at the signatures, sph_harm does not accept arrays in theta or pi. So that's fundamentally a switching point. There is no whether scalar or not difference. It does not even accept it. Hence I think we can establish that there is no way you can push arrays to the sph_harm.
Now what the output looks like is something else. I still don't see why shape changes but even if they do, like I mentioned, we have all SciPy doing this all the time. There is no problem with changing the output shape, JIT is not something we should care at this point because we have not even managed to get to a public API.
Let's say I have theta phi scalar and I used sph_harm(m, n, theta, phi, all=True) then what will happen the loops will spin once and we get spurious array dims?

Mixing them together in one function would only work at the Python level, not at the ufunc level and not at the C / C++ level.

That's OK, the difference is already in with your _all suffix now. So they are different functions. And we can still use a unified Python API.
But I must press on this point that _all really does not mean anything to the user. All harmonics, all elements, all inputs what does that actually mean?

I don't have a strong opinion regarding having separate functions or a single function with a flag, to separate a ufunc whose scalar kernel computes f(m, n, ...) for specific values of m and n, and the gufunc backed version whose scalar kernel can compute the full table of values of f(m, n, ...) for 0 <= m <= M, 0 <= n <= N. The argument about data dependent output shapes in the array API seems compelling though, especially since we're trying to establish a convention to use for all such functions.
I think the name sph_harm_all is imperfect since, like @ilayn said, it's not clear from the name what the function gives all of. I struggle to think of other choices though. The only other good candidate I've heard is sph_harm_table, which @izaid had also suggested. I could accept either of these provided the convention is consistently followed and well documented.
If we go with separate functions, I suppose something like pbvv_seq could become an alias for pbvv_all or whatever, and perhaps would eventually be deprecated.
First, to answer some questions:

I think this needs some actual examples, I am still having hard time understanding the underlying difference. If you look at the kernel of both functions. They are cephes.poch for some loop I'm not fully following. Hence I don't see how they are different yet.

They are indeed very similar. The difference between the kernel for sph_harm and that for what is now sph_harm_all, is that the former stores no intermediate results when calculating based on a recurrence relation, maintaining only the final result for a given m and n. The kernel for sph_harm_all stores all intermediate results in an array so that the entire table of results can be returned. The former has the benefit of being memory efficient if you really only need results for a few particular values for m and n, at the expense of requiring duplicated computation if you need results for a large range of m and n.

However, if you look at the signatures, sph_harm does not accept arrays in theta or pi. So that's fundamentally a switching point. There is no whether scalar or not difference. It does not even accept it. Hence I think we can establish that there is no way you can push arrays to the sph_harm.

I think the signatures you're looking at are those for the scalar kernels. These don't accept arrays for theta or phi for either sph_harm or sph_harm_all. It's the functions at the Python level that do. The difference in signatures at the Python level is that sph_harm(m, n, theta, phi) can take arrays for all arguments, and _sph_harm_all(m, n, theta, phi) can take arrays for theta and phi, but only ints for m and n.

@ilayn, can you share more on the benefits of offering single functions with flags to switch behavior in this case, compared to having pairs of separate functions? Since special is already a hodge-podge of hundreds of specific functions, I think the marginal cost of adding new functions is less than for a module with a smaller set of more general tools.

authors: ilayn izaid steppi


text:
Numpy 1.10 breaks scipy.io.netcdf_file()

Numpy 1.10.* seems to have broken something in scipy.io.netcdf_file. The following code fails silently and leads to extensive memory usage (several GBs). It works as expected with numpy <= 1.9.
from scipy.io import netcdf_file

nc = netcdf_file("data.nc")

It does not happen with all netcdf files, though. Here you can download a netcdf3 file that reproduces the error (be prepared to hit Ctrl+C before your memory is exhausted):
https://github.com/weatherfrog/notebooks/blob/master/data.nc?raw=true
Tested with Python 2.7/3.4 on Linux and OS X (64 Bit). scipy.__version__ == '0.16.0'.

I wonder if this is related to #6467? The report there is of slowdowns, not memory consumption though. A fix for that has been merged, is it possible that you can test with the current 1.10.x branch?

I will test with current 1.10.x branch and report back...

Tested again with numpy 1.10.4 (scipy 0.17.0). Bug is gone. So I'll close this issue.

thanks @weatherfrog

authors: charris rgommers weatherfrog


text:
bessel functions

I would assume these two print statements to be equivalent:
import numpy as np
from scipy.special import j1
x = .1
print(j1(x))
print((np.sin(x)/x**2 - np.cos(x)/x))

Output is (they're not):
0.440050585745
0.30116867894

Bessel function j1
There is no documentation as to what the Bessel function should be (i.e. if some constant prefactors are added). Either documentation needs to be improved or function fixed (if this is indeed an error). Any suggestions?
thanks!
julien

np.sin(x)/x**2 - np.cos(x)/x is the first spherical Bessel function, which is related to the Bessel functions of the first kind of order 3/2. See spherical_jn.

Thanks!
Perhaps it's a good chance to add a small line of documentation? (See Also, etc)

Documentation clarifications are always welcome, do you have time to make a
PR?

Added (see PR above). The following text would have made it easier for me, but there may be better wording. thanks!


text:
MAINT,TST: update and run scipy/special/utils/convert.py

Run the boost test_data importer.
Added new tests for special functions ivp, jvp, kvp, yvp

This should be reproducible with
git clone --depth=1 https://github.com/boostorg/math.git boostmath

rm -rf scipy/special/tests/data/boost/*
python scipy/special/utils/convert.py

# List all files then check if they appear in scipy/scipy/special/tests/test_data.py
find scipy/special/tests/data/boost -type f | sed 's!scipy/special/tests/data/boost/!        # TODO    !' | sort -r

# Verify all the new files are used in test_data.py
git diff --stat HEAD~ | grep "Bin 0"
git diff upstream/master -- scipy/special/tests/test_data.py

This results in a BIG diff because many tabs were converted to spaces,
git diff -a HEAD~ | wc
  86048  354276 13779286
git diff -a -w HEAD~ | wc
  29414  123487 4599674

To keep the diff smaller I revert any whitespace-only change via this hack
git diff -a -w HEAD~ > test.patch
git reset --hard HEAD~
git apply test.patch
git add scipy/special/tests/data/boost
git commit -m 'MAINT,TST: Run scipy/special/utils/convert.py'

I could probably reduces the number of lines change by another 30% if I reverted changes of 1 to 1.0 and 0 to 0.0 in a number of "updated" files but I didn't see any easy way to do this (maybe running sed over a hand curated list of files?)
Ideally this wouldn't be squashed, so inspecting the diff of convert.py and test_data.py would be easier. the tests would be broken by the first commit but fixed when the second commit is submitted.
I'm sorry the diff is such a mess, I'd suggest looking at the diff of the first commit only. Also I don't understand many of the pieces I touched (e.g. how is jn different from jv) so a semantic review of test_data.py would appreciated.

The lint test fails because I'm copied the style of the other tests in the file.
data(ivp, 'bessel_i_prime_int_data_ipp-bessel_i_prime_int_data', (0,1j), 2, rtol=1.2e-13, atol=1e-300),

(0,1j) should maybe have a space and should maybe line wrap but it is in the style of the file.

In theory you should be able to make exceptions when justified, e.g., # noqa: E501 for the line with the line length problem, and do something similar for the other. At least you can for flake8 as a tool, haven't looked into how our diff lint works

I think the whitespace and 0 -> 0.0 changes here are fine, given that they're in data files and generated code. The first commit in this PR is tidy enough. The thing left to do is verify that no tests got dropped by the regex change - should be a matter of comparing test count in master vs in this PR.

In theory you should be able to make exceptions when justified, e.g., # noqa: E501 for the line with the line length problem, and do something similar for the other. At least you can for flake8 as a tool, haven't looked into how our diff lint works

I don't think there's a need for that, if one decides that ignoring the flake8 output is fine (e.g. because style is consistent within the file), then we can just commit it. Subsequent PRs won't fail on it, that's the nice part of the new diffing tool.

I think the whitespace and 0 -> 0.0 changes here are fine, given that they're in data files and generated code. The first commit in this PR is tidy enough. The thing left to do is verify that no tests got dropped by the regex change - should be a matter of comparing test count in master vs in this PR.

You can also look at https://github.com/scipy/scipy/pull/12357/files#diff-a49868d52c69e2d53c607534a4572a8e and see that I only add lines :)

I wouldn't add tests for sinc. Our sinc is just NumPy's sinc, and NumPy's sinc is not good.
Now if you want to reimplement sinc along the lines of https://github.com/numpy/numpy/pull/12924 then that would be great.

... and to go into more detail: you're matching Boost's definition by doing
def sinc_x(x):
    return sinc(x / np.pi)

but numerically speaking, sin(pi * x) / (pi * x) and sin(x) / x have very different properties. This is because sin(pi * x) and sin(x) are very different-the former can easily be computed to ~1 ULP of accuracy (see e.g. our implementation in special), while that is not true for the latter (with any practical implementation). So this doesn't end up being an apples-to-apples comparison.

@person142 These are really solid comments. I pulled out the sinc changes and we can discuss in a followup PR.

@rgommers I think this can go in; one test failure is lint, the other is timeout.
After this goes in I'll have a 2nd pass to use some of the new test data files (sinc, ive, iv`)

Thanks @rgommers  :)

source: gharchive/pull-request

authors: larsoner person142 rgommers sethtroisi


text:
CI: pin to rtools40

There is a new rtools release on choco, version 4.3. This has caused a large proportion of Windows CI to stop working as the binaries are in a different place to the previous version available on choco. This meant that meson couldn't find pkg-config (different install location), so the whole build ground to a halt.
This PR pins the rtools version to 4.0. Decided to stick with that because 4.3 may bundle different gcc versions, etc.

I suspect that the upgrade will solely involve changing instances of
echo "c:\rtools40\ucrt64\bin;" >> $env:GITHUB_PATH

to

Merging because changes straightforward and it keeps CI going.

Would this affect testing with fortran on windows?

Not quite sure what the question is asking. The majority of the windows CI infra uses rtools to supply pkg-config, gcc, gfortran. The jobs started failing because the choco update meant rtools was in a different location, meaning that the build tools weren't found. This PR pins the version of rtools, so the tools that were working can be found again.
I am not sure if the new rtools package uses different versions of gcc/gfortran.

I was asking because we have started getting CI failures involving Fortran when running full tests on Windows 32 bit, see https://github.com/numpy/numpy/issues/23675. I was wondering if there could be a connection.

I think there will be a connection. The azure job installs rtools. Rtools43 is now the default on choco and that will be installed in a different location. I suspect you need to pin the version.

@andyfaff Just for your information, it turns out the rtools dropped support for 32 bit builds in version 4.2.

authors: andyfaff charris


text:
Refactoring: benchmarks/linalg.py

Hi all.
I did some refactoring to this file. Couple of instance attributes were defined outside init, so I moved them inside the init method.
Also: couple of PEP8 amendments.
Thank you.

Sorry, these refactorings appear wrong. Please study first how the benchmarking system works, before attempting to refactor.


text:
BENCH: Add dual annealing to scipy benchmark (see #9415)

@rgommers, thanks for the merging. This PR is for adding optimize.dual_annealing into the scipy optimize benchmark. I ran:
python runtests.py --bench -s optimize

with SCIPY_XSLOW=5 env var exported to test if what I added in benchmarks/benchmarks/optimize.py was ok.
This was to address #9415.

Thanks @sgubianpm, that works for me.

Thanks @rgommers for merging this benchmark addition. What about running the test with GOBench for now including shgo as an intermediate evaluation overall performance (something outside the repos for now) and enhance the scipy benchmark for optimization later one?

Thanks @sgubianpm , that makes sense for now. We'll want some viz capability at some point though. Perhaps there was something somewhere that I missed, because the original source of our benchmarks had some plots as well IIRC (@andyfaff do you remember?).
@Stefan-Endres said in https://github.com/scipy/scipy/pull/8098#issuecomment-436729990 that he's interested in adding performance profile plots as well.

When I added in the Global Optimisation benchmarks there was no plotting. Any plots would've been done by other code. The output could certainly be improved, both in terms of writing to a properly designed file (I think at the moment one has to parse stdout), and code that could plot the file.

both in terms of writing to a properly designed file (I think at the moment one has to parse stdout),

asv produces sane json files.

I ran the benchmark with only 5 runs to check and asv produced a file called global-bench-results.json which seems to be usable (I changed the extension for attaching it).
global-bench-results.txt

authors: andyfaff rgommers sgubianpm


text:
DOC: Include Numba in list of non-C ndimage examples

Numba is shown as the first example after this list, but was not included in this list. Hence we add it to the list.

Simple doc improvement -- in it goes, thanks @jakirkham

authors: jakirkham tylerjereddy


text:
500 on empty file uid|label|reaper upload

Fixes #993
Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@9301a8d). Click here to learn what that means.
The diff coverage is 80%.

@@            Coverage Diff            @@
##             master     #996   +/-   ##
=========================================
  Coverage          ?   90.23%
=========================================
  Files             ?       49
  Lines             ?     6690
  Branches          ?        0
=========================================
  Hits              ?     6037
  Misses            ?      653
  Partials          ?        0

I think we would rather 400 (client messed up) than 500 (we messed up).

We'd want to change the existing 500s as well. Inside placer.py there is a check state, and we have requireTarget and requireMetadata, would we also want to write one for requireFile? Not 100% sure that's the best way forward but would probably be cleaner.

Our of band, Harsha: we can't access file_fields that way because it breaks the ability of the Placer interface to incrementally process a metadata stream. Megan and I worked on some alternatives, check with us.

@hkethi002 and I went through moving the check back into the process_upload path.

authors: codecov-io hkethi002 kofalt nagem

repo: scitran/core

license: MIT


text:
Add test-lib-openshift.sh for testing in OpenShift

Add test-lib-openshift.sh with set of functions for testing docker images in OpenShift using 'oc'

Thanks!

authors: hhorak praiskup

repo: sclorg/container-common-scripts

license_source: bigquery


text:
Backend with Realtime Database

Drones use their radio link to communicate with their ground stations and controllers. Those radio links can communicate for a limited range. When a drone fly automatically through way points it may go away from radio link. Then ground station unable to monitor that drone any more. We want a system to monitor automatically flying drones using long range communication network “LORA” . Its mandatory to uniquely identifying each drone, for that we want to bind unique key to each drone. The final product should be a separate unit which can externally mound on any kind of drone.Currently we have a system consisting with seperate devices to mount on drones and web system to monitor their locations. That device send data like location, timestamp, and nodeid with AES encryption. This is an one way communication system which sends messages only from drones to the DroLa web System. For further developments we are seeking to improve the system as follows.We want a new platform to send messages in more general way instead of current fixed message format. Then we can introduce this system to other tracking solutions also. We want to design a bidirectional communication protocol and implement it including message passers for both Lora Node side and web server side.Current system is running on web sockets, drone locations are collected by Lora receivers and send them to the web server through a web socket server. we want to move to a realtime database instead of using web-sockets.
Required Knowledge: C++, Node Js, Python and ReactJs

@charithccmc would love to contribute to this issue, but I think #30 already does this in entirety. Would that PR be reviewed and merged? In that case, it'd be futile to redo all that work :)

authors: VibhorCodecianGupta charithccmc

repo: scorelab/drola


text:
Parse other way: date to "2 weeks ago"

Your package can parse things like this 2 weeks ago.
But could you do the same thing the other way around?
Transform a specific time-delta (now() - paste_date) to such a user-friendly string format like 2 weeks ago?

Going the other way around is out of the scope of dateparser. You might want to use an alternative for that, such as arrow, maya or pendulum.

authors: Gallaecio noMICROSOFTbuhtz

repo: scrapinghub/dateparser


text:
Any guarantees on retries?

Good day. We have a project that makes extensive use of the API and at certain times "429 Too many requests for user" occur and are raised with no indication of any retries, exponential or otherwise, taking place. In fact it seems no HTTP errors are retried for resource iterators whatsoever:
https://github.com/scrapinghub/python-scrapinghub/blob/master/scrapinghub/hubstorage/resourcetype.py#L134-L143
Although I see other places do handle such:
https://github.com/scrapinghub/python-scrapinghub/blob/master/scrapinghub/hubstorage/client.py#L20-L37
Is this by design?
UPD:
So at this point: https://github.com/scrapinghub/python-scrapinghub/blob/master/scrapinghub/hubstorage/client.py#L113-L116
the iterator request executed as is without retrier wrapping as it deemed not idempotent. At a later point here: https://github.com/scrapinghub/python-scrapinghub/blob/master/scrapinghub/hubstorage/resourcetype.py#L134-L143 it is not handled at all (scrapinghub.client.exceptions.ScrapinghubAPIError('Too many requests for user') in our case).

Hey, thanks for the report! I believe your observations are correct, and the requests sent by the resource iterators are considered to be not idempotent, thus not retried on any of the listed exceptions. It's not by design, this is something that would be nice to fix.
Worth to add that the issue affects only iterating over/listing a collection elements, and doesn't affect items/requests/logs. (While we call the origin iter_values method for collections, it's origin list method for items/requests/logs which uses apiget method underneath with setting is_idempotent=True by default)

Hi @vshlapakov, thank you for the reply.

doesn't affect items/requests/logs

I don't believe that's the case. Items object MRO is:
<class 'scrapinghub.client.items.Items'>
<class 'scrapinghub.client.proxy._DownloadableProxyMixin'>
<class 'scrapinghub.client.proxy._ItemsResourceProxy'>
<class 'scrapinghub.client.proxy._Proxy'>
<class 'object'>

It calls _DownloadableProxyMixin.iter() (-> _origin.iter_values() -> _origin.iter_json/msgpack()) and not _ItemsResourceProxy.iter(). Same for logs and requests.

My bad, I had an old version of the library installed locally, there was a different MRO before.
A PR is highly welcome if you have time.

This should be covered now, thanks again for the thorough investigation and the clean patch :raised_hands:

authors: hermit-crab vshlapakov

repo: scrapinghub/python-scrapinghub


text:
ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Hi,
I executed many tests to scrape thousand of urls, and confirmed the error Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory, the error occurred when I crawled about 2500 urls from https://majestic.com/reports/majestic-million and Scrapy script crashed. I tested on:

scrapy-playwright 0.0.3
or current master
or multiple-contexts branch from #13 with creating new context per domain thanks to example in context example

So, I guess this is a bug.
The detail of error:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa18150 node::Abort() [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 2: 0xa1855c node::OnFatalError(char const*, char const*) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 3: 0xb9715e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 4: 0xb974d9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 5: 0xd54755  [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 6: 0xd54de6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 7: 0xd616a5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 8: 0xd62555 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 9: 0xd6500c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
10: 0xd32eac v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
11: 0xd3366d v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const> const&, v8::internal::AllocationType) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
12: 0xbaf47f v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
13: 0xaf5340 node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
14: 0x9f21e6  [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
15: 0x1390d8d  [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]

Note:

One contributor of playwright said reusing the browser context may lead the error Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory and he recommends to use new context for new domain, more detail in playwright

Thanks for the report. I just merged and released support for multiple contexts (v0.0.4), but I don't think there's much more we can do on this side of things.
I'm closing this, since according to the linked upstream issue, this "can be fixed by creating a new context each time". Feel free to reopen if you think it's necessary.

Hi @elacuesta
The error still occurred with scrapy-playwright 0.0.4. The Scrapy script crawled about 2500 urls in 10k domains from majestic and crashed.
My main code:
domain = self.get_domain(url=url)

context_name = domain.replace('.', '_')
yield scrapy.Request(
    url=url,
    meta={
        "playwright": True,
        #"playwright_include_page": True,
        "playwright_page_coroutines": {
            "screenshot": PageCoroutine("screenshot", domain + ".png"),
        },
        # Create new content
        "playwright_context_name": context_name,
    },
)

My env:
Python 3.8.10
Scrapy 2.5.0
playwright 1.12.1
scrapy-playwright 0.0.04

The error:
2021-07-17 14:47:48 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.costco.com/>: HTTP status code is not handled or not allowed
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0xa18150 node::Abort() [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 2: 0xa1855c node::OnFatalError(char const*, char const*) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 3: 0xb9715e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 4: 0xb974d9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 5: 0xd54755  [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 6: 0xd650a8 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 7: 0xd2bd9d v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::RootIndex, int, v8::internal::Object, v8::internal::AllocationType) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 8: 0xd2be90 v8::internal::Handle<v8::internal::FixedArray> v8::internal::Factory::NewFixedArrayWithMap<v8::internal::FixedArray>(v8::internal::RootIndex, int, v8::internal::AllocationType) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
 9: 0xf5abd0 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::AllocationType) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
10: 0xf5ac81 v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Isolate*, v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
11: 0xf5b2cb v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::EnsureGrowable(v8::internal::Isolate*, v8::internal::Handle<v8::internal::OrderedHashMap>) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
12: 0x1051b38 v8::internal::Runtime_MapGrow(int, unsigned long*, v8::internal::Isolate*) [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
13: 0x140a8f9  [/home/ubuntu/.local/lib/python3.8/site-packages/playwright/driver/node]
Aborted (core dumped)
2021-07-17 14:48:34 [scrapy.extensions.logstats] INFO: Crawled 2533 pages (at 15 pages/min), scraped 2362 items (at 12 items/min)

Temporary fix: I added await page.context.close() to close current context in line 166 in handler.py. It will fix the error Allocation failed - JavaScript heap out of memory and the Scrapy script crawled all 10k domains. However, when I added the new code, the new error was:
2021-07-17 15:04:59 [scrapy.core.scraper] ERROR: Error downloading <GET http://usatoday.com>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
    return (yield download_func(request=request, spider=spider))
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 824, in adapt
    extracted = result.result()
  File "/home/ubuntu/python/scrapy-playwright/scrapy_playwright/handler.py", line 138, in _download_request
    result = await self._download_request_with_page(request, page)
  File "/home/ubuntu/python/scrapy-playwright/scrapy_playwright/handler.py", line 149, in _download_request_with_page
    response = await page.goto(request.url)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/playwright/async_api/_generated.py", line 6006, in goto
    await self._async(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/playwright/_impl/_page.py", line 429, in goto
    return await self._main_frame.goto(**locals_to_params(locals()))
  File "/home/ubuntu/.local/lib/python3.8/site-packages/playwright/_impl/_frame.py", line 117, in goto
    await self._channel.send("goto", locals_to_params(locals()))
  File "/home/ubuntu/.local/lib/python3.8/site-packages/playwright/_impl/_connection.py", line 36, in send
    return await self.inner_send(method, params, False)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/playwright/_impl/_connection.py", line 54, in inner_send
    result = next(iter(done)).result()
playwright._impl._api_types.Error: Navigation failed because page was closed!

authors: elacuesta phongtnit

repo: scrapy-plugins/scrapy-playwright


text:
more efficient handling of duplicate args

This PR adds a new feature: user can specify which Splash arguments don't change often, and scrapy-splash will handle them more efficiently:

[x] they won't be stored in disk queue multiple times;
[ ] they won't be sent to Splash multiple times (requires https://github.com/scrapinghub/splash/pull/442)

Current coverage is 87.42%

Merging #63 into master will decrease coverage by -1.86% as of 8413f6c

@@            master     #63   diff @@
======================================
  Files            9       9
  Stmts          420     469    +49
  Branches        79      91    +12
  Methods          0       0
======================================
+ Hit            375     410    +35
- Partial         23      27     +4
- Missed          22      32    +10

Review entire Coverage Diff as of 8413f6c

Powered by Codecov. Updated on successful CI builds.

authors: codecov-io kmike

repo: scrapy-plugins/scrapy-splash


text:
Vehicle infromations should be stored in a ledger

Storing the characteristics of the used vehicle on the blockchain makes it easier to reproduce the calculated PCF.
A chain code function should be able to write the characteristic and another function be able to read the data based on the ID.
The id should be used as key.
Example payload:
{
    "id": "0cafe6b0-8a14-11ee-b9d1-0242ac120002",
    "name": "My lovely DB5",
    "emissionPerKm": 40
}

Add reference values for example vehicles
{
    "id": "d7180626-bf12-4bd7-96b9-fd5ff4b51a3d",
    "name": "My Citybike",
    "emissionPerKm": 0
}

{
    "id": "3307a49f-d1d1-4ae8-80d9-b467936a939e",
    "name": "My MTB",
    "emissionPerKm": 0
}

{
    "id": "0cafe6b0-8a14-11ee-b9d1-0242ac120002",
    "name": "My lovely DB5",
    "emissionPerKm": 250
}

{
    "id": "b6eb42e9-2f55-4a99-8e2d-2812b337431f",
    "name": "My allday Golf",
    "emissionPerKm": 115
}

{
    "id": "e09c2259-994e-40f1-8b1d-c66a55ca0bb7",
    "name": "Always late train",
    "emissionPerKm": 93
}

Refs to id
[b6eb42e9-2f55-4a99-8e2d-2812b337431f] https://www.volkswagen.de/idhub/content/dam/onehub_pkw/importers/de/besitzer-und-nutzer/wissenswertes/wltp/WLTP_Emissionswerte_Emissionsnormen_DE_10-19.pdf (27.12.2023)
[e09c2259-994e-40f1-8b1d-c66a55ca0bb7] https://www.umweltbundesamt.de/bild/vergleich-der-durchschnittlichen-emissionen-0 (27.12.2023)

repo: scray/logbook-app


text:
move() does not return undefined

not sure how this ended up in the types, but i am pretty sure it is not correct :D

No idea how it ended up there either, I exported a lot of the code when working on it, so there might actually have been cases where it would return undefined, someone would have to verify that xD

just looking at the docs i dont see why this would be the one intent that returns something else other than a return code (okay, other than the spawn stuff). so i would assume this is just some weird side effect of the export?

authors: lmik thmsndk

repo: screepers/typed-screeps-arena


text:
🛑 Animate Me is down

In b62ab4b, Animate Me (https://animate-me.atanas.info) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Animate Me is back up in d21b794.

repo: scriptex/uptime


text:
[WIP] memory-opt: add check for input, output and return bytes for precompile

TODO
Run testool again.
Description
Add check for input, output and return bytes for precompile (as bus-mapping).
Issue Link
Previous testool result
https://circuit-release.s3.us-west-2.amazonaws.com/testool/default.1687933681.e55d0b0.html

\b run testool

repo: scroll-tech/zkevm-circuits


text:
Creating opt-in opportunity to generate PWA

It sounds very naturally to add some PWA features to the generated project:

App shell
App manifest + default icons

To give the full flexibility to extend a service worker if needed, I'd offer to use Workbox library in its "node module" form.

@webmaxru We would happy to work on this.

Whats the progress on this?

authors: Jiraci SanderElias webmaxru

repo: scullyio/scully


text:
Trying to run DCA on microbiome dataset

I'm attempting to run your DCA method on microbiome data containing 11K rows (samples) x 1200 columns (microbes).
In trying to read in the data using io.py read_dataset() I get the following error which can be traced back anndata.py
The error message is:
~/anaconda3/envs/common_microbe/lib/python3.7/site-packages/anndata/_core/anndata.py in _check_dimensions(self, key)
   1834         if "var" in key and len(self._var) != self._n_vars:
   1835             raise ValueError(
-> 1836                 "Variables annot. `var` must have number of columns of `X`"
   1837                 f" ({self._n_vars}), but has {self._var.shape[0]} rows."
   1838             )

ValueError: Variables annot. `var` must have number of columns of `X` (1232), but has 7391 rows.

Any help with how to apply DCA to a dataframe like this would be helpful. Thank you!

Hi! Sorry for not responding back then. read_dataset isn’t an exported function of anndata, or is it?
If you happen to have the file around still, please post the code that led to the error and upload/link to the file so we can reproduce.
I’m going to close this because it hasn’t come up anymore. I’ll reopen if it’s still an issue.

repo: scverse/anndata


text:
Does squidpy.gr.spatial_neighbors evaluate 3D coordinates?

Hello,
I have 3D coordinates in obsm['spatial']. Does squidpy.gr.spatial_neighbors support this? The function seems to be running on my dataset but I am not sure about the output.
Thanks!
...

Hi @schienstockd ,  sorry for the late reply. I quickly checked the code, as far as I could tell, yes, it should work on 3D (or even ND), see here or here, the coords should be directly taken from adata.obsm.

repo: scverse/squidpy


text:
improvement(graphs): Mark dependency changes

In performance tests it's important to note changes of dependencies, like kernel, drivers, stress tools.
This commit changes the way point is displayed (gets white background) and adds changed packages details to the tooltip.
closes: https://github.com/scylladb/argus/issues/490

@soyacz
can we add the the full scylla release as well ?

@soyacz
can we add the the full scylla release as well ?

We can. See also that kernel changes almost every week, I wonder if we shouldn't mark only big changes like major version updates only (I mean still change will be shown, but point won't be marked).

@soyacz
can we add the the full scylla release as well ?

We can. See also that kernel changes almost every week, I wonder if we shouldn't mark only big changes like major version updates only (I mean still change will be shown, but point won't be marked).

before I want to notice the change, we want to see them.
then we can indicate there was a change on the inside the popup
and last we can decide what to indicate on the graph
one more thing, not for this PR, we should be able to change the x axis to something else like driver version, or just plain data of the run, cause we don't have version change or date in the drivers CI (at least not yet, we didn't thought of it yet)

before I want to notice the change, we want to see them. then we can indicate there was a change on the inside the popup and last we can decide what to indicate on the graph

Summarizing:
Ok, any change in dependencies (except SUT) will be marked with different point color.
All the changes, including SUT will be shown in tooltip.
If this won't fit us well in the wild, we can adjust.

one more thing, not for this PR, we should be able to change the x axis to something else like driver version, or just plain data of the run, cause we don't have version change or date in the drivers CI (at least not yet, we didn't thought of it yet)

Generally, SUT is detected automatically based on which dependency changes the most often. If driver version will be changing the most often then it becomes the SUT. This feature is used e.g. in SM tests. We can adjust it in followup issues if we need something else.

before I want to notice the change, we want to see them. then we can indicate there was a change on the inside the popup and last we can decide what to indicate on the graph

Summarizing: Ok, any change in dependencies (except SUT) will be marked with different point color. All the changes, including SUT will be shown in tooltip. If this won't fit us well in the wild, we can adjust.

one more thing, not for this PR, we should be able to change the x axis to something else like driver version, or just plain data of the run, cause we don't have version change or date in the drivers CI (at least not yet, we didn't thought of it yet)

Generally, SUT is detected automatically based on which dependency changes the most often. If driver version/build date will be changing the most often then it becomes the SUT. This feature is used e.g. in SM tests. We can adjust it in followup issues if we need something else.

not sure automatic would fit the best, consider a case we run CI with multiple release of scylla, let say OSS and enterprise ?

run CI with multiple release of scylla, let say OSS and enterprise

package name for both is scylla-server so it works (just both anyway are separated for different releases on the graph)

run CI with multiple release of scylla, let say OSS and enterprise

package name for both is scylla-server so it works (just both anyway are separated for different releases on the graph)

in driver CI, we care to see the driver version change / commit, not necessarily scylla, in my example both driver, and scylla might be changing across time

run CI with multiple release of scylla, let say OSS and enterprise

package name for both is scylla-server so it works (just both anyway are separated for different releases on the graph)

in driver CI, we care to see the driver version change / commit, not necessarily scylla, in my example both driver, and scylla might be changing across time

I doubt in testing performance while changing both dependencies at the same pace, anyway, we can adjust to whatever we need later.

@fruch I adjusted the code, please see PR description to see the changes.

@fruch I adjusted things we agreed yesterday: SUT version only, error threshold instead 'limit'. Also I did various fixes for tooltip: enlarge font, make it be above mouse cursor (so it does not cover text).

@k0machi this one is unrelated to highlights (discussion) view widget and is relatively small. I propose to review it first so it can be merged and used.

authors: fruch soyacz

repo: scylladb/argus


text:
messages from kafka topic not inserted in scylla db

after loading the connector using  Kafka-connect-scylladb.json,  no messages are inserted in Scylla DB.

Followed the quickstart
https://github.com/scylladb/kafka-connect-scylladb/blob/master/documentation/QUICKSTART.md
but scylla db is not started using docker image, since  Scylla DB is already present in the  machine with below version:
ScyllaDB: cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4
checked plugin loader using:
 curl -sS localhost:8083/connector-plugins | jq .[].class | grep ScyllaDbSinkConnector
and got the ouput as mentioned in quick start:
"io.connect.scylladb.ScyllaDbSinkConnector"
below Kafka-connect-scylladb.json load connector config is used
{
     "name" : "scylladb-sink-connector",
     "config" : {
       "connector.class" : "io.connect.scylladb.ScyllaDbSinkConnector",
       "tasks.max" : "1",
       "topics" : "topic1,topic2,mqtt-malaria",
       "scylladb.contact.points" : "localhost:9042",
       "scylladb.keyspace" : "mqtest"
       }
}

tried the same message to check:
kafka-avro-console-producer
--broker-list localhost:9092
--topic topic1
--property parse.key=true
--property key.schema='{"type":"record","name":"key_schema","fields":[{"name":"id","type":"int"}]}'
--property "key.separator=$"
--property value.schema='{"type":"record","name":"value_schema","fields":[{"name":"id","type":"int"},
{"name":"firstName","type":"string"},{"name":"lastName","type":"string"}]}'

even after creating a table with keyspace maually, the messages are not inserted in DB.
Is that any config or setup is missed in the quick start? any config is need to add in Kafka? How to confirm Scylla is connected to kafka

Could you also post messages on the topic?
Command to do it:
kafka-avro-console-consumer --broker-list localhost:9092 --topic topic1 --from-beginning --property print.key=true

(note that on my system --broker-list is called --bootstrap-server)
And also print logs of connector?
Command to do it:
confluent local services connect log

(command as of Confluent 6.0.0, if it doesn't work maybe look at /tmp/confluent.XXXXXX/connect/logs/connect.log, where XXXXXX are some numbers)

Could you also post messages on the topic?
Command to do it:
kafka-avro-console-consumer --broker-list localhost:9092 --topic topic1 --from-beginning --property print.key=true

(note that on my system --broker-list is called --bootstrap-server)
And also print logs of connector?
Command to do it:
confluent local services connect log

(command as of Confluent 6.0.0, if it doesn't work maybe look at /tmp/confluent.XXXXXX/connect/logs/connect.log, where XXXXXX are some numbers)

posted messages are hitting in consumer, Scylla DB  connection issue exists.
{
     "name" : "scylladb-sink-connector",
     "config" : {
       "connector.class" : "io.connect.scylladb.ScyllaDbSinkConnector",
       "tasks.max" : "1",
       "topics" : "topic1,topic2,mqtt-malaria",
       "scylladb.contact.points" : "localhost:9042",
       "scylladb.keyspace" : "mqtest"
       }
}

what value does the  scylladb.contact.points  refers ?
checked confluent local services connect log
 (io.connect.scylladb.ScyllaDbSinkConnectorConfig:361)
[2021-03-18 04:41:57,520] INFO Trying to configure address translator for private network address and port. (io.connect.scylladb.ScyllaDbSessionFactory:152)
[2021-03-18 04:41:57,522] INFO Failed to configure address translator, provide a valid JSON string with external network address and port mapped to private network address and port. (io.connect.scylladb.ScyllaDbSessionFactory:54)
[2021-03-18 04:41:57,522] INFO Configuring public contact points=localhost:9042 (io.connect.scylladb.ScyllaDbSessionFactory:145)
[2021-03-18 04:41:57,546] ERROR WorkerConnector{id=scylladb-sink-connector} Error while starting connector (org.apache.kafka.connect.runtime.WorkerConnector:194)
java.lang.IllegalArgumentException: Failed to add contact point: localhost:9042
	at com.datastax.driver.core.Cluster$Builder.addContactPoint(Cluster.java:925)
	at com.datastax.driver.core.Cluster$Builder.addContactPoints(Cluster.java:945)
	at io.connect.scylladb.ScyllaDbSessionFactory.configurePublicContactPoints(ScyllaDbSessionFactory.java:148)
	at io.connect.scylladb.ScyllaDbSessionFactory.newSession(ScyllaDbSessionFactory.java:57)
	at io.connect.scylladb.ScyllaDbSinkConnector.start(ScyllaDbSinkConnector.java:48)
	at org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:186)
	at org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:211)
	at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:350)
	at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:333)
	at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:141)
	at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: localhost:9042: Temporary failure in name resolution
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
	at com.datastax.driver.core.Cluster$Builder.addContactPoint(Cluster.java:922)
	... 15 more
[2021-03-18 04:41:57,551] ERROR [Worker clientId=connect-1, groupId=connect-cluster] Failed to start connector 'scylladb-sink-connector' (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1339)
org.apache.kafka.connect.errors.ConnectException: Failed to start connector: scylladb-sink-connector
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$startConnector$5(DistributedHerder.java:1305)
	at org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:336)
	at org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:141)
	at org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

and passed message in topic1 as :
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"emailid"},{"type":"string","optional":false,"field":"ip"},{"type":"string","optional":false,"field":"browser"},{"type":"string","optional":false,"field":"pagetype"}],"payload":{"id":1,"emailid":"uaer@gmail.com","ip":"172.123.145.000","browser":"chrome6.0","pagetype":"home"}}}
message in consumer:
./bin/kafka-console-consumer --topic topic1 --from-beginning --bootstrap-server localhost:9092 --property print.key=true

null	{"schema":{"type":"map","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"emailid"},{"type":"string","optional":false,"field":"ip"},{"type":"string","optional":false,"field":"browser"},{"type":"string","optional":false,"field":"pagetype"}],"payload":{"id":1,"emailid":"uaer@gmail.com","ip":"172.123.145.000","browser":"chrome6.0","pagetype":"home"}}}

So the problem is that your messages in Kafka don't have a key properly set (the key is null, as printed by kafka-console-consumer). The key should be filled in and only contain fields that will be a part of PRIMARY KEY in destination Scylla cluster.
I see that you tried to use ValueToKey, but it fails (this is not a part of Scylla Sink Connector). My guess is that you are using org.apache.kafka.connect.storage.StringConverter instead of org.apache.kafka.connect.json.JsonConverter and because of that ValueToKey can't extract id field.

I have been successfully able to configure the connector with the following configuration:
{
  "name": "scylladb-sink-connector",
  "config": {
    "connector.class": "io.connect.scylladb.ScyllaDbSinkConnector",
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "transforms": "createKey",
    "topics": "topic1",
    "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields": "id",
    "scylladb.contact.points": "127.0.0.1",
    "scylladb.port": "9042",
    "scylladb.keyspace": "ks",
    "key.converter.schemas.enable": "true",
    "value.converter.schemas.enable": "true"
  }
}

And by sending the following message to a topic.
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"emailid"},{"type":"string","optional":false,"field":"ip"},{"type":"string","optional":false,"field":"browser"},{"type":"string","optional":false,"field":"pagetype"}]},"payload":{"id":1,"emailid":"uaer@gmail.com","ip":"172.123.145.000","browser":"chrome6.0","pagetype":"home"}}

Before, your posted JSON was invalid, as the payload field was erroneously in schema instead of being top level:

I have been successfully able to configure the connector with the following configuration:
{
  "name": "scylladb-sink-connector",
  "config": {
    "connector.class": "io.connect.scylladb.ScyllaDbSinkConnector",
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "transforms": "createKey",
    "topics": "topic1",
    "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
    "transforms.createKey.fields": "id",
    "scylladb.contact.points": "127.0.0.1",
    "scylladb.port": "9042",
    "scylladb.keyspace": "ks",
    "key.converter.schemas.enable": "true",
    "value.converter.schemas.enable": "true"
  }
}

And by sending the following message to a topic.
{"schema":{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"emailid"},{"type":"string","optional":false,"field":"ip"},{"type":"string","optional":false,"field":"browser"},{"type":"string","optional":false,"field":"pagetype"}]},"payload":{"id":1,"emailid":"uaer@gmail.com","ip":"172.123.145.000","browser":"chrome6.0","pagetype":"home"}}

Before, your posted JSON was invalid, as the payload field was erroneously in schema instead of being top level:

It was a documentation issue, noticed that one, now the messages are inserting. Thanks for reply!

Could you also post messages on the topic?
Command to do it:
kafka-avro-console-consumer --broker-list localhost:9092 --topic topic1 --from-beginning --property print.key=true

(note that on my system --broker-list is called --bootstrap-server)
And also print logs of connector?
Command to do it:
confluent local services connect log

(command as of Confluent 6.0.0, if it doesn't work maybe look at /tmp/confluent.XXXXXX/connect/logs/connect.log, where XXXXXX are some numbers)

Have you tried avro format, I was able to connect and insert the messages only on above tried json-schema-payload method , Is config varies for avro format.
In my case, the message is being published from snowplow-thrift , so from consumer message i need to create a similar avro-producer and to dumb in scylla db.
Here is my conumer message sample :
{"ip":{"string":"37.47.61.165"},"browser":{"string":"Mozilla/5.0 (Linux; Android 6.0.1; SM-A500FU) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Mobile Safari/537.36"}}

repo: scylladb/kafka-connect-scylladb


text:
feature(.travis.yaml): Creating CI via travis

Move old CCM test to old_tests directory
Create new session fixtures:

results_dir - The directory of the cluster
test_id - unique cluster ID
docker_cluster - Cluster with 3 nodes

test_config.py - Create one place to create test variables
.gitignore - Add tests/test_results to file

@Orenef11 you are removing files here which are modified in other people PRs. please sync with them if those files should be removed or not

@Orenef11 please undo the deletion of test files, and lets get this merged regardless of it the tests passes or not
so we could start seeing this running on the PRs we are opening, and then we could address the tests issues.

@fruch
All issues are fixed

repo: scylladb/scylla-ccm