text:
Invalid geometries after export/import

Context
The dataset with all geometries as valid generates, a different number of, invalid geometries when importing it depending on the export format.
Steps to Reproduce

Go to https://team.carto.com/u/saleiva/tables/chicago_census_tracts/public?redirected=true
Export:

CSV.
SHP.
GeoPackage.

Import all of them.
Run the queries:

SELECT count(*) FROM chicago_census_tracts_csv where st_isvalidreason(the_geom) != 'Valid Geometry'
SELECT count(*) FROM chicago_census_tracts_shp where st_isvalidreason(the_geom) != 'Valid Geometry'
SELECT count(*) FROM chicago_census_tracts_gpkg where st_isvalidreason(the_geom) != 'Valid Geometry'

Current Result
Tables count for invalid geometries:

chicago_census_tracts_csv: 0
chicago_census_tracts_shp: 3
chicago_census_tracts_gpkg: 36

Expected result
I would expect to have all geometries as valid, independently of the export format.
Additional info
Using QGIS Geometry Checker Plugin I wasn't able to detect invalid geometries from step 2 files. However, I was able to detect invalid geometries after exporting again the imported gpkg dataset at step 3.
cc @javisantana

Just curious and to let me manage priorities: who added this to Data services project?

Me.

This should be a priority since it's making some of the .CARTO we create for the use cases to not import correctly.

Hey, If the issue is in GDAL ping me ASAP so I can talk with Even

sure

thanks guys

I reproduced the issue but got different numbers:
Current Result
Tables count for invalid geometries:

chicago_census_tracts_csv: 0
chicago_census_tracts_shp: 1
chicago_census_tracts_gpkg: 5

Wondering what could have changed since then or if there's some source of variance in the process.

No modifications to the original table since the issue was reported:
 # SELECT * from cartodb.cdb_tablemetadata where tabname = 'saleiva.chicago_census_tracts'::regclass;
            tabname            |          updated_at
-------------------------------+-------------------------------
 saleiva.chicago_census_tracts | 2016-12-01 18:25:49.107569+00
(1 row)

and I checked again and got the same numbers

Invalid reasons

chicago_census_tracts_gpkg:

Hole lies outside shell[-87.5999603271484 41.6572640635485]
Self-intersection[-87.6329271739131 41.932908826087]
Self-intersection[-87.6247062332533 41.9068444417047]
Self-intersection[-87.614171 41.888397]
Self-intersection[-87.575521 41.795392]

chicago_census_tracts_shp:

Self-intersection[-87.632939 41.932914]

It looks like a precision problem to me.

Recursion test
I got the same numbers in local. Then I thought of running it recursively (feeding back the outputs to the inputs) and got the numbers again:

CSV: http://cdb.localhost.lan:8080/api/v2/sql?filename=chicago_census_tracts_csv_csv&q=SELECT+*+FROM+public.chicago_census_tracts_csv&skipfields=the_geom_webmercator&format=csv
SHP: http://cdb.localhost.lan:8080/api/v2/sql?filename=chicago_census_tracts_shp_shp&q=SELECT+*+FROM+public.chicago_census_tracts_shp&skipfields=the_geom_webmercator&format=shp
GPKG: http://cdb.localhost.lan:8080/api/v2/sql?filename=chicago_census_tracts_gpkg_gpkg&q=SELECT+*+FROM+public.chicago_census_tracts_gpkg&skipfields=the_geom_webmercator&format=gpkg

Tables count for invalid geometries:

chicago_census_tracts_csv_csv: 0
chicago_census_tracts_shp_shp: 2
chicago_census_tracts_gpkg_gpkg: 7

Same invalid reasons (holes and self-intersections). IMO that reinforces the hypothesis of a precision issue.

@rochoa in production and following the STR I only got 500 rows imported for SHP and GPKG formats, though the original file has 1031 rows. Do you know of any reason for that behavior?

Regarding the 500 rows apparent limit, it seems to be a different issue related to permissions when exporting public tables without api_key.
From the DB logs:
ERROR:  permission denied for relation geometry_columns
STATEMENT:  SELECT type, coord_dimension, srid FROM geometry_columns WHERE f_table_name = 'chicago_census_tracts' AND f_geometry_column='the_geom' AND f_table_schema = 'saleiva'
ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  SELECT pg_class.relname FROM pg_class WHERE oid = (SELECT pg_inherits.inhparent FROM pg_inherits WHERE inhrelid = (SELECT c.oid FROM pg_class c, pg_namespace n WHERE c.relname = 'chicago_census_tracts' AND c.relnamespace=n.oid AND n.nspname = 'saleiva'))

...

ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  FETCH 500 in OGRPGLayerReader0x2549e90
ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  CLOSE OGRPGLayerReader0x2549e90

and it has been introduced sometime between the 1st of December and today.

I'd consider it pretty important if that happens for every single export. Could be related to the onpremise release?

Exporting issues
Let's keep the api_key permissions discussion apart in its own ticket, as these are two separate issues.
Exporting the SHP
Exporting the SHP file in local, with no api_key:
http://cdb.localhost.lan:8080/api/v2/sql?filename=chicago_census_tracts_shp&q=SELECT+*+FROM+public.chicago_census_tracts&skipfields=the_geom_webmercator&format=shp
Using the Check validity feature of qgis 2.18, with the qgis method:

ogr2ogr binary
rows exported
invalid geoms (qgis)

ogr2ogr
1031
1

ogr2ogr2
1031
1

ogr2ogr2.1
1031
1

Exporting the GPKG
Exporting the GPKG file in local, with no api_key:
http://cdb.localhost.lan:8080/api/v2/sql?filename=chicago_census_tracts_gpkg&q=SELECT+*+FROM+public.chicago_census_tracts&skipfields=the_geom_webmercator&format=gpkg
I couldn't import the resulting gpkg back in qgis, so measuring invalid geoms as per st_isvalid(the_geom):
SELECT count(*) FROM chicago_census_tracts_gpkg where not st_isvalid(the_geom);

ogr2ogr binary
rows exported
invalid geoms (postgis)

ogr2ogr
1031
7

ogr2ogr2
1031
7

ogr2ogr2.1
1031
7

Actually qgis complains about the same error on the original table imported directly from postgres: segments 167 and 173 of line 0 intersect at -87.658697, 41.909483, though all rows are OK as per postgis.
So, instead of using qgis (which underneath uses ogr2ogr as well) let's focus on a possible precision loss and cumulative errors in export/import process and how they impact feature validity from postgis point of view.

(As suggested by Santana) Let's pick one invalid feature:
SELECT cartodb_id, st_isvalidreason(the_geom) FROM chicago_census_tracts_gpkg where st_isvalidreason(the_geom) != 'Valid Geometry' LIMIT 1;
 cartodb_id |                   st_isvalidreason
------------+-------------------------------------------------------
        967 | Self-intersection[-87.6783312117263 41.8422366970676]
(1 row)

and compare the original WKB with the BLOB stored in the sqlite file:
$ sqlite3 chicago_census_tracts_gpkg.gpkg
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .mode csv
sqlite> .output gpkg_blob_out.csv
sqlite> select hex(geom) from chicago_census_tracts_gpkg  where cartodb_id = 967;
sqlite> .output stdout

The binary representations of the original and the exported blob look very similar, except for the first few bytes (it's a 9 KB multi-polygon feature).
I tried this as well in sqlite:
SELECT Hex(ST_AsBinary(geom)) from chicago_census_tracts_gpkg where cartodb_id = 967;

but couldn't get any output. See https://www.gaia-gis.it/gaia-sins/spatialite-cookbook/html/wkt-wkb.html

After importing the GPKG and inspecting the WKB directly in the database, the diff shows subtle differences in "random" bytes, hopefully some LSB's in coordinates:

So at least the issue with GPKG (which matter most because of .carto format) is not exporting but importing GPKG files.

The next question is whether this bug happens before or after ogr2ogr command execution. This is the command line from my dev machine:
OSM_USE_CUSTOM_INDEXING=NO PG_USE_COPY=YES PGCLIENTENCODING=UTF-8  /usr/bin/ogr2ogr2.1 \
  -f PostgreSQL  PG:"host=localhost port=5432 user=development_cartodb_user_95dc74f3-84b8-4738-a16e-92bab6a9a9af dbname=cartodb_dev_user_95dc74f3-84b8-4738-a16e-92bab6a9a9af_db password=f351585ea8623d35033ea5684c462e724051626adevelopment_cartodb_user_95dc74f3-84b8-4738-a16e-92bab6a9a9af" \
  -t_srs EPSG:4326  -lco DIM=2 -lco PRECISION=NO /tmp/imports/20170105-17614-5uhokg/chicago_census_tracts_gpkg.gpkg  \
  -nln cdb_importer.importer_79ce31b0d36511e69bd9080027880ca6 -nlt PROMOTE_TO_MULTI  \
  -doo PRELUDE_STATEMENTS="SET statement_timeout TO '1h'" -doo CLOSING_STATEMENTS='SET statement_timeout TO DEFAULT' -update

(new lines added for readability)
Admittedly the option -lco PRECISION=NO stinks.
After executing it and checking the resulting table:
select count(*) from cdb_importer.importer_79ce31b0d36511e69bd9080027880ca6 where not st_isvalid(wkb_geometry);
NOTICE:  Self-intersection at or near point -87.67833121172626 41.842236697067641
NOTICE:  Self-intersection at or near point -87.658417636367616 41.909339727274777
NOTICE:  Self-intersection at or near point -87.632927173913075 41.932908826086972
NOTICE:  Self-intersection at or near point -87.624706233253292 41.906844441704735
NOTICE:  Self-intersection at or near point -87.614170999999985 41.888396999999991
NOTICE:  Self-intersection at or near point -87.575520999999981 41.795391999999993
NOTICE:  Self-intersection at or near point -87.704673999999983 41.83199399999998
 count
-------
     7
(1 row)

So at least we can discard cartodby'ation as the source FTM.

With -lco PRECISION=YES I got the same result :disappointed:

The PRECISION option is targeted at postgres output (lco stands for "layer creation option") and that option has been there for ages (w/o affecting CSV format for instance). See http://www.gdal.org/drv_pg.html

I made a small test with one of the "offending" features:
original.zip
# convert csv -> gpkg -> csv
ogr2ogr -f GPKG original.gpkg original.csv
ogr2ogr -f CSV converted.csv original.gpkg

# compare them, ignoring CR new lines
$ diff --strip-trailing-cr original.csv converted.csv  && echo "OK"
OK

It doesn't look like the issue is in the gdal gpkg driver, but a combination with postgis driver.

This seems to be a datum conversion issue; the geometry in the geopackage (or shapefile) is not interpretedt as EPSG:4326.
Omiting the -t_srs EPSG:4326 from the ogr2ogr import avoids the problem, but when it is present, the coordinates are slightly altered.

On second thought, It could be not an SRS misinterpretation,but that the presence of -t_srs introduces a transformation step (in this case applying an identity transformation if SRIDs are right) which introduces some roundoff errors.

seems like that's also an optimization in gdal, right? we might send a patch for this...

The underlying problem we have here is that in some cases (involving conversion to PostGIS with  ogr2ogr) in which the WKB form of geometry should be preserved exactly (because no coordinate transformation is involved and the data source preserves the their full precision) the coordinates are altered in their least significant digits.
This in turn can cause the invalid geometry result observed for complex-enough polygons, since tiny displacements derived from the changed coordinates can change the topology of lines.
Here's how to reproduce the coordinate-changing problem using the 2.1 version of  ogr2ogr used at CARTO.
We'll use this point to reveal the problem:

WKB: (hex) 0101000020E610000007A359D93EEB55C05812A0A696EB4440
Text: POINT(-87.675711 41.840535)

We'll be using a pnt.csv file with that point
id,geom
1,0101000020E610000007A359D93EEB55C05812A0A696EB4440

And a PostGIS table:
CREATE TABLE pnt(cartodb_id integer, the_geom geometry);
INSERT INTO pnt(cartodb_id, the_geom)
  VALUES (1, '0101000020E610000007A359D93EEB55C05812A0A696EB4440'::geometry);

We can export this data to a GPKG using CARTO (the SQL API):
curl "http://cdb.localhost.lan:8080/api/v2/sql?filename=pntexp&q=SELECT+*+FROM+pnt&skipfields=the_geom_webmercator&format=gpkg" > pnt.gpkg

or directly with ogr2ogr obtaining an equivalent result:
ogr2ogr2.1 -f GPKG /pnt.gpkg pnt.csv -oo GEOM_POSSIBLE_NAMES=wkb_geom -a_srs EPSG:4326

Now we have a GPKG that preserves the WKB geometry exactly:
$ sqlite3 pntexp.gpkg
sqlite> select * from gpkg_contents;
pntexp|features|pntexp||2017-01-10T11:26:53.000Z|-87.6757|41.8405|-87.6757|41.8405|4326
sqlite> select * from pntexp;
1|GP|1
sqlite> select hex(geom) from pntexp;
47500001E6100000010100000007A359D93EEB55C05812A0A696EB4440
sqlite> .quit

Now we convert the GPKG back to PostGIS with ogr2ogr using the -t_srs
option to transform the geometry to WGS84 (4326). Since has already that
reference as shown in the GPKG metadata above, the transformation should be
the identity.
ogr2ogr2.1 \
 -f PostgreSQL  PG:"host=localhost port=5432 user=development_cartodb_user_bd5106b3-3463-4182-a819-d8a53d6f95ca dbname=cartodb_dev_user_bd5106b3-3463-4182-a819-d8a53d6f95ca_db password=c21cde21e4abf48bb6ec3b0505bc8bca922cf476development_cartodb_user_bd5106b3-3463-4182-a819-d8a53d6f95ca" \
 -t_srs EPSG:4326  \
 /vagrant/pntexp.gpkg \
 -nln cdb_importer.pnt_imp

But the geometry has been (slightly) altered:
select * from cdb_importer.pnt_imp;
 fid | cartodb_id |                    wkb_geometry
-----+------------+----------------------------------------------------
   1 |          1 | 0101000020E610000006A359D93EEB55C05812A0A696EB4440
(1 row)

cartodb_dev_user_bd5106b3-3463-4182-a819-d8a53d6f95ca_db=> select st_astext(wkb_geometry) from cdb_importer.pnt_imp;
             st_astext
------------------------------------
 POINT(-87.6757109999999 41.840535)
(1 row)

If we omit the -t_srs (or if we replace it by -a_srs which assigns the SRS without transformation)
this problem doesn't happen:
ogr2ogr2.1 \
 -f PostgreSQL  PG:"host=localhost port=5432 user=development_cartodb_user_bd5106b3-3463-4182-a819-d8a53d6f95ca dbname=cartodb_dev_user_bd5106b3-3463-4182-a819-d8a53d6f95ca_db password=c21cde21e4abf48bb6ec3b0505bc8bca922cf476development_cartodb_user_bd5106b3-3463-4182-a819-d8a53d6f95ca" \
 /vagrant/pntexp.gpkg \
 -nln cdb_importer.pnt_imp2

cartodb_dev_user_bd5106b3-3463-4182-a819-d8a53d6f95ca_db=> select * from cdb_importer.pnt_imp2;
 fid | cartodb_id |                    wkb_geometry
-----+------------+----------------------------------------------------
   1 |          1 | 0101000020E610000007A359D93EEB55C05812A0A696EB4440
(1 row)

cartodb_dev_user_bd5106b3-3463-4182-a819-d8a53d6f95ca_db=> select st_astext(wkb_geometry) from cdb_importer.pnt_imp2;
          st_astext
-----------------------------
 POINT(-87.675711 41.840535)
(1 row)

Converting the same GPKG to CSV instead of PostGIS with ogr2ogr
is free of the problem as well:
ogr2ogr2.1 -f CSV pnt2.csv pntexp.gpkg -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT
cat pnt2.csv
WKT,cartodb_id
"POINT (-87.675711 41.840535)",1

Note that changes were introduced in OGR 1.11.0 to treat identity transformations: https://trac.osgeo.org/gdal/changeset/26274
(in principle it seems that this should have solve this problem)

re. https://trac.osgeo.org/gdal/changeset/26274 these lines look a bit fragile to me:
 	755	    /* Determine if we really have a transformation to do */
 	756	    bIdentityTransform = (strcmp(pszSrcProj4Defn, pszDstProj4Defn) == 0);

I'd compile it with debug traces enabled and/or run it through a debugger.

That text comparison seems really fragile, but it is not the problem in our case (the definitions are identical, maybe both have been extracted from the PROJ4 epsg file, because I don't think PostGIS spatial_ref_sys or the GPKG gpkg_spatial_ref_sys is being used at this point).
In our case the transformation is correctly identified as an Identity, but the coordinates have been converted to radians, and the error is introduced when converting back to degrees:
Debugging in ogrct.cpp from GDAL we have in our problematic conversion:
(gdb) s
1162	                x[i] *= dfTargetFromRadians;
(gdb) p x[i]
$5 = -1.5302298309770097
(gdb) s
1163	                y[i] *= dfTargetFromRadians;
(gdb) p x[i]
$6 = -87.67571099999995

Tomorrow I'll take a look at why are coordinates being converted to radians here.

I've been checking the radias/degrees conversions:
First, the deg->rad->deg conversions could been done preserving the original value (at least in our test case) if using the DEG_TO_RAD constant, but a different value is used here which corresponds to the less precise literal in the projection definition 0.0174532925199433
These values (with less precision than IEEE 64 bit floating point numbers) are used in a lot of projection definition files, so we don't have control over it.
What can be done is to try to detect deg->rad->deg round-trip conversions (allowing some tolerance in the conversion factors) and avoid them. And this precisely the intention of the deactivated and commented-out code here (which has been completely removed from trunk)
I've checked that code and it fixes our problem. I'll try to find out why was it dismissed.

For the record, here's a snippet showing how I got to debug our test case:
First, compile GDAL with debug symbols and open ogr2ogr with gdb:
wget http://download.osgeo.org/gdal/2.1.1/gdal-2.1.1.tar.gz
tar -xzf gdal-2.1.1.tar.gz
cd gdal-2.1.1
./configure --without-ld-shared --disable-shared --enable-static --with-ogdi=no --with-spatialite=yes --enable-debug
make
export GDAL_DATA=$(pwd)/data
gdb apps/ogr2ogr

Now we can set breakpoints as needed and execute the conversion of our test case:
b ogrct.cpp:859
...
r -f PostgreSQL PG:"host=localhost port=5432 user={DBUSER} dbname={DBNAME} password={PASSWORD}"  -t_srs EPSG:4326  pntexp.gpkg  -nln pnt_imp_dbg
...

Hey @rouault, could you shed some light on why this code hasn't been used? It would avoid the loss of precision due to the use of the degree definition common in WKT files when source and target are in the same long/lat units.
It seems that code originated in this ticket: https://trac.osgeo.org/gdal/ticket/5188

@jgoizueta: I'm not sure why the code has been commented. Probably because it looked like a hack to set bSourceLatLong = FALSE; and bTargetLatLong = FALSE. A potential fix would be to add a bNoOperation = TRUE flag ( bIdentifyTransform only applies to proj.4 related conversions, not angular unit conversions )

Thanks for the info and your suggestions @rouault !  I'll take a look at adding a new flag to skip conversions as you propose.

A note on why we haven't had this problem with CSV files:
With the options passed to ogr2ogr to import files into PostGIS , ogr2ogr does not regard the geometry columns as geometry, and they're imported verbatim as text (hex WKB representation of the geometry). Then the import process (cartodbfication etc.) detects the columns as geometric and converts it, but no coordinate transformation is involved.
This works nicely for files exported from CARTO since the geometry is always EPSG:4326 and they are transparently converted back when imported.
If we use ogr2ogr options to make it handle the geometry the same problem occurs as for gpkg:
ogr2ogr -f PostgreSQL PG:"host=localhost port=5432 user=USER dbname=DBNAME password=PASSwoRD"  -t_srs EPSG:4326  pnt.csv  -nln cdb_importer.pnt_imp_csv -oo GEOM_POSSIBLE_NAMES=the_geom -t_srs EPSG:4326 -s_srs EPSG:4326

Here we use GEOM_POSSIBLE_NAMES to specify the name of the geometry column, -s_srs to define its reference system and -t_srs to transform it when imported.

After looking carefully into the code I agree the existing (disabled) patch is rather hackish and a equivalent but cleaner changer is in order.
It would consist simply of adding a flag bNoTransformation to class OGRProj4CT to skip transformation completely, assigned in OGRProj4CT::InitializeNoLock as:
bNoTransformation = bIdentityTransform && bSourceLatLong && !bSourceWrap &&
                    bTargetLatLong && !bTargetWrap &&
                    fabs(dfSourceToRadians * dfTargetFromRadians - 1.0) < EPSILON;

And then used it in OGRProj4CT::TransformEx to avoid transforming the coordinates.
Now, since we're considering changing  ogr2ogr version I think we could create a patch for this too, include it in our next version build and submit it to upstream (so we can remove it from our build if/when accepted).

Here's a patch to fix this problem: https://github.com/OSGeo/gdal/pull/184

The fix has been committed by rouault (thanks!) in trunk r37138 and branches/2.1 r37139, with a fix in the test when running on 32bit Linux ( hex() adds a trailing 'L' in that case).
Proposed acceptance test:
(In a local or staging environment)

Load the CSV data into an account, check 1031 rows are imported and none are invalid

build ogr2ogr from branch 2.1, install it in the test environment

Export the data to CSV, SHP & Geopackage

Reimport each format, check it to have 1031 rows and no invalid geometries

@ethervoid in order to coordinate, acceptance test the 2 bugfixes and prepare the release: what commit/revision are you testing against?
At the time of writing this, it is https://github.com/OSGeo/gdal/commit/3c42908b43090c367450f641e916c23ed643b105 / r37152

@rafatower I've packaged this one: r37152 | rouault | 2017-01-15 14:07:29 +0000 (Sun, 15 Jan 2017) | 1 line

It looks like we still have a problem with the current patch:
Using the production version of GDAL 2.1, I've generated three new files (CSV, GPKG and SHP) using this file as base:
PGCLIENTENCODING=UTF-8 ogr2ogr2.1 -f CSV chicago_census.csv chicago_census_tracts_csv.csv -oo GEOM_POSSIBLE_NAMES=the_geom -s_srs EPSG:4326 -t_srs EPSG:4326
PGCLIENTENCODING=UTF-8 ogr2ogr2.1 -f GPKG chicago_census.GPKG chicago_census_tracts_csv.csv -oo GEOM_POSSIBLE_NAMES=the_geom -s_srs EPSG:4326 -t_srs EPSG:4326
PGCLIENTENCODING=UTF-8 ogr2ogr2.1 -f "ESRI Shapefile" chicago_census.shp chicago_census_tracts_csv.csv -oo GEOM_POSSIBLE_NAMES=the_geom -s_srs EPSG:4326 -t_srs EPSG:4326

The results after import them into CARTO and check for invalid geometries are the following:
chicago_census_tracts_csv: 0
chicago_census_tracts_shp: 8
chicago_census_tracts_gpkg: 10

I've packaged the GDAL in the last revision of the branch 2.1 and installed it. After generating the same three files with it, the results are:
chicago_census_tracts_csv: 0
chicago_census_tracts_shp: 3
chicago_census_tracts_gpkg: 0

So we still have a problem with the reprojection part even with the patch. After talk with @jgoizueta he has found the problem: the  condition
irb(main):004:0> a = 0.0174532925199433
=> 0.0174532925199433
irb(main):005:0> b = 1/0.017453292519943295
=> 57.29577951308232
irb(main):006:0> (a*b - 1.0) < Float::EPSILON
=> false
irb(main):007:0> (a*b - 1.0) == Float::EPSILON
=> true

We have input units defined but some factor A and output units defined by other factor B.
Using DBL_EPSILON as the tolerance to consider if A/B = 1 (i.e. if A and B define the same units) I was given enough tolerance for the floating point involved if A and B where identical, but not for the actual case we have found where A and B are defined with varying precision: degrees are defined as 0.0174532925199433 (radians per degree) in PROJ4 configuration files (epsg) and as 0.017453292519943295 in a typical shapefile's .prj.
I should have accounted to that and used a larger tolerance 😢
Now, for this particular case it just happens that A/B -1 == DBL_EPSILON so if a <= had been used instead of a < it would have worked! 😆   (but that's just a coincidence)
I think we can work out a better value for this tolerance by considering the minimum number of correct significant number that we should require the units definition to have.
I think we can look at that and make another change to GDAL to increase the tolerance. Otherwise, with the current fix we have GPPK (.carto) files working transparently but shapefiles with slight discrepancies (all depending on how are units defined for the particular format/file)

In the found case both definitions have at least 14 correct significant digits, so 10^-13 would be an appropriate tolerance for such cases. If we require N correct digits, we should use 10^(-N+1).

By looking at the GDAL code and configuration files, the pi/180 constant (radians per degree) is found defined as:

.0174532925199432958 default value used in code; correct to 18 rounded digits; enough to be casted to a double which is the best possible approximation to pi/180.
.017453292519943295 value used in code and in esri_extra.wkt; correct to 16 rounded digits; enough to be casted to a double which is the best possible approximation to pi/180. This is the value that is used int .prj files in EPSG:4326 shapefiles generated with ogr2ogr.
0.0174532925199433 most common value found in configuration files, including GPKG's gpkg_spatial_ref_sys and PostGIS spatial_ref_sys, correct to 15 rounded digits. This is used when assigning EPSG:4326 to a CSV with ogr2ogr, when generating GPKG, etc.
0.0174532925199432955 used in some configuration files (esri_StatePlane_extra), correct to 16 rounded digits.
0.01745329251994328 used in bsbdataset.cpp, correct to 15 rounded digits.

Note: for reference, 180/pi = 0.0174532925199432957692369076848861271344287188854172545609719144017100911460344944368224156963450948221230449250...
So, when input and output is in degrees, we can in general expect to have it defined by slightly different values with at least 15 correct digits.
As a result, I'd use a tolerance not smaller than 10^-14 in the check for same input/output units. In fact I see no problem having 10^-13 or even something larger to cover even less accurate definitions that could surface. This is only used only when the input and output is long/lat with equivalent datums, so I see no risk of regarding different units as the same.

I've found [this]
(https://github.com/OSGeo/gdal/blob/8dde8c78f51359e9de64850a15cf6c6e0af56713/gdal/ogr/ogr_srs_esri.cpp#L1586) in the code where the angular units are checked to be degrees:
The discrepancy in the degrees definition we have found is because when saving shapefiles, the degrees definition used for EPSG:4326 is converted to a more precise value to be saved in the .prj file:
/* -------------------------------------------------------------------- */
/*      reset constants for decimal degrees to the exact string ESRI    */
/*      expects when encountered to ensure a matchup.                   */
/* -------------------------------------------------------------------- */
    OGR_SRSNode *poUnit = GetAttrNode( "GEOGCS|UNIT" );

    if( poUnit != NULL && poUnit->GetChildCount() >= 2
        && std::abs(GetAngularUnits()-0.0174532925199433) < 0.00000000001 )
    {
        poUnit->GetChild(0)->SetValue("Degree");
        poUnit->GetChild(1)->SetValue("0.017453292519943295");
    }

Note how the absolute tolerance for units equivalence used here is 1E-11. Taking into account this is for a magnitude of the order of 1E-2 (0.017...) that would be equivalent to using 1E-10 for our relative tolerance (fabs(dfSourceToRadians * dfTargetFromRadians - 1.0) < 1E-10) which is the actual value use originally in code that was removed. (shame on me for not having maintained that code as was written)

Now we have two possible criteria to choose the new tolerance:
To admit all observed definitions of degrees could expect to have 14 correct digits in the definitions and use a value of 10E-13. This is the minimum value we should use to allow all those cases.
To be consistent with the ESRI definitions and allow for any variation introduced in that case we should use a value of 10E-9 (equivalent require no more than 10 correct digits).

IMHO I don't think we really want to do "conversions" from "ESRI degrees" to "whatever degrees". All degrees should be dealt as equivalent. The numerical differences in WKT definitions are a matter of history, but not really significant (ie we don't want 45 deg to be converted to 45.0000000001 deg when converting from a shapefile to a geopackage). So a rather relaxed test to ensure that the source and target units are the "same" should be OK.
I'm wondering if the ogrct code shouldn't even force the use of .0174532925199432958 when the unit is close-enough to degree.

My decision is to use 10-E9. The greater tolerance shouldn't be a problem (what can an angular unit that fits 359.999999 of them in a circle be other than degrees?)

Hi rouault glad to have your opinion here. I agree with you with having a relaxed acceptance of degrees.
I also think it's a good idea adjusting the conversion factors when close enough to degrees. It would make for more accurate and consistent coordinate conversions. I'll leave it out of this ticket, though. Here our purpose is to have exact round-trip conversions when possible and we'll achieve it with the tolerance change.

Testing again against gdal2.1-static-bin (2.1.2+svn.37167-precise1) in staging, counting invalid geometries:
SELECT count(*) FROM table where st_isvalid(the_geom) is false;

Results:
chicago_census_tracts_csv: 0
chicago_census_tracts_shp: 0
chicago_census_tracts_gpkg: 0

vamos!

I ran the following regression tests as well:

Import a non-trivial .carto file: OK
Duplicate a dataset (and add new data to it): OK
Create a table from a SQL query: OK
Create a sync table: OK
Force sync: OK
raster import: OK
Create a table with ogr2ogr 2.1.2+svn.37167-precise1: OK

Ready for production.

Deployed to production, should be fixed by now

authors: ethervoid javisantana jgoizueta rafatower rochoa rouault saleiva

repo: CartoDB/cartodb


text:
Geometry detector doesnt detect geoms if the dataset is not ~completely geocoded

If you have a dataset with some rows with the_geom = null, and some polygons in the last rows, the map is drawn but:

340 rows have the_geom populated
32485 rows don't have the_geom populated
We think that the sampling could not be enough in this case. Would it be worthy to be more aggressive here?
Ask me for the table sample if needed, it's private.
If I just get the_geom from the dataset and import it, this cannot be reproduced. File here: https://dl.dropboxusercontent.com/u/2879308/geoms_guessing.geojson
@Kartones @xavijam

Bump, it seems that we're taking samples and if they don't contain a point, the table is detected as non-georeferenced. This is not just making the message in the image below to appear, but blocking all the wizards for map styling.

Ask @saleiva or @javisantana to priorize it for next milestone if is happening a lot or just in this case, to check sampling and its retrieval are big enough and representative enough

this is a known issue and it's been like this since the beginning, low priority

A little bit more context:
in order to know what is the geometry type we look into the first columns. We do in that way because executing a query to get the geometry types could be pretty expensive depending on the table. There are cases like this where first rows are null so this happens.
BTW, this is a frontend issue

Happening for S/B 6320578 also.

There has been no activity on this issue for more several months. We are closing it. If you think this still needs to be addressed please open a new issue.

source: gharchive/issue

authors: Kartones alonsogarciapablo iriberri javisantana ztephm

license_type: permissive

license_source: github-api


text:
Adds hover tip for tracts in the AOI analysis

Closes #10352
This PR adds an explanation about the tracts in the Area of Influence analysis.

As suggested by @ohasselblad, I've also swapped the distance and units labels to make the form more understandable and correct:
Before the change:

After the change:

CR: @xavijam

🇳🇿

source: gharchive/pull-request

authors: javierarce xavijam


text:
Updating dependencies (CartoDB.js, DI.js,...)

Mainly for releasing this: https://github.com/CartoDB/cartodb/issues/12574
cc @IagoLast and @alonsogarciapablo because I'm updating tangram dependency 😬 .

Testing it in ded02.

I just merged https://github.com/CartoDB/cartodb/pull/12579 @xavijam which updates dependencies too. I'm releasing it right now.

Perfect, thanks @alonsogarciapablo 💋 ❤ .

authors: alonsogarciapablo xavijam


text:
Hiding signup link from header if it isn't SaaS

Ref. ticket #2191 : **"create an account" should be removed from open-source edition **
Assuming that:
SaaS ==> account_host: ‘cartodb.com’ in app_config.yml
I had to add account_host var to javascript from Cartodb.config[:account_host]
Tests modification adding account_host to db.config.

account_host:       'localhost.lan:3000'

account_host:       'cartodb.com'

Frontend tests were OK :+1: (details)

Please, @javisantana @CartoDB/frontend , could you check it? thanks!

This change probably requires that you update the new public header too

Oh, you're right @viddo. I'll do it. thanks!

:es:

account_host: 'cartodb.com'

account_host: 'localhost.lan:3000'

cc @viddo

Frontend tests were OK :+1: (details)

:+1:

Closing this PR because it continues in this 'general' one #2646

authors: Cartofante Xatpy javisantana viddo


text:
Accept multiple --group-match for OR search.

Related to :

https://github.com/awslabs/ssosync/pull/55

https://app.shortcut.com/cartoteam/story/198357/configure-aws-sso

repo: CartoDB/ssosync


text:
Getting relevant data from Layer

An important part of what consists of our Web SDK is to let users access the data behind the visualization they are watching on their computers' screen, whether it is the features rendered on screen, the extent of the features of the layer or the number of rows of the whole dataset.
So for that matter, new functions have to be added to our Layer specification. Replicating the ones that we had in CARTO-VL, we thought of adding those ones:
Layer.getViewportFeatures()
This method will return an array containing GeoJSON objects for the features that are visible within the current viewport.
Proposed usage:
const layer = new carto.viz.Layer('dataset');
layer.addTo(deckInstance);

// Get current viewport features
(await?) layer.getViewportFeatures();

Depending on whether we do it with an SQL query or filtering features within a layer, we would need to add an await keyword. Both approaches have upsides and downsides.

Filtering the rendered feature data from our layer
This approach is based on getting features from MVTLayer, filter them by the bounds of the current viewport and returning them in our function. It is completely synchronous, so no need to await for our results, but filtering happens on client. But given that our coordinates are local coordinates we would need to convert them into WGS84 coordinates before coordinate comparison

Requesting rows within viewport to SQL API
This approach is much easier for us given that SQL API will perform the filtering and then we will return the rows' data in our function. The downside is that the request can take too much time depending on the data that is matched, and we will need to add an await keyword to our execution statement.

Layer.getExtent()
This method will return the bounds of our layer in the map, the coordinates of the rectangular area that encloses all layer features.
Proposed usage:
const layer = new carto.viz.Layer('dataset');
layer.addTo(deckInstance);

// Get layer extent
await layer.getExtent();

Return value:
[
 { longitude: -180, latitude:  90 },
 { longitude:  180, latitude:  90 },
 { longitude:  180, latitude: -90 },
 { longitude: -180, latitude: -90 },
]

To get layer's extent, we would need to request it to SQL API, so there's no discussion for this method's approach.
Layer.getNumberOfRows()
I think that was @VictorVelarde who added this method to Clubhouse's task, but I find it kind of strange to ask our layer for the number of rows. It is probably a better fit to have a .getDataset() or .getSource() method, and then request the number of rows to that object. What do you think?
Proposed usage:
const layer = new carto.viz.Layer('dataset');
layer.addTo(deckInstance);

// Get layer extent
await layer.getDataset().getNumberOfRows();
await layer.getSource().getNumberOfRows();

Hi @jesusbotella  thx for your thoughts. Some comments on them.
Indeed, all of them look to me like methods belonging to an internal 'Source' object. The Layer methods would be probably just probably a facade on them, and a way to get some params for the queries.
viewportFeatures
Layer.getViewportFeatures()looks like a nice method name, that could be handled internally by the source, with the help of layer.getMap().getCurrentExtent() (not real names)
I would start with an await interface and the SQL API, as the first & quick approach. That could be probably enough for smaller datasets. From there, we can start exploring other options, like client-side filtering (that filtering would probably belong to MVTLayer, thus a deck.gl contribution). If we keep the SQL API approach, we should consider pagination as well.
extent
Layer.getExtentlooks nice too, although I would probably change the returned value, to something more explicit, like:
{
  lonMin: -180,
  latMin: -90,
  lonMax: 180,
  latMax: 90
}

or even the common xmin, ymin, xmax, ymax order [-180, -90, 180, 90]
xtra (number of rows...)
It's true, I added it as an example, and it should probably be developed together with some other 'basic stats' info related to the source (https://app.clubhouse.io/cartoteam/story/63820/data-basic-field-stats). So IMO we can postpone it

I'm gonna close this issue. We should better keep this "open discussions" in Clubhouse and leave GH mostly for PRs

authors: VictorVelarde jesusbotella

repo: CartoDB/toolkit


text:
rozśmieszyła mnie elewacja na początku art

https://ebvalaim.pl/2020/05/19/mierzenie-pozycji-iss/

Not related to the project.

repo: CasperLabs/casper-node


text:
no-tick Post release branch version bump, per versioning standards.

Bumping to 1.5.0 on master after the release of 1.4.0.

@Fraser999 has made a tool which bumps all versions but also crates versions. Not sure if we should use it for this PR. Here is how I've run it
cd ci/casper_updater/
cargo run --package=casper-updater -- --root-dir=. -b minor
Exsample PR after running it:
https://github.com/CasperLabs/test-casper-node-cicd/pull/6

Tried this and it bumps ALL libraries.  Not sure if we wanted to do that.  I know we were trying to only version those when they changed before, but that might have changed.  I did run his tool, but omitted the -b minor to not bump everything.
We might want to discuss this wider on a sprint entry sidebar on Monday, re: version bump of libraries..

bors r+

authors: piotr-putit sacherjj


text:
NDRS-556: add node ID enum

This PR introduces libp2p's PeerId type into the system.  It has been added alongside the existing node identifier in the form of an enum, with the expectation that we'll be able to support connections of either type (libp2p or our current TLS ones) while we transition the small_network component to use libp2p fully.
Once the transition is complete, the NodeId enum introduced here can be changed to a simple newtype struct.
Rather than making NodeId able to be randomly generated by any RNG, I strongly prefer to force the usage of our TestRng type, as was previously debated and agreed.  This caused a ripple effect whereby our CryptoRngCore trait has been removed in favour of a single, specific RNG type used throughout our crate.  In test configuration this is a TestRng and otherwise it is a ChaCha20Rng.
This still keeps things trivial should we wish to change from using that specific RNG.  It should only involve changing the alias in lib.rs.  As we designed the system to use not just a single RNG type, but a single RNG instance throughout, I don't see any advantage at all to using dyn CryptoRngCore everywhere: there is no intent to use different concrete RNG types in these places.

bors r+


text:
Login with Authentication key still asks for password

Bug report
Your Environment

Netflix add-on version: 1.12.0
Operating system version/name: LibreElec 9.2.6
Device model:  Raspberry Pi 4B

Used Operating system:

[ ] Android
[ ] iOS
[ ] Linux
[ ] OSX
[X] Raspberry-Pi
[ ] Windows

Describe the bug

When using Login with Authentication key, I get asked for the key file, and the PIN. Afterwards I still get prompted for a password. If I enter the password, it says password do not match.
Expected behavior

Login using key file and PIN only.
Actual behavior

Prompted for password.
Steps to reproduce the behavior

Configure plugin.
Browse to key file and enter.
Enter PIN

Possible fix

Debug log

The debug log can be found from this link:
http://paste.kodi.tv/lofizumoru
Additional context or screenshots (if appropriate)
Installation

[x] I'm using other Netflix Repo https://castagnait.github.io/repository.castagnait/
[ ] I'm using a different source

Other information

Screenshots

only one time the login ask the password
the account password is mandatory to input
if you do not type the right password you can not login with the Authentication key

The problem is that I do enter the right password, but it says it's wrong.
On Mon, 28 Dec 2020 at 19:46, Stefano Gottardo notifications@github.com
wrote:

only one time the login ask the password
the account password is mandatory to input
if you do not type the right password you can not login with the
Authentication key
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/CastagnaIT/plugin.video.netflix/issues/1003#issuecomment-751825778,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAWSQB4LVUREXVBDJRQWWJLSXDG6RANCNFSM4VMIQMSQ
.

I can do nothing this answer is given by the netflix website

you can try to change the password account and redo the auth key to see if will be accepted

That somehow did the trick! Thanks Stefano.
On Tue, 29 Dec 2020 at 14:13, Stefano Gottardo notifications@github.com
wrote:

you can try to change the password account and redo the auth key to see if
will be accepted
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/CastagnaIT/plugin.video.netflix/issues/1003#issuecomment-752070086,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAWSQBYJA6GCFMDDRSSGILTSXHIWBANCNFSM4VMIQMSQ
.

authors: CastagnaIT tomdeblende

license: MIT


text:
Netflix does not start

Bug report
Your Environment

Netflix add-on version: 1.10.0
Operating system version/name: OSMC (Buster), Kodi 18.8
Device model:  RPi 3B+

Used Operating system:

[ ] Android
[ ] iOS
[ ] Linux
[ ] OSX
[*] Raspberry-Pi
[ ] Windows

Describe the bug
I installed v1.0.1 of your repo using a zip install, then installed Netflix 1.10.0 and all the other suggested packages. When I go to start the addon, the circle goes round for a while then stops and nothing happens. Initially I tried to resolve it by installing python3-crypto. That didn't work so I removed it and installed python3-pycryptodome. That also failed. The logs are with python3-pycryptodome installed.
kodi.log
Expected behavior
Netflix should start
Actual behavior
The circle goes round for a bit then stops.
Steps to reproduce the behavior
Possible fix

I can confirm the same issue, the loading icon got stuck even though the contents are loaded. Therefore you can't see anything. It was working fine but the OSMC update in 10/2020 (a couple of days ago) might have interfered.
Workaround is to change the OSMC skin to the default one. Then I can view the content.

In my case I was using a Confluence skin. I have switched back to OSMC default, rebooted and there is no change. It looks like it is loading then does nothing.

                                     ImportError: No module named Crypto.Random

is missing a crypto package
why you do not want to install pycryptodomex?

I tried to resolve it by installing python3-crypto

Kodi 18 works only with python 2 not with python 3...

I didn't say I didn't want to install pycryptodomex. I tried googling around to see what package supplied the Crypto.Random package. During the upgrade I noticed python3 packages being installed so I assumed I'd need python3-pycryptodome. I am not a python ot Kodr expert.
The question is then, why isn't pycryptodomex as one of the packages pulled in as a dependency of the installation? It looks like the package needed is python-pycryptodome.
I have now installed it and Netflix worked once I rebooted it.

In case someone doesn't know how to install it: pip install pycryptodome

authors: CastagnaIT NickJH Quandm1195 VaultDeveloper


text:
Buddybuild crash report on build #525

Buddybuild detected a crash
Exception Class: RLMException
Object has been deleted or invalidated.

View Full Crash Details

Crash report

repo: CatchChat/Yep

license: mit

license_source: bigquery


text:
[REQUISITOS] Use Case - Fazer Upload de Vídeos

Descrição: O usuário realiza o upload de vídeos para o banco de dados.
Fluxo Básico:

O usuário (na seção de upload) anexa vídeos.
O usuário faz upload dos vídeos anexados.
O usuário recebe um popup de upload concluido.

É interessante verificar a necessidade de uma limitação para tamanhos enormes de vídeo e quantidade de uploads de um usuário

repo: CauBitten/SaraVideo


text:
🛑 Main Website is down

In 7d13f24, Main Website (https://aqurusmc.xyz) was down:

HTTP code: 502
Response time: 5151 ms

Resolved: Main Website is back up in c35c705 after 8 minutes.

repo: CcNicebruh/aqurusstatuspage


text:
Blog- App: Data model

In this milestone, I :

Created a new app
Initialize the project with Git.
Make sure that the project has Postgres database set up.
Built the project schema.

Created and ran the necessary migration files.
Ensured the table and column names match the ERD diagram.

Included Foreign keys.
Ensured that all columns that are foreign keys have a corresponding index.

The errors have nothing to do with my code. any ideas on why thy are there? I have not come across this before.


text:
🛑 MyNas is down

In 75b084b, MyNas (https://unraid.hiripple.com) was down:

HTTP code: 530
Response time: 61 ms

Resolved: MyNas is back up in a38fa61 after 10 minutes.

repo: CelestialRipple/ripplelog


text:
🛑 qinglong is down

In 4a7cccc, qinglong (ql.hiripple.com) was down:

HTTP code: 530
Response time: 117 ms

Resolved: qinglong is back up in c81d782 after 1 hour, 23 minutes.


text:
🛑 MyNas is down

In 6783576, MyNas (https://unraid.hiripple.com) was down:

HTTP code: 530
Response time: 93 ms

Resolved: MyNas is back up in 8ae4770 after 36 minutes.


text:
🛑 Nextcloud is down

In 613dac6, Nextcloud (https://nextcloud.hiripple.com) was down:

HTTP code: 530
Response time: 105 ms

Resolved: Nextcloud is back up in c18f1b3 after 1 hour, 1 minute.


text:
🛑 Alist is down

In 9a06ae9, Alist (https://alist.hiripple.com) was down:

HTTP code: 530
Response time: 8262 ms

Resolved: Alist is back up in 3b21f13 after 54 minutes.


text:
🛑 MyNas is down

In 278bfd3, MyNas (https://unraid.hiripple.com) was down:

HTTP code: 500
Response time: 133 ms

Resolved: MyNas is back up in d74422e after 4 hours, 1 minute.


text:
add a switch to disable PV in load test

Scale-out and Scale-up load tests both failed due ot PV/PVC processing of deployments&statefulsets. The deployments/statefulset which uses volumes will never get the pods in "Running" state.  As we are continuing to solve issue (tracked in https://github.com/CentaurusInfra/arktos/issues/978), this PR provides a switch to skip this issue temporarily.
When the env var  CL2_ENABLE_PVS is set to false, the test deployments&statefulsets will be created without attaching to PVs.
Verification:
This PR was tested on top of https://github.com/CentaurusInfra/arktos/pull/976, which fixed another error in load-test, and I got the following result:

Scale-up 100-node load test succeeded with commandline test-configs "--testconfig=testing/load/config.yaml --testoverrides=./testing/experiments/disable_pvs.yaml".
Scale-out 100-node load test succeeded with commandline test-configs "--testconfig=testing/load/config.yaml --testoverrides=./testing/experiments/disable_pvs.yaml".
Scale-up 100-node density test succeeded
Scale-out 100-node density test succeeded

/lgtm

/approve

repo: CentaurusInfra/arktos


text:
Standardize variants for callers of the same type

Description of feature
Not all callers report all variants in the same way. This needs some standardization.
Some links for inspiration:

SV: https://support.illumina.com/content/dam/illumina-support/help/Illumina_DRAGEN_Bio_IT_Platform_v3_7_1000000141465/Content/SW/Informatics/Dragen/OutputFilesMantaVCFInterpretation_fDG.htm
CNV: https://support.illumina.com/content/dam/illumina-support/help/Illumina_DRAGEN_Bio_IT_Platform_v3_7_1000000141465/Content/SW/Informatics/Dragen/CNVVCFFile_fDG_dtSW.htm => This can be done with bedgovcf
Repeat expansions: https://support.illumina.com/content/dam/illumina-support/help/Illumina_DRAGEN_Bio_IT_Platform_v3_7_1000000141465/Content/SW/RepeatGenotyping_fDG_dtSW.htm

Jasmine merging and normalization can be improved by following the steps in this pipeline: https://github.com/mkirsche/Jasmine/tree/master/pipeline

Overview of work to do:

[ ] ALT fields

[ ] Set all ALT fields to their corresponding SV (e.g. <DUP>, <DEL>...) => Get this from the INFO/SVTYPE field
[ ] Move insertion sequences to the INFO fields

[ ] FILTER fields

[ ] Streamline the requirements for filters to pass

[ ] QUAL fields

[ ] Streamline the quality

[ ] INFO

[ ] What can we keep and what is really necessary per variant type

[ ] INS
[ ] DEL
[ ] DUP
[ ] CNV
[ ] TRA
[ ] BND
[ ] Complex variants

[ ] Add the used caller (already done for CNV)
[ ] ...

[ ] FORMAT + sample

[ ] Standardize genotype

[ ] What to do with the CNV genotyping?
[ ] Gridss genotyping?
[ ] Is all genotyping done correctly and standardized between all callers?

[ ] What other fields are necessary per variant type

[ ] INS
[ ] DEL
[ ] DUP
[ ] CNV
[ ] TRA
[ ] BND
[ ] Complex variants

After a day of searching I could't really find a good resource to standardize VCF files that was easily available. I'm afraid we'll have to make our own tooling for this

I've started looking at the fields that we should include (at least for a first release) for the SV callers:

Paired and split read support:
The following FORMAT fields should be available in the merged vcf as FORMAT fields:

DELLY

##FORMAT=<ID=DR,Number=1,Type=Integer,Description="# high-quality reference pairs">
##FORMAT=<ID=DV,Number=1,Type=Integer,Description="# high-quality variant pairs">
##FORMAT=<ID=RR,Number=1,Type=Integer,Description="# high-quality reference junction reads">
##FORMAT=<ID=RV,Number=1,Type=Integer,Description="# high-quality variant junction reads">

available as
##FORMAT=<ID=PE,Number=2,Type=Integer,Description="Paired-read support for the ref and alt alleles in the order listed">
with PE=DR,DV
##FORMAT=<ID=SR,Number=2,Type=Integer,Description="Split-read support for the ref and alt alleles in the order listed">
with SR=RR,RV

SMOOVE

##FORMAT=<ID=PE,Number=1,Type=Integer,Description="Number of paired-end reads supporting the variant">
##FORMAT=<ID=SR,Number=1,Type=Integer,Description="Number of split reads supporting the variant">

available as
##FORMAT=<ID=PE,Number=2,Type=Integer,Description="Paired-read support for the ref and alt alleles in the order listed">
with PE=.,PE
##FORMAT=<ID=SR,Number=2,Type=Integer,Description="Split-read support for the ref and alt alleles in the order listed">
with SR=.,SR

MANTA

##FORMAT=<ID=PR,Number=.,Type=Integer,Description="Spanning paired-read support for the ref and alt alleles in the order listed">
##FORMAT=<ID=SR,Number=.,Type=Integer,Description="Split reads for the ref and alt alleles in the order listed, for reads where P(allele|read)>0.999">

available as
##FORMAT=<ID=PE,Number=2,Type=Integer,Description="Paired-read support for the ref and alt alleles in the order listed">
with PE=PR
##FORMAT=<ID=SR,Number=2,Type=Integer,Description="Split-read support for the ref and alt alleles in the order listed">
with SR=SR

genotype:
The following FORMAT fields should be available in the merged vcf as an FORMAT field:

DELLY

##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">

SMOOVE

##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">

MANTA

##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
available as
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">

Awesome thank you! I'll have a better look when I'm implementing that in svync :)

I've started looking at the fields that we should include (at least for a first release) for the SV callers:

The following INFO fields should be available in the merged vcf as an INFO field:

DELLY

##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">

available as
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
with SVLEN=|END-POS| or max(|END-POS|,|END2-POS2|)

SMOOVE

##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">

available as
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">
with SVLEN=|END-POS| or max(|END-POS|,|END2-POS2|)

MANTA

##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant described in this record">

The following ALT fields should be available in the merged vcf as an ALT field:

##ALT=<ID=DEL,Description="Deletion">
##ALT=<ID=DUP,Description="Duplication">
##ALT=<ID=INV,Description="Inversion">
##ALT=<ID=BND,Description="Translocation">
##ALT=<ID=INS,Description="Insertion">
##ALT=<ID=DEL,Description="Deletion">
##ALT=<ID=DUP,Description="Duplication">
##ALT=<ID=INV,Description="Inversion">
##ALT=<ID=DUP:TANDEM,Description="Tandem duplication">
##ALT=<ID=INS,Description="Insertion of novel sequence">
##ALT=<ID=CNV,Description="Copy number variable region">

I've started looking at the fields that we should include (at least for a first release) for the repeat caller:

The following fields should be available in the merged vcf:

##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the structural variant">
##INFO=<ID=RU,Number=1,Type=String,Description="The repeat unit of the STR">
##INFO=<ID=REPREF,Number=1,Type=Integer,Description="How many repeats are in the reference">
##INFO=<ID=RL,Number=1,Type=Integer,Description="The repeat length in basepairs">
##INFO=<ID=REPID,Number=1,Type=String,Description="The ID of the repeat">
##FORMAT=<ID=REPCN,Number=1,Type=String,Description="The mean copy number in this repeat expansion">
##FORMAT=<ID=REPCI,Number=1,Type=String,Description="The confidence interval for the size of the expanded allele">
##FORMAT=<ID=ADSP,Number=1,Type=String,Description="The amount of spanning reads consistent with the repeat allele">
##FORMAT=<ID=ADFL,Number=1,Type=String,Description="The amount of flanking reads that overlap the repeat allele">
##FORMAT=<ID=ADIR,Number=1,Type=String,Description="The amount of in-repeat reads that are consistent with the repeat allele">

available as
##INFO=<ID=SVTYPE,Number=1,Type=String,Description="Type of structural variant">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the structural variant">
##INFO=<ID=RU,Number=1,Type=String,Description="The repeat unit of the variant">
##INFO=<ID=REPREF,Number=1,Type=Integer,Description="Number of repeats in the reference">
##INFO=<ID=RL,Number=1,Type=Integer,Description="The repeat length in basepairs">
##INFO=<ID=REPID,Number=1,Type=String,Description="The ID of the repeat">
##FORMAT=<ID=REPCN,Number=1,Type=String,Description="The mean number of repeats">
##FORMAT=<ID=REPCI,Number=1,Type=String,Description="The confidence interval for the number of repeats">
##FORMAT=<ID=ADSP,Number=1,Type=String,Description="Spanning read support consistent with the repeat allele">
##FORMAT=<ID=ADFL,Number=1,Type=String,Description="Flanking read support that overlap the repeat allele">
##FORMAT=<ID=ADIR,Number=1,Type=String,Description="In-repeat read support consistent with the repeat allele">

You could consider adding SVLEN, but as a FORMAT field, because this can vary for every sample,
##FORMAT=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
with SVLEN=len(RU) x REPCN

or as an INFO field, if you make a comma separated list of all SVLENs for all the ALT alleles in the merged vcf
##INFO=<ID=SVLEN,Number=.,Type=Integer,Description="Difference in length between REF and ALT alleles">
with SVLEN=c(len(RU) x REPCN)

I've started looking at the fields that we should include (at least for a first release) for the CNV callers:

The following fields should be available in the merged vcf:

##INFO=<ID=SVTYPE,Number=1,Type=String,Description="The type of structural variant">
##INFO=<ID=END,Number=1,Type=Integer,Description="End position of the variant">
##INFO=<ID=SVLEN,Number=1,Type=Integer,Description="The total length of the structural variant">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=CN,Number=1,Type=Integer,Description="Copy number">

For the ID field I suggest the following formula:
Caller_SVTYPE_0000001
Caller_SVTYPE_0000002
Caller_SVTYPE_0000003

For the ID field I suggest the following formula:
Caller_SVTYPE_0000001 Caller_SVTYPE_0000002 Caller_SVTYPE_0000003
We could also consider a more specific ID like Caller_SVTYPE_CHROM_POS(2)_END(2)_REF_ALT that can then be used to list and filter recurrent false positive calls. This type of ID does not necessarily need to be stored in the vcf, we can calculate this on the fly in the filter step as well. Your thoughts?

IMO it will be easier to filter on the position and info fields instead of on the ID. Also how will we handle variants that are found by multiple callers and then merged by jasmine?

Yes but will we use another Caller ID or use a list of all callers in the ID? We could filter on positions and SVTYPE? there always is some uncertainty for SVs so a simple position filter won't filter out everything I think

authors: mvheetve nvnieuwk


text:
unfrobulate whit537's dev env

I've got myself in some kinda mess over on https://github.com/CenterForOpenScience/osf.io/pull/5294. Reticketing to avoid noise over there.

Picking up from https://github.com/CenterForOpenScience/osf.io/pull/5294#issuecomment-207053218 ...
I've made more room on my hard drive (fare thee well, old bitcoin wallet!).
$ df -h
Filesystem      Size   Used  Avail Capacity  iused   ifree %iused  Mounted on
/dev/disk1     112Gi   92Gi   20Gi    83% 24078175 5243551   82%   /
devfs          189Ki  189Ki    0Bi   100%      654       0  100%   /dev
map -hosts       0Bi    0Bi    0Bi   100%        0       0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%        0       0  100%   /home
$

I've restarted my computer.
I am at aa1042c, which was the last version I had working (https://github.com/CenterForOpenScience/osf.io/pull/5294#issuecomment-205874720).
I'm starting with a clean ps:
  PID TTY           TIME CMD
  423 ttys000    0:00.04 -bash
  487 ttys000    0:00.02 screen -D -RR -U
  515 ttys001    0:00.03 bash
  522 ttys002    0:00.03 bash
  529 ttys003    0:00.03 bash
  707 ttys003    0:02.63 mongod
  514 ttys004    0:00.05 bash
 1723 ttys004    0:00.00 pbcopy
  513 ttys005    0:00.05 bash
  509 ttys006    0:00.03 bash
  500 ttys007    0:00.03 bash
  501 ttys008    0:00.03 bash
  505 ttys009    0:00.06 bash
  499 ttys010    0:00.05 bash

I have tokumx running, as you can see.
I am running ./run-osf.sh (because inv assets fails without ... something from run-osf.sh) ...

Now assets ...
[osf.io]$ time inv clean_assets assets -d
...
real    0m41.080s
user    0m33.316s
sys     0m4.346s

(Sometimes it takes 3+ minutes, I don't know why.)

Now inv server ...

... and I'm reproducing the result at https://github.com/CenterForOpenScience/osf.io/pull/5294#issuecomment-207038745:

Okay! How about resetting the database?

Hrm ... I guess this is progress? Still getting the SyntaxError, but more of the UI shows up now ...

If I close the right sidebar I get the "Create Project" button. :no_mouth:

Width of the screen doesn't appear to be a factor.

... and opening the right sidebar again does not make the "Create Project" button go away again.

After creating a project, I am back to the completely unusable "My Projects" page.

So resetting the database helped because it remove all of my projects?
Indeed, if I manually remove the project (db.node.remove({'title':'Projectitude!'}) in the mongo cli), then I am back to https://github.com/CenterForOpenScience/osf.io/issues/5419#issuecomment-207073750.

Let's see about that SyntaxError ...

Its probably an exception from the API as HTML. Mithril is trying to parse it as JSON.

https://github.com/CenterForOpenScience/osf.io/pull/5294#issuecomment-207039157

The third time I hit the breakpoint (cf. ), it is while loading this URL:
http://localhost:8000/v2/users/me/nodes/?related_counts=children&embed=contributors&_=1460061498516
When I hit that directly, it does not appear to be an error. Is it HTML though?

The content-type of the response and the response itself look fine:

Where is Mithril finding a <?

BLAM!!!!!!!!!!!!!!
ESI.

Okay!

Now what's the fix?

(P.S. That is e.target at this breakpoint:

)

Is there a setting to turn off ESI? Am I supposed to be running Varnish in front of the API server?

$ cat api/base/settings/local.py
ENABLE_ESI=True
ENABLE_VARNISH = True
VARNISH_SERVERS = ['http://localhost:8193']  # This should be set in local.py or cache invalidation won't work
$

Left-overs from https://github.com/CenterForOpenScience/osf.io/pull/4745#issuecomment-167820503. :blush:

Sweet victory. :v:

Okay! Thanks for watching! :-)

Reopening to unfrobulate my waterbutler.

I've updated to the waterbutler at https://github.com/CenterForOpenScience/waterbutler/tree/63b7d469e5545de9f2183b964fb6264fd9a423a5. I've set up a virtualenv with:
$ virtualenv --python=python3.5 --prompt="[waterbutler] " env

I hit http://localhost:7777/data and I get a 500:
Then I env/bin/pip install -r requirements.txt and env/bin/inv server. When I
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/waterbutler/env/lib/python3.5/site-packages/tornado/web.py", line 1424, in _execute
    result = yield result
  File "/Users/whit537/workbench/cos/waterbutler/env/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/Users/whit537/workbench/cos/waterbutler/env/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/Users/whit537/workbench/cos/waterbutler/env/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "<string>", line 6, in _wrap_awaitable
TypeError: throw() takes 2 positional arguments but 4 were given

Hey @whit537,
Core python has a bug.  The workaround is to turn off debugging. See https://github.com/CenterForOpenScience/waterbutler#testing-configuration-optional

Thanks, @felliott.

That did the trick! :-)

And for my next trick ... https://github.com/CenterForOpenScience/osf.io/pull/5294#issuecomment-208551867 ...

I signed up a second contributor in a different browser session, and when I try to add the second contributor from the first session I get a 500 from the search endpoint.

[werkzeug]  INFO: 127.0.0.1 - - [11/Apr/2016 15:39:07] "GET /api/v1/user/search/?query=bar&page=0&_=1460407105699 HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/whit537/workbench/cos/osf.io/framework/routing/__init__.py", line 118, in wrapped
    data = fn(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/framework/auth/decorators.py", line 22, in wrapped
    return func(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/website/search/views.py", line 208, in search_contributor
    exclude=exclude, current_user=user)
  File "/Users/whit537/workbench/cos/osf.io/website/search/search.py", line 19, in wrapped
    return func(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/website/search/search.py", line 89, in search_contributor
    exclude=exclude, current_user=current_user)
  File "/Users/whit537/workbench/cos/osf.io/website/search/elastic_search.py", line 79, in wrapped
    return func(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/website/search/elastic_search.py", line 593, in search_contributor
    if current_user and current_user._id == user._id:
AttributeError: 'NoneType' object has no attribute '_id'

If I search for the full name of the second user I get an odd presentation in the add user dialog. The user is unregistered?

I believe I completed user registration successfully. The user has no unconfirmed emails and one primary email.

It's user that is None.
(Pdb) l
589         for doc in docs:
590             # TODO: use utils.serialize_user
591             user = User.load(doc['id'])
592
593             import pdb; pdb.set_trace()
594  ->         if current_user and current_user._id == user._id:
595                 n_projects_in_common = -1
596             elif current_user:
597                 n_projects_in_common = current_user.n_projects_in_common(user)
598             else:
599                 n_projects_in_common = 0
(Pdb) current_user
<User(u'foo@example.com') with id u'vy3kx'>
(Pdb) user
(Pdb)

I guess elasticsearch is confused? There is no user doc in Mongo with an _id of 4c79q.
(Pdb) doc['id']
u'4c79q'
(Pdb)

maybe try running inv rebuild_search

https://github.com/CenterForOpenScience/osf.io/pull/5294#issuecomment-208556593

That did it.

More unfrobulatin' ... this time ... elasticsearch?

Sorry, inv celery ...

I thought I ran inv requirements, trying that again ...

More research required ...
    warning: no previously-included files matching '*' found under directory 'tests'
  Could not find .egg-info directory in install record for modular-odm==0.3.0 from git+https://github.com/CenterForOpenScience/modular-odm.git@develop in ./env/lib/python2.7/site-packages (from -r /Users/whit537/workbench/cos/osf.io/requirements.txt (line 42))
  Found existing installation: mdx-del-ins 1.0
    Uninstalling mdx-del-ins:
      Successfully uninstalled mdx-del-ins
  Running setup.py install for mdx-del-ins
    warning: build_py: byte-compiling is disabled, skipping.

    warning: install_lib: byte-compiling is disabled, skipping.

  Could not find .egg-info directory in install record for mdx-del-ins==1.0 from git+git://github.com/aleray/mdx_del_ins.git in ./env/lib/python2.7/site-packages (from -r /Users/whit537/workbench/cos/osf.io/requirements.txt (line 44))
Successfully installed modular-odm mdx-del-ins
Cleaning up...

real    0m42.703s
user    0m13.943s
sys     0m1.661s
[osf.io]$ python
Python 2.7.9 (default, Jan  4 2015, 20:46:16)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrax
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pyrax
>>>

They pyrax import in that script is a year old.

I waxed my virtualenv and am rebuilding it ... maybe -ad to inv requirements?

😫
    sys.exit(main())
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/base.py", line 307, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
    return self.execute(command, argv)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/base.py", line 270, in __call__
    ret = self.run(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in init_worker
    self.import_default_modules()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 121, in import_default_modules
    tuple(maybe_list(self.app.conf.CELERY_INCLUDE))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 103, in import_task_module
    return self.import_from_cwd(module)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 112, in import_from_cwd
    package=package,
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 106, in import_module
    return importlib.import_module(module, package=package)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/whit537/workbench/cos/osf.io/scripts/osfstorage/usage_audit.py", line 18, in <module>
    import progressbar
ImportError: No module named progressbar
osf.io| $

Full traceback:
No handlers could be found for logger "framework.mongo.handlers"
Raven is not configured (logging is disabled). Please see the documentation for more information.
[elasticsearch]  WARNING: GET http://localhost:9200/_cluster/health?wait_for_status=yellow [status:N/A request:0.004s]
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 67, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/retry.py", line 222, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 155, in connect
    conn = self._new_conn()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
ProtocolError: ('Connection aborted.', error(61, 'Connection refused'))
[elasticsearch]  WARNING: GET http://localhost:9200/_cluster/health?wait_for_status=yellow [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 67, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/retry.py", line 222, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 155, in connect
    conn = self._new_conn()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
ProtocolError: ('Connection aborted.', error(61, 'Connection refused'))
[elasticsearch]  WARNING: GET http://localhost:9200/_cluster/health?wait_for_status=yellow [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 67, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/retry.py", line 222, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 155, in connect
    conn = self._new_conn()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
ProtocolError: ('Connection aborted.', error(61, 'Connection refused'))
[elasticsearch]  WARNING: GET http://localhost:9200/_cluster/health?wait_for_status=yellow [status:N/A request:0.001s]
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 67, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/retry.py", line 222, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connectionpool.py", line 349, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python2.7/httplib.py", line 1057, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 1097, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 1053, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 897, in _send_output
    self.send(msg)
  File "/usr/local/lib/python2.7/httplib.py", line 859, in send
    self.connect()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 155, in connect
    conn = self._new_conn()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/connection.py", line 134, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/urllib3/util/connection.py", line 88, in create_connection
    raise err
ProtocolError: ('Connection aborted.', error(61, 'Connection refused'))
[framework.sentry]  WARNING: Sentry called to log exception, but is not active
[framework.sentry]  WARNING: Sentry called to log message, but is not active: The SEARCH_ENGINE setting is set to 'elastic', but there was a problem starting the elasticsearch interface. Is elasticsearch running?
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/base.py", line 307, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
    return self.execute(command, argv)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/base.py", line 270, in __call__
    ret = self.run(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in init_worker
    self.import_default_modules()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 121, in import_default_modules
    tuple(maybe_list(self.app.conf.CELERY_INCLUDE))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 103, in import_task_module
    return self.import_from_cwd(module)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 112, in import_from_cwd
    package=package,
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 106, in import_module
    return importlib.import_module(module, package=package)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/whit537/workbench/cos/osf.io/scripts/osfstorage/usage_audit.py", line 18, in <module>
    import progressbar
ImportError: No module named progressbar

With elasticsearch running ...
No handlers could be found for logger "framework.mongo.handlers"
Raven is not configured (logging is disabled). Please see the documentation for more information.
Traceback (most recent call last):
  File "/Users/whit537/workbench/cos/osf.io/env/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/base.py", line 307, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 761, in handle_argv
    return self.execute(command, argv)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/celery.py", line 693, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
    return self(*args, **options)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/base.py", line 270, in __call__
    ret = self.run(*args, **kwargs)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
    state_db=self.node_format(state_db, hostname), **kwargs
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/worker/__init__.py", line 95, in __init__
    self.app.loader.init_worker()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 128, in init_worker
    self.import_default_modules()
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 121, in import_default_modules
    tuple(maybe_list(self.app.conf.CELERY_INCLUDE))
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 103, in import_task_module
    return self.import_from_cwd(module)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 112, in import_from_cwd
    package=package,
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/Users/whit537/workbench/cos/osf.io/env/lib/python2.7/site-packages/celery/loaders/base.py", line 106, in import_module
    return importlib.import_module(module, package=package)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/whit537/workbench/cos/osf.io/scripts/osfstorage/usage_audit.py", line 18, in <module>
    import progressbar
ImportError: No module named progressbar

(Pdb) l
114
115         def import_default_modules(self):
116             signals.import_modules.send(sender=self.app)
117             import pdb; pdb.set_trace()
118             return [
119  ->             self.import_task_module(m) for m in (
120                     tuple(self.builtin_modules) +
121                     tuple(maybe_list(self.app.conf.CELERY_IMPORTS)) +
122                     tuple(maybe_list(self.app.conf.CELERY_INCLUDE))
123                 )
124             ]
(Pdb) self.app.conf.CELERY_INCLUDE
()
(Pdb) self.app.conf.CELERY_IMPORTS
('framework.celery_tasks', 'framework.celery_tasks.signals', 'framework.email.tasks', 'framework.analytics.tasks', 'website.mailchimp_utils', 'website.notifications.tasks', 'website.archiver.tasks', 'website.search.search', 'api.caching.tasks', 'scripts.populate_new_and_noteworthy_projects', 'scripts.refresh_box_tokens', 'scripts.retract_registrations', 'scripts.embargo_registrations', 'scripts.approve_registrations', 'scripts.osfstorage.glacier_inventory', 'scripts.osfstorage.glacier_audit', 'scripts.triggered_mails', 'scripts.send_queued_mails', 'scripts.osfstorage.usage_audit', 'scripts.osfstorage.files_audit', 'scripts.analytics.tasks', 'scripts.analytics.upload')
(Pdb)

Looks like scripts.osfstorage.{usage,files}_audit have been in there for a couple months.

There's still no pyrax, either:
osf.io| $ python
Python 2.7.11 (default, Mar 18 2016, 15:44:22)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyrax
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pyrax
>>>

I have taken to https://www.flowdock.com/app/cos/osf-dev/threads/R_z9eQvG4v7r5X79r3_CVFZ152X.

Alright, workaround the celery import errors by overriding CELERY_IMPORTS in local.py.

What's next ... ?

15:21:47 apiserver.1     | Traceback (most recent call last):
15:21:47 apiserver.1     |   File "manage.py", line 9, in <module>
15:21:47 apiserver.1     |     init_app(set_backends=True, routes=False, attach_request_handlers=False)
15:21:47 apiserver.1     |   File "/Users/whit537/workbench/cos/osf.io/website/app.py", line 123, in init_app
15:21:47 apiserver.1     |     init_addons(settings, routes)
15:21:47 apiserver.1     |   File "/Users/whit537/workbench/cos/osf.io/website/app.py", line 47, in init_addons
15:21:47 apiserver.1     |     addon = init_addon(app, addon_name, routes=routes)
15:21:47 apiserver.1     |   File "/Users/whit537/workbench/cos/osf.io/website/addons/base/__init__.py", line 1035, in init_addon
15:21:47 apiserver.1     |     addon_module = importlib.import_module(import_path)
15:21:47 apiserver.1     |   File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
15:21:47 apiserver.1     |     __import__(name)
15:21:47 apiserver.1     |   File "/Users/whit537/workbench/cos/osf.io/website/addons/box/__init__.py", line 3, in <module>
15:21:47 apiserver.1     |     from website.addons.box import model, routes, views
15:21:47 apiserver.1     |   File "/Users/whit537/workbench/cos/osf.io/website/addons/box/model.py", line 4, in <module>
15:21:47 apiserver.1     |     from box import CredentialsV2, BoxClient
15:21:47 apiserver.1     | ImportError: No module named box

Alright, the learning make glorious here seems to be that the osf-dev channel on Flowdock is a veritable hotbed of hackery to get the Millenium Falcon that is osf.io to fly: check there first. 🌌


text:
Feature/fix addon icon

Purpose
Resize the Github and S3 icon so that it fits the size in the file widget. Solves #2931 and https://github.com/CenterForOpenScience/osf.io/issues/2940
Changes
After change:

@nanchen47 please resubmit this against the develop branch

@icereval done in https://github.com/CenterForOpenScience/osf.io/pull/2961


text:
Made text improvement in merging accounts modal [OSF-6721]

Purpose
The original confirmation message when adding an email to merge accounts was ambiguous so it should be edited to be more detailed.
Changes
Changed the text in accountSettings.js.

Side effects
N/A
Ticket
https://openscience.atlassian.net/browse/OSF-6721

Looks great. 🎉 🐧 🎉


text:
Add Institutions matching query to API search results [OSF-7370]

Purpose
Search results from the API were not equipped to deal with showing found Institutions matching the query search terms. Add the means for Institutions to show up in search results now without throwing a 500 error!
Changes

Update search views, urls, and serializers to also show Institution information in results
Add Institution results in the pagination view for search results
Add key to elastic_search.py's DOC_TYPE_TO_MODEL
Update elastic search when an institution is saved
Add tests

Side effects
None internally, but external uses of the search endpoint of the URL will now have to account for another key, and another type of serialized search result.
Ticket
https://openscience.atlassian.net/browse/OSF-7370

@erinspace QA says this doesn't seem to be happening any more. Can you see if other changes may have been made related to this? If need be, we can just merge the test if the test passes on current develop.

authors: brianjgeiger erinspace


text:
Classifai frontend on web browser did not reflect the latest changes

Current Version: Version 1.1.1
Description: A bug(#277) supposed to be fixed in (v1.1.1) were not reflected although the older version (v1.1.0) has been deleted and reinstalled with the latest one.
In the web browser,
when the saving format for Pascal VOC
image: 1.png
saved as 1_png.xml
but supposed to be saved as 1.xml

Solution: https://github.com/CertifaiAI/CertifAI-Knowledge-Base/wiki/Clear-Browser-Cache

authors: amalsyahmi-certifai codenamewei

repo: CertifaiAI/CertifAI-Knowledge-Base


text:
Copying and pasting dist folder's contents from Classifai_FrontEnd to classifai's webroot

This is an automatically created PR :robot::robot:

It will triggered once codes in dist folder from repo Classifai_FrontEnd has been pushed.
Files & sub-folder in dist folder of repo Classifai_FrontEnd will be copied into classifai's webroot

working on this

authors: codenamewei fariskamaludin-skymind

repo: CertifaiAI/classifai


text:
Exclude skirt vertices from bounds

Fixes #405

👍

authors: joseph-kaile kring

repo: CesiumGS/cesium-native


text:
Flip the range for fade-out

Previously, fading in tiles went from 0.0 --> 1.0 and fading out tiles went from 1.0 --> 0.0. When these values are used to threshold a dither texture in a renderer, this small discrepancy might cause floating point disagreement between fading in tiles and fading out tiles. So instead, we now want fading in and fading out tiles to have the exact same dither threshold (fade percentage) and simply invert the resulting sample if it is fading out. Since the dither texture is sampled in screen-space, this allows the fading out tiles and fading in tiles to blend together perfectly without appearing transparent.
@kring came up with the idea and it works quite nicely!

@kring Thanks for spotting that, I just simplified the logic that keeps fading out tiles from being unloaded. Now it just checks if a tile is in the fading-out list (actually an unordered_set) before deciding to unload it. The logic seems cleaner, the memory leak is fixed, and the extra std::unordered_set::find call looks fast enough in the profiler. What do you think?

That sounds good to me, thanks @nithinp7!


text:
Why does camera.flyto() not return a promise or take a callback?

Using viewer.flyTo(entity) returns a promise that gets resolved when the viewer's camera has zoomed to the entity. However, when using camera.flyto(options), there is no promise returned. Why is this, or is it just an error in the documentation?

@bluninja1234 camera.flyTo takes a complete function callback that is called when the flight is complete. It's a different API from the viewer.flyTo. See: https://cesium.com/docs/cesiumjs-ref-doc/Camera.html?classFilter=camera#flyTo.
For general questions please use the Cesium community forum: https://community.cesium.com/.

repo: CesiumGS/cesium


text:
Fixes BingMapsImageryProvider ineffective culture option problem.

Description
This PR fixes #11695
According to Cesium documentation (To construct a BingMapsImageryProvider, call BingMapsImageryProvider.fromUrl. Do not call the constructor directly.) I modified the code as below;
const viewer = new Cesium.Viewer("cesiumContainer");

const bing = await Cesium.BingMapsImageryProvider.fromUrl(
  "https://dev.virtualearth.net", {
    culture: "es-ES",
    key: "use-your-own-key",
    mapStyle: Cesium.BingMapsStyle.AERIAL_WITH_LABELS_ON_DEMAND,
});

viewer.scene.imageryLayers.removeAll();
viewer.scene.imageryLayers.addImageryProvider(bing);

When I evaluated the problem I saw that the problem was about metadata request and it did not tested well.  I just added culture parameter into metadata request which fixes the problem.

Issue number and link
Fixes #11695 11695
Author checklist

[x] I have submitted a Contributor License Agreement
[x] I have added my name to CONTRIBUTORS.md
[x] I have updated CHANGES.md with a short summary of my change
[x] I have added or updated unit tests to ensure consistent code coverage
[ ] I have update the inline documentation, and included code examples where relevant
[x] I have performed a self-review of my code

Thanks @ateshuseyin! Congrats on your first merged PR! 🎉

authors: ateshuseyin ggetz


text:
utils/common.ts/buildPosition Validation Error

https://github.com/CetusProtocol/cetus-clmm-sui-sdk/blob/3ff730264688ece50a6e31b499fa2d80ef9aab7c/src/utils/common.ts#L201
When I queried my Position, objects.error was not empty while object.data.content was correct.
The error was 'Could not find the referenced transaction events   TransactionEventsDigest(EoX7bG16cqGe6T6n7DbvJMpau9j6psCaoNDF4XQu3NqN)].'
I doubt why we need objects.error != null validation in buildPosition function.

Thanks for the feedback. Fixed in v5.1.2

repo: CetusProtocol/cetus-clmm-sui-sdk


text:
Google app

I rearranged things a little so we could keep the old app next to the newer App Engine version.

Looks good 👍

authors: SlowhandPVT ramjac


text:
Butterflynet nv22

Summary of changes

Changes introduced in this pull request:

to be rebased on main after NV22 for calibnet is merged

Reference issue to close (if applicable)

Closes
Other information and links

Change checklist

[ ] I have performed a self-review of my own code,
[ ] I have made corresponding changes to the documentation,
[ ] I have added tests that prove my fix is effective or that my feature works (if possible),
[ ] I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

This butterflynet will be reset once again for v1.26.0-rc1, holding for now - new assets will need to be uploaded and bundle + genesis updated.

repo: ChainSafe/forest


text:
refactor: class MonitoringService as close only

Motivation
Lodestar does not support start stop syntax in general. It's unnecessary for downstream classes to expose such behavior

Part of https://github.com/ChainSafe/lodestar/issues/4501

Description
MonitoringService as close only class

:tada: This PR is included in v1.9.0 :tada:

authors: dapplion wemeetagain

repo: ChainSafe/lodestar