text:
Using Geopandas with hvPlot doesn't work for a large dataset.

ALL software version info

MacOS - Sonoma 14.0
python = 3.11.8
notebook = 7.0.8
dask = 2023.11.0
datashader = 0.16.0
geopandas = 0.14.2
hvplot = 0.9.2
holoviews = 1.18.3
spatialpandas = 0.4.10
pyarrow = 14.0.2
pandas = 2.2.1
bokeh = 3.4.0

Description of expected behavior and the observed behavior
As part of the NumFOCUS SDG, I am modernizing the nyc_building example on the examples website to use the latest APIs. This example also involves using geopandas instead of spatialpandas to read the data stored as a parquet (.parq) file.
Switching from spatialpandas to geopandas for reading the file was not straightforward. The geometry column in the data file was not recognized by geopandas. To address this, I read the file using spatialpandas, converted it to a pandas DataFrame using .compute(), and then transformed it into a geopandas DataFrame. The geometry column was converted to a shapely object using a custom function. Finally, I saved this new DataFrame as a .parq file, which was then read directly using geopandas.
Code:
import geopandas as gpd
import spatialpandas as spd
import spatialpandas.io

ddf = spd.io.read_parquet_dask('./data/nyc_buildings.parq').persist().compute()

gdf = gpd.GeoDataFrame(ddf)

def convert_to_shapely(spgeom):
    try:
        return spgeom.to_shapely()
    except Exception as e:
        print(f"Error converting geometry: {e}")
        return None

gdf['geometry'] = gdf['geometry'].apply(convert_to_shapely)
gdf = gpd.GeoDataFrame(gdf, geometry='geometry')

gdf.to_parquet('new_nyc_buildings.parq')

gdf = gpd.read_parquet('new_nyc_buildings.parq')

gdf.head()

Despite these adjustments, plotting the new data file proved challenging. Using hvPlot.polygons to plot the entire dataset takes over 5 minutes to run with minimal code:
gdf.hvplot.polygons(tiles='CartoLight', rasterize=True)

Testing with a small sample of the data (gdf.head(1000)) produced results in a reasonable time, suggesting that hvPlot may have difficulties handling the large dataset (over 1 million rows).
Additionally, plotting with extra parameters, such as adding color mapping to the type categories in the data results in a Traceback error.
Complete, Minimal, Self-Contained Example Code that Reproduces the Issue
gdf.head(1000).hvplot.polygons(tiles='CartoLight', rasterize=True, c='type', cmap=color_key)

Stack traceback and/or browser JavaScript console output
WARNING:param.dynamic_operation: Callable raised "ValueError('input must be numeric')".
Invoked as dynamic_operation(height=300, scale=1.0, width=700, x_range=None, y_range=None)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/IPython/core/formatters.py:974, in MimeBundleFormatter.__call__(self, obj, include, exclude)
    971     method = get_real_method(obj, self.print_method)
    973     if method is not None:
--> 974         return method(include=include, exclude=exclude)
    975     return None
    976 else:

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/dimension.py:1286, in Dimensioned._repr_mimebundle_(self, include, exclude)
   1279 def _repr_mimebundle_(self, include=None, exclude=None):
   1280     """
   1281     Resolves the class hierarchy for the class rendering the
   1282     object using any display hooks registered on Store.display
   1283     hooks.  The output of all registered display_hooks is then
   1284     combined and returned.
   1285     """
-> 1286     return Store.render(self)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/options.py:1428, in Store.render(cls, obj)
   1426 data, metadata = {}, {}
   1427 for hook in hooks:
-> 1428     ret = hook(obj)
   1429     if ret is None:
   1430         continue

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:287, in pprint_display(obj)
    285 if not ip.display_formatter.formatters['text/plain'].pprint:
    286     return None
--> 287 return display(obj, raw_output=True)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:261, in display(obj, raw_output, **kwargs)
    259 elif isinstance(obj, (HoloMap, DynamicMap)):
    260     with option_state(obj):
--> 261         output = map_display(obj)
    262 elif isinstance(obj, Plot):
    263     output = render(obj)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:149, in display_hook.<locals>.wrapped(element)
    147 try:
    148     max_frames = OutputSettings.options['max_frames']
--> 149     mimebundle = fn(element, max_frames=max_frames)
    150     if mimebundle is None:
    151         return {}, {}

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:209, in map_display(vmap, max_frames)
    206     max_frame_warning(max_frames)
    207     return None
--> 209 return render(vmap)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/ipython/display_hooks.py:76, in render(obj, **kwargs)
     73 if renderer.fig == 'pdf':
     74     renderer = renderer.instance(fig='png')
---> 76 return renderer.components(obj, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/renderer.py:396, in Renderer.components(self, obj, fmt, comm, **kwargs)
    393 embed = (not (dynamic or streams or self.widget_mode == 'live') or config.embed)
    395 if embed or config.comms == 'default':
--> 396     return self._render_panel(plot, embed, comm)
    397 return self._render_ipywidget(plot)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/renderer.py:403, in Renderer._render_panel(self, plot, embed, comm)
    401 doc = Document()
    402 with config.set(embed=embed):
--> 403     model = plot.layout._render_model(doc, comm)
    404 if embed:
    405     return render_model(model, comm)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/viewable.py:736, in Viewable._render_model(self, doc, comm)
    734 if comm is None:
    735     comm = state._comm_manager.get_server_comm()
--> 736 model = self.get_root(doc, comm)
    738 if self._design and self._design.theme.bokeh_theme:
    739     doc.theme = self._design.theme.bokeh_theme

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/layout/base.py:320, in Panel.get_root(self, doc, comm, preprocess)
    316 def get_root(
    317     self, doc: Optional[Document] = None, comm: Optional[Comm] = None,
    318     preprocess: bool = True
    319 ) -> Model:
--> 320     root = super().get_root(doc, comm, preprocess)
    321     # ALERT: Find a better way to handle this
    322     if hasattr(root, 'styles') and 'overflow-x' in root.styles:

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/viewable.py:667, in Renderable.get_root(self, doc, comm, preprocess)
    665 wrapper = self._design._wrapper(self)
    666 if wrapper is self:
--> 667     root = self._get_model(doc, comm=comm)
    668     if preprocess:
    669         self._preprocess(root)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/layout/base.py:186, in Panel._get_model(self, doc, root, parent, comm)
    184 root = root or model
    185 self._models[root.ref['id']] = (model, parent)
--> 186 objects, _ = self._get_objects(model, [], doc, root, comm)
    187 props = self._get_properties(doc)
    188 props[self._property_mapping['objects']] = objects

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/layout/base.py:168, in Panel._get_objects(self, model, old_objects, doc, root, comm)
    166 else:
    167     try:
--> 168         child = pane._get_model(doc, root, model, comm)
    169     except RerenderError as e:
    170         if e.layout is not None and e.layout is not self:

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/pane/holoviews.py:429, in HoloViews._get_model(self, doc, root, parent, comm)
    427     plot = self.object
    428 else:
--> 429     plot = self._render(doc, comm, root)
    431 plot.pane = self
    432 backend = plot.renderer.backend

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/panel/pane/holoviews.py:525, in HoloViews._render(self, doc, comm, root)
    522     if comm:
    523         kwargs['comm'] = comm
--> 525 return renderer.get_plot(self.object, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/bokeh/renderer.py:68, in BokehRenderer.get_plot(self_or_cls, obj, doc, renderer, **kwargs)
     61 @bothmethod
     62 def get_plot(self_or_cls, obj, doc=None, renderer=None, **kwargs):
     63     """
     64     Given a HoloViews Viewable return a corresponding plot instance.
     65     Allows supplying a document attach the plot to, useful when
     66     combining the bokeh model with another plot.
     67     """
---> 68     plot = super().get_plot(obj, doc, renderer, **kwargs)
     69     if plot.document is None:
     70         plot.document = Document() if self_or_cls.notebook_context else curdoc()

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/renderer.py:217, in Renderer.get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs)
    214     raise SkipRendering(msg.format(dims=dims))
    216 # Initialize DynamicMaps with first data item
--> 217 initialize_dynamic(obj)
    219 if not renderer:
    220     renderer = self_or_cls

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/plotting/util.py:270, in initialize_dynamic(obj)
    268     continue
    269 if not len(dmap):
--> 270     dmap[dmap._initial_key()]

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:1217, in DynamicMap.__getitem__(self, key)
   1215 # Not a cross product and nothing cached so compute element.
   1216 if cache is not None: return cache
-> 1217 val = self._execute_callback(*tuple_key)
   1218 if data_slice:
   1219     val = self._dataslice(val, data_slice)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:984, in DynamicMap._execute_callback(self, *args)
    981     kwargs['_memoization_hash_'] = hash_items
    983 with dynamicmap_memoization(self.callback, self.streams):
--> 984     retval = self.callback(*args, **kwargs)
    985 return self._style(retval)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:552, in Callable.__call__(self, *args, **kwargs)
    550     return self.callable.rx.value
    551 elif not args and not kwargs and not any(kwarg_hash):
--> 552     return self.callable()
    553 inputs = [i for i in self.inputs if isinstance(i, DynamicMap)]
    554 streams = []

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1037, in Dynamic._dynamic_operation.<locals>.dynamic_operation(*key, **kwargs)
   1036 def dynamic_operation(*key, **kwargs):
-> 1037     key, obj = resolve(key, kwargs)
   1038     return apply(obj, *key, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1026, in Dynamic._dynamic_operation.<locals>.resolve(key, kwargs)
   1024 elif isinstance(map_obj, DynamicMap) and map_obj._posarg_keys and not key:
   1025     key = tuple(kwargs[k] for k in map_obj._posarg_keys)
-> 1026 return key, map_obj[key]

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:1217, in DynamicMap.__getitem__(self, key)
   1215 # Not a cross product and nothing cached so compute element.
   1216 if cache is not None: return cache
-> 1217 val = self._execute_callback(*tuple_key)
   1218 if data_slice:
   1219     val = self._dataslice(val, data_slice)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:984, in DynamicMap._execute_callback(self, *args)
    981     kwargs['_memoization_hash_'] = hash_items
    983 with dynamicmap_memoization(self.callback, self.streams):
--> 984     retval = self.callback(*args, **kwargs)
    985 return self._style(retval)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:552, in Callable.__call__(self, *args, **kwargs)
    550     return self.callable.rx.value
    551 elif not args and not kwargs and not any(kwarg_hash):
--> 552     return self.callable()
    553 inputs = [i for i in self.inputs if isinstance(i, DynamicMap)]
    554 streams = []

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/overlay.py:34, in Overlayable.__mul__.<locals>.dynamic_mul(*args, **kwargs)
     33 def dynamic_mul(*args, **kwargs):
---> 34     element = other[args]
     35     return self * element

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:1217, in DynamicMap.__getitem__(self, key)
   1215 # Not a cross product and nothing cached so compute element.
   1216 if cache is not None: return cache
-> 1217 val = self._execute_callback(*tuple_key)
   1218 if data_slice:
   1219     val = self._dataslice(val, data_slice)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:984, in DynamicMap._execute_callback(self, *args)
    981     kwargs['_memoization_hash_'] = hash_items
    983 with dynamicmap_memoization(self.callback, self.streams):
--> 984     retval = self.callback(*args, **kwargs)
    985 return self._style(retval)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/spaces.py:582, in Callable.__call__(self, *args, **kwargs)
    579     args, kwargs = (), dict(pos_kwargs, **kwargs)
    581 try:
--> 582     ret = self.callable(*args, **kwargs)
    583 except KeyError:
    584     # KeyError is caught separately because it is used to signal
    585     # invalid keys on DynamicMap and should not warn
    586     raise

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1038, in Dynamic._dynamic_operation.<locals>.dynamic_operation(*key, **kwargs)
   1036 def dynamic_operation(*key, **kwargs):
   1037     key, obj = resolve(key, kwargs)
-> 1038     return apply(obj, *key, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1030, in Dynamic._dynamic_operation.<locals>.apply(element, *key, **kwargs)
   1028 def apply(element, *key, **kwargs):
   1029     kwargs = dict(util.resolve_dependent_kwargs(self.p.kwargs), **kwargs)
-> 1030     processed = self._process(element, key, kwargs)
   1031     if (self.p.link_dataset and isinstance(element, Dataset) and
   1032         isinstance(processed, Dataset) and processed._dataset is None):
   1033         processed._dataset = element.dataset

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/util/__init__.py:1012, in Dynamic._process(self, element, key, kwargs)
   1010 elif isinstance(self.p.operation, Operation):
   1011     kwargs = {k: v for k, v in kwargs.items() if k in self.p.operation.param}
-> 1012     return self.p.operation.process_element(element, key, **kwargs)
   1013 else:
   1014     return self.p.operation(element, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:194, in Operation.process_element(self, element, key, **params)
    191 else:
    192     self.p = param.ParamOverrides(self, params,
    193                                   allow_extra_keywords=self._allow_extra_keywords)
--> 194 return self._apply(element, key)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:141, in Operation._apply(self, element, key)
    139     if not in_method:
    140         element._in_method = True
--> 141 ret = self._process(element, key)
    142 if hasattr(element, '_in_method') and not in_method:
    143     element._in_method = in_method

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/operation/datashader.py:1506, in rasterize._process(self, element, key)
   1503     op = transform.instance(**{k:v for k,v in extended_kws.items()
   1504                                if k in transform.param})
   1505     op._precomputed = self._precomputed
-> 1506     element = element.map(op, predicate)
   1507     self._precomputed = op._precomputed
   1509 unused_params = list(all_supplied_kws - all_allowed_kws)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/data/__init__.py:196, in PipelineMeta.pipelined.<locals>.pipelined_fn(*args, **kwargs)
    193     inst._in_method = True
    195 try:
--> 196     result = method_fn(*args, **kwargs)
    197     if PipelineMeta.disable:
    198         return result

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/data/__init__.py:1213, in Dataset.map(self, *args, **kwargs)
   1211 @wraps(LabelledData.map)
   1212 def map(self, *args, **kwargs):
-> 1213     return super().map(*args, **kwargs)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/dimension.py:697, in LabelledData.map(self, map_fn, specs, clone)
    695     return deep_mapped
    696 else:
--> 697     return map_fn(self) if applies else self

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:214, in Operation.__call__(self, element, **kwargs)
    210         return element.clone([(k, self._apply(el, key=k))
    211                               for k, el in element.items()])
    212     elif ((self._per_element and isinstance(element, Element)) or
    213           (not self._per_element and isinstance(element, ViewableElement))):
--> 214         return self._apply(element)
    215 elif 'streams' not in kwargs:
    216     kwargs['streams'] = self.p.streams

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/core/operation.py:141, in Operation._apply(self, element, key)
    139     if not in_method:
    140         element._in_method = True
--> 141 ret = self._process(element, key)
    142 if hasattr(element, '_in_method') and not in_method:
    143     element._in_method = in_method

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/holoviews/operation/datashader.py:1391, in geometry_rasterize._process(self, element, key)
   1389 agg_kwargs = dict(geometry=col, agg=agg_fn)
   1390 if isinstance(element, Polygons):
-> 1391     agg = cvs.polygons(data, **agg_kwargs)
   1392 elif isinstance(element, Path):
   1393     if self.p.line_width and ds_version >= Version('0.14.0'):

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/datashader/core.py:782, in Canvas.polygons(self, source, geometry, agg)
    780 if agg is None:
    781     agg = any_rdn()
--> 782 return bypixel(source, self, glyph, agg)

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/datashader/core.py:1329, in bypixel(source, canvas, glyph, agg, antialias)
   1327 schema = dshape.measure
   1328 glyph.validate(schema)
-> 1329 agg.validate(schema)
   1330 canvas.validate()
   1332 # All-NaN objects (e.g. chunks of arrays with no data) are valid in Datashader

File ~/Documents/development/holoviz-topics-examples/nyc_buildings/envs/default/lib/python3.11/site-packages/datashader/reductions.py:355, in Reduction.validate(self, in_dshape)
    353     raise ValueError("specified column not found")
    354 if not isnumeric(in_dshape.measure[self.column]):
--> 355     raise ValueError("input must be numeric")

ValueError: input must be numeric
:DynamicMap   []

Screenshots or screencasts of the bug in action

@Azaya89 thanks for this very nice bug report!
Could you also report the timings for the equivalent of this line gdf.hvplot.polygons(tiles='CartoLight', rasterize=True) but with SpatialPandas? I would like to know how much slower things got.

Here it is.

Oh I'm pretty sure it definitely takes more for the plot to render than 542 us. Can you get an estimate of the real time it takes for the plot to render? Also, I notice another difference is that the hvPlot call uses rasterize while in the last snippet it uses datashade.

Oh I'm pretty sure it definitely takes more for the plot to render than 542 us. Can you get an estimate of the real time it takes for the plot to render?

How do you mean? is it different from using timeit on the plot objects?

How do you mean? is it different from using timeit on the plot objects?

Yes, the render time is different. Feel free to just give a rough estimate of when the plot is done rendering on screen

Yes, the render time is different. Feel free to just give a rough estimate of when the plot is done rendering on screen

OK, I timed it myself and it took ~5 secs to run this cell and render the plots: tiles * shaded * legend * hover

Can you try to run this with HoloViews 1.19.0a2?

Can you try to run this with HoloViews 1.19.0a2?

I did and it ran significantly faster. However, adding the other parameters still caused the same error as before.

Coming back to the issue reported with:
gdf.head(1000).hvplot.polygons(tiles='CartoLight', rasterize=True, c='type', cmap=color_key)

@Azaya89 you haven't shared how color_key is derived in your example. Focusing on this bit of code from the NYC Buildings example:
cats = list(ddf.type.value_counts().compute().iloc[:10].index.values) + ['unknown']
ddf['type'] = ddf.type.replace({None: 'unknown'})
ddf = ddf[ddf.type.isin(cats)]
ddf['type'] = ddf['type'].astype('category').cat.as_known()

There, ddf is a spatialpandas.geodataframe.GeoDataFrame object. In your updated code, gdf is a geopandas.GeoDataFrame object. The latter has a type property that returns the geometry type of each geometry in the GeoSeries.
https://github.com/geopandas/geopandas/blob/747d66ee6fcf00b819c08f11ecded53736c4652b/geopandas/base.py#L233-L236
Therefore to access the column data you need to use __getitem__ / the [] syntax with gdf['type'].
Then, to get the plot displayed I had to add aggregator='count_cat' to c='type'. Somehow, I would have expected setting these two parameters to be identical to setting by='type' but it didn't work. Something to discuss I guess.
So here's the full code:
Code

import colorcet as cc
import datashader as ds
import geopandas as gpd
import hvplot.pandas

gdf = gpd.read_parquet('new_nyc_buildings.parq')
cats = list(gdf['type'].value_counts().iloc[:10].index.values) + ['unknown']
gdf['type'] = gdf['type'].replace({None: 'unknown'})
gdf = gdf[gdf['type'].isin(cats)]
colors    = cc.glasbey_bw_minc_20_maxl_70
color_key = {cat: tuple(int(e*255.) for e in colors[i]) for i, cat in enumerate(cats)}
gdf.hvplot.polygons(
    tiles='CartoLight', data_aspect=1,
    datashade=True, aggregator=ds.by('type'), cmap=color_key
)

Alternatively, this would also work:
gdf.hvplot.polygons(
    tiles='CartoLight', data_aspect=1,
    datashade=True, aggregator='count_cat', c='type', cmap=color_key
)

In the NYC Buildings example another aggregator is used with ds.by('type', ds.any()). If I try to use that instead of ds.by('type') I get ValueError: input must be categorical. Turning the type column into a categorical one fixes that, and the plot looks much closer to the plot currently displayed in the example:
Code

gdf['type'] = gdf['type'].astype('category')
gdf.hvplot.polygons(
    tiles='CartoLight', data_aspect=1,
    datashade=True, aggregator=ds.by('type', ds.any()), cmap=color_key
)

Finally, setting rasterize=True instead of datashade=True generates a plot that is far from what we'd expect:

And indeed, taking another simpler example and using HoloViews only, I can see that the output of a rasterize operation is not the expected one.
Code

import geopandas as gpd
import geodatasets
import datashader as ds
import holoviews as hv
import spatialpandas as spd
from holoviews.operation.datashader import datashade, rasterize

hv.extension('bokeh')

path = geodatasets.get_path("geoda.nyc_neighborhoods")
nyc = gpd.read_file(path)
nyc['boroname'] = nyc['boroname'].astype('category')  # required
spd_nyc = spd.GeoDataFrame(nyc)

polys = hv.Polygons(spd_nyc, vdims='boroname')
shaded = datashade(polys, aggregator=ds.by('boroname', ds.any()))
rasterized = rasterize(polys, aggregator=ds.by('boroname', ds.any()))

polys + shaded + rasterized

@Azaya89 you haven't shared how color_key is derived in your example.

Here's how it was constructed:
colors    = cc.glasbey_bw_minc_20_maxl_70
color_key = {cat: tuple(int(e*255.) for e in colors[i]) for i, cat in enumerate(cats)}

The same as the one you shared.
Thank you for this. I have incorporated it into my PR now, although I'm now curious why rasterize and datashade are giving very different outputs...

@Azaya89 you haven't shared how color_key is derived in your example.

Here's how it was constructed:
colors    = cc.glasbey_bw_minc_20_maxl_70
color_key = {cat: tuple(int(e*255.) for e in colors[i]) for i, cat in enumerate(cats)}

The same as the one you shared.
Thank you for this. I have incorporated it into my PR now, although I'm now curious why rasterize and datashade are giving very different outputs...

@Azaya89 what was important is how you computed cats.

although I'm now curious why rasterize and datashade are giving very different outputs...

This is a bug in HoloViews I think, I need to open a bug report.

@Azaya89 what was important is how you computed cats.

OK. Here:
cats = list(gdf['type'].value_counts().iloc[:10].index) + ['unknown']

authors: Azaya89 droumis hoxbro maximlt

repo: holoviz/hvplot


text:
MFA raw html for setup and login dialogs

Home Assistant release with the issue: 0.87.0

Last working Home Assistant release (if known): N/A
Operating environment (Hass.io/Docker/Windows/etc.): N/A

Component/platform: N/A

Description of problem:
I'm trying to make an MFA module for Duo Security.  This requires adding the below code to the setup and login dialogs.  Currently only a text based title and description are allowed.
        <script src='/Duo-Web-v2.js'></script>
        <iframe id="duo_iframe"
                title="Two-Factor Authentication"
                frameborder="0"
                data-host="%(host)s"
                data-sig-request="%(sig_request)s"
                >
        </iframe>

Additional information:
Thanks!

I am going to close it since it is not a bug. This place is only for bug report.
To answer your question: this is something you need to change in front end ha-form, you need allow it support load additional js file and raw html. However I have concern to include such support. The way we manage the localization, will let support raw html became security issue.

source: gharchive/issue

authors: JumpMaster awarecan

repo: home-assistant/home-assistant

license_type: permissive

license_source: github-api


text:
error while adding Lightify LED strip

Home Assistant release (hass --version):
0.48.0
Python release (python3 --version):
Python 3.4.3
Component/platform:
zha
Description of problem:
After calling the permit service, the strips flash 3 times as expected, but error below is output and no device is added.
Expected:
Device added to zha
Problem-relevant configuration.yaml entries and steps to reproduce:
zha:
  usb_path: /dev/ttyUSB1
  database_path: /home/ha/.homeassistant/zigbee.db

Have device powered on & unpaired
Issue permit service call

Traceback (if applicable):
2017-07-12 23:39:23 WARNING (MainThread) [bellows.zigbee.zdo] [0x19e9:zdo] Unsupported ZDO request 0x0036
2017-07-12 23:39:29 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification
2017-07-12 23:39:29 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-07-12 23:39:29 WARNING (MainThread) [bellows.zigbee.device] [0x19e9] Message on unknown endpoint 1
2017-07-12 23:39:29 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification
2017-07-12 23:39:53 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/ha/.homeassistant/deps/bellows/zigbee/device.py", line 38, in initialize
    epr = yield from self.zdo.request(0x0005, self._nwk)
  File "/home/ha/.homeassistant/deps/bellows/zigbee/application.py", line 259, in request
    v = yield from asyncio.wait_for(reply_fut, timeout)
  File "/usr/lib/python3.4/asyncio/tasks.py", line 381, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError

Additional info:
I found issue #7473 appears to be similar, and with a different device--below is output from trying to follow debug instructions in this message
$ bellows devices -D `pwd`/zigbee.db
Device:
  NWK: 0x848a
  IEEE: 7c:b0:3e:aa:00:a4:d2:d1
  Endpoints:
$ bellows zdo -D `pwd`/zigbee.db 7c:b0:3e:aa:00:a4:d2:d1 endpoints
[3]
$ bellows zdo -D `pwd`/zigbee.db 7c:b0:3e:aa:00:a4:d2:d1 get_endpoint 3
Device 7c:b0:3e:aa:00:a4:d2:d1 has no endpoint 3

It seems that it lists that it has endpoint 3, but no endpoint 3 exists?
Here is the product listed on zigbee.org as zha. I have SKU 73661

This is a duplicate of #7473

authors: kvermilion rcloran


text:
Added fan support.  Reduced number of calls to the thermostat to a mi…

Description:
See issue #8404 for most of the details.  The radiotherm underlying python module that the radiotherm climate component is using has a few issues.  Every state element access is implemented as a separate url call which the thermostat can't handle and there are no time outs supported.  I've reduced these as much as possible for now and reduce the number of url calls from ~7 per update() to 2-3 per update().   Given the constraint to use the existing library, a number of improvements that make this code work better (reducing calls even further to a single request, proper time outs, retry ability) can't be done.  The things I could fix include:

fixed multiple web calls in update() to only use a minimum of calls for a large speed increase
fixed constructor to defer web calls until update() to speed up start up times
added correct temperature resolution
added fan attributes so the web front end now allows for manual fan control.

There is still a lot of work to do for this component but there have been several requests for this fix by users so I'm putting this in as is for now.  Future work (which I'll open as a separate issue when I have time to finish it) includes:

asyncio support
automatic retry when set functions fail (which can happen sometimes)
fix auto mode which isn't really working right now
add proper hold mode as a switch that can be controlled from the web front end
full set of unit tests w/ a mock thermostat (there are 0 test cases for this right now)
move these fixes down to the 3rd party Python library.  The current package owner hasn't responded to issue posts since June so it's not clear whether changes can be merged down or not at this point.  Once I have a full set of fixes, I'll try and figure out how to get those propagated down or if the library should be forked.

NOTE: this was a previous PR that I closed because I somehow got my git branch into a strange state and could never fix it so I deleted the PR and the branch and started over.  There were a number of comments about using the existing 3rd party radiotherm library instead of moving that functionality into the HA component.  I disagreed with those since the only functionality being provided was a single web get/set call which tons of other components already do and they seem fine.  However I've changed the code to use the existing library as efficiently as it can given that constraint.
Related issue (if applicable): fixes #8404
Checklist:
If the code communicates with devices, web services, or third-party tools:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass

If the code does not interact with devices:

[x] Local tests with tox run successfully. Your PR cannot be merged unless tests pass

If there are any reviewers out there, can you give me some advice on the pep8 error?  The only solution I can find that makes it happy makes the code look bad:
    vol.Optional(CONF_AWAY_TEMPERATURE_COOL,
                 default=DEFAULT_AWAY_TEMPERATURE_COOL):
    vol.All(vol.Coerce(float), round_temp),

I tried that one - I tried every indentation combination I could think of and never got one that worked and didn't look terrible.  I think it's a bug in flake8...

Usually pylint gives you more helpful comments regarding indentation (i.e., tells you how much to indent). However, according to the travis build, pylint currently fails because of PRECISION_HALVES not being found. You should fix that (and other pylint errors), and then look at what pylint has to say to that indentation. Also, that's probably what causes the other travis build to fail.

@tinloaf PRECISION_HALVES is broken because someone just moved it out of climate and didn't import it back.  And pylint fails because of the mod that @pvizeli just pushed (i.e. not me).   I'd love to fix it so if you can suggest a fix that passes flake8 (besides the one I already submitted), that would be great.  pylint reports several indentations are OK (including the one that pvizeli just submitted) but flake8 fails them.
I could use some git help as well - the last time I tried to merge changes to my branch before pushing, I screwed everything up (I ended up with all my changes plus all the new dev changes).   What's the correct way to pull down the current HA dev branch into my branch so I can fix the broken climate init changes?

Assuming that you have the hass repository (not your fork) added as a remote named upstream, these are the steps to rebase (not merge!) your fork onto the new dev branch:
(assuming you are in your repository and on the branch representing this PR)
git fetch upstream
git rebase upstream/dev
(… Conflicts may happen here. Fix them and finish the rebase. …)
git push --force

If someone has moved PRECISION_HALVES to somewhere else, you probably need to change your patch to import it from that new place.

OK - I think this one is (finally) ready to go.  I still don't particularly like the indentation required by flake8 on lines 69 and 72 but what's there is the only option I can find that passes.

Out of curiosity, because I'm currently trying to get a picture of the "state machines" behind as many climate devices as possible: Can you explain what the difference between self._tstate and self._tmode is? Is _tmode what the user tells the device to do, and _tstate what the device actually currently does? (E.g.: The user can tell the device to heat, but the device can still be idle if the room is currently warmer than the target temperature.)

@tinloaf Yes - that's correct.  Modes are the commands to the device to set possible activity (cool, heat, off, fan on).  States are what the device is doing at a particular time (cooling, heating, fan blowing).  So if the mode is cool, the state will be idle until the thermostat decides the temperature has changed enough to turn on the AC at which point the state will go idle->cool.  Once the correct temperature is reached, the state will go cool->idle; but the mode has been cool for the whole time.

Thanks!

source: gharchive/pull-request

authors: MartinHjelmare TD22057 tinloaf


text:
Enable auth by default 🙈

Description:
This enabled the new authentication system by default. This will enable both the Home Assistant local auth provider and the Legacy API password auth provider. New users will be routed via an onboarding flow on initial boot to create a new user. This new user will be owner and is able to add more users.
Breaking change: It is no longer possible to use Home Assistant without authentication. To allow users to log in without entering a username or password, you can configure the trusted networks auth provider. If you were using another application to provide authentication, like NGINX, configure it to set the header x-ha-access: <API PASSWORD> on each request. In the future, this header will be replaced with a header containing a long lived access token (#15195).
(I don't use these platforms like NGINX, would be nice if people would contribute config examples)
Related issue (if applicable): #15703
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

To allow users to log in with entering a username or password, you can use the trusted networks auth provider.

Sounds not right.
To allow users to log in without entering password, you can use the trusted networks auth provider.
Example configuration for trusted networks auth provider:
homeassistant:
  auth_providers:
    - type: homeassistant
    - type: legacy_api_password
    - type: trusted_networks

My bad, that didn't sound right. I am going to add some docs right now . Will add the config example there.

Another thought, if user doesn't set api_password, we should either not load leagcy_api_password auth provider, or let it failed quietly.

@awarecan good point, I will open a PR.

I have a better idea, cooking up a PR :)

Legacy API password provider will now abort any login flow if API password has not been set: https://github.com/home-assistant/home-assistant/pull/16127

So the new user will see two auth providers, and the legacy one is broken out-of-box?

They will see an abort message saying that it won't work because there is no HTTP password. They already created an owner via onboarding, so they can log in with that.

Why we need legacy_api_password in default config? If they not working out of the box we need it not inside default config?

I think we should auto load legacy_api_pasword depends on if http.api_password exist

I like that idea.

Updated.

Pascal you were right, we should avoid to get people to configure auth themselves. The default settings should suffice. This is a good solution.

authors: awarecan balloob pvizeli


text:
Add doorsense sensor for August 3rd Gen Smart Lock Pro

Description:
The August 3rd Gen Smart Lock Pro comes with a DoorSense sensor to identify if the door is open or closed.
I added a new binary sensor within the august component to expose the sensor to HASS. A binary sensor will be created for each lock that has the doorsense sensor available as well. During initialization it checks if the status returned is not Unknown and only if it is Unknown it will add the device, otherwise the binary sensor will not be added.
I further also added debug logging in various areas of the coding for both locks and doorbells.
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Checklist:

[X ] The code change is tested and works locally.
[X ] Local tests pass with tox. Your PR cannot be merged unless tests pass

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[X] New dependencies are only imported inside functions that use them ([example][ex-import]).

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Hi @ehendrix23,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

Thank you for all the comments in the review. I went through and believe to have addressed all of them. I also checked and after changes it is still working locally on my HASS. :-)

This PR is borked with commits that have nothing to do with it. Recommend you redo.

This PR is borked with commits that have nothing to do with it. Recommend you redo.

WOW, I can see that. Any idea what I might have done wrong to caused this?
I know I did a rebase of my branch which I thought all came back clean.  Want to ensure that this does not happen again. :-)

There are full instructions here for rebasing: https://developers.home-assistant.io/docs/en/development_catching_up.html Most importantly, after you rebase, you need to force push to your branch on GitHub, and be sure NOT to pull first as git suggests.
You can fix this branch by running git rebase -i upstream/dev for interactive mode, then delete any commits that aren't related to your change. Once you're done force push up to GitHub.

Going back through my command history, looks like I indeed did a pull as git suggested. :-(
I still have the update files. Might it be better/easier instead to close this PR instead, delete the branch and then create a new branch and a new PR?

Yeah, that works

authors: armills bachya ehendrix23 homeassistant


text:
Update to homekit==0.12.0

Description:
This bumps the underlying homekit_controller code to use homekit==0.12.0. It uses the new Controller API in that release. This starts to remove assumptions about IP based homekit accessories in anticipation of BLE support in a future release.
Related issue (if applicable): fixes #
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[x] New dependencies have been added to the REQUIREMENTS variable (example).
[x] New dependencies are only imported inside functions that use them (example).
[x] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Hi @Jc2k,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

For the future, please don't squash commits after review has started to make it easier for readers to track changes.

authors: Jc2k MartinHjelmare homeassistant


text:
Solaredge new sensors

Description:
Adding additional monitored_conditions to the SolarEdge platform.
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#8213
Example entry for configuration.yaml (if applicable):
sensor:
 - platform: solaredge
    api_key: YOUR_API_KEY
    site_id: YOUR_SITE_ID
    monitored_conditions:
      - current_power
      - last_day_data
      - last_month_data
      - last_year_data
      - life_time_data
      # new conditions
      - site_details
      - inventory_inverters
      - inventory_meters
      - inventory_sensors
      - inventory_gateways
      - inventory_batteries

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.io

Alright, that didn't go as planned. Will close this one and open a new pull request.


text:
Move homekit_controllers HKDevice into connection module

Description:
This is a mechanical lift + shift that makes the review of my homekit controller config entry branch a bit easier, in and of itself it doesn't have any impact, add features, change behaviour or anything, it just makes an upcoming PR easier to review.
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

Please rebase and solve the merge conflict.


text:
Add support for sensor state STATE_UNAVAILABLE

Fixed integration with ESPhome, which caused an error if ESPhome did not update fast enough on startup
Description:
There is a race condition on the state of the sensor when starting home assistant, in case of ESPHome this results in an error along the lines of: ValueError: could not convert string to [insert casted value]: 'unavailable'.
This fix updates the plant sensor state to STATE_PROBLEM and sets the attribute of the STATE_UNAVAILABLE sensor to STATE_UNAVAILABLE.
We should not ignore STATE_UNAVAILABLE for a sensor related to a plant, as this may cause the user to think that everything is fine, while the plant is dying, which is why I chose to set the plant state to STATE_PROBLEM in this case.
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

Thanks!

authors: balloob cnrd


text:
user agent header

Description:
Add  user agent to header
Related issue (if applicable): fixes #5154

🐬

Nice 🐬

authors: Danielhiversen balloob


text:
No intent match for what’s with right single quote vs single quote

This is similar to #2086 but for the question prefix rather than the entity name.
When I ask Assist:

whats the temp

... or:

what's the temp

... on my PC I get the response I expect, however in the iOS companion app when I ask:

What’s the temp

I get a no_intent_match error. It looks like this is because of the iOS auto-correct inserted right single quotation mark (U+2019) vs single-quote/apostrophe (U+0027)
In the the following line:
https://github.com/home-assistant/intents/blob/40384abc2f67cb233cb058684a159bd0d57255ea/sentences/en/_common.yaml#L292
Would it be advisable to change it to this?
what_is: "(what's|what’s|whats|what is)"

Nice catch! The proper solution is to consider the character that iOS uses as punctuation so it would get stripped and ignored during recognition.
I've created a PR for this https://github.com/home-assistant/hassil/pull/137

That sounds like a much better approach! Thanks.

authors: paulhomes tetele

repo: home-assistant/intents


text:
[enhancement]: Add support for E330 professional cameras

Describe the solution you'd like
Hello,
I have E330 wired 24/7 4K cameras associated with my HomeBase 3, however it seems the plugin does not support these cameras. Please consider supporting them in future plugin releases. Thank you.

Yes please. I tried hours until find out, not really supported.
Any ETA? thx

it's been back ported in 2.3.6 hksv

it's been back ported in 2.3.6 hksv

Hi thank you. I installed that Version and now it at least appeasrs in Homebridge and Homekit after manuel insert. But nothing working. I don't have Settings at homebridge (as with HB2 + Cams) and i cant see Picture on Homekit. It sais "no answer, this camera is not responding" (self translated)..
And its only the S220 SoloCam which is showing up (but not work as described).
The 24/7 Camera E330 it not working at all. Shows not up under Devices in Homebridge. But in Config its there, but nearly no Options. I also set unbridge for both cams.

Thanks @lenusch  for connecting me to this thread.
I tried this version (Beta 2.3.7) and no luck with HB3 and/or E330 Cam  When I go to add it manually, it seems at though the HB3 is bridge. I don't think I can add the E330 Cam on its own.
I tried in the Plug In Settings using Unbridge Camera and then add it manually using homebridge instance (e.g. 123-45-678).  It adds the bridge still and then sees the camera as device.  I assume that is incorrect and it should just add camera. Not completely sure how the unbridge/bridge mode works.  But, when I added the camera, it just shows up in Home app as unsupported.

Yes its still not working

happy to see this being worked on, thanks folks.

I have still problems. I even can't watch both cams on Homekit, its kinda disturbing each other. And then i think to watch that nothing works anymore.

Wanted to check in to see if this was still being worked on. I have the same issue and would love support for these cameras.

ITS WORKIN FOR ME :-)
https://github.com/homebridge-eufy-security/plugin/issues/470#issuecomment-2074225255

authors: DrWhy44 captcook32 ibmaster lenoxys lenusch tranceme

repo: homebridge-eufy-security/plugin


text:
Missing /api/server/qrcode.svg after upgrade to v4.50.2

Feature Description
After upgrading homebridge-config-ui-x from version v4.41.2 to v4.50.2 the qr-code link is not available anymore.
Has the link changed or is this feature missing ?
I still can get the pairing code number from /api/server/pairing
But don´t know how to convert that into a qr-code link.
Homebridge system is running on
OS: Linux 4.14.52 arm ( Ubuntu 16.04.7 LTS 4.14.52 )
Node.js v16.19.0
Thanks for your attention.

Have you tried to 'Reset Layout' ?

repo: homebridge/homebridge-config-ui-x

license: MIT


text:
v9.2.2: fix: fix the options for the sass-loader

What does this PR do?
Follow up for #86 ... I forgot to move outputStyle inside sassOptions.
How should it be tested manually?
Build a target with SASS/SCSS and...
yarn test
# or
npm test

Pull Request Test Coverage Report for Build 319

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 316:
0.0%

Covered Lines:
654

Relevant Lines:
654

💛  - Coveralls


text:
Always overwrite the output path for JS files when the target is a library

What does this PR do?
When the targetFinder detects a library target, it will automatically set the output path for JS files (both for production and development) to [target-name].js.
The reason of the change is that libraries use the main key of the package.json and it should be a static path.
This change is breaking because if a library target didn't have its path overwritten, this change will updated to the new format.
How should it be tested manually?
Create a library target and use projext info targets to see the output settings, the path for JS files should be [target-name].js for both production and development.
And of course...
npm test
# or
yarn test

Pull Request Test Coverage Report for Build 142

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 140:
0.0%

Covered Lines:
1475

Relevant Lines:
1475

💛  - Coveralls


text:
refactor(project): use wootils' ObjectUtils instead of extend and the Utils deprecated methods

What does this PR do?
Wootils' ObjectUtils is already a wrapper on top of extend, so on this PR I replaced every call to extend with ObjectUtils.merge and ObjectUtils.copy.
I also removed the deprecated methods from Utils:

getPropertyWithPath was removed in favor of ObjectUtils.get.
setPropertyWithPath was removed in favor of ObjectUtils.set.
deletePropertyWithPath was removed in favor of ObjectUtils.delete.

How should it be tested manually?
This shouldn't change anything in your project if you weren't using the deprecated methods.
And don't forget...
yarn test
# or
npm test

Pull Request Test Coverage Report for Build 298

18 of 18 (100.0%)  changed or added relevant lines in 6 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 295:
0.0%

Covered Lines:
2169

Relevant Lines:
2169

💛  - Coveralls


text:
SQL Injection

SQL Injection
SQL 인젝션이란?
SQL 인젝션은 웹 사이트의 보안상 허점을 이용해 특정 SQL 쿼리문을 전송해 공격자가 원하는 데이터베이스의 중요한 정보를 가져오는 해킹 기법이다.
대부분 클라이언트가 입력한 데이터를 제대로 필터링하지 못하는 경우에 발생한다.
공격의 쉬운 난이도에 비해 피해가 상당하기 때문에 보안 위협 1순위로 불릴만큼 중요한 기법이다.
SQL 인젝션의 종류와 공격 방법
Error based SQL Injection

논리적 에러를 이용한 SQL 인젝션

가장 많이 쓰이고, 대중적인 공격 기법이다.
SELECT * FROM Users WHERE id = 'INPUT1' AND password = 'INPUT2'

SELECT * FROM Users WHERE id = ' ' OR 1=1 -- 'AND password = 'INPUT2'
--> SELECT * FROM Users

첫번째 쿼리문은 일반적으로 로그인 시 많이 사용되는 SQL 구문이다.
해당 구문에서 입력값에 대한 검증이 없음을 확인하고, 악의적인 사용자가 임의의 SQL 구문을 주입했다.
주인된 내용은 ' OR 1=1-- 으로 WHERE 절에 있는 ' 를 닫아주기 위한 ' 와 OR 1=1 라는 구문을 이용해 WHERE절을 모두 참으로 만들고, —를 넣어주어서 뒤의 구문을 주석 처리를 해준 것이다.
매우 간단한 구문이지만 결론적으로 Users 테이블에 있는 모든 정보를 조회하게 됨으로써 가장 먼저 만들어진 계정으로 로그인에 성공하게 된다.
보통 관리자 계정을 맨 처음으로 만들기 때문에 관리자 계정으로 로그인할 수 있게 되며, 악의적인 사용자는 관리자의 권한을 이용해 또 다른 2차피해를 발생시킬 수 있게 된다.
UNION based SQL Injection

UNION 명령어를 이용한 SQL 인젝션

UNION은 여러개의 SQL문을 합쳐 하나의 SQL 문으로 만들어주는 방법이다.
정상적인 쿼리문에 Union 키워드를 사용해 인젝션에 성공하면, 원하는 쿼리문을 실행할 수 잇게 된다.
Union Injection을 성공하기 위해서는 두 테이블의 컬럼 수가 같아야 하고, 데이터 형이 같아야 한다는 두 가지의 조건이 있다.
SELECT * FROM Board WHERE title like '%INPUT%' OR contents like '%INPUT%'

SELECT * FROM Board WHERE title LIKE '% '
UNION SELECT null, id, password FROM Users
-- %' AND contents '% UNION SELECT null, id, password FROM Users -- %'

첫번째 쿼리문은 Board라는 테이블에서 게시글을 검색하는 쿼리문이다.
입력값을 title과 content 컬럼의 데이터랑 비교한 뒤 비슷한 글자가 있는 게시글을 출력한다.
여기서 입력값으로 Union 키워드와 함께 컬럼 수를 맞춰서 SELECT 구문을 넣어주게 되면 두 쿼리문이 합쳐져서 하나의 테이블로 보여지게 된다.
현재 인젝션한 구문은 사용자의 id와 password를 요청하는 쿼리문이다.
인젝션이 성공하게 되면, 사용자의 개인정보가 게시글과 함께 화면에 보여지게 된다.
password를 그대로 DB에 저장하지는 않겠지만 인젝션이 가능하다는 점에서 이미 그 이상의 보안위험에 노출되어 있다.
이 공격도 역시 입력값에 대한 검증이 없기 때문에 발생한다.
Blind SQL Injection

Boolean based Blind SQL 인젝션

Blind SQL Injection은 데이터베이스로부터 특정한 값이나 데이터를 전달받지 않고 단순히 참과 거짓의 정보만 알 수 있을 때 사용한다.
로그인 폼에 SQL Injection이 가능하다고 가정했을 때, 서버가 응답하는 로그인 성공과 로그인 실패 메시지를 이용해 DB의 테이블 정보 등을 추출한다.
즉, 쿼리를 삽입하였을 때 쿼리의 참과 거짓에 대한 반응을 구분할 수 있을 때 사용되는 기술이다.
SELECT * FROM Users WHERE id = 'INPUT1' ANS password = 'INPUT2'

SELECT * FROM Users WHERE id = ' abc123' and
ASCII(SUBSTR((SELECT name FROM information_schema.tables
WHERE table_type='base table' limit 0,1),1,1)) > 100 --

위 쿼리는 Blind Injection을 이용해 DB의 테이블명을 알아내는 방법이다.
인젝션이 가능한 로그인 폼을 통해 악의적인 사용자는 임의로 가입한 abc123이라는 아이디와 함께 뒤의 구문을 주입한다.
해당 구문은 MySQL에서 테이블 명을 조회하는 구문으로 limit 키워드를 통해 하나의 테이블만 조회하고, SUBSTR 함수로 첫 글자만, 그리고 마지막으로 ASCII를 통해 ascii 값으로 변환한다.
만약 조회되는 테이블 명이 Users 라면 'U' 가 조회될 것이고, 뒤의 100이라는 숫자와 값을 비교하게 된다.
거짓이면 로그인 실패, 참이 될 때까지 뒤의 100이라는 숫자를 변경해가면서 비교한다.
참이 나오는 것을 통해 단기간 내에 테이블 명을 알아낼 수 있다.

Time based SQL

어떤 경우에는 응답의 결과가 항상 동일해 해당 결과만으로 참과 거짓을 판별할 수 없는 경우가 있을 수 있다.
이런 경우 시간을 지연시키는 쿼리를 주입(Injection)하여 응답 시간의 차이로 참과 거짓 여부를 판별할 수 있다.
사용되는 함수는 MySQL 기준 SLEEP 과 BENCHMARK 이다.
SELECT * FROM Users WHERE id = 'INPUT1' AND password = 'INPUT2'

SELECT * FROM Users WHERE id = ' abc123' OR
(LENGTH(DATABASE())=1 AND SLEEP(2))
-- ' AND password = 'INPUT2'

위 쿼리문은 Time based SQL Injection을 사용해 현재 사용하고 있는 데이터베이스의 길이를 알아내는 방법이다.
로그인 폼에 주입되었으며 임의로 abc123이라는 계정을 생성한다.
악의적인 사용자가 해당 구문을 주입하면, LENGH 함수는 문자열의 길이를 반환하고, DATABASE 함수는 데이터베이스의 이름을 반환한다.
주입된 구문에서, LENGTH(DATABASE()) = 1 가 참이면 SLEEP(2) 가 동작하고, 거짓이면 동작하지 않는다.
이를 통해 숫자 1 부분을 조작해 데이터베이스의 길이를 알 수 있다.
만약 SLEEP 이라는 단어가 치환 처리되었다면, 다른 방법으로 BENCHMARK 나 WAIT 함수를 사용할 수 있다.
Stored Procedure SQL Injection

저장된 프로시저에서의 SQL Injection

저장 프로시저(Stored Procedure)은 일련의 쿼리들을 모아 하나의 함수처럼 사용하기 위한 것이다.
공격에 사용되는 대표적인 저장 프로시저는 MS-SQL에 있는 xp_cmdshell로 윈도우 명령어를 사용할 수 있게 된다.
단, 공격자가 시스템 권한을 획득해야 하므로 공격난이도가 높으나 공격에 성공한다면 서버에 직접적인 피해를 입힐 수 있는 공격이다.
Mass SQL Injection

다량의 SQL Injection 공격

기존 SQL Injection과 달리 한번의 공격으로 다량의 데이터베이스가 조작되어 큰 피해를 입히는 것을 의미한다.
보통 MS-SQL을 사용하는 ASP 기반 웹 애플리케이션에서 많이 사용되며, 쿼리문은 HEX 인코딩 방식으로 인코딩해 공격한다.
보통 데이터베이스 값을 변조해 데이터베이스에 악성스크립트를 삽입하고, 사용자들이 변조된 사이트에 접속 시 좀비 PC로 감염되게 한다.
이렇게 감염된 좀비 PC들은 DDos 공격에 사용된다.
SQL 인젝션 대응방안
입력 값에 대한 검증
SQL Injection에 사용되는 기법과 키워드는 엄청 많다.
따라서 사용자의 입력 값에 대한 검증이 필요하다.
서버 단에서 화이트리스트를 기반으로 검증해야 하며, 블랙리스트를 기반으로 검증하게 되면 차단리스트를 많이 등록해야 하고, 하나라도 빠지면 공격에 성공하게 된다.
공백으로 치환하는 방법도 많이 쓰이는데, 이 방법도 취약한 방법이다.
예시로 SESELECTLECT 라고 입력 시 중간의 SELECT가 공백으로 치환되면 SELECT 라는 키워드가 완성된다.
공백 대신 공격 키워드와를 의미 없는 단어로 치환되어야 한다.
Prepared Statement 구문 사용
Prepared Statement 구문을 사용하게 되면, 사용자의 입력 값이 데이터베이스의 파라미터로 들어가기 전 DBMS가 미리 컴파일하여 실행하지 않고 대기한다.
그 후 사용자의 입력 값을 문자열로 인식하여 공격쿼리가 들어간다고 해도, 사용자의 입력은 이미 의미 없는 단순 문자열이기 때문에 전체 쿼리문도 공격자의 의도대로 작동하지 않는다.
Error Message 노출 금지
공격자가 SQL Injection을 수행하기 위해서는 데이터베이스의 정보(테이블, 컬럼 명)가 필요하다.
데이터베이스 에러 발생 시 따로 처리하지 않는다면, 에러가 발생한 쿼리문과 함께 에러에 관한 내용을 반환해주기 때문에 공격자에게는 힌트가 될 수 있다.
따라서 데이터베이스에 대한 오류가 발생할 때는 사용자에게 보여줄 수 있는 페이지를 제작하거나 메시지박스를 띄우도록 해야 한다.
웹 방화벽 사용
웹 공격 방어에 특화되어있는 웹 방화벽을 사용하는 것도 하나의 방법이다.
웹 방화벽은 소프트웨어형, 하드웨어형, 프록시형 세 가지 종류로 나눌 수 있다.
소프트웨어형은 서버 내에 직접 설치하는 방법이고, 하드웨어형은 네트워크 상에서 서버 앞 단에 직접 하드웨어 장비로 구성하는 것이며 마지막으로 프록시형은 DNS 서버 주소를 웹 방화벽으로 바꾸고 서버로 가는 트래픽이 웹 방화벽을 먼저 거치도록 하는 방법이다.

SQL 인젝션을 설명해보세요.
SQL 인젝션을 방어할 수 있는 기법은 어떤 것이 있나요?

repo: hongcheol/CS-study


text:
Support SDK retry in case of rate limit

Problem
There are customers with enough resources to run into Hookdeck API rate limit (240rqm) which blocks their Terraform flow. We should use Fern's SDK retries with backoff feature to make sure we're not a blocking the Terraform run here.
API
I suggest we allow 2 ways to configure the retry max attempts:
1: via environment variable HOOKDECK_SDK_MAX_ATTEMPTS
2: via option
provider "hookdeck" {
  api_key = var.hookdeck_api_key
  sdk_max_attempts = 5
}

Blocked by https://github.com/hookdeck/terraform-provider-hookdeck/issues/109

v0.6.0 of the Go SDK has been released with the latest version of the Fern Go generator.
We next need to complete #109, which will unblock this and a number of other features/PRs.

authors: alexluong leggetter

repo: hookdeck/terraform-provider-hookdeck


text:
Teams is not working

I'm experiencing following issue related to Teams

Dialog box that popups when tapping Invite button shows an infinite loader.
Plus Button (+ New) button under team collection is disabled.
Add new Button on the same page shows tool tip with message:
You do not have edit access to these collections.

This incident has been resolved. Team features are now up.

authors: lalkrishna liyasthomas

repo: hoppscotch/docs


text:
Persisting and recovering capacity scheduler

Pushing the work done by Sri on implementing fail over for the capacity scheduler.
Resolve #52

Is this working correctly?

There are tests failing on jenkins that were not failing on my machine I need to investigate them.

authors: berthoug jimdowling

repo: hopshadoop/hops

license_source: bigquery


text:
When would the code for GPT-J-6B be released?

Thanks a lot for your work on compression on LLMs, and looking forward for the code for GPT-J. When would it be available for it? It would be a great help for my experiment.

Thanks for your advice. We have no plan for GPT-J-6B so far.

authors: VainF mumuyeye

repo: horseee/LLM-Pruner


text:
CB-18850 adding vertical scale right to the ui right response

See detailed description in the commit message.

LGTM

authors: doktoric gregito

repo: hortonworks/cloudbreak


text:
Implemented product identifier using dxp-components.

Related Issues

Closes #180
Short Description and Why It's Useful

Added product identifier selection card in the settings page and made changes wherever the identifier is displayed in the app.
Screenshots of Visual Changes before/after (If There Are Any)

IMPORTANT NOTICE - Remember to update CHANGELOG.md with description of your change
Contribution and Currently Important Rules Acceptance

[x] I read and followed contribution rules

Closing as will be handled using reusable components.

repo: hotwax/preorder


text:
turbo isn't working with adsense

I didn't find the root cause for this issue, but if adsense on page, links aren't working with turbo.
(every link will trigger full page reload)
this start happen after 7.1 version (on 7.1 works ok, on 7.2-beta-1 or higher the problem exists).
if adsense isn't presenting, everything works as expected.

update - found the issue, the function doesNotTargetIFrame(anchor)
is getting as parameter empty string (because the link target is empty)
this cause the selector to get all iframes in, and then it's not handled by turbo.

the fix is simple,
just add this as another check.
if someone can create a PR, it would be great.
function doesNotTargetIFrame(anchor: HTMLAnchorElement): boolean {
if (anchor.target === "")
return true;

@eladmarg thank you for opening this issue.
Making an implementation change seems straightforward, but making that change without a test case wouldn't prevent future changes to re-introduce a regression.
Could you share some more information like an error stack trace or a JSFiddle with a minimum snippet of code to reproduce the behavior so that we can also include test coverage in the fix?

sure,
very simple,
when you queryselector by empty name, you can get iframes if exists on the page (document.getElementsByTagName(""))
so simply add to the page an iframe of google ads
this for instance:

<iframe frameborder="0" src="https://5642dc6e614dd1d0e97dec6070467bae.safeframe.googlesyndication.com/safeframe/1-0-40/html/container.html" id="google_ads_iframe_/4744526/ads_cat_top_desktop_0" title="3rd party ad content" name="" scrolling="no" marginwidth="0" marginheight="0" width="728" height="90" data-is-safeframe="true" sandbox="allow-forms allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-top-navigation-by-user-activation" role="region" aria-label="Advertisement" tabindex="0" data-google-container-id="1" style="border: 0px; vertical-align: bottom;" data-load-complete="true">

then you'll see that its get selected when you click on another anchor element.

Thank you for sharing that snippet!
I've extracted the  element and broken it across several lines to make reading it easier:
<iframe
  frameborder="0"
  src="https://5642dc6e614dd1d0e97dec6070467bae.safeframe.googlesyndication.com/safeframe/1-0-40/html/container.html"
  id="google_ads_iframe_/4744526/ads_cat_top_desktop_0"
  title="3rd party ad content"
  name=""
  scrolling="no"
  marginwidth="0"
  marginheight="0"
  width="728" height="90"
  data-is-safeframe="true"
  sandbox="allow-forms allow-popups allow-popups-to-escape-sandbox allow-same-origin allow-scripts allow-top-navigation-by-user-activation"
  role="region"
  aria-label="Advertisement"
  tabindex="0"
  data-google-container-id="1"
  style="border: 0px; vertical-align: bottom;"
  data-load-complete="true">

Note the name="" attribute. The presence of that attribute is at the root of the issue.
I'll incorporate some tests into a pull request.

fixed.

authors: eladmarg seanpdoyle

repo: hotwired/turbo


text:
Getting this error on Start.bat launch.

Start_img2img.bat launches fine.
Is my Python broken?

There seems to be a typo in the controlnet.json-dist file : missing , after "override_settings_restore_afterwards": "true" line 20.
I will push a fix soon. You will have to correct or delete your local controlnet.json file afterwards.

This should be alright now, be sure to remove the faulty controlnet.json file.

repo: houseofsecrets/SdPaint


text:
drm: output list of deleted files option

admins would like a list of the files that drm deletes... just in case a user asks about it later.

We could do this with a new --output option in drm.
There are two lists I can think of that might be of interest.
1) files that were deleted
2) files that were not deleted, because they didn't exist or there was some other I/O error

Perhaps one could break list 2 into sublists based on the type of error?

Working this issue in PR https://github.com/hpc/mpifileutils/pull/288

authors: adammoody gonsie

repo: hpc/mpifileutils


text:
Follow ups to #152 "Report user position to server"

This adds the code missing from #152 that is not required for the feature to work (you may notice some FIXMEs left in the code). This PR (we hope) won't change the API introduced in #152 on the User model, so dependent issues can start working with it in parallel.

[x] The UI to distinguish between showing the location on the map and sharing with the server
[x] A mechanism that actually deletes old user positions / deletes them when the site is left

[x] delete old entries too

[x] Handle failed requests (for example, when unauthorized) don't even send requests when not signed in

I would be lying if I told you that the two switches are visually appealing, but since they need to be reworked to look on mobile, I will defer that task for whoever does that :)


text:
Move 'change password' form to sidebar

Currently the 'change password' form (for non-openID users) is part of the the normal 'edit' view of user profiles.
It can be moved to the sidebar to unclutter the main content.

As a first step, moved to sidebar in 'edit' view.

license: mit


text:
setting up framework

Gems:

[x] rspec
[x] rubocop
[x] Scaffold
[x] Pry
[x] cancancan
[x] Twitter Bootstrap
[x] Better Errors (+ adv)
[x] and a couple of more..

Infrastructure:

[ ] Heroku
[ ] TravisCI
[ ] CodeClimate
[ ] Erbit
[ ] New Relic for Heroku

If possible, critical errors and warnings of the infrastructure services should be automatically reported to the "notifications" channel of the student slack.

will be done! :+1:

authors: derari jaSunny


text:
关于训练第11个epoch

用非常非常小的数据集想测试下，选用了20epoch，从10个后开始微调backbone，训练时候，前十个没问题，可以正常，但是到第11个报错如下：
Traceback (most recent call last):
File "C:/Users/Administrator/Desktop/python/siamban-master/tools/train.py", line 312, in
main()
File "C:/Users/Administrator/Desktop/python/siamban-master/tools/train.py", line 307, in main
train(train_loader, dist_model, optimizer, lr_scheduler, tb_writer)
File "C:/Users/Administrator/Desktop/python/siamban-master/tools/train.py", line 203, in train
outputs = model(data)  # 此处进行前向传播进行计算.
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\utils\distributed.py", line 43, in forward
return self.module(*args, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\models\model_builder.py", line 76, in forward
xf = self.backbone(search)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\models\backbone\resnet_atrous.py", line 192, in forward
p3 = self.layer3(p2)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
input = module(input)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\Desktop\python\siamban-master\siamban\models\backbone\resnet_atrous.py", line 104, in forward
residual = self.downsample(x)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
input = module(input)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
return self._conv_forward(input, self.weight)
File "C:\Users\Administrator\anaconda3\envs\py37\lib\site-packages\torch\nn\modules\conv.py", line 420, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
请问这是什么问题呢？

Fine-tuning the backbone requires a large amount of video memory. You can check if there are other processes occupying the video memory.

authors: BaBaWen zeduchen

repo: hqucv/siamban


text:
建议

导航页可不可以去掉登陆直接浏览？
设置一些必须登陆才能看到的网页？
增加分组设置
当切换到内网模式时只显示有内网地址的网址会不会更好。

准备这次做，3.准备以后研究一个方案后做，4.暂不做，用户需求不同，也不能确保在内网模式不访问外网地址

repo: hslr-s/sun-panel


text:
ion2 calender not showing ios-arrow-forword & ios-arrow-back with latest ionic 5

Ionic version:  (check one with "x")
[ ] 2.x
[ ] 3.x
[ ] 4.x
Ion2-calendar mode:  (check one with "x")
[ ] components mode
[ ] modal mode
I'm submitting a ...  (check one with "x")
[ ] bug report
[ ] feature request
[ ] help me
Current behavior:

Expected behavior:

Steps to reproduce:

Related code:
insert short code snippets here

Other information:
Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
insert the output from ionic info here

I have the same problem.

Same here

Same here, any idea?

i have same problem, which is the path for add this????

People who are facing the issue can use my fork from the PR. I have done the fix

@itsLucario Hi. Please write the command which I can install your fork by npm.

@itsLucario I tried to use your fork by following command
npm i https://github.com/itsLucario/ion2-calendar.git
but while importing it say module not found....
Can you please help me for the proper command to configure.

I have published my fork to @lucario/ion5-calendar
You can use it with npm i @lucario/ion5-calendar

Hello Hsuan,
Can you please let me know how can I set the default date in while using
range.
Also y does it give value undefined when i wont select range but only from
date?
Regards,
Sagar J Hedau
On Sat, Oct 31, 2020, 10:37 PM Hsuan Lee notifications@github.com wrote:

Closed #312 https://github.com/hsuanxyz/ion2-calendar/issues/312 via
#313 https://github.com/hsuanxyz/ion2-calendar/pull/313.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/hsuanxyz/ion2-calendar/issues/312#event-3944315225,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALBOT5MTVOAXI6N47R6L2SLSNQ75VANCNFSM4QDSHKYA
.

authors: AleksandrCherniavskiy DalterioRaffaele Metallink SagarJHedau dromazmj itsLucario marrahi neerajrk2509


text:
Правило: heading-level

Краткое описание
Validates heading level increments and order. Headings must start at h1 and can only increase one level at a time.
Плохие примеры

Heading 1

Subheading

Хорошие примеры

Heading 1

Subheading

Поле со свободным описанием.
в параметр передать список sectioningRoots и allowMultipleH1

Проверить в валидаторе

С включённым outline показывает ошибки

authors: balesniy nikolai-shabalin

repo: htmlacademy/linthtml-rules-htmlacademy


text:
http-server does not get PORT from process.env by default

My command: http-server dist -a mysite.com -S -C cert.pem -K key.pem -o
My .env file: PORT=8081
Expected: The url should be: https://mysite.com:8081
The result: https://mysite.com:8080
Environment:

http-server: 0.12.3
Node version: 14.4.0
Platform: Windows 10

Other information: http-server does not get PORT from process.env.PORT by default

It seems that http-server does not read .env file.  I try using dotenv module and adding dotenv to http-server.js in order to load .env file and it works correctly, http-server can get the PORT from .env file

That's correct that http-server does not read any .env file, if you export the .env into your environment, does it pick up the port correctly?

Closing for now, please re-open if this turns out to be a bug in http-server

so http-server.js won't read the .env file

Correct.

my docker-entrypoint.sh has this command - http-server dist -p 3000 --cors -c1 -g

You're passing the port 3000 explicitly, so that should ignore not just the .env file but also any PORT environment variable set inside the container.

so http-server.js won't read the .env file

Correct.

my docker-entrypoint.sh has this command - http-server dist -p 3000 --cors -c1 -g

You're passing the port 3000 explicitly, so that should ignore not just the .env file but also any PORT environment variable set inside the container.

Great, thanks for the confirmation. Just for clarification , am trying to pass sensitive data and not the port. I’ll use the node command to run the server then.

authors: duc-dev gursahibsahni thornjad zbynek

repo: http-party/http-server


text:
Update fs2-reactive-streams to 2.5.1

Updates co.fs2:fs2-reactive-streams from 2.5.0 to 2.5.1.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "co.fs2", artifactId = "fs2-reactive-streams" } ]

labels: library-update, semver-patch

Superseded by #114.


text:
Add REPORT as valid HTTP verb

Resolves #387.

#387 issue and this PR actually brings us to the need of rethinking how we validate HTTP verbs. REPORT used in RFC 6352 extends RFC 3253. And taking a glance look at last one it's obvious that our current way will fail on VERSION-CONTROL method.

@tarcieri you might wanna about some of the changes made with this pr ;D

@tarcieri I tend to merge this PR as failures on jruby now are pretty specific to travis and bundler. Somhow it simply fails to run on jruby on travis - but runs locally.

Looks good... well aside from the JRuby failure

@tarcieri master is also failing on jruby ;)) btw

@ixti I'm asking on the Bundler Slack about it.
Maybe we should pin to a known-working Bundler.

This appears to be the issue: https://github.com/bundler/bundler/issues/5340

This will be fixed in Bundler 1.14.2

Yeah. Thanks for digging into that. I have locked down bundler on travis for now to the one that worked last time for us ;))

Okay. I have no clue how to lock bundler...

Let me take a crack at just fixing that issue...

After some time and couple of rounds when I thought I finally got insane - I have managed travis to run with bundler / rubygems known to work :D

Looks good!

authors: ixti tarcieri

repo: httprb/http


text:
Problem loading MDO with Protégé

Hi @huanyu-li ,
I normally use Protégé to visualize owl files, and I had problems to load all the modules from the mdo-full.owl
Adding a catalog-v001.xml file like this:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<catalog prefer="public" xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
  <group id="Folder Repository, directory=, recursive=true, Auto-Update=false, version=2" prefer="public" xml:base="">
    <uri name="https://w3id.org/mdo/calculation/1.0"             uri="./mdo-calculation.owl"/>
    <uri name="https://w3id.org/mdo/core/1.0"                    uri="./mdo-core.owl"/>
    <uri name="https://w3id.org/mdo/provenance/1.0"              uri="./mdo-provenance.owl"/>
    <uri name="https://w3id.org/mdo/structure/1.0"               uri="./mdo-structure.owl"/>
    <uri name="https://w3id.org/mdo/full/1.0"                    uri="./mdo-full.owl"/>
  </group>
</catalog>

to the root folder made the trick. Might be you think it is useful to add such a file to the repository. Protégé users will be very happy!
Thanks

Hi, The catalog file is uploaded.

authors: huanyu-li lopeztarifa

repo: huanyu-li/Materials-Design-Ontology


text:
SMARTS0.4.17 does not support py3.8 very well

BUG REPORT
High Level Description

SMARTS0.4.17 does not support py3.8 very well. When make sanity-test, it can't run.
[]
When in the py3.7, it works. Unfortunately, but I found some mistakes.I'm not sure if it's because of the Python version.
（1）When I try to run Multi-agent Experiments in ULTRA，error is as follows:

SMARTS version
[0.4.17]
System information
[ubuntu18.04]

@JenishPatel99 @christianjans Can you please have a look at the ULTRA error.
I will in the meantime have a look at the make sanity-test error for py3.8. Thanks

Hi @Yuanzhuo-Liu, regarding the ULTRA error, was this all that was outputted? Or was there more error text above that observations = env.reset() line?
If there was more error text, please do share, I am currently unsure of exactly what it might be with just the image given.
However, if there was no more error text above the observations = env.reset() line, then this error is just due to the GRPC dependency's version. Essentially, the error is caused by the GRPC version we are using, and upgrading GRPC will solve the issue. See #704, #735, and #747 for more context on this issue.

Hi @Yuanzhuo-Liu:
I setup my environment in python3.8 and followed the setup instructions and did not see the make sanity-test error you have mentioned above. For me the make sanity-test passed without any error and I also was able to run some experiments.
This is how I installed python3.8 on the remote server:
sudo apt-get install python3.8
sudo apt install python3.8-distutils
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.8 get-pip.py
python3.8 -m pip install --upgrade pip setuptools wheel
sudo apt install python3.8-venv python3.8-dev
This is how I setup SMARTS in python3.8 virtual environment:
bash utils/setup/install_deps.sh
source /etc/profile.d/sumo.sh
python 3.8 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
python3.8 -m pip install -e .[train]
python3.8 -m pip install -e .[test]
make sanity-test
The output I got:

Hello, Yuanzhuo. As for the ULTRA issue, would you mind telling us if you are running the experiments with or without headless mode? Also, would you be able to also share the script which you used to run the experiments.
Thank you

Hi @Yuanzhuo-Liu:
I setup my environment in python3.8 and followed the setup instructions and did not see the make sanity-test error you have mentioned above. For me the make sanity-test passed without any error and I also was able to run some experiments.
This is how I installed python3.8 on the remote server:
sudo apt-get install python3.8
sudo apt install python3.8-distutils
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.8 get-pip.py
python3.8 -m pip install --upgrade pip setuptools wheel
sudo apt install python3.8-venv python3.8-dev
This is how I setup SMARTS in python3.8 virtual environment:
bash utils/setup/install_deps.sh
source /etc/profile.d/sumo.sh
python 3.8 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
python3.8 -m pip install -e .[train]
python3.8 -m pip install -e .[test]
make sanity-test
The output I get:

NOTE: Ignore the failing test test_smarts_doesnt_leak_after_memory_reset. This is due to some Xlib: extension "XFree86-DGA" missing for ShowBase Instance of Panda-3D module which is specific to the path issue on remote server I used and not due to SMARTS dependencies. (Source: The same test passes on Github's CI/CD and on other servers).

Thanks! The reason why my installation failed is that I used Conda instead of Venv.I'm so sorry...

Hello, @Yuanzhuo-Liu. As for the ULTRA issue, would you mind telling us if you are running the experiments with or without headless mode? Also, would you be able to also share the script which you used to run the experiments.
Thank you

Thanks!
The experiment I ran followed that 'SMARTS/ultra/doc/multiagent.md'.

Hi! @RutvikGupta ,Thank you very much for your patience. I'm really sorry that I still want to ask about the memory growth problem(#990 ray.memory_monitor.RayOutOfMemoryError). I still encountered this problem in 0.4.17, and you said you would solve it before. How does it work in this latest release?
Thank you

@Yuanzhuo-Liu, Thanks for confirming that your issue is resolved. I will have a look at Issue #990.

It seems that the experiments have been ran without headless mode, this means that we are expecting an output on Envision. My initial prediction is that the error has been caused by a runtime error. Usually, if a runtime error (or end of program) occurs, the termination signal error (i.e. the SIGTERM signal) will be outputted, if we don't use headless mode. However, it is a naïve error output.
There are a couple of things we can do:
If you are running ULTRA via github (i.e. if you git clone <smarts_repo>) you can run our custom tests (used for CI). To run a test, follow these commands:

cd <path_to_smarts_repo>/ultra
pytest -v tests/test_train.py
The test_train.py will execute unit tests for single and multi agent(s).

If all of the test passes then we can assume that there is a possibility that there has been a runtime error. According to your error output, there was an StopIteration exception right before the termination signal error. The StopIteration exception indicates that the next() method goes beyond the iterated items. In terms of ULTRA, the items that are iterated are the scenario paths.
If the tests fails, please let us know which of the tests failed and a error log of the failed tests will be great. However, if the tests pass then we can move below to the troubleshooting points.
To troubleshoot this issue, we can start by printing the scenarios object that is passed on to parent constructor, as show below:
https://github.com/huawei-noah/SMARTS/blob/6c3a39cef28916bbeddaea31135216aa1dc23210/ultra/ultra/env/ultra_env.py#L72-L79
Printing the scenarios object will either produce an empty list or a populated list with scenario paths. My assumption is that the list is empty thus when we cannot iterate through the scenarios object and it will end up throwing an exception
Note: If you are developing from ultra-develop branch then you might see some new changes, that is you haven't yet rebased to the latest commit. However, printing the scenarios object before the function call should work fine
The causes of an empty scenarios object are:

The scenarios has not been generated yet
The scenarios are generated, but not in the correct root directory (i.e. the directory where the scenarios are stored)
The scenarios are generated and in the correct root directory, but the name of the scenarios are incorrect. Note: We recently modified the way that the name of scenarios are constructed in #982

Sorry, for the long reply 😄, I hope these steps do help you with resolving the issue, if not, you can always contact us (@JenishPatel99 @christianjans) for further assistance.

It seems that the experiments have been ran without headless mode, this means that we are expecting an output on Envision. My initial prediction is that the error has been caused by a runtime error. Usually, if a runtime error (or end of program) occurs, the termination signal error (i.e. the SIGTERM signal) will be outputted, if we don't use headless mode. However, it is a naïve error output.
There are a couple of things we can do:
If you are running ULTRA via github (i.e. if you git clone <smarts_repo>) you can run our custom tests (used for CI). To run a test, follow these commands:

cd <path_to_smarts_repo>/ultra
pytest -v tests/test_train.py
The test_train.py will execute unit tests for single and multi agent(s).

If all of the test passes then we can assume that there is a possibility that there has been a runtime error. According to your error output, there was an StopIteration exception right before the termination signal error. The StopIteration exception indicates that the next() method goes beyond the iterated items. In terms of ULTRA, the items that are iterated are the scenario paths.
If the tests fails, please let us know which of the tests failed and a error log of the failed tests will be great. However, if the tests pass then we can move below to the troubleshooting points.
To troubleshoot this issue, we can start by printing the scenarios object that is passed on to parent constructor, as show below:
https://github.com/huawei-noah/SMARTS/blob/6c3a39cef28916bbeddaea31135216aa1dc23210/ultra/ultra/env/ultra_env.py#L72-L79
Printing the scenarios object will either produce an empty list or a populated list with scenario paths. My assumption is that the list is empty thus when we cannot iterate through the scenarios object and it will end up throwing an exception
Note: If you are developing from ultra-develop branch then you might see some new changes, that is you haven't yet rebased to the latest commit. However, printing the scenarios object before the function call should work fine
The causes of an empty scenarios object are:

The scenarios has not been generated yet
The scenarios are generated, but not in the correct root directory (i.e. the directory where the scenarios are stored)
The scenarios are generated and in the correct root directory, but the name of the scenarios are incorrect. Note: We recently modified the way that the name of scenarios are constructed in ULTRA Add task id to generated scenario directories #982

Sorry, for the long reply , I hope these steps do help you with resolving the issue, if not, you can always contact us (@JenishPatel99 @christianjans) for further assistance.

I sincerely thank you for such a long answer. I'm sorry that I didn't see it. I will try to solve the problem according to your suggestions.
Thank you again for

I also encountered @Yuanzhuo-Liu's first issue using python3.8 in a venv. It looks like TraverseGoal in scenario.py is just missing a @dataclass(frozen=True). I will make a PR.

authors: JenishPatel99 RutvikGupta Yuanzhuo-Liu christianjans saulfield

repo: huawei-noah/SMARTS


text:
请问sdk支持单片机平台吗?

有支持单片机的sdk吗?

我已经收到您的邮件啦，谢谢

该SDK不支持非Linux系统的，可以查看官网说明https://support.huaweicloud.com/sdkreference-iothub/iot_02_9980.html

authors: jiaotanghengjiao louiscrazy upupsky


text:
running hubble.audit with salt (nitrogen - 2017.7.0rc1) asserts

From @jrporcaro on July 6, 2017 22:34
I am using salt 2017.7.0rc1 (Nitrogen) with a default install of hubble 2017.4.1 on CentOS 7.2  using pygit2 and gitfs based hubble install. I did a saltutil.sync_all then a hubble.sync then a hubble.audit.
[root@master2 ~]# salt \* hubble.audit
master2:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1466, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 108, in audit
        show_compliance=show_compliance)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 406, in top
        load()
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 567, in load
        __nova__ = NovaLazyLoader()
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 667, in __init__
        self._load_all()
      File "/usr/lib/python2.7/site-packages/salt/loader.py", line 1611, in _load_all
        self._load_module(name)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 828, in _load_module
        module_name,
    ValueError: too many values to unpack

here is my versions_report:
Salt Version:
           Salt: 2017.7.0rc1

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.24.6
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.2
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-327.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

Copied from original issue: hubblestack/hubble-salt#83

This is probably in my customized loader, I expect that salt's loader changed in some way for nitrogen, and I need to update my calls. The crappy thing is that will probably require some sort of version check for salt....maybe I just need to pull in the whole loader so it doesn't change.

From @jrporcaro on July 19, 2017 1:52
I tried it with the gitfs and "develop" and here is what I got back
salt * hubble.audit
master2:
The minion function caused an exception: Traceback (most recent call
last):
File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1466, in
_thread_return
return_data = executor.execute()
File
"/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28,
in execute
return self.func(*self.args, **self.kwargs)
File "/var/cache/salt/minion/extmods/modules/hubble.py", line 113, in
audit
show_compliance=show_compliance)
File "/var/cache/salt/minion/extmods/modules/hubble.py", line 428, in
top
load()
File "/var/cache/salt/minion/extmods/modules/hubble.py", line 589, in
load
nova = NovaLazyLoader()
File "/var/cache/salt/minion/extmods/modules/nova_loader.py", line
1717, in init
super(NovaLazyLoader, self).init(_hubble_dir(),
NameError: global name '_hubble_dir' is not defined
On Mon, Jul 17, 2017 at 3:06 PM, Colton Myers notifications@github.com
wrote:

@jrporcaro https://github.com/jrporcaro could you do me a favor and
test with the develop branch of hubble-salt? I think I've resolved the
issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hubblestack/hubble-salt/issues/83#issuecomment-315884511,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHUIFf8S_qOFyzckYPT298JZm8O4cosWks5sO8zTgaJpZM4OQRPj
.

--
Jeff Porcaro
VP, Engineering
SaltStack

@jrporcaro could you do me a favor and test with the develop branch of hubble-salt? I think I've resolved the issue.

Whoops, that's my bad. Another fix incoming.

Alright, can you try one more time?

I tried with these settings:
fileserver_backend:
  - roots
  - git

gitfs_remotes:
  - https://github.com/hubblestack/hubble-salt.git:
    - base: develop
    - root: ''

and I am getting the following:
[root@master3 ~]# salt \* hubble.audit
master3:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1466, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 113, in audit
        show_compliance=show_compliance)
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 428, in top
        load()
      File "/var/cache/salt/minion/extmods/modules/hubble.py", line 589, in load
        __nova__ = NovaLazyLoader(_hubble_dir())
      File "/var/cache/salt/minion/extmods/modules/nova_loader.py", line 1718, in __init__
        opts=__opts__,
    NameError: global name '__opts__' is not defined

Thanks, investigating now.

@jrporcaro Alright, I finally have this fixed. Please try again, and close this issue if it's resolved.

This is now fixed. And should be closed. I do not have a close option because you re-entered this issue. Thanks for your work on this.

Sorry it took so long. Shouldn't break in the future, since I pulled in all the code in question, so I'm not longer relying on you guys not changing anything. :)

authors: basepi jrporcaro

repo: hubblestack/hubble


text:
Schema for donor metadata?

https://github.com/hubmapconsortium/search-api/issues/44 is tracking the indexing of donor metadata. Is there any documentation or schema about what that data should look like? (When there's a pointer for it, file a new issue in this repo to incorporate it.)

assigning to @shirey, but feel free to reassign to a better contact.
cc: @bherr2

With the portal index, I'm not really as concerned about this: Hitting unexpected data at index time is a lot less frustrating than having to deal with it in the front end. @bherr2 : If this is still something that would be helpful for you, reopen and nudge Bill.

repo: hubmapconsortium/search-schema


text:
No UMD. Fix #297

Sort of a policy question, but does it seem reasonable to you to say that UMD is just not a preferred format?

I really do not know enough about why/when to use the different library bundle formats. It seems like a small enough change that if we end up needing it in the future it would be easy to undo.

authors: keller-mark mccalluc

repo: hubmapconsortium/vitessce


text:
🛑 Akkoma is down

In af9cf1a, Akkoma (https://akkoma.chez.chat) was down:

HTTP code: 500
Response time: 5444 ms

Resolved: Akkoma is back up in df4a788 after 17 minutes.

repo: hufman/upptime


text:
前端install一直失败，yarn install和npm install都失败了

3181 verbose node v20.13.1
3182 verbose npm  v10.5.2
C:\001-Automation\django-vue3-admin\web> yarn install
➤ YN0000: · Yarn 4.2.2
➤ YN0000: ┌ Resolution step
➤ YN0001: │ Error: e-icon-picker@workspace:./lib: Workspace not found (e-icon-picker@workspace:./lib)
at St.getWorkspaceByDescriptor (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:210:3520)
at l1.getCandidates (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:140:115276)
at Pd.getCandidates (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:141:1311)
at Pd.getCandidates (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:141:1311)
at C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:210:8409
at Ky (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:140:53916)
at Fe (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:210:8389)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Promise.allSettled (index 5)
2024-05-13T09_46_34_343Z-debug-0.log
at async _c (C:\Users\jialinl\AppData\Local\node\corepack\v1\yarn\4.2.2\yarn.js:140:53244)

同样问题

@qn2060 我刚刚也遇到这个问题了，直接用e-icon-picker2.1.1就好了，他这个高版本应该是有问题，项目里引入这种star少的库风险确实太高。

已解决。
解决方式：将e-icon-picker先从package.json中移除，再单独安装e-icon-picker 2.1.1就好


text:
PyPI accelerate 0.17.0

Is there any release plan for accelerate 0.17.0? when will it be available in the PyPI?
Thanks!

There will be a release soon-ish yes. We just need to merge a couple of PRs related to the upcoming release of PyTorch 2.0.

@sgugger Thank you.

authors: rupeshs sgugger

repo: huggingface/accelerate