text:
Problem with cftime coordinates on sequence_dim

In #47, @naomi-henderson reported that cftime-based time coordinates did not work with her recipe. (Details in this notebook.
The error occurs on prepare_target. Some relevant traceback is:
---------------------------------------------------------------------------
OutOfBoundsDatetime                       Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime)
    193         try:
--> 194             dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
    195         except (KeyError, OutOfBoundsDatetime, OverflowError):

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_datetime_with_pandas(flat_num_dates, units, calendar)
    141             "Cannot decode times from a non-standard calendar, {!r}, using "
--> 142             "pandas.".format(calendar)
    143         )

OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
    112     try:
--> 113         result = decode_cf_datetime(example_value, units, calendar, use_cftime)
    114     except Exception:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime)
    196             dates = _decode_datetime_with_cftime(
--> 197                 flat_num_dates.astype(float), units, calendar
    198             )

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_datetime_with_cftime(num_dates, units, calendar)
    133     return np.asarray(
--> 134         cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
    135     )

src/cftime/_cftime.pyx in cftime._cftime.num2date()

TypeError: unsupported operand type(s) for +: 'cftime._cftime.DatetimeNoLeap' and 'NoneType'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-47-9bb7712f434d> in <module>
      1 # put basic info in target directory
----> 2 recipe.prepare_target()

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/pangeo_forge/recipe.py in _prepare_target()
    166
    167             try:
--> 168                 ds = self.open_target()
    169                 logger.info("Found an existing dataset in target")
    170                 logger.debug(f"{ds}")

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/pangeo_forge/recipe.py in open_target(self)
    271     def open_target(self):
    272         target_mapper = self.target.get_mapper()
--> 273         return xr.open_zarr(target_mapper)
    274
    275     def initialize_target(self, ds, **expand_dims):

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, decode_timedelta, use_cftime, **kwargs)
    686         backend_kwargs=backend_kwargs,
    687         decode_timedelta=decode_timedelta,
--> 688         use_cftime=use_cftime,
    689     )
    690

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    573
    574     with close_on_error(store):
--> 575         ds = maybe_decode_store(store, chunks)
    576
    577     # Ensure source filename always stored in dataset object (GH issue #2550)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in maybe_decode_store(store, chunks)
    477             drop_variables=drop_variables,
    478             use_cftime=use_cftime,
--> 479             decode_timedelta=decode_timedelta,
    480         )
    481

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    596         drop_variables=drop_variables,
    597         use_cftime=use_cftime,
--> 598         decode_timedelta=decode_timedelta,
    599     )
    600     ds = Dataset(vars, attrs=attrs)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    498             stack_char_dim=stack_char_dim,
    499             use_cftime=use_cftime,
--> 500             decode_timedelta=decode_timedelta,
    501         )
    502         if decode_coords:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta)
    338         var = times.CFTimedeltaCoder().decode(var, name=name)
    339     if decode_times:
--> 340         var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
    341
    342     dimensions, data, attributes, encoding = variables.unpack_for_decoding(var)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode(self, variable, name)
    461             units = pop_to(attrs, encoding, "units")
    462             calendar = pop_to(attrs, encoding, "calendar")
--> 463             dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
    464             transform = partial(
    465                 decode_cf_datetime,

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
    121             "if it is not installed."
    122         )
--> 123         raise ValueError(msg)
    124     else:
    125         dtype = getattr(result, "dtype", np.dtype("object"))

ValueError: unable to decode time units 'hours since 0001-01-16 12:00:00.000000' with "calendar 'noleap'". Try open

Examining this closely, it looks like there is already a dataset in the target, but it can't be opened. This makes it hard to debug. The notebook has been run non-sequentially, making it hard to debug. @naomi-henderson, it would be great if you could turn this into a reproducible example we can use to get the the bottom of the cftime issue.

Okay, here is my cftime trouble example.
This seems to be a tricky bug - which only shows up under certain conditions.  For simplicity, I am considering the case of one chunk per netcdf file and only two netcdf files.  The simplest case of a single netcdf file being converted into a single chunk of zarr is handled correctly with cftime. I needed to make an example with at least two netcdf files, even though the error shows up when storing the very first chunk/file.
For my own sanity, I found out that I can make this notebook work properly, generating an acceptable zarr store, with a minor edit of .../xarray/coding/times.py, adding 'noleap' to:
_STANDARD_CALENDARS = {"standard", "gregorian", "proleptic_gregorian"}
but this might not be a proper fix, as it might cause problems elsewhere ...
@rabernat, I have also noticed that, despite an indication to the contrary when printing out the recipe:
NetCDFtoZarrSequentialRecipe(sequence_dim='time', inputs_per_chunk=1, nitems_per_input=600, target=FSSpecTarget(fs=<fsspec.implementations.local.LocalFileSystem object at 0x7f17aa446a50>, root_path='/tmp/tmp37ghyji0'), input_cache=CacheFSSpecTarget(fs=<fsspec.implementations.local.LocalFileSystem object at 0x7f17aa446a50>, root_path='/tmp/tmpvna52uxv'), require_cache=True, consolidate_zarr=True, xarray_open_kwargs={'use_cftime': True}, xarray_concat_kwargs={}, delete_input_encoding=True)
there is no .zmetadata being saved - otherwize the zarr store looks fine.

Okay, here is my cftime trouble example.
This seems to be a tricky bug - which only shows up under certain conditions.  For simplicity, I am considering the case of one chunk per netcdf file and only two netcdf files.  The simplest case of a single netcdf file being converted into a single chunk of zarr is handled correctly with cftime. I needed to make an example with at least two netcdf files, even though the error shows up when storing the very first chunk/file.
For my own sanity, I found out that I can make this notebook work properly, generating an acceptable zarr store, with a minor edit of .../xarray/coding/times.py, adding 'noleap' to:
_STANDARD_CALENDARS = {"standard", "gregorian", "proleptic_gregorian"}
but this might not be a proper fix, as it might cause problems elsewhere ...
@rabernat, I have also noticed that, despite an indication to the contrary when printing out the recipe:
NetCDFtoZarrSequentialRecipe(sequence_dim='time', inputs_per_chunk=1, nitems_per_input=600, target=FSSpecTarget(fs=<fsspec.implementations.local.LocalFileSystem object at 0x7f17aa446a50>, root_path='/tmp/tmp37ghyji0'), input_cache=CacheFSSpecTarget(fs=<fsspec.implementations.local.LocalFileSystem object at 0x7f17aa446a50>, root_path='/tmp/tmpvna52uxv'), require_cache=True, consolidate_zarr=True, xarray_open_kwargs={'use_cftime': True}, xarray_concat_kwargs={}, delete_input_encoding=True)
there is no .zmetadata being saved - otherwize the zarr store looks fine.

I have a similar issue in the GPM IMERG recipe. For me recipe.prepare_target() works fine, the error is on store_chunk:
ValueError: unable to decode time units 'minutes since 2000-06-01 00:00:00.000000' with "calendar 'julian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

I could share a notebook where it can be reproduced, but it needs my username/password and #59.

I have a similar issue in the GPM IMERG recipe. For me recipe.prepare_target() works fine, the error is on store_chunk:
ValueError: unable to decode time units 'minutes since 2000-06-01 00:00:00.000000' with "calendar 'julian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

I could share a notebook where it can be reproduced, but it needs my username/password and #59.

@davidbrochart - do you have cftime installed?

@davidbrochart - do you have cftime installed?

Naomi's full traceback is

---------------------------------------------------------------------------
OutOfBoundsDatetime                       Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime)
    193         try:
--> 194             dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
    195         except (KeyError, OutOfBoundsDatetime, OverflowError):

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_datetime_with_pandas(flat_num_dates, units, calendar)
    141             "Cannot decode times from a non-standard calendar, {!r}, using "
--> 142             "pandas.".format(calendar)
    143         )

OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
    112     try:
--> 113         result = decode_cf_datetime(example_value, units, calendar, use_cftime)
    114     except Exception:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime)
    196             dates = _decode_datetime_with_cftime(
--> 197                 flat_num_dates.astype(float), units, calendar
    198             )

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_datetime_with_cftime(num_dates, units, calendar)
    133     return np.asarray(
--> 134         cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
    135     )

src/cftime/_cftime.pyx in cftime._cftime.num2date()

TypeError: unsupported operand type(s) for +: 'cftime._cftime.DatetimeNoLeap' and 'NoneType'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-8-dd7bdaf49ef5> in <module>
      1 #from cftime import DatetimeNoLeap
      2 # store first chunk
----> 3 recipe.store_chunk(0)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/pangeo_forge/recipe.py in _store_chunk(chunk_key)
    209             write_region = self.region_for_chunk(chunk_key)
    210             logger.info(f"Storing chunk '{chunk_key}' to Zarr region {write_region}")
--> 211             ds_chunk.to_zarr(target_mapper, region=write_region)
    212
    213         return _store_chunk

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region)
   1754             consolidated=consolidated,
   1755             append_dim=append_dim,
-> 1756             region=region,
   1757         )
   1758

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region)
   1479     writer = ArrayWriter()
   1480     # TODO: figure out how to properly handle unlimited_dims
-> 1481     dump_to_store(dataset, zstore, writer, encoding=encoding)
   1482     writes = writer.sync(compute=compute)
   1483

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1156         variables, attrs = encoder(variables, attrs)
   1157
-> 1158     store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
   1159
   1160

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    460             # there are variables to append
    461             # their encoding must be the same as in the store
--> 462             ds = open_zarr(self.ds.store, group=self.ds.path, chunks=None)
    463             variables_with_encoding = {}
    464             for vn in existing_variables:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, decode_timedelta, use_cftime, **kwargs)
    686         backend_kwargs=backend_kwargs,
    687         decode_timedelta=decode_timedelta,
--> 688         use_cftime=use_cftime,
    689     )
    690

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    573
    574     with close_on_error(store):
--> 575         ds = maybe_decode_store(store, chunks)
    576
    577     # Ensure source filename always stored in dataset object (GH issue #2550)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in maybe_decode_store(store, chunks)
    477             drop_variables=drop_variables,
    478             use_cftime=use_cftime,
--> 479             decode_timedelta=decode_timedelta,
    480         )
    481

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    596         drop_variables=drop_variables,
    597         use_cftime=use_cftime,
--> 598         decode_timedelta=decode_timedelta,
    599     )
    600     ds = Dataset(vars, attrs=attrs)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    498             stack_char_dim=stack_char_dim,
    499             use_cftime=use_cftime,
--> 500             decode_timedelta=decode_timedelta,
    501         )
    502         if decode_coords:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta)
    338         var = times.CFTimedeltaCoder().decode(var, name=name)
    339     if decode_times:
--> 340         var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
    341
    342     dimensions, data, attributes, encoding = variables.unpack_for_decoding(var)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode(self, variable, name)
    461             units = pop_to(attrs, encoding, "units")
    462             calendar = pop_to(attrs, encoding, "calendar")
--> 463             dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
    464             transform = partial(
    465                 decode_cf_datetime,

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
    121             "if it is not installed."
    122         )
--> 123         raise ValueError(msg)
    124     else:
    125         dtype = getattr(result, "dtype", np.dtype("object"))

ValueError: unable to decode time units 'hours since 1850-01-15 12:00:00.000000' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

Naomi's full traceback is

---------------------------------------------------------------------------
OutOfBoundsDatetime                       Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime)
    193         try:
--> 194             dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
    195         except (KeyError, OutOfBoundsDatetime, OverflowError):

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_datetime_with_pandas(flat_num_dates, units, calendar)
    141             "Cannot decode times from a non-standard calendar, {!r}, using "
--> 142             "pandas.".format(calendar)
    143         )

OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'noleap', using pandas.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
    112     try:
--> 113         result = decode_cf_datetime(example_value, units, calendar, use_cftime)
    114     except Exception:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime)
    196             dates = _decode_datetime_with_cftime(
--> 197                 flat_num_dates.astype(float), units, calendar
    198             )

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_datetime_with_cftime(num_dates, units, calendar)
    133     return np.asarray(
--> 134         cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
    135     )

src/cftime/_cftime.pyx in cftime._cftime.num2date()

TypeError: unsupported operand type(s) for +: 'cftime._cftime.DatetimeNoLeap' and 'NoneType'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-8-dd7bdaf49ef5> in <module>
      1 #from cftime import DatetimeNoLeap
      2 # store first chunk
----> 3 recipe.store_chunk(0)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/pangeo_forge/recipe.py in _store_chunk(chunk_key)
    209             write_region = self.region_for_chunk(chunk_key)
    210             logger.info(f"Storing chunk '{chunk_key}' to Zarr region {write_region}")
--> 211             ds_chunk.to_zarr(target_mapper, region=write_region)
    212
    213         return _store_chunk

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region)
   1754             consolidated=consolidated,
   1755             append_dim=append_dim,
-> 1756             region=region,
   1757         )
   1758

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region)
   1479     writer = ArrayWriter()
   1480     # TODO: figure out how to properly handle unlimited_dims
-> 1481     dump_to_store(dataset, zstore, writer, encoding=encoding)
   1482     writes = writer.sync(compute=compute)
   1483

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1156         variables, attrs = encoder(variables, attrs)
   1157
-> 1158     store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
   1159
   1160

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
    460             # there are variables to append
    461             # their encoding must be the same as in the store
--> 462             ds = open_zarr(self.ds.store, group=self.ds.path, chunks=None)
    463             variables_with_encoding = {}
    464             for vn in existing_variables:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, decode_timedelta, use_cftime, **kwargs)
    686         backend_kwargs=backend_kwargs,
    687         decode_timedelta=decode_timedelta,
--> 688         use_cftime=use_cftime,
    689     )
    690

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    573
    574     with close_on_error(store):
--> 575         ds = maybe_decode_store(store, chunks)
    576
    577     # Ensure source filename always stored in dataset object (GH issue #2550)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/backends/api.py in maybe_decode_store(store, chunks)
    477             drop_variables=drop_variables,
    478             use_cftime=use_cftime,
--> 479             decode_timedelta=decode_timedelta,
    480         )
    481

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    596         drop_variables=drop_variables,
    597         use_cftime=use_cftime,
--> 598         decode_timedelta=decode_timedelta,
    599     )
    600     ds = Dataset(vars, attrs=attrs)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    498             stack_char_dim=stack_char_dim,
    499             use_cftime=use_cftime,
--> 500             decode_timedelta=decode_timedelta,
    501         )
    502         if decode_coords:

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta)
    338         var = times.CFTimedeltaCoder().decode(var, name=name)
    339     if decode_times:
--> 340         var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
    341
    342     dimensions, data, attributes, encoding = variables.unpack_for_decoding(var)

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in decode(self, variable, name)
    461             units = pop_to(attrs, encoding, "units")
    462             calendar = pop_to(attrs, encoding, "calendar")
--> 463             dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
    464             transform = partial(
    465                 decode_cf_datetime,

/usr/local/python/anaconda3/envs/pangeo-forge/lib/python3.7/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
    121             "if it is not installed."
    122         )
--> 123         raise ValueError(msg)
    124     else:
    125         dtype = getattr(result, "dtype", np.dtype("object"))

ValueError: unable to decode time units 'hours since 1850-01-15 12:00:00.000000' with "calendar 'noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

@davidbrochart - do you have cftime installed?

Yes, and when I open_chunk I get cftime objects in the time dimension: cftime.DatetimeJulian(2000, 6, 1, 0, 0, 0, 0)...

@davidbrochart - do you have cftime installed?

Yes, and when I open_chunk I get cftime objects in the time dimension: cftime.DatetimeJulian(2000, 6, 1, 0, 0, 0, 0)...

@naomi-henderson, the link to your 'cftime trouble example' above (https://github.com/naomi-henderson/cmip6collect2/blob/main/NZS-cftime.ipynb) appears to be broken. (I get a 404 error when clicking it.)
Has this notebook been renamed or moved?
Curious to look into this issue, and would be helpful to have the minimal example as reference. Thanks!

@cisaacstern , that notebook was in reference to a very old version of the NetCDFtoZarrSequentialRecipe.  I haven't looked at the problem since then and I don't know if it is still an issue.  I will try to get the latest recipe and see if I still have cftime issues and, if so, try find a better example for you

Naomi, I believe that everything needed to support your recipe is in place in current master branch. If you specify nitems_per_input=None in the recipe construction (and add a metadata_cache storage target), it should be able to deal with a variable number of timesteps per file. Handling of encoding has been refactored in #86. So I am optimistic that your recipe will now work. Would you mind giving a try and reporting back with any errors you find?

@cisaacstern and @rabernat , I used nitems_per_input=None in the recipe construction and added a metadata_cache storage target, using the current master branch of pangeo-forge.  My recipe, using just NetCDFtoZarrSequentialRecipe,  now works on my four test cases, grabbing netcdf files from the GFDL AWS netcdf bucket, with no cftime issues!   I was also able to pick the chunks for the time dimension and pass the join='exact as a xarray_concat_kwargs.
But there are some new issues:

The caching of files in my '/tmp' is still hit-or-miss, especially for one of the datasets - sometimes it gets 1 or 2 of the netcdf files and then stalls out on the next file.  I guess it could be my internet connection causing the trouble, but I am getting these netcdf files from S3 - not some podunk server.  I don't have trouble with my regular scripts when downloading from S3 to a local directory using python requests.
TypeError: to_zarr() got an unexpected keyword argument 'safe_chunks' so I had to comment out the safe_chunks option in the 'NetCDFtoZarrRecipe' of 'recipe.py'. I also tried with the current master branch of xarray (xarray-0.17.1.dev49+g903278a) - same problem.
The 'https://pangeo-forge.readthedocs.io/en/latest/tutorials/netcdf_zarr_sequential.html' example is outdated in many ways ... in particular recipe.open_chunk returns a contextlib._GeneratorContextManager thing - not a dataset - so I didn't know how to look at a particular chunk to verify all was well.  If I knew how to deal with a Generator Context Manager thing, I could try to fix the documentation 8-)

My basic recipe:
recipe = NetCDFtoZarrSequentialRecipe(
    input_urls=input_urls,
    sequence_dim="time",
    target_chunks=target_chunks,
    nitems_per_input=None,
    process_chunk=set_bnds_as_coords,
    xarray_open_kwargs={'use_cftime':True},
    xarray_concat_kwargs={'join':'exact'}
)

Storage Targets:
import tempfile
from fsspec.implementations.local import LocalFileSystem
from pangeo_forge.storage import FSSpecTarget, CacheFSSpecTarget

fs_local = LocalFileSystem()

target_dir = tempfile.TemporaryDirectory()
target = FSSpecTarget(fs_local, target_dir.name)

cache_dir = tempfile.TemporaryDirectory()
cache_target = CacheFSSpecTarget(fs_local, cache_dir.name)

meta_dir = tempfile.TemporaryDirectory()
meta_store = FSSpecTarget(fs_local, meta_dir.name)

recipe.target = target
recipe.input_cache = cache_target
recipe.metadata_cache = meta_store

Execute recipe:
for input_name in recipe.iter_inputs():
    recipe.cache_input(input_name)
recipe.prepare_target()
for chunk in recipe.iter_chunks():
    recipe.store_chunk(chunk)
recipe.finalize_target()

Glad to hear that the initial cftime issue is resolved.
For the sake of organization, perhaps this issue should be closed, and the 3 new issues @naomi-henderson discovered opened as distinct, new Issues?

Thanks so much @naomi-henderson for trying this out!
Issues 1 and 2 are very likely related to your environment. The intermittent hanging in 1 sounds a lot like https://github.com/intake/filesystem_spec/issues/565; this was a bug in filesystem spec that was surfaced in part by our work on Pangeo Forge. It has been fixed in the latest fsspec master. It would be great if you could verify this.
2 is because we are now dependent on an as-of-yet unmerged xarray PR (https://github.com/pydata/xarray/pull/5065) which adds the safe_chunks option. Hopefully that will go in soon.
For development, you're probably best off creating a new environment that matches our CI, which should eliminate both problems:
https://github.com/pangeo-forge/pangeo-forge/blob/ba4dc7430137ae854b358698f0eb84fb4232c032/ci/py3.8.yml#L1-L39
(I have switched from conda to mamba and am never going back.)

3. example is outdated in many ways

Thanks for checking this. You're absolutely right that I have not bothered to update the tutorials after some recent changes. However, I hope that "outdated in many ways" is an exaggeration; I have strived to keep the API the same. The biggest change, as you noted, is the use of context managers for all openers. This allows us to keep better track of open / closed file objects and is in line with python best practice. So instead of.
ds = recipe.open_input(input_key)

you do
with recipe.open_input(input_key) as ds:
    # do something with ds
    display(ds)
    # If you want it in memory outside of the context manager, do
    ds.load()
# now the file is closed

(Same for open_chunk().) It would be fantastic if you could update the tutorials where needed. Going even further, perhaps you could turn your CMIP recipe into its own tutorial example notebook for the docs?
Thanks again for your helpful comments and real-world testing. Things are moving fast, so it's great to have this input.

For the sake of organization, perhaps this issue should be closed, and the 3 new issues @naomi-henderson discovered opened as distinct, new Issues?

Thanks for this suggestion @cisaacstern. I think 1&2 are related (see comment above). 3 is about stale documentation. Having dedicated issues for these would be useful; however, depending on @naomi-henderson's response, they might be resolved very quickly, so possibly not needed...

@rabernat, As usual, your comments are very clarifying, thanks!  This old lady brain gets easily bogged down!
Yes, "outdated in many ways" was an exaggeration, of course - that first example is very helpful!  Now that I know about context managers I will go through it again and make a pull request with my suggestions.
Okay, will use the new pangeo-forge environment - I had made a kernel with the old one and then just updated xarray, fsspec and pangeo-forge. I agree it is best to make a new kernel at this point. I will also give mamba a try because conda is taking way.... too.... long....
As for making a new tutorial example with CMIP6 - yes, I will give it a try.  I am concerned about all of the moving parts, GFDL's AWS collection included, but will try to create something robust.

@naomi-henderson - just checking whether things worked better with the updated environment?

@rabernat - I have been visiting grandkids in Virginia, so just getting back to this now.  I had updated the environment, but then had trouble with the 'https://aws-cloudnode.esgfed.org/thredds/dodsC/' OPenDAP server not working last week, so had been trying to use the s3:// urls directly - which have to be explicitly opened (unlike the gs:// urls).
It was easy to switch to mamba - thanks for the suggestion! - but then also needed to python3 -m pip install ipykernel  to add the py3.8.yml kernel, and then used mamba to add packages such as s3fs, pyarrow, fastparquet and even matplotlib.  But then the trouble with the esgfed opendap server not working while I was also changing the kernel was a bit hassle to sort out and I ran out of time.  Now that I am back, I will work on the first tutorial and adding another tutorial for CMIP6.
Have you decided to rework the recipe and pattern codes?  If so, perhaps I should wait?

I'm so happy you're able to visit your grandkids! How exciting! 😊
Sorry about your environment troubles.

so had been trying to use the s3:// urls directly - which have to be explicitly opened (unlike the gs:// urls).

This doesn't make sense to me. s3fs and gcsfs should be interchangeable here and behave identically. Could you give a more verbose example of what you mean?

Have you decided to rework the recipe and pattern codes? If so, perhaps I should wait?

I am working on that, but it's mostly an internal refactor. No reason not to try.
Anyway, nothing urgent here. Just checking in.

@rabernat ,
Okay, it was sloppy speak - but here is what I meant.
# Connect to AWS S3 storage
import s3fs
fs_s3 = s3fs.S3FileSystem(anon=True)

and then, for the cmip6-pds bucket, xarray can open the zarr stores directly:
url = "s3://cmip6-pds/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Amon/ua/gn/v20180827/"
ds = xr.open_zarr(url,consolidated=True)

but, for the esgf-world bucket, xarray cannot use the s3path directly, we must first use open:
s3path = "s3://esgf-world/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Amon/ua/gn/v20180827/ua_Amon_GISS-E2-1-G_historical_r1i1p1f1_gn_185001-190012.nc"
url    = fs_s3.open(s3path, mode='rb')
ds = xr.open_dataset(url)

otherwise, just using xr.open_dataset(s3path),  we get the error:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/file_manager.py in _acquire_with_cache_info(self, needs_lock)
    198             try:
--> 199                 file = self._cache[self._key]
    200             except KeyError:

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/lru_cache.py in __getitem__(self, key)
     52         with self._lock:
---> 53             value = self._cache[key]
     54             self._cache.move_to_end(key)

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('s3://esgf-world/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Amon/ua/gn/v20180827/ua_Amon_GISS-E2-1-G_historical_r1i1p1f1_gn_185001-190012.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-64-0d8d5c8ceff2> in <module>
      1 s3path = "s3://esgf-world/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Amon/ua/gn/v20180827/ua_Amon_GISS-E2-1-G_historical_r1i1p1f1_gn_185001-190012.nc"
      2 #url    = fs_s3.open(s3path, mode='rb')
----> 3 ds = xr.open_dataset(s3path)

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, *args, **kwargs)
    507
    508     overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 509     backend_ds = backend.open_dataset(
    510         filename_or_obj,
    511         drop_variables=drop_variables,

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose)
    543     ):
    544
--> 545         store = NetCDF4DataStore.open(
    546             filename_or_obj,
    547             mode=mode,

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in open(cls, filename, mode, format, group, clobber, diskless, persist, lock, lock_maker, autoclose)
    376             netCDF4.Dataset, filename, mode=mode, kwargs=kwargs
    377         )
--> 378         return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
    379
    380     def _acquire(self, needs_lock=True):

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in __init__(self, manager, group, mode, lock, autoclose)
    324         self._group = group
    325         self._mode = mode
--> 326         self.format = self.ds.data_model
    327         self._filename = self.ds.filepath()
    328         self.is_remote = is_remote_uri(self._filename)

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in ds(self)
    385     @property
    386     def ds(self):
--> 387         return self._acquire()
    388
    389     def open_store_variable(self, name, var):

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in _acquire(self, needs_lock)
    379
    380     def _acquire(self, needs_lock=True):
--> 381         with self._manager.acquire_context(needs_lock) as root:
    382             ds = _nc4_require_group(root, self._group, self._mode)
    383         return ds

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/contextlib.py in __enter__(self)
    111         del self.args, self.kwds, self.func
    112         try:
--> 113             return next(self.gen)
    114         except StopIteration:
    115             raise RuntimeError("generator didn't yield") from None

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/file_manager.py in acquire_context(self, needs_lock)
    185     def acquire_context(self, needs_lock=True):
    186         """Context manager for acquiring a file."""
--> 187         file, cached = self._acquire_with_cache_info(needs_lock)
    188         try:
    189             yield file

/usr/local/python/anaconda3/envs/pangeo-forge3.8/lib/python3.8/site-packages/xarray/backends/file_manager.py in _acquire_with_cache_info(self, needs_lock)
    203                     kwargs = kwargs.copy()
    204                     kwargs["mode"] = self._mode
--> 205                 file = self._opener(*self._args, **kwargs)
    206                 if self._mode == "w":
    207                     # ensure file doesn't get overriden when opened again

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

OSError: [Errno -36] NetCDF: Invalid argument: b's3://esgf-world/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Amon/ua/gn/v20180827/ua_Amon_GISS-E2-1-G_historical_r1i1p1f1_gn_185001-190012.nc'

so in my test cases, I was using the opendap URLs to avoid this extra step ...

The difference is between CDF and zarr, not s3fs and gcsfs. While I finally got my PR into xarray to use fsspec for the zarr backend, and interpret the path as necessary, this was not done for netCDF (et al) because:

the ambiguity around DAP paths, which look like fsspec remote paths, but should not be interpreted by fsspec
the mechanism for opening things is quite different for zarr (directories and no globs) versus netCDF

Since the method of passing file-like objects works OK, making everything work has not been a high priority.

thanks, @martindurant , yes - that makes sense.  So I am passing file-like objects for the CDF files (or using their OPeNDAP server) for my normal workflow - just thought it would be cleaner for the tutorial to not have the extra step.  I agree it is not high priority

So far so good! All of my CMIP6 netcdf -> zarr test cases went through with no issues (safe_chunks option still not available, but apparently that was not causing my other issues).  It looks like we could start using it for constructing zarr datasets from the esgf-world netcdf files and putting them in GC!  It might turn up some other interesting special cases.  I have just one quick question and then this issue could be closed.
The time independent CMIP6 datasets are usually in a single netcdf file and are not very large, so we don't need to worry too much about the chunk sizes.  The NetCDFtoZarrSequentialRecipe could probably be adapted to this degenerate case - is that what you suggest I use?
I will open pull requests for my suggested changes to the documentation and tutorials once I have had time to rework the first tutorial example and contribute a new CMIP6 specific example.

authors: cisaacstern davidbrochart martindurant naomi-henderson rabernat

repo: pangeo-forge/pangeo-forge


text:
fix: eslint warnings on whole examples fixed (#2765)

IMPORTANT: Please do not create a Pull Request without creating an issue first.
Any change needs to be discussed before proceeding. Failure to do so may result in the rejection of the pull request.
Please provide enough information so that others can review your pull request:
eslint warnings were cleaned by skipping warnings and by removing unused variables

Explain the details for making this change. What existing problem does the pull request solve?

Test plan (required)
Demonstrate the code is solid. If not, please add WIP: in its title.

Closing issues
Put closes #XXXX in your comment to auto-close the issue that your PR fixes (if such).
Self Check before Merge
Please check all items below before review.

[ ] Corresponding issues are created/updated or not needed
[ ] Docs are updated/provided or not needed
[x] Examples are updated/provided or not needed
[ ] TypeScript definitions are updated/provided or not needed
[ ] Tests are updated/provided or not needed
[ ] Changesets are provided or not needed

Congrats @sevkioruc 🍕

source: gharchive/pull-request

authors: omeraplak sevkioruc

repo: pankod/refine

license: MIT

license_type: permissive

license_source: github-api


text:
pex does not respect absense of --resolve-local-platforms on Mac M1, builds using local platform anyway

On an M1 Mac, when I don't specify --resolve-local-platforms, pex will still build dependencies using the local interpreter matching the --python flag, even when binary wheels exist.
When using rosetta Python (x86 emulation), this issue does not exist.
Reproduction - I have two venvs, one for m1 native (arm) and one for rosetta (x86)):
Note the binary wheel exists for grpcio for requested --platform. Using --complete-platform does not help.
# x86 works as expected - pulls the binary wheel and makes the pex
% pex --platform=manylinux2014_x86_64-cp-38-cp38 --python=python3.8 --pip-version=22.2.2 --resolver-version=pip-legacy-resolver grpcio==1.47.2 -o grpcio.pex
(x86-python) ~/src/issues/pex-python-flag
% unzip -l grpcio.pex | grep 'whl/$'
        0  01-01-1980 00:00   .deps/grpcio-1.47.2-cp38-cp38-macosx_10_10_x86_64.whl/
        0  01-01-1980 00:00   .deps/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/
        0  01-01-1980 00:00   .deps/six-1.16.0-py2.py3-none-any.whl/

# same command in arm just starts building a wheel
 pex --platform=manylinux2014_x86_64-cp-38-cp38 --python=python3.8 --pip-version=22.2.2 --resolver-version=pip-legacy-resolver grpcio==1.47.2 -o grpcio.pex -v
pex: Building pex :: Resolving distributions (grpcio==1.47.2) :: Resolving requirements. :: Resolving for:
  /Users/shalabh/src/issues/pex-python-flag/arm-python/bin/python3.8
pex: Hashing pex                                                                                pex: Hashing pex: 90.3ms
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (grpcio==1.47.2) :: Resolving requirements. :: Building distributions for:
  BuildRequest(target=LocalInterpreter(id='Users.shalabh.src.issues.pex-python-flag.arm-python.bin.python3.8', platform=Platform(platform='macosx_12_0_arm64', impl='cp', version='3.8.16', version_info=(3, 8, 16), abi='cp38'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.8.16', os_name='posix', platform_machine='arm64', platform_python_implementation='CPython', platform_release='21.6.0', platform_system='Darwin', platform_version='Darwin Kernel Version 21.6.0: Mon Aug 22 20:20:05 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T8101', python_full_version='3.8.16', python_version='3.8', sys_platform='darwin'), interpreter=PythonInterpreter('/Users/shalabh/src/issues/pex-python-flag/arm-python/bin/python3.8', PythonIdentity('/Users/shalabh/src/issues/pex-python-flag/arm-python/bin/python3.8', 'cp38', 'cp38', 'macosx_12_0_arm64', (3, 8, 16)))), source_path='/Users/shalabh/.pex/downloads/resolver_download.gmefuolk/Users.spex: Building /Users/shalabh/.pex/downloads/resolver_download.gmefuolk/Users.shalabh.src.issues.pex-python-flag.arm-python.bin.python3.8/grpcio-1.47.2.tar.gz to /Users/shalabh/.pex/built_wheels/sdists/grpcio-1.47.2.tar.gz/c7f888b067f2b1833a5670f20eeac2c6063039a23a99b799929de065c1b18692/cp38-cp38-macosx_12_0_arm64

<CTRL-C>

This has something to do with the --python flag because if I remove that, even arm quickly builds the pex from the pypi wheel.

@shalabhc this is by design but obviously not what you expect. Each --python, --interpreter-constraint, --platform and --complete-platform adds to a list of targets resolved for; i.e. they each add a "platform" to a multi-platform PEX.

The question then is what you are trying to do by adding --python?

More spin: --resolve-local-platforms is a modifier for --platform and --complete-platformand that is all. It has no bearing on either--pythonor--interpreter-constraint` which select 1 and 1+ local interpreters unconditionally. After all that selection you are left with 1 or more real local interpreters and 1 or more foreign platforms. For each real local interpreter if there is no wheel, that interpreter will be used to build an sdist into a wheel for that real local interpreter.
Does that make sense?

Ah I did not realize this. I assumed --python=python3.8 selected the interpreter on the target (it's under "PEX target environment options:"). Specifically I thought it meant "when this pex is run, ensure it is run with a binary called python3.8 and fail if python3.8 is not present in the target environment". I did not think it affected how the build itself was done.
I can exclude --python=python3.8 but how do I tell it run the pex with python3.8 in case the target has multiple interpreters present?
Here I think it makes sense because grpcio has a wheel for x86 but no wheel for arm. The only confusion that remains is: what is the point of building the wheel for arm when it is not going to be used? IIUC it will build the arm wheel but then discard it because the --platform specifies that only the manylinux wheel should be included in the pex.

The point is Pex thinks you're asking it to build a multiplatform PEX file that works on Linux x86_64 and Mac arm.

To require the PEX runs using python3.8 you need not do anything. Pex gained the ability to probe compatibility on boot a while back. That can take time though, so you can steer by setting the --python-shebang option. More direct, if you control the PEX execution command, change it from my.pex ... to /this/exact/python my.pex ....

Indeed I just noticed it is a fat pex on macos x86 (two wheels for grpcio):
 unzip -l grpcio.pex | grep 'whl/$'
        0  01-01-1980 00:00   .deps/grpcio-1.47.2-cp38-cp38-macosx_10_10_x86_64.whl/
        0  01-01-1980 00:00   .deps/grpcio-1.47.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/

I knew about multi platform pexes and created them intentionally with multiple --platform flags, but did not realize --python was also adding the local platform.
Thanks for the explanation!

You're welcome. I'll close this as answered, but please re-open if you have more questions.

source: gharchive/issue

authors: jsirois shalabhc

repo: pantsbuild/pex

license_source: bigquery


text:
Release 2.1.145

On the docket:

[ ] pex does not use .pip/pip.conf to resolve packages #336

Broken off of #2212 to make way for #2220.


text:
Cleanup test warnings.

Two fixes:

Avoid warnings about universal newlines.
These flood the end of the unit test log for ell unit test environments
except py27 and pypy2 as it stands.

Fixup an errant import of collections.Sequence.

The universal newline mode warnings can be seen here in a CI burn on main with 283 entries: https://github.com/pantsbuild/pex/runs/3405797418?check_suite_focus=true#step:7:1292


text:
Prepare the 2.0.1 release.

Fixes #796

FYI @Djailla


text:
The resource cannot be found.

I tried your plugin. Unfortunatly I was not able to get it working.
I am pretty sure I added the correct setting values, but when I try to get the status I only get the error "The resource cannot be found."
Is there anything else I can do in order to find out why its not working?

I installed it today and I'm having the same issue

Could you check the %AppData%\Roaming\Elgato\StreamDeck\Plugins\net.oksala.azuredevops.runner.DefaultPlugi.sdPlugin\log folder. There should be log file.

The resource cannot be found sounds a lot like issue with images in the plugin, but maybe the log contains more info.

Actually the logs do not show any errors.
But: I created a new button today and it now "magically" works.
Seems like a temporary bug or something.
I did not enter anything different then last time

I tried recreating one today and still get the resource error. I don't see any errors in my log either.
azuredevopsplugin20220316.log

I created a new button with no config and get the same error

I will send new version into Elgato during the weekend (maybe tomorrow) so it should arrive into shop pretty soon.

I'll give it another try next week or feel free to ping me when it's available. Glad to help however I can.
Thanks!

Submitted version 1.2.0 to Elgato which has improved logging. I will inform when the update is available at store. Usually it does not take long.

Version 1.2.0 is now available at the store. Please try again with that.

I can confirm it's working for me now. Thanks!

authors: bmacombe myMartek panuoksala


text:
Error if pageorloc formatting deviates from expected format

I have experienced the following error:
Traceback (most recent call last):
File "kindle2notion.py", line 200, in
ch = KindleClippings(CLIPPINGS_FILE)
File "kindle2notion.py", line 13, in init
self.clippings = self._getAllClippings(clippingsFile)
File "kindle2notion.py", line 18, in _getAllClippings
return self._parseClippings(allClippings)
File "kindle2notion.py", line 96, in _parseClippings
self.addToNotion(lastClip)
File "kindle2notion.py", line 185, in addToNotion
title = "Page: " + lastClip['Page'] + "\tDate Added: " +  str(lastClip['Date Added'].strftime("%A, %d %B %Y %I:%M:%S %p"))
TypeError: must be str, not NoneType
The cause is that my Kindle is storing my strings as:

Your Highlight on Location

Which is a bit of a deviation from what kindle2notion expects.
I think its probably easier to make the parsing here a bit more robust, probably by doing the following:

convert case down to lower to make it case agnostic
attempt to find substring "page" or "location" in order to check if it is a page or a location
filter out all characters which are not digits or hyphens in order to get the meaningful information

May suggest this as a push.

@j-chan-hkust After looking at your suggestion, I was able to fix part of the code. However, I still get an error while executing. Would you mind looking at the other issue I raised? Thanks!

Hey, thanks for raising this! Sorry for the late reply! Will look into implementing a feature to take all these subtle variations in formats into account.

@paperboi linked the commit here #284021522ddcd65689cdc1858ddb8f5387fd54d3

#10

@j-chan-hkust implemented your suggestions as a fix. Thanks for the help!

authors: j-chan-hkust paperboi pranavkaranjkar xbassi


text:
RuntimeError: CUDA error: invalid device ordinal

When I load model I have this error.
Traceback (most recent call last):
File "", line 1, in
File "test/env/lib/python3.9/site-packages/galai/init.py", line 39, in load_model
model._load_checkpoint(checkpoint_path=get_checkpoint_path(name))
File "test/env/lib/python3.9/site-packages/galai/model.py", line 63, in _load_checkpoint
load_checkpoint_and_dispatch(
File "test/env/lib/python3.9/site-packages/accelerate/big_modeling.py", line 366, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "test/env/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 701, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param)
File "test/env/lib/python3.9/site-packages/accelerate/utils/modeling.py", line 124, in set_module_tensor_to_device
new_value = value.to(device)
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Trying this with
model = galai.load_model("base")

it looks like there is a device map that expects 8 GPUs, if I'm seeing this right:
{'decoder.embed_tokens': 0,
 'decoder.embed_positions': 0,
 'decoder.layer_norm': 0,
 'decoder.layers.0': 0,
 'decoder.layers.1': 0,
 'decoder.layers.2': 0,
 'decoder.layers.3': 1,
 'decoder.layers.4': 1,
 'decoder.layers.5': 1,
 'decoder.layers.6': 2,
 'decoder.layers.7': 2,
 'decoder.layers.8': 2,
 'decoder.layers.9': 3,
 'decoder.layers.10': 3,
 'decoder.layers.11': 3,
 'decoder.layers.12': 4,
 'decoder.layers.13': 4,
 'decoder.layers.14': 4,
 'decoder.layers.15': 5,
 'decoder.layers.16': 5,
 'decoder.layers.17': 5,
 'decoder.layers.18': 6,
 'decoder.layers.19': 6,
 'decoder.layers.20': 6,
 'decoder.layers.21': 7,
 'decoder.layers.22': 7,
 'decoder.layers.23': 7}

Confirmed. Had same error and num_gpus = 1 resolved it.

Please mention that in your documentation / readme.

A model size between base and standard would be nice. I barely can't fit standard on my RTX 3090, I think.

Do you offer 8 bit versions/compatibility, like BLOOM?

I see, dtype='float16' does the job sorry. Please mention in readme. Many folks will want to try on a local gpu as well.

Hmm.. 8 bit would still be handy to play with larger models. Is that possible?

Num of GPUs defaults to None.

If you have less than the default number of GPUs (8)

Who has a default number of 8 GPUs?

people that work at Meta AI, probably XD

If you have less than the default number of GPUs (8), you have to specify how many when you load the model. Try: model = gal.load_model(name = 'base', num_gpus = 1)

why this isnt written on main page

galai 1.1.0 uses all available GPUs by default which should fix the issue. One can still manually specify the number of GPUs using num_gpus parameter. Setting num_gpus=0 (or keeping the default None if no GPUs are available) will load the model to RAM. 8 bit inference is not supported yet. Please reopen if you still experience any issues.

authors: FurkanGozukara KnutJaegersberg dcruiz01 dginev elter-tef metaphorz mkardas zzj0402

repo: paperswithcode/galai


text:
Update pgrx to the stable 0.12 version once it is released

          This feels unsafe. I'm not sure if we want to match all other cases to user mapping relation id. I made a PR to pgrx a while back to expose `USER_MAPPING_ID` to `pg_sys` but it would require we upgrade `pgrx`.

Originally posted by @rebasedming in https://github.com/paradedb/pg_analytics/pull/37#discussion_r1712662854

Great idea! I chatted with the pgrx folks and it should release within the week (As of August 12, 2024)

authors: Weijun-H philippemnoel

license: PostgreSQL


text:
Incorrect head block in Status

Describe the bug
The ethereum/devp2p suite now mostly fails with:
not ok 1 TestStatus
# peering failed: status exchange failed: wrong head block in status, want:  0x0e70e01064023f70f047dbf0b97e7109e4aa5df3d643d45f8e91e47d3d67a424 (block 999) have 0xd6f7972e854d4d7812fca9f911e9a1c1db3b89e2650a25b7782a1d8972ced000
not ok 2 TestGetBlockHeaders
# peering failed: status exchange failed: wrong head block in status, want:  0x0e70e01064023f70f047dbf0b97e7109e4aa5df3d643d45f8e91e47d3d67a424 (block 999) have 0xd6f7972e854d4d7812fca9f911e9a1c1db3b89e2650a25b7782a1d8972ced000
not ok 3 TestSimultaneousRequests
# peering failed: status exchange failed: wrong head block in status, want:  0x0e70e01064023f70f047dbf0b97e7109e4aa5df3d643d45f8e91e47d3d67a424 (block 999) have 0xd6f7972e854d4d7812fca9f911e9a1c1db3b89e2650a25b7782a1d8972ced000
not ok 4 TestSameRequestID
# peering failed: status exchange failed: wrong head block in status, want:  0x0e70e01064023f70f047dbf0b97e7109e4aa5df3d643d45f8e91e47d3d67a424 (block 999) have 0xd6f7972e854d4d7812fca9f911e9a1c1db3b89e2650a25b7782a1d8972ced000
not ok 5 TestZeroRequestID
# peering failed: status exchange failed: wrong head block in status, want:  0x0e70e01064023f70f047dbf0b97e7109e4aa5df3d643d45f8e91e47d3d67a424 (block 999) have 0xd6f7972e854d4d7812fca9f911e9a1c1db3b89e2650a25b7782a1d8972ced000

Link to CI run:
https://github.com/paradigmxyz/reth/actions/runs/4847074556/jobs/8637202357#step:7:11
The ethereum/sync test also fails with:
wrong chain head 0 (7245cf..5879f3), want 3000 (c95596..c97840)

Steps to reproduce
Run hive with the devp2p target:
./hive --client reth --sim devp2p --sim.limit "eth"

Node logs
No response
Platform(s)
No response
What version/commit are you on?
No response
Code of Conduct

[X] I agree to follow the Code of Conduct

Fixed by https://github.com/paradigmxyz/reth/pull/2527

repo: paradigmxyz/reth


text:
(fix available) Loading genesis.json from Geth fails due to very large U256: terminalTotalDifficulty

Describe the bug
Hi,
When attempting to load eth mainnet (or forks) genesis.json from geth you will get the error Error("invalid type: floating point `58750003716598360000000 on the field terminalTotalDifficulty
A quick hack is to edit the file and wrap the number in quotes "x" so that serde does not force reading it as a floating point.
After init, you have to run the client with reth --chain genesis.json or it will get the difficulty numbers wrong
I tried to make a deserializer to avoid this, but it seems the serde package is forcing non-quoted numbers into floating points
Steps to reproduce
reth init --chain genesis.json
Node logs
No response
Platform(s)
No response
What version/commit are you on?
No response
What database version are you on?
No response
What type of node are you running?
None
What prune config do you use, if any?
No response
If you've built Reth from source, provide the full command you used
No response
Code of Conduct

[X] I agree to follow the Code of Conduct

which commit are you on?
can you please share the json file?

which commit are you on? can you please share the json file?

Tested on master. The attached file was generated by 'geth --mainnet dumpgenesis'
eth.json

sorry can't reproduce on main, please provide sha
cargo run --bin reth -- init --chain ../genesis.json --datadir dummy                                                                                $ main
    Finished dev [unoptimized + debuginfo] target(s) in 0.64s
     Running `target/debug/reth init --chain ../genesis.json --datadir dummy`
2023-11-08T17:03:59.226724Z  INFO reth::cli: reth init starting
2023-11-08T17:03:59.226868Z  INFO reth::cli: Opening database path="dummy/db"
2023-11-08T17:03:59.230323Z  INFO reth::cli: Database opened
2023-11-08T17:03:59.230337Z  INFO reth::cli: Writing genesis block
2023-11-08T17:04:00.982056Z  INFO reth::cli: Genesis block written hash=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3

You're right. I blundered with a fork unknowingly and I had an older version. Sorry


text:
Use already recovered transaction during tracing

Describe the feature
with #5302
we're caching the senders of transactions, so we no longer need to recover them during tracing
TODO
use SealedBlockWithSenders during block level tracing
Additional context
No response

Can take this !

sweet!, lmk if you need pointers, this affects trace_ debug_ rpc calls that fetch the full blocks

@yash-atreya do let us know when you intend to take it on ^^

Hey @gakonst @mattsse
I was going to start working on this on Thursday.
Do you intend to include this in alpha.11? Will try to do it sooner then.

thursday's fine :)
thanks in advance 🙏

Hey @mattsse
Extremely sorry for the delay.
Needed some guidance to implement this. I have identified the block level tracing methods where the cached senders should be used instead of recovering the signers.

trace_block
Under the hood this function uses fn trace_block_until to generate the traces.
https://github.com/paradigmxyz/reth/blob/025511baa654bf664c3d3e684364707bc6db5ed2/crates/rpc/rpc/src/eth/api/transactions.rs#L762
trace_block_until calls self.block_by_id to get the SealedBlock
https://github.com/paradigmxyz/reth/blob/025511baa654bf664c3d3e684364707bc6db5ed2/crates/rpc/rpc/src/eth/api/transactions.rs#L781-L782
Enhancement:
Use self.block_with_senders(blockId) which returns SealedBlockWithSenders. This removes the need doing the following
https://github.com/paradigmxyz/reth/blob/025511baa654bf664c3d3e684364707bc6db5ed2/crates/rpc/rpc/src/eth/api/transactions.rs#L808 instead we can just do:

let tx = TransactionSignedEcRecovered::from_signed_transaction(tx, signer); // Signer is from block.senders[idx]

trace_filter
The above enhancement will also apply here as this function also uses trace_block_until.
Moreover, this function fetches blocks in the specified range:
https://github.com/paradigmxyz/reth/blob/025511baa654bf664c3d3e684364707bc6db5ed2/crates/rpc/rpc/src/trace.rs#L260
And the signers of the fetched blocks are recovered and used in the match filter, like so:
https://github.com/paradigmxyz/reth/blob/025511baa654bf664c3d3e684364707bc6db5ed2/crates/rpc/rpc/src/trace.rs#L267-L275
Enhancement:
Add new method fn block_range_with_senders which fetches the range of block and returns ProviderResult<Vec<BlockWithSenders>>. We do not need to recover the signers now. However this uses the db and not the cache.

replay_block_transactions
This method also uses trace_block_until under the hood and the initial enhancement applies here as well.

I hope I'm on the right track, would appreciate your input on this.

@yash-atreya sorry I forgot to respond to this one here.
your PR was great, so now we can the same thing for:
https://github.com/paradigmxyz/reth/blob/main/crates/rpc/rpc/src/eth/api/call.rs#L74-L74
and all the others that fetch blocks for tracing

@yash-atreya sorry I forgot to respond to this one here. your PR was great, so now we can the same thing for:
https://github.com/paradigmxyz/reth/blob/main/crates/rpc/rpc/src/eth/api/call.rs#L74-L74
and all the others that fetch blocks for tracing

@mattsse Thanks. I'll get on it.
Just wanted to confirm whether I should move forward with the following enhancement I suggested for trace_filter, which uses the db instead of the cache to retrieve the block range with senders. I'll create a separate PR for this.

Enhancement:
Add new method fn block_range_with_senders which fetches the range of block and returns ProviderResult<Vec>. We do not need to recover the signers now. However, this uses the db and not the cache.

Just wanted to confirm whether I should move forward with the following enhancement I suggested for trace_filter, which uses the db instead of the cache to retrieve the block range with senders. I'll create a separate PR for this.

need to think about it, but I believe we want this yes

Just wanted to confirm whether I should move forward with the following enhancement I suggested for trace_filter, which uses the db instead of the cache to retrieve the block range with senders. I'll create a separate PR for this.

need to think about it, but I believe we want this yes

Have created the PR in case you need it. #5647

Hey @mattsse, the last few PRs have been extremely fun, thank you for your guidance.
Feel free to assign me more issues, I'll be on the lookout as well.

ty!
if you don't find any interesting issue please ping dm on tg or twitter and I'll come up if something

authors: gakonst mattsse yash-atreya


text:
feat(cli): add db drop subcommand

Closes: https://github.com/paradigmxyz/reth/issues/796
This PR adds a subcommand to clear all tables from the database.

Codecov Report

Merging #801 (1fb0777) into main (55bc259) will decrease coverage by 0.09%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main     #801      +/-   ##
==========================================
- Coverage   73.10%   73.00%   -0.10%
==========================================
  Files         274      274
  Lines       28340    28377      +37
==========================================
  Hits        20717    20717
- Misses       7623     7660      +37

Flag
Coverage Δ

unit-tests
73.00% <0.00%> (-0.10%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

bin/reth/src/db/mod.rs
0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter lambdaclass-user


text:
Password Change Request does not handle 'email not confirmed' error

The CloudSessionLocalService class is not handling the EmailNotConfirmedException while processing a password reset request in the requestPasswordReset and doPasswordReset methods.

This issue was addressed in pull request #25. Closing issue.

repo: parallaxinc/Cloud-session-java-client

license: mit


text:
Use rust's own bool type in abstraction crate

The Bbool type was translating between the FFI's CK_BBOOL and rust's own bool, by providing a backing intermediate type. But the storage for a bool is guaranteed to be large enough for the job, and the point of conversion from byte back to bool is already structured to check for validity.

Ref for the data layout, which is also present in a comment in the commit: https://doc.rust-lang.org/reference/type-layout.html#primitive-data-layout

This looks fine by me. And since Rust's true is equivalent to CK_TRUE this should work.

This looks fine by me. And since Rust's true is equivalent to CK_TRUE this should work.

Funny enough, there is some ambiguity here. CK_TRUE is defined to 1, but the range of valid types is much more lax:

In Cryptoki, the CK_BBOOL data type is a Boolean type that can be true or false.  A zero value means
false, and a nonzero value means true.

Should the range error condition on the u8->bool be removed?

In Cryptoki, the CK_BBOOL data type is a Boolean type that can be true or false. A zero value means false, and a nonzero value means true.

That's why I love this spec 🙄

Should the range error condition on the u8->bool be removed?

Yeah probably to be as spec-compliant as possible (which I have droned on about in the past).

That's why I love this spec :roll_eyes:

If it's any consolation, I don't like it either.

repo: parallaxsecond/rust-cryptoki


text:
Added scroll-to-up button

Description:
I’ve added a Back to Top link for easier navigation within the README file. At the bottom of each main section, you’ll find a Back to Top link that will take you back to the beginning of the document.

Screenshot/Preview:

@param-code Please review and merge it under gssoc-ext level1 and hacktoberfest lables.

Description: I’ve added a Back to Top link for easier navigation within the README file. At the bottom of each main section, you’ll find a Back to Top link that will take you back to the beginning of the document.
Screenshot/Preview:
@param-code Please review and merge it under gssoc-ext level1 and hacktoberfest lables.

Sure.

authors: T-Fathima param-code

repo: param-code/counter-app


text:
Bug-Fix: Fixes Production

Fix #65
Solves the problem of deployment not found for the production

Done already!!

authors: ArinNigam param-code


text:
Support for withSponsored on various base-cms-marko-web-theme-monorail components

Section Feed example:

Various Forbes blocks:

 URGENT: Need "Sponsored" thumbnail!


text:
Object Null Reference

-Object Reference not set to be an instance of an object in these Services. Kindly initialize the db instance
Services are:

AssessmentCategory
ModuleAssessment
ModuleAssessmentMark
@maazk9119  Needs to fix this.

I have resolved it into feature-course-offering branch.
I am going to close this Issue

repo: parbatpk/parbat


text:
Add .ruby-version file

Allow people using rvm to switch automatically to the required Gemfile version of Ruby.

Already done here : https://github.com/parisrb/rubyparis.org/pull/15.


text:
New action request: actions-rs/audit-check

Action name
uses: actions-rs/audit-check
Reasons
actions-rs/audit-check

Intended usage
actions-rs/audit-check

I have checked the internal marketplace for similar actions and couldn't find one that works for us.

[X] I have checked the internal marketplace first

:robot: Found these results:

Check
Results
Links

CodeQL on the forked repo
:x:
CodeQL run

Security scan:

Check
Result

No Dependabot configuration found
⛔

No CodeQL Init found
⛔

No CodeQL Analyze found
⛔

:robot: Check completed with result: ⚠
Found 29 Dependabot security vulnerabilities:
15 where high and 14 where moderate.
Use this link to verify the alerts.


text:
feat: Fixes #390. Show shortened version if account name over 25 chars

Screenshots:

From my test we could allow up to 34 characters on the account list, eg:

compared to

and keep the 25 char limit on the header view.
Should we limit the amount of char in the name field at account creation/import?

It's different for me. If I change to allow up to 34 chars on the account list, it appears as follows. If I instead allow up to 31 chars it looks ok

But perhaps we should approach it a different way.

Totally, what about using something like text-overflow: ellipsis ?

Hey @ltfschoen, do you want to go on with this PR or have you had other thoughts to do it?

Sorry for delayed response. The text-overflow: ellipsis idea is brilliant idea

@Tbaut I've pushed a commit using text-overflow: ellipsis, it appears to work great

Thanks a lot, that works well.

One thing I could notice is that Capital letter and lower letter are cut, the big "T" isn't fully visible and the "p" and "g" also:

@Tbaut I've pushed a commit to prevent the clipping from occurring on Linux

authors: Tbaut ltfschoen

repo: paritytech/fether


text:
Reduce alloc in copy_to_overlay

I notice that the copy_to_overlay method clones the value vector in Hash index mode and clones both the key and value vectors in Btree index mode, which seems to be an unnecessary overhead, this PR tries to, instead, clone the pointer type Arc<Vec<u8>> which is much cheaper, please let me know your feedback, thanks!

Thank you for the PR.
I've ran the stress test with it and this does not seem to affect performance. Memory usage is sligly lower indeed.

Thank you for the PR. I've ran the stress test with it and this does not seem to affect performance. Memory usage is sligtly lower indeed.

Where can I found the stress test code? In my scenario, I have ~70M pairs to ingest upfront so it's a write-only scenario, and I have to commit a batch at 1GB or 2GB to work around the hard coded 16MB write queue size. This change does help my scenery a bit

Thank you for the PR. I've ran the stress test with it and this does not seem to affect performance. Memory usage is sligtly lower indeed.

Where can I find the stress test code? In my scenario, I have ~70M pairs to ingest upfront so it's a write-only scenario, and I have to commit batches at 1GB or 2GB to work around the hard-coded 16MB write queue size. This change does help my scenario a bit

cargo run -p parity-db-admin -- stress --help

@cheme  thanks!

authors: arkpar cheme hanabi1224

repo: paritytech/parity-db


text:
How to estimate validator and nominator rewards?

Is there any documentations available anywhere about how block reward is calculated?
It will be good if there is a way to calculate ROI.

I will soon publish a small document about the Staking Module which should answer this question; net yet written but I actually have a section dedicated to explaining how the reward and slash value is calculated + shared among validator and nominators.
at the moment the test cases are the only resource.

And One way or another this issue/question suits https://github.com/paritytech/substrate-developer-hub better than here.

@Kianenigma this repository is private and not visible for non-parity members.

@xlc Please see if this answers your question.

@Kianenigma I don't think it answers all my questions.
More specifically:

With 4 validators, each have 1000000 tokens, and session reward of 0.1%, session length of 10 blocks, era length of 5 sessions, all 100% availability, what is the expected rewards for each validators per session, and per era?
With same scenario as above, 3 nominators nominate each validator 1000 tokens, assuming validator does not take any cut, what is the expected rewards for each validators and nominators per session, and per era?
With same scenario as 1, one validator offline once per era, what is the expected reward for this validator and others that always available?

Basically I want some formulas that can be used to calculate ROI for validators and nominators. If I invest 100000 DOTs to become a validator or nominator, how many DOTs do I expected to make per year? If my availability is 99%, what is my expected rewards? To what point I will start losing DOTs due to off-line slash more than rewards?

This is some nice brain food. Let's see.
Note that I am answering this based on https://github.com/paritytech/substrate/pull/1915 which is not yet merged but I believe it is not that far. With this PR, nominators' vote plays a significant role in the staking system and basically their staked value is the one and only measure of electing validators (technically the value that the validator has in stake doesn not play any role regarding being elected; it does matter for reward though).
I will base all the below answer based on all stakers not forwarding their rewards to their staked account. Otherwise, this will change slot stake which then affects session reward which makes the naswer to this another step harder. Though, note that the same logic applies if slot stake changes. Slot stake is the minimum staked value among all stakers in the system and the reward of each session is slot_stake multiplied by a ration which in this case you defined to be 0.1%.

This is good to discuss but completely in the gray zone. First I have to discuss the situation where nominators are scarce. The Staking Module defines two storage configs: ValidatorCount and MinimumValidatorCount. With the current implementation, all validators are first filtered by having a none-zero and positiveapproval_score, which is sum of the staked fund of all nominators who voted for this particular validator.  Next:

If the number of available validators with this filter are more than ValidatorCount: Normal election algorithm executes and ValidatorCount validators are chosen (I call this code green/white).
If the number of available validators with this filter are less than ValidatorCount but more than MinimumValidatorCount: All of them are elected, and nominators who voted for this elected set are also recorded. (I call this code yellow)
if the number of available validators with this filter are less than ValidatorCount and MinimumValidatorCount, Nothing happens. The previous generation of validators remain in duty and more importantly, the nominator votes are not taken into account at all (meaning that potentially the next era reward will not be shared -- I call this code red).

Now, to properly answer 1, it must first become clear which of the above is the case for the 4 aforementioned validators.
To be a good complement of your question 2, I will assume code red, meaning that 4 validators are chosen and they DO NOT SHARE anything with anyone else. The calculation will be as follows:
Ignoring the first round, the session reward will be calculated with your numbers as:
// current_session_reward = concrete value of reward
// session_reward = the ratio
current_session_reward = slot_stake * session_reward
// if block production is on-time
current_era_reward = era_length * current_session_reward

Hence,
1_000_000 * 0.001 = 1000. This is because slot_stake == 1_000_000. With 5 sessions to get the actual reward, each validator will be paid 5000 per era. Since I've assumed constant slot_stake, this should not change and the same reward should be paid henceforth.

Is the session reward calculated as 1000000 * 0.1% for one validators? Or it is for all validators?

Yes (given all the conditions above) and this is paid Per Validator.

I will assume:

Again, slot_stake is constant and is the minumum that all entities have in stake. Let's change

4 nominators nominate each validator 1000 1_000_000 tokens.

to all nominators placing also 1_000_000 in stake, otherwise sessoion reward will be 1 and not that interesting anymore to be divided in an environment where no float calculation can be executed.
Ignoring the first round, the session reward will be calculated as 1_000_000 * 0.001 = 1000. Note that by this point the staking system does not even take into account if nominators/validator_cut exist or not. Similarly, 5000 is recorded at the end of the era and should be distributed. To illustrate how, this struct kinda shows the exposure of each validator at the end of the era:
{
    total: 5_000_000,
    own: 1_000_000,
    others: [ {who: N1, 1000_000}, {who: N2, 1000_000}, {who: N3, 1000_000}, {who: N4, 1000_000}]
}

And they are each rewarded by a ratio of the total 5000, proportional to what they contribute to exposure.total == 5_000_000. In this case it will be all even with 1/5 share for everyone. Hence, each validator will get 5000 / 5 => 1_000 per era, same for nominators.
Finally, since you mentioned that each nominator will vote for all of the selected 4 validator, this means that at the end of the era, each nominator gets rewarded 4 times for each of its elected validators. Hence, the total reward for a nominator will be 4000 per era.

Going offline will not

affect the reward for rest of the validators.
It will also not affect the reward of the validator that went offline. The first N offline reports are grace and once it reaches the threshold, it will be slashed and kicked out. This might raise the question of how much is the slash for each validator and nominators but I will leave this for another comment.
What will affect the rewards (of all validators) is the timeliness of block production. But I will leave this also for another comment.

Bringing this up to mention that the logic here as changed significantly, should be re-documented in the future. cc @thiolliere

logic hasn't change except the total reward per era and the how this total payout is distributed between validator and there associated nominator.
the total payout per era is computed using the current stake rate in npos (stake by validators and nominator / total tokens) and the era duration. See more about this function see more here: https://research.web3.foundation/en/latest/polkadot/Token Economics/#inflation-model
then once this total payout is computed it is distributed between validator not equally as it was the case before but depending on action that happen during the era. During the era ppl gets point if they produce a block (20 points), make a validity statement for parachain (20 points), reference an uncle (2 points), produce a referenced uncle (1 points).
(validity statement will be implemented on polkadot side, and also ppl does not get rewarded because of issue #2986).
See more here https://research.web3.foundation/en/latest/polkadot/Token Economics/#payment-details

Closing as I see both the Staking code and web3 research pages covering the topic sufficiently

authors: Kianenigma bkchr kianenigma thiolliere xlc

repo: paritytech/substrate


text:
Could not convert parameter tx between node and runtime: No such variant in enum MultiSignature

Error: 1002: Verification Error: Execution: Could not convert parameter `tx` between node and runtime: No such variant in enum MultiSignature: RuntimeApi, Execution: Could not convert parameter `tx` between node and runtime: No such variant in enum MultiSignature
    at t.RpcCoder._checkError (polkadot.01.9d35375d.js:1)
    at t.RpcCoder.decodeResponse (polkadot.01.9d35375d.js:1)
    at R.value (polkadot.01.9d35375d.js:1)
    at WebSocket.value (polkadot.01.9d35375d.js:1)

Error: 1: Unable to query dispatch info.: Execution, ApiCould not convert parameter `uxt` between node and runtime: No such variant in enum MultiSignature
    at t.RpcCoder._checkError (polkadot.01.9d35375d.js:1)
    at t.RpcCoder.decodeResponse (polkadot.01.9d35375d.js:1)
    at R.value (polkadot.01.9d35375d.js:1)
    at WebSocket.value (polkadot.01.9d35375d.js:1)


text:
Make all type information available via the metadata endpoint

When I request the metadata from my node, I see a description of the Balances transfer call that looks like this:
{
          "name": "Balances",
          "storage": {...}
          "calls": [
            {
              "name": "transfer",
              "args": [
                {
                  "name": "dest",
                  "type": "LookupSource"
                },
                {
                  "name": "value",
                  "type": "Compact<Balance>"
                }
              ], ...
            }
}

However, nowhere in the rest of the metadata is it described the structure of the LookupSource datatype. This means there is no automatic way to determine what datatype is expected by the node when making calls to this API. Authors of the python and javascript API clients must manually maintain a mapping of high to low level types as you can see here:
https://github.com/polkadot-js/api/tree/master/packages/types/src/interfaces/contracts
https://github.com/polkascan/py-scale-codec/tree/master/scalecodec/type_registry
Please make this information available in the existing metadata endpoint or perhaps a new endpoint.

Yes this will be solved by scale-info https://github.com/paritytech/scale-info
So that full type description will be available in the metadata.
cc @ascjones

Yes this will be solved by scale-info https://github.com/paritytech/scale-info
So that full type description will be available in the metadata.
cc @ascjones

@thiolliere Thanks for that information. Do you have an idea of roughly when this feature will be rolled into substrate? We don't have an urgent need for it now but it would be good to know whether this is an imminent feature or something that still has large challenges to overcome (I don't believe for a second it's a trivial issue).

@thiolliere Thanks for that information. Do you have an idea of roughly when this feature will be rolled into substrate? We don't have an urgent need for it now but it would be good to know whether this is an imminent feature or something that still has large challenges to overcome (I don't believe for a second it's a trivial issue).

I'm actively working on this, but I wouldn't expect it to be in master for a while yet. Not until March at least. I have a proof-of-concept branch https://github.com/paritytech/substrate/compare/aj-metadata-vnext to keep an eye on, but it is not in a working state at the moment.
Also it depends on all pallets being converted to the new pallet attribute syntax https://github.com/paritytech/substrate/issues/7882 so keep an eye on the progress there.

I'm actively working on this, but I wouldn't expect it to be in master for a while yet. Not until March at least. I have a proof-of-concept branch https://github.com/paritytech/substrate/compare/aj-metadata-vnext to keep an eye on, but it is not in a working state at the moment.
Also it depends on all pallets being converted to the new pallet attribute syntax https://github.com/paritytech/substrate/issues/7882 so keep an eye on the progress there.

Duplicates to #917

Duplicates to #917

closed in favor of https://github.com/paritytech/substrate/issues/917

closed in favor of https://github.com/paritytech/substrate/issues/917

Almost 2.5 years in the making :)
This feature will (hopefully) lower the maintenance burden on the API-layer teams significantly.
The Polkascan team really looks forward to this feature.

Almost 2.5 years in the making :)
This feature will (hopefully) lower the maintenance burden on the API-layer teams significantly.
The Polkascan team really looks forward to this feature.

Hopefully in the next few weeks I can provide a working branch that will produce the new metadata to get some feedback on it.

Hopefully in the next few weeks I can provide a working branch that will produce the new metadata to get some feedback on it.

authors: alexspurling ascjones emielvanderhoek thiolliere xlc


text:
Weight v1.5 Follow Ups

This PR does some follow ups to Weight v1.5

Improve math APIs exposed by Weight.

Mul and Div are always scalar over the fields.
We do not support Weight * Weight or Weight / Weight which is nonsensical in the context of multi-dimensional weights.

Remove Weight::one() which doesn't really make sense with multi-dimensional weights.
Remove Weight::new() and use just Weight::zero() instead.
Introduce macros to multiply by u8, u16, u32, u64, Percent, PerU16, Permill, Perbill, Perquintill

polkadot companion: https://github.com/paritytech/polkadot/pull/5949
cumulus companion: https://github.com/paritytech/cumulus/pull/1584

More follow up here: https://github.com/paritytech/substrate/pull/12157

Just a bit unclear why we implement standalone functions for operations that have standard traits, otherwise looks good.

because the traits are not const, and things like saturating, imply that we must implement things like saturating_mul(Self), which does not make sense.
Finally, it can help reduce the number of trait imports you need when using the Weight type and functions. So in this case, every trait which I could implement, i did, but only as a re-export of the internal functions which are all const.

bot merge


text:
macro stubs for all pallet:: macros to improve documentation visibility and discovery + revamp of pallet macro documentation

Description
This PR adds macro stubs in frame_support_procedural for all of the pallet:: macros, moves documentation for the specific pallet:: macros to their respective macro stubs (note that we do keep the majority of it where it was, only things like expansion info are truly moved), and finally includes the stubs as part of the pallet prelude. This results in several key developer experience improvements:

In the generated cargo docs, searching for a particular pallet macro in the search bar or browsing through the list of macros will result in actually finding the macro you are looking for instead of it saying no results found
Users of rust-analyzer and similar static analysis tools will see docs for pallet:: macros when they hover over them (right now they see nothing)
Users of rust-analyzer and similar static analysis tools will be able to right click a pallet:: macro and select "Go to Definition" successfully (right now it fails to find the definition)

For context, the reason we don't already have stubs for these macros is we use them essentially as markers that later have their AST nodes removed by the pallet expansion macro. In other words definitions for these macros don't actually need to exist in our codebase for them to function properly, since all the code associated with these markers is elsewhere in the pallet expansion macro machinery.
Note I ended up also substantially re-writing/fixing up the existing pallet macro documentation, as noted in my line comments.
Related Issues

Fixes #12333
https://substrate.stackexchange.com/questions/4959/invalid-generic-declaration-when-using-frame-macro-palleterror

Proof of Concept

Note that I've been going a step further with this -- it turns out the docs on that page had a lot of grammatical errors and formatting confusions so I've been fixing up the whole thing and adding a more navigable and complete table of all the pallet macros :)

btw I added B7-runtimenoteworthy to this because I think people will want to know that the docs have been revamped. Release notes bullet points would be something like:

docs for frame_support::pallet and all pallet::* macros substantially improved
new frame_support::pallet_macros module containing documentation stubs for all pallet::* attribute macros to improve discoverability and readability of pallet macro docs

Happy to change label to whatever ofc!

Note to reviewers: be on the lookout for any scenario where I misunderstand something from the original docs. A lot of the wording was pretty dicey in the original but I think I interpreted it all correctly.

bot rebase

bot rebase

bot rebase

bot merge

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:
https://forum.polkadot.network/t/polkadot-release-analysis/1026/2


text:
Assets Pallet: reintroduce fungibles::Destroy trait

This PR is an offshoot of #12310 , to maintain a way for users of the asset pallet still remove fungible assets.
It's a separate PR to allow it be reviewed easily without bloating #12310 further. But they should be both merged in together.

bot rebase

bot rebase

authors: jsidorenko tonyalaribe


text:
Removed assumptions about ancestry from fork tree prune method

Fork tree prune method was assuming that the user's predicate closure makes the find_node_index_where to NOT return the tree node immediately preceding the hash parameter (according to is_descendent_of closure).
Up to date the only users of this structure are GRANDPA and BABE in order to keep a tree of epoch changes announcements AND according to their predicate, the assumption is satisfied.
But this is a time-bomb.
If at some point a user doesn't satisfy the assumption, the tree would break and we loose part of the tree due to how prune logic is implemented.
(indeed this assumption was the cause of https://github.com/paritytech/substrate/pull/12758 bug)
This PR removes the assumption and makes the tree more resilient and independent for any use case.

THIS PR IS STILL IN DRAFT AND SHARED ONLY TO GAIN TEAM FEEDBACK
CODE IS A BIT HACKY AND PROBABLY WILL BE IMPROVED

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2084697

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2084698

A burn-in has been performed. Both full sync and warp went good

A burn-in has been performed. Both full sync and warp went good

If it worked, why did you closed the pr? :D

Jeez that was not intentional :-D

Not stale

authors: bkchr davxy paritytech-cicd-pr


text:
grandpa: Use storage proofs for Grandpa authorities

Currently, the Grandpa protocol requires light clients to request and check execution proofs for the Substrate runtime in order to get the authority set, when in fact the executed runtime API is just a getter for a value from module storage. This is somewhat inefficient and particularly difficult to check inside of a Substrate module acting as a light client for a different chain (as is the case for bridged chains).
This PR migrates the authority set in the srml-grandpa module to a well-known storage key and replaces the execution proofs in the Grandpa finality mechanism with storage proofs. For backwards compatibility, an on-chain storage migration can be enabled in the runtime through a feature flag.
Relevant section of chat history with @rphmeier

jimpo: I've noticed that the Grandpa finality proofs that light clients check are actually execution proofs of the GrandpaApi_grandpa_authorities runtime call
rphmeier: this is an optimization I'm fairly comfortable with, but we will want to standardize the storage location
rphmeier: we'd also want to watch out for Kusama and other deployed networks; we can't just blindly make this change in the runtime.
rphmeier: probably we'd have to feature-flag out the storage location change

(generally, I'm super glad that we're switching from proof-of-execution to proof-of-read here)

worth getting this in for substrate 2.0.0 and kusama pos?

For Substrate 2.0, yes, I'd say so. However, @jimpo is working on an alternate storage proof system for the trie (more compact format; omission of value where redundant), so I don't think we could call the GRANDPA light client stable until that's integrated as well.

@rphmeier Are you OK with the versioning scheme?

authors: gavofyork jimpo rphmeier svyatonik


text:
add samp image

All Submissions:

[x] Have you ensured there aren't other open Pull Requests for the same update or change?
[x] Have you created a new branch for your changes and PR from that branch and not from your master branch?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?

I moved this image from https://quay.io/repository/parkervcp/pterodactyl-images?tab=tags&tag=game_samp to here because there is a error with it because it is to old and it does not include GLIBCXX_3.4.22

what is needed:

So I made the image myself wit hwhat I found on Quay.io and now it is in there:

i have not tested the image I just saw someone in the discord with that problem
New Image Submissions:

[x] Have you added your image to the Github workflows?
[x] Have you updated the README list to contain your new image?

The persone with the problem has confirmed this image fixt his problem, id:351916271304048650

repo: parkervcp/yolks


text:
Reworked exit codes in docker scripts

Code
What does your code do?
Reworks entrypoint.sh and run-with-retry.sh to handle exit codes better for proper exit in GitHub Action
Where should the reviewer start?

entrypoint.sh
run-with-retry.sh

Any background context you want to provide?
v1 of Action was failing due to exit code being non-zero with cloud_sql_proxy not shutting down correctly.
Checks
Pre PR

[X] tested changes

Besides the fact @DuriDuri is going to have a 🐄 about the missing newlines looks fine.

true true

authors: DuriDuri cpheps

repo: parsleyhealth/cloud-sql-proxy-goose


text:
Is it possible to add watermark ?

Hi. Is it possible add a watermark photo to video ?

Use the built-in watermark filter, you should do something like this:
FFMpeg::open('your_video.mp4')
    ->addFilter(function ($filters) {
        $filters->watermark($watermarkPath, [
            'position' => 'relative',
            'bottom' => 50,
            'right' => 50,
        ]);
    })...

Can you give an example of the $filters? They don't have it on their docs in the section you linked. Thanks.

How you can define the watermark path ?

authors: AliN11 MarcVertigo aliasgarvanak pascalbaljet

repo: pascalbaljetmedia/laravel-ffmpeg