text:
Memory Usage Much Higher in Alpine Container than on Dedicated Machine for MC Server

I'm running several modded Minecraft servers and am seeing larger than normal memory usage for them. One server is running at around 1-1.5GB RAM usage on a dedicated machine and 6GB+ in the container.

You will need to provide way more information for this to be an issue we can look into, as there is nothing in this issue I can work with. Things such as machine specs, server configuration, all of the info that was asked for when you made this issue, logs and metrics showing this usage.

This can probably be closed. I'm still doing some more testing on my end. One of the mods on the Discord asked me to submit the GitHub report, but I'm not convinced it's a Ptero issue yet.

repo: Pterodactyl/Panel


text:
[主線] [PTT] 實作使用者設定

實作細節 / Details of Implement
根據目前架構，應該分成 delivery/http usecase repository 三個子 ISSUE 下去實作。

[x] http 部分請依照文件實作並且有初步的 testcode。

ISSUE #118

[ ] usecase 部分應該是直接將 repository 的結果輸出即可。

ISSUE #148

[ ] repository 因為 go-bbs 可能還沒完成，因此需要先 mock 一個暫時性的結果等待 go-bbs 完成之後進行串接。

ISSUE #148

期程 / Schedule

討論時間： 一週, 到 2/20
實作時間： 一週, 到 2/27
確認時間： 一週, 到 3/5

相關文件 / Documents
PTT 後端系統協定

我想試試看這部分, 請指派給我

我想試試看這部分, 請指派給我

OK，那在麻煩你幫忙開立相關的子ISSUE

這部分我想是都完成了
另外, 目前users的部分好像都還沒做post, 這是之後再開其他issue處理嗎?

有也可能是排入milestone 2

source: gharchive/issue

repo: Ptt-official-app/Ptt-backend

license_type: permissive

license_source: github-api


text:
Link to A Super-simple Guide to Mechanical Ventilation... is incorrect

Hi,
The link goes to a different (tho interesting) document...
Looking forward to this guide!
Christine
A Super-simple Guide to Mechanical Ventilation For Open Source Engineers Designing Ventilators for the COVID-19 Pandemic -- This was written by me (Robert L. Read). It's my best attempt at an introduction.

Fixed: https://docs.google.com/document/d/1p-oYUv_6FadipV67g2O5yfnQvpdvZDm4WadYVubhoH4/edit?usp=sharing


text:
Double spacing on referral letter [NOIPM]

As discussed, I've attached screenshots of the double spacing issue we are seeing in Complaint Manager.
I've attached a screenshot of a complaint referral letter as seen on our end once the letter is started. I redacted all complainant information, and included red squares where we are seeing double spacing. This example uses an unknown officer, but we are seeing the double spacing in between all officer info as well (Name, Rank, Supervisor, ect.) under the "Subject NOPD Employee Info" section.
I've also attached the admin template as I see it. As discussed, this issue first came about when I edited text in the admin template (nothing related to spacing though).

oipmdoublespacingandadmintemplatescreenshots.zip

We have determined that the issue is with Handlebars.js adding extra new lines
Relevant StackOverflow: https://stackoverflow.com/questions/21713227/how-to-remove-newlines-generated-by-handlebars
Adding ~s in the correct locations fixes the issue
We just now have to figure out where to make the fix.

instance_files_noipm commit
instance_files_hawaii_csoc commit

https://app.circleci.com/pipelines/github/PublicDataWorks/complaint-manager/2520/workflows/38f2f25a-b0c0-4231-ad20-3dbc7d75a926/jobs/19790

Link for the wiki: https://sites.google.com/thoughtworks.com/tw-invisible-institute-wiki/development-resources/how-to-guides/how-to-update-the-pdf-snapshot

Verified double spacing no longer appears on complaint referral letter.
Hawaii
https://github.com/PublicDataWorks/complaint-manager/assets/134408135/52f83e66-4ac8-408d-885c-a842badd46e8
NO
https://github.com/PublicDataWorks/complaint-manager/assets/134408135/ede4fc6b-646c-448c-ae38-220e10f9853e

Verified double spacing shown above no longer appears on complaint referral letter.
NOIPM
https://github.com/PublicDataWorks/complaint-manager/assets/134408135/3141c33d-fcb6-4e57-af7a-08cfc832c36b

Received confirmation from Christian that it works.

authors: heliofreitastw mariahsager-tw mariliasoaresTW rnair-tw thoughtworks-vwong

repo: PublicDataWorks/complaint-manager


text:
Push recently configured regions to production

Overview
There have been 10 new regions configured over the past few weeks. We need to also push these to production, but should only do so after a production deploy, since there was a change implemented that's needed for Texas to work properly. I've already updated the Terraform config to bump up the number of states.
Notes from first batch of 5
INSERT INTO "region_config"("name", "country_code", "region_code", "s3_uri", "version") VALUES ('Texas','US','TX','s3://districtbuilder-production-data-us-east-1/regions/US/TX/2020-10-12T16:56:43.962Z/','2020-10-12T16:56:43.962Z');
INSERT INTO "region_config"("name", "country_code", "region_code", "s3_uri", "version") VALUES ('Ohio','US','OH','s3://districtbuilder-production-data-us-east-1/regions/US/OH/2020-10-12T15:02:03.156Z/','2020-10-12T15:02:03.156Z');
INSERT INTO "region_config"("name", "country_code", "region_code", "s3_uri", "version") VALUES ('Florida','US','FL','s3://districtbuilder-production-data-us-east-1/regions/US/FL/2020-10-12T15:31:08.657Z/','2020-10-12T15:31:08.657Z');
INSERT INTO "region_config"("name", "country_code", "region_code", "s3_uri", "version") VALUES ('Wisconsin','US','WI','s3://districtbuilder-production-data-us-east-1/regions/US/WI/2020-10-12T16:31:54.519Z/','2020-10-12T16:31:54.519Z');
INSERT INTO "region_config"("name", "country_code", "region_code", "s3_uri", "version") VALUES ('North Carolina','US','NC','s3://districtbuilder-production-data-us-east-1/regions/US/NC/2020-10-12T16:03:59.098Z/','2020-10-12T16:03:59.098Z');

Grab the ids from above and fill in the empty strings below for chambers:
INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 36, ''),
('Texas Senate', 31, ''),
('Texas House of Representatives', 150, '');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 27, ''),
('Florida Senate', 40, ''),
('Florida House of Representatives', 120, '');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 16, ''),
('Ohio Senate', 33, ''),
('Ohio House of Representatives', 99, '');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 8, ''),
('Wisconsin State Senate', 33, ''),
('Wisconsin State Assembly', 99, '');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 13, ''),
('North Carolina Senate', 50, ''),
('North Carolina House of Representatives', 120, '');

Notes from second batch of 5
INSERT INTO public.region_config (id, name, country_code, region_code, s3_uri, version) VALUES ('f11da46d-8b3a-4dca-abc7-1a89f69dc253', 'Washington', 'US', 'WA', 's3://districtbuilder-production-data-us-east-1/regions/US/WA/2020-10-22T14:34:20.488Z/', '2020-10-22 14:34:20.488+00');
INSERT INTO public.region_config (id, name, country_code, region_code, s3_uri, version) VALUES ('d2db792f-5d26-4cc6-83c2-b72515e381e9', 'New Mexico', 'US', 'NM', 's3://districtbuilder-production-data-us-east-1/regions/US/NM/2020-10-22T15:13:06.857Z/', '2020-10-22 15:13:06.857+00');
INSERT INTO public.region_config (id, name, country_code, region_code, s3_uri, version) VALUES ('35a55a23-c680-492d-87c3-d0a7e91a761e', 'West Virginia', 'US', 'WV', 's3://districtbuilder-production-data-us-east-1/regions/US/WV/2020-10-22T15:41:21.936Z/', '2020-10-22 15:41:21.936+00');
INSERT INTO public.region_config (id, name, country_code, region_code, s3_uri, version) VALUES ('633032cb-be59-4f62-b361-4e82654420f6', 'Vermont', 'US', 'VT', 's3://districtbuilder-production-data-us-east-1/regions/US/VT/2020-10-22T16:02:10.920Z/', '2020-10-22 16:02:10.92+00');
INSERT INTO public.region_config (id, name, country_code, region_code, s3_uri, version) VALUES ('42dd1dbe-5771-48de-94f0-5d92fcc21a62', 'Alaska', 'US', 'AK', 's3://districtbuilder-production-data-us-east-1/regions/US/AK/2020-10-22T16:17:16.539Z/', '2020-10-22 16:17:16.539+00');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 10, 'f11da46d-8b3a-4dca-abc7-1a89f69dc253'),
('Washington State Senate', 49, 'f11da46d-8b3a-4dca-abc7-1a89f69dc253'),
('Washington House of Representatives', 98, 'f11da46d-8b3a-4dca-abc7-1a89f69dc253');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 3, 'd2db792f-5d26-4cc6-83c2-b72515e381e9'),
('New Mexico Senate', 42, 'd2db792f-5d26-4cc6-83c2-b72515e381e9'),
('New Mexico House of Representatives', 70, 'd2db792f-5d26-4cc6-83c2-b72515e381e9');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('US House of Representatives', 3, '35a55a23-c680-492d-87c3-d0a7e91a761e'),
('West Virginia Senate', 17, '35a55a23-c680-492d-87c3-d0a7e91a761e'),
('West Virginia House of Delegates', 67, '35a55a23-c680-492d-87c3-d0a7e91a761e');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('Vermont Senate', 13, '633032cb-be59-4f62-b361-4e82654420f6'),
('Vermont House of Representatives', 104, '633032cb-be59-4f62-b361-4e82654420f6');

INSERT INTO chamber(name, number_of_districts, region_config_id) VALUES
('Alaska Senate', 20, '42dd1dbe-5771-48de-94f0-5d92fcc21a62'),
('Alaska House of Representatives', 40, '42dd1dbe-5771-48de-94f0-5d92fcc21a62');

Acceptance criteria
All 10 new regions are available in production

We should also do Dane County now, after the deploy, remembering to set the hidden flag to true so it isn't visible in the states dropdown.

For Dane County:
INSERT INTO "region_config"("name", "country_code", "region_code", "s3_uri", "version") VALUES ('Dane County','US','WI','s3://districtbuilder-production-data-us-east-1/regions/US/WI/2020-10-15T17:48:30.686Z');

No chambers are needed, since projects won't be created in the usual way.

repo: PublicMapping/districtbuilder


text:
Help chat plugin

Task:                     To create a chat facility and add it to Helm-PHR-Theme plugin.
Related issue:       https://github.com/LeedsCC/Helm_PHR/issues/3  -  Add chat facility to Helm for trial
Use as reference:  https://www.click4assistance.co.uk/
ToolScript.txt

not top priority, tackle this in context of Helm reBranding Sprint please

@tony-shannon got it, thank you

Duplicate of https://github.com/LeedsCC/Helm_PHR/issues/3

authors: PhilBarrett kuvakina tony-shannon

repo: PulseTile/PulseTile-React-Core


text:
Deadline for Final Report

@Willian-Zhang  Add your "explore new station" part to final project by 9PM Dec 14, so that we can have some time to compile the final version before midnight. The final due is 9AM Dec 15.
My part is available at latest commit on master branch: https://github.com/Puriney/ineedabike/tree/a04e78a2d1405f1925fd750e9b67770680a40f2c

I know you have big data to be processed. Here is suggested steps for you to follow to avoid uploading large file to GitHub:

Follow the instruction on the html report to download the data required by my part which is a 7MB zip which would be a csv (44.5MB) once unzipped.
Compile the RMD to see whether your environment works. It takes 1-2 minutes on my old computer.
Add your part to RMD.
Compile the RMD to generate the html.

I suggest you write bash file similar to my get_data.sh to leave the burden of downloading data to TA. We just need the compiled html.

I'd commit my previous build 49c04f7 for now, and leave the final one to 12 Dec late PM / 13 Dev late PM

Add some paragraph to elaborate on your data analysis, instead of codes and figures only. I will not wait for your part by 9 PM today.  I did not see your updates as you planed on either Dec 12 or Dec 13. Tomorrow morning is the due.

Sorry for making a mistake on dates,
Good news is
I'm working on it.

authors: Puriney Willian-Zhang

repo: Puriney/ineedabike

license_source: bigquery


text:
fix incorrect name of finder file menu item

It should be like this:

After the changes:

Hi Deep. Thank you for the PR 🙏

source: gharchive/pull-request

authors: Deep-Codes PuruVJ

repo: PuruVJ/macos-web

license: MIT


text:
Added - Validating password by taking input from user.

Pull Request Template

Please Fill all the details. It will help us to track issues and Pull Request.

What have you Changed -
Issue no.(must) - #29
Self Check(Tick After Making pull Request)

[ ] This issue was assigned to me
[x] One Change in one Pull Request
[ ] My file is in proper folder
[ ] I am following clean code and Documentation.
[ ] I have added README.md with my script

OK, thanks.

repo: Py-Contributors/awesomeScripts


text:
Offline HTML (1st iteration)

Another attempt to improve offline documentation, now by building HTML without external dependences (see #347).
The README images were redrawn/recreated, and building downloadable zipped HTML on RTD is enabled. Now this should work on RTD, and the README should look fine on GitHub. However, since PyPI doesn't support relative paths for images, after this PR is merged into master, another PR will be needed to change these paths to their absolute URLs.
Two other experimental changes:

Sphinx relies on MathJax to render TeX  markup, and the problem is that by default MathJax scripts are loaded from jsDelivr CDN, so they will not work offline. Thus the docs recommend downloading and using a local copy. Here I'm trying to do so automatically (the full MathJax script is ~2 MB, so just including it in our repository wouldn't be great, regardless of its license).
We are using “latest DOI” from Zenodo everywhere, but in several places it would be more appropriate to use DOI for specific versions, especially since RTD keeps the docs for several previous releases. I couldn't find any clues how to retrieve such DOI from Zenodo by simple means, so here is also another script that tries to do so through their API.

Let's see how it all works...

Hello @MikhailRyazanov! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file doc/conf/mathjax.py:

Line 8:80: E501 line too long (80 > 79 characters)

OK, all pieces seem to work so far (except that RTD doesn't build for PRs not only PDF but even zipped HTML).

repo: PyAbel/PyAbel

license: mit


text:
Bandit for jenkins.

How to integrate bandint into jenkins?
I propose to develop an official plugin. Very necessary thing for jenkins

Duplicate of https://github.com/PyCQA/bandit/issues/220

authors: CheckB ericwb

repo: PyCQA/bandit


text:
Impossible to download Earth-scale data

Description

While improving PyPSA-Africa towards PyPSA-Earth, I've noticed that when I try to create a cutout using atlite, atlite returns an error because copernicus is not able to convert grib files into netCDF format, as requested by the atlite package, and the workflow crashes.
Expected Behavior

Atlite shall be able to derive the desiderable cutout for the Earth.
The error message below can be reproduced by using the following code:
import atlite
cutout = atlite.Cutout(path="world_2013.nc",
                       module="era5",
                       x=slice(-179.4, 179.7),
                       y=slice(-59.1, 87.3),
                       time="2013"
                       )
cutout.prepare()

Actual Behavior

The workflow stops because Atlite is not able to successfully download the data from copernicus.
The raw grib files seem to be procuced but they cannot be successfully converted into netCDF format, hence the procedure stops.
When creating the cutout for the world exactly :"area": [87.3, -179.4, -59.1, 179.7] in copernicus.
Error Message
The request you have submitted is not valid

Reason:  grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints Cannot create netCDF classic format, dataset is too large! Try splitting the input GRIB(s). grib_to_netcdf: Version 2.24.2 grib_to_netcdf: Processing input file '/cache/tmp/8599e6f4-dc3e-423e-a66e-72e4d44ac365-adaptor.mars.internal-1645467199.6672807-10228-8-tmp.grib'. grib_to_netcdf: Found 17520 GRIB fields in 1 file. grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate grib_to_netcdf: Creating netCDF file '/cache/data3/adaptor.mars.internal-1645468582.836742-10228-15-8599e6f4-dc3e-423e-a66e-72e4d44ac365.nc' grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $ grib_to_netcdf: Creating large (64 bit) file format. grib_to_netcdf: Defining variable 't2m'. grib_to_netcdf: Defining variable 'stl4'.

Your Environment

The atlite version used: 0.2.5
How you installed atlite (conda, pip or github): conda
Operating System: Linux
My environment: CentOS Linux
(output of `conda list`)
    ```
    # Name                    Version                   Build  Channel

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
affine                    2.3.0                      py_0    conda-forge
alsa-lib                  1.2.3                h516909a_0    conda-forge
ampl-mp                   3.1.0             h2cc385e_1005    conda-forge
amply                     0.1.4                      py_0    conda-forge
anyio                     3.4.0            py39hf3d152e_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               21.1.0           py39h3811e60_2    conda-forge
async_generator           1.10                       py_0    conda-forge
atlite                    0.2.5              pyhd8ed1ab_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.10.0             pyha770c72_0    conda-forge
bleach                    4.1.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.0               h9c3ff4c_0    conda-forge
bokeh                     2.4.2            py39hf3d152e_0    conda-forge
boost-cpp                 1.74.0               h312852a_4    conda-forge
bottleneck                1.3.2            py39hce5d2b2_5    conda-forge
branca                    0.4.2              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h7f98852_6    conda-forge
brotli-bin                1.0.9                h7f98852_6    conda-forge
brotlipy                  0.7.0           py39h3811e60_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
cartopy                   0.20.1           py39ha48bd0d_1    conda-forge
cdsapi                    0.5.1              pyhd8ed1ab_0    conda-forge
certifi                   2021.10.8        py39hf3d152e_1    conda-forge
cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
cftime                    1.5.1.1          py39hce5d2b2_1    conda-forge
charset-normalizer        2.0.8              pyhd8ed1ab_0    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge
coincbc                   2.10.5               hcee13e7_1    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
colorcet                  2.0.6              pyhd8ed1ab_0    conda-forge
conda                     4.11.0           py39hf3d152e_0    conda-forge
conda-package-handling    1.7.3            py39h3811e60_1    conda-forge
configargparse            1.5.3              pyhd8ed1ab_0    conda-forge
connection_pool           0.0.3              pyhd3deb0d_0    conda-forge
contextily                1.2.0              pyhd8ed1ab_0    conda-forge
country_converter         0.7.4              pyhd8ed1ab_0    conda-forge
countrycode               0.2                      pypi_0    pypi
cryptography              36.0.0           py39h95dcef6_0    conda-forge
curl                      7.80.0               h2574ce0_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.11.2           py39h3811e60_1    conda-forge
dask                      2021.3.1           pyhd8ed1ab_0    conda-forge
dask-core                 2021.3.1           pyhd8ed1ab_0    conda-forge
datashader                0.13.0             pyh6c4a22f_0    conda-forge
datashape                 0.5.4                      py_1    conda-forge
datrie                    0.8.2            py39h3811e60_3    conda-forge
dbus                      1.13.6               h48d8840_2    conda-forge
debugpy                   1.5.1            py39he80948d_0    conda-forge
decorator                 5.1.0              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
deprecation               2.1.0              pyh9f0ad1d_0    conda-forge
descartes                 1.1.0                      py_4    conda-forge
distlib                   0.3.3              pyhd8ed1ab_0    conda-forge
distributed               2021.4.1         py39hf3d152e_1    conda-forge
docutils                  0.18.1           py39hf3d152e_0    conda-forge
editdistance-s            1.0.0            py39h1a9c180_2    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
entsoe-py                 0.3.7              pyhd8ed1ab_0    conda-forge
esy-osm-pbf               0.0.1                    pypi_0    pypi
esy-osmfilter             1.0.11                   pypi_0    pypi
et_xmlfile                1.0.1                   py_1001    conda-forge
expat                     2.4.1                h9c3ff4c_0    conda-forge
filelock                  3.4.0              pyhd8ed1ab_0    conda-forge
fiona                     1.8.20           py39h427c1bf_2    conda-forge
folium                    0.12.1.post1       pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.28.2           py39h3811e60_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.6                h7f98852_0    conda-forge
fsspec                    2021.11.0          pyhd8ed1ab_0    conda-forge
gdal                      3.3.3            py39h0494519_0    conda-forge
gdrive                    2.1.0                haa36a5b_0    conda-forge
geographiclib             1.52               pyhd8ed1ab_0    conda-forge
geopandas                 0.10.2             pyhd8ed1ab_0    conda-forge
geopandas-base            0.10.2             pyha770c72_0    conda-forge
geoplot                   0.4.4              pyhd8ed1ab_0    conda-forge
geopy                     2.2.0              pyhd8ed1ab_0    conda-forge
geos                      3.9.1                h9c3ff4c_2    conda-forge
geotiff                   1.7.0                hcfb7246_3    conda-forge
geoviews                  1.9.2              pyhd8ed1ab_0    conda-forge
geoviews-core             1.9.2              pyha770c72_0    conda-forge
gettext                   0.19.8.1          h73d1719_1008    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
gitdb                     4.0.9              pyhd8ed1ab_0    conda-forge
gitpython                 3.1.24             pyhd8ed1ab_0    conda-forge
glib                      2.70.1               h780b84a_0    conda-forge
glib-tools                2.70.1               h780b84a_0    conda-forge
glpk                      5.0                  h445213a_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
googledrivedownloader     0.4                pyhd3deb0d_1    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gst-plugins-base          1.18.5               hf529b03_2    conda-forge
gstreamer                 1.18.5               h9f60fe5_2    conda-forge
harfbuzz                  2.9.1                h83ec7ef_1    conda-forge
hdf4                      4.2.15               h10796ff_3    conda-forge
hdf5                      1.12.1          nompi_h2750804_102    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
holoviews                 1.14.6             pyhd8ed1ab_0    conda-forge
hvplot                    0.7.3              pyh6c4a22f_0    conda-forge
icu                       68.2                 h9c3ff4c_0    conda-forge
identify                  2.3.7              pyhd8ed1ab_0    conda-forge
idna                      3.1                pyhd3deb0d_0    conda-forge
importlib-metadata        4.8.2            py39hf3d152e_0    conda-forge
importlib_metadata        4.8.2                hd8ed1ab_0    conda-forge
importlib_resources       5.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipopt                     3.14.4               h7ede334_0    conda-forge
ipykernel                 6.5.0            py39hef51801_1    conda-forge
ipython                   7.29.0           py39hef51801_2    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jedi                      0.18.1           py39hf3d152e_0    conda-forge
jinja2                    3.0.3              pyhd8ed1ab_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
json-c                    0.15                 h98cffda_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                4.2.1              pyhd8ed1ab_0    conda-forge
jupyter_client            7.1.0              pyhd8ed1ab_0    conda-forge
jupyter_core              4.9.1            py39hf3d152e_1    conda-forge
jupyter_server            1.12.0             pyhd8ed1ab_0    conda-forge
jupyterlab                3.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.8.2              pyhd8ed1ab_0    conda-forge
kealib                    1.4.14               h87e4c3c_3    conda-forge
kiwisolver                1.3.2            py39h1a9c180_1    conda-forge
krb5                      1.19.2               hcc1bbae_3    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libarchive                3.5.2                hccf745f_1    conda-forge
libblas                   3.9.0           12_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h7f98852_6    conda-forge
libbrotlidec              1.0.9                h7f98852_6    conda-forge
libbrotlienc              1.0.9                h7f98852_6    conda-forge
libcblas                  3.9.0           12_linux64_openblas    conda-forge
libclang                  11.1.0          default_ha53f305_1    conda-forge
libcurl                   7.80.0               h2574ce0_0    conda-forge
libdap4                   3.20.6               hd7c4107_2    conda-forge
libdeflate                1.8                  h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgdal                   3.3.3                h356f897_0    conda-forge
libgfortran-ng            11.2.0              h69a702a_11    conda-forge
libgfortran5              11.2.0              h5c6108e_11    conda-forge
libglib                   2.70.1               h174f98d_0    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libkml                    1.3.0             h238a007_1014    conda-forge
liblapack                 3.9.0           12_linux64_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm11                 11.1.0               hf817b99_2    conda-forge
libmamba                  0.18.2               h3985d26_0    conda-forge
libmambapy                0.18.2           py39h8bfa403_0    conda-forge
libnetcdf                 4.8.1           nompi_hb3fd0d9_101    conda-forge
libnghttp2                1.43.0               h812cca2_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.5                 hd57d9b9_0    conda-forge
librttopo                 1.1.0                h1185371_6    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.19               h780b84a_5    conda-forge
libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
libspatialite             5.0.1                h8796b1e_9    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
libtiff                   4.3.0                h6f004c6_2    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.2.1                h7f98852_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h72842e0_0    conda-forge
libxslt                   1.1.33               h15afd5d_2    conda-forge
libzip                    1.8.0                h4de3113_1    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
llvmlite                  0.36.0           py39h1bbdace_0    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.6.4            py39h107f48f_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     0.18.2           py39hfa8f2c8_0    conda-forge
mapclassify               2.4.3              pyhd8ed1ab_0    conda-forge
markdown                  3.3.6              pyhd8ed1ab_0    conda-forge
markupsafe                2.0.1            py39h3811e60_1    conda-forge
matplotlib                3.5.0            py39hf3d152e_0    conda-forge
matplotlib-base           3.5.0            py39h2fa2bec_0    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
memory_profiler           0.58.0                     py_0    conda-forge
mercantile                1.2.1              pyhd8ed1ab_0    conda-forge
metis                     5.1.0             h58526e2_1006    conda-forge
mistune                   0.8.4           py39h3811e60_1005    conda-forge
mock                      4.0.3            py39hf3d152e_2    conda-forge
more-itertools            8.12.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.3            py39h1a9c180_0    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
mumps-include             5.2.1               ha770c72_10    conda-forge
mumps-seq                 5.2.1               h47a8eb5_10    conda-forge
munch                     2.5.0                      py_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.27               ha770c72_1    conda-forge
mysql-libs                8.0.27               hfa10184_1    conda-forge
nbclassic                 0.3.4              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.9              pyhd8ed1ab_0    conda-forge
nbconvert                 6.3.0            py39hf3d152e_1    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
netcdf4                   1.5.8           nompi_py39h64b754b_101    conda-forge
networkx                  2.6.3              pyhd8ed1ab_1    conda-forge
nodeenv                   1.6.0              pyhd8ed1ab_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
notebook                  6.4.6              pyha770c72_0    conda-forge
nspr                      4.32                 h9c3ff4c_1    conda-forge
nss                       3.72                 hb5efdd6_0    conda-forge
numba                     0.53.1           py39h56b8d98_1    conda-forge
numexpr                   2.7.3           py39hbd72853_102    conda-forge
numpy                     1.21.4           py39hdbf815f_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjdk                   11.0.9.1             h5cc2fde_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openpyxl                  3.0.9              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.4            py39hde0f152_1    conda-forge
pandoc                    2.16.2               h7f98852_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
panel                     0.12.4             pyhd8ed1ab_0    conda-forge
param                     1.12.0             pyh6c4a22f_0    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.4.0            py39ha612740_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pluggy                    1.0.0            py39hf3d152e_2    conda-forge
ply                       3.11                       py_1    conda-forge
poppler                   21.09.0              ha39eefc_3    conda-forge
poppler-data              0.4.11               hd8ed1ab_0    conda-forge
postgresql                13.5                 h2510834_0    conda-forge
powerplantmatching        0.4.8              pyhd8ed1ab_0    conda-forge
pre-commit                2.15.0           py39hf3d152e_1    conda-forge
progressbar2              3.55.0             pyhd8ed1ab_0    conda-forge
proj                      8.1.1                h277dcde_2    conda-forge
prometheus_client         0.12.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.22             pyha770c72_0    conda-forge
protobuf                  3.19.1                   pypi_0    pypi
psutil                    5.8.0            py39h3811e60_2    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulp                      2.5.1            py39hf3d152e_1    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.3           py39h3811e60_1009    conda-forge
pycountry                 20.7.3             pyh9f0ad1d_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pygments                  2.10.0             pyhd8ed1ab_0    conda-forge
pyomo                     6.2              py39he80948d_0    conda-forge
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.6              pyhd8ed1ab_0    conda-forge
pyproj                    3.2.1            py39ha81a305_2    conda-forge
pypsa                     0.18.1             pyhd8ed1ab_0    conda-forge
pyqt                      5.12.3           py39hf3d152e_8    conda-forge
pyqt-impl                 5.12.3           py39hde8b62d_8    conda-forge
pyqt5-sip                 4.19.18          py39he80948d_8    conda-forge
pyqtchart                 5.12             py39h0fcd23e_8    conda-forge
pyqtwebengine             5.12.1           py39h0fcd23e_8    conda-forge
pyrsistent                0.18.0           py39h3811e60_0    conda-forge
pyshp                     2.1.3              pyh44b312d_0    conda-forge
pysocks                   1.7.1            py39hf3d152e_4    conda-forge
pytables                  3.6.1            py39h2669a42_4    conda-forge
pytest                    6.2.5            py39hf3d152e_1    conda-forge
python                    3.9.7           hb7a2778_3_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-utils              2.5.6              pyh44b312d_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pyviz_comms               2.1.0              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py39h3811e60_3    conda-forge
pyzmq                     22.3.0           py39h37b5a0c_1    conda-forge
qt                        5.12.9               hda022c4_4    conda-forge
rasterio                  1.2.10           py39hb37810a_0    conda-forge
ratelimiter               1.2.0                   py_1002    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
reproc                    14.2.3               h7f98852_0    conda-forge
reproc-cpp                14.2.3               h9c3ff4c_0    conda-forge
requests                  2.26.0             pyhd8ed1ab_1    conda-forge
rioxarray                 0.9.0              pyhd8ed1ab_0    conda-forge
rtree                     0.9.7            py39hb102c33_3    conda-forge
ruamel_yaml               0.15.80         py39h3811e60_1006    conda-forge
scikit-learn              1.0.1            py39h4dfa638_2    conda-forge
scipy                     1.7.3            py39hee8e79c_0    conda-forge
scotch                    6.0.9                h3858553_1    conda-forge
seaborn                   0.11.2               hd8ed1ab_0    conda-forge
seaborn-base              0.11.2             pyhd8ed1ab_0    conda-forge
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                59.2.0           py39hf3d152e_0    conda-forge
shapely                   1.8rc1           py39h2226dc9_0    conda-forge/label/shapely_dev
six                       1.16.0             pyh6c4a22f_0    conda-forge
sklearn                   0.0                      pypi_0    pypi
smart_open                5.2.1              pyhd8ed1ab_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snakemake-minimal         6.11.0             pyhdfd78af_0    bioconda
sniffio                   1.2.0            py39hf3d152e_2    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3                pyhd8ed1ab_0    conda-forge
sqlite                    3.36.0               h9cd32fc_2    conda-forge
statsmodels               0.13.1           py39hce5d2b2_0    conda-forge
stopit                    1.1.2                      py_0    conda-forge
tabulate                  0.8.9              pyhd8ed1ab_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.12.1           py39hf3d152e_1    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.0.0              pyh8a188c0_0    conda-forge
tiledb                    2.3.4                he87e0bf_0    conda-forge
tk                        8.6.11               h27826a3_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
toposort                  1.7                pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39h3811e60_2    conda-forge
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
tsam                      1.1.2                    pypi_0    pypi
typing_extensions         4.0.0              pyha770c72_0    conda-forge
tzcode                    2021e                h7f98852_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
virtualenv                20.4.7           py39hf3d152e_1    conda-forge
vresutils                 0.3.1                    pypi_0    pypi
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.2.1            py39hf3d152e_0    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
wrapt                     1.13.3           py39h3811e60_1    conda-forge
xarray                    0.20.2.dev19+g21e84842          pypi_0    pypi
xerces-c                  3.2.3                h9d8b166_3    conda-forge
xlrd                      2.0.1              pyhd8ed1ab_3    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-libxtst              1.2.3             h7f98852_1002    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xyzservices               2021.11.0          pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yaml-cpp                  0.6.3                he1b5a44_4    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zict                      2.0.0                      py_0    conda-forge
zipp                      3.6.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge
```

Can reproduce. From a first glance this might be an issue on the CDS API side. classic  netCDF is limited in size to ~2 GB, maybe the individual requests are exceeding this size and the api does not internally switch to netCDF v4 (which allows for larger sizes)? We'd need to investigate a bit more, maybe someone else has a clearer view?
Full stack trace:
2022-02-22 08:15:33,413 INFO Requesting data for feature influx...
2022-02-22 08:23:52,061 INFO CDS: Downloading variables
	 * geopotential (2013)

2022-02-22 08:59:55,175 ERROR Message: the request you have submitted is not valid
2022-02-22 08:59:55,176 ERROR Reason:
grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints

Cannot create netCDF classic format, dataset is too large!
Try splitting the input GRIB(s).
grib_to_netcdf: Version 2.24.2
grib_to_netcdf: Processing input file '/cache/tmp/74fcce81-8102-4388-b11b-f71e8c4d56a3-adaptor.mars.internal-1645514525.1224709-4276-8-tmp.grib'.
grib_to_netcdf: Found 17520 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '/cache/data5/adaptor.mars.internal-1645515842.3469903-4276-6-74fcce81-8102-4388-b11b-f71e8c4d56a3.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Defining variable 'stl4'.

2022-02-22 08:59:55,177 ERROR   Traceback (most recent call last):
2022-02-22 08:59:55,178 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 55, in handle_request
2022-02-22 08:59:55,179 ERROR       result = cached(context.method, proc, context, context.args, context.kwargs)
2022-02-22 08:59:55,179 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
2022-02-22 08:59:55,180 ERROR       result = proc(context, *context.args, **context.kwargs)
2022-02-22 08:59:55,180 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
2022-02-22 08:59:55,180 ERROR       return p(*args, **kwargs)
2022-02-22 08:59:55,181 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
2022-02-22 08:59:55,181 ERROR       return self.proc(context, *args, **kwargs)
2022-02-22 08:59:55,181 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 47, in internal
2022-02-22 08:59:55,181 ERROR       return mars(context, request, **kwargs)
2022-02-22 08:59:55,182 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 25, in mars
2022-02-22 08:59:55,182 ERROR       grib_to_netcdf(context, requests, info)
2022-02-22 08:59:55,182 ERROR     File "/home/cds/cdsservices/services/mars/grib_to_netcdf.py", line 42, in grib_to_netcdf
2022-02-22 08:59:55,183 ERROR       context.run_command(*cmd, exception=NetcdfException)
2022-02-22 08:59:55,183 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/context.py", line 209, in run_command
2022-02-22 08:59:55,184 ERROR       raise exception(call, proc.returncode, output)
2022-02-22 08:59:55,184 ERROR   home.cds.cdsservices.services.mars.__init__.py.exceptions.NetcdfException:
2022-02-22 08:59:55,185 ERROR   grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Input In [6], in <module>
----> 1 cutout.prepare()

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:102, in maybe_remove_tmpdir.<locals>.wrapper(*args, **kwargs)
    100 kwargs["tmpdir"] = mkdtemp()
    101 try:
--> 102     res = func(*args, **kwargs)
    103 finally:
    104     rmtree(kwargs["tmpdir"])

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:164, in cutout_prepare(cutout, features, tmpdir, overwrite)
    162 logger.info(f"Calculating and writing with module {module}:")
    163 missing_features = missing_vars.index.unique("feature")
--> 164 ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
    165 prepared |= set(missing_features)
    167 cutout.data.attrs.update(dict(prepared_features=list(prepared)))

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:46, in get_features(cutout, module, features, tmpdir)
     41     feature_data = delayed(get_data)(
     42         cutout, feature, tmpdir=tmpdir, lock=lock, **parameters
     43     )
     44     datasets.append(feature_data)
---> 46 datasets = compute(*datasets)
     48 ds = xr.merge(datasets, compat="equals")
     49 for v in ds:

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/base.py:571, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    568     keys.append(x.__dask_keys__())
    569     postcomputes.append(x.__dask_postcompute__())
--> 571 results = schedule(dsk, keys, **kwargs)
    572 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/threaded.py:79, in get(dsk, result, cache, num_workers, pool, **kwargs)
     76     elif isinstance(pool, multiprocessing.pool.Pool):
     77         pool = MultiprocessingPoolExecutor(pool)
---> 79 results = get_async(
     80     pool.submit,
     81     pool._max_workers,
     82     dsk,
     83     result,
     84     cache=cache,
     85     get_id=_thread_get_id,
     86     pack_exception=pack_exception,
     87     **kwargs,
     88 )
     90 # Cleanup pools associated to dead threads
     91 with pools_lock:

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:507, in get_async(submit, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, chunksize, **kwargs)
    505         _execute_task(task, data)  # Re-execute locally
    506     else:
--> 507         raise_exception(exc, tb)
    508 res, worker_id = loads(res_info)
    509 state["cache"][key] = res

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:315, in reraise(exc, tb)
    313 if exc.__traceback__ is not tb:
    314     raise exc.with_traceback(tb)
--> 315 raise exc

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:220, in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
    218 try:
    219     task, data = loads(task_info)
--> 220     result = _execute_task(task, data)
    221     id = get_id()
    222     result = dumps((result, id))

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/core.py:119, in _execute_task(arg, cache, dsk)
    115     func, args = arg[0], arg[1:]
    116     # Note: Don't assign the subtask results to a variable. numpy detects
    117     # temporaries by their reference count and can execute certain
    118     # operations in-place.
--> 119     return func(*(_execute_task(a, cache) for a in args))
    120 elif not ishashable(arg):
    121     return arg

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/utils.py:40, in apply(func, args, kwargs)
     38 def apply(func, args, kwargs=None):
     39     if kwargs:
---> 40         return func(*args, **kwargs)
     41     else:
     42         return func(*args)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:378, in get_data(cutout, feature, tmpdir, lock, **creation_parameters)
    374     return retrieve_once(retrieval_times(coords, static=True)).squeeze()
    376 datasets = map(retrieve_once, retrieval_times(coords))
--> 378 return xr.concat(datasets, dim="time").sel(time=coords["time"])

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/xarray/core/concat.py:220, in concat(objs, dim, data_vars, coords, compat, positions, fill_value, join, combine_attrs)
    217 from .dataset import Dataset
    219 try:
--> 220     first_obj, objs = utils.peek_at(objs)
    221 except StopIteration:
    222     raise ValueError("must supply at least one object to concatenate")

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/xarray/core/utils.py:196, in peek_at(iterable)
    192 """Returns the first value from iterable, as well as a new iterator with
    193 the same content as the original iterable
    194 """
    195 gen = iter(iterable)
--> 196 peek = next(gen)
    197 return peek, itertools.chain([peek], gen)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:368, in get_data.<locals>.retrieve_once(time)
    367 def retrieve_once(time):
--> 368     ds = func({**retrieval_params, **time})
    369     if sanitize and sanitize_func is not None:
    370         ds = sanitize_func(ds)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:192, in get_data_temperature(retrieval_params)
    190 def get_data_temperature(retrieval_params):
    191     """Get wind temperature for given retrieval parameters."""
--> 192     ds = retrieve_data(
    193         variable=["2m_temperature", "soil_temperature_level_4"], **retrieval_params
    194     )
    196     ds = _rename_and_clean_coords(ds)
    197     ds = ds.rename({"t2m": "temperature", "stl4": "soil temperature"})

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:301, in retrieve_data(product, chunks, tmpdir, lock, **updates)
    294 assert {"year", "month", "variable"}.issubset(
    295     request
    296 ), "Need to specify at least 'variable', 'year' and 'month'"
    298 client = cdsapi.Client(
    299     info_callback=logger.debug, debug=logging.DEBUG >= logging.root.level
    300 )
--> 301 result = client.retrieve(product, request)
    303 if lock is None:
    304     lock = nullcontext()

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/cdsapi/api.py:348, in Client.retrieve(self, name, request, target)
    347 def retrieve(self, name, request, target=None):
--> 348     result = self._api("%s/resources/%s" % (self.url, name), request, "POST")
    349     if target is not None:
    350         result.download(target)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/cdsapi/api.py:506, in Client._api(self, url, request, method)
    504             break
    505         self.error("  %s", n)
--> 506     raise Exception(
    507         "%s. %s."
    508         % (reply["error"].get("message"), reply["error"].get("reason"))
    509     )
    511 raise Exception("Unknown API state [%s]" % (reply["state"],))

Exception: the request you have submitted is not valid.
grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints

Cannot create netCDF classic format, dataset is too large!
Try splitting the input GRIB(s).
grib_to_netcdf: Version 2.24.2
grib_to_netcdf: Processing input file '/cache/tmp/74fcce81-8102-4388-b11b-f71e8c4d56a3-adaptor.mars.internal-1645514525.1224709-4276-8-tmp.grib'.
grib_to_netcdf: Found 17520 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '/cache/data5/adaptor.mars.internal-1645515842.3469903-4276-6-74fcce81-8102-4388-b11b-f71e8c4d56a3.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Defining variable 'stl4'.
.

I'm wondering whether the NC format is needed for atlite.
A possible solution could be to use the raw grib format when downloading data from copernicus, xarray should be able to load data accordingly, not sure about the format that may need some adaptations.
https://stackoverflow.com/questions/64937550/converting-grib-to-netcdf4

@euronion Super! thank you very much. Currently, I am a bit busy with other stuff and I cannot run the machine with copernicus waiting long time for the analysis, unfortunately. As I have free resources, I'll test that.
Thank you!

authors: davide-f euronion

repo: PyPSA/atlite


text:
Prepare data for pathway optimisation with sector coupling

Heating

Today's split of heating technologies in each country: district heating / gas boiler / oil boiler (JRC IDEES in "residential" and "tertiary")
Approximations of age distribution of heating infrastructure (assumption e.g. 25% 5 years old, 25% 10 years old, etc.)
Capital costs and fuel costs over time

Transport

Number of vehicles in each country (JRC IDEES)
Distribution of their ages, or make assumption (e.g. 25% 5 years old, etc.), could be different in different countries
EITHER cost prediction for EVs over time (e.g. from Bloomberg NEF or Fraunhofer IEE or EnergyPLAN) OR exogenous share of EV over time

Industry

Costs of different production pathways, e.g. blast furnace versus DRI + electric furnace, to determine substitution pathway (Christian Breyer has researched some of these costs) OR exogenous pathway

Other studies
Fraunhofer IEE (formerly IWES) has cost assumptions over time for vehicles, heating, etc., as does Palzer PhD thesis, see e.g. references in Synergies of Sector Coupling paper, e.g.
http://www.energiesystemtechnik.iwes.fraunhofer.de/de/projekte/suche/laufende/interaktion_strom_waerme_verkehr.html
has diesel versus petrol versus EV for DE until 2050, see e.g. Figure 0-9, costs in Table 10-35 (beware battery costs too high since study is old).

For the squashing of assets with the same attributes to reduce the number of optimization variables, my suggestion would be:

Prepare the prenetwork-brownfield as normal
Have a script that aggregates similar components before solving to prenetwork-brownfield-aggregated
Solve the aggregated prenetwork to postnetwork-aggregated
Have a script that disaggregates postnetwork-aggregated to postnetwork (which as the same components as prenetwork-brownfield)

For the aggregation, e.g. all generators with the same bus, marginal cost, efficiency, p_min_pu and p_max_pu would be aggregated to a single representative. The existing p_nom's would be added up. If one of the generators is p_nom_extendable, then the representative is also p_nom_extendable.
For the disaggregation, all the new capacity goes to the generator with p_nom_extendable (if more than one is p_nom_extendable, it can be split). The dispatch n.generators_t.p is split to the generators proportional to p_nom.
Here's code for identifying the mapping from generators to representative aggregated generators:
#attributes that should match
attributes = {"Link" : pd.Index(["marginal_cost","p_max_pu","p_min_pu"]).append(n.links.columns[n.links.columns.str.contains("efficiency")]).append(n.links.columns[n.links.columns.str.contains("bus")]),
              "Generator" : pd.Index(["marginal_cost","bus","p_max_pu","p_min_pu","efficiency"])}

for c in n.iterate_components(attributes.keys()):

    #store the mapping to representatives here
    mapping[c.name] = pd.Series()
    selection = c.df[attributes[c.name]]

    #to compare time-dependent, just check the average
    for attribute in attributes[c.name]:
        if attribute in c.pnl and not c.pnl[attribute].empty:
            selection.loc[c.pnl[attribute].columns,attribute + "_t"] = c.pnl[attribute].mean()

    for i in selection.index:
        found = False
        for j in mapping[c.name].index:
            if i[:-4] != j[:-4]:
                continue
            if selection.loc[i].equals(selection.loc[j]):
                print(i,"is same as",j)
                found = True
                break
        if not found:
            mapping[c.name][i] = i
        else:
            mapping[c.name][i] = j

Then for the mapping[c.name].unique() values, drop those elements that aggregate to them and add up their p_nom. Check for p_nom_exendable.

repo: PyPSA/pypsa-eur-sec


text:
Tone detector for syllables - Determine Thai tone rule for syllable?

Hello! How could I determine tone rule for syllable? Thank you

Determine a tone of a syllable from its consonants, vowels, and tone marks.
Interesting enhancement.

+1 haven't found any solution so far

Now, It is work-in-prcessing.

Now, It is work-in-processing.

Maybe, we can use SpellWord's _find_tone() and _find_tone_info() here:
https://github.com/cakimpei/khanaa/blob/main/src/khanaa/thai_spelling.py#L310

Maybe, we can use SpellWord's _find_tone() and _find_tone_info() here:
https://github.com/cakimpei/khanaa/blob/main/src/khanaa/thai_spelling.py#L310

It is word-level. I think it should have error from word-level and the code is hard to understand.

You can try the tone detector for Thai word/syllable by https://github.com/PyThaiNLP/pythainlp/pull/690.

authors: alexgolod bact korakot sferrka wannaphong

repo: PyThaiNLP/pythainlp


text:
1.0.0rc4 Save to TorchScript self.eval() device error

🐛 Bug
Likely self.eval() sends the model to the CPU, resulting in saving to a TorchScript file fails.
To Reproduce
Run the code sample below.
Code sample

Script can also be downloaded from: https://gist.github.com/NumesSanguis/388b4cfab2a8945afa85e8b79cd0c794
Most relevant code (extended to_torchscript to support trace):
# ...

    # use TorchScript trace
    def to_torchscript(...):
        # ...
                print(f"\n\nExample inputs device: {example_inputs.device}\n")
                scripted_module = torch.jit.trace(func=self.eval(), example_inputs=example_inputs, **kwargs)
        # ...
        return scripted_module
#...

if __name__ == '__main__':
    # ...
    # UNTIL HERE WORKS FINE

    # save model as TorchScript using eval()
    model.to_torchscript(file_path="example.pt", torchscript_approach='trace')

CLICK ME for full code

from __future__ import annotations  # don't crash on non-imported libraries when type checking
from typing import Dict, Optional, Union
import torch
from torch import nn as nn
import pytorch_lightning as pl
from pytorch_lightning import Trainer
from torch.utils.data import Dataset, DataLoader
from torch.nn import functional as F

class MyModel(pl.LightningModule):
    def __init__(self):  # **kwargs  # sample_rate
        super().__init__()
        # self.save_hyperparameters()
        # 1 sec of audio
        self.input_layer = nn.Linear(8000, 400, bias=True)
        self.hidden_layer = nn.Linear(400, 128, bias=True)
        self.output_layer = nn.Linear(128, 3, bias=True)

        self.criterion = nn.CrossEntropyLoss()

    def forward(self, input):
        x = F.relu_(self.input_layer(input))
        x = F.relu_(self.hidden_layer(x))
        output = self.output_layer(x)  # torch.sigmoid()
        return output

    def calculate_loss(self, prediction, target):
        loss = self.criterion(prediction, target)
        return loss

    def training_step(self, batch, batch_idx):
        input, target = batch
        # !!! use first batch to create an example input !!!
        if self.example_input_array is None:
            # we only need 1 sample, not a whole batch
            self.example_input_array = input
        prediction = self(input)
        loss = self.calculate_loss(prediction, target)
        return loss

    def validation_step(self, batch, batch_idx):
        input, target = batch
        prediction = self(input)
        loss = self.calculate_loss(prediction, target)
        return loss

    def test_step(self, batch, batch_idx):
        input, target = batch
        prediction = self(input)
        loss = self.calculate_loss(prediction, target)
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=1e-3)

    # TODO create pull request for this
    def to_torchscript(
            self, file_path: Optional[str] = None, torchscript_approach: Optional[str] = 'script',
            example_inputs: torch.Tensor = None, **kwargs
    ) -> Union[ScriptModule, Dict[str, ScriptModule]]:
        # training or eval/test?
        mode = self.training
        with torch.no_grad():
            if torchscript_approach == 'script':
                scripted_module = torch.jit.script(self.eval(), **kwargs)
            elif torchscript_approach == 'trace':
                if example_inputs is None:
                    example_inputs = self.example_input_array
                print(f"\n\nExample inputs device: {example_inputs.device}\n")
                scripted_module = torch.jit.trace(func=self.eval(), example_inputs=example_inputs, **kwargs)
            else:
                raise ValueError(f"torchscript_approach only supports 'script' or 'trace', but value given was:"
                                 f"{torchscript_approach}")
        # set back whether we were training or not
        self.train(mode)

        if file_path is not None:
            torch.jit.save(scripted_module, file_path)

        return scripted_module

# DATA
class SimpleDataset(Dataset):
    def __init__(self, sample_rate=8000):
        self.sample_rate = sample_rate

    def __len__(self):
        return 16

    def __getitem__(self, idx):
        # 0, 1 or 2
        target = torch.randint(0, 3, size=(1, )).squeeze()
        # size 8000/16000 of 0.0, 0.5, or 1.0
        input = torch.full((self.sample_rate,), (target.float()/2).item())
        # torch.empty(self.sample_rate,).fill_(target.float()/2)
        return input, target

class SimpleDatamodule(pl.LightningDataModule):
    def setup(self, stage: str = None):
        pass

    def train_dataloader(self):
        return DataLoader(SimpleDataset(), batch_size=4)

    def val_dataloader(self):
        return DataLoader(SimpleDataset(), batch_size=4)

    def test_dataloader(self):
        return DataLoader(SimpleDataset(), batch_size=4)

if __name__ == '__main__':
    sr = 8000
    checkpoint_location = "example.ckpt"
    # network
    model = MyModel()
    # data
    dm = SimpleDatamodule()
    # train
    trainer = Trainer(max_epochs=2, deterministic=True, gpus=1)  # gpus=1,
    trainer.fit(model, dm)
    # save
    trainer.save_checkpoint(checkpoint_location)
    # UNTIL HERE WORKS FINE

    # save model as TorchScript using eval()
    model.to_torchscript(file_path="example.pt", torchscript_approach='trace')

Expected behavior
Model and Tensor input both being evaluated at device GPU.
Actual behavior
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

CLICK ME for full Traceback

yes, even hidden code blocks!
$ python save_as_torchscript_device_error.py
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name         | Type             | Params
--------------------------------------------------
0 | input_layer  | Linear           | 3 M
1 | hidden_layer | Linear           | 51 K
2 | output_layer | Linear           | 387
3 | criterion    | CrossEntropyLoss | 0
/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
Epoch 1: 100%|█████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 148.42it/s, loss=0.954, v_num=55]

Example inputs device: cuda:0

Traceback (most recent call last):
  File "/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/torch/jit/__init__.py", line 692, in run_mod_and_filter_tensor_outputs
    outs = wrap_retval(mod(*_clone_inputs(inputs)))
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "<string>", line 3, in forward

      def addmm(self: Tensor, mat1: Tensor, mat2: Tensor, beta: number = 1.0, alpha: number = 1.0):
          return self + mat1.mm(mat2)
                 ~~~~~~~~~~~~~~~~~~~ <--- HERE

      def batch_norm(input : Tensor, running_mean : Optional[Tensor], running_var : Optional[Tensor], training : bool, momentum : float, eps : float) -> Tensor:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "eval_gpu_test.py", line 130, in <module>
    model.to_torchscript(file_path="example.pt", torchscript_approach='trace')
  File "eval_gpu_test.py", line 71, in to_torchscript
    scripted_module = torch.jit.trace(func=self.eval(), example_inputs=example_inputs, **kwargs)
  File "/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/torch/jit/__init__.py", line 953, in trace
    return trace_module(func, {'forward': example_inputs}, None,
  File "/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/torch/jit/__init__.py", line 1118, in trace_module
    _check_trace([inputs], func, check_trace_method,
  File "/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/torch/jit/__init__.py", line 734, in _check_trace
    traced_outs = run_mod_and_filter_tensor_outputs(traced_func, inputs, 'trace')
  File "/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/torch/jit/__init__.py", line 696, in run_mod_and_filter_tensor_outputs
    raise TracingCheckError(*graph_diagnostic_info(),
torch.jit.TracingCheckError: Tracing failed sanity checks!
Encountered an exception while running the trace with test inputs.
Exception:
	The following operation failed in the TorchScript interpreter.
	Traceback of TorchScript (most recent call last):
	  File "<string>", line 3, in forward

	      def addmm(self: Tensor, mat1: Tensor, mat2: Tensor, beta: number = 1.0, alpha: number = 1.0):
	          return self + mat1.mm(mat2)
	                 ~~~~~~~~~~~~~~~~~~~ <--- HERE

	      def batch_norm(input : Tensor, running_mean : Optional[Tensor], running_var : Optional[Tensor], training : bool, momentum : float, eps : float) -> Tensor:
	RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Environment

CUDA:

GPU:

GeForce GTX 1080 Ti

available:         True
version:           10.2

Packages:

numpy:             1.19.1
pyTorch_debug:     False
pyTorch_version:   1.6.0
pytorch-lightning: 1.0.0rc4
tqdm:              4.50.1

System:

OS:                Linux (Ubuntu 18.04)
architecture:

64bit
ELF

processor:         x86_64
python:            3.8.5
version:           #119-Ubuntu SMP Tue Sep 8 12:30:01 UTC 2020

Additional context
Might it be related to one of these eval() changes in v0.10?:
https://pytorch-lightning.readthedocs.io/en/stable/CHANGELOG.html?highlight=eval#changed

Changed the code under __main__ to the following to narrow down the problem:
if __name__ == '__main__':
    checkpoint_location = "example.ckpt"
    # network
    model = MyModel()
    print(f"network model device: {model.device}")
    # data
    dm = SimpleDatamodule()
    # train
    trainer = Trainer(max_epochs=2, deterministic=True, gpus=1)  # gpus=1,
    trainer.fit(model, dm)
    print(f"trainer fit model device: {model.device}")
    # save
    trainer.save_checkpoint(checkpoint_location)
    print(f"save checkpoint model device: {model.device}")
    # UNTIL HERE WORKS FINE
    print(f"before eval model device: {model.device}")
    model.eval()
    print(f"after eval model device: {model.device}")

Output:
$ python save_as_torchscript_device_error.py
network model device: cpu
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name         | Type             | Params
--------------------------------------------------
0 | input_layer  | Linear           | 3 M
1 | hidden_layer | Linear           | 51 K
2 | output_layer | Linear           | 387
3 | criterion    | CrossEntropyLoss | 0
/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The dataloader, val dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
/home/*user*/anaconda3/envs/pytorchlit10/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:45: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
Epoch 1: 100%|█████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 153.21it/s, loss=1.045, v_num=57]
trainer fit model device: cpu
save checkpoint model device: cpu
before eval model device: cpu
after eval model device: cpu

The model is never send to the GPU, even though gpus=1 is set in the Trainer?

If I remember correctly, the trainer moves the model to cpu after fit ends, at least for ddp it was like this. Can't test it right now but will verify later. Maybe it helps

@awaelchli Thanks for the input. If that was still the case, this code:
trainer.fit(model, dm)
print(f"trainer fit model device: {model.device}")

should output gpu. Instead it outputs cpu?
Is the model send back to the CPU after training is completed?

Is the model send back to the CPU after training is completed?

Yes.
I don't understand the error that you posted. It shows device mismatch in the batch norm, but I don't see any batch norm  layer in your model.
Then I discover this:
if self.example_input_array is None:
            # we only need 1 sample, not a whole batch
            self.example_input_array = input

in your training step. This is why you will get the error. example input array remains on cuda while model is moved to cpu.
In your torchscript override, move the example input to the correct device. Problem solved.

@awaelchli I see, thank you. I was kind of expecting the model to stay on the GPU. That's why I thought the model being back on the CPU is a bug.
If you would run .test() after .fit(), it would need to again copy the model over to the GPU. Or if you want to manually run some data through your trained model. This generates extra traffic and delays.
What is the reasoning behind PL always moving the model back to the CPU? Is the moving of the model considered to be a small enough cost?

authors: NumesSanguis awaelchli

repo: PyTorchLightning/pytorch-lightning


text:
Training steps vs training batches

I have some doubts about training steps and training batches.
When training a model using max_steps, the number of training batches showed in progressing bar using 1 GPU is the same as using 8 GPUs. If using X GPUs, is the actual batch size during one optimization step X-times larger, which results in a smaller number of training batches?
The batch size should be expanded in multi-GPU setting according to DDP of pytorch (PyTorch Distributed: Experiences on Accelerating Data Parallel Training). From my observation in 1, the number of training batches equals to total number of steps (I don't think they should be the same). I wonder the DDP training mechanisms between pytorch and pytorch-lightning are still the same.
In pytorch, each device gets a copy of the model and other states exactly like the original one. After backwards, gradients from all devices are accumulated together and broadcasted to every device. Then models are updated using same gradients. Such process acts like splitting a large batch, execute forward and backward pass on a smaller, splitted part in each GPU and then accumulate all gradients and update model parameters.
It seems like pytorch-lightning treats a single optimization step in X GPUs as X steps. If so, the number of training batches becomes the number of steps. Only difference is that model parameters are updated once, not X times as expected. Another coincidence is that learning rate scheduler setting interval='step' updates learning rate in every batch.
I hope anyone could explain this.

python version: 3.8.5
pytorch version: 1.8.1
pytorch-lightning version: 1.2.5
OS version: CentOS 8

When training a model using max_steps, the number of training batches showed in progressing bar using 1 GPU is the same as using 8 GPUs.

Because we show the progress bar on rank 0 only

I wonder the DDP training mechanisms between pytorch and pytorch-lightning are still the same.

yes they are! (they should be, or it's a bug)

Such process acts like splitting a large batch, execute forward and backward pass on a smaller, splitted part in each GPU and then accumulate all gradients and update model parameters.

no, you are describing DP, not DDP. In DDP we don't split the batches by number of GPUs.

The batch size should be expanded in multi-GPU setting according to DDP of pytorch (PyTorch Distributed: Experiences on Accelerating Data Parallel Training)

Could you link to the article, please? I'd like to take a look.
Could you take our example and run it.
python pl_examples/basic_examples/simple_image_clasifier.py --accelerator ddp --gpus 2
and let me know which observations / parameter settings you are not agreeing with?

Thanks for quick reply.
I understand the difference between DP and DDP. What I really mean is that, each GPU processes a batch from a distributed sampler, after all GPUs finish a batch, the gradients are then accumulated. For example, a dataloader with distributed sampler generates a batch of size 16, 8 GPUs can process 8 * 16 instances in one step, and then the gradients of these 128 instances are averaged to update model parameters. It's just like virtually splitting a batch of size 128 into 8 parts, processing all parts using 8 GPUs and merging their results. I suppose the resulting model trained by 8 GPUs with a batch size 16 should be same as the one trained by 1 GPU with batch size 128, regardless of randomness in sampling and ordering of instances.

Such process acts like splitting a large batch, execute forward and backward pass on a smaller, splitted part in each GPU and then accumulate all gradients and update model parameters.

no, you are describing DP, not DDP. In DDP we don't split the batches by number of GPUs.

The batch size should be expanded in multi-GPU setting according to DDP of pytorch (PyTorch Distributed: Experiences on Accelerating Data Parallel Training)
The working mechanism of DDP can be found in this paper 1.

I use following command to test the output of pytorch-lightning.
python ./simple_image_classifier.py \
    --num_nodes 1 \
    --accelerator 'ddp' \
    --gpus 2 \
    --batch_size 16 \
    --num_workers 1 \
    --max_epochs 2 \
    --accumulate_grad_batches 1 \
    --check_val_every_n_epoch 1 \
    --log_every_n_steps 1 \
    --flush_logs_every_n_steps 1 \
    --progress_bar_refresh_rate 1 \
    --auto_lr_find false \
    --replace_sampler_ddp false

Only 1 GPU:

2 GPUs:

As you were saying, the progress bar only shows for rank 0. MNIST has 55000 images in the training set, all 3438 = ceil(55000 / 16) batches are consumed by rank 0 GPU for only one GPU setting. However, in the later setting using 2 GPUs, rank 0 GPU only processes 1719 batches, which is not consistent with the output of progress bar. I got similar outputs when testing with even more GPUs. Why is that?
Further, I use max_steps=900 instead of max_epochs and 8 GPUs. I have such a result.

The training stops at 900 batches, not 900 steps. Indeed, the training number is expected to be 900 * 16 * 8 = 115200 images, approximately 2.09 epochs.
It seems like pytorch-lightning regards a step as a batch, which I think is a misunderstanding. A step should be one duration of a whole update of model parameters accomplished in optimizer.step() call. In such sense, a steps may contain multiple batches.

After carefully inspect the source code, I found the mistake made in the launch command. --replace_sampler_ddp false should not be set in this case. Enabling --replace_sampler_ddp gives me the expected results.
Thanks again.


text:
final cleanup for v0.8.0

What does this PR do?
rest final version, add link to docs
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

There are a number of places that reference changing the epoch count to starting at 0 in v0.8, are we still doing this?
https://github.com/PyTorchLightning/pytorch-lightning/blob/761252be78ba573ad1f5c700190eb89b13e41e99/pytorch_lightning/callbacks/gradient_accumulation_scheduler.py#L60

yes! this should be done. want to submit as soon as you can?

There are a number of places that reference changing the epoch count to starting at 0 in v0.8, are we still doing this?
https://github.com/PyTorchLightning/pytorch-lightning/blob/761252be78ba573ad1f5c700190eb89b13e41e99/pytorch_lightning/callbacks/gradient_accumulation_scheduler.py#L60

yes! this should be done. @Borda mind updating?

fixing in separate PR #2206

authors: Borda jeremyjordan williamFalcon


text:
tests: use MagicMock [wip]

What does this PR do?
experiment with setting all MagicMock sub-class so we can easier debug
eventually, we can try something similar to Trainer...
Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[ ] Did you read the contributor guideline, Pull Request section?
[ ] Did you make sure your PR does only one thing, instead of bundling different changes together?
[ ] Did you make sure to update the documentation with your changes [if needed]?
[ ] Did you write any new necessary tests [no need for typos, docs]?
[ ] Did you verify new and existing tests pass locally with your changes?
[ ] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

[ ] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[ ] Make sure the title is self-explanatory and the description concisely explains the PR
[ ] Add labels and milestones (and optionally projects) to the PR so it can be classified
[ ] Check that target branch and milestone are aligned!

Did you have fun?
Make sure you had fun coding 🙃

@Borda This will unfortunately not work. I don't think this is how MagicMock is supposed to be used. Here I made an example:
from unittest import mock
from unittest.mock import MagicMock

# Inherit from MagicMock (this PR) will not work:

class Example(MagicMock):

    def __init__(self):
        print("constructor")

    def foo(self):
        print("foo")

ex = Example()
ex.foo()

# This will throw an error!
# ex.foo.assert_called_once()

# A real mocking example that will work

class Example(object):

    def __init__(self):
        print("still a constructor")

    def foo(self):
        print("I got mocked, I will not print")

with mock.patch.object(Example, "foo") as mocked_foo:
    ex = Example()
    ex.foo()
    mocked_foo.assert_called_once()  # this is how mocks should be used!

from unittest import mock
from unittest.mock import MagicMock

# Inherit from MagicMock (this PR) will not work:

class Example(MagicMock):

    def __init__(self):
        print("constructor")

    def foo(self):
        print("foo")

ex = Example()
ex.foo()

# This will throw an error!
# ex.foo.assert_called_once()

it seems that this returns empty list too:
ex = Example()
ex.foo()
print(ex.method_calls)

from unittest import mock
from unittest.mock import MagicMock

# Inherit from MagicMock (this PR) will not work:

class Example(MagicMock):

    def __init__(self):
        print("constructor")

    def foo(self):
        print("foo")

ex = Example()
ex.foo()

# This will throw an error!
# ex.foo.assert_called_once()

it seems that this returns empty list too:
ex = Example()
ex.foo()
print(ex.method_calls)

authors: Borda awaelchli


text:
Raise exception for ddp_cpu not supported for TPUs

What does this PR do?

Fixes #7810
Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

[x] Was this discussed/approved via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[ ] Did you make sure to update the documentation with your changes? (if necessary)
[x] Did you write any new necessary tests? (not for typos and docs)
[x] Did you verify new and existing tests pass locally with your changes?
[ ] Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)
[ ] Did you list all the breaking changes introduced by this pull request?

PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

[x] Is this pull request ready for review? (if not, please submit in draft mode)
[x] Check that all items from Before submitting are resolved
[x] Make sure the title is self-explanatory and the description concisely explains the PR
[x] Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?
Make sure you had fun coding 🙃

@kaushikb11 still draft?

Hello @kaushikb11! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file pytorch_lightning/trainer/connectors/accelerator_connector.py:

Line 747:22: E126 continuation line over-indented for hanging indent


text:
newint.divmod() barfs with NotImplemented

>>> from builtins import int as _int
>>> i = int(1431216000)
>>> j = _int(i)
>>> i == j
True
>>> type(i)
<type 'int'>
>>> type(j)
<class 'future.types.newint.newint'>
>>> divmod(i, 1.0)
(1431216000.0, 0.0)
>>> divmod(j, 1.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../site-packages/future/types/newint.py", line 204, in __divmod__
    return (newint(value[0]), newint(value[1]))
TypeError: 'NotImplementedType' object has no attribute '__getitem__'
>>> import future
>>> future.__version__
'0.14.3'

On PyPy (2.5.1), this breaks datetime.utcfromtimestamp:
>>>> from builtins import int as _int
>>>> i = int(1431216000)
>>>> j = _int(i)
>>>> i == j
True
>>>> type(i)
<type 'int'>
>>>> type(j)
<class 'future.types.newint.newint'>
>>>> divmod(i, 1.0)
(1431216000.0, 0.0)
>>>> divmod(j, 1.0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/matt/Desktop/py-dimgx/.venv-pypy/site-packages/future-0.14.3-py2.7.egg/future/types/newint.py", line 204, in __divmod__
    return (newint(value[0]), newint(value[1]))
TypeError: 'NotImplemented' object is not subscriptable
>>>> import future
>>>> future.__version__
'0.14.3'
>>>> from datetime import datetime
>>>> datetime.utcfromtimestamp(i)
datetime.datetime(2015, 5, 10, 0, 0)
>>>> datetime.utcfromtimestamp(j)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/matt/.site/lib/pypy/lib_pypy/datetime.py", line 1530, in utcfromtimestamp
    t, frac = divmod(t, 1.0)
  File "/Users/matt/Desktop/py-dimgx/.venv-pypy/site-packages/future-0.14.3-py2.7.egg/future/types/newint.py", line 204, in __divmod__
    return (newint(value[0]), newint(value[1]))
TypeError: 'NotImplemented' object is not subscriptable
>>> from sys import version
>>> print(version)
2.7.9 (9c4588d731b7fe0b08669bd732c2b676cb0a8233, Apr 02 2015, 15:24:16)
[PyPy 2.5.1 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]

Curiously, this doesn't happen in cPython (session continued from above):
>>> from datetime import datetime
>>> datetime.utcfromtimestamp(i)
datetime.datetime(2015, 5, 10, 0, 0)
>>> datetime.utcfromtimestamp(j)
datetime.datetime(2015, 5, 10, 0, 0)
>>> from sys import version
>>> print(version)
2.7.9 (default, Dec 12 2014, 13:52:29)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]

newint seems to derive from long. Have you tried the same with a native long instead of a native int?

Using a long works in cPython:
>>> from datetime import datetime
>>> i = 1431216000L
>>> type(i)
<type 'long'>
>>> divmod(i, 1.0)
(1431216000.0, 0.0)
>>> datetime.utcfromtimestamp(i)
datetime.datetime(2015, 5, 10, 0, 0)

... as well as PyPy:
>>>> from datetime import datetime
>>>> i = 1431216000L
>>>> type(i)
<type 'long'>
>>>> divmod(i, 1.0)
(1431216000.0, 0.0)
>>>> datetime.utcfromtimestamp(i)
datetime.datetime(2015, 5, 10, 0, 0)

See also https://bitbucket.org/pypy/pypy/issue/2048/. However, despite the datetime oddity, future should probably support newint.__divmod__(<float>), if it is to be a true drop-in replacement for Python 2.7's int.

Thanks very much for finding and fixing this. Merged into the v0.14.x branch.

authors: QuLogic edschofield posita

repo: PythonCharmers/python-future


text:
Have you heard of the fighting game developers discord?

Bit inconvenient to make this a github issue, but its probably the best way to contact you about it.
Here's the link: https://discord.gg/tfhufFqh
Also, are you guys aware of Snopek's Rollback Netcode framework for godot?

Also, are you guys aware of Snopek's Rollback Netcode framework for godot?

Discussed in #26.

Oh totally forgot about the fact that I mentioned that already. Woops!

Bit inconvenient to make this a github issue, but its probably the best way to contact you about it.
Here's the link: https://discord.gg/tfhufFqh

Nope, I wasn't aware of this server. I went ahead and joined, thanks for the invite! Also just for future reference, the discussion page is open for posts like this.

authors: 20milliliter Pyxus ValorZard

repo: Pyxus/fray


text:
Need to setup profile page

Profile page

User

Watchlist
Sign out
Reset password and edit profile

@shivshankar9 Any update? If you face issue in the code let me know with your problem I am here to help you out.

Ok sure ,

@shivshankar9 Any update? If you face issue in the code let me know with your problem I am here to help you out.

I need to work on this issue, I can build a ui profile page like netflix @QAZIMAAZARSHAD

This issue is already assigned.
And first, resolve other issues that are assigned to you.

@shivshankar9 any new updates?

Is this still active? I would like to work on it under hacktoberfest

Thanks @QAZIMAAZARSHAD for assigning me this issue. Can you give some hints on how the profile page should be like?

Thanks @QAZIMAAZARSHAD for assigning me this issue. Can you give some hints on how the profile page should be like?

A profile page should contain user details like name, id, subscription validity, my downloads, an option to log out, It should also display how many devices are connected with this account.
You can implement your own ideas as well, all the above-mentioned features are not mandatory, you can add as many features you want.
Make sure the profile page follows the theme of the website.
Also, add the My profile button in the navbar of all pages.
Do check the responsiveness of your page in mobile view.
Visit some other websites or apps to get an idea how it should look like.

If you include the log-out button inside the profile page.
Then replace the logout button on the navbar with the My Profile button.

Do you want a static profile page or a dynamic one?
For Dynamic Profile Page, profiling needs to be done for every user involving making APIs and integrating database.

@QAZIMAAZARSHAD Do you want a static profile page or a dynamic one? For Dynamic Profile Page, profiling needs to be done for every user involving making APIs and integrating database.

Static page will be fine.

repo: QAZIMAAZARSHAD/Movie-Streaming-Website


text:
Hi, I am not clear with the steps to run the benchmarks.  Could you please help me with elaborated sequence of steps to be followed to run this test. My setup details are : cpu: AMD FX 4100 quad core:: gpu: Ellesmere

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Version used:
Environment name and version (e.g. rustc 1.26.0-nightly):
Operating System and version:
Hardware (e.g. CPU: Intel i5-6360U, GPU: Intel Iris Graphics 540, 8GB RAM)

Hello, do you know which open cl device is which? This can usually be found with the clinfo command

Ping @luckynikki

authors: adamisntdead luckynikki

repo: QCGPU/qcgpu-rust