text:
UART get/set sclk functions (IDFGH-12488)

Is your feature request related to a problem?
It's possible to individually get/set the struct values passed into uart_param_config except for the source_clk element.  For example, data_bits, parity, and stop_bits can be fetched and updated with the functions mentioned in the Multiple Steps portion of the documentation.
However, there are no functions like uart_set_sclk or uart_get_sclk to get or set the UART clock source.
Describe the solution you'd like.
I propose adding the following methods:

esp_err_t uart_get_sclk(uart_port_t uart_num, uart_sclk_t *sclk)
esp_err_t uart_set_sclk(uart_port_t uart_num, uart_sclk_t sclk)

Describe alternatives you've considered.
It's possible to get the UART sclk using the HAL function uart_hal_get_sclk but it requires a uart_hal_context_t struct.  Unfortunately, those are statically defined in uart.c with no way to access them.  The only alternative is to use the UART_LL_GET_HW and uart_ll_get_sclk low-level functions instead.
Additional context.
Some clock sources like UART_SCLK_RTC require additional steps when enabling or disabling them.  Moving this logic into uart_set_sclk would mean less work when dealing with low-level UART functions.
For context, I'm currently working on creating a UART autobaud library.  One of the main challenges is that the uart_ll_get_*_pulse_cnt functions return values in clock cycle units.  It's not possible to use this into a baud rate calculation without knowing the UART source clock's frequency.  While this can be done with low-level functions, I'd prefer to use higher-level functions to ensure portability and prevent upgrade issues.  For example, upgrade issues like the return type of uart_ll_get_sclk changing from uart_sclk_t to soc_module_clk_t. 😅

Hi @llamasoft,
According to what you are trying to do, I think it is better if we offer an API that can directly know the time passed between two pulses. How do you think?

I was mostly pointing out how all settings controlled by uart_param_config have corresponding get and set methods except for the source_clk member.  It means that the only way to set the value is with uart_param_config which will necessarily overwrite the other settings.  Once set, there's no way to retrieve the value without using low-level functions.
(The use for autobaud is tangential.  Autobaud is going to require low-level functions anyways so I don't mind using other low-level functions to get source_clk.)

authors: llamasoft songruo


text:
[TW#17988] SPIFFS long file names silently fail.

Setting SPIFFS_OBJ_NAME_LEN to 256 or 255 causes SPIFFS VFS operations to silently fail. Value at or below 192 work. Values above 192 were not tested.

Was this fixed?

source: gharchive/issue

repo: espressif/esp-idf

license_type: permissive

license_source: github-api


text:
idf_monitor does not respect port setting when flashing (IDFGH-2479)

Environment

Development Kit: none
Module or chip used: ESP32 (Heltec WiFi LoRa 32 V2)
IDF version: v4.1-dev-1757-g826b9f638
Build System: idf.py
Compiler version: xtensa-esp32-elf-gcc (crosstool-NG esp-2019r2) 8.2.0
Operating System: Linux
Using an IDE?: CLion, but just for coding. Building and flashing is done from the terminal.
Power Supply: USB

Problem Description
I am using two Heltec WiFi LoRa 32 (V2) connected to the same PC (/dev/ttyUSB0 and /dev/ttyUSB1). I open the monitor with the following command:
idf.py monitor -p /dev/ttyUSBx with x = 0 and x = 1 in a second terminal.
I use Ctrl+T, Ctrl+A/Ctrl+F to build and flash.
Expected Behavior
The code is rebuilt and flashed to the device connected to the port specified with the -p switch.
Actual Behavior
The code is rebuilt and always flashed to the device on port /dev/ttyUSB1. The -p setting is ignored.
Steps to reproduce

Connect two devices to the same PC. They should be assigned /dev/ttyUSB0 and /dev/ttyUSB1.
Open the monitor for both devices with the following two commands:

idf.py monitor -p /dev/ttyUSB0
idf.py monitor -p /dev/ttyUSB1

Press Ctrl+T, Ctrl+A in the monitor for /dev/ttyUSB0.
The code is built successfully, but flashed to the wrong device (/dev/ttyUSB1)

Code to reproduce this issue
Any project.
Debug Logs
No crashes.
Other items if possible
N/A

I just noticed that when reversing the order of the arguments to idf.py -p /dev/ttyUSBx monitor it works as it should. However, I think this behavior is strange a maybe a consistent handling of the parameters  regardless of the order should be considered.

Hi @KloolK. Yes you're right and flashing from IDF monitor should use the port specified for monitor. We already noticed this issue and the fix is under way. This GH issue will be closed when the fix gets pushed to GH.

I just noticed that when reversing the order of the arguments to idf.py -p /dev/ttyUSBx monitor it works as it should. However, I think this behavior is strange and maybe a consistent handling of the parameters regardless of the order should be considered.

No, this is the intended behavior. Arguments after idf.py are global and local after a subcommand. At this moment using a global port bypasses the related bug.
For example, port1 is a global port and port2 is a locally overridden one for monitor in
idf.py -p port1 monitor -p port2

Also, --print-filter is a local argument for monitor, i.e. idf.py monitor --print-filter xy can be used but idf.py --print-filter xy monitor makes no sense.

@KloolK Thanks for reporting and sorry for the slow turnaround. Really appreciate your patience in waiting for the fix. The fix has been merged internally in our repo. and this issue will be automatically closed when the fix lands on Github. For your information. Thanks.

authors: Alvin1Zhang KloolK dobairoland


text:
ESP-IDF Tools Installer v2.2 does not work due to two different issues (IDFGH-2670)

Environment
Windows 10 with latest updates applied. (Other environment items were not applicable.)
Problem Description
I cannot install ESP-IDF Tools and ESP-IDF using the official installer. I would like to start developing for the ESP32 but I cannot install the tools. I would prefer not to install the environment manually.
I encounter two issues:
Issue 1
Expected Behavior
ESP-IDF is copied to destination directory after extraction and setp continues.
Actual Behavior
Error message "Failed to copy ESP-IDF to the destination directory: E:\espidf\3.3" appears and setup halts.
Steps to reproduce

Download and run esp-idf-tools-setup-2.2.exe
Choose option to download ESP-IDF (tested with 3.3 release and 3.3 branch)

Debug Logs
log1.txt
Relevant part:

2020-02-10 19:00:44.329   --------
2020-02-10 19:00:44.332   Moving C:\Users\r\AppData\Local\Temp\is-EG1TH.tmp\esp-idf-v3.3.1 to E:\espidf\3.3
2020-02-10 19:00:44.332   Message box (OK):
Failed to copy ESP-IDF to the destination directory: E:\espidf\3.3
2020-02-10 19:01:53.031   User chose OK.
2020-02-10 19:01:53.031   Message box (Yes/No):
Installation log has been created, it may contain more information about the problem.
Display the installation log now?
2020-02-10 19:01:53.962   User chose Yes.
2020-02-10 19:01:53.989   CurStepChanged raised an exception.

Workaround
Go to the .espressif\releases directory, manually extract and copy esp-idf-v3.3.1.zip to ESP-IDF destination directory, and re-run installer.
Issue 2
Expected Behavior
ESP-IDF Tools are set-up.
Actual Behavior
Error message "Installation has failed with exit code 1" appears and setup halts.
Steps to reproduce

Download and run esp-idf-tools-setup-2.2.exe
Choose option to download ESP-IDF (tested with 3.3 release and 3.3 branch)
When installation fails, apply workaround from Issue 1
Re-run esp-idf-tools-setup-2.2.exe
Choose option to use the previously extracted ESP-IDF

Debug Logs
log2.txt
Relevant part:

2020-02-10 19:10:02.444   Installing Python environment:C:\Users\r\AppData\Local\Programs\Python\Python37\python.exe E:\espidf\3.3\tools\idf_tools.py install-python-env
2020-02-10 19:10:03.415   Done, exit code=1
2020-02-10 19:10:03.415   --------
2020-02-10 19:10:03.415   Running command: C:\Users\r\AppData\Local\Programs\Python\Python37\python.exe E:\espidf\3.3\tools\idf_tools.py install-python-env
usage: virtualenv [--version] [-v | -q] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-win,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_separated_list] [--clear] [--system-site-packages] [--copies]
[--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--clear-app-data] [--prompt prompt] [-h]
[dest]
virtualenv: error: unrecognized arguments: --no-site-packages
Creating a new Python environment in E:\espidf.espressif\python_env\idf3.3_py3.7_env
Traceback (most recent call last):
File "E:\espidf\3.3\tools\idf_tools.py", line 1349, in
main(sys.argv[1:])
File "E:\espidf\3.3\tools\idf_tools.py", line 1345, in main
action_func(args)
File "E:\espidf\3.3\tools\idf_tools.py", line 1156, in action_install_python_env
stdout=sys.stdout, stderr=sys.stderr)
File "C:\Users\r\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\Users\r\AppData\Local\Programs\Python\Python37\python.exe', '-m', 'virtualenv', '--no-site-packages', 'E:\espidf\.espressif\python_env\idf3.3_py3.7_env']' returned non-zero exit status 2.
2020-02-10 19:10:03.415   --------
2020-02-10 19:10:03.415   Message box (OK):
Installation has failed with exit code 1
2020-02-10 19:10:21.217   User chose OK.
2020-02-10 19:10:21.235   Message box (Yes/No):
Installation log has been created, it may contain more information about the problem.
Display the installation log now?
2020-02-10 19:10:36.417   User chose Yes.
2020-02-10 19:10:36.460   CurStepChanged raised an exception.

Thanks for reporting! The fix for Python environment installation is in review and will be released soon (along with a new version of the installer).
As a temporary workaround, you may downgrade virtualenv package as follows:
C:\Users\r\AppData\Local\Programs\Python\Python37\python.exe -m pip install --upgrade virtualenv=16.7.9

After this, retry the installation.

Hi @igrr,
I confirm the workaround works. Thanks!
What about Issue 1? Not as inconvenient but I've seen many reports for that as well.

@vmsh0 sure, checking that issue as well. Can you please link to the reports you've seen? To update them if it gets fixed.

One more detail, did E:\espidf\3.3 directory exist prior to running the installer? If yes, was it empty?

As a temporary workaround, you may downgrade virtualenv package as follows:
C:\Users\r\AppData\Local\Programs\Python\Python37\python.exe -m pip install --upgrade virtualenv==16.7.9

Or remove --no-site-packages flag from virtualenv inside idf_tools.py. The flag has been deprecated for quite some time (6 years).

@pavestru yes, that's the fix which is going to be merged. For the users of the Windows tools installer the problem is that it's not trivial to modify idf_tools.py since it is extracted by the installer (at least when installing v3.3.1). For other OSes, modifying idf_tools.py can also be used as a workaround for now.

Okay, I was able to reproduce the first issue as well. It happens because we are trying to use "RenameFile" function to move the extracted ESP-IDF directory into place, but this function doesn't work when the destination is on a different drive. This will also be fixed shortly.

Hi @igrr,
Thank you for looking into the issue. I suspected it had something to do with different drives. I'm happy you'll be able to push a fix!
I had found issue #4128 and also one more, which I can't find anymore. (And then many help requests on esp32.com.)

我也遇到了和楼主同样的问题，异常1和异常-2，非常困扰，等待乐鑫回复解决方法

@pavestru yes, that's the fix which is going to be merged. For the users of the Windows tools installer the problem is that it's not trivial to modify idf_tools.py since it is extracted by the installer (at least when installing v3.3.1). For other OSes, modifying idf_tools.py can also be used as a workaround for now.

would running this from CMD also work? it seems to be the last step of the install.
python -m virtualenv "C:\\Users\\user\\.espressif\\python_env\\idf4.0_py3.7_env"

@armandr Yes, running this from CMD should work, provided that python refers to the same Python installation you are selecting in the Installer (in case you have more than one).

The fix for the first issue has been merged in https://github.com/espressif/esp-idf/commit/7a18f02acd7005f7c56e62175a8d1968a1a9019d, backports to release/v4.1 and release/v4.0 are coming soon.

If you are experiencing this issue (on Windows), please try this version of the installer: https://dl.espressif.com/dl/esp-idf-tools-setup-2.3.exe

I tried the installer version listed above and it worked - the --no-site-packages problem disappeared, and the installer finished, creating the proper shortcuts in the Start Menu etc.

I confirm that the file copy issue ("Issue 1") has been resolved for me.
The installer also successfully installs the master branch (w/ Python 3.8), the 3.3.1 release version, but not the 4.0 branch version. Is 3.3.1 expected to work at all? I see the --no-sites-package fix hasn't been backported yet, so I find this surprising.
Also, what is the difference between "release version" and "branch version"? I though the first downloaded the SDK as a full zip package and the second cloned it from the Github branch, but it seems to me that the second one downloads a zip package as well.

Thanks for the 2.3 version!  I'm up and running now!  I can confirm it resolves the error:
virtualenv: error: unrecognized arguments: --no-site-packages"
Cleaned everything out, and let it install python 3.7, git, and esp-idf release/4.0
All is well now.

With https://dl.espressif.com/dl/esp-idf-tools-setup-2.3.exe installation was smooth. I used python 3.8, git 2.23.0, window .1.
Example hello_world running OK.

Ran into this issue too, the 2.3 version works fine.
I can see that windows-setup.rst has been updated to link to the 2.3 installer, but the online docs don't seem to have reflected that change yet. Might be worth looking into that so more folks don't run into this problem.

This is an unfortunate timing as we are transitioning to a new documentation host, still needs some work for the docs to go live: https://github.com/espressif/esp-idf/blob/master/README.md#setting-up-esp-idf

I am running into the following issue with version 2.2 and 2.3 of the installer. Not sure if this is the correct thread for this issue.
subprocess.CalledProcessError: Command '['C:\Users\user\AppData\Local\Programs\Python\Python38\python.exe', '-m', 'virtualenv', '--no-site-packages', 'C:\Users\user\.espressif\python_env\idf4.0_py3.8_env']' returned non-zero exit status 2.

@markwest which IDF version are you using with the 2.3 installer? If you are using an existing IDF directory (not downloading a new one using the installer), please pull the latest version of your branch (master, release/v4.1, release/v4.0). The fixes for this issue have been merged into the respective branches.
Alternatively, you can use the installer to download a fresh copy of IDF, in this case the installation should succeed.

@igrr I went back to the latest version of virtualenv (20.0.5), removed the existing IDF directory, and ran version 2.3 of the installer again. I did not encounter any issues with the install. Also, I successfully flashed and monitored the hello_world example. Thanks for your help.

I choosed to do it with the installer from https://dl.espressif.com/dl/esp-idf-tools-setup-2.3.exe
I started esp-idf_tools-setup-2.2.exe as admin
Available Python version: Python 3.7 (64-bit)
Available Git versions: Install Git 2.21.0
Available ESP-IDF versions:  ESP8266-master
But error show :
Installing Python environment:C:\Users\Python37\python.exe "D:\ESP8266_RTOS_SDK\tools\idf_tools.py" --idf-path D:\ESP8266_RTOS_SDK install-python-env
Done, exit code=1

@xuhongv It sounds like your setup is similar to mine. I was getting the same error. In my case, I did not use an existing ESP-IDF framework. Instead, I allowed the installer to download a new one, which worked.

ran with the same issue I installed v2.3 and then had exit code 2, did the process and then had exit code 4. Decided to follow @markwest recommendation, did everything from new and the installation went clean.
Thanks all.

As a temporary workaround, you may downgrade virtualenv package as follows:
C:\Users\r\AppData\Local\Programs\Python\Python37\python.exe -m pip install --upgrade virtualenv==16.7.9

Or remove --no-site-packages flag from virtualenv inside idf_tools.py. The flag has been deprecated for quite some time (6 years).

Hi, I met the same problem, and this method actually worked. after removing the flag, I am able to re-run the installer successfully on Windows 10.

It seems like that latest version of virtualenv has removed the --no-site-packages flag and the created virtualenv doesn't inherit system site-packages by default. Instead, the flag --system-site-packages is added to inherit system site-packages.

Just for the record
diff --git a/tools/idf_tools.py b/tools/idf_tools.py
index 158b5a764..32a7c57a1 100755
--- a/tools/idf_tools.py
+++ b/tools/idf_tools.py
@@ -1180,7 +1180,7 @@ def action_install_python_env(args):
             subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--user', 'virtualenv'],
                                   stdout=sys.stdout, stderr=sys.stderr)

-        subprocess.check_call([sys.executable, '-m', 'virtualenv', '--no-site-packages', idf_python_env_path],
+        subprocess.check_call([sys.executable, '-m', 'virtualenv', idf_python_env_path],
                               stdout=sys.stdout, stderr=sys.stderr)
     run_args = [virtualenv_python, '-m', 'pip', 'install', '--no-warn-script-location']
     requirements_txt = os.path.join(global_idf_path, 'requirements.txt')

This happened on MacOS also. The above workarounds worked.

I had "issue 1" with version 2.2 of the installer too (installing to D:).  2.3 fixed it. But the "getting started website still points to the buggy version 2.2"
https://docs.espressif.com/projects/esp-idf/en/stable/get-started/windows-setup.html

Did not notice the 2.3 version of tools setup. I ran the ver. 2.2 and ran into the Issue #2. I removed the '--no-site-packages' flag from 'idf_tools.py' and execute it again to make it work perfectly.
I realized the "stable" version of documentation still refering to the 2.2 version. (https://docs.espressif.com/projects/esp-idf/en/stable/get-started/windows-setup.html)
To see the 2.3 version link, change the doc. version in the lower-left part of the page to "lastest" or enter this link (https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/windows-setup.html)

Hello I would like to share my experience with you in case someone falls into the same problem as I did.
I've been trying to install esp-idf stable version v4.0 with https://dl.espressif.com/dl/esp-idf-tools-setup-2.3.exe but I've received many fails until I've got it working.
Things I did in order for the installation to succeed was:

Install python 3.7 from esp-idf-tools-setup-2.3
I had git already installed
Install esp-idf v4.0 (not release) from esp-idf-tools-setup-2.3
Install .espressif's tools from esp-idf-tools-setup-2.3
This way I've got installation working BUT! The esp-idf v4.0 was not the one that should have been. So next I deleted esp-idf folder and downloaded the latest esp-idf v4.0 with git.

I don't know if this problem came up because I didn't download the release/v4.0 version within esp-idf-tools-setup-2.3 installation.
After this if you want to work with any IDE you have to add tool's paths in "System environment variables".
I hope this will help anyone who get the same problem as me.
Regards, Kostas Ifantidis

@kifantidis,
Thanks for sharing this. Can you please describe the failures that you experienced before you got everything working?
Angus

The thing that I've noticed is that the timeout feature in esp_tls_conn_new(); function timeout configuration was missing. That was causing the socket to stay inside in infinite loop. Timeout also wasn't working for esp_https_ota_begin(); function.

Hi @kifantidis , OK, thanks for noting this. This is probably not related to the ESP-IDF Tools installer, although it might related to the exact ESP-IDF version. We would like to help investigate it. Could you please open a new issue and fill out the details asked for in the new issue template?

Namaste,
I too ran into these two different issues.
As suggested by many, used the latest esp-idf-tools-setup-2.3.exe installer along with ESP-IDF-Ver-4.0.
Installation completed successfully and the final installation logs on the ESP-IDF Command Prompt are as below:
Using Python in C:\Program Files (x86)\Python38-32
Python 3.8.2
Using Git in C:\Program Files\Git\cmd
git version 2.26.2.windows.1
Setting IDF_PATH: D:\espIdfVer40
Adding ESP-IDF tools to PATH...
D:.espressif\tools\xtensa-esp32-elf\esp-2019r2-8.2.0\xtensa-esp32-elf\bin
D:.espressif\tools\esp32ulp-elf\2.28.51.20170517\esp32ulp-elf-binutils\bin
D:.espressif\tools\cmake\3.13.4\bin
D:.espressif\tools\openocd-esp32\v0.10.0-esp32-20190313\openocd-esp32\bin
D:.espressif\tools\mconf\v4.6.0.0-idf-20190628
D:.espressif\tools\ninja\1.9.0
D:.espressif\tools\idf-exe\1.0.1
D:.espressif\tools\ccache\3.7
D:.espressif\python_env\idf4.0_py3.8_env\Scripts
D:\espIdfVer40\tools
Checking if Python packages are up to date...
Python requirements from D:\espIdfVer40\requirements.txt are satisfied.
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:
idf.py build
D:\espIdfVer40>

I ran esp-idf-tools-setup-2.3.exe with Python 3.8.3 - May 13, 2020 x84-64 and ESP-IDF-Ver-4.0 in Windows10 64bits
and I tested hello world example
Everything OK.

I just tried running  esp-idf-tools-setup-2.3.exe on Windows 10 x64 with previously installed miniconda python  v3.8.5, Git  2.30.0, selected esp-idf-4.1, and I get the following error (from the end of the setup log):
2021-01-06 16:25:44.893   Installation process succeeded.
2021-01-06 16:25:44.899   Setting PATH for this process: C:\Users\hari\miniconda3;C:\Program Files\Git\cmd;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Program Files\Git\cmd;C:\Users\hari\AppData\Local\Microsoft\WindowsApps;;C:\Modeltech_pe_edu_10.4a\win32pe_edu;C:\Users\Hari\Anlogic\TD4.6.4\bin
2021-01-06 16:25:44.899   PYTHONPATH=
2021-01-06 16:25:44.900   idf_tools.py exists in IDF directory
2021-01-06 16:25:44.900   idf_tools.py command: C:\Users\hari\miniconda3\python.exe "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py" --idf-path C:\Users\hari\esp32\esp-idf-v4.1
2021-01-06 16:25:44.900   Installing tools:C:\Users\hari\miniconda3\python.exe "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py" --idf-path C:\Users\hari\esp32\esp-idf-v4.1 install
2021-01-06 16:25:45.904   Done, exit code=1
2021-01-06 16:25:45.904   --------
2021-01-06 16:25:45.904   Running command: C:\Users\hari\miniconda3\python.exe "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py" --idf-path C:\Users\hari\esp32\esp-idf-v4.1 install
Installing tools: xtensa-esp32-elf, xtensa-esp32s2-elf, esp32ulp-elf, esp32s2ulp-elf, cmake, openocd-esp32, ninja, idf-exe, ccache
Installing xtensa-esp32-elf@esp-2020r2-8.2.0
Downloading xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
WARNING: Download failure
WARNING: Failed to download https://dl.espressif.com/dl/xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
Downloading xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
WARNING: Download failure
WARNING: Failed to download https://dl.espressif.com/dl/xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
Downloading xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
WARNING: Download failure
WARNING: Failed to download https://dl.espressif.com/dl/xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
ERROR: Failed to download, and retry count has expired
Traceback (most recent call last):
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 1387, in
main(sys.argv[1:])
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 1383, in main
action_func(args)
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 1167, in action_install
tool_obj.download(tool_version)
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 584, in download
raise DownloadError()
main.DownloadError
I googled this "Download failure: .... https"  error, and on  StackOverflow found the accepted fix is to install python openssl, which I did with "conda install  openssl".  Tried installation again, same error. Rebooted and tried again, same error.

I just tried running  esp-idf-tools-setup-2.3.exe on Windows 10 x64 with previously installed miniconda python  v3.8.5, Git  2.30.0, selected esp-idf-4.1, and I get the following error (from the end of the setup log):
2021-01-06 16:25:44.893   Installation process succeeded.
2021-01-06 16:25:44.899   Setting PATH for this process: C:\Users\hari\miniconda3;C:\Program Files\Git\cmd;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\AMD\ATI.ACE\Core-Static;C:\Program Files\Git\cmd;C:\Users\hari\AppData\Local\Microsoft\WindowsApps;;C:\Modeltech_pe_edu_10.4a\win32pe_edu;C:\Users\Hari\Anlogic\TD4.6.4\bin
2021-01-06 16:25:44.899   PYTHONPATH=
2021-01-06 16:25:44.900   idf_tools.py exists in IDF directory
2021-01-06 16:25:44.900   idf_tools.py command: C:\Users\hari\miniconda3\python.exe "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py" --idf-path C:\Users\hari\esp32\esp-idf-v4.1
2021-01-06 16:25:44.900   Installing tools:C:\Users\hari\miniconda3\python.exe "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py" --idf-path C:\Users\hari\esp32\esp-idf-v4.1 install
2021-01-06 16:25:45.904   Done, exit code=1
2021-01-06 16:25:45.904   --------
2021-01-06 16:25:45.904   Running command: C:\Users\hari\miniconda3\python.exe "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py" --idf-path C:\Users\hari\esp32\esp-idf-v4.1 install
Installing tools: xtensa-esp32-elf, xtensa-esp32s2-elf, esp32ulp-elf, esp32s2ulp-elf, cmake, openocd-esp32, ninja, idf-exe, ccache
Installing xtensa-esp32-elf@esp-2020r2-8.2.0
Downloading xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
WARNING: Download failure
WARNING: Failed to download https://dl.espressif.com/dl/xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
Downloading xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
WARNING: Download failure
WARNING: Failed to download https://dl.espressif.com/dl/xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
Downloading xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
WARNING: Download failure
WARNING: Failed to download https://dl.espressif.com/dl/xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip to C:\Users\hari.espressif\dist\xtensa-esp32-elf-gcc8_2_0-esp-2020r2-win64.zip.tmp
ERROR: Failed to download, and retry count has expired
Traceback (most recent call last):
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 1387, in
main(sys.argv[1:])
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 1383, in main
action_func(args)
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 1167, in action_install
tool_obj.download(tool_version)
File "C:\Users\hari\esp32\esp-idf-v4.1\tools\idf_tools.py", line 584, in download
raise DownloadError()
main.DownloadError
I googled this "Download failure: .... https"  error, and on  StackOverflow found the accepted fix is to install python openssl, which I did with "conda install  openssl".  Tried installation again, same error. Rebooted and tried again, same error.

@har-in-air this report looks like a duplicate of https://github.com/espressif/esp-idf/issues/6367.
Please check solution in https://github.com/espressif/esp-idf/issues/6367. In case it does not help open a new ticket. This ticket is for older version of windows tools installer and closed.

@har-in-air this report looks like a duplicate of https://github.com/espressif/esp-idf/issues/6367.
Please check solution in https://github.com/espressif/esp-idf/issues/6367. In case it does not help open a new ticket. This ticket is for older version of windows tools installer and closed.

@krzychb Thanks for the info.
I was able to get the Windows 10 install and example project build working with v2.3.3 tools installer when I used the "install Python" option, instead of pointing to my existing miniconda Python 3.8.5 installation.

@krzychb Thanks for the info.
I was able to get the Windows 10 install and example project build working with v2.3.3 tools installer when I used the "install Python" option, instead of pointing to my existing miniconda Python 3.8.5 installation.

authors: MitchBradley ORParga Zeni241 andalera armandr bl4krat dipen-1337lab efsieitch gsuberland har-in-air igrr kifantidis krzychb markhu markwest paulzhn pavestru projectgus ralphcastro504 sheldon123z vmsh0 xuhongv zhongziming1999 zzeroo


text:
ESP32C3 9bit-SPI Transmit Error (IDFGH-6865)

Environment

Development Kit:      [ESP32-C3 LUATOS/ESP32C3]
Module or chip used:  [ESP32-C3]
IDF version :v4.4
Build System:  idf.py
Compiler version :8.4.0
Operating System: Windows
(Windows only) environment type: PowerShell.
Using an IDE?: Yes Vscode
Power Supply: USB

Problem Description
I have a 3-wire TFT screen, his communication protocol is 9bit, the highest bit is cmd/dat control bit, and the remaining 8 bits are communication data.
Through the vscode-idf plug-in tool, the project was created through the template spi-master. According to the sending method of 9bit spi in the manual, the content after the spi initialization was deleted, and the following code was written
(After the "ESP_ERROR_CHECK(ret);" 441lines):
uint16_t data;
data = SPI_SWAP_DATA_TX(0x145, 9);
memset(&t, 0, sizeof(t));
t.length = 9 ;
t.tx_buffer=&data;
ret=spi_device_transmit(spi, &t);
assert(ret==ESP_OK);

It's works great. I use the Logic Analyzer to view the gpio statues:

the date seems Ok,but when I change the DATE in SPI_SWAP_DATA_TX，to 0X045:
data = SPI_SWAP_DATA_TX(0x045, 9);
memset(&t, 0, sizeof(t));
t.length = 9 ;
t.tx_buffer=&data;
ret=spi_device_transmit(spi, &t);
assert(ret==ESP_OK);

The logic analyzer detected the following information:

It can be seen that the highest bit (9bit) and the lowest bit (1bit) of the SPI output seem to be bound together, which is controlled by the ninth bit of DATE in SPI_SWAP_DATA_TX. I have tried many times with different data, and the result is the same , also checked the Mode and other settings, what could be the problem, thank you very much!
Debug Logs
none

Other items if possible
none

I discovered this issue a few days ago, but @londonlove beat me to opening the ticket.
I have been using ESP32-S3 (not C3 like londonlove) but the issue appears to be similar.
Using a buffer containing [0xff, 0xff] I tested sending various size transfers from 1 to 16 bits. Two cases fail: sending 1 bit results in a 0 bit being transmitted, and sending 9 bits results in the last bit being erroneously sent as a 0. On the original ESP32, this does not happen.

(captured using esp32-s3 with code at https://gist.github.com/DavidJRobertson/0e33a51ddb2462d7c48bdc5f14a355db)
I am also using IDF v4.4

With some further testing of the issue on ESP32-S3, the last bit is always sent as zero for any transaction with 8n+1 bits. There's something special happening when the last buffer byte only has one bit used.
Workaround is to detect the 8n+1 bits case and split it into two transactions (8n-1 and 2 bits respectively). Here is an example of the workaround: https://gist.github.com/DavidJRobertson/3d077b55dce744c4c87ae6c49d0fc2cb
The workaround can't fix the case of 1 bit transactions, but these are not of much practical use anyway.

Hi @londonlove , @DavidJRobertson ,
Thanks for reporting this issue. We'll check if it's an SW or HW issue..

Hi @londonlove , @DavidJRobertson , Thanks for reporting this issue. We'll check if it's an SW or HW issue..

Thanks，I haved change the IDF branch to v4.3，and the problem is also exist，Waiting for your response

Small question: is SPI host with DMA enabled or disabled?

Hi @liaozhelin ,
We confirmed this is a limitation in the HW. The architecture of HW is based on bytes, not for bit-based transactions.
Though it happens to support all other cases except 8n + 1 bits.. You may workaround this issue with the command/addr phases.

Could the software workaround be included in the IDF? This would at least give API compatibility between ESP32 and ESP32-C3/S3

Hi @DavidJRobertson ,
The workaround would take some execution time, space, and cannot fix the issue perfectly (can fix tx only, cannot use with the DMA mode). Finally the user still need to know this limitation, and is suggested to avoid this use cases by making a descriptor that meets HW limitation.
Due to the space usage (temporary buffer should be given by the user), if there should be a layer for compatibility, it should be a helper function above the driver, instead of integrated into the driver core. Please see and try the code below:
//User only writes to trans. Other fields to be used by the helper
typedef struct {
union {
    spi_transaction_t trans;
    spi_transaction_ext_t trans_ext;
};
uint8_t temp_buf[64];
} spi_transaction_arb_ext_t;

// Only support TX MSB first, with no command, address, dummy bits, shorter than 64 bytes, no DMA enabled.
esp_err_t spi_device_queue_trans_arb_len(spi_device_handle_t handle, spi_transaction_arb_ext_t *trans_desc, TickType_t ticks_to_wait) {
   if (trans_desc->trans.rx_buffer) {
        if (trans_desc->trans.length % 8 != 0) {
             return ESP_ERR_NOT_SUPPORTED;
        } else {
            return spi_device_queue_trans(trans);
        }
    } else if (trans_desc->trans.tx_buffer) {
        if (trans_desc->trans.length > 64 * 8) {
            return ESP_ERR_NOT_SUPPORTED;
        }
    }

   if (trans_desc->trans.rx_buffer) {
        return spi_device_queue_trans(handle, trans_desc, ticks_to_wait);
    } else if (trans_desc->trans.tx_buffer) {
        if (trans_desc->trans.length % 8 == 0) {
            return spi_device_queue_trans(handle, trans_desc, ticks_to_wait);
        }
    }

    int bit_remain = trans_desc->trans.length % 8;
    trans_desc->trans.cmd = cmd = ((uint8_t*)trans_desc->trans.tx_buffer)[0] >> (8-bit_remain);
    trans_desc->trans.flags |= SPI_TRANS_VARIABLE_CMD_ADR;

    trans_desc->trans_ext.command_bits = bit_remain;
    trans_desc->trans_ext.address_bits = 0;
    trans_desc->trans_ext.dummy_bits = 0;

    for (int i = 0; i < trans_desc->trans.length/8; i++) {
        trans->temp_buf[i] = ((uint8_t*)trans_desc->trans.tx_buffer)[i] << bit_remain | ((uint8_t*)trans_desc->trans.tx_buffer)[i+1] >> (8-bit_remain);
    }

    return spi_device_queue_trans(handle, trans_desc, ticks_to_wait);
}

Issue closed, feel free to re-open it if you find more problems.

authors: DavidJRobertson Maksons ginkgm londonlove


text:
esp_ota_begin called with large image size (1263952) causes a WDT reset whit CONFIG_ESP32_WIFI_SW_COEXIST_ENABLE = y (IDFGH-10062)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
4.4.1 and 4.4.2
Operating System used.
Linux
How did you build your project?
VS Code IDE
If you are using Windows, please specify command line type.
No response
Development Kit.
Custom board
Power Supply used.
USB
What is the expected behavior?
The ota partition is initialized
What is the actual behavior?
WDT reset info
Steps to reproduce.

Step  Set CONFIG_ESP32_WIFI_SW_COEXIST_ENABLE = y
Call esp_ota_begin with large image size (1263952)

Debug Logs.
D (38517) CMD_MAN: Start erase partition firmwareLen: 1263952     <--This is printed before esp_ota_begin
ets Jun  8 2016 00:22:57

rst:0x8 (TG1WDT_SYS_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:6608
load:0x40078000,len:14788
ho 0 tail 12 room 4
load:0x40080400,len:3792
0x40080400: _init at ??:?

entry 0x40080694
I (171) boot: ESP-IDF v4.4.1 2nd stage bootloader
I (171) boot: compile time 16:42:46
I (172) boot: chip revision: 1
I (175) boot.esp32: SPI Speed      : 40MHz
I (180) boot.esp32: SPI Mode       : DIO
I (185) boot.esp32: SPI Flash Size : 16MB
W (189) boot.esp32: PRO CPU has been reset by WDT.
W (195) boot.esp32: WDT reset info: PRO CPU PC=0x400945e1
0x400945e1: _xt_context_save at /home/famar/Works/toolchain/esp/esp-idf-v4.4.1/components/freertos/port/xtensa/xtensa_context.S:209

W (201) boot.esp32: WDT reset info: APP CPU PC=0x40081f6d
0x40081f6d: panicHandler at /home/famar/Works/toolchain/esp/esp-idf-v4.4.1/components/esp_system/port/panic_handler.c:213

I (207) boot: Enabling RNG early entropy source...
I (212) boot: Partition Table:
I (216) boot: ## Label            Usage          Type ST Offset   Length
I (223) boot:  0 otadata          OTA data         01 00 00009000 00002000
I (231) boot:  1 nvs              WiFi data        01 02 0000b000 00505000
I (239) boot:  2 ota_0            OTA app          00 10 00510000 00500000
I (246) boot:  3 ota_1            OTA app          00 11 00a10000 00500000

More Information.
I executed the same test using esp idf 4.4.2 and I got the same result.
st:0x8 (TG1WDT_SYS_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:6660
load:0x40078000,len:14848
ho 0 tail 12 room 4
load:0x40080400,len:3792
0x40080400: _init at ??:?

entry 0x40080694
I (171) boot: ESP-IDF v4.4.2 2nd stage bootloader
I (171) boot: compile time 08:38:19
I (172) boot: chip revision: 1
I (175) boot.esp32: SPI Speed      : 40MHz
I (180) boot.esp32: SPI Mode       : DIO
I (185) boot.esp32: SPI Flash Size : 16MB
W (189) boot.esp32: PRO CPU has been reset by WDT.
W (195) boot.esp32: WDT reset info: PRO CPU PC=0x40081f91
0x40081f91: panicHandler at /home/famar/Works/toolchain/esp/esp-idf-v4.4.2/components/esp_system/port/panic_handler.c:213

W (201) boot.esp32: WDT reset info: APP CPU PC=0x40081f91
0x40081f91: panicHandler at /home/famar/Works/toolchain/esp/esp-idf-v4.4.2/components/esp_system/port/panic_handler.c:213

I (207) boot: Enabling RNG early entropy source...

When CONFIG_ESP32_WIFI_SW_COEXIST_ENABLE is not set, esp_ota_begin ends succesfully in 4.4 s
Thanks

@ffamar

Do you still hit the issue with latest v4.4.4 release?
Can you share your sdkconfig?

Hi @ffamar , I would also like to ask the questions asked by @AxelLin . Besides, can you provide more contexts of your application? What happens before you call esp_ota_begin? Do you use Wi-Fi and Bluetooth, if yes, what are their activities before you call esp_ota_begin?

@esp-lrh I migrated my firmware to idf version 5.0.6 and the bug does not exist.

authors: AxelLin ffamar wmy-espressif


text:
fix(esp_netif): Fixed initializer order of ESP_IP macros for C++ (IDFGH-13520)

The designated initializer order has to be the same as member declaration order in C++. This change enables C++ users to use ESP_IP4ADDR_INIT and ESP_IP6ADDR_INIT macros.

sha=f371940992bf0fe0e1e273b8d37edaf92fa051cf

Thanks for contribution again, changes have been merged with https://github.com/espressif/esp-idf/commit/1889851d7c04e225017357ca230336323593a8c5.

source: gharchive/pull-request


text:
Is there any API for discover attributes? (TZ-204)

Hi, i used this library to develop a ZigBee gateway.
For the reason of want connect to most zigbee device, I need to discover the attributes for cluster.
However, I read all the examples and didn't find some api that can do this, there is something like esp_zb_zdo_find_on_off_light, but it's not what I want.
I searched for information on Google again and found the ZCL command 0x0c, but in the online documentation of esp_zigbee_lib, I did not find the implementation of this command; there is a related command esp_zb_zcl_get_attribute, but this command is only for local.

Hi,
Could you please provide a detailed description of the method you would like to use for discovering attributes for a cluster in a Zigbee network? Specifically, I'd like to know what parameters you can provide to find the attribute.
The ESP-Zigbee-SDK offers two essential functions to aid in this process: esp_zb_zdo_active_ep_req() is used to search for endpoints in the Zigbee network, and esp_zb_zdo_simple_desc_req() is employed to find the clusters associated with specific endpoints. Are you interested in a command that allows you to retrieve an attribute list based on a specific endpoint ID and cluster ID?

Hi @xieqinan, thanks for your reply.

Specifically, I'd like to know what parameters you can provide to find the attribute.

Yes, I can provide node address (16-bit short address), endpoint number, cluster ID, callback function, callback function context.

esp_zb_zdo_active_ep_req() is used to search for endpoints in the Zigbee network, and esp_zb_zdo_simple_desc_req() is employed to find the clusters associated with specific endpoints.

I have used these two functions, and the clusterID list provided by esp_zb_af_simple_desc_1_1_t as the input parameter of the callback function only contains two clusterIDs, which is too few, and some devices have many clusterIDs under one endpoint. Also, I also did not find an API that can discover all clusterIDs.

Are you interested in a command that allows you to retrieve an attribute list based on a specific endpoint ID and cluster ID?

Yes, I am interested in finding attribute by specific endpoint and specific clusterID.
In the documentation, there are many APIs related to HA creation, cluster creation, and attribute creation, but the APIs related to HA discovery, cluster discovery, and attribute discovery are not perfect. This is friendly for creating End devices, but unfriendly for creating coordinators.
I'm not very familiar with Zigbee, so sorry for any mistakes or omissions.

Hi,

I have used these two functions, and the clusterID list provided by esp_zb_af_simple_desc_1_1_t as the input parameter of the callback function only contains two clusterIDs, which is too few, and some devices have many clusterIDs under one endpoint. Also, I also did not find an API that can discover all clusterIDs.

The esp_zb_af_simple_desc_1_1_t provides a clusterID list whose size is app_input_cluster_count + app_output_cluster_count, rather than containing only two clusterIDs.

In the documentation, there are many APIs related to HA creation, cluster creation, and attribute creation, but the APIs related to HA discovery, cluster discovery, and attribute discovery are not perfect. This is friendly for creating End devices, but unfriendly for creating coordinators.

Thank you for your feedback and suggestion, the discovery feature will be added in August.

Hello,
The API for discovering attributes has been supported in esp-zigbee-sdk.

@xieqinan, can you give a hint on how to correctly use this API?
I have the following code:
                                esp_zb_zcl_disc_attr_cmd_t cmd;
                                cmd.address_mode = ESP_ZB_APS_ADDR_MODE_16_ENDP_PRESENT;
                                cmd.cluster_id = simple_desc->app_cluster_list[i]; // <-- id from esp_zb_zdo_simple_desc_req
                                cmd.direction = ESP_ZB_ZCL_CMD_DIRECTION_TO_SRV;
                                cmd.max_attr_number = 10;
                                cmd.start_attr_id = 0;
                                cmd.zcl_basic_cmd.dst_addr_u.addr_short = 0x5118; // <-- matches device
                                cmd.zcl_basic_cmd.dst_endpoint = 1;
                                cmd.zcl_basic_cmd.src_endpoint = 1;
                                auto seq = esp_zb_zcl_disc_attr_cmd_req(&cmd);

but in my callback on esp_zb_core_action_handler_register I get status_code=0x82 (ESP_ZB_ZCL_STATUS_UNSUP_GEN_CMD).
I'm trying using a Sonoff SNZB-02 sensor as my test-device currently; my final application requires me to discover the attributes.

It's odd that such a basic example does not exist. A simple coordinator that forms a network and then allows joining of new devices forever and display all the nodes, clusters and attributes of each such connected device in a nice, easy-to-read way.
And this example not be be locked behind closed API, like the esp_zb_zdo_find_on_off_light function is. Too much stuff is behind the closed source and without proper documentation and examples for the exposed cryptic functions. Also, why is this issue closed?
In my case, I'm trying to have my coordinator allow any type of off-the-shelf home automation (HA) zigbee device to connect and then read their data.

authors: Cycov Zmmfly petergjoel xieqinan

repo: espressif/esp-zigbee-sdk


text:
openstmap: move exit to bottom of menu again

Reverts c4631cfe5f8bdc0fad3ce3d163a790d81a6078c8, see #3169
PR as Draft until clarified wether to revert Math.round() in latLonToXY()

Thanks!

authors: gfwilliams nxdefiant

repo: espruino/BangleApps

license: MIT


text:
Split type_check.rs into smaller modules.

I have been updating type_check.rs recently and noticed the file size was getting over 3000 lines.  So I thought 'we should split that up, it's too big'.
This is the traditional idea; I was told that a source file should never go above 1000 lines and refactoring it into smaller modules is the answer.  But this was in the olden days, when a) compilers (and disks and/or filesystems) were slower, and b) before we had LSP for every language under the sun, allowing simple access to any parts of a codebase with fuzzy finding.
I went and split type_check.rs anyway.  But I'm thinking this actually won't make much difference to how I work anyway.  As above, I'd use LSP to find what I need and which file or module it's in doesn't really matter.  It's good to have similar code together so you're not jumping around when using prior art as inspiration, but a 3000 line file perhaps isn't the end of the world any more..?
So IDK, perhaps this PR isn't worth merging?  I'd be interested to hear opinions.  @mohammadfawaz @lethalgem
The changes are purely at the method level of granularity, nothing inside a method has changed, just which file/module it's in.  If you have changes to type_check.rs in a working branch then it might be a pain to merge since Git can't do it for new files.

I don't have any strongly held opinions here. Mostly looking to ya'll to learn and gain experience around when files should be split up.
to preface: I've heard this about both files and functions
In the past I've heard that files shouldn't be bigger than even a few hundred lines (and functions less than 20 lines, if you can believe it). That the bigger the file was, the more it was a hint to abstract away granular pieces so that the reader could understand the high level logic faster. I worked with a project like this. Nearly every file was kept to under 100 lines. I found it to be a minor annoyance. Just constantly clicking around different files to find where I needed to make changes.
Recently I've heard the opposite. That this brings in more indirection and makes it harder to find what you're looking for. Having related methods be in close proximity makes it easier to scan and end up where you need to be, faster.
Looking at the changes, I like splitting up this file. I almost never find myself scrolling through the entire type_check file. I either search, or cmd+click to follow the definitions of functions until I end up where I need to make changes. Plus, the files are well named and make it easy enough to navigate without the aid of the editor.

In the past I've heard that files shouldn't be bigger than even a few hundred lines (and functions less than 20 lines, if you can believe it).
Recently I've heard the opposite. That this brings in more indirection and makes it harder to find what you're looking for. Having related methods be in close proximity makes it easier to scan and end up where you need to be, faster.

Yeah, Java introduced the idea of a single class per file, and they have to have the same name, and many C++ projects I worked on were the same.  No matter how small your class was it had to go in its own file.  And if you couldn't see an entire function on your screen at once then it needed to be split up.  Nice ideas but I think they're from the before-times, when editors and IDEs were shit.
In the very early days of my career we had a codebase which put all function names at the start of the line, so you could find them using a ^name regex search:
int
count_frogs(struct pond_t* pond)
{
    ...
}

authors: lethalgem otrho

repo: essential-contributions/pint


text:
Bugfix/fav

What does this PR?
Where should the reviewer start?
Steps to reproduce
Issue number
Screenshots/Video

conflict @ue

authors: mistikk ue

repo: esteemapp/esteem-mobile


text:
Backported fix for prev arrow on decade and century view.

Closes #1720 and #1719

LGTM

authors: acrobat hebbet

repo: eternicode/bootstrap-datepicker

license_source: bigquery


text:
Add tests to ensure working code

Motivation
To ship code with the security that it won't break things.
Testing enviroment
There are two ways I test when using lua:

vusted
plenary.nvim

To use them we just have to create a tests folder (in the root for example) and for plenary.nvim a minimal_init.{lua|vim} (I prefer .lua) and testing files following the convention of name_of_the_file_spec.lua, the _spec.lua will be detected as tests files.
-- minimal_init.lua
local plenary_dir = os.getenv("PLENARY_DIR") or "/tmp/plenary.nvim"
local is_not_a_directory = vim.fn.isdirectory(plenary_dir) == 0
if is_not_a_directory then
  vim.fn.system({ "git", "clone", "https://github.com/nvim-lua/plenary.nvim", plenary_dir })
end

vim.opt.rtp:append(".")
vim.opt.rtp:append(plenary_dir)

vim.cmd("runtime plugin/plenary.vim")
require("plenary.busted")

Commands to run

vusted

vusted ./tests/freeze

plenary.nvim

nvim --headless --noplugin -u tests/minimal_init.lua -c "PlenaryBustedDirectory tests/freeze {minimal_init = 'tests/minimal_init.lua'}"

Tests files
-- ./tests/freeze/setup_freeze_spec.lua
local freeze = require("freeze")

describe("[freeze.nvim tests]", function()
  describe("setup", function()
    it("sets up with default config", function()
      local expected = {
	  dir = ".",
	  output = default_output,
	  theme = "default",
	  config = "base",
	  open = false,
      }
      local actual = require("freeze")
      freeze.setup(expected)
      assert.are.same(expected, actual.opts)
    end)
    it("sets up with custom config", function()
      local opts = {
        dir = "/tmp/freeze_images",
        theme = "rose-pine-moon",
      }
      freeze.setup(opts)
      local actual = freeze.opts
      assert.are.same(opts.dir, actual.dir)
      -- or assert.are.same("/tmp/freeze_images", actual.dir)
      assert.are.same(opts.theme, actual.theme)
      -- or assert.are.same("rose-pine-moon", actual.theme)
      assert.are.same(opts, actual)
    end)
  end)
end)

I like the bones proposed in #11 but I prefer to use Nix to handle my CI. Adopting Nix will also allow for us to specify and use devshells for development.
I am open to these suggestions but tests should be a separate PR from any CI changes (to include the included Makefile).
Additionally, I make no guarantees on running testing on Windows as this is not an environment I plan to support (although it should work fine).
Lastly, I will be offline for awhile so do not expect movement on this project for the near future.

@ethanholz I removed the workflow on the PR, and added the tests for using freeze command.

authors: AlejandroSuero ethanholz


text:
MongoDB - Update to avoid deprecation warnings

Replace update() with updateOne() or updateMany()
Replace remove() with deleteOne() or deleteMany()

https://github.com/ether/ueberDB/pull/352

Updated in #371

authors: JohnMcLear SamTV12345 fdemello

repo: ether/ueberDB


text:
updating gas limits and fp contracts

Adding a New Chain
This PR update Base Mainnet and Sepolia in the registry. It updates the gas limits to the latest values pulled from the SystemConfig contracts. It also adds Base Mainnet's newly deployed fault proof contracts.
Checklist

[ ] I have declared the chain at the appropriate Superchain Level.
[x] I have run just validate <chain-id> locally to ensure all local validation checks pass.
[x] I have run just codegen to ensure that the chainlist and other generated files are up-to-date and include my chain.

@mds1 and @bitwiseguy the gas limit questions are good to get to the bottom of so I recorded this issue and pulled them out of this PR because its more important to get these fault proof addresses documented.

repo: ethereum-optimism/superchain-registry


text:
Remove bitlist length bit from SSZ

There was a change recently that changed the bitfield types to Bitlist[N] in #1224. Part of this change was to also change SSZ to include special handling for the underlying data type, a byte array.
I've talked with a few folks about the rationale for the changes in #1224. From what I understand, these changes were included for readability in the Python specification/implementation. However, this added a change that isn't strictly related to readability: Bitlist bit length in SSZ encoding.

Bitlist[N]
Note that from the offset coding, the length (in bytes) of the bitlist is known. An additional leading 1 bit is added so that the length in bits will also be known.
as_integer = (1 << len(value)) + sum([value[i] << i for i in range(len(value))])
return as_integer.to_bytes((as_integer.bit_length() + 7) // 8, "little")

specs/simple-serialize.md
Prior to the add bitlist length in SSZ, we checked the number of bytes in the bitfield byte array to ensure it was the expected length. Maybe an issue here is that you can only check for fields in multiples of 8 bits (1 byte). So if we had a bitfield that represented indices in a committee where the committee length was 10 then we would check the bitfield had 2 bytes, but there were 6 bits unused. An additional layer of verification could be to set the 6 most significant bits to 0 in the second byte or simple disregard their values entirely. In either case, we had a solution without Bitlist length in SSZ. The length enforcement should be handled in the application layer rather than the serialization layer.
With these changes, we now have this added consequence of a non-primitive type handling in SSZ. Up until this PR, you could use SSZ and other encoders interchangeably with adding special rules. You could encode any data structure with a bitfield to JSON or others where the underlying bytes array would naturally be appropriately represented in the same way. We have to treat some byte arrays different than other byte arrays and this adds unnecessary complexity. The key takeaway here is that we could previously fully populate an SSZ yaml file to a data structure using the existing serialization tooling for yaml and/or json. i.e. byte arrays were simply byte arrays, not byte arrays with some extra metadata.
If its helpful, I had started collecting notes on this change and trying to piece together the puzzle of #1224. Link
There's a good bit about this in @rauljordan's article in the section called "Why Not More Basic Types!?" link
TL;DR - Bitfield length information in SSZ is unnecessary complexity and we should remove it. Let's keep simple serialize simple.

+1 on this. Although it adds further readability on perhaps the Python implementation, it pushes further complexity on languages, such as GO, which already have a harder time implementing SSZ due to a lack of generics. Having bitlists and bitvectors as primitives as @prestonvanloon mentioned also make implementation assumptions within SSZ that limit us from using the tooling we use in Prysm with other encoders. We kindly ask researchers to reconsider this change and possibly revert in favor of keeping SSZ simple.

Since I was the driver behind this change I am very happy to answer any questions about it. The way I understand it, the main criticism is this:

The length enforcement should be handled in the application layer rather than the serialization layer.

It was the main intention of this PR to have the length check in the SSZ layer. The reason is that we came across a number of these bitfields in different places, and it seemed that it would be much more natural to have this length check at a lower layer. We think it does increase readability of the spec.

You could encode any data structure with a bitfield to JSON or others where the underlying bytes array would naturally be appropriately represented in the same way. We have to treat some byte arrays different than other byte arrays and this adds unnecessary complexity.

I don't understand this part, could you clarify? My recommendation is that you just don't treat bitfields as bytes, but as a separate object, that is just a list of booleans with a special way of serializing it. That should be possible in any implementation, even if you serialize to yaml or JSON?
Here are some answers to your question in the Hackmd:

This appears to have moved from the phase 0 spec into the custody game spec for phase 1. It also appears to have been renamed to get_bit.

This is just temporary because there will be larger changes to the phase 1, and we will make it completely use the SSZ bitfields. There will be no more need for get_bit, it will all be handled by SSZ.

What isn’t clear is how does ssz know what the committee size is? Removing this validation loosens overall validation on justification and finalization during epoch processing. I recommend client implementors keep this validation and we restore it in the specification.

SSZ does not know the committee size, but it now knows the bitfield size. There was a bug where this size was not checked, which has now been fixed by Justin in #1264 .

Bitlists with padding bit seems like a change we don’t need given the added cost and no clear benefit.

The benefit is that we have a well defined length of every bitfield object.

I don't understand this part, could you clarify? My recommendation is that you just don't treat bitfields as bytes, but as a separate object, that is just a list of booleans with a special way of serializing it. That should be possible in any implementation, even if you serialize to yaml or JSON?

I'll try to come up with a contrived example.
Imagine that we have a bitlist of length 8 where the data is 0b00111111.
Also imagine we have a data structure that exists of primitive types.
type Data struct {
   Num uint64      `json:"num"`
   Bitfield []byte `json:"bitifeld"`
   Name string     `json:"name"`
}

Now if we look at the JSON representation of this object:
{
   "num": 5,
   "bitfield": "8w==", // Or 0xF3 if you prefer
    "string": "hello"
}

This JSON can populate the go struct without any special handling or processing. It just works.
Before #1226, the SSZ representation of the bitfield would be 0xF3 as well. These data representations are interchangeable between the two.
Now, you have this padding bit metadata. So the bitlist would be serialized to 0x3F01 in SSZ. If we were to put this same data in JSON and use that data to populate our primitive []byte field, we would have data that looks like
[]byte{0b00111111, 0b00000001}

So now we have to do a second round a processing this data.

Populate the structure with the data from JSON
Then go back and clean up any bitfields, removing the length bit.

Likewise with marshaling to JSON and remain compatible with SSZ primitives

Edit any bitfields to add the padding bit
Marshal to JSON

An alternative is to change our struct to
type Data struct {
   Num uint64      `json:"num"`
   Bitfield struct{
      Data []byte `json:"data"`
      BitsLength uint64 `json:"length"`
   } `json:"bitifeld"`
   Name string     `json:"name"`
}

But now our JSON object from earlier doesn't map so we have to change that representation to the following:
{
   "num": 5,
   "bitfield": {
       "data": "8w==", // Or 0xF3 if you prefer
        "length": 8,
     },
    "string": "hello"
}

SSZ does not know the committee size, but it now knows the bitfield size.

SSZ doesn't need to know either of these things. The bitlist is an unbounded/dynamic array of bytes in ssz. I offered a solution in the original issue description for handling expected bitlist lengths that were not a multiple of 8.

The benefit is that we have a well defined length of every bitfield object.

We already knew what to expect for bits length in processing.
I strong urge you to reconsider this change. It may improve readability in python but comes with a non-trivial increase in complexity.

What if you just have your bitfield Bitfield []byte already be in the serialized SSZ format? Then all the information would be contained in this data structure.

SSZ doesn't need to know either of these things.

The argument here is that SSZ is a strictly better typing library if it handles lengths of lists safely, i.e. it is aware of the exact length of a list and can catch an out of bounds access. This change makes bitlists behave in exactly the same way any other kind of lists behave. It pushes a very slight bit of complexity into the SSZ layer, and removes a larger amount of complexity from the application layer.

While I do think it is natural to enforce bitfield list lengths in consensus via Merkleisation length mix-ins, there is wiggle room to experiment with alternative serialisations if the currently suggested serialisation is too painful to deal with. Indeed, the consensus part of phase 0 (deposit contract, state transition function, fork choice rule) is agnostic to serialisation. In other words, serialisation is an implementation detail not part of the remit of the spec freeze.

Indeed, the consensus part of phase 0 (deposit contract, state transition function, fork choice rule) is agnostic to serialisation

I would think alternative serializations would have an impact to changing state transition function. Ex: if we don't enforce length check in the SSZ layer, state transition will now need to check block body's operation lengths, validator length for shuffling...etc

Go is somewhat painfull here due to the lack of static-type parametrization, but you can work around it reasonably well. I.e. you keep the leading bit in memory for bitlists to derive the length from. And for bitvectors you declare it as byte array (fixed size) and put a Count method on it to tell the bit length. Your SSZ lib can instantiate a default value, and derive the length from there, since there are no type/class/static-methods in Go to work with. Or you make Count run on a pointer receiver, and you can use a typed nil to call the bit-count function on.
Here's a POC: https://play.golang.org/p/fiCVRA5_1jS
This should enable you to read it from hex/yaml, like any other []byte and [N]byte. The functions attached to the type will handle the logic, no constructor or encode-decode overrides necessary. Go sure is different than Python, JS, etc.
Bitvectors cost you a few lines to define a different length, but there's simply no generics/parametrization/inheritance to do it with. And it's not a struct, so no embedding either. But this works well enough, although Go certainly is not "pretty" here.
Carelessly being able to convert from Bitvector[:] to []byte to Bitlist is the benefit however, and SSZ serialization/deserialization is basically free (copy over bytes in one go, check length, and done).

Hi @dankrad

What if you just have your bitfield Bitfield []byte already be in the serialized SSZ format? Then all the information would be contained in this data structure.

This is highly non-standard and enforces one more cognitive leap for someone attempting to read our serialization code + use our libraries. It adds more "gotchas" that shouldn't exist in the first place. I can't think of an encoding library that would do this. We certainly would never expect the canonical Go JSON encoder to enforce that a list of something never exceeds 10 elements, and for good reason.
@protolambda

Bitvectors cost you a few lines to define a different length, but there's simply no generics/parametrization/inheritance to do it with. And it's not a struct, so no embedding either. But this works well enough, although Go certainly is not "pretty" here.

Why do we need to work around that? Why not just handle it at the application layer in the first place? Aside from being quite messy and difficult to grok, using something akin to that Go playground would add large complexity to our SSZ and lack of readability. That also makes assumptions about the structure of our SSZ library. It is unreasonable to need to do that sort of logic for something that could have been avoided in the first place.
Our point is less about the difficulty of doing it in Go and more that this adds significant constraints to our tooling and forces SSZ to make assumptions that other serialization libraries would never enforce given they are supposed to be agnostic to use cases.
When approaching SSZ, we sought out to build it following the mindset and API's of other popular, agnostic serialization libraries that focus simplicity and do not take on application layer or use-case complexity. This change means we have to use SSZ for everything, and encourages lock-in, as it is what does length enforcement instead of it living in the application layer.
On behalf of our entire team and keeping in mind the goals of SSZ as a generic, agnostic, and simple serialization library, we kindly ask to please reconsider this change, as v0.7.1 SSZ maintained a higher level of simplicity and a more robust, agnostic specification with respect to these primitives than what we see in v0.8's design decisions.

My recommendation is that you just don't treat bitfields as bytes, but as a separate object, that is just a list of booleans with a special way of serializing it.

I think part of the friction is around the fact that practically speaking, any performant implementation is going to want to represent a bitlist as bytes, like a bitfield. But by imbuing a bitlist with its length information, we can no longer just use the bytes primitive as is. This is the first type that is not a primitive in any language to be added to ssz. The non-standard convention of the padding bit / length information practically requires an additional type to be supported.
Its this disconnect between this pseudo-bitfield and the widely-known pattern of bitfield usage that is a little troubling. The switch to thinking about bitfields as "lists of bits" creates different machinery and potentially different semantics than a normal bitfield.
Does the bitlist need to grow/shrink? Does bit access beyond the bitlength but within the bytelength throw an error? What operations does it support? These are new questions that need to be addressed by a 'bitlist' implementor. The lack of specification and unknown future usage of bitlists makes it difficult to build out what amounts to a custom data type library.
I'm curious if there are known tangible benefits from tracking the bitlist length? I understand the typing is more robust, but why do we need/want it within ssz? What will we need to be doing with bitlists that we can't be doing with bitfields/bytes in our application?

This is tricky because SSZ implicitly assumes that there are two levels of "staticness of length" of a type: (i) known at "compile time", (ii) not known until you receive the object. Case (i) corresponds to Vector[foo, n], case (ii) corresponds to List[foo,n]. But attestations are a strange third type, where the length is NOT known at compile time, but it's also known before we get the list, because it's proportional to the length of the committee size. So a variable length type is necessary, but we don't technically have to specify the exact length. So this specific kind of dynamicness that we see with attestations is worst-case for SSZ.
I could foresee usecases in the future where the fully dynamic bitlist is valuable; representing Merkle paths in SSZ objects is one big example.
I'm personally okay with going back to bitfields being an application-layer concept if desired; it's a small change either way.

Pros:

None?
Spec simplicity (depending on your perspective)

Cons:

When len(bitfield) % 8 == 0, the encoding size increases by 1 byte
The expected length can be inferred at runtime for verification already
Including the bitlist length in ssz encoding requires workarounds
Most implementations would verify the expected length in runtime processing at the application layer in addition to decoding any ssz container
Increased complexity of ssz libraries

I think its a clear win to revert this, especially if its a small change. Let me know what you think.

if we don't enforce length check in the SSZ layer, state transition will now need to check block body's operation lengths, validator length for shuffling...etc

Right, you can do that if you want :) The spec freeze does not prevent implementers from not enforcing the length check in their SSZ layer, and instead pushing it the check at a higher layer to keep the abstract state transition equivalent and maintain compatibility with the frozen spec. It's an implementation detail.

Pros:
* None?

* Spec simplicity (depending on your perspective)

Consistency with other kinds of lists (all of them have a clearly defined length, bitlist is the exception here -- why should a programmer have to think different about bitlists compared to other kinds of lists?)
Better readability of the spec (and also any other program that makes use of SSZ+bitlists in the future)
Type safety (In this particular case, the length is known from another place, but in general, it is preferrable that all types come with the same kind of guarantees)
assert len(bitlist) == expected_length is easier to read and reason about then verify_bitlist(...)

Cons:
* When len(bitfield) % 8 == 0, the encoding size increases by 1 byte

This difference is extremely small (one bit on average!)

* The expected length can be inferred at runtime for verification already

That's not a con, that's a neutral one. It only applies in this case.

* Including the bitlist length in ssz encoding requires workarounds
* Most implementations would verify the expected length in runtime processing at the application layer in addition to decoding any ssz container
* Increased complexity of ssz libraries

I think the increase in complexity here is a bit overblown :)

I think its a clear win to revert this, especially if its a small change. Let me know what you think.
Well, I think your list of pros and cons is a bit biased. I think we're all willing to engage in the discussion here, but please let's keep it at the technical layer and not an emotional one.

@prestonvanloon: Would you be happier if every Bitlist was replaced with Container{bytes, length} and the length check done in the state transition function?

@prestonvanloon Would you be happier if every Bitlist[N] was replaced by Container{bits: Bitvector[N], length: uint256}, with the length check done in the state transition function? :)

I just had a quick chat with @prestonvanloon about this. Currently, the solutions available to us seem:

Go with @protolambda's POC, which basically always uses the serialized version in memory. Then Bitlists can still be represented by bytes without a problem with naturally using yaml/json libraries.

In this case, the SSZ libraries should still present the appropriate methods for getting/setting bits and for getting the length, with the corresponding safety in terms of length checking.
We might want to consider to change Merkleizationif we go this way, because currently, the length checking bit is not added for Merkleization [Disadvantage: a bitlist of max length 2^n could have a bitlength of 2^n+1, needing 2^(n+1) bits at the bottom of the Merkle tree effectively...]

@JustinDrake's suggestion; another possibility would be to use this as the representation of the Bitlist inside SSZ, thus getting rid of the leading bit

Removing Bitlists entirely and going back to the status before the change, with separate Bitlist checks inside the spec

Thanks for the recap @dankrad. The offline discussion was much needed.
The primary motivation for using a bitlist abstraction is that the bit getters/setters are much more intuitive than bitwise operations. Different languages may represent and maintain the underlying data in different ways, i.e. python might be a list of booleans while others might use some container with bytes / length or use the ssz encoded format with the length bit present.
The trade off is clear: you cannot simply construct this bitlist abstraction without the length information.
With that said, our original position was option 3 as we did not fully understand the trade offs. Now, we are leaning more towards option 1 where we can keep the length information in the optimized form as suggested by @dankrad with an abstraction pattern suggested by @protolambda or a backing container in our implementation as @JustinDrake suggested.

I'll go ahead and close this.
Happy to reopen if anyone has any further ideas or comments. Thanks all!

authors: JustinDrake dankrad prestonvanloon protolambda rauljordan terencechain vbuterin wemeetagain


text:
Block header signature issue

state.latest_block_header.signature is different between producing and processing a block and this is causing block processing to fail because of a mismatch on block.state_root.
Producing
When calling per_block_processing, state.latest_block_header.signature is EMPTY_SIGNATURE.
It's impossible to have the real block signature here as you need the output of per_block_processing(..) (the updated state root) before you can produce the state.latest_block_header.signature and the value of state.latest_block_header.signature changes the state root, which changes the signature, and so on.
Processing
When calling per_block_processing, state.latest_block_header.signature is (hopefully) a real block signature.

Initially, I thought this could be fixed by ensuring that get_temporary_block_header(..) uses EMPTY_SIGNATURE instead of block.signature.
However, that raises another issue: all the hashes in latest block roots are just hashes of:

slot
previous_block_root
block_body_root

We no longer have signature, which represented a ton of information (state root, fork, etc).
I'm guessing this arose as a desire to remove previous_block_root as a parameter to per_slot_processing. Thinking about that some more, I'm not sure you can avoid passing the previous block (or its signature) to the next state. If the block needs to sign across the state, it's impossible for the state to contain the block signature. Your only solution is to store the block signature and pass it into a state that isn't included in the block signature (the next state).

Yeah, you're right there's a circular dependency. If we want the pre-state root of block N+1 to include a commitment to the signature, then the signature can't be over the post-state-root of block N as that's the same as the pre-state root of block N+1.
I'm inclined to say get_temporary_block_header using EMPTY_SIGNATURE is a pretty clean solution. But note that this does introduce a slight change to protocol properties: the block signature is no longer part of the block hash. I suppose this is meaningless because at least with the current BLS scheme there's only one possible valid signature for each block hash anyway, but it is something that we'd need to keep in mind in case we change the signature algo in the future (eg. we'd need to guard against DoS issues arising from a proposer issuing 1000 identical blocks with 1000 different but all valid signatures).

We no longer have signature, which represented a ton of information (state root, fork, etc).

I don't think we need to worry about this, because the state root itself does get included in the header (at the start of processing the next block).

there's a circular dependency

Argh, I messed up!

I suppose this is meaningless because at least with the current BLS scheme there's only one possible valid signature for each block hash anyway

One of the considerations discussed with @djrtwo is cleanliness around identifier schemes. We probably want to have a consistent "canonical" notion of message identifier, if anything for block explorers. If we exclude signatures then we may end up with the two types of identifiers used in the wild. Also, if we exclude signatures from block identifiers we may want to the same thing for attestations, transfers, etc.
Maybe the best approach is simply to revert to a (pre_state, previous_block_root, block) -> (post_state, error) state transition function with "oracle access" to previous_block_root. How much do you value a (pre_state, block) -> (post_state, error) state transition function @vbuterin?

Another approach would be to add a previous_block_signature field to BeaconBlocks.

Ok, I think I've found a satisfactory solution (see https://github.com/ethereum/eth2.0-specs/pull/799). Basically we remove latest_block_header and instead cache hash_tree_root(block) after checking block.state_root == hash_tree_root(state). This "hack" allows us to maintain a clean (pre_state, block) -> (post_state, error) state transition function.

I'm currently thinking that the cleanest way forward is to use signed_root(block) for both signing and prev_block purposes. This allows us to use the current mechanics and avoid any sort of oracle because the relevant block information is cached in the latest_block_header except for the state_root which cached at the next slot.
@JustinDrake We can use signed_root(obj) for "canonical" identifier for anything that is signed and hash_tree_root otherwise.

Basically we remove latest_block_header and instead cache hash_tree_root(block) after checking block.state_root == hash_tree_root(state).

The main challenge with this is that it introduces another point of ugliness which is the distinction between "signed post-state root" and "real post-state root".

Here's what seems to be a complete categorization of alternatives:

Roots that go in the state don't include signatures, and those signatures forever "disappear" from the state transition function's knowledge; they never get merkled in anywhere
Signatures get added in after calculating post-state roots
Signatures get passed as an argument to the next block's state transition function
The block gets passed as an argument to the next block's state transition function
The next block includes the signature of the previous block. At network layer, blocks have to be bundled with proposer signatures.

(1) is current status quo, (4) is original status quo. (5) seems bad because it means that "a chain" does not contain enough information to build on top of the chain (namely, the signature).

I'm opting for (1) with the modification that we cache block roots as signed_root(block) to be very explicit about what the root is and not have to have clients set block.signature = EMPTY_SIGNATURE before getting a block root.
PR here https://github.com/ethereum/eth2.0-specs/pull/816

This was fixed in 0.5.1

authors: JustinDrake djrtwo paulhauner vbuterin


text:
Implemented opcodes CALL and CALLCODE

Tickets
closes #230
depends on #280
What was implemented?
Implemented opcodes CALL and CALLCODE.
Cute Animal Picture

Pic credits: Shizzi

I have pushed

@SamWilsn @lightclient This is ready for review.

repo: ethereum/execution-specs


text:
Mechanism for keeping proofs up to date for cold state.

At present, the conceptual gossip mechanism for distributing new and updated state into the network looks roughly like this.
At each block, a proof P is generated which contains all of the state data touched during block execution.  This proof P is comprised of account leafs, contract storage leafs, and the intermediate trie data necessary to anchor that data to the Header.state_root.  A bridge node will then take this proof, and enumerate the leaf data, finding nodes that are interested in each piece of leaf data and distribute it to those nodes using OFFER/ACCEPT/STORE messages.  These nodes will then repeat this process, offering the proof data they received to a subset of their interested neighbors.  This process repeats until it hits the natural termination condition of none of the offered data being accepted.
The mechanism above ensures that the network contains updated proof information for each piece of leaf data at each block state root where that data was either created or modified.
Now, we consider an account which was modified at block N - 1 when we are currently on block N.  In this case, there can be nodes that will have received a proof against the state root at height N - 1 who need to update their proof to be valid against the state root at height N.
Suppose we have the following trie of data.

We use a binary trie for simplified visuals

    0:                           X
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0                           1
                 /   \                       /   \
               /       \                   /       \
             /           \               /           \
    2:      0             1             0             1
           / \           / \           / \           / \
          /   \         /   \         /   \         /   \
    3:   0     1       0     1       0     1       0     1
        / \   / \     / \   / \     / \   / \     / \   / \
    4: 0   1 0   1   0   1 0   1   0   1 0   1   0   1 0   1

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

Now we suppose that during block N-1 the state data C was touched.  The
corresponding proof that nodes which store C would receive would be as
follows.

The trie nodes with a * next to them are the nodes that actually changed.

    0:                         (N-1)*
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1
                 /   \
               /       \
             /           \
    2:      0*            1
           / \
          /   \
    3:   0     1*
              / \
    4:       0*  1

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

Now suppose that during block N the state data G was changed.  The
corresponding proof that nodes which store G would receive would be as
follows.

The trie nodes with a * next to them are the nodes that actually changed.

    0:                           N
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1
                 /   \
               /       \
             /           \
    2:      0             1*
                         / \
                        /   \
    3:                 0     1*
                            / \
    4:                     0*  1

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

The nodes storing C would need to receive the following proof in order to
update their proof from height N-1 to be valid against the state root at
heigh N.
    0:                           N
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1
                 /   \
               /       \
             /           \
    2:      0             1*

    3:

    4:

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

This visualization is intended to highlight that the data needed to update the
proof for C at height N-1 to be valid against height N is a subset of the
proof at height N.  This means that the data necessary to update proofs for
cold state is already present in the main proof P that is generated at each
block.  We just need to design a mechanism that allows nodes in posession of
cold state to receive the appropriate subset of the proof data they need in
order to update their proofs.

So if I understand correctly, the two extreme solutions (neither of which seem great) are that either bridge nodes will have to track what data they've sent to which peers and transmit the relative proofs to those peers with every new block witness - or any node interested in storing state data will need to access each new block witness and then parse it themselves to update their proofs accordingly?

the two extreme solutions (neither of which seem great) are that:

A: bridge nodes will have to track what data they've sent to which peers and transmit the relative proofs to those peers with every new block witness
B: any node interested in storing state data will need to access each new block witness and then parse it themselves to update their proofs accordingly?

I don't believe that A is a viable approach since a node may have the proof that needs updating but may not actually be "tracked" by any bridge node.
The solution we're looking for is shaped a little bit like B but with the desired property that a node would not need to process the full witness, but would be able to get access to only the subset they care about.  We should expect full proof sizes to be sufficiently large (10-100MB) that transmission of the full proof to every node in the network would exceed our allowable bandwidth limits (which aren't firmly defined).
Exploring this from the perspective of two nodes, alice and bob.  Bob has just received a proof for a block at height N for some amount of state that he stores.  Bob knows alice's node_d and radius.  This means that bob can approximate the part of the trie that alice is storing, aka the range [alice.node_id - alice.radius, alice.node_id + alice.radius].  In this model, bob can filter his proof down to the minimal proof to represent this segment of the trie and offer it to alice.  We currently have no concept of offering proofs against only intermediate trie data so we'd need to come of with a scheme for doing this.  Second, I'm not 100% sure that this covers all the various edge cases.  If bob's node_id is sufficiently far away from alice's, then it's likely that bob's proof may be omitting some intermediate trie data that alice needs.
Lets look at this case, where bob's proof data isn't enough.  Suppose that
Alice has the proof data for H at height N-1.  At height N the state data
at D and J is modified.  This leaves us with the following two proofs
                            Alice's proof

    0:                          N-1
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0                           1
                 /   \
               /       \
             /           \
    2:      0             1
                           \
                            \
    3:                       1
                            / \
    4:                     0   1

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

                      Full Proof at height N

    0:                           N*
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1*
                 /                           /   \
               /                           /       \
             /                           /           \
    2:      0*                          0*            1
           / \                         / \
          /   \                       /   \
    3:   0     1*                    0*    1
              / \                   / \
    4:       0   1*                0   1*

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

The part of the proof at height N that alice needs is this.
                          What Alice needs

    0:                           N*
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1*
                 /
               /
             /
    2:      0*

    3:

    4:

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

We'll suppose that two other nodes Bob and Carol are storing D and J
respectively.  The proofs we expect them to receive at height N look like this.
                            Bob's Proof

    0:                           N*
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1*
                 /
               /
             /
    2:      0*
           / \
          /   \
    3:   0     1*
              / \
    4:       0   1*

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

                           Carol's Proof

    0:                           N*
                                / \
                              /     \
                            /         \
                          /             \
                        /                 \
                      /                     \
                    /                         \
    1:             0*                          1*
                                             /   \
                                           /       \
                                         /           \
    2:                                  0*            1
                                       / \
                                      /   \
    3:                               0*    1
                                    / \
    4:                             0   1*

       A   B C   D   E   F G   H   I   J K   L   M   N O   P

Looking at the above, we can see that Bob has everything that Alice needs, but
Carol does not.  My intuition on this is that the further away two nodes are
from each other, the less likely it will be that they have the proof data
necessary to update the other's proof.  Conversely, two nodes that are very
close to each other should have a high likilihood of having the necessary data.
Again, based off of intuition, this suggests to me that a node who needs to
update their proof data from height N-1 to height N could adopt the
strategy of accepting any valid proof data for the region of the trie they
manage and incrementally populate their trie at height N until they are able
to connect it to the parts of their trie from heigh N-1.
Inverting this and looking at it from alice's perspective.  She has a complete proof against N-1.  Bob offers her some intermediate proof data against the state root at height N.  The proof data that bob gets is actually not enough to connect to her p

authors: njgheorghita pipermerriam

repo: ethereum/stateless-ethereum-specs


text:
fix: use beacon types for beacon rpc instead of history types

What was wrong?
I was looking into getting a foundation built out so we can have beacon tests on portal hive and I ran into a issue oh no.
Beacon RFC was out of date with the spec, then I found for FC, RFC, TRFC, PLCK we were returning history Key/Value types :scream:
How was it fixed?
By making variants of ContentInfo, PaginateLocalContentInfo, TraceContentInfo for beacon and using them

utp-test failing has nothing to do with us I am looking to resolve the main issue, but the will take a few days to resolve. I am going to merge this as it has been approved and it is clear the CI issue is external

repo: ethereum/trin