text:
Responsible Vulnerability Disclosure_  Critical Unrestricted File Upload vulnerability Found

Hello Team,
Being a responsible security researcher, I want to catch your attention to the vulnerability I found on your website.
Proof Of Concept :
I have found critical Unrestricted File Upload vulnerability on below  url:
Target URL: https://filebin.net/
As you may already know, it is possible to make a website vulnerable to XSS if you can upload/include a SWF file into that website.
I tried to upload a malicious file  "xssproject(1).swf"  on the target url and the file uploaded successfully.
This shows that the Target URL is vulnerable to Unrestricted File Upload vulnerability.
Uploaded Malicious File URL: https://filebin.net/g9z3k6ajxm2qxikm
Impact:
The consequences of unrestricted file upload can vary, including complete system takeover, an overloaded file system or database, forwarding attacks to back-end systems, client-side attacks, or simple defacement. It depends on what the application does with the uploaded file and especially where it is stored.
Mitigation:

Never allow users to upload executables (php, exe, ...etc)
Check the File Type and File Extention.
Analyse the uploaded file itself, recreate it and rename it.

Source of Malicious File:
https://github.com/fuzzdb-project/fuzzdb/blob/master/attack/file-upload/malicious-images/xssproject.swf
Reference:
https://www.owasp.org/index.php/Unrestricted_File_Upload
Notes:

Please find  supporting screenshots attached herewith.
Please Takedown the uploaded malicious file and Delete the content ASAP

Step1- Filebin's File upload page

Step 2- Uploading Malicious File

Step 3- Successfully Uploaded Malicious File

You call this "responsible disclosure", but posting this publicly on github is not THAT.
This is a public disclosure.
Also @Ratna037, you do understand that this is a file uploading service?
But as you say, how bad this is depends on what the application does with the uploaded file.

@atluxity ,
I tried to find respective's team contact details. But couldn't get that. So I posted here.
Let the respective team check this. If it's not that much critical or false positive, then let them close it.

@Ratna037 Hi, thank you for the report. Can you elaborate a bit on how bad this is? Specifically, how are you able to take over the environment hosting filebin using an executable file that you've uploaded?

@espebra   Hi,  For this vulnerability I have used malicious xssproject.swf file for an example only which is available on Github itself. Attacker can use any other malicious file for this and cause the harm to the web application.
It is possible to make a website vulnerable to XSS if you can upload/include a SWF file into that website.
The xssproject.swf file has Adobe Flash function ExternalInterface.call(...) in it's  actionscript code, which implements a JavaScript bridge to the hosting page.
Here is the actionscript code of xssproject.swf

According to the Adobe website, ExternalInterface.call() can accept a JavaScript function name as the first argument and a string which would be sent to that JavaScript function. Adobe says “When the call is to a JavaScript function, the ActionScript types are automatically converted into JavaScript types; when the call is to some other ActiveX container, the parameters are encoded in the request message.” Therefore, in our case, the string would be converted into JavaScript type.
So this function ExternalInterface.call() can inject malicious JavaScript to the hosting page, which makes the website vulnerable to XSS and attacker  can do any admin task also.
References:
[1] The other reason to beware ExternalInterface.call() (URL: http://lcamtuf.blogspot.co.uk/2011/03/other-reason-to-beware-of.html)
[2] Flash ExternalInterface.call() JavaScript Injection – can make the websites vulnerable to XSS (URL: http://soroush.secproject.com/blog/2011/03/flash-externalinterface-call-javascript-injection-–-can-make-the-websites-vulnerable-to-xss/)
[3] Source of Malicious  xssproject.swf File:
https://github.com/fuzzdb-project/fuzzdb/blob/master/attack/file-upload/malicious-images
What is XSS ?
Cross-site Scripting, also known as XSS, is a way of bypassing the SOP concept in a vulnerable web application.
The Cross-Site Scripting vulnerability is one of the few vulnerabilities that has made it in every OWASP Top 10 list of most critical web application security risks released.
XSS is a command injection of the client side, like the other user pointed out, it can result in any action that can be performed by the user. Mostly XSS is used for session hijacking where the attacker using javascript makes the victim transmit session cookies to an attacker-controlled server and from there the attacker can perform "session riding".
But XSS can also result in complete application takeover. Consider a scenario in which you inject javascript and it gets stored. The admin then loads that into a web browser (usually logs or CMS). If an XSS is present there you now have the admin session tokens. That is why XSS can be very dangerous.
An XSS attack is not a danger to the server. It's a danger to the reason you have a server. Not in a technical sense but very much a human one, as any kind of XSS attack originating from your site usually ends with your reputation down the toilet. A few test cases:

Someone redirects from your site to a fake login page. Now you have a potential mass security breach of user accounts on your site.

Someone puts a cryptominer on your site. This will make your visitors' machines work overtime and when spotted, makes you look either grossly greedy and/or grossly incompetent as a sysadmin. Neither of which is a good look.

Someone redirects traffic from your site to a competitor. I shouldn't have to explain why this is bad.

Someone puts some javascript in there that makes your site unusable or even crash browsers. Again, should be obvious why this is bad.

Someone puts DDOS code in your site to try take down your site or a third party. If aimed at you, should be obvious why this is bad. If aimed at someone else and your site is deemed culpable, your hosting provider can cut you off if you do not fix your site for breach of contract.

Someone replaces your ads with their own. If you rely on ad revenue, they're stealing that revenue.

Someone uses it to snoop on your users. Hel-lo, breach of GDPR.

Most of the possible consequences of XSS vulnerabilities affect the user, not your server. So if you don't care about your user getting their accounts on your website compromised or your users seeing content on your website which doesn't come from your server, sure, ignore those vulnerabilities.
But if your users have admin rights, then an XSS vulnerability can easily lead to unintentional admin actions. A classic case of that is a log viewer in your admin area which isn't XSS-proof. Some javascript snippets in your access logs might get executed by your admins and perform administrative actions under their account. This is why you sometimes see javascript snippets in the HTTP headers of the bots which try to hack your website.
Additional damage could have been done by sending a request to the hosting company to cancel the subscription of the server and of the domain and transfer money from his banking account
References:
[1] Cross Site Scripting (URL: https://www.netsparker.com/blog/web-security/cross-site-scripting-xss/ )
[2] How Dangerous is XSS ? (URL: https://security.stackexchange.com/questions/206520/how-dangerous-is-xss )

I'm all for feedback and reporting issues (even if it may be a false positive). Better safe than sorry and I thank you for coming forward with this. However, this sounds to me like you have no way of exploiting it, but you read about XSS in a book/blog-post and automatically think that allowing user content on a page is an exploit. All the generic references to what an XSS can be just serves as a minor insult IMHO.
If you could inject code into a page that would embed the file, then it would be an XSS. The SWG or any other binary file is never embedded in any page on filebin.net allowing it to be executed, i.e. it is a "dead", binary file. It is just a link to it, and it is served with Content-Type: application/octet-stream, so I see no problem. Nor do I see any way of executing the file on the host file system itself.

@ethlo Thanks for your valuable feedback. As I am in beginner learning phase, this can be addition to my knowledge.

I'm closing this now. If it was possible to upload and include, it would have been a vulnerability. I don't see how the current implementation is vulnerable. Please reopen if you find a way to include/inject code in the filebin app itself.

authors: Ratna037 atluxity espebra ethlo

repo: espebra/filebin


text:
ESP32 AT2.0 WiFi自动重连机制

你好：
在使用AT+CWJAP指令连接AP时，bssid为空 reconn=1的时候。模块实际进行重连时使用的是SSID还是BSSID。
实际问题是：如果把设备从第一个AP区域移动到第二个AP区域（两个AP SSID一致 BSSID不同），会不会自动连接第二个AP（第一个AP完全没信号）。或当第一个AP还有微弱信号会连接那个AP。

@RomeliFeng As far as i know, 802.11 probe request packet will be sent at the beginning of WiFi connection,
and the packet will include SSID item and BSSID item.
SSID or BSSID item will adopt NULL or 0xFF as the auto fill if user does not assign the valid value for them.

Normally,  SSID and BSSID should have at least one valid configuration for WiFi connection.

esp-at will do auto reconnection if WiFi disconnected, scan all the channel, and should be connected to second AP when first AP disappeared.

Which AP to be connecting depends on the AP's reply(probe response)

谢谢。和我自己测试的结果一致。客户现场的问题看来和客户自己的设备有关系。

source: gharchive/issue

authors: RomeliFeng ustccw

repo: espressif/esp-at

license_type: permissive

license_source: github-api


text:
ESP-AT Master compile error

I want to compile ESP8266_1MB  ESP-AT, configution use PLATFORM_ESP8266   ESP8266_1MB
compile log as follow
/mnt/hgfs/esp8266-share/esp-at/components/at/lib/libesp8266_at_core.a(at_pingCmd.c.obj):(.literal.esp_at_ping_process+0xc): undefined reference to ping_init' /mnt/hgfs/esp8266-share/esp-at/components/at/lib/libesp8266_at_core.a(at_pingCmd.c.obj): In function esp_at_ping_process':
(.text.esp_at_ping_process+0x78): undefined reference to `ping_init'
collect2: error: ld returned 1 exit status
make: *** [/mnt/hgfs/esp8266-share/esp-at/esp-idf/make/project.mk:510: /mnt/hgfs/esp8266-share/esp-at/build/esp-at.elf] Error 1
root@ubuntu:/mnt/hgfs/esp8266-share/esp-at#
root@ubuntu:/mnt/hgfs/esp8266-share/esp-at# git branch

master

@jack685 could you please provide detailed compile log?

hello ustcct,
My platform choose PLATFORM_ESP8266 and module_name choose ESP8266_1MB. esp-at checkout to Master version. export then make menuconfig and log as follow.
root@ubuntu:/mnt/esp8285/esp-at# make menuconfig
Python requirements from /mnt/hgfs/esp8266-share/esp-at/esp-idf/requirements.txt are satisfied.
MENUCONFIG
Traceback (most recent call last):
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1128, in _open
return open(filename)
FileNotFoundError: [Errno 2] No such file or directory: '$ESP_AT_PORJECT_PATH/main/interface/uart/Kconfig'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1158, in _enter_file
self._file = self._open(filename)
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1141, in _open
raise IOError(
OSError: Could not open '$ESP_AT_PORJECT_PATH/main/interface/uart/Kconfig' (ENOENT: No such file or directory). Perhaps the $srctree environment variable (which was unset) is set incorrectly. Note that the current value of $srctree is saved when the Kconfig instance is created (for consistency and to cleanly separate instances).
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/confgen.py", line 553, in
main()
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/confgen.py", line 225, in main
config = kconfiglib.Kconfig(args.kconfig)
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 653, in init
self._parse_block(None,           # end_token
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1632, in _parse_block
self._parse_block(_T_ENDMENU,
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1584, in _parse_block
prev_node = self._parse_block(None,            # end_token
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1632, in _parse_block
self._parse_block(_T_ENDMENU,
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1681, in _parse_block
self._parse_block(_T_ENDCHOICE,
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1583, in _parse_block
self._enter_file(sourced_file)
File "/mnt/hgfs/esp8266-share/esp-at/esp-idf/tools/kconfig_new/kconfiglib.py", line 1167, in _enter_file
.format(self._filename, self._linenr, e.message))
AttributeError: 'OSError' object has no attribute 'message'
make: *** No rule to make target 'menuconfig', needed by '/mnt/esp8285/esp-at/build/include/config/auto.conf'.  Stop.
root@ubuntu:/mnt/esp8285/esp-at#

but I compile it sucessfull when i checkout  to release/v2.1.0.0_esp8266 .

@jack685 please build the esp-at project according to https://github.com/espressif/esp-at/blob/master/docs/en/Compile_and_Develop/How_to_clone_project_and_compile_it.rst
Moreover, it it recommended that keep the default sdk version.

Closing as this question appears to be answered. Please reopen if the issue persists and you can supply more details. Thanks.

authors: FayeY jack685 ustccw


text:
Bluedroid + release/v5.1 + -O2 = build fail (IDFGH-10045)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.2-dev-350-g56123c52aa
Operating System used.
Linux
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
What is the expected behavior?
Successful build.
What is the actual behavior?
Build fail
Steps to reproduce.
Enable -O2 (CONFIG_COMPILER_OPTIMIZATION_PERF) and build
Build or installation Logs.
/esp-idf/components/bt/host/bluedroid/stack/include/stack/bt_types.h:35:34: error: 'strncpy' specified bound 33 equals destination size [-Werror=stringop-truncation]
   35 | #define BCM_STRNCPY_S(x1,x2,x3)  strncpy((x1),(x2),(x3))
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~
/esp-idf/components/bt/host/bluedroid/btc/profile/std/gap/btc_gap_ble.c:89:9: note: in expansion of macro 'BCM_STRNCPY_S'
   89 |         BCM_STRNCPY_S(param.get_dev_name_cmpl.name, name, BTC_MAX_LOC_BD_NAME_LEN + 1);

### More Information.

_No response_

By release/v5.1 in the title I meant it's affecting it and master. I didn't test other refs.

Is the above issue encountered while using the "esp_ble_gap_get_device_name" API? I am using the latest esp-idf v5.1 and I have not been able to reproduce the problem. Could you provide more log information and the specific commit ID of the esp-idf?

I am using the latest esp-idf v5.1（commit id 420ebd208ae9eb71cb71ebd22742d1175f11addc）and I have not been able to reproduce the problem.
Could you provide more log information and the specific commit ID of the esp-idf?

The commit ID is mentioned in the issue report: v5.2-dev-350-g56123c52aa
BTW, I can confirm this build issue when build for esp32c3.
Make sure you have CONFIG_COMPILER_OPTIMIZATION_PERF=y && CONFIG_BT_BLUEDROID_ENABLED=y.

Happens to me too on latest 5.1 release

Happens to me too on latest 5.1 release
Which example did you encounter this problem in, and did you make any changes to the code of the example?

Happens to me too on latest 5.1 release
Which example did you encounter this problem in, and did you make any changes to the code of the example?
Code which works fine without any optimisation, but with -O2 it failed. Generic bluedroid code I don't have anymore I deleted

The commit ID is mentioned in the issue report: v5.2-dev-350-g56123c52aa BTW, I can confirm this build issue when build for esp32c3. Make sure you have CONFIG_COMPILER_OPTIMIZATION_PERF=y && CONFIG_BT_BLUEDROID_ENABLED=y.

Thank you for reporting the bug. I have successfully reproduced it and the upcoming commit will fix the issue.

The commit ID is mentioned in the issue report: v5.2-dev-350-g56123c52aa BTW, I can confirm this build issue when build for esp32c3. Make sure you have CONFIG_COMPILER_OPTIMIZATION_PERF=y && CONFIG_BT_BLUEDROID_ENABLED=y.

Thank you for reporting the bug. I have successfully reproduced it and the upcoming commit will fix the issue.

I was having the bug on the esp32, maybe it's worth checking it now works for it too

The commit ID is mentioned in the issue report: v5.2-dev-350-g56123c52aa BTW, I can confirm this build issue when build for esp32c3. Make sure you have CONFIG_COMPILER_OPTIMIZATION_PERF=y && CONFIG_BT_BLUEDROID_ENABLED=y.

Thank you for reporting the bug. I have successfully reproduced it and the upcoming commit will fix the issue.

I was having the bug on the esp32, maybe it's worth checking it now works for it too

you are right,All chips utilizing Bluedroid will experience an impact.

for a temporary solution:
esp-idf v5.1
in file: \esp-idf\v5.1\components\bt\CMakeLists.txt
change line 747:
set(jump_table_opts "-fjump-tables -Wno-error=stringop-truncation")

What is the approximate stage to provide a fix for this issue?
When 5.0 was updated to 5.1, almost all projects were blocked.

for a temporary solution: esp-idf v5.1 in file: \esp-idf\v5.1\components\bt\CMakeLists.txt change line 747: set(jump_table_opts "-fjump-tables -Wno-error=stringop-truncation")

The commit ID is mentioned in the issue report: v5.2-dev-350-g56123c52aa BTW, I can confirm this build issue when build for esp32c3. Make sure you have CONFIG_COMPILER_OPTIMIZATION_PERF=y && CONFIG_BT_BLUEDROID_ENABLED=y.

Thank you for reporting the bug. I have successfully reproduced it and the upcoming commit will fix the issue.

@zhp0406
It is a big surprise that it takes so much time to fix a build failure .(This issue has been reported for almost 3 Months)

@AxelLin
I'm sorry, the fix for this bug has already been applied to the master branch . but due to some other reasons, the changes have not yet taken effect in version 5.1.
You can cherry-pick it over for use (commit 0634bb904ea8fdf014d173b8f142638491453775).
Thank you for bringing up the issue.

Thanks for reporting and sorry for the very slow turnaround, the fix on master branch is available at https://github.com/espressif/esp-idf/commit/126b98d88987e3a1b2fb4b9887218e914a0ac704, and already back ported to other release branches. Feel free to reopen, thanks.

available at 62f5905be25e8f096032fb0f62dd768e7f0008de on release/v5.1 branch

authors: Alvin1Zhang AxelLin KaeLL SlitheringMule caidamao habablaxe zhp0406

repo: espressif/esp-idf


text:
Bootloader update/upgrade on new ESP-IDF major version (IDFGH-13587)

Is your feature request related to a problem?
This is not really a feature request (although it might be, depending on your answer). I could not find any documentation on this.
When IDF is upgraded from v5 to v6 at some time in the future, will the (v5) boot loader still be able to load v6 images?
This is something I will need, because my ESP32s will be at unreachable places, where I cannot simply put an USB plug in.
Or alternatively, will there be an API that allows the boot loader to be upgraded OTA? I know I can come up with something like that myself, just overwrite the flash sectors. But it feels a bit odd to use the OTA system for the image and then have to kludge something myself for the boot loader.
Describe the solution you'd like.
No response
Describe alternatives you've considered.
No response
Additional context.
No response

#14195

Hi @eriksl!

will the (v5) boot loader still be able to load v6 images?

Yes, the v5.x bootloader will be able to load a v6.x application.

Or alternatively, will there be an API that allows the boot loader to be upgraded OTA?

We are working on this feature. It will be possible to do it, but with some limitations. Bootloader OTA update can not be done safely, if power is dropped in the middle of the writing to the bootloader partition then the device will become unbootable.

I know I can come up with something like that myself, just overwrite the flash sectors. But it feels a bit odd to use the OTA system for the image and then have to kludge something myself for the boot loader.

Yes, you are right the current OTA component requires not too many changes to support it for bootloader.

That is great news, thanks!

I close this issue. When the bootloader OTA update feature is released then you will see it in #14195. Thanks.

authors: KaeLL KonstantinKondrashov eriksl


text:
PROVIDE ( GPIO = 0x60004000 ); in tools/sdk/ld/esp32.peripherals.ld (IDFGH-1607)

I was told to move https://github.com/espressif/arduino-esp32/issues/2571 to this queue.
As per that bug,
I had to reapply this patch:
tools/sdk/ld/esp32.peripherals.ld
-PROVIDE ( GPIO = 0x3ff44000 );
+PROVIDE ( GPIO = 0x60004000 );
I got it from there https://github.com/NeoCat/ESP32-P3RGB64x32MatrixPanel/blob/master/README.md 'You should patch the tools/sdk/ld/esp32.peripherals.ld to avoid random pixel noise on the display as following
Please refer section 3.3. in https://www.espressif.com/sites/default/files/documentation/eco_and_workarounds_for_bugs_in_esp32_en.pdf for more details."
also present here http://gitlab.studiopresent.rs/ESP32-dotlab/framework/blob/master/tools/sdk/ld/esp32.peripherals.ld
and also found it here https://github.com/mrfaptastic/ESP32-RGB64x32MatrixPanel-I2S-DMA
https://qiita.com/h_nari/items/d52734440a7256072e8e when translated by google, says " I encountered GPIO 's unintelligible behavior. After examining the cause, it turned out to be a CPU bug (Eratta)"
I'm not sure if any of those people opened a bug or tried to get it changed, but in case none of them did, I'm filing this bug.
I am not knowledgeable about why this fix is needed, although the first and last link does give details that should help

Details:
Some ESP32 peripherals are mapped to two internal memory buses (AHB & DPORT).
When written via DPORT, consecutive writes to the same address may be lost.
Workarounds:
This issue is automatically worked around in the drivers of ESP-IDF V1.0 and newer.
Workaround Details:
When writing the same register address (i.e., FIFO-like addresses) in sequential instructions,
use the equivalent AHB address not the DPORT address.
(For other kinds of register writes, using DPORT registers will give better write
performance.)
Fixes:
This issue is fixed in silicon revision 1.

So this should only affect old rev0 chips and only consecutive writes to the same address. Because of this and the fact there is a performance trade-off, it is better to selectively work around this issue instead of applying a blanket fix.  This is what esp-idf drivers do.
Can you explain what problem this resolved for you?

@negativekelvin people using ESP32 to run RGBPanels via I2S DMA seem to have timing problems without it, I gave the links in my original message. For instance
Patching GPIOhttps://github.com/NeoCat/ESP32-P3RGB64x32MatrixPanel/blob/master/README.md to avoid eratta of ESP32
You should patch the tools/sdk/ld/esp32.peripherals.ld to avoid random pixel noise on the display as following:
/* PROVIDE ( GPIO = 0x3ff44000 ); */
PROVIDE ( GPIO = 0x60004000 );

Ok but have you confirmed this yourself and what do GPIO registers have to do with I2S DMA?
For instance https://github.com/NeoCat/ESP32-P3RGB64x32MatrixPanel/blob/master/README.md mostly uses REG_WRITE so changing the GPIO struct address would have no effect.

@negativekelvin  no, I did not. I just used multiple projects that told me I had to change this, so I did.
It might be possible they are all wrong and have been passing around an obsolete fix. If you think that's the case, feel free to close this.

Hi @marcmerlin ,
@negativekelvin is right that this is only a problem on ESP32 rev0, and only a problem with very specific workloads - writing to the exact same DPORT register twice in consecutive instructions.
I don't see anywhere in the code where this would happen, but it's not inconceivable that it could have happened at some point. Also, as @negativekelvin  points out, the library also uses register macros and for these the suggested "fix" would have no effect.
This advice looks like it was added in the initial commit 2 years ago, so my guess is that it was copied over from some other codebase where it may have been necessary on the rev0 chip:
https://github.com/NeoCat/ESP32-P3RGB64x32MatrixPanel/commit/6143785bc7c0e88439e0ef58a590d3a8f6d25677#diff-04c6e90faac2675aa89e2176d2eec7d8R14
If you do find a situation where changing this address fixes a problem, please let us know and we will reopen.

thanks for the look/review @projectgus , this wouldn't be the first time that people have been copying over an unnecessary workaround.

authors: marcmerlin negativekelvin projectgus


text:
IP_EVENT_ETH_LOST_IP missing (in esp_netif_types.h)

I am trying to implement better error handling for all network related events and it looks like there are events for when the IP was successfully assigned and also events when we loose ip (dhcp lease timeout and dhcp server doesnt respond anymore, or other failure condition).
I found these matching pairs:
IP_EVENT_STA_GOT_IP - IP_EVENT_STA_LOST_IP
IP_EVENT_PPP_GOT_IP - IP_EVENT_PPP_LOST_IP
IP_EVENT_ETH_GOT_IP - ??? when we loose ip?
I am missing a IP_EVENT_ETH_LOST_IP
Shouldnt the same error handling be also possible for the ethernet interface?

@0xFEEDC0DE64 Thanks for your suggestion. Could you please try this patch.
diff --git a/components/esp_netif/include/esp_netif_defaults.h b/components/esp_netif/include/esp_netif_defaults.h
index 7ef6526..903a678 100644
--- a/components/esp_netif/include/esp_netif_defaults.h
+++ b/components/esp_netif/include/esp_netif_defaults.h
@@ -55,7 +55,7 @@ extern "C" {
         ESP_COMPILER_DESIGNATED_INIT_AGGREGATE_TYPE_EMPTY(mac) \
         ESP_COMPILER_DESIGNATED_INIT_AGGREGATE_TYPE_EMPTY(ip_info) \
         .get_ip_event = IP_EVENT_ETH_GOT_IP, \
-        .lost_ip_event = 0, \
+        .lost_ip_event = IP_EVENT_ETH_LOST_IP, \
         .if_key = "ETH_DEF", \
         .if_desc = "eth", \
         .route_prio = 50 \
diff --git a/components/esp_netif/include/esp_netif_types.h b/components/esp_netif/include/esp_netif_types.h
index 044ca0d..7c68c71 100644
--- a/components/esp_netif/include/esp_netif_types.h
+++ b/components/esp_netif/include/esp_netif_types.h
@@ -88,6 +88,7 @@ typedef enum {
     IP_EVENT_AP_STAIPASSIGNED,         /*!< soft-AP assign an IP to a connected station */
     IP_EVENT_GOT_IP6,                  /*!< station or ap or ethernet interface v6IP addr is preferred */
     IP_EVENT_ETH_GOT_IP,               /*!< ethernet got IP from connected AP */
+    IP_EVENT_ETH_LOST_IP,              /*!< ethernet lost IP and the IP is reset to 0 */
     IP_EVENT_PPP_GOT_IP,               /*!< PPP interface got IP */
     IP_EVENT_PPP_LOST_IP,              /*!< PPP interface lost IP */
 } ip_event_t;

Hi @ESP-YJM
I update to v4.4-dev-1390-g1ca2afd98223, but I never receive the IP_EVENT_ETH_LOST_IP event.
Could you share how to test (verify) it?

IP_EVENT_ETH_LOST_IP will happen after you unplug the network cable ESP_NETIF_IP_LOST_TIMER_INTERVAL seconds later (default 120 seconds). This is the same with IP_EVENT_STA_LOST_IP.

Thanks, I plug the ethernet cable back too quickly in my previous test.

authors: 0xFEEDC0DE64 AxelLin ESP-YJM


text:
802.11 LR can be used in Japan? (IDFGH-5627)

In below page, 802.11 LR is subscribed. In Japan 802.11 LR can be used?
https://github.com/espressif/esp-idf/blob/58022f8/docs/en/api-guides/wifi.rst
ESP32 has Japanese certification "Gijyutukijyun Tekigou Syoumeisyo" about 802.11 a/b/g/n,
so 802.11 a/b/g/n can be used in Japan.
However the 802.11 LR can be used in Japan ?
Also ESP-Now can be used in Japan ?
I thine the target point are whether
･ PHY layer is changed or not
･ Carrier sense is executed or not
I would be grateful if you could give me some advice.

hi @uecken
ESP-NOW not change anything about PHY layer, which can be used.
While LR mode mainly change about the PHY Data Rate, which support 250Kbps & 500Kbps, others such as channel, tx power do not change, not quite sure about the data rate can be used in Japan.

@uecken  Can you elaborate on the the certification requirements?   As @MaxwellAlan noted, ESPNOW is completely covered in the upper layer and no PHY layer modifications are needed. LR mode is a proprietary mode and does not increase the Tx power of the device. The modulation is different for LR mode and is not compatible with any other standard WiFi device.

Thank you for your reply. LR mode changes only the PHY Data Rate?
In nay case, this LR mode is not standard WiFi.
So, if we use LR mode in Japan, need o be tested and meet requirements.
I have not known certification requirements for 2.4GHz. So, I investigated it.
Reference to https://www.tele.soumu.go.jp/j/sys/equ/tech/type/,
The 2.4GHz system needs to obey "2.4GHz帯高度化小電力データ通信システム(2,400～2,483.5MHz)"
https://www.tele.soumu.go.jp/resource/j/equ/tech/betu/43.pdf
Of course power, obw, spurious is needed, however Carrier sense is not needed in case the OBW is less than 26MHz.
If the LR mode uses less than 26MHz and other RF characteristic is same with 802.11a/b/g/n, it's not difficult to take Japanese certification.
By the way, LR mode uses like LoRa protocol?   Receiver sensitivity is +4dB than WiFi?

hi @uecken ,

LR mode changes only the PHY Data Rate?

Yes.

LR mode uses like LoRa protocol? Receiver sensitivity is +4dB than WiFi?

Yes.

What is the OBW?

OBW is Occupied Bandwidth (OBW), which is the same as 11b.

If PHY is same with 802.11, use Carrier Sense?

Yes.

@MaxwellAlan @sagb2015
Sorry for my late reply.
I understand your reply.
LR Phy is same with other WiFi. So i t its not needed additional certification.
BTW, Can I use LR mode from PlatformIO?
I am developping below system.
https://protopedia.net/prototype/2356
I want to cover 100 meter without repeater

hi @uecken ,
Sorry for that I'm not familiar with PlatformIO IDE.
Btw, I think the LR mode only depends on which IDF verion you are working on, since IDF release/v3.3 and later, all these IDF versions/branches are support LR mode.
Thanks.

@MaxwellAlan
Thanks.
It seems that platform-espressif32 uses ESP-IDF latest 4.3.1.
https://github.com/platformio/platform-espressif32/releases
Thank you! I will try 801.11LR.
OK, I want to ask how to use 11LR.
Which?
https://github.com/platformio/platform-espressif32
https://github.com/espressif/arduino-esp32/blob/master/libraries/WiFi/src/WiFiAP.cpp
Additionally, I also want to try 11b on PlatformIO IDE.
Regards.

hi @uecken ,
May be you can try esp-idf/examples/wifi/getting_started , both in SoftAP and Station example, after wifi init, you can call esp_err_t esp_wifi_set_protocol(wifi_interface_t ifx, uint8_t protocol_bitmap) to set operating protocol, eg: esp_wifi_set_protocol(0, WIFI_PROTOCOL_LR); and sorry for lack of some docs about LR mode.

thanks @MaxwellAlan !
Now I am installing ESP-IDF with PlatformIO.
Get started with ESP-IDF and ESP32-DevKitC: debugging, unit testing, project analysis¶
https://docs.platformio.org/en/stable/tutorials/espressif32/espidf_debugging_unit_testing_analysis.html
I will try!

I found the Arduino source code to use 802.11 LR.
First I will try this.
https://github.com/jnogues/ESP32-Long-Range-WiFi/blob/master/master.ino
BTW, 11b and 11LR is very good foy 2.4GHz IoT communication.
I have an idea to improve this WiFi more.
not execute carrier sense.
Under OBW 26MHz, carrier sense is not needed.

If not doing carrier sense, many device can send the data simultaniously.
implement power control
If power control are implemented like 3G system that uses DSSS.
simultaneous transmission will be success, I think.
Regard.

authors: MaxwellAlan sagb2015 uecken


text:
SDK Configuration editor (IDFGH-6781)

Turning off both Enable Modbus stack support for RTU mode and Enable Modbus stack support for ASCII mode in the SDK Configuration editor will result in an error compiling.

Hi @flair0626,
Thank you for this issue report. Could you be more specific and specify the ESP-IDF version (commit) you are using, the example and sdkconfig.h file you are using? The ESP-Modbus component requires at least one communication option to be selected. If not, it causes the compilation error (intentional behavior). So, if you selected at least one of the options Enable Modbus stack support for TCP comm mode and disabled the Enable Modbus Stack support for RTU mode, Enable Modbus Stack support for ASCII mode there should not be any error during compiling. However, I have checked this on master and maybe some backport fixes are not merged.
Thanks for the information.

@flair0626 Thanks for reporting, would you please help share if any more updates for https://github.com/espressif/esp-idf/issues/8408#issuecomment-1042769008? Thanks.

This was the first problem I stumbled across when I tried to optimize the sdkconfig.
A user that does not know anything about modbus and does not want to use it, will always try to disable all those options. Because, "I do not use it, I do not need it, it will save some space when I disable it".

@kriegste , @flair0626 ,
Sorry for late response. Unfortunately this issue will not going to be fixed in the esp-idf. The old freemodbus component is migrated to separate esp-modbus component.

Closing. Please feel free to raise a issue on https://github.com/espressif/esp-modbus (if any)

authors: Alvin1Zhang alisitsyn ghost kriegste mahavirj


text:
gatts_demo: re-enter advertising after client disconnects

In the existing tree, after a connected client disconnects the device
will never re-enter the advertising state. Additional clients will not
be able to discover or connect to the device. This patch forces the
device back into the advertising state upon disconnection.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Thanks Mike. Sorry for the delay, I've put this fix into our internal merge queue.

Thanks!

source: gharchive/pull-request

authors: CLAassistant mikeryan projectgus


text:
4 lights endpoints are not adding in Tuya Gateway (TZ-494)

Question
I am using this routine to add endpoints:
esp_zb_ep_list_t *esp_zb_ep_list = esp_zb_ep_list_create();
esp_zb_cluster_list_t *esp_zb_server_cluster_list = server_cluster();

for(int i=0; i<MAX_EP_SIZE; i++){
   esp_zb_ep_list_add_ep(esp_zb_ep_list,
		   esp_zb_server_cluster_list,
		   ENDPOINTS_LIST[i],
		   ESP_ZB_AF_HA_PROFILE_ID,
		   ESP_ZB_HA_ON_OFF_LIGHT_DEVICE_ID);
   ESP_LOGI(TAG, "ENDPOINTS_LIST[%d]= %d",i,ENDPOINTS_LIST[i]);
}
esp_zb_device_register(esp_zb_ep_list);

this code working perfectely in Ewelink app(Sonoff), showing 4 light switches, but not in Smart life app(Tuya).
Please help me
Additional context.
No response

If i set MAX_EP_SIZE = 3, its working. more than 3 not showing on Smart Life (Tuya) App.

It works with Ewelink app(Sonoff), but fails with Smart life app(Tuya). Should it be some issue with Tuya app?
We haven't tested such details with these ecosystems. As you may know, although Zigbee is a standard protocol, but unfortunately there are more or less some differences among Zigbee ecosystems.

It works with Ewelink app(Sonoff), but fails with Smart life app(Tuya). Should it be some issue with Tuya app?
We haven't tested such details with these ecosystems. As you may know, although Zigbee is a standard protocol, but unfortunately there are more or less some differences among Zigbee ecosystems.

Thank you for the reply Sir,
Yes, May be some issue with Tuya app. But i am also getting another issue, I am getting only 1 active endpoint instead of 4, during sending request esp_zb_zdo_find_on_off_light after joining network. Can you help me over this.

duplicate issue, @mundevx please check with the gataway vendor (https://github.com/espressif/esp-zigbee-sdk/issues/184#issuecomment-1880598546)

authors: chshu munish-nuos munxerpor

repo: espressif/esp-zigbee-sdk


text:
[run] Actual pause button?

It would be nice to be able to pause my run. I can stop it and then start it with the append option but that starts the distance tracker from 0. It would be nice to have a simple way of pausing it temporarily and then resuming it without hassle.

Yes, pause button would be nice. I guess it is more important when user is walking than when running. Battery does not last for all-day trip and pause could help.

Thanks - I guess once stopped, if the button is pressed again it could ask you if you want to resume or start a new run?

Now that I've had this watch a while I have gotten used to stopping and starting my runs. I just use the append button in order to combine them

authors: Darin755 gfwilliams pavelmachek

repo: espruino/BangleApps

license: MIT


text:
Add gpsrawtime

I am not sure if it is too much code for the boot.js.
Should I put it in a separate file?
Could someone review the code?
See also http://forum.espruino.com/conversations/372114/?offset=50#comment16367783

Hi - thanks for this, but I'm not sure all that code is really needed, and adding 'GPS-raw' could cause FIFO_FULL errors...
Looks like you're decoding the RMC nmea data to get the GPS time? But Espruino already does this and puts it in the time field of the GPS fix: https://github.com/espruino/Espruino/blob/master/libs/misc/nmea.c#L53-L61
So unless I'm missing something, all you need to do is Bangle.on('GPS', fix => print(fix.time)) and you'll be able to grab the latest time? Also RMC is on by default, so there's no need to send the PCAS03
There is actually already a widget which does this as far as I can see: https://github.com/espruino/BangleApps/blob/master/apps/gpsautotime/widget.js
But right now it's not enabled for Bangle.js 2 (although it should work). Do you think it's worth just enabling that? Although I know it doesn't have logging.

My intention was to get the time without a fix.
The app enables gps, wait for the for the first RMC message and disable gps.
If within 10 seconds no RMC message is received gps will also disabled.
I'll remove PCAS03.

My intention was to get the time without a fix.

Yes, that should be fine to do with the GPS event? The GPS event fires whenever there's data from the a GPS - it doesn't wait until there's a fix (or even a time - so the first few events time may be undefined).
Using GPS-raw every time any app starts will really slow things down - I don't see why you need this when the data is already being parsed internally by Bangle.js

Yes, you are right. I didn't see it, sorry.
I will take a look at gpsautotime.
Right now it's waiting for a fix, but that should be able to be changed.

Thanks!

Right now it's waiting for a fix

I didn't spot that! Wow, that really limits its use - but yeah, that should be an easy tweak


text:
前置シフトキーの２度おしで空白がはいるようにする。

#92 で、前置シフトの状態が画面上でわかるようになった。
それ以降、前置シフトロックのとりけしはBackspaceキーの役割にうつっている。

done @ 1c44ec4baabf8d69df08521e389796317f46c51d

repo: esrille/ibus-hiragana


text:
Delete Cargo.lock file from git repository.

Cargo.lock is regenerated at build time.

I guess you certainly can keep the Cargo.lock file if you decide so.
It just seems quite cumbersome to merge this file when I have to do it.
But I guess I did not have your own requirements in mind.
It might certainly make sense for you.

Thanks for the proposal. I will close this and keep the lock file for now.

authors: flxo jcdubois

repo: esrlabs/northstar


text:
Updated Readme

I've added further information on how to build northstar and start one of the sample container. Should help others to get easier started

looks nice...thanks

authors: marcmo taunusflieger


text:
Este Universal

browser/server/mobile/desktop/common
one cli to rule them all

On a side note - I quite like what Hapi.js does with its universe - lots of packages and general ecosystem that just feels nice. Maybe worth considering its direction and going toward Este universe that's a 1st choice for everybody wanting to either do something efficiently or start React.js journey? :P

I have zero experience with hapi. Can you highlight some of its benefits?

Would you like to know more about Hapi.js as a framework compared to Express or about their organisation structure I was referring to?

Probably both :-)

authors: grabbou steida

repo: este/este

license: mit

license_source: bigquery


text:
please respect unicode strings you are given

esprima.parse('a = "привет"') clearly preserves unicode strings in the format supplied
{ type: 'Program',
  body:
   [ { type: 'ExpressionStatement',
       expression:
        { type: 'AssignmentExpression',
          operator: '=',
          left: { type: 'Identifier', name: 'a' },
          right: { type: 'Literal', value: 'привет', raw: '"привет"' } } } ] }

esgen(esprima.parse('a = "привет"')) clearly does not
a = '\u043F\u0440\u0438\u0432\u0435\u0442';

Technically these js-strings are equivalent, but why forego readability?

Use verbatim: "raw" in your escodegen options if you want the literal representation preserved.

thank you!
Does fix the unicode representation, but wraps the expression in a redundant set of parens:
a = ('привет');
PS I have to ask though, why mangle the string literal in the first place?

It's not getting mangled. Those are the same strings. We only have the raw representation and the value. We can render the value using only Unicode escapes if we want, but we can't print all characters in a string literal safely. So I think we only print non-control ASCII characters and escape all others.

can't say I follow. What does render safely mean? Escodegen received both raw and value from the outside world, so someone somewhere both rendered and read them safely. AST is fair game, but I don't understand why it would need to mess with the value. (if you squint a little you might even say that escodegen is breaking its own type-signature by messing with values instead of AST nodes).
More generally here's where I'm coming from. I think escodegen is often used in projects that are required to consume and produce human-readable code, which, seeing how the world writes in more than one language, may on occasion include some unicode. Present behaviour is unexpected to say the least, because it presents itself to a customer as something completely baffling. At the minimum it seems like the wrong default.

:+1:

@vkz

What does render safely mean?

We can't have " in "-delimited strings. We can't have a carriage return or a paragraph separator. To avoid these characters, we can just Unicode escape everything. It's the same string. If you really want to avoid Unicode escapes, use {format: {escapeless: true}} as your escodegen options. But the only way to use the same escapes that you used in your original code is to use the {verbatim: "raw"} option.

right. Chance of us agreeing on the matter seems to be dwindling with each message, so let's not prolong the agony. If I may I'll leave you with these observations:
You already have the string representation in hand and it's clearly not broken, why fear then? Not saying you're wrong especially with security in the mix, but it is the wrong default.

But the only way to use the same escapes that you used in your original code

I'm using Russian not escapes just like other people could've used Japanese, Hindi, etc.

But you very well could have used escapes. See the following program for example.
("\0 \x00 \u0000 0 \x30 \u0030");

There are many ways I could have escaped that string but I chose that way. Preserving this requires the verbatim option. Are you getting it now?

var esprima = require("esprima"),
    esgen = require("escodegen").generate,
    uglify = require("uglify-js");

var exhibit1 = '"\0 \x00 \u0000 0 \x30 \u0030"',
    ast1 = esprima.parse(exhibit1),
    exhibit2 = "'привет'",
    ast2 = esprima.parse(exhibit2);

esgen(ast1);
uglify.AST_Node.from_mozilla_ast(ast1).print_to_string();

esgen(ast2);
uglify.AST_Node.from_mozilla_ast(ast2).print_to_string();

every time uglify-js does this horribly unsafe thing a puppy dies.
Having lost several hours of my life debugging this safe behaviour I feel some friendly trolling is warranted here. There'll be many a poor sobs down the road tripping over this, I'm sure.
NB: Esprima and Escodegen have saved me countless hours, I use them daily and I am absolutely grateful to their authors and maintainers for keeping them open source and providing support.

authors: fpirsch michaelficarra vkz

repo: estools/escodegen


text:
Dockerize

Watch for upstream issue etal/cnvkit#135 here.
See also etal/cnvkit#39.

We're pretty well dockerized now. Thanks @geetduggal

repo: etal/cnvkit-dnanexus


text:
feat: stop using insee for UL with more than 100 etablissements

Congrats Hajar for the amazing work
closes #810

The API client test would have to be updated, but I can do it.

authors: XavierJp johangirod

repo: etalab/annuaire-entreprises-site


text:
UX: button icon and placement change

Change the button that brings down the tool menu to a hamburger menu icon and put the button at the top right instead of the bottom right like it is now.

also this button is sometimes hidded under the toolbox,
maybe place on the top of toolbox or change place


text:
Change server interaction with the frontend to use id from the database

@KovaSs  Добавить описание задачи!

@VladKalachev  Я не помню в чем тут была суть дела, надо писать описание задачи сразу как заводится issue

repo: eternal-system/react-cli


text:
Add inherent support of parallel 'make'

Currently, if somebody wants to make his build parallel, he has to override the compile() method as follows:
    def compile(self):
        super().compile(options='-j')

We should somehow provide this automatically. What do you think?

Closed by #340.

repo: eth-cscs/reframe


text:
[WIP] Parallel and matrix evaluation

WIP code from the arXiv paper. Needs to be cleaned. Can also be optimzied (e.g., start evaluations in parallel while training)

Closing this since Xianzhe is working on the evaluation environment :)

repo: eth-easl/modyn


text:
added ky to Competition area

I added ky to the competition area

Thanks for the suggestion; I would add it but Sorhus and I don't seem to be on good terms for some reason so I'm avoiding using or linking to his projects for the time being.

repo: ethanent/phin


text:
Week 3 Step 6 pt1, site redesign

TLDR; Add functionality to limit number of comments displayed, also redesign for site.
LOOK AT LINK FIRST (it's very different): http://ethanimooney-step-2020.appspot.com/
ListCommentsServlet.java - Now can take a num  value from the get request and use it and FetchOptions to only retrieve that number of comments.
NewCommentServlet.java - changed redirect to new comments.html page.
comments.html - New page that now houses the comments feature.
index.html - New file, replaces indexOld.html as the new homepage. It looks like it's really weird and large but its fairly simple, I used a ul and li as my main way to display content, with spans to change the color or certain parts of the text. I also used nbsp; to simulate tab spaces
indexOld.html - the old index.html file
script.js - now passes a number with the /load-comments fetch request, for LoadCommentsServlet to determine how many comments to load. Added two eventListeners as a way to resize the form inputs in comment.html as they are typed in. Changed the way comments look when displayed.
style.css - New file, replaces styleOld.css as the new style file.
styleOld.css - the old style.css file.

Latest commit changes the script.js to utilize a different way of displaying comments, updated pom.xml file for the new gcp project, removes indexOld.html (forgot to do styleOld, will do in next commit), and adds a missing piece of styling to style.css

repo: ethanimooney/step-repo


text:
Codestyle formatting (Part 2)

Follow up of #68 (and that must be applied first). Adds npm run lint test to packages.json and has the more delicate commits regarding globals (Module and web3).
Todo: include the test in Travis so that future PRs are validated.
We could switch over to standard from semistandard either in this or a subsequent PR to keep diffs digestable.

Nice job!

authors: axic chriseth

repo: ethereum/browser-solidity


text:
add note about distributing bootnode ENRs prior to genesis

As discussed with @AgeManning and @nisdas, bootnode ENRs need to be distributed prior to learning genesis_validators_root. A note of guidance for how to do this in a standard way has been added.
Note: circleci cache got messed up so bumped version

@nisdas and @AgeManning: sanity check that this looks fine (and at least very similar to what you are doing already)

This looks good to me :+1:

Yeah this looks good to me also.
For us, we would likely update the bootnode's ENR after we know the value, as then the bootnode will also be discovered and participate in discovery with other nodes on the network.

made a quick note on setting bootnode ENRs to normal values after genesis info is know. thanks!

authors: AgeManning djrtwo nisdas


text:
Fix optimized state bugs

What was wrong?

Due to a typo, storage changes would be lost after calls to state_root(). This prevented syncing past block 50111, where the first non-zero storage read occurred. This did not affect the calculation of the state root itself, only storage reads.
Storage keys were not deleted when an account was self destructed due to the call to clear_destroyed_account() being on the wrong code path.

How was it fixed?

Fixed typo.
Moved call to clear_destroyed_account() to the correct place.

Cute Animal Picture

Thanks for adding the tests!

authors: lightclient ultratwo

repo: ethereum/execution-specs


text:
Tuple pattern

What was wrong?
Add tuple pattern matching.
This PR depends on #770
To-Do

[ ] OPTIONAL: Update Spec if applicable

[ ] Add entry to the release notes (may forgo for trivial changes)

[ ] Clean up commit history

I decided to include this in #770

repo: ethereum/fe


text:
feature: Showcase the nodes that cannot be parsed

Many clients don't accept peer information, would be useful to extend the dashboard API to add metadata so we can showcase this number somehow..

The data doesn't quite support us to do this, since we are filtering nodes which we don't care about, how do we know the nodes that cannot be parsed are indeed important?

repo: ethereum/node-crawler


text:
PyPI lists version 1.3.7 (not 1.6.0) as the latest

https://pypi.python.org/pypi/ethereum
This appears to have happened after the Nov. 13 release.  Version 1.6.0 is still available ( https://pypi.python.org/pypi/ethereum/1.6.0 ), and I remember seeing it as the latest version a few days ago.

This is a problem of the current (old) PyPI UI. It always shows the latest upload independent of version number.
The new PyPI interface shows the correct version https://pypi.org/project/ethereum/#description
Also pip correctly installs 1.6.0

authors: maurycyp ulope

repo: ethereum/pyethereum


text:
mapping type decoder

[ ] performance: after 400 items in storage, it become quite unusable https://github.com/ethereum/remix/pull/512
[x] warn user if storage is not complete
[ ] check geth preimage is working
[ ] does not work now if the client does not return the key.
[ ] issue https://github.com/ethereum/remix/issues/510
[ ] instead of requesting preimage(key) , we should be able to request for an array of key
[ ] the storage retrieve only the first 100 items (for geth/eth). we should change the API so the user can load more keys if she wants to. or maybe have an option in the ui to force to load the entire storage.

We will try to use a different solution for that instead of using preimage.

repo: ethereum/remix


text:
Error when passing big integer argument for a uint256 field

I have a function with a single uint256 argument.  When I try to call that function from the Run pane with the number 100000000000000000, I get the following error:
transact to <Contract.Function> errored: Error encoding arguments: Error: Assertion failed

That number is 0.1 ether in gwei.  It's much smaller than the max representable number for an unsigned 256-bit integer (2^256), so I'm not sure what assertion is failing.  Does anyone know what's going on?  Type casting issue, maybe?
As a workaround, when I surround the value with double quotes, it works.

that should be fixed by now ( remix.ethereum.org ) please retry and reopen if necessary


text:
Remove sleep from ethportal-peertest integration test

https://github.com/ethereum/trin/blob/30d03fd54891ac68d832cd92dc3ebf74492ef106/tests/self_peertest.rs#L14
This sleep really slows down the entire test suite for trin. We should remove it in favor of some other mechanism like a channel.

A possible solution here is to write a utility fn similar to https://github.com/ethereum/trin/blob/master/ethportal-peertest/src/utils.rs#L6-L30 that will send a portal_historyRoutingTableInfo request to the node, on something with like a 20millis loop, and returns true when the routing table is populated.

authors: jacobkaufmann njgheorghita

repo: ethereum/trin


text:
Flaky peertest failure, often on Windows

For some reason, in #411, I'm now able to reproduce the issue locally ~1/3 of the time, which is enough to start tracking it down.
There are at least two problems:

There is a panic happening, it appears to be in this line:
https://github.com/ethereum/trin/blob/8a9bb83a416aefbd0d095baa9346774d5ee97704/ethportal-peertest/src/scenarios.rs#L47-L48
During the panic, some other failure causes it to hang

TODO

[ ] Panic shouldn't cause a hang
[ ] calling portal_historyLocalContent shouldn't panic

Looks like it's as simple as: the content that is supposed to be there for the test to pass is simply not there. I haven't tracked down why. @ogenev - any ideas why the data would not be there sometimes? Is this maybe a result of the lack of retries?
The nasty panic handling which hangs is its own problem, but I think for now I'll disable the test_offer_accept so that master turns green.

@ogenev - any ideas why the data would not be there sometimes? Is this maybe a result of the lack of retries?

It is hard to say, I can only speculate without adding more logs and debugging but here are my observations:

We are transferring only one uTP DATA packet in this test and from the logs, it looks like the uTP packets flow as expected. So we can exclude missed uTP packets. I tested one DATA packet transfer before with the network simulator and as long as we don't drop packets, never got any issues. Problems start to appear when we try to transfer larger contents over multiple DATA packets and https://github.com/ethereum/trin/pull/422 aims to solve those issues.

Why this test fails mostly on windows? We had some issues with the temp data dir on win before, is it because of this, maybe when we receive the content, we can't store it on time successfully? Maybe it is some race condition where we are trying to verify the content before the storage succeeds?

It is possible that make_ipc_request(&peertest.bootnode.web3_ipc_path, &local_content_request).unwrap() fails because we are not giving enough time in the test for the data transfer to success, maybe the solution is as simple as adding     tokio::time::sleep(Duration::from_secs(1)).await; before the rpc request above.

I think adding some more log messages and fixing the panic hanging will give us a clearer picture.

Hm looking into this failure:
Aug 24 20:03:01.668 DEBUG trin_core::jsonrpc::service: New IPC client: (unnamed)
Aug 24 20:03:01.778 DEBUG trin_core::jsonrpc::service: Got new IPC request: Ok(JsonRequest { jsonrpc: "2.0", params: Array([String("0x000300720704f3aa11c53cf344ea069db95cecb81ad7453c8f276b2a1062979611f09c")]), method: "portal_historyLocalContent", id: 16 })
Aug 24 20:03:01.778 DEBUG trin_core::jsonrpc::handlers: Processing Portal JSON-RPC request: HistoryEndpoint(LocalContent)
Aug 24 20:03:01.779 DEBUG discv5::service: Received RPC response: Response: Response  to request: TALK: protocol: 757470, request: 1100c7b92b0ade0d0000000000100000000396f2 from: Node: 0x8a27..1b0f, addr: 127.0.0.1:9001
Aug 24 20:03:01.780 DEBUG trin_core::utp::stream: Closing connection, sent Packet { type: Fin, version: 1, extension: None, connection_id: 51129, timestamp: Timestamp(722132493), timestamp_difference: Delay(0), wnd_size: 1048576, seq_nr: 3, ack_nr: 38642 }
Aug 24 20:03:01.780 DEBUG trin_core::utp::stream: received Packet { type: State, version: 1, extension: None, connection_id: 51128, timestamp: Timestamp(722135370), timestamp_difference: Delay(2877), wnd_size: 1048576, seq_nr: 38643, ack_nr: 3 }
Aug 24 20:03:01.780 DEBUG trin_core::utp::stream: Handle packet: State. Conn state: FinSent
Aug 24 20:03:01.781 DEBUG ureq::stream: connecting to 127.0.0.1:50197 at 127.0.0.1:50197
Aug 24 20:03:01.781 DEBUG ureq::stream: created stream: TcpStream { addr: 127.0.0.1:50200, peer: 127.0.0.1:50197, socket: 1300 }

It looks like we are processing JSON-RPC request: HistoryEndpoint(LocalContent) before the uTP stream completes (send FIN packet) and because we are emitting uTP listener close event with a payload after uTP stream is closed, we are storing the accepted content after  the LocalContent request.
Adding some reasonable  tokio::sleep value before calling the LocalContent RPC endpoint in offer/accept test should at least solve this issue.

Hm, any way we can make the portal_historySendOffer call block until the stream is closed?
If we have to go the sleep route, then I'd say we should try optimistically first, and then sleep/retry, to reduce the total test time.

Why this test fails mostly on windows?

Actually, something changed such that I could reproduce it locally on Linux 1/2 to 1/3 of the time. So it may not be too fruitful to hunt down this angle.

Hm, any way we can make the portal_historySendOffer call block until the stream is closed?

We can emit an event from overlay and return a handle for run_trin but not sure if it is worth the effort because it is one more channel and has added complexity.

If we have to go the sleep route, then I'd say we should try optimistically first, and then sleep/retry, to reduce the total test time.

This sounds like a good idea!

We can emit an event from overlay and return a handle for run_trin but not sure if it is worth the effort because it is one more channel and has added complexity.

Ah I think we were talking about different things here. I was imagining having the json-rpc endpoint not return any result until after the offer was successfully accepted. The name & purpose of the endpoint seems specifically tied to sending just the first message in the whole exchange, though. So that probably doesn't make sense either.

authors: carver ogenev


text:
Backfill cold state during Beam sync

This issue is to track some ideas on how to fill in cold state with beam sync.  Mostly a brain dump that hopefully works towards some good ideas.
Goals

Preserve full node caches.
Swarming: syncing nodes should be able to serve other syncing nodes with some reasonably good probability.
Fill in all cold state within a reasonable amount of time.

In order to have a cache preserving behavior clients need to be pulling common state.  i.e. two clients that are beam syncing at the same time should end up pulling roughly the same cold state nodes.  Ideal case is for this property to hold even if one of the clients started their sync at a very different time than the other.
Questions

Is it beneficial to prioritize filling in the account state before filling in the contract state?

There might be an argument that account balances and nonces are more valuable information than the actual contract code and state.   Every contract call interacts with the account but doesn't necessarily interact with the state (but it does require the actual code).
Ideas
Prioritize cold state reads from alternative sources like Swarm.
When looking up cold state, we could try to prioritize fetching it from alternative sources such as Swarm.  This could help preserve full node caches while having the beneficial side effect of keeping cold state active in the swarm network.
Select cold state based on most recent processed block
In order to have multiple syncing clients converge on fetching the same state nodes we could use the most recent block's state reads as the locus for clients to swarm around.  Something like traversing the state trie outwards from the nodes that were read in the most recent block till it finds the closest account that it is missing and fetches it too.  The same procedure could be used for contract state.
This should have the following effects:

clients asking for data from full nodes will be asking for mostly the same data from the same part of the trie.
a client that is syncing should always be able to serve a state request to another client provided that the requesting client started syncing after them.

I thought we had an issue for this, but couldn't find it. Making this the "official" one for now. Forward progress is blocked on (actively progressing) https://github.com/ethereum/py-trie/pull/95.

I can't remember where the proposal is that suggested a 65 bytes index (account + subtrie identifier + subtrie index) -- where subtrie can only be storage for now. So I'll just write out a though I had about it here:
Anything beyond the 32 bytes is pretty useless, because there's effectively 0 chance that you will land directly inside an account with the first 32 bytes. I guess the trailing 32 bytes could mean that the subtrie index somehow applies to the nearest account. Even then, it's very unlikely to help with accounts like IDEX, because you're extremely unlikely to land immediately adjacent to the few giant accounts with the first 32 bytes, so you would still end up scanning them sequentially. There would be issues that it would take longer to eventually land near enough the giant accounts to collect all their storage.
So I think we still don't have a good "random selection" story that deals with accounts that have giant storage.

authors: carver pipermerriam

repo: ethereum/trinity


text:
Implement eth/64 protocol

What was wrong?
Trinity is currently on eth/63 whereas Geth has implemented eth/64 and eth/65 already. We are pushing for the inclusion of request ids as eth/66.
Before we add an implementation for the proposed eth/66 protocol, we should first implement eth/64 and eth/65.
How was it fixed?
This implements the eth/64 protocol according to EIP 2124. The implementation follows the general principle of how LESV1, LESV2 are implemented side by side.
This also adds a tiny cleanup to combine three different places with the same if/else construct to choose the concrete API version into a single helper function.
Real life testing also demonstrates we are having eth/64 peers alongside eth/63 peers in the pool.
Adding ETHPeer (eth, 64) <Session <Node(0x089581@45.40.132.135)> 1da28744-11a1-4ce4-82e3-c84449cd41df> to pool
Adding ETHPeer (eth, 63) <Session <Node(0x5629f4@54.39.133.125)> 656b4238-5afd-4df8-8f29-515022183c0b> to pool

To-Do

[x] Clean up commit history
[x] Update tests
[x] Fill in PR description
[x] Cleanup code
[x] Do the actual ForkID check and disconnect

[x] Add entry to the release notes

Cute Animal Picture

If you can show that ETH/64 is supported our-of-the-box by the majority of network nodes (I don't know whether it actually is...) then we can drop ETH/63

Yeah, I thought about that too but ended up abandoning the thought because:

Geth still supports eth/63
It's a good code exercise to support two different handshakes especially when we already remove LES

Alright, this should be ready for review.

I have reassigned this to you @gsalgado because I think @pipermerriam is very busy right now.

Just ran a last live test and had this error blasting out.
ERROR  2020-02-20 10:20:24,392           ETHPeerPool  unexpected error during peer connection
Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/commands.py", line 131, in decode
    payload = cls.serialization_codec.decode(payload_data)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/commands.py", line 64, in decode
    rlp.decode(data, strict=self.decode_strict, sedes=self.sedes, recursive_cache=True)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/rlp/codec.py", line 237, in decode
    obj = sedes.deserialize(item, **kwargs)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/eth_utils/functional.py", line 45, in inner
    return callback(fn(*args, **kwargs))
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/rlp/sedes/lists.py", line 89, in deserialize
    serial)
rlp.exceptions.ListDeserializationError: Deserializing list length (5) does not match sedes (6)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/protocol/common/peer.py", line 206, in maybe_connect_more_peers
    for backend in self.peer_backends
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/peer_pool.py", line 209, in _add_peers_from_backend
    await self.connect_to_nodes(iter(candidates))
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/peer_pool.py", line 401, in connect_to_nodes
    loop=self.get_event_loop(),
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/peer_pool.py", line 437, in connect_to_node
    peer = await self.connect(node)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/peer_pool.py", line 347, in connect
    timeout=HANDSHAKE_TIMEOUT,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/cancellable.py", line 20, in wait
    return await self.wait_first(awaitable, token=token, timeout=timeout)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/cancellable.py", line 43, in wait_first
    return await token_chain.cancellable_wait(*awaitables, timeout=timeout)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/cancel_token/token.py", line 163, in cancellable_wait
    await future
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/peer.py", line 492, in handshake
    token=self.cancel_token
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/handshake.py", line 323, in dial_out
    token=token,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/handshake.py", line 288, in negotiate_protocol_handshakes
    in zip(selected_handshakers, selected_protocols)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/async_generator/_util.py", line 53, in __aexit__
    await self._agen.athrow(type, value, traceback)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/multiplexer.py", line 381, in multiplex
    fut.result()
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/multiplexer.py", line 404, in _do_multiplexing
    await self._handle_commands(msg_stream, stop)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/multiplexer.py", line 421, in _handle_commands
    async for protocol, cmd in msg_stream:
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/cancellable.py", line 67, in wait_iter
    timeout=timeout,
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/cancellable.py", line 20, in wait
    return await self.wait_first(awaitable, token=token, timeout=timeout)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/cancellable.py", line 43, in wait_first
    return await token_chain.cancellable_wait(*awaitables, timeout=timeout)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.7/site-packages/cancel_token/token.py", line 178, in cancellable_wait
    return done.pop().result()
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/multiplexer.py", line 87, in stream_transport_messages
    cmd = command_type.decode(msg, msg_proto.snappy_support)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/p2p/commands.py", line 136, in decode
    ) from err
rlp.exceptions.DeserializationError: DeserializationError for <class 'trinity.protocol.eth.commands.Status'>

I had live tested with a mix of eth/64 and eth/63 and clients connected without errors. Need to check where this is coming from. It looks like someone is sending us an eth/63Status message where we expect an eth/64 one.
So I need to hold of merging this until I figured out what's wrong here.

@gsalgado Mind taking a look at the most recent commit that I added? This deals with the DeserializationError. From all I can tell, this has nothing to do with this PR per se. I think what happens is that we hit this error if we try to do a handshake with a client that advertises eth/64 but then actually responds with the wrong status message. This is now treated as a MalformedMessage.


text:
build: disable legacy peer deps

Closes #693

Release v7.1.1 addresses this.

repo: ethersphere/bee-js


text:
Enable bee to start as a Windows Service

Task
Currently bee cannot start as a Windows service, it fails with
Error 1053: The service did not respond to the start or control request in a timely fashion
It looks like that solution is to change how bee is being started, either this svc or this kardianos/service
Acceptance criterea
Bee can be started as a Windows Service

Hi @metacertain , could you please attach your PR to this issue?

@metacertain @vandot , I move this to the decommissioned pipeline. @metacertain , please attach your PR to this issue, so we can pick up the work easily when we work on it after the sprint.
Thanks

I have checked state of Windows service with @vandot, and have some pending changes.
Main problem with the current changes/branches that relate to this is that when Bee is started as a service is that we lose logging output. Even though logs are configured to STDOUT, they are not going anywhere. Two main options that can be considered are to either support logging to some file (which is quite a work in itself), or use Windows event log for logs for service (build-msi-2-eventlogger branch). Event log seems to be the place for logs from services.
Other problem is with actual way of starting Windows service.
There is a bit specific way on how the service should be started, and simple example can be found here: golang/sys/blob/master/windows/svc/example/service.go.
Issue I have found here is that name of the service must be global thing, i.e. user cannot create his own Windows service, and Bee can be started as Windows service (of course, unless we expose some additional config). This is not much of a problem, just a thing to note. Creating and starting service must be done with same service name, or the process will not start.
It is quite possible to have Windows-specific code, which will not interfere with other versions.
The kardianos/service library can be used here, mostly to simplify the amount of needed code for starting service. Because it does support many other systems, which we already have support for, we can only use it just for starting the service (removes much of boilerplate).
What I suggest here is to include new CLI command, which would be Windows-only, and would have commands for installing and removing Windows service, that can be called by system that provides Bee package (or manually by user). This would be something like:
# Install Windows service
bee service [install|create]

# Remove Windows service
bee service [uninstall|remove|delete]

Hi @zbiljic , thanks for the effort you put in writing this down.
So, what you suggest is to use https://github.com/kardianos/service as windows-specific code in our Bee codebase and then use windows event log to log our console output?
If the windows event log are the standard to use, this seems acceptable to me. Regarding the service, we should document this properly, but I don't see this as a blocker.
The only doubt that I am having is with regards to maintainability and effort to implement this. Could you please let me know the required effort (approximately) to implement logging and windows service creation, as well as maintenance requirements if we officially support windows.

Yes @Eknir, that is my suggestion. The use of this library can be considered as side thing, just to make things easier.
As far as the work goes, on the one hand we would have the changes required in the code, to add support for another CLI command (used just for Windows), and these other two sub-commands. The code required for sending logs and setting up actual service startup is already done (on separate branch). The other part is actually setting up required installers, which @vandot is familiar with.
This sub-command changes should be pretty trivial, I think most of the "work" be to actually decide on the name of the commands and their descriptions. Those changes can be done by anyone pretty easily. It should not take more than a day to add and test everything. If no problems arise it can be joined with other few changes for the log output also.
As for the setting up different packages for Windows, I cannot say, maybe @vandot could provide more info.
In regards to "official support for Windows", I cannot say that we should or should not support it. The Bee node does work on Windows (when started from terminal). This only comes down to supporting installing Windows service and supporting that way of using it. That to me seems like product decision.

Hi @zbiljic , thanks again (and sorry for the late answer -- I realize this caused delays). I would like to support the windows package, most importantly to allow our users to use bee-clef easily.
So, this should be the user story that needs to be achieved;
"As a user of Swarm who wants to earn gBZZ, I want to easily install and run Bee and Bee-clef, such that my computer will earn gBZZ in the background and my keys are secure and interoperable with Ethereum"

As far as having Clef also, that is one point that @vandot and myself discussed, however that should be separate from this issue. This issue is only related to having possibility for running Bee as a Windows Service.
Having bee-clef in tandem with bee would be whole separate issue in itself. Personally I have not looked at how Clef service works, or even if it possible to have it as Windows Service. As we found out for this issue, some custom code is required to have CLI application running as Windows Service. It very well might be possible that Clef does not have support for it (support being what was discussed for Bee). In that case alternatives would have to be explored.
Of course, there might that there is something that I am not aware, and invite @vandot to comment further on this.

I created an issue for bee-clef. https://github.com/ethersphere/bee-clef/issues/3 . I understand that this is a separate issue. Let's indeed not convolute this issue with non-related things. Perhaps you and @vandot could spend some time (not more than 4h) to explore the possibilities of this and report your findings in the above-mentioned issue?
Thanks @zbiljic

This issue is completed. Could somebody please provide a concise set of instructions (as a comment below), outlining which steps a user needs to take in order to use Bee as a service on Microsoft? You can follow the example from our documentation

This issue is completed. Could somebody please provide a concise set of instructions (as a comment below), outlining which steps a user needs to take in order to use Bee as a service on Microsoft? You can follow the example from our documentation

@Eknir Even though the Bee now supports starting it as a Windows service, the intention was not for the user to manually create the service. After future tickets for chocolatey and scoop documentation should be updated.

@Eknir Even though the Bee now supports starting it as a Windows service, the intention was not for the user to manually create the service. After future tickets for chocolatey and scoop documentation should be updated.

@zbiljic , putting this in the icebox. @vandot , is it on your radar to get bee in scoop/chocolatey?

@zbiljic , putting this in the icebox. @vandot , is it on your radar to get bee in scoop/chocolatey?

I think this issue didn't solve yet. I created my bee manifest for scoop with a custom service helper for fast installing/uninstalling bee service, then I encounter the issue, it can't start and just fails with 1053 error.
screenshot:

authors: Eknir chawyehsu vandot zbiljic

repo: ethersphere/bee


text:
Remove qhull

Build fails on osx.

test this please

lgtm :disappointed:

test this please

Build passes on ubuntu jenkins, osx currently down

test this please

test this please

authors: dymczykm tcies


text:
ImportError: cannot import name 'opentype'

Since the last update of 4 hours ago my CI broke because a module is using your pyasn1 module.
I received the following in my stack trace:
  File "/usr/local/lib/python3.5/dist-packages/oauth2client/_pure_python_crypt.py", line 24, in <module>
    from pyasn1_modules.rfc2459 import Certificate
  File "/usr/local/lib/python3.5/dist-packages/pyasn1_modules/rfc2459.py", line 20, in <module>
    from pyasn1.type import opentype
ImportError: cannot import name 'opentype'

Is this anything related to the last changes made in this module and a way this is fixable on my side or with a new release of this module?

Unfortunately I'm not using ASN myself. It must be in a module that I'm using. Right now I'm trying to find out what module is to see if I can do anything about it.

May be just pip install --upgrade pyasn1-modules ?
Or send pip list here to confirm my suspicion. ;-)

Upgrading pyasn1-modules did the trick! Thanks

Im getting error while upgrading from ubuntu 14 to ubuntu 16. My development server is on ubuntu 14 is has no issue, but now i upgraded my local system with ubuntu 16, and i started getting this error for pip install --upgrade google-cloud-storage package.
Tried pip install --upgrade pyasn1-modules. but same error.
My pip freeze output:
aioredis==1.0.0
asgiref==2.3.0
asn1crypto==0.24.0
async-timeout==2.0.1
attrs==17.4.0
autobahn==18.4.1
Automat==0.6.0
boto3==1.7.7
botocore==1.10.7
cachetools==2.0.1
certifi==2018.4.16
cffi==1.11.5
channels==2.1.1
channels-redis==2.1.1
chardet==3.0.4
click==6.7
constantly==15.1.0
cryptography==2.2.2
daphne==2.1.1
decorator==4.3.0
Django==2.0.4
django-cors-headers==2.2.0
django-fcm==0.1.1
django-filter==1.1.0
django-material==1.2.3
django-simple-history==2.0
django-uuidfield==0.5.0
djangorestframework==3.8.2
djangorestframework-jwt==1.11.0
docutils==0.14
fcm-django==0.2.18
future==0.16.0
geocoder==1.38.1
geographiclib==1.49
geopy==1.13.0
google-api-core==1.1.1
google-auth==1.4.1
google-cloud-core==0.28.1
google-cloud-storage==1.8.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
hiredis==0.2.0
hyperlink==18.0.0
idna==2.6
incremental==17.5.0
ipgetter==0.7
jmespath==0.9.3
msgpack==0.5.6
mysqlclient==1.3.12
Pillow==5.1.0
protobuf==3.5.2.post1
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.18
pyfcm==1.4.5
PyJWT==1.6.1
python-dateutil==2.7.2
pytz==2018.4
ratelim==0.1.6
requests==2.18.4
requests-toolbelt==0.8.0
rsa==3.4.2
s3transfer==0.1.13
six==1.11.0
Twisted==17.9.0
txaio==2.10.0
urllib3==1.22
zope.interface==4.5.0
Any help please...

My guess is that this is perhaps because you have another (older) pyasn1 installed somewhere on the system and picked up accidentally.
Your pip freeze looks good from the versions perspective.

You can try to import pyasn1_modules in Python's verbose mode too see which files it picks up:
$ python -v
>>> import pyasn1_modules

$ python -v >>> import pyasn1_modules import pyasn1_modules # directory /usr/lib/python2.7/dist-packages/pyasn1_modules  /usr/lib/python2.7/dist-packages/pyasn1_modules/__init__.pyc matches /usr/lib/python2.7/dist-packages/pyasn1_modules/__init__.py import pyasn1_modules # precompiled from /usr/lib/python2.7/dist-packages/pyasn1_modules/__init__.pyc

(Virtual_EnvPy35_FC) friendcolony@fc:~$ python -v
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__init__.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/codecs.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/codecs.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/codecs.cpython-35.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468be16d8>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/aliases.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b7d1d0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468be12e8>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/utf_8.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/utf_8.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/utf_8.cpython-35.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b82f60>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/latin_1.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/latin_1.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/latin_1.cpython-35.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b85a90>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/io.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/io.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/io.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/abc.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/abc.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/abc.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/_weakrefset.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/_weakrefset.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/_weakrefset.cpython-35.pyc'
import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b88748>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b85f28>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b85cc0>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/site.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/site.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/os.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/os.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/os.cpython-35.pyc'
import 'errno' # <class '_frozen_importlib.BuiltinImporter'>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/stat.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/stat.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/stat.cpython-35.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b323c8>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/posixpath.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/posixpath.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/posixpath.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/genericpath.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/genericpath.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/genericpath.cpython-35.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b35c88>
import 'posixpath' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b32710>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/_collections_abc.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/_collections_abc.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/_collections_abc.cpython-35.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b3d320>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b9cb00>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/_bootlocale.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/_bootlocale.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/_bootlocale.cpython-35.pyc'
import '_locale' # <class '_frozen_importlib.BuiltinImporter'>
import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b9cda0>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/types.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/types.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/types.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/functools.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/functools.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/functools.cpython-35.pyc'
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/collections/__pycache__/__init__.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/collections/__init__.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/collections/__pycache__/__init__.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/operator.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/operator.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/operator.cpython-35.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b26e48>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/keyword.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/keyword.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/keyword.cpython-35.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466f3c160>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/heapq.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/heapq.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/heapq.cpython-35.pyc'
import '_heapq' # <class '_frozen_importlib.BuiltinImporter'>
import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466f3c9b0>
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/reprlib.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/reprlib.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/reprlib.cpython-35.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466f3ce48>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b08898>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/weakref.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/weakref.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/weakref.cpython-35.pyc'
import 'weakref' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b2f198>
import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468afc518>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/collections/__pycache__/abc.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/collections/abc.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/collections/__pycache__/abc.cpython-35.pyc'
import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b044e0>
import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468af6940>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/__init__.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__init__.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/__init__.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/warnings.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/warnings.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/__pycache__/warnings.cpython-35.pyc'
import 'warnings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b04cf8>
import 'importlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468af6a20>
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/util.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/util.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/util.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/abc.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/abc.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/abc.cpython-35.pyc'
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/machinery.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/machinery.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/importlib/__pycache__/machinery.cpython-35.pyc'
import 'importlib.machinery' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466f61668>
import 'importlib.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466f61240>
# /usr/lib/python3.5/__pycache__/contextlib.cpython-35.pyc matches /usr/lib/python3.5/contextlib.py
# code object from '/usr/lib/python3.5/__pycache__/contextlib.cpython-35.pyc'
import 'contextlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466ef1eb8>
import 'importlib.util' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466f5ccc0>
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/google
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/google
# destroy google
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/google
# destroy google
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/google
# destroy google
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/google/logging
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/google
# destroy google
# possible namespace for /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/zope
# /usr/lib/python3.5/__pycache__/sitecustomize.cpython-35.pyc matches /usr/lib/python3.5/sitecustomize.py
# code object from '/usr/lib/python3.5/__pycache__/sitecustomize.cpython-35.pyc'
# code object from /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/cp437.py
# could not create '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/cp437.cpython-35.pyc': PermissionError(13, 'Permission denied')
# wrote '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/encodings/__pycache__/cp437.cpython-35.pyc'
import 'encodings.cp437' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468af60f0>
# zipimport: found 77 names in '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/pyasn1-0.4.2-py3.5.egg'
import 'sitecustomize' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468af3f98>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0468b99080>
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
# extension module 'readline' loaded from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/lib-dynload/readline.cpython-35m-x86_64-linux-gnu.so'
# extension module 'readline' executed from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/lib-dynload/readline.cpython-35m-x86_64-linux-gnu.so'
import 'readline' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f0466e9b7b8>
>>> import pyasn1_modules
# /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/pyasn1_modules/__pycache__/__init__.cpython-35.pyc matches /home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/pyasn1_modules/__init__.py
# code object from '/home/friendcolony/python_workspace/Virtual_EnvPy35_FC/lib/python3.5/site-packages/pyasn1_modules/__pycache__/__init__.cpython-35.pyc'
import 'pyasn1_modules' # <_frozen_importlib_external.SourceFileLoader object at 0x7f0466e9b978>
>>>

Output of python -v verbose mode with virtualenv activated.

As a small note, I noticed that after updating pyasn1-modules, it was only pyasn1 itself that I had to declare explicitly on a higher version. Whatever version of pyasn1-modules was pulled in then worked, perhaps through a peer dependency of a more complex library, which is probably how a lot of people are running into this error.

pip install -U pyasn1_modules works for me.

pip install -U pyasn1_modules
worked for me

Suggestted pip updates fail to work with virtualenv for me. I had to export CLOUDSDK_PYTHON="/usr/bin/python3"

authors: BHushanRathod DanielMartinus GinSoda etingof jon-torodash qrasulee valentas-kurauskas