text:
Zephyr does not define minimal C++ language standard requirement and does not track it

Describe the bug
Zephyr defines C11 as minimal supported standard for C language in C language support - language standards.
Similar documentation article: C++ language support does not have "language standard" section. Roadmap #31281 also does not say a word about minimal requirements. Thus, minimal C++ version supported is unknown. This is a serious problem since:

C++ language standards have much more changes between releases than C language.
Custom toolchains for embedded systems often concentrate on C language support (due to its popularity) leaving C++ language  with old standard (like dreadful C++98).

Concrete problem encountered:
SOF project uses Cadence Xtensa toolchain "xt-xcc" that is based on GNU 4.2.0 compiler for Intel Meteorlake audio firmware.
Meteorlake board has some C++ code that uses Zephyr #include <zephyr/sys/util.h> that provides C/C++ interropt helpers at the time of writing. "xt-xcc" is obsolete for some time and latest support language is C++98 standard.
Everything was working fine in our CI until PR #53405 , where author added constexpr keyword usage in commit deab09d7a6b0c51a52ffd8bd03f17e04cf39c25c .
Author @cfriedt in that PR claims (selective quote):

There should be a announcement for this and the code that suddenly requires new C++

As mentioned above, this is all in line with Zephyr's suggested ISO C and C++ standards (C11, C++11).

Proof of wrong:
Fuction template definition __z_log2_impl from include/zephyr/sys/util.h in some conditions returns value A and in other conditions value B. According to C++ standard ISO/IEC 14882:2011 §7.1.5.3:

The definition of a constexpr function shall satisfy the following constraints:

it shall not be virtual (10.3);
its return type shall be a literal type;
each of its parameter types shall be a literal type;
its function-body shall be = delete, = default, or a compound-statement that contains only

null statements,
static_assert-declarations
typedef declarations and alias-declarations that do not define classes or enumerations,
using-declarations,
using-directives,
and exactly one return statement;

every constructor call and implicit conversion used in initializing the return value (6.6.3, 8.5) shall be
one of those allowed in a constant expression (5.19).

Also, cppreference.com - constexpr confirms this:
Quote marked "(until C++14)":

the function body must be either deleted or defaulted or contain only the following:

null statements (plain semicolons)
static_assert declarations
typedef declarations and alias declarations that do not define classes or enumerations
using declarations
using directives
if the function is not a constructor, exactly one return statement

To Reproduce
Steps to reproduce the behavior:

Open my minimal reproduction project with Compiler Explorer:
https://godbolt.org/z/78sWY65v4
Play around with compiler version and C++ standard.
Note: compiler x86-64 gcc 12.2 compiles minimal example using C++11 without a warning!
Change compiler to: x86-64 gcc 11.3 to receive an error (related to multiple return statements in constexpr function):

header.h:17:1: error: body of 'constexpr' function 'constexpr int __z_log2_impl(T) [with T = int]' not a return-statement
17 | }
| ^

Change CMake line:
set_property(TARGET output.s PROPERTY CXX_STANDARD 11) to
set_property(TARGET output.s PROPERTY CXX_STANDARD 14)
and watch how everything compiles successfully.

Expected behavior

Zephyr defines minimal C++ language standard in the official documentation.
Right now it is C++14 if we do not roll back PR #53405 (probably - remaining C++ code needs verification too!)
Developers know, and check the C++ standard of the code they wright! Please note, that modern compilers GCC, Clang, MSVC tend to provide "backward compatibility" to old C++ standards what results in non-conforming code being compiled without an issue!
If the compiler supports C++17 or C++20 it is possible that it will compile the code marked -std=c++11 that should not compile with that standard due to compiler extensions they provide! We need to watch the conformance carefully, maybe some static analysis tools?

Impact
SOF project has to urgently upgrade toolchain to new Cadence Xtensa "xt-clang" based on clang, that allows to use up to C++17 standard. If we do not do this, we are unable upgrade Zephyr revisions since our code will not build!
Luckily, we had been preparing for toolchain change in https://github.com/thesofproject/sof/pull/7027 for some time and have only 2 .cpp source files at the time of writing.
Logs and console output
Compiling with xt-xcc:  xt-xcc.log
Compiling with xt-clang -std=c++11:  xt-clang C++11.log
Environment (please complete the following information):

OS: Windows, Linux

Additional context
N/A

@aborisovich - documentation only does so much. Why not contribute a PR? Perhaps a test so you are not inconvenienced the same way in the future. It will probably amount to only a few lines of code.

@aborisovich - documentation only goes so far.
Why not contribute a PR? Perhaps a test so you are not inconvenienced the same way in the future.
It will probably amount to only a few lines of code.

Subject is certainly not easy and we need some tool that would run as part of Zephyr CI. And it will certainly not be only a few lines of code. I'm looking for potential solutions. Maybe something like cppcheck added to Compliance Checks workflow would be good for checking standard conformance...

@aborisovich - can you extract just the relevant portion of your error from the log and put it into a markdown code block, so that the error is a bit more visible / searchable?

@cfriedt I never, ever meant to discredit anyone. I assume that as obvious thing, that we developers work with mutual respect.
Bugs, unspecified behaviors are constant part of our work. Moreover, I strongly approve what you do - C++ is a very needed language in Zephyr and I hope it will be replacing or complimenting C in future. When I wrote "you are doing something wrong" I meat it as a quote, not as an expression targeting you directly. By this quote I meant "Zephyr project systems providing API compatibility". I wanted to indicate importance of the problem I described above and potentially severe impact, so we can harden the processes in Zephyr.

@aborisovich - no harsh feelings. I'm glad that we can work together to make things better.
Please let me know if you need any help.

Using -std=c++11 -pedantic should get you similar earnings with GCC. I'm not sure which backwards compatibility extensions you're worried about, generally GCC and clang will warn if you use language features that are not supported by the standard dialect selected with -std

Zephyr docs don't explicitly say C11 or even C99 is required. C11 is still on the table as a pending requirement in RFC #30105
The docs simply state that C99 features are widely used but there is no requirement on it.

(and having said that it's clear I should just put that text in the docs and mark it as fixing this bug)

Zephyr defines C11 as minimal supported standard for C language in C language support - language standards.

That is not necessarily true. The documentation says that most parts of Zephyr only make use of the features available in C99 with some additional components requiring C11 -- i.e. the "minimum" compiler C standard support requirement varies depending on which component(s) you require.
The documentation does not designate a specific "minimum supported" (or "maximum allowed" for that matter) C standard version because some components may benefit from the features available in a more recent version of the standard; nevertheless, the recommendation is to stick with C99 in general and C11 at most.

Expected behavior

Zephyr defines minimal C++ language standard in the official documentation.

For the same reason described above, I would refrain from defining a hard limit; especially for C++ because newer C++ standards have many appealing features.
The documentation should only provide the general recommendation (and the de facto requirement for the core components of Zephyr) for the C++ standard conformance, which would be to stick with C++11 in general. It would be preferable to allow a more recent version of the C++ standard for specific components and modules.

I reopen this issue because it looks to me that you may missed the key point I am trying to show here.
This statement:

For the same reason described above, I would refrain from defining a hard limit; especially for C++ because newer C++ standards have many appealing features.

Is not a resolution to problem with using C++. I agree that new language feature are appealing, but please note what I wrote in the issue description:

Custom toolchains for embedded systems often concentrate on C language support (due to its popularity) leaving C++ language with old standard (like dreadful C++98).

@andyross, filling up documentation with "We support C++ standard whatever found in the give file you compile" is not a solution either. I would rather call it "discrediting a problem" than solving it.
This problem is not related to C, because Zephyr does great job of hiding C language features under #ifdef macros.
Example: What would happen to all zephyr applications if you suddenly add to very commonly used header (like some util header) alignof keyword (supported since C11).
All zephyr applications using C99 would stop to work and would be forced to use C11 compiler right?
You basically do the same for C++ right now, and cover the issue with a blanked - do not do this.

Hm... as of right now we guarantee that all the OS headers build correctly with C++98 compilers (specifically gcc and clang with -std=c++98), and we're testing this along with all the other .  It's true it was a hole, but I'm pretty sure it's been plugged.  Obviously the discussion as to when to deprecate C++98 will be contentious, but for now at least that's the requirement.
Which is to say: if you push a PR right now using alignof or rvalue references or whatever inside an #ifdef __cplusplus, it will fail in CI.  I think that's what you want, no?
(Note that there are other issues with C++ support at the toolchain level, like "how to link in the standard library" that are quite a bit farther behind.  So while the OS has support lots of platforms won't quite build C++98 correctly even if their compilers technically support it; xcc and xt-clang are both in this category for now.)

Maybe I need to make it simple and more concise:
Commit https://github.com/zephyrproject-rtos/zephyr/commit/deab09d7a6b0c51a52ffd8bd03f17e04cf39c25c should not every compile without setting Kconfig option CONFIG_STD_CPP14=y but it does (using CONFIG_STD_CPP11=y) because the compilers you use does not check standard conformity correctly.

Yeah, those problems are fixed as of #55290
Please open a new bug if you find a way to add non-C++98 code to an OS header without doing proper version detection.

Thanks @andyross . Did not notice a whole PR just a single commit modifying docs. Sorry for the hussle.

Using -std=c++11 -pedantic should get you similar warnings with GCC. I'm not sure which backwards compatibility extensions you're worried about, generally GCC and clang will warn (or error) if you use language features that are not supported by the standard dialect selected with -std. You just need to use the option to ask for warnings about extensions, which is -pedantic.

Indeed adding a -pedantic flag to newest gcc also resolves the issue.

authors: aborisovich andyross cfriedt jwakely stephanosio teburd


text:
LWM2M RD Client stop fn locks up if already suspended

When using LWM2M Client - the sample code flow makes a call to lwm2m_rd_client_stop in error handling cases
This can occur when the engine has already been suspended, and since some recent (~1yr change) this will block indefinitely
The rd client service needs to be looked at so that a call to stop can be made at any time and does not block on failure (or has a maximum timeout)
See discussion.
https://github.com/zephyrproject-rtos/zephyr/discussions/54478#discussion-4827052
To Reproduce
Using lwm2m-client sample code

Load the lwm2m-client sample
Add code to force close LTE client, outside of the lwm2m client loop (by calling lte_lc_offline)
Observe lem2m client enter LTE_NETWORK_ERROR state entered
Observe block on lwm2m_rd_client_stop call
Observe block occurring on ln 1360 of lwm2m_rd_client.c

Expected behavior
The stop call should not block, at least not indefinitily
Impact
No work around for handling error cases, other than using non-threadsafe checks on the state of the client - by the calling code - before calling stop

This looks like a duplicate of https://github.com/zephyrproject-rtos/zephyr/issues/56254

Closing in favor of https://github.com/zephyrproject-rtos/zephyr/issues/56254, please reopen if this is not a duplicate.

source: gharchive/issue

authors: GrantRolls carlescufi rlubos

repo: zephyrproject-rtos/zephyr

license_type: permissive

license_source: github-api


text:
shell: Enable backends via Kconfig file

Added functionality to enable active shell backends via Kconfig
file. When there will be more backends implemented user will
have an option to select only required ones.
It is no longer needed to select SERIAL in prj.conf.
Fixes #10190
Signed-off-by: Jakub Rzeszutko jakub.rzeszutko@nordicsemi.no

Codecov Report

Merging #10379 into master will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #10379      +/-   ##
==========================================
- Coverage    52.9%   52.85%   -0.06%
==========================================
  Files         210      210
  Lines       25732    25732
  Branches     5672     5672
==========================================
- Hits        13614    13600      -14
- Misses       9802     9816      +14
  Partials     2316     2316

Impacted Files
Coverage Δ

lib/posix/clock.c
33.33% <0%> (-38.89%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 316f860...b0a29f0. Read the comment docs.

source: gharchive/pull-request

authors: codecov-io jarz-nordic


text:
logging: Fix RTT log backend lagging if no host

In case it was detected that RTT host was not present
backend was still performing single sleep or busy wait
round before dropping data. That was degrading log
processing performance.
Fixes #13424.
Signed-off-by: Krzysztof Chruscinski krzysztof.chruscinski@nordicsemi.no

Codecov Report

Merging #14033 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14033      +/-   ##
==========================================
- Coverage   52.27%   52.24%   -0.04%
==========================================
  Files         307      307
  Lines       45467    45467
  Branches    10525    10525
==========================================
- Hits        23768    23752      -16
- Misses      16901    16913      +12
- Partials     4798     4802       +4

Impacted Files
Coverage Δ

subsys/testsuite/ztest/include/ztest_assert.h
38.88% <0%> (-38.89%)
:arrow_down:

subsys/testsuite/ztest/src/ztest.c
65.28% <0%> (-7.44%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80fd80a...afe6d7c. Read the comment docs.

@jarz-nordic @pizi-nordic ping

@carlescufi could you merge it? @jarz-nordic is out of office so he will not review that.

authors: codecov-io nordic-krch


text:
ci: send email to builds@ mailing list if master fails

CI always runs after new PRs are merged into master. Report failures to
mailing list to get more attention if master fails to build.

Is builds@zephyrproject.org the same as builds@lists.zephyrproject.org / https://lists.zephyrproject.org/g/builds?
Is there a low traffic list with just failures, no success email?

authors: marc-hb nashif


text:
doc: Bluetooth: Update PICS to 7.4.1

This PR updates/adds the following PICS to 7.4.1.
SM
L2CAP
GAP
GATT
MESH
This fixes #17158

Hi All reviewers,
I created PTS file from Launchstudio to check the consistency and found a few issues.
GAP > SM | 5:C.4 | If [GAP] (41/2a and 41/2b and 43/2a and 43/2b) are Not Supported then [SM] (5/5) is Excluded
Table 5:C.4 | If GAP 41/2a, GAP 41/2b, GAP 43/2a And GAP 43/2b Are Not Supported then SM 5/5 Is Excluded.

I don't think we support Link Key Conversion Function h7. I changed it to False.
16:C.3 | If [MESH] (12/5) is Not Supported then [MESH] (16/5-6) are Excluded

Could you comment on this?
SUM ICS > GAP Table 8A:C.2 | If SUM ICS 34/11 Is Not Supported then GAP 8A/3 Is Excluded.

I don't understand this one. From the GAP ICS 8A/3,

C.2: Mandatory IF (5/1 “Broadcaster, LE Only”) is supported; Optional IF SUM ICS 34/11 “CSSv7 LE Host” is supported; otherwise Excluded.

GAP 5/1 is True so it is a mandatory and SUM ICS 34/11 is No Longer Used, and our selection is valid. Missing something?

I don't think we support Link Key Conversion Function h7. I changed it to False.

We do actually support it, but it's only relevant for dual mode (LE + BR/EDR) support, and since we're doing an LE-only qualification I think the right action is to set it to False.

16:C.3 | If [MESH] (12/5) is Not Supported then [MESH] (16/5-6) are Excluded>
Could you comment on this?

Mesh 12/5 is "Proxy Client" which we don't support. Mesh 16/5-6 are Central role features, i.e. only relevant (for Mesh) if Proxy Client is supported. So for your Mesh qualification none of these should be enabled.

C.2: Mandatory IF (5/1 “Broadcaster, LE Only”) is supported; Optional IF SUM ICS 34/11 “CSSv7 LE Host” is supported; otherwise Excluded.
GAP 5/1 is True so it is a mandatory and SUM ICS 34/11 is No Longer Used, and our selection is valid. Missing something?

I don't think so, just that they failed to update this reference to SUM ICS 34/11 which doesn't exist anymore.

Mesh 12/5 is "Proxy Client" which we don't support. Mesh 16/5-6 are Central role features, i.e. only relevant (for Mesh) if Proxy Client is supported. So for our Mesh qualification none of these should be enabled.

I will mark it False.

Apart from that, GAP 8a/1-17 and GAP 20a/1-17 (except 8a/6 and 20a/6) should be set as discussed on slack with @jhedberg and @joerchan

@MariuszSkamra

(Broadcaster Link Layer Advertising Event Types)
TSPC_GAP_8_3  False ()  Non-Connectable and Non-Scannable Directed Event (C.1)
TSPC_GAP_8_4  False ()  Scannable Directed Event (C.1)
(Peripheral Link Layer Advertising Event Types)
TSPC_GAP_20_5 False ()  Connectable Undirected Event (C.3)
TSPC_GAP_20_6 False ()  Non-Connectable and Non-Scannable Directed Event (C.3)
TSPC_GAP_20_7 False (*) Scannable Directed Event (C.3)
(Peripheral Connection Modes and Procedures)
TSPC_GAP_23_2 False (*) Directed Connectable Mode (C.1)

All of the above belongs to Extended Advertising, not supported.

(Central Privacy Feature)
TSPC_GAP_36_2 False (*) Non-Resolvable Private Address Generation Procedure (C.1)
(Observer Privacy Feature)
TSPC_GAP_17_2 False (*) Non-Resolvable Private Address Generation
Procedure (C.1)
(Broadcaster Privacy Feature)
TSPC_GAP_11_3 False (*) Non-Resolvable Private Address Generation
Procedure (C.2)

Non-resolvable address generation is supported, API call bt_addr_le_create_nrpa.

Found a few misspellings, otherwise looks OK.

Thanks David, I fixed them.

Seems like the backporting of this failed because #15340 from @dbkinder was never backported, causing merge conflicts due to different beginnings to the files. I'll try to create the backport PR manually.

authors: MariuszSkamra jhedberg joerchan tedd-an


text:
Add CANopen support

This series of commits adds CANopen support to Zephyr through the use of the 3rd party CANopenNode protocol stack.
CANopenNode is under the GPLv2 license with linking exception. It is included as a module in Zephyr, but the west manifest still refers to the Vestas Wind Systems fork of the library. The library should be forked into the Zephyr Github project and the manifest updated to reflect this.
Only two changes were done to the CANopenNode fork: One for fixing an format specifier compilation warning (submitted upstream as https://github.com/CANopenNode/CANopenNode/pull/106) and one for making it into a Zephyr module (this will not be upstreamed).
A new API for retrieving the values of the CAN bus error counters was added to Zephyr to support CANopen. This may not be the final solution as @alexanderwachter is working on similar API changes.
This fixes #15278.

@henrikbrixandersen thanks for the PR. I think you need to submit the license for approval to the TSC, since this is a new license.

The checkpatch issues found in samples/subsys/canbus/canopen/objdict are due to auto-generated code for the CANopen object dictionary. Would it be okay to exclude this entire folder in checkpatch.conf?

@henrikbrixandersen thanks for the PR. I think you need to submit the license for approval to the TSC, since this is a new license.

@carlescufi I have read through the guidelines in https://docs.zephyrproject.org/latest/contribute/index.html#contribute-non-apache but found that they are more relevant for code under a different license that is actually included in the Zephyr Git repository (CANopenNode is included as a module)?
I'll be happy to fill in the format README template for TSC/governing board approval. Do I just put the text in a comment here on the issue and flag it somehow?

The checkpatch issues found in samples/subsys/canbus/canopen/objdict are due to auto-generated code for the CANopen object dictionary.

One option is to run the offending file through uncrustify which hopefully can "fix" the source and then submit the result.

One option is to run the offending file through uncrustify which hopefully can "fix" the source and then submit the result.

@jukkar Thanks for the tip. This would have to be done on every regeneration of the code in question, then (e.g. on all edits of the samples object dictionary). I'll try and see how it goes.

One option is to run the offending file through uncrustify which hopefully can "fix" the source and then submit the result.

@jukkar Thanks for the tip. This would have to be done on every regeneration of the code in question, then (e.g. on all edits of the samples object dictionary). I'll try and see how it goes.

Looks like uncrustify isn't sufficient for this. Apart from long lines, C99 comments etc., CO_OD adds typedefs for the complex types used in the object dictionary. It also prefers the types used in CANopenNode over the kernel types preferred in Zephyr (e.g. u8_t vs. uint8_t).

@henrikbrixandersen thanks for the PR. I think you need to submit the license for approval to the TSC, since this is a new license.

@carlescufi I have read through the guidelines in https://docs.zephyrproject.org/latest/contribute/index.html#contribute-non-apache but found that they are more relevant for code under a different license that is actually included in the Zephyr Git repository (CANopenNode is included as a module)?
I'll be happy to fill in the format README template for TSC/governing board approval. Do I just put the text in a comment here on the issue and flag it somehow?

@henrikbrixandersen we quickly discussed this in the TSC yesterday. The guidelines you linked above do apply for modules in the default zephyr upstream manifest, since those are considered part of the zephyr core codebase. Please send an email to: licensereview@lists.zephyrproject.org with the relevant info.

I have been asked a couple of question regarding the exact license of the CANopenNode protocol stack. The correct license identifier, as far as I can tell, is:
SPDX-License-Identifier: GPL-2.0-or-later WITH Classpath-exception-2.0
For reference, see the following pages:

https://github.com/CANopenNode/CANopenNode/blob/master/CANopen.c (license at the top of the file)
https://spdx.org/licenses/GPL-2.0-or-later.html
https://spdx.org/licenses/Classpath-exception-2.0.html

@alexanderwachter I have rebased the branch to use your CAN bus state API.

Thanks @henrikbrixandersen for the contribution. The governing board evaluated the CANopenNode license and decided not to accept it, on the grounds that it is a custom license not approved by the Open Source Initiative (OSI). The board is investigating updating project policy to clearly state that all licenses must be OSI-approved, so future components can be assessed more quickly.

CANopenNode has (mostly) been relicensed under the Apache-2.0 license. There are still a few files in the CANopenNode repository under the GNU GPL with Linking Exception license (LSS support and drivers for other platforms than Zephyr). It is expected that the LSS support files will be relicensed shortly and the drivers for other platforms split into separate repositories.
I am reopening this PR to have it reconsidered for being merged into Zephyr.

authors: MaureenHelm carlescufi henrikbrixandersen jukkar


text:
doc: identify release version in generated docs

It's not obvious which kernel release version you're reading about in the
documentation.  Add the version info in the breadcrumb header (instead
of "Home / Docs / Subsystems /" show as "Home / Docs (1.8) / Subsystems /").
(Depends on docs-theme PR-9, but can be merged now with no ill-effect)
Signed-off-by: David B. Kinder david.b.kinder@intel.com

"Home / Docs (1.8) / Subsystems /").

Why not  "Home / Docs / 1.8 / Subsystems /".
looks cleaner

"as you wish"

authors: dbkinder nashif


text:
Test: correct an error in test_pipe_user_thread2thread().

In test_pipe_user_thread2thread(), what should be tested is
the pipe, which allocated with k_object_alloc(), rather than
"pipe" and "kpipe". That two pipes are already teseted in
test_pipe_thread2thread().
Signed-off-by: Steven Wang steven.l.wang@linux.intel.com

@andrewboie , please take look at it. The original codes was written by you. I would like to make sure that I don't do anything wrong.

@mengxianglinx , please help to review this PR.


text:
Add additonal Stm32g0 socs and nucleog0b1 board.

dts/arm/st: add additional g0 device defintions
soc/arm/st_stm32: Add additional stm32g0 socs
drivers: gpio: add definition for stm32g0 porte
drivers: clock_control: stm32g0: add support for g051, g0b1, g0b0 socs
board: arm: add support for the nucleo_g0b1re board
drivers: clock_control: stm32: rm Kconfig for APB2 prescaler on g0 series
boards: stm32g0: add jlink and stm32cubeprogrammer runners.

Flashing of stm32g0b1 currently works with stm32cubeprogrammer and jlink.
Keil has already released a dsp for the new series, but some issues lead to pyocd flashing failure,
therefore i have added stm32g071 as target for pyocd for now.

@str4t0m Thanks for the work! Some changes required

I have seen that the STM32_open_pin_data repo has been updated today (stm32cubemx 6.2.0 release).

This is the first one that includes the definitions of the g05x socs.
And for g0b1 it now includes the 128k flash variant:
Additionally to the mcu/STM32G0B1R(C-E)Tx.xml they have now added a mcu/STM32G0B1R(B-C-E)Tx.xml  file...

Will we need to wait until the the pinctrl DTS files of the new release are merged in the hal before this pr can be merged, or could this pr be merged before?

I addressed all changes and rebased, @erwango could you have another look?


text:
ISO fixes and improvements for ISOAL and CIS

test: bluetooth: Fix hci_test_app iso_data_pool element size
Size of ISO RX data pool did not account for the EDTT timestamp.
Bluetooth: controller: Fixes for ISO from IAL testing

Prevent NULL-pointer dereferencing if datapath is created late
Support SDU fragment complete-counting for framed case

Bluetooth: controller: Unified ISO-AL TX SDU fragment count
Implemented incrementing TX SDU fragment count such that it indicates
the number of completed SDU fragments in the PDU being emitted for both
unframed and framed transmission.
tests: bluetooth: Reworked ISO-AL RX unit tests
Fixes incorrect use of the test framework and updates tests according
to the new test API.
Bluetooth: controller: fix for some problems in ISO-AL RX and HCI
HCI:

Discarded data from HCI ISO Data packets from controller to host if
data has been lost

ISO-AL:

Corrected iso_interval in latency calculations
Updated handling and release of SDUs for error conditions / padding
at the last PDU for the SDU
Updated prioritisation of error status in released PDUs
Included error spooling exit on based on payload number to SDU
mapping for unframed reassembly
Updated sequence number handling for framed recombination

Bluetooth: controller: Asynch stop of ISO resume ticker in disable
Do not use ull_ticker_stop_with_mark for ISO one-shot resume ticker, but
stop without checking result. If active this will stop it, otherwise it
is ignored. Also, this prevents calling lll_disable twice.
Bluetooth: controller: LLCP and ISOAL Fixes for EBQ LL tests

Prioritize CIS_REQ handling in (old) LLCP
Reject if CIS_REQ uses exisiting CIS ID
These fixes prevent assertions in /LL/CIS/PER/BV-38-C.

@Thalley, should we take out unit test commit from this PR - seems it may need some optimization/rework? Would require disabling that one unit test suite until new tests are committed.

@Thalley, should we take out unit test commit from this PR - seems it may need some optimization/rework? Would require disabling that one unit test suite until new tests are committed.

It is not necessary. If the test is running and passing, then we can ignore/postpone any/all my comments for the tests, as those would just be minor fixes or optimizations.

@mtpr-ot I think I've given the feedback I have (still some pending). Given the lack of understanding of many of the things in the controller, I am a bit vary of approving this, as I do not feel like I have the necessary insight.

I am a bit vary of approving this

@Thalley fair enough. Thanks for your contribution. We will let Vinayak be final approver.

@wopu-ot, please re-visit as you approval has become stale. We need to get this merged!

authors: Thalley mtpr-ot


text:
Allow Cortex-R52 to configure VFP

The Cortex-R52 processor supports single-precision floating-point instructions and can optionally support the double-precision floating-point instructions.
Currently, if selecting CONFIG_FPU=y and for instance CONFIG_VFP_DP_D16=y, then the -mfpu compiler option won't be set:
arm-zephyr-eabi-gcc: error: missing argument to '-mfpu='

Hi @stephanosio, may I get your review on this patch? some time ago we discussed on discord about the FPU support for Cortex-R52. Thanks.


text:
bluetooth: tester: Add nrf5340_hci_rpmsg.conf file

To apply when the hci_rpmsg controller with overlay nrf5340_cpunet_iso-bt_ll_sw_split.conf is used on the netcore.

@szymon-czapracki


text:
doc: Add INFINEON Prefix to ifx supported boards

Add INFINEON Prefix to boards supported by IFX so that documentation shows "INFINEON" properly

Any reason this is all caps?
@galak  : Just following how other vendors (like ESP, NXP) are defining in https://docs.zephyrproject.org/latest/boards/index.html

Any reason this is all caps?
@galak  : Just following how other vendors (like ESP, NXP) are defining in https://docs.zephyrproject.org/latest/boards/index.html

And this capitalization is aligned with our Marketing team's desires.

And this capitalization is aligned with our Marketing team's desires.

Just be aware there are some places that all caps is not accepted (main one I can think of is devicetree).

authors: galak ifyall praveenCY


text:
Bluetooth: BAP: Unicast client Split start and connect

Removes the CIS connection establishment from bt_bap_stream_start and move the behavior to a new funciton bt_bap_stream_connect.
This has 2 advantages:

The behavior of bt_bap_stream_start is much more clear and more aligned with the spec's behavior for the receiver start ready opcode. 2) It is possible to connect streams in both the enabling and the QoS configured state with bt_bap_stream_connect as per the spec. This allows us to pass additional PTS test cases.

To implement this new behavior, samples and tests have been updated. The CAP Initiator implementation has also been updated, and slightly refactored, to accomodate for the change in BAP, but the CAP initiator implementation should work the same for application, except that it's now possible to do unicast start on ASEs in any order (https://github.com/zephyrproject-rtos/zephyr/issues/72138).
fixes https://github.com/zephyrproject-rtos/zephyr/issues/72138
fixes https://github.com/zephyrproject-rtos/zephyr/issues/70089

Rebased to solve merge conflicts

Rebased to solve merge conflicts


text:
bluetooth: hci_nxp: move vendor specific setup to its dedicated place

Some vendor specific setup was done inside
the open() HCI function, those should be
inside setup() function instead.

Hi @jhedberg, this is to address your comment in here:
https://github.com/zephyrproject-rtos/zephyr/pull/72323#discussion_r1625459577


text:
boards: imx93_evk: add board.c to control board mux

i.MX 93 EVK has a series of digital muxes controlled by ADP5585 GPIO expander. This commit adds board level initialization to such muxes to enable selecting mux using Kconfig option.

this will be updated once #76195 is approved. DNM

this will be updated once #76195 is approved. DNM

Why?

Note that we HAVE TO override  board mux selection by other enabled Kconfig options. Otherwise, samples / tests will fail.

@CkovMk , would you please update board document to add the information for how to configure exp_sel?

updated based on comments and try fix ci compliance wrn.

authors: CkovMk JiafeiPan


text:
drivers: wdog: fix wrong logic in NXP drivers

This pull request fixes #79178
The wdog driver for NXP implements a wrong logic for the sleep and debug bits.
It does not stop the wdog if in debug or sleep state, leading to a false trigger.

Thank you for the PR. Kindly help resolve the Compliance failures.

I'm not sure it is worth going back to v3.6 to address as support for v3.6 ends next month.

I agree with @dleach02 here. However @VinFar if you are interested in sending a PR with this fix to 3.6, please feel free to do so (I'd be happy to review it)

bug was fixed in 3.7

I'm not sure it is worth going back to v3.6 to address as support for v3.6 ends next month.

I agree with @dleach02 here. However @VinFar if you are interested in sending a PR with this fix to 3.6, please feel free to do so (I'd be happy to review it)

I have to open a new PR for backporting this in Zephyr 3.6, right?

I have to open a new PR for backporting this in Zephyr 3.6, right?

Yes, that's correct


text:
nxp_imx: Move i.MX RT PLL selects to Kconfig.soc

PLL configuration options for i.MX RT SoCs were added in commit
3fd25c64c779ee06e43bfc7c8fb304cbe01d8e5d, but the selects were
incorrectly added to the SoC defconfig rather than Kconfig.soc. This
resulted in the PLL options not being configured and the part not
booting properly.
Signed-off-by: Maureen Helm maureen.helm@nxp.com
cc: @jhqian

Codecov Report

Merging #8170 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #8170      +/-   ##
==========================================
+ Coverage   64.54%   64.54%   +<.01%
==========================================
  Files         420      420
  Lines       40142    40142
  Branches     6765     6765
==========================================
+ Hits        25910    25911       +1
  Misses      11110    11110
+ Partials     3122     3121       -1

Impacted Files
Coverage Δ

boards/posix/native_posix/timer_model.c
100% <0%> (+1.78%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae829b2...e6b6ba1. Read the comment docs.

+1 to this.  Fixes the multiple warning spew when generating .config referring to "has selects or implies".

authors: MaureenHelm codecov-io mike-scott


text:
Research spreading of vdisk data sharding as it is

Would be great to investigate how well the data is really spread using the modulo index as we do, and it might be nice to also try out the jumpConsistentHash and see how well it would perform compared to the modulo index used right now.

NBD Vdisk Data Sharding
The data for vdisks (as in data blocks, 4 KiB by default for deduped vdisks), are sharded amongst several servers within a single cluster. Where it's not surprising to have 1 cluster contain 10 to 40 servers.
Non Deduped Vdisk Data Sharding
For non deduped vdisks we get the shard index content (a block) is to be stored on by taking acount the amount of avialable shards (servers) and the index of that block:
shardIndex = blockIndex % numberOfAvailableShards

Using the modulo index is however not the best way to guarantee well-distributed sharding. An alternative could be to use the jumpConsistentHash algorithm described in https://arxiv.org/pdf/1406.2294.pdf. We already use that algorithm for the LBA bucket sharding, and we could just as wel use it for the non deduped vdisk data sharding. It's a fast algorithm and performs better data-spreading.
Applying the algorithm above, getting the shard index would now be as follows:
shardIndex = jumpConsistentHash(blockIndex, numberOfAvailableShards)

Example/Experiment
Original Setup
Start nbdserver with a vdisk configured using 6 data storage servers:
bin/nbdserver -protocol tcp -address localhost:6666 -v

Then convert an ubuntu image:
qemu-img convert -p -n ubuntu-16.04-server-cloudimg-amd64-disk1.img  \
    -O nbd nbd+tcp://localhost:6666/e

Check block count for each server:
redis-cli -p 16380 HLEN nondedup:e
(integer) 36766
redis-cli -p 16381 HLEN nondedup:e
(integer) 36772
redis-cli -p 16382 HLEN nondedup:e
(integer) 36763
redis-cli -p 16383 HLEN nondedup:e
(integer) 36757
redis-cli -p 16384 HLEN nondedup:e
(integer) 36771
redis-cli -p 16385 HLEN nondedup:e
(integer) 36766
redis-cli -p 16386 HLEN nondedup:e
(integer) 36761

Observation: We have a pretty good data spreading. This does surprise me. Which might mean that we can keep using modulo index for non dedupe vdisk data sharding after all. I tried the same for a less big image (Lede), and the data spreading was even better, so yeah, I think it's safe to say we can keep the modulo index data spreading for non deduped vdisks.
Setup using the jumpConsistentHash algorithm
Start nbdserver with a vdisk configured using 6 data storage servers:
bin/nbdserver -protocol tcp -address localhost:6666 -v

Then convert an ubuntu image:
qemu-img convert -p -n ubuntu-16.04-server-cloudimg-amd64-disk1.img  \
    -O nbd nbd+tcp://localhost:6666/e

Check block count for each server:
redis-cli -p 16380 HLEN nondedup:e
(integer) 36765
redis-cli -p 16381 HLEN nondedup:e
(integer) 36759
redis-cli -p 16382 HLEN nondedup:e
(integer) 36803
redis-cli -p 16383 HLEN nondedup:e
(integer) 36764
redis-cli -p 16384 HLEN nondedup:e
(integer) 36730
redis-cli -p 16385 HLEN nondedup:e
(integer) 36894
redis-cli -p 16386 HLEN nondedup:e
(integer) 36641

Observation: The data spreading is worse than the modulo index approach. So yeah, It is defintely safe to say that we can keep the modulo index data spreading for non deduped vdisks as it has a better data spreading, and performs way better as well in terms of speed.
Deduped Vdisk Data Sharding
For deduped vdisks we get the shard index content (a block) is to be stored on by taking account the amount of available shards (servers) and the hash of that block:
shardIndex = firstByte(hash(block)) % numberOfAvailableShards

Again using simply the modulo index is a problem as it is not the best way to distribute data. But here there is also a second issue. By only ever using the first byte of a hash, we run into the danger that we might also get worse data spreading by only looking at the first byte which might for whatever reason be often very similar.
In order to improve deduped vdisk data sharding we therefore should ideally be able to look at the entire hash and shard somehow based on that. This needs to remain however very efficient and guarantee a great data distribution no matter what vdisk.
For now I do not have a suggestion on how to resolve data distribution for deduped vdisks, will require some more research and deep thinking.
Example/Experiment
I tried the same experiment setup as non deduped vdisks, but than for deduped vdisks.
When calculating the shard index as:
shardIndex = firstByte(hash(block)) % numberOfAvailableShards

I get:
redis-cli -p 16380 KEYS '*' | wc -l
   37376
redis-cli -p 16381 KEYS '*' | wc -l
   37594
redis-cli -p 16382 KEYS '*' | wc -l
   37364
redis-cli -p 16383 KEYS '*' | wc -l
   38523
redis-cli -p 16384 KEYS '*' | wc -l
   36498
redis-cli -p 16385 KEYS '*' | wc -l
   36555
redis-cli -p 16386 KEYS '*' | wc -l
   36553

And when calculating the shard index as:
shardIndex = jumpConsistentHash(firstByte(hash(block)), numberOfAvailableShards)

I get:
redis-cli -p 16380 KEYS '*' | wc -l
   39384
redis-cli -p 16381 KEYS '*' | wc -l
   36570
redis-cli -p 16382 KEYS '*' | wc -l
   34577
redis-cli -p 16383 KEYS '*' | wc -l
   31522
redis-cli -p 16384 KEYS '*' | wc -l
   40475
redis-cli -p 16385 KEYS '*' | wc -l
   34549
redis-cli -p 16386 KEYS '*' | wc -l
   43386

Conclusion
So while the modulo index based on the first byte of the hash, is not as perfect data spreadin, compared to the non-deduped vdisks, it is still better data spreading than when using the jumpConsistentHash algorithm compared to data spreading. Here It's easier for me to explain why the modulo index does work quite good, as the hash of a block is generated using a crypto hashing algorithm (blake2b), thus that hash value should aready be as random as possible, giving a good data spreading. It's also once again way faster to use the modulo index compared to the jumpConsistentHash, which is not surprising.
I think if we would find a way to involve all the bytes of a block hash in order to compute the shard index, we might get an even better data sharding, but for now I can't really think for a good algorithm for that.

@FastGeert commented:

Comment on bad rebalance
The failure module mapping is actually made on a per vdisk basis. So the way how you present it is not really correct. However, i also do not really like the way how we proposed it in the original spec.
Experiment, recalculate shard by rehashing the key combined with a random string
In this experiment I want to learn how cpu-intensive this approach would be, as we then do not have to recalculate shards anymore on a per vdisk basis.
Basic algoritm:
int getShard(blockaddress) {
    int shard = blockaddress % SHARD_COUNT;
    if ( isShardOk(shard))
        return shard;
    int shard_index = hash(blockaddress, getShardFailureSeed(shard))[0] % getShardCountAfterFailure(shard);
    int shard_counter = 0;
    for (int i=0; i< SHARD_COUNT; i++) {
        if ( ! isShardOk(i) )
            continue;
        if (shard_counter == shard_index)
            return i;
        shard_counter ++;
    }
}

I think there is a flaw in your proposed algorithm @FastGeert:
for (int i=0; i< SHARD_COUNT; i++) {
        if ( ! isShardOk(i) )
            continue;
        if (shard_counter == shard_index)
            return i;
        shard_counter ++;
    }

This part isn't consistent. Meaning that it could return a different result in deferent scenarios, even though the block address is the same. If first only shard B is not longer OK. We'll get shard C for that block (let's just assume that this is the shard_index returned by your shard_index computation line). But if now for some reason shard A also is no longer OK, that part of your algorithm would return shard D, even though the block really was moved and lives on shard C since the original rebalancing...

@FastGeert commented:

I know, its not finished yet. Still need to support multiple failing shards.

Ok @FastGeert, I have my naive solution ready which seems to work:
You can see and run it yourself @ https://play.golang.org/p/dgeUDzYe8A
The most important part of that file is the actual shardIndex algo:
func (s *storage) getShardIndex(blockIndex int64) int64 {
	shardCount := s.shardCount
	shards := make([]shardMetadata, shardCount)
	copy(shards, s.shardsMetadata)

	for {
		if len(shards) == 0 {
			panic("no shard available")
		}

		si := blockIndex % shardCount
		if !shards[si].disabled {
			return shards[si].index
		}

		// not here, let's remove the gap and continue trying
		shards = append(shards[:si], shards[si+1:]...)
		shardCount--
	}
}

I hope it can be implemented a bit more optimal (especially without having to copy an entire slice each time we want to get a shardIndex, but at least the algorithm does work as it is, so it gives us a good starting point.
Using this algorithm we'll always be able to get the same consistent shard index for a given block index. The data spreading isn't ideal, but it does seem to remain pretty stable and never really gets too much out of hand. What do you think?
CC @ChrisVdg
UPDATE:
I tried to "improve" the above implementation by using the jumpConsistentHash algorithm instead of modulo, but just like my initial attempts as documented in this proposal, it also performs here much worse compared to simply using the modulo operator (both in terms of speed and data spreading).
You can find the updated playground for the failed attempt here: https://play.golang.org/p/D4daZZtTL9

I have my naive solution ready which seems to work @ https://play.golang.org/p/dgeUDzYe8A

This solution might actually work and be very cheap if somehow we would be able to pre-compute a remapping slice only when re-loading configuration and when a shard gets disabled. As we would than simply be able to use 1 module index operator and it would give us the right index.
UPDATE: https://play.golang.org/p/IfaJghMHop is a slightly improved version upon the one above. We still require an allocation, but only if the default shard that we are appointed to using the module operator is offline. So for most blocks that should still mean we don't require an allocation at all.

Well, instead of reslicing, just skipping the gaps will be a cheaper solution, because it will prevent alot of costly object creation actions.

@FastGeert we can't just skip gaps as this will prevent us from getting persistent results, no matter how many more shards drop in the future.
Example
Let's say that you have 8 blocks and 4 servers:

A
B
C
D

0
1
2
3

4
5
6
7

Now if server C Drops, we'll get the following result using my proposed algorithm:

A
B
OFFLINE
D

0
1

3

4
5

7

6

2

If we would simply skip gaps we'll get:

A
B
OFFLINE
D

0
1

3

4
5

7

6

2

As you can see at the moment we'll get the same.
Good you might think, this is more efficient and it seems to work. So why am I not doing this?
Well let's see what now happens when also server B goes offline. Remember we don't ever rebalance data for which the server/shard is still online.
With my proposed algorithm we'll now get:

A
OFFLINE
OFFLINE
D

0

3

4

7

6

2

1

5

The respreading is not optimal. With this algorithm you'll notice that the less the amount of shards the more the worse the respreading get. However it will as far as I've tested remain reasonable at all times. And more importantly. It will only ever move data which was previously on an offline shard.
Let's now see what would happen with the data when we simply skip the gaps, as you propose:

A
OFFLINE
OFFLINE
D

0

1

2

3

4

5

6

7

Wow! The respreading is still optimal, great! Yes, but look at what happened to block 2. Remember it moved to server D after C went offline. But now it is suddenly on server D, even though server D is still online. Thus we just violated the design requirement where we only want to move data which otherwise no longer available due to being on an offline shard.
Like I said before, my proposed algorithm does come with its limits, but as far as I know it is performing reasonable. I'll be able to tell more benchmark numbers once I was able to use the algorithm in some real-world example.
Until than, here is a quick visualization to show that the data-spreading of 10 servers and 150 blocks, while not optimal, remains reasonable at all times:
shard #00 (count: 015) => xxxxxxxxxxxxxxx
shard #01 (count: 015) => xxxxxxxxxxxxxxx
shard #02 (count: 015) => xxxxxxxxxxxxxxx
shard #03 (count: 015) => xxxxxxxxxxxxxxx
shard #04 (count: 015) => xxxxxxxxxxxxxxx
shard #05 (count: 015) => xxxxxxxxxxxxxxx
shard #06 (count: 015) => xxxxxxxxxxxxxxx
shard #07 (count: 015) => xxxxxxxxxxxxxxx
shard #08 (count: 015) => xxxxxxxxxxxxxxx
shard #09 (count: 015) => xxxxxxxxxxxxxxx
===============
shard #00 (count: 017) => xxxxxxxxxxxxxxxxx
shard #01 (count: 017) => xxxxxxxxxxxxxxxxx
shard #02 (count: 017) => xxxxxxxxxxxxxxxxx
shard #03 (count: 017) => xxxxxxxxxxxxxxxxx
shard #04 (count: 016) => xxxxxxxxxxxxxxxx
shard #05 (count: 016) => xxxxxxxxxxxxxxxx
shard #06 (count: 016) => xxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 017) => xxxxxxxxxxxxxxxxx
shard #09 (count: 017) => xxxxxxxxxxxxxxxxx
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 019) => xxxxxxxxxxxxxxxxxxx
shard #02 (count: 019) => xxxxxxxxxxxxxxxxxxx
shard #03 (count: 019) => xxxxxxxxxxxxxxxxxxx
shard #04 (count: 019) => xxxxxxxxxxxxxxxxxxx
shard #05 (count: 019) => xxxxxxxxxxxxxxxxxxx
shard #06 (count: 019) => xxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 018) => xxxxxxxxxxxxxxxxxx
shard #09 (count: 018) => xxxxxxxxxxxxxxxxxx
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 022) => xxxxxxxxxxxxxxxxxxxxxx
shard #02 (count: 020) => xxxxxxxxxxxxxxxxxxxx
shard #03 (count: 000) => OFFLINE
shard #04 (count: 023) => xxxxxxxxxxxxxxxxxxxxxxx
shard #05 (count: 021) => xxxxxxxxxxxxxxxxxxxxx
shard #06 (count: 022) => xxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 021) => xxxxxxxxxxxxxxxxxxxxx
shard #09 (count: 021) => xxxxxxxxxxxxxxxxxxxxx
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 028) => xxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #02 (count: 024) => xxxxxxxxxxxxxxxxxxxxxxxx
shard #03 (count: 000) => OFFLINE
shard #04 (count: 026) => xxxxxxxxxxxxxxxxxxxxxxxxxx
shard #05 (count: 023) => xxxxxxxxxxxxxxxxxxxxxxx
shard #06 (count: 024) => xxxxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 025) => xxxxxxxxxxxxxxxxxxxxxxxxx
shard #09 (count: 000) => OFFLINE
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 032) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #02 (count: 000) => OFFLINE
shard #03 (count: 000) => OFFLINE
shard #04 (count: 031) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #05 (count: 027) => xxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #06 (count: 030) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 030) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #09 (count: 000) => OFFLINE
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 037) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #02 (count: 000) => OFFLINE
shard #03 (count: 000) => OFFLINE
shard #04 (count: 000) => OFFLINE
shard #05 (count: 035) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #06 (count: 042) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 036) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #09 (count: 000) => OFFLINE
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 049) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #02 (count: 000) => OFFLINE
shard #03 (count: 000) => OFFLINE
shard #04 (count: 000) => OFFLINE
shard #05 (count: 045) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #06 (count: 056) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 000) => OFFLINE
shard #09 (count: 000) => OFFLINE
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 072) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #02 (count: 000) => OFFLINE
shard #03 (count: 000) => OFFLINE
shard #04 (count: 000) => OFFLINE
shard #05 (count: 000) => OFFLINE
shard #06 (count: 078) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 000) => OFFLINE
shard #09 (count: 000) => OFFLINE
===============
shard #00 (count: 000) => OFFLINE
shard #01 (count: 000) => OFFLINE
shard #02 (count: 000) => OFFLINE
shard #03 (count: 000) => OFFLINE
shard #04 (count: 000) => OFFLINE
shard #05 (count: 000) => OFFLINE
shard #06 (count: 150) => xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
shard #07 (count: 000) => OFFLINE
shard #08 (count: 000) => OFFLINE
shard #09 (count: 000) => OFFLINE
===============

license_source: bigquery


text:
Remove Cargo.lock from version control

Closes #8
Signed-off-by: Aurabindo Jayamohanan mail@aurabindo.in

Thats weird, locally the tests are all passing for me. Whats the problem ?

@aurabindo rebuilded it.

Can you send the PR to develop branch and close this one?

Done, please see #9 but it fails too

authors: aurabindo joeirimpan

repo: zerodhatech/kiteconnect-rs

license: MIT


text:
suddenly connects, even though the user is offline (sometimes)

So I created the TikTok Live Connector as a tool to check whether this username is live or not within a certain period of time. This is what the code looks like
const { WebcastPushConnection } = require('tiktok-live-connector');

let tiktok = new WebCastPushConnection('username');
let connected = false;

setInterval(()=> {
  if(connected === true) return;
  tiktok.connect().then((stats) => console.log(stats.roomId)).catch((err) => console.error(err));
}, 60000);

tiktok.on('StreamEnd', () => {
  if(connected){
      connected = false;
  }
});

Sometimes this code runs very well, but sometimes it also reads that the user is live even though the user is offline. maybe someone has an idea about this?

Same problem here
I cannot find any reliable method to check live true/false

So I created the TikTok Live Connector as a tool to check whether this username is live or not within a certain period of time. This is what the code looks like
const { WebcastPushConnection } = require('tiktok-live-connector');

let tiktok = new WebCastPushConnection('username');
let connected = false;

setInterval(()=> {
  if(connected === true) return;
  tiktok.connect().then((stats) => console.log(stats.roomId)).catch((err) => console.error(err));
}, 60000);

tiktok.on('StreamEnd', () => {
  if(connected){
      connected = false;
  }
});

Sometimes this code runs very well, but sometimes it also reads that the user is live even though the user is offline. maybe someone has an idea about this?

This functionality is implemented in the python version.
https://github.com/isaackogan/TikTokLive?tab=readme-ov-file#checking-if-a-user-is-live

i just added a conditional value like this
Object.keys(stat.roomInfo).length > 2
so when the library suddenly connect to room even the user is offline, the roomInfo is empty so we can do like tiktok.disconnect() first and then we try to connect to check the user live or not.

authors: ZmoleCristian bakteribaik

repo: zerodytrash/TikTok-Live-Connector


text:
Add Libretro Wolfenstein3d Port

This works great... once. For some reason on exit it puts the framebuffer in an unuseable state, I'm pretty sure the module is done proper, but I think there is still some upstream stuff that needs to be sorted.
Note that this module is built based on the latest port changes from RetroPie.

I'll pull this into its own branch until it's stable.

I merged this into the chocolate-wolf3d branch and moved it from ports to the newly created libretrocores. I'll give this a try sometime later tonight.

appreciated. they haven't actually done any libretro porting really so its really just wolfenstein right now,  don't know if I'll hold my breath on them porting it anytime soon, but its easier to set up than wolf4sdl with the obvious glaring issue of the jacked up framebuffer (even after using runcommand)

Is it something that we can fix by simply running something like fbset -depth 32 and then fbset -depth 16 after the game closes out?

It may be possible, I haven't looked into it too extensively tbh.

authors: HerbFargus zerojay

repo: zerojay/RetroPie-Extra

license: mit


text:
QueryRowsPartial: assembly variables in the list?

type User struct {
	UserId            int64  `db:"user_id"`
	Nickname      string `db:"nickname"`
	Introduction  string `db:"introduction"`
}
user := User{}
DB.QueryRowPartial(&user, sqlGetUserNicknameByUserId, userId)
DB.QueryRowPartial(&user, sqlGetUserIntroductionByUserId, userId)

这样会拼装好 user 没有问题
users := make([]User, 0)
DB.QueryRowsPartial(&users, sqlGetUserNicknamesByUserIds, userIds)
DB.QueryRowsPartial(&users, sqlGetUserIntroductionsByUserIds, userIds)

这样会返回一个列表，前一半有 nickname，后一半有 introduction，与期望的不符，期望返回的列表的数据是完整的

可否提供一个函数，比如 QueryRowsRowPartial，实现返回一个拼装好 nickname 和 introduction 的列表
使用场景：introduction 这种字段是 text 类型，会单独建表存储，但是和 user 属于同一个模型，还有重构业务时，新的模型可能是从多个表里获取的
大致想法：查询 introduction 时，将结果存到一个 map 里，key 是 select 的第一个字段（user_id），value 是查询结果（user_id,introduction），然后根据 key 找到列表里的对应记录，将 value 拼到对象里，拼装逻辑可以和 QueryRowPartial 一样

Partial means only some columns are returned.

authors: kevwan wuyazi

repo: zeromicro/go-zero


text:
Unable to install in python 3.7.7 alpine3.11 docker

On my archlinux host i have
 python 3.7.3
 pipenv, version 2018.11.26

I have Pipfile and Pipfile.lock with many packages inside them. I install all the packages in a virtualenv using pipenv install --dev
I have psycopg2 2.8.4 and pyzmq packages which also get installed without any error.
But in the docker python:3.7.7-alpine3.11 i am not able to install all my packages from Pipfile and Pipfile.lock.
psycopg2 2.8.4 and pyzmq packages dont get installed and throw error.
pyzmq inside Pipfile.lock
"pyzmq": {
    "hashes": [
        "sha256:01b588911714a6696283de3904f564c550c9e12e8b4995e173f1011755e01086",
        "sha256:0573b9790aa26faff33fba40f25763657271d26f64bffb55a957a3d4165d6098",
        "sha256:0fa82b9fc3334478be95a5566f35f23109f763d1669bb762e3871a8fa2a4a037",
        "sha256:1e59b7b19396f26e360f41411a5d4603356d18871049cd7790f1a7d18f65fb2c",
        "sha256:2a294b4f44201bb21acc2c1a17ff87fbe57b82060b10ddb00ac03e57f3d7fcfa",
        "sha256:355b38d7dd6f884b8ee9771f59036bcd178d98539680c4f87e7ceb2c6fd057b6",
        "sha256:4b73d20aec63933bbda7957e30add233289d86d92a0bb9feb3f4746376f33527",
        "sha256:4ec47f2b50bdb97df58f1697470e5c58c3c5109289a623e30baf293481ff0166",
        "sha256:5541dc8cad3a8486d58bbed076cb113b65b5dd6b91eb94fb3e38a3d1d3022f20",
        "sha256:6fca7d11310430e751f9832257866a122edf9d7b635305c5d8c51f74a5174d3d",
        "sha256:7369656f89878455a5bcd5d56ca961884f5d096268f71c0750fc33d6732a25e5",
        "sha256:75d73ee7ca4b289a2a2dfe0e6bd8f854979fc13b3fe4ebc19381be3b04e37a4a",
        "sha256:80c928d5adcfa12346b08d31360988d843b54b94154575cccd628f1fe91446bc",
        "sha256:83ce18b133dc7e6789f64cb994e7376c5aa6b4aeced993048bf1d7f9a0fe6d3a",
        "sha256:8b8498ceee33a7023deb2f3db907ca41d6940321e282297327a9be41e3983792",
        "sha256:8c69a6cbfa94da29a34f6b16193e7c15f5d3220cb772d6d17425ff3faa063a6d",
        "sha256:8ff946b20d13a99dc5c21cb76f4b8b253eeddf3eceab4218df8825b0c65ab23d",
        "sha256:972d723a36ab6a60b7806faa5c18aa3c080b7d046c407e816a1d8673989e2485",
        "sha256:a6c9c42bbdba3f9c73aedbb7671815af1943ae8073e532c2b66efb72f39f4165",
        "sha256:aa3872f2ebfc5f9692ef8957fe69abe92d905a029c0608e45ebfcd451ad30ab5",
        "sha256:cf08435b14684f7f2ca2df32c9df38a79cdc17c20dc461927789216cb43d8363",
        "sha256:d30db4566177a6205ed1badb8dbbac3c043e91b12a2db5ef9171b318c5641b75",
        "sha256:d5ac84f38575a601ab20c1878818ffe0d09eb51d6cb8511b636da46d0fd8949a",
        "sha256:e37f22eb4bfbf69cd462c7000616e03b0cdc1b65f2d99334acad36ea0e4ddf6b",
        "sha256:e6549dd80de7b23b637f586217a4280facd14ac01e9410a037a13854a6977299",
        "sha256:ed6205ca0de035f252baa0fd26fdd2bc8a8f633f92f89ca866fd423ff26c6f25",
        "sha256:efdde21febb9b5d7a8e0b87ea2549d7e00fda1936459cfb27fb6fca0c36af6c1",
        "sha256:f4e72646bfe79ff3adbf1314906bbd2d67ef9ccc71a3a98b8b2ccbcca0ab7bec"
    ],
    "version": "==18.1.1"

psycopg2 inside Pipfile.lock
"psycopg2": {
    "hashes": [
        "sha256:4212ca404c4445dc5746c0d68db27d2cbfb87b523fe233dc84ecd24062e35677",
        "sha256:47fc642bf6f427805daf52d6e52619fe0637648fe27017062d898f3bf891419d",
        "sha256:72772181d9bad1fa349792a1e7384dde56742c14af2b9986013eb94a240f005b",
        "sha256:8396be6e5ff844282d4d49b81631772f80dabae5658d432202faf101f5283b7c",
        "sha256:893c11064b347b24ecdd277a094413e1954f8a4e8cdaf7ffbe7ca3db87c103f0",
        "sha256:92a07dfd4d7c325dd177548c4134052d4842222833576c8391aab6f74038fc3f",
        "sha256:965c4c93e33e6984d8031f74e51227bd755376a9df6993774fd5b6fb3288b1f4",
        "sha256:9ab75e0b2820880ae24b7136c4d230383e07db014456a476d096591172569c38",
        "sha256:b0845e3bdd4aa18dc2f9b6fb78fbd3d9d371ad167fd6d1b7ad01c0a6cdad4fc6",
        "sha256:dca2d7203f0dfce8ea4b3efd668f8ea65cd2b35112638e488a4c12594015f67b",
        "sha256:ed686e5926929887e2c7ae0a700e32c6129abb798b4ad2b846e933de21508151",
        "sha256:ef6df7e14698e79c59c7ee7cf94cd62e5b869db369ed4b1b8f7b729ea825712a",
        "sha256:f898e5cc0a662a9e12bde6f931263a1bbd350cfb18e1d5336a12927851825bb6"
    ],
    "index": "pypi",
    "version": "==2.8.4"

When i try to install the packages from Pipfile and Pipfile.lock inside the docker i get the following error

I have read somewhere its the issue with gcc compiler not available in alpine. SO i also tried the Dockerfile with RUN apk --no-cache add build-base
FROM python:3.7.7-alpine3.11

ENV PYTHONUNBUFFERED 1

RUN apk --no-cache add build-base

Still same error
Since it gets installed on archlinux, why its not getting installed on my docker image.
I want to keep the same configuration (python 3.7 and the same Pipfile and Pipfile.lock) throughtout my various projects, how to get this done in docker

Just FYI
This ends up with error...
$ docker run -it python:3.7.7-alpine3.11 sh -c "pip install pyzmq==18.1.1"

But this works, that highly probably means all you need is apk add build-base zeromq-dev.
$ docker run -it python:3.7.7-alpine3.11 sh -c "apk add build-base zeromq-dev && pip install pyzmq==18.1.1"

thank you

@sakurai-youhei thanks for finding the answer!

authors: minrk sakurai-youhei sant527

repo: zeromq/pyzmq


text:
Do not change case of environment variable names

Configo makes all environment variable names uppercased. In some cases its unacceptable.

Hi @nuclon,
First of all, thanks for your PR :)
However, I would rather make it configurable. Uppercase was implemented because of this scenario:
Source 1: MY_PROP: 123
Source 2: {"my": { "prop": 456 } }
it should resolve to 456, but without the uppercasing it will be 123

I think in your case source 2 is resolved to "my_prop", which is correct as for me.
But I agree that it should be configurable. Environment variable CONFIGO_CASE_SENSITIVE sounds okay for you?

@nuclon CONFIGO_UPPERCASE_KEYS?

Ok, but I think it's reasonable to keep current behaviour as default, so this variable should be set to 0 to override. Not sure it is good idea

@nuclon absolutely.
Also could you please add an integration test? I feel sorry for a lack of the docs how to contribute, my bad, will document the process soon.
Thanks!

hey @nuclon,
The change looks almost perfect. Only one thing I would like to change - do not pass the variable around, but put it to the constant near to flatten and use it there.
Motivation - users could use their own CompositeSources, and the code will fail because it's a property per CompositeSource, but it shouldn't be.
Could you please change it? Thanks!

@nuclon, all good, thanks for your change! I'll release it soon :)

authors: bsideup nuclon

repo: zeroturnaround/configo


text:
Implement memoisation

Memoisation is a technique for making backtracking parser performance linear over the input length at the cost of higher memory consumption and more overhead for less exponential grammars. It would be nice to be able to support it.
See also #74

Closing because #290 implements a memoisation combinator. However, I still believe there's a need for improved behaviour: it can be unintuitive to figure out where best to place the combinator right now.

repo: zesterer/chumsky


text:
some packet loss and other weirdness when running weave in NAT'ed VMs

MongoDB host machine: dell R620 server (40 cores, 128GB ram) Ubuntu 14.04, Docker 1.4.1 uname -a:
Linux london 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Client machine: Ubuntu 14.04 VMWare Fusion 7.1.1 VM  (hardware version 11) (provisioned 12GB RAM + 8 cores), Docker 1.5.0 uname -a:
Linux ubuntu 3.13.0-46-generic #77-Ubuntu SMP Mon Mar 2 18:23:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Both are using the latest weave release:
sudo weave version
weave script 0.9.0
weave router 0.9.0
weave DNS 0.9.0
weave tools 0.9.0

The client is able to insert small documents into MongoDB over the weave network, but unable to insert large documents, output from sudo tcpdump -i weave on the client machine shows messages about "need to frag"
Sample server setup (server physical IP is 192.168.130.95):
sudo weave stop
sudo weave reset
sudo weave launch
sudo weave expose 10.0.9.101/24
sudo weave run 10.0.9.19/24 -d -t --name=mongo1 -p :27017:27017 mongo:3.0.0 mongod --storageEngine wiredTiger

Client setup:
sudo weave stop
sudo weave reset
sudo weave launch 192.168.130.95
sudo weave expose 10.0.9.102/24

Small write over weave network from client:
docker run -it --rm mongo:3.0.0 mongo "10.0.9.19:27017/test" --eval "db.somecollection.insert({ a: 'bc'});"
MongoDB shell version: 3.0.0-rc8
connecting to: 10.0.9.19:27017/test
WriteResult({ "nInserted" : 1 })

Large write over the weave network from client (hangs):
docker run -it --rm mongo:3.0.0 mongo "10.0.9.19:27017/test" --eval "db.somecollection.insert({ a: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'});"
MongoDB shell version: 3.0.0-rc8
connecting to: 10.0.9.19:27017/test

Same large write over the physical network (works fine):
docker run -it --rm mongo:3.0.0 mongo "192.168.130.95:27017/test" --eval "db.somecollection.insert({ a: 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'});"
MongoDB shell version: 3.0.0-rc8
connecting to: 192.168.130.95:27017/test
WriteResult({ "nInserted" : 1 })

Output from client sudo tcpdump -i weave
listening on weave, link-type EN10MB (Ethernet), capture size 65535 bytes
10:50:55.405307 ARP, Request who-has 10.0.9.19 tell 10.0.9.102, length 28
10:50:55.407795 ARP, Reply 10.0.9.19 is-at be:1b:9e:e8:24:8e (oui Unknown), length 28
10:50:55.407832 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [S], seq 164960364, win 29200, options [mss 1460,sackOK,TS val 20069498 ecr 0,nop,wscale 7], length 0
10:50:55.409096 IP 10.0.9.19.27017 > 172.17.0.24.47892: Flags [S.], seq 2370793204, ack 164960365, win 43690, options [mss 65495,sackOK,TS val 15892260 ecr 20069498,nop,wscale 7], length 0
10:50:55.409137 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], ack 2370793205, win 229, options [nop,nop,TS val 20069499 ecr 15892260], length 0
10:50:55.409334 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [P.], seq 0:60, ack 1, win 229, options [nop,nop,TS val 20069499 ecr 15892260], length 60
10:50:55.410562 IP 10.0.9.19.27017 > 172.17.0.24.47892: Flags [.], ack 61, win 342, options [nop,nop,TS val 15892261 ecr 20069499], length 0
10:50:55.411040 IP 10.0.9.19.27017 > 172.17.0.24.47892: Flags [P.], seq 1:80, ack 61, win 342, options [nop,nop,TS val 15892261 ecr 20069499], length 79
10:50:55.411074 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], ack 80, win 229, options [nop,nop,TS val 20069499 ecr 15892261], length 0
10:50:55.412388 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [P.], seq 60:122, ack 80, win 229, options [nop,nop,TS val 20069500 ecr 15892261], length 62
10:50:55.414015 IP 10.0.9.19.27017 > 172.17.0.24.47892: Flags [P.], seq 80:274, ack 123, win 342, options [nop,nop,TS val 15892261 ecr 20069500], length 194
10:50:55.416389 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [P.], seq 122:955, ack 274, win 237, options [nop,nop,TS val 20069501 ecr 15892261], length 833
10:50:55.416545 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36
10:50:55.416597 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], seq 122:622, ack 274, win 237, options [nop,nop,TS val 20069501 ecr 15892261], length 500
10:50:55.416602 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [P.], seq 622:955, ack 274, win 237, options [nop,nop,TS val 20069501 ecr 15892261], length 333
10:50:55.416633 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36
10:50:55.418500 IP 10.0.9.19.27017 > 172.17.0.24.47892: Flags [.], ack 123, win 350, options [nop,nop,TS val 15892263 ecr 20069500,nop,nop,sack 1 {623:956}], length 0
10:50:55.620941 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], seq 122:622, ack 274, win 237, options [nop,nop,TS val 20069552 ecr 15892263], length 500
10:50:55.621070 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36
10:50:56.028061 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], seq 122:622, ack 274, win 237, options [nop,nop,TS val 20069654 ecr 15892263], length 500
10:50:56.028278 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36
10:50:56.845105 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], seq 122:622, ack 274, win 237, options [nop,nop,TS val 20069858 ecr 15892263], length 500
10:50:56.845278 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36
10:50:58.479914 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], seq 122:622, ack 274, win 237, options [nop,nop,TS val 20070267 ecr 15892263], length 500
10:50:58.480052 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36
10:51:01.748340 IP 10.0.9.102.47892 > 10.0.9.19.27017: Flags [.], seq 122:622, ack 274, win 237, options [nop,nop,TS val 20071084 ecr 15892263], length 500
10:51:01.748508 IP 10.0.9.19 > 172.17.0.24: ICMP 10.0.9.19 unreachable - need to frag (mtu 490), length 36

Server docker logs weave output
weave 2015/03/18 14:41:05.486129 Command line options: map[wait:20 iface:ethwe name:7a:46:86:82:f5:92]
weave 2015/03/18 14:41:05.486195 Command line peers: []
weave 2015/03/18 14:41:06.487123 Communication between peers is unencrypted.
weave 2015/03/18 14:41:06.487661 Our name is 7a:46:86:82:f5:92
weave 2015/03/18 14:41:06.518895 Sniffing traffic on &{312 65535 ethwe d6:9e:64:84:78:ca up|broadcast|multicast}
weave 2015/03/18 14:41:06.518944 Discovered our MAC d6:9e:64:84:78:ca
weave 2015/03/18 14:41:06.614431 Discovered local MAC 32:ac:71:ec:0d:24
weave 2015/03/18 14:41:06.990526 Discovered local MAC 7a:46:86:82:f5:92
weave 2015/03/18 14:41:08.091084 ->[192.168.70.25:51570] connection accepted
weave 2015/03/18 14:41:08.092523 ->[192.168.70.25:51570] completed handshake with 7a:8a:74:ed:7c:64
weave 2015/03/18 14:41:08.092873 ->[7a:8a:74:ed:7c:64]: connection added
weave 2015/03/18 14:41:08.095855 ->[7a:8a:74:ed:7c:64]: connection fully established
weave 2015/03/18 14:41:08.095909 EMSGSIZE on send, expecting PMTU update (IP packet was 60064 bytes, payload was 60056 bytes)
weave 2015/03/18 14:41:08.095928 ->[7a:8a:74:ed:7c:64]: Effective PMTU set to 1438
weave 2015/03/18 14:41:08.099198 ->[7a:8a:74:ed:7c:64]: Effective PMTU verified at 1438
weave 2015/03/18 14:41:08.735417 Discovered remote MAC 7a:8a:74:ed:7c:64 at 7a:8a:74:ed:7c:64
weave 2015/03/18 14:41:22.966496 Discovered local MAC aa:df:0e:2b:63:f1
weave 2015/03/18 14:41:47.170530 ->[7a:8a:74:ed:7c:64]: connection shutting down due to error: EOF
weave 2015/03/18 14:41:47.170646 Removed unreachable Peer 7a:8a:74:ed:7c:64 (v5) (UID 17900529838048737883)
weave 2015/03/18 14:41:47.170690 ->[7a:8a:74:ed:7c:64]: connection deleted
weave 2015/03/18 14:41:49.646219 ->[192.168.70.25:51585] connection accepted
weave 2015/03/18 14:41:49.647336 ->[192.168.70.25:51585] completed handshake with 7a:67:a3:ad:5c:1e
weave 2015/03/18 14:41:49.647479 ->[7a:67:a3:ad:5c:1e]: connection added
weave 2015/03/18 14:41:49.657602 ->[7a:67:a3:ad:5c:1e]: connection fully established
weave 2015/03/18 14:41:49.657720 EMSGSIZE on send, expecting PMTU update (IP packet was 60064 bytes, payload was 60056 bytes)
weave 2015/03/18 14:41:49.657740 ->[7a:67:a3:ad:5c:1e]: Effective PMTU set to 1438
weave 2015/03/18 14:41:49.699386 ->[7a:67:a3:ad:5c:1e]: Effective PMTU verified at 1438
weave 2015/03/18 14:41:50.059363 Discovered remote MAC 86:98:15:dd:e4:ef at 7a:67:a3:ad:5c:1e
weave 2015/03/18 14:41:50.311578 Discovered remote MAC 7a:67:a3:ad:5c:1e at 7a:67:a3:ad:5c:1e
weave 2015/03/18 14:46:42.586514 Discovered local MAC be:1b:9e:e8:24:8e
weave 2015/03/18 14:51:06.527351 Expired MAC d6:9e:64:84:78:ca at 7a:46:86:82:f5:92
weave 2015/03/18 14:52:06.527639 Expired MAC 32:ac:71:ec:0d:24 at 7a:46:86:82:f5:92
weave 2015/03/18 14:52:06.527653 Expired MAC 7a:46:86:82:f5:92 at 7a:46:86:82:f5:92
weave 2015/03/18 14:57:06.528694 Expired MAC aa:df:0e:2b:63:f1 at 7a:46:86:82:f5:92

Client docker logs weave output
weave 2015/03/18 14:41:48.704557 Command line options: map[iface:ethwe name:7a:67:a3:ad:5c:1e wait:20]
weave 2015/03/18 14:41:48.704654 Command line peers: [192.168.130.95]
weave 2015/03/18 14:41:49.705890 Communication between peers is unencrypted.
weave 2015/03/18 14:41:49.706632 Our name is 7a:67:a3:ad:5c:1e
weave 2015/03/18 14:41:49.738252 Sniffing traffic on &{63 65535 ethwe d6:53:0f:08:71:f5 up|broadcast|multicast}
weave 2015/03/18 14:41:49.738280 Discovered our MAC d6:53:0f:08:71:f5
weave 2015/03/18 14:41:49.738620 ->[192.168.130.95:6783] attempting connection
weave 2015/03/18 14:41:49.741230 ->[192.168.130.95:6783] completed handshake with 7a:46:86:82:f5:92
weave 2015/03/18 14:41:49.741735 ->[7a:46:86:82:f5:92]: connection added
weave 2015/03/18 14:41:49.749514 ->[7a:46:86:82:f5:92]: connection fully established
weave 2015/03/18 14:41:49.749593 EMSGSIZE on send, expecting PMTU update (IP packet was 60064 bytes, payload was 60056 bytes)
weave 2015/03/18 14:41:49.749608 ->[7a:46:86:82:f5:92]: Effective PMTU set to 1438
weave 2015/03/18 14:41:49.751365 ->[7a:46:86:82:f5:92]: Effective PMTU verified at 1438
weave 2015/03/18 14:41:50.152645 Discovered local MAC 86:98:15:dd:e4:ef
weave 2015/03/18 14:41:50.404767 Discovered local MAC 7a:67:a3:ad:5c:1e
weave 2015/03/18 14:41:53.299722 Discovered remote MAC 7a:46:86:82:f5:92 at 7a:46:86:82:f5:92
weave 2015/03/18 14:41:55.169164 Discovered remote MAC aa:df:0e:2b:63:f1 at 7a:46:86:82:f5:92
weave 2015/03/18 14:42:32.933803 EMSGSIZE on send, expecting PMTU update (IP packet was 927 bytes, payload was 919 bytes)
weave 2015/03/18 14:42:32.933829 ->[7a:46:86:82:f5:92]: Effective PMTU set to 490
weave 2015/03/18 14:42:32.935258 ->[7a:46:86:82:f5:92]: Effective PMTU verified at 490
weave 2015/03/18 14:42:33.135939 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:33.136136 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:33.340041 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:33.748170 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:34.564017 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:36.200074 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:39.476301 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:46.021177 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:55.819624 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:55.819747 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:56.020357 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:56.428777 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:57.244153 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:58.880001 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:59.108630 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:42:59.516569 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:00.332423 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:01.968884 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:02.148167 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:05.236450 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:08.692113 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:09.100339 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:09.916668 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:11.552377 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:11.780143 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:14.820745 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:21.363987 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:24.868180 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:34.468679 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:43:51.044114 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:44:00.645135 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:46:42.686522 Discovered remote MAC be:1b:9e:e8:24:8e at 7a:46:86:82:f5:92
weave 2015/03/18 14:50:55.416498 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:50:55.416623 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:50:55.621037 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:50:56.028239 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:50:56.845245 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:50:58.480026 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:51:01.748469 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:51:08.292302 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:51:21.380276 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:51:47.524948 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:51:49.740238 Expired MAC d6:53:0f:08:71:f5 at 7a:67:a3:ad:5c:1e
weave 2015/03/18 14:52:39.876569 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:52:49.740522 Expired MAC 7a:46:86:82:f5:92 at 7a:46:86:82:f5:92
weave 2015/03/18 14:54:24.580261 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:56:24.644420 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 14:56:49.741403 Expired MAC aa:df:0e:2b:63:f1 at 7a:46:86:82:f5:92
weave 2015/03/18 14:58:24.965048 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 15:00:25.283968 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 15:02:25.604498 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490
weave 2015/03/18 15:04:25.924619 Sending ICMP 3,4 (10.0.9.19 -> 10.0.9.102): PMTU= 490

During some debugging it became apparent that weave may be having issues with MTU discovery.  Between the client and server exists a firewall whose MTU is set to 1460, but weave is somehow choosing to use an MTU of 490.
When the same test is run from a client/server on the same subnet, there is no issue.
Did a raw tcpdump on the weave interface on server and client in this manner:

reset weave everywhere
sudo weave launch (on server)
sudo tcpdump -i weave -w [capture file] (on server)
sudo weave expose 10.0.9.101/24 (on server)
[run mongo db container command]
sudo weave launch (on client)
sudo tcpdump -i weave -w [capture file] (on client)
sudo weave connect 192.168.130.95 (on client .. connects weave network to server)
[run mongo client command to insert large document]

this hangs on step 9 as described above
the raw tcpdump captures for the client/server can be accessed here: https://drive.google.com/folderview?id=0Bxv3aLTVo2eKfkhNSUI3Y2JWb1pnM1VTdVBxdzZoQzBUTTZqTzkwbEVPdldIcTFMNVk4dm8&usp=sharing

Thanks for the pcap files.
There's something conspicuously odd in the client-side capture right from the start.  Although its is sending packets to the server at 10.0.9.19, the resulting packets sent by the sent always come back with the source IP address set to 172.17.0.32.  This is probably an address on the docker network bridge.  I don't know for sure that this is connected to the MTU issue, but I'm surprised the client and server are able to communicate at all.
That strange address is probably due to a poor interaction between the NAT that docker does in order to allow containers on the docker bridge to talk to the outside world, and how 'weave expose' is implemented.
It's hard to say whether this is really a bug or not, because the way you are trying to use weave seems strange. As I understand it, you want to run a mongo server in a container, and a mongo client in a container, talking over the weave network.  Why not do this with 'weave run' on both ends?  You already do that on the server side (and there the 'weave expose' and the port mapping option after 'weave run' appear to be redundant).  But on the client you are starting a container with 'docker run' and relying on 'weave expose' to allow it to talk to the server container.  This is unusual: The intended use of 'weave expose' is to allow things that are not in containers to talk to a weave network.
Does this make sense?  Is there a particular reason why you are trying to use 'weave expose' in that way?  If you use 'weave run' for both the client and server containers, do the large requests work?

I'm using weave expose on the hosts just so that I can confirm that the weave network is working easily using ping
Also, regardless if the mongo client is run from within a container or not, it's the same end result.
However your notes on that 172.17.0.32 address made me think twice, this is the VMWare VM NAT...
I just re-tested everything with these scenarios with the VM networking setup:

NAT only (VMWare's default mode), gives a NAT ip of 172.x, this exhibits the problems above
NAT + bridged interfaces, gets a NAT of 172.x and a bridged IP on the physical 192.168.0.0/16 network, this exhibits the same problem as described
bridged interface only, IP on physical 192.168.0.0/16, this is working.

So you're right, it's a problem isolated to my VM network configuration (albeit not working with the default VMWare networking setup of NAT)
But that seems logical, how else would the server host know?

The odd thing here is that quite a lot appeared to work, all the way to doing small inserts in mongo. Ideally at least weave should have detected that something isn't well. Even better if it could have coped and got things to work.

Perhaps 'weave should have detected' could apply equally to #416/#417, which had the similar property that a few things would work even though the network configuration was unhelpful.

@alex-sherwin, thanks for your testing.  I'm still very puzzled by this!
Some thoughts:

There's no real evidence that the MTU issue and the 172.17.0.32 destination address shown in the client pcap file are related.  Both are strange, so its tempting to think they are connected, but they might not be.

It's also hard to explain why it would work with vmware in bridged mode but not in NAT mode.  Weave is certainly supposed to work with NAT in the way, and we do a lot of development on VMs with NAT (using various VM technologies).  I doubt that the vmware NAT is doing anything special.  If weave breaks when hosts are talking across a vmware NAT, that sounds like a bug in weave.

The NAT + bridged case is strange too. Why wouldn't weave simply use the bridged network interface, and work as it does in the pure bridged case?  And if so, how could the presence of a NATted interface cause problems if it's not being used?

The pcap files from the client and server sides are captured from the weave bridges.  So they should show exactly what appears at the two endpoints of the weave network.  The packets are being modified in transit, but the weave router does not modify packets in this way. Despite the surprising way 'weave expose' was being used on the client side, that can't be the cause of the 172.17.0.32 destination address, because it happens on the other side of the weave bridge.  The only thing I can imagine causing the destination address rewriting is an iptables rule on the client.  But nothing mentioned on this issue suggests that such a rule has been created.

@alex-sherwin, if you are able to do some more testing for us, could you record pcap files as before, but in the case with bridged networking, where the large queries across the weave network succeed?  That will indicate whether the destination address rewriting is really related to the problem or not.

@dpw, I've re-run the exact same test when my VM is running with only a bridged interface.  As mentioned before, this all works with no issues.  Here are the pcap files for this scenario:
https://drive.google.com/folderview?id=0Bxv3aLTVo2eKfkVLcGpHU0RfM2paSElkR2FzRzQxS2QtMm9oSUcwRXdqSFdfSU13UTFwMVE&usp=sharing

@dpw, I've re-run the exact same test when my VM is running with only a bridged interface. As mentioned before, this all works with no issues. Here are the pcap files for this scenario:

Thanks!
On the client side, the destination IP address is still being replaced, so that is not directly the cause of the problem.  The difference in this capture is that there are no ICMP "frag needed" packets generated by weave, suggesting that the effective PMTU on the underlying network was not restricted to 490 as before.
So I have a theory about what is happening when it hangs:  The modification of the destination address is not sufficient to prevent the TCP connection operating (surprisingly).  But it is sufficient to prevent the kernel registering the ICMP "frag needed".  When the client sends a packet that is too large (due to the large write), weave drops it and sends a "frag needed" back to the client side, which ignores it, and after a while TCP resends the packet, which gets dropped with another "frag needed", and so on, hence the hang.  This is visible in the client capture where it hanged.
As I mentioned before, the modification of the destination address suggests an iptables rule.  So @alex-sherwin, could you show the output of sudo iptables-save in the client VM? It doesn't matter what kind of networking the client has for this.  Also, are you using any firewall software within the client VM?
The PTMU of 490 when vmware NAT is used seems strangely low, but is allowed.

@dpw, here's the output of iptables-save in the VM
sudo iptables-save
# Generated by iptables-save v1.4.21 on Fri Mar 20 13:23:21 2015
*nat
:PREROUTING ACCEPT [6:468]
:INPUT ACCEPT [6:468]
:OUTPUT ACCEPT [1022:77063]
:POSTROUTING ACCEPT [922:71063]
:DOCKER - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -j WEAVE
-A WEAVE ! -s 10.0.1.0/24 -o weave -j MASQUERADE
-A WEAVE -s 10.0.1.0/24 ! -o weave -j MASQUERADE
COMMIT
# Completed on Fri Mar 20 13:23:21 2015
# Generated by iptables-save v1.4.21 on Fri Mar 20 13:23:21 2015
*filter
:INPUT ACCEPT [75982:28474773]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [81854:30034942]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j DROP
-A FORWARD -i weave -o weave -j ACCEPT
COMMIT
# Completed on Fri Mar 20 13:23:21 2015

There is no other firewall of any kind in use in the VM or the VM host

Thanks.  I don't see anything obviously wrong there.  One question though:
...
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
...

That suggests docker is using the 172.17.x.x address range on its internal bridge.  Earlier, you said that the same address range is being used by the VMware NAT interface.  Could you double-check that?

I must have looked too quickly.. tey are close but different subnets.  VMWare NAT is 172.16 and docker bridge is 172.17
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38 errors:0 dropped:0 overruns:0 frame:0
          TX packets:118 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2568 (2.5 KB)  TX bytes:11810 (11.8 KB)

eth0      Link encap:Ethernet  HWaddr 00:0c:29:39:54:c1
          inet addr:172.16.158.101  Bcast:172.16.158.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe39:54c1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:381947 errors:0 dropped:0 overruns:0 frame:0
          TX packets:221838 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:508118046 (508.1 MB)  TX bytes:19471935 (19.4 MB)

So, I had been trying to reproduce the 172.17.x.x destination addresses from your capture files, without success: tcpdump on the client-side weave bridge just showed 10.0.x.x addresses.  But I was using a Fedora 21 test VM I had lying around to act as the client side.
So today tried with a fresh Ubuntu 14.04 VM.  And it does show the 172.17.x.x destination addresses on the client side!  In exactly the same test scenarios that fedora didn't.  The iptables rules look the same, so I my guess is that a kernel change or patch in one of the distros is responsible, but I haven't looked into that yet (I have docker 1.5 in both cases).
I haven't been able to reproduce the hangs yet.  The PMTU between my weave routers is 1500, so I have to make the "large write" case even large to trigger a "frag needed".  But the write still seems to complete successfully.  I'll work more on this, constraining the MTU to 490 in case that matters.
One thing I didn't appreciate until late yesterday is that the 'weave expose' on the client side is significant.  Without the 'weave expose', if you use 'weave run' for the client container, what happens is that the mongo client tries the request before 'weave run' has had time to set up the ethwe interface, and so the mongo client's attempt to connect to the server simply fails.  With 'weave expose', the ethwe interface still isn't set up in time, so the connection attempt goes to the container's eth0, over the docker bridge onto the host (being masqueraded in the process), and then onto the weave bridge and across the weave network.  I don't think this convoluted path is really anticipated to work, and while it doesn't explain the destination address rewriting, it certainly seems like ample opportunity for odd things like that to happen.  And so, another approach to fixing the hangs might be to run the client under 'weave run' but with a delay so that it does use the ethwe interface.  But I'm going to try to reproduce the hangs before I resort to that.

another approach to fixing the hangs might be to run the client under 'weave run' but with a delay so that it does use the ethwe interface

Instead of a simple delay, the recommended approach is to loop (with a delay) until the interface comes up.

Instead of a simple delay, the recommended approach is to loop (with a delay) until the interface comes up.

Yes, but I imagine that could be hard to achieve in the context of a stripped-down mongodb container.  Whereas it looks like a simple delay is expressible in the mongo shell.

I have found the cause of the hang: although the minimum IPv4 MTU is 68, Linux has a minimum limit on the PMTU of 552: http://cschramm.blogspot.co.uk/2012/12/why-does-linux-enforce-minimum-mtu-of.html
You can change this limit.  So @alex-sherwin, if you do
echo 200 >/proc/sys/net/ipv4/route/min_pmtu

in your client VM, your hang should go away.
I'm still trying to get my head around why Linux sets this limit.
It also goes some way to explaining where the "Effective PMTU set to 490" comes from: effectiveOverhead() = 28 + 6 + (6 + 6 + 2) + 16 = 62.  552 - 62 = 490.
The whole destination address modification thing seems to be a red herring.  If I tcpdump on the weave router containers ethwe interface, I don't see that rewriting.  And if I tcpdump on the docker0 bridge, everything is fully NATted.  tcpdumping on the weave bridge seems to show things in a strange halfway state.

Re-tested with a lower /proc/sys/net/ipv4/route/min_pmtu setting and it works this way

Re-tested with a lower /proc/sys/net/ipv4/route/min_pmtu setting and it works this way

Well, that at least confirms that I'm reproducing your problem.
I'm still not sure why the PMTU on the underlying network would drop to the default Linux minimum of 552.  I wondered if it might be because ICMP "frag needed" packets were being blocked, but I spent some time trying to reproduce that without success.
Also, I'm still trying to think of anything useful weave can do to work around this problem, but I don't have any bright ideas so far.  The best might simply be to show the problem in the weave status output.

@dpw, I'd be fine with that.. the worst part is always not knowing.  I (and hope anyone else) wouldn't be using a VMWare NAT in this manner for anything other then development purposes

@dpw

The best might simply be to show the problem in the weave status output.

How hard is that?

How hard is that?

Detect when PMTU for a connection falls such that the overlay MTU is below 552, stash the fact in the connection, and gossip it around.  Sounds doable, but I don't claim to know the gossipy side of things well.
However, I doubt that is the right solution.  When the problem occurs, weave will still be broken, and in circumstances that are already fairly clear from docker logs weave (now we know what to look for).  So we don't gain much at all.
One important consideration that is still not clear is: what conditions cause the PTMU to fall to 552, and how likely do we think they are to occur? If it seems rare, maybe we don't need to do anything.  If it seems potentially common, we should think further about whether weave can reasonably work around the problem.
Last week I talked to a friend who has lots of experience with MTUs and what happens to them when ICMP packets are being blocked, and it sounds like there is a good chance that is the cause.  More experimentation is needed though.
I've been working on other things, but I can drop them to return to this if it's a priority.

authors: alex-sherwin bboreham dpw rade

repo: zettio/weave


text:
Incompatibility with ZF 2.4

This change forced me to require this dependency of this package directly in my composer.json file instead of letting Composer figure out the best version.
Looks like Composer was resolving to Zend Framework 2.4 and therefore was choosing the previous version of this package (1.0.3), which is the one with a compatibility issue with ZF 2.4. It couldn't choose version 1.0.4 because its incompatible with ZF 2.4. So kind of a catch-22, unless I'm missing something?
Like I said, the only way I could fix this is by explicitly requiring this package version 1.0.4, which made Composer uninstall ZF 2.4 and install version 2.3.7.
I didn't test this, but I suspect that a clean install of Apigility through composer will duplicate the issue I had at the time of this writing.

The change you reference was made because ZF 2.4 deprecates the EventManager's triggerUntil method. You could get around it by changing your error_reporting to add ^E_DEPRECATED ^E_USER_DEPRECATED, which removes the problem.
That said, we pinned current master of zf-rest to ~2.3.0 and tagged v1.0.4, because there's a bigger BC break in ZF 2.4 which makes current master incompatible: Zend\Mvc\Controller\AbstractRestfulController::deleteList() now has a required $data argument. In zf-rest, on current master, it's an optional argument, which means we simply cannot update to 2.4 yet, and have it compatible with both ZF 2.3 and ZF 2.4 simultaneously.
#64 updates zf-rest to work for 2.4, and this change will be released with v1.1 early next week.

We'll just wait until next week in that case, thanks!

authors: gabrielsomoza weierophinney

repo: zfcampus/zf-rest


text:
i has some ideas to write paper like eco, but i donot know how to create the ECO.yaml from origin caffe model?

i has some ideas to write paper like eco,
but i donot know how to create the ECO.yaml from origin caffe model?

Hi @dotsonliu , actually there is a tool provided in this link. But this tool may not works due to caffe version variation.

@zhang-can Hi zhang, I did some modifying  based on ECO.But lacking of kinetics dataset ,the dataset is seeming much bigger,so what can I do to verify my modifying?can you give some advice about this problem?Thank you !

authors: chen-ming2019 dotsonliu zhang-can

repo: zhang-can/ECO-pytorch