text:
8bit encoding behaves incorrectly

Hi there :wave: I've been working to update Hagrid to lettre 0.10.0-alpha3. It has mostly been a smooth experience, in particular the builder API is much nicer now than it was before. Good job on that :+1:
However :)
Describe the bug
The 8bit encoder indiscriminately inserts a linebreak \r\n every 998 bytes, regardless of existing newlines or charset. This can easily be observed in the code.
Adding line breaks without considering the encoding changes the content in subtle ways. In most cases it'll be a benign extra linebreak in the mail, but there are more problematic scenarios. In particular, adding a linebreak in the middle of a multi-byte character will break the entire encoding.
I noticed this when an email test case of mine was encoded as a raw_message rather than message by FileTransport. This was caused by a Japanese character ら placed on a 998 byte boundary, and its utf8 encoded form e3 82 89 becoming e3 82 0d 0a 89.
To Reproduce
    #[test]
    fn eight_bit_encode_long() {
        let mut c = EightBitCodec::new();

        assert_eq!(
            &String::from_utf8(c.encode(b"Hello, worldaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!")).unwrap(),
            "Hello, worldaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!"
        );
    }

Expected behavior
I'm not sure what the correct behavior here would be. The 8bit encoder as is cannot actually decide where to insert line breaks, since that depends on the encoded charset. I can think of three ways to improve the situation:

make the encoder a no-op, and leave 8bit line length up to the api user. this will lead to "malformed mails that probably still work" situations, but matches real world requirements most easily.
throw an error when a line longer than 998 characters is encountered. simple to implement, ensures that outgoing mail is spec-compliant, but also gets in the way of what users want to do?
make the encoder charset-sensitive, and insert line breaks before multi-byte characters. In practice this will only work for utf8, but that's probably the 99.9% case anyways.

Environment:

0.10.0-alpha.3

Thanks for testing the alpha version. Looks like as part of #495 we'll have to decide what's the best way of fixing this.

In the meantime you should be able to work around the issue by using quoted printable or base64

authors: Valodim paolobarbolini

repo: lettre/lettre


text:
Added Title Bar notification

-Webpage Title blinks when user training finishes
-Unsure if this works in all cases (jutsu training, etc)

Need to change the 995 to 1000, if number not changed larger training time will cause the notification to start too early on AFK webpage tabs

source: gharchive/pull-request

repo: levimeahan/shinobi-chronicles

license: MIT

license_type: permissive

license_source: github-api


text:
Flickering when typing

Describe the bug
Sometimes the vertical line dissapears and after few milliseconds comes back again.
Timestamps of video when this occurs:

00:11
00:15
00:19
00:26

https://user-images.githubusercontent.com/34353640/114551068-2f0cde80-9c63-11eb-8459-98d8731b672a.mp4
To Reproduce
Vim config: https://github.com/gbaranski/dotfiles/tree/master/dot_config/nvim
Initializing gitsigns: https://github.com/gbaranski/dotfiles/blob/369481c7461a48c6b1c6bdd53c47acba592d8497/dot_config/nvim/lua/plugins.lua#L30
Steps to reproduce the behavior:

Run my config
Open some file
Spam enter to make new lines

Observed output
Include the output of :lua require('gitsigns').debug_messages().
I've received:
E5108: Error executing lua [string ":lua"]:1: attempt to call field 'debug_messages' (a nil value)

Additional context
System: Arch Linux
Neovim version: NVIM v0.5.0-dev+1243-g4a36ec6da
Terminal emulator: Alacritty

Include the output of :lua require('gitsigns').debug_messages().
I've received:
E5108: Error executing lua [string ":lua"]:1: attempt to call field 'debug_messages' (a nil value)

You need to set debug_mode = true in your config. I need to update the issue template to explain this.

Also the MP4 file says it is corrupt.

Also the MP4 file says it is corrupt.

Here is fixed video(you must open fullscreen to see the git signs)
https://user-images.githubusercontent.com/34353640/114564595-bdd42800-9c70-11eb-8dec-836f96aa61e9.mp4

Include the output of :lua require('gitsigns').debug_messages().
I've received:
E5108: Error executing lua [string ":lua"]:1: attempt to call field 'debug_messages' (a nil value)

You need to set debug_mode = true in your config. I need to update the issue template to explain this.

It keeps saying this a lot of times
run_job: git --no-pager --git-dir=/home/gbaranski/code/houseflow/.git show :0:lighthouse/src/tcp_server.rs
update(3): updates: 327, jobs: 331

Try setting _refresh_staged_on_update = false. That'll stop it running git show on every edit.

_refresh_staged_on_update = false

Awesome, that works, thanks :D

Great! Just be wary that the config field is experimental and will be changed/removed at some point. You may also encounter some issue with it enable (hopefully not) but please raise an issue if you do.

Maybe try increasing update_debounce or maybe setting use_decoration_api=false

source: gharchive/issue


text:
[Feature request] Modification of URLs

I'm currently working on an RFC to use Lexbor for parsing and manipulating URLs in PHP (https://wiki.php.net/rfc/url_parsing_api), and I realized during implementation that Lexbor doesn't support the modification of the URL components once they are parsed. This feature would be very useful to have, so I think the best would be to implement upstream.
For comparison, ADA URL supports this feature: https://cpp-api.ada-url.com/#autotoc_md18

Hi @kocsismate
It's not hard to add. This weekend I plan to commit the changes related to URL parsing performance (to be faster than ADA).
After that I can add API for object manipulation.

@lexborisov That's very kind of you, thank you! I'll incorporate your changes ASAP.
Actually, I was also about to have a look at implementing the feature myself, but it would surely need quite some investigation first on my end, as I'm not that familiar with the internals yet. So I'm happy if you have capacity for the request.
P.S. I created a very basic benchmark comparing Lexbor, PHP's proprietary URL parsing implementation, and uriparser. The actual results can be seen on the lines marked with green: https://github.com/php/php-src/actions/runs/10550036054/job/29225746422?pr=14461#step:14:174 There is another benchmark above for a very basic URL where Lexbor is around 3.5 times slower than PHP's native implementation. As far as I remember from last year, ADA also had similar results (although with quite different implementation of the PHP binding).
I know the results are not directly comparable because of the different requirements due to the different specifications, but at least we have some numbers against which we can compare your changes. :)

@kocsismate
Preamble
I've optimized the URL parsing. It cost almost nothing to organize fast and slow paths in the code.
I can still optimize parsing of domain name, path and so on, but I don't see the point.
And I don't see the point in it because I don't understand what tasks need wild URL parsing speeds?
I compared the result with the ADA URL parser (I was told it was the fastest). Without resorting to wild optimizations in the lexbor project, I was completely satisfied with the result.
Below the text is a comparison of lexbor URL parser and ADA URL parser.
URL list

ID
URL

0
https://example.com

1
https://it.a.long.domain.name.just.made.up.com

2
https://пример.рф

3
https://пример.домена.который.юникод.тестирует.рф

4
https://example.com/my

5
https://example.com/my/home/in/this/world/is/the/best/place/I/love/and/always/come/back/to

6
https://example.com/Ensure_the_combination_of_parser/serializer/and/API_guarantee_idempotence/For_example/a_non_failure_result_of_a/parse_then_serialize/operation_will_not_change/with_any_further/parse_then_serialize/operations_applied_to_it/Similarly/manipulating_a_non_failure_result_through_the_API/will_not_change_from_applying_any_number/of_serialize_then_parse/operations_to_it

7
https://example.com/my/../home

8
https://example.com/my/../home/./in/./this/world/../is/../the/best/./place/../I/./love/../and/always/../come/back/../to

9
https://example.com/%D0%A4/%D0%AF/%D0%99

10
https://example.com/%D0%A2%D0%B0%D0%BA/%D0%B3%D0%BE%D0%B2%D0%BE%D1%80%D0%B8%D0%BB%D0%B0/%D0%B2/%D0%B8%D1%8E%D0%BB%D0%B5/1805/%D0%B3%D0%BE%D0%B4%D0%B0/%D0%B8%D0%B7%D0%B2%D0%B5%D1%81%D1%82%D0%BD%D0%B0%D1%8F/%D0%90%D0%BD%D0%BD%D0%B0/%D0%9F%D0%B0%D0%B2%D0%BB%D0%BE%D0%B2%D0%BD%D0%B0/%D0%A8%D0%B5%D1%80%D0%B5%D1%80%2C/%D1%84%D1%80%D0%B5%D0%B9%D0%BB%D0%B8%D0%BD%D0%B0/%D0%B8/%D0%BF%D1%80%D0%B8%D0%B1%D0%BB%D0%B8%D0%B6%D0%B5%D0%BD%D0%BD%D0%B0%D1%8F/%D0%B8%D0%BC%D0%BF%D0%B5%D1%80%D0%B0%D1%82%D1%80%D0%B8%D1%86%D1%8B/%D0%9C%D0%B0%D1%80%D0%B8%D0%B8/%D0%A4%D0%B5%D0%BE%D0%B4%D0%BE%D1%80%D0%BE%D0%B2%D0%BD%D1%8B%2C/%D0%B2%D1%81%D1%82%D1%80%D0%B5%D1%87%D0%B0%D1%8F/%D0%B2%D0%B0%D0%B6%D0%BD%D0%BE%D0%B3%D0%BE/%D0%B8/%D1%87%D0%B8%D0%BD%D0%BE%D0%B2%D0%BD%D0%BE%D0%B3%D0%BE/%D0%BA%D0%BD%D1%8F%D0%B7%D1%8F/%D0%92%D0%B0%D1%81%D0%B8%D0%BB%D0%B8%D1%8F%2C/%D0%BF%D0%B5%D1%80%D0%B2%D0%BE%D0%B3%D0%BE/%D0%BF%D1%80%D0%B8%D0%B5%D1%85%D0%B0%D0%B2%D1%88%D0%B5%D0%B3%D0%BE/%D0%BD%D0%B0/%D0%B5%D0%B5/%D0%B2%D0%B5%D1%87%D0%B5%D1%80

11
https://example.com/мой/дом/моя/крепость

12
https://example.com/Так/говорила/в/июле/1805/года/известная/Анна/Павловна/Шерер/фрейлина/и/приближенная/императрицы/Марии/Феодоровны/встречая/важного/и/чиновного/князя/Василия/первого/приехавшего/на/ее/вечер

13
https://example.com/my/дом/is/my/крепость

Result
Parsing repetitions for each URL: 100000
Number of repetitions to identify average and median values: 10

URL ID
Lexbor Average
ADA Average
Lexbor Median
ADA Median

0
0.016228
0.014519
0.014263
0.012096

1
0.021192
0.011828
0.021203
0.011850

2
0.032634
0.060687
0.032390
0.060640

3
0.105112
0.24110
0.105268
0.241082

4
0.014445
0.013445
0.014411
0.013428

5
0.018582
0.014836
0.018538
0.014842

6
0.032811
0.027613
0.032939
0.027588

7
0.015834
0.016519
0.015749
0.016607

8
0.031256
0.043055
0.031213
0.043161

9
0.016267
0.018239
0.016279
0.018159

10
0.116674
0.107119
0.117256
0.106831

11
0.025567
0.056733
0.025793
0.056756

12
0.122021
0.296652
0.123431
0.296686

13
0.024461
0.048586
0.024380
0.048411

Hardware
Apple M1 Pro chip, 10-core CPU with 8 performance cores and 2 efficiency cores, 16GB.
OS 14.6.1 (23G93)
Compilation options
For Lexbor and Ada
-O2 -Wall -pedantic -pipe -fPIC

P.S.: I could get steamed up and completely tear apart ADA Parser, but I don't see the point, it's not a place where you need to spend a lot of effort on micro-optimizations.
I'll try these tests on an Intel processor a little later.

@kocsismate
Also, if I'm not mistaken, ADA doesn't validate URLs, i.e. it doesn't log errors. And this imposes a lot of restrictions on code optimization, you have to do a lot of different checks and logging of parsing.
I mean this: https://url.spec.whatwg.org/#writing
For an example, see path state paragraph 2.1 and 2.2. And there are enough such checks in the code, they make optimization very difficult.
Lexbor parser always validates URL data and saves errors to the log.

@lexborisov I upgraded the bundled Lexbor, but I found some BC breaks and some other problems:

-Werror=int-conversion was triggered: https://github.com/php/php-src/actions/runs/11210426284/job/31157304027?pr=14461#step:10:8
Member access within null pointer error was triggered, so I had to conditionally destroy the idna member in the parser struct: I changed lxb_unicode_idna_destroy(&parser->idna, false); to if (parser->idna != NULL) {lxb_unicode_idna_destroy(&parser->idna, false);} (sorry, somehow I could not link this diff on GitHub)
There are some port related test failures which I had to fix with https://github.com/php/php-src/pull/14461/commits/da184a6df579ce0b9791af70a24bc5b49f6ac947#diff-618caf650d8fa6adab7f8c7f92226c7708633f760d82b85aa58e43383c332f96R123. Apparently, the has_port field does not have an effect and I explicitly have to check for the 0 port value. I don't know if it's intentional, but I wanted to point this out.
Some tests now fail due to the path component being different, but I guess my changes are to blame (https://github.com/php/php-src/pull/14461/commits/da184a6df579ce0b9791af70a24bc5b49f6ac947#diff-618caf650d8fa6adab7f8c7f92226c7708633f760d82b85aa58e43383c332f96L136). I'll have another look at those again...

Regarding performance: Lexbor is much faster now indeed. Using GitHub runners for benchmarking surely won't provide reliable data, but at least we have an estimation

for example.com, Lexbor used to finish in 0.000931104. Now it's 0.000327715.
for https://username:password@www.google.com:8080/pathname1/pathname2/pathname3?query=true#hash-exists, Lexbor used to finish in 0.001555342. Now it's 0.000800032.

Hi @kocsismate

Ok, I'll fix it.
Here the correct solution is: parser->idna = lxb_unicode_idna_destroy(parser->idna, true); Inside the lxb_unicode_idna_destroy() function there is a check for NULL. I'll make a patch for it now.
I'll look at it, all right.
Yes, now a patch is just a string that is already ready to be “consumed”, it contains a “/” at the beginning.

As for performance, I can still increase it significantly, but I just don't see the point. Well, parsing URLs is not worth such optimization efforts.

@lexborisov Just wanted to say that the first point is not in your code. This is because we have a modified structure in our bundled fork and our modified Python code generation script wasn't re-run. So you can ignore the first point.

@kocsismate
I think you haven't updated the code for the Unicode module.  Do you have a NULL check in the lxb_unicode_idna_destroy() function?

Oh yes, you are right! I didn't have it until now... Thanks for the idea!

@kocsismate
A bit delayed, but the functions to change URLs have been added. Please, see https://github.com/lexbor/lexbor/blob/master/source/lexbor/url/url.h#L339

@lexborisov Thank you, this is awesome news! I'll have a look ASAP.

I tried the new functions, and they work well for me based on the tests I have! Thank you again very much!

authors: kocsismate lexborisov nielsdos

repo: lexbor/lexbor


text:
Change std::snprintf to _snprintf for VC11

Building on VC11 in multi-byte character mode failed with an error about std:: not containing snprintf. Changing this line to match the unicode version a few lines above it fixed the issue.

This issue was moved to nanodbc/nanodbc#33

repo: lexicalunit/nanodbc

license: mit

license_source: bigquery


text:
Replaced the "os" (ubuntu-18.04) for CI build, test-suite with a newe…

…r one (ubuntu-22.04)
Signed-off-by: Taras Drozdovskyi t.drozdovsky@samsung.com
Description
Due to the ubuntu 18.04 is deprecated, it was replaced with ubuntu 22.04 for CI.
Fixes #612
Type of change

[x] CI system update

Test Configuration:

OS type & version: Ubuntu 20.04
Hardware: x86-64
Toolchain: Docker v20.10 & Go v1.16
Edge Orchestration Release: v1.1.16

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

What about updating ubuntu and ubuntu_multistage in defdockerfiles with 22.04?

What about updating ubuntu and ubuntu_multistage in defdockerfiles with 22.04?

If it is necessary to update the base OS then I suggest using 20.04

If it is necessary to update the base OS then I suggest using 20.04

20.04 is also fine!

It looks like there are a lot of things to check to include 22.04.

Yes, you are absolutely right, additional study is needed for the 22.04 version. That's why I didn't want to use it as a base image. I hope soon I will be able to study 22.04 in more detail and adjust it for successful check of the our project

Let's use only 20.04 now. Please revise the commit message as well.

Let's use only 20.04 now. Please revise the commit message as well.

Done!

repo: lf-edge/edge-home-orchestration-go


text:
remove appimg vs baseosobj from contenttree

volumemgr handles all contenttree downloads. There really isn't a difference between one kind of contenttree and the next, and the distinction makes it hard to simplify baseos improvements.
This is a first attempt at eliminating that distinction, having volumemgr listen to contenttreeconfig of any type, and publish plain contenttreestatus, and have the other publishers and subscribers - baseosmgr and zedagent - simply use them as is.
cc @rvs @eriknordmark @zed-rishabh

I have not had the chance to put this through any paces yet. I want CI to do its thing, but more importantly, I want the people cc'ed above to have a first look at it, see what is obviously right or wrong.

Does anyone understand this failure? It is in a vendored package that isn't affected by this PR (nor is any other vendored package).

volumemgr has two publishers (pubContentTreeStatus, pubBaseOsContentTreeStatus) and two subscribers (subContentTreeConfig, subBaseOsContentTreeConfig) for handling content trees. volumemgr is not using pubBaseOsContentTreeStatus after your changes

Correct, that was the intent (although I forgot to remove pubBaseOsContentTreeStatus, and will fix it). Now it just has one thing to publish, a ContentTreeStatus, and one thing for others to subscribe to. No AgentScope.

baseosmgr is subscribing to volumemgr for all types of content trees that means baseosmgr will receive ContentTreeStatus of app images also. How are we planning to prevent that? (I am not sure will it break the existing baseosmgr workflow)

Correct. It receives all, but since it only responds to those for which it cares by UUID, it should ignore the rest. "Should" being the operative word, and I am hoping that you fine folks will see if I got it right.

IMO, we need to move the logic of publishing ContentTreeConfig from the baseosmgr to the zedagent. baseosmgr will publish the ContentTreeRefConfig to the volumemgr. With this approach, I think we will be able to clean up the types from the configs.

100%. This would be a first step. Once we have this in place, then volumemgr is much simplified and ready to handle it.

@deitch Thanks for the clarification. It makes sense to me.
I think we need some kind of flag in the content tree config in API between EVE and controller. It will help the controller differentiating app content trees from the base content tree while displaying the content tree status on the device status page on UI. After these changes, zedagent will receive the base content tree status as well from the volumemgr and it will be reported in the info message to the controller by EVE.

@deitch Thanks for the clarification. It makes sense to me.
I think we need some kind of flag in the content tree config in API between EVE and controller. It will help the controller differentiating app content trees from the base content tree while displaying the content tree status on the device status page on UI. After these changes, zedagent will also receive the base content tree status from the volumemgr and it will be reported in the info message to the controller by EVE.

Hey @zed-rishabh -- so you're saying we need some hint to the UI to be able to filer things based on whether it needs to be visible on screen or not? Perhaps the real solution here is that the part of the UI that renders status of baseOS should actually start feeding off of a different source -- mainly regular Volume/ContentTree stats, no?

Look like log test in Eden failed -- not sure if this is intermittent or not -- but surely this needs to be take a look at.

@deitch Thanks for the clarification. It makes sense to me.

I am glad it does to someone @zed-rishabh ; I was not totally sure it did! 😁

I think we need some kind of flag in the content tree config in API between EVE and controller. It will help the controller differentiating app content trees from the base content tree while displaying the content tree status on the device status page on UI. After these changes, zedagent will also receive the base content tree status from the volumemgr and it will be reported in the info message to the controller by EVE

Do we? I think that we need, when looking at a list of apps, to be able to know their status; same for baseos. In other words, going from one of those (app or base) to contenttree. But do we need it the other way? A tree is a tree is a tree?

I am not sure of it either. @giggsoff do you know the meaning of the "eden log test" failure in this PR? Is it something intermittent, or did I actually break something?

Hi, @deitch . It means, that we cannot get log from EVE for our action in the test (ssh into EVE).
What I see on my PC:
cat /var/persist/log/reboot-reason.log
Reboot from agent volumemgr[1455] in partition IMGA EVE version 0.0.0-volumemgr-all-contenttree-ae7a0a54-kvm-amd64 at 2020-12-17T09:00:08.347731466Z: fatal: agent volumemgr[1455]: Can not publish /var/run/volumemgr/ContentTreeStatus.sock since it it already used

So, I think, the problem in the same TopicType here and here.

Ah, the lingering publication that @zed-rishabh found. Thanks for finding it @giggsoff . I am updating now and will run tests again.

Updated. Let us see what it does.

Anyone else need to review this before @rvs merges it in?
My understanding is that CI has gotten good enough that I don't have to spend real time doing my manual tests?

@deitch please take a look at artifact from integration test. I see AllowNonFreePort: Unknown ObjType inside.

Thanks @giggsoff ; Adarsh found the same issue. I see what it is, finding a resolution.

Short summary. I am awaiting feedback from some others.
Basically, it looks up policy on whether or not to allow NonFreePort based on whether it is a baseos contenttree or appimg contenttree. But the essence of this PR is that there is no difference when it comes to download and verification: contenttree is a contenttree is a contenttree. I would like to just have a single policy setting.

Per the suggestions from @zed-rishabh and @rvs , I have updated this to eliminate the distinction between various objTypes when checking AllowNonFreePort. It basically just uses the policy from "network.allow.wwan.app.download" for all content tree, both app and baseos.
We probably should rename it to something more universal, but let's do that in a separate PR, since it will affect controller as well.
Ready for review.

Looks like it passes now.

We can also remove ObjType field from the ContentTreeConfig and ContentTreeStatus structures defined in types/contenttreetypes.go

Good catch there as well. Done.

Can we get this merged in?

Nope -- 'cuz Ring0 is failing again.

Moving this forward.
Now, baseosmgr successfully transmits the ContentTreeConfig to volumemgr, which indeed downloads it all and loads it into containerd. It then updates a ContentTreeStatus, which baseosmgr does find and respond to.
The problem is that when baseosmgr receives the status update, it looks for anything to process based not on the UUID, but on the content tree hash.
This creates 2 problems:

When baseosmgr first creates the config, it often doesn't even know the hash, especially if it is an OCI image, as we latch the hash from volumemgr via downloader.
Even if it does know the hash, that same image could be used in multiple places. E.g. if we have the same image used for baseosmgr and to run in an ECO as a VM. Pulling it down again (or reusing it), thus creating a new ContentTreeStatus with different UUID but same hash, should not cause baseosmgr to do anything again.

As an example, here is the BaseOsConfig created by zedagent:
{
  "UUIDandVersion": {
    "UUID": "a18ac2eb-f53f-4ab0-b925-81b0142044e9",
    "Version": "4"
  },
  "BaseOsVersion": "5.17.0-kvm",
  "ContentTreeConfigList": [
    {
      "ContentID": "a18ac2eb-f53f-4ab0-b925-81b0142044e9",
      "DatastoreID": "470d55a8-89da-4bb8-b50f-7dd8737d0eba",
      "RelativeURL": "lfedge/eve:5.17.0-kvm",
      "Format": 8,
      "ContentSha256": "",
      "MaxDownloadSize": 0,
      "GenerationCounter": 0,
      "DisplayName": "lfedge/eve:5.17.0-kvm"
    }
  ],
  "RetryCount": 0,
  "Activate": true
}

Notice that the ContentSha256 is blank.
Now the ContentTreeConfig created by baseosmgr due to the above config:
{
  "ContentID": "a18ac2eb-f53f-4ab0-b925-81b0142044e9",
  "DatastoreID": "470d55a8-89da-4bb8-b50f-7dd8737d0eba",
  "RelativeURL": "lfedge/eve:5.17.0-kvm",
  "Format": 8,
  "ContentSha256": "",
  "MaxDownloadSize": 0,
  "GenerationCounter": 0,
  "DisplayName": "lfedge/eve:5.17.0-kvm"
}

Same UUID as in the baseosconfig, content sha still is blank.
Now here is the ContentTreeStatus created by volumemgr in response to the above:
{
  "ContentID": "a18ac2eb-f53f-4ab0-b925-81b0142044e9",
  "DatastoreID": "470d55a8-89da-4bb8-b50f-7dd8737d0eba",
  "DatastoreType": "DsContainerRegistry",
  "RelativeURL": "lfedge/eve@sha256:3aa3a0f850103103ff2e60e2323fd38b758477d17daa9d7ea57f3deb4ccbb8ed",
  "Format": 8,
  "ContentSha256": "3aa3a0f850103103ff2e60e2323fd38b758477d17daa9d7ea57f3deb4ccbb8ed",
  "MaxDownloadSize": 0,
  "GenerationCounter": 0,
  "DisplayName": "lfedge/eve:5.17.0-kvm",
  "HasResolverRef": false,
  "State": 108,
  "TotalSize": 283379869,
  "CurrentSize": 283379869,
  "Progress": 100,
  "FileLocation": "",
  "NameIsURL": false,
  "Blobs": [
    "3aa3a0f850103103ff2e60e2323fd38b758477d17daa9d7ea57f3deb4ccbb8ed",
    "99c8e54e2fa6fe8b72986f189c369947a7a3e12d5fe8a85775e9ab50fac56b9e",
    "a0900be7ab37267ff0c841779d0c74c02faf6b29a71342a25337a4ac45ca98c2",
    "38c988dff6ee7b460fba8879eb1c223eee78b8500501d797b4d90b648827af00",
    "7ad2a41413fc25c73feba105bcd1e8a95c43018a3c5e81f8c3578bc7acdc4436",
    "894ecc33944315ceeecf4182ebb799f4e9fa5c37cc08eb20bc3c626cc4ddde4c",
    "1f6f442fb5ebf1c97cbbd445968094192a10567117f4fc57fd8ec0646a513d78",
    "c50d002e168b72acd691b324a6e1de95b0606374d22a3b6932739091ba1466d1",
    "c32fc83ffa1a4234babc50d5fb001dfb65da7bb4d07b678d5772b3b8c5e411d2",
    "b75fa0789eb2395c7774b95a9b68b486a34b4ff899855ee1e50256fa410c4c0e"
  ],
  "ErrorSourceType": "",
  "Error": "",
  "ErrorTime": "0001-01-01T00:00:00Z"
}

The content ID is identical (as it should be), and the hash is set (as volumemgr latched it).
Along comes baseosmgr and tries to match that ContentTreeStatus to the source ContentTreeConfig using hash, and it fails to find it.
I propose that we change baseosmgr to find content based on its UUID. The very response from volumemgr - ContentTreeStatus - not only gives the status, but also gives the content sha for that tree.

Updated. It now updates based on uuid, rather than hash. Testing.

And rebased.

Update: baseosmgr now correctly catches the update from volumemgr. It fails to install, though, because of a tag/name mismatch.

8ac34e67-81fd-4bd5-9d11-7be90823be5b-lfedge/eve:5.17.0-kvm - this is what baseosmgr looks for
8ac34e67-81fd-4bd5-9d11-7be90823be5b-lfedge/eve@sha256:3aa3a0f850103103ff2e60e2323fd38b758477d17daa9d7ea57f3deb4ccbb8ed - this is what is in containerd

The first is because it tries to build the reference based on ContentTreeStatus that is embedded in BaseOsStatus.
Digging through it, I cannot figure out where the right place to apply the latch to BaseOsStatus is, and if we do so, how it won't mess up the check that the RelativeURL is the same.

Hey @deitch -- I see new commits popping up here -- any further comments? Should we proceed with testing this?

Hey @deitch -- I see new commits popping up here -- any further comments? Should we proceed with testing this?

I am working on it right now @rvs . Some odd issues, but getting there. I will comment here when my local tests are clean.

I am working on it right now @rvs . Some odd issues, but getting there. I will comment here when my local tests are clean.

This is at the point where all of the interim steps are in place, with baseosmgr doing the right thing.
However, it calls the library to pull the image from containerd expecting a particular format, i.e. the one defined in edge-containers.
The image we are using for these, e.g. oci://docker.io/lfedge/eve:5.17.0-kvm, has a different internal format. We can support it, if we know what it is.
I think @rvs best knows what the layout is inside there. Is it layers with annotations? Or are they just files in certain predefined locations on disk? Let's describe the format here, and then we should be able to support it.

This is at the point where all of the interim steps are in place, with baseosmgr doing the right thing.
However, it calls the library to pull the image from containerd expecting a particular format, i.e. the one defined in edge-containers.
The image we are using for these, e.g. oci://docker.io/lfedge/eve:5.17.0-kvm, has a different internal format. We can support it, if we know what it is.
I think @rvs best knows what the layout is inside there. Is it layers with annotations? Or are they just files in certain predefined locations on disk? Let's describe the format here, and then we should be able to support it.

The image we are using for these, e.g. oci://docker.io/lfedge/eve:5.17.0-kvm, has a different internal format. We can support it, if we know what it is.

The high order bit here @deitch is that we need to support a format of a container image where rootfs is just a file inside of the image (and not a special kind of a layer). In fact I'd argue that it would be useful to support it to run user-defined VMs as well.
If we agree to that in principle -- we can discuss the location of the files that need to be extracted and used.
Please let me know what you think @deitch

I think @rvs best knows what the layout is inside there. Is it layers with annotations? Or are they just files in certain predefined locations on disk? Let's describe the format here, and then we should be able to support it.

Currently the file we need to extract from the container image is bits/rootfs.img however, as mentioned above, we are free to change that pathname inside the container.
That said, all official EVE release images we've published so far under lfedge/eve:<version> follow the above convention.

The image we are using for these, e.g. oci://docker.io/lfedge/eve:5.17.0-kvm, has a different internal format. We can support it, if we know what it is.

The high order bit here @deitch is that we need to support a format of a container image where rootfs is just a file inside of the image (and not a special kind of a layer). In fact I'd argue that it would be useful to support it to run user-defined VMs as well.
If we agree to that in principle -- we can discuss the location of the files that need to be extracted and used.
Please let me know what you think @deitch

I think @rvs best knows what the layout is inside there. Is it layers with annotations? Or are they just files in certain predefined locations on disk? Let's describe the format here, and then we should be able to support it.

Currently the file we need to extract from the container image is bits/rootfs.img however, as mentioned above, we are free to change that pathname inside the container.
That said, all official EVE release images we've published so far under lfedge/eve:<version> follow the above convention.

Oh this will fail as is. It should work for all non-OCI images (as it did before), but OCI will get most of the way there and then error out. We need to get the format working before that will work too.

Oh this will fail as is. It should work for all non-OCI images (as it did before), but OCI will get most of the way there and then error out. We need to get the format working before that will work too.

Looking at the edge-containers README it explains the format it uses for reading the images and how to find the content.
Basically, it looks at the following in descending order:

Special media-type, to indicate what is the kernel or rootfs. This obviously does not happen (at least today) with normal docker build.
"Normal" docker layer media-types, but have annotations, that indicate which file contains which elements (rootfs, kernel, initrd, etc.)

The image needs to contain some form of flags (i.e. annotations on the manifest) to indicate where to find the various parts.
This means that if the following both are true:

we want continue to use a regular OCI image, so that anything can read and extract it, not only something compliant with the OCI Artifacts spec
we want to place the files in those places, e.g. bits/rootfs.img for the rootfs

Then we should build those images with the extra annotations, so that eve can know where to find what it needs.
Actually, if it had the annotations, I think it would work already (but won't know until we test)
Where do we build this image? It is pretty easy to add the right annotations to it, whether using libraries or command-line tools.

Looking at the edge-containers README it explains the format it uses for reading the images and how to find the content.
Basically, it looks at the following in descending order:

Special media-type, to indicate what is the kernel or rootfs. This obviously does not happen (at least today) with normal docker build.
"Normal" docker layer media-types, but have annotations, that indicate which file contains which elements (rootfs, kernel, initrd, etc.)

The image needs to contain some form of flags (i.e. annotations on the manifest) to indicate where to find the various parts.
This means that if the following both are true:

we want continue to use a regular OCI image, so that anything can read and extract it, not only something compliant with the OCI Artifacts spec
we want to place the files in those places, e.g. bits/rootfs.img for the rootfs

Then we should build those images with the extra annotations, so that eve can know where to find what it needs.
Actually, if it had the annotations, I think it would work already (but won't know until we test)
Where do we build this image? It is pretty easy to add the right annotations to it, whether using libraries or command-line tools.

@rvs responded that the image is built using this Dockerfile and invoked by this Makefile

@rvs responded that the image is built using this Dockerfile and invoked by this Makefile

Will do @eriknordmark . As it is this still does not work, as it is blocked by some underlying issues relating to pulling artifacts. I am working it through. I will get the changes that you requested in, but we cannot merge anyways until the underlying piece is resolved.

Will do @eriknordmark . As it is this still does not work, as it is blocked by some underlying issues relating to pulling artifacts. I am working it through. I will get the changes that you requested in, but we cannot merge anyways until the underlying piece is resolved.

Hey @deitch -- is this getting attention as part of your latest work on baseOS image?

All of a piece. It's so tantalizingly close I can taste it.

Addressed the two concerns @eriknordmark raised today and pushed out. Now going back to look at the older ones.

Updated, addressing all of @eriknordmark 's comments (I hope I got them all correctly).
@rvs let's get this through the full test suite. Once it passes, we can merge it in. The modifications to eve for baseos-in-oci, then, will be very small and straightforward: new version of edge-containers library and a minor change or two.

This is strange. The failure is in unit tests of workers, but we didn't change any code in workers (other than fixing a log message typo, and that isn't where it is failing). 🤷♂

CI all green. Let's merge it in?

Thanks @eriknordmark
Working the last bits for baseos in OCI now. That will be a new (and much simpler) PR

authors: deitch giggsoff rvs zed-rishabh

repo: lf-edge/eve


text:
rework deferred sending to handle on success and use priority

Seems that proposed infoQueue in #2252 has quite same functionality as existence deferred logic and as discussed we want to send all messages when they comes, without scheduling them into queue.
In this PR I add priority functions to deferred logic to sort items by type (with TypePriorityCheck functions) and by time of add/modify of item (fifo).
Instead of direct sending I use SetDeferred and call HandleDeferred for sending one object from deferred (which may have more priority to send).
Signed-off-by: Petr Fedchenkov giggsoff@gmail.com

@petr-zededa Once this has run through the usual tests we probably want to do some more significant outage injection i.e., let an edge node boot without network connectivity (or it looses connectivity after the first get of its config), and let it sit there for minutes while it queues info messages for the configured app instances, volumes, etc, then make the network packets flow again to make sure they all go out without problems.
I don't know if we can easily create a test in eden to do this (create volumes, app instances, network instances etc, then reboot and drop all network packets for the next 5 minutes). Thoughts?

authors: eriknordmark giggsoff


text:
record failures and success from the try code

This seems to be missing, which confused me when I was looking at /persist/status/nim/DevicePortConfigList/global.json after I unplugged eth0.
Failures don't get updated until we start testing the alternatives.

This PR is broken. It doesn't record the failures in the current index in DevicePortConfigList. Withdrawing it for now.


text:
Implement configurators for VLANs and Bonds

This is the last missing piece needed to enable support for VLANs
and LAGs for EVE management and local network instances.
This functionality is described and designed in this document:
https://wiki.lfedge.org/display/EVE/EVE+VLAN+support+-+create+VLAN+and+bond+interfaces
A part of this commit is also a fix for Switch network VLANs,
where we forgot to configure the network uplink port as trunk, therefore
all tagged packets coming in and out of it would be dropped by bridge VLAN
filtering. As a result, VLANs were working correctly only for air-gapped
switch networks.
Test for VLANs and LAGs: https://github.com/lf-edge/eden/pull/753
Signed-off-by: Milan Lenco milan@zededa.com

If we have a vlan I assume we create vlan10 on top of eth0 and then we do a kvlan10 rename and create a vlan10 bridge. Is that correct?
In such a case do we also allow creating a network instance directly on eth0 (for untagged frames?) or not. That would end up causing a eth0->keth0 rename etc.

Actually it was not planned to support this case (design document is here), at least not for now. Mostly to simplify (both EVE and the controller) we decided that it would not be allowed to attach network instance to a port which has higher-layer adapters (see this diagram).
So if eth has VLANs then it will not be possible to use it directly anymore. Do you think that this is a too much of a limitations for the future users of EVE?

I don't think bonds cause the same potential complexity since a child of a bond is exclusive hence can not be used directly AFAIK. But bond0 could be used by both a local network instance and a switch network instance (just like vlan10 can be) which means we'd need a kbond0 rename AFAIU

Yes, renaming is done and bridge is created for bond IF there is nothing defined on top of it (like vlans).


text:
forgot to test agentsettings

This clearly needed end-to-end testing.
With this fix it works to set
debug.zedrouter.loglevel and
debug.zedrouter.remote.loglevel
However, @archishou the code hits at the new syntax should be
agent.zedrouter.debug.loglevel
and two issues:

That is not documented in docs/CONFIG-PROPERTIES.md
It fails with this in the logs:
{"file":"/pillar/types/global.go:330","func":"github.com/lf-edge/eve/pkg/pillar/types.(*ConfigItemSpecMap).parseAgentItem","level":"error","msg":"***parseAgentItem: ERROR: Unable to find agent name for per-agent setting. Key: debug.zedrouter.loglevel","pid":1986,"source":"zedagent","time":"2020-04-02T12:57:19.967666351Z"}

@kalyan-nidumolu can you file a bug story to fix that new syntax issues?

Hmm.. A very basic bug.. "TestParseGlobalItem" is missing the case for Agent Items..

No, there is a unitest for it, But the unittest sets up its own map; should use the one defined in global.go

In the initial version, we did not have that routine -
NewConfigItemSpecMap() .. So they were hand coded.. Missed changing it..
Will take care of it..
Thanks,
Kalyan
On Thu, Apr 2, 2020 at 11:16 AM Erik Nordmark notifications@github.com
wrote:

Hmm.. A very basic bug.. "TestParseGlobalItem" is missing the case for
Agent Items..
No, there is a unitest for it, But the unittest sets up its own map;
should use the one defined in global.go
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lf-edge/eve/pull/819#issuecomment-608022679, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AK4GJOKNU4GKTR6P2KFTNL3RKTI7RANCNFSM4L2NFSYA
.

I am confused now @eriknordmark @kalyan-nidumolu -- why are we nod adding a blurb to  docs/CONFIG-PROPERTIES.md ?

I am confused now @eriknordmark @kalyan-nidumolu -- why are we nod adding a blurb to docs/CONFIG-PROPERTIES.md ?

We should but I'm handing this (including the doc and fixing the tests) to @kalyan-nidumolu
I had to fix something so I could continue helping Avi debug things i.e., be able to enable the per-agent debug.

I am confused now @eriknordmark @kalyan-nidumolu -- why are we nod adding a blurb to docs/CONFIG-PROPERTIES.md ?

We should but I'm handing this (including the doc and fixing the tests) to @kalyan-nidumolu
I had to fix something so I could continue helping Avi debug things i.e., be able to enable the per-agent debug.

I will take care of this..

Got it! Thanks @kalyan-nidumolu !

authors: eriknordmark kalyan-nidumolu rvs


text:
support FOREIGN TABLE

https://www.postgresql.org/docs/9.6/static/sql-createforeigntable.html
FOREIGN TABLE not support SERIAL PRIMARY KEY
maybe replace
 dump.gsub!(/^    id integer NOT NULL,$/, '    id SERIAL PRIMARY KEY,')

by
dump.gsub!(/^CREATE TABLE (\w+) \(\n    id integer NOT NULL,$/, "CREATE TABLE \\1 (\n    id SERIAL PRIMARY KEY,")

Good idea - do you want to make a PR?

authors: ermolaev lfittl

repo: lfittl/activerecord-clean-db-structure


text:
No output while using "--verbose" parameter

System info:

OS: Debian (codename: bookworm)
hwinfo version: 21.82-1
Install method: Debian repository

Issue description

Used command:

hwinfo --all --map --verbose --log filename.txt
Expected result
Verbose hardware informations
Current result
No output.
Note: When used hwinfo without parameters, I recieved a sumarized output

Hi, you may were looking for this repository: https://github.com/openSUSE/hwinfo

Hi, you may were looking for this repository: https://github.com/openSUSE/hwinfo

I was searching for one with is maintained by Debian because thru it's reposirory it was installed

https://salsa.debian.org/debian/hwinfo

authors: lfreist mdziczkowski

repo: lfreist/hwinfo


text:
add scripts to make qcd templates

add two main scripts to make histograms in the lepton anti-isolated region, and to run linear extrapolations from the anti-isolated regions to the isolated region, along with the systematic uncertainties from signal contaminations.
1). collect the MC normalizations:
 python get_norms.py --fileset filefetcher/aram_samples_lpc.json
2). collect the histograms in the lepton anti-isolated CR:
python runner.py --workflow wptqcd --output hists_wptqcd_filefetcher_aram_samples_lpc_aI.coffea --json filefetcher/aram_samples_lpc_aIso.json
3). make the QCD templates in different W pT bins and different lepton charges (works well for muons now. more checks on electrons.)
python PrepareQCDTemplates.py
which should produce a root file output_qcd_templates.root and a lot of plots

@yongbinfeng can you pip install pre-commit and then pre-commit run --all-files and commit the results? Thanks!

A bit more flake8 and other comments - lemme see if I can fix them.

authors: lgray yongbinfeng

repo: lgray/wpt-coffea


text:
ctrl-f

Quality Assurance Checklist
To make reviews more efficient, please make sure the software feature meets the following standards and check everything off that meets the quality check. Once everything has been checked, the assigned reviewers will begin the review process. Edit this description to check off the list.
There are exceptions with all guidelines. As long as your decisions are justified, then you are good! Contact the reviewers or the leads about any exceptions.
Requirements

[x] Followed Coding Style Guide
[x] Presented/discussed in some capacity with others on the Controls team
[x] Code Build checks pass
[x] No merge conflicts
[x] Software feature has documentation for it updated in both ReadTheDocs and the comments
[x] Software feature has documentation for it updated
Testing

[ ] Software feature has test associated with it
[ ] Test provides useful information and uses relevant data to accurately represent Controls
[ ] Tested on hardware
NOTE: If test file already exists, use that one

[ ] If applicable, have you added the new feature to main.c?
[ ] Tagged appropriate issue ticket on this PR
[x] Did you have fun?

Things to Consider

[ ] Even if the above are checked, is this the best way of writing my code?

It's possible to write code that works, but are there ways to make code more efficient?

Closes #362

repo: lhr-solar/Controls


text:
能否实现弹出后还能继续对界面进行操作?

弹出后,操作空白区域还能对原本界面滑动,点击等等...

这个目前是不行的

authors: li-xiaojun zofnk

repo: li-xiaojun/XPopup


text:
底部弹窗，输入法软键盘弹出后切换到后台在回来的时候，软键盘隐藏，但是弹窗并没有掉下来

XPopup版本
2.6.7
手机系统和型号
部分机型，如三星，部分华为等等，挺多的
描述你的问题
需要弹出软键盘的弹窗，切换到后台，再次返回时软键盘消失，但是弹窗并没有顶下来。导致软键盘的区域是空白的。本来应该在底部的弹窗悬浮在半空。挺多机型都能必现

试试demo有没有这个问题

demo没有问题，但是demo里面的isViewMode(true)修改为isViewMode(false)大部分手机必现。但是isViewMode对于我而言有局限性，项目只能用isViewMode(false)的情况，大佬帮忙看看这个问题有修复的可能吗

@xhh1993 已修复。更新到2.8.2即可

@xhh1993 已修复。更新到2.8.2即可

最高不也才2.8.13吗？

13比2大，有什么问题吗

authors: li-xiaojun oudengding xhh1993


text:
[Plugins API] Combine Plugin.register and Plugin.use?

Hi! Have you considered changing the Plugins API so it combines the functionality of Plugin.register and Plugin.use into one function?
I'm currently using matter-js in another project and importing it as a module via webpack. I recently created a plugin for the project that I would like to be able to be used both in the browser as well as imported as a module. I noticed in your three examples listed in the wiki, you define them as IIFE's that just immediately run Plugin.Register, but this isn't possible in the case where I try importing them as a module.
The way I've currently gotten around the situation is by just defining the plugins object itself, and then just attaching it to exports or window depending on whats available. In the client that uses this code, I just call register, and then use consecutively.
Here is an example where I use it in the browser.
Here is an example of how I'm importing it via ES6 modules and using it that way.
It seems like the extra use call might be unnecessary, and you could potentially just combine it into a single register(...plugins) call. Would there ever be a case where you would want to register a plugin, but not use it?
Thanks for the great work!

The Plugin.register function is used so that other plugins can reference your plugin from anywhere by name. This avoids the need for bundlers for those that don't use them.
It doesn't make sense to combine Plugin.register and Plugin.use since they are run at different times for different reasons (register happens at plugin definition by the plugin author, use happens after by the plugin consumer).
The snippet you showed:
Matter.Plugin.register(MatterCollisionEvents);
Matter.use('matter-collision-events');

Is actually just equivalent to Matter.use(MatterCollisionEvents) because you can simply just pass a plugin object if it's more appropriate. Though I'd advise using register because it allows others to more easily consume your plugin, e.g. in browser or when building their plugins on top of yours, without complex dependency resolution or bundling.
Is there a reason you can't just call Matter.Plugin.register(MatterCollisionEvents) just before line 62 in your code?

Thanks for replying!!
Ah, I think I see what you mean. Plugin.register is useful when people are directly including it in the browser, so they don't have to reference the plugin from the window object.

register happens at plugin definition by the plugin author, use happens after by the plugin consumer

My confusion is around why one would register a plugin, but then not use it. It seems like if anyone was including a plugin on their page (and thus register the plugin in your use case), they will also install (and thus use()) it. And if that were the case, I don't see why you couldn't install plugins in the same function that adds them to the registry.
I looked briefly at the Plugin code. It looks like one potential problem is that use requires the dependency graph, which is generated by using the registry, which is populated by register. I'm wondering if you could get around this by requiring plugins be registered as Matter.Plugin.register(...plugins) upfront. I'm not sure how people are currently setting up plugins, but if I were to use plugins, I would already just call a number of Matter.use() statements in the same place, anyway.

Is there a reason you can't just call Matter.Plugin.register(MatterCollisionEvents) just before line 62 in your code?

I don't call Matter.Plugin.register at line 62 because with bundled code, this would require that I import Matter.js as a dependency within what gets exported as MatterCollisionEvents. As a result, anyone using the Matter.js plugin via a bundler will end up with Matter.js duplicated for every plugin that they use (assuming this is the canonical way to do things). This could be solved if Matter plugins support some sort of dependency injection, but currently matter plugins are just objects.
I could potentially first check if Matter exists so that it will register if it's in the browser, but that would cause a discrepancy in the "API" when used from the browser vs. used with a bundler:

In the browser, you simply call Matter.use('matter-collision-events') because they are already registered
With a bundler, you have to call Matter.register(MatterCollisionEvents); Matter.use('matter-collision-events'), or Matter.use(MatterCollisionEvents), both of which differ from option 1.

My confusion is around why one would register a plugin, but then not use it.

Because registering is the actual plugin definition, which is done by the author?

I'm wondering if you could get around this by requiring plugins be registered as Matter.Plugin.register(...plugins) upfront.

Yes, this is what is supposed to happen.

I would already just call a number of Matter.use() statements in the same place, anyway.

You should only call Matter.use once otherwise you may end up with plugins being installed in the wrong order.

As a result, anyone using the Matter.js plugin via a bundler will end up with Matter.js duplicated for every plugin that they use (assuming this is the canonical way to do things).

Actually this is avoided by configuring your bundler to declare that Matter.js is an external, so it doesn't get bundled.

This could be solved if Matter plugins support some sort of dependency injection, but currently matter plugins are just objects.

I don't see how? I think you'll always need to require matter-js to be able to tell it your plugin exists, either that or you use some special global (which is ugly).
I strongly suggest you switch over to the (newly released) matter-plugin-boilerplate which demonstrates exactly what I mean :)

Ah, I was unfamiliar with externals, thanks for the enlightenment. I hope it doesn't seem like I'm stepping on your toes, I'm definitely happy to change my plugin, these were just some thoughts that I had when I was making the plugins, in case you found them interesting.

Because registering is the actual plugin definition, which is done by the author?

I guess I'm don't really understand why that's important. When the author creates the plugin, he isn't using the plugin, he's just writing out the definition. There is nothing to technically "register" to at the time of definition, because it's just a file that's will eventually be included by a plugin user.
As a plugin user, I don't care that the plugin author has "registered" his plugin, because regardless, I just know that there is a plugin, and I am going to use it.

I don't see how? I think you'll always need to require matter-js to be able to tell it your plugin exists, either that or you use some special global (which is ugly).

Currently, your plugin definitions are defined as objects, but you could potentially have them defined as a function (or an ES6 class), and inject Matter as a dependency there. You have done this with the install function that is specified on the object, but then there is a requirement to run the Matter.Plugin.register function outside that install function. If you think about it, if we're already assuming that Matter is defined, you could actually do away with passing in the base Matter into the install method, since we know that that closure will have access to the Matter outside (which we're using to call Plugin.register)
As an example, Vue.js plugins seem to follow this idea. There is only a single use function, that can be called by either the plugin author OR the plugin user, and when you use a plugin, that plugin can then be referenced or used by the plugin user. This Vue.js plugin shows a simple example. Vue is injected as a dependency into the install function itself, so that the plugin doesn't necessarily need to contain Vue.js if it doesn't want to.
On lines 52-57, it checks - if it's in the browser, immediately use it. Because use has been called, all you now have to do is just include the script, and you don't have to call use again, as a plugin user. If it's being imported as a module, though, you will still need to call Vue.use() on the plugin itself - but the plugin source doesn't actually call it.
That example has been deprecated, but the one that replaces it actually exhibits the same ideas. It includes Vue as a dependency in other files - this plugin seems to be a lot more complex, but you can see similar patterns at the end of this file.
This quote that I wrote:

I'm wondering if you could get around this by requiring plugins be registered as Matter.Plugin.register(...plugins) upfront.

is referring to the fact that my guess that you had originally separated the register from the use so that you could build the dependency graph first, before you use them. I don't know if this is accurate because I didn't study the entirety of the code, so pardon me if I was wrong here.
One example of how you might specify a plugin with dependency injection is if you instead have a model where you simply have an install function, and all your other details are simply specified on that function itself.
so instead of:
var MyPlugin = {
    name: 'matter-my-plugin',

    version: '0.1.0',

    for: 'matter-js@^0.10.0',

    uses: [
        'matter-some-dependency@^0.1.0',
        'matter-another-dependency@^0.1.0'
    ],

    options: {
        something: true
    },

    install: function(base) {
        // patch the Matter namespace here
    },

    // implement your plugin functions etc...
};

Matter.Plugin.register(MyPlugin);

you have
function MatterPlugin(Matter) {
        // patch the Matter namespace here

    // implement your plugin functions etc...
};

MatterPlugin.name = 'matter-my-plugin',
MatterPlugin.version = '0.1.0',
MatterPlugin.for = 'matter-js@^0.10.0',
... etc etc

export default MatterPlugin

and then running Matter.use(...plugins) will just build the dependency graph, and then immediately load them.
again these were just thoughts I had and thought might be useful discussion. I'll change my plugin to follow the current implementation with the external bundling.

I hope it doesn't seem like I'm stepping on your toes

I'm all ears! The plugin system is still very new and it's a good idea to try iron out any kinks early.

As a plugin user, I don't care that the plugin author has "registered" his plugin, because regardless, I just know that there is a plugin, and I am going to use it.

So I should probably explain that the reason registering exists is because plugins can be required by other plugins as dependencies by name:
var MyPlugin = {
    // ...

   uses: [
        'matter-some-dependency@^0.1.0',
        'matter-another-dependency@^0.1.0'
    ],

    // ...
};

The end user of MyPlugin here only ever calls Matter.use(MyPlugin), they don't (and shouldn't) include the two dependencies in their use call, this is done automatically. So if register and use were the same, this would no longer work because those names will not resolve to anything.

If you think about it, if we're already assuming that Matter is defined, you could actually do away with passing in the base Matter into the install method, since we know that that closure will have access to the Matter outside (which we're using to call Plugin.register)

In theory yes but the idea is that technically any object can have plugins installed on it, not just Matter. This may be useful in the future.

Vue is injected as a dependency into the install function itself, so that the plugin doesn't necessarily need to contain Vue.js if it doesn't want to.

The main difference is that Matter.js plugins can reference other plugins by name like I described above. Since I don't think Vue.js plugins can, so it makes sense that this works for them (it's how I initially did it too).

On lines 52-57, it checks - if it's in the browser, immediately use it.

This means that the order of install is down to the order the plugins are loaded, which may be fine for Vue.js. This won't work for Matter.js plugins because they can have dependencies which need to be resolved and installed in a specific order. I don't want to offload figuring out the correct ordering to the end user as this would suck given the recursion involved.

my guess that you had originally separated the register from the use so that you could build the dependency graph first, before you use them.

Exactly yes this is why.

One example of how you might specify a plugin with dependency injection is if you instead have a model where you simply have an install function, and all your other details are simply specified on that function itself.

Your example is almost equivalent to how it is already, except you've moved the Plugin.install to be the plugin object itself right?
I don't see how this solves any of the above issues because given you have a reference to the plugin object, there's little difference between calling MyPlugin(Matter); and MyPlugin.install(Matter);? The crux of the problem here is discovering the reference to MyPlugin when all you have is the name my-plugin.
Just so you don't get me wrong, I'm all for finding a nicer solution to register and I know it's far from convenient. But given that I want the plugin system to support:

resolution by name
recursive dependencies

I'm not sure there's going to be a better solution until all platforms natively support resolving packages by name. Either that or suggest that the end user manually goes through all of their plugins and resolves / registers all of the possible dependencies themselves (which I think sucks).

Thanks for taking the time to patiently discuss and explain! Some more thoughts:
To give context on my thoughts, just in case I'm misunderstanding the flow, this is my current understanding of the usage of a plugin. Let's say there are three plugins, PluginA, PluginB, and PluginC, between which have some sort of dependencies. They're all made by different authors. In the definitions, we call register.
There is a user that is using Matter.js, and wants to use all three plugins. For simplicity lets just assume its in the browser. When they include the scripts, they just include them in any order they please. The plugins will call register, and get put into the registry
When their application starts up, when they set up the boilerplate for their app, they also call Matter.use(PluginA, PluginC, Plugin B). Matter will generate the dependency graph, and figure out which ones to install first. If there is an issue resolving a dependency, this is where you error out and tell the user.
Here, I'm a little fuzzy on:
The dependency resolution is important, because if, for example, PluginA has a dependency on PluginB, it may want to reference PluginB in PluginA's implementation. For example, PluginA needs a way to get the reference to the plugin object of PluginB, so that PluginA can use it within its install function. Plugin A may also use any methods that PluginB adds onto the baseMatter. Is that italicized statement accurate?

So if register and use were the same, this would no longer work because those names will not resolve to anything

It looks like in the docs, you already suggest that people call Matter.use(...plugins) once, and with all plugins. Wouldn't you be able to just, within the Matter.use function, call Plugin.register on each of the plugins, and then call Plugin.use on them afterwards? This disallows people to call Matter.use() in the order they wish, but I don't know if that's actually useful, because like you mentioned people probably don't want to manually resolve dependencies, similar to any package manager.

This means that the order of install is down to the order the plugins are loaded, which may be fine for Vue.js. This won't work for Matter.js plugins because they can have dependencies which need to be resolved and installed in a specific order

Yeah, you're right, the browser case shouldn't actually be applied to Matter. In this case, as a plugin author, you would simply just not call use. The base idea that I was trying to outline was that they don't require you to "static engine setup step" (register), and then run a "dynamic step" (use).

actually no, the dependency graph gets built inside Matter.use, whereas Matter.register only pairs a plugin name to a plugin object.

Oops!! You're right, sorry about that. I should've looked at the code more carefully.

there's little difference between calling MyPlugin(Matter); and MyPlugin.install(Matter);? The crux of the problem here is discovering the reference to MyPlugin when all you have is the name my-plugin, on any platform.

The primary difference is that the author of MyPlugin now no longer needs to include Matter.js in the MyPlugin definition. Instead, in Matter.js itself, you pass Matter directly into the plugin in your generic "load plugin" step (in this case, I believe it is within Plugin.use. Within this step as well, you can also set up the registry, or the map of string -> Plugin, and plugin users can just know to be able to directly call (something along the lines of) Matter.registry['matter-js-plugin']

I'm not sure there's going to be a better solution until all platforms natively support resolving packages by name

To clarify, what do you mean by platform? Like node (or bundlers) vs browser?
I'm also wondering if there is another issue that might come up if plugin developers start using dependencies.
Here's a scenario I'm thinking about:
There is MyPluginB@0.3.0, and then MyPluginA@0.0.3, but it has a dependency on MyPluginB@0.1.0 because the developer hasn't updated dependencies, and then there is MyPluginC@0.0.3 that depends on MyPlugin@0.2.0. And for the sake of discussion, each of the 0.x.0 updates on MyPluginB are breaking API changes (missing methods, etc)
The user wants to use all three plugins, and is in the browser, so he goes and downloads each of the three files. The dependency resolution will go through fine, because the dependency graph just includes 3 different versions of MyPluginB. However, the user has actually only downloaded the latest, 0.3.0 version of MyPluginB! When MyPluginA and MyPluginC try to run or try to get used, they try to reference methods that were supposed to be added on by MyPluginB, but were not included.
An easy solution is to just throw warnings or errors that there were multiple versions of MyPluginB that were included, but only one plugin instance. However, this ends up forcing the user of Matter.js to go and download each of the individual releases to include themselves.
Have you thought about somehow relegating the dependency resolution and plugin management work at build-time, i.e, just have plugin developers include modules via npm or yarn. I don't know if it's even possible since the plugins modify Matter at run-time, but it might be interesting.
I'm thinking something where, for example: I am developing MyPlugin, I will import any dependencies I need as npm modules, import these npm modules, and then specify my dependencies: [] array with the npm modules I import (instead of the names). In Matter.js, you can just build a tree and run a post-order traversal and call the install functions on each of the plugins (post-order, so that you start with the dependencies first, before calling install on the plugins themselves).

When their application starts up, when they set up the boilerplate for their app, they also call Matter.use(PluginA, PluginC, Plugin B)

Well, only if all three of those are 'top level'. If plugin B and C are only dependencies of A, then the user should only call Matter.use(PluginA) (this is why registering the others is important).

For example, PluginA needs a way to get the reference to the plugin object of PluginB, so that PluginA can use it within its install function. Plugin A may also use any methods that PluginB adds onto the baseMatter. Is that italicized statement accurate?

No not really, it's not (usually) the case that plugins directly reference functions from other plugins (though, it is possible and acceptable, it's not the problem that the plugin system is solving). The actual reason for dependencies inside plugins is that they usually build on some feature that another plugin will patch on Matter. So in other words, they're not usually 'hard' dependencies, they're implicit, an expectation of some new 'built-in' functionality.

It looks like in the docs, you already suggest that people call Matter.use(...plugins) once, and with all plugins.

Again, not all the plugins. Only the ones the user is interested in (the user does not pass the lower level dependencies into use).

Wouldn't you be able to just, within the Matter.use function, call Plugin.register on each of the plugins, and then call Plugin.use on them afterwards? This disallows people to call Matter.use() in the order they wish, but I don't know if that's actually useful

It's not just useful but sometimes required for things to work. The user must figure out the order for the top level plugins themselves because there's no automatic way to do it. Usually though, this order shouldn't be too complicated (often it may even be arbitrary) due to the dependency resolver doing the hard parts.
Now, if the order of Matter.use was guaranteed to be arbitrary, then everything you're saying makes sense, because then like you say we can get rid of Matter.register.
To be fair, it may actually be the case that the top level order is always arbitrary iff plugins correctly specify all of their dependencies... hmm. I need to think of a solid counter case.

The primary difference is that the author of MyPlugin now no longer needs to include Matter.js in the MyPlugin definition.

But the only reason they do though right now is because of needing to call Matter.Plugin.register, the plugin format itself makes no requirement of Matter.
Aside from that, I did think about using the format you suggest, but the current format is more inline with the style of the codebase. Also, the function-as-the-plugin-object format makes it look like it's some sort of JS constructor which is a little confusing, attaching static functions to it also feels a little weird. I prefer the plugin-as-a-namespace style.

Instead, in Matter.js itself, you pass Matter directly into the plugin in your generic "load plugin" step (in this case, I believe it is within Plugin.use

This is already the case though? The first argument of MyPlugin.install is the Matter namespace, which is what should be patched (dependency injection, as you say).

Within this step as well, you can also set up the registry, or the map of string -> Plugin

Yeah it could, but only if all plugins were passed by the user (not currently the case, like I discussed). It's an interesting thought though, if only I could guarantee that order didn't matter.

To clarify, what do you mean by platform? Like node (or bundlers) vs browser?

Yes, node vs browser vs what ever else can run JS.

An easy solution is to just throw warnings or errors that there were multiple versions of MyPluginB that were included, but only one plugin instance. However, this ends up forcing the user of Matter.js to go and download each of the individual releases to include themselves.

It's already handled and warnings will be thrown. Note that the system only installs the single latest version of a plugin that the user has loaded (as multiple versions of a single plugin are impossible to support simultaneously).

Have you thought about somehow relegating the dependency resolution and plugin management work at build-time, i.e, just have plugin developers include modules via npm or yarn. I don't know if it's even possible since the plugins modify Matter at run-time, but it might be interesting.

Nope because the system is designed to work on any platform. I don't want to tie anyone into using a bundler or package manger. Otherwise closer integration with npm would have been a neat idea :)

It's not just useful but sometimes required for things to work. The user must figure out the order for the top level plugins themselves because there's no automatic way to do it.

Sorry I didn't quite follow this. If we are assuming top level plugins don't have dependencies on each other, why does order matter?

But the only reason they do though right now is because of needing to call Matter.Plugin.register, the plugin format itself makes no requirement of Matter.

Yeah, the Matter requirement is only needed for Matter.Plugin.register.

Aside from that, I did think about using the format you suggest, but the current format is more inline with the style of the codebase. Also, the function-as-the-plugin-object format makes it look like it's some sort of JS constructor which is a little confusing, attaching static functions to it also feels a little weird. I prefer the plugin-as-a-namespace style.

That's fair, I won't belabor that point.

This is already the case though? The first argument of MyPlugin.install is the Matter namespace, which is what should be patched (dependency injection, as you say).

Yeah, I am referring to the requirement of Matter in the plugin definition to call Matter.register

It's already handled and warnings will be thrown. Note that the system only installs the single latest version of a plugin that the user has loaded (as multiple versions of a single plugin are impossible to support simultaneously).

That's a good point, now that I think about it, multiple versions are indeed impossible since they all modify Matter. I think you can still reuse this run-time check while also utilizing npm for the dependencies though.

Nope because the system is designed to work on any platform. I don't want to tie anyone into using a bundler or package manger. Otherwise closer integration with npm would have been a neat idea :)

Aren't you already tying plugin development to node/bundlers, though, since there is the need to require Matter for Matter.register? Otherwise, plugins developed specifically for the browser that don't use bundlers would have to assume there is a global Matter reference, which in turn would not be able to be used by bundlers.

I don't want to break up the discussion, but in case it's helpful, I just converted matter-collision-events to use Plugin.register, but am running into issues when trying to use it in another project. I've specified Matter.js as an external in the plugin itself, and the example runs fine in the browser.
For my other project, I'm building all my front-end code with bundles via webpack. This is my entry file. I'm importing both Matter and MatterCollisionEvents as a bundle, and I think because I import Matter as a bundle instead of including it as a script file, it is never set on window, so when MatterCollisionEvents runs, it throws errors saying that Matter isn't defined. Any thoughts on how I can get around this? I may change it so that it checks existence on the plugin side as a temporary fix, and just call register from my project's file.

Sorry I didn't quite follow this. If we are assuming top level plugins don't have dependencies on each other, why does order matter?

There are cases where it does matter. For example imagine a plugin that patches Matter.Render and renders some sort of HUD onto the canvas (HudPlugin). Now imagine another plugin that also patches Matter.Render and does some kind of post-processing effect, for example a bloom filter (BloomPlugin). The user must always use HudPlugin last, otherwise BloomPlugin will apply the bloom filter to the HUD which will make it blurry. Notice that there's no dependency here, so it's down to the user to order them correctly.
Another example. Imagine a plugin that patches Body.create and adds an option to generate vertices in some way, let's say TrianglePlugin. Now another that patches Body.create and makes the corners of the body's vertices rounded, ChamfterPlugin. For these two plugins to work together, the order must be TrianglePlugin before ChamfterPlugin otherwise the result will just be a triangle rather than a rounded one.
Hope those make sense?

Aren't you already tying plugin development to node/bundlers, though, since there is the need to require Matter for Matter.register?

Not at all. In the browser, Matter is global so there's no need to require anything.

Otherwise, plugins developed specifically for the browser that don't use bundlers would have to assume there is a global Matter reference, which in turn would not be able to be used by bundlers.

For this project the browser is first class and therefore I have avoided any requirement for using bundlers because they are not standardised. I do strongly suggest that production plugins should be  compatible with bundlers (which is why matter-plugin-boilerplate exists to make it easy), but it isn't a requirement due to this.
I'll put this another way: Matter and the plugin system are completely agnostic to what ever dependency resolution (as in, require etc) system you'd like to use.  There are many of them and none are standard. You're left to your own devices, as is standard practice for any browser based library.
If the author didn't bundle their plugin (and I strongly suggest they do for compatibility, it's just common sense), then it's ultimately down to the person using the bundler to handle resolving missing globals. This is usually a built in feature of bundlers too, for example see webpack's shimming modules feature.

authors: dxu liabru

repo: liabru/matter-js


text:
Multiple Storage Engines

Currently limited to one.

Update to list

Refactor recorder / storage engine to be simple when initialising?

Handled by multiple strategies.

repo: liampauling/flumine


text:
Migrate to CMake

Mostly for a better ecosystem. Especially, CLion has built-in CMake support.

https://github.com/bluealert/amy-cmake
我把您的代码改成cmake构建了。在macOS 10.13.1上编译完全通过。把mysql_service移到amy/detail里面了，把mysql_service.cpp删去了。也就是说，现在amy-cmake是一个纯头文件库了。

@bluealert Sorry for the late reply and thanks for your efforts! Would you like to send a pull request for your CMake changes?
I didn't quite get the mysql_service.cpp part, though. I don't think that file ever existed in Amy and Amy was designed to be a header only library from the very beginning.
BTW, it would be great if you can leave messages here in English so that our discussions can be more widely accessible! 🌏 🌍 🌎

I have changed the code too much. For example, I only support Boost.Asio, not support vanilla Asio. I also remove SConstruct file. To merge the code properly is too difficult? I think you do the support to CMake yourself would be less work?

Ah, I see, that's why you made a whole new repository instead of forking Amy. Yea, borrowing code from your repo seems to be easier. I will probably keep both SCons and CMake for a while before having all the options ported to CMake. Thanks for the explanation!

fixed in #17

authors: bluealert hhggit liancheng

repo: liancheng/amy


text:
merge from master

What is the purpose of the change
XXXXX
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[ ] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

ok

repo: liangliangchen/incubator-dubbo


text:
"CSP_VERSION" Is not defined

Is this supposed to be "LIBCSP_VERSION" which is set in waf?
https://github.com/libcsp/libcsp/blob/0dbdc8e7a055462124cca10264d85dbeaf367210/include/csp/csp.h#L75

No. The WAF build system is currently not maintained, since the switch to 2.0. we have people working on it.
As a workaround you can use mesok build system instead of simply modify WAF or the source to set CSP version to either 1 or 2.

So, CSP_VERSION is a build option to tell the build system which version of library to build, and it's a single digit number.  LIBCSP_VERSION was the version number of the library, but it's not used and removed from the devel branch.
CSP_VERSION is not used at all, so the following patch will fix this.  But we still have more issues with wscript.
diff --git a/wscript b/wscript
index 52373a8..c8bb0ea 100644
--- a/wscript
+++ b/wscript
@@ -69,6 +69,7 @@ def options(ctx):
                   help='Set minimum compile time log level. Must be one of: ' + str(valid_loglevel))
     gr.add_option('--with-rtable', metavar='TABLE', default='static',
                   help='Set routing table type: \'static\' or \'cidr\'')
+    gr.add_option('--version2', action='store_true', help='Build CSP version 2')

 def configure(ctx):
@@ -121,6 +122,11 @@ def configure(ctx):
     ctx.define_cond('CSP_WINDOWS', ctx.options.with_os == 'windows')
     ctx.define_cond('CSP_MACOSX', ctx.options.with_os == 'macosx')

+    if ctx.options.version2:
+        ctx.define('CSP_VERSION', 2)
+    else:
+        ctx.define('CSP_VERSION', 1)
+
     # Add files
     ctx.env.append_unique('FILES_CSP', ['src/*.c',
                                         'src/external/**/*.c',

@johandc, do you want to merge this?  I can make a pr if you want to keep waf.  Or are you dropping waf support for v2.0?

Gomspace would like to maintain WAF support for their build systems, and I don't see any reason the two build systems should not be able to coexist.
Surely I would merge that. Seems like a quick fix.
At space inventor I have moved to Meson.

Thanks to yashi for his help

authors: arrooney johandc yashi

repo: libcsp/libcsp


text:
rdp: Fix csp_rdp_queue_flush packet leak with multiple conn

A conn closing could delete packet from others conn, now we enqueue them again.
I was checking another issue during my debug, I found out that when another connection was closing it would dequeue and free the packet from other connection. That shouldn't be the case because it could cause a packet loss on RDP.

Good catch. Thanks.
Would you mind if I ask you to work on the commit log a bit?

Good catch. Thanks.
Would you mind if I ask you to work on the commit log a bit?

No problem !

I add some comments and csp_debug for fail error.
I was thinking on rewriting the function...
void flush_queue(csp_queue_handle_t * queue, csp_conn_t * conn) {
    csp_packet_t * packet;
    int queue_size = csp_queue_size(queue);
    while (queue_size--) {
        if (csp_queue_dequeue(queue, &packet, 0) == CSP_QUEUE_ERROR) {
            /* csp_queue_dequeue will fail if the queue is empty, so it's better to break here */
            break;
        }

        if (packet == NULL) {
            continue;
        }
        if ((conn == NULL) || (conn == packet->conn)) {
            csp_buffer_free(packet);
        } else {
            if (csp_queue_enqueue(queue, &packet, 0) != CSP_QUEUE_OK) {
                csp_print("csp_rdp_queue_flush : queue fail to enqueue, data lost !\n");
                /* Free the packet if re-enqueue fails */
                csp_buffer_free(packet);
            }
        }
    }
}

void csp_rdp_queue_flush(csp_conn_t * conn) {
    /* Flush conn data from TX queue */
    flush_queue(tx_queue, conn);
    /* Flush conn data from RX queue */
    flush_queue(rx_queue, conn);
}

I hope it make sense.

this could be a root cause of #491.


text:
Custom Database

Hi- I'm trying to classify my sequences using a custom LSU database but I am running into some errors that I'm not sure how to solve.
I've ostensibly formatted my database headers in the UNITE format eg.:

Schizosaccharomyces_pombe_Asco|Z19136.1|k__Fungi;p__Ascomycota;c__Schizosaccharomycetes;o__Schizosaccharomycetales;f__Schizosaccharomycetaceae;g__Schizosaccharomyces;s__pombe

Candida_glabrata_Asco|AY198398.1|k__Fungi;p__Ascomycota;c__Saccharomycetes;o__Saccharomycetales;f__Saccharomycetaceae;g__Candida;s__glabrata

This is the code I am running:

constax -i otu97repseqs_clean_AMFonly.fasta
-d AMFDB_UNITEFORMAT.fasta
-c 0.8 -b -t -n 120

And this is the output I am getting

Reformatting database
UNITE format detected
Traceback (most recent call last):
File "/apps/eb/constax/2.0.17/opt/constax-2.0.17-0/FormatRefDB.py", line 88, in
temp2 = temp[4].strip().split("__")
IndexError: list index out of range

Do you have any idea why I might be coming up with this error?
Thanks!
Phil

Hi @pbraileyjones ,
The formatting script splits the header at |, and takes the 5th element for taxonomy. You should just need to add one extra |, like in this header:
>Entoloma_flavidum|JQ281481|SH1510137.08FU|refs|k__Fungi;p__Basidiomycota;c__Agaricomycetes;o__Agaricales;f__Entolomataceae;g__Entoloma;s__Entoloma_flavidum

For example:
>Candida_glabrata_Asco|AY198398.1|XXX|reps|k__Fungi;p__Ascomycota;c__Saccharomycetes;o__Saccharomycetales;f__Saccharomycetaceae;g__Candida;s__glabrata

I hope that works,
Julian

Thanks! That was it!
Best,
Phil

authors: liberjul pbraileyjones


text:
Modify Tests to Run on Consumer Hardware

This PR modifies the FFCV tests to work on consumer hardware by adding cpu_count checks and lowering the batch and size in test_cuda_nonblocking.py.
The test_cuda_nonblocking.py changes could be moved to a pytest config file with a pytest extension like pytest-testconfig, assuming pytest doesn't this built in and I am missing it.
The dataset writer tests currently hang on my machine, I believe I fixed this issue by merging in the fix-198 and FFCV 1.0 branches.

This looks like a great and easy fix, thank you!

authors: andrewilyas warner-benjamin

repo: libffcv/ffcv


text:
Bullet Wrapper Triangle callbacks

Issue details
I would like to create my own TriangleIndexCallback. However, in the callback function internalProcessTriangleIndex(btVector3 triangle, int partId, int triangleIndex) I only get the first vector of the triangles. Is it possible to customize the bullet wrapper to pass an array (or a List) with all three vectors of the triangle? (e.g. internalProcessTriangleIndex(btVector3[] triangle, int partId, int triangleIndex))
Reproduction steps/code
public class MyInternalTriangleIndexCallback extends btInternalTriangleIndexCallback {

	@Override
	public void internalProcessTriangleIndex(btVector3 triangle, int partId, int triangleIndex) {
		Vector3 v0 = new Vector3(triangle.getFloats());
		System.out.println(triangleIndex + ": " + v0);
	}

	public static void main(String[] args) {
		LibGDXLoader libgdxLoader = new LibGDXLoader();
		libgdxLoader.loadAll();

		btTriangleMesh triangleMesh = new btTriangleMesh();
		triangleMesh.addTriangle(new Vector3(0, 0, 0), new Vector3(0, 1, 0), new Vector3(0, 2, 0));
		triangleMesh.addTriangle(new Vector3(1, 0, 0), new Vector3(1, 1, 0), new Vector3(1, 2, 0));
		triangleMesh.addTriangle(new Vector3(2, 0, 0), new Vector3(2, 1, 0), new Vector3(2, 2, 0));

		MyInternalTriangleIndexCallback callback = new MyInternalTriangleIndexCallback();
		Vector3 aabbMin = new Vector3();
		Vector3 aabbMax = new Vector3(2, 2, 2);
		triangleMesh.InternalProcessAllTriangles(callback, aabbMin, aabbMax);
	}

}

0: (0.0,0.0,0.0)
1: (1.0,0.0,0.0)
2: (2.0,0.0,0.0)

Version of LibGDX and/or relevant dependencies
1.9.10
Please select the affected platforms

[ x] Linux

https://github.com/libgdx/libgdx/wiki/Issue-Tracker

https://github.com/libgdx/libgdx/wiki/Issue-Tracker

I'm not sure but I guess that means I should post my request in the forum: https://www.badlogicgames.com/forum/viewtopic.php?f=11&t=28971
Then maybe this section should be changed accordingly: https://github.com/libgdx/libgdx/wiki/Bullet-Wrapper---Using-the-wrapper#working-with-bullet-wrapper ?

repo: libgdx/libgdx


text:
New TiledMapPacker output, fixes multiple tileset problem

Adds/Changes the following

Searches for maps in nested folders, regardless of depth

Creates an individual atlas for each map (uses the map name sans extension)

Fixes loading/rendering problems with multiple tilesets

Removes isBlended checking and addition of blended property

Removes unused methods, broken methods, minor cleanup throughout TiledMapPacker class

Fixes the problem of not locating tilesets with ../ in the map Note: This has only been fixed with the      new functionality (not --combine-tilsets) The old method of combining tilesets into one atlas still has this problem.

Old functionality can be called with the option "--combine-tilesets"

Adds two classes to test the options and renders the processed maps

Changes usage to: Usage: INPUTDIR [OUTPUTDIR] [--strip-unused] [--combine-tilesets] [-v]

Adds a new usage prompt:
 Usage: INPUTDIR [OUTPUTDIR] [--strip-unused] [--combine-tilesets] [-v]
 Processes a directory of Tiled .tmx maps. Unable to process maps with XML tile layer format.

     --strip-unused
          omits all tiles that are not used. Speeds up the processing. Smaller tilesets.
     --combine-tilesets
          instead of creating a tileset for each map, this combines the tilesets into some kind of monster tileset. Has problems with tileset location. Has problems with nested folders. Not recommended.
      -v
          outputs which tiles are stripped and included

Example:
Running TiledMapPacker on "./in ./out" with an "in" directory with the following structure:

in/overworld.tmx
in/dawn/Examples/Blank.tmx
in/dawn/Examples/Dungeon.tmx
in/dawn/Examples/Logo.tmx
in/dawn/Examples/Mine.tmx
in/dawn/Examples/Town.tmx
in/dawn/Examples/Underworld.tmx
in/world1Maps/dungeon1.tmx
in/world1Maps/dungeon2.tmx
in/world2Maps/town1.tmx
in/world2Maps/town2.tmx

would have the following structure in the "out" directory:

out/Blank.tmx
out/dungeon1.tmx
out/dungeon2.tmx
out/Dungeon.tmx
out/Logo.tmx
out/Mine.tmx
out/overworld.tmx
out/town1.tmx
out/town2.tmx
out/Town.tmx
out/Underworld.tmx
out/tileset/Blank.atlas
out/tileset/dungeon1.atlas
out/tileset/dungeon1.png
out/tileset/dungeon2.atlas
out/tileset/dungeon2.png
out/tileset/Dungeon.atlas
out/tileset/Dungeon.png
out/tileset/Logo.atlas
out/tileset/Logo.png
out/tileset/Mine.atlas
out/tileset/Mine.png
out/tileset/overworld.atlas
out/tileset/overworld.png
out/tileset/town1.atlas
out/tileset/town2.atlas
out/tileset/town2.png
out/tileset/Town.atlas
out/tileset/Town.png
out/tileset/Underworld.atlas
out/tileset/Underworld.png

(Note: maps without a tileset .png were completely empty maps)
Old functionality (now --combine-tilesets) can't handle the nested directories or parent (../) paths for tilesets but if it could run on the same in/out directories it would have the following output:

out/{same maps here}
out/tileset/packed.atlas
out/tileset/packed.png

With all of the tiles from every map inside the packed.png.
TL;DR
Changes TiledMapPacker to create one texture atlas per map, adds two classes to
test that I haven't messed things up more than they were.. I think. Fixes multiple tileset loading/rendering problem. Removes isBlended checking, property addition.

modified:   CHANGES
modified:   extensions/gdx-tools/src/com/badlogic/gdx/tiledmappacker/TiledMapPacker.java
new file:   extensions/gdx-tools/src/com/badlogic/gdx/tiledmappacker/TiledMapPackerTest.java
new file:   extensions/gdx-tools/src/com/badlogic/gdx/tiledmappacker/TiledMapPackerTestRender.java
modified: gdx/src/com/badlogic/gdx/maps/tiled/AtlasTmxMapLoader.java

Please don't create a new PR for every change. If you make new commits they will automatically be added to the existing PR.

Sorry.

Is that change in AtlasTmxMapLoader compatible with existing stuff?

I asked in IRC if AtlasTmxMapLoader was used for anything other than TiledMapPacker's processed maps but there wasn't anyone that knew with absolute certainty. From my understanding, AtlasTmxMapLoader is only used to process maps with an 'atlas' property and TiledMapPacker is the only thing that adds an 'atlas' property to tiled maps. I could try asking at another time.

People do use it without using the packer.

The call hierarchy in libGDX says that the only files that use AtlasTmxMapLoader are
TiledMapAtlasAssetManagerTest
TiledMapAtlasDirectLoaderTest
Although the asset manager test is the only one that shows up in my LwjglTestStarter.
They both point to data/maps/tiled-atlas-processed/test.tmx
and that tiled map has the tell-tale sign of being processed with tiledMapPacker since the first line has
property name="atlas" value="tileset/packed.atlas" property name="blended tiles" value=""

If you run TiledMapAtlasAssetManagerTest, there will be artifacts on my branch since it was created with the old TiledMapPacker. If you run the TiledMapPacker tool with tiled-atlas-src as the input and tiled-atlas-processed as the output, they will render better than master. This is what master's TiledMapAtlasAssetManagerTest looks like and this is what it renders like in my branch.
The more tiles you have, the more obvious the problems with multiple tilesets are. Here's a two layers, each with an entire tileset on it the one on the right is how it currently is in master. The center one has been processed with all the changes in my branch.
I've added the reprocessed maps to my branch.
I'd like to know what else uses AtlasTmxMapLoader to create tests to check for conflicts.

It is a public class... part of the public api. Anybody can use it in their
project. Just because libgdx itself doesn't use it, doesn't mean it isn't
used.

I know. However, the current implementation is broken. AtlasTmxMapLoader is currently expecting for the tileset index for each tileset to begin at zero and end at number of tiles in each tilset then the firstgid for the tileset would be added to that. Since it iterates through every texture region, there would be conflicts with almost every single tile.
The way I changed it, it now expects that the index of every tile is exactly the index that is used in the Tiled Map Editor. So every tile can now have a unique id, as it should.

As far as I know, the way it works now is that each tileset should be
packed in the atlas by the tileset name, with indices by tile. Whether the
packer packs them that way, I don't know.

It does look like that was the way AtlasTmxMapLoader was written. The packer has been packing every texture region with "packed" as the name, not the tileset's name. There hasn't been a change in the tiled-atlas-processed directory for 2+ years. I'm not sure about the packer since it used to be tiled-preprocessor...? So the AtlasTmxMapLoader rewrite might have broken the packer, even though it seems to have been written for the packer.
I still think having texture region indexes start at zero for every tileset is a mistake. Trying to debug the problem as it is was horrible enough. It seems much nicer to be able to make a map in Tiled, save the tile layer as CSV, look at the index for a particular tile, be able to look up that index in the atlas easily, and know that the index will line up inside the map's data structure. Kind of like having the same variable for one thing throughout a project, you see it somewhere, and you know what it is.

AtlasTmxMapLoader really has nothing to do with the packer. When I rewrote
the maps api, I deleted the packer and someone else revived it.
I would prefer we don't change the behavior of the loader as it is part of
the public api and there isn't a compelling reason to change it.

I still think the current AtlasTmxMapLoader implementation is broken since it doesn't do any kind of checking against tileset name to texture region name. There would still be conflicts with indexes when using more than one tileset.

I'll rewrite the packer and test it. So, AtlasTmxMapLoader should be okay with one atlas per map, a region name equal to the tileset name, with indexes from zero to last tile, right?

To get AtlasTmxMapLoader to load an atlas in line with the intent it was written, where it would load regions according to the tileset name/index as I understand it, this line in the loader would need to change to
String regionsName = name;

Since it gets the layer's name on this line. Once you do that, it works like you said it should.
Should I create a bug report with the tests, separate PR, or incorporate that into this PR?

I incorporated the change into this PR. I updated the test files for AtlasTmxMapLoader so they're in line to how you described it should work. The packer now outputs atlas regions formatted in the way you described. With the loader fix, and the updated processed maps, TiledMapAtlasAssetManagerTest is now rendering correctly.
Fixing AtlasTmxMapLoader fixed all of the problems TiledMapPacker was having with multiple layers, at least the problems I am aware of and actively testing for. Now there are no issues in the packer that I know of, except that tile layer format can't be XML since it gets the limitation from TmxMapLoader, from what I understand.
This branch is finished, pending review. Although I'm not sure if I should delete gdx-tools' TiledMapPackerTest and TiledMapPackerTestRender, I really only included them to verify that I didn't mess things up more than they were.

Alrighty, will take some time to fully review due to work and family obligations, but I'll do so soon.

authors: MobiDevelop oddlydrawn


text:
Box2D Shape implements Disposable

Hey i see Shape class got this  comment
* NOTE: YOU NEED TO DISPOSE SHAPES YOU CREATED YOURSELF AFTER YOU NO LONGER USE THEM! E.g. after calling body.createFixture();
 * @author mzechner */
public abstract class Shape implements Disposable {

dispose() method is already here.
I check on package except in World class (already impl Disposable) i dont find other occurence of dispose() or jniDispose()
Thanks for reading.

oh my bad , sry

Not sure it's worth adding the Disposable interface here, the only benefit I see is code documentation but the current docs are pretty clear. Shapes on Box2D are usually very short lived and the dispose() method is usually called just after fixture creation. This is different from the most common behavior of classes that implement Disposable in libGDX in which dispose() is called on longer lived resources when they are not needed anymore for example when changing to a different screen or quitting the game.

It still has some usefulness if you were to consider debug tools which track objects which implement Disposable and throw warnings when an object is finalized without being disposed first.

This doesn't hurt and adds consitency.

authors: PokeMMO badlogic fabiitch