text:
Database Migrations

It would be nice for upgrades to tracy to automatically apply database migrations so that users wouldn't need to delete their database when updating to the lastest binary.

@bennyncc

It appears the errors being thrown by AutoMigration are not being thrown.  Saw them finally when running go run main.go --debug-ui -v.  Might just need to change code here: https://github.com/nccgroup/tracy/blob/master/api/store/sqlite.go#L30 so that any errors are immediately thrown.  Also need to add default value for reason column to solve this particular problem.

What would be the proposed solution for when an error is thrown? Should tracy create a new database file since the migration can't happen with fields that are not null?

Closed in 4016ddcb5a5189eb48c60de9352c29158df6dfa3

If a user of Tracy is on a long running project where they have many saved tracers, they should be able to upgrade to a new version of Tracy without losing all of that data.  I propose that errors are just thrown, and a small amount of testing is done for version upgrades when database schema changes are made to ensure that migrations happen smoothly during the upgrade.

authors: bennyncc jacobheath

repo: nccgroup/tracy


text:
implementation, test case, doc for --check-new option

I'm using crypt on an underlying Amazon Cloud drive, wanting to keep most-recent backup copies of some local directory trees.  Experimenting with the options I saw that a second or later copy operation only picks local files to copy if they are new (never before copied up) or the size is different. I understand this is because ACD does not allow setting the modification time and so it is not checked, and crypt does not support comparing the file hashes.
I wrote a powershell script to extract file names from the output of cryptcheck and for example on a local tree with about 110,000 files it found 300 more files than the copy operation with existing options.  I extended this script to generate a copy command for each file, and it did work but it was slow and made poor use of the available bandwidth due to the overheads of running all those separate commands.
I starting looking at the rclone code for ways that I might improve on this.  I decided that the simplest way to pick out the files I want re-copied is to compare the local timestamp with the remote timestamp: when the local timestamp is more recent it means the local file was altered after its last "backup".
I implemented this as a new option, --check-newer.  The impact on the code is pretty small.  I did update the doc and, while my most extensive testing is using a powershell script, I also was able to work up a test case that exercises the changed logic.
I suppose there may be "better" ways to deal with the underlying concern, such as maybe hooking in the re-hashing used in cryptcheck to the equal function used to decide if a file needs to be copied from the src to the dest.  This being my first-ever exposure to golang and my second-ever attemtp to make a contribution on github I decided to stick with the simplest change that I thought would serve.

This appears at first glance very similar to the -u, --update flag.  Does that flag work for your use case?
If not can you describe how the semantics of --check-new and --update differ?
Thanks
Nick

D'OH!  On re-testing but focusing on --update, I see that it does seem to produce the same result as what I just did with --check-newer.

No worries, rclone has a lot of options!

source: gharchive/pull-request

repo: ncw/rclone

license: MIT

license_type: permissive

license_source: github-api


text:
Add A Bendo Transaction Cleaner

Not sure if this is a periodic, or timer-based thing.

successful transactions should delete their files. And they should be removed after some period of time. (1 day? 3 days? 1 week?)

source: gharchive/issue

authors: dbrower msuhovec

repo: ndlib/bendo


text:
Guard npm

Adding npm install to guardfile with guard-npm.
Remember to run bundle update hesburgh_infrastructure

:+1:

authors: danthewolfe jonhartzler

repo: ndlib/honeycomb

license_source: bigquery


text:
Updated README.md file

Updated browser links

Looks good!

repo: ndlib/presqt


text:
shakeArgs flags should be documented somewhere

Yes I know I can get it using shake --help but sometimes doing that is really annoying.

The website could also explain how to set them as shakeOptions and more details that are too long for a single line.

There's now https://github.com/ndmitchell/shake/blob/master/docs/CommandLine.md, which will go on the website next time I upload.

authors: ezyang ndmitchell

repo: ndmitchell/shake


text:
Added group of dashboards for aws

Summary
A concise description of the changes being introduced. If you are submitting a PR for a
new quickstart please review the pre merge checklist section to validate this PR is ready for review.

Pre merge checklist

[ ] Did you check you NRQL syntax? - Does it work?
[ ] Did you check your dashboard image quality? -  Do they look good?
[ ] Did you check that your alerts actually work?
[ ] Did you include an InstallPlan and Documentation reference?
[ ] Did you check your descriptive content for voice, tone, spelling and grammar errors?
[ ] Did you attach images of your dashboards to the PR so we can see them working?

Screenshots
Attach images of any visual changes, such as a dashboard here.

Dashboard 1
1. Validate dashboard

[ ] Copy the *_validate.json to an account

If the accountId does not have data, find an account ID that does, replace the accountId properties with the new acountId

2. Compare against infra-dashboard

[ ] Find a infra dashboard with data, copy its providerAccountId
[ ] Add a providerAccountId to the filter on the migrated dashboard
[ ] Document differences

If changes are basic, make them directly to the template
If changes need to be applied to all dashboards: change script, re-migrate

3. Validate expected differences

[ ] All widget titles are sentence case
[ ] "Dashboard version x.y.z" is removed from markdown (if applicable)
[ ] inventory widget(s) are removed (if applicable)
[ ] event-feed widget(s) are removed (if applicable)

4. Take a screenshot

[ ] Take a screenshot of the dashboard

Make sure it does not include any PII

[ ] Add screenshot to dashboard directory as a *.png file

5. Add config.yml

[ ] Validate if a quickstarts/**/* directory/file exist

If not, move the one from the dashboard directory
If so, compare it to the one in the dashboard directory; make changes as necessary

6. Confirm contents of config.yml

[ ] Documentation link
[ ] Appropriate installPlan
[ ] Logo (Preferably a logo.svg file in the same directory)
[ ] Description (can be generic)
[ ] Reference to the dashboard(s)

7. Clean up

[ ] Remove the *_validate.json file from the dashboard directory
[ ] Remove the config.yml file from the dashboard directory
[ ] Clean up any *_validate dashboard that you created in NR1

8. Move the PR from draft

[ ] Create the actual PR, ensure all PR Pre merge checklist items are complete

Dashboard 2
1. Validate dashboard

[ ] Copy the *_validate.json to an account

If the accountId does not have data, find an account ID that does, replace the accountId properties with the new acountId

2. Compare against infra-dashboard

[ ] Find a infra dashboard with data, copy its providerAccountId
[ ] Add a providerAccountId to the filter on the migrated dashboard
[ ] Document differences

If changes are basic, make them directly to the template
If changes need to be applied to all dashboards: change script, re-migrate

3. Validate expected differences

[ ] All widget titles are sentence case
[ ] "Dashboard version x.y.z" is removed from markdown (if applicable)
[ ] inventory widget(s) are removed (if applicable)
[ ] event-feed widget(s) are removed (if applicable)

4. Take a screenshot

[ ] Take a screenshot of the dashboard

Make sure it does not include any PII

[ ] Add screenshot to dashboard directory as a *.png file

5. Add config.yml

[ ] Validate if a quickstarts/**/* directory/file exist

If not, move the one from the dashboard directory
If so, compare it to the one in the dashboard directory; make changes as necessary

6. Confirm contents of config.yml

[ ] Documentation link
[ ] Appropriate installPlan
[ ] Logo (Preferably a logo.svg file in the same directory)
[ ] Description (can be generic)
[ ] Reference to the dashboard(s)

7. Clean up

[ ] Remove the *_validate.json file from the dashboard directory
[ ] Remove the config.yml file from the dashboard directory
[ ] Clean up any *_validate dashboard that you created in NR1

8. Move the PR from draft

[ ] Create the actual PR, ensure all PR Pre merge checklist items are complete

Dashboard 3
1. Validate dashboard

[ ] Copy the *_validate.json to an account

If the accountId does not have data, find an account ID that does, replace the accountId properties with the new acountId

2. Compare against infra-dashboard

[ ] Find a infra dashboard with data, copy its providerAccountId
[ ] Add a providerAccountId to the filter on the migrated dashboard
[ ] Document differences

If changes are basic, make them directly to the template
If changes need to be applied to all dashboards: change script, re-migrate

3. Validate expected differences

[ ] All widget titles are sentence case
[ ] "Dashboard version x.y.z" is removed from markdown (if applicable)
[ ] inventory widget(s) are removed (if applicable)
[ ] event-feed widget(s) are removed (if applicable)

4. Take a screenshot

[ ] Take a screenshot of the dashboard

Make sure it does not include any PII

[ ] Add screenshot to dashboard directory as a *.png file

5. Add config.yml

[ ] Validate if a quickstarts/**/* directory/file exist

If not, move the one from the dashboard directory
If so, compare it to the one in the dashboard directory; make changes as necessary

6. Confirm contents of config.yml

[ ] Documentation link
[ ] Appropriate installPlan
[ ] Logo (Preferably a logo.svg file in the same directory)
[ ] Description (can be generic)
[ ] Reference to the dashboard(s)

7. Clean up

[ ] Remove the *_validate.json file from the dashboard directory
[ ] Remove the config.yml file from the dashboard directory
[ ] Clean up any *_validate dashboard that you created in NR1

8. Move the PR from draft

[ ] Create the actual PR, ensure all PR Pre merge checklist items are complete

Dashboard 4
1. Validate dashboard

[ ] Copy the *_validate.json to an account

If the accountId does not have data, find an account ID that does, replace the accountId properties with the new acountId

2. Compare against infra-dashboard

[ ] Find a infra dashboard with data, copy its providerAccountId
[ ] Add a providerAccountId to the filter on the migrated dashboard
[ ] Document differences

If changes are basic, make them directly to the template
If changes need to be applied to all dashboards: change script, re-migrate

3. Validate expected differences

[ ] All widget titles are sentence case
[ ] "Dashboard version x.y.z" is removed from markdown (if applicable)
[ ] inventory widget(s) are removed (if applicable)
[ ] event-feed widget(s) are removed (if applicable)

4. Take a screenshot

[ ] Take a screenshot of the dashboard

Make sure it does not include any PII

[ ] Add screenshot to dashboard directory as a *.png file

5. Add config.yml

[ ] Validate if a quickstarts/**/* directory/file exist

If not, move the one from the dashboard directory
If so, compare it to the one in the dashboard directory; make changes as necessary

6. Confirm contents of config.yml

[ ] Documentation link
[ ] Appropriate installPlan
[ ] Logo (Preferably a logo.svg file in the same directory)
[ ] Description (can be generic)
[ ] Reference to the dashboard(s)

7. Clean up

[ ] Remove the *_validate.json file from the dashboard directory
[ ] Remove the config.yml file from the dashboard directory
[ ] Clean up any *_validate dashboard that you created in NR1

8. Move the PR from draft

[ ] Create the actual PR, ensure all PR Pre merge checklist items are complete

repo: nduvieilh/newrelic-quickstarts


text:
elastic teams

https://github.com/elastic/engineering/blob/master/development_constitution.md

努力创新
坚守原则
不要害怕犯错
积极沟通

license: mit


text:
fix ghost card cause

closes #166

Good catch, thank you!

authors: lennczar mohamedalichelbi

repo: near-multicall/ui


text:
non-english input is not displayed properly

If you try inputting some non-latin text (e.g. cyrillic) you will see that the view is wrong. This is due to the fact that we use dialoguer::prompts::input::Input::interact_text() method and as it's doc-comment says: it "Enables the user to enter a printable ascii sequence and returns the result." (ascii is the key word here)
This is not much of a problem as most users only type latin, but if someone wants to enter some args in e.g. Russian that will cause a bit of pain for the user.
[example of interact_text() method in our codebase](https://github.com/near/near-cli-rs/blob/525703fcdaca6080d3898a35c25bafd8fc2ffc25/src/commands/execute_command/change_method/call_function_type/mod.rs#L54)

Thanks for reporting it. I wonder if dialoguer will consider fixing it

I think dialoguer (the crate that is used for the inputs here) is not going to fix it, so I believe we should switch to inquire in interactive-clap. Contributions are welcome!

authors: adsick frol

repo: near/near-cli-rs


text:
Change license to GPL

near-sdk-rs depends on nearcore which has GPL license, which means it should also be GPL.

Hey @nearmax I know there have been a number of conversations about making crates more independent. I'll admit I haven't followed the discussions between @willemneal and @evgenykuzyakov but I do see the root Cargo.toml has commented out dependencies on nearcore.
https://github.com/near/near-sdk-rs/blame/master/Cargo.toml#L21-L22
This makes me want to ask Willem and Eugene if this issue still stands?

@mikedotexe, the dependencies in nearcore were published and we depend on them in the sdk and the simulator.  Though the crates in nearcore are published with APACHE, so we should update those as well.

Created https://github.com/near/nearcore/issues/4140 to discuss GPL in nearcore, because I can't find a proper issue

authors: evgenykuzyakov mikedotexe nearmax willemneal

repo: near/near-sdk-rs


text:
Please remove the input selector from button,input { overflow: visible} rule

Right now, normalize.css has a rule to make all input elements have overflow:visible. This has no effect in Chrome and Safari, and it has an effect in IE/Edge/Firefox which makes them diverge from Chrome/Safari, which can then cause interop problems for sites that use normalize.css.  So, it's a net negative right now.  Could it be removed?
Investigation:
Below is a testcase I used to test the effect of this rule (with explicitly-sized tiny widgets whose contents are larger than the explicit size, where possible), on every input type that is listed on MDN:
overflow unspecified (default):
http://jsfiddle.net/dholbert/sghn2tz4/embedded/result,css,html
overflow:visible added (as in normalize.css)
http://jsfiddle.net/dholbert/sghn2tz4/1/embedded/result,css,html
Notable differences between these versions, on each major browser engine:

In IE11 and Edge 18, for input type="button,reset,submit", the overflowing button label becomes visible if you provide overflow:visible. There are no other differences.
in Firefox 68 (nightly), for input type="file", the "Browse" button-label and "..." filename area become visible if you provide overflow:visible There are no other differences.
In Chrome 75 (dev) and Safari 12, there's no difference between the two versions.

Most notably, the widgets that start overflowing via overflow:visible in IE/Edge/Firefox are always clipped in Chrome/Safari, even with overflow:visible specified.
Conclusions:
In the majority-market-share browser engine family (Blink/WebKit), overflow:visible has no effect on input elements.  And in other engines, the only effect is to cause these engines to diverge from the Blink/WebKit behavior.
So, unless there's a reason to keep this rule that I'm not seeing, it'd be great if we could remove it, to avoid introducing web compatibility issues https://github.com/mozilla/send/issues/1295 .
Thanks!

For convenience, I've posted a pull request that I think addresses this in https://github.com/necolas/normalize.css/pull/780 .  (Let me know if I'm missing something in my analysis, or if you've got any other concerns about this.)

Just tested this in current Edge and Firefox, and it seems overflow: visible no longer has any meaningful effect there either. (In Edge, interestingly enough, it causes a very slight positioning discrepancy.) So I'm removing it from my normalize in https://github.com/garrettw/vitals

authors: dholbert garrettw


text:
Retrieve errorred object from Form

When using Railjet with server-rendered pages the POST actions can be verified with Forms. These forms however return JSON hashes with the errors on invalid forms. This works really well with javascript front-ends, but not with server-rendered pages, because a model with Class object.errors should contain those errors. This model can then be used on the view again.
As a start I'm proposing something like..
form(Form::MyForm, Class.instance)

begin
  form = form(Form::MyForm)
rescue Form::Invalid -> e
  instance = find(bla)
  e.form.error_object
end

The #form helper was built with JSON API in mind so I'm not surprised it doesn't work in your case 😉

These forms however return JSON hashes

If you use the helper then yeah, that's correct. But forms themselves are behaving very much like AR models. If you initialise Form yourself and call #valid?, then the #errors will actually return an instance of ActiveModel::Errors. Which is exactly what AR model #errors would return.
Railjet also monkey-patches AM::Errors to add #<< method which merges two Errors objects together.
Otherwise you could just assign form errors to model errors and pass that to the view.
I didn't try it but I think something like this could work:
form = FooForm.new(params[:foo])

if form.valid?
  # save to DB
  # redirect or whatever
else
  # option 1)
  @foo = Foo.find(params[:id]).tap { |o| o.errors << form.errors }
  # options 2)
  @foo = Foo.find(params[:id]).tap { |o| o.errors = form.errors }

  # render edit page again
end

authors: roald zlw

repo: nedap/railjet


text:
Using the default "l" command caused the table to become skewed by TableParser

I got bad output using OS X Yosemite version 10.10.5 (14F27) with this doing a simple PID lookup. When I removed the "l" parameter that is hardcoded in, I was able to get the proper output, even though it's much less output.
I don't know if anyone else has experienced this. But I have version 0.1.2 installed.
old output of a process:
UID: [ '407128372' ],
PID: [ '2', '70505' ],
PPID: [ '70505', '70504' ],
CPU: [ '70504' ],
PRI: [ '0' ],
NI: [ '31', '0' ],
VSZ: [ '2464092' ],
RSS: [ '692' ],
WCHAN: [ '692', '-' ],
STAT: [ 'S+' ],
TT: [ '+', 's003' ],
TIME: [ '0:00.05' ],
COMMAND: [ '0:00.05', '-bash' ] }
"fixed" out put of a process:
{ PID: [ '80018' ],
TTY: [ 'ttys003' ],
TIME: [ '0:00.53' ],
CMD: [ 'node', 'node-site.jslib' ] }

@AugustYadon Sorry for the late response. It's very hard to tell just from your description since it works for everyone else.
So for a better knowing of your situation, can you do two things for me?

1 clone this project to your local and run the test npm test to see if all test cases pass
2 type the ps command manually in your terminal ( with the default arguments |) instead of using this module, and copy the result here, so that I can figure what caused the problem.

Thanks.

close due to inactive

authors: AugustYadon neekey

repo: neekey/ps


text:
stdout of ChildProcess.exec("ps") is cut short not giving the complete args to process

Found on version 0.1.4
/lib/index.js Line 74
If a process is started with a particularly long argument/ multiple arguments,
e.g.node /really/long/absolute/path/to/my/script/myscriptname.js
The object returned ps.lookup for that process will return a truncated version of  the
argument passed to the script e.g. /really/long/absolute/pa
`  else {
ChildProcess.exec('ps ' + args, function (err, stdout, stderr) {
  if (err || stderr) {
    return callback(err || stderr.toString());
  }
  else {
    stdout = stdout.toString();
    callback(null, stdout || false);
  }
});`

Function that is causing the problem is the ChildProcess.exec.
Suggested solutions are to change from exec to spawn?
http://www.hacksparrow.com/difference-between-spawn-and-exec-of-node-js-child_process.html
Minor workaround is is to pass an option of { psargs: " " } to the lookup function to limit the size of
stdout returned

@peteroreil this is a know issue for 0.1.4 you can switch to 0.1.5 for the moment, or wait for a few days I will release 0.1.6 soon which fixes a small issue brought in 0.1.5 for kill()

authors: neekey peteroreil


text:
[Proposal] Update and discuss behavior of SVD

Proposal
This proposal is in two parts, addressing TL current SVD implementation.
Note, the SVD is typically written as:
$$
U,S,V^T = \ldots
$$
The important part here is the $V^T$.
Currently, TL returns $V$ not $V^T$, as TL uses torch.svd not torch.linalg.svd, and torch.svd also returned $V$.
https://github.com/neelnanda-io/TransformerLens/blob/347e65235140d942301452a55505c9215b36806a/transformer_lens/FactoredMatrix.py#L118-L125
To address this, we can do one or both of:

small: torch.svd in FactoredMatrix is deprecated and should be replaced by torch.linalg.svd. This can be done and keep all current behavior. Note, right now TL uses Vh in FactoredMatrix but should actually be called V, so if this is done, we should also do the renaming.
big: change the implementation to instead return V_T as torch.linalg.svd does. This would be a breaking change

Pitch
I'd like to make a pitch for doing both of the above. That is, switch to torch.linalg.svd, and return $V^T$ directly. This would let you do:
U, S, V_T = model.OV.svd()
torch.dist(model.OV, U @ torch.diag(S) @ V_T)

like in the docs, and at least for me, behaves more "like expected"
Note: the above is only conceptual. Right now, I dont think you can use FactoredMatrix directly and instead must use model.OV.AB, and the call to diag depends on the sizes of the others, and should probably be .diag_embed
The code change would be very small. Something like:
Ua, Sa, V_Ta = torch.linalg.svd(model.OV.A,full_matrices=False)
Ub, Sb, V_Tb = torch.linalg.svd(model.OV.B,full_matrices=False)

middle = Sa.diag_embed() @ V_Ta @ Ub @ Sb.diag_embed()
Um, Sm, V_Tm = torch.linalg.svd(middle,full_matrices=False)

Uf = Ua @ Um
Sf = Sm
V_Tf = V_Tm @ V_Tb

return Uf, Sf, V_Tf

I can submit a PR, but thought it might be best to discuss first
Alternatives
The first plan should be considered the 'safe and easy' alternative.
Checklist

[X] I have checked that there is no similar issue in the repo (required)

Not directly related, but may be worth clarifying in the docs if this change is done is the following behavior:
model.OV.svd returns the "thin"/"compact" SVD, having performed the more efficient calculation directly.
In principle, one could just do the full SVD torch.linalg.svd(model.OV.AB)and then take the first r=rank columns of $U$ and rows of $V^T$ yourself.
These do not need to directly match! The SVD is not unique relative to $\pm$ of the left and right singular vectors.

Happy to just send a pull-request for the first option, keeping current behavior, while this is discussed

repo: neelnanda-io/TransformerLens


text:
Move towards consistent commenting/docstring style

Description
Convert ActivationCache and HookedTransformer docstrings/comments to the consistent Google Python Style Guide.
Note the only changes made are:

Make sure docstrings have titles (and they end in a full stop).
Put all args under "Args:" (some were missing this section title so they didn't show on the Sphinx docs)
Some very minor spelling mistakes/moving titles to the imperative tense.
Use rewrap to make all the comments the same length (100, as per Pylint config in the project)

There are still other things missing (e.g. sometimes args are not documented) but it's a good start.
Type of change
Please delete options that are not relevant.

[x] This change requires a documentation update

Usual checklist didn't seem to make sense (as it's only changing comments/docstrings), but I've manually gone through and also made sure all wording changes are minor (so it's just formatting).

Seems good! Surely there's a linter or something that might help with this stuff? We could make it default in the dev container?

Yeah pylint will flag stuff like this - but I don't think we can make it mandatory yet as it'll cause headaches for ongoing PRs (as quite a lot of the codebase has squiggly lines :) ). At least re-wrap did most of the hard work for now.
I've just re-checked and will merge now, as leaving a big PR around like this will create merge conflicts otherwise.

authors: alan-cooney jbloomAus


text:
48 rotate image.cs

File(s) Modified: 48-Rotate-Image.cs
Language(s) Used: c#
Submission URL: _https://leetcode.com/submissions/detail/792756112/_

Oops, branched from another branch instead of main, so closing the PR.

repo: neetcode-gh/leetcode


text:
Create 0133-clone-graph.kt

File(s) Modified: 0133-clone-graph.kt
Language(s) Used: Kotlin
Submission URL: _https://leetcode.com/problems/clone-graph/submissions/793825719/_

The submission urls for the other ways of solving this problem (included all 3 ways in the file):
https://leetcode.com/problems/clone-graph/submissions/793816844/
https://leetcode.com/problems/clone-graph/submissions/793824774/

Thanks, @a93a!


text:
Easier solution: 0049-group-anagrams.py

I have written a more straightforward and easier to comprehend solution for problem "Group Anagrams", without compromising the BigO complexities.
Compared to Neetcode's version, this solution can take any type (and number) of different string characters as input and is not limited by 26 English characters.
Instead of using an array to store character counts, we use a hashmap, however, it still eventually converts the counter hashmap to array form for sorting, and then to tuple. But these are all O(1) operations due to the limited number of possible keys (characters). We convert from Dict -> List + sort() -> Tuple to be able to use it as hashmap key.
More importantly, this way we also avoid the "clever" solution of using ASCII codes to modify the correct array index, as we only need to store the counts in a hashmap (like in the Easy "Valid Anagram" problem).

File(s) Modified: 0049-group-anagrams.py
Language(s) Used: python3
Submission URL: _https://leetcode.com/problems/[problem-name]/submissions/xxxxxxxxx/_

Important
Please make sure the file name is lowercase and a duplicate file does not already exist before merging.

Hi could you correct your submission link?

@Zirafnik Thank you for your contributions 👍


text:
refactor:  239-Sliding-Window-Maximum.js

File(s) Modified: 239-Sliding-Window-Maximum
Language(s) Used: javascript
Submission URL: _https://leetcode.com/submissions/detail/786810213/_

Thanks, @aakhtar3! Although your JavaScript solution looks like a duplicate.


text:
Add the ability to insert some predefined HTML at the cursor location.

We are using Squire to implement an editor to which different modules could send arbitrary HTML fragments to insert into the editor. I believe it would be helpful to add an insertHTML method to the Editor.

Commit merged, this is done.

authors: farooqu neilj

repo: neilj/Squire


text:
Fixed MosaicDefinitionTransaction.create() to accept MosaicNonce and MosaicId

Fixed MosaicDefinitionTransaction.create() to accept MosaicNonce and MosaicId
Fixed MosaicDefinitionTransaction unit test payload with new MosaicId

Pull Request Test Coverage Report for Build 163

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
10 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.9%) to 70.555%

Files with Coverage Reduction
New Missed Lines
%

src/model/namespace/NamespaceId.ts
2
66.67%

src/service/NamespaceService.ts
8
21.43%

Totals

Change from base Build 161:
-0.9%

Covered Lines:
840

Relevant Lines:
1110

💛  - Coveralls

Pull Request Test Coverage Report for Build 162

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.04%) to 71.408%

Totals

Change from base Build 161:
-0.04%

Covered Lines:
850

Relevant Lines:
1110

💛  - Coveralls

authors: coveralls evias


text:
Task/g628 metadata receipt

Applied pagination changes for Metadata and Receipt endpoints

ReceiptRepository should provide 2 or 3 independent searches so the user knows exactly what are the accepted parameters and the returned types.
https://github.com/nemtech/symbol-sdk-typescript-javascript/blob/task/g628_metadata_receipt/src/infrastructure/ReceiptRepository.ts#L27
Java as an example:
https://github.com/nemtech/symbol-sdk-java/blob/metadata-receipt/sdk-core/src/main/java/io/nem/symbol/sdk/api/ReceiptRepository.java#L32


text:
Nens auth client

Hoi Reinout,
Ik heb een poging gedaan om de nens-auth-client te installeren. Leek me interessant om zelf eens te proberen. Ik denk dat ik aardig ver ben gekomen, maar ik loop momenteel tegen deze fout aan::
invalid_token: The token is not valid as it was issued in the future
Ik hoop dat dit ligt aan bepaalde configuratie waar ik niet bij kan. Als het ergens anders aan ligt, dan hoor ik het graag

Ik weet niet of jij iets had gedaan al Reinout, maar vandaag werkt de client wel.

Ik heb niks gedaan :-) Ik zal nu even naar je PR kijken.
Dit soort "het komt uit de toekomst" errors kan gebeuren als de interne klok van je docker niet klopt met het OS waarin je browser draait,

Die JWT tokens die op de achtergrond worden gebruikt hebben allemaal een ingebouwde hard afgedwongen timestamp met beperkte geldigheid (minuutje of twee ofzo). Als windows en docker dan hun tijd niet netjes coördineren kan de docker een timestamp gebruiken die bij windows niet meer klopt.
Dat dit alles zo nauwkeurig kan werken is met dank aan "Father Time" (https://en.wikipedia.org/wiki/David_L._Mills) die afgelopen januari is overleden :-)

authors: JJFlorian reinout

repo: nens/bro-hub


text:
Improve Cypher language support in Code Editor

We should update the integrated Cypher Editor to use the latest version of Neo4j Cypher Language Support:
https://github.com/neo4j/cypher-language-support?tab=readme-ov-file
this gives us access to a lot of new features and makes it a lot easier for users to write Cypher in NeoDash:

Support for plugging in a schema, so that label names, rel types and properties are autocompleted.
Autocompletion of Cypher keywords (e.g. MATCH, WHERE, RETURN)
Live syntax validation and formatting.

Demo of what is possible:
https://neo4j.github.io/cypher-language-support/

This would be updating "@neo4j-cypher/react-codemirror": "^1.0.3" to 1.0.4 and testing that everything works, right? If so, I could take care of it.

authors: JonanOribe nielsdejong


text:
update simple template

We seem to have let this fall behind the MP template.  We need to update it now and then keep it up to date going forward.

Harshit has suggested maybe we get rid of simple.  In that case we need to make sure the marketplace template is easily consumable via GitHub (console and CLI).

clean up properly and enhance documentation

This is done.  Simple has been removed and documentation has been improved for the marketplace template.

authors: benofben edrandall-dev


text:
#inspect for model objects should truncate properties

I have some largeish text fields in my nodes and the output is large

Address, I believe, in #1023 so going to move to that issue


text:
JsSymbol

Rendered: https://github.com/chrisbajorin/rfcs/blob/symbol-primitive/text/0000-symbol-primitive.md

@kjvalencik  In regards to Eq, are you just looking for a mention that strict equality should hold for all primitive values, or is this something that fits within the reference-level explanation? I see Eq currently has a blanket implementation on Handle<T>, so I'm not quite sure how this fits in code-wise.

Updated to reflect our discussion in https://github.com/neon-bindings/neon/pull/761

@chrisbajorin This looks really great! I'm going to put it into final comment period! Thanks for your contributions!

BTW, I just wanted to note that the description() method can be technically thought of as a convenience method, since you can implement it with other primitives, i.e. the Neon equivalent of:
const getter = Object.getOwnPropertyDescriptor(Symbol.prototype, 'description').get;
getter.call(sym)

Not a crucial point but it makes me feel confident that the description() method is a nice convenience to have and doesn't fundamentally add any complexity.

authors: chrisbajorin dherman kjvalencik

repo: neon-bindings/rfcs


text:
Error replaying WAL record in regression test

pageserver log in CI run: https://github.com/neondatabase/neon/actions/runs/3462211769
2022-11-14T14:36:31.513003Z ERROR handle_pagerequests{tenant_id=9aab6a83f75803fd7782e0f0366fe938 timeline_id=64b1c0b58cb4576a5f8bccd9780db3cd}:handle_get_page_at_lsn_request{rel=1663/16384/2608 blkno=81 req_lsn=0/9E1ED80}:apply_wal_records{tenant_id=9aab6a83f75803fd7782e0f0366fe938 pid=575}: wal-redo-postgres: 2022-11-14 14:36:31.512 UTC [575] WARNING:  will not overwrite a used ItemId
2022-11-14 14:36:31.512 UTC [575] CONTEXT:  WAL redo at 0/9E0C770 for Heap2/MULTI_INSERT: 1 tuples flags 0x02
2022-11-14 14:36:31.512 UTC [575] PANIC:  failed to add tuple
2022-11-14 14:36:31.512 UTC [575] CONTEXT:  WAL redo at 0/9E0C770 for Heap2/MULTI_INSERT: 1 tuples flags 0x02

2022-11-14T14:36:31.513043Z ERROR handle_pagerequests{tenant_id=9aab6a83f75803fd7782e0f0366fe938 timeline_id=64b1c0b58cb4576a5f8bccd9780db3cd}:handle_get_page_at_lsn_request{rel=1663/16384/2608 blkno=81 req_lsn=0/9E1ED80}:apply_wal_records{tenant_id=9aab6a83f75803fd7782e0f0366fe938 pid=575}: wal-redo-postgres: ---------------------------------------
seccomp: bad syscall 14
---------------------------------------

@hlinnaka , I can't find such error in the attached CI run. Is it the right link?
Can you point me out to the log where you have found this error?

Took me a while to find it too.. When you re-run a job, the results are appended to the same Allure report, but the Overview only shows the latest result for each test.  If you click on the "Timeline" and look for the failed pg_regress test there, you'll get here:
https://neon-github-public-dev.s3.amazonaws.com/reports/pr-2653/debug/3462211769/index.html#testresult/2facc8c78fabe5ab
And there you'll find the pageserver.log that contains that error: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-2653/debug/3462211769/data/attachments/4a3e38e1bdd2c19c.gz

I just got this error on my laptop, when I was running the test_gc_aggressive test:
022-12-28T17:37:24.575545Z  INFO PUT /v1/tenant/95f187af2f3d82a0aebea695b3c7ceb6/timeline/c18c950e2eb89971dc3a6abc4dfe8740/checkpoint 200 OK
2022-12-28T17:37:24.579946Z  INFO manual_gc{tenant=95f187af2f3d82a0aebea695b3c7ceb6 timeline=c18c950e2eb89971dc3a6abc4dfe8740}: starting on 1 timelines
2022-12-28T17:37:24.580006Z  INFO manual_gc{tenant=95f187af2f3d82a0aebea695b3c7ceb6 timeline=c18c950e2eb89971dc3a6abc4dfe8740}:gc_timeline{timeline=c18c950e2eb89971dc3a6abc4dfe8740 cutoff=0/
23C3E30}: GC starting
2022-12-28T17:37:24.580036Z  INFO manual_gc{tenant=95f187af2f3d82a0aebea695b3c7ceb6 timeline=c18c950e2eb89971dc3a6abc4dfe8740}:gc_timeline{timeline=c18c950e2eb89971dc3a6abc4dfe8740 cutoff=0/
23C3E30}: GC completed removing 0 layers, cutoff 0/23C3E30
2022-12-28T17:37:24.580087Z  INFO PUT /v1/tenant/95f187af2f3d82a0aebea695b3c7ceb6/timeline/c18c950e2eb89971dc3a6abc4dfe8740/do_gc 200 OK
2022-12-28T17:37:24.612879Z  INFO Starting Level0 compaction in LSN range 0/1698CC1-0/23C6079 for 10 layers (10 deltas in total)
2022-12-28T17:37:24.612931Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000001698CC1-0000000002385B19
2022-12-28T17:37:24.612954Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__0000000002385B19-00000000023B1141
2022-12-28T17:37:24.612977Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023B1141-00000000023B4FE1
2022-12-28T17:37:24.613010Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023B4FE1-00000000023B7599
2022-12-28T17:37:24.613023Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023B7599-00000000023BA261
2022-12-28T17:37:24.613035Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023BA261-00000000023BCE91
2022-12-28T17:37:24.613052Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023BCE91-00000000023BF5A9
2022-12-28T17:37:24.613063Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023BF5A9-00000000023C1E69
2022-12-28T17:37:24.613073Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023C1E69-00000000023C3E31
2022-12-28T17:37:24.613087Z  INFO compact includes 000000000000000000000000000000000000-FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF__00000000023C3E31-00000000023C6079
2022-12-28T17:37:24.841733Z ERROR handle_pagerequests{tenant_id=95f187af2f3d82a0aebea695b3c7ceb6 timeline_id=c18c950e2eb89971dc3a6abc4dfe8740}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=785 req_lsn=0/23B11C8}:apply_wal_records{tenant_id=95f187af2f3d82a0aebea695b3c7ceb6 pid=1765470}: wal-redo-postgres: 2022-12-28 19:37:24.841 EET [1765470] WARNING:  will not overwrite a used ItemId
2022-12-28 19:37:24.841 EET [1765470] CONTEXT:  WAL redo at 0/23B11C8 for Heap/HOT_UPDATE: off 84 xmax 1580 flags 0x60 ; new off 108 xmax 0
2022-12-28 19:37:24.841 EET [1765470] PANIC:  failed to add tuple
2022-12-28 19:37:24.841 EET [1765470] CONTEXT:  WAL redo at 0/23B11C8 for Heap/HOT_UPDATE: off 84 xmax 1580 flags 0x60 ; new off 108 xmax 0

This was with lots of local changes, so it's very possible that this was an unrelated bug in my code changes. But symptoms look similar.

Did some digging on this. I can reliably reproduce this on my laptop now, although again this is with lots of local changes, so it's very likely it's just a bug in those changes.
Relevant records from Original WAL:
I used -A1 to print the next record after each one, to get the "end LSN" of each interersting record in the output.
./pg_install/v14/bin/pg_waldump test_output/test_gc_aggressive/repo/pgdatadirs/tenants/142ce5ba3a362979b21cb226414db19c/test_gc_aggressive_pg_node/pg_wal/000000010000000000000002 | grep "rel 1663/12990/16384 blk 125" -A1

rmgr: Heap        len (rec/tot):     60/    60, tx:       1371, lsn: 0/023A7880, prev 0/023A7858, desc: LOCK off 21: xid 1371: flags 0x00 LOCK_ONLY EXCL_LOCK , blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):    129/   129, tx:       1371, lsn: 0/023A78C0, prev 0/023A7880, desc: UPDATE off 21 xmax 1371 flags 0x00 ; new off 22 xmax 0, blkref #0: rel 1663/12990/16384 blk 940, blkref #1: rel 1663/12990/16384 blk 125
rmgr: Btree       len (rec/tot):     64/    64, tx:       1371, lsn: 0/023A7948, prev 0/023A78C0, desc: INSERT_LEAF off 223, blkref #0: rel 1663/12990/16389 blk 38
--
rmgr: Heap2       len (rec/tot):     56/    56, tx:          0, lsn: 0/023C1CA8, prev 0/023C1C80, desc: PRUNE latestRemovedXid 1371 nredirected 0 ndead 1, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       1801, lsn: 0/023C1CE0, prev 0/023C1CA8, desc: HOT_UPDATE off 107 xmax 1801 flags 0x60 ; new off 108 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Transaction len (rec/tot):     34/    34, tx:       1801, lsn: 0/023C1D30, prev 0/023C1CE0, desc: COMMIT 2022-12-28 21:11:33.863868 EET
--
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D83F0, prev 0/023D83C8, desc: PRUNE latestRemovedXid 1801 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       2176, lsn: 0/023D8430, prev 0/023D83F0, desc: HOT_UPDATE off 87 xmax 2176 flags 0x60 ; new off 109 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D8480, prev 0/023D8430, desc: PRUNE latestRemovedXid 1533 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 864
--
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D9E18, prev 0/023D9DF0, desc: PRUNE latestRemovedXid 2176 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       2207, lsn: 0/023D9E58, prev 0/023D9E18, desc: HOT_UPDATE off 16 xmax 2207 flags 0x60 ; new off 110 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Transaction len (rec/tot):     34/    34, tx:       2207, lsn: 0/023D9EA8, prev 0/023D9E58, desc: COMMIT 2022-12-28 21:11:34.118436 EET
--
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023E9C38, prev 0/023E9C10, desc: PRUNE latestRemovedXid 1801 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap2       len (rec/tot):     56/    56, tx:          0, lsn: 0/023E9C78, prev 0/023E9C38, desc: PRUNE latestRemovedXid 1284 nredirected 0 ndead 1, blkref #0: rel 1663/12990/16384 blk 711
rmgr: Heap        len (rec/tot):     79/    79, tx:       2509, lsn: 0/023E9CB0, prev 0/023E9C78, desc: HOT_UPDATE off 19 xmax 2509 flags 0x60 ; new off 109 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       2510, lsn: 0/023E9D00, prev 0/023E9CB0, desc: HOT_UPDATE off 76 xmax 2510 flags 0x60 ; new off 108 xmax 0, blkref #0: rel 1663/12990/16384 blk 711

Pageserver log (I added extra logging to print the records being replayed:
022-12-28T19:11:35.063008Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: wal-redo-postgres: 2022-12-28 21:11:35.062 EET [1797955] WARNING:  will not overwrite a used ItemId
2022-12-28 21:11:35.062 EET [1797955] CONTEXT:  WAL redo at 0/23E9D00 for Heap/HOT_UPDATE: off 19 xmax 2509 flags 0x60 ; new off 109 xmax 0
2022-12-28 21:11:35.062 EET [1797955] PANIC:  failed to add tuple
2022-12-28 21:11:35.062 EET [1797955] CONTEXT:  WAL redo at 0/23E9D00 for Heap/HOT_UPDATE: off 19 xmax 2509 flags 0x60 ; new off 109 xmax 0

2022-12-28T19:11:35.063073Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: wal-redo-postgres: ---------------------------------------
seccomp: bad syscall 14
---------------------------------------

2022-12-28T19:11:35.064588Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: WAL redo process closed its stderr unexpectedly, 6 records
2022-12-28T19:11:35.064635Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: rec at 0/023D8430: Postgres { will_init: false, rec: b":\0\0\0\0\0\0\0\xc8\x83=\x02\0\0\0\0\x10\t\0\0\xaa\x9eGv\0 \x04\0\x7f\x06\0\0\xbe2\0\0\0@\0\0}\0\0\0\xff\x08k\0l\0\t\x07\0\0\x01\0\0\0" }
2022-12-28T19:11:35.064670Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: rec at 0/023D8480: Postgres { will_init: false, rec: b"O\0\0\0\x80\x08\0\0\xf0\x83=\x02\0\0\0\0@\n\0\0\t,\xae\x99\0 \x0f\0\x7f\x06\0\0\xbe2\0\0\0@\0\0}\0\0\0\xff\x12\x04\0*\0\x03\x80\x02(\0\0\0\0\x18\0\x01\x80\x08\0\0W\0\0`\0\0\0\0\0\0\0\0m\0" }
2022-12-28T19:11:35.064707Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: rec at 0/023D9E58: Postgres { will_init: false, rec: b":\0\0\0\0\0\0\0\xf0\x9d=\x02\0\0\0\0\x10\t\0\0\xc4\xab\x9f\xc7\0 \x04\0\x7f\x06\0\0\xbe2\0\0\0@\0\0}\0\0\0\xff\x08W\0m\0\x80\x08\0\0\x01\0\0\0" }
2022-12-28T19:11:35.064737Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: rec at 0/023D9EA8: Postgres { will_init: false, rec: b"O\0\0\0\x9f\x08\0\0\x18\x9e=\x02\0\0\0\0@\n\0\0\xaa\x84;I\0 \x0f\0\x7f\x06\0\0\xbe2\0\0\0@\0\0}\0\0\0\xff\x12\x04\0*\0\x03\x80\x02(\0\0\0\0\x18\0\x01\x9f\x08\0\0\x10\0\0`\0\0\0\0\0\0\0\0n\0" }
2022-12-28T19:11:35.064771Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: rec at 0/023E9C78: Postgres { will_init: false, rec: b":\0\0\0\0\0\0\0\x10\x9c>\x02\0\0\0\0\x10\t\0\0\xf3O<\x0f\0 \x04\0\x7f\x06\0\0\xbe2\0\0\0@\0\0}\0\0\0\xff\x08k\0l\0\t\x07\0\0\x01\0\0\0" }
2022-12-28T19:11:35.064802Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:apply_wal_records{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}: rec at 0/023E9D00: Postgres { will_init: false, rec: b"O\0\0\0\xcd\t\0\0x\x9c>\x02\0\0\0\0@\n\0\0{\r/)\0 \x0f\0\x7f\x06\0\0\xbe2\0\0\0@\0\0}\0\0\0\xff\x12\x04\0*\0\x03\x80\x02(\0\0\0\0\x18\0\x01\xcd\t\0\0\x13\0\0`\0\0\0\0\0\0\0\0m\0" }
2022-12-28T19:11:35.064846Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}: error applying 6 WAL records 0/23D8430..0/23E9D00 (411 bytes) to base image with LSN 0/023C1D30 to reconstruct page image at LSN 0/241AC68
2022-12-28T19:11:35.067188Z  INFO handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}:kill{tenant_id=142ce5ba3a362979b21cb226414db19c pid=1797955}:kill_and_wait_impl{pid=1797955}: wait successful exit_status=exit status: 1
2022-12-28T19:11:35.067370Z  INFO handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=929 req_lsn=0/23C7D70}:launch{tenant_id=142ce5ba3a362979b21cb226414db19c pg_version=14}: old temporary datadir /home/heikki/git-sandbox/neon/test_output/test_gc_aggressive/repo/tenants/142ce5ba3a362979b21cb226414db19c/wal-redo-datadir.___temp exists, removing
2022-12-28T19:11:35.078159Z  INFO PUT /v1/tenant/142ce5ba3a362979b21cb226414db19c/timeline/d172374febcb10d810110c0ec320b84d/checkpoint 200 OK
2022-12-28T19:11:35.080391Z  INFO manual_gc{tenant=142ce5ba3a362979b21cb226414db19c timeline=d172374febcb10d810110c0ec320b84d}: starting on 1 timelines
2022-12-28T19:11:35.102710Z  INFO handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=929 req_lsn=0/23C7D70}:launch{tenant_id=142ce5ba3a362979b21cb226414db19c pg_version=14}: running initdb in /home/heikki/git-sandbox/neon/test_output/test_gc_aggressive/repo/tenants/142ce5ba3a362979b21cb226414db19c/wal-redo-datadir.___temp
2022-12-28T19:11:35.067332Z ERROR handle_pagerequests{tenant_id=142ce5ba3a362979b21cb226414db19c timeline_id=d172374febcb10d810110c0ec320b84d}:handle_get_page_at_lsn_request{rel=1663/12990/16384 blkno=125 req_lsn=0/23E9D00}: reconstruct error path

Matching up the LSNs from the "rec at ..." lines from the pagserver log with the pg_waldump output, these are the records being replayed:
base image at LSN 0/023C1D30
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D83F0, prev 0/023D83C8, desc: PRUNE latestRemovedXid 1801 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       2176, lsn: 0/023D8430, prev 0/023D83F0, desc: HOT_UPDATE off 87 xmax 2176 flags 0x60 ; new off 109 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D9E18, prev 0/023D9DF0, desc: PRUNE latestRemovedXid 2176 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       2207, lsn: 0/023D9E58, prev 0/023D9E18, desc: HOT_UPDATE off 16 xmax 2207 flags 0x60 ; new off 110 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023E9C38, prev 0/023E9C10, desc: PRUNE latestRemovedXid 1801 nredirected 1 ndead 0, blkref #0: rel 1663/12990/16384 blk 125
rmgr: Heap        len (rec/tot):     79/    79, tx:       2509, lsn: 0/023E9CB0, prev 0/023E9C78, desc: HOT_UPDATE off 19 xmax 2509 flags 0x60 ; new off 109 xmax 0, blkref #0: rel 1663/12990/16384 blk 125

Note how there are two records with "new off 109". It looks like the WAL sequence is invalid, inserting an updated tuple to offset 109 twice.
Or did the PRUNE record prune out offset 109, making it available for reuse? To rule that out, I modifed pg_waldump to print more details on PRUNE records:
0: base image at LSN 0/023C1D30
1: rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D83F0, prev 0/023D83C8, desc: PRUNE latestRemovedXid 1801 nredirected 1 ndead 0; redirected: [ 107 ] nowdead: [ ] nowunused: [ ], blkref #0: rel 1663/12990/16384 blk 125
2: rmgr: Heap        len (rec/tot):     79/    79, tx:       2176, lsn: 0/023D8430, prev 0/023D83F0, desc: HOT_UPDATE off 87 xmax 2176 flags 0x60 ; new off 109 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
3: rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023D9E18, prev 0/023D9DF0, desc: PRUNE latestRemovedXid 2176 nredirected 1 ndead 0; redirected: [ 87 ] nowdead: [ ] nowunused: [ ], blkref #0: rel 1663/12990/16384 blk 125
4: rmgr: Heap        len (rec/tot):     79/    79, tx:       2207, lsn: 0/023D9E58, prev 0/023D9E18, desc: HOT_UPDATE off 16 xmax 2207 flags 0x60 ; new off 110 xmax 0, blkref #0: rel 1663/12990/16384 blk 125
5: rmgr: Heap2       len (rec/tot):     58/    58, tx:          0, lsn: 0/023E9C38, prev 0/023E9C10, desc: PRUNE latestRemovedXid 1801 nredirected 1 ndead 0; redirected: [ 107 ] nowdead: [ ] nowunused: [ ], blkref #0: rel 1663/12990/16384 blk 125
6: rmgr: Heap        len (rec/tot):     79/    79, tx:       2509, lsn: 0/023E9CB0, prev 0/023E9C78, desc: HOT_UPDATE off 19 xmax 2509 flags 0x60 ; new off 109 xmax 0, blkref #0: rel 1663/12990/16384 blk 125

So no, that's not it. The WAL sequence is definitely bogus.
I think this means that Postgres was operating on an incorrect page version. When Postgres issued the GetPage request for step 5, it incorrectly got a too old page image, which didn't include the changes between 1-4. Because of that, it performed the PRUNEing again in step 5, doing the same redirection as in record 1, and then the UPDATE into the same target offset 109. (I'm a little surprised that replaying that the PRUNE record in step 5 didn't cause an error already.)
So where did Postgres get a too-old page version? Two possibilites:

Compute used too-old LSN in the GetPage request, because of a bug in the last-written LSN cache
Compute sent the request correctly, but the pageserver processed it incorrectly.

If the problem is reliably reproduced you can add tracing of all get_page_at_lsn calls, then find one related with this page (rel 1663/12990/16384 blk 125) and check if  req_lsn is correct. So it will give answer to whether compute or pageserver are responsible for this error.
I wonder  if your branch includes my commit [1137b58b4d476370c36af93f5da46f9ae2562303] ?

Ok, found the issue. It was indeed a bug in my local changes, not present in 'main'. The bug was in how get_reconstruct_data scanned the frozen layers in the layer map, I did some refactorings that broke that so that it skipped frozen in-memory layers in the search.
So this was different from whatever the issue on CI was.

It seems the trail has gone cold on the original issue. We've fixed numerous bugs since then, and we're not currently seeing htis in the CI, so I assume it was fixed.

authors: hlinnaka knizhnik lubennikovaav

repo: neondatabase/neon


text:
Calculate synthetic size worker

New version of the PR contains following changes:

Handle errors in tenant_size_model code instead of panicking
add background worker that periodically spawns synthetic size calculation.
add new pageserver config param calculate_synthetic_size_interval
fix for #3179

TODO:

[x] remove test test_metric_collection_multiple_branches . It is there only to illustrate the bug.
[x] clean the code (remove debug info! messages)

It is not explained why it was decided to calculate logical size in background.
Before it is calculated incrementally.

We still track logical size at the end of the branch incrementally. However, the synthetic size calculation needs the logical size at an older point in time.

May be I missed something but logical=synthetic size is one we are limiting for free tier and what customers are paying for. If so, is it acceptable to calculate it once per 10 minutes?

Logical size is different from the synthetic size.

So I do some massive insert, what to check used size but see... old value.
Once again sorry if it is explained somewhere and I just missed it.

We don't have a good explanation of synthetic size anywhere, so the confusion is understandable.
I wrote a small doc on what the synthetic size is: https://github.com/neondatabase/neon/pull/3328. It needs more work, but I hope it helps.

authors: hlinnaka lubennikovaav


text:
pageserver: flush deletion queue on detach

Problem
If a caller detaches a tenant and then attaches it again, pending deletions from the old attachment might not have happened yet.  This is not a correctness problem, but it causes:

Risk of leaking some objects in S3
Some warnings from the deletion queue when pending LSN updates and pending deletions don't pass validation.

Summary of changes

Deletion queue now uses UnboundedChannel so that the push interfaces don't have to be async.

This was pulled out of https://github.com/neondatabase/neon/pull/5397, where it is also useful to be able to drive the queue from non-async contexts.
Why is it okay for this to be unbounded?  The only way the unbounded-ness of the channel can become a problem is if writing out deletion lists can't keep up, but if the system were that overloaded then the code generating deletions (GC, compaction) would also be impacted.

DeletionQueueClient gets a new flush_advisory function, which is like flush_execute, but doesn't wait for completion: this is appropriate for use in contexts where we would like to encourage the deletion queue to flush, but don't need to block on it.

This function is also expected to be useful in next steps for seamless migration, where the option to flush to S3 while transitioning into AttachedStale will also include flushing deletion queue, but we wouldn't want to block on  that flush.

The tenant_detach code in  mgr.rs invokes flush_advisory after stopping the Tenant object.

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

I also wonder if we should somehow measure the queue length in a metric or warn if the queue length is larger than some limit. There is the risk of having a memory "leak" like the one fixed in https://github.com/neondatabase/neon/pull/5472 ...

We do already have metrics that enable calculating queue length  -- this was a good reminder for me to add charts for those to the pageserver dashboard.  I'll follow up to add an alert for queue length.

Push:

Merge main into this branch
Since the LocationConf stuff merged, we should also do this flush-on-detach behavior when we shut down a tenant in upsert_location -- added that.


text:
pageserver: fix cont lsn jump on vectored read path

Problem
Vectored read path may return an image that's newer than the request lsn under certain circumstances.
  LSN
    ^
    |
    |
500 | ------------------------- -> branch point
400 |        X
300 |        X
200 | ------------------------------------> requested lsn
100 |        X
    |---------------------------------> Key

Legend:
* X - page images

The vectored read path inspects each ancestor timeline one by one starting from the current one.
When moving into the ancestor timeline, the current code resets the current search lsn (called cont_lsn in code)
to the lsn of the ancestor timeline (here).
For instance, if the request lsn was 200, we would:

Look into the current timeline and find nothing for the key
Descend into the ancestor timeline and set cont_lsn=500
Return the page image at LSN 400

Myself and Christian find it very unlikely for this to have happened in prod since the vectored read path
is always used at the last record lsn.
This issue was found by a regress test during the work to migrate get page handling to use the vectored
implementation. I've applied my fix to that wip branch and it fixed the issue.
Summary of changes
The fix is to set the current search lsn to the min between the requested LSN and the ancestor lsn.
Hence, at step 2 above we would set the current search lsn to 200 and ignore the images above that.
A test illustrating the bug is also included. Fails without the patch and passes with it.
Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] If it is a core feature, I have added thorough tests.
[ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
[ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

[ ] Do not forget to reformat commit message to not include the above checklist

The test looks convincing enough, but I cannot make the connection to the actual vectored read path change (min).
Does the old Timeline::get_value_reconstruct_data have this problem? There, we move to ancestor if cont_lsn is less than ancestor_lsn, and cont_lsn is originally request_lsn + 1. Okay, I think these are now aligned.

It's quite tricky to see the alignment between the read paths since they're structured differently. I think going through the example in the cover letter is the best way of understanding the issue. The test produces a very similar setup, albeit with a lot of boilerplate. Let me know if you wish to chat about it - the bug is quite obvious once you know about it :)

I'll merge now to get some validation for the change before next weeks release.

Great explanation and test case!
It took me a while to understand why the non-vectored path is not suffering from the same bug. I believe it's because `get_reconstruct_data' has this in the loop:

            // Recurse into ancestor if needed
            if is_inherited_key(key) && Lsn(cont_lsn.0 - 1) <= timeline.ancestor_lsn {
                trace!(
                    "going into ancestor {}, cont_lsn is {}",
                    timeline.ancestor_lsn,
                    cont_lsn
                );

                timeline_owned = timeline.get_ready_ancestor_timeline(ctx).await?;
                timeline = &*timeline_owned;
                prev_lsn = None;
                continue 'outer;
            }

Wouldn't hurt to explicitly test for the same scenario in the non-vectored case too. Or maybe we already have a test for that?
It would be useful to cross-check that get_reconstruct_data produces the same result as get_vectored. For example, at the end of get_reconstruct_data, call get_vectored with a corresponding single-item KeySpace, and assert that the result is the same.

We had some discussion with @problame about this change in slack thread: https://neondb.slack.com/archives/C033RQ5SPDH/p1717665056195329 about min(request_lsn + 1, ancestor_lsn + 1) vs. min(cont_lsn + 1, ancestor_lsn + 1).

authors: VladLazar hlinnaka koivunej


text:
Add option allowing running multiple endpoints on the same branch.

This is used by safekeeper tests.

This is used by safekeeper tests.

Which tests exactly? Is it only run_concurrent_computes?

Yes, only that so far.

authors: arssher petuhovskiy


text:
API incorrectly calculates what loos are in a given radius / bounding box

Problem
At present, metres are converted into radians before being passed to Mongo's $geoNear.
We don't really want to represent our distance as radians while providing an interface that uses metres, however, as the amount of metres a radian represents in longitude varies greatly at different latitudes. Our simple conversion method doesn't take this into account and so the results returned are misleading.
As we use $geoNear with a radius to do our bounding box queries too, using the same method of conversion, searches within a bounding box of coordinates are also affected.
Suggested Remedy
Do we want to be using metres for toilet range searches in our API?

For nearby toilets, almost definitely, as this represents how accessible toilets are for a user far better than coordinates could. Mongo has support for metre radii, which we should try and use here.
For coordinate bounds, it is more complicated. We should see how this could be used on the front-end before fixing this implementation but, to fix this, either:

The approach for nearby toilets could be used as previously (and clipped to the bounding box, which is not done at the moment, although this is a separate issue) or
Simple numerical conditions could be tested against the toilet's coordinates (greater than / less than) to eliminate toilets outside the bounds, assuming they are rectangular, which is not checked for at present)

After some discussion:
Let's:
a) Make an attempt to talk radii in meters all the way through.
b) Immediately remove the /in/<bbox> endpoint and all its supporting code, until we can prove a use-case for it.

authors: Hoolean marxian

repo: neontribe/gbptm


text:
Enable terraform-ls for .tf files

The .tf extension seems to be what Hashicorp suggest in most of their tutorials and documentation.

tf by default relates to some mud game files or something, don't think that pr is correct.

Yeah the tf filetype isn't Terraform, it's TinyFugue.

Ah fair enough. My bad.

authors: SG60 gegoune lithammer

repo: neovim/nvim-lspconfig


text:
Create initial package for AMF based on the new CRD

Create initial package for AMF based on the new CRD

Also Olivier Smith wants to support and help Anh Thu out.

solved: https://github.com/nephio-project/free5gc-packages/pull/10

https://github.com/nephio-project/free5gc-packages/pull/11
fixed the naming

authors: gvbalaji henderiw

repo: nephio-project/nephio


text:
Rename interface PackageDraft to PackageRevisionDraft

This is a cleanup PR that renames the interface PackageDraft to PackageRevisionDraft. This interface is used to manage package revision drafts. The interface is renamed to improve the readability of the Porch code.

/approve

/assign @kispaljr @Catalin-Stratulat-Ericsson @JamesMcDermott

/approve

/lgtm

authors: JamesMcDermott efiacor liamfallon

repo: nephio-project/porch


text:
Better and platform agnostic path handling in commands

I think we have two goals:
1. Express and append paths in a native form

Common case to express paths properly depending on platform
Should use the native os.pathsep for appending

2. Convert/Enforce to a non-native form after expansion

Case where env variable is either not representing a PATH or has non-native expectations
This may mean the need to control the os.pathsep  (which we have already)

For better illustration here samples of non-handled cases and their intentions:
def commands():
    # Case 1.
    env.PATH.append("{root}/bin")     # This really should be "{root}\\bin" on some platforms

   # Case 2.
   env.LANGUAGE_COMMENTS.append(r"//")    # This does not represent a path, should be literal
   env.CMAKE_MODULE_PATH.append("{root}/cmake")    # Tricky: This path needs to use forward slashes and ; regardless of platform.

Solution A
One way to deal with 1. is to make the native path conversion implicit.
So imagine that we assumed any environment variable is by default a path that should be native.
Our sample would need to be explicit in the 2. cases:
# We got this already. Necessary in rezconfig / package for case 2. later on
with scope("config") as c:
    c.env_var_separators = {
        "CMAKE_MODULE_PATH": ";",
    }

def commands():
    # Case 1.
    env.PATH.append("{root}/bin")     # Converts to native path with os.pathsep

   # Case 2.
    env.LANGUAGE_COMMENTS.append(literal(r"//"))    # Literal with os.pathsep
    # This needs special treatment.
    # Can't use str.replace() since it must happen after expansion(?)
    env.CMAKE_MODULE_PATH.append(unixpath("{root}/cmake")
    #                                   ^ we don't have this

Solution B
The alternative solution would be to always be explicit, and assume literal otherwise:
# Necessary in rezconfig / package for case 2. later on
with scope("config") as c:
    c.env_var_separators = {
        "CMAKE_MODULE_PATH": ";",
    }

def commands():
    # Case 1.
    env.PATH.append(nativepath("{root}/bin"))     # Can't use os.path.join

   # Case 2.
    env.LANGUAGE_COMMENTS.append(r"//")    # No special treatment

    # Still need to handle this case
    env.CMAKE_MODULE_PATH.append(unixpath("{root}/cmake")
    #                                   ^ we don't have this

Solution C
This is an extension of Solution A, but we give the user control over the behavior in rezconfig so we could even act as Solution B.
Like before we implement three additional bindings:

nativepath -> Converts to native path
unixpath -> Converts to forward slashes
windowspath -> Converts to backwards slashes

But we add a definition in config that handles paths.
Similar to the env_var_separators environment variables do have special types and I wouldn't mind this information to be hidden away in config, especially since we can always override it per package.
# rezconfig.py

env_var_separators = {
        "CMAKE_MODULE_PATH": ";",
}

# Tries functions, else uses default
env_var_default_functions = ["nativepath"]
env_var_functions = {
    "CMAKE_MODULE_PATH": ["expandvars", "unixpath"],    # Accepts list for chaining
    "LANGUAGE_COMMENTS": ["literal"],
}

Our packages become idiomatic:
def commands():
    # Case 1.
    env.PATH.append("{root}/bin")     # Assumes native path

   # Case 2.
   env.LANGUAGE_COMMENTS.append(r"//")    # Handled by rezconfig
   env.CMAKE_MODULE_PATH.append("{root}/cmake")    # Handled by rezconfig

And we should be able to express any other case in rezconfig or the package directly.

Literal is wrong as default. I probably mean expandvars.

I agree with the general direction of C, but I'd be up for simplifying it a
little, and generalising the settings (which will make more sense below).
RE env_var_functions: I don't think we should have this 'chaining' -
expandvars should always be implied (knowing that expansion won't happen if
literal() is used). Exposing 'expandvars' to the user like this is
confusing IMO. So instead I would just have a single function name for each
entry - "nativepath", "unixpath" etc ("posixpath" instead?) - implying that
these all do variable expansion anyway.
RE generalization. So there is a need to give more control over env-vars in
general. Right now for eg, append/prepend always works in
set-if-first-then-append mode - ie, PYTHONPATH gets set the first time it's
appended/prepended, then appended/prepended on all subsequent ops. We
should have a way of refining this behaviour - for eg, a useful default
would be, "append/prepend, but retain the original, pre-rez-env value as
well, and keep it appended to the result". You could have other modes as
well, like "retain" (which would do what the "parent_variables" setting
does currently).
In any case, without going too much into those details, I think the
settings need to be structured in a way that promotes this idea of more env
var control, and we could start doing this now. What I mean is, instead of
this:
env_var_separators = {
"CMAKE_MODULE_PATH": ";",
}
env_var_default_functions = ["nativepath"]
env_var_functions = {
"CMAKE_MODULE_PATH": ["expandvars", "unixpath"],    # Accepts list
for chaining
"LANGUAGE_COMMENTS": ["literal"],
}
We would have:
env_var_settings = {
"": {
"path_modify_op": "nativepath"  # ie append, prepend
},
"_PATH": {
"set_op": "nativepath"  # eg, so "env.MAYA_PLUGIN_PATH = ..." uses
nativepath
},
"CMAKE_MODULE_PATH": {
"pathsep": ";",
"path_modify_op": "unixpath"
}
}
Note that the globbing would allow for less verbose settings, by sharing
settings across vars (eg, "CMAKE_MODULE_PATH" would inherit settings from
"", then "_PATH", in that order. Which actually isn't useful in this
specific example, but you get the idea).
Lastly - I don't know if you already implied this or not, but nativepath()
should be able to take any sensible path syntax (posix or windows) and
convert it to the native equivalent. That way, studios can just default use
whichever they are comfortable with. We should check for ambiguous cases
(like "/hey/this\is\weird"), and raise an error for those. Do we also
need to deal with drives in Windows (eg C://), is that even still a thing?
I have no idea.
On Sat, Aug 31, 2019 at 12:48 AM Blazej Floch notifications@github.com
wrote:

Literal is wrong as default. I probably mean expandvars.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nerdvegas/rez/issues/707?email_source=notifications&email_token=AAMOUSXPP6CWELITP7KL7YTQHEXLLA5CNFSM4ISODKG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5R34SQ#issuecomment-526630474,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMOUSVSMCMLKL5MNDEKQ5TQHEXLLANCNFSM4ISODKGQ
.

I agree. Thanks for taking the proposal further.
On Wildcards and inheritance
I wonder how the order would work - I can imagine this brings up some challenges, like:
{
  "*": { ... },        # A
  "*_PATH": { ... },   # B
  "HELLO_*": { ... },  # C
}

In this case what would the the right initial setting given HELLO_PATH and the fact that dicts are unordered?
Maybe a better approach would be to have ordered settings with explicit inheritance (although not as nicely looking):
OrderedDict([
  ("CMAKE_MODULE_PATH", {...} ),  # D
  ("*_PATH", {...} ),             # B
  ("HELLO_*", {                   # C
      "inherits": "*",
      ...
  })
  ("*", {...} ),                  # A, usually at the end.
])

Then the rule is:

If it has the explicit key use it (e.g. env_var_settings["CMAKE_MODULE_PATH"])
Else iterate the keys and cancel if a match is found (can skip non wildcards)
If there is inheritance, then follow it and apply in reverse order.

The benefit is now we can express both inheritance and explicit rules that don't want to inherit.
Looking at this I wonder how strong the relationship between variables is. Maybe we don't need inheritance at all, due to the complexity involved.
It shouldn't be too much work to have separate settings for each special case. Since it is python you could also define you little helpers variables.
But I would be fine with having inheritance if there are strong opinions about it.
I'll put some more thought into the ops and try to come up with all the crazy cases that we might want to handle across platforms.

I would avoid the ordered inheritance for two reasons - 1) it just looks
odd as you mention, but 2) it forces you to put things in order that are
not related (FOO and BAH for eg).
In terms of inheritance - we at least need to specify global defaults, and
perhaps that, combined with specific env-var overrides, is enough. What we
could do is use the schema I've already suggested, but support only full
env-var names as keys, or "*" (which is the global defaults). Doing it this
way also means that we don't break compatibility if we decide later that we
do indeed want more fine-grained inheritance as I described.
If we did want inheritance, then a fairly simple set of rules can be used
to determine order:

Globbed before non-globbed
shorter before longer
lastly, alphabetical

Thus "" would be used, then "PATH_" to override, then "FOO_PATH" to
override again. Ambiguous cases like "PATH_" and "_HELLO" would just fall
back to alphabetical (ie arbitrary but deterministic).
In summary, I'd be happy to use suggested schema but support only '*' and
full env-var names, leaving the option for further inheritance open in case
we decide we want it.
Thx
A
On Thu, Sep 5, 2019 at 5:13 AM Blazej Floch notifications@github.com
wrote:

I agree. Thanks for taking the proposal further.
On Wildcards and inheritance
I wonder how the order would work - I can imagine this brings up some
challenges, like:
{
"": { ... },        # A
"PATH": { ... },   # B
"HELLO*": { ... },  # C
}
In this case what would the the right initial setting given HELLO_PATH
and the fact that dicts are unordered?
Maybe a better approach would be to have ordered settings with explicit
inheritance (although not as nicely looking):
OrderedDict([
("CMAKE_MODULE_PATH", {...} ),  # D
("PATH", {...} ),             # B
("HELLO", {                   # C
"inherits": "",
...
})
("", {...} ),                  # A, usually at the end.
])
Then the rule is:

If it has the explicit key use it (e.g.
env_var_settings["CMAKE_MODULE_PATH"])
Else iterate the keys and cancel if a match is found (can skip non
wildcards)
If there is inheritance, then follow it and apply in reverse order.

The benefit is now we can express both inheritance and explicit rules that
don't want to inherit.
Looking at this I wonder how strong the relationship between variables is.
Maybe we don't need inheritance at all, due to the complexity involved.
It shouldn't be too much work to have separate settings for each special
case. Since it is python you could also define you little helpers variables.
But I would be fine with having inheritance if there are strong opinions
about it.
I'll put some more thought into the ops and try to come up with all the
crazy cases that we might want to handle across platforms.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nerdvegas/rez/issues/707?email_source=notifications&email_token=AAMOUSRPWRGG2NOE7I7GCQ3QIACEZA5CNFSM4ISODKG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD54U4UY#issuecomment-528043603,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAMOUSRGMQTWFKY4EUHLQDDQIACEZANCNFSM4ISODKGQ
.

@nerdvegas - I agree with everything you said.
I'd prefer the first pass implementation (* and specific vars inheriting it) for now and then we can expand on it like you proposed. Maybe it is not that much work to begin with doing the full thing. We'll see.
Let's open the next and bigger can of worms:
Settings and Ops
This is a rather complex topic since we want to support so many use cases so I'll build up to the suggestion based on every possible use case imaginable. It also gives us opportunity to migrate existing settings in a generic configuration.
I don't want to jump into implementation details yet (hope this is doable at all :) ), but let's see if we
can come up with use cases best demonstrated via an sample.
References
#703
#698
#661
(more ?)
Use cases
(cases handled by the sample are crossed)

Path operations:

[x] Handle paths in platform independent fashion
[x] Handle paths in explicit platform dependant fashion
[ ] Handle differences in Drive handling (C:\ -> C:/ vs /C/) (Needed?)
[ ] Handle platform dependent path prefix (C:\path -> /mnt/c/path) (Needed?)

[x] Define platform dependent settings
[x] Migrate all_*_variables
[x] Define path separators (like ; and :)
[ ] Stable de-duplication of separated values (Needed?)
[x] Implicit variables, system variables and there placement with context variables

Sample
(Note: This is a completely artificial example to showcase many possible use cases)
env_var_settings = {

    # -------------------------------------------------------------------------
    # Global control inherited(!) by all other variables.

    "*": {

        # Migration of all_parent_variables
        #
        # Full control over variables is achieved given the following parent_variables & actions:
        #
        # clear (No parent variables)
        #  [prepend_all] [prepend_parent->FAILS] [append_parent->FAILS] [$REZ_VARS] [append_all]
        #
        # prepend (parent before rez variables)
        #  [prepend_all] [prepend_parent] [$PARENT_VARS] [append_parent] [$REZ_VARS] [append_all]
        #
        # append (parent after rez variables)
        #  [prepend_all] [$REZ_VARS] [prepend_parent] [$PARENT_VARS] [append_parent] [append_all]
        #
        "parent_variables": "clear"

        # Migration of all_resetting_variables
        # Note: Actually had a problem with understanding the naming, but kept for consistency
        "resetting_variables": False

        # Any variable (!!!) on this level may be "its expected type" OR dict of
        # ("platform": "its expected type").
        # Note we are only handling platform.system differences.
        "pathsep": {
            "windows": ";",
            "linux":   ":",
            # Any other platform, except specific one …
            "*":       "🍎",
        },
        # Default append, prepend and set operation
        "modify_op": "nativepath",
        # Interestingly this could now also be implemented as followed :-)
        # but we probably still need nativepath on commands() side.
        "modify_op": {
            "windows": "windowspath",
            "*": "posixpath",
        },
    },

    # -------------------------------------------------------------------------
    # Creation of variables without having to rely on bootstrap file or package
    # (Studio must make decision of when to use implicit packages vs. rezconfig)

    # Set custom variables without package.

    "STUDIO_REZCONFIG_VERSION": {
        "parent_variables": "clear",
        "prepend_all": "2019.08.23.2",
    },

    "STUDIO_STORAGE": {
        "parent_variables": "clear",
        "prepend_all": {
            "windows": "P:\\",
            "*":  "/mnt/production",
        }
    },

    # Sample where a user might run a local STUDIO_CACHING_SERVER
    # [Parent] [This] [Rez]
    "STUDIO_CACHING_SERVERS": {
        "parent_variables": "prepend",
        "append_parent": ["192.168.2.2", "cache.studio.com"], # Note multiple things as list
    },

    # Sample where user might override for debugging, rez is default authority but
    # ultimately use sensible default
    # [Parent] [Rez] [This]
    "STUDIO_LOCATION": {
        "parent_variables": "prepend",
        "append_all": "UNKNOWN",
    },

    # Artificial case where REZ packages always have priority, but we guarantee a fallback
    # coming from Parent Environment or This config
    # [Rez] [Parent] [This]
    "STUDIO_USER_GROUPS": {
        "parent_variables": "append",
        "append_parent": {
            "windows": ["UPDATING"],
            "linux": ["WORKING"],
            "osx": ["BROKE"],
        }
    },

    # -------------------------------------------------------------------------
    # Common use cases
    # We now can control the "always-on" tools and their order very precisely

    "PATH": {
        # In this sample: Rez first
        "parent_variables": "append",

        "prepend_all": {
            "windows": "C:\\ImportantTools",
            "*": "/opt/important_tools"
        },

        # Here after the system paths
        "append_all": {
            "windows": ["C:\\WindowsFallbackTools"],
        },
    },

    "PYTHONPATH": {
        "parent_variables": "clear",
    },

    "CMAKE_MODULE_PATH": {
        "parent_variables": "clear",

        # Nice.
        "pathsep": ";",
        "modify_op": "posixpath",
    },

    "PATHEXT": {
        "pathsep": ";",
        "parent_variables": "prepend",
        "append_parent": {
            # Does not pollute other platforms
            "windows": [".PY", ".PS1"],
        }
    }
}

Naming, features and details are like always up for discussion. Let me know if I missed anything.

I'll close this in favor for #737 which has a broader context.

authors: bfloch nerdvegas

repo: nerdvegas/rez


text:
printing material

Hi,
Do you have any instruction on the printing materials and assemble?

您好有组装相关的图纸么

当然，您可以使用我们的 自然协议纸 ，它演示了合适的打印材料。如果你把你的电子邮件发给我，我可以给你发np2的详细组装协议，它或多或少与协议文件中描述的相同，但视觉效果可能会有所帮助。

可以也发我一份么229161309@qq.com

Hi there, I'm just hopping on this thread as well. I am trying to produce some fixtures for NPXL 2.0 implants as well. We dont have access to SLS printing, as far as I'm aware, but we do have a bambu labs FFF printer with pretty good resolution (seems good enough for this part) and an anycubic MSLA printer. So id be curious to hear your thoughts on what other materials might be acceptable. david.protter@colorado.edu

Hi David, I have sent you the protocol, and SLA also works since we do most of our prototypes with that material

authors: ChangQing-SH cagjony dprotter youyikang


text:
Normalize depth for packed samples

This resolves #811 .

Hmm! I don't have write access! Do you mind giving me access? I'll not to burn down the building.

repo: nerfstudio-project/nerfstudio


text:
Error: moleculec: not found

Describe the bug
A moleculec not found error occurs on c-sharedlib contracts. Molecule is not listed as a prerequisite of Capsule.
Screenshots
username@dev1:~$ capsule new project
New project "project"
Created file "capsule.toml"
Created file "deployment.toml"
Created file "README.md"
Created file "Cargo.toml"
Created file ".gitignore"
Initialized empty Git repository in /home/username/project/.git/
Created "/home/username/project"
Created tests
     Created library `tests` package
New contract "project"
     Created binary (application) `project` package
Rewrite Cargo.toml
Rewrite ckb_capsule.toml
Done
username@dev1:~$ cd project
username@dev1:~/project$ capsule new-contract sharedlib --template c-sharedlib
Setup C environment
Cloning into '/home/username/project/contracts/c/deps/ckb-c-stdlib'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 205 (delta 1), reused 4 (delta 1), pack-reused 196
Receiving objects: 100% (205/205), 108.79 KiB | 2.09 MiB/s, done.
Resolving deltas: 100% (93/93), done.
New contract "sharedlib"
Rewrite Makefile
Rewrite ckb_capsule.toml
Done
username@dev1:~/project$ capsule build
Building contract project
    Updating crates.io index
   Compiling cfg-if v0.1.10
   Compiling cc v1.0.41
   Compiling buddy-alloc v0.3.0
   Compiling molecule v0.6.0
   Compiling ckb-allocator v0.1.1
   Compiling ckb-standalone-types v0.0.1-pre.1
   Compiling ckb-std v0.6.3
   Compiling project v0.1.0 (/code/contracts/project)
    Finished dev [unoptimized + debuginfo] target(s) in 6.29s
Building contract sharedlib
curl -L -o build/blockchain.mol https://raw.githubusercontent.com/nervosnetwork/ckb/d75e4c56ffa40e17fd2fe477da3f98c5578edcd1/util/types/schemas/blockchain.mol
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2357  100  2357    0     0  21623      0 --:--:-- --:--:-- --:--:-- 21824
moleculec --language c --schema-file build/blockchain.mol > build/blockchain.h
/bin/sh: 1: moleculec: not found
make: *** [Makefile:30: build/blockchain.h] Error 127
error: process exit with code Some(2)

Additional context
Please post the following informations:
username@dev1:~/project$ capsule --version
Capsule 0.4.1 303dfcb
username@dev1:~/project$ ckb-cli --version
ckb-cli 0.35.0 (3875086 2020-08-11)username@dev1:~/project$

Rerun your command with  RUST_BACKTRACE=1 RUST_LOG=capsule=trace environment, then post the output as a file.

username@dev1:~/project$ capsule build
Building contract project
    Updating crates.io index
   Compiling cfg-if v0.1.10
   Compiling cc v1.0.41
   Compiling buddy-alloc v0.3.0
   Compiling molecule v0.6.0
   Compiling ckb-allocator v0.1.1
   Compiling ckb-standalone-types v0.0.1-pre.1
   Compiling ckb-std v0.6.3
   Compiling project v0.1.0 (/code/contracts/project)
    Finished dev [unoptimized + debuginfo] target(s) in 6.29s
Building contract sharedlib
curl -L -o build/blockchain.mol https://raw.githubusercontent.com/nervosnetwork/ckb/d75e4c56ffa40e17fd2fe477da3f98c5578edcd1/util/types/schemas/blockchain.mol
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2357  100  2357    0     0  21623      0 --:--:-- --:--:-- --:--:-- 21824
moleculec --language c --schema-file build/blockchain.mol > build/blockchain.h
/bin/sh: 1: moleculec: not found
make: *** [Makefile:30: build/blockchain.h] Error 127
error: process exit with code Some(2)
username@dev1:~/project$ capsule --version
Capsule 0.4.1 303dfcb
username@dev1:~/project$ ckb-cli --version
ckb-cli 0.35.0 (3875086 2020-08-11)username@dev1:~/project$ cd ..
username@dev1:~$ rm -rf project
username@dev1:~$ RUST_BACKTRACE=1 RUST_LOG=capsule=trace capsule new project
New project "project"
Created file "capsule.toml"
Created file "deployment.toml"
Created file "README.md"
Created file "Cargo.toml"
Created file ".gitignore"
Initialized empty Git repository in /home/username/project/.git/
Created "/home/username/project"
Created tests
     Created library `tests` package
New contract "project"
     Created binary (application) `project` package
Rewrite Cargo.toml
Rewrite ckb_capsule.toml
Done
username@dev1:~$ cd project/
username@dev1:~/project$ RUST_BACKTRACE=1 RUST_LOG=capsule=trace capsule new-contract sharedlib --template c-sharedlib
Setup C environment
Cloning into '/home/username/project/contracts/c/deps/ckb-c-stdlib'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 205 (delta 1), reused 4 (delta 1), pack-reused 196
Receiving objects: 100% (205/205), 108.79 KiB | 1.29 MiB/s, done.
Resolving deltas: 100% (93/93), done.
New contract "sharedlib"
Rewrite Makefile
Rewrite ckb_capsule.toml
Done
username@dev1:~/project$ RUST_BACKTRACE=1 RUST_LOG=capsule=trace capsule build
Building contract project
    Updating crates.io index
   Compiling cfg-if v0.1.10
   Compiling cc v1.0.41
   Compiling buddy-alloc v0.3.0
   Compiling molecule v0.6.0
   Compiling ckb-allocator v0.1.1
   Compiling ckb-standalone-types v0.0.1-pre.1
   Compiling ckb-std v0.6.3
   Compiling project v0.1.0 (/code/contracts/project)
    Finished dev [unoptimized + debuginfo] target(s) in 6.72s
Building contract sharedlib
curl -L -o build/blockchain.mol https://raw.githubusercontent.com/nervosnetwork/ckb/d75e4c56ffa40e17fd2fe477da3f98c5578edcd1/util/types/schemas/blockchain.mol
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2357  100  2357    0     0   5572      0 --:--:-- --:--:-- --:--:--  5558
moleculec --language c --schema-file build/blockchain.mol > build/blockchain.h
/bin/sh: 1: moleculec: not found
make: *** [Makefile:30: build/blockchain.h] Error 127
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: process exit with code Some(2)

Stack backtrace:
   0: ckb_capsule::util::cli::run
   1: <ckb_capsule::recipe::c::C<R> as ckb_capsule::recipe::Recipe>::run_build
   2: capsule::run_cli
   3: capsule::main
   4: std::sys_common::backtrace::__rust_begin_short_backtrace
   5: std::rt::lang_start::{{closure}}
   6: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/core/src/ops/function.rs:259
      std::panicking::try::do_call
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/std/src/panicking.rs:381
      std::panicking::try
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/std/src/panicking.rs:345
      std::panic::catch_unwind
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/std/src/panic.rs:382
      std::rt::lang_start_internal
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/std/src/rt.rs:51
   7: main
   8: __libc_start_main
   9: _start', src/bin/capsule.rs:404:17
stack backtrace:
   0: rust_begin_unwind
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/std/src/panicking.rs:483
   1: core::panicking::panic_fmt
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/core/src/panicking.rs:85
   2: core::option::expect_none_failed
             at /rustc/7f7a1cbfd3b55daee191247770627afab09eece2/library/core/src/option.rs:1234
   3: capsule::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

You can execute the following command to install moleculec:
cd contracts/c && make install-tools
Would you like to contribute a document about setup C contracts?

Would you like to contribute a document about setup C contracts?

Sure, whatever I can do to help.

authors: jjyr jordanmack

repo: nervosnetwork/capsule


text:
fix: skip asking proof if no updates for the last state

Skip asking proof from a peer if its proved last block number is not greater its current last block number.
Skip sampling blocks if there no enough blocks, just take all blocks.

Codecov Report
Base: 34.27% // Head: 35.10% // Increases project coverage by +0.83% :tada:

Coverage data is based on head (1c53208) compared to base (4962850).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop      #66      +/-   ##
===========================================
+ Coverage    34.27%   35.10%   +0.83%
===========================================
  Files           22       22
  Lines         3475     3475
  Branches       489      490       +1
===========================================
+ Hits          1191     1220      +29
+ Misses        2164     2135      -29
  Partials       120      120

Flag
Coverage Δ

unittests
35.10% <100.00%> (+0.83%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/protocols/mod.rs
0.00% <ø> (ø)

src/protocols/light_client/mod.rs
15.93% <100.00%> (+15.93%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

authors: codecov-commenter yangby-cryptape

repo: nervosnetwork/ckb-light-client


text:
Some lines seem to be unreachable code.

I have run fuzz tests again and again, the following if condition is never entered:
https://github.com/nervosnetwork/merkle-mountain-range/blob/8206b586c3534cce2a4ddbd0ffbc5e2a9ce04727/src/mmr.rs#L404-L406
I could NOT prove but I think it is unreachable.

it's reachable, I added an unit test: https://github.com/nervosnetwork/merkle-mountain-range/pull/36

Yes, better to return CorruptedProof directly, and I noticed that line 408 checked the iterator empty again, so these lines can be deleted directly, cc @jjyr

authors: quake yangby-cryptape

repo: nervosnetwork/merkle-mountain-range