text:
Function Calling Fine-tuning Support

Dear trl team,
As function calling fine-tuning becomes increasingly important for teaching language models to use tools effectively in agentic workflows, I'm pleased to see Hugging Face has unified the encoding and processing of various tool-calling messages through chat templates (https://huggingface.co/blog/unified-tool-use) .
I would like to inquire about trl's current or planned support for function calling fine-tuning. I haven't found related documentation in SFTTrainer (the conversation data format fields only include user and assistant roles). I believe supporting function calling would require at least these features:

Support for processing and encoding messages with user, assistant, and tool roles (already supported by the tokenizer library)
Loss calculation specifically for assistant parts (including both general messages and tool-calling messages)

Alternatively, is it possible to implement function calling fine-tuning using existing SFTTrainer features? For example, through a custom collate_fn similar to vlm_sft?
Thank you for your clarification!

Thanks for suggesting this enhancement @jc-ryan. I agree it would be a great addition. cc @Rocketknight1

Support for processing and encoding messages with user, assistant, and tool roles (already supported by the tokenizer library)

It would probably be necessary to update the functions in trl/data_utils.py to include tool arg. This could be done as a first step.

Loss calculation specifically for assistant parts (including both general messages and tool-calling messages)

Can you elaborate on this part? Should the loss be calculated differently for the tool part?

Alternatively, is it possible to implement function calling fine-tuning using existing SFTTrainer features? For example, through a custom collate_fn similar to vlm_sft?

I'm not sure I understand this point. With SFT at least, training is offline, so you shouldn't have to call the tools, right?

Loss calculation specifically for assistant parts (including both general messages and tool-calling messages)"

@qgallouedec Sorry for not being clear earlier. Regarding loss calculation, for tool using, there are several role messages including system, user, assistant (tool_call), tool (tool call results), and assistant (plain text). We should mainly calculate the loss for the assistant (tool_call) and assistant (plain text) parts while masking the loss for other roles.
For the specific data format, you can refer to mistral: https://docs.mistral.ai/capabilities/finetuning/#2-function-calling-instruct

Alternatively, is it possible to implement function calling fine-tuning using existing SFTTrainer features? For example, through a custom collate_fn similar to vlm_sft?

This part means I noticed that in the SFTTrainer documentation, you can write custom collate_fn for vlm_sft to support multimodal model input formats and loss masking (https://huggingface.co/docs/trl/sft_trainer#a-custom-collator-for-processing-multi-modal-data). So I'm not sure if this same approach can be applied to handle function calling data.

Thanks for the clarification. Yes, you're right, the loss is calculated over the entire sequence by default (assistant and user). I think it's ok as a first approach, even if it's probably better to hide these parts in the loss.
For any contributor who wants to tackle this improvement, I suggest

update the functions in trl/data_utils.py to include tool arg.
update the trainers one by one to include tooling in their preprocessing stage (start with SFT)
find a clean way to hide the “tool” part of the assistant in the loss (data collator isn't IMHO). As a preliminary step, I'd also say prove that it's really worth doing.

@qgallouedec
Let's clarify before proceeding with the PR.
given a dataset with three roles: user, assistant, and tool. The goal is to get the SFTtrainer and others to use a list of functions and feed that into the tokenizer (or processor). This will allow create a system prompt that tells the model which functions it has access to and how to use them.
and in data_utils.py, we’ll need to implement a function that checks whether the dataset uses tools. After that, we can modify the SFTtrainer and (other scripts) to preprocess all that before training.

Support for processing and encoding messages with user, assistant, and tool roles (already supported by the tokenizer library)

update the functions in trl/data_utils.py to include tool arg.
update the trainers one by one to include tooling in their preprocessing stage (start with SFT)

Here's something I don't get. given a conversational dataset with 3 roles of user, assistant and tool like this dummy dataset I made from the blog the tools dataset can be pre-processed like this:
from datasets import load_dataset
dataset = load_dataset("August4293/tool_dummy_dataset")
tools = [get_current_temperature, get_current_time] #functions for the model to call
processed_text = processor.apply_chat_template(
    chat,
    tools=tools
)

which results in:
<|begin_of_text|><|im_start|>system
You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> {"type": "function", "function": {"name": "get_current_temperature", "description": "get_current_temperature(location: str) - Gets the temperature at a given location.

    Args:
        location(str): The location to get the temperature for, in the format "city, country"", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \"city, country\""}}, "required": ["location"]}}
{"type": "function", "function": {"name": "get_current_time", "description": "get_current_time(location: str) - Gets the current time at a given location.

    Args:
        location(str): The location to get the time for, in the format "city, country"", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the time for, in the format \"city, country\""}}, "required": ["location"]}} </tools>Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}}
For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
<tool_call>
{"name": <function-name>, "arguments": <args-dict>}
</tool_call><|im_end|>
<|im_start|>user
Hey, what's the weather like in Paris right now?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "get_current_temperature", "arguments": {"location": "Paris, France"}}
</tool_call><|im_end|>
<|im_start|>tool
<tool_response>
22.0
</tool_response>
<|im_end|><|im_start|>assistant<|im_end|>

if it's already supported by the tokenizer library, why do we need to modify SFTTrainer to preprocess?

authors: August-murr jc-ryan qgallouedec

repo: huggingface/trl


text:
fix gradient checkpointing when using PEFT

What does this PR do?

Enables proper handling of gradient_checkpointing_kwargs when using PEFT without the quantization, such as PEFT+DeepSpeed.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
[FEATURE] Why home page only render pages that type is posts

Describe the feature you want 描述你的功能需求
A website may have many type pages, such as blogs, notes, etc...
But now in layouts/index.html limit the page type:
      {{- $pages := where .Site.RegularPages "Type" "posts" -}}

So if I have a folder /content/notes to store my note, it will not show in home page, only if I manully set front matter type: posts.
I don't think that is a good design, how do you think about that?
Useful reference 有价值的参考
I think there are 2 better design:

Show all pages not only posts by change above code to:

{{- $pages :=  .Site.RegularPages -}}

Give a config to set what types or sections will show in home page, e.g.

{{- $pages := where .Site.RegularPages "Type" "in" site.Params.mainSections -}}

@dilless Thanks for your suggestions at first!
Usually, as a blog-type site, the home page will display the content of the post instead of the page, and the same is true for hexo, wordpress, etc. FixIt's predecessor, LoveIt, did the same design, and I think it's reasonable, so I'm not going to change it.

However, if you just want to keep your blog post links in their original format, like /notes/xxx/, maybe you could simply modify the permalinks in the config file:
# Permalinks config (https://gohugo.io/content-management/urls/#permalinks)
[Permalinks]
  posts = "notes/:filename"
  # or posts store in /notes
  notes = "posts/:filename"

Of course, if you have to modify it, you could create a file with the same path on your blog to override the design of theme.

@Lruihao Thanks for your comment. I will solve this problem with your suggestion.

source: gharchive/issue

authors: Lruihao dilless

repo: hugo-fixit/FixIt

license: MIT


text:
feat(i18n): add French translations

Hi,
Here is a proposal for the French translations.
Let me know

Thank you!

authors: frajibe razonyang

repo: hugomods/search


text:
PDF not loading

What happened?
Hi, I have tried your plugin, and it works well in most cases, except when I try to view a pdf.
How to reproduce the bug
The loading indicator is spinning indefinitely.
I have tried the following solution, and it works, although I am sure it can be done better (I'm not a front-end guy):
 @elseif ($mediaType === 'pdf')

        <iframe x-ref="mediaFrame" class="rounded-lg" src="{{ $media }}" type="application/pdf" width="100%" height="2000" @load.window="loading = false"></iframe>

Package Version
3.1.0.0
PHP Version
8.2
Laravel Version
10
Which operating systems does with happen with?
macOS
Notes
No response

Hi,
I don't have any problems with PDFs, maybe it's due to the fact that you're on Mac with Safari I suppose? I have no way of testing on Safari.
In any case, I've made some changes to the plugin and released a new release. Can you tell me if this has corrected your problem?
Release : v3.1.0.1

Hi, I have updated the package, but the PDF is still not loading, just the loading indicator is spinning.
Indeed, I am on a Mac, but on Chrome.
In Firefox the PDF is loaded, but the PDF does not take the full height of the modal (I use ->slideOver())

Ok I managed to reproduce the problem on chrome, I corrected it and it works correctly for me now. Can you update the package and confirm that the problem is solved?
Also, I've slightly reworked the style for the PDFs so that they take up the correct height.
Release : v3.1.0.2

Still not loading, now I get "No preview available", and sometimes the loading indicator is just spinning

I've updated the package again, and now I have no problems with either Chrome or Firefox, and the PDF loads perfectly. Is it good for you?

Nice! Glad you like it, thanks for your issue and sorry for the inconvenience.

authors: hugomyb molcsab

repo: hugomyb/filament-media-action


text:
Mexc Connector Bug Report

Describe the bug
There is an issue observed with MEXC connector, if the order was partially filled and Humming bot canceled it - then it cannot fetch status update on this order.
Failed to cancel order <order id> error appears in log
Error fetching status update for the lost order ... Partially Cancelled
I would appreciate if anyone can fix it.
MEXC is growing exchange and connector deserves to be GOLD :)
Steps to reproduce

connect to MEXC
using pure_market_making create some orders on any paid
wait when one of order will be partially filled and bot will attempt to cancel it
-> here comes the error

Release version
1.21.0
Type of installation
Docker
Attach required files
logs_MM_JOY_test.log

@rapcmia @fengtality @nikspz I talked with @leetjoy and he wants to fund this bounty. I think that has to be considered as P2 and the solution should include:

cancel order method --> Review how the order is processed when sending the cancelation. (specially when having partial fills)
evaluate the solution proposed in this PR https://github.com/hummingbot/hummingbot/pull/6584/files for the methods:
_is_order_not_found_during_status_update_error
_is_order_not_found_during_cancelation_error

Once the connector solution is ready we have to validate it by running a bot for > 1 day without issues

https://hummingbot.org/bounties/sponsors/
Okay, then to validate this as P2 Bounty (50k HBOT) it should be funded with 55k HBOT (50k +10%) or 133USDC (111 USDC +20%) using these wallets:
https://hummingbot.org/bounties/sponsors/#foundation-bounty-wallets

@nikspz can I fund this bounty by sending USDC from CEX?

@leetjoy For BEP20, 0xE517b826a26B439eCE92f3220628eC007049d915 can be used for USDC.

hi @leetjoy Are you still willing to fund the bounty?

hi @leetjoy Are you still willing to fund the bounty?

bounty funded
tx: 0xf3868f71b20055aadc03a1c136f7f9c670b3b38367fe513091aff1a9b969f14e

Severity: P2
Bounty: 111 USDC (Developer portion)
Rationale:

Related to the MEXC exchange connector issue
MEXC connector fails to fetch status updates for partially filled and canceled orders, resulting in log errors
Please read issue description as the reference

Bounty Opened for developers who want to be assigned

@yancong001 this might be of your interest

ok, i will fix this bug

here https://github.com/hummingbot/hummingbot/pull/6691

Fix has been deployed to release version 1.23.0

thank you guys, will test this out.

@nikspz
how to update to version 1.23?

@nikspz
how to update to version 1.23?

https://hummingbot.org/installation/post-installation/#updating-hummingbot

authors: cardosofede leetjoy nikspz rapcmia yancong001

repo: hummingbot/hummingbot


text:
(feat) Introduces the CoinCap RateOracle rate source

Before submitting this PR, please make sure:

[X] Your code builds clean without any errors or warnings
[X] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:
This PR is associated to bounty issue: https://github.com/hummingbot/hummingbot/issues/6517.
This PR introduces the CoinCap rate source for the rate oracle. It is possible to obtain an API key here.
CoinCap adds an alternative to CoinGecko, and its API allows for price streaming, which is included in this PR.
The PR had to make significant changes to the configuration modules in order to allow configuring a secret API key in the client config map.
Tests performed by the developer:

Unit tests added.
Manually tested the functionality.

Tips for QA testing:
Obtain an API key here and run a full test on the functionality of the rate provider. Please not that the global token must be configured to USD in order for rate --token {TOKEN} to work. That is because CoinCap only supports USD as a quote, and that rate functionality cannot perform conversions. However, the rate --pair {BASE}-{QUOTE} will work with pairs that don't include USD as quote.

Closing for now to pass CoinAlpha QA


text:
(fix) account for single order level in linear distribution

Before submitting this PR, please make sure:

[X] Your code builds clean without any errors or warnings
[X] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:

accounts for case where n_levels equals 1 in linear distribution

@fengtality the solution is good and we should test if has to be replicated in other distributions, but you need to add a test to cover that line that you are adding

authors: cardosofede fengtality


text:
triangle storage 64bit vs 32bit musings

So I threw an obj file with 16m vectors/faces at PBR for fun, and thus I spent today's hack time getting more familiar with golang's memory profiling.
There's a large amount of GC (space) overhead as it builds things, but lets put that aside for now since fixing the GC usage still doesn't allow loading the file and I can PR those fixes without discussion.
surface.Triangle takes up quite a bit of memory: 11 vectors + material + bounds. In my test file normals and textures don't even exist for these particular triangles.
I considered turning Triangle.Normals/Textures into pointers (which would slow down processing), or maybe something like a SimpleTriangle and fiddling with interfaces where normals/textures actually exist... but I noticed in every obj file I've looked at all of the numbers are float32 safe.
Do you know if this is true in a broader case? It makes sense for a lot of the internal numbers to be 64bit, especially the samples which can be large averaged numbers, but it might be a huge savings (and potential performance boost as it should reduce memory bandwidth) to use 32bits for the underlying data where possible.
Sadly, it's not an easy change. for shiggles I started switching out 64's for 32's but there's a lot of it, and preserving 64 where it should be 64 might require adding a vec64 structure since vec is nice and cleanly reused everywhere :)
Also, sadly, it looks pretty fiddly in go to make 32 vs 64 a compile time option :( Haven't read through all the possibilities yet.
At any rate, wondering if you've had any thoughts along these lines? I can see via the googles that it's a big area of discussion, though some notes about GPU acceleration being better at 32bit.
thanks!

That's a super neat thought; in another program I'd say 32 vs 64 is a minor YAGNI optimization but since we're talking about trillions of operations on millions of triangles, halving their space might lead to a significant speedup.
There's no getting around the loss in precision, but I wonder if that loss would be visually apparent. It might be worth doing an absolutely rough first cut of find-replacing all the float64s to 32s, just to see what happens.

authors: dormando hunterloftis

repo: hunterloftis/pbr


text:
Thank you for your contribution and we look forward to your future updates！

Thank you for your contribution and we look forward to your future updates！

Thx. Feel free to open an issue or submit a pull request for any improvements or new features you'd like to see.


text:
🛑 Split a bill is down

In ca84e19, Split a bill (https://bill.thekor.eu/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Split a bill is back up in 565f4bd after 1 day, 22 hours, 26 minutes.

repo: hupratt/upptime


text:
Make clusters clickable hurricane-response/response-theme#22

Description

Addresses issue #22

Makes clusters clickable - both on the shelters page and the distribution points page.
Code largely adapted from mapbox cluster documentation. Eventually, it would be nice to de-dupe code between those two layout pages.
On click, recenters map on clicked cluster centroid and zooms until the cluster splits up (either into more clusters or individual points).
Renamed ZOOM to INITIAL_ZOOM so it's obvious that variable is only used for initialization and doesn't store the state of the current zoom. Cleaned up some whitespace in the file.

List of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] This pull request relates to a ticket.
[x] This code is tested.
[x] I want my code added to the response theme.

@nihonjinrxs not sure if you're the right person to tag - any idea on timeline for feedback here?

repo: hurricane-response/response-theme


text:
Allow PHP 8.1

Once hussainweb/drupalqa#6 is complete, allow php8.1 is a valid input for php-version.
—
Created via Raycast

Available in v1.2.0

repo: hussainweb/drupalqa-action


text:
fix issue with setting cookies for wrong domain

I had a problem with cookies being set for instagram.com instead of i.instagram.com, this line fixes the issue.

Thank you

authors: IvanMMM marek-vybiral

repo: huttarichard/instagram-private-api


text:
could you update the demo to show a set zoom level?

It seems too challenging to achieve a simple task: fire up a viewcontroller, set an imagescrollview with a preset zoom and offset.
I know i had it working before, but at present it doesn't seem to want to work.
thank you, drew..

I found the copy i had where i had the initial VC display showing a zoomed offset image, and kick me in the ass for not copying that folder before tweaking it.
The challenge with the demo is that the pod project needs a cleaning before any changes to it manifest in the running of the demo. So until you realize this, NO code changes are touched.  So.. that means that i don't know what code was being run that allowed the initial view to show a zoomed control.
As soon as i cleaned the project, the desired result was lost.  Back to square one.

The line in private func configureImageForSize(_ size: CGSize), is the culprit:
zoomScale = minimumZoomScale
Commenting it out provides proper behaviour.

Currently, I don't have a good solution for this issue. If you set initial zoomScale, it will be reset by the first orientation change. So a workaround solution is set imageScrollViewDelegate (that I have just added, am going to pushed). In this function of delegate func imageScrollViewDidChangeOrientation(imageScrollView: ImageScrollView), set your initial zoomScale.
imageScrollView.imageScrollViewDelegate = self

extension ViewController: ImageScrollViewDelegate {
	func imageScrollViewDidChangeOrientation(imageScrollView: ImageScrollView) {
               imageScrollView.zoomScale = 2
   	}
}

I've just updated version 1.7, that has delegate to do that.

This would force a scale of 2 with each change, with negates the base use-case for the repo.  I have a solution in play and will revisit this as the project matures a bit more.

Could you please share you solution?

authors: huynguyencong wdcurry

repo: huynguyencong/ImageScrollView

license: mit

license_source: bigquery


text:
chore: NebulaGraph prompt optmization

Was preparing for a demo project of NebulaGraphQAChain to find out the prompt needed to be optimized a little bit.
Please @hwchase17 kindly help review.
Thanks!

Thanks @hwchase17 for the blazing fast merge!


text:
Logic layer config file aliasing

Example config:
{
    "aliases": {
        "cubes": [
            {
                "name": "example",
                "cube": "Example"
            }
        ]
    }
}

Just want to check: this allows multiple aliases per cube?

Not currently... I'll add that.

@hwchen new example:
{
    "aliases": {
        "cubes": [
            {
                "name": "Example",
                "alternatives": ["ex1", "ex2"]
            }
        ]
    }
}

looks good, thanks!

authors: MarcioPorto hwchen

repo: hwchen/tesseract


text:
Opposite parallax?

Hi!
This is a great plugin and I have it all working which is good but at the moment the image I have the parallax on is moving TOWARDS the mouse at the moment and I want it to move AWAY from the mouse move (like in the opposite direction) if you get me?
Is this possible? And if so, how do I go about doing it as I am at a total loss...
Any help would be greatly appreciated.
Thank you.

just use negative values and it will move in opposite of mouse.

On May 31, 2018, at 10:19, Emma Dalby notifications@github.com wrote:
Hi!
This is a great plugin and I have it all working which is good but at the moment the image I have the parallax on is moving TOWARDS the mouse at the moment and I want it to move AWAY from the mouse move (like in the opposite direction) if you get me?
Is this possible? And if so, how do I go about doing it as I am at a total loss...
Any help would be greatly appreciated.
Thank you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Ahhh.. thank you so much!! I have been looking at this for way too long and must of missed that.
Thanks so much for the quick reply too man I really appreciate it.
Kind Regards,
Emma Dalby
digital by default.
http://www.wearedigitalbydefault.co.uk/ www.wearedigitalbydefault.co.uk
01283 791007
From: Felix Pflaum notifications@github.com
Sent: 31 May 2018 16:24
To: hwthorn/parallaxify parallaxify@noreply.github.com
Cc: Emma Dalby emma@wearedigitalbydefault.co.uk; Author author@noreply.github.com
Subject: Re: [hwthorn/parallaxify] Opposite parallax? (#16)
just use negative values and it will move in opposite of mouse.

On May 31, 2018, at 10:19, Emma Dalby <notifications@github.com mailto:notifications@github.com > wrote:
Hi!
This is a great plugin and I have it all working which is good but at the moment the image I have the parallax on is moving TOWARDS the mouse at the moment and I want it to move AWAY from the mouse move (like in the opposite direction) if you get me?
Is this possible? And if so, how do I go about doing it as I am at a total loss...
Any help would be greatly appreciated.
Thank you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/hwthorn/parallaxify/issues/16#issuecomment-393567286 , or mute the thread https://github.com/notifications/unsubscribe-auth/Aj4KrSNtNLmOqwflLx8YgIyQe9ZnGKo6ks5t4Ar7gaJpZM4UVPV8 .  https://github.com/notifications/beacon/Aj4KrSeAlhmRt3of3W4z_0gffs9_yoIWks5t4Ar7gaJpZM4UVPV8.gif

Replied you on email already emma, not exactly it goes with opposite direction. but i think try this to get some idea http://jsfiddle.net/X7UwG/854/ and maybe you can achieve with this plugin as well.

Thanks for the help guys. It's sorted now and looks great! :)

Thanks to a previos ticket, here's the code:
  // defining your own positioning function
  $.parallaxify.positionProperty.inverted = {
      setPosition: function($element, left, originalLeft, top, originalTop) {
        $element.css('transform', 'translate(' + (originalLeft - left) + 'px, ' + (originalTop - top) + 'px)');
      }
  };
  // using your own positioning function
  $('#parallax').parallaxify({
      positionProperty: 'inverted',
      responsive: true,
      motionType: "natural",
      mouseMotionType: "gaussian",
      motionAngleX: 90,
      motionAngleY: 90,
      alphaFilter: 0.5,
      adjustBasePosition: true,
      alphaPosition: 0.025
  });

Hi @miroamarillo
Sorry, only just seen this. I just used negative values instead of positive ones as suggested above by @hwthorn
Worked great.

@emmadalby Thanks!

No problem 😊

authors: emmadalby hwthorn miroamarillo yeomann

repo: hwthorn/parallaxify


text:
Emply div with height and padding

Problem:
There are empty div as the above image shown when hideEyeDrop, hideAdvancedSliders, hideColorGuide, hideInputType, hideColorTypeBtns are selected.
Expect:
The div would disappear ("display: none").

fixed in 2.2.24

Fixed in 2.2.24


text:
industrial-edge-datacenter and industrial-edge-factory applications show "out of sync" in cluster-wide GitOps instance

The two applications named above show out of sync perpetually in ArgoCD.  This is because there are RoleBindings and ClusterRoleBindings that are part of the installation that are managed by the operators and are not represented in the GitOps structure.

Closing.  This should be fixed

authors: claudiol mhjacks

repo: hybrid-cloud-patterns/industrial-edge


text:
Fix docs for Application secret type

Please describe the changes this PR makes and why it should be merged:
The documentation for the OAuth2 application were stating the secret is a boolean, which does not make sense. They now say it is a string as supposed to be.
Semantic versioning classification:

[ ] This PR changes the library's interface (methods or parameters added)

[ ] This PR includes breaking changes (methods removed or renamed, parameters moved or removed)

[x] This PR only includes non-code changes, like changes to documentation, README, etc.

PR 👨🌾 🌱

👨🌾🌼✨

authors: iCrawl robflop


text:
feat-bbio-card-emulation-iso-14443-a-raw-mode

BBIO card emulation ISO 14443-A - RAW MODE

From the PPS, we can receive/send raw TPDU command instead of APDUs.
In a next version, we will add the RATS as well

An example is given in contrib/bbio_hydranfc/bbio_hydranfc_card_emulator_iso_14443_a_raw_example.py

I see RFAL is modified (API added) please add a comment (of major modification) in file https://github.com/hydrabus/hydrafw_hydranfc_shield_v2/blob/master/src/hydranfc_v2/rfal/STSW-ST25RFAL002.txt
Note: I plan to fix build warning too (before to release v0.3)

Remarks taken into account
hydranfc_v2_bbio_card_emulator.X aim is to emulate a card, but not in stand alone mode: the response is chosen by the terminal connected to the hydra. This is why it is not in the same file as hydranfc_v2_bbio_c.X.

authors: bvernoux gvinet


text:
[Bug]: PRIMARY KEY and UNIQUE INDEX not always respected

What's wrong?
I use the latest container version of hydra (https://github.com/orgs/hydradatabase/packages/container/hydra/139632640?tag=14-22e7434a7e34d12a57a346e5d6c1d74da75f5046).
I have a simple columnar table containing metric data. This data consists of multiple varchar fields (basically segments of the identifier of the metric) and a float8 value.
I was looking for a way to prevent duplicate insertion of the same metric. Therefore I added an other column containing an int8 value, which is a 64-bit hash of the varchar fields (concatenated) mentioned above. My idea was to set a primary key or unique index on the int8 column to prevent multiple inserts for the same metric.
Unfortunately, this did not work out as expected. I add ~40000 metric with copy from. Then I wait a little bit and run the same copy from again. Sometimes it fails (with key violation, as expected), sometimes it is successful. After successful runs, I can verify the result by querying for one of the hash values and I do get back two (or more, if I ran the copy from more than 2 times) identical rows, which should by definition not be possible (also the size of the table does increase, so cleary the data has been stored multiple times).
This leads me to two questions:

Is this expected behavior or is this a bug?
What is the recommended way with Hydra to prevent duplicate entries (what a primary key would normally guarantee)?

hi @breml,
are you able to provide a sample schema and constraints (create statements) that you are able to reproduce this on?  that would help quite a bit!
thanks

Hi @JerrySievert
The DB schema looks like this (with primary key):
CREATE TABLE metric (
	hash int8 NOT NULL,
	key_part1 varchar NOT NULL,
	key_part2 varchar NOT NULL,
	key_part3 varchar,
	key_part4 varchar,
	key_part5 varchar,
	key_part6 varchar,
	key_part7 varchar,
	value float8 NOT NULL,
	PRIMARY KEY (hash)
) USING columnar;

and with unique key:
CREATE TABLE metric (
	hash int8 NOT NULL,
	key_part1 varchar NOT NULL,
	key_part2 varchar NOT NULL,
	key_part3 varchar,
	key_part4 varchar,
	key_part5 varchar,
	key_part6 varchar,
	key_part7 varchar,
	value float8 NOT NULL,
) USING columnar;
CREATE UNIQUE INDEX unique_metrics_idx ON metric (hash);

In both cases, I observed the same behavior. The value for murmur_hash is calculated externally from the DB on the application side. The hash is formed on the concatenated string (separated by a separator) of all the fields except the value (see below pseudo code):
hash(join([]strings{key_part1, key_part2, key_part3, key_part4, key_part5, key_part6, key_part7}, separator))

The data is inserted in batches with copy from. For now I work with a small data set of ~40k rows. After the initial data load, I retried. Sometimes all batches fail on the second run (expected), sometimes only some batches fail (not expected), sometimes all batches are re-added successfully (not expected). In the "failing" cases, I can see, that the additional data is in fact in the table and that I get multiple rows with the same value for hash even though there is a primary key / unique key constraint set on hash.
In both scenarios I observed,

if I'm reading this correctly, you're generating insert statements in your application that are then being run as \copy from?
is there any chance you have an example of the failure specifically that you can share?

I think, there are no insert statements involved. The application is written in Go and I am using github.com/jackc/pgx/v5 as the DB driver. I am using the pgx.CopyFromRows function, which basically reads from a 2-dimensional slice (array) in memory. The DB driver is using the PGCOPY\n\377\r\n\000 sequence (https://github.com/jackc/pgx/blob/master/copy_from.go#L143), so I guess this COPY is used (see https://www.postgresql.org/docs/current/sql-copy.html, section Binary Format).

I've tried to break it down to its constituent pieces:
CREATE TABLE metric (
  hash int8 NOT NULL,
  PRIMARY KEY(hash)
);

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/jackc/pgx/v5"
)

func main() {
	// urlExample := "postgres://username:password@localhost:5432/database_name"
	conn, err := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
	if err != nil {
		fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
		os.Exit(1)
	}
	defer conn.Close(context.Background())

	rows := [][]interface{}{
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
		{int32(36)},
		{int32(29)},
	}

	copyCount, err := conn.CopyFrom(
		context.Background(),
		pgx.Identifier{"metric"},
		[]string{"hash"},
		pgx.CopyFromRows(rows),
	)

	if err != nil {
		fmt.Fprintf(os.Stderr, "error: %v\n", err)
		os.Exit(1)
	}

	fmt.Fprintf(os.Stdout, "%d rows copied\n", copyCount)
}

but while copy relaxes some constraints (CHECK specifically), it does not remove uniqueness, and I have not been able to make it fail on heap nor columnar tables.
are you able to provide a reproduction of the issue, as I apparently am not able to reproduce locally?

Hi @JerrySievert
Thanks for the minimal example. With your code, I am not able to produce the issue (more about why I think this is the case further down).
I already worked on something similar. I now have an example, that fails reproducible on my setup (more about the setup further down, since now I think there might be some relevance).
Go Code
package main

import (
	"context"
	"flag"
	"fmt"
	"os"
	"strconv"

	"github.com/brianvoe/gofakeit/v6"
	"github.com/jackc/pgx/v5"
	"github.com/spaolacci/murmur3"
)

var (
	dropTable            = flag.Bool("drop", false, "drop the table before creating it")
	createTablePK        = flag.Bool("create-table-pk", false, "create the table with pk before inserting")
	createTableUniqueKey = flag.Bool("create-table-unique-key", false, "create the table with unique key before inserting")
	rows                 = flag.Int("rows", 1000, "number of rows per batch to insert")
)

func main() {
	flag.Parse()

	ctx := context.Background()
	conn, err := pgx.Connect(ctx, os.Getenv("DATABASE_URL"))
	die(err)
	defer conn.Close(ctx)

	if *dropTable {
		_, err = conn.Exec(ctx, "drop table if exists metric;")
		die(err)
	}

	if *createTablePK {
		_, err = conn.Exec(ctx, `CREATE EXTENSION IF NOT EXISTS columnar;`)
		die(err)
		_, err = conn.Exec(ctx, `create table if not exists metric (
				hash int8 NOT NULL,
				key_part1 varchar NOT NULL,
				key_part2 varchar NOT NULL,
				value float8 NOT NULL,
				key_part3 varchar,
				key_part4 varchar,
				key_part5 varchar,
				key_part6 varchar,
				key_part7 varchar,
				PRIMARY KEY (hash)
			) USING columnar;`)
		die(err)
	}

	if *createTableUniqueKey {
		_, err = conn.Exec(ctx, `CREATE EXTENSION IF NOT EXISTS columnar;`)
		die(err)
		_, err = conn.Exec(ctx, `create table if not exists metric (
				hash int8 NOT NULL,
				key_part1 varchar NOT NULL,
				key_part2 varchar NOT NULL,
				value float8 NOT NULL,
				key_part3 varchar,
				key_part4 varchar,
				key_part5 varchar,
				key_part6 varchar,
				key_part7 varchar
			) USING columnar;`)
		die(err)
		_, err = conn.Exec(ctx, `CREATE UNIQUE INDEX unique_metrics_idx ON metric (hash);`)
		die(err)
	}

	if len(flag.Args()) != 1 {
		fmt.Fprintf(os.Stderr, "Usage: %s <offset>\n", os.Args[0])
		os.Exit(1)
	}

	offset, err := strconv.Atoi(flag.Args()[0])
	if err != nil {
		fmt.Fprintf(os.Stderr, "Unable to convert offset to int: %v\n", err)
		os.Exit(1)
	}

	idx := offset * *rows
	key_elements := []string{"aaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "cccccccccccccccccccccc", "ddddddddd", "eeeeeeeeeeeeeee", "fffff", "gggg", "hhhhhhhhhhhhhhhhh", "iiiiiiiii", "jjjjjjjjjjjjjjjjjjjjjj"}

	metricRows := make([][]any, 0, *rows)
	for row := 0; row < *rows; row++ {
		key_part1 := key_elements[idx%10]
		key_part2 := key_elements[(idx/10)%10]
		key_part3 := key_elements[(idx/100)%10]
		key_part4 := key_elements[(idx/1000)%10]
		key_part5 := key_elements[(idx/10000)%10]
		key_part6 := key_elements[(idx/100000)%10]
		key_part7 := key_elements[(idx/1000000)%10]

		metricRow := []any{
			hash(key_part1, key_part2, key_part3, key_part4, key_part5, key_part6, key_part7),
			key_part1,
			key_part2,
			float64(gofakeit.Number(0, 100)),
			key_part3,
			key_part4,
			key_part5,
			key_part6,
			key_part7,
		}
		metricRows = append(metricRows, metricRow)
		idx++
	}

	_, err = conn.CopyFrom(
		ctx,
		pgx.Identifier{"metric"},
		[]string{
			"hash",
			"key_part1",
			"key_part2",
			"value",
			"key_part3",
			"key_part4",
			"key_part5",
			"key_part6",
			"key_part7",
		},
		pgx.CopyFromRows(metricRows),
	)
	if err != nil {
		fmt.Printf("failed: %v\n", err)
	}
}

func hash(key_part1, key_part2, key_part3, key_part4, key_part5, key_part6, key_part7 string) int {
	separator := []byte{124}

	hasher := murmur3.New64()
	hasher.Write([]byte(key_part1))
	hasher.Write(separator)
	hasher.Write([]byte(key_part2))
	hasher.Write(separator)
	hasher.Write([]byte(key_part3))
	hasher.Write(separator)
	hasher.Write([]byte(key_part4))
	hasher.Write(separator)
	hasher.Write([]byte(key_part5))
	hasher.Write(separator)
	hasher.Write([]byte(key_part6))
	hasher.Write(separator)
	hasher.Write([]byte(key_part7))

	return int(hasher.Sum64())
}

func die(err error) {
	if err != nil {
		panic(err)
	}
}

It might be possible to strip the example further down, but I did not yet try to do so.
Execution
Execution (fish shell syntax):
What this basically does is:

drop table
create table with pk
insert with offset 0
loop 10 times, insert with offsets 1 to 10
again loop 10 times, insert with offsets 1 to 10

Based on the bellows output, it successfully inserts for offset 1 to 5 a second time and only starting from offset 6, the constraint starts to kick in.
$ go run ./cmd/pocs/hydra_pk_issue_177 -drop -create-table-pk 0; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)
7
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)
8
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)
9
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)
10
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)

To me, this looks like a race condition and more specifically, the race condition seams to be related to different DB connections.
Setup
The hydra container (version mentioned in previous comment) is running on a "rather" slow server (AMD Embedded R-Series RX-421BD Radeon R7; 4 cores; 20 GB RAM) and the data is stored on spinning discs (not SSD). I use this system for my lab work. The above Go program is executed on my notebook and the connection to the server is going of the network.
The reason why I am mentioning this is, that with a rather slow system it might be "easier" to hit such race conditions.
Additionally, I played with some of the "parameters". For example if I reduce all the key parts to single letter ("a", "b", etc.), the test does not fail (meaning the duplicate key does kick in immediately). Also if I reduce the size of each batch from 1000 to e.g. 100, it does no longer fail. I also played with the unique key constraint and this one behaves somewhat differently. I was not yet able to find a configuration that fails with the above code (but with my real code I do see the issue).
So again, this might be related to the setup, more specifically to the spinning disk, which take some time to persist the data (but this is just wild guessing).
I hope, you find these additional details helpful.

Just as an addition, I simplified the program a little bit, such that there is just one data column instead of 7 key parts.
In this case, I need to add 20000 bytes per row to trigger the failing state (instead of ~200 bytes spread among 7 columns):
package main

import (
	"context"
	"flag"
	"fmt"
	"os"
	"strconv"
	"strings"

	"github.com/jackc/pgx/v5"
	"github.com/spaolacci/murmur3"
)

var (
	dropTable            = flag.Bool("drop", false, "drop the table before creating it")
	createTablePK        = flag.Bool("create-table-pk", false, "create the table with pk before inserting")
	createTableUniqueKey = flag.Bool("create-table-unique-key", false, "create the table with unique key before inserting")
	rows                 = flag.Int("rows", 1000, "number of rows per batch to insert")
	dataBytes            = flag.Int("data-bytes", 20000, "number of data bytes to insert")
)

func main() {
	flag.Parse()

	ctx := context.Background()
	conn, err := pgx.Connect(ctx, os.Getenv("DATABASE_URL"))
	die(err)
	defer conn.Close(ctx)

	if *dropTable {
		_, err = conn.Exec(ctx, "drop table if exists metric;")
		die(err)
	}

	if *createTablePK {
		_, err = conn.Exec(ctx, `CREATE EXTENSION IF NOT EXISTS columnar;`)
		die(err)
		_, err = conn.Exec(ctx, `create table if not exists metric (
				hash int8 NOT NULL,
				data varchar,
				PRIMARY KEY (hash)
			) USING columnar;`)
		die(err)
	}

	if *createTableUniqueKey {
		_, err = conn.Exec(ctx, `CREATE EXTENSION IF NOT EXISTS columnar;`)
		die(err)
		_, err = conn.Exec(ctx, `create table if not exists metric (
				hash int8 NOT NULL,
				data varchar,
			) USING columnar;`)
		die(err)
		_, err = conn.Exec(ctx, `CREATE UNIQUE INDEX unique_metrics_idx ON metric (hash);`)
		die(err)
	}

	if len(flag.Args()) != 1 {
		fmt.Fprintf(os.Stderr, "Usage: %s <offset>\n", os.Args[0])
		os.Exit(1)
	}

	offset, err := strconv.Atoi(flag.Args()[0])
	if err != nil {
		fmt.Fprintf(os.Stderr, "Unable to convert offset to int: %v\n", err)
		os.Exit(1)
	}

	idx := offset * *rows
	data := strings.Repeat("a", *dataBytes)

	metricRows := make([][]any, 0, *rows)
	for row := 0; row < *rows; row++ {
		metricRow := []any{
			hash(strconv.Itoa(idx)),
			data,
		}
		metricRows = append(metricRows, metricRow)
		idx++
	}

	_, err = conn.CopyFrom(
		ctx,
		pgx.Identifier{"metric"},
		[]string{
			"hash",
			"data",
		},
		pgx.CopyFromRows(metricRows),
	)
	if err != nil {
		fmt.Printf("failed: %v\n", err)
	}
}

func hash(key string) int {
	hasher := murmur3.New64()
	hasher.Write([]byte(key))

	return int(hasher.Sum64())
}

func die(err error) {
	if err != nil {
		panic(err)
	}
}

Execution ():
$ go run ./cmd/pocs/hydra_pk_issue_177 -drop -create-table-pk 0; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
6
7
8
9
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)
10
failed: ERROR: duplicate key value violates unique constraint "metric_pkey1" (SQLSTATE 23505)

I was able to reproduce, thank you!
but, your hypothesis about it being because of system speed is not accurate, I was able to reproduce it on my fastest machine with my fastest nvme.  I'll try to hunt down the reasons and get a fix out as soon as I can.

Very good, I am glad you could reproduce the issue.
About the system speed was just an assumption that it is more likely to trigger the issue than on fast machines. But in the end it does not matter to me if this assumption was correct or not, the only thing that matters is, that the issue can be reproduced and finally fixed. I look forward to a fixed version of this.

to put it more succinctly, on a multi-insert (like copy), it was only doing a constraint check on creation of a new stripe.

I've merged #182 which fixes this - so closing.
docker image should now be updated as well.

@JerrySievert Unfortunately, the latest version does not seem to fix the issue completely.
I use image ghcr.io/hydradatabase/hydra:latest, which resolves to
$ kubectl get pod hydra -o json | jq '.status.containerStatuses[0].imageID'
"ghcr.io/hydradatabase/hydra@sha256:9d062d8884af7a3502e7ad6ac640bac989a76c13aff84d8228cbfb4a5d1ec606"

If I execute the above mentioned program again, it got better, since it does no longer fail on every run, but I still observed, that it failed some times. For example I got the following outputs:
$ go run ./cmd/pocs/hydra_pk_issue_177 -drop -create-table-pk 0; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/
$i; end
1
2
3
4
5
6
7
8
9
10
1
2
3
4
5
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)
6
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)
7
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)
8
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)
9
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)                                                                                10
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)

and also:
$ go run ./cmd/pocs/hydra_pk_issue_177 -drop -create-table-pk 0; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end; for i in (seq 1 10); echo $i; go run ./cmd/pocs/hydra_pk_issue_177/ $i; end
1
2
3
4
5
6
7
8
9
10
1
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)
2
failed: ERROR: duplicate key value violates unique constraint "metric_pkey" (SQLSTATE 23505)
3
4
5
6
7
8
9
10

Side question: Is there an easy way to get the version of hydra with SQL, sucht that I can get the version always as first call in my tests to make sure, that the container is updated successfully?

I'd recommend pulling explicit tags, e.g. ghcr.io/hydradatabase/hydra:14-22d647ed4062184698b3a1fa8b37e1cef6ddaff7 to make sure you are using a specific version. These tags correspond to the Postgres and git commit of the build.

authors: JerrySievert breml wuputah

repo: hydradatabase/hydra


text:
Update the French documentation

Pull Request Checklist:

[ ] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes #xyz

[x] (If applicable) Documentation has been added / updated (for bug fixes / features).
[ ] (If applicable) Tests have been added.
[ ] CHANGELOG.rst has been updated (with summary of main changes).

[ ] Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

This is a first batch of translations that were missing from the current documentation.
I've changed the Makefile to create files for the Docstrings, but I haven't gone around to translate them yet.
This PR is only for the translations themselves, but I've used a script to automate most of the process that could be implemented in a GitHub workflow eventually. We'll probably also want to share my script somewhere, to help other users translate their PRs.
This partly addresses #63, but not fully.

Does this PR introduce a breaking change?
Other information:

I don't think that we need to translate the ChangeLog?

@RondeauG
/home/docs/checkouts/readthedocs.org/user_builds/xhydro-fr/checkouts/163/docs/notebooks/climate_change.ipynb:941: WARNING: File not found: 'notebooks/local_Frequency_analysis.ipynb'

@Zeitsperre I see that quite a few translations get ignored. Would you know why?

authors: RondeauG Zeitsperre

repo: hydrologie/xhydro


text:
Unnecessary Title Creation Dialog

When users create a resource, they are prompted for a title, then given the chance to edit it. Can skip the initial extra prompt and go right to resource creation, where a Cancel button should be available:

Skip this dialog

We added this dialog on purpose because as things currently work, once you get to the resource editing page the resource has already been created on disk. This was a way to avoid the creation of a bunch of empty resources.

I agree it would be cool to avoid the title prompt and go straight to the landing page.  When @sblack-usu implemented this, he and I wracked our brains about how to do this, and this was the best solution we came up with.  If there is another way that will quickly get to the landing page in edit mode without creating a resource until a user actually adds content, then that would be better.

Ok I'll have a look and check back.

Related to https://github.com/hydroshare/hydroshare/issues/3415

Note to developer (or me if participating in design) https://github.com/hydroshare/hydroshare/issues/3415 ensure there are no backend remnants as a part of canceling resource creation.

Upon review with the group, this is as designed closing

authors: cuahsimarko dtarb horsburgh martinseul

repo: hydroshare/hydroshare


text:
Discover Paging Not Realistic

Paging can be performed only in very small increments. Getting to the 10th or 100th page requires clicking approximately that many times.

Issue resolved as a part of the Discover redesign

authors: cuahsimarko hydrocheck


text:
4191 meta xml file check fix for check_irods_files

The check for the metadata XML files works fine in the debug URL but it was not working in the management command. Turns out the management command is running off a BaseResource object instead of a CompositeResource object. The BaseResource implementation of is_aggregation_xml_file always returns false. The implementation of is_aggregation_xml_file in CompositeResource has an implementation to check if it's a logical file metadata XML file. The solution is to use a CompositeResource object instead of a BaseResource object in the management command when the resource is a Composite type.
I also included an update to the resource debug URL because of a change to downloads that caused the previous debug URL to route to a download which will always fail.

Pull Request Checklist:

[x] Positive Test Case Written by Dev

[x] Automated Testing

[x] Sufficient User and Developer Documentation

[ ] Passing Jenkins Build

[ ] Peer Code review and approval

Positive Test Case

Run the check_irods_files management command and confirm no _meta.xml and _resmap.xml files are reported that are associated with a logical file.
Navigate to a resources debug URL /debug/resource/<resource_id>/

retest this, please

@dan-palmer1 or @cbcunc - Looks like Jenkins didn't pick up this PR to run the tests.  Any ideas what is going on?

METRIC
VALUE

NEW BUGS
0

NEW SECURITY HOTSPOTS
0

NEW CODE SMELLS
0

NEW DUPLICATED LINES DENSITY
0.0

NEW VULNERABILITIES
0

https://sonarqube.cuahsi-workstation.com:9000/dashboard?id=hydroshare-4193

fixes #4191

authors: hydrocheck sblack-usu


text:
node-pre-gyp support requested

Hope for pre-released packages, compile from source code is not always available

@windschaser please see https://github.com/hyj1991/v8-profiler-node8/pull/34/, I added a full binary compilation pipeline there using GitHub Actions that could likely be reused for this package. Builds binaries for Win/Mac/Linux including 32bit windows binaries, and handles uploading them as a release asset.

Landed in v8-profiler-next@1.7.0.

authors: Widdershin hyj1991 windschaser


text:
WIP Brave 4.1 APIs

Very very rough and untested attempt at using the Brave 4.1 APIs.
Next steps is to actually test this out and add abstractions where necessary.

Codecov Report

Merging #24 into master will decrease coverage by 54.57%.
The diff coverage is 30.76%.

@@             Coverage Diff              @@
##             master     #24       +/-   ##
============================================
- Coverage     91.17%   36.6%   -54.58%
+ Complexity       91      10       -81
============================================
  Files            18       6       -12
  Lines           306     153      -153
  Branches         17      10        -7
============================================
- Hits            279      56      -223
- Misses           23      93       +70
  Partials          4       4

Impacted Files
Coverage Δ
Complexity Δ

.../ratpack/zipkin/internal/ZipkinHttpClientImpl.java
0% <0%> (-70%)
0 <0> (-6)

...ck/zipkin/internal/MethodCapturingRequestSpec.java
0% <0%> (-61.54%)
0 <0> (-6)

...ck/zipkin/internal/RatpackCurrentTraceContext.java
0% <0%> (ø)
0 <0> (?)

...tpack/zipkin/internal/DefaultSpanNameProvider.java
100% <100%> (ø)
2 <2> (?)

.../main/java/ratpack/zipkin/ServerTracingModule.java
84.61% <100%> (-6.01%)
5 <1> (-10)

...k/zipkin/internal/DefaultServerTracingHandler.java
87.5% <87.5%> (-12.5%)
3 <3> (+1)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0ec625...b12b67a. Read the comment docs.

@llinder wow! I started working on the same thing last night - but clearly you're a lot further along than me. 😄

@llinder looks like there's a bunch of stuff that's going to be outright removed as part of this change. If that's the direction we're going in, then I suppose we might be looking at a major version bump (since there would be breaking changes)?

@hyleung glad to hear you are looking into this as well!  Right now I'm just considering this a spike.  If the directions seems good then we can add parts back in that make sense for Brave 4.
My hope is that the changes will have minimal to no effect for most folks.  For anyone interacting with the Brave 3 APIs and wishing to upgrade there will definitely be some breaking changes.  We could always go a different direction and keep both Brave 3 and Brave 4 APIs in the project but that feels overly complicated.  My preference is to just use Brave 4 and do a version bump.  Then again I don't want to make decisions for other folks so hopefully we will get some feedback.
Also I don't want to feel like I'm taking over so feel free to steer this as you see fit and I will follow your lead :)

@llinder at the moment (we'll see how long it lasts) I'm trying to take an incremental approach but yeah, I have a hunch that over time a version bump with breaking changes might be the easiest. If it comes to that, them maybe we just get one more release out to get things synced up to the last 3.x Brave release (plus any PRs that might be relevant) leave it at that...

I think this can all be removed. It was basically an attempt at using the
MDC context for span context propagation. It isn't really working as
intended. Will switch to the Ratpack MutableRegistry approach instead.

reminds me to make a base test class for CurrentTraceContext impl.  first
guess is that MDC would need some sort of dance to restore properly on
close.. anyway nice try very interesting!
https://github.com/openzipkin/brave/blob/master/brave/src/test/java/brave/features/log4j2_context/Log4JThreadContextTest.java
^^ has a dispatch approach which could still do log correlation, but uses a
more explicit means of attaching scope

This proxy thing is probably from my obsession about being able to test
everything (iirc) - it's just some indirection to provide a seam for
testing.

heh powermockito is a sledge hammer I sometimes use (but it can become a
cure worse than the disease)

By the way, do people prefer I keep commit logs for all changes or is squashing ok?  I tend to do the later but having the history to look back on might help.

@llinder ¯\(ツ)/¯  generally i'm pretty cool with whatever folks want to do on their own forks (tho' I do prefer a squash before the final merge).
Having said that, if you're up for it, I'd love to be able to collaborate on this PR (even if it's just pitching in on tidying things up, writing tests, etc.) - in which case, obviously squashes on WIP is a a bit problematic.

@hyleung cool I will avoid squashes to make collaboration easier.

I will try it out locally, looking good so far!

Thanks for the comments and catching my mistakes @edwardsoo-ss!

@llinder Dude you're on 🔥!
I was thinking about this a bit and it looks like you're on the path to pretty much on the path to doing away with brave-http altogether - i.e. getting down to the nitty gritty of annotating those spans yourself. What do you figure is the benefit of this approach over using brave-http's APIs (all the adapters and whatnot)?

brave-http will mostly go away once more generic Client/ServerHandler
support is in
https://github.com/openzipkin/brave/pull/350/files#diff-e0f8554fb22e6ddd2bc65c6ea53965e2
If it remains, it would only include helpers related to http tags I think.
On Thu, Apr 6, 2017 at 11:18 AM, Ho Yan Leung notifications@github.com
wrote:

@llinder https://github.com/llinder Dude you're on 🔥!
I was thinking about this a bit and it looks like you're on the path to
pretty much on the path to doing away with brave-http altogether - i.e.
getting down to the nitty gritty of annotating those spans yourself. What
do you figure is the benefit of this approach over using brave-http's
APIs (all the adapters and whatnot)?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/hyleung/ratpack-zipkin/pull/24#issuecomment-292057862,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAD61_U19sHye9oN1J_KxPpVzGOWPfcWks5rtFlpgaJpZM4MyV-_
.

@adriancole ah, ic...so it would be a case of implementing (in this case) RatPack specific subclasses of these handlers? In any case, the logic around dealing with starting/finishing spans, making sure they're correctly annotated, etc. - that would reside in the base classes that are provided by Brave (e.g. ServerHandler.handleRecieve, ServerHandler.handleSend, right?

@adriancole https://github.com/adriancole ah, ic...so it would be a
case of implementing (in this case) RatPack specific subclasses of these
handlers? In any case, the logic around dealing with starting/finishing
spans, making sure they're correctly annotated, etc. - that would reside in
the base classes that are provided by Brave (e.g.
ServerHandler.handleRecieve, ServerHandler.handleSend, right?

The end goals of that PR are the following... not all would apply here

be able to switch out B3 if needed (ex for internal things or another
format) that's the Propagation type
be able to switch out the context implementation (even if ratpack
decorates, would be nice for folks to be able to weave in SLF4J etc)
separate tagging implementation from tagging policy (ex be able to reduce
the tags emitted without knowing how they are created.. ex turn off
http.url or add a custom tag)
have a single test framework that makes sure things mostly work
regardless of how they are implemented

So, once this is in, we'd have tests to say if you could add response.size
tag, for example, or to disable client address parsing regardless of
library.
I'm kindof meh about sharing base classes for this as some of the code
might be doable in another way.. sharing some config classes (especially
saying which propagation to use and what tags to record) might be helpful
regardless as that would allow someone a single switch.
what do you think?

Thanks @hyleung but to his credit @adriancole is the one on 🔥 I'm just attempting to keep up with his breakneck pace :)   PR 350 looks like a great direction and I could see a lot of value using the pluggable span name, tags and address parsers here as well.
I mostly left brave-http out of the picture in this PR to get a better feeling for the core Brave 4 APIs without abstractions.  In general I found it way easier to use as apposed to Brave 3 and that the extra abstractions didn't provide as much value.  On the contrary, having a consistent way to handle span name, tag and address parsing seems very desirable, it just wasn't my initial focus.  Now that something has mostly been proved out with other frameworks I think it totally makes sense to apply them here as well.

@adriancole as always, thanks for the additional context :)
Re. brave-http, base classes, etc., for me the primary thing is being able to verify that a particular implementation has the correct behaviour. Especially at the beginning (hell, it's probably still the case now) when I wasn't super familiar with what needed to happen at a lower level, being able to implement a few adapters and interfaces and have it just plug in was handy. So having a way of verifying an implementation, whether it be that "single test framework" or otherwise would definitely be helpful.

@llinder could I trouble you with one small thing on this PR before it's merged - can we split the ParallelBatch stuff into a separate PR? I had a question from https://github.com/hyleung/ratpack-zipkin/pull/23, plus I'd prefer not to have it mixed in with the Brave upgrade.
Other than that, let's get this merged in soon. What I'll do is probably one more 1.x release (for https://github.com/hyleung/ratpack-zipkin/pull/21), then bump it to 2.0.0-SNAPSHOT and merge this in.
Does that sound reasonable?

Sounds good.  I will split the parallel batch stuff off.  I would like to get test coverage up a little more so I will try to tackle some of that this week as well.

@llinder I did have one bit of the code that I had trouble writing a test for.
Other than that, I'm ok with addressing coverage after merge (I'm happy to help out with that).  I think the JaCoCo doesn't deal with lambdas very well so the coverage report is a bit misleading sometimes.

@hyleung I removed the parallel batch stuff for now.  Also updated the dependency injection to use the Tracing component as suggested by @adriancole.  Lastly I updated the Ratpack request so that the tracing context headers are injected into the request.
Hopefully this is in a fairly good state to merge.  I will keep tracing Brave #350 work and incorporate new additions as they are released.

Hopefully this is in a good state to.. MERGED
Nice work and thanks for being the earliest into the new apis. I know it is
especially tricky with async.

@adriancole haha…to be clear, there are still a few bits an pieces (e.g. features that were removed) that I'd like to add back before we do a release - but we figured it would be better to make some forward progress.
@llinder I'm gonna add a couple of issues for the missing features (probably later today). If there's anything that you think needs to be included for an initial 2x. release feel free to add to the list (even if it's just a placeholder so we remember to discuss it) 😄

authors: adriancole codecov-io edwardsoo-ss hyleung llinder

repo: hyleung/ratpack-zipkin


text:
Поле "Ссылка" на событие.

Хорошо бы для автора отзывов иметь поле встреча (событие), к которому оно относится, а на странице отзыва (не автора) видеть эту ссылку и в случае чего перейти на него.
Из-за структуры отзывов (боковое меню) может быть несколько похожих событий или событий с одинаковыми именами. Встреча 2024, Встреча 2024 и т.п.
Такая ссылка позволит более точно идентифицировать событие или часть событий (раздел) в рамках одного события

Ничего не понял. Речь про возможность респонденту переименовать встречу?

authors: Lyumih nin-jin


text:
reedis连接池耗尽，无法新建立连接

如图，在tcp服务里使用redis连接池，报错提示 Connection pool exhausted. Cannot establish new connection before wait_timeout。
似乎是并发太高时造成的，该怎么解决呢？

加大 max_connection

加大 max_connection

感谢,加大之后没报错了

authors: limingxinleo tyhcmzy

repo: hyperf/hyperf


text:
fix build on osx

fix #693

fixed in my pending PR

repo: hyperion-project/hyperion


text:
Changelog mentions wrong feature name

Moved from #1957 (which maybe never got looked at bc. failing CI?):
There's a typo in CHANGELOG.md:

put Stream impl for Body behind stream feature

stream should be unstable-stream.

This should be a simple change! Sorry I missed the PR, though it would probably be better to not include the newline normalization.

Can I change it?

authors: hamzamuric jplatte seanmonstar

repo: hyperium/hyper


text:
does not compile with latest Rust nightly

 Compiling hyper v0.3.12
/home/nathan/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.3.12/src/server/mod.rs:185:20: 185:30 error: wrong number of type arguments: expected 1, found 0 [E0243]
/home/nathan/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.3.12/src/server/mod.rs:185     _guard: Option<JoinHandle>,
                                                                                ^~~~~~~~~~
error: aborting due to previous error
Build failed, waiting for other jobs to finish...
Could not compile `hyper`.

rustc --version
rustc 1.0.0-nightly (abf0548b5 2015-04-15) (built 2015-04-16)

See #459

@seanmonstar thanks

authors: fuchsnj seanmonstar


text:
Upgrade the Quorum platform code for Helm3

Description
As a developer I want to add helm3 related code for Quorum network so that the deployment is deployed & cleaned properly.
Please refer to PR : #1035 and #950 containing similar change for Quorum
Steps:

Take latest code from feature/helm3 branch
Run the Quorum network, ensuring that helm3 is installed.
Update the reset network code
Application charts (examples/supplychain-app/charts) should also be ugraded to have end-to-end testing of the platform.
Run reset network and verify all deployment components are deleted

Acceptance Criteria

Running reset network resets the whole network deployments
Testing for all 4 possible combination of consensus+transaction-manager of Quorum.
The cluster after reset, should work with helm2 (no CRD's should be left after cleanup)

4. Application charts (examples/supplychain-app/charts) should also be ugraded to have end-to-end testing of the platform.
=> The upgradation of the application code has been moved to a seperate issue.

authors: abevers suvajit-sarkar


text:
Initial code

The purpose of this repository is to provide a thin shim between FireFly and an ERC1155 contract exposed via ethconnect. It must take logical inputs from FireFly (namespace, pool name & UUID, token type) and pack them into the standard fields provided by ERC1155 (token ID, URI). It must also take the events emitted from ERC1155 (URI, transfer) and map them to logical events needed by FireFly (pool creation, mint, transfer).
The API is very flat and non REST-ful at the moment, so it's very open to input. The FireFly tokens API itself was proposed on a recent community call, but the API for this micro-service will have a much smaller service area and notably very few GETs (because very little is queryable from Solidity and the goal is for this service to be stateless). However, we would like this API to become standard across all token implementations on all chains (as much as possible), so it's worth a close look.
Current proposed APIs:

POST /pool - Create a new token pool (inputs: type, namespace, name, client_id)
POST /mint - Mint new tokens (inputs: pool_id, to, amount)
POST /transfer - Transfer tokens (inputs: pool_id, token_index, from, to, amount)
GET /balance - Get token balance (inputs: pool_id, token_index, account)

All POST APIs are async and return 202 immediately, and results are reported by a websocket event of the form {event: string, data: any}, with the following event types:

token-pool - Token pool created (outputs: pool_id, type, namespace, name, client_id)
token-mint - Tokens minted (outputs: pool_id, token_index, to, amount)
token-transfer - Tokens transferred (outputs: pool_id, token_index, from, to, amount)

The TypeScript code is tightly coupled to the Solidity contract in terms of how data is packed and unpacked. For the purposes of this first pass, this reference contract is assumed, which is in turn based on a fungible/non-fungible example from Enjin. The actual contract will be finalized for review in a future PR.
At a high level, the 256-bit token ID from the contract is used to pack the token type, pool ID, and (for non-fungible only) token index. The string URI from the contract is used to pack the FireFly data such as namespace, name, and UUID.

Do I need licenses in every file?

Another side note on API naming (worth considering along with the rest of the initial design here): how will this service be named when plugging it into a FireFly config? Should it be named "https" like the data exchange plugin? Although this implementation is backed by ERC1155, the goal is for the HTTP API to be generic enough for a variety of token deployments. Possible config sample:
tokens:
  type: https
    https:
      url: http://localhost:6000

I added a few more commits to clean up the batching and fix the issue noted with broadcast vs. unicast. Probably a few minor items still to address, but these were the most significant changes.

Do I need licenses in every file?

Yes :)
https://github.com/hyperledger-labs/hyperledger-community-management-tools/blob/main/repo_structure/README.md#L46-L47

@nguyer yep, added the licenses and (I think) addressed all the previous review comments.

@awrichar Could you also please update CODEOWNERS to match the list of maintainers for this repo?

authors: awrichar nguyer


text:
did method registry

To support different did methods in plugins we need to support did method pluggability. This PR changes the default methods to be static and registered into a registry of methods at start-up. This allows plugins to define new did methods that can be registered. The bulk of changes in this PR is updating the current code to use a did method registry.
Signed-off-by: Adam Burdett burdettadam@gmail.com

Thought provoking question: If we load a DID from the wallet that was created with a plugged in method but the current instance of ACA-Py does not have the same plugin loaded, should we fail on loading the DID or when we attempt to use the DID?
As currently structured, the DID Method includes info about key types and support for key rotation. This would need to be stored alongside DIDs in order for us to restore a DID without knowing the Method. This doesn't seem favorable so failing to load the DID seems like the right choice.

Also making note of this here: the Indy wallet does not hold a reference to an injection context. This means it is unable to load the set of supported methods from the context. Many of its operations are hard coded for did:sov and did:key. This PR leaves those operations as hard coded to support did:sov and did:key and does not introduce the injection context on the Indy wallet.

@burdettadam can you catch this one up with the latest main branch?

Codecov Report

Merging #1955 (797724b) into main (a16347d) will decrease coverage by 0.01%.
The diff coverage is 81.37%.

@@            Coverage Diff             @@
##             main    #1955      +/-   ##
==========================================
- Coverage   93.55%   93.54%   -0.02%
==========================================
  Files         539      539
  Lines       34526    34528       +2
==========================================
- Hits        32300    32298       -2
- Misses       2226     2230       +4

We have a single PR with a related key type registry now, #1986. Closing this one in favor of the new one.

authors: burdettadam codecov-commenter dbluhm ianco

repo: hyperledger/aries-cloudagent-python


text:
Storage Optimization

Epic to be able to optimise the database storage needed for the Besu.

[x] Final cleanup and archive besu bonsai light work
[x] Test Pruning + Near-head Checkpoint Sync on Mainnet
[x] Optimised storage presentation slides
[x] Bonsai light mode to allow checkpoint sync near head block for pivot block
[x] Bonsai lightweight mode to prune old chain data
[x] Test Besu and Teku with a multiplexer
[x] Spike pruning existing rocksDB chain database
[x] Setup Teku and Besu with near-head checkpoint sync
[x] Bonsai light mode to update the chain
[x] Bonsai light mode to prune transaction receipts
[x] Investigate EL/CL multiplexers
[x] Investigate how syncing works with CL in context of optimised storage
[x] Understand how storage is distributed for bonsai
[x] Determine how shared storage consistency will be handled
[x] Investigate how we would sync head world state
[x] Investigate whether scylladb database could be used for shared storage
[x] Investigate shared ZFS filesystem
[x] Investigate using EBS multi-attach to share storage
[x] Performance benchmark of the eth_call RPC daemon
[x] Investigate Ethereum light client
[x] Identify which external databases for shared database
[x] Optimise the reader database API
[x] Spike on external databases for shared storage
[x] Create a tool to give stats on the anatomy of the database usage
[x] Change shared db spike API to use gRPC
[x] Investigate possible external databases for shared storage
[x] [SPIKE] Besu shared database using Storage plugin API
[x] [Discovery] Understand storage engine

Final approach we decided on using near-head checkpoint Sync on Mainnet

repo: hyperledger/besu


text:
Optimize RocksDB WAL file

PR description
Currently Besu uses the default RocksDB WAL options, with the result that usually the WAL files could occupy ~15GB of disk space, due to the fact that different column families are flushed at very different intervals, this has also the bad effect of slow restarts, when Besu has to parse an reapply all these WAL files.
Capping the WAL files total size to 1GB and reusing old files when possible, has the advantage of a faster restart, and a more linear disk usage, instead of the classic saw tooth shape that we see today.
Fixed Issue(s)

From https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-(WAL)

DBOptions::max_total_wal_size
In order to limit the size of WALs, RocksDB uses DBOptions::max_total_wal_size as the trigger of column family flush. Once WALs exceed this size, RocksDB will start forcing the flush of column families to allow deletion of some oldest WALs. This config can be useful when column families are updated at non-uniform frequencies. If there's no size limit, users may need to keep really old WALs when the infrequently-updated column families hasn't flushed for a while.

So the ~15GB the WALs occupy, consist mostly of unneeded data, since the data has already been written for all the column families but one that is infrequently flushed, and that single column family is enough to keep these files around, setting the limit forces a regular flush of that column family.


text:
Update CI For Asset Transfer

Pushes the scripting logic into a standalone script and adds tests for the applications which by side-effect also test the invoke and query functions on the chaincode.
Signed-off-by: Brett Logan brett.t.logan@ibm.com

Converting back to draft for now. Because all of these invokes happen on the same asset asset13 we can't invoke twice, and I don't want to keep restarting the network in between. My short answer was create a file for calling deleteAsset but I don't I imagine we wouldn't want to include a delete.js file alongside invoke.js and query.js let me think through this.
I may just end up splitting each test out into separate job executions

Here we go, this is a much more proper CI pipeline

Converting back to draft one more time, pending another discussion

Now that Typescript chaincode and apps are remove, going to simplify this greatly as well as introduce Javascript and Go Linters.

what is typescript-application being removed, but typescript CC being kept?
btw
#268 had similar updates

what is typescript-application being removed, but typescript CC being kept?
btw
#268 had similar updates

After speaking with the chaincode maintainers, there are typescript specific constructs, specifically the annotations that need to be demoed via chaincode. So they elected to keep typescript chaincode to demo these typescript specific constructs. None of them apply to applications however. So we will keep one example of typescript chaincode (basic).

repo: hyperledger/fabric-samples


text:
System chaincode APIs

System chaincode requires a special stub to access privilege APIs, so we need to define these APIs. For example, a scc may call functions to deploy a user-chaincode.
User-chaincode may call system-chaincode functions.

APIs will fall out of work for #1127 and #1133. For example, If Uber chaincode woud need an API to Deploy, Invoke and Query. The APIs for most part will be implementations available in code today (ie, hopefully most of material for the implementation is in exisiting code).

authors: binhn muralisrini

repo: hyperledger/fabric


text:
performance degradation

Commit level 42c6fe9ff5bceb4258eb9754fa33fef5f668ecd2 can only perform 25 invokes per second with security and privacy enabled and both peer and crypto set to error.  The test is performed with SoapUI with 100 threads.
After trace back, found that commit e3c27dcf853863bda1f962addf65e6db43232521 performs 152 Invokes per second but commit 329127cf14f2ce7637308e3c7693b5190abd8111 can only perform 25 Invokes per second.
commit e3c27dcf853863bda1f962addf65e6db43232521
Author: srderson sranderson@gmail.com
Date:   Thu Mar 3 12:14:42 2016 -0500
Initial table shim API for Go. Issue #477

commit 329127cf14f2ce7637308e3c7693b5190abd8111
Merge: 3c9ff3a 2cd84df
Author: Binh Q Nguyen binhn@us.ibm.com
Date:   Thu Mar 3 15:18:45 2016 -0500
Merge pull request #745 from adecaro/next_conf

Confidentiality 1.2

@adecaro : after bracketing, issues seems to be with commit 329127c

@dongmingh @muralisrini I have just submitted a PR to address this issue. Please, let me know if it fixes the problem. Thanks :)

With the commit from @adecaro ,
commit 085dcb505674780746bd65df05dafa878d77bc41
Author: Angelo De Caro adc@zurich.ibm.com
Date:   Thu Mar 10 17:15:54 2016 +0100
removed unused imports

With Tcert=200, threads=100, 5 min run, the throughput are as follow:
peer and crypto set to error, security and privacy enabled, tps=121 and 151 for invoke and query resp.
peer and crypto set to error, security and privacy disabled, tps=185 and 257 for invoke and query resp.
peer and crypto set to info, security and privacy enabled, tps=104 and 140 for invoke and query resp.
peer and crypto set to info, security and privacy disabled, tps=162 and 252 for invoke and query resp.

that's closer to the original numbers. @adecaro : do you know what's causing the delta degradation (given that @dongmingh is able to reproduce the original values with the previous level of code) ?

With peer and crypto set to error and security and privacy enabled, tps = 147/236 (invoke/query) about two weeks ago, see #684

@muralisrini, the degradation, at this point, is clearly related to the fact that we are now using asymmetric encryption which is slower then symmetric encryption.

thanks, @adecaro

I'm on commit tag 3ef73d319b01fe84c0c88973350a240fc490d82e and seeing similar results. One node, after deploying chaincode_example02, with security and privacy disabled, invoke/query throughput is 173/198 per second. With security and privacy enabled, invoke/query throughput is 2.5/2.2 per second. This is on Linux s390x with 4 CPUs and 16GB memory, SLES 12.1.
One thing needs to be fixed is that with security and privacy enabled, even with log level at critical, the following hex numbers are still being printed for each invoke/query:
block 2d 2d 2d 2d 2d 42 45 47 49 4e 20 45 43 44 53 41 20 50 55 42 4c 49 43 20 4b 45 59 2d 2d 2d 2d 2d 0a 4d 46 6b 77 45 77 59 48 4b 6f 5a 49 7a 6a 30 43 41 51 59 49 4b 6f 5a 49 7a 6a 30 44 41 51 63 44 51 67 41 45 54 76 43 6d 6b 53 50 46 72 31 52 4f 7a 77 56 63 52 68 37 61 41 31 6b 74 39 43 69 31 0a 73 76 4e 77 7a 65 4f 4f 6a 78 4b 37 30 61 4b 59 65 32 63 6a 72 6e 48 38 44 2b 78 63 4f 67 6c 42 58 36 73 32 4f 6b 65 50 4c 71 67 64 4d 44 78 35 6e 36 47 79 61 46 50 42 68 67 3d 3d 0a 2d 2d 2d 2d 2d 45 4e 44 20 45 43 44 53 41 20 50 55 42 4c 49 43 20 4b 45 59 2d 2d 2d 2d 2d 0a
Logging should be async so it shouldn't have too much impact but you never know.

@gongsu832, thanks for the feedback. PR #813 addresses also that.

Hi @dongmingh, can this issue be closed?

@dongmingh, what is the your status? Can this be closed as @adecaro inquired..

@markparz - we were looking at some current numbers and suspect there has been some regression in this area. We were going to collect numbers at old commit level and compare to confirm. If regression confirmed, we have to keep this open.
Agree, @dongmingh ?

@muralisrini what are you using to determine performance regressions ?

@jeremyeder : nothing formal. We just have some measurements from the commit level after the first fix.

The issue was fixed on commit 1f65f6499e99246042125fe26426b177459ac673, dated 3/24.  It performed at 128/198 tps for invoke/query with security and privacy enabled.  However, I re-tested it with commit 0beb9ff56e57f8d8ecf4b13b958fe410f0241b7b, dated 4/15, the performance drops to 60/55 tps for invoke/query.  The test environment is on 1 vagrant using SoapUI to generate transactions.  Something was introduced between these two commits to cause the performance drop.

The date for commit 1f65f6499e99246042125fe26426b177459ac673 is 3/21 instead of 3/24.
commit 1f65f6499e99246042125fe26426b177459ac673
Merge: 13fe451 929cc6a
Author: Sheehan Anderson sranderson@gmail.com
Date:   Mon Mar 21 11:33:40 2016 -0400
Merge pull request #858 from srderson/issues/776/comment_cleanup

Clean up utxo chaincode comments

@dongmingh, I have just benchmarked the crypto layer only of 0beb9ff56e57f8d8ecf4b13b958fe410f0241b7b and I didn't see any particular degradation in the performance.
I wasn't able to run the same benchmark on 1f65f6499e99246042125fe26426b177459ac673 because it refers to openblockchain and the compilation failed in hyperledger.
The current master has also good numbers. Again, it's just the crypto layer.

@adecaro, One more data point, if I comment out all CloseClient() from both rest_api.go and devops.go,  the tps remains the same, 60, for both invoke and query.
I will test today's master and report here.

@dongmingh, I have just benchmark the master of openblockchain and I see very similar numbers to those I get from hyperledger. Again this is the crypto layer only. It might be then that the degradation comes from other layer on the stack. More investigation is needed.

@dongmingh .. given you have the env. would it be possible to run the two levels so we can home in on the difference ?

With today's commit: 6bc272efc1f0aec1a062b32f0ef7621489673d43, the tps are
security/privacy disabled, invoke/query = 255/535
security/privacy enabled, invoke/query = 58/48
security/privacy enabled, CloseClient() commented out, invoke/query = 60/56

@dongmingh, is there a way to profile hyperledger when you submit your transactions? That would help to understand where are the bottlenecks.

@adecaro, in the past, we have done profiling using GO tool and learned that it does not give us the granularity we need to point to specific functions which are the bottleneck.  Therefore we build instrument into the golang code to get the time used by the user specified functions now.  The data (excel file) I sent you last week is the output from the instrument.  It shows that getTCertfromDER and loadUnusedTCerts (delete) use a large portion of time when security and privacy are enabled.  However, since the performance tps are about the same if commenting out CloseClient() in both rest_api.go and devops.go, it seems that there are something else cause the performance issue in addition to the two functions in crypto.

@dongmingh, Indeed. Thanks for the clarification :)

authors: adecaro dongmingh gongsu832 jeremyeder markparz muralisrini


text:
Parallelize Unit Tests in CI

Parallelizes unit tests. Beyond making unit take less time to execute, this also means if you have a failure you don't have to wait 40 minutes for units to rerun as you only have to rerun the failed test.
Signed-off-by: Brett Logan brett.t.logan@ibm.com

The parallel integration tests have been a large help to productivity. I'm definitely in favor of doing the same for unit tests. Of course, we should continue to try to improve unit test time and flakes in general, but let's move forward with the parallel unit tests as well.

I wish we had a pull request like this a few years ago.

I'm going to stand my ground here. It doesn't make sense to me to spin up 8-10 VMs to run tests that developers are expected to run locally.
If we want to improve things, let's attack the source of the problems. For unit tests, we know what needs to change but we seem to be unwilling to do it.

@sykesm The orderer unit tests have been failing a lot, and it is quite painful to rerun all the unit tests each time.
It is quite possible to do both - invest in unit test improvements AND parallelize in CI for quicker turnaround and re-test. I don't understand the argument that we can only do one or the other.

The orderer unit tests have been failing a lot, and it is quite painful to rerun all the unit tests each time.

Then we need to focus on fixing them. Same is true of gossip. Most of those tests share the same authors and test approach where many concurrent go routines are used in a way that results in non-deterministic execution.

It is quite possible to do both - invest in unit test improvements AND parallelize in CI

Agreed. However, other than gossip (which can't be run in parallel because of resource conflicts), the tests are already running in parallel (see the -p flag on go test). That means the effort should be focused on test improvements instead of distributing the parallelism across virtual machines.
This is not a new discussion. I have been very vocal about this for two years. I fundamentally oppose a culture where we make it easy for maintainers to ignore persistent problems by simply clicking a button to re-run tests.
Facts:

We currently have 27 items labled ci-flakes on the board.
19 ci-flake issues have been opened since December 2018. Most have been opened by @caod123, @btl5037, or @sykesm (me). One was opened by @denyeart.
3 ci-flake issues have been resolved with a fix
1 ci-flake was closed as a duplicate
Only 1-2 two flakes were updated with information about a recurrence.

Given "orderer unit tests have been failing a log", those facts imply that maintainers aren't looking into the flakes.
In terms of test execution, FAB-17508 highlights one example of a watermelon in our tests. While the elapsed time in CI will differ from what I observed in my VM, simply caching a configuration object in our tests brings the execution time of the units down to 17m33s. Compare that to the   15m27s execution time of one of the parallel executions in this PR. Does the 2m6s difference really justify the consumption of 2 additional VMs? I don't believe it does.
In summary, before we try to "optimize" by doing multi-image parallelization

let's fix the watermelons in the tests
let's focus on improving the stability of the tests

222 ci-flakes have been opened since December 2018. I've opened 8 of them. That tells me that the remaining 27 are the difficult ones that require some serious time commitment. Regardless of the Jiras, we all know that gossip and orderer flakes are the frequent ones for recurrence.  The strategy of inflicting pain on all developers the last two years, in hopes that somebody will volunteer to fix these flakes, has not worked. It has only caused frustration and turned developers off. We need to change our approach.
I propose:

Let's actually invest some dedicated resource in fixing the gossip and orderer flakes.
I am good to wait for FAB-17508.
If there is not a fundamental improvement a few weeks from now, let's proceed with the unit test parallelization.

The benefit of this is trivialized by #758. Thank you Matt for investigating and both you and Derrick for patching.
Closing

authors: btl5037 denyeart sykesm yacovm


text:
Correct path for installing marble chaincode

Hi Team,
The current path for marble chaincode under the heading "Using CouchDB" is invalid. Change it from
github.com/hyperledger/fabric/examples/chaincode/go/marbles02
to
github.com/chaincode/marbles02/go
How Has This Been Tested?

I followed the instruction under "Using CouchDB" with this path and the chaincode was installed.
Checklist:

[x] I have added a Signed-off-by.
[x] I have either added documentation to cover my changes or this change requires no new documentation.
[x] I have either added unit tests to cover my changes or this change requires no new tests.
[x] I have run golint and have fixed valid warnings in code I have added or modified. This tool generates false positives so you may choose to ignore some warnings. The goal is clean, consistent, and readable code.

Signed-off-by: Wahab Jawed wahabjawed@gmail.com

Thanks for the contribution, but it's only a read-only mirror. You can go to Gerrit and submit the patch.

authors: ryjones wahabjawed


text:
Add submitter and observer traits

Adds the public traits for the submitter and submitter observer. These will be used in future implementations.

Rolled into #1346

repo: hyperledger/grid


text:
refactor: reorganize project

Description
This fixes the error with cargo install:
package `.../client/tests/integration/smartcontracts/executor_custom_data_model/Cargo.toml` is a member of the wrong workspace

move all binary crates to /bins
move all library crates to /libs
move executors, smart contracts and triggers to /samples
unnest iroha_executor and iroha_trigger from iroha_smart_contract
rename crate directories to match their actual names (except nested utility crates)

Benefits

binaries can be installed with cargo install
separation between binaries and libraries
more predictable structure and names

Checklist

[x] I've read CONTRIBUTING.md
[x] I've used the standard signed-off commit format (or will squash just before merging)
[x] All applicable CI checks pass (or I promised to make them pass later)
[ ] (optional) I've written unit tests for the code changes
[ ] I replied to all comments after code review, marking all implemented changes with thumbs up

I think you should break this into several PRs

can this error be fixed in less invasive way?

Superseded by #4863.

authors: Erigara mversic nxsaken

repo: hyperledger/iroha


text:
Fix Seth for Sawtooth 1.1 Consensus

Sawtooth 1.1 Consensus does not work.
The DevMode consensus engine times-out and does not connect
to the validator and times-out.
The solution is to open port 5050 in container devmode-engine-rust.
Signed-off-by: danintel daniel.anderson@intel.com

For the record, here are the exact error messages for DevMode consensus errors:
devmode-engine-rust | ERROR | devmode_engine_rust: | ReceiveError: TimeoutError
devmode-engine-rust exited with code 1

repo: hyperledger/sawtooth-seth


text:
Finish library interface

Currently, Solang has a library interface, but it is only used for tests. In the long run we should be able to use our library interface as a the ground through for all tests, the solang binary and anything else. Feel free to comment and collect ideas here regarding the direction we want to go with this interface.

The library interface is also used by the solang binary, but the library interface has far too many public items.
The library interface is also used by the language server, so it needs the entire ast exposed (but not codegen)
The solang binary may save intermediates like llvm ir, assembly of cfg. This interface could be much cleaner
The solana interface is extra ugly because currently Solana creates on binary for all contracts - this will soon change

Depends on #964 - once that is implemented, this will be much simpler

authors: seanyoung xermicus

repo: hyperledger/solang


text:
Autoconnect flag

It'd be nice to be able to disable the auto-connection feature for cases where it'd be handy to control that at the application level.

You can do that today using the info object. I think the method is info.reconnect(false)

How do I get the info object from a swarm instance?

Oh, I think I get it. When I get the peer event, that's where I'd invoke info.reconnect(false) which would prevent the initial connection?

Except that you don't have info in peer event :joy:

@indutny ah you wanna configure peer stuff before a connection happens?

I probably don't. Was just browsing through the issues to see if there is anything useful for my case.

you use the companion object next to the connection in the 'connection' event to manage all this

authors: RangerMauve indutny mafintosh

repo: hyperswarm/hyperswarm


text:
Prevent (re-)delivering RabbitMQ messages to Celery workers that're being terminated

This is the same as https://github.com/hypothesis/lms/issues/5304 but for Checkmate.

Closing as won't fix, same as we did for https://github.com/hypothesis/lms/issues/5304

repo: hypothesis/checkmate


text:
Hide floating windows and only show tiled windows while holding keybind

Description
I mostly work with tiled windows, but I have set some rules for certain programs like my file manager or my download managers to float by default. However, because of the nature of Hyprland, floating windows will always be on top of tiled windows. Sometimes I need to see something in the tiled window covered under the floating window, and currently I need to either move the window out of the way or I just turn it into a tile window or even close the floating window all together. What I want is a functionality that, when assigned to a keybind and held, hides all floating windows on the current workspace. Something like Windows' functionality that allows you to hide all currently open windows while you hover your cursor on the bottom right of your screen.
Is this even possible to implement? Perhaps maybe something like detecting if the client is floating=1 and is in the current active workspace, and then setting opacity to 0?

hyprctl -r keyword 'windowrulev2' 'opacity 0,floating:1', to revert hyprctl -r keyword 'windowrulev2' 'opacity 1,floating:1'
needs lates hyprland-git to work properly. You can combine this with transparent binds.

authors: farelmapm vaxerski

repo: hyprwm/Hyprland


text:
Add .ani support

Add support of Windows animate cursor format .ani

No?

authors: VerySweetBread vaxerski

repo: hyprwm/hyprcursor


text:
Fails to build on 32-bit architectures (e.g., armv7, i386, powerpc)

Regressed by 39ad021c7599. See also https://github.com/hyprwm/hyprlang/issues/8
$ export CXXFLAGS=-m32
$ cmake -G Ninja -B /tmp/hyprpaper_build
$ cmake --build /tmp/hyprpaper_build
[...]
src/config/ConfigManager.cpp:135:35: error: call to constructor of 'const Hyprlang::CConfigValue &' is ambiguous
    config->addConfigValue("ipc", {1L});
                                  ^~~~
/usr/include/hyprlang.hpp:192:9: note: candidate constructor
        CConfigValue(const INT value);
        ^
/usr/include/hyprlang.hpp:193:9: note: candidate constructor
        CConfigValue(const FLOAT value);
        ^
/usr/include/hyprlang.hpp:289:67: note: passing argument to parameter 'value' here
        void addConfigValue(const char* name, const CConfigValue& value);
                                                                  ^
src/config/ConfigManager.cpp:136:38: error: call to constructor of 'const Hyprlang::CConfigValue &' is ambiguous
    config->addConfigValue("splash", {0L});
                                     ^~~~
/usr/include/hyprlang.hpp:192:9: note: candidate constructor
        CConfigValue(const INT value);
        ^
/usr/include/hyprlang.hpp:193:9: note: candidate constructor
        CConfigValue(const FLOAT value);
        ^
/usr/include/hyprlang.hpp:194:9: note: candidate constructor
        CConfigValue(const STRING value);
        ^
/usr/include/hyprlang.hpp:289:67: note: passing argument to parameter 'value' here
        void addConfigValue(const char* name, const CConfigValue& value);
                                                                  ^
2 errors generated.

should be done with above

authors: jbeich vaxerski

repo: hyprwm/hyprpaper


text:
Feature: cache build result

Since the output file name is stable, I guess I can add a simple cache strategy like the tool make: If the output file's mtime is after the input file's, it is usually safe to skip this build task. To escape from this I can add a new option --cache=false.

I found one caveat, that if the file was process with plugins, and then run it again without plugins, the cache will hit and return the result of using plugins. In that case maybe the user just use --cache=false.

repo: hyrious/esbuild-dev


text:
Games started with errors after added VmwareHardenedLoader

Hi All/Author,
The reason I needed the VMwareHardenedLoader is because I am trying to play Maplestory inside my VM Windows 10 Pro,
but after I did all the required steps, the Games show either
Initialization error
or

Well you don't have a proper graphics driver to run it. So basically VMware Video Driver is necessary but it basically defeats the purpose of this repository

authors: Kuroyabumozu nxvvvv

repo: hzqst/VmwareHardenedLoader