text:
How do I load a single file into kiara using the python API

what plugin(s) do I need to have installed? Core? Onboarding? What version(s) does this explanation apply to, has anything changed recently?
what exactly do I write in kiara? what's the operation name, what arguments do I have to give it, what exact format are they in

relative paths? absolute paths? URLs? special URIs from github or something?

what optional things can I give to that operation? Is this a thing where aliases come to play?
is there anything else I need to know/think about

Yes I know some of this is in the operation docs, but I really need concrete examples of how to load a file from github and from the user's filesystem.

The only 'stable' file import operation at the moment is import.local.file. It's one of the few modules included in the kiara package (not even kiara_plugin.core_types), so in theory you don't need any plugin installed at all (although I haven't really tested not having core_types installed in a while, so there might be breakage, but feel free to report an issue if that's the case).
The single argument is path (kiara operation explain import.local.file), and it takes either an absolute path, or a relative one from the current directory. It does not support urls or anything non-local.

what optional things can I give to that operation? Is this a thing where aliases come to play?
No, nothing.

is there anything else I need to know/think about
No. Except I'm thinking about a more generic module that can take all kinds of strings and is smart enough to figure out how to retrieve the file from wherever it is (in the onboarding plugin). The main difficulty with that is to figure out a good module interface that will work for as many cases as possible, and it might still turn out to be a bad idea in the first place.
Anyway, that module that is not in any way ready to be used. Happy to consider ideas anyone might have in that regard.

If you want to load files from github, there is only the very alpha one in the onboarding plugin at the moment, but I wouldn't recommend to use that atm; it is an area I intend to focus on in the near/medium-term future, and create a set of modules that complement each other well and are able to get retrieve and import datasets from any of the potential sources we've identified.
For the type of functionality you need but that isn't ready yet, it might be a good idea to create your own plugin project and add very basic modules that do what you need. The advantage there is that with those we don't need to think too much about their interface (inputs/outputs schema) yet, and you can replace them fairly easily once there is an 'official' one. And they can be used as input to designing the 'official' one in the first place.
The other advantage is that you have full control over the module, so you don't have to worry that the module interfaces changes under your feet while in development.

Is this a thing where aliases come to play?

You can assign an alias to the imported file. In the cli you'd do it something like:
kiara run import.local.file path=pyproject.toml --save file=my_alias

In Python, you'd do:

Can I have an example using the python API (kiara.api.KiaraAPI.instance()) rather than the CLI please?

Yeah, I'm about to write it up, one sec.

Up, finished my comment (above). Happy to change the docstring for that function if you have any suggestions. I guess one area to write up would be the whole concept of storage, but that would probably be too much for this particular comment and needs to go into its own sections in the future docs.

Btw, you don't need to store a value if you don't want to persist it and don't need a (human-readable) alias in the UI (often it's not necessary for temp data). The value will be available until you restart the Python process.

So the relevant things I learned here was to not use the operations in the onboarding module, and that you have to have files locally. So I've got a bunch of rewriting to do in my app prototype. I'll write this up in a how-to doc and send a PR shortly.
In the future, please could you avoid giving CLI examples, I find it really confusing and hard to follow, because it's similar but not quite the same as usage via the Python API. I think the consensus from Mariella's research was that no end user wants to use the CLI, and I don't want to spend extra time documenting it.

Ah, and you might rather use queue_job instead of run_job if you don't want the operation to block:
import time
from kiara.interfaces.python_api import KiaraAPI

api: KiaraAPI = KiaraAPI.instance()
inputs = {
    "path": "/home/markus/projects/kiara/kiara/pyproject.toml"
}
job_id = api.queue_job("import.local.file", inputs=inputs)

job = api.get_job(job_id)

# some way to pull, ideally we'd have an event system of some sort, but there is none yet
while job.finished is None:  # will add an 'is_finished()' method to this object in next version, as this only returns a date if the job is finished which might be unintuitive
    time.sleep(1)
    job = api.get_job(job_id)

results = api.get_job_result(job_id)

file_result = results["file"]
api.store_value(file_result, "alias_from_python_2")

source: gharchive/issue

repo: DHARPA-Project/kiara-website

license_type: permissive

license_source: github-api


text:
Recommendation for storing and versioning AIPs without the use of BagIt

For the versioning of AIPs the plan is to recommend the use of OCFL.
Assuming the following structure for an original submission information package example.sip.001.tar stored as version v00000 and an AIP urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar stored as version v00001:
├── 0=ocfl_object_1.0
├── inventory.json
├── inventory.json.sha512
├── v00000
│   └── example.sip.001.tar
└── v00001
    └── urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar

The inventory.json could look as follows:
{
    "digestAlgorithm": "sha512",
    "fixity": {
        "md5": {
            "f97f90b429a84bdd0bfb88b6d037b351": [
                "v00000/example.sip.001.tar"
            ],
            "ed7f48df08c5c1f134c02dcfd9ff6098": [
                "v00001/urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar"
            ]
        },
        "sha256": {
            "47782fd210d3933bda9045d923c4370b2632c39826096bfa86b2860d07397742": [
                "v00000/example.sip.001.tar"
            ],
            "ac1d70378c2be8cf818b490292b51ccabba55b2192004eda67a7822f60072612": [
                "v00001/urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar"
            ]
        }
    },
    "head": "v00001",
    "id": "urn:uuid:81bd3aa2-7350-44f6-ad54-d8181858605a",
    "manifest": {
        "c676d28b0d0a5aa345aea7995fbdf36b06981923af85d2234de5157c11173c032435fc3da4f513c717bba4bb912d0f9c7165750c46ed821bffdc22def79606c7": [
            "v00000/example.sip.001.tar"
        ],
        "1aef284e408d991ba6abf9973c4bcb02c1a2a94c951cd119cd040249878ac2d2ce790d78c0416327fd92efaab8f0536be1b0fb0a2f17a8cbe0069b72d16f7988": [
            "v00001/urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar"
        ]
    },
    "type": "https://ocfl.io/1.0/spec/#inventory",
    "versions": {
        "v00000": {
            "created": "2024-04-09T21:08:54Z",
            "message": "Original SIP",
            "state": {
                "c676d28b0d0a5aa345aea7995fbdf36b06981923af85d2234de5157c11173c032435fc3da4f513c717bba4bb912d0f9c7165750c46ed821bffdc22def79606c7": [
                    "v00000/example.sip.001.tar"
                ]
            }
        },
        "v00001": {
            "created": "2024-04-09T21:08:55Z",
            "message": "AIP (ingest)",
            "state": {
                "1aef284e408d991ba6abf9973c4bcb02c1a2a94c951cd119cd040249878ac2d2ce790d78c0416327fd92efaab8f0536be1b0fb0a2f17a8cbe0069b72d16f7988": [
                    "v00001/urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar"
                ]
            }
        }
    }
}

Note that there is an overlap of fixity information which is provided in the METS already.
The question for voting is if the container files example.sip.001.tar for the original SIP and urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar for the AIP should be wrapped in a bagit container, for example:
├── bag-info.txt
├── bagit.txt
├── data
│   ├── metadata
│   │   ├── descriptive
│   │   │   └── ead.xml
│   │   ├── metadata.json
│   │   └── preservation
│   │       └── premis.xml
│   ├── METS.xml
│   ├── processing.log
│   ├── representations
│   │   └── 1710641a-bfa1-48cc-b41f-4220606679ae
│   │       ├── data
│   │       │   └── example.pdf
│   │       ├── metadata
│   │       │   └── preservation
│   │       │       └── premis.xml
│   │       └── METS.xml
│   └── state.json
├── manifest-sha256.txt
├── manifest-sha512.txt
├── tagmanifest-sha256.txt
└── tagmanifest-sha512.txt

Note that this way fixity information would possibly be provided in up to four layers:

OCFL
BAGIT
METS
PREMIS

To reduce complexity and redundancy, the proposal is store the E-ARK information package as TAR files instead of wrapping them as bagit containers as shown in the example above.
The E-ARK AIP container file urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a.tar would then have the following form, for example:
urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a
├── metadata
│   ├── descriptive
│   │   └── ead.xml
│   ├── metadata.json
│   ├── other
│   │   ├── processing.log
│   │   └── state.json
│   └── preservation
│       ├── premis_202401094-230854Z_event_sipcreation.xml
│       └── premis_20240409-230854Z_event_ingest.xml
├── METS.xml
├── representations
│   └── 09502a26-f822-407c-ad0a-4d7e64052a91
│       ├── data
│       │   └── example.pdf
│       ├── metadata
│       │   └── preservation
│       │       └── premis.xml
│       └── METS.xml
└── schemas
    ├── csip.xsd
    ├── ead3.xsd
    ├── IP.xsd
    ├── mets_1_11.xsd
    ├── premis-v2-2.xsd
    └── xlink.xsd

The suggestion is:
As part of the general AIP recommendations, the proposal is to store the E-ARK information package as TAR files instead of wrapping them as bagit containers.

Hello,
I am not quite understanding the use cases that are meant to be supported. In my view, AIPs live independently of SIPs, but might be affected by them, such as they might be affected by other operations such as metadata enrichment, file format convertion or even redaction or destruction by retention processes. Note also that AIPs might be created by SIPs, or by operations, such when creating an AIC (and AIP relative to a collection or a case with only metadata).
Also, I am not very confortable with this level of complexity, the versioning should be an acessory level, that you could take it or leave it without much change to the overall format of the AIP.
As such, I would like to suggest the following:

Leave the AIP layout and storing as it was
Have versioning support nested in an optional sub-folder under the AIP (named for example as "versions"), even at the cost of redundancy
Versions use the OCFL format, and are always expanded (i.e. no use of TAR or other containers), so it is quite clear what changes occured from previous version.
The AIP usual folders, such as metadata and representation, will always provide the "current" version of the AIP.
SIPs or any other "submission" might be archived in another folder (e.g. named "submissions") and have the submission datetime (in ISO 8601 with Z timezone but without ':') recorded as SIP updates tend to have the same file name. The SIP should be archived inside the AIP as it was received.
Relationships between submissions, action and AIP versions should be described in PREMIS. As such, it should be possible for PREMIS events to refer to a specific version of an AIP as the source or outcome object.
Versions and Submissions are optional, their use will have a significant impact in storage and processing and their use should be defined by implementation and easy to switch on and off.

Given this, I would like to suggest the following layout of an AIP with two versions, were the first version was created by a SIP and the second version is just an update of the descriptive metadata.
urn+uuid+81bd3aa2-7350-44f6-ad54-d8181858605a
├── metadata
│   ├── descriptive
│   │   └── ead.xml
│   ├── metadata.json
│   ├── other
│   │   ├── processing.log
│   │   └── state.json
│   └── preservation
│       ├── premis_202401094-230854Z_event_sipcreation.xml
│       └── premis_20240409-230854Z_event_ingest.xml
├── METS.xml
├── representations
│   └── 09502a26-f822-407c-ad0a-4d7e64052a91
│       ├── data
│       │   └── example.pdf
│       ├── metadata
│       │   └── preservation
│       │       └── premis.xml
│       └── METS.xml
└── schemas
│   ├── csip.xsd
│   ├── ead3.xsd
│   ├── IP.xsd
│   ├── mets_1_11.xsd
│   ├── premis-v2-2.xsd
│   └── xlink.xsd
└── versions
│   ├── 0=ocfl_object_1.0
│   ├── inventory.json
│   ├── inventory.json.sha512
│   ├── v00000
│   │      └── metadata/descriptive/ead.xml
│   │          (all other files in AIP originaly received from the SIP)
│   └── v00001
│          └── metadata/descriptive/ead.xml
|              (descriptive metadata updated on the digital preservation archive)
└── submissions
    └──2024-04-10T11-57-00Z
       └── example.sip.001.zip

I don't have strong feelings, and I don't have skin in the game. I'm not quite able to make like for like comparison because one or two points in @luis100 response aren't clear to me. I think you're suggesting no BAGIT and no use of TAR for OCFL versions. I'm inclined to agree about BAGIT; I'm not sure we gain much from its use and much of the metadata is redundant. I agree that using TAR archives in versions is perhaps a bit messy and obscures the content/metadata changed.

Versions and Submissions are optional, their use will have a significant impact in storage and processing and their use should be defined by implementation and easy to switch on and off.

Storage and processing impact IS and implementation detail to some degree. Institutional policy/budget/choice will also be a factor. Making them optional appears a sensible decision.

Note that the OCFL format does not belong to the AIP. This is just one possible way how to store the original SIP (in this example as v00000) if you want to keep it, and the versioned AIPs are separate instances of AIPs. We moved away from integrating the versions into the AIP since E-ARK3.
Packaging as TAR/ZIP/etc. is a technical implementation detail that depends on storage system and requirements. It is adequate if the packages need to be transferred and may be a good approach if you have a tape system where the AIPs are stored for the long-term. However, if the AIPs are still being updated, the continuous re-packaging causes a lot of processing and redundancy.
The question here was about the use of BagIt to wrap E-ARK AIPs. In E-ARK, the manifest is included in the METS, but bagit has a simpler, non-XML format (payload manifest) for this purpose.

The AIP working group discussed the use of BagIt and recommends to take it out of the main recommendation for AIP packaging. Instead, it would be moved to an appendix where it will be explained how to wrap E-ARK information packages using BagIt. As optional BagIt packaging is also relevant for the SIP, it should be added to the CSIP rather than to the AIP. This decision is independent from the use of OCFL which will be dealt with in a separate issue.
The suggestion is:

Move the BagIt recommendation for packaging to a CSIP appendix

Board members acknowledgment of the issue:
Tick the box in front of you name to indicate that you have looked at the suggestion.

[ ] Karin Bredenberg (Kommunalförbundet Sydarkivera, chair)
[ ] Anders Bo Nielsen (National Archives of Denmark)
[ ] Anja Paulič (National Archives of Slovenia)
[ ] Arne-Kristian Groven (National Archives of Norway)
[ ] Gregor Zavrsnik (Geoarh)
[ ] Janet Anderson (Highbury Research & Development Ltd.)
[ ] Maya Bangerter (Swiss Federal Archives)
[ ] Miguel Ferreira (KEEP Solutions)
[ ] Stephen Mackey (Penwern Limited)
[ ] Sven Schlarb (Austrian Institute of Technology)

Voting
(Decision making will be carried out on the basis of majority voting by all eligible members of the Board. In the case of a tied vote, decisions will be made at the discretion of the Chair)
Tick the box in front of you name to say yes to the suggestion.

[ ] Karin Bredenberg (Kommunalförbundet Sydarkivera, chair)
[ ] Anders Bo Nielsen (National Archives of Denmark)
[ ] Anja Paulič (National Archives of Slovenia)
[ ] Arne-Kristian Groven (National Archives of Norway)
[ ] Gregor Zavrsnik (Geoarh)
[ ] Janet Anderson (Highbury Research & Development Ltd.)
[ ] Maya Bangerter (Swiss Federal Archives)
[ ] Miguel Ferreira (KEEP Solutions)
[ ] Stephen Mackey (Penwern Limited)
[ ] Sven Schlarb (Austrian Institute of Technology)

6 DILCIS Board members have acknowledge the issue
6 DILCIS Board members agree with the solution
The BagIt recommendation for packaging will be removed to to a CSIP appendix

authors: carlwilson karinbredenberg luis100 shsdev

repo: DILCISBoard/E-ARK-AIP


text:
Reformat the README

We don't need all the descriptions for each of the dependencies. There are also some spelling errors through out the document that need to be fixed.

Fixed

repo: DIRECTLab/EVPRE

license: MIT


text:
Require user's confirmation when creating new A or C from Suggester whose label is identical to an already existing one

[Problem and rationale:]
It's great that it is so easy to create entities, but it is also dangerous for the coherence of the database, and thus the analysis. I have myself created 2 or 3 spurious Cs by mistake even during short work in InkV.
PLOGE are usually new ones even with the same label, so there any protection by confirmation message would be spurious. In STR it is not so relevant, they are not so often created (or not from Suggester).
By contrast, A and C really hold the database together for analysis and ensure it is correct and in correct hierarchies.
[Request:]
Therefore, I suggest to implement, upon new A and C creation from Suggester, a check of existing A (in the case of a new A) or C (in the case of a new C) for the same label (full match only). If it is the same, prompt the user with the message:
"[Concept | Action] with the same label already exists. Are you sure you want to create a new one?"
Enter to confirm the creation, Escape to cancel.

@adammertel: I think we can drop this, since the newly created A or C opens in Detail and can be easily deleted there if it was created by mistake. Closing.

repo: DISSINET/InkVisitor


text:
GitHub actions / pytest fails

Describe the bug
The GitHub action with the pytest fails, not due to the test itself but because pytest causes problems.

Closed by #35

repo: DLR-AE/LoadsKernel


text:
Check token validation / expiration on plan sharing

Some users encounter issues when a plan has been shared with them and they don't follow the link to sign up to the tool.
If they come to register later, the tool doesn't allow them to sign up as they haven't followed the invitation link.
The invitation should expire OR if they come up to register not via the invitation, this action should invalidate the invitation and remove any access permissions to the shared plan
The user should always be able to register

Same issue as #2422. Think it may be fixed as creating an account is possible now when a previous unused plan share invite has been issued.
Close out both?

We're not sure if this is still an issue. @briri will check and update the issue.

authors: mariapraetzellis sjDCC

repo: DMPRoadmap/roadmap

license: mit

license_source: bigquery


text:
Issue 2418

Fixes #2418 .
Changes proposed in this PR:

Change text in app/views/paginable/plans/_privately_visiblehtml.erb line 64 from 'Make a Copy' to 'Copy'

@briri Please squash & merge this because it's accidentially on top of the  tag work

Just replaced the original since the change was a one-liner.
Pulled the latest from dev. Created a 'issue_2418' branch locally. Made the text change and committed changes. Then force pushed back to overwrite the original branch.

source: gharchive/pull-request

authors: briri martaribeiro xsrust


text:
make alert subscription optional

Currently not providing the alarms topic causes the scripts to fail. Checking the topic to ensure its provided is now used to skip the subscription.
Types of changes

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist

[x] I have read the CONTRIBUTING.md doc.
[x] I have added necessary documentation (if appropriate).
[x] Any dependent changes have been merged and published in downstream modules.

As the sns_topic is a required variable to provide the aws_db_event_subscription resource, that pull request fix the module to be more coherent.

authors: caiovfernandes matthosking-mydeal

repo: DNXLabs/terraform-aws-db-monitoring


text:
🛑 Fakultas Hukum is down

In 518a6be, Fakultas Hukum (https://lawfaculty.unhas.ac.id) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Fakultas Hukum is back up in ec449e1 after 12 minutes.


text:
🛑 Unhas Main is down

In 95e2639, Unhas Main (https://unhas.ac.id) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Unhas Main is back up in e5a43c0.


text:
🛑 Fakultas Kedokteran Gigi is down

In 557c3c5, Fakultas Kedokteran Gigi (https://dent.unhas.ac.id) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Fakultas Kedokteran Gigi is back up in fd070e4 after 3 minutes.


text:
🛑 Apps is down

In 562bf42, Apps (https://apps.unhas.ac.id) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Apps is back up in 5934150.


text:
Misc edit community and collection fixes

References
https://github.com/DSpace/dspace-angular/issues/665
https://github.com/DSpace/dspace-angular/issues/666
Description
This PR addresses some issues with the Community & Collection edit pages.
Instructions for Reviewers
List of changes in this PR:

(DSOSelectorModalWrapperComponent): the mechanism to automatically select the current Community/Collection in the new/edit menu has been fixed
(ComcolMetadataComponent): the request method used by the edit Community/Collection page has been changed from PUT to PATCH to repair the edit functionality
(ComcolMetadataComponent): an appropriate error message is shown when the user doesn't have sufficient edit rights. I didn't add error message for other cases, because I don't think a 'Bad Request' or an 'Internal Server Error' message would have an added value to most end users.
(ComcolMetadataComponent): I've split up the logo <> metadata logic, to make sure both can happen at the same time

I've tested the following cases, both for communities and collections:

creating a new community/collection with metadata and with a logo
creating a new community/collection with metadata and without a logo
adding a community/collection logo without editing the metadata
deleting a community/collection logo without editing the metadata
adding a community/collection logo while editing the metadata
deleting a community/collection logo while editing the metadata
editing community/collection metadata without changing the logo
trying one of the above as a user without permission, to verify the error message (I used a local REST server to test this, to be able to easily deny the permission)

@atarix83 That is an issue in master as well: #706
Seems recent, because it wasn't there when I tested this PR last week.
But you're right it makes sense to fix it in the context of this PR. We'll look in to it.

@atarix83 it should be fixed now. Can you verify?

authors: artlowel samuelcambien

repo: DSpace/dspace-angular


text:
Hybrid model for fast light sim

PDFastSim_dune.fcl is modified to allow the merged library to be used for the non-active volume part. In the photpropservices_dune.fcl, the dune10kt_vd_1x8x14_photonvisibilityservice_ArXe service is also declared for the library.

The summary of the absorption length, reflectivity for the semi-analytical model and the the contribution of the merged library are listed in the slides here.

Thank you, Jiaoyang. I think there is one issue with the library, though. I also invite @msorel to comment on that. Protodune data (both SP and DP) saw a cut in Ar light and the fact that not all late late goes into Xe. We are implementing that for the semi-analytical model using the parameters updated in this PR. Unfortunately, since there is just one library, we cannot do the same at the detsim stage for the external volume. If we want to keep things consistent, this has to go into the merged library.

repo: DUNE/duneopdet


text:
Add thread message positions

Pull Request Etiquette

[X] I have checked the PRs for upcoming features/bug fixes.
[X] I have read the contributing guidelines.

Changes

[X] Internal code
[X] Library interface (affecting end-user code)
[X] Documentation
[ ] Other: _____

Closes Issue: #2305
Description
This adds the position property from the message object.

Older messages have missing positions, I assume this is before discord added the property, should it be documented ?


text:
Add support for HKDF (RFC 5869).

Implementation based off the pbkdf2 code, except this is fast, and not
for password hashing.

Thanks for submitting this. I don't have time to take a look right now, but I'll try to take a look tomorrow.
Thanks!

@calvinmetcalf Fixed.  Still getting a hang of this rust thing.
I also went and changed the HKDF-Expand code to eliminate the helper function, since unlike PBKDF2, just generating T(N) in the loop body is compact enough.  The newer code is easier to read/maintain I hope.

Thanks!

authors: DaGenix Yawning

repo: DaGenix/rust-crypto


text:
Date values parsed in wrong format

Hello, @DaSpawn thank you for a great work with this package. I've looked deeply in it and tried several documents. There is my fork: https://github.com/kikill95/xlsx-stream-reader
Can you please take a look and let me know what is wrong?
I am working with example/example4.js and example/example5.xlsx files. In workBookReader.workBookSharedStrings I got the following:
[ 'a', 'b', 'c', '11/04/2014', '04/01/2016' ]

But I believe this should be
[ 'a', 'b', 'c', '09/01/1981', '11/04/2014', '04/01/2016' ]

And on 'row' for workSheetReader I received in "row.values" something strange instead of '09/01/1981': I got this: "29595"
I know that you had experienced with this more than me, so maybe this is easy for you to recognize where is the issue.
p.s. I have tried your master branch and received the same results

Also, I saw one another issue https://github.com/DaSpawn/xlsx-stream-reader/issues/8
I tried to resave (and even reexport), but same issue here =(
p.s.s. in my fork I've also fixed https://github.com/DaSpawn/xlsx-stream-reader/issues/13 and https://github.com/DaSpawn/xlsx-stream-reader/issues/11 and this https://github.com/DaSpawn/xlsx-stream-reader/issues/10

Interesting issue..  I did see this in the past, I will have to revisit as soon as I can
Thank you for looking at/working on those issues.  I have been incredibly busy with work, would you like to bring your changes into this project/be added as a collaborator?

Sorry if I comment on a closed issue, but may I ask if you could publish the 1.0.0 version on npm? @DaSpawn

I published to NPM (had to bump to version 1.0.1 first)

Thank you ! 👍

authors: DaSpawn kikill95 lambrojos

repo: DaSpawn/xlsx-stream-reader


text:
[BUG] ImportError: cannot import name 'startfile' from 'os' (/usr/lib/python3.10/os.py)

Describe the bug
Won't start
To Reproduce
Try to start
Expected behavior
GUI showing up
Log
$ speech-translate
Traceback (most recent call last):
  File "/home/user/.local/bin/speech-translate", line 5, in <module>
    from speech_translate.__main__ import main
  File "/home/user/.local/lib/python3.10/site-packages/speech_translate/__main__.py", line 38, in <module>
    from .ui.window.main import main  # pylint: disable=wrong-import-position
  File "/home/user/.local/lib/python3.10/site-packages/speech_translate/ui/window/main.py", line 23, in <module>
    from speech_translate.linker import bc, sj
  File "/home/user/.local/lib/python3.10/site-packages/speech_translate/linker.py", line 13, in <module>
    from speech_translate.utils.helper import generate_color, str_separator_to_html, wrap_result
  File "/home/user/.local/lib/python3.10/site-packages/speech_translate/utils/helper.py", line 9, in <module>
    from os import makedirs, path, startfile
ImportError: cannot import name 'startfile' from 'os' (/usr/lib/python3.10/os.py)

Desktop (please complete the following information):

OS: Linux Mint 21.2 x86_64
App Installation version: module
App / Python version: Python 3.10.12

Additional context
See https://stackoverflow.com/q/62578790

whoops i'm sorry about that. Forgot that it's a windows only import

it should be fixed now, you can try pip install -U git+https://github.com/Dadangdut33/Speech-Translate.git --upgrade --force-reinstall --no-deps


text:
깃허브 퍼스널 엑세스 토큰 노출에 따른 보안 위협

관련 대화: https://discord.com/channels/775115965964222492/1289773425207803964/1294869211331428434
별도의 백엔드 없이 프론트엔드에서 바로 깃허브 API 연동을 하기 위해서는 엑세스 토큰이 사용자에게 노출된다. 우선은 엑세스 토큰에 읽기 권한만 부여하여 프로젝트를 진행하기로 했지만, 여전히 보안 측면에서 모범 사례에서 벗어나는 관행이다. 뿐만 아니라 나중에라도 실수로 엑세스 토큰에 쓰기 권한을 주었을 경우 우리 깃허브 조직 아래에 있는 데이터가 위험에 빠질 수 있다.
본 티켓을 통해 깃허브 엑세스 토큰을 좀 더 안전하게 관리할 수 있는 방법을 탐구하자.

@Sunjae95 @HC-kang 당장 blocker는 아니지만, 프로젝트 후반에 재고해보면 좋을 것 같아서 일단 티켓만 생성해놓겠습니다. 혹시 조사하신 부분이 있으시면 댓글로 간단히 달아놓으시면 좋을 것 같습니다.

토큰을 Fine-grained personal access tokens과 Personal access tokens 두가지 방법으로 만들수있습니다. 참고
제가 생각하기엔 Fine-grained personal access tokens은 beta버전이기에 적합하지않고 Personal access tokens으로 api 요청을 해야한다 가정하고 조사했는데요.
Discord에서 @HC-kang 님도 언급하신 내용처럼 프로젝트가 읽기가 필요한 api 토큰이기에 아래이미지처럼 토큰을 생성시 권한을 제한하는것이 좋아보입니다.

org의 read 권한으로 토큰생성해서 팀과 멤버에 대해서만 팀생성 api테스트했을 때 403 오류가 나와서 정상작동하는 걸로 확인돼요.
토큰 옵션 설정 범위 참고
제가 파일읽기 api는 어디 속해있는지 찾지못해서 이 부분 @HC-kang 님이 확인후 공유해주시면 감사합니다 😊

디렉토리 정보 조회에 대한 API는 Get a tree이고, 저희 레포(leetcode-study)가 Public 레포인지라 별도의 토큰 발급은 불필요합니다!

제가 생각하기엔 Fine-grained personal access tokens은 beta버전이기에 적합하지않고 Personal access tokens으로 api 요청을 해야한다 가정하고 조사했는데요.

@Sunjae95 PAT를 쓴다면 GitHub에서 권고하는 데로 fine-grained personal access tokens를 선택해야합니다.

If possible, GitHub recommends that you use a fine-grained personal access token instead of a personal access token (classic)

https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api?apiVersion=2022-11-28#authenticating-with-a-personal-access-token

https://github.com/DaleStudy/leaderboard/issues/60 이 이슈를 해결하기도 하고 토큰을 실제로 사용할 순간이 왔습니다.
일단은 임시로 @DaleSeo 님이 만든 기간 제한의 PAT를 사용하되 어떤 방법이 리스크가 적은지 조사를 병행해야겠습니다.

https://github.com/DaleStudy/leaderboard/issues/60 이 이슈를 해결하기도 하고

@sounmind 님, 흠... 두 이슈가 관련이 있지만 그래도 완전히 별개의 이슈가 아닐까요? #60 은 브라우저 환경에서 process 전역 변수를 사용했기 때문에 발생하는 버그이고요. 본 이슈는 어떻게 안전하게 GitHub API와 연동하는냐는 좀 더 깊은 고려가 필요한 문제로 보입니다.

@DaleSeo 말씀이 맞습니다. 별개의 이슈입니다.
Github API를 사용하려고 보니, 관련 코드에서 #60 이슈가 발생한 것이고,
Github API를 사용하기 위한 방법도 다시 알아봐야 했는데 #14 이슈가 다시 언급됐네요.

현재 상황

PAT를 프론트엔드에서 직접 사용 중
토큰이 노출되어 보안 위험 존재

가능한 대안
1. PAT 현행유지

현재처럼 계속 사용
단점

토큰 관리 소요와 부담이 큼
보안 위험 지속

2. 신규 계정의 제한된 PAT 사용

새로운 깡통 계정 생성 / 팀 멤버로 추가
읽기 전용, 무기한 PAT 발급
장점

가장 적은 노력으로 개선 가능
토큰 노출시 피해 최소화

단점

여전히 토큰 노출됨
계정 관리 소요 필요

3. GitHub App 사용

GitHub App 생성 및 설정 필요
매 요청마다 Private Key로 JWT 생성 후 Installation Token 발급하는 로직 필요
장점: 토큰 만료로 인한 보안 강화
단점

수정사항이 매우 큼
Private Key 노출되어야 하므로 여전히 보안 위험 존재

4. GitHub Actions 활용

leetcode-study 레포에서 Action으로 JSON(등등) 생성/업데이트
leaderboard에서는 public JSON만 참조
장점: 토큰 불필요, 가장 안전한 구조
단점: 저장소에 알고리즘 풀이 외의 코드(데이터)가 추가되어야 함

권장 순위

GitHub Actions (4)
신규 계정의 제한된 PAT (2)
GitHub App (3)
현행 유지 (1)

authors: DaleSeo HC-kang Sunjae95 sounmind

repo: DaleStudy/leaderboard


text:
Bug: Client server settings reset

I have the application completely set up.
After clicking on "Manual download" and downloading the media, the client settings are reset and you have to enter them again.

Can you describe the menu selection procedure in more detail?

My quick setup for Download settings looks like this:

Setup source (ID/username)
Setup download folder
Manual download

Yes, after you press Manual download  and wait to finish its work, you'll have to set up client settings again. It's not supposed to be like this, right?

Can you show the screenshots with comment?

authors: DRMPN DamianMorozov

repo: DamianMorozov/TgDownloader


text:
update

Describe what this PR does / why we need it
Does this pull request fix one issue?

Describe how you did it
Describe how to verify it
Special notes for reviews

update

repo: DamonChow/spring-cloud-alibaba


text:
Team Meeting 11/13/23

Agenda:

POC Demo

Feedback

Use GitHub Issue Tracker to track tasks instead of Gdocs
Refine documents using other team feedback
Start setting up CI
Commit more often with smaller changes instead of one big commit
Use co-author commits for overleaf

TODO

Start refining documents
Star working on closing github issues

Next Meeting

Sometime next week

repo: DangJustin/CapstoneProject


text:
Reflect @angular/cdk dependency requirement in readme

Please reflect @angular/cdk dependency requirement in readme.
As of 6.0.5 the app won't work without it.

there is a peerDependencies in package.json

authors: DanielYKPan Deilan

repo: DanielYKPan/date-time-picker


text:
About the heatmap and paf value

hi, thanks for your nice work, but i still have some questions about the code.
1). keypoint_map[map_y, map_x] += math.exp(-exponent)
As mentioned in the original paper, it takes maximun of the confidence maps, but in the code, it takes sum of the confidence maps.  So I wonder why you take the sum, maybe it's a trick？I modified this line of code like this: keypoint_map[map_y, map_x] = max(keypoint_map[map_y, map_x] , math.exp(-exponent)), and the experiment is still going on.
2)paf_map[0, y, x] = x_ba
paf_map[1, y, x] = y_ba
paf values takes the average values, but your code implements didn't take the average.

You can leave max, it shouldn't make a lot of difference.
Paf corresponds to single person, so average values between pafs of two persons makes them unsuitable for both, so just set one of pafs.

Hope, it helped.


text:
Fix Windows build

Fix build error reported in https://github.com/Dao-AILab/flash-attention/issues/814
I don't have a way to test this on Windows, but @bottler pointed out that long keyword is the likely problem. Is there a way to run Windows CI on this PR? @tridao

I got it compiling on windows by changing to this:

@shaltielshmid I think that your version would also be a valid fix, equally as good as this one.

@sgrigory I successfully built FA2 on windows using this PR. The Github Actions log is located here and the release is here
Thanks for the PR!

authors: bdashore3 bottler sgrigory shaltielshmid

repo: Dao-AILab/flash-attention


text:
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.5.1

镜像域名 (镜像仓库)

[ ] 请确保包含域名(domain)

镜像标签

[ ] 请确保包含镜像的标签(tag)

镜像存在

[ ] 请确保这个镜像真实存在

yes

repo: DaoCloud/public-image-mirror


text:
docker.io/phpipam/phpipam-cron:latest

白名单级别
只需要这一个镜像 (如 docker.io/library/busybox)
镜像仓库地址
https://hub.docker.com/r/phpipam/phpipam-cron
这是 镜像仓库 认证可信的 验证过的发布者(Verified Publisher)或者赞助的项目(Sponsored OSS) 么?
是的 - 不需要填下面的信息 - 如果实际上不是将不再继续核查
项目源码地址 或 组织地址
No response
官网 或 文档 或 项目源码 中哪提及对应的镜像的地址 (需要证明这个镜像和源码有实际关联)
No response
补充说明
No response

不是受镜像仓库验证的镜像, 也没有提供对应的资料, 推荐使用 #2328 里其他镜像源


text:
Discussion on BTC module construction

Hello everyone,
I have noticed that our repository already includes some content related to Bitcoin (BTC). To better continue and expand the task mechanism we use in dapp learning, I have organized the related topics and tasks into a detailed outline, which I have published in this Issue. The purpose of this is to facilitate task claiming, thereby not only improving our work efficiency but also ensuring the quality of content.
Below is the outline I propose, with each section detailing the topics and subtopics that need to be explored. Please review it and claim any sections you are interested in the comments. If you have any suggestions or additional content you’d like to add, please feel free to bring it up.
1. Basic Features
1.1 Payment Addresses

P2PKH
P2SH-P2PKH
P2WPKH
P2TR
Base58
Bech32
Bech32m

1.2 Wallets

Standard Wallets
HD Wallets

1.3 Transactions

Transaction Construction
Fee Estimation
Signature Algorithms

1.4 Tools

Blockchain Explorers
API Services
Data Analysis Tools

2. Advanced Features
2.1 PSBT (Partially Signed Bitcoin Transaction)

PSBT Protocol Introduction
Creating and Parsing PSBT

2.2 Taproot

Schnorr Signatures
MAST
Privacy and Efficiency Improvements

2.3 Multisig Addresses

Multisig Address Process
Multisig Address Creation
Signing Process

3. Protocol Introduction
3.1 BRC20
Protocol Introduction

Functionality and Purpose
Comparison with Other Standards

Code Implementation
3.2 ARC20
Protocol Introduction

Functionality and Purpose
Application Scenarios

Code Implementation
3.3 Runes
Protocol Introduction

Design Philosophy
Key Features

Code Implementation
3.4 SRC20
Protocol Introduction

Standard Specifications
Interaction Modes

Code Implementation
4. Ecosystem Applications
4.1 Babylon
Protocol Introduction
Core Logic
4.2 RGB
Protocol Introduction
Core Logic
4.3 Lightning Network
Protocol Introduction
Core Logic
etc...

Looks good!
Maybe we can add more content to Payment Address, such as multi-signature address, in the future

Looks good! Maybe we can add more content to Payment Address, such as multi-signature address, in the future

After the basic content is complete, we can add more advanced usage
PRs welcome

Looking forward to the content related to Taproot
Another question, how to claim the related tasks?

Looking forward to the content related to Taproot
Another question, how to claim the related tasks?

We will sort out the relevant outlines to ensure the quality of the content, all discussions can be held under this issue
I think you can directly reply to the content you are interested in, Welcome your PR

I think I can help organize the taproot address related information😎

Let's buidl together!

I added the principles and applications of MAST in Taproot. More discussion and guidance are welcome.https://github.com/Dapp-Learning-DAO/Dapp-Learning/pull/1144

The cryptocurrency world is ruthless, but the world is full of love. We will continue to build✊

I added the principles and applications of MAST in Taproot. More discussion and guidance are welcome.#1144

The content is quite complete and good, but the analysis of specific cases can be increased

repo: Dapp-Learning-DAO/Dapp-Learning


text:
[FEATURE] Add extend options for marks

Is this a unique feature?

[X] I have checked "open" AND "closed" issues and this is not a duplicate

Is your feature request related to a problem/unavailable functionality? Please describe.
Yes
Proposed Solution
Add .extend function to every mark
Screenshots
No response
Do you want to work on this issue?
Yes
If "yes" to the above, please explain how you would technically implement this (issue will not be assigned if this is skipped)
No response

Should marks also be extendible like plugins, to have a certain API to follow and you can add own marks?

@maZahaca the main point is add extend functionality for custom rendering of marks and adding some default props to each mark.
Ex: some devs want to extend props for highlight to add custom colors for text and backgroud.

Ok, so adding other marks sounds like another task. But can we define certain api protocol for it, as we have for plugins?


text:
when aternos account is in more than 5 servers list_servers() returns empty list

i deleted a few servers from the accounts list and my previous issue was resolved, it appears that theirs an unlisted max number that it can have before it stops working.

Thank you so much for the provided information.
I'll check your servers HTML code today and answer here.
5-servers limit is very strange.

And also, could you enable logging as I said in #30 ? Just add to the beginning:
import logging
logging.basicConfig(level=logging.DEBUG)

yup


text:
Add ability for user to not display "Cancel" button

I've updated to 0.1.33 and noticed that there is now a "Cancel" button on the DownPicker.  However, I don't  need a "Cancel" button and added a property "shouldDisplayCancelButton" that defaults to "YES" (so that nothing changes for the normal user), but can be set to "NO" if a user doesn't want it to be shown.

Done! Thanks for you kind work.

Any update on that updated pod?  =)

Done! Sorry for the delay, had some issues :)

Thank you very much!

authors: Darkseal rbaumbach

repo: Darkseal/DownPicker


text:
[Feature] Update-VsTeamWorkItem to add a tag

Feature request: Update-VsTeamWorkItem to add a tag to a workitem.
I envision it would work something like this:
Update-VsTeamWorkItem -Id 1 -Tag TestTag1
Note: I removed the array (multiple) tags, as that's more complex than I am comfortable with doing.

@joeywas regarding your PR, this issue can be closed, right? Hope you are not too frustrated that you made effort for your change. But I am happy that the current solution works for you. Version is already published and should work for you.

Yes, it can be closed. Not frustrated, i am still fairly new to
collavorating on github so it's good learning experience!
Joseph
On Tue, Sep 3, 2019, 10:04 Sebastian Schütze notifications@github.com
wrote:

@joeywas https://github.com/joeywas regarding your PR, this issue can
be closed, right? Hope you are not too frustrated that you made effort for
your change. But I am happy that the current solution works for you.
Version is already published and should work for you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/DarqueWarrior/vsteam/issues/183?email_source=notifications&email_token=AAG2MWSNYIV6PK3YXFA4J4TQH2KILA5CNFSM4IL72MQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5Y4FWQ#issuecomment-527549146,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAG2MWQ5HU2TZRKLNR4MTETQH2KILANCNFSM4IL72MQQ
.

authors: SebastianSchuetze joeywas

repo: DarqueWarrior/vsteam


text:
PyPI/conda-forge release

Hi! I'm wondering if you all might be considering a release of the package on PyPI and/or conda-forge?
Happy to help guide the process if a lack of knowledge is the main barrier here, have done this for other bioinformatics-oriented tools.

Hi Eric! Glad you find the software useful. It is mostly bandwidth on our end that has prohibited a pypi release. If you'd be willing to contribute in that way, we'd be most grateful!

Happy to help, there are some specific things I could do, such as registering a package name on PyPI first and ensuring that you all have the relevant permissions to push packages to that PyPI package project -- followed by handing over ownership to you all. Releases to PyPI are much lighter weight than doing a peer-reviewed paper :joy_cat:. Let me know if you'd like that.
I also know the conda-forge process and bots well enough to have them do auto pulls from PyPI to conda-forge as well, which will ensure that you have distributions on both PyPI and conda-forge. We can proceed with that afterwards.
Let me know how you'd like to proceed!

authors: HWaymentSteele ericmjl

repo: DasLab/RiboGraphViz


text:
add cbmem boot time measurment doc

Signed off by Przemysław Banasiak [przemyslaw.banasiak@3mdeb.com]

@PrzemyslawBanasiak13 @damiankaras @macpijan based on this PR I have prepared additional PR with minor changes, which fits the current construction of test documentation. Therefore, this topic is closed.

repo: Dasharo/docs


text:
[IA-3224] Tab Navigation in Cloud Env Modal

There was an extra, non-visible close button (circled in red in the image below) that was being focused on when a user tried to tab through the modal. I fixed this by changing the tabindex value of the close on the TitleBar component depending on whether the passed in hideCloseButton property. If I did not render the element at all, then the spacing of the arrow on the modal would be affected.

Tested by going through the modal to make sure that the close was not able to be tabbed to if it was not visible and was able to be tabbed to if it was.
I also included a small fix to the Environments page, as I noticed that the total disk price was not being displayed properly on the page.

It would still be nice to figure out the focus when going from one screen to the next in the wizard (I know you already spent some time on it).

It would still be nice to figure out the focus when going from one screen to the next in the wizard (I know you already spent some time on it).

Yeah, the IA team and I discussed making another ticket to try and figure that out in the future, but other work has come up that is more pressing

authors: cahrens cstrandw

repo: DataBiosphere/terra-ui


text:
Update ua parser js

Updated the ua-parser-js library, which was flagged as a security issue in WX-942.
While we don't depend on ua-parser-js directly, two packages react-json-view and outdated-browser-rework were using it. I tested that these still work by interacting with these components/pages on my local front end. I can indeed still view JSON, and I am redirected when using an oudated browser.

This does not upgrade existing instances of this package. It adds a new (unused) instance of the package. You can tell because the old version still shows up in yarn.lock.
See https://github.com/DataBiosphere/terra-ui/wiki/Security-and-Maintenance#dependabot-security-updates for instructions on how to update transitive dependencies.

My mistake. Will look into it this afternoon and resubmit.

Fixes made, followed the documentation this time around 😅 . Repeated the testing and was still able to interact successfully with relevant components.

Ah, I see. The extra updates are from yarn dedupe.
Not sure I agree that a yarn dedupe step should be included in Dependabot updates, but it should be fine and can't fault this change for following the documented process.

authors: THWiseman nawatts


text:
[Serverless] Always call TellDaemonRuntimeStarted before TellDaemonRuntimeDone

What does this PR do?
https://github.com/DataDog/datadog-agent/blob/785d1bb777158919b2a78dc92112c46cf940b9b7/pkg/serverless/serverless.go#L187-L196
There's a bug here. If this is the first invocation of this Lambda function, and the context starts in a timed out state, it is possible TellDaemonRuntimeDone can be called before TellDaemonRuntimeStarted. TellDaemonRuntimeDone relies on TellDaemonRuntimeDoneOnce to be not nil, and TellDaemonRuntimeDoneOnce is not initialized until TellDaemonRuntimeStarted is called. This causes a segmentation error. 😢
https://go.dev/play/p/RXUKJ1OAoWg
Motivation
SLES-842
Additional Notes

Possible Drawbacks / Trade-offs
I am not 100% confident this is the only place we are calling HandleInvocation
Describe how to test/QA your changes

Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the config template has been updated.

I can't believe this worked.
--- FAIL: TestHandleInvocationShouldNotSEGSEVWhenTimedOut (0.00s)
    serverless_test.go:68:
                Error Trace:    serverless_test.go:68
                                                        panic.go:838
                                                        panic.go:220
                                                        signal_unix.go:818
                                                        once.go:57
                                                        daemon.go:324
                                                        serverless.go:196
                                                        serverless_test.go:78
                Error:          Expected no panic, instead got
                Test:           TestHandleInvocationShouldNotSEGSEVWhenTimedOut
                Messages:       runtime error: invalid memory address or nil pointer dereference
FAIL
FAIL    github.com/DataDog/datadog-agent/pkg/serverless 1.654s

repo: DataDog/datadog-agent


text:
[workloadmeta] Fix panic in image_util

What does this PR do?
Fixes the following panic during SBOM generation:
2023-11-14 13:24:47 UTC | CORE | INFO | (pkg/util/containers/metrics/provider/registry.go:99 in collectorDiscovery) | Container metrics provider discovery process finished
2023-11-14 13:24:47 UTC | CORE | WARN | (pkg/workloadmeta/collectors/internal/containerd/containerd.go:175 in stream) | error getting image: image "eu.gcr.io/datadog-staging/datadog-agent@sha256:9d4fadd7f8b445c11d444fc91e9d6c30f306ed07608604584a226291bda9c9b4": not found
2023-11-14 13:24:47 UTC | CORE | WARN | (pkg/workloadmeta/collectors/internal/containerd/containerd.go:175 in stream) | error getting image: image "sha256:df4eef994d45da531c16a05922fb9bcd4e1cb3071d4ccb48ba795829863beae8": not found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3cd31b6]

goroutine 454 [running]:
github.com/DataDog/datadog-agent/pkg/workloadmeta.updateSBOMRepoMetadata(0x0?, {0x0?, 0x7bd52af?, 0x5?}, {0x0?, 0x0?, 0xc001ce9000?})
	/omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/workloadmeta/image_util.go:23 +0x36
github.com/DataDog/datadog-agent/pkg/workloadmeta.(*ContainerImageMetadata).Merge(0xc000840500, {0x8fc4950?, 0xc00222a900?})
	/omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/workloadmeta/types.go:940 +0x5f
github.com/DataDog/datadog-agent/pkg/workloadmeta.(*store).handleEvents(0xc001e22a20, {0xc003470060, 0x1, 0x0?})
	/omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/workloadmeta/store.go:700 +0xcb4
github.com/DataDog/datadog-agent/pkg/workloadmeta.(*store).Start.func1()
	/omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/workloadmeta/store.go:98 +0x145
created by github.com/DataDog/datadog-agent/pkg/workloadmeta.(*store).Start
	/omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/workloadmeta/store.go:91 +0x95

Motivation
Agent pods are crashlooping in staging with this error
Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes
Follow the set up steps outlined in this PR: https://github.com/DataDog/datadog-agent/pull/20437
Then delete an image from a node. The agent running on that node should not crash as a result
Reviewer's Checklist

[ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
[ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
[ ] A release note has been added or the changelog/no-changelog label has been applied.
[ ] Changed code has automated tests for its functionality.
[ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
[ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
[ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
[ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
[ ] If applicable, the k8s/<min-version> label, indicating the lowest Kubernetes version compatible with this feature.
[ ] If applicable, the config template has been updated.

I think we should either revert https://github.com/DataDog/datadog-agent/pull/20437 or merge https://github.com/DataDog/datadog-agent/pull/20874 as the change introduced in 7.50.0 has a release note announcing a bugfix which is clearly not fixed.

Closing in favor of https://github.com/DataDog/datadog-agent/pull/20874

authors: AliDatadog sblumenthal


text:
[SLVS-4017][serverless-init] Fix incorrect _dd.origin tag for Azure + GCP serverless services

What does this PR do?

Runs a check for the cloud service and sets the correct _dd.origin value when we call ModifySpan.
Live staging example where Container App traces are marked with _dd.origin: lambda: traces (click the trace JSON tab and then open in a new tab to inspect)
Correct _dd.origin for Cloud Run: https://ddserverless.datadoghq.com/api/v1/trace/8871360755274513971?colorBy=service&env=dev&graphType=flamegraph&shouldShowLegend=true&sort=time&spanID=8781288428123163570&spanViewType=metadata&timeHint=1708981920802.639&view=spans
Motivation

Traces from serverless-init were always set as _dd.origin: lambda due to code being reused from the Lambda Extension. There was some hardcoding we did initially for AWS Lambda that was carried over.
Additional Notes

Added a unit test and manually tested.
Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


text:
[EBPF-377] Change ebpf errors metrics to prometheus.Counter

What does this PR do?

Changes the ebpf errors telemetry metrics to Counter instead of Gauge (the metric count total number of errors in different ebpf helper functions)
Renamed the metrics to prevent type confusion on the backend
Introduced ebpfErrorsTelemetry interface to allow easy mocking in UTs

Motivation
Monotonic counter is a more suitable type for this kind of metric, since we measure number of errors which increase over time and never decrease, hence it is more intuitive to see those metrics in our dashboards as counters
Additional Notes
I will add additional UTs to check the delta calculation is done properly this time
Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


text:
[comp/core/tagger/tagstore] Fix Lookup funcs documentation

What does this PR do?
Deletes some comments that no longer apply from the documentation of a couple of TagStore functions.
Describe how to test/QA your changes
Skip.

/merge

/merge -c


text:
[DBM] Fix SQL table collection bug

What does this PR do?
This PR bumps pkg go-sqllexer to v0.0.12 to fix a bug where table name is incorrectly collected on statements CREATE TABLE IF NOT EXISTS <table>. Prior to this fix, keyword IF was incorrectly parsed as a table name.
Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


text:
Update 6.8.1 changelog

What does this PR do?
Update 6.8.1 changelog

Codecov Report

Merging #2801 into 6.8.x will decrease coverage by 4.54%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            6.8.x   #2801      +/-   ##
=========================================
- Coverage   61.64%   57.1%   -4.55%
=========================================
  Files         263     383     +120
  Lines       17195   24556    +7361
=========================================
+ Hits        10600   14022    +3422
- Misses       6033    9610    +3577
- Partials      562     924     +362

Impacted Files
Coverage Δ

...g/util/kubernetes/apiserver/metadata_controller.go
55.3% <0%> (ø)

pkg/util/kubernetes/apiserver/diagnosis.go
18.18% <0%> (ø)

pkg/util/containers/cri/diagnosis.go
28.57% <0%> (ø)

pkg/clusteragent/custommetrics/store.go
65.04% <0%> (ø)

pkg/util/hostname_nix.go
0% <0%> (ø)

pkg/util/docker/global.go
0% <0%> (ø)

pkg/util/pipe_nix.go
0% <0%> (ø)

pkg/util/kubernetes/apiserver/apiserver.go
9.95% <0%> (ø)

pkg/collector/corechecks/system/iostats_nix.go
71.6% <0%> (ø)

pkg/util/kubernetes/kubelet/pod.go
82.35% <0%> (ø)

... and 127 more

@mfpierre Thanks for this! Can you also:

make the same changelog update on the master branch (IMO a cherry-pick to master works here since the change was already reviewed)
create a github release (i.e. https://github.com/DataDog/datadog-agent/releases/tag/6.8.1) with the same changelog? (you'll have to convert it to markdown, see our internal docs)

authors: codecov-io mfpierre olivielpeau


text:
[OTEL-2239] Convert comp/serializer/compression into new style components

What does this PR do?
This PR modifies comp/serializer/compression into the new "component" style https://datadoghq.dev/datadog-agent/components/overview/.  One kind of odd thing it does is creates a selector package - this package contains build time flags to select which implementation to use based on build tags.  This is existing logic that I just ported over - I am not trying to change the behavior of anything.
Motivation
We want to split up this package so we can pull in different implementations to avoid cgo dependencies in other packages.
Describe how to test/QA your changes
Unit tests continue to pass

Wondering of we can simplify things a bit more.
Looking at the tags that we use to build the different Agent builds in the build_tags I can see that most if not all flavors use zlib and zstd. Could we simplify the selector permutations?
I understand that the otel Agent do not use build tags and wants to avoid the use CGO, so you are deliberate using the zlib implementation, but the rest of the Agents allow to select the compression base on configuration option so I would assume that both build tags are present for non Otlp Agent

I agree in principle but frankly I don't know much about the agent and its different distributions.  My team doesn't work on this code and I chose to keep the logic the same since I don't know why this piece was included in the first place.  Would be easy to tear out in the future

Could we remove the component from the TODO list of components to migrate?
https://github.com/DataDog/datadog-agent/blob/d838cbd3b8e820062d0be6ed48f3911dba16f387/tasks/components.py#L146

Already done!
https://github.com/DataDog/datadog-agent/pull/30688/files#diff-f048224c30106b98e0d1e2937484c4667dab647d7175f85407c2889746fed8e0L146

I agree in principle but frankly I don't know much about the agent and its different distributions. My team doesn't work on this code and I chose to keep the logic the same since I don't know why this piece was included in the first place. Would be easy to tear out in the future

I believe shortly OTel will be looking to incorporate the Native Go zstd implementation to use instead of the cgo one. When that happen I imagine those build tags would be obsolete and we should probably just have something like otel_no_cgo. When set it will choose the native Go zstd package, when unset it will choose cgo.

/merge


text:
[SAM] Hangs for the entire lambda timeout, then responds

Problem
Hi, I'm using AWS SAM and I'm running locally with sam local start-api. However, when I add the dd lambda extension layer, it causes my handler to run for the full timeout before responding normally. From my local logs, I can see that my handler is returning a value almost immediately, so the delay is from the dd lambda wrapper. This makes it very difficult to work locally, which is important to me. My guess is maybe it's failing to authenticate with datadog and timing out? But, it's not emitting any auth errors. Is there any way to get more information about why it's hanging?
Sequence of events

I start my local API using sam build, then sam local start-api.
Send request using postman
My lambda runs and returns a response almost immediately. There are a few logs from datadog:

Lambda function 'MyFunction' is already running
START RequestId: 21d0304a-779d-4b7a-b895-5b1142303bff Version: $LATEST
2024/08/23 16:27:05 {"status":"error","message":"datadog: Couldn't convert X-Ray trace context: Couldn't read trace id from X-Ray: invalid x-ray trace id; expected 3 components in id"}
2024/08/23 16:27:05 Datadog Tracer v1.65.1 INFO: DATADOG TRACER CONFIGURATION { <long configuration json> }
2024/08/23 16:27:05 MyFunctionHandler: returning response

The lambda runs for the full 30 second timeout, then it logs the following and sends the response back to postman.

END RequestId: 7bb76fa1-2e9e-4ba5-9486-d8cbeac70bcc
REPORT RequestId: 7bb76fa1-2e9e-4ba5-9486-d8cbeac70bcc  Init Duration: 0.15 ms  Duration: 29984.04 ms   Billed Duration: 29985 ms       Memory Size: 128 MB       Max Memory Used: 128 MB

Relevant code
bootstrap.go:
func main() {
	lambda.Start(ddlambda.WrapFunction(handler, nil))
}

func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
        // Do handler logic

        log.Printf("MyFunctionHandler: returning response")
        return events.APIGatewayProxyResponse{
                StatusCode: 200,
                Body:       "Success",
        }, nil
}

template.yaml:
Resources:
  MyLambdaFunction:
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: go1.x
    Properties:
      CodeUri: myproject/
      Handler: bootstrap
      Runtime: provided.al2023
      Architectures:
        - x86_64
      Layers:
        - arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension:64
      Policies:
        - AWSSecretsManagerGetSecretValuePolicy:
            SecretArn: "arn:aws:secretsmanager:us-east-1:<my account>:secret:DD_API_KEY_MY_PROJECT"
      Environment:
        Variables:
          DD_SITE: 'datadoghq.com'
          DD_API_KEY_SECRET_ARN: 'arn:aws:secretsmanager:us-east-1:<my account>:secret:DD_API_KEY_MY_PROJECT'

Datadog Go Lambda package version: github.com/DataDog/datadog-lambda-go v1.19.0
Go version: go1.22.6 darwin/amd64

Hey @distinctdan - thanks for reaching out!
I suspect that the reason is because the AWS Lambda Runtime Interface Emulator is being used by SAM.
Unfortunately AWS does not emulate the telemetry API in the runtime interface emulator.
The telemetry API is how Lambda Extensions communicate with the AWS Lambda Runtime, and we require it to tell us important things like:

logs from your function
runtime metrics
and most crucially
runtime events, like when your Lambda function is finished processing.

Today we can't support the emulator because of this limitation. Instead, we recommend using a conditional variable to skip the Lambda Extension and Lambda Instrumentation when emulating locally.
This has been an issue for a long time, so we'd appreciate if you leave a comment asking for this feature from AWS and mention it to your TAM.
Thanks!
AJ

Thanks for the info! I've commented on their issue.
Since this is a known issue where we know DataDog isn't compatible with SAM local, I would request a documentation update on the DD side for the following:

Warning message that this doesn't work.
Workaround code to conditionally turn on DD only when not running locally.

Thanks @distinctdan - I've learned that we document the DD_LOCAL_TESTING env var here, but I'm not sure exactly the full breadth of support and I don't expect Datadog's instrumentation to work locally due to the disparities between Lambda and other local testing options.
It may prevent the extension from hanging in your case though. Either way, our recommendation is to disable Datadog in a local emulation environment.
Thanks!

Thanks for the info, but unfortunately it looks like that flag doesn't change the behavior for me. At the very least, I would expect the DD lambda to emit some kind of log saying it can't connect instead of silently failing. I know it's not y'all's fault that this doesn't work, but I've spent several hours tracking this down, and some documentation or error logs on the datadog side would have made this a lot easier.

Hi @distinctdan, thanks for the note.
The connection from the extension to the telemetry API can be established, the issue is that no runtime events are passed (only logs). So for the extension, there's no way to know that we'll never receive those events.
I will ask our documentation team to more clearly describe our recommendation to not use Datadog's instrumentation locally at this time.
Thanks!

I see, yeah that's pretty unfortunate. Ok, thanks for the help!

Hi @distinctdan - if you try out our new next-generation Lambda extension locally, you can specify a short flush interval via DD_SERVERLESS_FLUSH_STRATEGY=periodically,1 which should no longer wait for any events. I don't expect the data to be correct or coherent in Datadog, but it should at least allow the function to run: https://github.com/DataDog/datadog-lambda-extension/pull/377
I hope this helps!

authors: astuyve distinctdan

repo: DataDog/datadog-lambda-go


text:
RUMM-2209: Prepare version 1.4.0 release

What does this PR do?
Update changelog and version for the version 1.4.0 release.
Review checklist (to be filled by reviewers)

[x] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[x] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[x] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report

Merging #89 (c5f37ce) into release/1.4.0 (f1288ce) will increase coverage by 0.45%.
The diff coverage is n/a.

@@                 Coverage Diff                 @@
##             release/1.4.0      #89      +/-   ##
===================================================
+ Coverage            89.96%   90.40%   +0.45%
- Complexity             147      148       +1
===================================================
  Files                   20       20
  Lines                  448      448
  Branches                50       50
===================================================
+ Hits                   403      405       +2
+ Misses                  28       27       -1
+ Partials                17       16       -1

Impacted Files
Coverage Δ

...in/kotlin/com/datadog/gradle/plugin/DatadogSite.kt
100.00% <0.00%> (+13.33%)
:arrow_up:

repo: DataDog/dd-sdk-android-gradle-plugin


text:
RUM-6314: Use layout text to display textview overflow correctly

What does this PR do?
Fix the Session Replay issue that text view overflow could not be displayed properly
Motivation
RUM-6314
Demo

App screen
Before
After

Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[ ] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report
Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 70.05%. Comparing base (26c28e2) to head (150f273).

Files with missing lines
Patch %
Lines

...id/sessionreplay/recorder/mapper/TextViewMapper.kt
0.00%
0 Missing and 1 partial :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2279      +/-   ##
===========================================
- Coverage    70.06%   70.05%   -0.01%
===========================================
  Files          732      732
  Lines        27224    27221       -3
  Branches      4580     4580
===========================================
- Hits         19073    19068       -5
+ Misses        6877     6868       -9
- Partials      1274     1285      +11

Files with missing lines
Coverage Δ

...id/sessionreplay/recorder/mapper/TextViewMapper.kt
89.90% <0.00%> (ø)

... and 31 files with indirect coverage changes

authors: ambushwork codecov-commenter

repo: DataDog/dd-sdk-android


text:
Update dependency version in envoyproxy/envoy to 0.1.11

Hi 👋 - hope this is the right place. I opened this issue here rather than in Envoy because it seems like @dgoffredo is a maintainer here, and is the one updating the version in Envoy. If this is not the right place, my apologies.
It seems like Envoy is still using the 0.1.8 version of this library here. This is true as of the latest release (yesterday, 1.19.4). It should probably be updated to 0.1.11. I'm not too sure how involved this is, or I would open a PR - plus I think it's probably more likely to be accepted by the maintainers here.

Hi, Ben. A recent release of dd-trace-cpp contains changes that would break the Datadog-related unit tests within Envoy.
My teammate is working on a PR within Envoy that will make the tests compatible with more recent versions of dd-trace-cpp.
Then, we will update the version of dd-trace-cpp used within Envoy.
Is there a feature or fix in more recent versions of dd-trace-cpp that you're looking for in Envoy?

Great to hear.
The issue we're running into is that with Envoy 1.27 the span names are different than with 1.26. In 1.26 the span names are envoy.proxy, while in 1.27 they show up as (what looks to be) the name of the service. This in turn leads some metrics to go missing (specifically trace.envoy.proxy.hits). At least - I think that's what's happening. I haven't got too much hands on experience with tracing / cpp though.
That said, when I dug a bit into the tracing code and old PRs, I saw that the changes here (specifically the change to setOperation) don't exist in release/v1.27 of envoy. I think the backport here was probably missing some things.
It's a little late for me, but would be happy to try updating the backport tomorrow if this sounds like it's the right direction.

Yes, the "operation name is wrong in v1.27 and v1.28" was fixed and backported onto those release branches. You can pick up those changes whenever Envoy produces another release in either series. That will fix that problem.
The "setting the 'resource.name' tag doesn't work anymore" issue was fixed on the main branch, but the process of proposing backports has stalled. Perhaps I can propose the backport PRs myself.

I haven't worked on Envoy much, but I found these docs if it's helpful. Specifically, they say that if you add backport/review as a label, that starts the process (apparently).

Just to follow up, I opened backport PRs against 1.27, 1.28, hope that's ok:

https://github.com/envoyproxy/envoy/pull/30892
https://github.com/envoyproxy/envoy/pull/30893

Interesting experience getting everything running locally, but tests were passing locally for both patches.

Hello everyone,
I work for Wayfair, and we had a support case with DD number 1339031. I've been following multiple issues in the past two months, and it seems that the problem is finally resolved. If I understood correctly, we would still need new builds for Envoy 1.27.x and 1.28.x versions, and they will be included in the next Istio patch versions. Is that correct?
Are you planning to upgrade the official documentation once this is done? It appears that the latest supported version is v.1.17, and that was what the support was referring to when we were complaining about traces being broken in Istio v.1.19.
Could you also describe any changes related to the DD tracing library that we can expect in Istio versions newer than v1.17, such as the one related to the resource name not being taken from the tag resource.name? These changes may require some of additional adjustments on our side, and we need to communicate this to developers.
Including these details in the official documentation would help to clarify that it's related to DD Istio integration.

Hi, Njegos. I don't know which case number 1339031 refers to. There is Zendesk, which I believe Datadog uses to communicate with customers about Support issues, and then we use another system internally when Support interacts with Engineering. What is the issue that you're referring to with number 1339031?
The table in the environment variables section of the documentation you linked is indeed not up to date. I wasn't aware that it exists.
The changes to Envoy that have caused these issues (the unexpected change in operation name, the inability to set the resource name via the "resource.name" tag, and an unintended sampling override) were made to move Envoy from an older OpenTracing-based Datadog tracing library to this newer tracing library (dd-trace-cpp). So, the version numbers tabulated will not be comparable for present and future versions of Istio that use the newer Envoys. On the other hand, dd-trace-cpp has major version zero (0.x.y), so there might not be any confusion.
I'll see to it that the documentation is updated. We have a work item to revisit our Istio documentation more extensively, as methods of deployment have changed since we initially developed the integration.
None of these problematic changes were intentional, and so I can't anticipate what others, if any, might be found. Here's a list of recent Envoy Datadog tracing issues (i.e. issues starting with Envoy v1.27) that I know about:

The default Datadog span name changed for spans created by Envoy: https://github.com/envoyproxy/envoy/pull/29932
The ability to set the Datadog resource name by setting the "resource.name" tag stopped working: https://github.com/envoyproxy/envoy/pull/30503
When Envoy's sampling rate (not to be confused with Datadog's sampling rate) is less then 100%, then drop decisions made by Envoy override any decision extracted from incoming request headers (this can break traces): https://github.com/envoyproxy/envoy/pull/30577

Backports onto Envoy v1.27 and v1.28 are merged for all of these issues ((2) thanks to @Smeb :tada:)
The relationship between Istio and the Datadog tracing library is the following:

Istio has a version of Envoy that it calls istio-proxy. istio-proxy is based on a release of Envoy. The correspondence between Istio releases and the Envoy releases on which istio-proxy is based are tabulated here.
Envoy builds dd-trace-cpp from source and combines the library with other Datadog-specific code within Envoy (this is where I put the bugs).
The version of dd-trace-cpp on which Envoy depends is determined by envoy/bazel/repository_locations.bzl within the Envoy source tree.
In summary:

A release of Istio tracks a particular release of Envoy.
A release of Envoy contains both Datadog-specific code and a particular version of dd-trace-cpp, determined by the build.
dd-trace-cpp contains the core Datadog tracing code, and is released as a source tarball on GitHub.

Hi David,
Thank you for the detailed updates and explanations. To be honest, I don't personally use DataDog, but our developers heavily rely on it. In early October, we initiated Istio upgrades, but had to revert in most of the clusters due to the "issue" with tracing.
Starting from Istio v1.18, we observed significant visual differences in Datadog tracing. Some of the main issues include, but are not limited to:

the function changing from envoy.proxy to <service-name>_<url for cluster>.
the istio.operation tag seems to be dropped or turned off.
runtime-id and process_id are missing from traces.

More details are available in the linked issue. When we approached support, we were informed that this is related to the DataDog tracing library that Envoy uses and that DataDog doesn't support Istio versions beyond v1.17, according to their documentation.
I plan to open a new support case with additional details that I cannot share here. I will mention this issue and then discuss it with our account manager.
Our primary concern is understanding when we can expect the same tracing functionality we had in previous versions. If not, could this be documented?

@njegosrailic please share the support case number with me once it is created, so I can inquire about it internally.
Your first bullet point is the "operation name" issue discussed previously. The other two changes of behavior that you mention are news to me. istio.operation I haven't heard of before. The runtime-id and process_id tags are added by the underlying tracing library, so it is a surprise that they are missing.

Hi @dgoffredo,
I have created a new support case https://help.datadoghq.com/hc/en-us/requests/1453323 and I'm providing the old one https://help.datadoghq.com/hc/en-us/requests/1339031 for reference.
Thank you for the help.

This will be resolved when https://github.com/envoyproxy/envoy/pull/31526 is merged.

envoyproxy/envoy now uses dd-trace-cpp v0.1.12.

Note - this is only true for the latest release (1.29) which isn't currently available in Istio. Should we backport the changes again? The changes look non-trivial, but I guess they're all in the same place.

I recommend that we do not backport a newer version of dd-trace-cpp onto Envoy release branches v1.27 and v1.28.
There have been several Datadog tracing problems in Envoy since v1.27, but all of them are due to changes made in Envoy's Datadog extension, not in the separate dd-trace-cpp library.

As a customer of Datadog this warning sent me down the path to finding this issue:

For me this indicates I should do something about it but after reading this thread I think I'm hearing this is an envoy/DD relationship. If it's true that as a customer I don't have the ability to fix it the warning may be a bit much.

Thanks for the details @dgoffredo ! I created an internal support ticket: 1565944 if that helps.

authors: Smeb brettcurtis dgoffredo njegosrailic

repo: DataDog/dd-trace-cpp


text:
vSphere docs updates

What does this PR do?
Adds resource_filters option + additional notes for vSphere docs to reflect changes that shipped with DD Agent 6,7.18+
Motivation
Jira request / docs channel request
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

@cswatt Removed a few extra options and created a card to add the new config options (will be a separate PR) - good for final review 👍

repo: DataDog/integrations-core


text:
chore: improve CodeCov configuration

Upload all reports to CodeCov at once at the end of the validation workflow, so that it stops producing mid-flight coverage reports that don't account for the entire set of reports.
Also, configure coverage collection during integration test execution.
Noticed the linter configuration was outdated and largely inert, so I fixed it up and applied A TON of linter fixes. The rule set is obviously subject to discussion & may also be updated in the future.

Cannot merge because the changes made in the workflows make the Complete job fail in the merge_group CI because it does not take into account the new jobs that are skipped

authors: RomainMuller eliottness

repo: DataDog/orchestrion


text:
optimise OwnedHeaders::insert

We pass length of header's key so in such case librdkafka doesn't calculate its length and relies on the passed value. Additionally it adds null byte at the end - https://github.com/confluentinc/librdkafka/blob/10f988f493695ee2ff180d9927f1e881ab98a36d/src/rdkafka_header.c#L96 so no point of creating c string in rust code.
I think that issue is there since forever - https://github.com/fede1024/rust-rdkafka/commit/56366ae8c1443318aa4036097c6fbc879f93e238#diff-e070e0bbef3fd7da263a02197f5b3ba50d6b086f1a882a639dcf70a03db48704R376.

replaced by https://github.com/DataDog/rust-rdkafka/pull/9.

repo: DataDog/rust-rdkafka


text:
feat(GCP): Exfiltrates a Compute Image by sharing it

What does this PR do?
Introduces gcp.exfiltration.share-compute-image where data is exfiltrated by sharing a Compute Image.
Motivation
Inspired by https://www.mitiga.io/blog/google-cloud-platform-exfiltration-a-threat-hunting-guide mentionned in
To answer before getting out of draft

[ ] Can we make it not "Slow"?

Thanks for the PR! Looks great overall, left a few comments

will be released as part of v2.12.0 shortly

authors: christophetd vthiery

repo: DataDog/stratus-red-team


text:
[golang] Unskip parametric tests for injection/extraction of p in tracestate

Motivation
Unskips the parametric tests for golang for asserting the functionality of the p key in the tracestate.
Changes

Unskips the tests.
Target the correct commit version of go

Workflow

⚠ Create your PR as draft ⚠
Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
Mark it as ready for review

Test logic is modified? -> Get a review from RFC owner. We're working on refining the codeowners file quickly.
Framework is modified, or non obvious usage of it -> get a review from R&P team

:rocket: Once your PR is reviewed, you can merge it!
🛟 #apm-shared-testing 🛟
Reviewer checklist

[ ] Relevant label (run-parametric-scenario, run-profiling-scenario...) are presents
[ ] No system-tests internal is modified. Otherwise, I have the approval from R&P team
[ ] CI is green, or failing jobs are not related to this change (and you are 100% sure about this statement)
[ ] A docker base image is modified?

[ ] the relevant build-XXX-image label is present
[ ] To R&P team: locally build and push the image to hub.docker.com

[ ] A scenario is added (or removed)?

[ ] Get a review from R&P team
[ ] Once merged, add (or remove) it in system-test-dasboard nightly

Done in a different PC, closing.

repo: DataDog/system-tests


text:
Fix issue when running single test due to missing docs

Description
When running a single test it was failing due to a missing entry in the docs dictionary:
runner           |   File "/app/conftest.py", line 440, in _pytest_junit_modifyreport
runner           |     junit_modifyreport(
runner           |   File "/app/utils/scripts/junit_report.py", line 35, in junit_modifyreport
runner           |     test_doc = _docs[nodeid]
runner           | KeyError: 'tests/appsec/iast/test_iast_sql_injection.py::TestIastSqlInjection::test_secure_sql'

Check list
Your PR is not ready to be reviewed? Please save it as a draft :pray:

[x] Follows the style guidelines of this project (See how to easily lint the code)
[ ] CI is passing
[x] There is at least one approval from code owners

Yes to all? Feel free to merge it whenever you want :heart:

@cbeauchesne The Ruby tests are broken in master, right? Can we merge this one?

authors: manuel-alvarez-alvarez smola


text:
🛑 Gitlab is down

In 6f1f613, Gitlab (https://gitlab.data-ensta.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Gitlab is back up in 72482df after 9 minutes.

repo: DataEnsta/upptime


text:
🛑 Angry is down

In d4962c6, Angry (https://angry.data-ensta.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Angry is back up in 2b34096 after 9 minutes.


text:
First medfilt proposal

Maybe not a one liner, but readable, I think. With 2 simple doctests.

@grzanka I've just realized... I've only removed scipy from requirements.txt.is it hidden somewhere else?

Check https://github.com/DataMedSci/beprof/blob/master/install_deps.sh

https://github.com/DataMedSci/beprof/blob/master/.travis/install_linux.sh

And setup.py

Well, I have some more 'cleaning' to do then. Like:
PYTHON2_CMD="apt-get install -y libblas-dev liblapack-dev gfortran"
PYTHON3_CMD="apt-get install -y libblas-dev liblapack-dev gfortran"```
gfortran was required for numpy or?

Probably scipy required gfortran.

Ok, I have some trouble then - should I remove whole block?
https://github.com/DataMedSci/beprof/blob/master/.travis/install_linux.sh#L14
line 14 - 18? Similar problem in _deps.sh

they might be useful in the future... I would comment out those...?

Remove it. We can retrieve it from the repo history.
21.06.2016 12:57 AM "ant6" notifications@github.com napisał(a):

they might be useful in the future... I would comment out those...?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/DataMedSci/beprof/pull/76#issuecomment-227294727, or mute
the thread
https://github.com/notifications/unsubscribe/AHCHjWpxBG8Q2ArWt4KrxWt0LdZurT3Tks5qNxr0gaJpZM4I6Llq
.

See my last comment here: https://github.com/DataMedSci/beprof/pull/76/files#diff-f5ada602d1e1e10fbf2362ecd2a5f645R52

Just pushed it ;)

Thanks. I'll merge once tests will pass

repo: DataMedSci/beprof


text:
Documentation for draw()

On page for draw() API method
replace

Please note that the string options require or newer

with

Please note that the string options require DataTables 1.10.8 or newer

Thanks. Fix committed and will be pushed out soon.

authors: DataTables mpryvkin

repo: DataTables/DataTables


text:
[FEATURE] Azure Synapse support?

A new user of Data Vault modelling paradigm here. First of all, huge thanks for implementing the dbtvault project.
I was wondering if it would be possible to add support for Azure Synapse Data Warehouse? As Azure takes more market share in cloud computing, I believe supporting Synapse will help propel the familiarity of Data Vault modelling and help end users like us to adopt it rapidly by using dbtvault.

Hi,
I’m interested in contributing for MS Fabric.
What do you think?

authors: GolamRashed MiguelElGallo

repo: Datavault-UK/automate-dv


text:
Dev

Updated docs and Github CI
Package version (main branch)
version: 0.3.5
Description
Updated documentation styling and Continuous Integration scripts with Github.
Problem

docs were unreadable
package wasn't tested on Windows

Solution

changed docs style
written CI GA script

Affected modules

docs

Unit tests

n/a

Package check

[n/a] All tests passed
[x] Documentation updated
[n/a] All tutorials are working properly

(Optional) Additional info
n/a

I will open this PR with a package update ;)

repo: DataverseLabs/pyinterpolate


text:
Rename temperature metric

Fix a typo in temperature metric name and conform to standards set by official Prometheus exporters by shortening temperature to temp.
bsbmp_temperature_celcius -> bsbmp_temp_celsius
One can't mistake temp (temperature) for temporary as there is celsius in metric name.
Resolves #15

LGTM, Thanks.

authors: David-Igou krystiancha

repo: David-Igou/bsbmp-exporter


text:
Unintuitive handling of .markdownlintignore files

It took me a while to realize that .markdownlintignore is not treated as an ignore file but rather in a custom way.
For example, this doesn't work ❌:
node_modules
/ignored-folder

This works ✅:
**/node_modules/**
ignored-folder/**

It's described in the README but I still think it would be much better if the standard "ignore" format was used.

And my "✅" updated example is even wrong – the second pattern should be **/ignored-folder/** (I learned this painfully when lint-staged was reporting markdownlint errors for MD files in the ignored folder; it uses an absolute path so my original glob didn't match it).

This is explained here: https://github.com/igorshubovych/markdownlint-cli/issues/108#issuecomment-661944570
As you note, everything you want to do is possible with the current implementation. It's just slightly different than usual. I haven't come up with a way around this yet, though I'll keep thinking about it. Would using a different file name for the ignore file have helped avoid the expectations you had that were wrong?￼

Yes, for example, if it was an ignoreGlobs key in .markdownlint.json, that would make it cleaner.
But even better would be to support the ignore format, ideally with hierarchy as well (https://github.com/DavidAnson/markdownlint-cli2/issues/2). I don't know how ripgrep or fd do that but for example, fd can find *.md files instantly between the 180k files I have in the current repo (most of that gitignored node_modules):
$ time fd -g '**/*.md' .
fd -g '**/*.md' .  0.02s user 0.02s system 270% cpu 0.014 total

The ignore configuration can't (currently) be in one of the config files like you suggest because those files are only found and parsed as part of the command-line globbing (and it's too late to start ignoring then).
Regarding your timings, the current implementation of markdownlint-cli2 is fast, also! I'm trying to avoid an implementation (like markdownlint-cli uses) that will be slow. :)
The obvious choice to implement this behavior consistent with .gitignore - and the one I made for markdownlint-cli - is with this package: https://github.com/kaelzhang/node-ignore. HOWEVER, that basically returns a filter function you feed every file to find out if it should be ignored. As covered in the related issue, it's too late at that point to avoid enumerating large directories like node_modules which means things can be very slow even when there are only a small number of Markdown files.
I want to do the ignoring as an integral part of the globbing operation which is why markdownlint-cli2 behaves like it does today. I can definitely name the ignore file something like .markdownlint-cli2-extra-globs, but I don't yet see how to change the implementation.

The ignore configuration can't (currently) be in one of the config files like you suggest because those files are only found and parsed as part of the command-line globbing (and it's too late to start ignoring then).

That makes sense, thanks for the explanation.

Regarding your timings, the current implementation of markdownlint-cli2 is fast, also!

cli2 is fast and I love that but achieves it by explicit globbing. My point was that fd can be equally fast while respecting the hierarchy of .gitignore files, which is a much harder problem.

I can definitely name the ignore file something like .markdownlint-cli2-extra-globs

I'd probably vote for that but it's a bit ugly and "proper" ignore support would be better, which leads me to:

[...] but I don't yet see how to change the implementation.

I also don't know if there's a Node.js module that would process ignore files and be fast at the same time 😀. For example, globby suffers from exactly the same problem: https://github.com/sindresorhus/globby/issues/50#issuecomment-467897064.
We once needed to find project.json files in a monorepo quickly (this also needs to avoid many such instances in node_modules) and I think we used vscode-ripgrep for that – it's a package that downloads the ripgrep binary and uses that. The Rust magicians somehow managed to implement globbing + gitignoring in a blazing fast way so we utilized their work.
I think fd is more appropriate tool for the job and I just found that there's an npm package for it: https://www.npmjs.com/package/fd-find, so maybe that's some option.
(I realize that plain Node.js implementation would be better but Rust binaries are cross-platform and usually unbelievably fast.)

Quick comments since I don't have a lot of time right now:
CLI2 uses the globby package you reference above, so all of that discussion is directly relevant here.￼ I chose this package after some research, but I can look at the alternatives mentioned in that thread.￼
I had an idea this morning that I feel optimistic about. I will remove support for .markdownlintignore and add a ignoreGlobs property to .markdownlint-cli2.jsonc. ￼It will still be fast when that file is in the base directory for the same reason that ignores are fast today. It will be slower in child directories, but that is less common and will still enable that scenario with the added benefit of using the same syntax everywhere. It will additionally enable the use of CLI2 without any command line parameters by specifying the command line globs in the config file in the base directory.￼￼￼
I have a philosophical objection to using precompiled binaries from a Node application. It hurts platform-independence and is much harder to audit from a security perspective. That is a fine idea, but not something I am really open to.￼

Since I believe ignore files would be the best UX, I tried to do another search for packages that would be able to do it. My conclusion:

Most packages simply don't handle .ignore files in a performant way.
Those that pride themselves on speed are often wrappers around ripgrep or fd, like https://www.npmjs.com/package/fast-replace.
https://www.npmjs.com/package/ignore-walk seems promising – at least it seems to do the right thing. Not sure about its performance or how to combine it with globs but it's probably the most promising package I've encountered.
https://www.npmjs.com/package/deglob combines globs and .gitignore support but probably doesn't support hierarchical ignore files.
https://www.npmjs.com/package/glob-gitignore specifically focuses on performance, see their Why section, but it doesn't seem to support ignore files, just gitignore-like array of patterns.

So nothing directly usable, I think.
I was also thinking about making fd/ripgrep opt-in, like markdonwlint-cli2 --use-fd, which would have this binary file dependency but be 1000x faster in return. The trouble is, probably, that to make the base cli2 reasonably fast, you'll need to invent your own performant way to ignore files, like the ignoreGlobs in .markdownlint-cli2.jsonc you mentioned, and this will be inherently incompatible with ignore-based tools.

Thanks for the investigation!

ignore-walk does not seem to accept globs for search
glob-gitignore does not seem to read ignore files
deglob seems to do what we want, but is missing the streaming results API that globby offers

CORRECTION: deglob only seems to handle ignore files at the root: https://github.com/standard/deglob/blob/61d0f01ddc86a5d9361246e176b2503b0acda835/index.js#L98
I was going to try deglob, though I'm worried about not being able to start linting until the entire search is done. However, root-only is a problem.
So I think I'll explore the .markdownlint-cli2.jsonc option I suggest above.

authors: DavidAnson borekb


text:
Include all formatters in Docker image

Creating this to track the discussion that happened in #93 about only the default formatter being available in the published Docker image (link to last comment of that discussion: https://github.com/DavidAnson/markdownlint-cli2/pull/93#issuecomment-1321302005).
In a CI pipeline the current way to deal with this is to install the desired formatter before invoking markdownlint-cli2 (this requires specifying a different entrypoint. See an example here: https://github.com/DavidAnson/markdownlint-cli2/tree/next/formatter-codequality#use-in-gitlab-ci

@mschoettle You can (and please do!) now validate this using the container image tagged "next" in Docker Hub. That image happens to include your Code Quality formatter as well. :)

authors: DavidAnson mschoettle


text:
Group minor/patch Dependabot updates

Major versions will still create individual PRs, but minor/patch ones are grouped together like https://github.com/canada-ca/ore-ero/pull/1915

Thank you! I'm glad this capability exists, but I actually do appreciate that the current approach gives each dependency change its own PR and its own validation once accepted. It's a bit more work for me to accept things individually, but I'm OK with it and that's how I'm inclined to stay for now.

That's fine, the other part of this, is that even though CI is free, this reduces the load and rebase churn

It's a good point, but GitHub has never complained so far and I don't know how to tell Dependabot to split a PR if something fails. :)
Something that feels wasteful for no reason is how the checks run twice sometimes because a) branch was pushed and b) PR was updated. I want each of these events to cause checks to run, but don't need duplicate checks when the same commit happens to tick both boxes. I haven't looked into this, but suspect there is a way to be more efficient.

It's a good point, but GitHub has never complained so far and I don't know how to tell Dependabot to split a PR if something fails. :)

Seems kind of like the current way of giving it a mention:

@dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)

Something that feels wasteful for no reason is how the checks run twice sometimes because a) branch was pushed and b) PR was updated. I want each of these events to cause checks to run, but don't need duplicate checks when the same commit happens to tick both boxes. I haven't looked into this, but suspect there is a way to be more efficient.

I think you're OK for that case, as Dependabot Pushes aren't built https://github.com/DavidAnson/markdownlint/blob/b2305efafb034b1f328845aec9928b5363ffd646/.github/workflows/ci.yml
Feel free to close this, but I'll keep it open for now

I'm closing this because I don't see myself deciding to merge these unless something else changes. Thanks for the suggestion!

authors: DavidAnson nschonni

repo: DavidAnson/markdownlint


text:
how to map buttons?

i want to know how to remap de buttons specialy de r zr zl l

This is where the mapping happens. To remap, you'll have to change these and recompile.

repo: DavidPagels/retro-pico-switch


text:
Check if there is an issue with units in any profile model fitting

This came from the first attempt at the NFW mass profile for fitting the revamped hydrostatic mass profiles (#1260) - I set the $\rho_{0}$ start parameter and default prior to be in units of Msun/Mpc^3 (because it is easy to conceptualise), but the fits are garbage and the scale radius parameter ($R_{s}$) is weirdly tiny.
I think the model function is not receiving the 'quantity' version of the model parameters (i.e. the ones with units), so the mismatch between the profile radius being in kpc and the density being in terms of cubic mega parsecs is causing a huge issue.
I thought that units were preserved in XGA model fits, so just need to review the whole situation (and other models that use mismatched units like this, the density profiles for instance).

Yeah so I was altering the start parameters, but they are automatically converted to the original start parameter units - so it wasn't helping diagnose the problem.
When I actually changed the units in the model init to msun/kpc^3, we find that an actual fit is possible (if not perfect):

I will have to continue reviewing the actual why of this - though I fear what the repercussions might be

I AM SO HAPPY TO REPORT THAT I HAVE REVERSED THE PROBLEM:

I think this is now down to a plotting bug in the BaseProfile method - the fitted model has sensible parameter distributions and also seems to be producing believable mass estimates (4x10^14 Msol) within 1200kpc (a randomly chosen radius).
So now I'll take a look at the plotting method

It is now working as intended