text:
Systems Biology Gene Sets (Evidence)

@MichaelaEBI commented on Tue May 15 2018
@MichaelaEBI commented on Tue Mar 06 2018
Include Systems Biology-derived gene sets in the Open Targets platform under "Pathways & Systems Biology"

[x] Map out process to include Systems Biology gene sets
[x] Curate papers
[x] Define JSON for SysBio evidence
[x] Define scoring model for SysBio evidence (plan ahead for CRISPR & network module data)
[ ] Implement scoring model for SysBio evidence for 19.02 OT release
[x] Generate sample data (JSON) from IBD Key Driver Gene Set paper
[x] Identify and curate sample papers that identify key driver genes for various diseases
[x] Validate and process sample JSON file using pipeline code
[x] Define changes that need to be made throughout the platform

Entry for the scoring table for the 19.02 release: "Systems Biology (p-values or rank-based scores are used for scoring if provided (scaled to [0.5,1], otherwise a score of 0.5 is assigned)."

Has been implemented for 19.02.

repo: opentargets/platform


text:
--gen step hangs at the end when index is already existing

The --gen step seems to hang after INFO - join workers if the --gen step is executed without first removing the .._gene-data index.

@afaulconbridge is the --gen step supposed to work if the index already exists? Or should it actually raise an error in this case?

It should work fine, regardless of the index existing or not. I suspect this is another instance of opentargets/platform#177 causing problems.
For 19.04 and onwards, the multiprocessing on the --gen step has been removed, since in practice it didn't give a performance increase. This makes it more robust, since there is less opportunity for deadlock.

Ok, will close it for now since it is unlikely to return after 19.04

Actually, one question: what is it supposed to do if the index already exists? Should it drop the index and recreate it?

Yes, if it exists its designed to drop and recreate it. This ensures that it is created according to the appropriate configuration. Theres separate tickets to document and externaltise that configurations opentargets/platform#367 and opentargets/platform#504

source: gharchive/issue

authors: afaulconbridge pieterlukasse

license_type: permissive

license_source: github-api


text:
Generate pypi website scraper dataset

Scrape pypi for all the info needed in the research phase that can not be gotten from pypi json api
the dataset output format and location should be documented
This should be a new command like python -m score.cli pypi-web

Fields unique to the PyPi Webscraper:

summary
author
license
requires
dev_status
first_release
last_release
releases
monthly_downloads_pypi

All fields gathered by the PyPi Webscraper:
All the fields are type string except for releases

name:
version:
summary:
author:
license:
maintainer:
maintainer_email:
requires:
dev_status:
first_release:
last_release:
releases:
monthly_downloads_pypi:
monthly_downloads_conda:

To run the scraper you can run:
python cli.py scrape-pypi-web --letter 0-9
Where scrape-pypi-web is a call to the web scraper and --letter (optional) specifies the range of letters that you would like to scrape
The output destination would be as follows:
./score/output/web/letter={letter}/pypi_packages.parquet

repo: openteamsinc/Score


text:
Signed the manifesto

I'm happy to do what I can to help.

Added in https://github.com/opentffoundation/manifesto/pull/1031.

Thanks!

source: gharchive/pull-request

authors: Iwaslazkis flavius-dinu

repo: opentffoundation/manifesto


text:
[mac-frame] const-qualify the getter methods and other enhancements

This commit updates the Mac::Frame implementation by ensuring getter
methods are declared as const. Also to simplify the code this commit
defines new methods for some commonly used logic like reading/writing
a uint16_t value at a given buffer pointer or getting the Frame
Control Field (FCF). Simple getter/setter are also in-lined.

Pushed a new commit with following commit log:
[mac-frame] `const`-qualify the getter methods and other enhancements

This commit updates the `Mac::Frame` implementation changing getter
methods `const`. Simple getter/setter are also in-lined.

Also to simplify the code this commit defines new functions under
`ot::Encoding::LittleEndiaon and `ot::Encoding::BigEndian` for
commonly used  logic like reading/writing a `uint16_t`/`uint32_t`
value from/to a given buffer pointer.

repo: openthread/openthread


text:
[udp] forbid udp duplicate open

This commit forbid opening a UDP socket twice.

@jwhui Comments are resolved now. PTAL again.


text:
Starter module

Fixes #13

I wanted to include integration tests but first let's discuss it in #14

Can this get a review?

LGTM - although looks very simple, I guess it is just a placeholder for more dependencies later?

yes, It's about aligning with spring conventions and for adding needed dependencies.

authors: objectiser pavolloffay

repo: opentracing-contrib/java-spring-cloud

license_source: bigquery


text:
make "required" carriers match their calling patterns

Right now we have two "required" carriers: SplitBinaryCarrier and SplitTextCarrier. The current spec snapshot describes them here. I'll copy in the relevant passage:

Required Inject/Join formats
At a minimum, all platforms require OpenTracing implementations to support two formats: the "split text" format and the "split binary" format. The "split" refers to two common components of propagated spans:

The core identifying information for the Span, referred to as the "tracer state" (for example, in Dapper this would include a trace_id, a span_id, and a bitmask representing the sampling status for the given trace)
Any Baggage (per Span's ability to set Baggage items that propagate across process boundaries)

The "text" and "binary" designations refer to two flavors of encoding:

The text format is a platform-idiomatic map from (unicode) string to string
The binary format is an opaque byte array (and presumably more compact and efficient)

Note that there is no expectation that different tracing systems Inject and Join Spans in compatible ways. Though OpenTracing is agnostic about the tracing implementation, for successful inter-process handoff it's essential that the processes on both sides of a propagation use the same tracing implementation.

The carriers are "split" so that

OpenTracing users can opt-out of propagating baggage, and also so that
OpenTracing users can put the (potentially fixed-length) tracer state in a different place than the baggage in their respective wire formats.

The former can be accomplished via a flag or other control mechanism on the carrier or on the inject call. The latter doesn't really make sense in the text format, and really derives from a very particular requirement that Uber has, and it may make more sense for TChannel to define its own Carrier which it will prefer (and which Uber's OT implementation can support).
As such, here's a new proposal for the bulit-in carriers... the intention here is to make sure that the user can (a) always opt-out of Baggage, and (b) fit in with either HTTP headers or binary protocols.
I'll sketch these out in Go, but any language would do:
type MixedMapCarrier struct {
    // Inject() should include / Join() should extract Baggage items in
    // MixedMap iff BaggageEnabled==true.
    BaggageEnabled bool  // tempted to call this "CarryBaggage" :)

    // A map in which to store Tracer State and – depending on
    // BaggageEnabled – Baggage items.
    //
    // IMPORTANT: This map is *not* only used by the carrier... it may have
    // entries devoted to entirely unrelated things (e.g., arbitrary HTTP
    // Headers) and, as such, any carrier items must be distinguishable (e.g.,
    // via a shared prefix string).
    MixedMap map[string]string
}

type BinaryCarrier struct {
    // Inject() should include / Join() should extract Baggage items in
    // Buffer iff BaggageEnabled==true.
    BaggageEnabled bool

    // A buffer in which to store Tracer State and – depending on
    // BaggageEnabled – Baggage items.
    Buffer []byte
}

Thoughts? @yurishkuro I'm particularly curious what you think about the general proposal for TChannel above, etc.

Passing options in through the carrier types seems pretty fishy to me. Can't you just have two carrier types, one of which doesn't support attributes?

@bensigelman

I'm particularly curious what you think about the general proposal for TChannel above, etc.
I think I can live with that.

Can't you just have two carrier types, one of which doesn't support attributes?
Another alternative is to make them completely independent, i.e. instead of the carrier having two parts as before, have two calls, inject tracer state, and inject baggage. An RPC that doesn't want to propagate baggage simply doesn't call the 2nd method (incidentally this solves the TChannel issue via standard carriers again).

instead of the carrier having two parts as before, have two calls, inject tracer state, and inject baggage.

I like this idea. Carriers should carry data and that's it, keep any logic-y stuff out of it.
The current Inject function assumes both tracer state and baggage gets injected. It can just calls the two individual functions for injecting state and function. This way you don't break the existing API and get something like this.
func (t *Tracer) Inject(sp Span, format interface{}, carrier interface{}) error
  t.InjectTracerState(sp, format, carrier)
  t.InjectBaggage(sp, format, carrier)
}

func (t *Tracer) InjectTracerState(sp Span, format interface{}, carrier interface{}) error {
  // Inject tracer state the Span ID and sampled status
}

func (t *Tracer) InjectBaggage(sp span, format interface{}, carrier interface{}) error {
  // Inject the baggage
}

@tschottdorf fair point about keeping logic out of the carriers...
FWIW, there is also something similar to what @michaelsembwever proposed with this implementation helper PR in Java: https://github.com/opentracing/opentracing-java/pull/11/files#diff-f2ad9c00db83aa2576fe12ea2f7f0413R21 ... the part I'm really pointing to is that the Tracer itself could be configured to always support XOR ignore the baggage.
@yurishkuro's idea to create separate methods for injecting tracer state and baggage makes some sense to me... then again, Join(...) is trickier. @yurishkuro, I think Join(...) would run into trouble if different formats were used for the tracer state and baggage; this makes me think we want to strongly encourage the caller to do the injection in a single method call with a single carrier instance. Given that idea, here's yet another way to make this work:
type Tracer interface {
...
    Inject(sp Span, format interface{}, carrier interface{}) error
    InjectWithOptions(sp Span, format interface{}, carrier interface{}, InjectOptions opts) error
...
}

type InjectOptions struct {
    IncludeTracerState bool
    IncludeBaggage bool
}

different formats were used for the tracer state and baggage

I can't really imagine a case where this would happen, would you mind elaborating?

@bg451

different formats were used for the tracer state and baggage

I can't really imagine a case where this would happen, would you mind elaborating?

Sure... @yurishkuro was saying that, in TChannel, the tracer state would be embedded in binary form at some fixed offset in the RPC body, but the baggage would be key:value string pairs.

Ah okay. I didn't catch that in previous discussions, thanks!

Not sure where we are with this... I was thinking more about the above and realized that Baggage can still be opt-in without touching inject/join. An individual Tracer implementation can simply not-support Baggage, or it would even be possible to write a pass-through Tracer adapter that basically just delegates everything but drops all calls to SetBaggageItem(...).
Having done something like that, I think we'd be better off with the following two required carriers:
// SharedMapCarrier is a generic key:value text map in which to store/smuggle
// Tracer State and Baggage.
//
// IMPORTANT: The contents of this map are *not* restricted to Tracer State and
// Baggage: it is potentially shared with other entirely unrelated things
// (e.g., arbitrary HTTP headers). As such, any carrier map entries must be
// distinguishable in some way (e.g., via a shared prefix string).
type SharedMapCarrier map[string]string

// BinaryCarrier is an opaque binary container in which to store Tracer State and
// Baggage.
type BinaryCarrier bytes.Buffer

Inject() and Join() would stay exactly as they are today. Thoughts?

An individual Tracer implementation can simply not-support Baggage, or it would even be possible to write a pass-through Tracer adapter that basically just delegates everything but drops all calls to SetBaggageItem(...)

I am not sure I buy this scenario. It's not about what the tracer supports, but what a composition of different frameworks in a service requires. Imagine a service that talks to some 3rd party storage systems. Not only they may not have any use for you app's baggage, but they also may have highly optimized proprietary protocol that doesn't even support baggage, only some restricted form of tracing context. However, the same service may talk to other services in the stack that do support baggage. So the same tracer will be used by all these RPC frameworks, but with different requirements and different carriers.

If services in the middle of the stack choose not to support Baggage, then Baggage doesn't really work... the whole point is that it cuts across the stack, right?
BTW, yet another possibility would be a pair of baggage-less formats... the IPC programmer would control whether the carrier includes baggage by requesting the SharedMapCarrier XOR the SharedMapNoBaggageCarrier (would need a better name).

What I meant was service A calling B, and service A calling storage S. Both A and B support baggage, but the storage S does not (protocol limitations, or lack of plugin framework in S that make baggage unusable there). So the same tracer in A must deal with injecting a span with and without storage.
It feels like of all the alternatives the simplest would be to expose a parameter in the Inject method:
func Inject(span Span, carrier interface{} includeBaggage bool)

I don't believe this has to be symmetrical with the extractor, because extractor just takes what has been transmitted over IPC - (A) if the baggage was transmitted, it will read it from carrier and store in the span, (B) if the baggage was not transmitted, it will keep the baggage empty. I am not sure what benefit we have by not extracting the baggage in (A).

@yurishkuro IMO the less we can add special parameters/flags for baggage, the better. Going back to your example, if S doesn't support baggage, can't it just not-use (or not-understand) the baggage data sent in by A? Or, in the unfortunate case where S requires its own custom carrier – OT compatibility bedamned – that carrier could exclude baggage entirely.
I just have the sense that we are talking about a really obscure corner case, and furthermore that there are plenty of plausible ways to make that corner case working without introducing a baggage-specific param/flag at Inject-time. And in the process we could make the carriers structurally simpler, perhaps reducing to nothing more than a string map and a byte buffer.

Fair enough, I concede that this may be an obscure corner case that can be dealt with via custom carrier. Btw, out spec does not show examples of custom using custom carriers, leaving that API completely up to the tracer implementation. The downside of that is swapping the tracers becomes not an O(1).

I was thinking more about the above and realized that Baggage can still be opt-in without touching inject/join. An individual Tracer implementation can simply not-support Baggage, or it would even be possible to write a pass-through Tracer adapter that basically just delegates everything but drops all calls to SetBaggageItem(...).

I am behind this sentiment.

furthermore that there are plenty of plausible ways to make that corner case working without introducing a baggage-specific param/flag at Inject-time.

Yes, and that's what I've demonstrated in https://github.com/opentracing/opentracing-java/pull/11/files#diff-f2ad9c00db83aa2576fe12ea2f7f0413R21
There's plenty of room to solve these problems behind-the-scenes. Additional flags or methods are not required (in the API).
We still have a problem with the serialisation of required carriers needing a specification so that different implementations in different languages, given the a fixed rpc/http framework, deserialise in a compatible way regardless of who serialised.

We still have a problem with the serialisation of required carriers needing a specification so that different implementations in different languages, given the a fixed rpc/http framework, deserialise in a compatible way regardless of who serialised.

I don't see why we need such specification. There is no goal of making two tracing systems (e.g. Zipkin vs. Dapper) inter-operate via serialization spec. The only requirement is the same tracing implementation (even in different languages, e.g. Zipkin Java vs. Zipkin Python) must understand its own serialization format.

The only requirement is the same tracing implementation (even in different languages, e.g. Zipkin Java vs. Zipkin Python) must understand its own serialization format.

That is correct.

Which becomes the problem for that implementation, not for OpenTracing spec.

But what is the common space here?
Tracing implementations are just going to push the problem back onto OpenTracing. This is about how OpenTracing serialisation works for each of the different Tracing solutions. With OpenTracing serialisation you could well already have broken the vanilla Tracing serialisation (eg the need to separate state and baggage). Therefore such specification does not belong to the Tracing solution, but to OpenTracing instead. If the code lives in OpenTracing this is even more evident. If OpenTracing is also delegating the serialisation into the rpc/http framework back onto the Tracing own instrumentation libraries, then you could say the common space is then within that Tracing solution. In this situation it is easier for the OpenTracing Internal Specfication to simply reference the Tracing solution's documentation.
At the end of the day I suspect it'll be more pragmatic to let OpenTracing provide a cohesive set of docs, re-using (referencing) what it can, rather than trying to force others to do what we think is correct.

We can certainly write an "OpenTracing opinion" about how the serialization could be done, but we cannot, nor should we IMO, enforce how it's done. I disagree that OT breaks the existing serialization, precisely because it does not dictate how it's done. OT tracer that wants to be compatible with Finagle can continue encoding trace context as 4 B3_xxx headers.
I may be misunderstanding what you are trying to suggest. The existing k/v text carrier is meant to be tracer-specific encoding. The RPC framework is expected to store those k/v pairs in the "application headers" segment of the request (e.g. HTTP headers) without any additional encoding or serialization.

In the spirit of keeping the discussion on-topic, I wonder if we could fork off a separate issue about whether OpenTracing should (or must!) have an opinion about how implementations do serialization. If we agree that such a discussion can happen independent of the actual issue here – namely, what the required carriers should look like – I will create a new issue and these two threads can happen in parallel.
In the meantime, I will go back to @yurishkuro's (on-topic) message above:

Fair enough, I concede that this may be an obscure corner case that can be dealt with via custom carrier.

So you are ok with the (simpler) single text map and single byte buffer carriers (at least for v1.0RC2)?

Btw, out spec does not show examples of custom using custom carriers, leaving that API completely up to the tracer implementation. The downside of that is swapping the tracers becomes not an O(1).

There are some typos above which might be getting in the way of my understanding, but I think you're pointing out that our docs/spec doesn't give any guidance about custom carriers... My hope was that code that relies on custom carriers would fall back on one of the required carriers if the tracing impl didn't support the non-standard/custom carrier. Imagine a pirate-friendly RPC framework called "ArrrrPC" – pseudocode might look like this:
func sendArrrrPC(rpcHandle, span) {
  try {
    // First try the custom ArrrrPC format
    span.tracer.inject(span, arrrrpc.ARRRRPC_FORMAT, rpcHandle);
  } catch (opentracing.UnsupportedFormat) {
    // Fall back on TEXT_FORMAT as needed
    textCarrier = opentracing.TextCarrier();
    span.tracer.inject(span, opentracing.TEXT_FORMAT, textCarrier);
    rpcHandle.headers.merge(textCarrier);
  }
  ... actually send the RPC ...
}

If the above makes sense, I agree it should be part of the docs.

Apologies for the typos, let me reword:

our spec does not show examples of registering custom carriers, leaving that API completely up to the tracer implementation. The downside of that is swapping the tracers becomes not an O(1).

Re the ARRRRPC_FORMAT example (love the name), the assumption is that its requirements can be satisfied, perhaps less efficiently, by standard formats. But as we have seen in TChannel use case, no standard carrier can be used as a fallback there, especially if we decide not to separate state and baggage.
Anyway, I don't think this is, strictly speaking, on-topic either.

if we could fork off a separate issue about whether OpenTracing should (or must!) have an opinion about how implementations do serialization.

Yes. FTR I never advocated for OpenTracing to dictate how implementations must serialise, but allow implementations to update OpenTracing docs along with OpenTracing code, as they implement.

But as we have seen in TChannel use case, no standard carrier can be used as a fallback there, especially if we decide not to separate state and baggage.

In the delegation model proposed in opentracing-java it was intentional that standard carriers did not have to be used. (That's kind of the idea with a delegation model).

@michaelsembwever even with delegation, the assumption is that a custom handler is delegating to some "standard" handlers via some "standard formats". My point is that if the standard formats are (1) a single map, and (2) a single byte array, then neither of these can be made to work for TChannel, which requires a fixed-length byte array for trace context, and a string k/v map for baggage.

the assumption is that a custom handler is delegating to some "standard" handlers via some "standard formats".

since when?
this sounds like the spec trying to stick its nose into the implementation.

I think we're disconnecting again. Who is implementing the delegation in your scenario, the end user or the tracer implementor?  I am talking about the end user (e.g. the author of TChannel RPC). Since the end user has no knowledge about tracer internal, the best she can do when implementing a custom injector is to delegate to known/standard formats that are guaranteed to be supported by the tracer.

Since the end user has no knowledge about tracer internal…

If the end-user working with a custom rpc (that has no known/standard format) is expected to solve this without knowing of the tracer internals I think we're kinda screwed.
The normal situation is that end-users delegate to a known/standard format.
There arises situations where the end-user can't, and must accordingly implement the injector/extractor without such delegation. TChannel seems a good example of this.

The normal situation is that end-users delegate to a known/standard format. There arises situations where the end-user can't, and must accordingly implement the injector/extractor without such delegation.

That's exactly what I was saying. However, in the existing Go API where text/binary carriers are "split", it was possible to implement TChannel use case via delegation (just not very efficiently).
@bensigelman as I said earlier, I don't have strong objection to "un-splitting" the carriers.

Alright... I will wait to see if there are immediate objections to the great un-splitting. If not we can start a Github Issue timer and go from there.

If there are no objections within 48 hours, we will proceed with the two required carriers described above (SharedMapCarrier and BinaryCarrier).

Ok, no objections... I will start working through this over the weekend (hopefully).

authors: bensigelman bg451 michaelsembwever tschottdorf yurishkuro


text:
🛑 [CN] OpenUPM Website Docs is down

In a04a1f4, [CN] OpenUPM Website Docs (https://openupm.cn/docs/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: [CN] OpenUPM Website Docs is back up in 9608702 after 35 minutes.

repo: openupm/upptime-openupmcn

license: MIT


text:
🛑 [CN] OpenUPM Website Package Detail is down

In 0cb0b82, [CN] OpenUPM Website Package Detail (https://openupm.cn/packages/com.littlebigfun.addressable-importer/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: [CN] OpenUPM Website Package Detail is back up in 2989f29.

repo: openupm/upptime


text:
想與 zonble 和 lukhnos 聊一下關於 swift 與 objC 的取捨問題。

首先請教 @zonble 好了。
請問您將程式庫由 objC 轉成 swift 的原因是因為方便維護、還是因為 swift 相對於 objC 有什麼壓倒性的優勢嗎？
我昨晚 google 爬文發現 objC 在執行效率方面比 swift 快大約六倍：
https://bunn.dev/benchmark/2020/07/25/objc-vs-swift.html
然後就是我想知道 @lukhnos 是否贊成整個小麥注音專案全部 swift 化、還是說只是部分 swift 化。
畢竟很多牽涉到核心處理模組的內容都是 lukhnos 完成的。

P.S.: 該問題純屬好奇。至於威注音，目前會繼續 ObjC、對 High Sierra 保持相容性。
將來 cook.py 的部分會用 ObjC 重寫，徹底解除對 Python 的依賴。
也还可能擇日會將 SwiftyOpenCC 換用 ObjC 重寫。

C 與 C++ 語言有一大堆的 undefined behavior，像 Swift 這些現代語言首先要解決的就是 undefined behavior 的問題。在小麥注音上面，最重要的地方在於 Swift 有嚴格的形態檢查。
以小麥注音來說，在轉換成 Swift 之前，其實很多 API 的呼叫所傳遞的參數根本型態就是錯誤的，但是 ObjC compiler 並不能夠檢查出這些錯誤。比方說，在選字窗的地方，對 NSView 呼叫 setNeedDisplay 的時候，應該要傳入 NSRect 型態，表示這個 view 應該哪裡應該重繪，原本有很多地方卻寫成 setNeedDisplay:YES，傳入 BOOL 型態，這個 API 呼叫根本就寫錯了，在傳入錯誤的參數型態的時候，就可能出現無法預期的行為，很可能根本就不會重繪，程式會動就只是碰巧而已。
嚴格的形態檢查中，還有一項很重要的，就是對 nil 的檢查。同樣是選字窗 UI 的部份，很多邏輯其實在沒有指定 delegate 的狀況下，根本就不該繼續往下走。原本有些邏輯整個建立在 delegate 不會 assign 到 nil 的假設上，但是寫 code 不太應該這樣寫，雖然每塊磚頭都是歪的，堆出來的房子還是可以住，但是你想稍微動一下哪塊磚頭，這種房子就很容易就垮了。
我覺得小麥基本上大概有兩個地方特別需要效能，第一個是 FastLM 載入 data.txt 的 parser 的部份，另外一個則是 Walker 裡頭的那段演算法，其餘地方，都是正確性遠高於效能。以選字窗這種 UI 來說好了，輸入法的選字窗又不像遊戲畫面需要一秒鐘更新 60 次，但是 ObjC 的型態一搞錯就是沒事就會 crash 的。用 C++ 寫的引擎是小麥的核心，不過應該得要再封裝一層。
這些年蘋果的新的 API 都慢慢只有 Swift 版本了，先不說 SwiftUI，像是 Combine 與 SwiftCrypto，都只有 Swift 的版本，如果想使用一些比較先進的蘋果技術，那麼就不可能停留在純 ObjC 的世界了。Swift 這幾年另外一個重點在於 concurrency，不過這對小麥影響還好，小麥畢竟不是那種同時兼有幾十條網路連線的哪種 app。
我的你的專案沒有任何評論。

@zonble 感謝您的講解。
剩下的問題只有一個了，就是小麥注音到目前為止哪怕用 Swift 重寫也要堅持使用 Voltaire 選字介面（而非目前 macOS 10.9-12 的 IMKCandidates）的原因是什麼。如果有功能是 IMKCandidates 無法實現的功能的話，請問是哪些小麥注音的功能？

authors: ShikiSuen zonble

repo: openvanilla/McBopomofo


text:
(develop branch) Symbolic links for raw, original and compressed when using NFS share

My actions before raising this issue

[x] Read/searched the docs
[x] Searched past issues

Expected Behaviour

Reduce storage by using links on raw, original, and compressed subfolders when using NFS share.
Current Behaviour

As far as I understood from engines/task.py (develop), data is copied from share to task/raw, then clustered and zipped to task/original and compressed to task/compressed, resulting in 4 times the same data.
Possible Solution

Is it possible to add an option to only use symbolic links? This is crucial when we have GBs of images and videos. Some related issues are #1210 and #204. Since folders task/original and task/compressed contain zipped chunks, it might be more difficult to mitigate, but for task/raw it's possible. There is an argument to the zip command called --symlinks, but I don't know exactly whether it's reflected onto the python ZipFile class. It's gonna be more difficult to the compressed folder, since it manipulates the images (by scaling the color range to 8 bits and converting it to 8 bits before compression).
Steps to Reproduce (for bugs)

n/a
Context

Each video has around 2 GB of size, and we have dozens of them. Each will be annotated by some students. We expect multiple users for the same dataset (in order to compare their annotations).
Your Environment

Git hash commit (git log -1): commit 76fc8e442d74ae8a733fa722c894d33bc037ddb6 (HEAD -> develop, origin/develop)
Docker version docker version (e.g. Docker 17.0.05): Docker version 19.03.5, build 633a0ea838
Operating System and version (e.g. Linux, Windows, MacOS): #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017

@anderflash , thanks for the request. It is a critical problem. Many users complain about that and it is time to find a solution.

@anderflash , please let us know if it is an issue for us. We have optimized working with data. It should not be an issue. I will close the issue as outdated. If the issue isn't resolved, don't hesitate to reopen.

authors: anderflash nmanovic

repo: openvinotoolkit/cvat


text:
Convert class and instance binary masks to COCO instances format

Hi everyone!
I am wondering if there is a way to convert dataset to coco-instances format by using class and instance masks.
common_semantic_segmentation format accepts only single mask, so this makes difficult to distinguish instances from the same class on the image. Is there a way to provide an instance mask with the dictionary {instance_id :class_id} or two binary masks for dataset conversion?
Thank you!

Hi @maritum,
Thanks for your interests on our project!
Sorry for inconvenience but there is no high-level API for this functionality. Instead, I created a simple Jupyter-notebook example to solve your problem using Datumaro. Please refer to this: https://github.com/vinnamkim/datumaro/blob/cs/make-coco-instance-mask/notebooks/08_assign_label.ipynb

Hi @vinnamkim! Thank you again for your help. It works great!
I am curious if you have any recommendations for memory optimization when using dm.Dataset.from_iterable() for large datasets. I am currently processing it by chinks and saving a temporary dataset which I then merge.Is there built-in functionality for this in datamaro?
Thank you in advance!

Hi @maritum,
Sorry, I don't get your point about memory optimization. Could you give me more details? I guess if you built your dataset with dm.Dataset.from_iterable(), the main bottleneck for the memory would be Image or Mask. If you give a raw image data to create Image (e.g. Image(data=np.array(...))), it is recommended to create it with providing an image file path, e.g., Image(path=<path/to/image>) to save the memory. On the other hand, Mask can be compressed by run-Length encoding (RLE) as follows.
import numpy as np
import pycocotools.mask as mask_utils
import datumaro as dm

binary_mask = np.array(
    [
        [1, 1, 1],
        [1, 1, 0],
        [1, 0, 0],
    ], dtype=np.uint8, requirements="F"
)

# You can use dm.RleMask as the compressed version of dm.Mask
mask = dm.RleMask(
    id=0,
    group=0,
    image=binary_mask,
    label=0
)

This example shows how to use RleMask rather than using Mask.

Because there has been no response, I'll close this ticket.

authors: maritum vinnamkim

repo: openvinotoolkit/datumaro


text:
Access kube-apiserver with ovn pod's secret

Instead of using node kubeconfig file, use the ovn pod token to access kube-apiserver.
Retrieve kube-apiserver URL from pod env variables.
Fixes #602

@girishmg does your https://github.com/openvswitch/ovn-kubernetes/pull/605 supercede parts of this PR?

@dcbw Yes, it does. We can close this PR once #605 is merged.

Since #605 is merged.

authors: dcbw girishmg pliurh shettyg

repo: openvswitch/ovn-kubernetes


text:
Invalid swagger file should not crash api create

Unable to parse swagger file is expected on invalid swagger file...instead during testing we see:
error: Unable to create API: API route creation failure: {"error":"API configuration failure: {"error":"The action did not produce a valid response and exited unexpectedly."}"} (code 0)

{
"swagger": "2.0",
"info": {
"title": "/",
"version": "1.0.0"
},
"paths": {
"/test1": {
        }
}

}

@nwspeete-ibm - i tried to recreate the above failure manually using the following swagger file
{
  "swagger": "2.0",
  "info": {
  "title": "/",
    "version": "1.0.0
  },
  "paths": {
    "/test1": {
     }
  }
}

$ wsk -i api-experimental create -c nic.badswagger.json
error: Unable to parse swagger file: Swagger file is invalid (missing basePath, info, paths, or swagger fields
Run 'wsk --help' for usage.

Error message is missing a ), but it fails as expected.  Does this swagger file fail with the manual command in your environment?

Running manually works great: error: Unable to parse swagger file: Swagger file is invalid (missing basePath, info, paths, or swagger fields
Run 'wsk --help' for usage.

took out other parameters and this passes. closing.

authors: mdeuser nwspeete-ibm

repo: openwhisk/openwhisk


text:
luci-app-vnstat2: add application

This adds a LuCI application for vnStat 2.x.
Depends-on: https://github.com/openwrt/packages/pull/10935

Removed unnecessary and wrong ucitrack definition.

I updated the check of vnstat output and the rpcd acl configuration.
I'm still unsure if the rpcd acl configuration is correct. Looking at other applications with client-side implementation, it is not really clear to me how it is supposed to be set up. For example, when uci files need to be specified (as right now any application seems to have access anyway), and whether items that already are in the "write" section should also be in the "read" section.

repo: openwrt/luci


text:
Cleanup: Use upstream TransformInterpreterPassBase

Transform dialect interpreter passes must still be defined in IREE, but they can use the upstream mlir::transform::TransformInterpreterPassBase implementation.

Great, glad to see the upstream parts shaved off this yak, thanks!

authors: matthias-springer nicolasvasilache

repo: openxla/iree


text:
[StableHLO] Port gather canonicalization pattern

This pattern folds gather into slice (+ reshape) and is ported from MHLO.
Issue: https://github.com/openxla/iree/issues/12678

Not sure why the windows runtime tests failed:
[ RUN      ] LoopTest.WaitAnyBlocking
C:\a\iree\iree\runtime\src\iree/base/loop_test.h(795): error: Value of: status
Expected: error code OK
  Actual: 0000000000000004, whose error code is DEADLINE_EXCEEDED: DEADLINE_EXCEEDED

Have we seen this before @benvanik @ScottTodd?

Not sure why the windows runtime tests failed:
[ RUN      ] LoopTest.WaitAnyBlocking
C:\a\iree\iree\runtime\src\iree/base/loop_test.h(795): error: Value of: status
Expected: error code OK
  Actual: 0000000000000004, whose error code is DEADLINE_EXCEEDED: DEADLINE_EXCEEDED

Have we seen this before @benvanik @ScottTodd?

Haven't seen that before, no. Unrelated to your PR though... maybe just re-run the failed job?

authors: ScottTodd kuhar


text:
Migrate Session to database for active/active support

The daemon makes use of CGI::Session with a file based storage backend which will not work in a dynamic active/active cluster setup where clients might bounce between nodes.

Should be implemented using the new sql-layer, concept might be taken from existing MySQL driver for CGI::Session

repo: openxpki/openxpki


text:
Feature request: ACME server

It would be really nice with support for acme / certbot clients to issue certificates.

The ACME protocol has a veeeery large footprint but we have a working beta here in our lab. As this is integrated into our commercial "protocol gateway" component its unlikely that it will become OpenSource in the near future - if you are interessted please contact me via PM.

ACME protocol has a veeeery large footprint
@oliwel
Maybe it would make sense to be part of this acme2certifier ca_handler.py - list?

ACME protocol has a veeeery large footprint

@oliwel Maybe it would make sense to be part of this acme2certifier ca_handler.py - list?

thank you for this pointer - as this project has a generic EST client and we have a generic EST server it should work out of the box, so feel free to integrate it.

@flybyray: Certificate enrollment via the generic est ca handler possible; we use OpenXPKI in our release regression since a few months.  We also released a dedicated ca handler integrating into OpenXPI via the RPC server API.  Feel free to test if you are interested...

authors: flybyray grindsa oliwel pertoft


text:
Docker cache

http://thenewstack.io/understanding-the-docker-cache-for-faster-builds/
I think we need some way of busting the cache, so it's not using an old version of openzipkin when building. (In fact, my docker build didn't even clone openzipkin at all)

My first idea is doing the clone outside the Dockerfile. This comes with the benefit of being able to add .git to .dockerignore.

Once we do releases, we wouldn't be building from source, and instead
letting the user choose snapshot or some pinned version from bintray. Does
this change anything?

Even in that case I'd prefer downloading on the host instead of in the container, to keep the images super lean. (Obviously the git clone will become a wget etc.)

Wait, no. In an ideal world we'd have automated builds, which means everything would be inside the containers, even the downloadings. In that case #16 is only a temporary thing to use while we build the docker containers from source.

While we build the docker images from source, this is fixed by #16. The proper long-term solution is tracked in #18, so I'm closing this issue.

authors: abesto adriancole eirslett

repo: openzipkin/docker-zipkin


text:
WIP: moves Elasticsearch Armeria client wiring to spring

Complexity around logging and health checks are unrelated to the functionality of the Elasticsearch client. For example, there is no requirement to use health checks at all. Having this code in the elasticsearch module complicates configuration and particularly zipkin-aws which needs to collaborate on client config (via spring).
What this change will do, is segregate the fairly volatile parts of our setup, notably around client-side load balancer health checks and wire logging, such that we can contain the change. Also important is being able to expose the resultant client so that for example an implicit request to fetch the AWS provided endpoint can be signed, made, logged and in the same trace context as the production requests.
What's left here is moving the parts and tests related to client config.. this PR will break until that's done.

I dare say this is ready.. checking now

https://github.com/openzipkin/zipkin-aws/pull/135 with fingers crossed will now work

meh not completely unwound yet.. false alarm

good notes thanks!
On Tue, Jul 30, 2019, 5:48 PM Anuraag Agrawal notifications@github.com
wrote:

@anuraaga approved this pull request.
In
zipkin-storage/elasticsearch/src/main/java/zipkin2/elasticsearch/ElasticsearchStorage.java
https://github.com/openzipkin/zipkin/pull/2711#discussion_r308143185:

@@ -82,21 +58,22 @@
BODY
}

/**

A list of elasticsearch nodes to connect to, in http://host:port or https://host:port format.

Note this value is only read once.

*/
public interface HostsSupplier {
List get();

public interface LazyHttpClient extends Supplier, Closeable {

/** If the client has been created, this will close it and any resources used by it. */
@Override public default void close() throws IOException {
}

/**

* Lazily creates and instance of the http client configured to the correct elasticsearch host

an instance
In
zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/HostsConverter.java
https://github.com/openzipkin/zipkin/pull/2711#discussion_r308595967:

is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express

or implied. See the License for the specific language governing permissions and limitations under

the License.

*/
+package zipkin2.server.internal.elasticsearch;

+import java.net.URI;
+import java.util.Collections;
+import java.util.List;
+import java.util.logging.Logger;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import zipkin2.elasticsearch.ElasticsearchStorage;
+
+final class HostsConverter {

static final Logger LOG = Logger.getLogger(ElasticsearchStorage.class.getName());

Realize this was probably copied, but since it's zipkin-server, I think
it's supposed to be either log4j2 or slf4j
In
zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/LazyHttpClientImpl.java
https://github.com/openzipkin/zipkin/pull/2711#discussion_r308625683:

int timeout, HttpLogging httpLogging) {

this.protocol = protocol;
this.initialEndpoints = initialEndpoints;
this.customizers = customizers;
this.factory = factory;
this.timeout = timeout;
this.httpLogging = httpLogging;
}

@Override public void close() {
EndpointGroup endpointGroup = EndpointGroupRegistry.get("elasticsearch");
if (endpointGroup != null) {

 endpointGroup.close();

 EndpointGroupRegistry.unregister("elasticsearch");

}
factory.close();

Could put this in a finally or local try-with-resources in case
endpointGroup.close() throws for some reason, though it's extremely
careful.
In
zipkin-server/src/main/java/zipkin2/server/internal/elasticsearch/LazyHttpClientImpl.java
https://github.com/openzipkin/zipkin/pull/2711#discussion_r308627828:

+import com.linecorp.armeria.client.logging.LoggingClientBuilder;
+import com.linecorp.armeria.common.HttpHeaders;
+import com.linecorp.armeria.common.SessionProtocol;
+import com.linecorp.armeria.common.logging.LogLevel;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Supplier;
+import zipkin2.elasticsearch.ElasticsearchStorage.LazyHttpClient;
+import zipkin2.server.internal.elasticsearch.ZipkinElasticsearchStorageProperties.HttpLogging;

+import static com.linecorp.armeria.client.endpoint.EndpointSelectionStrategy.ROUND_ROBIN;
+
+final class LazyHttpClientImpl implements LazyHttpClient {

final SessionProtocol protocol;
final Supplier initialEndpoints;

Maybe configuredEndpoints - only because this is different than what
awaitInitialEndpoints does.
In
zipkin-server/src/test/java/zipkin2/server/internal/elasticsearch/ITElasticsearchAuth.java
https://github.com/openzipkin/zipkin/pull/2711#discussion_r308630021:

+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.Assert.fail;
+import static zipkin2.server.internal.elasticsearch.ITElasticsearchHealthCheck.YELLOW_RESPONSE;
+import static zipkin2.server.internal.elasticsearch.ZipkinElasticsearchStorageConfiguration.QUALIFIER;

+public class ITElasticsearchAuth {

static final BlockingQueue CAPTURED_REQUESTS = new LinkedBlockingQueue<>();

@ClassRule public static ServerRule server = new ServerRule() {
@Override protected void configure(ServerBuilder sb) throws Exception {

 sb.https(0);

 sb.tlsSelfSigned();

 sb.serviceUnder("/", (ctx, req) -> {

   CompletableFuture<HttpResponse> responseFuture = new CompletableFuture<>();

   req.aggregate().thenAccept(agg -> {

Should be equivalent to do
CompletableFuture responseFuture = req.aggregate().thenApply(agg -> {
...
return HttpResponse.of(YELLOW_RESPONSE);
});
and you don't need to worry about exceptionally
In
zipkin-server/src/test/java/zipkin2/server/internal/elasticsearch/ITElasticsearchAuth.java
https://github.com/openzipkin/zipkin/pull/2711#discussion_r308630485:

+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.Assert.fail;
+import static zipkin2.server.internal.elasticsearch.ITElasticsearchHealthCheck.YELLOW_RESPONSE;
+import static zipkin2.server.internal.elasticsearch.ZipkinElasticsearchStorageConfiguration.QUALIFIER;

+public class ITElasticsearchAuth {

static final BlockingQueue CAPTURED_REQUESTS = new LinkedBlockingQueue<>();

@ClassRule public static ServerRule server = new ServerRule() {
@Override protected void configure(ServerBuilder sb) throws Exception {

 sb.https(0);

 sb.tlsSelfSigned();

 sb.serviceUnder("/", (ctx, req) -> {

   CompletableFuture<HttpResponse> responseFuture = new CompletableFuture<>();

   req.aggregate().thenAccept(agg -> {

Though fine to just leave it and possibly update with 0.90
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/openzipkin/zipkin/pull/2711?email_source=notifications&email_token=AAAPVV3JC5BNPH5L4OJ54ELQCAE53A5CNFSM4IHQT3K2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB72GOZI#pullrequestreview-267675493,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAPVV72ZKO34Z42EH4QBRDQCAE53ANCNFSM4IHQT3KQ
.

I've disabled the client-side health checking in an attempt to at least get to something functional as we can't keep holding the whole repo hostage and until things work, that's what's happening.
against a local node, this works except there's one of these things.. aws doesn't work and I can't quite tell why not as logging isn't working.
2019-07-30 20:43:00.095  WARN 12226 --- [-worker-nio-2-1] i.n.u.ReferenceCountUtil                 : Failed to release a message: ByteBufHttpData{buf=PooledUnsafeDirectByteBuf(freed)}

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[netty-buffer-4.1.38.Final.jar!/:4.1.38.Final]
	at com.linecorp.armeria.unsafe.ByteBufHttpData.release(ByteBufHttpData.java:116) ~[armeria-0.89.0.jar!/:?]
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:113) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:193) ~[zipkin-storage-elasticsearch-2.15.1-SNAPSHOT.jar!/:?]
	at zipkin2.elasticsearch.internal.client.HttpCall.lambda$doEnqueue$0(HttpCall.java:125) ~[zipkin-storage-elasticsearch-2.15.1-SNAPSHOT.jar!/:?]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onComplete(DeferredStreamMessage.java:326) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.notifySubscriberOfCloseEvent(FixedStreamMessage.java:126) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.TwoElementFixedStreamMessage.doNotify(TwoElementFixedStreamMessage.java:110) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.TwoElementFixedStreamMessage.doRequest(TwoElementFixedStreamMessage.java:72) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.request(FixedStreamMessage.java:88) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:309) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onSubscribe(DeferredStreamMessage.java:309) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.subscribe(FixedStreamMessage.java:115) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:93) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:84) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DeferredStreamMessage.safeOnSubscribeToDelegate(DeferredStreamMessage.java:272) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DeferredStreamMessage.delegate(DeferredStreamMessage.java:118) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.DeferredHttpResponse.delegate(DeferredHttpResponse.java:39) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.HttpResponse.lambda$from$0(HttpResponse.java:74) ~[armeria-0.89.0.jar!/:?]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FilteredStreamMessage$FilteringSubscriber.onComplete(FilteredStreamMessage.java:290) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:194) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:363) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:306) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:253) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:239) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:369) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.closeAction(HttpResponseDecoder.java:296) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.close(HttpResponseDecoder.java:279) [armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.close(HttpResponseDecoder.java:267) [armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.client.Http1ResponseDecoder.channelRead(Http1ResponseDecoder.java:212) [armeria-0.89.0.jar!/:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) [netty-codec-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302) [netty-codec-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:154) [netty-handler-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

I think the above error is interfering with self-tracing as I don't get any client spans, only the spans above it.
also I think there's something about the raw body decorator which is incompatible with the client usage here.. https://github.com/openzipkin/zipkin-aws/pull/135/files#diff-c2299f75862484516908a6b48313a9e1R59
this is interfering with my ability to figure out what if anything is wrong with the signature code when sent against the elasticsearch domains endpoint

even when removing the health check stuff there's still a problem I've not seen in any other http client where we can have no results due to dns lag. it seems possibly a good idea to still block on dns to resolve rather than error DnsAddressEndpointGroup
com.linecorp.armeria.client.endpoint.EndpointGroupException: com.linecorp.armeria.client.endpoint.dns.DnsAddressEndpointGroup@7ede1e6a is empty

ok there's no more problem with DNS crashing requests.
The below is still a problem with the reference count thing regardless of local use or not. I will try to hunt which of the changes tripped this
2019-07-30 21:09:28.757  WARN 15594 --- [-worker-nio-2-1] i.n.u.ReferenceCountUtil                 : Failed to release a message: ByteBufHttpData{buf=PooledUnsafeDirectByteBuf(freed)}

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[netty-buffer-4.1.38.Final.jar!/:4.1.38.Final]
	at com.linecorp.armeria.unsafe.ByteBufHttpData.release(ByteBufHttpData.java:116) ~[armeria-0.89.0.jar!/:?]
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:113) ~[netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:193) ~[zipkin-storage-elasticsearch-2.15.1-SNAPSHOT.jar!/:?]
	at zipkin2.elasticsearch.internal.client.HttpCall.lambda$doEnqueue$0(HttpCall.java:125) ~[zipkin-storage-elasticsearch-2.15.1-SNAPSHOT.jar!/:?]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:98) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onComplete(DeferredStreamMessage.java:326) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:363) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.notifySubscriberOfCloseEvent(FixedStreamMessage.java:126) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.TwoElementFixedStreamMessage.doNotify(TwoElementFixedStreamMessage.java:110) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.TwoElementFixedStreamMessage.doRequest(TwoElementFixedStreamMessage.java:72) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.request(FixedStreamMessage.java:88) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:309) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onSubscribe(DeferredStreamMessage.java:309) ~[armeria-0.89.0.jar!/:?]
	at com.linecorp.armeria.common.stream.FixedStreamMessage.lambda$subscribe$1(FixedStreamMessage.java:117) ~[armeria-0.89.0.jar!/:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515) [netty-transport-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.38.Final.jar!/:4.1.38.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

ok this is now operable. I really don't want any more variables like client health check until aws works again as this effort is bleeding time. Let's do that stuff in a separate PR when the time is right.

I'll take a look at the signature tomorrow too, hoping just step-debugging the two versions of the signature will be enlightening enough without actually starting something in the cloud.
Without the health check, if an endpoint goes down we'll round-robin through them all and fail 1/N of requests as it won't fall back in case of connection failure - I believe the pseudo address set thing allowed it to fallback in case of connection failure. I don't know if this is too much of a regression or affects when the time is right.

I've added a backport test of signed output comparing against boto. (aws --debug es describe-elasticsearch-domain  --domain-name zipkin)
https://github.com/openzipkin/zipkin-aws/pull/135
Let's move the conversation there and we can get back to more cleanups and tidies once things are working. thanks for the support

authors: adriancole anuraaga

repo: openzipkin/zipkin


text:
Fixes to detect missing dependency by ServerRecv annotations

#932 allows to detect missing dependency with intermediate spans.
But it introduces another missing dependency by ServerRecv only spans usually generated by finagle including zipkin-web and zipkin-query itself.
This change reduces the impact of this regression by detecting the dependency without intermediate spans.
Walking trace tree within zipkin-query may detect such dependencies. However, it will have serious performance regression.
This change also reduces performance regression by modifying the query to use already existing index trace_id, span_id on zipkin_spans table.
Note that it avoids loopback dependency tests by ugly query because finagle generated annotations have no-meaningful CA and SA indicating unintentional loopback.
Please tell me if I misunderstand.

Hmm I wonder if the CA and SA pointing to themselves is a configuration bug?

Looking at the finagle code, it seems finagle indeed logs ClientAddr as the same service name of the local process.. It records the client-side of the socket.
Yeah seems if we look at "CA" (ex to track uninstrumented clients of the root span), we likely have to deal with this curious "SA" = "CA" cas of finagle..

I agree that supporting finagle and also supporting "ca" to resolve uninstrumented root spans requires work like this.
I do feel we are digging a hole and more edge cases will make this SQL quite unwieldy. I'm cool merging for now as we can clean that up later. Ex I have alternative SQL (with client side joins) readying for a pull request soon on the java project.

Thanks for the help, Yuki!

authors: adriancole t-yuki


text:
support / read the docs button label

The button label for read the docs is "ask questions", probably duplicated from the get support button label.

Fixed in Commit fcd32d5

authors: JeremyTellier qrkourier

repo: openziti/desktop-edge-ui


text:
add s1-etad prep module for testing S1-ETAD products

This PR adds s1_etad.py, which is a wrapper around the s1etad module from ESA. Given an s1reader.Sentinel1BurstSlc object, this sub-module will search, read the ETAD corrections, and/or resample into the full resolution, following the s1etad documentation (https://s1etad.readthedocs.io/en/latest/notebooks/S1-ETAD-Product_Quick_User_Guide.html).
Below is a simple example of usage.
import s1reader

orbit_dir = '~/data/aux/S1/aux_poeorb'
eta_dir = './ETAD'
slc_file = './SLC/S1A_IW_SLC__1SSV_20160922T230629_20160922T230656_013171_014F06_1463.zip'

# read SLC burst
orbit_file = s1reader.get_orbit_file_from_dir(slc_file, orbit_dir)
slc_burst = s1reader.load_bursts(slc_file, orbit_file, swath_num=1)[0]

# read ETA burst
eta_burst, eta = s1reader.s1_etad.get_eta_burst_from_slc_burst(slc_burst, eta_dir, verbose=True)

The title of this PR seems to tell that I should expect a notebook. Are you going to upload that in the branch?

Thank you @seongsujeong. I have a notebook, but will upload it to another independent repo, as notebooks can easily increase the code repo size. I modified the PR title to avoid this confusion.

@yunjunz how can I test this PR? Are the ETAD product already available and downloadable? Or do you have a Notebook with a workflow? Merci

@yunjunz how can I test this PR? Are the ETAD product already available and downloadable? Or do you have a Notebook with a workflow? Merci

@vbrancat with the link I sent offline, you should be able to download the S1ETAD products and test based on the note I wrote at the beginning of this PR.

Here is the notebook example: https://github.com/opera-adt/notebooks/pull/1

@seongsujeong do you have time to look at this PR? It might be interesting for the work you are doing for the azimuth FM-rate mismatch

authors: vbrancat yunjunz


text:
Reconcile manifests with templates.

Description of the change: Unknown field pruning was not enabled in several CRD templates (it's disabled by default in v1beta1), and new bundleLookups status field was added to the template schema, but not the corresponding generated manifest.
Motivation for the change: Reconciling differences between manifests and templates in preparation for a new release per the instructions at https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/design/release.md.
Reviewer Checklist

[ ] Implementation matches the proposed design, or proposal is updated to match implementation
[ ] Sufficient unit test coverage
[ ] Sufficient end-to-end test coverage
[ ] Docs updated or added to /docs
[ ] Commit messages sensible and descriptive

/retest

/lgtm

/retest

/assign @ecordell

/assign @ecordell

/lgtm
/approve

authors: benluddy ecordell


text:
Bug 1952576: csv_succeeded metric not present

csv_succeeded metric is lost between pod restarts.
This is because this metric is only emitted when CSV.Status is changed.

Description of the change:
Emit csv_succeeded/csv_abnormal metric during every CSV sync loop.
Motivation for the change:
csv_succeeded metric is lost between pod restarts.
Reviewer Checklist

[ ] Implementation matches the proposed design, or proposal is updated to match implementation
[ ] Sufficient unit test coverage
[ ] Sufficient end-to-end test coverage
[ ] Docs updated or added to /doc
[ ] Commit messages sensible and descriptive

/ok-to-test

How can I fix bugzilla/invalid-bug label on the PR?

@josefkarasek The bug bot is complaining about the current state of the BZ: https://github.com/operator-framework/operator-lifecycle-manager/pull/2213#issuecomment-866827324:
expected the bug to target the "4.9.0" release, but it targets "---" instead

In order to fix this, update the BZ's "Target Release" dropdown button to the 4.9.0, instead of the default value (empty release, "---"), save the BZ, and then comment /bugzilla refresh here.

/bugzilla refresh

Also, @timflannagan we don't need the bug number in the PR title right? We'll only need it when we downstream the PR?

Closing in favor of #2216

authors: anik120 josefkarasek timflannagan


text:
add support for finalizers

Sorry the template for a feature request got lost when I changed the type from bug...
I'd like to request support for finalizers.
Some CRD type need to do complex cleanup via a finalizers.
The operator sdk could provide support for it with the following:

optional flag in the operator-sdk add controller to specify that code to support finalizer for that type should be generated.
add logic to the reconciliation cycle to detect if a CR is being deleted. If it's the case hand control to a dedicated function
generate stub for the dedicated finalizer function
if the function returns successfully, handle the removing of the finalizer entry from the array of finalizer of the to-be-deleted CR.

@raffaelespazzoli There is related discussion in #460 and there is a proposal PR open in #489. In the interest of keeping the discussion in one place, would you mind contributing your ideas there?
You may notice there hasn't been much discussion lately, but we are definitely interested in adding better support for finalizers (and we'd happily accept contributions to the proposal and eventual implementation :slightly_smiling_face:)

authors: joelanford raffaelespazzoli

repo: operator-framework/operator-sdk


text:
Provide Metrics list

Provide documented list of standard metrics emitted by operator SDK.

For the record, the list of metrics that should be documented for ansible operators should be something like this (this is just me looking at kiali's operator metrics - all of which come directly from the ansible operator SDK 0.17.0):
Prometheus query: "{namespace="kiali-operator",app="kiali-operator"}"
ansible_operator_reconciles_bucket
ansible_operator_reconciles_count
ansible_operator_reconciles_sum
go_gc_duration_seconds
go_gc_duration_seconds_count
go_gc_duration_seconds_sum
go_goroutines
go_info
go_memstats_alloc_bytes
go_memstats_alloc_bytes_total
go_memstats_buck_hash_sys_bytes
go_memstats_frees_total
go_memstats_gc_cpu_fraction
go_memstats_gc_sys_bytes
go_memstats_heap_alloc_bytes
go_memstats_heap_idle_bytes
go_memstats_heap_inuse_bytes
go_memstats_heap_objects
go_memstats_heap_released_bytes
go_memstats_heap_sys_bytes
go_memstats_last_gc_time_seconds
go_memstats_lookups_total
go_memstats_mallocs_total
go_memstats_mcache_inuse_bytes
go_memstats_mcache_sys_bytes
go_memstats_mspan_inuse_bytes
go_memstats_mspan_sys_bytes
go_memstats_next_gc_bytes
go_memstats_other_sys_bytes
go_memstats_stack_inuse_bytes
go_memstats_stack_sys_bytes
go_memstats_sys_bytes
go_threads
kiali_info
process_cpu_seconds_total
process_max_fds
process_open_fds
process_resident_memory_bytes
process_start_time_seconds
process_virtual_memory_bytes
process_virtual_memory_max_bytes
rest_client_request_latency_seconds_bucket
rest_client_request_latency_seconds_count
rest_client_request_latency_seconds_sum
rest_client_requests_total
scrape_duration_seconds
scrape_samples_post_metric_relabeling
scrape_samples_scraped
scrape_series_added
up
workqueue_adds_total
workqueue_depth
workqueue_longest_running_processor_seconds
workqueue_queue_duration_seconds_bucket
workqueue_queue_duration_seconds_count
workqueue_queue_duration_seconds_sum
workqueue_retries_total
workqueue_unfinished_work_seconds
workqueue_work_duration_seconds_bucket
workqueue_work_duration_seconds_count
workqueue_work_duration_seconds_sum

/remove-lifecycle stale

/remove-lifecycle stale
IMO, this should be documented somewhere - otherwise, why expose metrics? As my old boss told me, "If it isn't documented, it doesn't exist" :) These metrics can be helpful to track performance issues in operators, but it would be nice to know WHAT ever metrics there is and WHAT they actually measure.

/remove-lifecycle stale
/remove-lifecycle rotten

@jmazzitelli these metrics first need to be documented in https://github.com/kubernetes-sigs/controller-runtime/tree/master/pkg/metrics, then SDK docs can simply point to these docs since no extra metrics are added on top of that set. Can you create an issue in controller-runtime to do so?

What then are these? I thought these are SDK-specific, no?
workqueue_adds_total
workqueue_depth
workqueue_longest_running_processor_seconds
workqueue_queue_duration_seconds_bucket
workqueue_queue_duration_seconds_count
workqueue_queue_duration_seconds_sum
workqueue_retries_total
workqueue_unfinished_work_seconds
workqueue_work_duration_seconds_bucket
workqueue_work_duration_seconds_count
workqueue_work_duration_seconds_sum

@jmazzitelli these metrics first need to be documented in https://github.com/kubernetes-sigs/controller-runtime/tree/master/pkg/metrics, then SDK docs can simply point to these docs since no extra metrics are added on top of that set. Can you create an issue in controller-runtime to do so?

Looks like there was already a PR - see "document exposed metrics" PR: https://github.com/kubernetes-sigs/controller-runtime/pull/811

Though I don't know how it helps the operator author - what are the "workqueues" used for in the operator SDK? I understand a workqueue in general, but from the operator perspective, what is the SDK using workqueues for? What do these metrics tell the operator user?

These are all controller-relevant metrics, which is why I suggested they be documented in controller-runtime. https://github.com/kubernetes-sigs/controller-runtime/pull/811 doesn't seem to actually document any metrics, rather it adds descriptions to them at runtime.
The following are set up by the ansible-operator (and helm-operator) internally, and so should be discussed in SDK docs:
ansible_operator_reconciles_bucket
ansible_operator_reconciles_count
ansible_operator_reconciles_sum
kiali_info

/language ansible
/language helm

authors: estroz jmazzitelli mattmahoneyrh


text:
bump controller-tools to v0.2.0

Description of the change: bump controller-tools to v0.2.0 (beta) and update CRD generator API/unit tests.
Motivation for the change: controller-tools k8s deps versions should match those of controller-runtime when possible. #1143 requires some changes in this release too.

Two changes that need discussion:

CRD manifests can contain multiple specified versions, so file naming should change from <group>_<version>_<kind>.yaml to something else. For example, controller-tools uses <group>_<resource>.yaml.
The CRD generator permits legacy support for spec.version but not simultaneously with spec.versions. The checkCRDVersions() function permits both forms as long as spec.version is the first element in spec.versions, which is correct behavior. How should we handle this?

@estroz Can we rebase or cherry pick #1652 into the refactor/controller-runtime-v0.2.0 branch?
And update the tag from // +genclient:nonNamespaced to // +kubebuilder:resource:path=<lowercase-kind-plural>,scope=Namespaced
Or do you want to do that as a separate PR?

@estroz For .1 I think we should follow the convention that kubebuilder has, so it will make it easier to migrate, do they have anything established already?
For 2. I think the log about the deprecation and error out if spec is not met, sounds good to me.

CRD file naming changes were quite extensive and touched both Helm and Ansible docs/tests. PTAL.
/cc @LiliC @hasbro17 @AlexNPavel @joelanford @jmrodri @fabianvf

@LiliC kubebuilder would generate a CRD file name cache.example.com_memcacheds.yaml. IMO cache_memcacheds_crd.yaml looks much nicer and contains all the necessary uniqueness/information that the kubebuilder name gives. WDYT?

@LiliC kubebuilder would generate a CRD file name cache.example.com_memcacheds.yaml. IMO cache_memcacheds_crd.yaml looks much nicer and contains all the necessary uniqueness/information that the kubebuilder name gives. WDYT?

Since we are moving to kubebuilder at some point, I would suggest we go the kubebuilder way as we probably will have to change it then anyways.

authors: LiliC estroz hasbro17


text:
dpinger fails to start with DHCPv6 on WAN

If WAN is set to DHCPv6 (with options "Request only an IPv6 prefix", "Prefix delegation size 60") dpinger fails to start with a PHP Error.
In my case WAN gets IPv6 fe80::20d:b9ff:fe48:58bc from the cable modem.
Workaround: set "Monitor IP" on the WAN_DHCP6 Gateway to some reachable IPv6 Address
OPNsense 19.7

I have the same issue, it affects all gateways with link-local addresses (including static ones): https://forum.opnsense.org/index.php?topic=13515.0
Seems to be caused by improper handling of zone indices. Worked in 19.1.

Same here.
I had suddenly 2 stopped instances of dpinger for WAN_DHCP6 which can't work.
My workaround so far was disabling interface, delete the double entries for gateways, reboot, enabling interface, gateway monitoring off.

There's something more going on with the gateways and giving priority.
I played a bit around and noticed that I can't set priority for dynamically created GWs over PPPoE. ipv4 and ipv6 GWs are set to 254 after connection goes up.
I changed to lower level (50) to give my PPPoE higher prio. It doesn't let me do this, because far gateway is unticked (GW-IP from ISP). Second try with activating far gateway does create a new gateway (with GW-IP given from ISP set to static) and whole connection goes nuts, because both dynamically created GWs are completely gone.
I deactivated PPPoE-IF completely and you can see (red dot) the leftover GW-IP.
Second thing is wrong displaying in WebIF. vigornic shows as active, but never was ticked as upstream-GW, but gw_winsim is.
Change default gw is activated under system->settings if that info is needed.

Seems to be fixed in 19.7.1 (well, works for me). I guess this line from the changelog refers to it:
system: scoped IPv6 "%" could confuse mwexecf(), use plain mwexec() instead
Can someone confirm and close?

Yeah, works now for me, too but I have now a new one with 2 WANs. :)
-> https://github.com/opnsense/core/issues/3601

Thanks for confirming.

authors: abplfab fichtner maurice-w mr44er

repo: opnsense/core


text:
Fix sorting of merged signals in package panel

Fix: #1683
Signed-off-by: David Mundelius david.mundelius@tngtech.com
Summary of changes
In the function getContainedExternalDisplayPackageInfosWithCount() in src\Frontend\Components\AggregatedAttributionsPanel\accordion-panel-helpers.ts instead of sorting the AttributionIds before combining ordinary and merged attributions, the packageCardIds are now sorted at the end.
Context and reason for change
Previously, instead of sorting all signals by count in the package panel, the merged signals always appeared after the other ones.
How can the changes be tested
In src\Frontend\Components\AggregatedAttributionsPanel_tests_\accordion-panel-helper.test.ts, run the test 'sorts ordinary and merged attributions according to the count' for the function ''getExternalDisplayPackageInfosWithCount'. Inste
Note: Please review the guidelines for contributing to this repository.
For manual testing, open the file 'example.opossum' in OpossumUI and choose the root directory in the Audit View. Then the TypeScript 3.9.6 signal with count 2 should appear at the top of the list.

btw: No need to sign-off the pull request description: Signed-off-by: David Mundelius [david.mundelius@tngtech.com](mailto:david.mundelius@tngtech.com)

authors: benedikt-richter davidmundelius

repo: opossum-tool/OpossumUI


text:
supermarket.chef.io release...

Hi @juliandunn,
v0.5.3 of this cookbook on github and on supermarket.chef.io is different. Can you please help to fix it?
TIA
-Suleyman Kutlu

I just released 0.5.4 that includes your changes.

authors: juliandunn snkutlu

repo: opscode-cookbooks/squid


text:
fix: name indexes

The migrations break for people that use a custom table name.  I've named the indexes to make sure the index drops still work with different table names.

Hey, can you specify the conditions under which the Eloquent's default index name generation logic fails?

Thanks a lot, by the way! Forgot to leave a comment. :)

authors: Tarpsvo jgile

repo: optimistdigital/nova-page-manager


text:
Update all dependencies.

@jordangarcia - I ran npm update --save and npm update --save-dev and it went ahead and updated/organized all of the dependencies.

@jordangarcia - tests passing now.

Checked every dependency change log and there is nothing breaking that is being introduced. Also, new dependencies are being added because npm will soon be removing peer dependencies. (https://github.com/npm/npm/issues/6565)

:+1:
On Wednesday, June 17, 2015, Baraa Hamodi notifications@github.com wrote:

Merged #97 https://github.com/optimizely/nuclear-js/pull/97.
—
Reply to this email directly or view it on GitHub
https://github.com/optimizely/nuclear-js/pull/97#event-332854189.

authors: bhamodi jordangarcia

repo: optimizely/nuclear-js

license: mit


text:
Set Aqua logging level

Description
Set AQUA logging through environment variables and CLI argument
CLI
--log-level=DEBUG
CLI help message

Example
ads aqua model list --compartment_id ocid1.compartment.oc1..<ocid> --log-level=DEBUG
2024-03-19 13:31:22,714 - ads.aqua - DEBUG - This is a test
2024-03-19 13:31:22,742 - ads.aqua - INFO - Fetching custom models from compartment_id=ocid1.compartment.oc1..<ocid>.
2024-03-19 13:31:22,742 - ads.aqua - INFO - query datasciencemodel resources where (compartmentId = 'ocid1.compartment.oc1..<ocid>' && lifecycleState = 'ACTIVE' && (freeformTags.key = 'OCI_AQUA' && freeformTags.key = 'aqua_fine_tuned_model'))
2024-03-19 13:31:22,742 - ads.aqua - INFO - tenant_id=None
2024-03-19 13:31:22,959 - ads.aqua - INFO - Fetch 0 model in compartment_id=ocid1.compartment.oc1..<ocid>.

ENV
LOG_LEVEL="DEBUG"

Closed this PR as this is duplicated with https://github.com/oracle/accelerated-data-science/pull/795

repo: oracle/accelerated-data-science


text:
OGG lacks my Local deployment on the second start

Hi!
I try to create a simple OGG Microservices container using the following docker-compose file:
version: '3'
services:
  ogg:
    image: oracle/goldengate-microservices:12.3.0.1.4
    container_name: ogg-12.3.0.1.4
    environment:
      - OGG_SCHEMA=oggadmin
      - OGG_ADMIN=oggadmin
      - OGG_ADMIN_PWD=q12345678
      - OGG_DEPLOYMENT=Local
    ports:
      - 9100-9104:9100-9104
    volumes:
      - T:\docker-data\ogg\conf:/u02/ogg/Local/etc/conf
      - T:\docker-data\ogg\trail:/u02/ogg/Local/var/lib/data
    cap_add:
      - SYS_RESOURCE

I use Docker for Windows so I have a path T:\docker-data\ogg\conf for configs and a path for trail files T:\docker-data\ogg\trail located on a host machine.
The very first start seems ok. Here's docker logs output: first_start.log. And I can see my Local deployment:

Then I docker-compose down my deployment and up again. Since the second start I don't see my Local deployment:

Here's a docker logs output for the second start:
second_start.log.
I'm expecting to see my configuration saved between ups and downs. I'm not good at OGG file structure and maybe I have to provide some other paths to several files in order to make them persistent using volume.

Hi @alesana-san
Thanks for your question.
There's quite a lot of other configuration data for the OGG Service Manager and the deployments for the installation. You can have a separate volume for each deployment, like this:
    volumes:
      - T:\docker-data\ogg\ServiceManager:/u02/ogg/ServiceManager
      - T:\docker-data\ogg\Local:/u02/ogg/Local

or keep all of the Service Manager and deployment data together:
    volumes:
      - T:\docker-data\ogg:/u02/ogg

If you really want trail data on another separate volume, you can mount a trails volume like so:
    volumes:
      - T:\docker-data\ogg:/u02/ogg
      - T:\docker-data\ogg-trails:/u02/ogg/Local/var/lib/data

I tried an option with
    volumes:
      - T:\docker-data\ogg:/u02/ogg
      - T:\docker-data\ogg-trails:/u02/ogg/Local/var/lib/data

But container failed with OutOfMemoryException. docker logs are attached.
out1.txt
output.txt

Same goes for:
    volumes:
      - T:\docker-data\ogg:/u02/ogg

out1.txt
output.txt

authors: alesana-san sbalousek

repo: oracle/docker-images


text:
$CONTAINERCONFIG_DIR/RCU.$RCUPREFIX.suc invalid check..

In the below script $CONTAINERCONFIG_DIR/RCU.$RCUPREFIX.suc is not created after successful execution of RCU. making it an invalid check.
OracleFMWInfrastructure/dockerfiles/12.2.1.2/container-scripts/createOrStartInfraDomain.sh

@kulbhushan1 Make sure if RCU ran successfully but you ran into an error when you create the domain to use a new unique RCU Prefix.

@kulbhushan1 are you still facing this issue?

Please reopen if still facing this issue


text:
Update NBT to 0.9.15

What does this PR do?
Update NBT to 0.9.15

Deprecated, as current NBT version is now 0.9.16

repo: oracle/graalvm-reachability-metadata


text:
As a user, I want to see the output more clearly so i can distinguish…

… between the outcome of each command

I did an npm install on all the packages. package.lock.json was updated for vscode-extension package.

Please also ignore the Pre-publish Build - master check, this is an error in our ci configuration.

repo: oracle/netsuite-suitecloud-sdk


text:
Add sample running Oracle WLS Kubernetes Operator on Azure Kubernetes Service

Please review
weblogic-kubernetes-operator/docs-source/content/samples/simple/azure-kubernetes-service/_index.md

Tested on AKS.

The PR has a non-image binary file. It's the test war: kubernetes/samples/scripts/create-weblogic-domain-on-azure-kubernetes-service/testwebapp.war. I've been working with the understanding that our security rules don't allow a checked-in binary file. Let me double-check that this is still the expectation.

Hello @rjeberhard , because today's changes included a modification to the script to remove the ClusterRoleBinding actions, I converted this to a Draft PR.  A Draft PR indicates I am not ready to have this merged.  I will test the changes, resolve any issues, and mark it Ready for review as soon as possible.

Thanks, @edburns. I verified with @markxnelson that it is unfortunately correct that the security policy doesn't let us merge changes with non-image binary files. In other samples, we've included the source for the war rather than the war itself.

Hello @rosemarymarano @rjeberhard @jshum2479 , this PR is ready for review.

Added clean up section and fixed changes suggested by @rosemarymarano .  Ready for review again.

@mriccell  may I please kindly have your +1 on this, or let me know what I must change to earn it?

Hello @mriccell , 39ed4677d has your requested changes.

authors: edburns rjeberhard

repo: oracle/weblogic-kubernetes-operator


text:
403 Error when logging in

I just tried logging into the https://upload.theorangealliance.org/setup.html website and it gives me an "invalid username/password" error when entering myTOA info.  I can log into myTOA with that password combo, and it works on the old DataSync tool.  It appears to be giving a 403 error which I have copied below.
VM21:1 POST https://www.googleapis.com/identitytoolkit/v3/relyingparty/verifyPassword?key=<key> 403

Thanks!! Fixed.

authors: jvens ofekashery

repo: orange-alliance/TOA-DataSync


text:
Please import the branch from tfoote/orbbec_ros

This is the branch used for the ROS packages. It's currently resides on my fork: https://github.com/tfoote/OpenNI2/tree/orbbec_ros
Since this is a slightly diverged branch I cannot submit it as a PR it needs to be pushed as a new branch by someone with push access.
Or if someone could create a branch with an appropriate name I'd be happy to submit it as a pull request against that branch.

I see your changes in the ros branch history. Is this done?

Yeah, it looks like they are there. Closing.

authors: BrannonKing tfoote


text:
Add a CLI tool

It would be great to have a CLI tool to manage logs.
The basic commands would be:
$ ipfs-log create
QmFoo1

$ ipfs-log append QmFoo1 "hello world"
QmFoo2

$ ipfs-log values QmFoo2
[{ payload: "hello world", ... }]

$ ipfs-log create --id 'logB'
QmFoo3

$ ipfs-log append QmFoo3 "hi"
QmFoo4

$ ipfs-log join QmFoo2 QmFoo4
QmFoo5

$ ipfs-log values QmFoo5 --size 2
[{ payload: "hello world", ... }, { payload: "hi", ... }]

I would use yargs to wrap the commands.
This one is up to grabs and if anyone wants to work on this, feel free to assign it to yourself or claim it here by saying so in a comment.

hey @haadcode could I take a look at this?

@grantlouisherman I'm not @haadcode but YES :)

Hey @aphelionz @haadcode ipfs-log create command. Did you envision this would be the creation of a Log?

authors: aphelionz grantlouisherman haadcode

repo: orbitdb/ipfs-log


text:
Add "X-Forwarded-For" header for requests with TLS

Is your feature request related to a problem? Please describe.
Using TLS for local development does not work out-of-the-box for applications usually deployed behind a reverse proxy. The reverse proxy (e.g. Traefik) provides a header X-Forwarded-For which is supported by most applications to generate TLS-links.
Describe the solution you'd like
Pass the header X-Forwarded-For like other reverse proxies do.

The proxy is transparent and source IPs aren't affected, so XFF isn't needed. Why does this matter? Any specific examples?

Hi @kdrag0n,
I'm developing with Symfony and basically want the same behaviour as the official Symfony CLI also provides. This might be something like https://github.com/symfony-cli/symfony-cli/blob/e4dd0651f978e258436e35fab04724de0beff939/local/proxy/proxy.go#L304

In PHP Laravel, application always get "http" instead of "https", I think it's the same problem here?

Lots of weird problems with LinkAce. Their docs recommend to set X-Forwarded-For: https://www.linkace.org/docs/v1/setup/setup-with-docker/advanced-configuration/

Fixed for the next version.

Thanks so much for all your efforts!👍👍

Thank you @kdrag0n .

Released in v1.4.0.

authors: JannesMeyer daFish kdrag0n shavenking

repo: orbstack/orbstack


text:
Change to NPM configuration today maybe causing issues

Hey, so starting today, vanilla Cookie Consent started misbehaving in a few ways that I suspect are related to the changes in the way it is being distributed via NPM.
We are using version 2.7.0, and on every deploy, we pull the code down from npm via yarn, and deploy the code from the dist directory. After a deploy earlier today, we got reports of issues from users, which you can read in the following discussions:
https://boardgamegeek.com/thread/2891715/pages-not-displaying-older-apple-oss
https://boardgamegeek.com/thread/2891626/your-privacy
Since we hadn't changed anything related to Cookie Consent, it was pretty confusing, until I tried loading up the site in firefox 78, and saw that now the minified js was using the logical OR assignment (||=), even in the version 2.7.0 code. This was not supported by Firefox until version 79.
The problems reported in the second thread did not seem to be directly related to users using incompatible older browsers--the site was working, but they were having trouble saving the consent preferences. I don't know exactly what caused this issue, but it can't be a coincidence that this bug came up at the same time as the other one.
For now, we are solving this problem by compressing the js from src ourselves, and deploying it. But there is definitely some issue with the way this package is being distributed.
I hope this is a helpful report. Let me know if you have any other questions about it.

Thanks for letting me know!
Looks like there was an issue with the minification tools' config. parameters. I'll make sure the minified output is in es5 format and that it runs in IE9-IE11.
Will release a minor revision soon.

OK, just to give you a quick correction on my initial report, I was inadvertently using the "latest" version of Vanilla Cookie Consent.
When version 2.8.1 went out yesterday, two things happened. As reported in
https://boardgamegeek.com/thread/2891715/pages-not-displaying-older-apple-oss, the code stopped working entirely on old browsers, probably due to minimization. I fixed that by deploying src instead.
However, there was a separate issue, https://boardgamegeek.com/thread/2891626/your-privacy, that was NOT fixed by deploying src. I think that this may be a legit bug in 2.8.1, separate from the minimization, that occurs for users in modern browsers, but I cannot reproduce it myself.
For now, I'm rolling back to 2.7.2 on production to try to avoid this other bug. Sorry I don't have more details about it, but perhaps by reading that thread, you may get some idea of what the issue might be.

v2.8.2 fixes the minified files.
I was not able to reproduce the issue described in the second thread. If v2.7.x works fine for you, you might as well stay with it. v2 will no longer receive feature updates, only bug fixes — as I'd like to focus on v3.

authors: karptonite orestbida

repo: orestbida/cookieconsent


text:
Improved Ukrainian translation

Fixed errors.
Finished translation of english sentences.

@biodranik I didn't change .po files. Will update

@biodranik Copied changes to content.uk.po file.


text:
fix(remote) Fix detection of GitLab merge request sha

Description
GitLabMergeRequest::merge_commit() now returns the commit's sha instead of None if the Merge Request was not merged using a merge commit but with fast-forward.
Motivation and Context
In the generation of the changelog commits were not linked to their GitLab MR if the commit was merged using fast-forward.
This change allows to identify the corresponding GitLab MR independently if the MR was merged with a merge commit or with fast-forward.
How Has This Been Tested?

Existing tests are sufficient and still pass.
Screenshots / Logs (if applicable)
Types of Changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation (no code change)
[ ] Refactor (refactoring production code)
[ ] Other

Checklist:

[x] My code follows the code style of this project.
[ ] I have updated the documentation accordingly. [imho not necessary]
[x] I have formatted the code with rustfmt.
[x] I checked the lints with clippy.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 43.35%. Comparing base (c449229) to head (ad03aea).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #968   +/-   ##
=======================================
  Coverage   43.35%   43.35%
=======================================
  Files          21       21
  Lines        1712     1712
=======================================
  Hits          742      742
  Misses        970      970

Flag
Coverage Δ

unit-tests
43.35% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter hackenbergstefan

repo: orhun/git-cliff


text:
Merge presets by default

Currently presets override your settings by default.
This causes a variety of unintuitive behaviours such as random spawn resetting the difficulty to moki.
If the already implemented merge behaviour would be the only available action, such cases would be handled more intuitively

Behavior changed with bfcb388c1637940c33a12fa00c97edb0de30668a

authors: SiriusAshling timoschwarzer

repo: ori-rando/ui


text:
🛑 TTS / Tillgängliga röster is down

In 3563c38, TTS / Tillgängliga röster ($TTS_ENDPOINT/voices) was down:

HTTP code: 0
Response time: 0 ms

Resolved: TTS / Tillgängliga röster is back up in 9b81ff8 after 15 minutes.

repo: oribisoftware/upptime


text:
🛑 TTS (Oribi Speak talsyntes) is down

In 386fd80, TTS (Oribi Speak talsyntes) ($TTS_ENDPOINT/speak) was down:

HTTP code: 0
Response time: 0 ms

Resolved: TTS (Oribi Speak talsyntes) is back up in 858af96 after 24 minutes.


text:
🛑 TTS (Oribi Speak) is down

In fab65f4, TTS (Oribi Speak) ($TTS_ENDPOINT/speak) was down:

HTTP code: 0
Response time: 0 ms

Resolved: TTS (Oribi Speak) is back up in 4822a5b after 6 minutes.


text:
Default transaction setting is to not use log (apparently not a transaction at all)

Please view orientDB repository issue #3556
Apparently having log off in transactions
request.AddDataItem((byte)0); // use log 0 = no, 1 = yes

can lead the DB to an inconsistent state.

You could also add the newly added comment in the documentation at that line as a comment to prevent this changing back by mistake

using-tx-log tells if the server must use the Transaction Log to recover the transaction. 1 = true, 0 = false. Use always 1 (true) by default to assure consistency. NOTE: Disabling the log could speed up execution of transaction, but can't be rollbacked in case of error. This could bring also at inconsistency in indexes as well, because in case of duplicated keys the rollback is not called to restore the index status.


text:
Using variable in ST_Intersects causes InvalidShapeException

OrientDB Version: 3.2.17 & 3.0.8 running in container
Java Version:  1.8 from container base image
OS: RHEL 8.6 Host
Expected behavior
Should return line records that intersect
Actual behavior
Error: com.orientechnologies.orient.core.exception.OStorageException: Error on executing command: select from line let $geom = (SELECT geom from #14:0) where $geom <> null AND ST_Intersects(aline, $geom) = true
DB name="trick"
Error: org.locationtech.spatial4j.exception.InvalidShapeException: can't make Geometry from: null
Steps to reproduce
I'm trying to find all the "lines" that intersect a specific polygon.  My database structure is:
poly {
  geom: OPolygon
}

line {
 aline: OLineString
}

create class poly
create property poly.geom embedded OPolygon
create index poly.geom on poly(geom) spatial engine lucene
create class line
create property line.aline embedded OLineString
create index line.aline on line(aline) spatial engine lucene
insert into poly set name = 'Poly1', geom = ST_GeomFromText('POLYGON((0 0, 10 0, 10 5, 0 5, 0 0))')
insert into line set name = 'Line1', aline = St_GeomFromText('LINESTRING(2 10, 2 -5)')

select from line let $geom = (SELECT geom from #14:0) where $geom <> null AND ST_Intersects(aline, $geom) = true

The last select statement throws the error mentioned above (in 3.0.8 it's a nullpointer exception).  I am expecting it to return the line since they intersect.
Replacing the $geom in the ST_Intersects with 'POLYGON((0 0, 10 0, 10 5, 0 5, 0 0))' works.  I'm not sure why I can't use a variable there.

Hi @tglman  ,
I would like to start contributing to orientdb , I have significant amount of experience in working with geospatial tools(postgis + psql , qgis etc) . So I was thinking I will start with this issue.
Is this  still a valid issue? and any tips and suggestions for  starting contributing to orientdb?

repo: orientechnologies/orientdb