text:
Consider status code 429 as recoverable errors to avoid resharding

Signed-off-by: Harkishen-Singh harkishensingh@hotmail.com

Refers: #7078
We consider 400 status code same as 200 status code for the purpose of allowing resharding. However, if the code is already rate-limiting (i.e.,429), this will make the matter worse. Hence, take this as recoverable errors and fall in the backoff loop. Also, 100 milliseconds is short for a 429, so maybe 1 second be a good balance.
cc @csmarchbanks @cstyan

This is a breaking change.
I think this is a good idea, because it is very anniyoing in practice, but should be configurable.

@roidelapluie with configurable, do you mean to give users the ability to enable to disable backoff for rate-limiting?

retryability, yes. (but I am not a remote write maintainer)

First, thanks for working on this, I think something like this behavior is certainly useful as dropping 429s indiscriminately is not ideal.
That said, I am not sure we ever decided what the best behavior on a 429 is. If you encounter a 429 in regular operation, chances are your remote write will just continue to fall behind until you force a restart dropping a significant chunk of data. Right now the 429 behavior can be useful because dropping some samples basically means less resolution in your data which might be better for many people not having any recent data at all.
Configuration of what to do on a 429 would be ok, but are there other ideas for handling rate limits more gracefully? Sending a new request that will probably fail 30ms later doesn't seem great. Limited number of retries + respect for the Retry-After header could be one option? This might end up relating to #7912.

I didn't get the retry after thing. Is it something that remote storage will respond to the remote write component, a time after which only, it should retry?

The Retry-After response header may be returned by remote storage to tell the client (remote write) how long to wait before trying another send. It is called out in https://tools.ietf.org/html/rfc6585#section-4.
To be clear, I am not saying implement the Retry-After logic. I am just not sure the 429 behavior should be the exact same as 5XX retry behavior, and would like a discussion. I am curious what @cstyan thinks too.

Ah, I feel like rate limiting is a thing from the remote storage. That means, the remote storage should get more control as to how it wants a particular request to behave. This gives itself the ability to handle the situation and come out of it. So, the response header should be a good way out here.

If there's a standard for how to respond to 429s via the Retry-After header I think we should support it. I'm hesitant to include an option for Max-Retries given the work we've put into remote write to attempt to never lose data, but not strongly opposed.

Ah, any suggestions on how we should move ahead here?

Supporting Retry-After when we see a 429 seems like the way to go to me, @csmarchbanks ?

@csmarchbanks @cstyan I have added the support for Retry-After and Max-Retries (since the discussion seems to be favourable towards some max-retries limit) as input from the response headers from remote storage systems. Can you please give towards the implementation?

@csmarchbanks @cstyan I have added the support for Retry-After and Max-Retries (since the discussion seems to be favourable towards some max-retries limit) as input from the response headers from remote storage systems. Can you please give towards the implementation?

Is this a breaking change and do we want users to opt out?

I don't think existing users would break due to this (as we return default backoff in case the header is not set).

I think retrying 429s classifies as a breaking change if anyone is depending on them not being retried. It should certainly be listed as a [CHANGE].
I would be fine with a temporary flag to allow opting out for a couple of releases, but I am not sure it is required. Happy to defer to your judgement as the release shepherd :)

I don't think existing users would break due to this (as we return default backoff in case the header is not set).

Well it would break users who would not expect a retry at all.

@roidelapluie got it!

LGTM after the last nits. I think that the logs would log that there is a 429, so that should be good for users to understand what is going on.

@csmarchbanks , a note to check the logs as that is changed to log 429 if it is received.

LGTM, we could add later a flag to disable this retries if we see issues in the wild.

This is indeed a breaking change. From what I can we see, we never drop data here? This means Prometheus will just keep falling behind if it is being constantly rate-limited? This assumes that Prometheus doesn't send data to a remote system that basically limits it constantly.
This is not true, we limit people in GrafanaCloud pretty regularly and they're completely fine with it. Further, in Cortex, we return 429 not just on a samples per second limit and do it for other cases too. For example, we have a limit for active series and if a user sends more than their active series limit, we return a 429. In this case, Prometheus would never be able to proceed because it would constantly try to send the sample that creates a new series and it would fail forever.
Now I am not sure if Cortex has the right behavior with 429 for other limits, it seemed pretty right when we built it ;) I still think it has the right behavior, but given Cortex has a major chunk of remote-write use-cases, we shouldn't roll this change out until we fix it there. WDYT @roidelapluie @csmarchbanks?

My choice would be not to break users and threat 429 and non-recoverable by default, with the option to opt-in for this retries. That default could be changed in Prometheus 3.x, I do not think we should depend on only changes in remote writes to make that switch.

429 not just on a samples per second limit and do it for other cases too

@gouthamve I am sorry but isn't this very specific to cortex.

But, unless the Retry-After is specified, it would continue as earlier.

@Harkishen-Singh I don't think so. Before this PR, a 429 was considered a non recoverable error. After this PR, a 429 is always retried until succeed, either it contains the Retry-After or not.

@Harkishen-Singh this is false, as if

429 not just on a samples per second limit and do it for other cases too

@gouthamve I am sorry but isn't this very specific to cortex?
That looks like a custom usage of 429 (I can be wrong). But, unless the Retry-After is specified, it would continue as earlier.

This is not exact: before, we would not retry at all. Now, with 429 and no retry-after, we would retry with the defaut backoff.

Note: #8474 is open.

Sorry, I think this change is only safe if you don't retry without the header, so it would default to the same behavior as previously for 429, and it should be behind a flag probably. Otherwise this will retry even without the header from the remote storage backend, which could be very problematic when 429's are returned for rate limiting or even reasons that are not ever going to succeed on retry.

We are putting it behind a flag for this release, and will likely put rules such as what you suggest or limited number of retries in future releases. See #8474 and #8477.

authors: Harkishen-Singh csmarchbanks cstyan gouthamve pracucci ranton256 roidelapluie


text:
Use a 128 sample buffer in memSeries

This is intended as a performance improvement for instant and short range queries, i.e. most rule evals. It expands the memSeries buffer from 4 to 128 samples, so it covers the whole last chunk, avoiding decoding the whole chunk for the samples most likely to be queried by rule evaluation.
It increases memory utilization by 2 KB per head series, improving instant and short range matrix query times by 20%+. Memory utilization could be reduced, either by lowering the buffer size (not very useful, as the iterator would still have to decode the part of the chunk not covered by the buffer) or by selectively using larger buffers for queried series only.
Before and after benchmark results and comparison are attached.
The complete change (including the Prometheus side of things) is in free/prometheus.

It increases memory utilization by 2 KB per head series

By past measurement that's the current overall cost of one head series, i.e. this would double Prometheus's memory usage. Did you try this out in a live Prometheus server?

By past measurement that's the current overall cost of one head series, i.e. this would double Prometheus's memory usage.

It is actually exactly a 2 KB per series increase in memory usage. (The iterator buffers are pooled and only one iterator is live at a time for each query, so they really don't matter.) So the memory increase is pretty easy to estimate for every Prometheus instance.

Did you try this out in a live Prometheus server?

Yes I did, in fact. One with ~50K series. (It also includes all my other proposed Prometheus performance improvements (prometheus/prometheus#4248, prometheus/prometheus#4271, prometheus/prometheus#4315 and prometheus/tsdb#351) so it's not exactly an apples to apples comparison, but also shouldn't affect steady state memory usage. Regardless, on this particular server (with a limited number of series, relatively expensive recorded rules and very little user queries) steady state memory usage is higher by an expected ~100 MB.
Here's the memory profile, for reference: pprof.prometheus.alloc_objects.alloc_space.inuse_objects.inuse_space.022.pb.gz. It shows 115 MB allocated by newMemSeries (new series creation, in 2.25 KB chunks), compared to 189 MB allocated from below (*memSeries).append (appending to existing series), so about a 60% increase in memory used by TSDB. (I use a 5s scrape interval and 10s eval interval, the percentage will likely be higher with e.g. 1m scrape intervals.)
If this turns out to be unacceptable, one could instead stick with the 4 sample buffer by default and switch to a dynamically allocated slice for queried series only. On the one hand that would limit the memory usage increase by buffering fewer series (those for which iterator speed matters) and on the other hand might use smaller buffer sizes for short-lived series. There would be a small increase in mutex contentions the first time a series is queried (because the samples need to be decoded into the buffer), but that would only be an issue (if at all) for one-off queries that touch previously unqueried series. (Rule eval would presumably do the locking either when the series is created, and thus the chunk is almost empty, or at startup.)

@free would you have time to resolve the conflict and I will look into this.
I am running some benchmarks as part of another PR so have an isolated machine to compare the results.
Can also run a proper Prombench test as well with these changes.

Conflicts resolved, tests are passing. I haven't had the time to build and test a Prometheus instance with this latest version, but I wouldn't expect any issues (or differences from my earlier test).
Like I said in my earlier comments, this change could be adjusted to dynamically allocate a buffer only for series that are being queried (as opposed to all series). And the buffer size could be configurable, e.g. if you know you're scraping every 10 seconds and your rules use maximum ranges of 5 minutes, then you could use a buffer size of 30 + epsilon instead of 128, as you'd only be interested in the last 30 samples when doing rule evaluation.
Oh, and finally, as part of the merge I've had to revert @brian-brazil's #444, because sampleBuf is now a circular buffer and so the computation of the index may not be worth the memory savings (especially seeing as how this change trades memory usage for CPU anyway).

Like I said in my earlier comments, this change could be adjusted to dynamically allocate a buffer only for series that are being queried (as opposed to all series)

This is still potentially all series, as it's easy for a user to request all of them and cause a sudden bump in memory usage.
For a Prometheus with 10M series that's an extra 4GB of RAM (have to double the usage due to GC). I don't think that's a good tradeoff.

For a Prometheus with 10M series that's an extra 4GB of RAM (have to double the usage due to GC). I don't think that's a good tradeoff.

4GB on its own looks like a big, scary number. But when considered in relation to the baseline memory usage of a Prometheus with 10M series, I doubt it looks all that bad. I run a Prometheus with only 130K series and a baseline memory usage of ~2.5GB, so an extra 2KB per series is merely 10% overhead. The percentage may be larger for a Prometheus instance with 10M series, but I for one would gladly trade 10% RAM for 20% CPU. And if I had to put money on it, I'd guess most Prometheus deployments are closer to 130K than 10M series.
Even better, it would be great to have a knob I could turn where I am free to choose between 20% more RAM or 20% more CPU usage. Or anything in-between. Which is fully doable.

This is still potentially all series, as it's easy for a user to request all of them and cause a sudden bump in memory usage.

It is just as easy for a user to issue 10 parallel queries going back 1 year or covering a few thousand series. I'm sure you can easily get a Prometheus with 10M series (or 130K series, for that matter) to blow up that way, no 2KB/series overhead needed. And you don't need some perfect storm for that to happen, just some clueless user selecting a 1 year range on a Grafana dashboard.
Plus, if this is the main concern, it should be relatively easy to either add a counter to each series and only allocate the buffer once the series was queried more than N times (with some exponential decay thrown in). Or limit the buffering to series queried by rule evaluation. Or a few other ways.

And if I had to put money on it, I'd guess most Prometheus deployments are closer to 130K than 10M series.

We have many users with excessive cardinality, and we hear a lot from them.

It is just as easy for a user to issue 10 parallel queries going back 1 year or covering a few thousand series.

As of 2.5.0, there's protections for that. That's also memory usage only for the lifetime of the query, not the lifetime of the series.

Or limit the buffering to series queried by rule evaluation.

Some users do put big queries in rule evaluation. I don't think it's safe operationally to have a varying value for this.

the initial bench results show some gains and losses in performance.
benchcmp -mag -best old128SampleBuffer.txt new128SampleBuffer.txt
benchmark                                                                                old ns/op      new ns/op      delta
BenchmarkRangeQuery/expr=a_ten,steps=1-16                                                                              121834         100264         -17.70%
BenchmarkRangeQuery/expr=rate(a_hundred[1d]),steps=1000-16                                                             1525009607     1280496030     -16.03%
BenchmarkRangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1-16                                      143303         169895         +18.56%
BenchmarkRangeQuery/expr=changes(a_hundred[1d]),steps=1000-16                                                          2049532384     2427188884     +18.43%
BenchmarkRangeQuery/expr=a_ten_unless_b_ten{l=~'.*[0-4]$'},steps=1-16                                                  319685         270065         -15.52%
BenchmarkRangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1000-16                                    518656298      610938446      +17.79%
BenchmarkRangeQuery/expr=sum_without_(l)(rate(a_ten[1m]))_/_sum_without_(l)(rate(b_ten[1m])),steps=100-16              967917         1134796        +17.24%
BenchmarkRangeQuery/expr=rate(a_hundred[1m])_+_rate(b_hundred[1m]),steps=1000-16                                       125753845      107611840      -14.43%
BenchmarkRangeQuery/expr=holt_winters(a_one[1d],_0.3,_0.3),steps=100-16                                                8085431        9422173        +16.53%
BenchmarkRangeQuery/expr=sum_without_(l)(h_one),steps=100-16                                                           1416127        1220579        -13.81%
BenchmarkRangeQuery/expr=a_hundred_-_b_hundred,steps=1000-16                                                           94911985       109603131      +15.48%
BenchmarkRangeQuery/expr=abs(a_hundred),steps=1-16                                                                     1233212        1070040        -13.23%
BenchmarkRangeQuery/expr=sum_by_(l)(h_one),steps=10-16                                                                 130891         150399         +14.90%
BenchmarkRangeQuery/expr=abs(a_one),steps=1-16                                                                         32564          37414          +14.89%
BenchmarkRangeQuery/expr=rate(a_one[1m])_+_rate(b_one[1m]),steps=10-16                                                 81303          70909          -12.78%
BenchmarkRangeQuery/expr=-a_one,steps=1-16                                                                             35849          31373          -12.49%
BenchmarkRangeQuery/expr=a_ten_unless_b_ten{l=~'.*[0-4]$'},steps=10-16                                                 342533         300642         -12.23%
BenchmarkRangeQuery/expr=a_one_-_b_one,steps=1000-16                                                                   1054326        934382         -11.38%
BenchmarkRangeQuery/expr=sum_without_(l)(rate(a_one[1m])),steps=1-16                                                   47953          42934          -10.47%
BenchmarkRangeQuery/expr=sum_without_(l)(rate(a_hundred[1m])),steps=1000-16                                            28271676       25552649       -9.62%
BenchmarkRangeQuery/expr=rate(a_ten[1m]),steps=1000-16                                                                 2043487        1847455        -9.59%
BenchmarkRangeQuery/expr=holt_winters(a_one[1d],_0.3,_0.3),steps=10-16                                                 1840960        2030539        +10.30%
BenchmarkRangeQuery/expr=sum_without_(l)(h_one),steps=10-16                                                            262000         237881         -9.21%
BenchmarkRangeQuery/expr=sum_by_(le)(h_ten),steps=100-16                                                               3617587        3982531        +10.09%
BenchmarkRangeQuery/expr=-a_one,steps=100-16                                                                           44151          40182          -8.99%
BenchmarkRangeQuery/expr=abs(a_one),steps=10-16                                                                        35829          39178          +9.35%
BenchmarkRangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1000-16                                               773132         707293         -8.52%
BenchmarkRangeQuery/expr=label_join(a_hundred,_'l2',_'-',_'l',_'l'),steps=100-16                                       5921697        6466325        +9.20%
BenchmarkRangeQuery/expr=a_one_or_b_one{l=~'.*[0-4]$'},steps=1000-16                                                   740113         805824         +8.88%
BenchmarkRangeQuery/expr=a_one_-_b_one,steps=100-16                                                                    148185         136132         -8.13%
BenchmarkRangeQuery/expr=sum_without_(le)(h_ten),steps=1-16                                                            1066798        980667         -8.07%
BenchmarkRangeQuery/expr=a_one,steps=1000-16                                                                           124525         115186         -7.50%
BenchmarkRangeQuery/expr=-a_ten,steps=10-16                                                                            109560         101547         -7.31%
BenchmarkRangeQuery/expr=sum_without_(l)(rate(a_ten[1m]))_/_sum_without_(l)(rate(b_ten[1m])),steps=10-16               293979         316468         +7.65%
BenchmarkRangeQuery/expr=sum_without_(le)(h_ten),steps=10-16                                                           1255500        1170190        -6.79%
BenchmarkRangeQuery/expr=rate(a_hundred[1m]),steps=1-16                                                                1069964        999694         -6.57%
BenchmarkRangeQuery/expr=rate(a_one[1m]),steps=1-16                                                                    34317          36665          +6.84%
BenchmarkRangeQuery/expr=rate(a_one[1m]),steps=10-16                                                                   36003          38342          +6.50%
BenchmarkRangeQuery/expr=a_hundred_-_b_hundred,steps=100-16                                                            11154411       11873247       +6.44%
BenchmarkRangeQuery/expr=a_one_-_b_one,steps=10-16                                                                     55085          51814          -5.94%
BenchmarkRangeQuery/expr=a_one_unless_b_one{l=~'.*[0-4]$'},steps=1-16                                                  144936         137033         -5.45%
BenchmarkRangeQuery/expr=sum(a_one),steps=100-16                                                                       103514         97969          -5.36%
BenchmarkRangeQuery/expr=a_one_-_b_one,steps=1-16                                                                      47120          44797          -4.93%
BenchmarkRangeQuery/expr=label_replace(a_one,_'l2',_'$1',_'l',_'(.*)'),steps=1000-16                                   675299         710159         +5.16%
BenchmarkRangeQuery/expr=sum_by_(l)(h_one),steps=100-16                                                                342576         360116         +5.12%

benchmark                                                                                old bytes     new bytes     delta
BenchmarkRangeQuery/expr=histogram_quantile(0.9,_rate(h_hundred[5m])),steps=1000-16                                    10640763      12621088      +18.61%
BenchmarkRangeQuery/expr=rate(a_hundred[1d]),steps=1000-16                                                             3183988       2804738       -11.91%
BenchmarkRangeQuery/expr=sum_without_(l)(h_hundred),steps=1000-16                                                      10581144      11118767      +5.08%
BenchmarkRangeQuery/expr=sum_without_(le)(h_hundred),steps=1000-16                                                     60444334      58485049      -3.24%

Looking at this from a slightly different angle, I find it mind-boggling that everything in TSDB is encoded as soon as it's written, requiring the query engine to decode and cache that data (currently for every single query; in some ideal future across rule evaluations, but not across API queries). If the TSDB could keep the most recently written samples unencoded, Prometheus would be able to access those samples without wasting any CPU and without needing to cache them. Saving both memory and CPU.
I know I'm over-simplifying to some extent, but there are quite a few decisions in the query engine-TSDB interaction which puzzle me. Off the top of my head, this is one of them; the other is the TSDB being designed to be an ACID database, flushing every single write, and yet Prometheus expressly providing no guarantees regarding persistence. I have a feeling that sorting out some of these inconsistencies could yield significant improvements, either in terms of resource utilization or in terms of features/guarantees/whatever.

the initial bench results show some gains and losses in performance.

That looks rather confusing to me. In my earlier benchmarking there were virtually no losses in performance across all benchmarks (there was literally 1 instance of +4%, 1 instance of +3% and the rest were all below 2%). Could you limit the benchmarking to one of the significant slowdowns and compare CPU profiles?
In particular, the third benchmark in your listing -- BenchmarkRangeQuery/expr=label_replace(a_ten,_'l2',_'$1',_'l',_'(.*)'),steps=1-16 -- was 18% faster in my earlier test, but 18% slower in your test. I doubt the difference stems from the number of cores (which I suppose the 16 in your case and 4 in my case stand for).

this looks like an interesting change and I would like to continue the tests.
pinged you on IRC.

Fair enough, I must agree with all of your points.
One last question, though. Would any of you change your mind if buffering only applied to time series touched by recording rules AND was not tunable but merely an on-off switch? I.e. one could specify on the command line whether they wanted lower memory usage or faster rule evaluation. The default would be as it is now and the only overhead would be one nil pointer per time series.

For now it seems that there isn't enough use-cases to justify adding this so closing and will revisit when/if needed.
@free thanks for putting the time and effort with this and everyone for the feedback.

source: gharchive/pull-request

authors: brian-brazil fabxc free krasi-georgiev

repo: prometheus/tsdb

license_type: permissive

license_source: bigquery


text:
Add Cerebras provider

Cerebras' API is OpenAI-compatible, so it would be easy to add it like the groq one.
Docs: https://inference-docs.cerebras.ai/api-reference/chat-completions

123

source: gharchive/issue

repo: promplate/demo

license: MIT

license_source: github-api


text:
outputPath is not working

Tried setting up outputPath in the config as both string and array of string following this guide https://www.promptfoo.dev/docs/configuration/parameters/#output-file. However, both didn't work
tests:
  - file://datasets/customer/mini_default_test.yaml

outputPath:
  - reports/customer_output.html
# outputPath: 'reports/customer.html'

Suspect that it is caused by outputPath is not read in readConfigs function
https://github.com/promptfoo/promptfoo/blob/e6089ec7594282f9be6094a52e8483e044cd7e78/src/config.ts#L357-L390

@mldangelo I went ahead with a PR https://github.com/promptfoo/promptfoo/pull/1734

Shipped in 0.89.4!

authors: albertlieyingadrian typpo

repo: promptfoo/promptfoo


text:
Add bits::u128 and bits::i128

The num::u128 and i128 strategies are present, but the bits::u128 and i128 strategies are missing. Having this would complete the set of strategies for bits for all primitive integer types.

I would like to work on this. Could you please assign it to me?

@tltsutltsu thank you for volunteering! you have been assigned

@tltsutltsu if you'd still like to work on this, let me know but I'm going to remove the assignment for now

repo: proptest-rs/proptest


text:
Website Monitoring feedback 202302 and 202303

I have expanded the scope of this issue to be feedback on the various website-monitoring reports that have come in during 202302 and 202303.  I'll consider this done when we have a first draft that I would feel comfortable sharing with other leaders and not needing to be there to answer/explain it.  After that we can develop a separate process for how we report ongoing observations, questions, and suggestions.

This concerns https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-7/ipfs/README.md#website-monitoring
First off, thanks for adding this!  Good stuff.
A few things that I think would be helpful to document:

What is the configuration of the node monitoring these sites?  For example, is it stalk Chromium phantoms node?  (I think we should be explicit that Companion (for intercepting IPFS URLs) is not in the mix.

Is the cache cleared between each run?

I assume "Page Load" is https://github.com/macbre/phantomas/blob/devel/docs/metrics.md#performancetimingpageload .  I don't find their docs helpful.  There is so much that goes into loading a page.  That said, I assume this is the "Load" metric that shows up in one's web inspector (screenshot - red vertical bar).  I could imagine it would be better to get DOMContentLoaded (blue vertical) since that isn't as susceptible to the JS processing on the page I believe (but does capture the network traffic up front fetching JS).  (That said, this isn't my expertise and I know there are a lot of intricacies.  @lidel will likely have a good suggestion here.). Regardless, I'd love to be more specific than "page load", or at least point people to something like https://developer.mozilla.org/en-US/docs/Web/API/PerformanceNavigationTiming so they have more insight into what that means)

Week over week trends - It would be great to have a mechanism to detect if this radically changes week over week.  One idea would be to pick a few sites and a few regions and plot the p50 and p90 of time to first bye (since that shouldn't be susceptible to the content of the page).

Other sites I could imagine adding:

ipfs.tech
blog.ipfs.tech
docs.libp2p.io
blog.libp2p.io
Or maybe it would be worth just syncing to whatever set of sites are being pinned to the collab cluster

Agree with @BigLep, we should  add 1 paragraph that defines what we mean by "The time it took to fully load the page.".
I know  @guseggert was looking into doing way more than measuring how long HTML took to transfer, and more than DOMContentLoaded – he was waiting for DOMContentLoaded AND also for all images on a page to load as well.  Not sure what ended up being the final metric, but we should document it to remove guesswork/confusion.
+ small nits:

add https://en.wikipedia-on-ipfs.org/wiki/ to reduce bias
replace ipfs.io with ipfs.tech  (context: https://github.com/protocol/bifrost-infra/issues/2018)

Great, thanks for your the feedback!

What is the configuration of the node monitoring these sites? For example, is it stalk Chromium phantoms node? (I think we should be explicit that Companion (for intercepting IPFS URLs) is not in the mix.

We run website measurements every six hours. Each of these measurements we consider a measurement run.
In each run, we start six Kubo nodes around the world in different AWS regions. As soon as their API is reachable, we wait for 10s to let the node settle, and then request these websites one by one. We request each website three times. Then we wait 10 minutes and request the set of websites again. We thought that this may simulate a warm(er) Kubo node that has a "better" routing table. The graphs in the weekly report don't yet distinguish between cold and warm nodes.
Actually, we're not only starting six Kubo nodes but twelve because we want to test the most recent stable Kubo version (v0.18.0) and the most popular according to our network crawls (v0.17.0 - up until last week, now it's also v0.18.0).
Another detail: in each run, we are also requesting every website via plain HTTP without going through the local Kubo node. This means we could compare both protocols.
We can easily change all of the above parameters (4x a day, 6 regions, settle times, 3 retries, 2 Kubo versions, etc.).
We are running the Kubo nodes on AWS's t3.medium instances and request the website through Kubo's local gateway. E.g., a website request looks like this: http://127.0.0.1:8080/ipns/filecoin.io. We thought that this would come close to the Companion's performance. However, IIUC if a user browses to, e.g., https://filecoin.io, the x-ipfs-path header gets intercepted and only the remaining resources will be loaded via IPFS. I think, with our experiments, we're simulating the case where a user would directly browse to ipns://filecoin.io.

Is the cache cleared between each run?

Yes, between each retry we run ipfs repo gc. However, the first website request is likely slower than the subsequent ones because BitSwap will discover the provider immediately in later requests.

I assume "Page Load" is https://github.com/macbre/phantomas/blob/devel/docs/metrics.md#performancetimingpageload

That's right :+1:

That said, I assume this is the "Load" metric that shows up in one's web inspector (screenshot - red vertical bar).

Not sure which screenshot you're referring to :/

Regardless, I'd love to be more specific than "page load"

Totally agree! Just to clarify what the performanceTimingPageLoad measures. I just looked it up (source) and it measures the difference between

loadEventStart - representing the time immediately before the current document's load event handler starts. The load event is fired when the whole page has loaded, including all dependent resources such as stylesheets, scripts, iframes, and images. This is in contrast to DOMContentLoaded, which is fired as soon as the page DOM has been loaded, without waiting for resources to finish loading.
navigationStart - a depracated feature "representing the moment [...] right after the prompt for unload terminates on the previous document"

@lidel phantomas already measures the DOMContentLoaded 👍 So I'll just replace the performanceTimingPageLoad metric with the DOMContentLoaded one?

Week over week trends - It would be great to have a mechanism to detect if this radically changes week over week. One idea would be to pick a few sites and a few regions and plot the p50 and p90 of time to first bye (since that shouldn't be susceptible to the content of the page).

Yup, I also wanted to have that but couldn't come up with a nice visualization that nicely captures all dimensions (latency, datetime, region, website, data points). What you're suggesting is actually a nice trade-off I think.
Suggestion: take ipfs.tech + en.wikipedia-on-ipfs.org/wiki/ p50 + p90 timeToFirstByte latencies from eu-central-1 and report these numbers every week.

Other sites I could imagine adding:

Updated list of websites: https://github.com/protocol/probelab-infra/pull/17

@dennis-tra : again, this is awesome, and thanks for the great reply.
A few followups...
Questions around the definition of a run

How are you combining all the datapoints for a <region,site,run>?  If I understand correctly, the <region,site,run> tuple is collecting 12 datapoints: 2 kubo version x 3 tries x 2 modes in http (direct-to-domain )and non-http (through local Kubo HTTP gateway)?

However, IIUC if a user browses to, e.g., https://filecoin.io, the x-ipfs-path header gets intercepted and only the remaining resources will be loaded via IPFS. I think, with our experiments, we're simulating the case where a user would directly browse to ipns://filecoin.io.

Yeah, you're right that there are some differences with companion vs. hitting the local Kubo HTTP gateway, but I think your approach is good/clean/simple.  Companion with time learns which domains are available via non-HTTP, and so future domain accesses is able to re-route the URL to the configured IPFS HTTP gateway.  (I believe there's more nuance here, but I think what you're capturing here is good/fine.)

Another implication of your "companion" approximation by loading through the local Kubo HTTP gateway is that page resources that don't live under the root domain will be fetched via HTTP.  In the companion case, they may get intercepted and fetched using non-http.

As a contrived example, for ipfs.tech, any assets (images, CSS, JS, etc) under ipfs.tech will continue to be fetched via non-http when you do http://127.0.0.1:8080/ipns/ipfs.tech, but if the top-level ipfs.tech page has <img src="https://libp2p.io/logo.png"/> that will be fetched via HTTP since Phanotms doesn't have the intelligence to know that libp2p.io assets can be fetched via non-http.  I think this is fine, but I think this caveat should be documented.

It's great we're clearing Kubo's cache.  What about the browser (phantoms) cache?

Events we're measuring around the page lifecycle
Doh, I added the screenshot to my original post - thanks.
load vs DOMContentLoaded - I've got a few things to say around this but am not necessarily laying out the points in the best way.

I think "load" event is actually what we want because it is doing what Lidel and Gus were looking at of waiting for all the images, JS, CSS, etc. to load.  We want all this network traffic for comparing fetching a site via non-http (local Kubo HTTP gateway) and HTTP (hitting the domain direct).
That said, in its current presentation, it will cause confusion because "load" times aren't necessarily comparable across sites.  Yes, smaller load time is better, but it also generally doesn't mater from a UX regard if the above the fold content is interactable sooner.  There are more objective assertions that can be made around DOMContentLoaded where studies have done to show that users report a site is "fast" if that event happens within X ms.  (I'm not up on the latest studies here.)  Something like DOMContentLoaded is more comparable in that sense.  That said, our goal here isn't to provide feedback to site creators on whether their site is meeting ideal UX targets....
By default when some people see https://github.com/protocol/network-measurements/blob/master/reports/2023/calendar-week-7/ipfs/README.md#performance-timing-page-load, they're going to think "What?  How is protocol.ai so slow?  It takes 1 second for me to load in my browser?  There must be something up with the measurements, or fetch via non-http is really slow?"
Most aren't going to know that the "load" event means.  We're going to need to caveat this.
Some may not realize that the "load" time is heavily affected by your browser cache and whether you've visited this site recently.
The critical thing we want to show is how does "load" time compare when using http vs. non-http.  At the minimum, non-http should have higher availability.  (If it doesn't, then what good is our tech?). It's ideal for it to also be more performant, but I'm not expecting we'll hit that given 1) most website assets are small and don't take advantage of block deduplication or tricks of fetching blocks from multiple peers and 2) so much has gone into optimizing HTTP for websites over the decades with CDNs, etc.
My big picture comment here is that I actually think the metrics you have are good, but we have to provide a view comparing http with non-http for a given site.

We can compare TTFB across the sites because that is more compatible (because not affected by page content), and it may lead to interesting questions as to "why is TTFB 3x slower for website X than website Y".

Week-over-week reporting
I like the idea of what you're suggesting but would add a bit:
Sites

ipfs.tech
en.wikipedia-on-ipfs.org/wiki/

Regions:

eu-central-1
us-east-1 or 2 (I added this given geography can matter a lot when it comes to latency)

Modes: (I added this, and maybe we do one graph for HTTP and one for non-http).

non-http
HTTP

Metric

p50 timeToFirstByte Latency
p90 timeToFirstByte Latency

The reason I think we want to plot HTTP vs non-http is:

non-http captures improvements in Kubo (or whatever IPFS implementation is used) (e.g., code optimizations, network routing improvements, data transfer improvements)
http mode benefits from all of the above plus HTTP specific things like CDNs.
Breaking them out lets credit be given where credit is due.  For example, I don't want Kubo/IPFS getting credit when the website TTFB goes down just because someone improved the HTTP CDN.

Meta
We have lots of good/important notes about this topic at this point.  Where is the durable place it's going to live?  While I do think we need some explanatory text in the report, I'm also fine for it to link to a more thorough document for more info.  You don't need to block on my review, but please include me when this is available for review.
Again, good stuff, and thanks!

I know @guseggert was looking into doing way more than measuring how long HTML took to transfer, and more than DOMContentLoaded – he was waiting for DOMContentLoaded AND also for all images on a page to load as well. Not sure what ended up being the final metric, but we should document it to remove guesswork/confusion.

(sorry didn't mean to close)

I have expanded the scope of this issue to be feedback on the various website-monitoring reports that have come in during 202302 and 202303.  I'll consider this done when we have a first draft that I would feel comfortable sharing with other leaders and not needing to be there to answer/explain it.  After that we can develop a separate process for how we report ongoing observations, questions, and suggestions.

Fresh observations from looking at week 9
Time First Byte

(nit) The asterisk "*" doesn't resolve to a link on the page.
(nit) "Note: the color scales are different in each graph." → "Note: the color scales are different in each percentile graph."
(nit) "Definitely check the sample size that went into the measurement of each website. The numnber is stated at the bottom of the graph."  This seems redundant.  (I agree with you though anytime we're talking about percentiles we need to be showing the sample size).
I like the idea of the "change since last week percentage", but I don't think it's very useful in practice because there isn't an ability to dive in to see what the trend is.  If previous weeks were very stable and then we see a sudden sustained jump or a slow creep up that is indicative of a problem.  That is what leads me to going with a smaller number of sites points that we plot over time.
I think the waters are further muddied here by combining http and non-http.  I agree that combining tells us if there is a root underlying issue like a provider of the content is misbehaving (e.g., like we had with Fleek) as that affects the http and non-http path but after that, getting insight likely requires separating them out.
We talk about when the stopwatch ends.  Lets also talk about when it starts.

DOMContentLoaded

Per before, I agree that DOMContentLoaded is a more comparable metric between sites because we want all sites to "DOMContentLoad" fast since that better mirrors the customer experience.  That said, this has a lot to do with a site's page layout, included JS, etc.  I don't think with these reports that we want to get into the business of helping site creators realize that their sites could be better optimized or are slow compared to other properties.   As a result, I don't think we should be surfacing here.  (I'm sorry I wasn't more clear on that last time, and certainly feel free to disagree.)
(Ignore if we drop this graph.)  If we keep it, why are we saying "These numbers don't account for the network latencies!".  I thought the stopwatch starts before the network calls have been made to load the page: https://developer.mozilla.org/en-US/docs/Web/API/PerformanceNavigationTiming

HTTP vs. Kubo

I don't think our sample sizes are big enough to use p99.
I think this is most meaningful to see week-over-week trends so we can see if there issues in Kubo, DHT, Gateway provider infra.

One idea would be to have a p90 graph for each tuple of <region,site>.  That graph would have two lines: HTTP and non-http.  This would be a matrix of graphs.  They could be small (200x200 pixels), but big enough for someone to see a trend line.  They can then click it to load a larger version of it.
I agree that we also want to make clear the sample size if looking at percentiles (thanks for adding to it).  Maybe we can add it as a column to the table of graphs above so someone can spot check to know that that sample size is reasonable.

In practice, here is what I'm suggesting:

Feel free to disagree and I'm up for discussing other options.

Also, what are the thoughts capturing all the meta-details?  Per before, we have lots of good/important notes about this topic. I want to make sure we have a durable place for it.  This could be in the report itself or listed somewhere else.
I think we also need a way where we can capture notes/investigations that were done.  I don't think slack threads will scale since it won't make it easy to see prior answers/investigations.  Ideally there is a self-surface way for anyone to see what investigations have done for a given week, or what our callouts/observations are for that week.  I think that likely means having an accompanying github issue, page, or Notion doc for each weekly report.  Some content will carry forward between reports and that's ok.  I'm happy to discuss this more.

Hi @BigLep,
sorry for the late reply. I have been working on improving our measurement setup. I spent some time last week putting our website measurement infrastructure on a new foundation and I'm much more confident about the data we are gathering now. I plan to document everything in this repository (note that we have created a ProbeLab GitHub organization, so this and other private repositories will eventually be migrated to that org). I have also explained my reasoning for the new setup here.
Because of this new setup, we don't have enough data to report in this week's report.
Some notes regarding the metrics we want to report: Further up in this issue, we focussed on the TTFB and domContentLoaded metrics. While working on our website monitoring infrastructure last week I read up on how to measure website performance and came across this list:
https://developer.mozilla.org/en-US/docs/Learn/Performance/Perceived_performance
To quote the website:

Performance metrics
There is no single metric or test that can be run on a site to evaluate how a user "feels". However, there are a number of metrics that can be "helpful indicators":
First paint
The time to start of first paint operation. Note that this change may not be visible; it can be a simple background color update or something even less noticeable.
First Contentful Paint (FCP)
The time until first significant rendering (e.g. of text, foreground or background image, canvas or SVG, etc.). Note that this content is not necessarily useful or meaningful.
First Meaningful Paint (FMP)
The time at which useful content is rendered to the screen.
Largest Contentful Paint (LCP)
The render time of the largest content element visible in the viewport.
Speed index
Measures the average time for pixels on the visible screen to be painted.
Time to interactive
Time until the UI is available for user interaction (i.e. the last long task of the load process finishes).

I think the relevant metrics on this list for us are First Contentful Paint, Largest Contentful Paint, and Time to interactive. First Meaningful Paint is deprecated (you can see that if you follow the link) and they recommend: "[...] consider using the LargestContentfulPaint API instead.".
First paint would include changes that "may not be visible", so I'm not particularly fond of this metric.
Speed index seems to be very much website-specific. With that, I mean that the network wouldn't play a role in this metric. We would measure the performance of the website itself. I would argue  that this is not something we want.
Besides the above metrics, we should still measure timeToFirstByte. According to https://web.dev/ttfb/ the metric would be the time difference between startTime and responseStart:

In the above graph you can also see the two timestamps domContentLoadedEventStart and domContentLoadedEventEnd. So I would think that the domContentLoaded metric would just be the difference between the two. However, this seems to only account for the processing time of the HTML (+ deferred JS scripts).
We could instead define domContentLoaded as the time difference between startTime and domContentLoadedEventEnd.

The revised measurement setup currently gathers the following data:

timeToFirstByte - as defined above
firstContentfulPaint
largestContentfulPaint
The full PerformanceNavigationTiming object

We could also include:

Time to interactive
domContentLoaded - as defined above

If I had the choice between only two metrics, I would choose timeToFirstByte and largestContentfulPaint.

Just want to note that the ask for week-over-week graphs was not unheard! I'm also working on this and will come back here when I have news. I'll try to address all your remarks from the last comment.
Also, I don't have a better place to discuss these things right now. Instead of GH we could use Notion or discuss.ipfs.io. I'll chat with @yiannisbot and @iand and come back here with a proposal. I'd document this discussion in the new location.

Explained the new website measurement methodology here: https://github.com/dennis-tra/tiros

Thanks @dennis-tra for the update.
Infra
Good call on getting good underpinning.
Thanks for sharing https://github.com/dennis-tra/tiros
I like all the defenses to prevent caching.
The only issue I'm wondering there is about: what advantage is Kubo getting for successive runs because it is already connected with the providers of all the website content.  I assume it's effectively getting to bypass content routing via DHT/IPNI and rely on Bitswap's content discovery.
Metrics we're reporting
Agreed on TTFB since that does measure HTTP vs. non-HTTP and is not dependent on the site's initial HTML payload.  It is comparable across sites.
Per before, "I don't think with these reports that we want to get into the business of helping site creators realize that their sites could be better optimized or are slow compared to other properties."  I'm a bit worried we're heading into these waters by talking about firstContentfulPaint, largestContentfulPaint, etc.  I think we should message our rationale with something like "We're including this metric because it helps a site owner see about the impact of using IPFS protocols over HTTP before their site is interactive.  HTTP vs. IPFS protocols have some impact on this metric, but they aren't the only culprit.  There have been many tools developed over the last decades of the modern web to help."  With a message like that, I guess it makes me think we should aslo prefer Time to interactive rather than largestContentfulPaint.  (I can see arguments either way but I would give preference ot interactivity rather than rendering of largestContentfulPaint because of how annoying the user experience is when you can't interact with the page to scroll or click.  Anyways, I'll defer here.)
Discussion place for weekly reports
I like the idea of having a discuss post per week (e.g., https://discuss.ipfs.tech/t/ipfs-measurement-report-calendar-week-10-2023/16114/2 ).  A couple of things:

We should maybe make this its own Discuss category so someone can subscribe to the category for notifications.
(nit) but I think it would be good to get an ISO date into the title: "🌌 IPFS Measurement Report - 2023-03-12"

The only issue I'm wondering there is about: what advantage is Kubo getting for successive runs because it is already connected with the providers of all the website content. I assume it's effectively getting to bypass content routing via DHT/IPNI and rely on Bitswap's content discovery.

We (ProbeLab) also discussed this previously and also assume that a subsequent request will likely be served directly via Bitswap. Since we're tracking if it's the first, second, third, etc., request in the Kubo node's lifetime, we could produce a graph that only considers the first requests. The sample size would be very small, though. Alternatively, we could actively disconnect from the content provider after each request. However, I don't think Kubo gives us information from which peer it fetched the data. If that were the case, we could certainly do that. Then we'd always measure the worst-case performance where we'd need to reach out to the DHT (although we could still, by chance, be connected to another providing peer).
On another note, we're complementing these website measurements with DHT performance measurements, where we directly measure the publication and lookup latencies.

I don't think with these reports that we want to get into the business of helping site creators realize that their sites could be better optimized or are slow compared to other properties.

I also think so, and that's exactly why I argued against the Speed index metric. firstContentfulPaint, and largestContentfulPaint will include latencies for subsequent requests from e.g., script, img, link tags, which may then also be served by Kubo (though, I saw that some websites make cross-origin requests which would then not be served by Kubo). That's why I think these two metrics will depend on Kubo's performance and are worth measuring - but it's certainly muddy.
However, that's also the case for the TTI metric. From the docs:

Time to Interactive (TTI) is a non-standardized web performance 'progress' metric defined as the point in time when the last Long Task finished and was followed by 5 seconds of network and main thread inactivity.

And a "Long Task":

Long tasks that block the main thread for 50ms or more cause, among other issues:

Delayed Time to interactive (TTI).
High/variable input latency.
High/variable event handling latency.
Janky animations and scrolling.

A long task is any uninterrupted period where the main UI thread is busy for 50ms or longer. Common examples include:

Long-running event handlers.
Expensive reflows and other re-renders.
Work the browser does between different turns of the event loop that exceeds 50 ms.

Especially the list of common examples sounds very website specific to me. I could imagine a SPA spending too much time on the main thread rendering the page. This wouldn't have something to do with Kubo's performance IMO.

I think measuring the TTI definitely won't hurt, so I'll try to track it regardless of whether we will eventually report it 👍

We should maybe make this its own Discuss category so someone can subscribe to the category for notifications.

Totally! We could also rename "Testing & Experiments" to something like "Measurements" (just to limit the number of categories). Who owns the forum? I believe I don't have the necessary permissions to create new categories.

(nit) but I think it would be good to get an ISO date into the title: "🌌 IPFS Measurement Report - 2023-03-12"

seems like I can't edit the post anymore :/ will do for the next 👍

Thanks @dennis-tra:
Kubo nodes having established connections to peers in successive runs

I think it's fine to leave as it.  I think we can make the case that it's expected that as IPFS protocols become more prevalent that your likelihood of being connected to a peer that already has the content goes up.  This is the "advantage" that the Kubo/IPFS path gets, just as the HTTP path is getting the advantage of server-side CDN caching.
It does hit on that we need to have a way for someone to find out all the persistent callouts/caveats about a given graph (i.e., someone should be able to see our hypothesis that runs 2+ for a given site will be more performant because of existing connections).  They don't have to be "in your face", but it should be self-service for someone to learn all the caveats (similar to what we're doing in https://www.notion.so/pl-strflt/IPFS-KPIs-f331f51033cc45979d5ccf50f591ee01 ).

Forum
For now I created a measurements subcategory: https://discuss.ipfs.tech/c/testing-and-experiments/measurements/39
I moved and renamed https://discuss.ipfs.tech/t/ipfs-measurement-report-2023-week-10-2023-03-12/16114
I also gave you moderation access.

It does hit on that we need to have a way for someone to find out all the persistent callouts/caveats about a given graph (i.e., someone should be able to see our hypothesis that runs 2+ for a given site will be more performant because of existing connections). They don't have to be "in your face", but it should be self-service for someone to learn all the caveats (similar to what we're doing in https://www.notion.so/pl-strflt/IPFS-KPIs-f331f51033cc45979d5ccf50f591ee01 ).

Totally, with our planned probelab website we want to give detailed description about the graphs and their measurement methodology.

authors: BigLep dennis-tra guseggert lidel

repo: protocol/network-measurements


text:
TypeError: reader.skipField is not a function

Since 3.21.1 I receive
TypeError: reader.skipField is not a function
on
deserializeBinary(data).toObject()
with 3.21.0 it is working
Node 16.17.1

Same with
reader.readUint64String is not a function
Working with 3.21.0

This definitely looks like the closure optimizations taking out too much. In the deeper details, all of our tests pass closure deps side, but it looks like our jasmine config for commonjs doesn't include most of our binary/ tests.

I just published a hotfix release (3.21.2) to npm and will close this out now.  Please re-open and ping me if the problem persists.

authors: Nihcep TA2k dibenede

repo: protocolbuffers/protobuf-javascript


text:
Batch awaitable

Add BatchAwaitable to wait for all stored Awaitable. SagaMonitor now uses BatchAwaitable to ensure all dispatch actions are completed and dispatch results are collected.

Codecov Report

Merging #19 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #19   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          41     41
  Lines         758    764    +6
=====================================
+ Hits          758    764    +6

Impacted Files
Coverage Δ

SwiftRedux/Middleware+Saga/Redux+SagaMonitor.swift
100% <100%> (ø)
:arrow_up:

SwiftRedux/Core/Redux+Awaitable.swift
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90d10de...e0d8915. Read the comment docs.


text:
Unpkg URL in "How to use" section appears broken

The script url in the Readme appears to have a broken redirect:  https://unpkg.com/protomaps@latest/dist/protomaps-leaflet.min.js to https://unpkg.com/protomaps@1.23.1/dist/protomaps-leaflet.min.js
Changing the URL to the latest release in the repo, 1.24.0, also does not work: https://unpkg.com/protomaps@1.24.0/dist/protomaps-leaflet.min.js

fixed, thanks

authors: bdon hampelm

repo: protomaps/protomaps-leaflet


text:
add OCHRE

Insert URLs to the project here:

[x] The projects is active, documented, open source licensed, shows usage from external parties and is directly targeting environmental sustainability. Find more details in the Contribution Guide.

All issues labeled as Good First Issue of the project listed on OpenSustain.tech will be visible on ClimateTriage.com. This is a great way to welcome new community members to your project.
All new projects listed will be posted on our Mastodon channel.

@mnblonsky @jmaguire1

repo: protontypes/open-sustainable-technology


text:
Can't connect kafka

Describe the bug

Can't connect my Stage Environment Kafka.
Set up

docker-compose installation

version: "3.7"
services:
kafka-ui:
container_name: kafka-ui
image: "provectuslabs/kafka-ui:latest"
restart: always
privileged: true
ports:
- "8099:8080"
environment:
KAFKA_CLUSTERS_0_NAME: Stage
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: 10.2.20.143:9092

version: vv0.4.0 521ba0c

Steps to Reproduce
Steps to reproduce the behavior:

create container with docker-compose up -d
check container logs

 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' 2022-05-18 20:50:35,957 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.0.Final
2022-05-18 20:50:35,982 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 13.0.9 on 373c8e33779a with PID 1 (/kafka-ui-api.jar started by kafkaui in /)
2022-05-18 20:50:35,983 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.6.3, Spring v5.3.15
2022-05-18 20:50:35,983 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to default profiles: default
2022-05-18 20:50:38,790 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
2022-05-18 20:50:38,891 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Finished Spring Data repository scanning in 91 ms. Found 0 LDAP repository interfaces.
2022-05-18 20:50:40,168 INFO  [main] c.p.k.u.s.DeserializationService: Using SimpleRecordSerDe for cluster 'Stage'
2022-05-18 20:50:42,407 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
2022-05-18 20:50:42,630 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration:
Using generated security password: 35b6dd3f-87da-4515-bf27-74f9979869ad
2022-05-18 20:50:42,824 WARN  [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
2022-05-18 20:50:43,120 INFO  [main] o.s.l.c.s.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations
2022-05-18 20:50:44,149 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080
2022-05-18 20:50:44,195 INFO  [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 9.109 seconds (JVM running for 10.11)
2022-05-18 20:50:44,250 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: Stage
2022-05-18 20:50:44,278 INFO  [parallel-1] o.a.k.c.a.AdminClientConfig: AdminClientConfig values:
bootstrap.servers = [10.2.20.143:9092]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2022-05-18 20:50:44,403 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka version: 2.8.0
2022-05-18 20:50:44,403 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka commitId: ebb1d6e21cc92130
2022-05-18 20:50:44,403 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1652907044401

here is the dasboard

topics

Connection via Conduktor

Connection via CLI
bin/kafka-console-producer.sh --broker-list 10.2.20.143:9092 --topic my_topic

Expected behavior

Successfully connected kafka-ui app.
Screenshots

Additional context

Hi, it's not an issue with the app, but a docker-network related one.
if you're planning to run it on your local machine, you can try docker's host network mode.
https://docs.docker.com/compose/compose-file/compose-file-v3/#network_mode
Feel free to leave a reply if the problem persists, either here or on discord.

I think thats not the problem cause i also tried with jar without docker. results and logs were same.
export KAFKA_CLUSTERS_0_NAME= Stage
export  KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS= 10.2.20.143:9092
java -jar kafka-ui-api-v0.4.0.jar

Its kind of connection problem.

Okay then, you should've mentioned it beforehand :)
Please share the full log when you run it as a jar file. The one you provided is just a little bit of it.

Set up
export KAFKA_CLUSTERS_0_NAME=Stage
export  KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=10.2.20.143:9092
java -jar kafka-ui-api-v0.4.0.jar

Steps to Reproduce

Run with above commands.
Check logs

 _   _ ___    __             _                _          _  __      __ _
| | | |_ _|  / _|___ _ _    /_\  _ __ __ _ __| |_  ___  | |/ /__ _ / _| |_____
| |_| || |  |  _/ _ | '_|  / _ \| '_ / _` / _| ' \/ -_) | ' </ _` |  _| / / _`|
 \___/|___| |_| \___|_|   /_/ \_| .__\__,_\__|_||_\___| |_|\_\__,_|_| |_\_\__,|
                                 |_|
2022-05-20 13:11:01,833 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.0.Final
2022-05-20 13:11:01,857 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 13.0.11 on C02FV4V3MD6R with PID 21758 (/Users/keremcan.kabadayi/Downloads/kafka-ui-api-v0.4.0.jar started by keremcan.kabadayi in /Users/keremcan.kabadayi/Downloads)
2022-05-20 13:11:01,858 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.6.3, Spring v5.3.15
2022-05-20 13:11:01,858 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to default profiles: default
2022-05-20 13:11:03,165 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
2022-05-20 13:11:03,211 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Finished Spring Data repository scanning in 37 ms. Found 0 LDAP repository interfaces.
2022-05-20 13:11:03,843 INFO  [main] c.p.k.u.s.DeserializationService: Using SimpleRecordSerDe for cluster 'Stage'
2022-05-20 13:11:04,547 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
2022-05-20 13:11:04,716 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration:

Using generated security password: bda90694-ace1-4553-ad51-6ffc9dc66db5

2022-05-20 13:11:04,817 WARN  [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
2022-05-20 13:11:05,009 INFO  [main] o.s.l.c.s.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations
2022-05-20 13:11:05,310 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080
2022-05-20 13:11:05,326 INFO  [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 4.326 seconds (JVM running for 5.173)
2022-05-20 13:11:05,350 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: Stage
2022-05-20 13:11:05,362 INFO  [parallel-1] o.a.k.c.a.AdminClientConfig: AdminClientConfig values:
        bootstrap.servers = [10.2.20.143:9092]
        client.dns.lookup = use_all_dns_ips
        client.id =
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS

2022-05-20 13:11:05,433 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka version: 2.8.0
2022-05-20 13:11:05,433 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka commitId: ebb1d6e21cc92130
2022-05-20 13:11:05,433 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1653041465431
2022-05-20 13:12:08,873 ERROR [parallel-1] c.p.k.u.s.MetricsService: Failed to collect cluster Stage info
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1653041528867, tries=2, nextAllowedTryMs=1653041528968) timed out at 1653041528868 after 2 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 31 due to node 6 being disconnected
2022-05-20 13:12:08,873 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: Stage
2022-05-20 13:12:08,883 DEBUG [parallel-14] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: Stage
2022-05-20 13:13:11,933 ERROR [parallel-14] c.p.k.u.s.MetricsService: Failed to collect cluster Stage info
org.apache.kafka.common.errors.TimeoutException: Call(callName=describeConfigs, deadlineMs=1653041591780, tries=1, nextAllowedTryMs=1653041592031) timed out at 1653041591931 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled describeConfigs request with correlation id 51 due to node 4 being disconnected
2022-05-20 13:13:11,934 DEBUG [parallel-14] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: Stage
2022-05-20 13:13:11,938 DEBUG [parallel-9] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: Stage

Additional context

I need to connect VPN for this Stage Kafka. I can connect with 3rd party apps i mentioned above Conduktor, Kafka CLI and Kouncil
I can ping ip 10.2.20.143

Thanks for your attention.

@keremcankabadayi thanks. Do you run this kouncil tool as a docker container as well?

Yes, running docker container now.
CONTAINER ID   IMAGE                           COMMAND                  CREATED        STATUS        PORTS                                                                             NAMES
7080f97d3117   consdata/kouncil:latest         "java -Djava.securit…"   39 hours ago   Up 39 hours   0.0.0.0:8099->8080/tcp                                                            kouncil

@keremcankabadayi can you ping/traceroute kafka's IP from within kafka-ui container? via docker exec -it kafka-ui sh

➜  apps docker exec -it kafka-ui sh
/ $ ping 10.2.20.143
PING 10.2.20.143 (10.2.20.143): 56 data bytes
64 bytes from 10.2.20.143: seq=0 ttl=42 time=57.378 ms
64 bytes from 10.2.20.143: seq=1 ttl=42 time=58.189 ms
64 bytes from 10.2.20.143: seq=2 ttl=42 time=66.688 ms
64 bytes from 10.2.20.143: seq=3 ttl=42 time=54.729 ms
64 bytes from 10.2.20.143: seq=4 ttl=42 time=54.852 ms
64 bytes from 10.2.20.143: seq=5 ttl=42 time=104.845 ms
64 bytes from 10.2.20.143: seq=6 ttl=42 time=154.781 ms
64 bytes from 10.2.20.143: seq=7 ttl=42 time=100.420 ms
64 bytes from 10.2.20.143: seq=8 ttl=42 time=105.424 ms
64 bytes from 10.2.20.143: seq=9 ttl=42 time=107.621 ms
64 bytes from 10.2.20.143: seq=10 ttl=42 time=63.469 ms
64 bytes from 10.2.20.143: seq=11 ttl=42 time=95.412 ms
64 bytes from 10.2.20.143: seq=12 ttl=42 time=95.873 ms
64 bytes from 10.2.20.143: seq=13 ttl=42 time=56.913 ms
64 bytes from 10.2.20.143: seq=14 ttl=42 time=49.255 ms
64 bytes from 10.2.20.143: seq=15 ttl=42 time=73.346 ms
64 bytes from 10.2.20.143: seq=16 ttl=42 time=53.972 ms
^C
--- 10.2.20.143 ping statistics ---
17 packets transmitted, 17 packets received, 0% packet loss
round-trip min/avg/max = 49.255/79.598/154.781 ms

@keremcankabadayi wanna discuss this on discord to make it quicker? or somewhere else if you don't have one.

The problem seems to be about having a huge amount of topics present in kafka.

workaround for this issue, http://localhost:8098/ui/clusters/<cluster_name>/topics/**<topic_name> you can visit directly topic without all topics loads.

@keremcankabadayi hey, can we talk on discord once again? I have a beta build to test it out

authors: Haarolean keremcankabadayi

repo: provectus/kafka-ui


text:
MSK support broken for 0.5.0

MSK support was working for us in version 0.4.0, but now throws an exception in 0.5.0.  The dashboard shows the cluster is down and the associated pages all timeout and only show a blank, white page.  I can easily reproduce this by switching between versions 0.4.0 and 0.5.0.  This is reproducible on clusters without authentication and data is sent via PLAINTEXT.  The broker version is 2.2.1.
Set up
We deploy using docker-compose with a config that looks much like this:
version: '2'
services:
  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8080:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=Prod
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=mskbroker1:9092,mskbroker2:9092...
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper1:2181,zookeeper2:2181...

I've tried setting KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=PLAINTEXT, but this did not help.
The following exception is thrown in the logs:
kafka-ui  | 2022-12-28 22:42:45,463 ERROR [parallel-18] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [e3d29b7e-17]  500 Server Error for HTTP GET "/api/clusters/<redacted>/brokers"
kafka-ui  | java.lang.IllegalStateException: Error while creating AdminClient for Cluster <redacted>
kafka-ui  |     at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$3(AdminClientServiceImpl.java:45)
kafka-ui  |     Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
kafka-ui  | Error has been observed at the following site(s):
kafka-ui  |     *__checkpoint ⇢ Handler com.provectus.kafka.ui.controller.BrokersController#getBrokers(String, ServerWebExchange) [DispatcherHandler]
kafka-ui  |     *__checkpoint ⇢ com.provectus.kafka.ui.config.CustomWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ com.provectus.kafka.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
kafka-ui  |     *__checkpoint ⇢ HTTP GET "/api/clusters/<redacted>/brokers" [ExceptionHandlingWebHandler]
kafka-ui  | Original Stack Trace:
kafka-ui  |             at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$3(AdminClientServiceImpl.java:45)
kafka-ui  |             at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3776)
kafka-ui  |             at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
kafka-ui  |             at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onError(FluxHide.java:142)
kafka-ui  |             at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
kafka-ui  |             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2063)
kafka-ui  |             at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
kafka-ui  |             at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
kafka-ui  |             at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
kafka-ui  |             at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
kafka-ui  |             at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:187)
kafka-ui  |             at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
kafka-ui  |             at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
kafka-ui  |             at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
kafka-ui  |             at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
kafka-ui  |             at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
kafka-ui  |             at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
kafka-ui  |             at java.base/java.lang.Thread.run(Thread.java:833)
kafka-ui  | Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 7

Hello @jsmaupin,
can you please check docker  image with master ?

@jsmaupin please try the aforementioned solution. Ping me in this issue if that doesn't help.

Sorry for the delayed response.  This appears to be fixed at TOT.  I tested using the tag 0048a7eabdd078606768487af74c4403f8142b60

@jsmaupin glad to hear :)

authors: Haarolean iliax jsmaupin


text:
Explorer shows address in public key field

Summary of Bug
The "Account Details" view of Explorer is showing the address for the account in the field labeled Public Key.
Example:
https://explorer.provenance.io/accounts/pb1q4jdcwg350cqsdzn6jy8fxln9mx5rg7dm5mzs7

Version
v2.2.4
Steps to Reproduce
Search for & view any address in the Explorer. The data values for fields "address" and "public key" are currently identical (showing the address value, not the public key).

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

This will need an FE change, but the data itself will need BE changes.

authors: minxylynx vwagner

repo: provenance-io/explorer-frontend


text:
AWS IAM Access Denied

Steps to Reproduce

prowler aws --log-level ERROR
aws
single account

Expected behavior
Execute without access denied messages
Actual Result with Screenshots or Logs
2024-11-28 10:15:19,249 [File: backup_service.py:164] 	[Module: backup_service]	 ERROR: eu-west-2 -- ClientError[155]: An error occurred (AccessDeniedException) when calling the GetBackupSelection operation: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/ScannerRole/security_scanner is not authorized to perform: backup:GetBackupSelection on resource: arn:aws:backup:eu-west-2:XXXXXXXXXXXX:backup-plan:XXXXXXXX-2434-428d-XXXXXXXX because no identity-based policy allows the backup:GetBackupSelection action
2024-11-28 10:19:20,721 [File: glue_service.py:248] 	[Module: glue_service]	 ERROR: eu-west-1 -- AccessDeniedException[231]: An error occurred (AccessDeniedException) when calling the GetMLTransforms operation: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/ScannerRole/security_scanner is not authorized to perform: glue:GetMLTransforms because no identity-based policy allows the glue:GetMLTransforms action
https://github.com/prowler-cloud/prowler/blob/master/permissions/prowler-additions-policy.json
Does not contain nor does the ViewOnlyAccess or SecurityAduit either:-
backup:GetBackupSelection
glue:GetMLTransforms
Now I think this error is inaccurate as it doesnt show access denied in cloudtrail and the policy has backup:List
2024-11-28 10:15:19,800 [File: backup_service.py:202] 	[Module: backup_service]	 ERROR: eu-north-1 -- ClientError[190]: An error occurred (AccessDeniedException) when calling the ListRecoveryPointsByBackupVault operation: Insufficient privileges to perform this action.
I'm also seeing in Cloudtrail fms.amazonaws.com ListPolicies Access Denied error messages too but not on prowler output
How did you install Prowler?
From pip package (pip install prowler)
Environment Resource

Fargate Task

OS used

Apline Linux

Prowler version
4.6.0
Pip version
na
Context
No response

Hi @garym-krrv,
After some research, I've confirmed that what you mentioned is correct. The backup:GetBackupSelection and glue:GetMLTransforms permissions were indeed missing. I have opened #5962 to add them.
Regarding the other permissions you mentioned, such as the ListRecoveryPointsByBackupVault action, I wasn't able to replicate the issue. It seems that this is covered by the backup:List* permission in the extra policy, so there shouldn't be any issue there.
As for the FMS permission, the reason you're not seeing anything in the Prowler output is that it's an expected and handled error. You can find more information in the FMS service. It is used for the only FMS check fms_policy_compliant.
Let me know if you need further clarification!

repo: prowler-cloud/prowler


text:
SwitchyOmega使用"pd自动切换"导致百度盘网页"列表加载失败"

问题描述(必要)
proxyee-down嗅探模式关闭, SwitchyOmega使用"pd自动切换"导致百度盘网页"列表加载失败"
导致无法加载百度盘内的内容
SwitchyOmega使用直接连接可以加载
SwitchyOmega配置是导入的教程bak配置
版本号(必要)
Firefox ver 61.0.1 64bit
proxyee-down ver 2.54
SwitchyOmega ver 2.5.15
操作系统(必要)
Win10专业版 64bit
相关截图

相关日志
error.2018-07-06.log

sorry新装电脑忘记了firefox证书问题. 我试下证书先

repo: proxyee-down-org/proxyee-down


text:
Update requirements.txt

When i build the docker image from the dockerfile from pyorde/vmware_exporter, and use the image, the container crashes out with this error.
ModuleNotFoundError: No module named 'requests'
Traceback (most recent call last):
File "/usr/local/bin/vmware_exporter", line 5, in
from vmware_exporter.vmware_exporter import main
File "/usr/local/lib/python3.7/site-packages/vmware_exporter/vmware_exporter.py", line 21, in
import requests
i've forked the project, and added requests in requirements.txt
now when building the image from the fork, the problem is solved.

same issue here, not sure how requests was pulled in earlier but since it's clearly used in vmware_exporter/vmware_exporter.py it should be in requirements.txt, no?


text:
Downgrade python version requirements to 3.10.15

This is what we use in our ComfyUI Image and I think that this version requirement is good enough for other cases as well.

Addressed in https://github.com/pschroedl/ComfyUI-SAM2-Realtime/pull/3

authors: leszko pschroedl


text:
Partial Decompilation Of iTime.cpp

Decompiled most of the functions.

Not sure why it didn't delete that when fetching; should be fixed now, as long as merging with the latest and some progress on behaviour.

Not sure why it didn't delete that when fetching; should be fixed now, as long as merging with the latest and some progress on behaviour.

repo: pslehisl/bfbbdecomp


text:
Otelfiber - Any reason to use this repository?

Hello.
I have a honest question. Is there any reason to use this repository instead of the contrib/otelfiber?
Shoudn't we just keep one project to build a stronger ecosystem?
Thanks!

@lucasoares that is excellent question. I didn't know about that library. Back then when I created mine there were zero libs like this.
Will check if otelfiber covers everything, if so then I will archive this one and provide proper description.
Thanks again for pointing this out :) Will keep you posted.

Nice! If otelfiber do not covers everything, we could migrate features from here to there..
But it is a library mantained by the fiber/opentelemetry teams themselves.

Looks like it covers everything. I will update Readme file and archive this repository. Thank you so much @lucasoares for point it out!

authors: lucasoares psmarcin

repo: psmarcin/fiber-opentelemetry


text:
Ldp::Gone when copying permissions to the FileSet

2017-04-12 11:17:05 -0400 (63783) Performed InheritPermissionsJob from Inline(default) in 101.07ms
2017-04-12 11:17:05 -0400 (63783) Error key ldp_gone is not present in the i18n file. You may want to add it.
2017-04-12 11:17:05 -0400 (63783) Rendering 500 page due to exception: Ldp::Gone

Setup
Create a new work with public visibility
Edit newly created work just after it's finished ingesting
Change visibility to "private"
Select "Yes" to copy visibility to file set
You get an error page. Looking in the log, you'll see the above error.

The error only occurs when going from either Public or Institution to Private. Going from Private to anything seems to work fine. I'm pretty sure this was fixed in HydraHead:
https://github.com/projecthydra/hydra-head/commit/06a1bfc17ad3326f3a9f8002688ec0c9a97cbaab#diff-1c76c5c523cfcdcc9af6d152ec53efef
But it required updating to HydraHead 10.4.

repo: psu-stewardship/scholarsphere


text:
Add UUID

Add the uuid of the server so that it is copied faster when the support of a host requests it

Based on the edits made to README, I'm gonna assume you opened this PR against the wrong repository.

faite pas attentions ou  README pls

authors: DiumStream-tech TrixterTheTux

repo: pterodactyl/whmcs


text:
Write to HDFS hive partition.

In HdfsState.java, would you please add a way to config to write dynamic path.
I need this to write to hdfs hive partition by day.
It is like
drwxr-xr-x   - hdfs tbuser          0 2013-11-18 23:20 /staging/anvil/processed/parsed/access/dc=2013-11-18
drwxr-xr-x   - hdfs tbuser          0 2013-11-19 23:15 /staging/anvil/processed/parsed/access/dc=2013-11-19
drwxr-xr-x   - hdfs tbuser          0 2013-11-20 11:31 /staging/anvil/processed/parsed/access/dc=2013-11-20
Thanks
HdfsState.Options options = new HdfsState.Options()
.withFileNameFormat(fileNameFormat)
.withRecordFormat(recordFormat)
.withRotationPolicy(rotationPolicy)
.withFsUrl("hdfs://localhost:54310")
.withPath(path);

I have similar use case where data needs to be saved in a directory created by using values from the message being processed. My use case requires partitioning data by timestamp of the event. Events in the pipeline can get delayed by several hours or days. Pipeline gets events that occurred few seconds ago to few days. I need to save these events in different directories based on their timestamp. I am looking for some way of doing it in Trident implementation.

@rs-01 : I have almost the same problem, did you solve it?

authors: fescandell rs-01 xelllee

repo: ptgoetz/storm-hdfs


text:
ToolTip support

Hi @ptmt, would you mind if I created a PR to add tool tip support?

No example yet because I'm not sure if that's necessary.

Merged, thanks

authors: beretboat ptmt

repo: ptmt/react-native-desktop


text:
Dimensions.addEventListener does not function

Description
Dimensions.addEventListener method does not fire on window resize.
Reproduction Steps and Sample Code
import { Dimensions } from 'react-native';

const handler = (dims) => {
    console.log(dims); // Never called
}

Dimensions.addEventListener('change', handler);

Solution
I have yet to dive into what might be causing this issue but I'll begin to debug.
Additional Information

React Native version: (macos) 0.16.1
Development Operating System: MacOS 10.13.3

I've found that the Dimensions.get("window") just returns the full screen size rather than the size of the window. Probably related..!

@shirakaba you were right! Sorry took me so long. Finally got time to fix minor issues.

authors: jtag05 ptmt shirakaba

repo: ptmt/react-native-macos

license: mit


text:
warnings - componentwillmount has been renamed and is not recommended for use

[x] I have searched existing issues
[x] I am using the latest multi slider version

Steps to Reproduce

Expected Behavior

Actual Behavior

This is fixed in master, but waiting to be released after someone tries out the pending pull request

authors: ptomasroos shachardevops

repo: ptomasroos/react-native-multi-slider


text:
Add other installer types and UI tweaks

Added "edit" button for installer list, made UI fill vertical space. Updated to version 1.5

Note the MainPage.xaml diff looks huge, but that's just because the Grid.Row value changed on a lot of things. Viewing in the VS diffing tool (vs. GitHub diffing) makes it more obvious what has changed.

repo: ptorr-msft/WinGetYamlGenerator


text:
All Lens are not displayed

System Information (please complete the following information):

OS: Mac OS ( Venture 13.4 (22F66) )
Macbook: 14 inc 2021, Apple M1 Pro.
Snap Camera Version: 1.21
Signature Patch applied: yes

Please follow these steps to provide detailed information

Open Snap Camera
Select all Lenses that are not working in your app
Copy and attach the Snap Camera log file to this ticket

(Mac OS): /Users/{YOUR_USER_NAME}/Library/Preferences/Snap/Snap Camera/Log.txt

Log.txt

Extra information:

I do all these steps from this video -> https://www.youtube.com/watch?v=6ezl5hCqm5w&lc=Ugw98NJUzgbj2w3jSBZ4AaABAg.9zLfueTs8U09zMGnPJooB6
I also run errorfix.sh here the output of terminal:

Sashas-MacBook-Pro ~ % sh ~/Downloads/errorfix.sh
Generating MD5 checksum of the Snap Camera binary file
/Users/sashazhgut/Downloads/errorfix.sh: line 22: 8dc456e29478a0cdfaedefac282958e7: value too great for base (error token is "8dc456e29478a0cdfaedefac282958e7")
/Users/sashazhgut/Downloads/errorfix.sh: line 24: 94218d95ced39db2f86f25ef2daa7384: value too great for base (error token is "94218d95ced39db2f86f25ef2daa7384")
Making the binary executable.
Removing the macOS code signing.
Removing extended file attributes.
Re-Signing the application.
You should be able to open Snap Camera now.

I updated the errorfix.sh script, please download and run it again and share the output.
Make sure the docker server is running in the background.

Hm it can't find the file server.js, is the file present in the server directory?
Did you edit any files?

server.js is presented:

I modified only docker compose file ( since I have M1 chip):

I can remove everything and re-run from scratch or move snap-server folder move from application folder to f.e. Downloads.

I've never seen that error before.
Did you whitelist the server folder in your settings?
Video Tutorial

yeap

Try to delete the webapp (also delete under "images" in left menu) and restart server

when I do this, I got error that I can not restart application:
Cannot stop Docker Compose application. Reason: compose [stop] exit status 1. could not find webapp: not found the same situation when I try to start snap-camera server.
So, what I do:
I remove server from application and then put again after that I update to latest version of the docker. Also I add chmod 777 to ssl folder. And after that webapp started. IDK which step gave the result...
for now in terminal I see these errors with connection:

should I remove all images and containers and re-run docker compose up?

If there are still problems, you should remove everything including the volumes.
This command should undo and reset everything if you don't want to do it manually:
docker-compose down --rmi all -v --remove-orphans

Hello,
So, I run this command and it's doesn't work. What I do:

I restarted docker
increase version of MySQL

After these steps I was able to run everything.
TY for your help, this issue we can close.
p.s. Maybe you know how can I fix problem that snap camera video doesn't displayed in Meets, Zoom etc?

p.s. Maybe you know how can I fix problem that snap camera video doesn't displayed in Meets, Zoom etc?

For macOS 14 check out:
https://github.com/ptrumpis/snap-camera-server/discussions/76
For other macOS versions continue here:
https://github.com/ptrumpis/snap-camera-server/discussions/78

authors: Lamadros ptrumpis

repo: ptrumpis/snap-camera-server


text:
Add SWAPI

Thank you for taking the time to work on a Pull Request for this project!
To ensure your PR is dealt with swiftly please check the following:

[x] Your submissions are formatted according to the guidelines in the contributing guide
[x] Your additions are ordered alphabetically
[x] Your submission has a useful description
[x] The description does not end with punctuation
[x] Each table column should be padded with one space on either side
[x] You have searched the repository for any relevant issues or pull requests
[x] Any category you are creating has the minimum requirement of 3 items
[x] All changes have been squashed into a single commit

Thanks

authors: semperry yannbertrand

repo: public-apis/public-apis


text:
[WIP] Stargaze VIP Program

What's left:

[ ] Add a secondary index for block height <> name, either another Map or IndexedMap.
[ ] Use hook in Stargaze Names to burn vNFT if a name association is changed. The minter contract would have to be added as a Name hook.
[ ] Add owner to minter to update params.
[ ] Add pause functionality to minter.
[ ] E2E testing, especially with instantiate2 contract creation.

Codecov Report
Patch coverage: 9.85% and project coverage change: -51.29% :warning:

Comparison is base (536724e) 90.56% compared to head (cb30c9c) 39.28%.
Report is 139 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main      #16       +/-   ##
===========================================
- Coverage   90.56%   39.28%   -51.29%
===========================================
  Files           4       15       +11
  Lines         106      308      +202
===========================================
+ Hits           96      121       +25
- Misses         10      187      +177

Files Changed
Coverage Δ

contracts/fair-burn/src/helpers.rs
100.00% <ø> (+37.50%)
:arrow_up:

contracts/fair-burn/src/state.rs
88.88% <ø> (ø)

contracts/vip/collection/src/bin/schema.rs
0.00% <0.00%> (ø)

contracts/vip/collection/src/contract.rs
0.00% <0.00%> (ø)

contracts/vip/minter/src/bin/schema.rs
0.00% <0.00%> (ø)

contracts/vip/minter/src/contract.rs
0.00% <0.00%> (ø)

contracts/vip/minter/src/sudo.rs
0.00% <0.00%> (ø)

contracts/vip/program/src/bin/schema.rs
0.00% <0.00%> (ø)

contracts/vip/program/src/contract.rs
0.00% <0.00%> (ø)

packages/sg-std/src/lib.rs
0.00% <0.00%> (ø)

... and 4 more

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter shanev

repo: public-awesome/core


text:
Add an AUTHORS file

An AUTHORS file or CONTRIBUTORS file can serve to provide credit to authors and their organisations, in addition to show the diversity of collaborators.
Non-mandatory.

Added a draft, feel free to edit 🙂

authors: bvhme ruphy


text:
UAT | Scrutiny | Only 1 case visibile at a time

Only 1 case visible at a time in the Scrutiny inbox.
Once you either send back the case or Admit the case, the next case is visibile.
To Reproduce
Steps to reproduce the behavior:

Create 2 cases and submit
Complete payment
Login as FSO
Go to My Cases
You will see the first case
Admit the case
Go back to list of cases and you will see the second case.

Expected behavior: All cases to be visible together.
Screen Recording:

This issue is fixed. I have tested in QA env.
Please check video link for reference - https://jam.dev/c/bb288c98-92c1-4315-a163-e4451ef3a5d2

authors: Taherabharmal nandinilp

repo: pucardotorg/dristi


text:
fix: fit channel avatars to consistent size

Before
After

this doesn't happen for me. are you sure you're not overriding some css properties?

I'm not touching any styles, just rendering the <FarcasterEmbed /> directly. Have you tried specifically with the /latenightcrew channel? It depends on the channel avatar, some are square so they just work.

authors: pugson zingerj

repo: pugson/react-farcaster-embed

license: Unlicense


text:
DateJS Folder Size is to big

Hi
Why that  DateJS Folder "vendors\DateJS" Size is 32 MB.
If I need to remove the Unneeded files from it. what can I delete?

Why you want to remove it ? It's a library, that's the the reason it takes so much.
We will close this one for now but if you have further questions you can reopen it whenever you want, or create another one of course.

many files are just repeated over and over for just a very small change for some cultures.
is there another lib or we should stick with this. or I can remove all cultures that I will not use in my application.

Doesn't the moment library do about the same thing??

it's fantastic it's just a few KB .. https://momentjs.com/

Why vendors file is versioned?

authors: AbdullahAlimam AndreiCN SamMousa florentingarnier

repo: puikinsh/gentelella


text:
make etasrcp location

etasrcp is the new location where Princeton ETAS Recap items will be housed. https://bibdata.princeton.edu/locations/holding_locations/etasrcp
It should match up with the etas location.
Still trying to figure out if setting recap_edd to true for the location should be ok.  @mzelesky @kevinreiss ?

Confirmed with @kevinreiss over slack that setting the recap_edd flag was ok for our etas recap items.

This location is in bibdata.princeton.edu and bibdata-staging.princeton.edu


text:
Various fixes and refactors

Fixes issue where running Lando generates an incorrect holding_library.rb file.
Allows all specs to be run with test runners like vim-test.
Moves LocationProcessorService and IndexFunctions out of marc_to_solr and into the main app where they are used exclusively.
Refactors LocationProcessorService from a module of class methods into a real class called LocationMapsGeneratorService.
Extracts indexer build method into a new IndexerService class.
Adds tests for IndexerService and LocationMapsGeneratorService
Moves all specs from marc_to_solr into the main app specs.
Instructs rubocop to exclude the generated translation map files.

Coverage increased (+0.9%) to 82.447% when pulling 6cc3375a22ad81c8082dbee94f838d08d3e7277e on spec-fixes into bf071725b2d5f36dba469de821ad1fdd477bb3c4 on alma.

Coverage increased (+0.9%) to 82.447% when pulling 6cc3375a22ad81c8082dbee94f838d08d3e7277e on spec-fixes into bf071725b2d5f36dba469de821ad1fdd477bb3c4 on alma.

authors: coveralls eliotjordan