text:
Cannot upgrade from 2.2.3 to >= 2.3

I have a container with a solo installation of management-center 2.2.3 (open source version, no license) running without issues. I have the broker connection parameter setup in environment variables.
I tried to upgrade to 2.3 (and 2.3.3 also) but the frontend shows a dialog to setup the broker connection. The same values as the ones I was using give the following error (shown in the snackbar), and I cannot connect:
Error creating connection "_My Mosquitto broker_". Reason: BaseMosquittoProxyClient: Timeout
If I revert back to 2.2.3 everything works properly.
I tried a new solo installation, with new volumes for the config, but I get the same problem. The container logs show (I think there is a regression with issue-768543431):
Starting Management Center for Eclipse Mosquitto
No license key found or provided.
Fri, 11 Mar 2022 08:14:33 GMT express-session deprecated undefined resave option; provide resave option at start.js:195:24
Fri, 11 Mar 2022 08:14:33 GMT express-session deprecated undefined saveUninitialized option; provide saveUninitialized option at start.js:195:24
Started Mosquitto proxy at http://localhost:8088
"CEDALO_MC_PLUGIN_DIR" is not set. Skipping loading of plugins
Mosquitto proxy server started on port 8088
No license key found or provided.
No license key found or provided.
No license key found or provided.
...

If I delete the dialog (using the Chrome's Developer tools), I can see that the frontend shows the following message:
We couldn't retrieve the system status information.
Please make sure that the user "" has the rights to read the "$SYS" topic on the selected broker.

I cleared all the cookies, cache, etc., but nothing works (but revert to 2.2.3).
In Chrome, the console shows these errors:

Is there some new configuration I am missing for versions >= 2.3?
I checked all the documentation but found nothing.

This should be fixed now in the new version 2.3.4.

(will update the 2.3 image later)

The update works now! Thank you!

authors: cleancoderocker mereshow

repo: cedalo/management-center


text:
Runnable example temurin spring

Added docker files for building and running a minimal chimera app developed using spring

@Gizgio83 please update also the README describing the new available templates

@Gizgio83 please update also the README describing the newly available deployment template

Adressed in afb926c543af405200f65a0d9e6afb6b15d0426a

source: gharchive/pull-request

authors: AuPath Gizgio83 marioscrock

repo: cefriel/chimera-deployment-templates

license_type: permissive

license_source: github-api


text:
Regex pattern

Add class with Regex Pattern constants, very useful if you use the same Regex a lot.
This should be at shared module

The feature requested has been added to the API

source: gharchive/issue

repo: celesteoficial/java-library


text:
CI: docgen doesn't build on v0.16.0, v0.17.1, v0.17.2

Docgen didn't build on releases

CI for doc gen failed due to previous step of go-releaser failing. Reference:
https://github.com/celestiaorg/celestia-node/actions/runs/11256100269
Error from CI:
⨯ release failed after 8m19s               error=scm releases: failed to publish artifacts: failed to upload celestia-node_Darwin_arm64.tar.gz after 1 tries: POST https://uploads.github.com/repos/celestiaorg/celestia-node/releases/178898505/assets?name=celestia-node_Darwin_arm64.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}]

Should be fixed by https://github.com/celestiaorg/celestia-node/pull/3877

@jcstein is it fixed now?

it seems to be resolved beginning with https://github.com/celestiaorg/celestia-node/releases/tag/v0.20.0-arabica
will reopen if issues!

authors: cristaloleg jcstein renaynay walldiss

repo: celestiaorg/celestia-node


text:
ipld: Rework Getleaves method to avoid using recursion when requesting leaves

We can avoid making external calls to dag recursively by using ipld.Walker

Closing this issue because received benchmarks are shown that internal implementation with recursion works better:
For single execution:
BenchmarkGetLeavesWithRecursion    	       	1	    364784 ns/op	  291648 B/op	    5622 allocs/op
BenchmarkGetLeavesWithWalker-12    	       1	 238430906 ns/op	199863296 B/op	 1954714 allocs/op
For multiple executions:
BenchmarkGetLeavesWithRecursion-12    	    4353	    294017 ns/op	  291518 B/op	    5620 allocs/op
BenchmarkGetLeavesWithWalker-12    	     384	   2864354 ns/op	 1424676 B/op	   22229 allocs/op


text:
refactor!: have knuu as an object

This PR proposes a structural change to the knuu pkg to have it as an object.
This helps to have more idiomatic code and enable us to add unit tests easier. Moreover, since every test can have their own knuu object and handle everything in isolation.
Note 1: in order to keep it backward compatible with the current tests, some wrapper functions with a global var is added, and they are in a number of files with _old.go postfix so we can delete the files when the users are ready to use this one.
Note 2: There was an attempt not to change everything, i.e. the refactor tries to focus on introducing an object structure for knuu. So, we need a number of refactor rounds on the knuu pkg and in general on the repo.
Closes #328

Since this is a relatively nuanced why of keeping backwards compatibility I would recommend you add some comments to the top of the _old files and their corresponding files that captures how the files are setup and should be used while we are in this period of maintaining backwards compatibility.
Otherwise I think it puts a lot of pressure and burden on the team to keep the context of this refactor in their minds while working on fixes and features.

Is it possible to add a deprecation annotation to the old functions so that we see in the code when we use those functions?

Is it possible to add a deprecation annotation to the old functions so that we see in the code when we use those functions?

good idea, done.

authors: MSevey mojtaba-esk smuu

repo: celestiaorg/knuu


text:
Currency Name is not shown for transaction activity other than the exchange activity in “Activity” Section when user has selected currency other than USD.

Frequency: 100%
App version: Staging build v1.5.2
Does not repro: Play store build v1.5.2.
Repro on: Samsung Galaxy Note 5 (7.0), Samsung Galaxy s5 (6.0.1)
Pre-condition: User must have Exchange activity in Activity section.
Repro Steps:

Launch the app.
Tap on Cog icon.
Tap on Currency option.
select a currency other than the USD.
Navigate back to wallet tab.
Observed.

Impact: Inconsistency in the Activity section as currency name is shown for only exchange activity/transaction, not for other transactions.
Investigation:

Issue does not repro when USD currency is selected
Issue does not repro for other activities such as Sent transaction,Received transactions, Verification fee.

Current Behavior: Currency name is shown for exchange activity.
Expected Behavior: Currency name should not display for only one activity
Attachment: Android_ExchangeActivityIssue.png

@jeanregisser Verified with iOS Test flight build v1.5.2 (17) and observed the same issue.

Currency Name is not shown for transaction activity other than the exchange activity in “Activity” Section when user has selected currency other than USD.

Attachment: IOS_CurrencyIssue.png

Thanks @Lss-Ankit, fixing this now.
Currency code shouldn't appear for exchange items in the feed to be consistent with the other items (Having the 3 letters code makes the value quite unreadable and we decided to drop it for now).

@jeanregisser Verified with iOS test flight build v1.5.2 (18) and found that the issue is fixed as Currency Name is not shown for Exchange Activity.
**Note: ** will close the ticket when an issue is fixed for Android.

authors: Lss-Ankit jeanregisser

repo: celo-org/celo-monorepo


text:
When confirming recipient's account in Send flow, app should auto-focus to subsequent textbox when entering characters.

Expected Behavior
When a phone number is mapped to multiple accounts the sender must confirm the last four characters of the recipient's account. After typing a character into a textbox, app should autofocus to the next textbox.

[ ] Also, should replace the word "digits" with "characters".

Current Behavior
Sender must tap each of the 4 textboxes when inputting characters.

duplicate of #4116


text:
Users SBAT to see record of requests they made

Expected Behavior
After making a payment request, user should be able to see a confirmation of this, and be able to check if their requests have been completed or not.
Current Behavior
After a request has been made, no confirmation/record of this provided

dupe of #1170 , closing

authors: isabellewei nityas


text:
Merge Release 1 back into master

Description
Branch nambrot/master-into-release-1 represents a merge of master into release/celo-core-contracts/1. The reason why it is separate is to make obvious the merge conflict resolution that happened since we want a clean merge-commit (instead of the typical squash merge). Thus the recommended sequence is:

[x] Branch nambrot/master-into-release-1 from release/celo-core-contracts/1
[x] Merge master into nambrot/master-into-release-1 without resolving conflicts
[x] Resolve conflicts in follow-up commits to make clear how they were resolved
[ ] Open this PR to allow reviewers to review those merge resolutions
[ ] Upon approval of this PR, close it, update release/celo-core-contracts/1 to nambrot/master-into-release-1
[ ] Open another PR that represents the merge of release/celo-core-contracts/1 to master

Listed in https://github.com/celo-org/celo-monorepo/issues/4812
The relevant diff to review becomes https://github.com/celo-org/celo-monorepo/pull/5328/files/a675fd719f7fe5ddd676affd69b263f21508b85f..1907dc071c5bf0fe4aaf1e60653523a4bdae66ed (i.e. the commits after the merge commit)

Ok with the approval, I will now update release/celo-core-contracts/1 to nambrot/master-into-release-1


text:
add initial version of W3C tracestate

This is the initial implementation of W3C tracestate in Go.
This is my first time writing program in Go, couple notes/questions:

I want to discuss if want to put thread safety guard on Tracestate.Fork(), or leave it as a responsibility for the API consumer.
Currently the Tracestate object will be copied every time when a new span got created, I plan to implement COW (copy on write) to improve perf.
My next pull request will implement the ochttp integration, which will propagate tracestate in HTTP headers.

@rakyll @ramonza, would you give me some hints? I'm getting the following AppVeyor build error which claims gcc compiler missing:
runtime/cgo
exec: "gcc": executable file not found in %PATH%
FAIL	go.opencensus.io/exporter/jaeger [build failed]
FAIL	go.opencensus.io/exporter/prometheus [build failed]
FAIL	go.opencensus.io/exporter/stackdriver/propagation [build failed]
FAIL	go.opencensus.io/exporter/zipkin [build failed]
FAIL	go.opencensus.io/internal [build failed]
FAIL	go.opencensus.io/plugin/ocgrpc [build failed]
FAIL	go.opencensus.io/plugin/ochttp [build failed]
FAIL	go.opencensus.io/plugin/ochttp/propagation/b3 [build failed]
FAIL	go.opencensus.io/plugin/ochttp/propagation/tracecontext [build failed]
FAIL	go.opencensus.io/stats [build failed]
FAIL	go.opencensus.io/stats/view [build failed]
FAIL	go.opencensus.io/tag [build failed]
FAIL	go.opencensus.io/trace [build failed]
FAIL	go.opencensus.io/trace/propagation [build failed]
FAIL	go.opencensus.io/zpages [build failed]
Command exited with code 2

@ramonza, agree, that's what I've mentioned in the PR description "My next pull request will implement the ochttp integration, which will propagate tracestate in HTTP headers.", I will add the actual propagation and the test cases in the next PR.

I would prefer to have it in one PR because I don't think just this alone makes sense on its own -- it doesn't add any functionality and I'm worried that if users see it they will be confused as to why its there.

I am going to look into the AppVeyor failure, it is unrelated.

Close this as it is a dup with #890 which @rghetia is working on.

authors: ramonza reyang

repo: census-instrumentation/opencensus-go

license_source: bigquery


text:
Allow users to use custom display name prefix for Stackdriver

Equivalent to https://github.com/census-instrumentation/opencensus-go/pull/583.

Codecov Report

Merging #1057 into master will decrease coverage by <.01%.
The diff coverage is 50%.

@@             Coverage Diff              @@
##             master    #1057      +/-   ##
============================================
- Coverage      81.6%   81.59%   -0.01%
  Complexity     1140     1140
============================================
  Files           182      182
  Lines          5359     5362       +3
  Branches        501      502       +1
============================================
+ Hits           4373     4375       +2
  Misses          855      855
- Partials        131      132       +1

Impacted Files
Coverage Δ
Complexity Δ

...ats/stackdriver/StackdriverStatsConfiguration.java
100% <ø> (ø)
2 <0> (ø)
:arrow_down:

...rter/stats/stackdriver/StackdriverExportUtils.java
94.11% <100%> (ø)
23 <0> (ø)
:arrow_down:

...r/stats/stackdriver/StackdriverExporterWorker.java
68.51% <100%> (+0.29%)
13 <0> (ø)
:arrow_down:

...er/stats/stackdriver/StackdriverStatsExporter.java
64.38% <30%> (-0.41%)
11 <2> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d681cc...69b5d9a. Read the comment docs.

Holding off this change - we're going to discuss about this a little in tomorrow's meeting.

Closing for now - with our new Metrics exporting approach this should no longer be needed.

repo: census-instrumentation/opencensus-java


text:
Initial HTTP instrumentation commit (same as PR/1020)

this is same PR as https://github.com/census-instrumentation/opencensus-java/pull/1020
by HailongWen
The only difference is in  README.md file.

Work restored in https://github.com/census-instrumentation/opencensus-java/pull/1537

Codecov Report

Merging #1537 into master will increase coverage by 0.16%.
The diff coverage is 94.21%.

@@             Coverage Diff             @@
##             master   #1537      +/-   ##
===========================================
+ Coverage     82.94%   83.1%   +0.16%
- Complexity     1659    1708      +49
===========================================
  Files           251     257       +6
  Lines          7720    7841     +121
  Branches        754     763       +9
===========================================
+ Hits           6403    6516     +113
- Misses         1097    1100       +3
- Partials        220     225       +5

Impacted Files
Coverage Δ
Complexity Δ

...java/io/opencensus/contrib/http/HttpExtractor.java
0% <0%> (ø)
0 <0> (?)

.../io/opencensus/contrib/http/HttpServerHandler.java
100% <100%> (ø)
2 <2> (?)

...n/java/io/opencensus/contrib/http/HttpHandler.java
100% <100%> (ø)
5 <5> (?)

...io/opencensus/contrib/http/util/HttpTraceUtil.java
100% <100%> (ø)
33 <33> (?)

...io/opencensus/contrib/http/HttpSpanCustomizer.java
75% <75%> (ø)
6 <6> (?)

.../io/opencensus/contrib/http/HttpClientHandler.java
94.11% <94.11%> (ø)
3 <3> (?)

...census/implcore/trace/export/SpanExporterImpl.java
91.78% <0%> (-1.37%)
8% <0%> (ø)

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71dba4a...5aa0721. Read the comment docs.

Please fix the build.

I'll split the PR and resubmit.

authors: bogdandrutu codecov-io rghetia


text:
Trying to run on EC2 instance.

Hi, we are trying to run HarmBench on EC2 instance, and we are having issues in setting up slurm, can you share your slurm.conf file and some insights on how we could setup slurm.

Hi @ArmaanSeth , you can run a job using the .py on a local mode instead of slurm. What's your current cmd?

I am working on ubuntu and am trying to run for llama2-7b, can you tell me the gpu requirements to run the benchmarks. So I can use that as an reference.

Hi @ArmaanSeth , we have an example GCG notebook so that you can quickly test GCG on your side.
For the method, what objective are you trying to solve in choosing the method?

I am trying to run all the 3 steps, I have got the slurm working but each sbatch job is loading the llama model in memory resulting in CUDA out of memory error.

Hi @ArmaanSeth,
Our hyperparameters were set using 80GB A100 GPUs, so GPUs with less memory may require increased parallelism to load models. In configs/model_configs/models.yaml, you can set the number of GPUs required to load each model. The models are partitioned across GPUs using device_map='auto' in the load_model_and_tokenizer function.

authors: ArmaanSeth justinphan3110cais mmazeika

repo: centerforaisafety/HarmBench

license: MIT


text:
Netarkivet Cleaning v2

Added CLI for cleaning (sentence filtering + quality filtering) and deduplication. As its config is a bit large, I used hydra for config management (it sounds like most are comfortable with it).
Split netarkivet into two folder v1 and v2. Added application script to version 2 of netarkivet.
removed flake8 from pre-commit (probably better to add it in another pre-commit with the file changes).
bumped version id.
Note: you should ignore cleaning-v1 files (no changes, just moved) and dataset validation scripts (see #162).

Coverage ReportFileStmtsMissCoverMissingsrc/dfm   init.py10100% src/dfm/cleaning   init.py30100%    clean_cli.py1311310%20–320   dedupe_cli.py1401400%21–299   deduper.py1561690%121–126, 149, 277, 280, 313, 408, 423, 446, 551, 578–581   deduper_utils.py26388%58–61   quality_filter.py2543985%92–97, 279, 343, 348, 422, 432, 491–506, 587–593, 732, 765–768, 793, 825, 909, 923   sentence_filter.py781482%93, 145, 150, 175, 183, 188–202, 220src/dfm/data   init.py00100% src/dfm/description   init.py20100%    description_patterns.py483233%500, 541, 581, 764–770, 781–795, 804–821, 830–847   generate_description.py12120%1–40   match_counter.py520100% src/dfm/dfm_tokenizers   init.py20100%    tokenizer_config.py340100%    train_tokenizer.py67593%52, 120, 122, 162–163TOTAL100639261%

Coverage ReportFileStmtsMissCoverMissingsrc/dfm   init.py10100% src/dfm/cleaning   init.py30100%    clean_cli.py1311310%20–320   dedupe_cli.py1401400%21–299   deduper.py1561690%121–126, 149, 277, 280, 313, 408, 423, 446, 551, 578–581   deduper_utils.py26388%58–61   quality_filter.py2543985%92–97, 279, 343, 348, 422, 432, 491–506, 587–593, 732, 765–768, 793, 825, 909, 923   sentence_filter.py781482%93, 145, 150, 175, 183, 188–202, 220src/dfm/data   init.py00100% src/dfm/description   init.py20100%    description_patterns.py483233%500, 541, 581, 764–770, 781–795, 804–821, 830–847   generate_description.py12120%1–40   match_counter.py520100% src/dfm/dfm_tokenizers   init.py20100%    tokenizer_config.py340100%    train_tokenizer.py67593%52, 120, 122, 162–163TOTAL100639261%

@KennethEnevoldsen There are quite a lot of changed files here - which ones would you like reviewed? 🙂

@saattrupdan, with the merge of #162, it should be notably less. As noted

Note: you should ignore cleaning-v1 files (no changes, just moved) and dataset validation scripts (see https://github.com/centre-for-humanities-computing/danish-foundation-models/pull/162).

The primary files to focus on are the src/application/netarkivet/cleaning-v2/* and src/dfm/cleaning/*

@saattrupdan or @s1945pk do you have the time to review this sometime this week? Otherwise, I will ask Lasse to review it

@saattrupdan or @s1945pk do you have the time to review this sometime this week? Otherwise, I will ask Lasse to review it

Yep, I'll get around to it asap!

Coverage ReportFileStmtsMissCoverMissingsrc/dfm   init.py10100% src/dfm/cleaning   init.py30100%    clean_cli.py1321320%20–375   dedupe_cli.py1411410%21–337   deduper.py1561690%121–126, 149, 277, 280, 313, 408, 423, 446, 551, 578–581   deduper_utils.py26388%58–61   quality_filter.py2543985%92–97, 279, 343, 348, 422, 432, 491–506, 587–593, 732, 765–768, 793, 825, 909, 923   sentence_filter.py781482%93, 145, 150, 175, 183, 188–202, 220src/dfm/data   init.py00100%    load_datasets.py88880%5–344src/dfm/description   init.py20100%    description_patterns.py483233%500, 541, 581, 764–770, 781–795, 804–821, 830–847   generate_description.py12120%1–40   match_counter.py500100% src/dfm/dfm_tokenizers   init.py20100%    tokenizer_config.py340100%    train_tokenizer.py67593%52, 120, 122, 162–163TOTAL109448256%

It is fixed! Will set it to merge once the tests pass

Coverage ReportFileStmtsMissCoverMissingsrc/dfm   init.py10100% src/dfm/cleaning   init.py30100%    clean_cli.py1321320%20–375   dedupe_cli.py1411410%21–338   deduper.py1561690%121–126, 149, 277, 280, 313, 408, 423, 446, 551, 578–581   deduper_utils.py26388%58–61   quality_filter.py2543985%92–97, 279, 343, 348, 422, 432, 491–506, 587–593, 732, 765–768, 793, 825, 909, 923   sentence_filter.py781482%93, 145, 150, 175, 183, 188–202, 220src/dfm/data   init.py00100%    load_datasets.py88880%5–344src/dfm/description   init.py20100%    description_patterns.py483233%500, 541, 581, 764–770, 781–795, 804–821, 830–847   generate_description.py12120%1–40   match_counter.py500100% src/dfm/dfm_tokenizers   init.py20100%    tokenizer_config.py340100%    train_tokenizer.py67593%52, 120, 122, 162–163TOTAL109448256%

authors: KennethEnevoldsen saattrupdan


text:
Connect/disconnect events

Hi,
Is it possible to track connect/disconnect (subscribe/unsubscribe) events on server side? For example, user connected to websocket/subscribed to channed and the 'connect' event will be emitted on server side. The same thing with disconnect.
Socket.io has this feature:
...
io.on('connection', function (socket) {
  socket.emit('news', { hello: 'world' });
  socket.on('my other event', function (data) {
    console.log(data);
  });
});

I understand that Centrifugo runs in separate process with Go, but maybe it is possible with API callbacks and so on. I need this feature to track when user is online in my application. It will be awesome to have this feature.
Thanks

@nanom1t hello, no - this is not possible at moment in a way you want as Centrifugo runs as separate service and though introducing some sort of webhooks is theoretically possible - I don't think it's a good idea - when you need such a tight integration with your service it's better to use sth integrated in code.
What is your concrete use case for this btw?

@FZambia Thanks for your response.
I'm working on Node.js application, which is similar to small social network (not very large for now) and I'm already using Centrifugo in my project.
Each user have own profile in my app. I just want to show when user is online or show last login time on his profile. Now user subscribes for his personal channel in Centrifugo and receives messages from server during all session time.
So I thought that it will be awesome if I can listen to connect (disconnect) events to channel and change user status at that time (for example, from offline to online).
I'm not sure that it is the best way to that, but I think it will work for me.
Thanks

Showing when user is online looks like a presence request to his personal channel. You can also update last login time sending AJAX request to your app manually when user opens a page - without Centrifugo at all. And periodically update that last active status - also without Centrifugo.
I don't see a clean way how having connect/disconnect events can help you a lot with online status. There can be races (fast reconnect) that must be handled somehow. Also there can be massive reconnects that can result in self ddos. And many other problems I suppose.

repo: centrifugal/jscent

license: mit


text:
ceph-nfs: bind mount ganesha log directory

The current ganesha log directory is only present in the container
and not bind mount on the host.
Signed-off-by: Dimitri Savineau dsavinea@redhat.com

jenkins test ceph-container-all_daemons

jenkins test ceph-container-purge

jenkins test ceph-container-purge

jenkins test centos-container-all_daemons

jenkins test centos-container-purge

jenkins test centos-container-update

repo: ceph/ceph-ansible


text:
Tracker issue for intree rbd to ceph csi online migration

Describe the bug
This issue track the few changes required to support in tree rbd migration to ceph csi ( Project # https://github.com/ceph/ceph-csi/projects/18) , this allows an in tree rbd user to do  online migration without changing any existing templates. This feature is tracked in kubernetes for v1.23 as alpha under https://github.com/kubernetes/enhancements/pull/2963.
As discussed in CSI calls, currently identified changes are:

[ ] get clusterid from mon
[ ] use a volume handle passed in and support static volume mounting, unmounting and delete operations on existing volumes.

The volume handle would include the mon, pool and image which could be used for above operations.
Additional Context:
https://github.com/ceph/ceph-csi/issues/1725

@humblec any specific reason not to use/suggest the CSI migration tool which is getting developed by @Yuggupta27 and @subhamkrai?

get clusterid from mon
use a volume handle passed in and support static volume mounting, unmounting, and delete operations on existing volumes.

IMO This is an extra burden to the cephcsi supportability point of view
can you please provide more details with examples of why we are taking this approach?

@humblec any specific reason not to use/suggest the CSI migration tool which is getting developed by @Yuggupta27 and @subhamkrai?

@Madhu-1 For offline migration ( taking down the app and untangling from pvc and attaching old pv after renaming to new pvc)  , the  tool can also be used,   as mentioned in the issue, this is an online migration for existing in tree users, the users will be capable to continue using the templates as they do today and everything will be taken care by the kubernetes, they dont want to change any existing templates like SC, PVC...etc . They will be unaware that, the migration is happening ...etc, thats why the migration framework in kubernetes (https://github.com/kubernetes/kubernetes/pull/95361) becomes handy.  Most of the kube intree drivers have been migrated or more are in migration path.
At any point there want to do offline migration , they could make use of the tool as well..  so users are left with good amount of choices and have options for both online and offline migration and make use of Ceph CSI. !!

get clusterid from mon
use a volume handle passed in and support static volume mounting, unmounting, and delete operations on existing volumes.

IMO This is an extra burden to the cephcsi supportability point of view

I dont think so,  these helpers will be ONLY called for migrated volumes and while csi driver process those , that too with very limited usage like mount , unmount ..etc.  Its is same as we currently support static volumes today.  None of the existing functionality would be broken with this change , so it shouldnt be a burden. Also snapshot , cloning ..etc are not supported on these static kind of volumes.

can you please provide more details with examples of why we are taking this approach?

The intree driver has monitor field available with it, so our configmap can be read to get the clusterid if its a migration request.  Almost like we do read clusterid mapping for replication cases...etc.
Once the clusterID is available , rest of the information is available for continuing to support it as static volume.
the volume handle would be same as our static volume case but just got a couple of strings append to it which carry the monitor and pool in it , very bare minimum stuff without any fancy operations on it.

I dont think so, these helpers will be ONLY called for migrated volumes and while csi driver process those , that too with very limited usage like mount , unmount ..etc. Its is same as we currently support static volumes today. None of the existing functionality would be broken with this change , so it shouldnt be a burden. Also snapshot , cloning ..etc are not supported on these static kind of volumes.

yes if we go for offline tools we can support all these features.

I dont think so, these helpers will be ONLY called for migrated volumes and while csi driver process those , that too with very limited usage like mount , unmount ..etc. Its is same as we currently support static volumes today. None of the existing functionality would be broken with this change , so it shouldnt be a burden. Also snapshot , cloning ..etc are not supported on these static kind of volumes.

we don't support DeleteVolume for the static PVC but you have added it to the list. as I already mentioned in https://github.com/ceph/ceph-csi/issues/1725#issuecomment-823225160 not sure it's worth supporting this one with extra effort.
If we really need/want to support this one please  make sure we support only the ceph version mentioned in the readme

I dont think so, these helpers will be ONLY called for migrated volumes and while csi driver process those , that too with very limited usage like mount , unmount ..etc. Its is same as we currently support static volumes today. None of the existing functionality would be broken with this change , so it shouldnt be a burden. Also snapshot , cloning ..etc are not supported on these static kind of volumes.

yes if we go for offline tools we can support all these features.

Yep, iow, at any point in time ( even after online migration has been enabled) , if users want to go with offline migration (now or  later or even only for few volumes) , they can take that route based on the production downtime and feasibility of making other changes for the  consumers of current volumes and its templates.
Online can be an intermediate path as well for many deployments.

I dont think so, these helpers will be ONLY called for migrated volumes and while csi driver process those , that too with very limited usage like mount , unmount ..etc. Its is same as we currently support static volumes today. None of the existing functionality would be broken with this change , so it shouldnt be a burden. Also snapshot , cloning ..etc are not supported on these static kind of volumes.

we don't support DeleteVolume for the static PVC but you have added it to the list. as I already mentioned in #1725 (comment) not sure it's worth supporting this one with extra effort.

As this delete will be only for "migration volumes" it will be a very shim helper function and can be done with it.

If we really need/want to support this one please make sure we support only the ceph version mentioned in the readme

Sure @Madhu-1 , thats the plan :+1:

@humblec I have added one more todo to Remove E2E hacks and do valid migration testing with Kubernetes 1.23 as its released

repo: ceph/ceph-csi


text:
Bulk delete of CSI based rbd volumes through project delete leaves behind some undeleted pvs

Description of problem:
Bulk delete of CSI based rbd volumes leaves behind some undeleted pvs.
Created a project with ~120 app pods, each bound with one csi-rbd volume. To initiate bulk delete, deleted the whole project. Even though the pods and PersistentVolumeClaims are deleted, ~17 PersistentVolumes still stayed undeleted and in "Released" state. The volumes still existed on the ceph backend as well.
Snip of the error message seen in the "csi-rbdplugin-provisioner-0/csi-provisioner log:
I0531 11:42:34.935223       1 controller.go:1097] delete "pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2": started

I0531 11:42:36.012411       1 controller.go:192] GRPC error: rpc error: code = Internal desc = rbd pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2 is still being used
E0531 11:42:36.012435       1 controller.go:1120] delete "pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2": volume deletion failed: rpc error: code = Internal desc = rbd pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2 is still being used

-----
-----

I0531 12:31:02.507663       1 controller.go:192] GRPC error: rpc error: code = Internal desc = rbd pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2 is still being used
E0531 12:31:02.507692       1 controller.go:1120] delete "pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2": volume deletion failed: rpc error: code = Internal desc = rbd pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2 is still being used
W0531 12:31:02.507750       1 controller.go:726] Retrying syncing volume "pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2" because failures 8 < threshold 15
E0531 12:31:02.507793       1 controller.go:741] error syncing volume "pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2": rpc error: code = Internal desc = rbd pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2 is still being used
I0531 12:31:02.508404       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2", UID:"b1b9f608-8397-11e9-96cc-062db78a205c", APIVersion:"v1", ResourceVersion:"591678", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = Internal desc = rbd pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2 is still being used

PV list of un-deleted volumes

oc get pv|grep cirros
pvc-423aeea5-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc001                csi-rbd1                              110m
pvc-4ccfcb4a-8397-11e9-96cc-062db78a205c   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc006                csi-rbd1                              110m
pvc-50abd71c-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc008                csi-rbd1                              110m
pvc-5c46b88d-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc015                csi-rbd1                              109m
pvc-5dbe8f50-8397-11e9-96cc-062db78a205c   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc016                csi-rbd1                              109m
pvc-6584e985-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc021                csi-rbd1                              109m
pvc-6a734cb8-8397-11e9-96cc-062db78a205c   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc024                csi-rbd1                              109m
pvc-6c7fac9c-8397-11e9-96cc-062db78a205c   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc025                csi-rbd1                              109m
pvc-72e7ad64-8397-11e9-96cc-062db78a205c   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc028                csi-rbd1                              109m
pvc-8fa59e88-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc045                csi-rbd1                              108m
pvc-946a0f61-8397-11e9-9704-0ae6b1651af2   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc048                csi-rbd1                              108m
pvc-9fbbe61e-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc055                csi-rbd1                              107m
pvc-a3487c70-8397-11e9-9704-0ae6b1651af2   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc057                csi-rbd1                              107m
pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc066                csi-rbd1                              107m
pvc-bc316634-8397-11e9-9704-0ae6b1651af2   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc073                csi-rbd1                              107m
pvc-c2c24fe6-8397-11e9-96cc-062db78a205c   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc076                csi-rbd1                              106m
pvc-df7ec39f-8397-11e9-983e-028dfec6f770   2Gi        RWO            Delete           Released   cirros/claimrbd-pvc094                csi-rbd1                              106m

sh-4.2 rbd ls -p rbd
pvc-423aeea5-8397-11e9-983e-028dfec6f770
pvc-4ccfcb4a-8397-11e9-96cc-062db78a205c
pvc-50abd71c-8397-11e9-983e-028dfec6f770
pvc-5c46b88d-8397-11e9-983e-028dfec6f770
pvc-5dbe8f50-8397-11e9-96cc-062db78a205c
pvc-6584e985-8397-11e9-983e-028dfec6f770
pvc-6a734cb8-8397-11e9-96cc-062db78a205c
pvc-6c7fac9c-8397-11e9-96cc-062db78a205c
pvc-72e7ad64-8397-11e9-96cc-062db78a205c
pvc-8fa59e88-8397-11e9-983e-028dfec6f770
pvc-946a0f61-8397-11e9-9704-0ae6b1651af2
pvc-9fbbe61e-8397-11e9-983e-028dfec6f770
pvc-a3487c70-8397-11e9-9704-0ae6b1651af2
pvc-b14bbc93-8397-11e9-9704-0ae6b1651af2
pvc-bc316634-8397-11e9-9704-0ae6b1651af2
pvc-c2c24fe6-8397-11e9-96cc-062db78a205c
pvc-df7ec39f-8397-11e9-983e-028dfec6f770

Version-Release number of selected component (if applicable):
Ceph versions from toolbox pod
-----------------------------------

sh-4.2 ceph versions
{
    "mon": {
        "ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)": 6
    },
    "mds": {},
    "overall": {
        "ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)": 10
    }
}

Ceph pod image
----------------
[nberry@localhost logs]$ oc describe pod rook-ceph-mon-a-69466447ff-kc272|grep -i image
    Image:         ceph/ceph:v14.2.0-20190410
    Image ID:      docker.io/ceph/ceph@sha256:212804b78633f61997440aa37fec87e144081ed60a5ee918c35070548bfc8c22
      CONTAINER_IMAGE:                ceph/ceph:v14.2.0-20190410
    Image:         ceph/ceph:v14.2.0-20190410
    Image ID:      docker.io/ceph/ceph@sha256:212804b78633f61997440aa37fec87e144081ed60a5ee918c35070548bfc8c22
      CONTAINER_IMAGE:                ceph/ceph:v14.2.0-20190410

CSI pod image
----------------

[nberry@localhost logs]$ oc describe pod csi-rbdplugin-8cphn|grep -i image
    Image:         quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
    Image ID:      quay.io/k8scsi/csi-node-driver-registrar@sha256:ffecfbe6ae9f446e5102cbf2c73041d63ccf44bcfd72e2f2a62174a3a185eb69
    Image:         quay.io/cephcsi/rbdplugin:v1.0.0
    Image ID:      quay.io/cephcsi/rbdplugin@sha256:e04fa3d1d210ef07f3287d58362067b17b313a0b5e46ce0879b3142df484a12b
[nberry@localhost logs]$ oc describe pod rook-ceph-tools-76c7d559b6-d92mr|grep -i image
    Image:         rook/ceph:master
    Image ID:      docker.io/rook/ceph@sha256:11aca02d842184bfad5a405ee0c0446f9d2a3b5b4418b4ede0a3211f20d05aa4

Rook version ( from github, master branch)
---------------
[nberry@localhost logs]$ oc describe pod rook-ceph-tools-76c7d559b6-d92mr|grep -i image
    Image:         rook/ceph:master
    Image ID:      docker.io/rook/ceph@sha256:11aca02d842184bfad5a405ee0c0446f9d2a3b5b4418b4ede0a3211f20d05aa4

oc version
------------

$ export KUBECONFIG=/home/nberry/aws-install/may31/auth/kubeconfig
(ocs-ci) [nberry@localhost ocs-ci]$ oc version
Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0-201905091432+4910781-dirty", GitCommit:"4910781", GitTreeState:"dirty", BuildDate:"2019-05-09T19:19:42Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+f2cc675", GitCommit:"f2cc675", GitTreeState:"clean", BuildDate:"2019-05-18T00:35:43Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.1.0-rc.5   True        False         138m    Cluster version is 4.1.0-rc.5

How reproducible:
Attempted once. The volumes were not yet deleted even after ~50 mins (last checked).
Steps to Reproduce:

Created a CSI based rook-ceph setup using ocs-ci

Create secret, storageclass using both "admin" and "kubernetes" user. Yamls are attached for reference in next comment.

Create a new-project (here cirros)
oc new-project cirros
Now using project "cirros" on server "https://api.vavuthu-ocs.qe.rh-ocs.com:6443".

Run a script to create pod + pvc in a new-project. Copied the script for volume & pod creation in next comment.

for i in {001..100}; do sh cirros-create-with-pvc-with-io.sh rbd-pvc$i 2; done

Created 20 more volumes.

Initiated PVC delete by deleting the whole project

oc delete project cirros

This initiated pod deletion followed by PVC/PV deletion.

Checked the PV list and ceph backend. Atleast 17 PVs were not yet deleted and there were delete failures in CSI logs

Some data for analysis:

First delete  request sent at (from logs) ~ I0531 11:42:10.62865
Out of 120 volumes, only 17 were left behind.

Actual results:
Some PVs failed to get deleted from OC side and the ceph backend.
They still listed under OC as "Released" and in ceph backend under: (pool name = rbd)
rd ls -p rbd
Expected results:
All the POds and volumes need to get removed once the hosting project(namespace) is deleted permanently.

Logs attached
issue-399.zip

@nehaberry current RBD version used  is quay.io/cephcsi/rbdplugin:v1.0.0 please use quay.io/cephcsi/cephcsi:canary

@Madhu-1 we are using the rook image from upstream master.  Do we need to change the image names for these params in operator-openshift-with-csi.yaml
   - name: ROOK_CSI_CEPHFS_IMAGE
    - name: ROOK_CSI_RBD_IMAGE
    - name: ROOK_CSI_REGISTRAR_IMAGE
    - name: ROOK_CSI_PROVISIONER_IMAGE
    - name: ROOK_CSI_SNAPSHOTTER_IMAGE
    - name: ROOK_CSI_ATTACHER_IMAGE

Please let me know if I have missed anything.
Correct ... Current images seem to be this:

name: ROOK_CSI_CEPHFS_IMAGE
value: "quay.io/cephcsi/cephfsplugin:v1.0.0"
- name: ROOK_CSI_ENABLE_RBD
value: "true"
- name: ROOK_CSI_RBD_IMAGE
value: "quay.io/cephcsi/rbdplugin:v1.0.0"
- name: ROOK_CSI_REGISTRAR_IMAGE
value: "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
- name: ROOK_CSI_PROVISIONER_IMAGE
value: "quay.io/k8scsi/csi-provisioner:v1.0.1"
- name: ROOK_CSI_SNAPSHOTTER_IMAGE
value: "quay.io/k8scsi/csi-snapshotter:v1.0.1"
- name: ROOK_CSI_ATTACHER_IMAGE
value: "quay.io/k8scsi/csi-attacher:v1.0.1"

using canary image needs more changes in rook. we are going to update rook once we release  ceph-csi v1.1.0
CC @humblec

@Madhu-1 when will ceph-csi v1.1.0 be available?

@hnallurv https://github.com/ceph/ceph-csi/issues/393

@hnallurv its actually https://github.com/ceph/ceph-csi/issues/353 and as per our schedule its going to land on Jun 7th. .  https://github.com/ceph/ceph-csi/issues/393 is for next minor release.

@Madhu-1 so do you suggest that temporarily changing the image version to quay.io/cephcsi/cephcsi:canary for all the below variables be helpful to get past this bug ? I understand that it is WIP for the canary build as well

name: ROOK_CSI_CEPHFS_IMAGE
- name: ROOK_CSI_RBD_IMAGE
- name: ROOK_CSI_REGISTRAR_IMAGE
- name: ROOK_CSI_PROVISIONER_IMAGE
- name: ROOK_CSI_SNAPSHOTTER_IMAGE
- name: ROOK_CSI_ATTACHER_IMAGE

@Madhu-1 so do you suggest that temporarily changing the image version to quay.io/cephcsi/cephcsi:canary for all the below variables be helpful to get past this bug ? I understand that it is WIP for the canary build as well

name: ROOK_CSI_CEPHFS_IMAGE

name: ROOK_CSI_RBD_IMAGE
name: ROOK_CSI_REGISTRAR_IMAGE
name: ROOK_CSI_PROVISIONER_IMAGE
name: ROOK_CSI_SNAPSHOTTER_IMAGE
name: ROOK_CSI_ATTACHER_IMAGE

@nehaberry need to do some change in rook code itself changing template won't fix the issue.

@Madhu-1 ok. Please let us know once the code changes are made. Thanks :)

@nehaberry are you still seeing this issue?

@nehaberry are you still seeing this issue?

Hi @Madhu-1 yeah last when I tested lst week, some PVs were still being left behind. The corresponding bugzilla was updated.
https://bugzilla.redhat.com/show_bug.cgi?id=1716276

assigning to @ShyamsundarR as he is looking into it.

https://github.com/ceph/ceph-csi/pull/510 should fix this issue.

authors: Madhu-1 ShyamsundarR hnallurv humblec nehaberry


text:
deploy: add API for rbd/CSIDriver and rbd/ceph-csi-config ConfigMap

This is similar to #2551 when OpenShift SecurityContextConstraints are exposed over an API.
The rbd/CSIDriver  and rbd/ceph-csi-config ConfigMap are deployment artifacts that should be made available when Ceph-CSI is deployed. Automated deployers (like Rook) can consume this API and remove any redundant maintained deployment files in the future.
Updates: #2542

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)
/retest all: run this in case the CentOS CI failed to start/report any test
progress or results

/retest ci/centos/mini-e2e-helm/k8s-1.21

@mergifyio rebase


text:
e2e: introduce ResourceDeployer interface

Add and use a simple ResourceDeployer interface for deploying YAML
artifacts in the e2e suite. This reduces the duplication of reading files,
replacing the namespace and other operations.
Adding new resources is now simple for each deployment, preventing
gocyclo complexity warnings in the future. (The NFS-Provisioner needs
to deploy more artifacts, causing warnings.)

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)
/retest all: run this in case the CentOS CI failed to start/report any test
progress or results

/retest ci/centos/mini-e2e-helm/k8s-1.22

/retest ci/centos/mini-e2e-helm/k8s-1.22

Ceph cluster did not become ready in time.

/retest ci/centos/mini-e2e-helm/k8s-1.23

/retest ci/centos/mini-e2e-helm/k8s-1.23

Failed with:
Apr 15 16:44:13.378: backend images not matching kubernetes resource count,image count 22 kubernetes resource count 21

/retest ci/centos/mini-e2e/k8s-1.21

@mergifyio rebase


text:
doc: add additional provisioner name

this commit adds another provisioner name, which is deployed via rook.

we dont need this comment, as this examples are specific to standalone cephcsi deployment

Thanks for the review Madhu, closing the PR in that case.


text:
helm: add cli argument instanceid

Describe what this PR does
This PR adds the existing cli argument --instanceid as an configurable helm value  instanceID.
This allows configuring the instance id parameter of Ceph CSI without manually changing the deployment or helm chart.
Is there anything that requires special attention
Is the change backward compatible?
Yes
Are there concerns around backward compatibility?
No, the default values in code have not been changed but are now overrideable via helm and cli arguments.
Checklist:

[X] Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
[X] Reviewed the developer guide on Submitting a Pull
Request
[X] Pending release
notes
updated with breaking and/or notable changes for the next major release.
[X] Documentation has been updated, if necessary.
[X] Unit tests have been added, if necessary.
[X] Integration tests have been added, if necessary.

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

Please take a look. There's a lint failure https://github.com/ceph/ceph-csi/actions/runs/9388846826/job/25883583707?pr=4666
a new live is required in both values.yaml

The new line at the end of the files values.yaml should be added now.

Can you please mention the new config in docs https://github.com/ceph/ceph-csi/tree/devel/charts/ceph-csi-rbd#configuration similar for cephfs Sorry for not noticing this earlier.

Please also add a point in PendingReleaseNotes.
https://github.com/ceph/ceph-csi/blob/devel/PendingReleaseNotes.md

@zerotens can you please check CI failures?

@mergifyio queue

Unable to update: user zerotens is unknown.
Please make sure zerotens has logged in Mergify dashboard.

Done.

@Mergifyio refresh

@Mergifyio queue

@mergifyio rebase

@mergifyio queue

authors: Madhu-1 Rakshith-R iPraveenParihar zerotens


text:
Validate filter value

Hi all, Congrats for the project!
Is there a way to validate filter values?
For example, I want to filter by name, but only if the name value is a string with more than 3 characters.
Is there a standard way to do this?
I couldn't find any documentation about this.

I haven't tried, but I assume you could just raise an error inside the filter callback. See http://jsonapi-resources.com/v0.9/guide/resources.html#Applying-Filters for how to implement a custom filter.

Thank you @scottgonzalez ! I have already looked at that doc.
What I was doing is something like:
      filter :name, verify: ->(values, context) {
        raise JSONAPI::Exceptions::InvalidFilterValue.new(:name, values) if ...
      }

But I'm wondering if there is a standard way to validate the filter values, and return an error with all the "wrong" filters, something like:
{
    "errors": [
        {
            "title": "Invalid filter value",
            "detail": "[\"10\"] is not a valid value for name.",
            "code": "107",
            "status": "400"
        },
        {
            "title": "Invalid filter value",
            "detail": "[\"10\"] is not a valid value for last_name.",
            "code": "107",
            "status": "400"
        }
    ]
}

BTW: It would be nice if we could something like:
filter :name, validate: { lenght: { min: 3, max: 20 } }

JR does not support filters validation, but you can implement it yourself by overriding verify_filter.
something like code below can do the trick:
class JSONAPI::Validators::LengthValidator
  def initialize(filter_name, values, options)
    @filter_name = filter_name
    @values = values
    @options = options
  end

  def valid?
    return false if !options[:allow_empty] && values.empty?
    values.each do |value|
      return false if options[:min] && value.length < options[:min]
      return false if options[:max] && value.length > options[:max]
    end
    true
  end
end

class ApplicationResource < JSONAPI::Resource
  def self.verify_filter(filter, raw, context = nil)
    result = super
    options = _allowed_filters.fetch(filter, {})
    options.fetch(:validate, {}).each do |validator, opts|
      klass = "JSONAPI::Validators::#{validator.to_s.classify}Validator".constantize
      unless klass.new(result[0], result[1], opts).valild?
        raise JSONAPI::Exceptions::InvalidFilterValue.new(filter, raw)
      end
    end
  end
end

class UserResource < ApplicationResource
  attribute :name
  filter :name, validate: { length: { min: 3, max: 20 } }
end

Thank you @senid231! That looks good. I'm just using verify for now. So I'll close this issue

authors: edipox scottgonzalez senid231

repo: cerebris/jsonapi-resources


text:
Uplift Form Input Styles

Issue Description
Uplift styles for the Form Input component.
Issue Type

[ ] New Feature
[X] Enhancement
[ ] Bug
[ ] Other

Closing in favor of tracking this on #1207

authors: benbcai bjankord

repo: cerner/terra-core


text:
ActionHeader Port from clinical

Summary
There exists an action header in the terra-clinical repo already. We would like to push this into terra-core and deprecate the clinical package. To make upgrading easiest, we would like to try to maintain the api for passivity.
Existing Props

Prop
Type
Required
Default
Description

title
string
optional
undefined
Text to be displayed as the title in the header bar

level
number
optional
1
Sets the heading level. One of 1, 2, 3, 4, 5, 6. (ADDED)

onClose
func
optional
undefined
Callback function for when the close button is clicked. The back button will not display if this is not set. On small viewports a back button will be displayed instead of a close button when a separate onBack callback is not set.

onBack
func
optional
undefined
Callback function for when the back button is clicked. The back button will not display if this is not set.

onMaximize
func
optional
undefined
Callback function for when the expand button is clicked. The expand button will not display if this is not set or on small viewports. Only the expand button will be rendered if onMaximize and onMinimize are set.

onMinimize
func
optional
undefined
Callback function for when the minimize button is clicked. The minimize button will not display if this is not set or on small viewports. Only the expand button will be rendered if both onMaximize and onMinimize are set.

onNext
func
optional
undefined
Callback function for when the next button is clicked. The previous-next button group will not display if neither this or onPrevious are set.

onPrevious
func
optional
undefined
Callback function for when the previous button is clicked. The previous-next button group will not display if neither this or onNext are set.

keepCloseButton
bool
optional
undefined
A Boolean indicating if close button should be displayed on small viewports when separate onBack callback is not set.

children
element
optional
undefined
Child element to be displayed on the right end of the header. This is intended to be used with the CollapsibleMenuView.

Changes

Currently the back button, and the expand/collapse buttons appear to be mutually exclusive as they share the same spot and change based off screen size https://github.com/cerner/terra-clinical/blob/master/packages/terra-clinical-action-header/src/ActionHeader.jsx#L129 . They will no longer be mutually exclusive as the back button, and expand/collapse will now be in 2 different sockets.
The existing action header uses functions specific to each special button (maximize, minimize, next, previous) to determine if they should be shown. These props will be kept for passivity, but it will be documented that they will be removed in the next major release, and instead the consumer should be passing them into their collapsible menu as children.
Button styles have been changed to utility
@mjhenkes  , we discussed with @neilpfeiffer, and he thought we should put the buttons from legacy function props in their same existing position (between the back button and title) for now.

How to deprecate the props
Seen here, https://discuss.reactjs.org/t/proposal-add-isdeprecated-to-react-proptypes/4037 something like this would work
import React from 'react';

class Component extends React.Component {
  render() {
    const {oldProp, newProp} = this.props;
    if (process.env.NODE_ENV !== 'production' && typeof oldProp !== 'undefined') {
      console.warn('oldProp is deprecated, use newProp instead');
    }
    ...
  }
}

React chose not to add an isDeprecated function for prop-types https://github.com/facebook/react/pull/6357 . Dan Abramov did make a package for this https://github.com/Aweary/react-is-deprecated but its not worth pulling in
Questions

Do the buttons' icon need to be themable? I'm assuming atleast some of them do since I see a vertical ... and a horizontal version in different spots.
Digging in more now, I see ActionHeader has a dependency on the terra-clinical-header. Is this also a component we want ported over directly (I don't see anything about it in callouts)? I assume we don't want a dependency from core -> clinical. We could probably get most of the functionality we need utilizing arrange, heading, and truncating the text.

I think we should decouple action-header from the header component, I'm guessing there are going to be theming differences between the two that require them to be decoupled.

Just to clarify, we are NOT planning to create a core version of terra-clinical-header, but rather integrate it as part of the action-header component itself. Is this correcxt?

@alex-bezek - I updated the props table above to include a new item level since consumers will need to be able to change up which-ever h1/2/3... level they need for correct screenreader accessibility expectations based on nesting order.
It is based on these comments realized for section-header: https://github.com/cerner/terra-core/pull/1313#issuecomment-370591792 and https://github.com/cerner/terra-core/pull/1313#discussion_r172290552

@ChaseSinclair We are not planning on creating a core version of terra-clinical-header. Anything terra-clinical-header provided terra-clincal-action-header should be coded directly into terra-action-header.

Do the buttons' icon need to be themable?

Yeah, let's assume so and make the icons like the back, close, more, prev, next, etc icons themable.
+1 on tech design.

+1 on the tech design

+1 on tech design

authors: ChaseSinclair alex-bezek bjankord mjhenkes neilpfeiffer


text:
Update eslint config terra dependency

Summary
Updated the eslint-config-terra dependency from v^3.0.0 to @cerner/eslint-config-terra v^4.0.0, along with the associated config. Additionally updated multiple files to fit eslint rules and to ensure that the lint tests pass.
Closes #2999
Thank you for contributing to Terra.
@cerner/terra

@DMcginn sorry 😅

authors: DMcginn emilyrohrbough


text:
Two certificates map to same kubernetes sercet, end up making so many certificate requests one by one

In test environment we created a two certificates, which mapped to same volume on app. Initially both certificates had same parameters. I think because of that following issue didn't happen.
Later we change key of one certificate.  That trigger some race condition in cert manager. Because of that it issued new certificates request one by one to certificate authority for those two certificates.  Over the weekend it end up making thousands of new certificate requests.
It seems to me cert manager should have handled this issue gracefully.

Either by identifying that two certs are mapped to same volume.
Or, it should not have made new cert request before renewal time, in our case it was large. Which could have avoided this issue.
Please let  me know if you need more info on this.

Expected behaviour:

Steps to reproduce the bug:

Anything else we need to know?:
Environment details::

Kubernetes version:
Cloud-provider/provisioner:
cert-manager version:
Install method: e.g. helm/static manifests

/kind bug

Hi, thanks for describing the issue.
This is a duplicate of https://github.com/cert-manager/cert-manager/issues/4846 so I'm going to close this issue to keep the discussion in one place.
There is a fix coming in https://github.com/cert-manager/cert-manager/pull/5567

repo: cert-manager/cert-manager


text:
cert-manager-webhook to provide logs when handling request

Pull Request Motivation
Add log message (with Info log level) in conversion, validation and mutation web hook handlers, to facilitate troubleshooting for people willing to verify if requests are received or not.
Add test to verify log message for conversion and validation.
Related issue: https://github.com/cert-manager/cert-manager/issues/5197
Kind
/kind feature
Release Note
Webhook now logs requests to mutating/validating webhook (with `--v=5` flag)

/assign @irbekrm

/ok-to-test

Thanks for making the changes @tobotg
This now looks good to me, output with --v=5 flag to webhook:
I0424 15:09:54.082206       1 server.go:299] cert-manager/webhook "msg"="request received by mutating webhook" "allowed"=true "kind"="Certificate" "name"="foo" "namespace"="default" "response uuid"="687ed146-c06b-4b38-a8c9-5742b4394ba7"
I0424 15:09:54.082320       1 server.go:299] cert-manager/webhook "msg"="request received by validating webhook" "allowed"=true "kind"="Certificate" "name"="foo" "namespace"="default" "response uuid"="81a6adde-a289-4784-b7a0-c2a5c8c3cda7"
I0424 15:09:54.084468       1 server.go:299] cert-manager/webhook "msg"="request received by validating webhook" "allowed"=true "kind"="Certificate" "name"="foo" "namespace"="default" "response uuid"="9ed7b6c9-a54b-4d3d-a887-b39443849650"
I0424 15:09:54.087962       1 server.go:299] cert-manager/webhook "msg"="request received by mutating webhook" "allowed"=true "kind"="Certificate" "name"="foo" "namespace"="default" "response uuid"="011086dc-7870-4e12-a379-0530e99df548"
I0424 15:09:54.093317       1 server.go:299] cert-manager/webhook "msg"="request received by mutating webhook" "allowed"=true "kind"="Certificate" "name"="foo" "namespace"="default" "response uuid"="4994eff3-9d75-4fa2-acbf-fa0f04c233aa"
I0424 15:09:54.096092       1 server.go:299] cert-manager/webhook "msg"="request received by validating webhook" "allowed"=true "kind"="Certificate" "name"="foo" "namespace"="default" "response uuid"="56303f33-3260-4516-9d5f-f49c0df35c1a"

I think this will be helpful for folks debugging cert-manager webhook.

I have also updated the release note as it's a user facing change

authors: irbekrm tobotg


text:
Improve smart contracts based on auditing pre-report

Improved smart contracts based on feedback from auditing.
Tested by deploying on local environment following debugging instruction in README

fail cases are from the solidity tests


text:
Code change for #75 - Parameterize the NotBefore validity of token for NTP sync issues more than 1 sec

This change is

Submitting new PR with single clean commit for now -10 change


text:
Add support to parse picklist values

When generating documentation for custom objects, include pick list values in documentation
TODO:

[x] Ensure documentation generator to include pick list values

@cesarParra  not sure if you have an integration test to verify but believe the changes are completed. Let me know

@cesarParra not sure if you have an integration test to verify but believe the changes are completed. Let me know

@RaviVaranasi Yeah, the CI tests will run once I approve.

@cesarParra Unsure of the last commit but should be ready for review again

authors: RaviVaranasi cesarParra

repo: cesarParra/apexdocs


text:
🛑 echo-server.madacluster.tech is down

In 4077dbb, echo-server.madacluster.tech (https://echo-server.madacluster.tech) was down:

HTTP code: 0
Response time: 0 ms

Resolved: echo-server.madacluster.tech is back up in 6ad4ffe after 3 minutes.

repo: cesarempathy/upptime


text:
Release version 3.3.0

[x] Admins to redirect URL to 'service unavailable page' (if option is available)
[x] Admins to backup production database, then run database update script (if required)

Release checklist: https://docs.google.com/spreadsheets/d/1KzLWKT7ZIPoQfBnuNYmqyex_fsASqASp3MUCLVp1ix8/edit#gid=849693722
Release report: https://docs.google.com/document/d/1xuvUC954UbGkOOCi_uLQ0M_qlFKe1lbmpBcR0pXjiGw/edit

Version 3.0.0 ready for release.

Content freeze email sent out to editors on 7 November:
"Dear CVS Editors,
as of today, please do not use the CVS Editor since a new version of the CESSDA Controlled Vocabulary Service will be released soon. Be aware that iff you do any changes in the Editor, there will not be included in the new release. We will inform you when you will be able to use it again.
Thank you for your understanding and kind regards,
Maja Dolinar
CVS Service Owner"

CVS 3.3.0 has been released, please update the About page

I am not sure how to update the About page, when I click on the Edit button on this page https://vocabularies.cessda.eu/about I am not seeing any text to edit, only blank. @matthew-morris-cessda


text:
Feature request: More than 10 entries

Nothing more than that, just my use case where i have used up all 10, and would like more if it's an easy win.

How many points (entries) would you need?
Could you be a bit more specific about your use case?

I was the guy that didn't read that this ramped up between the two temperature points. So I took your example and have two points of every temperature, to get a more square graph ( and less ramp )
I was wondering if it was possible to have more than 10 points, as in my use case I'm doubling up my entries to make the more hard cut points

See Profile-with-more-than-10-points

Awesome,
i suspect i could also have more profiles, with 12 hours (making sure i have a start at 00:00 and end 23:59) but all the points in the first 12 hours, then another with more points in the second 12 hours.


text:
removes style tags and all in-line style attributes

@kurtw @richaagarwal @jimmynotjim @sebworks @KimberlyMunoz @anselmbradford
Testing

view this page

Then run ./cfgov/manage.py sheer_index -r -p posts
should now look like this with all style tags and inline styles stripped:

:+1:

:+1: based on the visual. We can probably worry about image and quote layouts when these posts move to Wagtail.

authors: jimmynotjim kave richaagarwal

repo: cfpb/cfgov-refresh


text:
Fix linter warnings

Changes

Minor linter fixes.

Review

@kave

👍

authors: anselmbradford jimmynotjim


text:
Alternate landing page

Changed the name to index2 to prevent any overwriting. We can use my fork and then delete it after testing instead of messing with the main repo.
Sound good? @ascott1 @stephanieosan or whomever else accepts these?
...note I'm submitting this from my personal lappy, and I just upgraded to 10.10 and apparently it screws up your dev environment. If either of you got around that upgrade fine, I need to know how you got around it!!

repo: cfpb/owning-a-home


text:
🛑 Grafana is down

In b0bf203, Grafana ($GRAFANA) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Grafana is back up in 69cd908 after 5 minutes.

repo: cfsausgit/upptime


text:
Dio can not request https://domain:port

Package
dio
Version
4.0.6
Operating-System
Android, iOS
Adapter
Default Dio
Output of flutter doctor -v
[!] Flutter (Channel main, 3.20.0-17.0.pre.53, on macOS 14.2 23C64 darwin-arm64, locale en-CA)
    ! Warning: `dart` on your path resolves to /opt/homebrew/Cellar/dart/3.0.4/libexec/bin/dart, which is not inside your current Flutter SDK checkout at /Library/flutter. Consider adding
      /Library/flutter/bin to the front of your path.
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.3)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.3.4)
[✓] VS Code (version 1.71.2)
[✓] VS Code (version 1.87.2)
[✓] Connected device (4 available)
[✓] Network resources

Dart Version
3.20.0
Steps to Reproduce
Use Dio to request(Get Method) https://www.abc.com:5001/api/test.
It will throw the error "No address associated with hostname". But the webapi is running well, I am sure I can use postman to get data from same API URL.
If I use my domain(https://www.abc.com/api/test) as the default port(443), then Dio works as expected.
Why Dio dose not support https://domain:port this way?
Expected Result
get the json output
Actual Result
Get the error “No address associated with hostname”

You're using a outdated package version and an unknown Dart version. We no longer support Dio 4.x.
Besides, please check your internet connection and permission on your devices.

You're using a outdated package version and an unknown Dart version. We no longer support Dio 4.x.
Besides, please check your internet connection and permission on your devices.

Dart SDK version: 3.4.0-190.0.dev (dev) (Thu Feb 29 08:14:33 2024 -0800) on "macos_arm64"
I create a new empty flutter project, choose Dio 5.4.2. It is same error.

@AlexV525

Postman result

repo: cfug/dio


text:
🎨 Using shared analysis options for all packages

Uses the base analysis options for all packages to avoid redundant definitions.
New Pull Request Checklist

[x] I have read the Documentation
[x] I have searched for a similar pull request in the project and found none
[x] I have updated this branch with the latest main branch to avoid conflicts (via merge from master or rebase)
[ ] I have added the required tests to prove the fix/feature I'm adding
[ ] I have updated the documentation (if necessary)
[ ] I have run the tests without failures
[x] I have updated the CHANGELOG.md in the corresponding package

Unfortunately, I don't have ideas about how to test compute with these changes.

It may be better to just use https://pub.dev/packages/lints and adjust a few rule that we want additionally or don't want? This way new rules would be used automatically.

It may be better to just use https://pub.dev/packages/lints and adjust a few rule that we want additionally or don't want? This way new rules would be used automatically.

That's for sure, but it cannot be used in a base config like this, since we don't have a pubspec here. Otherwise we may revert the config change and change dart files only in this PR.

I walked around other monorepos like fluttefire, flutter_local_notification, and melos itself, adding a pubspec seems fine here.

We could symlink the same file to all repos. Then we could use imports. I am not sure whether that’s a good idea.

We could symlink the same file to all repos. Then we could use imports. I am not sure whether that’s a good idea.

We should avoid symlinks because it usually breaks the development on Windows. Added new commits.

Tbh I think we can just add lints to every package. There should not be that much customization of rules required.

But this involves us changing Github workflows to run pub get in the root repository before anything else.

I'm surprised not seing any exceptions in tests. Maybe it only affects the local development.

Tbh I think we can just add lints to every package. There should not be that much customization of rules required.

That's what we have done previously.

It works perfectly fine without the pubspec.yaml in root for me.

This is what happened if you don't have a pubspec in the root which doesn't have .dart_tool too.

It works perfectly fine without the pubspec.yaml in root for me.

This is what happened if you don't have a pubspec in the root which doesn't have .dart_tool too.

Yea but this is only when you open it right? We can just name it differently and it is probably fine. Something like analysis_options_base.yaml might work.

Deleted and scheduled for another run.

authors: AlexV525 kuhnroyal ueman


text:
swift-deps-index.json became invalid after updating from 0.15.0 to 0.20.0

I'm using RxSwift and one of my files imports:
import RxRelay
import RxSwift

But the gazelle plugin is adding these to my build file:
        "@swiftpkg_rxswift//:Sources_RxBlocking",
        "@swiftpkg_rxswift//:Sources_RxCocoa",
        "@swiftpkg_rxswift//:Sources_RxRelay",
        "@swiftpkg_rxswift//:Sources_RxSwift",
        "@swiftpkg_rxswift//:Sources_RxTest",

My Package.resolved has:
    {
      "identity" : "rxswift",
      "kind" : "remoteSourceControl",
      "location" : "https://github.com/ReactiveX/RxSwift",
      "state" : {
        "revision" : "9dcaa4b333db437b0fbfaf453fad29069044a8b4",
        "version" : "6.6.0"
      }
    },

RxSwift has not been updated in 6+ months and the Package.swift was last added two years ago:
https://github.com/ReactiveX/RxSwift/blob/main/Package.swift
I believe this is a recent regression with rules swift package manager.
In my MODULE.bazel I have:
bazel_dep(name = "rules_swift_package_manager", version = "0.20.0")

This is blocking me because this target depends on XCTest which I can't link into my application.

Following up here, @luispadron and I figured out on Slack that my swift-deps-index.json became invalid when I updated my rules_swift_package_manager version from 0.15.0 to 0.20.0
When I deleted this file and Package.resolved, and then re-generated them with my swift_update_packages target, the correct dependencies could be added by the plugin.
At a minimum, the file should include versioning information and the plugin should throw an error if the file was generated from an incompatible version telling the user to re-run the swift update packages action.
Preferably, though, as I've mentioned previously, we could just produce this file from the repository rule and not check it into the repository. Bazel would automatically regenerate it when the version changes because the module versions are included in the directories in the bazel-gen and bazel-out directories.

Related to #703.

At a minimum, the file should include versioning information and the plugin should throw an error if the file was generated from an incompatible version telling the user to re-run the swift update packages action.

In the short-term, I agree that we should add a version to the file and fail with a nice message if the version of rules_swift_package_manager requires a newer version of the index file. The index file version should be independent of the rules_swift_package_manager version. It may be as simple as "file_version": X where X is an integer. This will be incremented whenever an incompatible change occurs to the file.

More discussion on removing need for this file in: https://github.com/cgrindel/rules_swift_package_manager/discussions/743

It may be as simple as "file_version": X where X is an integer. This will be incremented whenever an incompatible change occurs to the file.

My main concern with detaching this from the version of rules_spm is that it just adds another version we can mess up on updating. I also don't know how we could catch this in CI for all cases. I almost want update_swift_pkgs to be a target you need to run whenever you update rules_spm until the discussion above is addressed.
For now though, at a minimum we should fail to build with an appropriate warning if this file is out-of-date. @AttilaTheFun is this version addition something you'd be able to contribute?

authors: AttilaTheFun cgrindel luispadron


text:
Add .mergify.yml

Integrating Mergify for automatic merge.
https://doc.mergify.io/index.html

Jenkins, test this please.

Jenkins CI test (for commit d260b5f40dbfbf27c29dc9480a8e56578392fb43, target branch master) failed with status FAILURE.

Jenkins, test this please.

Jenkins CI test (for commit d260b5f40dbfbf27c29dc9480a8e56578392fb43, target branch master) failed with status FAILURE.

Added ChainerCV CIs as requirements.

Jenkins, test this please.

Jenkins CI test (for commit d7a6e2c58dcc1233f4317a7c38710d253d376e3f, target branch master) succeeded!

LGTM.

@asi1024 This pull-request is marked as to-be-backported, but the corresponding backport PR could not be found. Could you check?

@asi1024 This pull-request is marked as to-be-backported, but the corresponding backport PR could not be found. Could you check?

authors: asi1024 chainer-ci niboshi

repo: chainer/chainer


text:
Use COCO2017

https://github.com/chainer/chainercv/issues/651

How to use 'train', 'val', 'minival', 'valminusminival' of 2014? If there are any correspondences, it would be nice to write them.

Is saying correspondence in doc enough?
train + valminusminival of 2014 == train of 2017 and minival of 2014 == val of 2017

I marked this PR as no-compat since the default value of year is 2017.

COCO was never included in any release, so this is not no-compat

I see. I removed the label.

repo: chainer/chainercv


text:
Completed webpack 5 upgrade

By the strength of our code and the wisdom of our scripts, we have conquered the treacherous upgrade from webpack version 4 to version 5. Let it be known that our bundles are now more optimized and our builds more efficient. Onwards, brave developers, to new horizons of performance and stability!

This branch has been reworked and you contribution split in #999 and #1004. Thanks!

authors: millette tahini

repo: chairemobilite/transition


text:
Sanic crashes when other asyncio tasks are running

Sanic version: 0.4.1
Python: 3.6.0
OS: Fedora Linux 25
I have an app were I have to start serving requests before the database (in this case MongoDB) connection fully started, replying with an adequate error message on requests. The problem is that sanic stops listening for tcp connections (the port closes), I don't know exactly what is causing it, but here is an example code that causes the problem:
import asyncio
import uvloop
from sanic import Sanic
from sanic.response import json
from motor.motor_asyncio import AsyncIOMotorClient
from pymongo.errors import CollectionInvalid, ServerSelectionTimeoutError

DB_NAME = 'testdb'
COLLECTION_NAME = 'testcoll'

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

loop = asyncio.get_event_loop()
conn = AsyncIOMotorClient(
    host='127.0.0.1',
    port=27017,
    connectTimeoutMS=2000,
    serverSelectionTimeoutMS=5000,
)
db = conn['testdb']
collection = db['testcoll']

app = Sanic()

async def start_db():
    print('connecting to mongodb!')
    try:
        await db.create_collection(COLLECTION_NAME)
    except CollectionInvalid:
        pass
    except ServerSelectionTimeoutError:
        await asyncio.sleep(0.5)
        print('trying to connect to mongodb again...')
        loop.create_task(start_db())
        return

    # Indexes are supposed to be created here

    print('connected to mongodb!')

@app.route('/')
async def handler(request):
    data = await collection.find().to_list(None)
    return json(data)

def main():
    server = app.create_server(
        host='127.0.0.1',
        port=5000,
        debug=True,
        loop=loop,
    )
    loop.create_task(start_db())
    asyncio.ensure_future(server, loop=loop)

    # Start other asyncio things here
    loop.run_forever()

if __name__ == '__main__':
    main()

with sanic debug enabled, a warning about tcp server resource unclosed appear on the terminal:
connecting to mongodb!
~/.local/lib/python3.6/site-packages/sanic/app.py:524: DeprecationWarning: Passing a loop will be deprecated in version 0.4.0 https://github.com/channelcat/sanic/pull/335 has more information.
  DeprecationWarning)
2017-03-06 16:15:19,518: DEBUG:
sanic drawing here
2017-03-06 16:15:19,518: INFO: Goin' Fast @ http://127.0.0.1:5000
trying to connect to mongodb again...
/usr/lib64/python3.6/traceback.py:352: ResourceWarning: unclosed resource <TCPServer closed=False 0x7f7dcbf4b448>; object created at (most recent call last):
  File "example_sanic.py", line 64, in <module>
    main()
  File "example_sanic.py", line 60, in main
    loop.run_forever()
  File "~/.local/lib/python3.6/site-packages/sanic/app.py", line 509, in create_server
    return await serve(**server_settings)
  filename, lineno, name, lookup_line=False, locals=f_locals))
connecting to mongodb!
trying to connect to mongodb again...

EDIT: Forgot to mention, this error happens after the first mongo connection retry

I think your issue is similar too https://github.com/channelcat/sanic/issues/435, try using the 'before_server_start' hook to initialize your mongo client. An example shown with aiocache here https://github.com/channelcat/sanic/blob/master/examples/cache_example.py

I should have explained it better, this is not exactly my use case, I just created it as a minimal example that causes the same problem.
My use case is that I'm starting a amqp listener (and writer) in the same process as sanic. I don't need multiple workers, I want everything in one thread, I manage workers using docker swarm mode (I have load balancing on nginx).
The problem with the before_server_start hook is that it requires sanic to start everything. I think it's not very practical to do this, for example, if another framework takes the same approach as sanic and requires everything asyncio related to be started in it's hooks, how would I start both sanic and this framework on the same event loop?

You'd start Sanic first and then use get_event_loop to get the currently running event loop? Is there a big issue with doing that? Also the hook only exists as a nice way for people to establish all the things there app needs. You don't necessarily have to use it since (with Python 3.6) get_event_loop gets the already running event loop.

It should'nt make any difference if you get the event loop before sanic start, but anyway, the error still happens because app.create_server does not execute anything before the loop starts (it is a coroutine). I changed the code to do what you said:
import asyncio
import uvloop
from sanic import Sanic
from sanic.response import json
from motor.motor_asyncio import AsyncIOMotorClient
from pymongo.errors import CollectionInvalid, ServerSelectionTimeoutError

DB_NAME = 'testdb'
COLLECTION_NAME = 'testcoll'

asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

conn = AsyncIOMotorClient(
    host='127.0.0.1',
    port=27017,
    connectTimeoutMS=2000,
    serverSelectionTimeoutMS=5000,
)
db = conn['testdb']
collection = db['testcoll']

app = Sanic()

async def start_db(loop):
    print('connecting to mongodb!')
    try:
        await db.create_collection(COLLECTION_NAME)
    except CollectionInvalid:
        pass
    except ServerSelectionTimeoutError:
        await asyncio.sleep(0.5)
        print('trying to connect to mongodb again...')
        loop.create_task(start_db(loop))
        return

    # Indexes are supposed to be created here

    print('connected to mongodb!')

@app.route('/')
async def handler(request):
    data = await collection.find().to_list(None)
    return json(data)

def main():
    server = app.create_server(
        host='127.0.0.1',
        port=5000,
        debug=True,
    )

    asyncio.ensure_future(server)
    loop = asyncio.get_event_loop()
    loop.create_task(start_db(loop))

    # Start other asyncio things here
    loop.run_forever()

if __name__ == '__main__':
    main()

The other option would be to run the app.run method, but this method is blocking, it runs the loop.run_forever method, so I can't get the event loop after that.

I'm still not entirely sure if this is what your looking for but there is some trickery with creating an event loop and using app.run() as app.run() always creates a It always creates a new loop here: https://github.com/channelcat/sanic/blob/master/sanic/server.py#L343 .  I've constructed a minimal bug with the following
from sanic import Sanic
from sanic.response import text
import uvloop
import asyncio
class ChattyPolicy(uvloop.EventLoopPolicy):
    def get_event_loop(self):
        loop = super().get_event_loop()
        print('getting event loop {}'.format(id(loop)))
        return loop

    def set_event_loop(self, loop):
        super().set_event_loop(loop)
        print('setting event loop {}'.format(id(loop)))

    def new_event_loop(self):
        loop = super().new_event_loop()
        print('new event loop {}'.format(id(loop)))
        return loop

asyncio.set_event_loop_policy(ChattyPolicy())
app = Sanic()
loop = asyncio.get_event_loop()

async def initalizer():
    return 'initalized'

f = asyncio.ensure_future(initalizer())

@app.route('/')
async def root(request):
    print(await f)
    return text('hello world')

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8000, loop=loop)

try using the ChattyPolicy to see if that shows you more than one id. For example I get the following output
new event loop 12603544
setting event loop 12603544
getting event loop 12603544
getting event loop 12603544
getting event loop 12603544
2017-03-07 13:12:57,593: INFO: Goin' Fast @ http://0.0.0.0:8000
setting event loop 12869464
2017-03-07 13:12:57,596: INFO: Starting worker [3451]
2017-03-07 13:13:00,328: ERROR: Traceback (most recent call last):
  File "path/to/sanic.py", line 329, in handle_request
    response = await response
  File "i_test.py", line 34, in root
    print(await f)
  File "uvloop/future.pyx", line 241, in __await__ (uvloop/loop.c:104128)
RuntimeError: Task <Task pending coro=<Sanic.handle_request() running at /path/to/sanic.py:329>> got Future <Task pending coro=<initalizer() running at i_test.py:27>> attached to a different loop

2017-03-07 13:14:00,328: ERROR: NoneType

Yes, I'm aware that using app.run always create a new event loop, that is why I'm using the app.create_server method. The problem is that after the ResourceWarning error message sanic stops listening on the port, this only happens when there are other tasks running in the same event loop.
Running the last example I gave with @abuckenheimer custom event loop policy, it seems that only one event loop is being created. I got the following log running it:
new event loop 93971158371896
setting event loop 93971158371896
getting event loop 93971158371896
getting event loop 93971158371896
getting event loop 93971158371896
~/.local/lib/python3.6/site-packages/sanic/app.py:524: DeprecationWarning: Passing a loop will be deprecated in version 0.4.0 https://github.com/channelcat/sanic/pull/335 has more information.
  DeprecationWarning)
2017-03-07 15:24:34,720: DEBUG:
sanic logo
2017-03-07 15:24:34,720: INFO: Goin' Fast @ http://127.0.0.1:5000
connecting to mongodb!
~/.local/lib/python3.6/tokenize.py:381: ResourceWarning: unclosed resource <TCPServer closed=False 0x7fefeb0e24c8>; object created at (most recent call last):
  File "example_sanic.py", line 83, in <module>
    main()
  File "example_sanic.py", line 79, in main
    loop.run_forever()
  File "~/.local/lib/python3.6/site-packages/sanic/app.py", line 509, in create_server
    return await serve(**server_settings)
  return readline()
trying to connect to mongodb again...
connecting to mongodb!
trying to connect to mongodb again...
connecting to mongodb!

You are right! I'm debugging the wrong problem. will look into this

I suggest you give my approach a shot and please give me feedback.
My method does not allow for multiprocessing but this is no problem for production purposes as you will want to start multiple independent Python interpreter instances using a supervisor such as supervisord anyway
https://gist.github.com/nszceta/087a14d8896e47bc6e7e441fd60b6ff4

After some testing, I discovered that the problem is not related to sanic code, the same problem happened with a similar code using aiohttp. The one causing this bug is the uvloop module, removing the line below fixes the problem (it makes the process run with the default python event loop).
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())

I created an issue on the uvloop project to track this error MagicStack/uvloop#81

@r0fls can we close this issue?

Thanks @nszceta. Closing, will reopen if needed.

authors: abuckenheimer nszceta r0fls rudineirk seemethere

repo: channelcat/sanic


text:
Move memento code to census repo

The code is currently located at https://github.com/mincheoly/memento-cxg. Move diffexp code to Census repo, with API entrypoint under api/experimental package. Move builder code to tools/scripts/memento.

Merged this to the epic branch. Closing.

repo: chanzuckerberg/cellxgene-census


text:
Replace existing layer_descriptions field with new schema 2.0.0 X_normalization

layer_descriptions is being depricated and replaced by X_normalization here
Change dataset validator to check for presence of X_normalization field:
https://github.com/chanzuckerberg/corpora-data-portal/blob/44b98b49bd3a38bf19d9f73b9b75e14d2b4e0586/backend/corpora/common/dataset_validator.py#L134-L146
Change raw layer extraction (See #1251  for the logic needed to identify the appropriate layer)
https://github.com/chanzuckerberg/corpora-data-portal/blob/7139b4f7a123544b7982dde74b8ddf17436f11de/backend/corpora/dataset_processing/process.py#L257-L267
Store X_normalization as a dataset column and serve on GET /dp/v1/collections/{id}
Add field to dataset entitity on frontend
Definition of Done

Add @ambrosejcarr to review raw layer extraction and mean genes per cell calculation

"Change raw layer extraction" portion of this ticket is done here: https://github.com/chanzuckerberg/corpora-data-portal/pull/1351

"Change dataset validator" portion of this ticket is done here: https://github.com/chanzuckerberg/single-cell-data-portal/pull/1363

authors: leslieypark seve

repo: chanzuckerberg/corpora-data-portal


text:
Show project and viz name in viz list

Description

This adds the info to the backend needed for the ultimate design.
NOTE: the table render design still needs work.
NOTE: I'm not sure on the best way to avoid N+1 queries here. Please advise.
Test and Reproduce
Outline the steps to test or reproduce the PR here.

try endpoint, http://localhost:3000/visualizations.json?domain=library
see data for both heatmap and phylo tree
see correct titles for  heatmap and phylo tree
try same in viz list UI

I changed the PR so that the name of a visualization is stored in the database, eliminating the N+1 queries at read time. Likewise, the sample ids of a phylo tree are stored.
To make sense of the class, I refactored phylo tree methods such as job_command to be all private when not called outside the class  . Please sanity check.

Did you discuss this naming strategy with Jenn?

No, that's a good idea. @happyimadesignr: I came up with an auto naming algorithm for showing the heatmaps in the visualization list.   It will makes a string such as "Patient 016 (CSF) and 015 (CSF)" from the two samples "Patient 016 (CSF)", "Patient 015 (CSF)". What do you think?

Did you run a test phylo tree?

I ran it but it never seems to get past:  "Computation in progress. Please check back later!". Same in master. Can I test in staging or is the a better way?

Hey @gregdingle and @tfrcarvalho

I think it's nice to include the project name, but I would put it in the table column instead of the name. I don't thinking including the samples names in the title will be scalable for a lot of samples.
For Phylo trees, they already have a name, so it should display the name the user made.
For Heatmaps, I was thinking that it could just be called "Heatmap" or "Untitled Heatmap" and the user can name it. Then the table details will help differentiate the heatmaps from each other if the user doesn't explicitly name it. Or we can also have users name it when they save for the first time. For the minimum version, why don't we called it Heatmap and complete the table columns? What do you think?

https://chanzuckerberg.invisionapp.com/d/main#/console/16827472/350949082/preview

@happyimadesignr

I think it's nice to include the project name, but I would put it in the table column instead of the name.

Yea, that was always my intention. I didn't have time to add the columns when I first wrote this PR.

For Heatmaps, I was thinking that it could just be called "Heatmap" or "Untitled Heatmap" and the user can name it.

Ok, that will work, and is simpler. We can keep the auto name idea in mind for we add the ability to name heatmaps.
About the project name, how do you want to handle the case when there multiple projects for one visualization?

authors: gregdingle happyimadesignr

repo: chanzuckerberg/idseq-web


text:
Yf/genus aggregation

doing genus aggregation on the db load
Test Plan
pr = PipelineRun.find(31)
pr.pipeline_output.destroy!
pr.load_results_from_s3
and check results!!

cool, i want to read more carefully when i get to a big screen, particularly as to what happens for species that have no genus

@boris-dimitrov in that case, no genus counts for it.

I think it would be useful to create a pseudo-genus for those taxons, so that we can get count and z score for all of them as a category, on the report.   The reason this will be useful is that we will have expand/collapse function showing species under their genuses, with sorting and filtering working on the genus metrics.  Sort of an officially supported “Other” category could be useful.  Don’t we currently assign -999 genus id for those guys?

Agree with Boris and yes when we generate the taxon lineage table we do fill all missing values with -999.

I think anything map to other we actually dont show on report.
Sent from my iPhone

On Oct 18, 2017, at 7:35 PM, cdebourcy notifications@github.com wrote:
Agree with Boris and yes when we generate the taxon lineage table we do fill all missing values with -999.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub, or mute the thread.

authors: boris-dimitrov cdebourcy yunfangjuan


text:
GitHub API: 422 unprocessable entity

422 Client Error: Unprocessable Entity for url: https://api.github.com/repos/chaosbot/Chaos/labels

http://chaosthebot.com:8081/chaos-stderr.log
We had this issue with status updates, but now it's returned in the form of label creation. Ideas?

I believe that's the error it returns that when the label already exists. Let me check on that, though.

Yea, we should log the response body if there's an error. It should say there. My guess is that the labels already exist. Reference: https://developer.github.com/v3/#client-errors

I hadn't seen the client errors page before, thanks. I'd still prefer to not log a bunch of errors, and instead list the labels once, then add missing ones. I implemented a similar solution for the blocklist PR.

https://github.com/chaosbot/Chaos/pull/451#issuecomment-305423130
I am going to rework this PR, with the @hongaar ideas https://github.com/chaosbot/Chaos/pull/451#issuecomment-305489318

Will be fixed by #486.

authors: PlasmaPower Swizz andrewda md678685

repo: chaosbot/Chaos


text:
Ideas for Projects for She Code Africa

CHAOSS is very interested in participating in the April-May cohort of She Code Africa's Contributhon, but before we can complete our application to be a mentor organization, we need to define what project they will be working on. I think this is a little bit different than our usual GSOC projects because of the shorter timing, and the fact that we have a CHAOSS member who is a past participant in this special project. We will hopefully be able to participate as a mentorship organization, but it depends on if the project gets sufficiently funded from sponsorship orgs.
Our current idea is to create a slackbot that would help point newcomers to some resources upon joining our Slack community.
@ruth-ikegah were you interested in being a mentor for this? It has been a really long time since I wrote a slackbot :laughing: but I could try to help out on the technical side if needed.
If we think this is a project we want to submit, we will have to draw up a list of subtasks and technical requirements (operating systems, etc).
The application (along with the project and subtasks) is due February 28, so we are kind of tight on time.
Would appreciate any other insights and comments @germonprez and @sgoggins.

Thanks @ElizabethN. This sounds like a great project idea. I'd sort of suggest only 1 (maybe 2) project ideas. The slackbot sounds great and I really liked that idea when it was talked about in the DEI audit team meeting.

Another idea I have for a project, we chatted about graphically representing our CHAOSS metrics (the one in the PDF) which will need design skills.

I really like this idea, too! Is this something you would be able to act as a mentor, @Ruth-ikegah? I think you'd be awesome as a mentor!

Yes, I can! We can talk more about it on slack or our 1:1 @ElizabethN

OK so after talking about this even more with @germonprez and the DEI Working Group, we think it would be fine to go ahead with our slackbot project idea and have you mentor, @Ruth-ikegah. We have plenty of people who can support from a technical standpoint that we shouldn't have any trouble. We can discuss at our weekly meeting tomorrow @Ruth-ikegah but let's do it! :tada:

Hi all, I'm going to close this issue as the ideas have become actual projects :)

authors: ElizabethN Ruth-ikegah germonprez

repo: chaoss/community


text:
Update event-demographics.md

Updated description to include the sentence:
This metric is applicable to virtual, in-person, and hybrid events.
Signed-off-by: Matt Germonprez germonprez@gmail.com

@germonprez just for the record, I'm including this PR in the Translations issue with all the minor DEI changes. (I hope that makes sense, let me know if it doesn't!) :D

authors: ElizabethN germonprez

repo: chaoss/wg-dei


text:
🛑 de-3 (itsTyrion) is down

In 19eb47f, de-3 (itsTyrion) (https://de-3-mirror.chaotic.cx/no-failover/chaotic-aur/lastupdate) was down:

HTTP code: 503
Response time: 117 ms

Resolved: de-3 (itsTyrion) is back up in a86f8d6.

repo: chaotic-aur/chaotic-uptimes


text:
🛑 secret (Garuda) is down

In 145644f, secret (Garuda) (https://builds.garudalinux.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: secret (Garuda) is back up in 6d5b3fa after 9 hours, 19 minutes.


text:
feed CNVkit reference data

Hi,
I was wondering if there is a way to supply control *.cnn when setting up a bcbio run.  I am interested in germline SV detection and want to use the same references every time run a new case.  However, it seems from the documentation the controls *.cnn file will recalculated every time bcbio runs.
Am I correct in this statement or is there I feature I missed?  Could a feature be made where this could be supplied?

Thanks much for starting this discussion. This is definitely something we'd like to support and I started with a basic framework to specify these. We need to work on the actual implementation within CNVkit which requires two additions:

Identify the target and antitarget BED regions from the input background file so these are consistent with the newly generated coverage bins for input samples.
Substitute in the supplied background reference CNN instead of calculating background.

I'll continue to work on implementing this and track progress here to let you know when it's ready. Thanks again for the suggestion.

The CNVkit distribution includes a script reference2bed.py which unpacks the target and antitarget regions from the reference.cnn file into two BED files, with filename extensions added automatically. (Sorry if that complicates reuse here.) The same internal functions are used by CNVkit's batch command, so if bcbio uses this script, the behavior will be the same.

Thanks again for starting this discussion, and thanks Eric for the tip on reference2targets.py, that nicely filled in the missing piece for us.
The latest development version of bcbio supports specifying a pre-built background CNN with:
algorithm:
   background:
       cnv_reference: /path/to/your/background.cnn

Please let us know if you run into any problems and thanks again for the suggestion.

authors: chapmanb etal schneiderthomas

repo: chapmanb/bcbio-nextgen