text:
Conversions to stdlib types

Noda Time provides the following methods for conversion to/from BCL types:

Instant.FromDateTimeOffset()
Instant.FromDateTimeUtc()
Instant.ToDateTimeOffset()
Instant.ToDateTimeUtc()
LocalDate.FromDateOnly()
LocalDate.FromDateTime()
LocalDate.ToDateOnly()
LocalDate.ToDateTimeUnspecified()
LocalDateTime.FromDateTime()
LocalDateTime.ToDateTimeUnspecified()
LocalTime.ToTimeOnly()
LocalTime.FromTimeOnly()
Offset.FromTimeSpan()
Offset.ToTimeSpan()
OffsetDateTime.FromDateTimeOffset()
OffsetDateTime.ToDateTimeOffset()
ZonedDateTime.FromDateTimeOffset()
ZonedDateTime.ToDateTimeOffset()
ZonedDateTime.ToDateTimeUtc()
ZonedDateTime.ToDateTimeUnspecified()

When thinking about how to port these to Python, some of them are relatively self-explanatory:

.ToTimeSpan() and .FromTimeSpan() become .to_timedelta() and .from_timedelta()
.ToDateOnly() and .FromDateOnly() become to_date() and from_date()
.ToTimeOnly() and .FromTimeOnly() become to_time() and from_time()

Where things become slightly more nuanced is when you are dealing with conversions to/from DateTime or DateTimeOffset.
In Noda Time, the distinguishing features of these methods are:

The type they convert to/from, e.g. DateTime or DateTimeOffset
For DateTime conversions, these are further distinguished by the DateTimeKind.

In Python's stdlib, we don't have a real equivalent to DateTimeOffset. We just have timezone-aware datetimes. We also don't have a DateTimeKind to deal with - just the notion of aware and naive datetimes.
Take Instant as an example. In Noda Time, we have FromDateTimeOffset() and FromDateTimeUtc(). In Python, it probably makes sense to have a single from_aware_date_time() method. Similarly, the ToDateTimeOffset() and ToDateTimeUtc methods could probably can be represented in Python with a single to_utc_datetime() method.

I've spent a lot of time thinking about this and not actually doing it. I'm going to just bite the bullet and get something out there.
This is the result of that thinking I've been doing... subject to change still.

Class
Noda
Pyoda
Comments

Duration
FromTimeSpan()
from_timedelta()

Duration
ToTimeSpan()
to_timedelta()

Instant
FromDateTimeOffset()
from_aware_datetime()
Same as FromDateTimeUtc(). Should raise if a naive datetime is passed.

Instant
FromDateTimeUtc()
from_aware_datetime()
Same as FromDateTimeOffset(). Should raise if a naive datetime is passed.

Instant
ToDateTimeOffset()
to_datetime_utc()
Same as ToDateTimeUtc()

Instant
ToDateTimeUtc()
to_datetime_utc()
Same as ToDateTimeOffset()

LocalDate
FromDateOnly()
from_date()

LocalDate
ToDateOnly()
to_date()

LocalDate
FromDateTime()
from_naive_datetime()
Accepts a naive datetime and extracts the date from it. No time zone conversion is performed. Should raise if an aware datetime is passed.

LocalDate
ToDateTimeUnspecified()
to_naive_datetime()
Implicit conversion to Gregorian calendar!!! Fractional second precision may be lost.

LocalDateTime
FromDateTime()
from_naive_datetime()
Raises if an aware datetime is passed.

LocalDateTime
ToDateTimeUnspecified()
to_naive_datetime()
Note that there is an implicit conversion to gregorian, and that fractional seconds precision may be lost.

LocalTime
ToTimeOnly()
to_time()

LocalTime
FromTimeOnly()
from_time()

Offset
FromTimeSpan()
from_timedelta()

Offset
ToTimeSpan()
to_timedelta()

OffsetDateTime
FromDateTimeOffset()
from_aware_datetime()
Note that this has a fixed offset from utc!!!

OffsetDateTime
ToDateTimeOffset()
to_aware_datetime()
Note that the tzinfo is a timezone instance with a fixed utcoffset.

ZonedDateTime
FromDateTimeOffset()
from_aware_datetime()
In Noda Time, the offset is fixed and unchanging... We can probably make this look up the zone id though in the default provider. Raises if a naive datetime is passed, or if the timezone cannot be found.

ZonedDateTime
ToDateTimeOffset()
to_aware_datetime()
Represents the same date and time in the same timezone.

ZonedDateTime
ToDateTimeUtc()
to_datetime_utc()
Represents the same instant in time (expressed as utc) rather than the same local time.

ZonedDateTime
ToDateTimeUnspecified()
to_naive_datetime()
Represents the same local time rather than the same instant in time.

repo: chrisimcevoy/pyoda-time


text:
pull in aws creds from default directory

hi, nice library.
It would be nice if it searched for your aws creds in the usual places Ie. '~/.aws' so that you don't have to enter them as arguments to a search each time.
cheers

That would be cool.  Might look at adding...
Open to a pull request

Checked in #3 that includes this.
Thanks for the idea!

source: gharchive/issue

repo: chriskinsman/DynamoDbExportCsv

license: mit

license_type: permissive

license_source: bigquery


text:
volume 的询问

你好
volume是只能注册到本机的master吗，能不能让volume注册到另外一台服务器上的master上呢。

No. Yes.

那怎么将volume注册到另外一台服务器上呢
/usr/local/weed volume -dir="/data/seaweed/vol/vol2" -mserver="192.168.1.15:9333" -port=9084 -ip="192.168.1.16" $>>/data/seaweed/logs/vol4.log

这个是我的配置信息，
我使用以上的配置信息注册不到15服务器上。
我是需要把master加上16的ip地址吗？

can not tell. Very likely it is the network problem.

authors: chrislusf xiaoqi-hhm

repo: chrislusf/seaweedfs


text:
adding header 'Access-Control-Allow-Origin'

Could you please include the feature adding individual headers like 'Access-Control-Allow-Origin' for responses? Otherwise, like many users (sea Google search) we have to use nginx or apache in front of seaweedfs. That slows the great speed of seaweedfs down and eats unnecessary resources .
In the meantime, where do we find this "response part" in the source code, so we would manually add this part and rebuild everything?
Thanks

Add Access-Control-Allow-Origin: * to all requests?

Yes, to all responses, e. g. add header(s) via command line and seafeedfs would include that header when serving images/data.
We have a bunch of servers via sub domains serving a ton of images. Sub domains are treated as separated domains and we have a new use case/feature set where seaweedfs will not work due to CORS policy.
We can include it manually and rebuild seaweedfs again though. But where?

Assuming you are using volume servers, add the fix similar to this line: https://github.com/chrislusf/seaweedfs/blob/fee3319cbf4ed88f79c6e7fe19bada2bddabbb85/weed/server/volume_server_handlers_read.go#L246

I've just started using Filer. Looks great, but I'm getting a CORS error using the javascript Fetch API:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8888/. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

Part of the reason I've gone for SeaweedFS is the ease of setup, so I'd like to avoid having to put a webserver in front of it all. Was a fix added?
ps. I do this on one of my go REST servers (using github.com/gorilla/handlers):
cors := handlers.CORS(
        handlers.AllowedHeaders([]string{"content-type"}),
        handlers.AllowedMethods([]string{"GET"}),
        handlers.AllowedOrigins([]string{"*"}),
        handlers.AllowCredentials(),
    )

@glowkeeper you want to add this to the filers, not volume servers, correct?
How about just doing this:

add Access-Control-Allow-Origin: * to all requests that has a "Origin" header.

Indeed (filers) - that sounds perfect - but forgive me, I'm new to Seaweed

how do I do that?

On Sat, 31 Oct 2020, 23:11 Chris Lu, notifications@github.com wrote:

@glowkeeper https://github.com/glowkeeper you want to add this to the
filers, not volume servers, correct?
How about just doing this:

add Access-Control-Allow-Origin: * to all requests that has a
"Origin" header.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/chrislusf/seaweedfs/issues/1271#issuecomment-720000163,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACGV6IJFSTZU6TLB7C53T23SNSKSZANCNFSM4MFRQD5Q
.

I added CORS support for filer and volume in 306062b4e7cacf9fbeae227965eeeef08ccfeb92 and 05acc3329411d50e62a973cb9e93edfa7216995f

I added CORS support for filer and volume in 306062b and 05acc33

Great. So how do I enable Access-Control-Allow-Origin: * on filer? Forgive me - but I haven't tracked down (yet) any docs related to it  - is that something I can add to filer.toml (for example)?

It is hard coded. Use nginx if customization is needed.

I'm confused! Does filer enable Access-Control-Allow-Origin: *, or not? My experience, so far, is it doesn't (obvs). But then, how was CORS support enabled?
Anyway - here's my requirement:

Currently, I'm trying to access filer on http://localhost:8888/ from some javascript running on http://localhost:9004/subDomain.
Ultimately, I'd like to access filer on  http://my.server.org:8888/ from a script running on http://possibly.anywhere.net:9004/subDomain.

I'm bypassing CORS errors with  1) by using local cors proxy.
However, like the original poster, I want to avoid having to proxy filer for scenario 2) because I want to keep things as simple as possible for a possibly large set of users who may use my javascript app' for their own filer setups.
Is it possible to enable Access-Control-Allow-Origin as a configurable option in filer.toml?
(ps. maybe re-open the issue?)

Does filer enable Access-Control-Allow-Origin: *, or not?

Yes. I added CORS support for filer and volume in 306062b and 05acc33, which hard coded "*".

So why do I get CORS errors, then?
On Sun, 1 Nov 2020 at 18:57, Chris Lu notifications@github.com wrote:

Does filer enable Access-Control-Allow-Origin: *, or not?
Yes. I added CORS support for filer and volume in 306062b
https://github.com/chrislusf/seaweedfs/commit/306062b4e7cacf9fbeae227965eeeef08ccfeb92
and 05acc33
https://github.com/chrislusf/seaweedfs/commit/05acc3329411d50e62a973cb9e93edfa7216995f,
which hard coded "*".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/chrislusf/seaweedfs/issues/1271#issuecomment-720135145,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACGV6INUVTYUOT5VHMBPINLSNWVQJANCNFSM4MFRQD5Q
.

So why do I get CORS errors, then?

Very likely the version you are using does not have the fixes. If not, create an issue with details on how to reproduce it.

I am also not getting CORS headers and therefore can't use seaweedfs :/
Here are steps to replicate:
$ docker run -p 8333:8333 --rm chrislusf/seaweedfs version
version 30GB 2.68 96514f0f00f1ca6f0f253c1c4080a45887d09ad9 linux amd64

$ docker run -p 8333:8333 --rm chrislusf/seaweedfs server -s3
(...)
 s3.go:201] Start Seaweed S3 API Server 30GB 2.68 96514f0f00f1ca6f0f253c1c4080a45887d09ad9 at http port 8333

$ curl  -H 'Origin: http://something' localhost:8333 --verbose
*   Trying 127.0.0.1:8333...
* Connected to localhost (127.0.0.1) port 8333 (#0)
> GET / HTTP/1.1
> Host: localhost:8333
> User-Agent: curl/7.74.0
> Accept: */*
> Origin: http://something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 206
< Content-Type: application/xml
< Server: SeaweedFS S3 30GB 2.68
< X-Amz-Request-Id: 1635684778631633576
< Date: Sun, 31 Oct 2021 12:52:58 GMT
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host localhost left intact
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><DisplayName></DisplayName><ID></ID></Owner><Buckets></Buckets></ListAllMyBucketsResult>

# Notice no Access-Control-Allow-* headers

Any help would be greatly appreciated!

@jareks You are listing all buckets. Is this supposed to work?
Getting each object works.
$ curl -v -H 'Origin: http://something' "http://localhost:8333/harbor/docker/registry/v2/repositories/chrislusf/seaweed_agent/_uploads/63de3955-8ba3-4814-8312-bec6d1f8c8a3/hashstates/sha256/12278234" --output -
*   Trying ::1:8333...
* Connected to localhost (::1) port 8333 (#0)
> GET /harbor/docker/registry/v2/repositories/chrislusf/seaweed_agent/_uploads/63de3955-8ba3-4814-8312-bec6d1f8c8a3/hashstates/sha256/12278234 HTTP/1.1
> Host: localhost:8333
> User-Agent: curl/7.77.0
> Accept: */*
> Origin: http://something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: Content-Disposition
< Content-Disposition: inline; filename="12278234"
< Content-Length: 108
< Content-Type: application/octet-stream
< Date: Sun, 31 Oct 2021 17:03:33 GMT
< Etag: "da75852f48c2ddf9e632550bac2f5cb4"
< Last-Modified: Fri, 29 Oct 2021 21:10:31 GMT
< Server: SeaweedFS Filer 30GB 2.75
< X-Amz-Storage-Class: STANDARD

@chrislusf many thanks for such a quick response!
Indeed, the real problem is not with getting list of all buckets, but with uploading files from browser:
POST http://host/bucket-name/
Content-Type: multipart/form-data
Very simplified example (which results in error - but still no CORS headers, so browser wont report that to javascript):
$ curl -F a=b -H 'Origin: http://sometething' 10.152.183.195:9000/bucket --verbose
*   Trying 10.152.183.195:9000...
* Connected to 10.152.183.195 (10.152.183.195) port 9000 (#0)
> POST /bucket HTTP/1.1
> Host: 10.152.183.195:9000
> User-Agent: curl/7.74.0
> Accept: */*
> Origin: htething
> Content-Length: 137
> Content-Type: multipart/form-data; boundary=------------------------84bd241c5ea20644
>
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Accept-Ranges: bytes
< Content-Length: 226
< Content-Type: application/xml
< Server: SeaweedFS S3 30GB 2.68
< X-Amz-Request-Id: 1635703620667861319
< Date: Sun, 31 Oct 2021 18:07:00 GMT
<
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host 10.152.183.195 left intact
<Error>MissingFields<Message>Missing fields in request.</Message><Resource>/bucket</Resource><RequestId>1635703620667848530</RequestId><BucketName>bucket</BucketName></Error>

Maybe it's simply this endpoint for POSTing file that's not adding CORS headers?

what about normal successful requests?

Same thing, browser reports missing CORS.
It only works fine if I add Access-Control-Allow-Origin through my proxy (traefik).

authors: chrislusf glowkeeper jareks kfe908


text:
Add mTLS support for both master and volume http server.

This adds TLS support to the master server, and mTLS support to both the master and volume server conditional on specifying the correct certificate files in security.toml.  If this is acceptable, I'm happy to also update the necessary documentation.

Looks good! Currently the existing clients are only using http to connect to volume and master servers. Is this intended for use cases without filer?

Correct.  We can't currently do grpc, but need mTLS.

source: gharchive/pull-request

authors: bercknash chrislusf


text:
failed to download data from ib

it just shows that i have connected to the client, but just timeout when i downloaded
the data from ib  it just warned me that it times out , then the  graphical interfaces of my client shows
the status of history_data_center is  inactive, i just want to use ib-api, but it never works, so
i really need your help to download the data from ib-api, Looking forward to hearing from you

when i use quandl to download data, it warns me that" We could not recognize your API key. Please check your API key and try again", i just don't kown how to deal with it

license: MIT

license_source: github-api


text:
Sprite transparency clashing

If player and another sprite are overlapping while moving the player the overlapped sprite is showing through the player (possibly only on each other frame, not 100% sure). See attached gif

I am using the gb studio on windows10, the issue occurs in the example project.
The issue was noticed during preview, not sure if the issue persists in built projects, .gb or .html.

That's actually a limitation of the Game Boy hardware itself I believe, the original Game Boy has no ability to control z-index order of sprites, and instead uses the X Coordinate of the sprites to determine the draw order. The alternative would be to make it so no actors can overlap but I tried it that way too and prefer it as is (also you need to check less tiles on the collision detection which was starting to be a problem). Don't think I can do much about this one sorry.

If we can't always draw the player last, then an alternative solution might be to lock the player to tile grid - maybe make that an option for the project?
By locking player to tile is to make sure that player always ends up on a grid. So if I press left for one frame I shouldn't move just one pixel to the left, but move one tile (pokemon does that, don't remember if the gb zeldas do that, but it was pretty common in games from that time). This would eliminate the need for checking for extra sprite tiles.
Now thinking about it, this would only fix the issue if we would also be able to choose if the whole sprite is solid, and not only the bottom two tiles. So, maybe not really a fix.
Nevertheless, would love the ability to make sprites stick to tile grid.

There are other possible workarounds that involve partial software rendering, and I don't know precisely how feasible they'd be because of the CPU overhead of software bit shifting. You might end up having to cut the frame rate on DMG to 30 fps.

Horizontally shift any sprite that's in front of another sprite so that they're 8-pixel aligned. For sprites with the same X coordinate, Z-order works as expected: lower OAM index in front.
Allocate new background tiles for any sprite that's occluded and draw it to the background, as seen in Prehistorik Man.

This Z-order quirk is not an issue for Game Boy Color software.

The latest dev builds now have the option to use a custom color palette which outputs the game as a Game Boy Color ROM instead, as @pinobatch says this issue doesn't occur there so one option is to enable a custom palette (you can leave the colors as defaults or change them whichever you prefer)

authors: Level0gamedev chrismaltby pinobatch

repo: chrismaltby/gb-studio


text:
Kernel Panic with long script

Describe the bug
The game starts fine, but as soon as I enter the battle room of my game, there is a fatal crash, described as a kernel panic by the GB Studio emulator.
To Reproduce

In the "On Init" portion of battle avatar, attach a script to the B button with 7 different attack options, controlled by a switch case. Which case is executed is controlled by the player setting their attack options in game.
Each of these 7 cases contains around 10-15 events, including launching a projectile.
the system works when there are only 6 calls to the launch projectile event, but as soon as I add the 7th call, the game crashes.
If I remove all the attack logic, and have only 7 launch projectile calls, it works. It also works if I remove one of the 7 projectile calls, and leave the rest of the attack logic.

Expected behavior
If this truly is a bug, I would simply desire the game not to crash. However, if this is simply a case of my script being too long for the game to process, perhaps a warning in the GUI when scripts become too long would be helpful.
Screenshots

Platform (please complete the following information):

Windows 10
GB Studio 3.0.2

Additional context

I believe that this was caused by my script being too long, because if I removed everything else being done as a part of the attack system and only had 7 projectile event calls, it worked fine. As a result, I am not entirely sure this is a bug so much as me running out of memory for the script. I was trying to do quite a lot in a single script.
For the time being, I was able to remedy the issue by changing how the attacks work, simultaneously shrinking the script and removing 3 of the 7 projectile calls.

That screen is more trouble than it's worth. It also sometimes show up when trying to load saved data, lol.

That screen is more trouble than it's worth. It also sometimes show up when trying to load saved data, lol.

It’s not more trouble than it’s worth because if it wasn’t this screen, the game would just crash or show you garbled tiles

That screen is more trouble than it's worth. It also sometimes show up when trying to load saved data, lol.

It’s not more trouble than it’s worth because if it wasn’t this screen, the game would just crash or show you garbled tiles

^ This. Having this screen gives game devs a chance to debug their games, and has already resulted in bug fixes behind the scenes as well.

That screen is more trouble than it's worth. It also sometimes show up when trying to load saved data, lol.

It’s not more trouble than it’s worth because if it wasn’t this screen, the game would just crash or show you garbled tiles

^ This. Having this screen gives game devs a chance to debug their games, and has already resulted in bug fixes behind the scenes as well.

authors: BonkersTBobcat darkhog maxoakland patrickmollohan


text:
Add getInitialState to README example

The main example in the README doesn't work, as it has no initial state for canSubmit.
This PR simply adds a getInitialState method returning canSubmit: false, as it may help newcomers have a working prototype just by copying the readme example.

Great stuff, thanks a lot :-)

Thank you, I'm enjoying using Formsy :)

authors: christianalfoni gusaiani

repo: christianalfoni/formsy-react


text:
CORS

I'm trying to make a call to the Yelp API, but keep getting the following error:
XMLHttpRequest cannot load https://api.yelp.com/v2/search?term=food&location=new york. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:3000' is therefore not allowed access. The response had HTTP status code 400.
Any suggestions on how to work around this?

You need something like this:
http://enable-cors.org/server_expressjs.html
http://stackoverflow.com/questions/18310394/no-access-control-allow-origin-node-apache-port-issue

I think you rather want to proxy that API call through express @tadwhitaker . It is Yelp not having the necessary headers to you can not do much about that. But Node can be a proxy handling the request.
So:
var request = require('request');
app.get('/yelp', function (req, res) {
  request('/api.help.com....', function (error, response) {
    res.send(response.body); // Check request API, do not remember this
  })
});

Can any-body advice me the way of handling server changes, with no rerun the server manualy,
and working with routes on it, thanks a lot !!

authors: Odywan55 armandomendivil christianalfoni tadwhitaker


text:
DSR—Cursor Position Report (CPR)

Implement: https://vt100.net/docs/vt510-rm/DSR-CPR.html

done (duplicate of other ticket that is done)

repo: christianparpart/libterminal


text:
Tmux Navigator not working with CodeWhisperer

After installing the Fig replacement, CodeWhisperer, it would seem that the Tmux Navigator no longer works alongside NeoVim.

Unfortunately this plugin seems to be incompatible with Fig (see https://github.com/christoomey/vim-tmux-navigator/issues/339 for notes and potential workarounds). As I'm not a fig user myself I've not had an opportunity to investigate to understand the root cause.

repo: christoomey/vim-tmux-navigator


text:
Event-Bus: Add optional BroadcastChannel polyfill factory

Bring your own polyfill! Adds an optional factory function to the event-bus options, which is used by the
global-broadcast and broadcast strategy to access a BroadcastChannel.
The provided id should be used as a channel name.
If no factory function is provided, the native API will be used when available.
If not available, it falls back to an no-op implementation of BroadcastChannel,
which will continue to deliver messages in the same browser context.

So quick after it came up in chat! Awesome!

authors: christoph-fricke hangtwenty

repo: christoph-fricke/xsystem


text:
Add Blur to propertyWired assertions

This PR allows the user to have wire:model.blur in their Livewire and still have assertPropertyWired return correctly.
Prior Behavior:

If a user has wire:model.blur="blur" in their livewire view
And the user writes a test with ->assertPropertyWired('blur')
The test will fail as wire:model.blur does not meet regex expression

What's Changed:

Added blur| to the regex in:
assertPropertyWired
assertPropertyNotWired

Awesome, thanks a lot 👍

Any chance you would add hacktoberfest-accepted label? :). Although I needed this personally from your repo, I may as well get credit in Hacktoberfest too!
Thanks for the package as well as the Pest course at Laravel

Sure. Like this?

authors: bretterer christophrumpel


text:
[Install issue]: installing chromadb 0.14.6 causes onnxruntime install error

What happened?
I'm using poetry to install chromadb as part of a pkg. I get this new error which didn't occur before.
Versions
Chroma 0.14.6, Python 3.11.5, MacOS 13.5.2
Relevant log output
• Installing onnxruntime (1.16.2): Failed

  RuntimeError

  Unable to find installation candidates for onnxruntime (1.16.2)

  at ~/.local/pipx/venvs/poetry/lib/python3.11/site-packages/poetry/installation/chooser.py:73 in choose_for
       69│
       70│             links.append(link)
       71│
       72│         if not links:
    →  73│             raise RuntimeError(f"Unable to find installation candidates for {package}")
       74│
       75│         # Get the best link
       76│         chosen = max(links, key=lambda link: self._sort_key(package, link))
       77│

Cannot install onnxruntime.

I get the same error!

I get the same error!

fixed by freezing onnxruntime to 1.16.1
(apparently 1.16.2 was released yesterday, so there probably wasn't a build for Apple Silicon M1)

@pchalasani, thanks for reporting this. Did you install onnxruntime manually, or do you have another dep that bumps it to >1.16.1? I just tried to install a fresh chromadb copy, and I noticed that 1.16.1 was installed, so I'm curious to understand how you ended up with 1.16.2.

@thilomichael, looking at the release in GH, it seems they have the packages but perhaps their release process failed to upload to pypi.

Thanks for tracking that down. It seems like there is an issue with their process. There is already in issue in their repository: https://github.com/microsoft/onnxruntime/issues/18378
So waiting will probably resolve this issue (that are the best kind of issues ;) )
Until then freezing onnxruntime to 1.16.1 does the trick!

@pchalasani, thanks for reporting this. Did you install onnxruntime manually, or do you have another dep that bumps it to >1.16.1? I just tried to install a fresh chromadb copy, and I noticed that 1.16.1 was installed, so I'm curious to understand how you ended up with 1.16.2.

I think it got bumped to 1.16.2 when I did a fresh install of my project poetry install or poetry update (I don’t recall).

We'll do a minor fix to pin the version until they resolve this issue. @pchalasani, thanks for finding it out.

somebody already reported this - https://github.com/microsoft/onnxruntime/issues/18378

The onnxruntime issue is resolved now.

Thanks @snnn for the speedy fix. I'll close this out.

authors: HammadB pchalasani snnn tazarov thilomichael

repo: chroma-core/chroma


text:
Chroma not retrieving collection previously stored on disk and creating it a second time from scratch each time

As above. Using the following code. After it exits I would expect the database to be persisted on disk as instructed in the code, but in practice it never finds it when the script is run a second time and embeds everything again from scratch. Inspecting the database files on disk does indicate additional records being added each time the script is run. It looks like the database was persisted on exit but for some reason those records were not retrieved when the script was re-run, even though the collection name being used ("stored_tweets" in the example below) never changes.
"""
Download your tweet archive from Twitter.
There will be a file called data/tweets.js. It will contain a single variable
assigned to an array of tweet objects.
Edit it, leave only the array and rename it to tweets.json.
This requires having chromadb and InstructorEmbedding installed via pip.
"""
from chromadb.config import Settings
#from chromadb.utils import embedding_functions
import chromadb.utils.embedding_functions as embedding_functions
import chromadb
import json
import time

dirname = "mytweets"
embeddings = embedding_functions.InstructorEmbeddingFunction()

client = chromadb.Client(Settings(chroma_db_impl="duckdb+parquet",
                                    persist_directory=dirname))

alltweets = json.load(open("tweets_test.json"))
tweets = [t['tweet'] for t in alltweets if not t['tweet']['full_text'].startswith("RT")]
total = len(tweets)
print(f"we have {total} tweets.")

coll = client.get_or_create_collection("stored_tweets", embedding_function=embeddings)
print(f"we have {coll.count()} in the collection.")

if coll.count() != total:
    i = 0
    batch_size = 10 #that's how much my gpu can do at a time

    toembed = [t["full_text"] for t in tweets]
    ids = [str(i) for i in range(total)]

    while i < len(toembed):
        coll.add(documents=toembed[i:i+batch_size], metadatas=None, ids=ids[i:i+batch_size])
        i += batch_size
        print(f"embedded: {i}")

print(f"we have {coll.count()} in the collection.")

I am also facing same issue when using with Langchain, after saving index on disk, on reloading it's not loading the embeddings. Please suggest the solution.

@arbuge @imranshabbir is this using the new retrieval langchain feature? will investigate!

@arbuge @imranshabbir is this using the new retrieval langchain feature? will investigate!

No, not using langchain at all. See code above. Playing with Chroma directly here.

here is a very minimal example on latest
test2.py
import chromadb
from chromadb.config import Settings
from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction

client = chromadb.Client(
    Settings(
        chroma_db_impl="duckdb+parquet",
        persist_directory="savedir",  # Optional, defaults to .chromadb/ in the current directory
    )
)

print(client.heartbeat())
collection = client.create_collection("test22")
collection.add(
    embeddings=[
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
    ],
    metadatas=[
        {"uri": "img1.png", "style": "style1"},
        {"uri": "img2.png", "style": "style2"},
        {"uri": "img3.png", "style": "style1"},
        {"uri": "img4.png", "style": "style1"},
        {"uri": "img5.png", "style": "style1"},
        {"uri": "img6.png", "style": "style1"},
        {"uri": "img7.png", "style": "style1"},
        {"uri": "img8.png", "style": "style1"},
    ],
    documents=["doc1", "doc2", "doc3", "doc4", "doc5", "doc6", "doc7", "doc8"],
    ids=["id1", "id2", "id3", "id4", "id5", "id6", "id7", "id8"],
)

test3.py
import chromadb
from chromadb.config import Settings
from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction

client = chromadb.Client(
    Settings(
        chroma_db_impl="duckdb+parquet",
        persist_directory="savedir"
    )
)

print(client.heartbeat())
collection = client.get_or_create_collection("test22")
print(collection.count())

logs
~/src/chroma main *9 ?4 > python test2.py                                                                              py base 05:44:50 PM
Using embedded DuckDB with persistence: data will be stored in: savedir
1680471942775868000000
No embedding_function provided, using default embedding function: SentenceTransformerEmbeddingFunction
~/src/chroma main *9 ?5 > python test3.py                                                                           4s py base 05:45:45 PM
Using embedded DuckDB with persistence: data will be stored in: savedir
1680471949575832000000
No embedding_function provided, using default embedding function: SentenceTransformerEmbeddingFunction
8

This seems to work OK... and there is a test for this.
What version are you running?

I'm using 0.3.16

can you try latest? 0.3.19? sorry about the trouble!

@arbuge i am using the langchain for uploading the documents in one class and for reading the documents in other class, so what's happening is, when i am terminating the program the read object is automatically persisting itself (i have not added any persistence call added) and overwriting the index created by the write object, and when i am running the program again, it will not find the embeddings created by write object as its already overwritten by read

Same problem here on 0.3.18. @jeffchuber's test returns count 0 from test3.py

I upgraded to 0.3.20 now but it made no difference. The issue is still there.

@skandium @arbuge can you all try running client.persist() the explicitly call saving to disk? sorry about the trouble

Thanks @jeffchuber. Modifying test2.py like this works on 0.3.20:

import chromadb
from chromadb.config import Settings
from chromadb.utils.embedding_functions import SentenceTransformerEmbeddingFunction

client = chromadb.Client(
    Settings(
        chroma_db_impl="duckdb+parquet",
        persist_directory=DB_DIR,  # Optional, defaults to .chromadb/ in the current directory
    )
)

print(client.heartbeat())
collection = client.create_collection("test22")
collection.add(
    embeddings=[
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
        [1.1, 2.3, 3.2],
        [4.5, 6.9, 4.4],
    ],
    metadatas=[
        {"uri": "img1.png", "style": "style1"},
        {"uri": "img2.png", "style": "style2"},
        {"uri": "img3.png", "style": "style1"},
        {"uri": "img4.png", "style": "style1"},
        {"uri": "img5.png", "style": "style1"},
        {"uri": "img6.png", "style": "style1"},
        {"uri": "img7.png", "style": "style1"},
        {"uri": "img8.png", "style": "style1"},
    ],
    documents=["doc1", "doc2", "doc3", "doc4", "doc5", "doc6", "doc7", "doc8"],
    ids=["id1", "id2", "id3", "id4", "id5", "id6", "id7", "id8"],
)

client.persist()

Note that the files chroma-collections.parquet and chroma-embeddings.parquet are only created in DB_DIR after the client.persist() call. Before that, it only creates an index folder.

@skandium glad to hear it!

Note that the files chroma-collections.parquet and chroma-embeddings.parquet are only created in DB_DIR after the client.persist() call. Before that, it only creates an index folder.

This behavior is expected in the current version.

This works, thanks!

Note that the files chroma-collections.parquet and chroma-embeddings.parquet are only created in DB_DIR after the client.persist() call. Before that, it only creates an index folder.

Correct, that's what was happening. Issue is resolved by adding client.persist() and those files are indeed created there. Closing this issue now as solved now.

@jeffchuber Hey, I facing the same issue even though I use client.persist().  When I re-run the app and do search on the db, getting ValueError: Collection candidates does not exist.
when i deploy the app with the database in google cloud run, it has same issue.  At first time, I can fetch or search data, later it raises ValueError: Collection candidates does not exist.
chromadb~=0.3.26
langchain==0.0.262

chromadb_client = chromadb.Client(Settings(
    chroma_db_impl='duckdb+parquet',
    persist_directory=Config.CHROMA_DB_PERSIST_DIR
))

        chromadb_client.get_or_create_collection(name=collection_name,
                                                 embedding_function=langchain_embedding_function).add(
            embeddings=embeddings, metadatas=metadata, ids=ids)
        chromadb_client.persist()

Im still facing this issue, but Chromadb Client()  does not have the persist() method anymore, also tried with PersistentClient() but its still not working.

@raibd can you share some more code/context? thanks@

@jeffchuber Actually, I think this is an issue with semantic kernel and not chromadb. I explain the issue here https://github.com/microsoft/semantic-kernel/issues/4566

Im still facing this issue, but version 0.5.0  Chromadb Client()  does not have the persist() method anymore. When using PersistentClient(), i have this log INFO     chromadb.api.segment:segment.py:189 Collection xxx is not created.

authors: arbuge imranshabbir jeffchuber liz1989 raibd sherwinwater skandium


text:
Explicit persist

@atroyn please review to assure this does what you expect.
Context: if a user deleted the client mid-script, the __del__ would not get run via garbage collection until after the program exited. In rare cases (like in tests) - you want to be able to explicitly call .persist. This pipes that through.

Tested with LangChain, works well.

authors: atroyn jeffchuber


text:
[ENH] Distributed HNSW segment respects updates/upsert/deletes

Description of changes
Summarize the changes made by this PR.

Improvements & Bug fixes

...

New functionality

...

Test plan
How are these changes tested?

[ ] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#2161  👈
#2160
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @HammadB and the rest of your teammates on  Graphite


text:
[ENH] CIP-2: Auth Providers Proposal

Description of changes
Summarize the changes made by this PR.

New functionality

Auth Provide Client and Server Side Abstractions
Basic Auth Provider

Test plan
TBD
Documentation Changes
Docs should change to describe how to use auth providers on the client and server. CIP added in docs/

Taz - we can move this to "Accepted" and merge the CIP separately. My only feedback is on the storage of credentials in plaintext, but we can refine this as we build the implementation. Congrats on landing the first community contributed CIP!

@HammadB, I agree plain text is not great. I can add support for htpasswd style file. This would give all in all 4 alternatives on how to configure basic auth.
As a side node, in cloud settings often credentials, as passed in plain text through secret injection - file and env, are generally accepted approaches for the injection.
I will make it a point to state in the docs that while credentials might not be encrypted, for prod use, we recommend using secret management (Vault, AWS KMS etc.) to inject the credentials at runtime.

@HammadB I think we're nearly there. Just a bit of cleanup for mypy errors and it should be good for merge.

I agree with @HammadB 's comments. my only other question, which I think the answer is yes, is how we can re-compose this same functionality into other middleware in the future (eg a grpc service)

I agree with @HammadB 's comments. my only other question, which I think the answer is yes, is how we can re-compose this same functionality into other middleware in the future (eg a grpc service)

@jeffchuber, I will add an abstraction for the middleware (on top of ChromaAuthMiddleware) so we can uniformly plug auth across APIs with minimal glue code specific to the API (e.i. gRPC).

@HammadB, I think this is now mostly done. Going to catch some sleep and clean up all the mypy errors. Local tests are passing.

@HammadB docs updated - https://github.com/chroma-core/docs/pull/117

Looks great! Thanks for the work!

authors: HammadB jeffchuber tazarov


text:
No guide for KEYBOARD_CUSTOM_KEY

I've read that the SDK added support for generic layouts.
Looking through the Colore docs I cannot see how the current grid implementation allows for generic layouts.
Could I get some help with this please?
As an example: the Blade and Ornata both have very different key layouts. If I reference a key using the grid, I will not get the same key on the Blade as I would on the Ornata.

The IKeyboard interface provides index-based indexers and key-based indexers for setting colors. Setting a color via a key (for example: keyboard[Key.A] = Color.Red will set the "A" key to the color red regardless of the user's current keyboard layout. This is supported as far as the SDK supports it, there have been issues in the past where it's not working properly due to bugs in the SDK/firmware.
Are you using v5.2 or v6.0 of Colore?
As for the documentation, we currently only have the "Getting Started" guide. But there's also the raw API documentation you can look at for v5.2 and v6.0.
For the keyboard specifically, check out the API docs for IKeyboard (or the v6.0 version).

authors: JoniSykes Sharparam

repo: chroma-sdk/Colore


text:
How to set onClick in element

What versions are you running?
I want rewrite onClick, not setAttribute href
$ go list -m github.com/chromedp/chromedp
<!--replace this line with the output-->
$ google-chrome --version
<!--replace this line with the output-->
$ go version
<!--replace this line with the output-->

What did you do? Include clear steps.

// not work
chromedp.SetAttributes(node.FullXPath(), map[string]string{
	"onClick": `(function(event) {
		event.preventDefault();
		window.location.href = "` + newUrl + `";
	})`,
}),
// not work
chromedp.SetJavascriptAttribute(node.FullXPath(), "onClick", `(function(event) {
	event.preventDefault();
	window.location.href = "`+newUrl+`";
})`),

What did you expect to see?
I want convert script to chromedp
const element = await page.$('a#mylink');
await element.evaluate((el, newUrl) => {
  el.onclick = function(e) {
    e.preventDefault();
    window.location.href = newUrl;
  }
}, 'https://example.com');

Thanks ^^
What did you see instead?

Try chromedp.Evaluate:
	js := `
const el = document.querySelector('a#mylink');
el.onclick = function(e) {
  e.preventDefault();
  window.location.href = "https://example.com";
}`

	if err := chromedp.Run(ctx,
		chromedp.Evaluate(js, nil),
	); err != nil {
		panic(err)
	}

authors: ZekeLu tuwibu

repo: chromedp/chromedp


text:
Albums Not Appearing

The "My Albums" view is wildly inaccurate compared to the list of Albums that I have on Google Photos itself. Some albums I have created in the past are present, but they are interspersed with 'albums' that I did not create (not by the Google Photos Assistant); usually illogical groupings of photos with a title of the date of one of the photos (but not all photos in the group). Worse, recent albums that I have shared with relatives and friends via Google Photos, do not appear in PhotoView at all! You can see the individual photos in the "timeline" album but that's not what anyone wants.
I was hoping that the Roku/Google Photos/Photoview experience would be better than the dreadful iCloud Slideshow fiasco of Apple TVs, but so far the jury is out.
Any fix in sight?
Simon

Have you reviewed the FAQ's in the app? There is a section explicitly on "Shared Albums".  Grouping which are displayed in "Albums" come directly from Google's API and should match what you see here: https://photos.google.com/u/1/albums
Again, Shared Albums will not show, it's not supported by Google and out of my control.

Yes Chris - as per the faq, I am not trying to view a lot albums that have been shared with me but an album that I have shared with others.
Photoview is not displaying all my own albums. As per my original post, nearly all albums appear along with some odd albums that have a date as a title and a few photos in them. Specifically an album I created a week or so ago with 180 pictures in it does not appear. It's it because some people have made comments about the photos which somehow changes the status of the album. Nobody else has added content.
Simon
Sent from my Verizon, Samsung Galaxy S9+ smartphone
-------- Original message --------From: Chris Taylor notifications@github.com Date: 5/31/18  2:36 PM  (GMT-05:00) To: chtaylo2/Roku-GooglePhotos Roku-GooglePhotos@noreply.github.com Cc: shrowell simon.rowell@gmail.com, Author author@noreply.github.com Subject: Re: [chtaylo2/Roku-GooglePhotos] Albums Not Appearing (#120)
Have you reviewed the FAQ's in the app? There is a section explicitly on "Shared Albums".  Grouping which are displayed in "Albums" come directly from Google's API and should match what you see here: https://photos.google.com/u/1/albums
Again, Shared Albums will not show, it's not supported by Google and out of my control.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
{"@context":"http://schema.org","@type":"EmailMessage","potentialAction":{"@type":"ViewAction","target":"https://github.com/chtaylo2/Roku-GooglePhotos/issues/120#issuecomment-393632773","url":"https://github.com/chtaylo2/Roku-GooglePhotos/issues/120#issuecomment-393632773","name":"View Issue"},"description":"View this Issue on GitHub","publisher":{"@type":"Organization","name":"GitHub","url":"https://github.com"}}
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/chtaylo2/Roku-GooglePhotos","title":"chtaylo2/Roku-GooglePhotos","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/chtaylo2/Roku-GooglePhotos"}},"updates":{"snippets":[{"icon":"PERSON","message":"@chtaylo2 in #120: Have you reviewed the FAQ's in the app? There is a section explicitly on "Shared Albums".  Grouping which are displayed in "Albums" come directly from Google's API and should match what you see here: https://photos.google.com/u/1/albums\r\n\r\nAgain, Shared Albums will not show, it's not supported by Google and out of my control. "}],"action":{"name":"View Issue","url":"https://github.com/chtaylo2/Roku-GooglePhotos/issues/120#issuecomment-393632773"}}}
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"hideOriginalBody": "false",
"originator": "37567f93-e2a7-4e2a-ad37-a9160fc62647",
"title": "Re: [chtaylo2/Roku-GooglePhotos] Albums Not Appearing (#120)",
"sections": [
{
"text": "",
"activityTitle": "Chris Taylor",
"activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@chtaylo2",
"facts": [
]
}
],
"potentialAction": [
{
"name": "Add a comment",
"@type": "ActionCard",
"inputs": [
{
"isMultiLine": true,
"@type": "TextInput",
"id": "IssueComment",
"isRequired": false
}
],
"actions": [
{
"name": "Comment",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n"commandName": "IssueComment",\n"repositoryFullName": "chtaylo2/Roku-GooglePhotos",\n"issueId": 120,\n"IssueComment": "{{IssueComment.value}}"\n}"
}
]
},
{
"name": "Close issue",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n"commandName": "IssueClose",\n"repositoryFullName": "chtaylo2/Roku-GooglePhotos",\n"issueId": 120\n}"
},
{
"targets": [
{
"os": "default",
"uri": "https://github.com/chtaylo2/Roku-GooglePhotos/issues/120#issuecomment-393632773"
}
],
"@type": "OpenUri",
"name": "View on GitHub"
},
{
"name": "Unsubscribe",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n"commandName": "MuteNotification",\n"threadId": 341112379\n}"
}
],
"themeColor": "26292E"
}

PhotoView shows my ARCHIVE ALBUM, which contains a lot of odd, unwanted images, and not my carefully edited CURRENT ALBUMS. Am I doing something wrong? Us there a way to fix this? Thanks, Chris

@mtuman - I'm only aware of the ability to archive photos, not albums. Where are you seeing the ability to do this?  https://photos.google.com/albums is the web front end for albums on Google. If you click the three DOTS beside and select "Delete" it will delete that album, removing it from your account. .. and you will not see it on the Roku app.

Chris. These are the photos I now see in PhotoView:
https://get.google.com/albumarchive/105245055429849927852
When I want to see the photos here:
https://photos.google.com/albums
Hope I am making sense. The photos under ALBUMARCHIVE are far more
disorganized, & not a place I edit - unlike those under ALBUMS. Hence, I am
now unable to use PhotoView.
Thanks, Chris
On Fri, Jul 6, 2018, 12:56 AM Chris Taylor notifications@github.com wrote:

@mtuman https://github.com/mtuman - I'm only aware of the ability to
archive photos, not albums. Where are you seeing the ability to do this?
https://photos.google.com/albums is the web front end for albums on
Google. If you click the three DOTS beside and select "Delete" it will
delete that album, removing it from your account. .. and you will not see
it on the Roku app.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/chtaylo2/Roku-GooglePhotos/issues/120#issuecomment-402925526,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Am-uHH7APvC0F2IHwdYzVU_MZ0xhdBT2ks5uDu3kgaJpZM4UVPY7
.

I'm running into a similar issue.  If I create a "shared album" it will not show even if I was the one that created it.   If I create an album and then share it it will show though if the other person adds photos. But the photos added by the other person will not show.
I had to get the person to send me the files and add them myself before they would show.

authors: chtaylo2 mtuman rhaskett shrowell


text:
wttr.in service down

The URL does not fetch anything anymore curl nor via the website...

Check now please

Works!

authors: chubin mukundzare


text:
如何安装到已存在的viterpress站点

你好，我有个已经存在的使用vitepress站点，想为站点添加blog页面，没大看明白你写的操作文档。
npm install -D vitepress @chunge16/vitepress-blogs-theme

之后需要怎么操作呢？文档讲的不是很清楚，还需要添加什么配置么？希望指导一下，谢谢

这里有个使用模板，你可以看看[vitepress-blogs-theme-template.png]
2024年2月4日 20:59，Jaaron Ge @.***> 写道：
你好，我有个已经存在的使用vitepress站点，想为站点添加blog页面，没大看明白你写的操作文档。
npm install -D vitepress @chunge16/vitepress-blogs-theme
之后需要怎么操作呢？文档讲的不是很清楚，还需要添加什么配置么？希望指导一下，谢谢
—
Reply to this email directly, view it on GitHubhttps://github.com/chunge16/vitepress-blogs-theme/issues/6, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD73NFAKH3ADD7DVGYXRGKDYR6A3NAVCNFSM6AAAAABCY3D5WKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYTOMBYGYYDKOA.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

已经存在的使用vitepress站点，安装该插件后，具体可以参考 模板库

根目录新建 blog和里面的文件
添加多个配置文件

这里有个使用模板，你可以看看[vitepress-blogs-theme-template.png]
2024年2月4日 20:59，Jaaron Ge @.***> 写道：
你好，我有个已经存在的使用vitepress站点，想为站点添加blog页面，没大看明白你写的操作文档。
npm install -D vitepress @chunge16/vitepress-blogs-theme
之后需要怎么操作呢？文档讲的不是很清楚，还需要添加什么配置么？希望指导一下，谢谢
—
Reply to this email directly, view it on GitHubhttps://github.com/chunge16/vitepress-blogs-theme/issues/6, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD73NFAKH3ADD7DVGYXRGKDYR6A3NAVCNFSM6AAAAABCY3D5WKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYTOMBYGYYDKOA.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


text:
桌面Linux 有乱码

您好，我按照您的readme ,在vimrc中set guifont = Droid\ Sans\ Mono\ for\ Powerline\ Nerd\ Font Complete,
设置完之后，发现还是有乱码的存在。注：我的是Ubuntu 14.04，在~/.vimrc 下设置的

@lykimage 在终端点击右键设置字体，你不需要在vimrc里面加，我已经在vimrc里面设置了

您好，首先谢谢您的回复。我右键终端profile 设置字体当中有Droid\ Sans\ Mono等字体，但是没有找到您的这个字体。在~/fonts下有这个字体。后缀是otf的，查阅相关资料，没搞清楚怎么安装这个，只好在问问您

对了，我用这个命令 fc-cache 执行过了

@lykimage vimplus的安装脚本会帮你执行fc-cache的，你的ubuntu系统是64位还是32位的？

64位的

照着这篇文章设置字体即可- http://blog.csdn.net/qq_30115765/article/details/52623935

如果没字体的话，可以照着这个文档安装 - https://github.com/ryanoasis/nerd-fonts

authors: chxuan lykimage zhhuabj

repo: chxuan/vimplus


text:
[Enhancement] WebSocket API and Web Remote

WebSocket API
WebSocket API documentation
Web Remote
If these take too long save for a future patch after 3.0.0 releases.

This app will be sunseted very soon and we are NOT maintaining this app
unless of a security issue. We encourage you to try Cider, our new music app built from scratch. Issues filled on that repository will be seen and will be acted upon. Please join our discord and see the #nightly-builds channel for a current build.
Cider can be found here

authors: booploops quacksire

repo: ciderapp/Apple-Music-Electron


text:
kubeProxyReplacement=false does not work and doesn't detect that kube-proxy is already installed

cilium-cli version: v0.16.21
I'm attempting to upgrade from cilium v1.16.3 to v1.16.4.
I get the following:
ℹ  Using Cilium version 1.16.4
🔮 Auto-detected cluster name: fedoracluster
🔮 Auto-detected kube-proxy has not been installed
ℹ Cilium will fully replace all functionalities of kube-proxy

My cluster disagrees:
kubectl get po -A | grep kube-proxy
kube-system                       kube-proxy-fedora                                      1/1     Running     0             21h

Trying to override this doesn't work either:
cilium-cli upgrade --set kubeProxyReplacement=false
It gives the same output.
I think this indicates that there are at least 2 issues; that disabling kube-proxy replacement doesn't work, and that auto-detection doesn't work.
Note: it seems as though the install target also has the same issue.

As additional info, the cluster is running rke2 (with the default CNI disabled).

repo: cilium/cilium-cli


text:
Restart of cilium pods causes restart of hubble-relay pod and loss of metrics

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
When I restart the cilium pods the hubble-relay pod also restarts with the following log:
time="2024-06-13T23:33:19Z" level=warning msg="Error while receiving peer change notification; will try again after the timeout has expired" connection timeout=30s error="rpc error: code = Unavailable desc = error reading from server: EOF" subsys=hubble-relay
time="2024-06-13T23:33:48Z" level=info msg="Stopping server..." subsys=hubble-relay
time="2024-06-13T23:33:48Z" level=info msg="Server stopped" subsys=hubble-relay

The previously existing metrics (e.g. hubble_drop_total) are lost.
Also tested with rollout restart and showed the same behavior with:
updateStrategy.rollingUpdate.maxUnavailable: 1 (with 2 nodes)
minReadySeconds: 10
Cilium Version
1.15.4, 1.15.6
1.13.4 and 1.14.12 do not exhibit this behavior
Kernel Version
5.15.0-1064-azure
Kubernetes Version
Tested on AKS. K8s 1.28.9
Regression
No response
Sysdump
No response
Relevant log output
time="2024-06-13T19:22:14Z" level=info msg="Stopping server..." subsys=hubble-relay
time="2024-06-13T19:22:14Z" level=info msg="Server stopped" subsys=hubble-relay

Anything else?
No response
Cilium Users Document

[ ] Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

[X] I agree to follow this project's Code of Conduct

It seems to me that the hubble-relay is killed because it becomes "unhealthy" during the agent upgrade. The health check asserts that the hubble relay is connected to an agent (any of then, it uses a service). We don't set a specific liveness probe interval, so the default 10s is used and the default failureThreshold value of 3. However, the default period before the relay attempts to reconnect is 30s as mentioned by the warning log. So the relay only tries to re-connect when its about to be killed for being unhealthy.
I would recommend decreasing .Values.hubble.relay.retryTimeout and perhaps .Values.hubble.relay.dialTimeout so that the relay re-connects faster after losing its original connection so it has more chance to become healthy before its restarted itself.

Great, thank you. Exactly the same recommendations as @soggiest

15s for retryTimeout and dialTimeout resolved the issue

15s for relay.retryTimeout and relay.dialTimeout resolved the issue

Nice, would you mind opening a PR to fix this? Maybe pick 20s as the timeout values?

Will do

https://github.com/cilium/cilium/pull/33894 should resolve this. If it doesn't, then we can re-open this. If it doesn't work, I believe the best path forward is to simply not tie the relay health to the health of it's peers at all, meaning not relying on peer status for hubble relay health, and relying on rely to appropriately handle the case where it has no peers by responding with an gRPC error of "unavailable" or similar.

authors: bdecoste chancez dylandreimerink rolinh

repo: cilium/cilium


text:
RFC: Spiffe Integration

What is this?
This is a request for comments PR regarding the Cilium-SPIFFE integration that has been proposed in this design document. This PR covers the "Identity Generation Hardening" part of such proposal. We already have some prototypes of the "Upgrading connections to mTLS" part but we prefer start discussing only the former one to keep it more scoped.
How to test?
If you want to test it by yourself you can follow these steps. Otherwise you can watch this recorded demo.

Download, compile and install cilium-agent based on this PR
Relax labels validations by updating the CNP CRD.

$ kubectl apply -f examples/crds/v2/ciliumnetworkpolicies.yaml

Add the --enable-spiffe flag to cilium-agent
Deploy Spire. I prepared a yaml manifest with all components needed to deploy it. It's based on the spire k8s quickstart tutorial but uses a custom spire-agent image that implements a POC of the spire privileged API.

$ kubectl apply -f https://gist.githubusercontent.com/mauriciovasquezbernal/820e8e11fa546c2b4e04a3a9105b47ed/raw/017ea2d66cdd1e7227ebde57f39d82ca9f1eb0f8/spire.yaml

Check that spire pods are running

$ kubectl get pods -n spire
NAME                READY   STATUS    RESTARTS   AGE
spire-agent-rqsp9   1/1     Running   0          14s
spire-server-0      1/1     Running   0          14s

Create some registration entries. This  creates some registration entries for testing.

$ ./create_registration_entries.sh

Restart cilium-agent. (This is something that should be improved in the code to avoid this step)

$ sudo systemctl restart cilium

Create some pods

$ kubectl run podubuntu --image=ubuntu:latest -- sleep 1000000
$ kubectl run podpraqma --image=praqma/network-multitool:latest -- sleep 100000

Restrict all egress traffic

$ kubectl apply -f https://gist.githubusercontent.com/mauriciovasquezbernal/820e8e11fa546c2b4e04a3a9105b47ed/raw/017ea2d66cdd1e7227ebde57f39d82ca9f1eb0f8/deny-all-egress.yaml

Check that ping is failing

$ kubectl exec podpraqma -- ping <podubuntu IP>

Deploy image based policy that allows those two pods to talk to each other

$ kubectl apply -f https://gist.githubusercontent.com/mauriciovasquezbernal/820e8e11fa546c2b4e04a3a9105b47ed/raw/017ea2d66cdd1e7227ebde57f39d82ca9f1eb0f8/image-based.yaml

Ping now works

$ kubectl exec podpraqma -- ping <podubuntu IP>
PING 10.11.0.173 (10.11.0.173) 56(84) bytes of data.
64 bytes from 10.11.0.173: icmp_seq=1 ttl=63 time=0.124 ms

TODO

[ ] Update code once Spire "privileged" PR is ready (https://github.com/spiffe/spire/issues/2246 & https://github.com/spiffe/spire-api-sdk/pull/7)
[ ] Tests (both unit and integration)
[ ] Documentation
[ ] How to deploy spire. Create helm templates?, leave it up to the user?
[ ] Improve label validation (undo validation relax done for this POC)
[ ] How to initialize the spiffe watcher in a lazy way (avoid trying to connect to it when spire is not deployed yet)
[ ] Rebase once https://github.com/cilium/cilium/pull/15123 is merged
[ ] Rebase once https://github.com/cilium/cilium/pull/16585 is merged

ref https://github.com/cilium/cilium/issues/4016
cc @nyrahul @jrajahalme @jrfastab @navarrothiago @rscampos

@mauriciovasquezbernal Go mod issues need fixing:
diff --git a/go.mod b/go.mod
index 258288a..d99b24e 100644
--- a/go.mod
+++ b/go.mod
@@ -118,10 +118,10 @@ replace (
 	github.com/miekg/dns => github.com/cilium/dns v1.1.4-0.20190417235132-8e25ec9a0ff3
 	github.com/optiopay/kafka => github.com/cilium/kafka v0.0.0-20180809090225-01ce283b732b

-	go.universe.tf/metallb => github.com/cilium/metallb v0.1.1-0.20210607221240-b4c60b959dd7
-
 	github.com/spiffe/spire-api-sdk => github.com/kinvolk/spire-api-sdk v1.0.0-pre.0.20210610130402-7108593e3cec

+	go.universe.tf/metallb => github.com/cilium/metallb v0.1.1-0.20210607221240-b4c60b959dd7
+
 	// Using private fork of controller-tools. See commit msg for more context
 	// as to why we are using a private fork.
 	sigs.k8s.io/controller-tools => github.com/christarazi/controller-tools v0.3.1-0.20200911184030-7e668c1fb4c2
diff --git a/go.sum b/go.sum
index 3daa888..3d5ba64 100644
--- a/go.sum
+++ b/go.sum
@@ -627,8 +627,6 @@ github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0Lh
 github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 h1:DowS9hvgyYSX4TO5NpyC606/Z4SxnNYbT+WX27or6Ck=
 github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
 github.com/keybase/go-ps v0.0.0-20190827175125-91aafc93ba19/go.mod h1:hY+WOq6m2FpbvyrI93sMaypsttvaIL5nhVR92dTMUcQ=
-github.com/kinvolk/spire-api-sdk v1.0.0-pre.0.20210531191218-f6af4bcf4c8e h1:TxTUdiVF31jdA2SrUdJOlH3lyIrWcaY1DlAjr6ui7BE=
-github.com/kinvolk/spire-api-sdk v1.0.0-pre.0.20210531191218-f6af4bcf4c8e/go.mod h1:2wSTZ6oEnKqI3uBST05Mmm751+yoHEvgxomYKYOQ6Ko=
 github.com/kinvolk/spire-api-sdk v1.0.0-pre.0.20210610130402-7108593e3cec h1:hBk3Ug1Vo+IZ0Yjkwtze8eyX0NS9l9SbtA4+Uhf67q4=
 github.com/kinvolk/spire-api-sdk v1.0.0-pre.0.20210610130402-7108593e3cec/go.mod h1:2wSTZ6oEnKqI3uBST05Mmm751+yoHEvgxomYKYOQ6Ko=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 23b9e8c..096e596 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -650,7 +650,7 @@ go.etcd.io/etcd/client/v3/credentials
 go.etcd.io/etcd/client/v3/internal/endpoint
 go.etcd.io/etcd/client/v3/internal/resolver
 go.etcd.io/etcd/client/v3/yaml
-# go.mongodb.org/mongo-driver v1.4.4
+# go.mongodb.org/mongo-driver v1.4.6
 go.mongodb.org/mongo-driver/bson
 go.mongodb.org/mongo-driver/bson/bsoncodec
 go.mongodb.org/mongo-driver/bson/bsonoptions
@@ -673,8 +673,8 @@ go.uber.org/zap/internal/bufferpool
 go.uber.org/zap/internal/color
 go.uber.org/zap/internal/exit
 go.uber.org/zap/zapcore
-# go.universe.tf/metallb v0.9.6 => github.com/cilium/metallb v0.1.1-0.20210607221240-b4c60b959dd7
 go.uber.org/zap/zapgrpc
+# go.universe.tf/metallb v0.9.6 => github.com/cilium/metallb v0.1.1-0.20210607221240-b4c60b959dd7
 ## explicit
 go.universe.tf/metallb/pkg/allocator
 go.universe.tf/metallb/pkg/allocator/k8salloc
@@ -1295,6 +1295,6 @@ sigs.k8s.io/structured-merge-diff/v4/value
 sigs.k8s.io/yaml
 # github.com/miekg/dns => github.com/cilium/dns v1.1.4-0.20190417235132-8e25ec9a0ff3
 # github.com/optiopay/kafka => github.com/cilium/kafka v0.0.0-20180809090225-01ce283b732b
-# go.universe.tf/metallb => github.com/cilium/metallb v0.1.1-0.20210607221240-b4c60b959dd7
 # github.com/spiffe/spire-api-sdk => github.com/kinvolk/spire-api-sdk v1.0.0-pre.0.20210610130402-7108593e3cec
+# go.universe.tf/metallb => github.com/cilium/metallb v0.1.1-0.20210607221240-b4c60b959dd7
 # sigs.k8s.io/controller-tools => github.com/christarazi/controller-tools v0.3.1-0.20200911184030-7e668c1fb4c2
please run 'go mod tidy && go mod vendor', and submit your changes
Error: Process completed with exit code 1.

Looks like generated deepcopy functions do not match the change to label regexes. However, we should expand the definition of the regexes to capture the allowed spiffe labels as well, maybe using the OR syntax so that non-spiffe labels would still be validated using the old rergex?
Ungenerated deepcopy source code:
diff --git a/pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml b/pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml
index bf0fc66..313ee45 100644
--- a/pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml
+++ b/pkg/k8s/apis/cilium.io/client/crds/v2/ciliumnetworkpolicies.yaml
@@ -181,7 +181,7 @@ spec:
                               description: MatchLabelsValue represents the value from
                                 the MatchLabels {key,value} pair.
                               maxLength: 63
-                              #pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
+                              pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
                               type: string
                             description: matchLabels is a map of {key,value} pairs.
                               A single {key,value} in the matchLabels map is equivalent
@@ -962,7 +962,7 @@ spec:
                               description: MatchLabelsValue represents the value from
                                 the MatchLabels {key,value} pair.
                               maxLength: 63
-                              #pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
+                              pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
                               type: string
                             description: matchLabels is a map of {key,value} pairs.
                               A single {key,value} in the matchLabels map is equivalent
@@ -2380,7 +2380,7 @@ spec:
                                 description: MatchLabelsValue represents the value
                                   from the MatchLabels {key,value} pair.
                                 maxLength: 63
-                                #pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
+                                pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
                                 type: string
                               description: matchLabels is a map of {key,value} pairs.
                                 A single {key,value} in the matchLabels map is equivalent
@@ -3176,7 +3176,7 @@ spec:
                                 description: MatchLabelsValue represents the value
                                   from the MatchLabels {key,value} pair.
                                 maxLength: 63
-                                #pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
+                                pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$
                                 type: string
                               description: matchLabels is a map of {key,value} pairs.
                                 A single {key,value} in the matchLabels map is equivalent
Please run make generate-k8s-api and submit your changes
Error: Process completed with exit code 1.

I'm switching back to draft until the basic tests have been fixed. Feel free to mark as ready again once they are fixed or if you think it doesn't make sense to fix them first.

Looks like generated deepcopy functions do not match the change to label regexes. However, we should expand the definition of the regexes to capture the allowed spiffe labels as well, maybe using the OR syntax so that non-spiffe labels would still be validated using the old rergex?

I updated the PR to have a proper SPIFFE ID validation in the code. The validation in the CRD is still missing. I added it to the TODO list in the PR description.

I'm switching back to draft until the basic tests have been fixed. Feel free to mark as ready again once they are fixed or if you think it doesn't make sense to fix them first.

Sure.
Probably I wasn't clear enough about the purpose of this PR. Before diving into the implementation details I'd like to get general feedback on this approach, specifically the following points. Once we agree this path makes sense we can discuss all the details.

Does it make sense to use the SPIFFE ID as the label key in the CNP letting the value empty? For instance:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
  endpointSelector:
    matchLabels:
      spiffe://example.org/image/praqma/network-multitool: ""
  egress:
  - toEndpoints:
    - matchLabels:
        spiffe://example.org/image/ubuntu: ""

I added a new enable-spiffe flag in cilium-agent. I think it should be off by default, shouldn't it?

I added a new enable-spiffe flag in cilium-agent. I think it should be off by default, shouldn't it?

Yes :+1:

Does it make sense to use the SPIFFE ID as the label key in the CNP letting the value empty? For instance:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
 endpointSelector:
    matchLabels:
      spiffe://example.org/image/praqma/network-multitool: ""
  egress:
  - toEndpoints:
    - matchLabels:
        spiffe://example.org/image/ubuntu: ""

If we contrast this with the way that Cilium integrates objects from other external systems like Kubernetes, I would expect would be to have a dedicated key like io.cilium.spiffe.id and specify the SPIFFE id in the value, like this:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
  endpointSelector:
    matchLabels:
      io.cilium.spiffe.id: spiffe://example.org/image/praqma/network-multitool
  egress:
  - toEndpoints:
    - matchLabels:
        io.cilium.spiffe.id: spiffe://example.org/image/ubuntu

Here's a link to some kubernetes constructs being used in policy:
https://docs.cilium.io/en/latest/policy/kubernetes/#multi-cluster

I added a new enable-spiffe flag in cilium-agent. I think it should be off by default, shouldn't it?

Yes, at least initially I think this makes sense.

Does it make sense to use the SPIFFE ID as the label key in the CNP letting the value empty? For instance:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
 endpointSelector:
    matchLabels:
      spiffe://example.org/image/praqma/network-multitool: ""
  egress:
  - toEndpoints:
    - matchLabels:
        spiffe://example.org/image/ubuntu: ""

If we contrast this with the way that Cilium integrates objects from other external systems like Kubernetes, I would expect would be to have a dedicated key like io.cilium.spiffe.id and specify the SPIFFE id in the value, like this:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
  endpointSelector:
    matchLabels:
      io.cilium.spiffe.id: spiffe://example.org/image/praqma/network-multitool
  egress:
  - toEndpoints:
    - matchLabels:
        io.cilium.spiffe.id: spiffe://example.org/image/ubuntu

@joestringer  in this case perhaps would be better as:
     - matchLabels:
         io.cilium.spiffe.id: example.org/image/ubuntu

(just to avoid duplicating spiffe

Here's a link to some kubernetes constructs being used in policy:
https://docs.cilium.io/en/latest/policy/kubernetes/#multi-cluster

I added a new enable-spiffe flag in cilium-agent. I think it should be off by default, shouldn't it?

Yes, at least initially I think this makes sense.

If we contrast this with the way that Cilium integrates objects from other external systems like Kubernetes, I would expect would be to have a dedicated key like io.cilium.spiffe.id and specify the SPIFFE id in the value, like this:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
  endpointSelector:
    matchLabels:
      io.cilium.spiffe.id: spiffe://example.org/image/praqma/network-multitool
  egress:
  - toEndpoints:
    - matchLabels:
        io.cilium.spiffe.id: spiffe://example.org/image/ubuntu

@joestringer I think it doesn't work because it's possible for an endpoint to have multiple spiffe-based labels. For instance, I don't think we are able to define the following policy using that approach.
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
  endpointSelector:
    matchLabels:
      spiffe://example.org/image/praqma/network-multitool: ""
  egress:
  - toEndpoints:
    - matchLabels:
        spiffe://example.org/image/ubuntu: ""
        spiffe://example.org/image/alpine: ""

Hi! I have been working with the AccuKnox folks (@nyrahul @navarrothiago @rscampos) on this PR and they are going to take over the task. I'll still be around to reply any questions and participate in the discussions.

@mauriciovasquezbernal are you saying that a single workload may have multiple SPIFFE IDs associated and you want to match that the peer workload has both SPIFFE IDs to be able to allow the traffic? The example you provided is a bit confusing because it seems to specify SPIFFE IDs with OS images, but presumably a single workload can only have one OS image.

@mauriciovasquezbernal are you saying that a single workload may have multiple SPIFFE IDs associated and you want to match that the peer workload has both SPIFFE IDs to be able to allow the traffic? The example you provided is a bit confusing because it seems to specify SPIFFE IDs with OS images, but presumably a single workload can only have one OS image.

I think even if a workload/endpoint may have multiple SPIFFE IDs, I don't see how these multiple SPIFFE IDs could be used in the same rule since the connection/session will use/have just a single associated SPIFFE ID.

:+1: Then if you're trying to denote "allow SPIFFE ID X or SPIFFE ID Y", I think it could be done through two separate rules:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "image-based"
spec:
  endpointSelector:
    matchLabels:
      io.cilium.spiffe.id: example.org/image/praqma/network-multitool
  egress:
  - toEndpoints:
    - matchLabels:
        io.cilium.spiffe.id: example.org/image/ubuntu
  - toEndpoints:
    - matchLabels:
        io.cilium.spiffe.id: example.org/image/alpine

Guys, first of all, thank you for all the suggestions. We did some tests based on io.cilium.spiffe.id. In order to use io.cilium.spiffe.id: example.org/ns/default/sa/default in CNP, we have to create a label for the endpoint using io.cilium.spiffe.id as a fix key and example.org/ns/default/sa/default as a value, right?
When we do so, we are not able of having more than one Spiffe ID for an endpoint at a given time once the io.cilium.spiffe.id assume only one value. This is why it was defined as a key in first place. Would be a problem?
Even the endpoint having more than one Spiffe ID label, we just need one Spiffe ID label to do the authz as follows:
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "spiffe-based"
spec:
  endpointSelector:
    matchLabels:
      spiffe://example.org/ns/default/sa/default: ""
  egress:
  - toEndpoints:
    - matchLabels:
        spiffe://example.org/ns/default/sa/foo2: ""
  - toEndpoints:
    - matchLabels:
        spiffe://example.org/pod-name/podfoo3: ""
        spiffe://example.org/ns/default/sa/foo3: ""

Here I just use an example based on these selectors:namescape, service account and pod-name.

We discussed this during the community meeting today. I had previously raised the question above whether it's possible to associate multiple SPIFFE IDs with a single endpoint and at least above in the PR I didn't get a clear answer. If this is the case, then the only way to encode this into labels would be to put the entire spiffe id into the key. The discussion during the meeting implied that multiple SPIFFE IDs may be associated with a single endpoint, but is there an upstream SPIFFE reference for this so we can better understand that model? Specifically something like a concepts document that describes the relationship between SPIFFE IDs and endpoints/connections?

but is there an upstream SPIFFE reference for this so we can better understand that model? Specifically something like a concepts document that describes the relationship between SPIFFE IDs and endpoints/connections?

@joestringer here is https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/

I couldn't see an explicit declaration of how many SPIFFE IDs may be associated with a single workload in the above concepts page. Maybe "A SPIFFE ID is a string that uniquely and specifically identifies a workload. " is supposed to imply the relationship but it's quite vague.
However, upon further digging into the SPIFFE Workload API, I did find the underlying gRPC protocol definition:
https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE_Workload_API.md#521-fetchx509svid

The X509SVIDResponse response consists of a mandatory svids field, which MUST contain one or more X509SVID messages (one for each identity granted to the client). The crl and federated_bundles fields are optional.

The "one for each identity granted to the client" seems to imply that a single client (by which I assume this is a SPIFFE workload, equivalent to pod in the model here) can have multiple identities.
The section on Default Identity seems to go further and describe the notion that it's up to the workload to understand the availability of multiple identities and how it should use those identities, or otherwise default to using the first one that is provided from the SPIFFE workload API:
https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE_Workload_API.md#53-default-identity
So yes, it seems that multiple SPIFFE IDs can be associated with a single workload.

I have another question:
Let's say that we set up the spiffe identities using this system. We associate ids with workloads by encoding the spiffe ID as a key for labels associated with the endpoint. What happens if someone then manually creates a new pod with labels including a spiffe ID, something like the below?
apiVersion: v1
kind: Pod
metadata:
  name: example
  "spiffe://example.org/image/ubuntu": ""
spec:
  containers:
    - name: busybox
      image: busybox:1.25
      command:
        - sleep
        - 10000

Hello @joestringer , thank you for the observations. All good points. I did some validation on this. Please check the slides for more information. To summarize:
1- We change label source for spiffe ID labels from “k8s” to “spiffe” to avoid error during the k8s validation. Doing so, we are no more constrained with the size limits of label key.
2- Using a different source for spiffe labels, we also avoid the risk of someone specifying that labels from userspace. Details in the slides.
Note that I have validated these points using actual code changes.

I've requested access to the doc. I think in terms of the label source, that part makes sense - then based on the source we can know that the SPIFFE ID comes from a legitimate SPIFFE server. What I'm still missing is how we ensure that the label match in the CiliumNetworkPolicy only matches on the SPIFFE-sourced labels, and not on k8s-sourced labels.

Actually, on second thought for:

1- We change label source for spiffe ID labels from “k8s” to “spiffe” to avoid error during the k8s validation. Doing so, we are no more constrained with the size limits of label key.

This doesn't change the limits on the LabelSelector in the policy. If SPIFFE IDs are longer, will it still be possible to match on them using policy?

@joestringer
I gave you permission to access the doc, could you try again please?
Let me split the comment in 3 parts:
Part 1. Use keyword spiffe as source in CNP:
After read your comment, I tried it and worked as well. I used spiffe as source in CiliumNetworkPolicy. I saw these two examples here and here. Is it a correct way of using it?
In this way, the policy is going to match just labels with spiffe source.
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "spiffe-based"
spec:
  endpointSelector:
    matchLabels:
      spiffe:io.spiffe.id.example.org/sa/foo1: ""
  egress:
  - toEndpoints:
    - matchLabels:
        spiffe:io.spiffe.id.example.org/sa/foo3: ""

Part 2. Label manually created by the user:
A user can manually create a label as follows, but is not possible to specify the source of the label, right? (Like this: "spiffe:io.spiffe.id.example.org/foo3": "").
Doing so, If a user try to create a label, the default source is going to be k8s. In this way the CNP used in Part 1 is not going to match.
apiVersion: v1
kind: Pod
metadata:
  name: podfoo3-personification
  labels:
    io.spiffe.id.example.org/foo3: ""
spec:
  containers:
  - name: podfoo3
    image: praqma/network-multitool
    command: ["bin/sh", "-c", "sleep 10000"]

Part 3. CNP label validation:
Using any source,k8s or spiffe in our example, a validation in going to happen in 2 places for our use case: 1) validateLabelKey() selector.go; and 2) ValidateLabelName() validation.go. Because Spiffe ID may use multiples "/", this is not a "qualified name" for Kubernetes defined here at IsQualifiedName() - this func is called by validateLabelKey() and ValidateLabelName(). This is why Mauricio changed a little the validation in these 2 places: selector.go and validation.go.
Based on these changes, the len of name part of the label is not checked. I tried with a big Spiffe ID (pod creation and CNP) and worked.
Any feedback is welcome, thank you for highlighting these points :D

:+1: Part 1 / 2 makes sense.
I think for Part 3 you've addressed technically how the validation and internal storage of the SPIFFE IDs would work though I'd probably have to take a closer look to reason about how it's actually working against different inputs. I didn't yet quite understand how the policy can match on bigger SPIFFE IDs though, if you place a long (say ~100 character) SPIFFE match in the policy under matchLabels, can Cilium successfully create the LabelSelector based on that input so that it can actually match on the spiffe label which is internally associated with an identity?

hey @joestringer, thank you for the answer
Yeah, it's possible to have a long Spiffe ID label. I recorded a PoC here. I think after the validation (that we bypassed), Cilium allows a long Spiffe ID to be used.

Hey @joestringer and folks,
I'm going to send another PR from my Github account since the actual PR was sent by @mauriciovasquezbernal. Thank you @mauriciovasquezbernal for all the work.
I'm going to maintain the Mauricio's description and commit the new changes made on Spiffe ID label.
Could you folks please close this one? I'll refer this #16626 in the new PR.
cc @mauriciovasquezbernal @nyrahul @navarrothiago
Tks :)

authors: aanm joestringer jrajahalme mauriciovasquezbernal navarrothiago nyrahul pchaigno rscampos


text:
Cilium monitor filters by multiple types

fixes #2869

I'v just merged #2864

authors: nebril tgraf


text:
bpf: add multicast in MAX_OVERLAY_OPTIONS

Fix mismatched brackets in multicast code and add ENABLE_MULTICAST in MAX_OVERLAY_OPTIONS to check such mismatches during compile time in future.

/test

ci-ginkgo: https://github.com/cilium/cilium/issues/32075

authors: harsimran-pabla julianwiedmann


text:
helm: clarify text for serviceNoBackendResponse

The kernel requirement is actually >= 5.2. And as RHEL 8.6 also has the relevant support, we can just remove the text that describes the limitation.

(discussed in https://github.com/cilium/cilium/pull/28157/files#r1818545597)

/test


text:
Bugtool: Add gops output

Add gops output for cilium agent to know what happens to the agent in
case of something wrong.

Fixes #3608
Signed-off-by: Eloy Coto eloy.coto@gmail.com

test-me-please


text:
🛑 Collection Analysis Datasette is down

In f981a81, Collection Analysis Datasette (https://collection-analysis.cincy.pl/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Collection Analysis Datasette is back up in bff2045 after 9 minutes.

repo: cincinnatilibrary/uptime-reports


text:
The content type must be a string

When I keep the sonarqube url  link in the action it's throwing as below. can you please look into this earlier it was working.
{"error":"The content type must be a string at least one character in length.":
example:
  - name: Send success notification
    if: always()
    uses: cinotify/github-action@main
    with:
      to: 'XXXXXXXXX@gmail.com'
      subject: 'testone'
      body: |
        'Test  '
        Job Details:
        Job Name: ${{ github.job }}
        Workflow: ${{ github.workflow }}
        Commit: ${{ github.sha }}
        Repository: ${{ github.repository }}
        type: 'text/html'
        SonarQube_URL: 'https://xxxxxxx/dashboard?branch=staging&id=xxxx_xxx_AYx87CNo85rvmfbTxLEv'

sure - it looks like type is nested under body - is that intentional? I also have set this up to fallback to text/plain if a content type is not provided - https://github.com/cinotify/api/blob/ac25e35fa9501ba81c8b2a6e2ee74124d4351c1e/src/mail.js#L6

@jshawl  I am also receiving the same error and has been fine until a few days ago.  Thanks!

sorry about that @derrick-ejan-rs @chandrasekhar7 - I have pushed a fix for this, do you mind retrying the build? thank you!

sorry about that @derrick-ejan-rs @chandrasekhar7 - I have pushed a fix for this, do you mind retrying the build? thank you!

@jshawl  we had a new build run and it was successful this time but no email was received.   I will report back if it occurs again on another build of ours.

sorry about that @derrick-ejan-rs @chandrasekhar7 - I have pushed a fix for this, do you mind retrying the build? thank you!

@jshawl we had a new build run and it was successful this time but no email was received. I will report back if it occurs again on another build of ours.

We ran a new build again and step did not fail but no email was received.  We are sending the email to two different email addresses and neither had the notification.

@derrick-ejan-rs are you able to provide more information about your setup? you are welcome to email me at jesse@jesse.sh if it is private or if you don't want to include those details on this public github thread. thank you!

@jshawl looks like its working now. Thanks for the quick fix.

That is great news! Thanks for reaching out and feel free to open a new issue if you have any questions or suggestions!

Working now thankyou so much

authors: chandrasekhar7 derrick-ejan-rs jshawl

repo: cinotify/github-action


text:
Update circe-core, circe-generic, ... to 0.14.2

Updates

io.circe:circe-core
io.circe:circe-generic
io.circe:circe-parser
io.circe:circe-testing

from 0.14.1 to 0.14.2.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.circe" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequest = { frequency = "@monthly" },
  dependency = { groupId = "io.circe" }
}]

labels: library-update, early-semver-minor, semver-spec-patch, commit-count:1

Codecov Report

Merging #324 (18c69b6) into master (9d4c9f3) will decrease coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #324      +/-   ##
==========================================
- Coverage   95.31%   95.23%   -0.08%
==========================================
  Files           6        5       -1
  Lines          64       63       -1
==========================================
- Hits           61       60       -1
  Misses          3        3

Impacted Files
Coverage Δ

...3+/main/scala/io.circe.config/compat/package.scala

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d4c9f3...18c69b6. Read the comment docs.

Superseded by #338.

authors: codecov-commenter scala-steward

repo: circe/circe-config


text:
Update org.everit.json.schema to 1.14.0

Updates com.github.everit-org.json-schema:org.everit.json.schema from 1.13.0 to 1.14.0.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.github.everit-org.json-schema", artifactId = "org.everit.json.schema" } ]

labels: library-update, semver-minor

Codecov Report

Merging #139 (9cd88fb) into master (bbbb5b1) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #139   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            2         2
  Lines           35        35
  Branches         2         4    +2
=========================================
  Hits            35        35

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbbb5b1...9cd88fb. Read the comment docs.

repo: circe/circe-json-schema


text:
Update scalafmt-core to 3.5.3

Updates org.scalameta:scalafmt-core from 3.5.0 to 3.5.3.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scalameta", artifactId = "scalafmt-core" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequest = { frequency = "@monthly" },
  dependency = { groupId = "org.scalameta", artifactId = "scalafmt-core" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:n:2

Superseded by #105.

repo: circe/circe-schema


text:
Question: When do you updates for google-chrome and chromedriver?

Hi.
I'm using a circleci/ruby:2.5.0-node-browsers image and using a chrome-driver in our product for testing. Also I'm using a  chrome-driver for version ChromeDriver 74.0.3729.6 in a local machine.
However, the image provided seems to use Google Chrome 64.0.3282.186 and ChromeDriver 2.37.544315 (730aa6a5fdba159ac9f4c1e8cbc59bf1b5ce12b7).
$ docker run -it circleci/ruby:2.5.0-node-browsers bash
circleci@1a4f2b18ace9:/$ _XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.

circleci@1a4f2b18ace9:/$ chromedriver -v
ChromeDriver 2.37.544315 (730aa6a5fdba159ac9f4c1e8cbc59bf1b5ce12b7)
circleci@1a4f2b18ace9:/$ google-chrome --version
Google Chrome 64.0.3282.186

I found some PullRequests and Issues related by this.

https://github.com/circleci/circleci-images/pull/174
It said you needed to pin the version of chorme, and currently this image has been used.
https://github.com/circleci/circleci-images/issues/355
https://github.com/circleci/circleci-images/pull/358

Then, the last PullRequest has fixes to update the versions of chrome and chrome-driver, and it has been merged. However, I could not point the image to use newer chrome-driver in CircleCI process, so I added some command to install newer chrome and driver with a base of -browser image by myself.
I'm just wondering when to release the newer one.

@toriumi0118 we likely won't be modifying how we install chrome / chromedriver in these images, as we're working on a new set of images that will address these types of issues in a better way, but you can use some orbs we'e been rolling out as part of that work to install newer versions of chrome / chromedriver:
https://circleci.com/orbs/registry/orb/circleci/circleci-images

I see. thanks!

repo: circleci/circleci-images