text:
Error: token invalid 401

Describe the problem
After upgrading from v0.14.4 to v0.21.3 I am unable to access the dashboard and get this error:
I suspect this is related to the new IdP changes but I am not sure what.
To Reproduce Steps to reproduce the behavior:
1. Pull latest `https://github.com/netbirdio/netbird.git`

2. Copy `infrastructure_files/setup.env.template` to `infrastructure_files/setup.env` (overwrite existing file from old version)

3. Fill in correct values noting these changes:

   1. Generate a client secret for the `netbird-client` in Keycloak and `NETBIRD_AUTH_CLIENT_SECRET`
   2. Create a new client `netbird-backend` and follow the steps here: https://docs.netbird.io/selfhosted/identity-providers#step-8-create-a-net-bird-backend-client

4. Re-run `./configure.sh`

5. Run `docker compose pull && docker compose down && docker compose up -d`

Expected behavior
Things work as they used to
Additional context
I tried setting NETBIRD_MGMT_IDP="none" and the dashboard loads but no clients can connect with this error:
netbird up --management-url https://netbird.XXX:33073 --admin-url https://netbird.XXX:443
Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details

i have the same error with a different return code:

Request failed with status code 401. Please refresh the page if the issue continues.
token invalid

docker management logs:

2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0
2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401

that's my setup.env:
## example file, you can copy this file to setup.env and update its values
##
# Dashboard domain. e.g. app.mydomain.com
NETBIRD_DOMAIN="XXXXX"

# -------------------------------------------
# OIDC
#  e.g., https://example.eu.auth0.com/.well-known/openid-configuration
# ------------------------------------------
NETBIRD_AUTH_DEVICE_AUTH_PROVIDER=hosted
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://XXXXX/auth/realms/netbird/.well-known/openid-configuration"
NETBIRD_USE_AUTH0=false
NETBIRD_AUTH_CLIENT_ID="netbird-client"
NETBIRD_AUTH_AUDIENCE="netbird-client"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID="netbird-client"
NETBIRD_MGMT_IDP="keycloak"
NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend"
NETBIRD_IDP_MGMT_CLIENT_SECRET="XXXXX"
NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://XXXXX/auth/realms/netbird"

# -------------------------------------------
# Letsencrypt
# -------------------------------------------
# Disable letsencrypt
#  if disabled, cannot use HTTPS anymore and requires setting up a reverse-proxy to do it instead
# e.g. hello@mydomain.com
NETBIRD_LETSENCRYPT_EMAIL="XXXXX"
# -------------------------------------------
# Extra settings
# -------------------------------------------
# Disable anonymous metrics collection, see more information at https://netbird.io/docs/FAQ/metrics-collection
NETBIRD_DISABLE_ANONYMOUS_METRICS=false
# DNS DOMAIN configures the domain name used for peer resolution. By default it is netbird.selfhosted
NETBIRD_MGMT_DNS_DOMAIN=netbird.selfhosted

when i try to connect in the client side:
sudo netbird up --management-url https://XXXXXXXX:33073
Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details

for the netbird-backend, i didn't do the management steps because, the services accounts roles tab is missing

new management logs:
2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0
2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3101701186: GET /api/users status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 335186381: GET /api/peers status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 449313820: GET /api/groups status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 780097130: GET /api/users status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3241716118: GET /api/peers status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 376855912: GET /api/users status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3584425485: GET /api/groups status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 997841889: GET /api/users status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3647231711: GET /api/users status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2307183365: GET /api/peers status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 418149988: GET /api/groups status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3370837239: GET /api/groups status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2633305945: GET /api/users status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1883989961: GET /api/users?service_user=false status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1727792694: GET /api/groups status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 127726435: GET /api/users status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1972559671: GET /api/groups status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3935102461: GET /api/peers status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 549383702: GET /api/peers status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3104191757: GET /api/users status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3006261035: GET /api/groups status 401
2023-06-29T00:30:18Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=
2023-06-29T00:30:21Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2288542812: GET /api/users status 401
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1587588152: GET /api/peers status 401
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 584506530: GET /api/groups status 401
2023-06-29T00:33:55Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=

Originally posted by @UncleJ4ck in https://github.com/netbirdio/netbird/issues/959#issuecomment-1612271029

@UncleJ4ck It seems that there was a warning in the logs regarding cache warming up failure, specifically due to an error with obtaining the Keycloak token. The error message indicated a status code of 401.
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401
It appears that the configuration of your netbird-backend client may be incorrect. To address this issue, I recommend following the steps outlined in the documentation at https://docs.netbird.io/selfhosted/identity-providers#keycloak to ensure it is properly configured.

i fixed the error, because there was no service accounts roles but after that the error 2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401 disappeared as you can see here: https://github.com/netbirdio/netbird/issues/959#issuecomment-1612278033

I've got the same error with Authentik. Runninh with a fresh install.

@Akruidenberg @UncleJ4ck can you confirm that you have added the Scopes variable to your setup.env files?
Keycloak
NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"

Authentik
NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"

Yes i confirm

Yes.

I also encountered this issue using authentik on a fresh install. I fixed the issue by setting a JWKS signing key in authentik under providers -> netbird -> protocol settings -> signing key. You may need to reconfigure / restart netbird.

I've been seeing this issue with Zitadel IDP behind Traefik since I set up the install a few versions ago and up until current release.
I do not have the cache warming up error from OP, management starts as so:
netbird-management  | time="2023-07-04T00:39:15Z" level=info msg="loading OIDC configuration from the provided IDP configuration endpoint https://zitadel.DOMAIN.nz/.well-known/openid-configuration" netbird-management  | time="2023-07-04T00:39:16Z" level=info msg="loaded OIDC configuration from the provided IDP configuration endpoint: https://zitadel.DOMAIN.nz/.well-known/openid-configuration" netbird-management  | time="2023-07-04T00:39:16Z" level=info msg="overriding HttpConfig.AuthIssuer with a new value https://zitadel.DOMAIN.nz, previously configured value: https://zitadel.DOMAIN.nz" netbird-management  | time="2023-07-04T00:39:16Z" level=info msg="overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://zitadel.DOMAIN.nz/oauth/v2/keys, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/keys" netbird-management  | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://zitadel.DOMAIN.nz/oauth/v2/token, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/token" netbird-management  | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://zitadel.DOMAIN.nz/oauth/v2/device_authorization, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/device_authorization" netbird-management  | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: zitadel.DOMAIN.nz, previously configured value: " netbird-management  | 2023-07-04T00:39:16Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081 netbird-management  | 2023-07-04T00:39:16Z INFO management/server/account.go:671: single account mode enabled, accounts number 1 netbird-management  | 2023-07-04T00:39:16Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073 netbird-management  | 2023-07-04T00:39:16Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443 netbird-management  | 2023-07-04T00:39:19Z INFO management/server/account.go:858: warmed up IDP cache with 1 entries netbird-management  | 2023-07-04T00:39:38Z INFO management/server/account.go:1256: overriding JWT Domain and DomainCategory claims since single account mode is enabled netbird-management  | 2023-07-04T00:39:38Z INFO management/server/account.go:1256: overriding JWT Domain and DomainCategory claims since single account mode is enabled
The dashboard will always log in correctly and management actually responds as you can see above for some hours after restarting management, I'm guessing until token expiration. After that happens management starts responding with 401 as so:
netbird-management  | 2023-07-04T23:38:16Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 125147462: GET /api/peers status 401 netbird-management  | 2023-07-04T23:38:16Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1167195797: GET /api/groups status 401 netbird-management  | 2023-07-04T23:38:29Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2431638793: GET /api/users status 401
I have been messing with config files over several versions and weeks to try and debug this further, I had a feeling that it was something to do with the token being invalid due to a mismatch in URL vs. URL:443 or something but I can't seem to figure that out and I'm not sure if that's the issue at all.

EDIT: Actually I just came across #806 and #808 which I think is likely to be my issue. I've recreated my containers now and will test that.

This has solved my issue and it appears to be unrelated to this issue. Apologies for the noise.

the issue still exists in the new version 0.21.9

dashboard_1   | 2023/07/22 13:59:57 [crit] 35#35: *131 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 159.223.19.103, server: 0.0.0.0:443
dashboard_1   | X - - [22/Jul/2023:13:59:57 +0000] "GET / HTTP/1.1" 200 347 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:13:59:58 +0000] "GET /private/api/v1/service/premaster HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:13:59:58 +0000] "GET /favicon.ico HTTP/1.1" 200 11587 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
dashboard_1   | X  - - [22/Jul/2023:14:02:52 +0000] "GET / HTTP/1.1" 200 602 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:14:02:52 +0000] "GET / HTTP/1.1" 200 602 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
dashboard_1   | X - - [22/Jul/2023:14:14:53 +0000] "GET /.env HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:14:14:53 +0000] "GET /.env HTTP/1.1" 200 347 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"

> some logs before they looks fine
management_1  | 2023-07-22T12:37:45Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
management_1  | 2023-07-22T12:37:45Z INFO management/server/account.go:672: single account mode enabled, accounts number 0
management_1  | 2023-07-22T12:37:46Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
management_1  | 2023-07-22T12:37:46Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443

Hello folks; sorry for the delay in replying.
The issue is probably connected to a wrong Audience configuration. We added logs in the management system, but unfortunately they were set to debug.  Can you all edit your docker-compose.yml and add the following line to the management service command list:
"--log-level", "debug",
The whole command should look similar to this:
    command: [
      "--port", "443",
      "--log-file", "console",
      "--log-level", "debug",
      "--disable-anonymous-metrics=false",
      "--single-account-mode-domain=netbird.selfhosted",
      "--dns-domain=netbird.selfhosted"
      ]

After that run docker-compose up -d to recreate the container.
If you have an issue with the audience, you should see logs like these:
2023-07-22T16:56:48+02:00 DEBG management/server/jwtclaims/jwtValidator.go:158: error parsing token: invalid audience
2023-07-22T16:56:48+02:00 DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: invalid audience

If that is the case, please review the settings in the management.json if they match your configuration.
I've opened the PR #1026 to fix this log-level issue.

i added the log-level debug into the docker-compose file and i recreated the container, but i don't have this type of error
Attaching to infrastructure_files_management_1
management_1  | time="2023-07-22T16:52:28Z" level=info msg="loading OIDC configuration from the provided IDP configuration endpoint https://domain/realms/netbird/.well-known/openid-configuration"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="loaded OIDC configuration from the provided IDP configuration endpoint: https://domain/realms/netbird/.well-known/openid-configuration"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding HttpConfig.AuthIssuer with a new value https://domain/realms/netbird, previously configured value: https://domain/realms/netbird"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://domain/realms/netbird/protocol/openid-connect/certs, previously configured value: https://domain/realms/netbird/protocol/openid-connect/certs"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://domain/realms/netbird/protocol/openid-connect/token, previously configured value: https://domain/realms/netbird/protocol/openid-connect/token"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://domain/realms/netbird/protocol/openid-connect/auth/device, previously configured value: https://domain/realms/netbird/protocol/openid-connect/auth/device"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: domain, previously configured value: "
management_1  | 2023-07-22T16:52:28Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
management_1  | 2023-07-22T16:52:28Z DEBG management/server/file_store.go:248: took 0 ms to persist the FileStore
management_1  | 2023-07-22T16:52:28Z INFO management/server/account.go:672: single account mode enabled, accounts number 0
management_1  | 2023-07-22T16:52:28Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
management_1  | 2023-07-22T16:52:28Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443

for the dashboard logs, i don't see any errors all requests have 200 as a response

Hello @UncleJ4ck do you mind joining our Slack channel it may be faster for us to troubleshoot the issue.

Sure

fixed, after fixing keycloak

authors: Akruidenberg UncleJ4ck bcmmbaga hadleyrich mlsmaycon thijsa


text:
Instance operator - container environment refactor

Description
This PR continues the instances operator refactor by renewing the container environment section.
How Has This Been Tested?

[ ] Locally
[ ] Staging

/deploy-staging

source: gharchive/pull-request

repo: netgroup-polito/CrownLabs

license_type: permissive

license_source: github-api


text:
feat: support npm modules in edge functions

Summary
Needs https://github.com/netlify/edge-bundler/pull/475. Setting as draft until that lands.

Superseded by https://github.com/netlify/cli/pull/6018.

repo: netlify/cli

license: MIT


text:
Use pr base branch to configure editorial workflow for different environments (stage/production)

I have been looking at how to configure the cms for different environments develop -> stage -> production.
What I would like to achieve is to have a dedicated repository that will contains the contents. This repository will contain long living branches per environment (master/stage/develop).
At build time I will have a script that builds the yaml configuration assigning the right branch per environment.
However this configuration wouldn't work with editorial workflow.
I noticed that when retrieving unpublished contents to render the editorial workfloe, the api only looks at the branch names. However when we retrieve the PRs, we could filter by base by using the configuration. This way we can separate editorial workflows per environments.
I may be missing something, but I wonder if this approach would work.
Would this be a right implementation? If so, I will be happy to contribute on it.

That's a very interesting idea. If I understand it correctly, you want to show only PRs with a specific base branch in the editorial workflow, is this correct? I think your approach can work if that's the idea. Feel free to open a PR so we can play with the code.

@calavera Yes that is what I mean. I see that the branch in the config is not used to filter the PRs for the editorial workflow. I will try to open a PR asap for this one.

source: gharchive/issue

authors: calavera rewop

repo: netlify/netlify-cms


text:
Only default locale editable i18n

Describe the bug
In the admin ui, only the default locale is shown.
Therefor it is not possible to edit other locale.
To Reproduce

Goto Admin UI -> any Collection
Open an entry

Expected behavior
Other locales are shown.
Screenshots

Applicable Versions:

Netlify CMS version: 2.10.160
Git provider: gitlab
OS: fedora
Browser version: Mozilla Firefox 90.0.2

CMS configuration
backend:
  name: gitlab
  repo: tw1t611/bestnotebook-tech
  branch: main

# publish_mode: editorial_workflow # will not publish into main
media_folder: public/images # Media files will be stored in the repo under public/images
public_folder: /images # The src attribute for uploaded media will begin with /images

i18n:
  structure: multiple_folders
  locales: [en, de]
  default_locale: en

collections:
  - label: "Pages"
    name: "pages"
    folder: "public/content/pages"
    create: true
    i18n: true
    fields:
      - { label: "Title", name: "title", widget: "string" }
      - { label: "Path", name: "path", widget: "string" }
      - { label: "Description", name: "description", widget: "text" }
      - label: "Content"
        name: "content"
        widget: "list"
        summary: "{{fields.name}}"
        fields:
          - { label: "Name", name: "name", widget: "string" }
          - { label: "Content", name: "content", widget: "markdown" }

  - label: "Blog"
    name: "blog"
    folder: "public/content/posts"
    create: true
    i18n: true
    fields:
      - { label: "Slug", name: "slug", widget: "string" }
      - { label: "Title", name: "title", widget: "string" }
      - { label: "Description", name: "description", widget: "text" }
      - { label: "Body", name: "body", widget: "markdown" }

  - label: "Notebooks"
    name: "notebooks"
    folder: "public/content/notebooks"
    create: true
    i18n: true
    fields:
      - { label: "Name", name: "name", widget: "string" }
      - { label: "Model", name: "model", widget: "string" }
      - { label: "Variant", name: "variant", widget: "string" }
      - { label: "Description", name: "description", widget: "text" }
      - { label: "Content", name: "content", widget: "markdown" }
      - label: "CPU"
        name: "cpu"
        widget: "relation"
        collection: "cpus"
        search_fields: ["name"]
        value_field: "name"

  - label: "CPUs"
    name: "cpus"
    folder: "public/content/cpus"
    create: true
    fields:
      - { label: "Name", name: "name", widget: "string" }
      - { label: "Cores", name: "cores", widget: "number" }

Additional Context

Tried adding the locale folders
Tried pushing content for other locale

public/content/posts/de/asdf.md exists

Content was downloaded to local branch

Hi @tw1t611, you should set the i18n configuration on specific fields too, see https://www.netlifycms.org/docs/beta-features/#i18n-support.
Closing, if my suggestion doesn't work please comment and I'll re-open

works, ty :)


text:
Next.js website with Netlify Forms is not deploying

Describe the bug:
Next.JS website with a form fails to deploy. The site builds perfectly, but in the deploying phase, the log states:
Post processing - HTML

...5 times and the build fails. Post processing is completely off. Upon further investigation it was found that Netlify Forms is causing problems. This is the behaviour I could notice:

Deploy fails without any change in default config.
Deploy succeeds if you turn off form detection.
Deploy succeeds if you remove data-netlify=true from the form in the TypeScript file, but keep form detection on.
Deploy fails with a setup similar as the previous point, but you add a static HTML version of the form with the required attribute in the public folder.

To Reproduce:
Steps to reproduce the behaviour:

Clone this repo: https://github.com/Hrishikesh-K/reapertips-develop.
Try the various combinations of settings as mentioned above.

Expected behaviour:
The website should deploy fine regardless of the form.
Versions:

Next.js: 11.1.0
plugin (if installed at fixed version): N/A - pulled latest automatically (3.8.0 and 3.9.0 tested at the time of writing)

Additional context:
A customer reported the problem: https://answers.netlify.com/t/next-js-deployment-failed/43308/

The specific customer fail (https://app.netlify.com/sites/reapertips/deploys/612f8b60e0e2cf00077e61f0) seems toreport an incorrect form
</form>: unexpected 's' after '[:equal, "'No, it'"]' worker=Deploy::ProcessWorker job_id=d619bb595919662f60e100c6 batch_id=nil queue_name=deploy_process process_id=8 sidekiq=true thread_id=m3uc deploy_id=612f8b60e0e2cf00077e61f0 site_id=4c5161e1-9be9-43eb-9d80-9df2ec8d126c site_subdomain=reapertips account_id=612a71aa1630d0e1fb8a5998 logtype=processing reseller_id=53ce9ee971f1c34344000001

logs

I can confirm I got the website to deploy successfully 's' from the radio value. It's a bit weird however that the form is reported to be incorrect for such a small thing.

To conclude, this is not an error with the plugin. So, this is actually an incorrect repo to file this issue, and it might be moved in the future.

About the actual issue:
While yes, there could have been better info about the logs, I believe you should refrain from using ' like characters in HTML attributes, because it's supposed to be a separator for an attribute.
But, glad to know the problem is now solved! Feel free to open another thread in the forums if you've any more questions.

authors: DaveKeehl Hrishikesh-K ingride

repo: netlify/netlify-plugin-nextjs


text:
implemented multiplier in transform-generic

This PR allows to add in the configuration file a multiplier to apply to particular fields in a flow log.
For example, if we know that we only get a sampling of the flow logs (e.g. 1/20), then the administrator may want us to multiply the measured number of bytes and/or packets in the flow log entries in order to get a closer estimate of the actual number of bytes and/or packets.
Each field that requires a multiplier must be individually marked.
This PR implements this functionality in the transform-generic stage.

Codecov Report

Merging #345 (d3ed936) into main (cba2541) will decrease coverage by 0.02%.
The diff coverage is 53.84%.

@@            Coverage Diff             @@
##             main     #345      +/-   ##
==========================================
- Coverage   68.89%   68.87%   -0.03%
==========================================
  Files          87       87
  Lines        5064     5099      +35
==========================================
+ Hits         3489     3512      +23
- Misses       1355     1368      +13
+ Partials      220      219       -1

Flag
Coverage Δ

unittests
68.87% <53.84%> (-0.03%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/api/transform_generic.go
0.00% <ø> (ø)

pkg/pipeline/transform/transform_generic.go
71.01% <53.84%> (-23.11%)
:arrow_down:

pkg/pipeline/extract/timebased/heap.go
97.82% <0.00%> (-2.18%)
:arrow_down:

pkg/test/utils.go
72.66% <0.00%> (+2.87%)
:arrow_up:

pkg/config/config.go
63.33% <0.00%> (+13.33%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: KalmanMeth codecov-commenter

repo: netobserv/flowlogs-pipeline


text:
ByteBuf Leak in SslHandler when running in paranoid mode

Expected behavior
No leak
Actual behavior
[ERROR] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1484)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1233)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)
#2:
	Hint: 'httpServerCodec' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1484)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1233)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)
#3:
	io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:736)
	io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1494)
	io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
	io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:198)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1338)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1233)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)
#4:
	io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:706)
	io.netty.handler.ssl.SslHandler.toByteBuffer(SslHandler.java:1494)
	io.netty.handler.ssl.SslHandler.access$300(SslHandler.java:166)
	io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:198)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1338)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1233)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:363)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
	io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2149)
	io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1333)
	io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1233)
	io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)
: 1 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

Steps to reproduce
Unknown - seeing this log printed in our service with PARANOID leakDetection enabled
Minimal yet complete reproducer code (or URL to code)
Unknown
Netty version
4.1.25
JVM version (e.g. java -version)
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)

OS version (e.g. uname -a)
Tue Feb 2 23:52:07 PST 2021 x86_64 x86_64 x86_64 GNU/Linux

Not seeing any of our application code in the stacktrace.

I think you are going by the fact that the buffer was created by the SslHandler. If a handler creates and sends a buffer on the pipeline, it has passed the liability of release to the handlers after it in the pipeline.

Hint: 'httpServerCodec' will handle the message from this point.

Do you add a handler with the name httpServerCodec? Perhaps check in your pipeline if you are releasing all buffers.

@NiteshKant:
Yes, we do have a handler with that name. It's actually just:

import io.netty.handler.codec.http.HttpServerCodec;
pipeline.addLast("httpServerCodec", new HttpServerCodec());

We are only calling retain() when we add the ByteBuf to the List<Object> out. Example:
  @Override
  protected void encode(ChannelHandlerContext ctx, ByteBuf msg, List<Object> out) throws Exception {
    BinaryWebSocketFrame binaryWebSocketFrame = new BinaryWebSocketFrame(msg);
    binaryWebSocketFrame.retain();
    out.add(binaryWebSocketFrame);
  }

Correct me if I'm wrong, we are calling retain() and then passing the liability to the next Handler in the pipeline. This usage should not result in any leaks, right?
Do you have any other suggestions to go about debugging this?

If the contents of out are sent on the pipeline such that they are released by another handler, then adding a retain() here is essentially calling the leak unless each frame created here is released at least twice.

please provide a reproducer...

Closing this issue as it looks like the additional retain as discussed in https://github.com/netty/netty/issues/11289#issuecomment-845536923 is causing the leak. Please reopen if this isn't true.

authors: NiteshKant normanmaurer svaithia

repo: netty/netty


text:
Vulnerability found in Netty 4.1.73 version

Hi,
Veracode scan reported Medium Vulnerabilities found in netty.

netty-handler-4.1.73.Final.jar

Files:

io/netty/handler/ssl/JdkAlpnSslUtils.java : 51
io/netty/handler/ssl/JdkSslClientContext.java : 292
io/netty/handler/ssl/JdkSslContext.java : 74
io/netty/handler/ssl/JdkSslServerContext.java : 271
io/netty/handler/ssl/OpenSslX509TrustManagerWrapper.java : 159
io/netty/handler/ssl/SslUtils.java : 188 & 190

CWE ID: 757
Reason: Selection of Less-Secure Algorithm During Negotiation ('Algorithm Downgrade') (check CWE for more details).

netty-resolver-4.1.73.Final.jar

Files:

io/netty/resolver/HostsFileEntriesProvider.java : 307

CWE ID: 73
Reason: External Control of File Name or Path. This call contains a path manipulation flaw. The argument to the function is a
filename constructed using untrusted input. (check CWE for more details).

netty-common-4.1.73.Final.jar

CWE ID: 377

Files:

io/netty/util/internal/PlatformDependent.java : 1438 & 1440

Reason: Insecure Temporary File. Creating and using insecure temporary files can leave application and system data vulnerable to
attack. In particular, file names created by the tmpnam family of functions can be easily guessed by an attacker (More
details on CWE)

CWE ID: 331

Files:

io/netty/util/internal/NativeLibraryLoader.java: 350

Reason: Standard random number generators do not provide a sufficient amount of entropy when used for security purposes.
Attackers can brute force the output of pseudorandom number generators such as rand(). (Check CWE for more details).

CWE ID: 117

Files:

io/netty/util/internal/logging/Log4JLogger.java: 328
io/netty/util/internal/logging/CommonsLogger.java: 305

Reason: Improper Output Neutralization for Logs. A function call could result in a log forging attack. Writing untrusted data into a log file allows an attacker to forge log entries or inject malicious content into log files. (More details on CWE).

CWE ID: 732

Files:

io/netty/util/internal/PlatformDependent.java: 1442

Reason: The software specifies permissions in a way that causes the resource to be accessible to an unintended sphere of control.
A common manifestation of this is setting overly lax file permissions. (More details on CWE).

Please let me know if these are false Positives or needs to be addressed.
Thanks,
Shyam Joshi

Thanks for reporting. However, all of them are false alerts.
Also, please do a little bit of self-research before reporting tool-generated alerts.


text:
Fix checkstyle warning

Build was failing for me

Thanks was my fault.

authors: jabley normanmaurer


text:
[Netty 5] Revisit AbstractReferenceCounted

If #8543 results in ReferenceCounted remaining on the public API, we should put in some verbiage for what implementations are expected to do.
For other C++ programs, the pattern I see is that retaining an object is a memory order relaxed atomic increment, and released is an acq-rel decrement. (with a minor optimization of a relaxed decrement if it knows it's not the last one).
I believe that decrementing a ref below zero is a bug, and at least one exception should be raised, but perhaps with not much more guarantees than that.  I believe incrementing a ref past int max should have a similar stipulation.  Lastly, I believe incrementing a ref from zero to above one is an invalid state and error behavior is undefined.
cc @dklempner

For other C++ programs, the pattern I see is that retaining an object is a memory order relaxed atomic increment, and released is an acq-rel decrement

And wdyt about providing an implementaion with only relaxed reference counting ops ie no full barriers: most thread-per-core arch would benefit by it.

@franz1981 that would definitely be useful, the increment mode could be reduced to plain rather than opaque.    In the past I have made just a dummy class to do this:
class Ref {
   long counter;
}

Java doesn't have standard way of doing this.


text:
Give explicit error on missing BouncyCastle PKIX library

Motivation:
If the BounyCastle PKIX library is missing from the classpath, Netty throws an exception like this: Failed to generate a self-signed X.509 certificate using Bouncy Castle. This is not really helpful as the user has to go through Stacktrace to find out what went wrong. To combat this, I added a util method that checks if org.bouncycastle.cert.X509v3CertificateBuilder is available in the classpath or not. Based on this, we can return explicit clear exceptions which helps in debugging.
org.bouncycastle.cert.X509v3CertificateBuilder is part of the BounyCastle PKIX library.
Modification:
Added isBouncyCastleAvailable() method to check for org.bouncycastle.cert.X509v3CertificateBuilder availability in classpath.
Result:
Clear explicit error on missing BouncyCastle PKIX library

@hyperxpro thanks

authors: hyperxpro normanmaurer


text:
Get uploaded size while upload is in progress 3.10

Proposal to fix issue #3636 (this has to be fix in all versions: version 3.10)
Motivations:
Currently, while adding the next buffers to the decoder (decoder.offer()), there is no way to access to the current HTTP object being decoded since it can only be available currently once fully decoded by decoder.hasNext().
Some could want to know the progression on the overall transfer but also per HTTP object.
While overall progression could be done using (if available) the global Content-Length of the request and taking into account each HttpContent size, the per HttpData object progression is unknown.
In addition, currently, the Content-Length is not provided by the client and not taken into account for Attribute.
Modifications:

For HTTP object, AbstractHttpData has 2 protected properties named definedSize and size, respectively the supposely final size and the current (decoded until now) size.
This provides a new method definedSize() to get the current value for definedSize. The size attribute is reachable by the length() method.

Note however definedSize is only defined if Content-Length is provided (not mandatory). So definedSize might be 0.

In the InterfaceHttpPostRequestDecoder (and the derived classes), I add a new method: decoder.currentPartialHttpData() which will return a InterfaceHttpData (if any) as the current Attribute or FileUpload (the 2 generic types), which will allow then the programmer to check
according to the real type (instance of) the 2 methods definedSize() and length().

This method check if currentFileUpload or currentAttribute are null and returns the one (only one could be not null) that is not null.
Note that if this method returns null, it might mean 2 situations:
a) the last HttpData (whatever attribute or file upload) is already finished and therefore accessible through next()
b) there is not yet any HttpData in decoding (body not yet parsed for instance)

Add Content-Length in default behavior in Multipart Encoder and handling in Multipart Decoder for Attribute (missing parts).

Add constructor for Attribute and Factory to create Attribute if possible with definedSize.

Result:
The developper has more access and therefore control on the current upload.
The coding from developper side could looks like in the example in HttpUloadServerHandler.
Master piece for #3661 (complete version for 3.10, master version will need to be redone once this 3.10 version could be accepted).

@normanmaurer @Scottmitch @trustin
Once tests passed, please review

@normanmaurer ping

@normanmaurer I suppose we should close this one since, as for 4.0, this would lead to addiing some new methods to the interface, right?

We should keep public interface compatibility in 4.0 and only do public interface additions/modifications in 4.1+.

Ok, it was just to confirm ;-)
So I close this PR too...

authors: Scottmitch fredericBregier


text:
HPACK Encoder headerFields improvements

Motivation:
HPACK Encoder has a data structure which is similar to a previous version of DefaultHeaders. Some of the same improvements can be made.
Motivation:

Enforce the restriction that the Encoder's headerFields length must be a power of two so we can use masking instead of modulo
Use AsciiString.hashCode which already has optimizations instead of having yet another hash code algorithm in Encoder

Result:
Fixes https://github.com/netty/netty/issues/5357

LGTM

Am 29.06.2016 um 03:16 schrieb Scott Mitchell notifications@github.com:
Motivation:
HPACK Encoder has a data structure which is similar to a previous version of DefaultHeaders. Some of the same improvements can be made.
Motivation:
Enforce the restriction that the Encoder's headerFields length must be a power of two so we can use masking instead of modulo
Use AsciiString.hashCode which already has optimizations instead of having yet another hash code algorithm in Encoder
Result:
Fixes #5357
You can view, comment on, or merge this pull request online at:
https://github.com/netty/netty/pull/5466
Commit Summary
HPACK Encoder headerFields improvements
File Changes
M codec-http2/src/main/java/io/netty/handler/codec/http2/internal/hpack/Encoder.java (65)
M codec-http2/src/test/java/io/netty/handler/codec/http2/internal/hpack/TestCase.java (2)
M codec/src/main/java/io/netty/handler/codec/DefaultHeaders.java (10)
M common/src/main/java/io/netty/util/AsciiString.java (2)
M microbench/src/main/java/io/netty/microbench/http2/internal/hpack/EncoderBenchmark.java (8)
Patch Links:
https://github.com/netty/netty/pull/5466.patch
https://github.com/netty/netty/pull/5466.diff
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@buchgr PTAL as well

@carl-mastrangelo mind taking a look?

LGTM, have before/after numbers?

=== with PR

Benchmark                (duplicates)  (limitToAscii)  (maxTableSize)  (sensitive)  (size)   Mode  Cnt        Score        Error  Units
EncoderBenchmark.encode          true            true            4096         true   SMALL  thrpt    5   356927.366 ±  56001.273  ops/s
EncoderBenchmark.encode          true            true            4096         true  MEDIUM  thrpt    5    47351.315 ±   8528.331  ops/s
EncoderBenchmark.encode          true            true            4096         true   LARGE  thrpt    5     2720.457 ±    193.968  ops/s
EncoderBenchmark.encode          true            true            4096        false   SMALL  thrpt    5  1437241.861 ± 132873.605  ops/s
EncoderBenchmark.encode          true            true            4096        false  MEDIUM  thrpt    5   849782.491 ±  86691.285  ops/s
EncoderBenchmark.encode          true            true            4096        false   LARGE  thrpt    5   144427.332 ±   7858.671  ops/s
EncoderBenchmark.encode          true           false            4096         true   SMALL  thrpt    5  2830273.724 ± 140489.445  ops/s
EncoderBenchmark.encode          true           false            4096         true  MEDIUM  thrpt    5   488952.888 ±  43357.410  ops/s
EncoderBenchmark.encode          true           false            4096         true   LARGE  thrpt    5    36376.959 ±   3110.164  ops/s
EncoderBenchmark.encode          true           false            4096        false   SMALL  thrpt    5  4482576.931 ± 137792.996  ops/s
EncoderBenchmark.encode          true           false            4096        false  MEDIUM  thrpt    5  1435410.956 ±  84211.360  ops/s
EncoderBenchmark.encode          true           false            4096        false   LARGE  thrpt    5   194170.322 ±  19504.159  ops/s
EncoderBenchmark.encode         false            true            4096         true   SMALL  thrpt    5   368044.726 ±  34911.272  ops/s
EncoderBenchmark.encode         false            true            4096         true  MEDIUM  thrpt    5    44510.592 ±   1875.679  ops/s
EncoderBenchmark.encode         false            true            4096         true   LARGE  thrpt    5     2489.113 ±    127.246  ops/s
EncoderBenchmark.encode         false            true            4096        false   SMALL  thrpt    5   338468.082 ±  12103.634  ops/s
EncoderBenchmark.encode         false            true            4096        false  MEDIUM  thrpt    5    40525.168 ±   2354.368  ops/s
EncoderBenchmark.encode         false            true            4096        false   LARGE  thrpt    5     2365.145 ±    160.677  ops/s
EncoderBenchmark.encode         false           false            4096         true   SMALL  thrpt    5  2663665.857 ± 364377.319  ops/s
EncoderBenchmark.encode         false           false            4096         true  MEDIUM  thrpt    5   478849.332 ±  41604.573  ops/s
EncoderBenchmark.encode         false           false            4096         true   LARGE  thrpt    5    35103.798 ±   1318.136  ops/s
EncoderBenchmark.encode         false           false            4096        false   SMALL  thrpt    5  2379261.970 ±  57179.624  ops/s
EncoderBenchmark.encode         false           false            4096        false  MEDIUM  thrpt    5   411287.298 ±  22104.069  ops/s
EncoderBenchmark.encode         false           false            4096        false   LARGE  thrpt    5    30752.950 ±   2407.385  ops/s

=== before PR

Benchmark                (duplicates)  (limitToAscii)  (maxTableSize)  (sensitive)  (size)   Mode  Cnt        Score        Error  Units
EncoderBenchmark.encode          true            true            4096         true   SMALL  thrpt    5   352301.863 ±  54569.696  ops/s
EncoderBenchmark.encode          true            true            4096         true  MEDIUM  thrpt    5    47410.641 ±   4530.055  ops/s
EncoderBenchmark.encode          true            true            4096         true   LARGE  thrpt    5     2841.511 ±    113.221  ops/s
EncoderBenchmark.encode          true            true            4096        false   SMALL  thrpt    5  1277058.234 ±  74586.996  ops/s
EncoderBenchmark.encode          true            true            4096        false  MEDIUM  thrpt    5   414200.033 ±  30139.221  ops/s
EncoderBenchmark.encode          true            true            4096        false   LARGE  thrpt    5    66775.632 ±   3148.258  ops/s
EncoderBenchmark.encode          true           false            4096         true   SMALL  thrpt    5  2747344.162 ± 193883.568  ops/s
EncoderBenchmark.encode          true           false            4096         true  MEDIUM  thrpt    5   487397.227 ±  42936.079  ops/s
EncoderBenchmark.encode          true           false            4096         true   LARGE  thrpt    5    35687.226 ±   3862.617  ops/s
EncoderBenchmark.encode          true           false            4096        false   SMALL  thrpt    5  3124704.076 ±  90154.430  ops/s
EncoderBenchmark.encode          true           false            4096        false  MEDIUM  thrpt    5   720691.100 ±  31043.752  ops/s
EncoderBenchmark.encode          true           false            4096        false   LARGE  thrpt    5    74676.306 ±   3189.223  ops/s
EncoderBenchmark.encode         false            true            4096         true   SMALL  thrpt    5   366475.332 ±  38861.898  ops/s
EncoderBenchmark.encode         false            true            4096         true  MEDIUM  thrpt    5    44272.063 ±   2756.406  ops/s
EncoderBenchmark.encode         false            true            4096         true   LARGE  thrpt    5     2504.013 ±    536.302  ops/s
EncoderBenchmark.encode         false            true            4096        false   SMALL  thrpt    5   527139.189 ±  49906.776  ops/s
EncoderBenchmark.encode         false            true            4096        false  MEDIUM  thrpt    5    39418.695 ±   2531.731  ops/s
EncoderBenchmark.encode         false            true            4096        false   LARGE  thrpt    5     2259.957 ±     73.728  ops/s
EncoderBenchmark.encode         false           false            4096         true   SMALL  thrpt    5  2623078.195 ±  56393.871  ops/s
EncoderBenchmark.encode         false           false            4096         true  MEDIUM  thrpt    5   446502.577 ±  32295.207  ops/s
EncoderBenchmark.encode         false           false            4096         true   LARGE  thrpt    5    34913.448 ±   2797.255  ops/s
EncoderBenchmark.encode         false           false            4096        false   SMALL  thrpt    5  1417427.482 ±  45159.113  ops/s
EncoderBenchmark.encode         false           false            4096        false  MEDIUM  thrpt    5   200201.677 ±   9379.406  ops/s
EncoderBenchmark.encode         false           false            4096        false   LARGE  thrpt    5    17524.141 ±   1062.549  ops/s

LGTM

./qps_server --directexecutor --address=localhost:8080
./qps_client --directexecutor --address=localhost:8080

Best out of 3.
Before
Channels:                       4
Outstanding RPCs per Channel:   10
Server Payload Size:            0
Client Payload Size:            0
50%ile Latency (in micros):     414
90%ile Latency (in micros):     452
95%ile Latency (in micros):     476
99%ile Latency (in micros):     644
99.9%ile Latency (in micros):   1513
Maximum Latency (in micros):    125695
QPS:                            93670

After
Channels:                       4
Outstanding RPCs per Channel:   10
Server Payload Size:            0
Client Payload Size:            0
50%ile Latency (in micros):     400
90%ile Latency (in micros):     440
95%ile Latency (in micros):     462
99%ile Latency (in micros):     616
99.9%ile Latency (in micros):   1333
Maximum Latency (in micros):    49407
QPS:                            97566

Seems like a slight win!

thanks @buchgr @carl-mastrangelo @nmittler @normanmaurer !
4.1 (6af56ff)

authors: Scottmitch buchgr carl-mastrangelo nmittler normanmaurer


text:
Fix the tcnative lib loading problem in OSGi

Motivation:
As the issue #5539 say, the OpenSsl.class will throw java.lang.UnsatisfiedLinkError: org.apache.tomcat.jni.Library.version(I)I when it is invoked. This path try to resolve the problem by modifying the native library loading logic of OpenSsl.class.
Modifications:
The OpenSsl.class loads the tcnative lib by NativeLibraryLoader.loadFirstAvailable(). The native library will be loaded in the bundle netty-common's ClassLoader, which is diff with the native class's ClassLoader. That is the root cause of throws UnsatisfiedLinkError when the native method is invoked.
So, it should load the native library by the its bundle classloader firstly, then the embedded resources if failed.
Result:
First of all, the error threw by native method problem will be resolved.
Secondly, the native library should work as normal in non-OSGi env. But, this is hard. The loading logic of Library.class in netty-tcnative bundle is simple: try to load the library in PATH env. If not found, it falls back to the originally logic NativeLibraryLoader.loadFirstAvailable().
BTW: this path depend on netty-tcnative-1.1.33.Fork19 version (with Bundle-NativeCode manifest entry added).
Signed-off-by: XU JINCHUAN xsir@msn.com

Actually, in my personal view, the NativeLibraryLoader.load will NOT work properly in OSGi. Because the ClassLoader which to load the native library is belong to netty-common bundle, NOT the ClassLoader of the native library.
So, if the netty would be compatible with OSGi, we have to customize the ClassLoader of native bundles (I have no idea about that), or we have to modify the NativeLibraryLoader -- rather than calling the System.load() or System.loadLibrary() directly, it should return the native library absolute path and let the caller invoke the System.load() with the absolute path. Something like this:
// Find/extract the embedded native lib. Or return the libname if not found.
String libPath = NativeLibraryLoader.prepareLibrary(libname);
System.load(libPath);  // Load the native lib into current ClassLoader.

@xsir I think the second idea would be ok... Are you interested to provide a PR ?

@normanmaurer I have considered the second idea for a while. The solution has a drawback: the callers of NativeLibraryLoader have to be modified to satisfy the new approach, and the NativeLibraryLoader signature will be changed too.
I have a new idea, but it has a little bit tricky:

Defines a new a simple helper Class, which is a wrapper for System.load() and System.loadLibrary(), will be injected into the native library's ClassLoader in step 2(a);
When loading the native library, will find the helper Class in the native library's ClassLoader at first:
a) if not found, turn the helper Class into a byte[] and define in the native library's ClassLoader, then will get a NEW helper Class instance;
b) else, use the helper Class instance;
Use the helper Class instance to load the native library.

That is, the native library will be loaded in propitiate ClassLoader! Ant this approach has a little side effect to non-OSGi environment application (in the step 1, the helper will be found cause it has been loaded).
I am not sure it is a good solution. What do you think?
I will submit a PR soon, thank you!

You might be able to convince @normanmaurer changing method signature in NativeLibraryLoader is ok because it is internal :)

@johnou Thank you, I just trying to make a better solution, make the NativeLibraryLoader working properly in both OSGi and non-OSGi env.
@normanmaurer I just submit a new commit, please review it and thanks.

The new patch leave the caller of NativeLibraryLoader untouched, such as OpenSsl.class in handler module.

@Scottmitch @normanmaurer The new commit trying to make the NativeLibraryLoader working properly in both OSGi and non-OSGi env. That will add a little bit overhead: find class, define the helper class into the library's ClassLoader if NOT found.
Another solution is to modify the caller process of NativeLibraryLoader, just as OpenSsl.class do in my first commit -- the modification is up to the library's loading helper, the Library.initialize(name) in tcnative for example.
So, as you see, I think the former solution is better: it hidden the detail of platform into itself, this make the caller simpler. Thanks!

@xsir - few more comments and lgtm ... good work.
@normanmaurer - assigned to you ... plz pull in when satisfied

@xsir can you please explain to me again why we need to use reflection hacks for this ? is there no other way ?

Actually, there has another solution to this issue, I have explained to @Scottmitch (forgive my poor English, btw):

netty-tcnative case, which provided a loading helper class org.apache.tomcat.jni.Library. Just like #9f2a213, we should modify OpenSsl, instead of calling the NativeLibraryLoader, trying the helper class first (Depends on a properly Bundle-NativeCode manifest entry). The helper class Library in netty-tcnative does NOT support loading library by absolute path.
netty-transport-native-epoll case, we should modify Native, instead of calling the NativeLibraryLoader, trying the System.load() first (Depends on a properly Bundle-NativeCode manifest entry).

This solution is more simple, but there has a drawback: the library loading process is diff between OSGi and non-OSGi. In the OSGi, the library in $PATH has a higher priority than the one in the bundle. The non-OSGi is just the reverse. That's why I submitted a more complicated solution:)
Thanks for all of the comments!

Got it... I will wait for @jasontedor to comment :)

@normanmaurer ACKing your ping, still catching up after being on vacation last week. I'll try to take a look on Friday if that's acceptable to you?

@jasontedor sure thing... take your time and thanks!

Squashed and cherry-picked into 4.1 (00f74b92fa4e10e26ea8b99f2bb95681085a3dc8) and 4.0 (cca236a8b22095b887e0877d77a4b274805f0d0d)
@xsir thanks a lot!

authors: Scottmitch jasontedor johnou normanmaurer xsir


text:
Remove support for epoll level-triggered

Motivation:
Netty uses epoll edge-triggered by default forever and there is really not reason why someone should use level-triggered (its considered a implementation detail).
Modifications:

Remove code that was related to level-triggered mode
Adjust testclass names

Result:
Fixes https://github.com/netty/netty/issues/9349

This currently depend on https://github.com/netty/netty/pull/9825.

@netty-bot test this please

@njhill yes lets see if it worth it for 4.1


text:
Crashes when deploying to Heroku

I'm stuck when deploying to Heroku. I click the "Deploy to Heroku" button in the tutorial. The app looks to install correctly, however upon launching it crashes. I'm very new to all this so may be a mistake on my end, but its not obvious where I've gone wrong. (Thanks for developing this, its awesome!)
Heroku Application Log:
2020-11-21T00:15:04.183959+00:00 app[web.1]:     at Module._compile (internal/modules/cjs/loader.js:1015:30)
2020-11-21T00:15:04.183960+00:00 app[web.1]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
2020-11-21T00:15:04.183960+00:00 app[web.1]:     at Module.load (internal/modules/cjs/loader.js:879:32)
2020-11-21T00:15:04.183961+00:00 app[web.1]:     at Function.Module._load (internal/modules/cjs/loader.js:724:14)
2020-11-21T00:15:04.183962+00:00 app[web.1]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
2020-11-21T00:15:04.183962+00:00 app[web.1]:   code: 'MODULE_NOT_FOUND',
2020-11-21T00:15:04.183963+00:00 app[web.1]:   requireStack: [ '/app/server/index.js' ]
2020-11-21T00:15:04.183963+00:00 app[web.1]: }
2020-11-21T00:15:04.267621+00:00 heroku[web.1]: Process exited with status 1
2020-11-21T00:15:04.430984+00:00 heroku[web.1]: State changed from starting to crashed
(I manually restarted all dynos here)
2020-11-21T00:22:44.035582+00:00 heroku[web.1]: State changed from crashed to starting
2020-11-21T00:22:47.163946+00:00 heroku[web.1]: Starting process with command node server/index.js
2020-11-21T00:22:49.762919+00:00 heroku[web.1]: Process exited with status 1
2020-11-21T00:22:49.805578+00:00 heroku[web.1]: State changed from starting to crashed
2020-11-21T00:22:49.714414+00:00 app[web.1]: internal/modules/cjs/loader.js:834
2020-11-21T00:22:49.714464+00:00 app[web.1]:   throw err;
2020-11-21T00:22:49.714465+00:00 app[web.1]:   ^
2020-11-21T00:22:49.714466+00:00 app[web.1]:
2020-11-21T00:22:49.714466+00:00 app[web.1]: Error: Cannot find module 'socket.io'
2020-11-21T00:22:49.714466+00:00 app[web.1]: Require stack:
2020-11-21T00:22:49.714467+00:00 app[web.1]: - /app/server/index.js
2020-11-21T00:22:49.719268+00:00 app[web.1]:     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:831:15)
2020-11-21T00:22:49.719274+00:00 app[web.1]:     at Function.Module._load (internal/modules/cjs/loader.js:687:27)
2020-11-21T00:22:49.719275+00:00 app[web.1]:     at Module.require (internal/modules/cjs/loader.js:903:19)
2020-11-21T00:22:49.719276+00:00 app[web.1]:     at require (internal/modules/cjs/helpers.js:74:18)
2020-11-21T00:22:49.719276+00:00 app[web.1]:     at Object. (/app/server/index.js:5:16)
2020-11-21T00:22:49.719276+00:00 app[web.1]:     at Module._compile (internal/modules/cjs/loader.js:1015:30)
2020-11-21T00:22:49.719277+00:00 app[web.1]:     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1035:10)
2020-11-21T00:22:49.719277+00:00 app[web.1]:     at Module.load (internal/modules/cjs/loader.js:879:32)
2020-11-21T00:22:49.719278+00:00 app[web.1]:     at Function.Module._load (internal/modules/cjs/loader.js:724:14)
2020-11-21T00:22:49.719293+00:00 app[web.1]:     at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
2020-11-21T00:22:49.719294+00:00 app[web.1]:   code: 'MODULE_NOT_FOUND',
2020-11-21T00:22:49.719295+00:00 app[web.1]:   requireStack: [ '/app/server/index.js' ]
2020-11-21T00:22:49.719295+00:00 app[web.1]: }
For reference here is the Heroku Build Log:
build-log.txt

see forked version for solution from hthetiot

I don't know heroku deployment. Why doesn't it find the socket.io dependency here? Did you fix the issue since then?

The easyrtc adapter moved again to the networked-aframe repository. I'm closing all issues in this repository.

authors: ghoople vincentfretin

repo: networked-aframe/naf-easyrtc-adapter


text:
Chore/style cleanup utility functions

Summary
This PR focuses on stylistic improvements and cleanup of utility functions.
Goal
Ensure better package maintainability, scalability, and ease of contribution for new developers.
Code Style Adjustments

Refactored variable names and formatting to improve readability.
Improved indentation, spacing, and overall syntax compliance with PEP8.

Utility Function Cleanup

Introduced @assign_algorithms decorator to dynamically assign algorithms to BackendInterface, reducing manual updates and boilerplate.
Simplified BackendInterface by replacing explicit algorithm declarations with automatic mapping from the ALGORITHMS list.

Even ALGORITHMS could be obviated by dynamically retrieving algorithm names directly from the modules using introspection (__str__, dir(), or __dict__), which might be too "auto-magical" for now but worth considering for a future PR.

Improved ParallelGraph's readability and consistency.
Reduced documentation and maintenance overhead through a more scalable, dynamic approach to algorithm registration.
Minor logic refactoring for efficiency.
Fixed bug in chunking:

Removal of Unused Imports/Code

Eliminated unused imports, reducing code complexity and improving load time.

@dpys

@Schefflera-Arboricola -- all type annotations have now been removed. If there are no other issues, then it might be helpful for us to get this merged sooner rather than later so that I can begin working on some of the other features that we discussed. WDYT?

I don't feel strongly about any of these comments, but these are some suggestions for choices during the style cleanup.

Thank you so much for the review! I've committed your formatting suggestions (though it appears now the linter test is failing?) and agree with all of your comments. It seems like the only thing that might warrant further discussion is the use of enums.

What is the program that is creating the .history files?  :)

I think we're getting bogged down here in linting land... :( Can we change the .gitignore file in a separate PR?
Yes, happy to revert the .gitignore to main :) We can address this on a separate PR as well once this PR gets over the finish line.

What is the program that is creating the .history files? :)

The Local History extension of VScode creates .history folders to store automatic backups of your files so you can revert to previous versions, similar to how PyCharm's Local History works.

@MridulS -- it seems like removing the ruff dependency with that direct push to main is causing the style / format check to unconditionally fail for all PR's? Was that something that you were planning to revert?

@MridulS -- it seems like removing the ruff dependency with that direct push to main is causing the style / format check to unconditionally fail for all PR's? Was that something that you were planning to revert?

Can you try moving the update-get-info precommit hook at the end in .pre-commit-config.yaml file, because that hook internally runs a ruff command and @MridulS removed the ruff dependency. But, I think if we run the ruff hook before update-get-info hook then ruff would be installed in the pre-commit environment for the update-get-info hook to use. I think this should resolve the failing style test. Also, I did the same thing in the PR that @MridulS made but that PR is made from the main branch of nx-parallel so i think that's the reason the style test is passing for the push but not for the pull_request. (PR https://github.com/networkx/nx-parallel/pull/84)

authors: Schefflera-Arboricola dPys dschult

repo: networkx/nx-parallel


text:
da edits

[x] add all FlashX citations, including FlashGraph
[ ] in Fig4, what did you use for c?  please specify.

what is c in Fig 4? did you add another subfigure?
BTW, In the caption of Fig 4, LOL in external memory takes 23 minutes to find the embedding on a dataset with 64M dimension, which is 1TB (64M * 2000 * 8), I think.

c is the sparsity constant in your sparse random projections.
any errors you find just fix and make a pull request or push the changes
directly.
On Sat, Jun 18, 2016 at 2:53 PM, Da Zheng notifications@github.com wrote:

what is c in Fig 4? did you add another subfigure?
BTW, In the caption of Fig 4, LOL in external memory takes 23 minutes to
find the embedding on a dataset with 64M dimension, which is 1TB (64M *
2000 * 8), I think.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/neurodata/LOL/issues/52#issuecomment-226959004, or mute
the thread
https://github.com/notifications/unsubscribe/AACjcgeHlkW4F2vDizbH55QrioSuNDdJks5qND6ngaJpZM4I41ax
.

--
the glass is all full:  half water, half air.
neurodata.io, jovo calendar
https://calendar.google.com/calendar/embed?src=joshuav%40gmail.com&ctz=America/New_York

please check if the description is clear

authors: jovo zheng-da

repo: neurodata/LOL

license_source: bigquery


text:
Adding SGM algorithm implementation to the match submodule

What does this implement/fix? Explain your changes.
This PR adds the Seeded Graph Matching (SGM) algorithm to the match submodule of graspy. Specifically, this is an implementation of algorithm 1 (hard seeding) from "Seeded graph matching" (Fishkind, 2018).  Proof of effectiveness of the implementation can be found in the figures below, where I compared the results from my code to the results reported by Fishkind in Table 1 and Figure 2, respectively.
In the first figure shown, for 15 selected QAP instances, 30 independent experiments were run for each m = 1,2,3,4 seeds. The average objective function values over the 30 runs was compared to the results reported in Table 1 of SGM through the metric
Performance = [(ofv_ali) - (ofv_fk)] / optimal.
Values >0: My code performing worse
Values <0: My code performing better
For the second figure shown, I plotted the match ratio (the fraction of vertices that were correctly matched by SGM) as a function of the number of seeds for different rho (correlation) values. Each seed and rho combination was run twice, and the average match ratio value of these two runs was plotted. Although Fishkind ran each seed-rho combination 150 times, I only ran them twice due to runtime constraints. This is likely why my results may appear to be slightly less smooth than Fishkind's Figure 2.

Any other comments?

@asaadeldin11 request a review from me after you get it peer reviewed and address any comments from that?
@SHAAAAN here is SGM code that looks like it works, if you reach the point where you can't proceed on padding/soft seeding, you can use this code temporarily (before merging) with the knowledge that details of the implementation may change later. But like I said, I don't see a reason why you can't test padding with current FAQ implementation. or soft "seeding" without seeds, but just tracking which nodes get matched to which.

@asaadeldin11 request a review from me after you get it peer reviewed and address any comments from that?
@SHAAAAN here is SGM code that looks like it works, if you reach the point where you can't proceed on padding/soft seeding, you can use this code temporarily (before merging) with the knowledge that details of the implementation may change later. But like I said, I don't see a reason why you can't test padding with current FAQ implementation. or soft "seeding" without seeds, but just tracking which nodes get matched to which.

I see. I'll try it later.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@e048a9a). Click here to learn what that means.
The diff coverage is 90.38%.

@@            Coverage Diff            @@
##             master     #323   +/-   ##
=========================================
  Coverage          ?   88.73%
=========================================
  Files             ?       41
  Lines             ?     2707
  Branches          ?        0
=========================================
  Hits              ?     2402
  Misses            ?      305
  Partials          ?        0

Impacted Files
Coverage Δ

graspy/cluster/gclust.py
93.61% <ø> (ø)

graspy/inference/latent_position_test.py
100% <ø> (ø)

graspy/embed/ase.py
100% <ø> (ø)

graspy/embed/omni.py
96.55% <ø> (ø)

graspy/plot/__init__.py
100% <ø> (ø)

graspy/embed/svd.py
100% <ø> (ø)

graspy/inference/latent_distribution_test.py
98.75% <ø> (ø)

graspy/utils/utils.py
91.46% <100%> (ø)

graspy/simulations/__init__.py
100% <100%> (ø)

graspy/__init__.py
100% <100%> (ø)

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e048a9a...1e30ba0. Read the comment docs.

Consider adding more detail to your tutorials - would be good to have a short explanation of what's happening for that code block

@asaadeldin11 looks pretty good overall, I will try to give more thorough comments this weekend. But main comment right now is, were you going to refactor FAQ to be a special case of SGM, or something like that? As is, I wouldn't want to have that much redundant code here.

@asaadeldin11 looks pretty good overall, I will try to give more thorough comments this weekend. But main comment right now is, were you going to refactor FAQ to be a special case of SGM, or something like that? As is, I wouldn't want to have that much redundant code here.

My opinion is that we take the current sgm class, rename it to "gmp" or "graph_matching", and remove faq. I wasn't sure if that's what you wanted to do so I decided to leave it as is for the time being. If you agree with what I said above, I can remove the redundancies.

@asaadeldin11 I like your idea even better, how about we just call the new class GraphMatch (or something like that if you have a better idea). And the gmp flag can be on by default now, which I think makes sense for Graspy anyway. I vote you keep the FAQ tutorial as a separate one and just call it "graph matching without seeds" or "intro to graph matching" or whatever you think makes sense, and the tutorial you have here can just be an additional one

note that I cleared the notebook output so that netlify will run the notebook with the current version of graspy for this PR when the docs build.

@bdpedigo Let me know what you think of the changes and if u like it I'll remove the faq and sgm files

@asaadeldin11
I like it. I propose:

you remove FAQ/SGM files
make any additional changes you want to make
clear all tutorial notebook output (you know what I mean by this?)
I have a few tiny changes it'll just be faster for me to make
you rerun benchmarks just to ensure we didn't mess anything up performance-wise
merge!

Sound good? I bet we could have it merged by early next week

@asaadeldin11
I like it. I propose:

you remove FAQ/SGM files
make any additional changes you want to make
clear all tutorial notebook output (you know what I mean by this?)
I have a few tiny changes it'll just be faster for me to make
you rerun benchmarks just to ensure we didn't mess anything up performance-wise
merge!

Sound good? I bet we could have it merged by early next week

For clearing the notebook do i just delete the cells that display results like graphs and numbers?

@asaadeldin11 can you resolve merge conflicts?

Best proof we have that it works:

Code
Compare to figure 2 here

repo: neurodata/graspy


text:
Add CLI support to generate morphometrics file.

Following discussions in issue #421, this PR aims to implement CLI to generate morphometrics file.
CLI COMMAND : axondeepseg_morphometrics -i path_to_image.png  -f  axon_morphometrics.xlsx -s pixel_size  -a axon_shape
DOCUMENTATION - https://axondeepseg--434.org.readthedocs.build/en/434/documentation.html#morphometrics
Fixes #421
Test the PR
Kindly refer to the RTD and test the functionalities as mentioned under the Morphometrics section of the RTD.
DONE: -

[x] Add a flag to select the shape of the axons
[x] Documentation: Add literature for axon shape (circle and ellipse)
[x] Documentation: Add an example regarding usage for CLI
[x] Feature: Should generate morphometrics from directory different from image directory
[x] Feature: Morphometrics of images from multiple folders much like axondeepseg CLI.
[x] Documentation: Morphometrics of images from multiple folders much like axondeepseg CLI.
[x] Add tests

Does anyone know if there is a way to render RTD from the forked repo and include in this PR so it would be easy to review this PR ?

Does anyone know if there is a way to render RTD from the forked repo and include in this PR so it would be easy to review this PR ?

Does anyone know if there is a way to render RTD from the forked repo and include in this PR so it would be easy to review this PR ?

link of the built RTD for the present PR is in the checks:

Does anyone know if there is a way to render RTD from the forked repo and include in this PR so it would be easy to review this PR ?

link of the built RTD for the present PR is in the checks:

@mariehbourget, I've added some literature studies here, but these are not sufficient. If you happen to know literature where axon shape being circle or ellipse are used for morphometrics, feel free to contribute in the documentation of this PR or add them in the comments, which I will add on in the RTD. Thanks!!

@mariehbourget, I've added some literature studies here, but these are not sufficient. If you happen to know literature where axon shape being circle or ellipse are used for morphometrics, feel free to contribute in the documentation of this PR or add them in the comments, which I will add on in the RTD. Thanks!!

This PR is ready for review.

Thanks @mariehbourget for the review of this PR. I've addressed your review suggestions. In addition, as discussed in the last meeting I've changed the default name of the morphometrics file from Morphometrics to axon_morphometrics.

What’s the current statut for that PR @mariehbourget @vs74 ? Is merging #399 essential before completing this feature?

Thanks, @mariehbourget for testing this PR.

One question though, should we extend the functionality to multiple files in a single folder?
It would make sense since our CLI for segmentation has this option. But we would need a mechanism to avoid overwriting the xls file with each image.

Thanks for pointing out the possibility of the functionality of generating multiple files in a single folder I also gave a thought about it while implementing this PR, but since it required overwriting the excel files, I didn't implement it.
Right now, I've asked the user about this here. Let's see what his response is.
If the user, doesn't require generating multiple files in a single folder, then we can merge this PR and open a new issue about the additional functionality. For this, we've to fix the renaming condition of the excel files generated. For instance, an excel file could be renamed as 'image_name_axon_morphometrics.xlsx'. This would meet the user's urgent requirement.
On the contrary, if he requires generating multiple files in a single folder, it will take 1-2 days for this PR to be ready as renaming of the morphometrics file and some tests need to added in this PR.

Looks like the user has images in different directories ( see here). I say for now, we can merge this PR If it is alright.
I'll open a follow-up issue on adding support for generating morphometrics when all the images are within a single directory.

Looks like the user has images in different directories ( see here). I say for now, we can merge this PR if it is alright.
I'll open a follow-up issue on adding support for generating morphometrics when all the images are within a single directory.
Let me know what do you think @mariehbourget.

Yes, that is ok with me, thanks! I'll approve the PR and let you do the merge.

authors: jcohenadad mariehbourget mathieuboudreau vs74

repo: neuropoly/axondeepseg


text:
qMRGenBatch: How is the sensitivity analysis plotted variable chosen?

@agahkarakuzu you're the one who wrote qMRGenBatch right? If so this question is for you.
I ran qMRGenBatch for two models today, inversion_recovery and vfa_t1.
For IR, the sensitivity analysis plot (fig 4) of the generated script showed me what I expected (Fitted T1 vs input T1).

For VFA, the sensitivity analysis plot didn't show me the variable that I as a user would have wanted to see (it showed me M0 vs M0).

What influences which variable is chosen in the qMRGenBatch for the sensitivity analysis to be performed, and how can we change it so that it is always a specific parameter of interest (in this case, for T1 mapping methods, show sensitivity for the T1 vals)? I tried looking in the qMRGenBatch code, but it was a little bit to abstract for me at first glance.

'M0' is certainly the first variable defined in Model.xnames. Careful: you
have to invert the order everywhere in vfa_t1.m (obj.lb, obj.ub,  obj.st,
obj.fx).
.
Le mar. 3 juil. 2018 à 18:24, Mathieu Boudreau notifications@github.com a
écrit :

@agahkarakuzu https://github.com/agahkarakuzu you're the one who wrote
qMRGenBatch right? If so this question is for you.
I ran qMRGenBatch for two models today, inversion_recovery and vfa_t1.
For IR, the sensitivity analysis plot (fig 4) of the generated script
showed me what I expected (Fitted T1 vs input T1).
[image: fig4]
https://user-images.githubusercontent.com/1421029/42232284-83eb0bc2-7ec8-11e8-9601-518d596efd00.png
For VFA, the sensitivity analysis plot didn't show me the variable that I
as a user would have wanted to see (it showed me M0 vs M0).
[image: fig4_2]
https://user-images.githubusercontent.com/1421029/42232289-89476c96-7ec8-11e8-90e1-961a85084ba9.png
What influences which variable is chosen in the qMRGenBatch for the
sensitivity analysis to be performed, and how can we change it so that it
is always a specific parameter of interest (in this case, for T1 mapping
methods, show sensitivity for the T1 vals)? I tried looking in the
qMRGenBatch code, but it was a little bit to abstract for me at first
glance.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/neuropoly/qMRLab/issues/244, or mute the thread
https://github.com/notifications/unsubscribe-auth/AHbLZD5AqdsnlFRCu4nmbBnyGWlPxjjFks5uC5rFgaJpZM4VBQun
.

Ah ha that makes sense, thanks @tanguyduval ! Maybe this should be documented in the wiki for developpers when creating new models (i.e. make the first variable in Model.xnames the most likely "interesting" variable of your model).
I'm curious to check if this could also be improved in any other models.

@mathieuboudreau qMRgenBatch calls qMRusage to retrieve commands for sensitivity analysis and single voxel curve. As @tanguyduval mentioned and you noted, documenting an ordering rule for independent variables is necessary.
To that end, I tried to explain it in my half-baked adding a new model guideline.

Thanks Agah, great wiki

@agahkarakuzu ah ha, thanks Agah! I should have checked to see if it was already there before suggesting it ;)

repo: neuropoly/qMRLab

license: mit


text:
User Conditioningd

유저들이 바라는 조건에 맞는 장소 추천
#앉아서 보고 듣기
SeeAndHear=['자동차극장',
 '극장',
'공연예술 극장',
'영화관',
 '경기장',
 '교육 컨설턴트']

#운동하기
Exercise = [ '스포츠 단지',
'건강 센터']

#복지
Welfare =['사회 복지 센터',
 '문화센터',
 '여행사']

#구경하기
Watch =['국립박물관',
 '미술관',
 '박물관',
 '역사 박물관',
 '전쟁 박물관',
 '현대미술관']

# 쇼핑하기
Shopping=[ '시가 전문점',
 '쇼핑몰',
 '커피용품 판매점',
'시장']

#놀기
Play = ['노래방',
'놀이 공원',
 '놀이공원',
     '테마파크']

#의미있는 곳
MeaningfulPlace = ['성지',
 '성당',
 '불교사찰',
 '문화유산보존지역',
 '관광 명소',
 '역사 유적지',
 '시청',
 '역사유적지',
 '역사적 명소',
'문화 유산 보존',
 '예술',
'정부청사',
'재단']

#밖 인공
OutdoorArtificial = ['다리',
 '대광장',
'고궁',
 '고분/왕릉',
 '타워']

#밖 자연
OutdoorNature=['공원',
'산',
 '국립공원',
 '산봉우리',
 '생태 공원']

#음식 먹기
EatingFood = ['와인 바',
 '음식점',
 '이탈리아 음식점',
 '일본 음식점',
 '딤섬 전문 레스토랑',
 '바 & 그릴',
 '술집',
 '중국 음식점',
 '카페',
 '칵테일바',
 '커피숍/커피 전문점',
 '수산 시장',
 '패밀리 레스토랑',
 '프랑스 음식점',
 '한국식 소고기 전문 음식점',
 '한식 고기구이 레스토랑'
 '재즈바']

hotel = ['호텔']

#1.음식 먹기
EatingFood = [
'딤섬 전문 레스토랑',
'음식점',
'이탈리아 음식점',
'일본 음식점',
'중국 음식점',
'패밀리 레스토랑',
'프랑스 음식점',
'한국식 소고기 전문 음식점',
'한식 고기구이 레스토랑'
]
#2. 마시기
Drinking = [ '바 & 그릴',
'술집',
'와인 바',
'재즈바',
'카페',
'칵테일바',
]

#3. 의미있는 곳
MeaningfulPlace = ['문화유산보존지역',
'불교사찰',
'성당',
'역사유적지',
'역사적 명소'
]
#4. 밖 인공,자연
Outdoors = ['고궁',
'관광명소',
'관광지',
'다리',
'대광장',
'타워',
'공원',
'국립공원',
'산'
]
#5. 앉아서 보고 듣기
SeeAndHear=['공연예술 극장',
'극장',
'영화관',
'자동차극장'
]
#6. 구경하기
Watch =['미술관',
'박물관'
]
#7. 놀기
Play = ['노래방',
'놀이공원'
]
#8. 쇼핑하기
Shopping=['쇼핑몰',
'시가 전문점',
'시장',
'커피용품 판매점',
]
#9. 운동하기
Exercise = ['건강 센터',
'스포츠 단지',
'스포츠단지'
]
#10. 복지
Welfare =['문화센터',
'정부청사'
]


text:
New I/O Banner

New banner for Instant Observability site

Closes #1979

repo: newrelic/developer-website


text:
feat: bump minimal agent version for agent upgrade scenarios

The new minimum version is supported by SUSE 15.6, else the pipelines will fail on the agent upgrade scenario.

Closing as this will be done when a new release needs to be published.

repo: newrelic/infrastructure-agent


text:
change source for nvidia-gpu

Summary
Migration of NVIDIA-GPU from docs to Install Framework
The owners of this repo are not experts in the subject matter of the quickstarts. We review for the quickstart to be functional and for security risks. If you are seeking feedback on the content of the quickstart, please seek out a subject matter expert. If you are not an internal NR contributor, we can do our best to connect you with a content reviewer.
Pre merge checklist

[x] Did you check you NRQL syntax? - Does it work?
[x] Did you include a Data source and Documentation reference?
[x] Are all documentation links publicly accessible?
[x] Did you check your descriptive content for voice and tone?
[x] Did you check your descriptive content for spelling and grammar errors?
[x] Did you review your content with a subject matter expert? (e.g. a Browser agent quickstart is reviewed with a member of the Browser Agent team)

Dashboards

[ ] Does the PR contain a screenshot for each of your dashboards?
[ ] Do your screenshots show data?
[ ] Has the sanitization script been run on each dashboard?

Alerts

[ ] Did you check that your alerts actually work?

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

can be merged once this PR is merged:
https://source.datanerd.us/nr1-dev-experience/shared-component-framework-configs/pull/299

@abhinav1602, Could you sign "Contributor License Agreement (cla)".

repo: newrelic/newrelic-quickstarts


text:
Fix Rack instrumentation for "kwarg middlewares" in Ruby 3.0

Overview
Prior to this change, in Ruby 3.0, using a middleware with keyword arguments would fail with an ArgumentError.
This fix is pretty much the same as https://github.com/rack/rack/pull/1505
Reviewer Checklist

[ ] Perform code review
[ ] Add performance label
[ ] Perform appropriate level of performance testing
[ ] Confirm all checks passed
[ ] Add version label prior to acceptance

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

The GHA failures seems to be due to this branch lacking the fix from #696

@walro yes, it does look like that is the case, and it is preventing our tests from running on those older ruby versions. If you merge (or rebase) the new changes in dev into your branch, this should allow the tests to run, so we can confirm that the tests pass on all ruby versions the agent supports.

f you merge (or rebase) the new changes in dev into your branch, this should allow the tests to run, so we can confirm that the tests pass on all ruby versions the agent supports.

Rebased and pushed!

authors: CLAassistant tannalynn walro

repo: newrelic/newrelic-ruby-agent


text:
Fix small issues and typos in Instructions

These fixes are only for some typos and other small issues that I noticed while preparing for FS so it can be merged without much work and review.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks @olanguyenvan for the PR!

authors: CLAassistant olanguyenvan tangollama