text:
[Plugin] Vault - No implementation available for apiRef{plugin.vault.service}

Expected Behavior
Seeing the vault secrets list on my component when the proper annotation is set.
Actual Behavior
Instead of seeing the secret list, there's an error card with the following message:
No implementation available for apiRef{plugin.vault.service}
Steps to Reproduce
Create a component on a fresh install of backstage with the proper annotation
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  annotations:
    vault.io/secrets-path: my-team/backstage

Install and configure the Vault plugin.
I don't know how to provide more context about that issue.
Context
I'm using the Renew Token method as shown here:
https://github.com/backstage/backstage/tree/master/plugins/vault-backend#renew-token>
I have found the https://github.com/backstage/backstage/issues/4101 issue and check if I don't have duplicate versions of the plugin, although I do have duplicate versions of @backstage/backend-common and @backstage/core-components due to plugins that didn't yet upgrade to the latest versions. Could this be the issue?
Your Environment

Browser Information:  Firefox 105.0b1 (64-bit)

Output of yarn backstage-cli info:

OS:   Linux 5.18.17-1-MANJARO - linux/x64
node: v16.14.0
yarn: 1.22.19
cli:  0.18.1 (installed)
backstage:  1.5.0

Dependencies:
  @backstage/app-defaults                          1.0.5
  @backstage/backend-app-api                       0.2.0
  @backstage/backend-common                        0.13.5, 0.15.0
  @backstage/backend-plugin-api                    0.1.1
  @backstage/backend-tasks                         0.3.4
  @backstage/backend-test-utils                    0.1.27
  @backstage/catalog-client                        1.0.4
  @backstage/catalog-model                         1.1.0
  @backstage/cli-common                            0.1.9
  @backstage/cli                                   0.18.1
  @backstage/config-loader                         1.1.3
  @backstage/config                                1.0.1
  @backstage/core-app-api                          1.0.5
  @backstage/core-components                       0.11.0, 0.9.5
  @backstage/core-plugin-api                       1.0.5
  @backstage/errors                                1.1.0
  @backstage/integration-react                     1.1.3
  @backstage/integration                           1.3.0
  @backstage/plugin-api-docs                       0.8.8
  @backstage/plugin-app-backend                    0.3.35
  @backstage/plugin-auth-backend                   0.15.1
  @backstage/plugin-auth-node                      0.2.4
  @backstage/plugin-catalog-backend-module-github  0.1.6
  @backstage/plugin-catalog-backend-module-ldap    0.5.2
  @backstage/plugin-catalog-backend                1.3.1
  @backstage/plugin-catalog-common                 1.0.5
  @backstage/plugin-catalog-graph                  0.2.20
  @backstage/plugin-catalog-import                 0.8.11
  @backstage/plugin-catalog-node                   1.0.1
  @backstage/plugin-catalog-react                  1.1.3
  @backstage/plugin-catalog                        1.5.0
  @backstage/plugin-explore-react                  0.0.20
  @backstage/plugin-explore                        0.3.39
  @backstage/plugin-github-actions                 0.5.8
  @backstage/plugin-home                           0.4.24
  @backstage/plugin-kubernetes-backend             0.7.1
  @backstage/plugin-kubernetes-common              0.4.1
  @backstage/plugin-kubernetes                     0.7.1
  @backstage/plugin-org                            0.5.8
  @backstage/plugin-permission-backend             0.5.10
  @backstage/plugin-permission-common              0.6.3
  @backstage/plugin-permission-node                0.6.4
  @backstage/plugin-permission-react               0.4.4
  @backstage/plugin-proxy-backend                  0.2.29
  @backstage/plugin-scaffolder-backend             1.5.0
  @backstage/plugin-scaffolder-common              1.1.2
  @backstage/plugin-scaffolder                     1.5.0
  @backstage/plugin-search-backend-module-pg       0.3.6
  @backstage/plugin-search-backend-node            1.0.1
  @backstage/plugin-search-backend                 1.0.1
  @backstage/plugin-search-common                  1.0.0
  @backstage/plugin-search-react                   1.0.1
  @backstage/plugin-search                         1.0.1
  @backstage/plugin-shortcuts                      0.3.0
  @backstage/plugin-stack-overflow                 0.1.4
  @backstage/plugin-tech-radar                     0.5.15
  @backstage/plugin-techdocs-backend               1.2.1
  @backstage/plugin-techdocs-module-addons-contrib 1.0.3
  @backstage/plugin-techdocs-node                  1.3.0
  @backstage/plugin-techdocs-react                 1.0.3
  @backstage/plugin-techdocs                       1.3.1
  @backstage/plugin-user-settings                  0.4.7
  @backstage/plugin-vault-backend                  0.2.1
  @backstage/plugin-vault                          0.1.2
  @backstage/release-manifests                     0.0.5
  @backstage/test-utils                            1.1.3
  @backstage/theme                                 0.2.16
  @backstage/types                                 1.0.0
  @backstage/version-bridge                        1.0.1

So the plugin isn't completely properly installed. If your packages/app does render the <EntityVaultCard> in the static JSX root, then that should be happening automatically.
Could it be that you are breaking up your entity page with functional components? That's a typical gotcha. Like here, that could NOT have been const CicdContent = () => ... ie a function, it HAS to be static JSX. That's a gotcha with the way that autodiscovery of plugins happens.

Hello @freben, thanks a lot for your help. It was indeed what happened, I had made the following function :
function OptionalVaultCard(): JSX.Element {
  const { entity } = useEntity();

  if (isVaultAvailable(entity)) {
    return <Grid item md={6} xs={12}>
      <EntityVaultCard />
    </Grid>
  }
  return <></>
}

Removing it and using the <EntityVaultCard> directly worked. If I may ask, is there a way to disable the MissingAnnotationEmptyState display when the annotation is missing? This is what I was trying to achieve. If there's no way to achieve that it's absolutely ok, I'm just curious :smile:
Thanks a lot for your help!

Answering my own question :
    <EntitySwitch>
      <EntitySwitch.Case if={isVaultAvailable}>
        <Grid item md={6} xs={12}>
          <EntityVaultCard />
        </Grid>
      </EntitySwitch.Case>
    </EntitySwitch>

Sorry about the noise! And thanks again!

Hi, i'm running into the same issue but in my case I need the actual entity object and idk how to get it without refactoring into a functional component and use useEntity hook. Any suggestions?

authors: Depado depado freben pzsette

repo: backstage/backstage


text:
🐛 Bug Report:

📜 Description
Running the container with unpredictable uid and gid (the new node unprivileged user suggested in the documentation) can lead to inconsistent behavior, especially when mounting volumes.
👍 Expected behavior
We should edit the documentation to create a predictable uid and gid for the container:
# Create user and group with predictable uid and gid
RUN groupadd --gid 1000 node && useradd --uid 1000 --gid 1000 node

👎 Actual Behavior with Screenshots
I run a job to fix permissions of a mounted volume and get:
k logs fix-permission-4chd5
chown: unknown user/group node:node

👟 Reproduction steps

Start with a previous deployment and a mounted folder with files created as root
Update the container as suggested in the backstage.io doc
try to run a job to let node user chown volume files

apiVersion: batch/v1
kind: Job
metadata:
  name: fix-permission
spec:
  template:
    spec:
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: my-volume-claim
      containers:
      - command:
        - chown
        - -R
        - node:node
        - /workdir
        image: busybox
        name: fix-permission
        resources: {}
        volumeMounts:
        - mountPath: /workdir
          name: my-volume
      restartPolicy: Never
status: {}

📃 Provide the context for the Bug.
No response
🖥 Your Environment
No response
👀 Have you spent some time to check if this bug has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
No response

I did my homework without paying enough attention. The image already uses 1000:1000 coming from the root node image. Closing this. Hopefully the thread can help someone else having the same troubles

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
🚀 Create a CLI tool to simplify the Backstage onboarding with plugin install, image creationa dn pushing it to image repo.

🔖 Feature description
Currently we have figure which plugins to install, which code do we have to inject from the documentation. Once that is done, we have build our another image and then run it. Instead of this, we can create a CLI tool which can take the list of plugins as  inputs and then add those relevant code blocks and then build the image.
🎤 Context
In my case when i tried searching and installing backstage in POC phase, i saw that onboarding experience can be improved if we can automate and simplify the process of initialising and running the project
✌ Possible Implementation
The CLI will do the following :

Pull the latest code from Github
based on the flag inputs, will add the necessary code in the relevant folders
package the application and create a docker image
based on the image repo flag and url , it will also push the code to the upstream repository.

👀 Have you spent some time to check if this feature request has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
None

There have been some major efforts for the new backend and frontend systems, making the amount of code in your main packages being almost nothing. And there's an ongoing workstream to try to get dynamic loading of plugins into place, which reduces it to no code at all.
But having code ownership will always be a thing. You'll want to make custom plugins and adaptations on top of the open source offering.
Closing this as being worked on :)

authors: freben krishnaduttp


text:
🐛 Bug Report: New scalable events service depends on events-backend being up first

📜 Description
Under the new implementation for the scalable EventsService, calling subscribe depends on the events-backend plugin being up and running to receive the API request - there a few event modules (eg. eventsModuleGithubEventRouter, eventsModuleGitlabEventRouter, etc) that broke since modules are initialized before their corresponding plugin and so the subscribe() calls in those modules fail with a "503 service unavailable" since the events-backend has not started up yet.
A simple fix for those modules is to wrap the calls in a lifecycle.addStartupHook however this brings about a boarder issue of how other plugins that interact with the EventsService also now have an indirect dependency of having the events-backend be initialized first.
For example we have catalog providers that are hooked up to events and our catalog backend startup fails when hitting the subscribe calls if the events-backend is not up yet. There's probably a few more places in this repo where we not handling the subscribe call failing correctly but I'm not sure if the solution here is to ensure we have proper error handling everywhere we call subscribe - some of which is part of startup code so not sure if there should also be a retry mechanism since we don't know if either the events backend is simply down or not yet fully started.
👍 Expected behavior
See description
👎 Actual Behavior with Screenshots
See description
👟 Reproduction steps
Install the events-backend with one of the events modules that hooks up a sub topic router such as https://github.com/backstage/backstage/blob/master/plugins/events-backend-module-github/src/service/eventsModuleGithubEventRouter.ts - when starting up the backend the module will fail to initialize with a 503 on the subscribe call
📃 Provide the context for the Bug.
No response
🖥 Your Environment
Backstage v1.32.1
👀 Have you spent some time to check if this bug has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
Yes I am willing to submit a PR!

wanted to open this issue first to discuss the problem before putting up fixes for the broken modules in question since not sure if the appropriate fix is to address it everywhere we're currently calling subscribe as part of startup code or if a wider fix to the EventsService is more pragmatic since the order of init limitation makes the API a bit messy now
cc: @Rugvip to get your thoughts when you get a chance 🙏

@kuangp ah yep thank you, this is gonna be an issue indeed! 👍
Given the behavior of the subscribe function I think it makes sense to retry. It's actually already part of the polling to try to upsert subscriptions if they don't exist, just not the initial request: https://github.com/backstage/backstage/blob/232a9600f07d40e5a55b1f5eb8cb85500ea6ec07/plugins/events-node/src/api/DefaultEventsService.ts#L211-L228
I think the best fix here might be to simply skip the initial subscription creation and leave it to the polling. Just gotta take care to keep the 404 behavior to handle the case where the events backend isn't present: https://github.com/backstage/backstage/blob/232a9600f07d40e5a55b1f5eb8cb85500ea6ec07/plugins/events-node/src/api/DefaultEventsService.ts#L175-L181

I ran into this error after upgrading to the latest Backstage release. Is there a workaround I can use, or do I need to downgrade?

Hey guys, just wanted to loop in the conv and say that we're also experiencing the same issue when tried to upgrade from 1.27.7 to 1.32.0.

Fix is on the way in #27227

I'm sure this will be out quickly, but is there a workaround in the meantime? I am seeing this on 1.31.3 I believe.

I am still seeing this issue on 1.32.1
Not sure if it was required, but I also set up useEventBus: auto in my app-config.yaml under events

It’s fixed in 1.32.2
On Thu, Oct 24, 2024 at 12:35 Cody Fidler @.***> wrote:

I am still seeing this issue on 1.32.1
Not sure if it was required, but I also set up useEventBus: auto in my
app-config.yaml under events
—
Reply to this email directly, view it on GitHub
https://github.com/backstage/backstage/issues/27207#issuecomment-2436193948,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFWH2SMNLXSGW6J5463YVDZ5FDWZAVCNFSM6AAAAABQCMLWQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWGE4TGOJUHA
.
You are receiving this because you commented.Message ID:
@.***>

🤔 I can still see this error in the latest version of Backstage.
[backend]: 2024-11-27T09:11:44.810Z catalog warn Event subscribe request failed with status 404, events backend not found. Will only receive events that were sent locally on this process.

authors: Ginger0913 LegendSebastianoL Rugvip Silthus kuangp kunickiaj polarp


text:
Search: DefaultCatalogCollator should index spec.profile section in group and user entities

Feature Suggestion
DefaultCatalogCollator is only designed for entities of kind Component. It would be neat to support other build in entity kinds too.
Users and groups have a displayName field in spec.profile. It is important to index this field too, otherwise it's hard to search for users by their name. For example, Backstage comes with the test user Calum Leavy, which you can't find by searching for calum right now. You can neither find him by full name with a space in between: Calum Leavy. You have to explicitly enter the entity name Calum.Leavy to find him (with a dot).

Possible Implementation
Extend DefaultCatalogCollator.execute() with different logic for each build in kind.

Context

@backstage/techdocs-core

Thank you for opening this issue @Fox32!

I wonder what your thoughts on this is. Is this special, or something we should support out of the box? Is it a separate collator with a separate document type, and a custom <CatalogResultListItem>? I could imaging that the <CatalogResultListItem> of a user contains an avatar.
What is the general suggestion on this. If I want to index additional fields in my entity, should I replace the DefaultCatalogCollator, or create an additional collator that just index my custom field?

Maybe the guiding light on "what goes in DefaultCatalogCollator vs. what should be left to app integrators" is what's required (or even optional) in the Well-Known Kinds documentation?
Although I guess for your User example, that doesn't help much, since the spec.profile field is quite open-ended.
I think there's still lots of room to flesh out the DefaultCatalogCollator to cover the 80% use-case though, so anything that maps to making the experience make sense for the well-known types are definitely welcome.
For anything that seems like an obvious thing to include, but doesn't currently map to the well-known kinds documentation (e.g. your avatar suggestion for User), I almost feel like it should put weight toward making it a more standard/recommended part of the kind's schema?
I dunno.  What do you think, @Fox32 @emmaindal @freben?

Our discussion when developing the DefaultCatalogCollator was to look into the required fields of all different Kinds and focusing on supporting those in the DefaultCatalogCollator. Looking at those again, it seems like we are missing a few required ones. Here is a collection of all required ones for all kinds:

[x] kind [required] (All)
[x] spec.type [required] (Component, API, Group, Resource)
[x] spec.lifecycle [required] (Component, API)
[x] spec.owner [required] (Component, API, Resource, System, Domain)
[ ] metadata.title [required] (Template, API)
[ ] spec.parameters [required] (Template)
[ ] spec.children [required] (Group)
[ ] spec.memberOf [required] (User)

With that said and with a little bit more thinking around this I would say - the DefaultCatalogCollator should support the required fields, if we want to extend a Collator beyond that, it should be created as its own. Could still be contributed, so that others can use it of course. And I'm not saying its the right thing to do!

I don't worry too much about open endedness I think; let's not support any random unknown extension field from the start, sure. So the candidates can be the fields that are in the core catalog-model.
But optionality is probably not tied to the suitability as a search parameter. Optionality is more about whether we want every single org everywhere to be forced to supply a field whether they want to or not. Those that do, probably care about them.
Just from an end user POV, it's probably fairly reasonable to want to be able to search by full name and email. Or turned the other way around, one might be frustrated or surprised if unable to search by exact email or name match, if you can clearly see it front and center on the user page. I also believe it's not too crazy to have metadata.name that's fairly useless or opaque (perhaps machine generated), while the full name and email are fine from the source.

To clarify the last part, I've seen sentiments similar to "Why can't we have @-signs in the name, we don't have user IDs in our HR system. We key off of full emails where the name part can have collisions across domains. Ok we'll just put the UUID in the name for now or generate something by kebab-case-converting the sanitized email" :) Or so.

Hmm, playing around with it a bit more, im not sure if the decision to only include required fields is a good idea. And it's also not followed properly 😅 For example, description is indexed, even though it's optional and I think it's a very valuable field 😉 Also run into tags not being indexed, but that would be quite useful too. Like searching for all tags:go services that are related to payments.
In my option I would expect that the DefaultCatalogCollator supports every reasonable field that is part of the core data model. Reasonable excludes for me stuff like annotations and labels that have a technical background. I wrote my own DocumentDecorator to extend the index a bit and it seems to work quite well, but I actually think it should work out of the box.
Another thing I noticed: the DefaultCatalogCollator indexes the owner field, but we might also want to take the relations into account - not sure though.

The drawback with description is also that it risks increasing the signal to noise ratio a whole lot, while, in my humble opinion, very very rarely providing intent-matching useful hits :)
But I know that I'm a very opinionated, niche, technical user who almost without exception is interested solely in searching among IDs and sometimes MAYBE some id-like or enum-shaped things. But even then only as an opt-in. The rest just confuses my result set :)

I think that case is already supported quite well. If you know what you are looking for, you can search inside a specific field, like title:playlist, ignoring all other fields.

Hi 👋🏻, just sharing a thought:
For this specific case, my first thought was: when using the Elastic Search engine, could the problem be solved with a custom analyser that disregards the "dot" in the field when searching? But then another question arose, is this something supported by other engines as well? So this seems to be a workaround, not the solution...
Is there any reason the DefaultCatalogCallator doesn't accept which fields to index as an optional setting? Required fields can be the standard fields, but optionally add others?
Sorry if this question has already been answered, I may have missed it in the previous comments.

@Fox32 where do you see this issue now after all the comments?

Hi, @Rugvip do you have any additional suggestions for this case?

Related: https://github.com/backstage/backstage/issues/11407

Related: https://github.com/backstage/backstage/issues/14530

Hey everyone following along on this issue!  I'm consolidating this feature idea (as well as others) into a single issue.
I encourage you to continue the discussion (and perhaps, implementation? 😉) here: https://github.com/backstage/backstage/issues/14961

authors: Fox32 camilaibs emmaindal freben garyniemen iamEAP


text:
Support RDS login via AWS IAM role

The AWS RDS database service provides the ability to request a temporary (15 min) password that can be used to login to a mysql or ps database. https://aws.amazon.com/premiumsupport/knowledge-center/users-connect-rds-iam/
Knex also provides the ability to configure the password dynamically and expire a configuration after a period of time.
As I understand it, now it is only possible to configure the backend database with a fixed password, and it can only be set at startup.
Feature Suggestion
It would be great if it were possible to configure the DatabaseManager to use dynamic configuration like the following.
const knex = require('knex')({
  client: 'postgres',
  connection: async () => {
    const { token, tokenExpiration } = await someCallToGetTheToken();
    return {
      host : 'your_host',
      port : 3306,
      user : 'your_database_user',
      password : token,
      database : 'myapp_test',
      expirationChecker: () => {
        return tokenExpiration <= Date.now();
      }
    };
  }
});

Possible Implementation
Maybe this might be configurable in the backend app rather than in the app-config.yaml.
Context
By using the AWS get-password feature, we can improve security by not writing a password to the filesystem. Also using an expiring password means that if the password is compromised it is no longer valid in a short period of time.
Also using this temporary password, solves password rotation security requirements.

I haven't looked into more than reading about how it might be done. I wasn't totally sure it wasn't already possible. I have some time tomorrow so I might take a look then and see if I can put together a patch.


text:
Export airbrake plugin helper function

Hey, I just made a Pull Request!
This PR adds a isAirbrakeAvailable helper function to the Airbrake plugin, much like the helper functions that exist in other plugins (see isAdrAvailable and isGithubActionsAvailable for examples of near identical help functions that exist in other plugins). It is for use in the EntityPage so that if you are using the airbrake plugin you can opt to not show the airbrake tab on your entitie's page if the entity's airbrake id is not set in the catalog-info.yml.
Add it to the packages/app/src/components/catalog/EntityPage.tsx like so:
import { EntityAirbrakeContent, isAirbrakeAvailable } from "@backstage/plugin-airbrake"
.
.
.

const websiteEntityPage = (
  <EntityLayout>
    <EntityLayout.Route path="/" title="Overview">
      {overviewContent}
    </EntityLayout.Route>
        <EntityLayout.Route if={isAirbrakeAvailable} path="/airbrake" title="Airbrake">
              <EntityAirbrakeContent />
         </EntityLayout.Route>

The airbrake tab will appear on the entity pages that have the airbrake id set in the catalog-info.yml and appear like so:

And if the id is not set it will not appear:

:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[x] Added or updated documentation
[x] Tests for new functionality and regression tests for bug fixes
[x] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Issue
https://github.com/simplybusiness/backstage/issues/1796

Hey :wave: I think this might need rebasing against the mainline branch as there's some changes in here that don't seem like your own or relevant to this PR 😅

Hey 👋 I think this might need rebasing against the mainline branch as there's some changes in here that don't seem like your own or relevant to this PR 😅

Ah! Thank you! Just rebased! 😄 benjdlambert

source: gharchive/pull-request

authors: LightningStairs benjdlambert


text:
[Azure DevOps] Updated to use encodeURIComponent

Hey, I just made a Pull Request!
Small updated based on this comment: https://github.com/backstage/backstage/pull/20729#discussion_r1405804594
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Thanks @Rugvip and also to @freben for the comment about this 🚀


text:
Update wording in user-settings README to clarify example location

Hey, I just made a Pull Request!
The README for the user-settings plugin referenced an example screen in the Demo app, where it was linking to the example app in the main Backstage repository.
Discord link for more details: https://discord.com/channels/687207715902193673/1279569368875008176/1280616399256948971
:heavy_check_mark: Checklist

[ ] A changeset describing the change and affected packages. (more info)
[x] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

@darylgraham this needs a changeset too, you can fix by running yarn changeset and selecting the correct package and patch bump :tada:

@darylgraham this needs a changeset too, you can fix by running yarn changeset and selecting the correct package and patch bump 🎉

@benjdlambert is a change set needed for this? The contributing guide says they aren’t needed for changes that do not affect code.

Changesets are also not needed for changes that do not affect the published version of each package, for example changes to tests or in-line source code comments.

Happy to create one, just wanted to be sure of the process.

Yeah, in order to update the readme in places like https://npmjs.com/package/@backstage/plugin-user-settings then it needs a release to be triggered.

Thanks @benjdlambert now I understand. Changeset added.

@darylgraham sorry, also need to DCO sign that last commit too. some more info here

Sorry, I thought I had all of my commits being signed off. Not sure how that one got through.

@darylgraham this needs a changeset too, you can fix by running yarn changeset and selecting the correct package and patch bump 🎉

@benjdlambert is a change set needed for this? The contributing guide says they aren’t needed for changes that do not affect code.

Changesets are also not needed for changes that do not affect the published version of each package, for example changes to tests or in-line source code comments.

Happy to create one, just wanted to be sure of the process.

@darylgraham this made me LOL as this is mentioned in the paragraph just below the one you quoted here 🤣

@darylgraham this needs a changeset too, you can fix by running yarn changeset and selecting the correct package and patch bump 🎉

@benjdlambert is a change set needed for this? The contributing guide says they aren’t needed for changes that do not affect code.

Changesets are also not needed for changes that do not affect the published version of each package, for example changes to tests or in-line source code comments.

Happy to create one, just wanted to be sure of the process.

@darylgraham this made me LOL as this is mentioned in the paragraph just below the one you quoted here 🤣

Argh! I just went back to check - sure enough, I had read the first part of the next paragraph and just blanked on the sentence that specifically mentions the readme file. Gah! Thankfully you guys are here to pick up my mistakes while I’m learning all this stuff 😂

authors: awanlin benjdlambert darylgraham


text:
fix(Scaffolder): The filename or extension is too long. #2527

Fixes #2527, thanks @freben for the help

I think that we need to check if that property exists, as it looks like if the binary fails then it causes the integration tests to fail.
It's because the library is not installed part of the e2e tests.
Also will need a changeset too :)

I've updated and rebased my branch on top of latest master. Not sure what's wrong again with CI.

Is master broken?
[Check Failed] package.json has mismatched field, dependencies[@backstage/core], change from "^0.4.1" to "^0.4.2"?
[Check Failed] package.json has mismatched field, devDependencies[@backstage/cli], change from "^0.4.2" to "^0.4.3"?
[Check Failed] package.json has mismatched field, devDependencies[@backstage/test-utils], change from "^0.1.5" to "^0.1.6"?

This is coming from the FOSSA plugin which was merged just now. https://github.com/backstage/backstage/pull/3674
Investigating further. Should be required to bump versions since we made a new release just today!

Thank you! Will close this though, as we've just moved away from using nodegit at all in #3805

authors: OrkoHunter Rugvip XBeg9 benjdlambert


text:
feat(plugin-auth-backend): resolve CVE-2021-39171

Hey, I just made a Pull Request!
passport-saml@^2.0.0 installs a version that flags code scans with CVE-2021-39171. This bumps the package to 3.1.0, which includes a resolution for this CVE.
Due to the 2.0.0 -> 3.1.2 upgrade, the auth.saml.cert config parameter is now required.

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[ ] Tests for new functionality and regression tests for bug fixes
[ ] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

Looking into the typing issue now.

Hi! Thanks for looking at this. Haven't reviewed yet, but see https://github.com/backstage/backstage/pull/7015 which addresses the same. Due to the stability level of this package and the breaking change that this implies, it's awaiting merge a little longer.

@freben I'm wondering if because it's a CVE do we not just merge the fix and get it out sooner rather than later and just accept that we broke the stability index because of a security issue? Not really sure what other projects do here tbf.

@freben your change is identical. I don't know why I couldn't find it before I did this work ha. Yours has a better change log message too, so I'll close this out in favor of yours.
🍻

authors: benjdlambert ellisio freben


text:
catalog-react: mark useEntityPermission as alpha

Marked useEntityPermission as alpha since the underlying permission framework is under active development.

Hm there's more steps than this to doing it right?
https://github.com/backstage/backstage/blob/b129681aba2ecf488d40b1120547860e7d2e4088/packages/backend-common/package.json#L12
see the alphaTypes and files at the bottom etc

@freben you know more then me 👍 I'll add that

This hook is already used in plugin-catalog, shall we update that usage as part of this PR? Surprised this doesn't break CI 🤔 .

💡 Ahhh, got it! Makes sense to me, thanks for the clarification 👍 .

authors: freben jhaals mtlewis


text:
A crash occurs. (Zelda Tears of the Kingdom) Inf. Zonai Device Duration

Inf. Zonai Device Duration (No Blink + Despawn)
ryujinx/yuzu
When running, if the quantity changes, it crashes and the game shuts down.

What codes are you using paste your entire file.

license: MIT


text:
Can not get esbuild configuration working, after migrating from TheBrainFamily/cypress-cucumber-preprocessor

Current behavior
I'm trying to migrate to this version of cypress-cucumber-preprocessor using the recommended esbuild method. However, I can't seem to get it working at all.
What is special about our setup is that our cypress setup is independent from the application it is testing. This should not be a problem as we only use it for e2e testing. This however does mean, that a lot of assumed configurations or packages are not presenting our setup.
# package.json
{
  "name": "integrations",
  "license": "UNLICENSED",
  "private": true,
  "description": "Integration testing using Cypress",
  "main": "index.js",
  "scripts": {
    "open": "cypress open",
    "test": "cypress run"
  },
  "dependencies": {
    "@badeball/cypress-cucumber-preprocessor": "latest",
    "@bahmutov/cypress-esbuild-preprocessor": "latest",
    "@esbuild-plugins/node-modules-polyfill": "latest",
    "cypress": "latest",
    "cypress-file-upload": "latest",
    "cypress-iframe": "latest",
    "cypress-otp": "latest",
    "esbuild": "latest",
    "uuid": "latest"
  },
  "cypress-cucumber-preprocessor": {
    "nonGlobalStepDefinitions": true,
    "stepDefinitions": [
      "[filepath]/steps.{js,ts}"
    ]
  }
}

# cypress config

const { defineConfig } = require("cypress");

module.exports = defineConfig({
  video: false,
  // chromeWebSecurity: false,
  defaultCommandTimeout: 10000,
  env: {
    TAGS: 'not (@ci-broken or @unstable)',
  },
  retries: {
    runMode: 2,
    openMode: 0,
  },
  e2e: {
    setupNodeEvents: require('./cypress/plugins/index.js'),
    specPattern: 'cypress/e2e/**/*.{feature,features,spec.js}',
    baseUrl: 'https://app.localtest.me',
  },
})

# cypress/plugins/index.js
/// <reference types="cypress" />
// ***********************************************************
// This example plugins/index.js can be used to load plugins
//
// You can change the location of this file or turn off loading
// the plugins file with the 'pluginsFile' configuration option.
//
// You can read more here:
// https://on.cypress.io/plugins-guide
// ***********************************************************

const createBundler = require("@bahmutov/cypress-esbuild-preprocessor");
const preprocessor = require("@badeball/cypress-cucumber-preprocessor");
const createEsbuildPlugin = require("@badeball/cypress-cucumber-preprocessor/esbuild");
// const nodePolyfills = require('@esbuild-plugins/node-modules-polyfill').NodeModulesPolyfillPlugin;
const generateOTP = require('cypress-otp');

// This function is called when a project is opened or re-opened (e.g. due to
// the project's config changing)

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = async function setupNodeEvents(on, config) {
  await preprocessor.addCucumberPreprocessorPlugin(on, config);

  on(
    "file:preprocessor",
    createBundler({
      plugins: [
        // nodePolyfills(),
        createEsbuildPlugin.default(config),
      ],
    })
  );

  on('task', { generateOTP });

  // Make sure to return the config object as it might have been modified by the plugin.
  return config;
}

There are no other config files.
When running this, cypress keeps crashing with this error:
Build failed with 6 errors:
node_modules/esbuild/lib/main.js:1709:17: ERROR: Could not resolve "fs"
node_modules/esbuild/lib/main.js:1710:17: ERROR: Could not resolve "os"
node_modules/esbuild/lib/main.js:1711:19: ERROR: Could not resolve "path"
node_modules/esbuild/lib/main.js:1858:28: ERROR: Could not resolve "child_process"
node_modules/esbuild/lib/main.js:1859:21: ERROR: Could not resolve "crypto"

When I try to run it with nodePolyfills as described here, we get an other error:
process is not defined

Which leads me further down a rabbit hole, that I think is the wrong rabbit hole.
Desired behavior
Get cypress running with @badeball/cypress-cucumber-preprocessor using the suggested esbuild-cjs approach.
Test code to reproduce
See code above.
Versions

Cypress version: 10.3.1
Preprocessor version: 11.5.1
Node version: 16.14.2

Checklist

[x] I've read the FAQ.
[x] I've read Instructions for logging issues.
[x] I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).

I can't reproduce the issue with the provided information alone. I suspect that there's something going on with your actual step definitions. In any case, you will have to provide me a cloneable repository that allows me to observe the error.

authors: Bertg badeball

repo: badeball/cypress-cucumber-preprocessor


text:
Badge request: Cirrus CI

Hello!
Please, add Cirrus CI badge(s).
https://cirrus-ci.org/
Badges from this service itself: https://cirrus-ci.org/guide/writing-tasks/#embedded-badges
/cc @fkorotkov
Thanks!

Hello @AlexWayfer!
Do you know if Cirrus CI has a proper json/xml/... API or has plans to implement one? We could use the badges of the service itself to construct our own, but this approach is more brittle (see #1329 and #1985 for examples of Shields's badges breaking because of changes in the service's own badges). 😉
Cheers,
Pyves

Do you know if Cirrus CI has a proper json/xml/... API or has plans to implement one?

I don't know, so I've mentioned @fkorotkov from Cirrus CI: I think he can help.

Awesome, let's wait for his thoughts then!

Sorry I'm a bit out of context. Cirrus CI already provides a way to get shields-like badges on it's own. See documentation.
shileds.io gets a status from different APIs and shows badges? Basically just proxies? Do I understand it correctly? How then shileds.io handles authentication to show badges for private repositories? Our endpoint verifies that a request comes from a user with read permissions to a repository.
Do we need to implement some kind of special API?

Hi @fkorotkov!
Yes, most of the time we make an API request and then format the data into the badge. Compared to vendor native badges, the badges we host are more consistently customizable in terms of label, color, and shape, though for many users it makes no difference.
For security reasons, we don't accept secret read–write tokens on the public servers. Users who wish to host their own copy of Shields could configure their own server with a global token. We have this in place for many other services.
The best way to do this on the public server is for you to let users generate a status-only API key. That's a token which provides access solely to read build status, and thus would be safe to include in a public web page. If you give us a way to reject other tokens, like giving all the status-only keys a status_ prefix – we can help keep your users secure.
UptimeRobot has this, for example: it's called a monitor-specific API key. They all start with m which means we outright reject the user tokens which give access to more.
In terms of a JSON status endpoint, we could suggest something if it would be helpful. Most services offer latest build on default branch + latest build on a specific branch. If you have workflows, or details on tests passed / failed, you could include that as well.

Ok, cool! Seems we'll need to do something on our side for these status-only API key. Will put on our roadmap. 🙌

Will put on our roadmap. raised_hands

I think, it's private. Do you have any predictions for this task?

We did implement a proper API access but not yet a way to have scoped access tokens for such a use case. Created https://github.com/cirruslabs/cirrus-ci-docs/issues/200 to publicly track it.

We did implement a proper API access but not yet a way to have scoped access tokens for such a use case. Created cirruslabs/cirrus-ci-docs#200 to publicly track it.

Thank you!

authors: AlexWayfer PyvesB fkorotkov paulmelnikow

repo: badges/shields

license_source: bigquery


text:
add logoSize to globalQueryParams

After I merged https://github.com/badges/shields/pull/9191 I deployed it to staging, gave it a quick test and realised we missed something.
This param was only taking effect for static badges. It was ignored for dynamic ones.
This change fixes it, but it does also raise follow up questions which I'll note in an issue

The endpoint badge does not support logoSize for now. Maybe you can also support logoSize for the endpoint badge in this PR.


text:
Image preprocessing before upload to s3

Hi, I was wondering whether it was possible to perform some image preprocessing with lwip (https://github.com/EyalAr/lwip#usage) prior to upload?

Hi @kenoung!
What solution did you find?

@panasevich I didnt. I needed this because the (cropper)[https://github.com/fengyuanchen/cropper/blob/master/README.md] I was using provided me with coordinates. So I switched to a different (library)[https://github.com/Foliotek/Croppie] that did the cropping on the front end.
Sorry I can't help!

authors: kenoung panasevich

license: mit


text:
All groups option is missing in customer group discount.

Terms

[X] Before you openning this issue, i have checked if the issue has already been reported.

Bagisto Version(s) affected
v2.0.0-BETA-1
Issue Description
While trying to create a customer group offer for all the groups in the product edit page on the admin panel. but it is showing only separate options.
Preconditions

Admin should be logged in.
Admin should have at least one product.

Steps to reproduce

Go to the 'Admin panel'
Click on the 'edit/create product' option inside the Catalog/products
Click on the customer group create option
See the error

Expected Result
Should be shown an option to create a customer group for all types of groups.
Actual Result
The option is missing. Here
Screenshots
No response
Additional context
No response

Now, this issue is fixed. so I am closing this issue.
And I have attached a screenshot Here, for future reference.

repo: bagisto/bagisto


text:
There is a UI issue with the Invoice Datagrid in the Admin-end customer section. The alignment is not appropriate.

Describe the bug
When checking the Admin End Customer Section, upon editing and reviewing the Invoice section, it's apparent that the UI of the invoice Datagrid is not appropriate.
To Reproduce
Steps to reproduce the behavior:

Go to Admin -> Customers -> Edit any customer with order's done and Invoice generated.
Scroll Down and check the Invoice Datagrid.
See the UI issue in the Datagrid.

Expected behavior
The UI must be proper and also the allignment should be in proper position.
Screenshots
Please find the screenshot here

Checked and found that the issue has been fixed and working fine now. Now the Invoice Datagrid UI has been fixed. Please check the screenshot for reference.
Screenshot
https://prnt.sc/Xjz7MxkZ2kyn
https://prnt.sc/5SjXJtxsLfe6

Checked and Found that the issue has been fixed and PR has been merged and hence I am closing the issue. Please check the screenshot below.
LTR -> https://prnt.sc/l2fGtPBzfOfl
RTL  -> https://prnt.sc/YbXXEhuulEc9


text:
Handle "github:name" version numbers

Found a use case when the following package was used
"bars": "github:jstrace/bars"

should ignore these dependencies

seems this is coming from deps-ok
at Object.checkNpmDependency (/deps-ok/src/utils.js:83)
test project https://github.com/bahmutov/left-behind

Made separate issue https://github.com/bahmutov/deps-ok/issues/22

repo: bahmutov/next-update


text:
fields with custom classNames improvement

Is your feature request related to a problem? Please describe.
I've use some third-party custom component (like element UI) for my work.
and auto class is not fully-compatible for components with wrapper.
And i noticed #1189 has a nice workaround for these feature,
but fields' property still has same name.
Describe the solution you'd like
I have 2 ideas.

set classNames will effect fields, but it will harm codes a lot.
performs new injected property such as vClass, it will configable,
and use like<el-input name="myProp" v-validate="'required'" :class="vClass.name" />

I prefers option 2
Describe alternatives you've considered
I write scss with mixins for compatible with bootstrap classes.

This was addressed by ValidationComponents added in 2.1, the slot-scope of the ValidationProvider exposes a classes field which is an object that can be bound to the class attribute.
https://baianat.github.io/vee-validate/guide/components/validation-provider.html#scoped-slot-data
The directive API probably won't be implementing this.

authors: logaretm yoyoys

repo: baianat/vee-validate


text:
Simulation mode documentation.

I added some documentation as it was not obvious that the Rust SGX SDK could be run in simulation mode (and on my Macbook Pro).
Finally I could see that there was a setting

Thanks for your contribution @davidp94 !

repo: baidu/rust-sgx-sdk


text:
merge的一个问题（多个tablet进行merge可能导致时序混乱）

a和b进行merge，先分别执行unload
b的unload失败进而重新被load起来成为ready状态，
接着b和c被选中进行merge，分别进行unload，
之后a的unload成功，发现b仍处于unload状态（此时b的unload是为c而准备的，但a不知道，以为b还在为a而unload，因而将自己转为OnMerge状态，苦苦等待着b的unload结束开始merge） :cry:
b和c的unload成功后merge出d来，b就从tablet界消失了，a却一直处于OnMerge状态。。。好可怜的a啊，求程序猿解救！

另外，ts的restart事件是不是也要注意下？

我来跟进

unload失败的原因都是超时，这个时候是不会发生一个失败的unload比一个成功的unload更早返回， 不会存在这个问题

当时看日志确认看到了这个问题啊。。。现场没了，原因可能分析有问题，但确认存在这么个现象。

“确认存在这么个现象”
为什么还close啊？

其实我是为了测试如果存在一个owner没有subscribe的issue，当它被关闭时owner是否会收到通知。

回忆一下当时时什么场景。。是不是在并发控制的情况下才出这个问题的

可能是当时我搞出来的bug。。。已经没有现场了，如果复现再开这个issue吧。

authors: bluebore taocp xupeilin

repo: baidu/tera


text:
关于dmkit返回值

result.actions_lists[0].custom_reply.event_name如果是‘DM_RESULT’则为dmkit返回，这个位置还能有其他返回值吗？如果有的话是做什么的呢，有没有文档

还有个问题
service_http_post的post参数怎么传，文档上写的第三个参数为传递的数据，可是怎么写呢？？没有一个示例么，比如我想发一个id=200，name='abc'，该怎么写呢？

repo: baidu/unit-dmkit


text:
a doubt about "Line 97: priors = priors[pos_idx].view(-1, 4)" in multibox_loss.py

hi, i have a doubt, such as the title. the shape of priors is (num_priors, 4), and pos_idx is (num, num_priors, 4). this code how to run success? thank u!

@AllenMao I am confused with that problem,too. It works well because both of 'priors' and 'pos_idx' are Variable. If they are tensor, it will fail to run success.
Besides, I also find that this code only works under <0.4.0 version. I tried with pytorch-0.3.1, it works well.
@bailvwangzi what's your opinion?

@huangzsdy: I got the same problem not because they are tensor. It is because of dimensions are not appropriate. For example when running on PASCAL
priors: [8732, 4]
pos_idx: [32, 8732, 4]
How can priors[pos_idx] work no matter Variable or Tensor?
If you said it works with Variable, can I like this?
va_pos_idx = Variable(pos_idx)
va_priors = Variable(priors)
va_priors = va_priors[va_pos_idx].view(-1, 4)
priors = torch.Tensor(va_priors)
Thank you

@huangzsdy i used pytoch-0.3.1.

@lthngan My environment is Python 3.6.2 :: Anaconda custom (64-bit), pytorch0.3.1.post2. It works well.
When the program run to this code, the type of priors or pos_idx is Variable, not tensor.
If you repeat these operations in python shell, it will be clear.
like this:
#>>> torch.version
'0.3.1.post2'
#>>> priors = torch.randn(8732, 4)
#>>> pos_idx = priors > 0
#>>> pos_idx = pos_idx.unsqueeze(0).expand_as(torch.randn(32,8732,4))
#>>> priors[pos_idx]
Traceback (most recent call last):
File "", line 1, in
priors[pos_idx]
RuntimeError: inconsistent tensor size, expected src [8732 x 4] and mask [32 x 8732 x 4] to have the same number of elements, but go
t 34928 and 1117696 elements respectively at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/TH/generic/THTensorMath.c:197
but if they are Variable, it works well
#>>> from torch.autograd import Variable
#>>> priors = Variable(priors)
#>>> pos_idx = priors > 0
#>>> pos_idx = pos_idx.unsqueeze(0).expand_as(torch.randn(32,8732,4))
#>>> priors[pos_idx]
Variable containing:
8.5339e-01
5.7070e-01
4.0415e-01
⋮
8.4663e-01
2.7078e-01
2.4516e+00
[torch.FloatTensor of size 561664]
#>>> type(priors)
<class 'torch.autograd.variable.Variable'>
#>>> type(pos_idx)
<class 'torch.autograd.variable.Variable'>
However, It failed when i used pytorch0.4.0(python3.6.2). It came out this error,'IndexError: too many indices for tensor of dimension 2'.
If you want to run through this code with pytorch0.4.0 , you can add 'priors = priors.expand_as(pos_idx)' before this code.
The above is just what I found through the experiment. I am confused with this problem,too. If you figure it out, pls let me know...thx.

@AllenMao What's your python version ? What 's your error message ?

@huangzsdy the same as you.
priors = priors.expand_as(pos_idx) print(priors[pos_idx].view(-1, 4))
and
priors = Variable(priors) print(priors[pos_idx].view(-1, 4)), their results is same on pytoch-0.3.1.
if use 'priors = priors.expand_as(pos_idx)', it came out error in later. so i use pytoch-0.3.1 and run well.

I think I've found a solution, and it works.
My python version is 3.6.9 and pytorch version is 1.1.0.
Firstly, as @huangzsdy said, add priors = priors.expand_as(pos_idx) before priors = priors[pos_idx].view(-1, 4) in multibox_loss.py
Secondly, change loss_c[pos] = 0 to loss_c[pos.view(-1, 1)] = 0.
Thirdly, change loss_l /= N loss_l_repul /= N loss_c /= N to loss_l /= N.float() loss_l_repul /= N.float() loss_c /= N.float().
And finally, change all loss.data[0] or loss_*.data[0] to loss.data or loss_*.data in train.py. Besides, if your loss value is Nan, you should reduce the value of the learning-rate.

authors: AllenMao geyuxi huangzsdy lthngan


text:
Update dependencies, drop support for python <3.8

What?
Bumping dependencies versions to their latest versions:

numpy from 1.19.5 to 1.22.3
pandas from 1.1.5 to 1.4.2
matplotlib from 3.3.4 to 3.5.2

Incidentally drops support for python 3.6 & 3.7
Why?
The current version of the dependencies are from over ~1.5 years ago (around Jan 2021)
Working with this module in combination with other up-to-date modules is getting increasingly difficult as the last year has seen some significant changes in the Python ecosystem.
As far as python versions are concerned, numpy has dropped support for:

Python 3.6 in 1.20.0 (~ Jan. 2021)
Python 3.7 in 1.22.0 (~ Dec. 2021)

Python 3.8 itself was released in Oct. 2019 and the current industry standard seems to be 3.9 (eg: Debian stable currently ships with 3.9). With 3.10 around the corner, it seems reasonable to drop support for 3.6, and 3.7 as numpy did.
WDYT?

On a side note, great module 💯 .
Coming from a home-brewed pymc3 implementation, this has been a pleasure to work with.

☝
The latest seaborn version is 0.11.2 (this module uses 0.10.1),  however a breaking change introduced in 0.11.0 breaks the colouring in _plot_positive_lift() in a way that doesn't seem to be easily fixable.

@godric Hey, so sorry I missed this for so long. I need to update my notifications or something. Looks good to me!

@bakermoran, no worries happy to hear from you
So since my original PR, numpy (in 1.24) dropped np.float which seaborn uses.
I've fixed my branch (specified a working range of version) so it would pass the tests.
It should be ready to merge now.
I was wondering if you would be willing to let go of that red shading?
If so I would be happy to create a new PR which would update all dependencies to their latest versions (and add more recent python3 version to the test matrix)

I was wondering if you would be willing to let go of that red shading?
If so I would be happy to create a new PR which would update all dependencies to their latest versions, and add more recent python3 version to the test matrix.

Actually I did just that in
https://github.com/godric/BayesABTest/tree/reupdated-deps

@godric Sorry I've turned on email notifications for this now to make sure I don't miss anything.
I've merged this PR as is. Open to turning that fork into a PR here. Can you help me understand the problem a bit? Is the goal just to upgrade seaborn, which removed functionality we use here? Or is there a breaking change in another package?

@godric This should be available in v1.0.9 now. Thanks for the help here!

authors: bakermoran godric

repo: bakermoran/BayesABTest


text:
Add pool to allowlist

Adding this pool to the allowlist: https://app.balancer.fi/#/polygon/pool/0xbc0144caf275099e5812b42724d4f432771efd3c000200000000000000000e92
Description
Kindly review this PR. Thanks for your work!
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Dependency changes
[ ] Code refactor / cleanup
[ ] Documentation or wording changes
[ ] Other

How should this be tested?
Please provide instructions so we can test. Please also list any relevant details for your test configuration.

[ ] Test A
[ ] Test B

Visual context
Please provide any relevant visual context for UI changes or additions. This could be static screenshots or a loom screencast.
Checklist:

[ ] I have performed a self-review of my own code
[ ] I have requested at least 1 review (If the PR is significant enough, use best judgement here)
[ ] I have commented my code where relevant, particularly in hard-to-understand areas
[ ] If package-lock.json has changes, it was intentional.
[ ] The base of this PR is master if hotfix, develop if not

@Barukimang  This pool was already allowlisted by this automatic PR.

authors: Barukimang agualis


text:
ExpressJS prior to Sails 0.12-rc* Candidates has Static Asset Problems

We recently upgraded SailsJS from 0.11 to 0.12-rc3 because the ExpressJS version paired with 0.11 (Express 3.16.0) had a pretty serious bug with sending static files.  We would get "pending" in the browser on a regular basis which would hang rendering in development mode.
We researched the issue and were happy to find that SailsJS bumped ExpressJS to 3.20.2 which resolves the issue. To replicate, simply have about 10-15 static files (images, JS, CSS - whatever) render upon page load with Express 3.16.0.

I can confirm I was having the same issue in my Sails 0.11 app. Only noticed once I had enough third party Bower libraries in the pipeline to go above that threshold. Every few hard refreshes would result in a page stuck half-loaded. I only got around this by forcing the development environment to concat all JS and CSS assets.

Ahhhhhhhhhh
@crh3675 would you be my hero and post something in about your findings in: https://github.com/balderdashy/sails/issues/2616

Simply put, it is ExpressJS.  I didn't dig too deep into it but 3.16.0 was the issue.

Thanks again for the post @crh3675.  This is fixed in the master branch (Express locked to 3.20.2).

Glad to contribute!

Can this be pushed into an .11.x release. This is a major bug for production environments in the .11.x version.

repo: balderdashy/sails


text:
opt.expires.toUTCString is not a function

Sails version: 0.12.3
Node version: 4.3.1
NPM version: 2.14.12
Operating system: osx 10.10.5

Just had my sails app crash with the following error, a restart did fix it but I'd like to understand what would cause it as never seen it before
2016-05-20T12:23:39.537233Z app: Sending 500 ("Server Error") response:
2016-05-20T12:23:39.541862Z app:  TypeError: opt.expires.toUTCString is not a function
2016-05-20T12:23:39.542355Z app:     at Object.serialize (/app/node_modules/sails/node_modules/express-session/node_modules/cookie/index.js:136:56)
2016-05-20T12:23:39.542710Z app:     at setcookie (/app/node_modules/sails/node_modules/express-session/index.js:581:21)
2016-05-20T12:23:39.542988Z app:     at ServerResponse.<anonymous> (/app/node_modules/sails/node_modules/express-session/index.js:204:7)
2016-05-20T12:23:39.543339Z app:     at ServerResponse.writeHead (/app/node_modules/sails/node_modules/express-session/node_modules/on-headers/index.js:46:16)
2016-05-20T12:23:39.543676Z app:     at ServerResponse.writeHead (/app/node_modules/sails/node_modules/compression/node_modules/on-headers/index.js:55:19)
2016-05-20T12:23:39.544040Z app:     at ServerResponse._implicitHeader (_http_server.js:157:8)
2016-05-20T12:23:39.544566Z app:     at ServerResponse.res.end (/app/node_modules/sails/node_modules/compression/index.js:94:14)
2016-05-20T12:23:39.545147Z app:     at ServerResponse.res.send (/app/node_modules/sails/node_modules/express/lib/response.js:188:10)
2016-05-20T12:23:39.545448Z app:     at ServerResponse.res.json (/app/node_modules/sails/node_modules/express/lib/response.js:237:15)
2016-05-20T12:23:39.545829Z app:     at ServerResponse.res.send (/app/node_modules/sails/node_modules/express/lib/response.js:139:21)
2016-05-20T12:23:39.546206Z app:     at /app/api/controllers/SessionController.js:82:18
2016-05-20T12:23:39.546568Z app:     at wrapped (/app/node_modules/newrelic/lib/transaction/tracer/index.js:155:28)
2016-05-20T12:23:39.547068Z app:     at Promise.linkTransaction (/app/node_modules/newrelic/lib/instrumentation/promise.js:289:65)
2016-05-20T12:23:39.547483Z app:     at Promise.wrapped [as __NR_wrapper] (/app/node_modules/newrelic/lib/transaction/tracer/index.js:155:28)
2016-05-20T12:23:39.547919Z app:     at __NR_wrappedThenHandler (/app/node_modules/newrelic/lib/instrumentation/promise.js:461:26)
2016-05-20T12:23:39.548300Z app:     at tryCatcher (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/util.js:11:23)
2016-05-20T12:23:39.548661Z app:     at Promise._settlePromiseFromHandler (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/promise.js:491:31)
2016-05-20T12:23:39.549155Z app:     at Promise._settlePromise (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/promise.js:548:18)
2016-05-20T12:23:39.549524Z app:     at Promise._settlePromise0 (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/promise.js:593:10)
2016-05-20T12:23:39.549824Z app:     at Promise._settlePromises (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/promise.js:676:18)
2016-05-20T12:23:39.550094Z app:     at Async._drainQueue (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/async.js:125:16)
2016-05-20T12:23:39.550943Z app:     at Async._drainQueues (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/async.js:135:10)
2016-05-20T12:23:39.551228Z app:     at Immediate.Async.drainQueues (/app/node_modules/sails/node_modules/waterline/node_modules/bluebird/js/release/async.js:16:14)
2016-05-20T12:23:39.551528Z app:     at Immediate.wrapped [as _onImmediate] (/app/node_modules/newrelic/lib/transaction/tracer/index.js:155:28)
2016-05-20T12:23:39.551949Z app:     at processImmediate [as _immediateCallback] (timers.js:383:17) [TypeError: opt.expires.toUTCString is not a function]

Just had this happen to me too... any ideas?

The stack trace shows that the issue is a non-Date object being given for expires to the session middleware's cookie option.

Yeah, looks like I was setting expires incorrectly via Moment, just needed to do .toDate(). 👍

I believe in my case it was because I was setting a cookie with a number rather that date so I had
req.session.cookie.expires = 24 * 60 * 60 * 1000 * 30;

rather than
var days = 24 * 60 * 60 * 1000 * 30;
req.session.cookie.expires = new Date(Date.now() + days);

authors: Peter-Barrett dottodot dougwilson


text:
Find, Destroy or Update methods ignore undefined type attributes

Waterline version: 0.13.0-rc11
Node version: 4.5
NPM version: 2.15.9
Operating system: MacOs

Hi guys, I found an issue that I'm not sure if it's a bug or something that you have considered. If you do something like this:
User.update({id: undefined}, {name: 'joe'}).exec()

This is gonna update ALL the users in the database, because the mongo query is gonna be something like this {}. The same thing happens with find/destroy methods.
So imagine that in your code you don't realize data X value is undefined and you lauch a destroy against it, you'll end up dropping all the collection with no clue about what just happened.
Thanks for your time.

I found the same issue before. Good thing it was not in prod.

We just updated an entire database collection on accident IN prod due to this.

This is intentional, but it's not too late to change the behavior for v1, so let's definitely discuss.
tldr; I wrote a big long thing, so if you don't have time to read it all, please just skip down to "Questions" below  :)
Why it works this way now
The "newRecord" and "valuesToSet" side of things have always done this (stripped keys with undefined values).  I realize this isn't what we're talking about here-- just making sure that's clear from the get-go so we're on the same page (and for anyone stumbling on this issue down the road.)
For criteria (and specifically for the where clause), the reason to do it this way is for use cases where you want to apply an optional constraint.
That might be for convenience within your code-- e.g. letting us do this:
// Look up all facilities if the logged-in user is an admin,
// otherwise look up just the facilities she owns.
// (Also, if the name search or country filters were specified, apply them)
var facilities = await Facility.find({
  owner: loggedInUser.isAdmin ? undefined : this.req.session.userId,
  name: inputs.name ? { contains: inputs.name } : undefined,
  country: inputs.country ? { contains: inputs.country } : undefined,
  isSoftDeleted: false
});

Instead of this (having to build the where clause separately), which is considerably more verbose:
// Look up all facilities if the logged-in user is an admin,
// otherwise look up just the facilities she owns.
// (Also, if the name search or country filters were specified, apply them)
var constraintsByAttrName = {
  isSoftDeleted: false
};

if (!loggedInUser.isAdmin) {
  constraintsByAttrName.owner = loggedInUser.id;
}

if (inputs.name) {
  constraintsByAttrName.name = inputs.name;
}

if (inputs.country) {
  constraintsByAttrName.country = inputs.country;
}

var facilities = await Facility.find(constraintsByAttrName);

We could try refactoring it a bit, which makes the code more concise -- but doing so makes it considerably more opaque, since you still have to address the non-fungibility of undefined vs. omission:
// Look up all facilities if the logged-in user is an admin,
// otherwise look up just the facilities she owns.
// (Also, if the name search or country filters were specified, apply them)
var constraintsByAttrName = {
  owner: loggedInUser.isAdmin ? undefined : this.req.session.userId,
  name: inputs.name ? { contains: inputs.name } : undefined,
  country: inputs.country ? { contains: inputs.country } : undefined,
  isSoftDeleted: false
};

_.each(_.keys(constraintsByAttrName), (attrName)=>{
  if (constraintsByAttrName[attrName] === undefined) {
    delete constraintsByAttrName[attrName];
  }
});

var facilities = await Facility.find(whereClause);

You probably also noticed in the example above that part of the reason it got way worse was because our use case involved an optional "AND" filter as a parameter.  That's where this makes the most difference, it seems.  Here's another example of where relying on undefined-stripping saves code complexity / development time / frustration:
{
  description: 'Archive posts more than 45 days old (within the additional geographical boundaries, if specified.)',
  inputs: {
    minLat: { type: 'number' },
    maxLat: { type: 'number' },
    minLong: { type: 'number' },
    maxLong: { type: 'number' },
  },
  fn: async function(inputs, exits){
    await Post.update({
      createdAt: { '<': Date.now()-(1000*60*60*24*45) },
      minLat: inputs.minLat,
      maxLat: inputs.maxLat,
      minLong: inputs.minLong,
      maxLong: inputs.maxLong,
    }).set({ isArchived: true });

    return exits.success();
  }
}

As you can imagine, the above code would be a lot longer if we couldn't rely on the stripping-out of undefined keys in criteria.  (This is definitely less of a big deal now that we can take advantage of async/await -- but it's still a material advantage that helps keeps code more concise.)
Of course, we have to contrast that with the downsides.
Thoughts
All that said, I've had similar experiences to what @Josebaseba @Xandrak and @luislobo are describing, and I think some kind of configurable warning, or failing with an error (e.g. assuming this was disable-able using .meta()) would be worth exploring.  I'm just wary about doing any of that lightly, because the last thing we want is to have to change it back (it's a pretty insidious breaking change as is, and it'd be really frustrating to have it flip-flop more than once).
We'd also need to think through how any change here would affect recursive parsing-- or possibly just try adding the warning and then try a ton of different things  (currently we're relying on this rhs-stripping logic in the generic where clause parse logic, which takes care of a few other edge cases involving "fracturing" the where clause, if memory serves.)
One last thing on that:  There are a few other cases where the where clause parsing in FS2Q does special stuff:  nin/in/and/or arrays are automatically cleansed of undefined values, and are automatically collapsed.  For the recursion to work as implemented, it currently relies on undefined constraints to be understood as no-ops, which it then puzzles over when cleansing the array for the parent predicate term (or/and).  So any change here would need to be carefully examined to make sure that all still makes sense.

See https://github.com/balderdashy/waterline/blob/5a74dfb8cb8b3926aef1966db89e72c82a7d8779/lib/waterline/utils/query/private/normalize-where-clause.js#L177-L229 for more on that.

Here's a more concrete example:
// If labels array turns out to be empty, it's treated as a "void" term,
// and thus (since it's at the top level, i.e. an "and") no issues are matched.
await Issue.find({
  primaryLabel: { in: _.without(inputs.labels, 'bug') }
});

// Similar to above, but in this case, it's more clear how its treatment depends on the context:
//
// In this case, if the array passed to `in` turns up empty, it's still understood as void,
// which then collapses that disjunct in the "or" predicate into `false`.
// Since `false` is the same thing as the void term, it matches _nothing_.
//
// But this time, since we're in an "or" context instead an "and", we don't make the
// entire `where` clause void.  Instead, this acts like `||false`, which means it's effectively
// disregarded and pruned out of the "or" predicate, leaving only the second disjunct.
//
// So at the end of the day, if the array provided for `in` turns up empty, we end up
// fetching _all_ issues that were created by user #13.
await Issue.find({
  or: [
    { primaryLabel: { in: _.without(inputs.labels, 'bug') } },
    { creator: 13 }
  ]
});

// Here's another one.  In this case, you only fetch issues that WEREN'T
// created by members of the specified organization.  But if the only users
// that exist are members of the specified organization (or perhaps if there're
// no users at all), then this ends up as an empty array.  Since that's inside of
// a "nin", rather than an "in", it is understood as "universal" rather than "void".
//
// In other words, since there are no users in the specified org, excluding those
// users is basically the same as matching any user-- any "creator".
// Furthermore, in this case, since we're effectively inside an "and"
// (b/c we're at the top level of the "where" clause), the universal term is
// understood like `&&true` -- and effectively ignored.
//
// So at the end of the day, all issues that have `isDeleted: false` are matched
// and returned.
var orgMembers = await User.find({ memberOf: inputs.organizationId });
await Issue.find({
  isDeleted: false,
  creator: { nin: _.pluck(orgMembers, 'id') }
});

// Finally, this example demonstrates how all the pieces fit together,
// as well as how the stripping of `undefined` values works within
// "and"/"or" predicates, and within "in"/"nin" constraints.
//
// This example matches all issues whose primary label is one of the specified
// labels (except "bug"), and also matches any other issues which were
// created by the logged-in user.  If there is no logged-in user, then the `undefined`
// is ignored.  (In an "or" predicate, an `undefined` disjunct operates like `||false`.
// In an "and" predicate, an `undefined` conjunct operates like `&&true`.  i.e. in
// either case, it is ignored.)
await Issue.find({
  or: [
    { primaryLabel: { in: _.without(inputs.labels, 'bug') } },
    this.req.session.userId ? { creator: this.req.session.userId } : undefined
  ]
});

Questions
@luislobo @Xandrak @Josebaseba @sgress454:

Did the explanation above make sense, or did I go too far into the weeds?
Has this ever been a problem for you when updating/destroying by something other than primary key value?  Or has the behavior only felt unexpected when attempting to update/destroy a particular record by id?
Has this behavior ever felt unexpected when doing a .find(), .findOne(), .count(), .avg(), or .sum()?  Or is it just with update and destroy?

Hi @mikermcneil, thanks for the answer it's good to know that this is an intentional feature and not a bug.
First answering your question:

It makes sense
For us it's a problem (updating/deleting), we're afraid of upgrading to sails1 and finding that we've updated all the users' table because of a {email: undefined} for example
That's a good question, I think that I'll go with the same behavior in all of them.

So, I'll add an option in the .meta()some thing like: removeUndefined: true (default false). Why? Because you're right deleting the undefined values can make the code easier to read/understand but facing an accidental drop of an entire table is very dangerous.

I like @Josebaseba suggestion, but, in addition to that, I would add an exception whenever a "value" in a where clause is undefined, in the case removeUndefined: false. If you are "wanting" undefined NOT to be removed, you are worried about getting undefined values from your inputs, so you should be warned/stopped in those cases.

@mikermcneil The explanation makes sense to me, but I don't count :P.
As far as impact, I've only experienced the problem when running an .update() on something other than a primary key.  When it really bites you is when you do something like .update({ name: someObj.name }) while assuming that someObj.name will always be defined.  It's usually due to a logical bug in the app, but it's hard to figure that out and in the meantime, you might have blown your database.
After mulling this over for quite some time, I don't think that the answer is to strip out undefined values or to throw warnings / errors.  I think what we should do is transform undefined values into the base type for that attribute (or null if allowNull is set for the attribute).  That is, after all, what being "undefined" means in Waterline 0.13.x.
When I see this example:
var facilities = await Facility.find({
  owner: loggedInUser.isAdmin ? undefined : this.req.session.userId,
});

I don't intuitively read it as, "If the logged in user is an admin, then find all facilities".  I read it as, "If the logged in user is an admin, then find all facilities that don't have an owner".  Casting undefined values to the proper base value would support that.  It would mean an extra step if you did intend to find all facilities when the user is an admin, but I'd argue that it's worth it for the clarity provided in all cases.
I took some time to look into how various other software handles this case, and it's all over the map.  The Mongo client app returns an error if you run a query with an undefined value, but the Node driver seems to translate it into an {$exists: false} clause, so that only records that don't have that field are returned (you could think of undefined as Mongo's "base type" for every field).  The Node Postgres driver allows you to bind undefined to a parameterized query, e.g.
pool.query('select * from test where age = $1', [undefined], console.log)

and you'll just always get zero results (even if there are records where age is null).  Moving on to the ORM world, if you try the Postgres example with Knex, it throws an error about "Undefined binding(s) detected when compiling SELECT query".
So, good arguments all around.  Casting to the base type is the most intuitive for me and seems like the most consistent thing with our scheme in general, and has the benefit of not throwing errors which might not be seen until production.  And in debugging, in my recent experience the question of "why didn't any rows update?" is a lot easier to figure out then, "why did ALL of my rows update"?

@sgress454 That is valid, unless, you do have values that are NULL, and are affected by your update, but you didn't intend to update those. In the case of an update or delete, I still prefer to fail because the only way to recover from that is a database restore, if you have such thing ;).
I don't think that doing any smart thing about an undefined is desirable in our case.
Although we do Pull Requests review and even sometimes Pair review, not all developers are senior or "awake" 100% of the time... Errors happen, validations are skipped, data is deleted. Massive Devops suicide happen.

Also, databases might even handle NULL differently, depending on it's configuration (ansinull, in some of them) so, it's not that easy neither to assume something like that.
Some references
https://stackoverflow.com/a/1843460/358634
http://sqlanywhere-forum.sap.com/questions/9285/is-null-vs-isnull-vs-null-correct-usage
http://dcx.sybase.com/index.html#1201/en/dbreference/nulls.html
https://www.postgresql.org/docs/8.3/static/functions-comparison.html

@luislobo Throwing errors when undefined values are detected is certainly the safest way to go.  If we were out of beta I'd be more hesitant since it'd be a significant breaking change, but in this case, I'm on board.
So it sounds like the current proposal is:

The addition of a meta option removeUndefined, which if true strips undefined values from criteria objects (as in the current behavior).  Defaults to false.
If removeUndefined is false and an undefined value is detected in a criteria object, an Error is thrown.

Examples:
// This would throw an error.
User.find({ name: undefined }).exec(...)

// This would retrieve all User records.
User.find({ name: undefined }).meta({ removeUndefined: true }).exec(...)

@mikermcneil @sgress454  Thanks for taking the time to address this! Let me clarify that it was actually my code error that brought this to light--unfortunately for us, it was in production. So I am not blaming Sails or anything. This could have been avoided with a subtle switch from using a .then() to a .each() after a find(). I should have been expecting the possibility of multiple values to be returned, but for whatever reason it slipped my mind. Please excuse the following, hastily written example 😉 :
...
// code before fix
model.find(findThis)
.then((foundModel) => {
  model.update({ id: foundModel.id }, updateModelValues)
})
...
// code after fix
model.find(findThis)
.each((foundModel) => {
  model.update({ id: foundModel.id }, updateModelValues)
})
...

We are now wrapping every update/destroy in our API within a check to make sure the criteria passed in is never undefined.
At any rate, we did not, however, expect the default behavior for an undefined criteria value to update the entire collection. We did expect an error to be thrown due to thinking that we do not ever send in undefined values on purpose. This may specific only for our team. Personally, I would still be wary of such methods that affect entire collections at once. I realize this is not the case for everyone--just my two cents.
@mikermcneil To answer your previous questions:

Yes, it makes sense.
In our case it only happened by id, but we have yet to test anything else.
Currently, just on .update() 😄 , but I think I would echo @Josebaseba's #3 answer as well.

@Josebaseba @luislobo Thanks for your input and bringing this issue to light.

@sgress454 I agree with your last comment.

@mikermcneil @sgress454 guys, this is a big deal, right now my production DB got all the collection updated at the same time, even if I went query by query making sure that everything was under control and right now I'm screwed... Is there anything that I can do to turn off this "feature"?

@josebaseba sorry to hear the upgrade is causing you trouble.  Thanks for following up.
We're relying on undefined-stripping in "where" clauses a lot, and in new code, it definitely saves time without introducing problems (it reduces the complexity of writing code to do optional filters quite a lot).  Thus I'm still not convinced that we should change this default behavior.
That said, I understand you might need an immediate change to help through the upgrade. I like @sgress454's proposal, with three tweaks:
• the flag needs to be an option that you enable to change the default behavior. The default behavior will continue to be what it is today: that undefined conjuncts are stripped from the 'where' clause in any kind of Waterline query.
• the option should be explicit in its name to clarify that it only applies to the 'where' clause:failOnUndefinedConjunctsInWhereClause, which defaults to false. Setting it to true causes an error instead of the default (current) behavior
• instead of a meta key (no one would ever want to type that all the time anyway), we should do this only as a model setting (no way to override on a query-by-query basis). That way, we minimize complexity while making it useful as a one line of code paste-in (in your config/models.js file)
@josebaseba if you've got a more immediate-term need and can't wait for the new model setting (it'll need to be added, tested, and documented), I think the best bet is to modify the code in normalize-where-clause I highlighted above and use that.

Note about implementation: Since this isn't a meta key, implementing it is easier. We already have access to the orm in that file linked previously, so we need only to look up the WLModel, check if it has the model setting as true, and if so, skip the undefined stripping. (No need to pass in more arguments or anything)

(BTW since that's such a mouthful, let's also, in parallel, down the road, see about coming up with a definition of what we want "strict mode" to look like in the future. It might make sense to lump a few similar features with the same motivations into that bucket)

@mikermcneil I like it making it a model setting, and defaulting to NOT allow the stripping of undefined conjuncts.

Thanks for the quick answer Mike, I like the idea of making it a model setting too. I'm gonna try to modify the code with a fork now.
Luckily I found the buggy entry point, now I have to figure out how to modify all the documents back to their previous values changing the DB as little as possible.
Another case that as a dev we must be careful with this feature:
// req.param('id') === 10
if (something) { res.ok(); } // we miss here a return
// req.param('id') -> undefined
User.update({id: req.param('id')}, {someData})

This wasn't my case but I found that issue while I was developing, it can be tricky. Our issue was related with the fact that the findOne doesn't work anymore if it finds more than one value so after that error, a bad update and... 💥 💥
Anyway, thanks guys!

@mikermcneil bump

ping ping ping

@mikermcneil  @sgress454 I didn't follow all the latest commits in Waterline (and friends), but may be you already handled what we have been discussing here?

bump! @mikermcneil Is there anything I can do to help with this one? I don't know its current status, thoguh

SailsBot, don't close it yet :)  #1545 seems to fix it!

Thanks @luislobo I'm hoping against hope for it to be accepted 😆

cough

+1
I think this is really important, any developer code accidentally passing undefined could lead to great frustration. I'm not sure what the solution is but I don't think this should be the default behavior.

I think right now, as this behavior is already in the wild, changing it's current way of working back is a breaking change.
My vote is to implement what @mikermcneil proposed in https://github.com/balderdashy/waterline/issues/1509#issuecomment-326016388 with the following changes/additions:
Hipothesis:

There is a configuration option that, when set, disables the current behavior of cleaning up undefined values from where conditions in find/updates/delete etc...

Where can it be configured:

The configuration can be set globally in /config/models.js
The configuration can be set in a model by model basis
It can be changed in a case by case scenario using a .meta() call.

Why the total opposition to this being a breaking change? I'm still not understanding the design decision and see this as undesired behavior. The reasoning given for maintaining the existing behavior seems to involve preventing developer frustration and keeping concise. I would think that there would be much more developer frustration when someone accidentally has their application do this:
await User.update({ emailAddress: undefined }, { rank: SUPER_ADMIN });
Maybe someone else has an example of another ORM and how they handle this behavior?
--
If there was to be a breaking change:
The proposed configuration change given in the PR could be added in the next version.
A warning could be created whenever find / update / delete etc. finds an undefined value, stating that this behavior will act differently in a future release, with a link to some article explaining the current behavior and the future behavior.
After the next major version, the undefined value in an find / update / delete etc. is not stripped out, and a warning is issued to the console explaining the behavior.
I wonder if any kind of upgrade script would be able to search the files for the breaking change in order to warn the developer when they upgrade to a new version.

@luislobo @AdJesumPerMariam Thanks for the feedback guys.
I think I was wrong above-- we shouldn't complicate things further with a new setting here.  The maintenance burden and cognitive load for users isn't worth it, and this is an area where we can afford to be opinionated-- that said, I agree that the current implementation is a bit "loose" (I accidentally deleted hundreds of user records myself a couple of weeks ago).
So, as far as I can tell, the most common situation where this is actually an issue is when attempting to update or destroy one thing (but instead, you end up accidentally updating/destroying multiple things at once.)  I think the best solution for this is to go ahead and implement updateOne() and destroyOne() and leave the current behavior of .update() and .destroy() as-is.
That way, since updateOne/destroyOne do a .count() first, then throw if ≥2 records are matched
That won't solve everything, but it'll cover the majority of issues.

It'll still be possible to make mistakes like:
await User.update({
  accountType: inputs.accountType,//« what if it's `undefined`
  createdAt: { '<': Date.now()-(1000*60*60*24*90) }
})
.set({
  accountStatus: 'inactive'
});

I'm open to discussing what it would look like to have the example above throw in future versions of Waterline (only for update and destroy), but it's a breaking change that we can't make until Sails v2.  (I can say for certain that making this change now without a major version bump would break at least 2 projects I've worked on personally)
But we can totally add updateOne() and destroyOne() now without breaking anybody's apps

Thanks for looking into this @mikermcneil.
Maybe a clear warning in the documentation for update, destroy, etc. about what will happen if a value is undefined, then at least for v1 people are warned to implement protections for that. There could also be a link to updateOne and destroyOne when those are implemented as "safer" alternatives.

I like having updateOne and deleteOne methods.
But I think the problem lies with acting on documents/records that don't match with your "expected" criteria.
The same issues causing a condition to match a different set of documents/records could happen to one single document/record
await Person.update({
  SSN: personSSN
})
set({
   someField: 'value'
});

Imagine having one record. SSN: '123456', and the calling service/application tries to update SSN:'847327'. For some bug/issue, the API receives personSSN == undefined, it should not update the SSN correspoding to 123456, because it doesn't match, but in this particular case it will.
I know it's an edge case, but that can happen, right?

@luislobo good point and that makes sense-- updateOne and destroyOne should not automatically trim conjuncts (instead they should throw).  Since they're new methods, that'll be safe to do without it being a breaking change

In my opinion, right now the best solution without breaking anything till the v2 would be adding a global flag in the models.js that throws an error when an undefined value is found in any query.
migrate: 'safe',
schema: true,
removeUndefined: false // default true

There's no urgency adding the flag in the .meta() method.
In my case I need to be sure that this "feature" is not executed again so I'm using my own waterline fork that has this behaviour. I've a team of tech people working with sails and I can't be sure 100% that the undefinegate won't come back to my database again... Even if I warned them tons of times :)
I'm looking forward to kill my fork and move on as soon as posible.

I agree with @Josebaseba, it would be great to be able to opt out of the "auto Remove Undefined From Where Clauses" feature.
Although I see it's convenience, it's consequences can be, as demonstrated, complicated.
Writing a little bit more code to handle specific criteria cases is not that bad for us. For example, we do have a "if this kind of user use this criteria and if not, use this other", and we are fine with that.

I was curious to see what the best practices around this issue are in other Node ORM's. I made a simple test of 6 ORM packages, including Waterline, and included the results below.
Waterline:
var users = await User.update({ 'id': undefined }, { lastName: 'Spaghetti' }).fetch();

update `user` set `lastName` = 'Spaghetti' limit 9007199254740991

Loopback:
User.updateAll({id: undefined}, {name: 'Fred'}, function(err, info) {});

UPDATE `User` SET `name`='Fred'

Same behavior as Waterline, intentionally changes {id: undefined} to {}.
Configuration option "normalizeUndefinedInQuery" available to modify this behavior.
https://github.com/strongloop/loopback/issues/637#issuecomment-146574151
Sequelize:
await User.update({ favorite_color: 'red' }, { where: { username: undefined } });

UPDATE `users` SET `favorite_color`='red',`updatedAt`='2018-06-02 13:22:51' WHERE `id` = NULL

Bookshelf (Knex Query Builder)
await User.where({ 'id': undefined }).save(
        { name: 'Fred' },
        { method: 'update', patch: true }
);

    Error: Undefined binding(s) detected when compiling UPDATE query: update `bookshelf_users` set `name` = ? where `id` = ?
            at QueryCompiler_MySQL.toSQL (/home/src/orm_test/books/node_modules/knex/lib/query/compiler.js:131:13)

TypeORM:
await getConnection()
        .createQueryBuilder()
        .update(User)
        .set({ firstName: "Mickie", lastName: "Mouse" })
        .where("id = :id", { id: undefined })
        .execute();

UPDATE `user` SET `firstName` = 'Mickie', `lastName` = 'Mouse' WHERE `id` = NULL

Node-ORM2 (No longer actively maintained)
Person.find({ id: undefined }, function (err, people) {});

SELECT `name`, `surname`, `age`, `male`, `id` FROM `person` WHERE `id` IS NULL

authors: AdJesumPerMariam Josebaseba Xandrak davidreghay luislobo mikermcneil sgress454

repo: balderdashy/waterline


text:
Dockerfile: Switch to using base images by arch

Private DTs do not have base images, so switching
to using images by arch simplifies automated
testing for them.
We also remove UDEV since it's not needed
for this particular application and throws
the following warning in the application logs:
balena-hello-world  Unable to start udev,
container must be run in privileged mode to start udev!
Change-type: patch
Signed-off-by: Alexandru Costache alexandru@balena.io

@balena-ci I self-certify!


text:
fin: toggle hub twice to make flashing consistant

We have had problems with the flashing of balena fins using the deviceInteractor. We have tried to isolate the problem to no effect.
When looking at the fin fullfillment rig code I saw this: https://github.com/balena-io-hardware/balena-fulfillment-rig/blob/master/core/app/utils/dut-power/index.js#L26
If it's true, and the usb hub driver doesn't always respect the uhubctl commands, doubling up the commands might help. This was the only difference I could find between the fin flashing procedure in the testbotsdk and the fin-fulfillment rig (software) - and the fulfillment rig never seems to have the problem. So hopefully this will help!
Change-type: patch
Signed-off-by: Ryan Cooke ryan@balena.io

@balena-ci rebase

@balena-ci I self-certify!

repo: balena-io-hardware/testbotsdk


text:
[WIP]: docs: Embedding masterclasses into documentation

This is a WIP to look at the feasibility of incorporating the Masterclasses into the main docs. The major advantage of this is they will be searchable.
I've pulled the masterclass README via fetch-external.sh, so it is pulling directly from the GitHub repos so they can continue to live in balena-projects.
There is some minor formatting that would need to be done. For example, the headings should be updated for search and navigation as there are multiple h1 tags.  Some text/links might need tweaking to be generic, as could be accessed in multiple places (I've not really checked this, so may not be an issue). Image paths will also need to be updated to be absolute rather than relative as currently.
I've located it in the navigation where it was at present though obviously, we can modify the navigation as necessary.
There's currently an issue with the external docs link for the cli pages as these also match the existing cli page.
I pushed to staging to give an idea (though as noted above there are some formatting issues)
https://www.balena-staging.io/docs/learn/more/masterclasses/overview/
If we think this idea is worth pursuing, I'll resolve the formatting issues above in the relevant repos.
Change-type: minor
Signed-off-by: Gareth Davies gareth@balena.io

As a better example, I updated a branch of the fleet management masterclass and it is now pulling from that so see this:
https://www.balena-staging.io/docs/learn/more/masterclasses/fleet-management-masterclass/

I like this a lot! I think this'll make discovery much easier, and it's always better to share docs directly rather than disparate GH projects.
One small concern I have is that aiui this'll only redeploy on docs merge, whereas the masterclasses may change faster than the docs (this is a general concern for fetched docs).
I also think as a step 2 it'd be worth looking into pulling these automatically using a repo tag or something, as hopefully we'll have these MCs proliferate :)

Thanks @xginn8, yeh all these external docs have these issues so we need to find a good way to deploy this - webhooks might be the simplest solution there and I guess I could bring this discussion to an arch call.
I thought you would mention repo tags/topics 😆 - I do agree completely but I did a bit of work on that and couldn't make it work (including your earlier comment here https://github.com/balena-io/docs/pull/1216#issuecomment-549616375). We now have topics for all masterclasses (I've asked for a couple to be removed that are not complete) but it seems the GitHub API for topics is lacking in that regard.
The response from https://api.github.com/search/topics?q=balena-masterclass is simply:
{
    "total_count": 1,
    "incomplete_results": false,
    "items": [
        {
            "name": "balena-masterclass",
            "display_name": null,
            "short_description": null,
            "description": null,
            "created_by": null,
            "released": null,
            "created_at": "2019-12-06T20:38:30Z",
            "updated_at": "2019-12-06T20:38:30Z",
            "featured": false,
            "curated": false,
            "score": 82.770805
        }
    ]
}

The only way I could get the repo information was via https://api.github.com/orgs/balena-io-projects/repos but there doesn't seem a way to filter this via topic (other than manually parsing each result).

Love the idea, and webhooks would be the best way to do this as you say. Watch for any repo with a -masterclass suffix to have a master merge, then pull it in automatically.

I've updated this PR to point to all the formatted branches of the PR so we can test it on staging see https://resin-docs-preview.herokuapp.com/docs/learn/more/masterclasses/overview/
I'll submit the PRs for each formatted branches in upstream repos then simply a case of updating to use those once merged. Formatting was mostly a case of fixing headers (which need to be like that for the docs) but I also tidied a few things up in line with Markdownlint such as named fenced code blocks - there are no notable content changes.
Given the frequency of updates of these masterclasses (i.e. low) and the frequency of updates to the docs (usually daily), I'm not sure the lack of integration to trigger a build is a showstopper for this right now.

@garethtdavies mind updating the skeleton as well? https://github.com/balena-io-projects/balena-masterclass-skeleton

Updated now to use master branches so this is ready to merge. Deployed to staging for one final test and read through https://resin-docs-preview.herokuapp.com/docs/learn/more/masterclasses/overview/
I've created an issue to track the triggering of a docs build from external docs here https://github.com/balena-io/docs/issues/1354, which I think we can implement in due course.

@balena-ci retest

@balena-ci retest

@hedss, not sure if you had any more feedback or wanted a final review of this before it is merged in?
Preview here: https://resin-docs-preview.herokuapp.com/docs/learn/more/masterclasses/overview/

@garethtdavies Have read through. This is great, thank you very much :)

authors: garethtdavies hedss xginn8

repo: balena-io/docs


text:
Consider allowing templating in environment variables in balena run command

Templating would allow exposing information normally accessible as docker inspect <container> to the container. Basic idea is to allow something like
balena run -e "MY_CONTAINER_ID={{.ContainerId}}" myimage

We might consider doing this via a dedicated argument (like -et) to not break existing -e behaviour,
Related to https://github.com/moby/moby/issues/8427
Also related to our recent use case when we needed supervisor app to know its own container ID: https://github.com/balena-os/meta-balena/pull/1590
cc @robertgzr @petrosagg @CameronDiver

One thing I just found when doing some initial exploration of how this could be done is that all things related to templates that are currently supported by balena-engine/moby are implemented on the CLI side.
Possible ways of moving forward with this would be duplicating some code into the engine repo or introduce a dep on the cli pkg/repo (but and I really don't want to do that)

@robertgzr do you also anticipate any difficulties obtaining container info (like container ID) before the moment when environment variables have to be set for a container process?

Current decision is to expose container ID in an environment variable using --cidenv command line option. See #174.

authors: robertgzr roman-mazur

repo: balena-os/balena-engine


text:
[LS Code Actions] "Document All" code action does not work for ballerina class method definitions

Description:
Please refer the session below and note that the [LS Code Actions] "Document All" code action does not add documentation for the ballerina class/object method definition.

Steps to reproduce:
Use the sample code below and invoke "Document All" code action on top of the partition method.
import ballerina/io;

# Description
class Partitioner {

    # Description
    #
    # + criterias - Parameter Description
    # + datasetFilePath - Parameter Description
    public function partition(string datasetFilePath, function... criterias) {
        stream<string[], io:Error>|error csvStream2 = io:fileReadCsvAsStream(datasetFilePath);
        // Loop through the stream and print the content.
        if (csvStream2 is error) {

        } else {
            error? e = csvStream2.forEach(function(string[] val) {
                                              io:println(val);
                                          });
        }
    }
}

# Description
# + return - Return Value Description
public function main() returns error? {

}

Affected Versions:
OS, DB, other environment details and versions:
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

Document all should only be suggested for top level constructs

authors: NipunaRanasinghe nadeeshaan

repo: ballerina-platform/ballerina-lang