text:
fix context undefined in useResponseCache ifDef enabled fn

🚨 IMPORTANT: Please do not create a Pull Request without creating an issue first.
Any change needs to be discussed before proceeding. Failure to do so may result in the rejection of
the pull request.
Description
Conntext not available when evaluating responseCache if function. Reason is that javascript function constructor does not inherit lexical scope. Instead we must pass the context as a named arg.
Fixes #6925
Type of change
Please delete options that are not relevant.

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as
expected)
[ ] This change requires a documentation update

Screenshots/Sandbox (if appropriate/relevant):
Adding links to sandbox or providing screenshots can help us understand more about this PR and take
action on it as appropriate
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can
reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Test Environment:

OS:
@graphql-mesh/...:
NodeJS:

Checklist:

[ ] I have followed the
CONTRIBUTING doc and the
style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works, and I have added a
changeset using yarn changeset that bumps the version
[ ] New and existing unit tests and linter rules pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

Further comments
If this is a relatively large or complex change, kick off the discussion by explaining why you chose
the solution you did and what alternatives you considered, etc...

cc @ardatan

Thanks for the PR!
Do you think we can add tests to prevent future regressions?

Thanks for the PR! Do you think we can add tests to prevent future regressions?

I'll take a crack at it, sure.

Thanks for the PR! Do you think we can add tests to prevent future regressions?

@ardatan The response-cache plugin does not have a tests folder or any tests defined currently. I attempted to copy the tests from the rate-limit plugin and adapt them to response cache but quickly encountered setup issues. When trying to pass the response-cache plugin to the envelop factory method, I got an error saying that the response-cache plugin must be used with graphql-yoga. So I'm not sure the best way to create a test harness. If you can help lay the groundwork here I'd be happy to contribute a unit test outlining the expectation for the request context being accessible in the ifdef setting.
I'm currently manually testing this with the help of the patch package lib

https://www.npmjs.com/package/patch-package

Here is the diff
diff --git a/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js b/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js
index 5c98d77..a4acb6a 100644
--- a/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js
+++ b/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js
@@ -20,7 +20,7 @@ function generateSessionIdFactory(sessionIdDef) {
 function generateEnabledFactory(ifDef) {
     return function enabled(context) {
         // eslint-disable-next-line no-new-func
-        return new Function(`return ${ifDef}`)();
+        return new Function("context", `return ${ifDef}`)(context);
     };
 }
 function getBuildResponseCacheKey(cacheKeyDef) {
diff --git a/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js b/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js
index 94f1947..3f0641f 100644
--- a/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js
+++ b/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js
@@ -18,7 +18,7 @@ function generateSessionIdFactory(sessionIdDef) {
 function generateEnabledFactory(ifDef) {
     return function enabled(context) {
         // eslint-disable-next-line no-new-func
-        return new Function(`return ${ifDef}`)();
+        return new Function("context", `return ${ifDef}`)(context);
     };
 }
 function getBuildResponseCacheKey(cacheKeyDef) {

with this diff in place and patch package installed, you can effectively modify the built library and observe the changed behavior. Before the patch my ifdef function fails when I reference context request headers and after the patch is applied -- no issues.

Thanks for the PR! Do you think we can add tests to prevent future regressions?

@ardatan The response-cache plugin does not have a tests folder or any tests defined currently. I attempted to copy the tests from the rate-limit plugin and adapt them to response cache but quickly encountered setup issues. When trying to pass the response-cache plugin to the envelop factory method, I got an error saying that the response-cache plugin must be used with graphql-yoga. So I'm not sure the best way to create a test harness. If you can help lay the groundwork here I'd be happy to contribute a unit test outlining the expectation for the request context being accessible in the ifdef setting.
I'm currently manually testing this with the help of the patch package lib

https://www.npmjs.com/package/patch-package

Here is the diff
diff --git a/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js b/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js
index 5c98d77..a4acb6a 100644
--- a/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js
+++ b/node_modules/@graphql-mesh/plugin-response-cache/cjs/index.js
@@ -20,7 +20,7 @@ function generateSessionIdFactory(sessionIdDef) {
 function generateEnabledFactory(ifDef) {
     return function enabled(context) {
         // eslint-disable-next-line no-new-func
-        return new Function(`return ${ifDef}`)();
+        return new Function("context", `return ${ifDef}`)(context);
     };
 }
 function getBuildResponseCacheKey(cacheKeyDef) {
diff --git a/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js b/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js
index 94f1947..3f0641f 100644
--- a/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js
+++ b/node_modules/@graphql-mesh/plugin-response-cache/esm/index.js
@@ -18,7 +18,7 @@ function generateSessionIdFactory(sessionIdDef) {
 function generateEnabledFactory(ifDef) {
     return function enabled(context) {
         // eslint-disable-next-line no-new-func
-        return new Function(`return ${ifDef}`)();
+        return new Function("context", `return ${ifDef}`)(context);
     };
 }
 function getBuildResponseCacheKey(cacheKeyDef) {

with this diff in place and patch package installed, you can effectively modify the built library and observe the changed behavior. Before the patch my ifdef function fails when I reference context request headers and after the patch is applied -- no issues.

ping @ardatan , thoughts on the above?

It seems integration tests are not failing which is ok for now. Let's merge this, then we can think of individual unit tests later on.
Thanks for the PR!

It seems integration tests are not failing which is ok for now. Let's merge this, then we can think of individual unit tests later on. Thanks for the PR! At least, could you create a changeset by using yarn changeset so we can bump the version and release?

changeset complete 👍

@ardatan apologies looks like I had some prettier formatting issues with my initial change. I've ran prettier and I think it should pass all your PR checks 🤞

@ardatan looks like load test failed on node 21, and the e2e tests timed out in node 20.  These don't appear to be related to my code changes. lmk if there is anything more you'd like me to do to facilitate here.

Hey @mmadson, I've fixed the flaky tests! Sorry about that. Can you please rebase on master?

authors: ardatan enisdenjo mmadson

repo: ardatan/graphql-mesh


text:
Fix licensing config/review screen on small displays

Add scrollbar to License configuring/reviewing screens. This fixes visibility of back/next buttons on mobile chrome (or very tiny android screens?)

Merging in order to get the deployed branch

source: gharchive/pull-request

authors: elliotsayes thiagocarvalhodev

repo: ardriveapp/ardrive-web

license_type: permissive

license_source: github-api


text:
Fixes #33

As described in the issue, this prevent strange behaviour

PIN_MISO was already set as input

authors: agdl landfillbaby

repo: arduino/arduino-examples


text:
Library release removal request

Library name
HCSR04 ultrasonic distance sensor
Version to remove
1.0.1

Hi @x0x0200. I am confirming your request.
Unlike submissions, this type of request is still handled manually. The Library Manager index maintainer occasionally comes through to clear all the outstanding requests. They will close this issue when it is done. You will get a notification from GitHub at that time.
After that, you can monitor the indexing of the library via its dedicated logs web page.
Information about that here:
https://github.com/arduino/library-registry/blob/main/FAQ.md#can-i-check-on-library-releases-being-added-to-library-manager

Done:
2024/02/29 14:53:37 Loaded 6704 libraries from DB
Removing HCSR04 ultrasonic distance sensor@1.0.1
Success!

source: gharchive/issue

repo: arduino/library-registry


text:
Update repositories.txt

Hello, I would like to add my library to Library Manager

Closing as duplicate of https://github.com/arduino/library-registry/pull/1803


text:
Update repositories.txt

I made improvements to the previous library

@ArduinoBot


text:
[Dev] Remove the in-repo typings for vscode and move them to @types

Heyo - saw your interview and decided I could be that first person to take a serious look around. Thought I'd do some cleanup of obvious things. Mainly moves type definitions which were inside the repo out into the @types/[module]. Will comment inline.

@orta really appreciate this. I'll just update the README.

authors: dzucconi orta

repo: aredotna/case

license: mit

license_source: bigquery


text:
Headers exchanges do not route messages based on headers

Headers exchanges as implemented currently ignore message headers, and instead route messages based on the :routing_key passed to the #publish method, just like Direct and Topic exchanges.
It also looks like the code that binds exchanges and queues together doesn't support headers exchanges either.
Sadly I don't have time to provide a fix myself at the moment, so a issue report will have to do for now.

I'm having the same problem

authors: fernandomcunha jimeh


text:
Auto commenting feature for new PR raised

This feature aims to address the problem of delayed and inconsistent communication following the raising of PRs. By automatically commenting on PRs as soon as they are raised, it ensures that contributors receive immediate feedback and acknowledgment for their efforts. This fosters a positive and supportive environment for collaboration, encouraging continued participation and engagement from contributors.
Could you please assign me this issue under GSSOC'24.

@arghadipmanna101 Introducing a groundbreaking feature crafted to revolutionize the way we communicate following the submission of PRs! No more waiting in frustration for acknowledgment. With this innovation, comments are automatically injected into PRs the moment they're created, ensuring contributors receive instant recognition for their dedication. Say hello to a dynamic, supportive collaboration environment that ignites passion and drives ongoing engagement from all contributors.
Would you be so kind as to assign me this captivating issue under GSSOC'24?

Please add the labels of gssoc and appropriate level as per gssoc guidelines on this issue.

authors: KomalSrivastava suhanipaliwal

license: MIT


text:
Ensure we test fully loaded sockets

cc: @stgeke -- running these tests currently.

Note to self: this change causes new timeouts on Travis for some reason... it really shouldn't affect that though, the number of cores from the test matrix doesn't enter into anything in the non verylong unit-tests AFAIK

repo: arghdos/spyJac


text:
[FEATURE] add __repr__ to argilla.settings._resource.SettingsProperties

Is your feature request related to a problem? Please describe.

These are used for error messaging but don't show the relevant information.
Describe the solution you'd like

I would like to see it as normal printed settings for the dataset.
Describe alternatives you've considered

NA.
Additional context

There might be other classes that need a __repr__.

Cool project!! I would like to work on this if it is okay. I am pretty inexperienced but this should be self-explanatory. I could check for other classes too but let me start with this. 🙂

Hi @phershbe, that would be very welcome. You can tak e a look here to see how you can contribute but feel free to ask for help when needed. https://docs.argilla.io/latest/community/contributor/

@phershbe are you still interested in tackling this?

@davidberenstein Sorry about this. Yeah I am, I spent quite a bit of time and went through the code base and checked out the __repr__ methods for different classes. I guess that in this case it is going to want to mention the properties but I didn't feel like I knew enough to ask a question and didn't want to submit a really sloppy pull request. I unassigned myself a few minutes ago, I will study whatever is in the pull request when it gets made. Sorry again about not letting you know before.

hi @davidberenstein1957, Request you to pls confirm if the below behavior is the expected behaviour for the below unit test.
def test_settings_repr(self):
        settings = rg.Settings(
            fields=[
                rg.TextField(name="text", title="text"),
                rg.ImageField(name="image", title="image"),
            ],
            metadata=[
                rg.FloatMetadataProperty("source"),
            ],
            questions=[
                rg.LabelQuestion(name="label", title="text", labels=["positive", "negative"]),
                #rg.MultiLabelQuestion(name="label", title="text", labels=["positive", "negative"]),
                rg.RatingQuestion(name="rating", title="text", values=[1, 2, 3, 4, 5]),
                rg.TextQuestion(name="text", title="text"),
                rg.SpanQuestion(name="span",
                                title="text", field="text",
                                labels=["Apparatus", "Method", "Machine", "Manufacture", "Design"],
                                visible_labels=3),
            ],
            vectors=[rg.VectorField(name="text", dimensions=3)],
        )
        print(settings.__repr__())
        assert (
            settings.__repr__() == f"Settings(guidelines=None, allow_extra_metadata=False, "
            "distribution=OverlapTaskDistribution(min_submitted=1), "
            f"fields={settings.fields}, questions={settings.questions}, vectors={settings.vectors}, metadata={settings.metadata})"
        )

Without fix
Settings(guidelines=None, allow_extra_metadata=False, distribution=OverlapTaskDistribution(min_submitted=1), fields=<argilla.settings._resource.SettingsProperties object at 0x7fd80ff7ea00>, questions=<argilla.settings._resource.QuestionsProperties object at 0x7fd80ff7e6d0>, vectors=<argilla.settings._resource.SettingsProperties object at 0x7fd80ff7edf0>, metadata=<argilla.settings._resource.SettingsProperties object at 0x7fd80ff7ea30>)
With Fix
Settings(guidelines=None, allow_extra_metadata=False, distribution=OverlapTaskDistribution(min_submitted=1), fields=[TextField(name=text, title=text, description=None, type=text, required=True) , ImageField(name=image, title=image, description=None, type=image, required=True) ], questions=[LabelQuestion(name=label, title=text, description=None, type=label_selection, required=True) , RatingQuestion(name=rating, title=text, description=None, type=rating, required=True) , TextQuestion(name=text, title=text, description=None, type=text, required=True) , SpanQuestion(name=span, title=text, description=None, type=span, required=True) ], vectors=[VectorField(name=text, title=text, dimensions=3)], metadata=[FloatMetadataProperty(name=source, title=source, visible_for_annotators=True)])

That looks great!

Thanks @davidberenstein1957 , raising PR now.

repo: argilla-io/argilla


text:
fix: dataset regex pattern, allow upper case

Fixes #1918

Hi @jamnicki and thanks for the contribution
We cannot support uppercased dataset names without decoupling them from the elasticsearch index creation. Elasticsearch introduces this limitation in the index naming and we're using the dataset name as part of the elasticsearch index.
We need first to decouple that relation and then relax the name pattern validation for the dataset name.

authors: frascuchon jamnicki


text:
Import HTML

Use sanitize-html to reduce HTML to a restricted set of tags, attributes, and styles, possibly add whitespace normalization, extract the plain text and map HTML tags to a list of properties. This feature would also allow pasting styled text (e.g. from Word or Google Docs) into the editor.

Thanks, my intention is to support HTML content in a few stages:
[1] To store the imported HTML as a zero-width annotation (this has now been written but not yet pushed). This allows HTML to be viewed in the document but does not allow the user to annotate the textual content of the imported HTML block
[2] To convert the imported HTML elements to matching stylistic annotations. The sanitize-html library would help establish the correctness and security of the HTML for this operation. The limitations of the approach would be that managing complex layout structures like DIVs, TABLEs, etc., could be complicated in standoff, as sibling order is crucial to the rendering, and this needs work in SPEEDy
[3] Another approach I am thinking through is to combine standoff annotation with embedded HTML ... should be fairly easy to import, but the export process will need to be carefully considered.

Thanks for feedback! Embedded HTML is another issue, I was looking for a way to convert existing documents to standoff-properties data format. I mostly use Markdown but HTML is more common and Markdown can be converted to HTML. The pandoc document converter gives a good summary of most common layout types among document formats (with pandoc you get import from dozens of document formats). Support of this would be enough:

Inline elements

bold (b, strong), italic (i, em)
external links (a)

Block elements

headings (h1, h2...)
paragraphs (p)
quotations (blockquote)

Support of more block elements (horizontal rule, lists, code blocks, tables) might be added later.

authors: argimenes nichtich

repo: argimenes/standoff-properties-editor


text:
fix(build): add step to Makefile to generate signing gpg key

This PR fixes the github action draft release failure in signing binaries.

/cherry-pick master

repo: argoproj-labs/argocd-image-updater


text:
Add a new custom resource to manage the argocd-notifications-cm

Is your feature request related to a problem? Please describe.
Currently, Argo CD Operator supports Notifications as Beta. One of the primary reasons for this is, Argo CD Operator does not manage the configuration(templates, triggers, services, subscriptions e.t.c.,.) of the the argocd-notifications-cm.
Triggers, Templates, Services and Subscriptions can be so huge at times that it creates a bad user experience if they are added to the Argo CD CR and managed by the Argo CD Reconciler of the Argo CD Operator.
Describe the solution you'd like

It is better to create a new CR - NotificationsConfiguration that can be used by the Users to update the their custom templates, triggers e.t.c.,.
A new reconciler will watch this CR to update the argocd-notifications-cm.
argocd-notifications-cm should be created with default triggers, templates e.t.c.,. OOTB.

Additional context
Sample NotificationsConfiguration resource
apiVersion: argoproj.io/v1alpha1
kind: NotificationsConfiguration
metadata:
 name: default-notifications-configuration
spec:
 triggers:
  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'Unknown'
      send: [my-custom-template]
 templates:
  template.my-custom-template: |
    message: |
      Application details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}.

Sorry for replying on a close issue. But does this feature also work with extremely large created argocd notifications configmaps? We have a situation where we needed to do a server side apply of the configmap because it became large. Does the shift to CRD's still allow us for those large configmaps?

authors: GijsvanDulmen iam-veeramalla

repo: argoproj-labs/argocd-operator


text:
Missing securityContext

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
How to control securityContext
I have k8s cluster with installed pod security policy.
My cluster setup for operator 0.15 and argocd 2.0.4
apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: argocd-1
spec:
  image: quay.io/argoproj/argocd
  version: v2.0.4
  redis:
    image: redis
    version: 6.2.4-alpine
  dex:
    image: quay.io/dexidp/dex
    version: sha256:01e996b4b60edcc5cc042227c6965dd63ba68764c25d86b481b0d65f6e4da308
  ha:
    enabled: false
    redisProxyImage: haproxy
    redisProxyVersion: "2.0.4"

In CRD there should be a possibility to run redis image as non root.
How to control securityContext ?
The same here I have with haproxy image in ha setup.

This same problem exists for quite a few configurables required for modifying argocd for argocd-vault-integration.
You can't set security context, mountSAtoken doesn't work, serviceAccountName doesn't work.

Probably this will be addressed by #1493. Making the operator compatible with restrictive pod security policy should work for most of the cases. On the request to make it configurable via ArgoCD CR needs to be discussed.

authors: bmhkb4 djkormo svghadi


text:
downloading artifact from s3 in ui, timed out waiting for condition

Checklist:

[x] I've included the version.
[x] I've included reproduction steps.
[] I've included the workflow YAML.
[x] I've included the logs.

What happened:
Installed the latest 2.5.0rc7 via install.yaml on eks 1.14 and added to the install.yaml  the diff shown in the output below, so the archivedLogs and s3 config are enabled (workflow-controller-configmap):
336,344d322
< data:
<   config: |
<     artifactRepository:
<       archiveLogs: true
<       s3:
<         bucket: "example-argo"
<         keyPrefix: "example"
<         endpoint: "s3.amazonaws.com"
<

to gain ui access on localhost to the ui in kubernetes
kubectl port-forward svc/argo-server 2746:2746 -n argo
run a basic hello world workflow via argo cli and the workflow completes as expected, and clicking on artifacts link in ui shows main-logs object as expected, but when you click to download the actual artfact in the ui, the browser eventually returns a "timed out waiting on condition"
What you expected to happen:
I expect clicking on the link to download the requested artifact.
How to reproduce it (as minimally and precisely as possible):
install.yaml with a s3 config similar to above and run any workflow, and then try and download the resulting main-log artifact.
Logs
argo-server log shows:
time="2020-01-31T23:02:06Z" level=info msg="S3 Load path: artifact368826374, key: example/local-script-gd5zj/local-script-gd5zj/main.log"
time="2020-01-31T23:02:06Z" level=info msg="Creating minio client s3.amazonaws.com using IAM role"
time="2020-01-31T23:02:06Z" level=info msg="Getting from s3 (endpoint: s3.amazonaws.com, bucket: example-argo, key: example/local-script-gd5zj/local-script-gd5zj/main.log) to artifact368826374"
time="2020-01-31T23:02:06Z" level=warning msg="Failed get file: Get https://s3.amazonaws.com/example-argo/?location=: x509: certificate signed by unknown authority"

Message from the maintainers:
If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

@sarabala1979 before I investigate - have you seen similar before please?

generally this kind of error is because the CA that signed s3.amazonaws.com isn't being used in the request.  Depending on your situation & security requirements (https://serverfault.com/questions/444186/is-it-safe-to-use-s3-over-http-from-ec2-as-opposed-to-https), this can likely be fixed by using insecure: true under your s3: configuration section.
Otherwise, it would appear that argo needs to have the CA when making the s3 request.  I haven't looked at the code or docker image to know more.

It looks like the image is from scratch (https://github.com/argoproj/argo/blob/master/Dockerfile#L87-L91), which wouldn't have CA's by default.  I also don't see them COPY'd in.
So unless the CA is built into the binary (which would seem odd), then I'm guessing no certs exist in the docker.  You can always create a secret and mount them into the container and set AWS_CA_BUNDLE env var - requiring only yaml changes.
All this said, I'm guessing this worked previously so I'm probably missing how they got in.

@ddseapy thanks for your suggestions. I tried with insecure: true, i get http: named cookie not present in browser window, and it does not seem to log anything on the argo-server nor the workflow-controller. So I tried to remove the insecure: true line to revert back and now i get "http: named cookie not present" message in browser even after i kubectl delete -f install.yaml and reinstall, and submit a new workflow and try and download that new artifact.
I assume the secret should have the contents of /etc/ssl/certs/ca-certificates.crt? I could give that a try, but I am starting to suspect there is something else going on (as well)? Thoughts welcome.

@haghabozorgi I am hitting the same http: named cookie not present error with insecure: true on rc7.  This is a bug/regression, it worked as recent as 2.4.3. @alexec let me know if you want me to create a separate ticket for this.
Yes, that's generally where ca certs are at on machines.  If AWS_CA_BUNDLE points to wherever that file is within the docker then i'd suspect you wouldn't see that error.  A configmap might be a bit easier than a secret actually to get it into the docker.  But I agree that this would be a quick hack to get past the error.  One of the argo maintainers can probably point to where that cert is supposed to come from.

@ddseapy is there extra config needed to get the artifact store setup in 2.4.3? I was trying to confirm if this is working in 2.4.3 but if i insert the same configmap snippet i see no artifacts list in the archive tab.

One quite way of doing this is to add the following to the argo-server deployment:
      containers:
      - volumeMounts:
        - mountPath: /etc/ssl/certs/ca-certificates.crt
          name: ssl-certs
          readOnly: true

      volumes:
      - hostPath:
          path: /etc/ssl/certs/ca-certificates.crt
          type: ""
        name: ssl-certs

@haghabozorgi I don't think the config is different.  hopefully @tcolgate 's fix works for you.

The executor probably has a valid system CA as it's build on a debian image. the argocli container is more stripped down.

I think this is probably a bug and that the argo-server image may need to be built on the same base as the argocli. This is not straight forward, I'll own it.

@alexec are you also testing with insecure: false or should I make a separate ticket?  the certificate issue wouldn't fix that.

I'm working of a fix that will change from scratch - what would be useful is for someone to test it for me.

@tcolgate thoughts on a solution

Option 1 - update the manifests to include the example you provided?
Option 2 - use the same debian slim image as the executor?

@jessesuen - thoughts on this please?

https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b

@ddseapy  when I add the volume mount and volume snippet per @tcolgate's  suggestion, I still get http named cookie not present when trying to download the artifact from the ui

I don't have access to S3 is a way to reliably test this. I've create a PR, would you be able to use the Dockefile there to try it out?

@haghabozorgi i'm afraid I'm not sure what the cause of that issue is.  I made the other ticket to track that.

@alexec  I am building from your repo now, but not sure if i can test properly given the issue I mentioned above. I assume i will see the same http named cookie not present message?

@haghabozorgi yes.  if you deploy without @tcolgate's fix, and see "http named cookie not present" instead of the original error "x509: certificate signed by unknown authority", then the PR works.

@alexec  using the image from your docker file results in http named cookie not present message.

Ok, but we know that is a good fix for the certs.

@alexec  based on @ddseapy  comments it seems the http named cookie not present is a separate issue, but that seems to be the case whether insecure flag is set to true or false. The end result being the user is still not able to download artifacts from ui.

Ok - should we close this issue once the PR is merged?

i'ts up to ya'll.  if you want to leave this open, i'll close the other one.  if you want to close this one, i'll leave the other one open.

@alexec @tcolgate have you been able to download artifacts through the UI with rc7?  If so, I'm curious why your setup might be different from @haghabozorgi and mine.

I have, but we're using MinIO over HTTP.

I use minio over http as well.

Can you please check you have secrets set-up?
---
apiVersion: v1
data:
  config: |
    artifactRepository:
      archiveLogs: true
      s3:
        bucket: my-bucket
        endpoint: minio:9000
        insecure: true
        accessKeySecret:
          name: my-minio-cred
          key: accesskey
        secretKeySecret:
          name: my-minio-cred
          key: secretkey
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
---
apiVersion: v1
kind: Secret
metadata:
  labels:
    app: minio
  name: my-minio-cred
stringData:
  accesskey: admin
  secretkey: password
type: Opaque

as far as i can tell it's ok.
apiVersion: v1
data:
  config: |
    containerRuntimeExecutor: docker
    artifactRepository:
      archiveLogs: true
      s3:
        accessKeySecret:
          key: accesskey
          name: ddseapy-minio
        secretKeySecret:
          key: secretkey
          name: ddseapy-minio
        bucket: ds-argo-artifacts
        endpoint: ddseapy-minio
        insecure: true
        region: us-east-1
    metricsConfig:
      enabled: true
      path: /metrics
      port: 8080
    persistence:
      archive: true
      connectionPool:
        maxIdleConns: 100
        maxOpenConns: 0
      nodeStatusOffLoad: true
      postgresql:
        host: ddseapy-postgresql
        port: 5432
        database: argo
        tableName: argo_workflows
        userNameSecret:
          name: ddseapy-argo-workflow-controller
          key: postgresqlUsername
        passwordSecret:
          name: ddseapy-argo-workflow-controller
          key: postgresqlPassword
kind: ConfigMap
metadata:
  name: ddseapy-argo-workflow-controller
  namespace: ddseapy
---
apiVersion: v1
data:
  accesskey: YWRtaW4=
  secretkey: cGFzc3dvcmQ=
kind: Secret
metadata:
  labels:
    app: minio
  name: ddseapy-minio
  namespace: ddseapy
type: Opaque

@ddseapy I redacted your paste as you shared un-encrypted credentials. If this is a production system, then you should immediately change your password.

I'm wondering where this error is coming from - can you open your browser console and share the HTTP request and response please?

no, that is simply admin password base64 encoded.

Oh - what auth mode are you using? Server?

Ok. I've reproduced this.

Fix implemented.

@alexec  does your fix address downloading from s3 or is it just for the minio over http use case?

minio is an s3-compatible api.  while, i don't have the ability to test s3, and it sounds like neither does @alexec it almost certainly fixes the error for s3 as well..

@alexec can we please re-open? I am testing install.yaml from master with same s3 snippet in my original comment on this issue and am still seeing http: named cookie not present in browser when I try and download main-logs for example. the url in the browser show http://localhost:2746/artifacts/argo/local-script-j7wg9/local-script-j7wg9/main-logs?Authorization=null

If bug persists - please re-open.

@haghabozorgi master install.yaml points to latest docker tag.  It might be that the docker wasn't on dockerhub yet.  Do you mind setting the docker tags to 2.5.0-rc8 in the yaml just to make sure.
With rc8 I am able to download logs, though all other artifacts are downloaded as .gz, when they are .tgz so I have to rename them after downloading.  I'll open a separate ticket.

The "named cookie" error should be fixed in rc8.

@alexec @ddseapy  thank you both for all your efforts, sincerely appreciate it. confirmed, it is working with rc8. I think we can close this.

Yay!

Hi,
i found another solution if it can help..
This is my config-map used in the controler to allow the container init to push logs on s3
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
[....]
data:
  artifactRepository: |
    archiveLogs: true
    s3:
      bucket: argoworkflow-logs
      endpoint: minio.minio.svc.cluster.local
      insecure: false
[....]
  executor: |
    resources:
[....]
    env:
      - name: SSL_CERT_FILE
        value: /run/secrets/kubernetes.io/serviceaccount/ca.crt

This is the conf of the argo-server deployment who mount the ca.crt and allow the UI to get logs from s3 after the pod is killed.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
[....]
spec:
  template:
    spec:
      containers:
          volumeMounts:
            - name: kube-root-ca
              mountPath: /etc/ssl/certs/kube-root-ca.crt
              subPath: ca.crt
[....]
      volumes:
        - name: kube-root-ca
          configMap:
            name: kube-root-ca.crt
            defaultMode: 0755

authors: alexec ddseapy haghabozorgi tcolgate xrafhue

repo: argoproj/argo-workflows


text:
v3.0.3 cherry-pick

[ ] Peixuan Ding: feat(cli): resubmit workflows by label and field selector (#5807) 507b92cf9

[ ] Yuan Tang: chore: Surface parse duration error in operator (#5814) 3f808661c

[ ] Nicoló Lino: docs: fix typo in async-pattern page (#5810) a65cc3e27

[ ] Radolumbo: fix: Add note about hyphenated variables (#5805) bdd44c723

[ ] Peixuan Ding: feat(cli): Retry workflows by label selector and field selector (#5795) b9a79e065

[x] markterm: fix: Node set updating global output parameter updates global. #5699 (#5700) 8f2acee32

[ ] Peixuan Ding: feat(controller): Add validation for ContainerSet (#5758) 076ff18a8

[x] Yuan Tang: fix: Reset workflow started time to current when retrying workflow. Fixes #5796 (#5798) 4b3a30f4e

[ ] Peixuan Ding: build: fix codegen failure on master branch (#5799) 418395b7f

[x] Wen-Chih (Ryan) Lo: fix: change log level to warn level (#5790) 4af011318

[ ] Alex Collins: test: Refactoring test JSON marshalling. (#5785) 0283fafd0

[ ] AIKAWA: fix(docs): Fix yaml snippet (#5788) 7e974dcda

[ ] sa-: docs: fix typo (#5781) 01f37b81d

[ ] Yuan Tang: feat: Support bucket lifecycle for OSS artifact driver (#5731) 4a55e6f0b

[ ] kennytrytek: feat: Emit WorkflowNodeRunning Event (#5531) 3cdb22a1e

[ ] Niklas Hansson: upgrade github.com/gogo/protobuf (#5780) 66c770993

[ ] Iven: chore: Prevent free-port.sh from killing browsers (#5791) 26f08c10a

[ ] Alex Collins: docs: Add training doc (#5777) 4aff34c9c

[ ] dinever: fix(ui): Fix an UI dropdown flickering issue (#5772) cb55cba07

[ ] Christophe Blin: docs: Update managed-namespace.md (#5768) 9dd3c6a49

[ ] dinever: feat(cli): Stop workflows by label selector and field selector (#5752) 60a64c825

[ ] Yuan Tang: docs: Redirect Argo community documents to argoproj/argoproj (#5734) 0efb57d0c

[ ] dinever: fix(ui): Fix the UI crashing issue (#5751) 05af5edfc

[x] Radolumbo: fix(ui): Remove the ability to change namespaces via the UI in Managed Namespace Mode. Closes #5577 (#5729) 407740046

[ ] dinever: fix(ui): Fix workflow summary page unscrollable issue (#5743) 2a050348b

[ ] Michael Crenshaw: chore: Fix expr array equality operator (#5744) 11e045dca

[x] Michael Ruoss: fix(ui): Fix greediness in regex for auth token replacement (#5746) 500d93387

[ ] dinever: fix(server): Fix the issue where GetArtifact didn't look for input artifacts (#5705) 284adfe16

[ ] dinever: fix(ui): Fix workflow list table column width mismatch (#5736) 511bbed2b

[ ] Tianchu Zhao: chore(url): move edge case paths to /argo-workflows/ (#5730) 0a1bff19d

[ ] Yuan Tang: fix(executor): Remove unnecessary check on resource group (#5723) 2b8740943

[ ] Brandon: docs: Added clarifying points for multi-namespace usage (#5725) e387b41f3

[ ] Alex Collins: ci: Fix argoexec-dev target (#5714) 0ebf6c0a4

done

@sarabala1979 these commits may have not been backported to v3.0.2 - can you please make sure it makes it into v3.0.3?
e3d1d1e822c01e2765bab2d57d9537849cd0f720
4c3b0ac530acaac22abb453df3de09e8c74068fb

Can we also add:
461b0b3cda111da1461c217d4a375c9e8a6fba50

And we need to also add this to fix the build:
Alex Collins: ci: Fix argoexec-dev target (#5714) 0ebf6c0

We should also add  e332be5.

@sarabala1979 Can you please include #5678 in this version. #5736  and #5678 and related, they both should go in the same release. Thanks


text:
chore(deps): bump github.com/vektra/mockery from v2.26.0 to v2.42.0

Motivation
This is part of a series of PRs to update build dependencies until the build no longer panics on Golang 1.22
Modifications
The new version of mockery generated some new files and no other changes were made.
Verification
No verification, the tests will fail if the mock changes do not work.

@isubasinghe Hey, I need help on the nix change for this PR as well. Thanks!

@isubasinghe I'm unsure how to get the new SHA. I wasn't able to see it fail when I run nix develop in the github codespace.

@isubasinghe I'm unsure how to get the new SHA. I wasn't able to see it fail when I run nix develop in the github codespace.

error: hash mismatch in fixed-output derivation '/nix/store/akhi4m5mxiz1m5352ddc59q6w2kxjvw0-source.drv':
specified: sha256-3wJ7XY/wfVXB4nEHLqEtNRscow/KrAo79sr2OgJisSY=
got:    sha256-wwt7rhHWPlYtvudWKb8vk8t19MeN7AMfMugs0XeBDVk=
Weird that you aren't getting these error msgs, could be do to with a buggy version of nix then?

https://github.com/argoproj/argo-workflows/pull/12719#issuecomment-1988234901

@isubasinghe Thanks, I've updated the SHA. It should be good to go once CI passes.

authors: blkperl isubasinghe


text:
feat: mark memoized node as cached

Fixes https://github.com/argoproj/argo-workflows/issues/13848
Motivation
when using memoization, its much easier to debug workflows when you can see that nodes have been cached. currently in large workflows you'd have to go over all the nodes one by one to understand if they are cached, or you could go to the timeline tab.
both of those options offer a bad user experience, that becomes worse as the workflows get larger.

Modifications
changed the icon of successful cached nodes to a database icon (from font awesome) to mark them as cached.

Verification
tested in the UI that it works using the examples/memoized-simple.yaml example
screenshots:
first workflow (no cache hit)

second workflow (cache hit)

I like the choice of icon - for the abstract concept of "cached" it seems to align with the rest of the world https://www.google.com/search?q=icon+to+represent+cache
But replacing the check mark within the node itself might be confusing for existing and new users alike - it looks like it's a step related to database access, and two workflows running the same thing will look quite different. I would suggest an additive approach e.g. an overlaid database symbol - quick sketch from a google slide

I've added the requested change, I'm not much of a UI UX designer so if you have any changes you'd like to have to the colors and what not, please share

this is the new icon


text:
fix(controller): Consider nested expanded task in reference

Fixes https://github.com/argoproj/argo-workflows/issues/5574

@alexec Do you guys have an ETA of when a new release will be cut including this fix?
Cheers


text:
feat(controller): HTTP Template and Agent support feature

Checklist:

[ ] My organization is added to USERS.md.

@bobgy
As we discussed, here the eng build for your testing.
sarabala1979/argocli:agent
sarabala1979/workflow-controller:agent
sarabala1979/argoexec:agent
HTTP Template example
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: good
            template: http
            arguments:
              parameters: [{name: url, value: "http://openlibrary.org/people/george08/lists.json"}]
        - - name: bad
            template: http
            continueOn:
              failed: true
            arguments:
              parameters: [{name: url, value: "http://openlibrary.org/people/george08/nofound.json"}]

    - name: http
      inputs:
        parameters:
          - name: url
      http:
       # url: http://dummy.restapiexample.com/api/v1/employees
       url: "{{inputs.parameters.url}}"

Thank you! I will check it out early next week.

Awesome @sarabala1979! I tried your images and got the agent and HTTP template working!
I was comparing your example with container based example using a simple 10 tasks executed in sequence example:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: http-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: good
            template: http
            arguments:
              parameters: [{name: url, value: "http://openlibrary.org/people/george08/lists.json"}]
        - - name: bad
            template: http
            continueOn:
              failed: true
            arguments:
              parameters: [{name: url, value: "http://openlibrary.org/people/george08/nofound.json"}]
        - - name: good3
            template: http
            arguments:
              parameters: [{name: url, value: "http://openlibrary.org/people/george08/lists.j
... (repeat until good10)

    - name: http
      inputs:
        parameters:
          - name: url
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["hello world {{inputs.parameters.url}}"]
        resources:                # limit the resources
          limits:
            memory: 32Mi
            cpu: 100m

and a similar workflow replacing http template with:
    - name: http
      inputs:
        parameters:
          - name: url
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["hello world {{inputs.parameters.url}}"]
        resources:                # limit the resources
          limits:
            memory: 32Mi
            cpu: 100m

After the initial run, it turns out that Pod latency was no longer the blocker:
hw2 took 20s to run
hello-world took 34s to run

I tried several times, and they all finish with roughly the same time 20s~.
Let me try to understand this, I initially thought that the entire flow for HTTP template would be event based and therefore can be significantly faster. However, it seems that
k get workflow -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startedAt}{"\t"}{.status.finishedAt}{"\n"}'
container-67w5w 2021-06-21T11:07:50Z    2021-06-21T11:09:30Z 1m40s
container-6lv7b 2021-06-21T11:09:58Z    2021-06-21T11:11:43Z 1m45s
container-8thcs 2021-06-21T11:18:24Z    2021-06-21T11:20:24Z 2m
container-fphgj 2021-06-21T11:18:21Z    2021-06-21T11:20:02Z 1m41s
container-sdlcb 2021-06-21T11:18:25Z    2021-06-21T11:20:15Z 1m50s

http-5m4tb      2021-06-21T11:18:21Z    2021-06-21T11:20:02Z 1m41s
http-6vh5l      2021-06-21T11:18:25Z    2021-06-21T11:20:16Z 1m39s
http-87r6j      2021-06-21T11:09:58Z    2021-06-21T11:11:33Z 1m35s
http-njcjs      2021-06-21T11:18:24Z    2021-06-21T11:20:04Z 1m40s
http-v5fss      2021-06-21T11:07:50Z    2021-06-21T11:09:41Z 1m51s

Time needed for both are very similar, and if I check the events, roughly every 10s an update event is emitted.
Therefore, it seems that the bottleneck is on workflow controller sync period.
I wonder if this is my configuration problem? Can we make workflow controller event driven?
Otherwise, if we calculate resource usage, the http template solution might use more resource, because it has one pod running for the entire workflow lifetime.
While the container based solution got several pods, but they don't fill up the entire workflow lifetime.

@Bobgy

Can we make workflow controller event-driven?
The current Workflow Controller is already event-driven like Workflow/POD/Taskset update will trigger the Reprocessing.

Let try on my environment.

Thanks for reviewing. I really appreciate it if I had received this review early so that we would be saved a lot of time in reviewing and addressing the previous review commands.

The WorkflowTaskSetSpec cannot be updated using patch. As both the operator and the agent will be updating the task-set, conflicts will be common. These can be avoided by using patch instead of update. The same applies to the agent itself, it will often conflict with the operator, but if it patches status (rather than update), this can never happen.

We can do that.  But status is defined as a subresource and Agent is only updating (MVP scope) and the controller is updating Spec So, We didn't have a conflict issue. I am fine changing the Patch.

By examining a workflow we can determine two facts:
If any HTTP templates in it, then we know we'll need to create a task-set.
If the workflow is incomplete, and there are any pending node status for HTTP templates, we know we need to create an agent (if not, we know we can delete it).

Currently, implementation will not change workflow execution way like parallel/sequence/Step/DAG.  The user didn't see any difference in execution. The workflow will create Taskset and agent pod when it sees the first HTTP template.

If any HTTP templates in it, then we know we'll need to create a task-set.

If workflow uses WorkflowTemplate or  ClusterWorkflowTemplate, We can't determine if the workflow has an HTTP template.

The current implementation is tightly coupled to creating workflow pods. This has resulted in a lot of code for wait containers, env vars for templates, that can be simplified:

The current Agent is using executor code similar to the data template pattern which needs some env variables for initializing.  if we want just simple POD, We can refactor the Agent implementation to exclude the executor and having a simple HTTP call function.

I've just had a thought. This needs to be tested with RBAC turned on, Docker for Desktop does not support RBAC. Can you test with K3S?

I am using kind which has RBAC support like k3s. I have added RBAC on argo-role and workflow-role for workflowtaskset


text:
Experiencing issues with the PNS executor

Hi @jessesuen, we are experimenting with the Argo PNS executor from PR #1214 and running it as the KubeFlow Pipeline backend. The Workflow runs smoothly for most of the containers, except we are experiencing some race condition with the last container in every Workflow. Below are the workflow definition we have and the corresponding error logs from the argoexec.
Failed wait container logs:
$ kubectl logs pipeline-flip-coin-8fn8h-3450342589 -n kubeflow wait
time="2019-03-11T18:44:36Z" level=info msg="Creating PNS executor (namespace: kubeflow, pod: pipeline-flip-coin-8fn8h-3450342589)"
time="2019-03-11T18:44:36Z" level=info msg="Executor (version: v2.3.0+83942fc.dirty, build_date: 2019-03-11T18:38:46Z) initialized with template:\narchiveLocation:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: mlpipeline-minio-artifact\n    bucket: mlpipeline\n    endpoint: minio-service.kubeflow:9000\n    insecure: true\n    key: artifacts/pipeline-flip-coin-8fn8h/pipeline-flip-coin-8fn8h-3450342589\n    secretKeySecret:\n      key: secretkey\n      name: mlpipeline-minio-artifact\ncontainer:\n  command:\n  - echo\n  - tails and 13 <= 15!\n  image: alpine:3.6\n  name: \"\"\n  resources: {}\ninputs:\n  parameters:\n  - name: random-number-2-output\n    value: \"13\"\nmetadata: {}\nname: print-2-3-4\noutputs:\n  artifacts:\n  - name: mlpipeline-ui-metadata\n    path: /mlpipeline-ui-metadata.json\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: mlpipeline-minio-artifact\n      bucket: mlpipeline\n      endpoint: minio-service.kubeflow:9000\n      insecure: true\n      key: runs/ac598e95-442d-11e9-a431-ba1598362405/pipeline-flip-coin-8fn8h-3450342589/mlpipeline-ui-metadata.tgz\n      secretKeySecret:\n        key: secretkey\n        name: mlpipeline-minio-artifact\n  - name: mlpipeline-metrics\n    path: /mlpipeline-metrics.json\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: mlpipeline-minio-artifact\n      bucket: mlpipeline\n      endpoint: minio-service.kubeflow:9000\n      insecure: true\n      key: runs/ac598e95-442d-11e9-a431-ba1598362405/pipeline-flip-coin-8fn8h-3450342589/mlpipeline-metrics.tgz\n      secretKeySecret:\n        key: secretkey\n        name: mlpipeline-minio-artifact\n"
time="2019-03-11T18:44:36Z" level=info msg="Waiting on main container"
time="2019-03-11T18:44:36Z" level=warning msg="Polling root processes (1m0s)"
time="2019-03-11T18:44:37Z" level=warning msg="Failed to open /proc/20/root: open /proc/20/root: permission denied"
time="2019-03-11T18:44:37Z" level=warning msg="Failed to open /proc/20/root: open /proc/20/root: permission denied"
time="2019-03-11T18:44:37Z" level=warning msg="Failed to open /proc/20/root: open /proc/20/root: permission denied"
time="2019-03-11T18:44:37Z" level=info msg="main container started with container ID: fbecd8dcbd49bc961e2462fb76552546d7c5d3739faba488ccd6f3158bec7aad"
time="2019-03-11T18:44:37Z" level=info msg="Starting annotations monitor"
time="2019-03-11T18:44:37Z" level=info msg="Starting deadline monitor"
time="2019-03-11T18:44:37Z" level=error msg="executor error: Could not find associated pid for containerID fbecd8dcbd49bc961e2462fb76552546d7c5d3739faba488ccd6f3158bec7aad\ngithub.com/argoproj/argo/errors.New\n\t/go/src/github.com/argoproj/argo/errors/errors.go:49\ngithub.com/argoproj/argo/errors.Errorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:55\ngithub.com/argoproj/argo/errors.InternalErrorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:65\ngithub.com/argoproj/argo/workflow/executor/pns.(*PNSExecutor).getContainerPID\n\t/go/src/github.com/argoproj/argo/workflow/executor/pns/pns.go:257\ngithub.com/argoproj/argo/workflow/executor/pns.(*PNSExecutor).Wait\n\t/go/src/github.com/argoproj/argo/workflow/executor/pns/pns.go:147\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).Wait\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:839\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:32\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
time="2019-03-11T18:44:37Z" level=info msg="No sidecars"
time="2019-03-11T18:44:37Z" level=info msg="Saving logs"
time="2019-03-11T18:44:37Z" level=info msg="Annotations monitor stopped"
time="2019-03-11T18:44:37Z" level=info msg="S3 Save path: /argo/outputs/logs/main.log, key: artifacts/pipeline-flip-coin-8fn8h/pipeline-flip-coin-8fn8h-3450342589/main.log"
time="2019-03-11T18:44:37Z" level=info msg="Creating minio client minio-service.kubeflow:9000 using static credentials"
time="2019-03-11T18:44:37Z" level=info msg="Saving from /argo/outputs/logs/main.log to s3 (endpoint: minio-service.kubeflow:9000, bucket: mlpipeline, key: artifacts/pipeline-flip-coin-8fn8h/pipeline-flip-coin-8fn8h-3450342589/main.log)"
time="2019-03-11T18:44:38Z" level=info msg="Deadline monitor stopped"
time="2019-03-11T18:44:39Z" level=info msg="No output parameters"
time="2019-03-11T18:44:39Z" level=info msg="Saving output artifacts"
time="2019-03-11T18:44:39Z" level=info msg="Staging artifact: mlpipeline-ui-metadata"
time="2019-03-11T18:44:39Z" level=info msg="Copying /mlpipeline-ui-metadata.json from container base image layer to /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-03-11T18:44:39Z" level=error msg="executor error: could not chroot into main for artifact collection: container may have exited too quickly\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor/pns.(*PNSExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/pns/pns.go:122\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:329\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:234\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:220\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:54\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"
time="2019-03-11T18:44:39Z" level=info msg="Alloc=3802 TotalAlloc=14044 Sys=70078 NumGC=6 Goroutines=10"
time="2019-03-11T18:44:39Z" level=fatal msg="could not chroot into main for artifact collection: container may have exited too quickly\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor/pns.(*PNSExecutor).CopyFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/pns/pns.go:122\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).stageArchiveFile\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:329\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).saveArtifact\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:234\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveArtifacts\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:220\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:54\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333"

Workflow yaml file:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: pipeline-flip-coin-
spec:
  arguments:
    parameters: []
  entrypoint: pipeline-flip-coin
  serviceAccountName: pipeline-runner
  templates:
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-output
            value: '{{tasks.random-number.outputs.parameters.random-number-output}}'
        dependencies:
        - random-number
        name: condition-2
        template: condition-2
        when: '{{tasks.random-number.outputs.parameters.random-number-output}} > 5'
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-output
            value: '{{tasks.random-number.outputs.parameters.random-number-output}}'
        dependencies:
        - random-number
        name: condition-3
        template: condition-3
        when: '{{tasks.random-number.outputs.parameters.random-number-output}} <=
          5'
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
        name: random-number
        template: random-number
    inputs:
      parameters:
      - name: flip-output
    name: condition-1
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-output
            value: '{{inputs.parameters.random-number-output}}'
        name: print
        template: print
    inputs:
      parameters:
      - name: flip-output
      - name: random-number-output
    name: condition-2
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-output
            value: '{{inputs.parameters.random-number-output}}'
        name: print-2
        template: print-2
    inputs:
      parameters:
      - name: flip-output
      - name: random-number-output
    name: condition-3
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-2-output
            value: '{{tasks.random-number-2.outputs.parameters.random-number-2-output}}'
        dependencies:
        - random-number-2
        name: condition-5
        template: condition-5
        when: '{{tasks.random-number-2.outputs.parameters.random-number-2-output}}
          > 15'
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-2-output
            value: '{{tasks.random-number-2.outputs.parameters.random-number-2-output}}'
        dependencies:
        - random-number-2
        name: condition-6
        template: condition-6
        when: '{{tasks.random-number-2.outputs.parameters.random-number-2-output}}
          <= 15'
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
        name: random-number-2
        template: random-number-2
    inputs:
      parameters:
      - name: flip-output
    name: condition-4
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-2-output
            value: '{{inputs.parameters.random-number-2-output}}'
        name: print-2-3
        template: print-2-3
    inputs:
      parameters:
      - name: flip-output
      - name: random-number-2-output
    name: condition-5
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{inputs.parameters.flip-output}}'
          - name: random-number-2-output
            value: '{{inputs.parameters.random-number-2-output}}'
        name: print-2-3-4
        template: print-2-3-4
    inputs:
      parameters:
      - name: flip-output
      - name: random-number-2-output
    name: condition-6
  - container:
      args:
      - python -c "import random; result = 'heads' if random.randint(0,1) == 0 else
        'tails'; print(result)" | tee /tmp/output
      command:
      - sh
      - -c
      image: python:alpine3.6
    name: flip
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      parameters:
      - name: flip-output
        valueFrom:
          path: /tmp/output
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: flip-output
            value: '{{tasks.flip.outputs.parameters.flip-output}}'
        dependencies:
        - flip
        name: condition-1
        template: condition-1
        when: '{{tasks.flip.outputs.parameters.flip-output}} == heads'
      - arguments:
          parameters:
          - name: flip-output
            value: '{{tasks.flip.outputs.parameters.flip-output}}'
        dependencies:
        - flip
        name: condition-4
        template: condition-4
        when: '{{tasks.flip.outputs.parameters.flip-output}} == tails'
      - name: flip
        template: flip
    name: pipeline-flip-coin
  - container:
      command:
      - echo
      - heads and {{inputs.parameters.random-number-output}} > 5!
      image: alpine:3.6
    inputs:
      parameters:
      - name: random-number-output
    name: print
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
  - container:
      command:
      - echo
      - heads and {{inputs.parameters.random-number-output}} <= 5!
      image: alpine:3.6
    inputs:
      parameters:
      - name: random-number-output
    name: print-2
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
  - container:
      command:
      - echo
      - tails and {{inputs.parameters.random-number-2-output}} > 15!
      image: alpine:3.6
    inputs:
      parameters:
      - name: random-number-2-output
    name: print-2-3
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
  - container:
      command:
      - echo
      - tails and {{inputs.parameters.random-number-2-output}} <= 15!
      image: alpine:3.6
    inputs:
      parameters:
      - name: random-number-2-output
    name: print-2-3-4
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
  - container:
      args:
      - python -c "import random; print(random.randint(0,9))" | tee /tmp/output
      command:
      - sh
      - -c
      image: python:alpine3.6
    name: random-number
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      parameters:
      - name: random-number-output
        valueFrom:
          path: /tmp/output
  - container:
      args:
      - python -c "import random; print(random.randint(10,19))" | tee /tmp/output
      command:
      - sh
      - -c
      image: python:alpine3.6
    name: random-number-2
    outputs:
      artifacts:
      - name: mlpipeline-ui-metadata
        path: /mlpipeline-ui-metadata.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-ui-metadata.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      - name: mlpipeline-metrics
        path: /mlpipeline-metrics.json
        s3:
          accessKeySecret:
            key: accesskey
            name: mlpipeline-minio-artifact
          bucket: mlpipeline
          endpoint: minio-service.kubeflow:9000
          insecure: true
          key: runs/{{workflow.uid}}/{{pod.name}}/mlpipeline-metrics.tgz
          secretKeySecret:
            key: secretkey
            name: mlpipeline-minio-artifact
      parameters:
      - name: random-number-2-output
        valueFrom:
          path: /tmp/output

Related issues: #970
cc: @animeshsingh
Is this a BUG REPORT or FEATURE REQUEST?:
BUG REPORT
What happened:
Filehandle not being secured before the main container started.
What you expected to happen:
Filehandle should be secured before the main container started.
How to reproduce it (as minimally and precisely as possible):
Run the workflow definition above with the PNS executor
Anything else we need to know?:
Environment:

Argo version: 2.3.0
Kubernetes version : 1.13.4

Update:
It looks like the Share Process Namespace from the "wait" container doesn't have enough permission to copy files in "main" for the the last workflow container. My temporary work around is adding SYS_PTRACE and set privileged: true to the "wait" container. However, occasionally I still experiencing some race condition where the "wait" container is trying to copy the output files when it's not ready. Below are the error logs when the race condition happened.
time="2019-03-14T20:29:00Z" level=info msg="Creating PNS executor (namespace: kubeflow, pod: pipeline-flip-coin-fzmf4-745333038)"
time="2019-03-14T20:29:00Z" level=info msg="Executor (version: v2.3.0+83942fc.dirty, build_date: 2019-03-14T20:25:57Z) initialized with template:\narchiveLocation:\n  archiveLogs: true\n  s3:\n    accessKeySecret:\n      key: accesskey\n      name: mlpipeline-minio-artifact\n    bucket: mlpipeline\n    endpoint: minio-service.kubeflow:9000\n    insecure: true\n    key: artifacts/pipeline-flip-coin-fzmf4/pipeline-flip-coin-fzmf4-745333038\n    secretKeySecret:\n      key: secretkey\n      name: mlpipeline-minio-artifact\ncontainer:\n  args:\n  - python -c \"import random; result = 'heads' if random.randint(0,1) == 0 else 'tails';\n    print(result)\" | tee /tmp/output\n  command:\n  - sh\n  - -c\n  image: python:alpine3.6\n  name: \"\"\n  resources: {}\ninputs: {}\nmetadata: {}\nname: flip\noutputs:\n  artifacts:\n  - name: mlpipeline-ui-metadata\n    path: /mlpipeline-ui-metadata.json\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: mlpipeline-minio-artifact\n      bucket: mlpipeline\n      endpoint: minio-service.kubeflow:9000\n      insecure: true\n      key: runs/cb43580d-4697-11e9-a431-ba1598362405/pipeline-flip-coin-fzmf4-745333038/mlpipeline-ui-metadata.tgz\n      secretKeySecret:\n        key: secretkey\n        name: mlpipeline-minio-artifact\n  - name: mlpipeline-metrics\n    path: /mlpipeline-metrics.json\n    s3:\n      accessKeySecret:\n        key: accesskey\n        name: mlpipeline-minio-artifact\n      bucket: mlpipeline\n      endpoint: minio-service.kubeflow:9000\n      insecure: true\n      key: runs/cb43580d-4697-11e9-a431-ba1598362405/pipeline-flip-coin-fzmf4-745333038/mlpipeline-metrics.tgz\n      secretKeySecret:\n        key: secretkey\n        name: mlpipeline-minio-artifact\n  parameters:\n  - name: flip-output\n    valueFrom:\n      path: /tmp/output\n"
time="2019-03-14T20:29:00Z" level=info msg="Waiting on main container"
time="2019-03-14T20:29:00Z" level=warning msg="Polling root processes (1m0s)"
time="2019-03-14T20:29:00Z" level=info msg="Secured filehandle on /proc/18/root"
time="2019-03-14T20:29:00Z" level=info msg="containerID 3130eaf160ff149e0b35aaef5718001026b52413476615e41d82ef6ac92b86f2 mapped to pid 18"
time="2019-03-14T20:29:00Z" level=info msg="main container started with container ID: 3130eaf160ff149e0b35aaef5718001026b52413476615e41d82ef6ac92b86f2"
time="2019-03-14T20:29:00Z" level=info msg="Starting annotations monitor"
time="2019-03-14T20:29:00Z" level=info msg="Main pid identified as 18"
time="2019-03-14T20:29:00Z" level=info msg="Successfully secured file handle on main container root filesystem"
time="2019-03-14T20:29:00Z" level=info msg="Waiting for main pid 18 to complete"
time="2019-03-14T20:29:00Z" level=info msg="Starting deadline monitor"
time="2019-03-14T20:29:00Z" level=info msg="Stopped root processes polling due to successful securing of main root fs"
time="2019-03-14T20:29:01Z" level=info msg="Main pid 18 completed"
time="2019-03-14T20:29:01Z" level=info msg="Main container completed"
time="2019-03-14T20:29:01Z" level=info msg="No sidecars"
time="2019-03-14T20:29:01Z" level=info msg="Saving logs"
time="2019-03-14T20:29:01Z" level=info msg="Deadline monitor stopped"
time="2019-03-14T20:29:01Z" level=info msg="Annotations monitor stopped"
time="2019-03-14T20:29:01Z" level=info msg="S3 Save path: /argo/outputs/logs/main.log, key: artifacts/pipeline-flip-coin-fzmf4/pipeline-flip-coin-fzmf4-745333038/main.log"
time="2019-03-14T20:29:01Z" level=info msg="Creating minio client minio-service.kubeflow:9000 using static credentials"
time="2019-03-14T20:29:01Z" level=info msg="Saving from /argo/outputs/logs/main.log to s3 (endpoint: minio-service.kubeflow:9000, bucket: mlpipeline, key: artifacts/pipeline-flip-coin-fzmf4/pipeline-flip-coin-fzmf4-745333038/main.log)"
time="2019-03-14T20:29:03Z" level=info msg="Saving output parameters"
time="2019-03-14T20:29:03Z" level=info msg="Saving path output parameter: flip-output"
time="2019-03-14T20:29:03Z" level=info msg="Copying /tmp/output from base image layer"
time="2019-03-14T20:29:03Z" level=error msg="executor error: open /tmp/output: no such file or directory"
time="2019-03-14T20:29:03Z" level=info msg="Alloc=3858 TotalAlloc=12562 Sys=70078 NumGC=5 Goroutines=9"
time="2019-03-14T20:29:03Z" level=fatal msg="open /tmp/output: no such file or directory"

@Tomcli - is it possible to construct a smaller, portable workflow which can reproduce this? Also, there's some caveats to PNS that people need to be aware of, which is: collection of artifacts from the base image layer is subject to race conditions when the main container exits too quickly.
Basically the main container needs to be running for a few seconds for the wait sidecar to reliably secure the filehandle on it's root filesystem. If the main container exits too quickly, then the wait sidecar may not have been able to secure the file handle to successfully collect artifacts.

My temporary work around is adding SYS_PTRACE and set privileged: true to the "wait" container. However, occasionally I still experiencing some race condition where the "wait" container is trying to copy the output files when it's not ready

Yes I don't expect privileged mode to help.
However, an alternative workaround is to output the artifacts into an emptyDir volume, mounted in the main container. In v2.3, when a volumes are used, they are now mirrored to the wait sidecar, which eliminates the race with artifact collection, because the wait sidecar has access to the volume long after the main container completed.

My temporary work around is adding SYS_PTRACE and set privileged: true to the "wait" container.
Actually I'm wrong. SYS_PTRACE is needed when the user id of the main container is different than the wait sidecar.

However, occasionally I still experiencing some race condition where the "wait" container is trying to copy the output files when it's not ready

I'm also experiencing this race condition. Trying to find a solution, but it does seem timing related.

Hi @jessesuen, Thanks for the reply. Since adding SYS_PTRACE and privileged mode is not that we want and we don't have a better way to work around PNS, right now we switched to use the k8sapi executor with emptyDir as our temporary solution.

Just to be clear, privileged is unnecessary, but SYS_PTRACE is. The latter is much more secure than having privileged pods.

@Tomcli I fixed the SYS_PTRACE issue, and also figured out the timing related issue about failing to upload the artifact. PNS should be working much more reliably now in latest version of the PR:
https://github.com/argoproj/argo/pull/1214
Given that privileged pods is unnecessary, I think you may want to reconsider PNS.

Thanks @jessesuen - we will give it a try. With respect to k8sapi  executor - you have a viewpoint? Ideally that should be the solution to use with CRI-O?

@animeshsingh there are pros and cons to each executor:

Docker:

+ supports all workflow examples
+ most reliable and well tested
+ very scalable. communicates to docker daemon for heavy lifting
- least secure. requires docker.sock of host to be mounted (often rejected by OPA)

kubelet

+ secure. cannot escape privileges of pod's service account
+ medium scalability - log retrieval and container polling is done against kubelet
- additional kubelet configuration may be required
- can only save params/artifacts in volumes (e.g. emptyDir), and not the base image layer (e.g. /tmp)

K8s API

+ secure. cannot escape privileges of pod's service account
+ no extra configuration
- least scalable - log retrieval and container polling is done against k8s API server
- can only save params/artifacts in volumes (e.g. emptyDir), and not the base image layer (e.g. /tmp)

PNS

+ secure. cannot escape privileges of service account
+ artifact collection can be collected from base image layer
+ scalable - process polling is done over procfs and not kubelet/k8s API
- processes will no longer run with pid 1
- artifact collection from base image may fail for containers which complete too fast
- cannot capture artifact directories from base image layer which has a volume mounted under it
- immature

IMO, PNS is the closest thing to the docker executor, without the security concerns, and is what I recommend, except for the fact that it is most immature.

Thanks @jessesuen for this comparison. Would the overhead of going through k8s apis bypass the demerits introduced through some randomness using PNS? Given that workflows are expected to be long running jobs, as opposed to a serverless model where bypassing k8s api has its merits vis a vis response, would it matter too much?  Also how important it is to store the artifacts in base image layer?

My feeling is PNS is the best compromise between security and functionality.

Would the overhead of going through k8s apis bypass the demerits introduced through some randomness using PNS?

The "randomness" of failing to collect artifacts is usually a non-issue unless containers are completing too quick. Even then, you can mitigate this by outputting the artifact to an emptyDir, and this would never be an issue.

Also how important it is to store the artifacts in base image layer?

Not necessary at all. It's just slightly more convenient not to have to define a emptyDir volume to collect artifacts.
Closing bug since PNS has merged.

@jessesuen

artifact collection from base image may fail for containers which complete too fast

this is causing a bunch of race conditions in our stuff, should we open a separate issue for this on PNS, or do you have any recommendations on how to deal with it properly?

@booninite yes. To ensure that the wait sidecar is able to collect outputs, instead of outputting outputs into the base image layer (such as /tmp), output artifacts into an empty dir (which gets mirrored into the wait sidecar). This ensures that the wait sidecar can collect the artifact without subject to timing problems.

@jessesuen we are still experiencing intermittent artifact passing issues using emptyDir.  Does the emptyDir additionally need to be mounted to a path that does not exist in the base image?

@jessesuen the emptyDir isn't a full proof solution - are there folks actually using PNS executor in real world scenarios?

We are running ~5k workflows per month that all use PNS. We only see consistent issues with extremely short duration steps, under 15 seconds.

https://circleci.com/gh/argoproj/argo/72?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

I see this same issue trying to pass a single file between my workflows, is the volume mount the solution?

yeah, seeing this with pns as well. Not sure what to do here...

Having the same issue. Running K3OS with CR-IO. So I can't use the docker executor. The other two, kubelet and k8api, simply won't work. Kubelet gives me a certificate error, which the helm chart doesn't give an option for ignoring and k8api give me errors like "function not found"...

@sarabala1979 is the workaround for this emptyDir?

I was finally able to get it running using the k8sapi
spec:   volumes:     - name: source       emptyDir: {}
Sadly this brakes the functionality of the built-in git solution, because apparently it can not write into a volume. I had to write my own git clone script. Also this kind of makes the artifact passing redundant, as I could just use this volume in every stage.

Any workaround for this?

@alexec Is this issue also fixed, as I see https://github.com/argoproj/argo/issues/4230 is fixed/closed?

Test it and see?

I'm experiencing a similar problem on some of my containers, on af03a74fb334c88493e38ed4cb94f771a97bffc5. Other containers doing almost identical work succeed, and if I keep retrying the workflow everything succeeds eventually. Seems particular to PNS. Here's an example wait container log:
time="2020-12-20T23:40:18.772Z" level=info msg="Waiting on main container"
time="2020-12-20T23:40:24.920Z" level=info msg="main container started with container ID: 053703ca0cd2eceaf68f64286fe641f09f7bedfc3547f887f8f1eca5f8617706"
time="2020-12-20T23:40:24.920Z" level=info msg="Starting annotations monitor"
time="2020-12-20T23:40:24.927Z" level=info msg="containerID 053703ca0cd2eceaf68f64286fe641f09f7bedfc3547f887f8f1eca5f8617706 mapped to pid 41"
time="2020-12-20T23:40:24.927Z" level=info msg="Main pid identified as 41"
time="2020-12-20T23:40:24.927Z" level=warning msg="Failed to secure file handle on main container's root filesystem. Output artifacts from base image layer will fail"
time="2020-12-20T23:40:24.927Z" level=info msg="Waiting for main pid 41 to complete"
time="2020-12-20T23:40:24.927Z" level=info msg="Starting deadline monitor"
time="2020-12-20T23:41:53.967Z" level=info msg="Main pid 41 completed"
time="2020-12-20T23:41:54.006Z" level=info msg="Main container completed"
time="2020-12-20T23:41:54.012Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2020-12-20T23:41:54.012Z" level=info msg="Capturing script exit code"
time="2020-12-20T23:41:54.018Z" level=info msg="Getting exit code of 053703ca0cd2eceaf68f64286fe641f09f7bedfc3547f887f8f1eca5f8617706"
time="2020-12-20T23:41:54.080Z" level=info msg="Annotations monitor stopped"
time="2020-12-20T23:41:54.118Z" level=info msg="Deadline monitor stopped"
time="2020-12-20T23:41:57.058Z" level=error msg="executor error: could not get container status: timed out waiting for the condition"
time="2020-12-20T23:41:57.058Z" level=info msg="Killing sidecars"
time="2020-12-20T23:41:57.079Z" level=info msg="Alloc=5590 TotalAlloc=15483 Sys=70848 NumGC=5 Goroutines=10"
time="2020-12-20T23:41:57.159Z" level=fatal msg="could not get container status: timed out waiting for the condition"

I think timed out waiting for the condition might be a new issue. Has anyone run the v2.11.8 executor vs the v2.12.2 executor?

We do not see Failed to get main, so that discounts #4523.

OK. Diagnosis - There is a timeout trying to determine if the pod has finished. We allow three attempts at 1-second intervals. The main container has completed (which we use the shared process namespace to determine), but we ask the Kubernetes API for the actual result. The API has not been updated yet. This could be mitigated by increasing the amount of time we allow the executor to poll for on line 375 of pns.go. The core team short-staffed through until 2021. @foobarbecue would you be interested in submitting a fix?

@alexec Sure, I can play with the timing and see if I come up with a good PR-worthy solution.

Thank you!

Hi guys,
I was experiencing a lot the same issue recently. Following the comment from @alexec above I've tried to install a previous argo version and everything works well as usual.
The downgraded version I've installed is using workflow-controller and argoexec v2.11.8

Maybe fixed in https://github.com/argoproj/argo/pull/4954.

Maybe fixed in https://github.com/argoproj/argo/pull/4954.

v3.0 will have a controller envvar name PNS_PRIVILEGED.

authors: 0-duke Kampe Tomcli alexec animeshsingh bharathguvvala booninite foobarbecue ggogel guoweis-outreach jessesuen shimmerjs

repo: argoproj/argo


text:
Workflow stuck when Failed pod status switches back to Pending

hecklist:

[x] I've included the version.
[x] I've included reproduction steps.
[x] I've included the workflow YAML.
[ ] I've included the logs.

What happened:
I have a large workflow that I need to run, represented as a DAG with a large number of mutually independent nodes (probably not the best way to be represented), that run in parallel. Each of the tasks in the DAG could take from a few minutes to a few hours to complete, and there are between 200 to 600 of such tasks scheduled in the workflow.
Whenever I run the workflow (tried both as a Workflow and CronWorkflow), all of the tasks complete, some of them after failing and retrying. However, there is always at least one task that ends up looking like this (see graphs(133:133)(0)):
[...]
 ├-✔ graphs(132:132)              graph
 | ├-✖ graphs(132:132)(0)         graph        g-20200422110754-000001-kg42p-1065241647  33m       pod termination
 | └-✔ graphs(132:132)(1)         graph        g-20200422110754-000001-kg42p-595615410   24m
 ├-✔ graphs(133:133)              graph
 | ├-◷ graphs(133:133)(0)         graph        g-20200422110754-000001-kg42p-2541540603  33m       ContainerCreating
 | └-✔ graphs(133:133)(1)         graph        g-20200422110754-000001-kg42p-2071914366  25m
 ├-✔ graphs(134:134)              graph
 | ├-✖ graphs(134:134)(0)         graph        g-20200422110754-000001-kg42p-2203744947  33m       pod termination
 | └-✔ graphs(134:134)(1)         graph        g-20200422110754-000001-kg42p-2807886326  7m
[...]

The first pod to attempt the task seems to be stuck in ContainerCreating, however the second attempt seems to be completed successfully. Whenever one pod manifests this behaviour, the workflow remains in the Running state, even though the parent task graphs(133:133) is marked as completed.
According to Kubernetes, the pod actually does not exist anymore, and a workflow like this one will remain in the Running status forever.
I can verify externally that the parent task graphs(133:133) has indeed completed, as it has populated a bucket in GCS with the output for that task.
See the Anything else we need to know section for additional info.
What you expected to happen:
Argo should set the message of graphs(133:133)(0) to pod termination or similar, and the workflow should complete and be labeled as Succeeded.
How to reproduce it (as minimally and precisely as possible):
This is the workflow that manifests the issue for me (it is not directly runnable, as I had to remove sensitive info):
apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
  entrypoint: begin
  templates:
  - name: begin
    dag:
      tasks:
      - name: graphs
        template: graph
        arguments:
          parameters:
          - name: task-id
            value: "{{item}}"
        withSequence:
          count: 200
  - name: graph
    inputs:
      parameters:
      - name: task-id
    metadata:
      annotations:
        "cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: preemptible
              operator: In
              values:
              - "True"
            - key: cpu
              operator: In
              values:
              - "32"
    container:
      image: <image name>:<image tag>
      args: ["execute", "--task-id {{inputs.parameters.task-id}}"]
      resources:
        requests:
          cpu: 1.00
          memory: 3.50Gi
        limits:
          cpu: 1.00
          memory: 24.0Gi
      imagePullPolicy: Always
    resubmitPendingPods: true
    retryStrategy:
      limit: 10
      retryPolicy: "Always"
      backoff:
        duration: 15
        factor: 1

Anything else we need to know?:

The workflow is run on a GKE cluster on a node pool with a Cluster Autoscaler.
The node pool consists of preemptible machines. However, the workflow seems to behave correctly when a node is shut down, as the pod is labeled with ⚠, the message is pod deleted, and the next attempt succeeds.
Going through the Stackdriver logs for both the cluster and the workflow-controller, I reconstructed this timeline:

[10:16:00] Pod is created and started successfully
[10:46:39] Cluster Autoscaler decides to kill the node on which the pod is running
[10:46:41] Workflow controller sets the phase of the pod from Running to Failed and the message to pod termination.
[10:46:44] Pod is labeled as completed
[10:46:55] New pod created and successfully transition its phase to Pending
[10:47:00] New pod transitioning its phase to Running
[10:47:15] The old pod's phase is updated from Failed -> Pending, and the message is set to ContainerCreating
[11:12:14] New pod Succeeds and the parent task's phase is also set to Succeeded

As you can see, for some reason the pod that previously failed and was replaced is labeled as Pending again, from which it will never recover. I believe that is the reason why the workflow never succeeds.
Environment:
Running on GKE.

Argo version:

$ argo version
argo: v2.7.3
  BuildDate: 2020-04-15T22:52:59Z
  GitCommit: 66bd0425280c801c06f21cf9a4bed46ee6f1e660
  GitTreeState: clean
  GitTag: v2.7.3
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

Workflow controller/Argo server/Argoexec are all using the v2.8.0-rc3 image, but I have tried with v2.7.3 and v2.8.0 as well and the problem remains.

Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-10-15T19:16:51Z"
  compiler: gc
  gitCommit: 20c265fef0741dd71a66480e35bd69f18351daea
  gitTreeState: clean
  gitVersion: v1.15.5
  goVersion: go1.12.10
  major: "1"
  minor: "15"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-03-31T02:49:49Z"
  compiler: gc
  gitCommit: a5bf731ea129336a3cf32c3375317b3a626919d7
  gitTreeState: clean
  gitVersion: v1.15.11-gke.5
  goVersion: go1.12.17b4
  major: "1"
  minor: 15+
  platform: linux/amd64

Message from the maintainers:
If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

Something that is invaluable when debugging similar errors is the full Workflow object that causes this error after it finishes running (or in this case stops running any further).
You can get it by running kubectl get wf <NAME> -o yaml. If it contains any sensitive company data and you're not comfortable sharing it publicly, you can share it with me privately on the project Slack in @simon

Possible workaround for this is to omit resubmitPendingPods: true

#3214

Discussed on Slack and fixed.

authors: alexec mmaggiolo simster7


text:
Failed to generate download URL for artifact : No module named 'ax.platform_client'

A 1.0.1 cluster fails to download a completed log, failing with an import error with following stacktace:
2017-10-06T00:15:29 ERROR ax.devops.artifact.rest 49 DummyThread-53: Error occurred: code = ERR_API_INTERNAL_ERROR, message = Internal server error, detail = Failed to generate download URL for artifact (id: 81794a18-dafe-4531-9437-3e1ea388f92f): No module named 'ax.platform_client'
Traceback (most recent call last):
  File "/ax/src/ax/devops/artifact/artifactmanager.py", line 1121, in download_artifact
    location = Cloud().get_bucket(bucket).generate_signed_url(key)
  File "/ax/src/ax/cloud/cloud.py", line 157, in get_bucket
    return AXS3Bucket(bucket_name, aws_profile, region=region)
  File "/ax/src/ax/cloud/aws/aws_s3.py", line 100, in __init__
    self._region = region if region else self._get_bucket_region_from_aws()
  File "/usr/lib/python3.5/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python3.5/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/usr/lib/python3.5/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python3.5/site-packages/six.py", line 686, in reraise
    raise value
  File "/usr/lib/python3.5/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/ax/src/ax/cloud/aws/aws_s3.py", line 144, in _get_bucket_region_from_aws
    return _do_get_region(Cloud().meta_data().get_region())
  File "/ax/src/ax/cloud/cloud.py", line 120, in meta_data
    from ax.aws.meta_data import AWSMetaData
  File "/ax/src/ax/aws/meta_data.py", line 10, in <module>
    from ax.platform_client.env import AXEnv
ImportError: No module named 'ax.platform_client'

Check with @shrinandj

@shrinandj Please look into this

This applies to all artifacts attempted to be downloaded from UI, not just logs.

The import is failing:
from ax.platform_client.env import AXEnv

but this file and class does exist:
$ find . -name env.py
./platform/source/lib/ax/platform_client/env.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# Copyright 2015-2016 Applatix, Inc. All rights reserved.
#

"""
Module library to identify python running environment.
"""

import os

from future.utils import with_metaclass

from ax.util.singleton import Singleton

class AXEnv(with_metaclass(Singleton, object)):

I'm wondering if the use of that Singleton metaclass is causing problems

It can't import platform_client because platform_client doesn't even exist in the container!
bash-4.3# ls -l /ax/src/ax
total 68
-rw-r--r--    1 root     root             0 Oct  4 01:07 __init__.py
drwxr-xr-x    2 root     root          4096 Oct  4 20:17 __pycache__
-rw-r--r--    1 root     root            85 Oct  4 01:07 _version.py
drwxr-xr-x    1 root     root          4096 Oct  6 00:04 aws
drwxr-xr-x    2 root     root          4096 Oct  4 01:07 axdb
drwxr-xr-x    2 root     root          4096 Oct  4 01:07 axevent
drwxr-xr-x    2 root     root          4096 Oct  4 01:07 cadvisor
drwxr-xr-x    1 root     root          4096 Oct  4 20:17 cloud
drwxr-xr-x    3 root     root          4096 Oct  4 01:07 cluster_management
drwxr-xr-x    1 root     root          4096 Oct  4 20:17 devops
-rw-r--r--    1 root     root          4863 Oct  4 01:07 exceptions.py
drwxr-xr-x    3 root     root          4096 Oct  4 01:07 kubernetes
drwxr-xr-x    2 root     root          4096 Oct  4 01:07 meta
-rw-r--r--    1 root     root          3533 Oct  4 01:07 notification_center.py
drwxr-xr-x    3 root     root          4096 Oct  4 01:07 test
drwxr-xr-x    1 root     root          4096 Oct  4 20:17 util
-rw-r--r--    1 root     root           950 Oct  4 01:07 version.py

This is a packaging issue with how the container image was built.

FILE: build/ax/build/devops.py
       if not re.search(‘devopsbuilder’, container_build_path):
            # DevOps source code is not protected (for the time being). Platform code is.
            # Do not copy platform code into devops containers to maintain this protection
            # and prevent exposure of our platform source to customers.
            shutil.rmtree(“{}/src/ax/platform”.format(docker_build_dir))
            shutil.rmtree(“{}/src/ax/platform_client”.format(docker_build_dir))

So.. the above code is in the v1.0 branch that explicitly avoids copying the platform and platform_client dirs in the axartifactmanager container. That causes the problem.
The above code was removed as part of commit 5a0c84af62c521c75e4bed1c353252d347c09711 on master. Therefore it works on master.
@sytianhe , can you cherry-pick the above commit (or the relevant parts of it) into the v1.0 branch?

That commit is about changing the devops builder to use platform builder. I don't see why it doesn't work without the commit as we have been using the devops builder all along.

authors: jessesuen pratulw shrinandj sytianhe


text:
rename output to value

@PJ-Finlay To fix the tiny issue here :raised_hands:

Woops just fixed it: https://github.com/argosopentech/argos-translate/commit/2f0bd0497baa75451dabb65f30a4e9c1636c274a

Great! :grin: :speedboat:
closing the PR then :pray: :+1:

authors: PJ-Finlay mmokhi

repo: argosopentech/argos-translate


text:
🛑 ideasoft test site is down

In be8bf30, ideasoft test site (https://ops.ideasoft.dev) was down:

HTTP code: 0
Response time: 0 ms

Resolved: ideasoft test site is back up in 4012b14 after 31 minutes.

repo: aribatturan/hcdev


text:
Suggestion - remove capitalizing github projects.

I think that this is pretty annoying, that all github projects names are being capitalised, and I hope this will change in the future

I will look into this.

I would like to solve this issue. Kindly assign this task to me

authors: Bamboooz arifszn hkrvikrant

repo: arifszn/gitprofile