text:
Swagger2/Open API Swagger UI support request

Framework version: XX
Implementations: Jersey / Spring / Spring Boot / Spark

Spring Boot
Scenario
I am interested in having Swagger2 support for Swagger UI with Sam local running Lambda SpringBoot.
https://springframework.guru/spring-boot-restful-api-documentation-with-swagger-2/
I tried to add the Swagger UI, however there is a conflict with SAM local security and the server times out, because of security authentication that fails.
It appears that the SAM Local security conflicts with the Swagger2 UI security being disabled, and the SAM Local security proxy wins, therefore disabling Swagger2 UI.
Expected behavior
This example shows the Swagger UI being enabled in this Spring Boot project.
1.)
https://github.com/quantum-fusion/springboot_swagger_example-master-cassandra
Actual behavior
This Lambda example shows that the Swagger UI is disabled, in this SpringBoot project.
2.)
https://github.com/quantum-fusion/aws-serverless-java-container/blob/master/samples/springboot/pet-store/README.md
Steps to reproduce
Run both springboot projects both 1.) and 2.)
Full log output

Hi, sorry for the delay, I lost this thread...
I'm in the process of publishing this now.  It's a little specific to my setup, but feel free to customize.  It will be at https://github.com/davidmco65/APIGatewaySwaggerUtils.
Thanks,
David

Let me see if I can get to the bottom of this. This library, and API Gateway + Lambda, are capable of returning static HTML content. Generating and returning a Swagger UI as an HTML page from the Lambda function should work fine.
Generating Swagger from the code is a very interesting idea. However, as @davidmco65 mentioned, it is a task better performed at compile time. It is not a core feature I'd want to build into the runtime framework. Perhaps we have a case here for building a CLI utility for this framework.
Can you tell me a little bit more about what the issue is with SAM local security? I can look into the SAM local code to find the issue if you can elaborate on this @quantum-fusion: "The security mechanism that is used with Swagger I found was conflicting with the Sam local proxy security."

I will jump on this, and get you a test case to reproduce the failure case.

The use case, is like this, Spring Boot applications can support the Swagger2 maven plugin
When the Swagger2 plugin is integrated within a SpringBoot project, that uses Lambda, the Swagger2 plugin that generates HTML, has a timeout with SAM local, due to a security authorization that times out with the SAM local proxy security.

ok. So when you build a project with the Swagger2 maven plugin it packages correctly. However, when you try to run it in SAM local, the application does not respond?
What I want to clarify is, the plugin works at compile time and builds successfully, the issue happens when you try and load the page. Is that only through SAM local or also on Lambda + API GW?

Yes, the project compiles correctly.
The problem is found at runtime, when the swagger ui plugin is called to generate the page.
The page is not accessible, due to spring security requiring a login token
I have found noticeable differences between your Lambda Spring Boot example, and mine, the main difference is that the spring security token in your project is missing and is not shown in the console logs.
The issue happens when you try to load the page, and the security token is required for login and password by spring security.  This does not normally occur in a project without Lambda with Spring Boot standalone.  The reason is that Swagger over rides the security, and allows all incoming requests to be authorized without a login token from spring security.
I believe that Sam Local shows that the service times out, with the AWS proxy, because of the need for a spring security token.  I do not know why it thinks it wants a login name and password for spring security, it appears that the over ride for the swagger plugin is disabled when integrated with SAM local and the proxy. I do not know why it does this.

I can recreate this scenario, or allow you to do the integration yourself.  Let me know if you want me to reproduce, or if you want to go through the integration steps to see for your self.

If you have a repo with the Swagger plugin enabled that I just grab, package, and test with SAM local send me the link - it would save me a lot of time and ensure that I replicate exactly what you are doing

Yes, I will do this, and give it to you after lunch in about an hour.
I would ask that you do a thorough code review, because there may be a couple things that I am missing.

If you want to see Swagger + SpringBoot in action, in the meanwhile, try this project:
It will look and feel exactly the same, if everything is working properly.
https://github.com/quantum-fusion/springboot_swagger_example-master-cassandra
The integrated version with Lambda will show up in a few hours, stay tuned:
https://github.com/quantum-fusion/swagger-lambda

Github push today was a complete train wreck, and wouldn't let me upload a directory to GitHub without lots of pain.
Therefore download the swagger-lambda.tar file from Dropbox.
https://www.dropbox.com/s/ms4wzqjt3z63ob8/swagger-lambda.tar?dl=0
The URL calls were as follows:
http://127.0.0.1:3000/v2/api-docs
http://127.0.0.1:3000/petstore/pets
http://127.0.0.1:3000/pets
Analysis:
The first and last URL was not recognized, see exceptions that were hit.
The second URL had Function returns an invalid response, possibly due to the sam.yaml , not including the full updated path, with the added petstore path qualifier.
This is the trace information from the console:
sam local start-api --template sam.yaml
A newer version of the AWS SAM CLI is available!
Your version:   0.2.4
Latest version: 0.2.6
See https://github.com/awslabs/aws-sam-local for upgrade instructions
2018/02/01 13:50:29 Connected to Docker 1.35
2018/02/01 13:50:29 Fetching lambci/lambda:java8 image for java8 runtime...
java8: Pulling from lambci/lambda
5be106c3813f: Pull complete
e240967675e1: Pull complete
b1666cffee1b: Pull complete
7bf690717622: Pull complete
Digest: sha256:7c5ab3fa13154c56d66c45de06a304139fe544dff502ee1bc68c47eeab79240c
Status: Downloaded newer image for lambci/lambda:java8
Mounting com.server.springboot.StreamLambdaHandler::handleRequest (java8) at http://127.0.0.1:3000/{proxy+} [OPTIONS GET HEAD POST PUT DELETE PATCH]
You can now browse to the above endpoints to invoke your functions.
You do not need to restart/reload SAM CLI while working on your functions,
changes will be reflected instantly/automatically. You only need to restart
SAM CLI if you update your AWS SAM template.
2018/02/01 13:52:25 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/01 13:52:25 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/01 13:52:38 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517511145483025000 as /var/task:ro inside runtime container
START RequestId: 436d034e-44fe-4cd8-bacd-094cda1a751c Version: $LATEST
javax.servlet.ServletException: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:91)
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyRequestDispatcher.forward(AwsProxyRequestDispatcher.java:64)
at org.springframework.boot.web.support.ErrorPageFilter.forwardToErrorPage(ErrorPageFilter.java:183)
at org.springframework.boot.web.support.ErrorPageFilter.handleException(ErrorPageFilter.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at com.amazonaws.serverless.proxy.internal.servlet.FilterChainHolder.doFilter(FilterChainHolder.java:85)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.doFilter(AwsLambdaServletContainerHandler.java:191)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:145)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:139)
at com.server.springboot.StreamLambdaHandler.handleRequest(StreamLambdaHandler.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:350)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:888)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:281)
at lambdainternal.AWSLambda.(AWSLambda.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
Caused by: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:88)
... 25 more
END RequestId: 436d034e-44fe-4cd8-bacd-094cda1a751c
REPORT RequestId: 436d034e-44fe-4cd8-bacd-094cda1a751c	Duration: 22091.41 ms	Billed Duration: 22100 ms	Memory Size: 1512 MB	Max Memory Used: 18 MB
2018/02/01 13:53:12 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/01 13:53:12 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/01 13:53:23 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517511192143388000 as /var/task:ro inside runtime container
START RequestId: c78cdec5-7670-49c3-9b40-e0fd7b04f37e Version: $LATEST
2018/02/01 13:53:44 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/01 13:53:44 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/01 13:53:47 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/01 13:53:47 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
END RequestId: c78cdec5-7670-49c3-9b40-e0fd7b04f37e
REPORT RequestId: c78cdec5-7670-49c3-9b40-e0fd7b04f37e	Duration: 21706.46 ms	Billed Duration: 21800 ms	Memory Size: 1512 MB	Max Memory Used: 18 MB
2018/02/01 13:54:09 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517511224063183000 as /var/task:ro inside runtime container
2018/02/01 13:54:11 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517511227284211000 as /var/task:ro inside runtime container
START RequestId: 40309963-b9d5-4a85-82f9-3473fb134a55 Version: $LATEST
START RequestId: 3d2ca59a-59c2-483d-b981-3d7b277e5e58 Version: $LATEST
END RequestId: 40309963-b9d5-4a85-82f9-3473fb134a55
REPORT RequestId: 40309963-b9d5-4a85-82f9-3473fb134a55	Duration: 33276.18 ms	Billed Duration: 33300 ms	Memory Size: 1512 MB	Max Memory Used: 18 MB
2018/02/01 13:54:45 Function returned an invalid response (must include one of: body, headers or statusCode in the response object): invalid character '-' after top-level value
2018/02/01 13:55:09 Function com.server.springboot.StreamLambdaHandler::handleRequest timed out after 60 seconds
2018/02/01 13:56:52 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/01 13:56:52 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/01 13:57:01 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517511412241522000 as /var/task:ro inside runtime container
START RequestId: 48705a57-a4f4-4bbb-8b88-b50c1fa51d89 Version: $LATEST
javax.servlet.ServletException: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:91)
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyRequestDispatcher.forward(AwsProxyRequestDispatcher.java:64)
at org.springframework.boot.web.support.ErrorPageFilter.forwardToErrorPage(ErrorPageFilter.java:183)
at org.springframework.boot.web.support.ErrorPageFilter.handleException(ErrorPageFilter.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at com.amazonaws.serverless.proxy.internal.servlet.FilterChainHolder.doFilter(FilterChainHolder.java:85)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.doFilter(AwsLambdaServletContainerHandler.java:191)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:145)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:139)
at com.server.springboot.StreamLambdaHandler.handleRequest(StreamLambdaHandler.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:350)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:888)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:281)
at lambdainternal.AWSLambda.(AWSLambda.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
Caused by: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:88)
... 25 more
END RequestId: 48705a57-a4f4-4bbb-8b88-b50c1fa51d89
REPORT RequestId: 48705a57-a4f4-4bbb-8b88-b50c1fa51d89	Duration: 21699.99 ms	Billed Duration: 21700 ms	Memory Size: 1512 MB	Max Memory Used: 18 MB

see line 24 of sam.yaml , it does not include the reference to the base path /petstore that is referenced in the petstore controller file.
I added the reference to /petstore as the base path, so that swagger can generate docs based on a base path for each controller.
Please let me know if sam.yaml file needs to be updated to include a base path qualifier, and if so, what the syntax should look like.

@SAPessi
@davidmco65
See these comments and the project on dropbox.
Please let me know when you have reviewed the code, and comments.
Do let me know if you have any additional questions.

When the swagger UI is running correctly, this URL should generate the HTML and run in the web browser.
http://127.0.0.1:3000/swagger-ui.html

See Tutorial on SpringBoot and Swagger API.
https://springframework.guru/spring-boot-restful-api-documentation-with-swagger-2/

See error generated from SAM local gateway.
When URL was called http://127.0.0.1:3000/petstore/pets
2018/02/01 13:54:45 Function returned an invalid response (must include one of: body, headers or statusCode in the response object): invalid character '-' after top-level value
2018/02/01 13:55:09 Function com.server.springboot.StreamLambdaHandler::handleRequest timed out after 60 seconds

My theory is that this may have generated the error Function returned an invalid response, due to adding the /petstore base path in the petstore controller, but not updating the sam.yaml file line 24 to include the /petstore base path.
Please advise.

I do not know why the swagger functions generate the exceptions.

@SAPessi This is the completed writeup and tar ball you were looking for, that should save you some time and work.  I have recreated the problem, please let me know when you have taken a look at this. Thanks @quantum-fusion

Safari URL: localhost:3000/petstore/pets
Safari response: {"message":"Gateway timeout"}
Theory: It is a possibility that sam.yaml needs to be changed from line24:
Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/pets'
TO:
Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/petstore/pets'
Results:
Showed that adding the /petstore base path did not work.
See console results BEFORE change to adding /petstore:
$ sam local start-api --template sam.yaml
A newer version of the AWS SAM CLI is available!
Your version:   0.2.4
Latest version: 0.2.6
See https://github.com/awslabs/aws-sam-local for upgrade instructions
2018/02/02 06:39:06 Connected to Docker 1.35
2018/02/02 06:39:06 Fetching lambci/lambda:java8 image for java8 runtime...
java8: Pulling from lambci/lambda
Digest: sha256:7c5ab3fa13154c56d66c45de06a304139fe544dff502ee1bc68c47eeab79240c
Status: Image is up to date for lambci/lambda:java8
Mounting com.server.springboot.StreamLambdaHandler::handleRequest (java8) at http://127.0.0.1:3000/{proxy+} [OPTIONS GET HEAD POST PUT DELETE PATCH]
You can now browse to the above endpoints to invoke your functions.
You do not need to restart/reload SAM CLI while working on your functions,
changes will be reflected instantly/automatically. You only need to restart
SAM CLI if you update your AWS SAM template.
2018/02/02 06:39:29 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/02 06:39:29 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/02 06:39:39 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517571569191686000 as /var/task:ro inside runtime container
START RequestId: 7bb28b2f-5e7e-4b24-98d0-c8ee1d9bbb65 Version: $LATEST
javax.servlet.ServletException: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:91)
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyRequestDispatcher.forward(AwsProxyRequestDispatcher.java:64)
at org.springframework.boot.web.support.ErrorPageFilter.forwardToErrorPage(ErrorPageFilter.java:183)
at org.springframework.boot.web.support.ErrorPageFilter.handleException(ErrorPageFilter.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at com.amazonaws.serverless.proxy.internal.servlet.FilterChainHolder.doFilter(FilterChainHolder.java:85)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.doFilter(AwsLambdaServletContainerHandler.java:191)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:145)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:139)
at com.server.springboot.StreamLambdaHandler.handleRequest(StreamLambdaHandler.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:350)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:888)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:281)
at lambdainternal.AWSLambda.(AWSLambda.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
Caused by: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:88)
... 25 more
END RequestId: 7bb28b2f-5e7e-4b24-98d0-c8ee1d9bbb65
REPORT RequestId: 7bb28b2f-5e7e-4b24-98d0-c8ee1d9bbb65	Duration: 16962.89 ms	Billed Duration: 17000 ms	Memory Size: 1512 MB	Max Memory Used: 18 MB
2018/02/02 06:40:07 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/02 06:40:07 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/02 06:40:13 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517571607394102000 as /var/task:ro inside runtime container
START RequestId: 8e8bb8b2-12a7-46a0-a479-b008973f5393 Version: $LATEST
See results AFTER adding /petstore base path:
sam local start-api --template sam.yaml
A newer version of the AWS SAM CLI is available!
Your version:   0.2.4
Latest version: 0.2.6
See https://github.com/awslabs/aws-sam-local for upgrade instructions
2018/02/02 06:43:47 Connected to Docker 1.35
2018/02/02 06:43:47 Fetching lambci/lambda:java8 image for java8 runtime...
java8: Pulling from lambci/lambda
Digest: sha256:7c5ab3fa13154c56d66c45de06a304139fe544dff502ee1bc68c47eeab79240c
Status: Image is up to date for lambci/lambda:java8
Mounting com.server.springboot.StreamLambdaHandler::handleRequest (java8) at http://127.0.0.1:3000/{proxy+} [OPTIONS GET HEAD POST PUT DELETE PATCH]
You can now browse to the above endpoints to invoke your functions.
You do not need to restart/reload SAM CLI while working on your functions,
changes will be reflected instantly/automatically. You only need to restart
SAM CLI if you update your AWS SAM template.
2018/02/02 06:43:57 Invoking com.server.springboot.StreamLambdaHandler::handleRequest (java8)
2018/02/02 06:43:57 Decompressing /Users/hottelet/Desktop/swagger-lambda/samples/springboot/pet-store/target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
2018/02/02 06:44:03 Mounting /private/var/folders/60/l9x5s_456sx9nspbd1l90_fm0000gn/T/aws-sam-local-1517571837480165000 as /var/task:ro inside runtime container
START RequestId: 168a0a23-0739-4011-8f41-41676cbf82b0 Version: $LATEST
javax.servlet.ServletException: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:91)
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyRequestDispatcher.forward(AwsProxyRequestDispatcher.java:64)
at org.springframework.boot.web.support.ErrorPageFilter.forwardToErrorPage(ErrorPageFilter.java:183)
at org.springframework.boot.web.support.ErrorPageFilter.handleException(ErrorPageFilter.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:130)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at com.amazonaws.serverless.proxy.internal.servlet.FilterChainHolder.doFilter(FilterChainHolder.java:85)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.doFilter(AwsLambdaServletContainerHandler.java:191)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:145)
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:139)
at com.server.springboot.StreamLambdaHandler.handleRequest(StreamLambdaHandler.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:350)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:888)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:281)
at lambdainternal.AWSLambda.(AWSLambda.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94)
Caused by: java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse
at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:52)
at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.forward(AwsLambdaServletContainerHandler.java:88)
... 25 more
END RequestId: 168a0a23-0739-4011-8f41-41676cbf82b0
REPORT RequestId: 168a0a23-0739-4011-8f41-41676cbf82b0	Duration: 16227.89 ms	Billed Duration: 16300 ms	Memory Size: 1512 MB	Max Memory Used: 18 MB

@SAPessi Please let me know when you have questions, and you have received the tar ball file test case for SpringBoot with Lambda with Swagger, without error.

I'm testing the application. Few issues:

I fixed an issue in serverless-java-container where we weren't handling response wrappers correctly. The root cause of this exception:

java.lang.ClassCastException: org.springframework.boot.web.support.ErrorPageFilter$ErrorWrapperResponse cannot be cast to com.amazonaws.serverless.proxy.internal.servlet.AwsHttpServletResponse

An exception is being thrown deep in your application, but I cannot get it to print it out in the logs. Still working on figuring this one out

Your JsonHelper is wildly inefficient, you are instantiating an ObjectMapper at each call. Have a single object mapper as a class member, lazily load it, and re-use it each time. Even better, if you don't need any special configuration, use the default mapper provided by the framework: LambdaContainerHandler.getObjectMapper()

Another issue is that spring security is trying to access the session. We intentionally throw an UnsupportedOperationException when you try to access the session - you should not rely on session when coding with Lambda. You can disable session processing:
@EnableWebSecurity
public class TestSiteSecurity extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.sessionManagement().disable();
     }
}

3.) Your JsonHelper is wildly inefficient, you are instantiating an ObjectMapper at each call. Have a single object mapper as a class member, lazily load it, and re-use it each time. Even better, if you don't need any special configuration, use the default mapper provided by the framework: LambdaContainerHandler.getObjectMapper()
Are you saying that Jackson parser is not needed?  I was able to get rid of Jackson parser, by having properly formatted REST API calls using Swagger API.  Note that incoming messages are properly formatted as POST or GET objects by expecting the data model as inputs.
Note: this will not always be the case, in the instance of custom formatted messages, or dynamic schema.

serverless-java-container already includes jackson as a dependency, you can just rely on the included one. I'm still debugging. I'll send you the diff to the app's code here once I'm done with all the issues

I would be happy to run a benchmark to see if the swagger UI is working properly, and also download a new build to test locally.

@SAPessi
I would need the fixes to the serverless code in the main branch.
I would need the modified code with changes provided back to me, so that I can run my own diffs.
How would you like to provide both builds or a single build?

I'm still working on root-causing the issue. Spring boot is painful to run with SAM local. SAM local reads the output of the function from the stdout. However, spring boot fills the stdout with its logging messages. It will take me a while to get to the bottom of this. I'm working on the spring branch of serverless-java-container. I will commit my changes once I've got it working.

I am rooting for you!  Let me know if you need to do a shared screen session, or conference call.  https://calendly.com/technologyventureslimited (Schedule appointment)
I am supposed to have an interview with AWS Virginia next week.

Let me know what you think about the added @RequestMapping for ("/petstore")
--PetsController.java----
@EnableWebMvc
@RestController
@RequestMapping("/petstore")
@ComponentScan("com.server.springboot.controller")
public class PetsController {
@RequestMapping(path = "/pets", method = RequestMethod.POST)
public Pet createPet(@RequestBody Pet newPet) {
if (newPet.getName() == null || newPet.getBreed() == null) {
return null;
}
    Pet dbPet = newPet;
    dbPet.setId(UUID.randomUUID().toString());
    return dbPet;
}

@RequestMapping(path = "/pets", method = RequestMethod.GET)
public Pet[] listPets(@RequestParam("limit") Optional<Integer> limit, Principal principal) {

I also added the /petstore base path to the sam.yaml file.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Example Pet Store API written with SpringBoot with the aws-serverless-java-container library
Resources:
PetStoreFunction:
Type: AWS::Serverless::Function
Properties:
Handler: com.server.springboot.StreamLambdaHandler::handleRequest
Runtime: java8
CodeUri: target/serverless-spring-boot-example-1.0-SNAPSHOT.jar
MemorySize: 1512
Policies: AWSLambdaBasicExecutionRole
Timeout: 60
Events:
GetResource:
Type: Api
Properties:
Path: /{proxy+}
Method: any
Outputs:
SpringBootPetStoreApi:
Description: URL for application
Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/petstore/pets'
Export:
Name: SpringBootPetStoreApi

ok, I've figured it out. The plugin you are using to generate the Swagger UI runs at compile time to generate the swagger-ui.html page. By default, the page would be included in the application's jar by the spring boot maven plugin in the repackage goal - I can see the file under META-INF/resources in the repackaged jar. However, the repackaged jar is not compatible with AWS Lambda. This is not an issue with serverless Java container so I will have to de-prioritize it, I don't have any more time for this today.
All other issues are fixed in the latest commit to the spring branch of this repo.
Latest source code for the app attached here:
pet-store.zip
In the sample app I have disabled Spring security because it was just getting in the way during testing, there were no issues with it.

I will take a look at the updated code, and after I have reviewed the code, if there are bugs with SpringBoot , we may have to get @philwebb involved.

@SAPessi

This is not an issue with serverless Java container so I will have to de-prioritize it, I don't have any more time for this today.
Question, is the serverless updates in the latest build, and available for download?  Please advise.

@quantum-fusion no, the commits are on the spring branch of this repo. You should be able to just clone the branch in local, then run mvn install. Once that process installs 0.1-SNAPSHOT on your local machine, you can mvn package the pet store code I sent you, it will find the latest version on your local box automatically.

The petstore pom.xml for the Swagger implementation is expecting:

org.springframework.boot
spring-boot-maven-plugin

The example that you provided with the maven shade plugin pets.zip , has substantial differences as compared to the original swagger example. (https://springframework.guru/spring-boot-restful-api-documentation-with-swagger-2/ )

If you wanted a test case, you could use the one here: https://github.com/springframeworkguru/springboot_swagger_example

The repackage goal, there isn't one.
This is the build goal:

org.springframework.boot
spring-boot-maven-plugin

The spring-boot-maven-plugin has to be used. as shown here: https://springframework.guru/spring-boot-restful-api-documentation-with-swagger-2/

A lot of the dependencies you were including were already provided by other packages in serverless-java-container, you didn't need them there. The problem you have is with how the swagger package works. As you can see this dependency is included at compile time:
<dependency>
 <groupId>io.springfox</groupId>
 <artifactId>springfox-swagger-ui</artifactId>
 <version>2.6.1</version>
 <scope>compile</scope>
</dependency

It generates an HTML page at compile time and then the page gets packaged into the jar by the spring boot (see pom file). The jar generated by the spring boot maven plugin is not compatible with Lambda, the handler class is not included. We need to figure out a way to run the swagger ui generation at compile time and still include it in the Lambda jar.

I do not know the answer to this.

I know it is dissappointing, however I am wondering if we might get Phil Webb interested in this?

Neither do I, it would require a lot of deep diving on how spring boot build and packaging works. I will put it on my backlog but won't have time any time soon.

I was hoping we could just use shade plugin, even with the springfox-swagger-ui plugin enabled, with compile as the option, the documentation did not generate, when I enabled it again.
See https://stackoverflow.com/questions/26975818/what-is-scope-under-dependency-in-pom-xml-for

What about other scope options, like provided?

There is a system scope.

If anyone in the world, knows the answer to this question, it is Stéphane Nicoll

Should we approach him?

We could explain what we are trying to do, and ask him the question, he will respond.

https://spring.io/team/snicoll

This is the official issues for Spring-Boot https://github.com/spring-projects/spring-boot/issues

Is there some reason, why you need a shaded maven jar, and not a spring boot jar?  I know the maven shaded jar, is flattened, to include all of the dependencies, however the spring boot jar, has an automatic spring bean detector, called spring bean auto discovery?

I don't think this is an "issue" with spring boot. The repackage goal of spring boot was never meant to create a jar that could run inside Lambda. My suggestion would be to take a look at the swagger-ui source and figure out how it executes - once you have that information, you have two options:

you can start trying to run it as part of your build for Lambda in the shade plugin
Find a way to include the Lambda handler in the jar generated by spring boot and see if you can make it work in Lambda

Currently, I am trying to figure out why running swagger-ui as a standalone with the spring-boot build, isn't working.  There is still something fundamental missing.  The compile goal for the dependencies, is just how it packages the binaries.  I could have just deleted the  flags, and see what that does.  However, I believe those dependencies are being explicit.
I did notice the following when doing experiments with package options either maven shade plugin vs. spring boot plugin:

spring boot with the spring boot package option, will not run SAM local, due to an incompatibility.
I also noticed, when I tried to get swagger running with your package options included in petstore.zip that swagger would not turn back on, even after turning off the disable feature for httpSecurity.sessionManagement().disable();

I would have to further investigate these issues, and have been busy going on job interviews for the time being.

@SAPessi I have found the SpringFox guys.  We could submit the question to them, and ask them if they can support Maven shade plugin.  (https://github.com/springfox/springfox/issues )

Why don't you first go on their Gitter chat and see if they have done it before outside of spring boot?

I just posted a question for the spring fox guys. (https://github.com/springfox/springfox/issues/2227 )
I will have to pickup this thread on Monday, and I will let you know what I find.

@SAPessi I found the Springfox documentation section 3.5 http://springfox.github.io/springfox/docs/current/#springfox-swagger-ui
The documentation states, that it is looking for the endpoint path by pulling in the dependency creates a webjar containing the swagger-ui static content. It adds a JSON endpoint /swagger-resources which lists all of the swagger resources and versions configured for a given application. The Swagger UI page should then be available at http://localhost:8080/swagger-ui.html
The spring fox owner @dilipkrish responded with "Not sure what would need to happen here in this library. Plus shading the library isn't really doing to buy you much because all the spring/jackson dependencies are provided dependencies except for a select few."

@SAPessi I am going to suggest that we start over with a fresh SpringBoot project that uses Swagger 2 fox plugin for maven, with the SpringBoot maven plugin.
See full example:
https://springframework.guru/spring-boot-restful-api-documentation-with-swagger-2/
See Github repo:
https://github.com/springframeworkguru/springboot_swagger_example
It is a possibility that AWS Lambda framework may support this project, however, the feasibility question remains, can you support the Spring Boot maven plugin, instead of the shade plugin for the AWS Lambda serverless frameworks?

The serverless java container library is capable of serving static content via Spring: See this unit test.
How the static content is included (or generated) is not a concern of the library. You don't have to use the shade plugin. There is nothing that ties this library to it.
As long as you package your code in a way that Lambda can understand, it should work. The serverless-java-container library does not (and should not) control your build/package process.

Fixes for the bugs discovered while investigating this are now merged in master. Closing.

Have the same issue (/swagger-ui.html returns Gateway timeout). Did you @quantum-fusion solve somehow?

authors: SAPessi davidmco65 ianaz quantum-fusion

repo: awslabs/aws-serverless-java-container


text:
MXNet Nightly Builds Moved to S3

Hi,
MXNet nightly builds have been moved from PyPI to S3 (https://github.com/pypa/pypi-support/issues/243). If this project requires the nightly builds for CI validation, please make sure that your CI pipeline now takes the nightly builds from https://dist.mxnet.io/python. Furthermore, as MXNet 1.7 and 2.0 are currently in progress simultaneously, we are releasing nightly builds for both 1.x and 2.x. Take mxnet-cu102 for example:
If you need the nightly builds for 1.x:
pip install --pre "mxnet-cu102<2" -f https://dist.mxnet.io/python

If you need the nightly builds for 2.x:
pip install --pre "mxnet-cu102>=2" -f https://dist.mxnet.io/python

Feel free to close this issue if the necessary changes have been made, or If the project doesn't rely on MXNet nightly builds. Thanks.

cc @lostella

Thanks, @shza!

Hi team. Is the related infra change ready?

@szha #801 should set us up for mxnet<2, we will maybe add a workflow to test also on mxnet>= 2

source: gharchive/issue

authors: jaheba lostella szha

repo: awslabs/gluon-ts

license_type: permissive

license_source: github-api


text:
Unused variables in hgt_nc.py

Hi, I found the following when playing with the example codes:

In file examples/customized_models/HGT/hgt_nc.py, the final linear projection self.out is not used. Might need to add self.out(h[self.target_ntype]) to h[self.target_ntype] before the loss function

The self.configs.lr does not affect the learning rate of an optimizer. Instead, it was controlled by the default variable lr in create_optimizer(self, lr=0.001)

Has been fixed in the PR #179.

repo: awslabs/graphstorm


text:
ReadError: not a gzip file

I was trying to get the 4 testing images following the notebook, but when downloading the IAM images it says "ReadError: not a gzip file.  Could anyone kindly help here ?

Hi please check https://github.com/awslabs/handwritten-text-recognition-for-apache-mxnet/issues/52

authors: TRokieG jonomon


text:
A better literate documentation mechanism

The literate documentation system we have today (where we include a piece of a file in a subdirectory) is not good enough. It requires a lot of overhead, making the additional file for every example you want to include and then referencing it. The overhead makes people skip it for small one-line examples, which makes it miss the point (because the one-line examples inevitably go out of date). This issue describes mechanism which will both lower the bar to entry, and make it impossible to write uncompiled code samples.
I will describe it using an example, which I think will be more clear than a lot of prose:
Current mechanism

test/integ.some-example.lit.ts

import mylib = require('../lib');

class SomeStack extends cdk.Stack {
  constructor(...) {
     super(this, id);

     ///! show
     const myconstruct = new mylib.Construct(this, 'hello');
     // relevant example goes here
    /// !hide
  }
}

const app = new cdk.App();
new SomeStack(app, 'mystack');
app.run();

README.md

Welcome to the readme for my library!

Here is how you use it:

[example of using my library](test/integ.some-example.lit.ts)

As you can see above, ...

Proposed mechanism

test/docfixture.default.tpl

import mylib = require('../lib');

class SomeStack extends cdk.Stack {
  constructor(...) {
     super(this, id);

     /* HERE */
  }
}

const app = new cdk.App();
new SomeStack(app, 'mystack');
app.run();

README.md

Welcome to the readme for my library!

Here is how you use it:

` ` `ts
const myconstruct = new mylib.Construct(this, 'hello');
// relevant example goes here
` ` `

If you want to integrate with another resource:

` ` `ts
/// <prereq>
// This won't be displayed in the README but it will be compiled.
const other = new otherlib.Resource();
/// </prereq>

const myconstruct = new mylib.Construct(this, 'hello');
myconstruct.connectTo(other);
` ` `

Bottom line

We will insert all MarkDown blocks fenced by ```ts into a template and the resulting code must compile.
Additional instructions (using a different template than the default, extra lines of code just to make this example work) can be given via special comment syntax, which also won't end up in the final code. Proposal: ///.
It is not allowed to have code blocks in a README and NOT have a fixture file to compile them against. That is: all code must be compile-checked.
Implemented by jsii.

Extensions:
We should probably extend the same courtesies to code blocks encountered in doc comments.

I am wondering if it would be nicer to simple compile the readme.md file during build (and check in the result), so we can have the readme completely self contained (with fixtures and all).

Will we add the compile step to checkin? I.e., one cannot check in a file with a code example that does not compile?

I am wondering if it would be nicer to simple compile the readme.md file during build (and check in the result), so we can have the readme completely self contained (with fixtures and all).

I am not sure what you mean by "compiling the readme", or what checking in the result means, but I did plan on applying the mechanism describe above during the jsii build.

What I meant is that maybe there's a README.md.src that includes both markdown and also code blocks with directives (//!) to the renderer:
Let's define our first CDK stack.

'''sh
//!exec
//!hide
npm i @aws-cdk/cdk, @aws-cdk/aws-sqs
'''

Create a file `stack.ts`:

'''ts
//!new-file:stack.ts
import { Stack } from '@aws-cdk/cdk';

export class MyStack extends Stack { }
'''

Now, create a file `app.ts`:

'''ts
//!new-file:app.ts
import { App } from '@aws-cdk/cdk';
import { MyStack } from './stack';

const app = new App();
new MyStack(app, 'MyFirstStack');
'''

'''sh
//!exec
//!hide
tsc
'''

Now, let's add a resource to our stack:

'''ts
//!edit-file:stack.ts
//!delete-lines:4-4
export class MyStack extends Stack {
  constructor(..) {
    super();
    new sqs.Queue(this, 'MyQueue');
  }
}
'''

YMMV of course... Just made up those directives...
Then, we have a little renderer that reads this README.md.src, executes all the directives and produces a README.md file as a result without directives or hidden stuff.
The nice thing here is that we can do pretty crazy things and generate portions of the readme, markdown, common headers and footers, etc.
We can decide if we check in the READM.md file. Not required. Eventually the important thing is that the README in npmjs, pypi and our docs website is the rendered version.

At first glance, I hate this (all the annotation work!) and I shudder to think about the security implications of letting people run arbitrary shell commands on our boxes just by submitting a PR :).
But those are details, and it occurs to me that you're trying to solve a slightly different problem than I am.
I'm interested in having code snippets appear in the docs syntax-checked. The snippets just appear by themselves in either the README or the doc blocks of methods and classes. That really doesn't need much more than a project template and some holes. It doesn't need sequencing, it doesn't need file editing, it doesn't need arbitrary package installs.
You seem to be solving the problem of a walkthrough (a la the getting started guide, or the workshop site), where we're taking a user by the hand, telling them "do this now, do that now, run this command, modify this file you created previously to add these lines, etc.", and having that validated to make sure it's still correct.
That problem is more generic, and will need more flexibility. We can probably combine the two in some way.

I don't README.md.src is necessary.
If we did that, we wouldn't have a README visible on GitHub, which is not great, or we would have to commit the generated README.md (also not great).
That decisions doesn't need to impact what gets published to NPM or PyPI, those markdowns could be post-processed, loaded from the JSII Assembly.

While @eladb's solutions strikes me as more flexible, it also feels a little over-complicated to me. I also worry about it being kind of brittle (edit commands for example, are quite brittle).

authors: Doug-AWS RomainMuller eladb rix0rrr

repo: awslabs/jsii


text:
URL is 404

What is the URL of the document?
https://github.com/awslabs/kubeflow-manifests/blob/main/docs/deployment/cognito/README.md#40-building-manifests-and-deploying-kubeflow
Which section(s) is the issue in?
Can't open the URL which is a part of instruction of deploying kubeflow with Conginto
from this page: https://github.com/awslabs/kubeflow-manifests/blob/main/docs/deployment/cognito/README-automated.md
What needs fixing and describe the solution you'd like?
Fix it with right URL.
Additional context

We are working on the .md files in main branch
For now please use :

website: https://awslabs.github.io/kubeflow-manifests/docs/deployment/cognito/guide/
or
The release branch : https://github.com/awslabs/kubeflow-manifests/tree/release-v1.4.1-aws-b1.0.0/docs/deployment/cognito

authors: akartsky narijeong

repo: awslabs/kubeflow-manifests


text:
Fix nightly clippy warning

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT license.

Thanks, I was puzzled about this on my other PR 😄 (I don't know if I can review things here, but I figured I could try)

Thanks! Looks like it worked.

source: gharchive/pull-request

repo: awslabs/mls-rs


text:
docs: Fix incorrect description for s2n_cleanup

See #649.

My mistake. Incorrect advice.


text:
Defining the ExplicitHashKey for a Kinesis Sink

The current configuration around the KPL allows us to define the PartitionKey for the Kinesis Sink. This can be seen in the documentation here. However, we are not able to define the ExplicitHashKey so that we can control to which shards specific rows are assigned. Currently, we would have to instead make sure a given rows PartitionKey mapped with an MD5 hash function generates the correct 128 integer within a given shard range. This process is more complex and ideally we'd like to have something as follows:
kinesis
  .selectExpr("CAST(rand() AS STRING) as partitionKey", "CAST(explicitHashKey AS STRING)", "CAST(data AS STRING)").as[(String,String,String)]
  .groupBy("data").count()
  .writeStream
  .format("aws-kinesis")
  .outputMode("append")
  .option("kinesis.region", "us-east-1")
  .option("kinesis.streamName", "sparkSinkTest")
  .option("kinesis.endpointUrl", "https://kinesis.us-east-1.amazonaws.com")
  .option("checkpointLocation", "/path/to/checkpoint")
  .start()

This would remove the step of having to make sure the partition key is in the correct shard range if we want to send certain rows to specific shards.

Changes merged to main branch.  Thanks for your contribution.

authors: hwanghw ttrankle

repo: awslabs/spark-sql-kinesis-connector


text:
SQS enabled not recognized as a property

Type: Bug
Component:
"Secrets Manager", "SQS"
Describe the bug
Without any addition configuration, starting the app will give
org.springframework.beans.factory.BeanDefinitionStoreException: Failed to process import candidates for configuration class [com.xxx.xxx.inboundmeasurementsapi.InboundMeasurementsApiApplication]: class path resource [io/awspring/cloud/sqs/config/SqsBootstrapConfiguration.class] cannot be opened because it does not exist when importing starter secrets manager.
Imports
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
...
    <properties>
        <java.version>17</java.version>
        <aws.sdk.version>2.17.280</aws.sdk.version>
        <spring.cloud.version>3.0.0-M3</spring.cloud.version>
    </properties>
...
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>io.awspring.cloud</groupId>
                <artifactId>spring-cloud-aws-dependencies</artifactId>
                <version>${spring.cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
...
        <dependency>
            <groupId>io.awspring.cloud</groupId>
            <artifactId>spring-cloud-aws-starter-secrets-manager</artifactId>
        </dependency>

Sample
Disabling sqs works, but the spring annotation configuration doesn't recognize enabled as a valid property for sqs, but it will for sns and others.
Perhaps the issue is that these are enabled by default even though I've only imported spring-cloud-aws-starter-secrets-manager
The workaround is to use the enabled flag anyway.
spring:
  cloud:
    aws:
      sqs:
        enabled: false

Without enabled: false it will attempt to autoconfigure the sqs despite not importing the cloud-aws modules for sqs

Hi @fireXtract, thanks for bringing this up.
Yeah, I think the problem is that we depend only on @ConditionalOnClass(SqsAsyncClient.class) to trigger autoconfiguration, and since the SDK contains that class it must be triggering it erroneously.
We should make it also conditional on a project's class such as SqsTemplate so that it doesn't trigger autoconfiguration when the SQS module is not used.
As for why the sqs.enabled configuration property is not showing as valid I'm not sure, but I can take a look too.
I should look into this later this week or early next.
Thanks again, and if you have anything else to add feel free to.

authors: fireXtract tomazfernandes

repo: awspring/spring-cloud-aws


text:
Cleanerjs crash on docker image v1.2.2 & v1.2.1

Summary
Send a PDF using default config crash module Cleaner.js
Steps To Reproduce

Send multipage PDF using curl
Check logs
See error

[2021-01-22T11:06:41] ERROR (parsr-api/6 on parsr-bb9d57c85-bxwmb): /opt/app-root/src/dist/src/Cleaner.js:201
            throw new Error("Module " + moduleClass.moduleName + " has unresolved dependencies (" + unresolvedStr + ").");
            ^

Error: Module lines-to-paragraph has unresolved dependencies (words-to-line-new).
    at Cleaner.checkDependenciesAndAdd (/opt/app-root/src/dist/src/Cleaner.js:201:19)
    at /opt/app-root/src/dist/src/Cleaner.js:136:19
    at Array.forEach (<anonymous>)
    at Cleaner.parseLatestConfig (/opt/app-root/src/dist/src/Cleaner.js:123:16)
    at new Cleaner (/opt/app-root/src/dist/src/Cleaner.js:88:18)
    at main (/opt/app-root/src/dist/bin/index.js:72:19)
    at Object.<anonymous> (/opt/app-root/src/dist/bin/index.js:204:1)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)

[2021-01-22T11:06:41] INFO  (parsr-api/6 on parsr-bb9d57c85-bxwmb): Process exited

Expected behavior
no crashes
Environment

Reference commit/version: v1.2.2 & v1.2.1
Other platform details: docker

This happens because some old config was cached, I wonder if when some module is not configured it should always have a fallback instead of crash

This happens because some old config was cached, I wonder if when some module is not configured it should always have a fallback instead of crash

repo: axa-group/Parsr


text:
🛑 Webhost prancer is down

In d7598b0, Webhost prancer ($URL_BLITZEN) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Webhost prancer is back up in 52389d3 after 2 hours, 37 minutes.

repo: axinjanet/upptime

license: MIT


text:
do more precise glibc fallback + error reporting

This would be a followup to #1210
We compute per-platform (in fact per-asset) glibc requirements, but for ease of usage / migration that gets smashed down to a single global glibc version when we hand it to the shell/npm/homebrew installer.
Ideally they should remember the requirements per-platform, and when they select a platform check that precise info.
Also if they try to do static-musl fallback, but there's no static-musl, we should complain about glibc version and not static-musl, because that's confusing.

We ended up doing this in #1346.

authors: Gankra mistydemeo

repo: axodotdev/cargo-dist


text:
GH workflow doesn't run when multiple tags are pushed

For proto (https://github.com/moonrepo/proto), we publish individual versions of each crate (using cargo release). This tags just fine, which you can see on the tags page: https://github.com/moonrepo/proto/tags
I've noticed this a few times now, that when I publish multiple tags, the GH dist/relase action does not run, resulting in no GH releases. On the tags page, notice that I just published proto_cli-v0.1.7 alongside proto_node-v0.1.3 and a few others, but no release was created.
I wouldn't be surprised if GH has some condition that these workflows only run for 1 tag...

An alternative is maybe using workflow_dispatch (https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch). This is what I use on other projects.

Looks like I was right: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push

Note: An event will not be created when you push more than three tags at once.

hey @milesj! thanks so much for the thorough report-  that's pretty frustrating that github has implemented a magic number of "three" tags.
i'll take a look into the workflow dispatch for this type of situation. for fun tho- if you are willing to share- could you say more about how you are using tags in your repo? i'm curious to learn more! (also could hop on a call if you'd rather, just let me know!)

So we're not really using tags for everything, cargo release just creates them. We have a few ideas on moving forward:

Don't create tags with cargo release.
Update the release workflow to only use "v#" instead of "crate-v#", and then manually tag when needed.

oooooh interesting. i am not a cargo-release user (it has made some decisions that mean i'm unlikely to become one) so i appreciate you sharing. does it create tags for every member of the workspace? (i am also gonna go look up the docs more specifically now around this)

I'm still learning cargo-release, it's a bit different than what I'm used to in Node.js world.
But basically cargo release patch --execute will bump the versions of all crates, then tag each one. I believe it supports a "unified version" approach but we want individual versions.

To be clear this was indeed an intended/supported workflow that I tested but of course OF COURSE I only tested up to 3 tags at once so I never hit this absolutely brutal and arbitrary restriction from github 😭

GH really hates monorepos huh.

authors: Gankra ashleygwilliams milesj


text:
:load a library in repl doesn't put shape factory

> :l path-to-stdlib
> open arith::nat
> def two : Nat => 2
In file <stdin>:1:17 ->

  1 │   def two : Nat => 2
    │                    ╰╯

Error: No rule inferring the type of 2

The problem is that ReplCompiler#compileToContext uses a clean ResolveInfo which has nothing in ResolveInfo#imports, so the StmtPreResolve will not load the shapeFactory to the ResolveInfo.
https://github.com/aya-prover/aya-dev/blob/e3f27d12ed40fe1e1c3d65916d07d91fd24558e1/base/src/main/java/org/aya/resolve/visitor/StmtPreResolver.java#L70-L81

Replace open arith::nat with open import arith::nat seems to fix it?

Replace open arith::nat with open import arith::nat seems to fix it?

No, it seems not

Actually, now that I think about it, I think importing all the modules make sense. But still the opening needs to be fixed.

❯ java -jar cli-console/build/libs/cli-console-0.36.0-SNAPSHOT-fat.jar -i
Aya 0.36.0-SNAPSHOT (1ddf5c379a0bba27ec8307d639783391426d8c65)
> :l cli-impl/src/test/resources/shared
Compiling tests
  [Info] Resolving source file dependency
    Done in 221ms
  [Tyck] relation::nullary::empty (relation/nullary/empty.aya)
  [Tyck] relation::unary::negation (relation/unary/negation.aya)
  [Tyck] relation::binary::path (relation/binary/path.aya)
  [Tyck] data::sum::base (data/sum/base.aya)
  [Tyck] data::sum (data/sum.aya)
  [Tyck] data::unit::base (data/unit/base.aya)
  [Tyck] data::unit (data/unit.aya)
  [Tyck] data::interval (data/interval.aya)
  [Tyck] data::fmset::base (data/fmset/base.aya)
  [Tyck] arith::nat::base (arith/nat/base.aya)
  [Tyck] data::list::base (data/list/base.aya)
  [Tyck] data::list::properties (data/list/properties.aya)
In file data/list/properties.aya:25:7 ->

  23 │   def ++-assoc (xs ys zs : List A) : (xs ++ ys) ++ zs = xs ++ (ys ++ zs) elim xs
  24 │   | [ ] => refl
  25 │   | x :< xs => pmap (x :<) (++-assoc xs ys zs)
     │          ╰╯

Warning: The name `xs' shadows a previous local definition from outer scope

In file data/list/properties.aya:29:7 ->

  27 │   private def rev'-map (f : A -> B) (buf xs : List A) : map f (rev' buf xs) = rev' (map f buf) (map f xs) elim xs
  28 │   | [ ] => refl
  29 │   | x :< xs => rev'-map f (x :< buf) xs
     │          ╰╯

Warning: The name `xs' shadows a previous local definition from outer scope

In file data/list/properties.aya:35:7 ->

  33 │   private def rev'-++ (buf xs : List A) : rev' buf xs = rev xs ++ buf elim xs
  34 │   | [ ] => refl
  35 │   | x :< xs =>
     │          ╰╯

Warning: The name `xs' shadows a previous local definition from outer scope

In file data/list/properties.aya:44:7 ->

  42 │   def rev-distrib-++ (xs ys : List A) : rev (xs ++ ys) = (rev ys ++ rev xs) elim xs
  43 │   | [ ] => refl
  44 │   | x :< xs =>
     │          ╰╯

Warning: The name `xs' shadows a previous local definition from outer scope

  [Tyck] arith::nat::properties (arith/nat/properties.aya)
  [Tyck] arith::nat (arith/nat.aya)
  [Tyck] arith::bool::base (arith/bool/base.aya)
  [Tyck] arith::int::base (arith/int/base.aya)
  [Tyck] arith::bool::properties (arith/bool/properties.aya)
  [Tyck] arith::bool (arith/bool.aya)
  [Tyck] prelude (prelude.aya)
  [Tyck] data::vec::base (data/vec/base.aya)
  [Tyck] data::vec::properties (data/vec/properties.aya)
In file data/vec/properties.aya:18:7 ->

  16 │   def vmap-distrib-++ (f : A -> B) (xs : Vec n A) (ys : Vec m A) : vmap f (xs ++ ys) = vmap f xs ++ vmap f ys elim xs
  17 │   | [] => refl
  18 │   | x :> xs => pmap (f x :>) (vmap-distrib-++ f xs ys)
     │          ╰╯

Warning: The name `xs' shadows a previous local definition from outer scope

  [Tyck] data::vec (data/vec.aya)
  [Tyck] algebra::free-group (algebra/free-group.aya)
  [Tyck] type-theory::qiit (type-theory/qiit.aya)
  [Tyck] relation::binary::quotient (relation/binary/quotient.aya)
I like these modules :)
    Library loaded in 5527ms
🎉
> open arith::nat
> 1 + 1
2
> :q
See you space cow woof woof :3

repo: aya-prover/aya-dev


text:
🛑 Özyasar is down

In c9dea11, Özyasar (https://www.ozyasar.com.tr/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Özyasar is back up in 7db951f after 25 minutes.

repo: aydgn/upptime


text:
chore(library): implement turporepo

chore(library): implement turborepo

Fixes #2
Description of changes:

set up dependencies to implement turborepo
update npm scripts

Tag a reviewer: @fazzaamiarso
Tasks:

[x] I have ran the build command to make sure apps are working: npm run build
[x] I have ran the tests to make sure nothing is broken: npm run test
[x] I have ran the linter to make sure code is clean: npm run lint
[x] I have reviewed my own code to remove changes that are not needed

@ayoayco I've tested them and they're looking good on my side. Do we need anything else? improve the docs maybe?

I think this one is ready. Any improvement needed can be worked on in other tasks.
Let me just see why the pipeline is failing 😅

@fazzaamiarso for some reason turbo run test was giving us a timeout error in the pipeline. This does not happen on the local though. What I did for now is create a new script "test-ci" which does the npm workspace commend npm run test --workspaces --if-present and this works, but we don't get the benefit of turbo running our tests in our pipelines. Maybe for another issue?

@ayoayco  is it possible for me to get write access to this repo? I wanted to push to this branch but blocked

Merged! The branch is autodeleted

authors: ayoayco fazzaamiarso

repo: ayoayco/astro-reactive-library


text:
Add border to highlighted words in editor

Highlighted words in editor have poor visibility. For example:

Adding a border around the highlighted word makes it much more visible:

I currently have the above in my settings as follows:
    "workbench.colorCustomizations": {
        "[Ayu Mirage]": {
            "editor.wordHighlightBorder": "#d9d7cd",
            "editor.wordHighlightStrongBorder": "#ffcb65"
        },
        "[Ayu Light]": {
            "editor.wordHighlightBorder": "#d9d7cd",
            "editor.wordHighlightStrongBorder": "#ffcb65"
        },
        "[Ayu Dark]": {
            "editor.wordHighlightBorder": "#d9d7cd",
            "editor.wordHighlightStrongBorder": "#ffcb65"
        },
    }

repo: ayu-theme/vscode-ayu

license: mit

license_source: bigquery


text:
usb3

my SSD works when I plug it in to the usb2 ports, but not the usb3 ports. on the 4.13 kernel. usb3 works sometimes on the 4.4 kernel:
(4.4)
Dec 03 01:51:32 rock64 kernel: xhci-hcd xhci-hcd.8.auto: Cannot set link state.
Dec 03 01:51:32 rock64 kernel: usb usb5-port1: cannot disable (err = -32)
Dec 03 01:51:34 rock64 kernel: xhci-hcd xhci-hcd.8.auto: Cannot set link state.
Dec 03 01:51:34 rock64 kernel: usb usb5-port1: cannot disable (err = -32)

@shawnl I'm assuming you figured out usb3 on mainline? Is this working?

My ssd was faulty.

cool, thanks!

authors: itdaniher shawnl


text:
Determine pages titles and order in sidebar menu

The current sidebar may include pages that could be combined.

[ ] Edit page titles
[ ] Determine display order of pages in sidebar menu

Let's iterate on this structure by commenting with deletions (strikethroughs) and additions (bold)
Proposed Page Structure:

Welcome
Identity (= Overview)

Company
Products
Fellowships
Community Groups
Partners

Communication (= Overview)

Voice and tone
Website
Social media
Press release (wait for future release)
Newsletters (wait for future release)

Brand Design (= Overview)

Logo
Colors
Typography
Video (wait for future release)
Photography
Blog
Print

Support
Contribute
Resources
Release History

Fixed in #24

repo: azavea/azavea-design-system


text:
Assemble incident report for outage

Ran management command that was memory intensive
Determine timeline through Panopta and Slack
Determine remediation items based on changes made last night

Looks good. Please ensure that the document we created makes its way to the public Incident Reports directory in Google Drive.
cc: @rbreslow

That's where it is.

authors: hectcastro rbreslow

repo: azavea/district-builder-dtl-pa


text:
Integrate with SML/NJ

Problem

No integration with SML/NJ currently. Have to go to the terminal and manually run it every time.
Solution

the baseline would be that i can have a window with the smlnj interactive system, and be able to send either a whole buffer from a millet window, or a selected segment.  ideally i could also go back from the interactive window by clicking on error messages.

(from Bob Harper on the Project Savanna Discord)

This is a bit non trivial especially with larger projects. For instance, if i have a project with a mlb file
a.sml
b.sml
c.sml

and i'm in the middle of b.sml and request SML/NJ output, we should be clever enough to feed SML/NJ only a.sml and part of b.sml (up to the current cursor?)
Feeding the interpreter only part of a file is also a bit tricky, since what if the file starts with e.g. a structure Foo = struct ... that goes until the end of the file? How should invoking SML/NJ when the cursor is between the struct and matching end work?

repo: azdavis/millet


text:
是否使用了无标签数据的gt？

您好，感谢您的工作，在阅读您的代码后我有一个问题:
我发现在pvrcnn_ssl.py中，您在filter_middle_pseudo_label函数中有两处使用了gt_boxes。
`# iou_flag to filter out pseudo-labels that overlap with gt
if len(train_box) != 0:
iou = iou3d_nms_utils.boxes_bev_iou_cpu(train_box, test_box)
iou_value = np.sum(iou, axis=0)
            iou_flag = iou_value == 0
        else:
            iou_flag = np.array([True] * len(test_box))`

以及
`ori_gt_box = batch_dict['gt_boxes']
new_gt_box_list = []
for bs_idx in range(batch_size):
cur_gt_box = ori_gt_box[bs_idx]
cur_gt_box = cur_gt_box[torch.sum(cur_gt_box, 1) != 0]
if label_mask[bs_idx]:   # labeled scene do not mine pseudo-label
new_gt_box_list.append(cur_gt_box)
continue
        if len(cur_gt_box) > 0:
            cur_gt_box = [bb.unsqueeze(0) for bb in cur_gt_box if bb[0] != 0]
            cur_gt_box = torch.cat(cur_gt_box) if len(cur_gt_box) > 0 else torch.zeros((0, 8)).to(ori_gt_box.device)
            cur_gt_box = cur_gt_box if len(cur_gt_box.shape) > 1 else cur_gt_box.unsqueeze(0)
            cur_pseudo_box = select_mid_box[bs_idx]
            new_gt_box = torch.cat((cur_gt_box, cur_pseudo_box), dim=0)
            new_gt_box_list.append(new_gt_box)`

想询问您是出于什么原因这么做，据我所知在半监督设置下不能使用无标签数据的label，期待您的回复，谢谢您！

您好，感谢您的关注！
按照我们在README中提供的数据处理流程，您所说的“gt_boxes”并不包含任何无标签数据的gt信息，也不涉及到使用无标签数据的label，谢谢！

我跟这位同行有同样的疑惑，所以这行代码中的cur_gt_box 包含的是什么信息呢？https://github.com/azhuantou/HSSDA/blob/main/pcdet/models/detectors/pv_rcnn_ssl.py#L507
您好像并没有在这里区分该样本是有标签还是无标签数据，换句话说，该样本是无标签数据会cat cur_gt_box 和 cur_pseudo_box。
根据我的理解，您在https://github.com/azhuantou/HSSDA/blob/main/pcdet/models/detectors/pv_rcnn_ssl.py#L407这一段代码里是计算了预测的box和GT之间的IoU，然后用这个IoU_flag筛选除GT_box之外的预测结果（KITTI数据集标签是不完整的），也就是cur_pseudo_box。如果我的理解有误，请您指出~感谢回复！

懂了，cur gt 是前一个”interval“ 产生的”hard“伪标签。
另外，为啥要mask Ped，直接屏蔽Ped这一类，不是变成了单阈值吗？这样Ped要么当作"gt_box"，要么被移除。感谢您的回复！

authors: Whale-ice azhuantou dof-pikes

repo: azhuantou/HSSDA


text:
feat: New preset Docusaurus

Adds Docusaurus as a new deliver preset option.
As we're working in changes related to the examples repository, you'll have to trust me that I didn't cheat in the testing:

This PR relates to: https://github.com/aziontech/azion-samples/pull/110

:tada: This PR is included in version 2.11.0-stage.2 :tada:
The release is available on:

npm package (@stage dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.11.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

authors: PabloDiehl tiagokrebs

repo: aziontech/vulcan


text:
Not permitted to proxy as 'azkaban' through Azkaban

@HappyRay @kunkun-tang
I get  this issue
azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib'
	at azkaban.utils.Props.getString(Props.java:426)
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:234)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:748)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:591)
	at azkaban.execapp.JobRunner.run(JobRunner.java:552)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
25-08-2017 18:20:14 CST foo ERROR - Missing required property 'azkaban.native.lib' cause: null

From the doc http://azkaban.github.io/azkaban/docs/latest/
Execute-As-User
With a new security enhancement in Azkaban 3.0, Azkaban jobs can now run as the submit user or the user.to.proxy of the flow by default. This ensures that Azkaban takes advantage of the Linux permission security mechanism, and operationally this simplifies resource monitoring and visibility. Set up this behavior by doing the following:-
Execute.as.user is set to true by default. In case needed, it can also be configured to false in azkaban-plugin’s commonprivate.properties
Configure azkaban.native.lib= to the place where you are going to put the compiled execute-as-user.c file (see below)
Generate an executable on the Azkaban box for azkaban-common/src/main/c/execute-as-user.c. it should be named execute-as-user Below is a sample approach
scp ./azkaban-common/src/main/c/execute-as-user.c onto the Azkaban box
run: gcc execute-as-user.c -o execute-as-user
run: chown root execute-as-user (you might need root privilege)
run: chmod 6050 execute-as-user (you might need root privilege)
I have done in exec-server  by step, while in the {azkban-exec-server-3.33.0}/plugins/jobtypes
then I get other issue:
java.lang.RuntimeException: Not permitted to proxy as 'azkaban' through Azkaban
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:240)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:748)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:591)
	at azkaban.execapp.JobRunner.run(JobRunner.java:552)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
25-08-2017 20:58:20 CST foo ERROR - Not permitted to proxy as 'azkaban' through Azkaban cause: null

So where is the problem? thx!

@HappyRay
@kunkun-tang

azkaban/jobExecutor/ProcessJob.java:226
@lhcg Please check the above code.  The two names, including root and azkaban are blacklisted. You should give another name a try.

How will you guys suggest us to do parameters passing between jobs then ?
Right now I am facing an issue where a job that should write to JOB_OUTPUT_PROP_FILE and is being executed by a user other then azkaban is failing on permission to write the json file inside the execution dir which is owned by azkaban user.

We haven't seen this permission issue.  #1292 and #1325 may be related.
cc @reallocf

@HappyRay @reallocf Just wanted to let you know that I was able to kinda fix that: The issue was we had copied project directories from azkaban 3.0.0 to azkaban 3.34.0 and this triggered the permission issue. Once we had a clean installation and a new deployment of one the previous failing projects we did not observe the permissions issue anymore.
This leaves me with one open issue which is how we should migrate the projects from 3.0.0 to recent version. It seemed to us that when trying to use 3.0.0 maintained DB with recent Azakaban web and exec there is some error message popping up when it tries to load the project to the exec from DB. We will try it soon and I will update whether this is true enough.

@prokod There are some DB schema changes. Please take a look at azkaban-db/src/main/sql/upgrade.3.20.0.to.3.22.0.sql for example. Also take a look at the changes made to the sql directory azkaban-db/src/main/sql

Hi @HappyRay so my previous statement that it works as expected was an incorrect one.
It appears I had set execute.as.user to false and this is why I did not have issues with the specific workflow I was testing.
When I reinstated execute.as.user=true then some of the directories under the execution are created with user.to.proxy user as owner and then azkaban user cannot write to those directories for instance:
07-09-2017 16:24:46 CEST preprocessing-hourly-forward ERROR - Could not open log file in executions/75/forwardFlow for job preprocessing-hourly-forward_calcCheck
java.io.FileNotFoundException: /opt/azkaban-exec-server-3.34.0/executions/75/forwardFlow/_job.75.preprocessing-hourly-forward_calcCheck.log (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.<init>(FileAppender.java:110)
	at org.apache.log4j.RollingFileAppender.<init>(RollingFileAppender.java:79)
	at azkaban.execapp.JobRunner.createFileAppender(JobRunner.java:317)
	at azkaban.execapp.JobRunner.createLogger(JobRunner.java:266)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:569)
	at azkaban.execapp.JobRunner.run(JobRunner.java:552)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

In this example forwardFlow dir owner is same as user.to.proxywhich is different thenazkaban` user.
Any ideas how to resolve that?

@reallocf should have fixed this. He can share more details.

Hey @prokod!
A couple thoughts. First, if you're coming from 3.0.0, I'd suggest upgrading to 3.30.1 which is our most recent stable release. 3.32.2 should also be good - we've been running it here in production for almost a month now and it will likely be marked as a stable build soon. We're only now releasing 3.34.0 so there might be additional bugs other than the one that might be here.
Now for your problems with execute-as-user, could you please share an example of a flow that is failing with your setup? Between 3.32.2 and 3.34.0 we updated the execute-as-user.c file and it's possible that something regressed.
Thanks and sorry for the delay! :smiley:

So here is the state of things:

In Azkaban 3.34.0 we get a permission denied error between jobs (looking at the flowLog tab for the execution) and the details are in my last comment.
In Azkaban 3.30.1 and 3.32.2 we do not see the above error, it passes this stage but within the job it fails on the following line:

echo "{\"param.doRun\": \"$DO_RUN\", \"param.input.partitions\": \"$INPUT_PARTITIONS\", \"param.spark.app.sessionLookAheadCalc\": \"$SESSION_LOOK_AHEAD_CALC\"}" > $JOB_OUTPUT_PROP_FILE

with the following error
08-09-2017 09:25:40 CEST preprocessing-hourly-forward_calcCheck INFO - {"param.doRun": "false", "param.input.partitions": "450", "param.spark.app.sessionLookAheadCalc": "true"}
08-09-2017 09:25:40 CEST preprocessing-hourly-forward_calcCheck INFO - /opt/azkaban-exec-server-3.32.2/executions/77/forwardFlow/../scripts/calcCheck.sh: line 286: /opt/azkaban-exec-server-3.32.2/executions/77/forwardFlow/preprocessing-hourly-forward_calcCheck_output_4944073578545338712_tmp: Permission denied
08-09-2017 09:25:40 CEST preprocessing-hourly-forward_calcCheck INFO - Process completed unsuccessfully in 7 seconds.

Btw, we are running Azkaban on CentOS 6

An example project is here:
preprocessing-hourly_2.10_cdh5-0.11.9-SNAPSHOT-issue-1390.zip
This is  a stripped down one with only first two jobs, the second fails on our 3.30.1/3.32.2 installations

@prokod that's interesting. I'm not seeing the same behavior when I try to run your job on our testing cluster here.
How is your execute-as-user executable permissioned?

It is according to Azkaban 3 doc (I hope):
chmod 6050
ownership root:azkaban

Yes, that's the proper permissioning. Hmm, I'm not totally sure what's happening in your environment @prokod. I'm not familiar with the nuances of CentOS 6 - we're running our production instances on Redhat and I'm not sure if there's any substantive differences that could be the culprit there.
I'm working now on a fix on top of 3.34.0 that should make execute-as-user work as expected going forward. For now, I'd suggest running with execute.as.user=false unless you can figure out a workaround in your particular environment. My fix should be out in 3.35.0 or 3.36.0, so sometime later this month. Hopefully you can fully upgrade and turn execute-as-user on then.
Sorry that it's not working as expected for you out of the box!

@reallocf what's your proposed change?

The fix I'm working on has to do with changing the permissions of the JOB_OUTPUT_PROP_FILE and JOB_PROP_FILE so they can be written to via the new execute-as-user executable. Because these files have been generated by the Azkaban process, they need to be updated to allow read/write access for the user.
Luckily, I can leverage some of the code I wrote for updating the permissions of the /executions/<exec_id> directory so it shouldn't be too much new work. Expect a PR in the next few days (as long as oncall doesn't take too much time).

Got it. Thanks.
Shall we hold off the current deployment and patch the release so that we don't break the users that rely on this feature? What do you recommend?

Do we have an integration test for this feature yet?

We do not need to hold off on current deployment - because we haven't upgraded the execute-as-user executable on any cluster except Yugioh, there won't be a problem.

I'm also working on an integration test to ensure we don't regress in the ability to properly access this file in the future.
Question: @HappyRay do you know of any ways to unit test things like this? I haven't been able to think of any great way to test thing like this that are so reliant on the filesystem.

@lhcg i have the same issue as you ,are you sovled it? can you tail me the detail methon?thx

@lhcg hi! i had solved this issue, modify the ProcessJob.java 204 line transfer ture for false
final boolean isExecuteAsUser = this.sysProps.getBoolean(EXECUTE_AS_USER, false);
in  this  azkaban-common-0.1.0-SNAPSHOT.jar

The permission-ing approach taken in this project has a hidden assumption which is a crucial one! The user azkaban-exec service is running under (usually azkaban) should have umask 002 which means that by default created file/dir will have also write permission set.

See   https://blog.csdn.net/zhang527836447/article/details/99602022
By default, Azkaban does not allow root and Azkaban to perform jobs as proxy users.

authors: HappyRay kunkun-tang lhcg lijixiang123 prokod reallocf zlessa

repo: azkaban/azkaban


text:
What's supposed to happen when an exec server dies?

Is there any mechanism to mark jobs and flows that it was executing as failed?

It looks like there is an ExecutorHealthChecker that periodically pings exec servers but it looks like it just sends emails.

Can the exec server resume executing the flows that it was executing when it restarts?

When an exec server starts up with the same host and port as it has when it died then it will reuse the executor id, but I can't see that it does anything with flows that the database records as being executed by it.
It seems like at the moment it's a manual process to find flows that are recorded as running but actually aren't running and kill them - is that right?

I should clarify that I'm talking about when using the polling model.

@toby-allsopp It is actually currently marking as failed only and hence not able to resume the job again.

authors: sarsjits toby-allsopp


text:
Carnaval

Begins	Friday before Ash Wednesday (51 days to Easter)
Ends	Ash Wednesday midday (45 days before Easter)

Thanks for the PR! Let me have a look so I can merge it :)

Hi @icaroce
Looks indeed that this should be 51 days. Can you also adjust the corresponding unit tests? Then I will merge your PR :)
Cheers! Sacha

authors: icaroce stelgenhof

repo: azuyalabs/yasumi


text:
cpp and c output not functioning

Traceback (most recent call last):
File "/usr/local/bin/shellsploit", line 9, in
load_entry_point('cmdline-shellsploit==0.1', 'console_scripts', 'shellsploit')()
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/shellsploit.py", line 247, in main
start()
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/shellsploit.py", line 33, in start
shellsploit()
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/shellsploit.py", line 82, in shellsploit
B3mB4m().control(shellcode)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 595, in control
self.control( string)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 595, in control
self.control( string)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 1054, in control
self.control( string)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 1054, in control
self.control( string)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 1054, in control
self.control( string)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 923, in control
self.control( string)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/control.py", line 979, in control
CplusplusFile( self.disassembly, True)
File "/usr/local/lib/python2.7/dist-packages/cmdline_shellsploit-0.1-py2.7.egg/shell/Outputs/Cplusplus.py", line 41, in CplusplusFile
""" % ( time.strftime("%d/%m/%Y"), time.strftime("%H:%M:%S"), shellcode)
NameError: global name 'time' is not defined
This error occurs with both cpp output after generation and c output.

Thanks for report, fixed.


text:
Modern Beta doesn't work on 1.18.2

It seems that all of the Bippity Boppity World Generation and Feature placement magic has somehow broken this Mod since when I try to run this with the most up to date Fabric 0.13.3 and Fabric API 0.48.0 it just tells me:"This crash may be caused by an invalid Java Runtime configuration. Check any custom configuration settings." Without even giving me a crash log. This proves that they should add these sorts of changes for new major Updates and not in the middle of the old update.

Tried to see if I could update this. There have been significant changes to internal world gen code. There were several errors caused by attempting to build for 1.18.2. There will likely need to be some very big changes to the mod, and the author is probably the only person with the familiarity with it necessary to do that.

Yes, my free time is a lot more limited these days, but the update is in progress. The world gen changes are indeed very significant, so I'll be working through the snapshots up to 1.18.2 to make sure I can manage it.

Yes, my free time is a lot more limited these days, but the update is in progress. The world gen changes are indeed very significant, so I'll be working through the snapshots up to 1.18.2 to make sure I can manage it.

If there's any parts you can give some basic instructions for that I can help with (assuming it's not more effort to tell me how to do it than to do it yourself) feel free to hit me up. Discord is xdMatthewbx#1337

Unfortunately, I've spent some time on working on the update and arrived at an impasse. The 1.18.2 world gen changes are above and beyond anything Mojang have done; so, I'm going to put the update on hold for now. I understand the reason for the changes, complete control of the noise sampling used for world generation. But I don't get the point when the noise settings are now beyond convoluted, to the point that I wonder if the intended end user (random players) could even make sense of the .json worldgen files.
I'm going to shift my focus to addressing the issues that affect the current 1.18.1 version. I apologize and appreciate the support.

hey, I'm kind of naughty, because someone taught me similar method-
to download the "testing/dev version" at few moments ago. I'm sorry
moderators I'm going to peak this mod for 1.18.2,
please remember this is still beta, devs is still fixing some problems,
meaning may cause some issue in this mod. In my case, it works perfectly fine:
disclaimer: me using windows 10

go here: https://github.com/b3spectacled/modern-beta-fabric/actions/runs/2178226759
scroll down as you can see there are The only Single Package, call 'build-artifacts' a .zip file
click and download the package
once you got the package then Extract the .zip file
inside 'build-artifacts' .zip, you can see 4 Different .jar,
You MUST pick modern-beta-fabric-3.0.3+1.18.2.jar {cut 'ctrl+x'} to the Minecraft 'mods' folder..}
(named include-dev,-sources,-sources-dev Is NOT What You Looking For)
installing .jar regular method once again (open with winRAR archiver might coming-
handy, use your mouse pull straight out)

Good luck for installing! Congratulations if you successfully installed!!
I wish this method of installation in future also be help.
Have a nice day :)

Regarding above: not a full release! I think I speak for the dev when I say use the above in worlds you value at your own risk! It is being developed by this developer and is legitimate but it is an early version and if it were ready for a full release it would be one. Do not blame anyone but yourself if the above beta version breaks your world!

Updated for 1.18.2, finally! Thanks for your support!

authors: Nolij Zockerhopper b3spectacled vrgtics


text:
babel-eslint not compatible with no-unused-vars: [2, {vars: local}]

Ok I have this setup:
❯ tree -a -I node_modules
.
├── A
│   ├── .eslintrc
│   └── a.js
├── B
│   ├── .eslintrc
│   └── b.js
└── package.json

A/.eslintrc:
---
rules:
  no-unused-vars: [2, {vars: local}]  # Allow defining globals

A/a.js:
var a = 1;

B/.eslintrc:
---
parser: babel-eslint

B/b.js:
var b = 2;

package.json:
{
  "dependencies": {
    "babel-eslint": "^3.1.19",
    "eslint": "^0.24.0",
    "esprima": "^2.4.0",
    "esprima-fb": "^15001.1.0-dev-harmony-fb"
  }
}

Running eslint on dir A and B gives the expected result:
❯ $(npm bin)/eslint A B

B/b.js
  1:4  error  b is defined but never used  no-unused-vars

✖ 1 problem (1 error, 0 warnings)

however, running it on dir B and A gives an additional unexpected error on a.js:
❯ $(npm bin)/eslint B A

B/b.js
  1:4  error  b is defined but never used  no-unused-vars

A/a.js
  1:4  error  a is defined but never used  no-unused-vars

✖ 2 problems (2 errors, 0 warnings)

This only happens when using babel-eslint as the parser for directory B. I've tried the default parser, esprima, and esprima-fb and it does not throw the error for a.js. The result shouldn't change given the order of directories given to eslint right?

I don't think this would be a babel-eslint since it's only the parser and changes some tokens etc - It doesn't define any rules or anything. I'm not sure what happens when you try to lint 2 directories with different configs other than in http://eslint.org/docs/user-guide/configuring#configuration-cascading-and-hierarchy - where it merges the config. Maybe that's what is happening here.

@nzakas - this isn't really a babel-eslint right?

If it works OK with the default parser, and Esprima, then it must be a babel-eslint issue.

Ok it's because no-unused-vars: [2, {vars: local}] isn't compatible with babel-eslint - nothing to do with different directories/configs. Just do $(npm bin)/eslint B with the babel-eslint parser and the rule and it will error.

Ok after doing some more looking it seems like it's because we are doing opts.sourceType = "module"; for escope. Based on that I tried
ecmaFeatures:
    modules: true
rules:
  no-unused-vars: [2, {vars: local}]

and it also will error with the regular parser.

I see.. the monkey patching basically affects all subsequent linting.

FWIW, Babel is probably going to get the ability to load ESLint rules rather than hacking around ESLint internal APIs and monkeypatching them.

What would be necessary in the ESLint side to prevent you from needing to monkey patch or otherwise load rules directly?

There's no way for people using Babel to ever use ESLint as a first class citizen. It'd mean putting all of this into ESLint core.

Yikes! Okay, at some point I'll dig in and see what's going on there.

Yeah that part is comment attachment, supporting no-unused-vars for es7 (decorators, comprehensions) and flow by changing escope, changing estraverse visitors

I guess we'll just close this for now with the workaround being what @kentor said:

For now I can just change my script to switch the order to have the directory that uses babel-eslint be the last.

authors: hzoo kentor nzakas sebmck

repo: babel/babel-eslint


text:
Add link to edit docs page in github

Saw a lot of other docs sites have this - I think it's good so it's easy for others to fix typos, etc.
We can change the wording too (or move it somewhere else if you don't like that).

@thejameskyle

Updated


text:
Add JSX fragment syntax support

Q
A

Bug fix?
no

Breaking change?
no

New feature?
yes

Deprecations?
no

Spec compliancy?
yes

Tests added/pass?
yes

Fixed tickets

License
MIT

This PR adds support for https://github.com/facebook/jsx/pull/93.
I'm also currently working on corresponding Babel PR as well, just leaving this up here first for review, since I'm not sure if I'm abusing $FlowIgnore. 😆
UPDATE: Both PR's ready for review and Babel one can be found at https://github.com/babel/babel/pull/6552

@Kovensky Hopefully https://github.com/facebook/jsx/pull/94 will be merged soon, so that my PR for Babel/Babylon can include those changes. :)

Sorry I've been inactive with this PR for a while! I'm going to open a corresponding PR for this on the Babel repository in the coming days. :)

:man_dancing:

repo: babel/babylon


text:
Fix sys.all_columns with correct column types

Description
The previous implementation of sys.all_columns did not match documented
column types for many columns.  This change fixes those discrepancies.
Note: the only exception is the system_type_id which is currently
implemented in Babelfish as type int, so casting to tinyint will cause
an error.
Also, implements the scale, max_length, and precision columns with
actual values using existing helper functions, where before a default was used.
This change also adds tests for the static columns of sys.all_columns and updates tests for BABEL-1756 to reflect the new return type of int.
Task: BABELFISH-402
Signed-off-by: Ross [C] Dutkiewicz dutkiert@amazon.com
Issues Resolved
None
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This is a re-creation of a previous PR that could not be merged due to merge conflicts.  Conflicts were resolved locally and this PR was created to succeed https://github.com/babelfish-for-postgresql/babelfish_extensions/pull/180.


text:
🛑 SM Service is down

In 5450a79, SM Service (http://smservice.de) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SM Service is back up in e868aaa after 20 minutes.

repo: bachmannschumacher/upptime


text:
Shieldy stopped working

The bot stopped working.
I wrote directly and it doesnt answer.
Something went wrong.
Can you check your deploy?

It is started to repond again.

It is started to repond again.

So please close the issue :)

Looks resolved

authors: KlausDevWalker backmeupplz tomerghelber

repo: backmeupplz/shieldy