text:
Getting go: inconsistent vendoring error when a module is located within function

Expected Behaviour
It should work? 🤷♂
Current Behaviour
Step 17/32 : RUN CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Running in 56939001fb88
go: inconsistent vendoring in /go/src/handler:
	github.com/openfaas/templates-sdk@v0.0.0-20200723092016-0ebf61253625: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt

run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
The command '/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1
[0] < Building notifier-fn done in 2.64s.
[0] Worker done.

Total build time: 2.64s
Errors received during build:
- [notifier-fn] received non-zero exit code from build, error: The command '/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .' returned a non-zero code: 1

Steps to Reproduce (for bugs)
We were following the template documentation guide as described here.

$ faas-cli template store pull golang-http
$ faas-cli new notifier-fn --lang golang-http --prefix Dentrax
$ cd notifier-fn
$ go mod init github.com/Dentrax/foo
Use a dependency (i.e. github.com/slack-go/slack) inside handler.go

Context
go.mod
module github.com/Dentrax/foo

go 1.16

require (
    github.com/openfaas/templates-sdk v0.0.0-20200723092016-0ebf61253625
    github.com/slack-go/slack v0.8.1
)

notifier-fn.yml:
version: 1.0
provider:
  name: openfaas
  gateway: http://foo
functions:
  notifier-fn:
    lang: golang-http
    handler: ./notifier-fn
    image: Dentrax/notifier-fn:latest
    build_args:
      GO111MODULE: on
    environment:
      SLACK_WEBHOOK_URL: https://bar

What we have tried so far that did not work
Each step we applied these in sequential:

faas-cli up -f notifier-fn.yml

faas-cli up -f notifier-fn.yml --build-arg GO111MODULE=on

Single handler.go file:

$ go mod init ...

$ go mod vendor + $ go mod tidy? (w/ vendor dir)

Using GO_REPLACE.txt in parent dir

Your Environment

FaaS-CLI version
CLI:
 commit:  6b5e7a14a598527063c7c05fb4187739b6eb6d38
 version: 0.13.6

Docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.7
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:12:42 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
Engine:
Version:          20.10.2
API version:      1.41 (minimum version 1.12)
Go version:       go1.13.15
Git commit:       8891c58
Built:            Mon Dec 28 16:15:28 2020
OS/Arch:          linux/amd64
Experimental:     false
containerd:
Version:          1.4.3
GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version:          1.0.0-rc92
GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version:          0.19.0
GitCommit:        de40ad0

Are you using Docker Swarm (FaaS-swarm ) or Kubernetes (FaaS-netes)? NO

Operating System and version (e.g. Linux, Windows, MacOS): macOS 11.0.1

We are not sure why the given template documentation does not work as we expected. I think we are missing some important detail on the documentation. Actually, the command that failing above is passing on our local workspace: $ CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler . Any ideas?
cc: @developer-guy

@alexellis  I started a looking at a fix that would  simply aligned the http and the middleware templates. But there is an issue.
The http template includes a vendor folder, this is a problem in vendor module. It produces this error due to Go validating that the mod file and the vendor folder are in sync, this produces the following error
#25 0.437 go: inconsistent vendoring in /go/src/handler:
#25 0.437 	github.com/openfaas/templates-sdk@v0.0.0-20200723092016-0ebf61253625: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
#25 0.437 	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt
#25 0.437
#25 0.437 run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory

To work around this error, you can delete the vendor folder that is included in the template. When you remove the vendor folder, I have a PR that works in these two scenarios

pure modules downloaded at build time

faas-cli build -f publisher.yml --build-arg GO111MODULE=on

modules disabled and using vendor

cd publisher && go mod vendor && cd ..
faas-cli build -f publisher.yml --build-arg GO111MODULE=off

But you can't use modules and vendor, for example this will fail
cd publisher && go mod vendor && cd ..
faas-cli build -f publisher.yml --build-arg GO111MODULE=on

This behavior is effectively implemented by using
RUN if [ -d ./function/vendor ] && [ "${GO111MODULE}" == "off" ]; then \
    echo "moving handler vendor" && \
    mv -f ./function/vendor .; \
    else \
    echo "using modules or vendor not found "; \
    fi

the biggest issue is that the template currently defaults to modules "off", which means the default vendor folder  that i just removed is required to make the default "echo" handler work. To make it work we would need to

turn modules on by default
leave modules off, but push the vendor folder from the template base and into the handler folder

One thing to note, i tried using go mod tidy to synchronize the go.mod and vendor during the Docker build, but it still results in and error
#20 [build  8/14] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
#20 sha256:48ad39e8ada37838ac158b13cce56b5bb4cd5dc818c058f9c1f1bc40c024a10f
#20 0.308 cat: can't open 'function/GO_REPLACE.txt': No such file or directory
#20 DONE 0.3s

#21 [build  9/14] RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor . && go mod tidy;     else     echo "using modules or vendor not found ";     fi
#21 sha256:6300638d9169e81111fb8f382fd8a274d66ee6ec190eb958125fe17c0a0361ed
#21 0.357 moving handler vendor
#21 0.539 go: downloading github.com/openfaas/templates-sdk v0.0.0-20200723092016-0ebf61253625
#21 0.631 go: found handler/function in handler/function v0.0.0-00010101000000-000000000000
#21 2.024 go: downloading github.com/openfaas/templates-sdk v0.0.0-20200723110415-a699ec277c12
#21 2.037 go: downloading github.com/nats-io/nats.go v1.10.0
#21 2.140 go: downloading github.com/nats-io/jwt v1.1.0
#21 2.140 go: downloading github.com/nats-io/nkeys v0.1.4
#21 2.140 go: downloading github.com/nats-io/nuid v1.0.1
#21 2.141 go: downloading github.com/nats-io/nats-server/v2 v2.1.9
#21 2.141 go: downloading github.com/golang/protobuf v1.4.3
#21 2.141 go: downloading google.golang.org/protobuf v1.25.0
#21 2.182 go: downloading golang.org/x/crypto v0.0.0-20200323165209-0ec3e9974c59
#21 2.388 go: downloading github.com/google/go-cmp v0.5.0
#21 2.487 go: downloading golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e
#21 2.517 go: downloading golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
#21 DONE 2.9s

#22 [build 10/14] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run "gofmt -s -w" on your Golang code"; exit 1; }
#22 sha256:310a7c685f0ed22d53cc736699d8299ff5543963680ef2eefbb5349f9c7dc1ac
#22 DONE 0.4s

#23 [build 11/14] WORKDIR /go/src/handler/function
#23 sha256:7c444425d527f8de94638a1c47c4fe729f0f40b6ca22c8c74f391558668fdded
#23 DONE 0.0s

#24 [build 12/14] RUN GOOS=linux GOARCH=amd64 go test ./... -cover
#24 sha256:0c7c7857b64200523e078c51cca979b4108c14c4694b7e51299949370e2f4c68
#24 4.240 ?   	function	[no test files]
#24 DONE 4.3s

#25 [build 13/14] WORKDIR /go/src/handler
#25 sha256:f548c022b481872d9036a04d81a4cd4f559d77290190882ef0fd5bf90e13b517
#25 DONE 0.0s

#26 [build 14/14] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
#26 sha256:63f06c1f4ebe4dd80dbd0f24d262bb9d0cb9f39f60acfc1fda7eb06bbbb9ad72
#26 0.492 go: inconsistent vendoring in /go/src/handler:
#26 0.492 	handler/function@v0.0.0-00010101000000-000000000000: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
#26 0.492 	handler/function: is replaced in go.mod, but not marked as replaced in vendor/modules.txt
#26 0.492 	github.com/nats-io/nats.go@v1.10.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
#26 0.492
#26 0.492 run 'go mod vendor' to sync, or use -mod=mod or -mod=readonly to ignore the vendor directory
#26 ERROR: executor failed running [/bin/sh -c CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .]: exit code: 1

The cause is the replace we use in the template go.mod

Do you know if these steps affect caching? Can you try some things and find out.

Do you know if these steps affect caching? Can you try some things and find out.

@alexellis what are some things that you consider as proof? just running the build twice and seeing that step reported as cached?

Here is a second build of my test function, all of the steps report as cached
$ faas-cli build -f publisher.yml --build-arg GO111MODULE=on
[0] > Building publisher.
Clearing temporary build folder: ./build/publisher/
Preparing: ./publisher/ build/publisher/function
Building: publisher:latest with golang-http template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:21a3073c4b533f5a50a7af91c6fc092fd2e018454c212c837e9a49b96a9ee5d5
#1 transferring dockerfile: 2.01kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:10bc899bd437f5c8b9341111d3de92a13762288bed34430aa2ff9bafe5407c5a
#2 transferring context: 102B done
#2 DONE 0.0s

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 ...

#4 [internal] load metadata for ghcr.io/openfaas/of-watchdog:0.8.4
#4 sha256:a36b81af230f4aa9af61173e476a504ddf679531cc6fb84c21a9f883d305ce12
#4 DONE 0.2s

#3 [internal] load metadata for docker.io/library/alpine:3.13
#3 sha256:5ecd52757bf8256597d3bd40d120a58086e4bc5517721ddf15a4b534c85228a0
#3 DONE 0.6s

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 DONE 0.6s

#6 [stage-2 1/7] FROM docker.io/library/alpine:3.13@sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
#6 sha256:ac117f80319f3dfc5f3a89bed8fde570a7095ef011b4aa4f1832954fe5fae646
#6 DONE 0.0s

#10 [build  1/14] FROM docker.io/library/golang:1.15-alpine3.13@sha256:a025015951720f3227acd51b0a99a71578b574a63172ea7d2415c60ae5e2bc0a
#10 sha256:945f28b33676aa8e1e61782ed1c14580cb6577c9dae69198435ffaa8642d6acc
#10 DONE 0.0s

#12 [watchdog 1/1] FROM ghcr.io/openfaas/of-watchdog:0.8.4@sha256:9a0b595f77724005ef7336d68a5a6347426d1931c82f238916496f27d7ea2e74
#12 sha256:0b83d079b8d19e155cadf89ef4c3782ffd5a248d98dbd78ee94c18245505455c
#12 DONE 0.0s

#17 [internal] load build context
#17 sha256:f377bbe6ad24eb025bf76e33d6219983965cc07c9dd78e02186bb94c2e8061ba
#17 transferring context: 16.92kB done
#17 DONE 0.0s

#15 [build  5/14] RUN mkdir -p /go/src/handler
#15 sha256:49827b6b29acd15ceb89996f6858464d41ef41d45bf4439caa09ad778a541c9b
#15 CACHED

#27 [stage-2 6/7] COPY --from=build --chown=app /usr/bin/fwatchdog         .
#27 sha256:a15f3a00af062c11966cc8bf9b62eec7189a69a7a5d6bd292268f5c04e62e80f
#27 CACHED

#13 [build  3/14] COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
#13 sha256:3d4b56eeab2f493ee25bcffa0a77d52bf329e8d5bcf1a7f4f9e60d76207b024c
#13 CACHED

#16 [build  6/14] WORKDIR /go/src/handler
#16 sha256:2d57cb8be74105b3bf574dde9b3acfb2afb9ef7c3db0babaf2976e90bd189f0a
#16 CACHED

#9 [stage-2 4/7] WORKDIR /home/app
#9 sha256:8d07b104cc02b839e2f8b214eb64d1906c976d87c5178fc380b99ff4ee79f52d
#9 CACHED

#14 [build  4/14] RUN chmod +x /usr/bin/fwatchdog
#14 sha256:0ce54538eaf598232623e0d0131601ae53ed928d280203d11398984ad325296d
#14 CACHED

#26 [stage-2 5/7] COPY --from=build --chown=app /go/src/handler/handler    .
#26 sha256:96883fc3eb65c24f97b0347d0c5bfb829b3bd29c2f2df68a7681a0257c0a09d3
#26 CACHED

#11 [build  2/14] RUN apk --no-cache add git
#11 sha256:e895da5279e30155cbfeb55e93d54399f92e5b71ef96937c7955731c811f752e
#11 CACHED

#25 [build 14/14] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
#25 sha256:f103a02f229f25654f54cbde12afa99aa0c721c85804b51c23e539cb95e044e9
#25 CACHED

#21 [build 10/14] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run "gofmt -s -w" on your Golang code"; exit 1; }
#21 sha256:b1d524d09ab70ce20b7f57efa46db80d3cf0e49ea734b78f152828451eacc2cd
#21 CACHED

#23 [build 12/14] RUN GOOS=linux GOARCH=amd64 go test ./... -cover
#23 sha256:2d9eb2ff7aff1812f647f4c7f9e1b420821e818717ce1145553f6b44635b0073
#23 CACHED

#19 [build  8/14] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
#19 sha256:4d72ddb9b23a8417c2db4826354610584a497c771deef7debf52befd032d9638
#19 CACHED

#20 [build  9/14] RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor . && go mod tidy;     else     echo "using modules or vendor not found ";     fi
#20 sha256:d1efda4d0f61d94cc51ed7d079ca194f3ff8a5dae60a6d9af48ffe4bc320efb8
#20 CACHED

#22 [build 11/14] WORKDIR /go/src/handler/function
#22 sha256:10fbc5207db3831f3e06a80dc0c6f960ca19f7dbd8192ff2435b41aaa663a832
#22 CACHED

#24 [build 13/14] WORKDIR /go/src/handler
#24 sha256:ee155e1a79e01bab85a5c637cbf925c23746c9aedd5b1288b142e5628df20daf
#24 CACHED

#8 [stage-2 3/7] RUN mkdir -p /home/app     && chown app /home/app
#8 sha256:392db714e3258ded26fd35e24dbf9da4b34d5f1b72bafa6651edb2c67c6e337a
#8 CACHED

#7 [stage-2 2/7] RUN apk --no-cache add ca-certificates     && addgroup -S app && adduser -S -g app app
#7 sha256:f3920e39e6cb6756252ef8376a8a6a562e10f4156f6387b0e5aa5f3204bbbeb5
#7 CACHED

#18 [build  7/14] COPY . .
#18 sha256:b2081521828c5a6f4f2df32505d0dfa0fd8e3b7f26e1e4e23ec1cfbdc2e60fb5
#18 CACHED

#28 [stage-2 7/7] COPY --from=build --chown=app /go/src/handler/function/  .
#28 sha256:0f57dc249c06f1ea6f7b251e043aaa88a4977b712ae899c41a93513ed9e7578a
#28 CACHED

#29 exporting to image
#29 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#29 exporting layers done
#29 writing image sha256:ae862458b5fc8f845899815ba391511beb488b07239a5d79c2f284f60a351e9b done
#29 naming to docker.io/library/publisher:latest done
#29 DONE 0.0s
Image: publisher:latest built.
[0] < Building publisher done in 2.03s.
[0] Worker done.

Total build time: 2.03s

Same with modules off  and a vendor folder exists
$ faas-cli build -f publisher.yml --build-arg GO111MODULE=off
[0] > Building publisher.
Clearing temporary build folder: ./build/publisher/
Preparing: ./publisher/ build/publisher/function
Building: publisher:latest with golang-http template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:e574877995c10303eb346abe88f875984739ad086d155a9a1760ba721dc461b9
#1 transferring dockerfile: 1.99kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:c2879ad2d201614c4ca5f0b0a93e9bd95ddcd2d3244b2afe07de6e6ac0eda3b2
#2 transferring context: 102B done
#2 DONE 0.0s

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 ...

#4 [internal] load metadata for ghcr.io/openfaas/of-watchdog:0.8.4
#4 sha256:a36b81af230f4aa9af61173e476a504ddf679531cc6fb84c21a9f883d305ce12
#4 DONE 0.2s

#3 [internal] load metadata for docker.io/library/alpine:3.13
#3 sha256:5ecd52757bf8256597d3bd40d120a58086e4bc5517721ddf15a4b534c85228a0
#3 ...

#5 [internal] load metadata for docker.io/library/golang:1.15-alpine3.13
#5 sha256:58dd66b3cb84deb634620790045bb91934c3fb27560259f0b5bb69b42224edb1
#5 DONE 0.6s

#3 [internal] load metadata for docker.io/library/alpine:3.13
#3 sha256:5ecd52757bf8256597d3bd40d120a58086e4bc5517721ddf15a4b534c85228a0
#3 DONE 1.4s

#6 [stage-2 1/7] FROM docker.io/library/alpine:3.13@sha256:a75afd8b57e7f34e4dad8d65e2c7ba2e1975c795ce1ee22fa34f8cf46f96a3be
#6 sha256:ac117f80319f3dfc5f3a89bed8fde570a7095ef011b4aa4f1832954fe5fae646
#6 DONE 0.0s

#10 [build  1/14] FROM docker.io/library/golang:1.15-alpine3.13@sha256:a025015951720f3227acd51b0a99a71578b574a63172ea7d2415c60ae5e2bc0a
#10 sha256:945f28b33676aa8e1e61782ed1c14580cb6577c9dae69198435ffaa8642d6acc
#10 DONE 0.0s

#12 [watchdog 1/1] FROM ghcr.io/openfaas/of-watchdog:0.8.4@sha256:9a0b595f77724005ef7336d68a5a6347426d1931c82f238916496f27d7ea2e74
#12 sha256:0b83d079b8d19e155cadf89ef4c3782ffd5a248d98dbd78ee94c18245505455c
#12 DONE 0.0s

#17 [internal] load build context
#17 sha256:22e4a759ec2fb76c5ae310d48e674f49bb5419d3eeca06c2b27c8987047be299
#17 transferring context: 496.45kB done
#17 DONE 0.0s

#20 [build  9/14] RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor .;     else     echo "using modules or vendor not found ";     fi
#20 sha256:59dd69b01bd2f54de6b035b838eaf41765d90591e42e1add8f2597a30122102d
#20 CACHED

#22 [build 11/14] WORKDIR /go/src/handler/function
#22 sha256:0f0ac30cb75d30ea26d73d4c9c2f1fd1e5b0deec9cdb447c7136b29d4beba8f2
#22 CACHED

#18 [build  7/14] COPY . .
#18 sha256:5f280c8982f74eb09a72e9966ab26deb656de679d2275697d355679904bac9c7
#18 CACHED

#16 [build  6/14] WORKDIR /go/src/handler
#16 sha256:2d57cb8be74105b3bf574dde9b3acfb2afb9ef7c3db0babaf2976e90bd189f0a
#16 CACHED

#23 [build 12/14] RUN GOOS=linux GOARCH=amd64 go test ./... -cover
#23 sha256:db1d87d696d6f1822011c62aa89d515dadbf2eb0b3eb304223308a73644ce7ef
#23 CACHED

#8 [stage-2 3/7] RUN mkdir -p /home/app     && chown app /home/app
#8 sha256:392db714e3258ded26fd35e24dbf9da4b34d5f1b72bafa6651edb2c67c6e337a
#8 CACHED

#26 [stage-2 5/7] COPY --from=build --chown=app /go/src/handler/handler    .
#26 sha256:3d4993c2f2577fd2fb316fc0705cdf656ddb2108d79f88520308dcc9bc09d238
#26 CACHED

#14 [build  4/14] RUN chmod +x /usr/bin/fwatchdog
#14 sha256:0ce54538eaf598232623e0d0131601ae53ed928d280203d11398984ad325296d
#14 CACHED

#21 [build 10/14] RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run "gofmt -s -w" on your Golang code"; exit 1; }
#21 sha256:700ed50eb6b399e541605fae19f2ba57943b7925d06b01eb79436008cc735ea6
#21 CACHED

#27 [stage-2 6/7] COPY --from=build --chown=app /usr/bin/fwatchdog         .
#27 sha256:361a3b5061edb01c935a8e442bfb9bf1cbc4f162ce840bba5b7f2071c3f50c1c
#27 CACHED

#13 [build  3/14] COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
#13 sha256:3d4b56eeab2f493ee25bcffa0a77d52bf329e8d5bcf1a7f4f9e60d76207b024c
#13 CACHED

#15 [build  5/14] RUN mkdir -p /go/src/handler
#15 sha256:49827b6b29acd15ceb89996f6858464d41ef41d45bf4439caa09ad778a541c9b
#15 CACHED

#11 [build  2/14] RUN apk --no-cache add git
#11 sha256:e895da5279e30155cbfeb55e93d54399f92e5b71ef96937c7955731c811f752e
#11 CACHED

#19 [build  8/14] RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
#19 sha256:e8470771778a6d5753f72db5f421edc35846630310fc09596f8dae241fda872d
#19 CACHED

#7 [stage-2 2/7] RUN apk --no-cache add ca-certificates     && addgroup -S app && adduser -S -g app app
#7 sha256:f3920e39e6cb6756252ef8376a8a6a562e10f4156f6387b0e5aa5f3204bbbeb5
#7 CACHED

#9 [stage-2 4/7] WORKDIR /home/app
#9 sha256:8d07b104cc02b839e2f8b214eb64d1906c976d87c5178fc380b99ff4ee79f52d
#9 CACHED

#24 [build 13/14] WORKDIR /go/src/handler
#24 sha256:ff1630159ddb306a357f8bcae32cdabc0d1b3852179bfba444e47dba0b644aee
#24 CACHED

#25 [build 14/14] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
#25 sha256:ef4ce81a267fab2bf49828dc2184db924b15090ed4b8241a8172996473624d2e
#25 CACHED

#28 [stage-2 7/7] COPY --from=build --chown=app /go/src/handler/function/  .
#28 sha256:18d8c2e6684e3b4a3ead499390a5c901e4dd1438d59332d61b0889d5a412add7
#28 CACHED

#29 exporting to image
#29 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#29 exporting layers done
#29 writing image sha256:7374b12db800120b5c6de39d05bb1abfc3d72edb7b96b09cc22b0b25f72a606f done
#29 naming to docker.io/library/publisher:latest done
#29 DONE 0.0s
Image: publisher:latest built.
[0] < Building publisher done in 2.38s.
[0] Worker done.

Total build time: 2.38s

What about with buildkit disabled?

@alexellis i will post the logs in a moment, but the docker docs are prtty clear about cache invalidation for RUN statements
https://docs.docker.com/engine/reference/builder/#:~:text=The cache for RUN instructions isn’t invalidated automatically during the next build

The cache for RUN instructions isn’t invalidated automatically during the next build. The cache for an instruction like RUN apt-get dist-upgrade -y will be reused during the next build. The cache for RUN instructions can be invalidated by using the --no-cache flag, for example docker build --no-cache.
The cache for RUN instructions can be invalidated by ADD and COPY instructions.

There is no reason to expect the if/else to cause a cache invalidation any more than the line immediately before it to copy the go mod replace statements.

Here are the logs when buildkit is disabled
DOCKER_BUILDKIT=0 faas-cli build -f publisher.yml --build-arg GO111MODULE=off
[0] > Building publisher.
Clearing temporary build folder: ./build/publisher/
Preparing: ./publisher/ build/publisher/function
Building: publisher:latest with golang-http template. Please wait..
Sending build context to Docker daemon  569.9kB
Step 1/36 : FROM --platform=${TARGETPLATFORM:-linux/amd64} ghcr.io/openfaas/of-watchdog:0.8.4 as watchdog
 ---> 4715559e6ad8
Step 2/36 : FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.15-alpine3.13 as build
 ---> b8d8ad7b4ab7
Step 3/36 : ARG TARGETPLATFORM
 ---> Using cache
 ---> e54de4e3e329
Step 4/36 : ARG BUILDPLATFORM
 ---> Using cache
 ---> 002f6d91ad3b
Step 5/36 : ARG TARGETOS
 ---> Using cache
 ---> 003ab4a52c89
Step 6/36 : ARG TARGETARCH
 ---> Using cache
 ---> 0978e1807ae2
Step 7/36 : RUN apk --no-cache add git
 ---> Using cache
 ---> fd831584b24f
Step 8/36 : COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
 ---> Using cache
 ---> ce43ffb34a27
Step 9/36 : RUN chmod +x /usr/bin/fwatchdog
 ---> Using cache
 ---> 26f87ae9e6d1
Step 10/36 : ENV CGO_ENABLED=0
 ---> Using cache
 ---> 03e5437e99fc
Step 11/36 : RUN mkdir -p /go/src/handler
 ---> Using cache
 ---> 89964a100c26
Step 12/36 : WORKDIR /go/src/handler
 ---> Using cache
 ---> 633c9382007b
Step 13/36 : COPY . .
 ---> Using cache
 ---> 8e51bfc34593
Step 14/36 : ARG GO111MODULE="off"
 ---> Using cache
 ---> 02efddd77e3d
Step 15/36 : ARG GOPROXY=""
 ---> Using cache
 ---> 638e050b3780
Step 16/36 : ARG GOFLAGS=""
 ---> Using cache
 ---> 2844cdb0943e
Step 17/36 : RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0
 ---> Using cache
 ---> bdce1d761aa5
Step 18/36 : RUN if [ -d ./function/vendor ]; then     echo "moving handler vendor" &&     mv -f ./function/vendor .;     else     echo "using modules or vendor not found ";     fi
 ---> Using cache
 ---> ba70a5a38292
Step 19/36 : RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }
 ---> Using cache
 ---> ccf7a5efd4ab
Step 20/36 : WORKDIR /go/src/handler/function
 ---> Using cache
 ---> 89d0e8bc709e
Step 21/36 : RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover
 ---> Using cache
 ---> d94f2ed7084d
Step 22/36 : WORKDIR /go/src/handler
 ---> Using cache
 ---> 59aac535b520
Step 23/36 : RUN CGO_ENABLED=${CGO_ENABLED} GOOS=${TARGETOS} GOARCH=${TARGETARCH}     go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
 ---> Using cache
 ---> 33eff2c16a04
Step 24/36 : FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.13
 ---> 28f6e2705743
Step 25/36 : RUN apk --no-cache add ca-certificates     && addgroup -S app && adduser -S -g app app
 ---> Using cache
 ---> 16d4eccf075b
Step 26/36 : RUN mkdir -p /home/app     && chown app /home/app
 ---> Using cache
 ---> 7d98c7f0dc19
Step 27/36 : WORKDIR /home/app
 ---> Using cache
 ---> c9a5f5a1b0c0
Step 28/36 : COPY --from=build --chown=app /go/src/handler/handler    .
 ---> Using cache
 ---> 9146e303eade
Step 29/36 : COPY --from=build --chown=app /usr/bin/fwatchdog         .
 ---> Using cache
 ---> 3f68b293e5be
Step 30/36 : COPY --from=build --chown=app /go/src/handler/function/  .
 ---> Using cache
 ---> 5f63bdb1bbc1
Step 31/36 : USER app
 ---> Using cache
 ---> 75ae534882fd
Step 32/36 : ENV fprocess="./handler"
 ---> Using cache
 ---> bb0f94d490d7
Step 33/36 : ENV mode="http"
 ---> Using cache
 ---> c51108fc209d
Step 34/36 : ENV upstream_url="http://127.0.0.1:8082"
 ---> Using cache
 ---> 114a47a9d7e2
Step 35/36 : ENV prefix_logs="false"
 ---> Using cache
 ---> 40b392698500
Step 36/36 : CMD ["./fwatchdog"]
 ---> Using cache
 ---> 2fa0b5044715
Successfully built 2fa0b5044715
Successfully tagged publisher:latest
Image: publisher:latest built.
[0] < Building publisher done in 2.99s.
[0] Worker done.

Total build time: 2.99s

Fixed by https://github.com/openfaas/golang-http-template/pull/56

@tatemz are you looking to use Go modules with vendoring or non-modules with vendoring? or no vendoring and just Go modules?

I use faas-cli in a mono repo atm where my functions live in folder with subfolders organized by type. Ideally, I feel that putting a go.mod in each function handler is an idea situation in order that each function could manage it's own third party deps. That said, I don't like the idea of vendor directories littered throughout my project, but maybe that's just the way it's gotta be.
In short, I think plain modules without vendoring is what my gut is telling me to be the ideal. 🤷

it should "just work" if you add the go.mod and go.sum to each function, no vendor folder required.
I think you are right about the mix of "copy" + replace to copy a shared folder from the same monorepo, but i would need to try it to say for sure

Another note. I think this can be more idempotent:
RUN cat function/GO_REPLACE.txt >> ./go.mod || exit 0

With a bit of refactoring:
RUN for replacement in $(cat function/GO_REPLACE.txt); do go mod edit -replace $replacement; done

This way any replacements that may already exist with that package name would be updated rather than duplicated (which would cause an error)
Thoughts?

Looks like we have resolved this issue, so I'm going to close it.
If anyone lands here, needing help. Raise your own issue please.

/lock: resolved

authors: Dentrax LucasRoesler alexellis tatemz

repo: openfaas/golang-http-template


text:
Break out Crops Service, Simplify User Mutation, Add TOS

What's changed

Put crop service in its own thing
Simplify Update User Mutation
Add TOS, Move Privacy (Close #113)

What's next

Look into Contributor Agreement
Add Sign TOS and Privacy on the sign up page
Send email to mailchimp based on sign up (look at Mailchimp API woohoo)

Coverage decreased (-0.44%) when pulling 222ddf13d839e230b3d5dec8d2db87b48eb7c56d on simonv3:master into 3b90691507e59b666417b93b9f2630b4ed52c8dd on openfarmcc:master.

Coverage decreased (-0.44%) when pulling 222ddf13d839e230b3d5dec8d2db87b48eb7c56d on simonv3:master into 3b90691507e59b666417b93b9f2630b4ed52c8dd on openfarmcc:master.

Coverage decreased (-0.44%) when pulling 1dcc4a96d2c06f7d038ed940854701903f1ea8f9 on simonv3:master into 3b90691507e59b666417b93b9f2630b4ed52c8dd on openfarmcc:master.

source: gharchive/pull-request

repo: openfarmcc/OpenFarm

license: mit

license_type: permissive

license_source: bigquery


text:
Export the data in pdf row by row

Hi,
I want to extract the table data in pdf file.
So, if the code is like following, how can I get this data into the object and export it?
Please provide some APIs.
There are some functions used in-
"fin-hypergrid-master/demo/simple-column-sorting-pipeline.html" but I can not use them to extract.
<!doctype html>

 <script src="https://openfin.github.io/fin-hypergrid/build/fin-hypergrid.js"></script>
 <script>
     var grid = new fin.Hypergrid('#fin-grid', {
         data: [
             { 'symbol':'APPL', 'name':'Apple Inc.', 'prevclose':'93.13' },
             { 'symbol':'MSFT', 'name':'Microsoft Corporation', 'prevclose':'51.91' },
             { 'symbol':'TSLA', 'name':'Tesla Motors Inc.', 'prevclose':'196.40' },
             { 'symbol':'IBM', 'name':'International Business Machines Corp', 'prevclose':'155.35' }
         ]
     });

 </script>

grid.

@tuv123 Hello! The data that is passed into the grid, is the same data we would return back to you when exporting it. It is the array of objects used during instantiation or setData

source: gharchive/issue

repo: openfin/fin-hypergrid


text:
fix: build test fails

Related issues

Fixes #3787
Fixes #3789

Screenshots

openfoodfacts.github.scrachx.openfood.models.entities.additive.AdditivesWrapperTest > map_returnsListOfCorrectlyMappedAdditives FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AdditivesWrapperTest.kt:46

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_LanguageCode FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:67

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_Names FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:75

openfoodfacts.github.scrachx.openfood.models.entities.additive.AdditivesWrapperTest > map_returnsListOfCorrectlyMappedAdditives FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AdditivesWrapperTest.kt:46

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_LanguageCode FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:67

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_Names FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:75

openfoodfacts.github.scrachx.openfood.models.entities.additive.AdditivesWrapperTest > map_returnsListOfCorrectlyMappedAdditives FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AdditivesWrapperTest.kt:46

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_LanguageCode FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:67

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_Names FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:75

I think these files are undergoing refactor by @VaiTon.
@VaiTon Can you please confirm?

openfoodfacts.github.scrachx.openfood.models.entities.additive.AdditivesWrapperTest > map_returnsListOfCorrectlyMappedAdditives FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AdditivesWrapperTest.kt:46

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_LanguageCode FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:67

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_Names FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:75

I think these files are undergoing refactor by @VaiTon.
@VaiTon Can you please confirm?

openfoodfacts.github.scrachx.openfood.models.entities.additive.AdditivesWrapperTest > map_returnsListOfCorrectlyMappedAdditives FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AdditivesWrapperTest.kt:46

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_LanguageCode FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:67

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_Names FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:75

I think these files are undergoing refactor by @VaiTon.
@VaiTon Can you please confirm?

I commited something but I finished ref them.

openfoodfacts.github.scrachx.openfood.models.entities.additive.AdditivesWrapperTest > map_returnsListOfCorrectlyMappedAdditives FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AdditivesWrapperTest.kt:46

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_LanguageCode FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:67

openfoodfacts.github.scrachx.openfood.models.entities.allergen.AllergensWrapperTest > map_returnsListOfCorrectlyMappedSubAllergens_Names FAILED
    com.google.common.truth.ComparisonFailureWithFacts at AllergensWrapperTest.kt:75

I think these files are undergoing refactor by @VaiTon.
@VaiTon Can you please confirm?

I commited something but I finished ref them.

I commited something but I finished ref them.

In that case, I will give them another look if the already existing PR fails to close the issue.

I commited something but I finished ref them.

In that case, I will give them another look if the already existing PR fails to close the issue.

@VaiTon @kartikaysharma01 what's the status on that one ?
I'd really like to make the build green again (MBGA)

@VaiTon @kartikaysharma01 what's the status on that one ?
I'd really like to make the build green again (MBGA)

It's clear for review.

authors: VaiTon kartikaysharma01 teolemon

repo: openfoodfacts/openfoodfacts-androidapp

license_source: github-api


text:
Add animation when deleting an item from a list

What

Add animation when deleting an item from a list

Why

Avoid surprise from users

Part of

Lists

Screenshot/Mockup/Before-After

Can i work on this issue?

sure thing, just assigned you 👍

I have resolved this issue
https://user-images.githubusercontent.com/46370525/160696467-69e76a3b-6f6b-4cd3-8d97-8abdf5a54444.mp4
https://github.com/openfoodfacts/smooth-app/pull/1411

I have resolved this issue, you can now merge my PR @teolemon

authors: IRFAN-GujjAR teolemon

repo: openfoodfacts/smooth-app


text:
This product provokes an error in the app

What

https://world.openfoodfacts.org/product/0804531111606/coconut-drink-taste-of-inspirationstas-brandvara-rood-drink-co-ltd
Link for a scannable barcode: barcode-804531111606 (in our case, the initial 0 was dropped)

I'm able to reproduce it only by scanning the barcode. It's not a search issue


text:
RankingFloatingActionButton  partially off-screen for some translations

What

The RankingFloatingActionButton is partially off-screen for certain translations, such as Portuguese and English, due to the text size.

Steps to reproduce the behavior

Go to 'Scan' tab
Click on 'Search' textfield
Search something, ex: emmenthal
Scroll down
See error

Expected behavior

Why

RenderFlex overflow may cause usability issues.

Screenshot/Mockup/Before-After
https://github.com/openfoodfacts/smooth-app/assets/16384897/e459f845-be78-4aed-ade7-94cf81c68c5b
Smartphone model

Device: iphoneSe (2 generation)
OS: 16.2
App Version: 4.13.1

Fixed by #5117 thanks to @gmmoraes!

authors: gmmoraes monsieurtanuki


text:
Product addition card button not clickable (card should be fully clickable)

What

Product addition card button not clickable anymore (anyway card should be fully clickable, until we torch everything)

@g123k @monsieurtanuki

In 4.17.1?
Because I've fixed it in the release

I've relaunched the build, let's see

Still there @g123k

@g123k are we dropping this one until the new scanner lands ?


text:
feat: 3765 - now using the "crop_image" package

What

The purpose of this PR is to integrate the crop_image package instead of (old) copies of its code.
The next step will be to have crop_image make a method public (cf. https://github.com/deakjahn/crop_image/pull/25) We're all set now.

Part of

#3765

Files
Deleted files:

crop_grid.dart
rotated_crop_controller.dart
rotated_crop_image.dart
rotation.dart

Impacted files:

background_task_image.dart: minor refactoring
crop_page.dart: renamed and moved; now using the crop_image package
image_compute_container.dart: renamed and moved; refactored
image_crop_page.dart: minor refactoring
product_image_viewer.dart: minor refactoring
pubspec.lock: wtf
pubspec.yaml: added crop_image, upgraded image and related packages
uploaded_image_gallery.dart: minor refactoring

Codecov Report

Merging #3926 (48a43ed) into develop (a8366fd) will increase coverage by 0.33%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop    #3926      +/-   ##
===========================================
+ Coverage    10.62%   10.96%   +0.33%
===========================================
  Files          268      264       -4
  Lines        13506    13093     -413
===========================================
  Hits          1435     1435
+ Misses       12071    11658     -413

Impacted Files
Coverage Δ

...ooth_app/lib/background/background_task_image.dart
0.00% <0.00%> (ø)

...mooth_app/lib/helpers/image_compute_container.dart
0.00% <0.00%> (ø)

packages/smooth_app/lib/pages/crop_page.dart
0.00% <0.00%> (ø)

...th_app/lib/pages/image/uploaded_image_gallery.dart
0.00% <ø> (ø)

packages/smooth_app/lib/pages/image_crop_page.dart
0.00% <ø> (ø)

...th_app/lib/pages/product/product_image_viewer.dart
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thank you @M123-dev for the review!

authors: codecov-commenter monsieurtanuki


text:
Error in system creation when using charge_from_molecules and molecule is MOL2 loaded by OETK

Thanks to @slochower, for finding this problem.
Reproducing example, using 0.3.0 toolkit:
MOL-sybyl.mol2.gz
CB7-sybyl.mol2.gz
from openforcefield.topology import Molecule, Topology
from openforcefield.typing.engines.smirnoff import ForceField
from simtk import unit
guest = Molecule('MOL-sybyl.mol2')
host  = Molecule('CB7-sybyl.mol2')

ff = ForceField("test_forcefields/smirnoff99Frosst.offxml")

guest_system = ff.create_openmm_system(guest.to_topology(), charge_from_molecules=[guest])
host_system  = ff.create_openmm_system(host.to_topology(), charge_from_molecules=[host])

returns:
(openforcefield) jwagner@MBP-S$ python test.py
Traceback (most recent call last):
  File "test.py", line 11, in <module>
    guest_system = ff.create_openmm_system(guest.to_topology(), charge_from_molecules=[guest])
  File "/Users/jwagner/projects/OpenForceField/openforcefield/openforcefield/typing/engines/smirnoff/forcefield.py", line 997, in create_openmm_system
    parameter_handler.create_force(system, topology, **kwargs)
  File "/Users/jwagner/projects/OpenForceField/openforcefield/openforcefield/typing/engines/smirnoff/parameters.py", line 2367, in create_force
    charges_from_charge_mol = self.assign_charge_from_molecules(temp_mol, kwargs['charge_from_molecules'])
  File "/Users/jwagner/projects/OpenForceField/openforcefield/openforcefield/typing/engines/smirnoff/parameters.py", line 2339, in assign_charge_from_molecules
    temp_mol_charges[ref_idx] = charge_mol.partial_charges[charge_idx]
  File "/Users/jwagner/miniconda3/envs/openforcefield/lib/python3.6/site-packages/simtk/unit/quantity.py", line 748, in __setitem__
    self._value[key] = value / self.unit
ValueError: assignment destination is read-only

We can somehow shake off the read-only nature of the partial charge array by making a deepcopy.
Fixed code:
from openforcefield.topology import Molecule, Topology
from openforcefield.typing.engines.smirnoff import ForceField
from simtk import unit
guest = Molecule('MOL-sybyl.mol2')
host  = Molecule('CB7-sybyl.mol2')
import copy
guest.partial_charges = copy.deepcopy(guest.partial_charges)
host.partial_charges = copy.deepcopy(host.partial_charges)
ff = ForceField("test_forcefields/smirnoff99Frosst.offxml")

guest_system = ff.create_openmm_system(guest.to_topology(), charge_from_molecules=[guest])
host_system  = ff.create_openmm_system(host.to_topology(), charge_from_molecules=[host])

I suspect the problem is either in

OpeneyeToolkitWrapper.from_openeye, where we initialize storage and load partial charges
FrozenMolecule.partial_charges or TopologyMolecule.partial_charges, setting their values as read-only
ToolkitAM1BCCHandler, in being lazy about how temp_mol_charges is initialized

Thanks @j-wags for looking into this.

I'd never noticed this problem in my files - weirdly changing the example to use Molecule.from_file instead of just the constructor works fine!

This error is no longer raised. The fix from #347 likely resolved this.

authors: SimonBoothroyd j-wags slochower

repo: openforcefield/openforcefield

license: MIT


text:
Rename chargeincrement attribute to charge_increment

In the current spec of ChargeIncrementModel (not currently implemented), we are using the attribute chargeincrement. Should we rename it according to the naming conventions of the other attributes (i.e. charge_increment)?

Yes. I think so. That was laziness on my part -- We recently changed the indexed values of chargeincrement from charge#increment to chargeincrement#, and I should have put an underscore in the middle.

This was implemented here, but rolled back because the PR was getting too big, and this change will need to be accompanied by a SMIRNOFF spec update
https://github.com/openforcefield/openforcefield/pull/349#discussion_r292707035

authors: andrrizzi j-wags


text:
Add cyclobutane parameters

I've added the cyclobutane parameters we discussed in issue #65. These changes correspond to the openforcefield PR#83

LGTM.

authors: bannanc davidlmobley


text:
Fix typo

Change Domonstrate to Demonstrate

Thanks for the help.

authors: lvlds thinkernel

repo: openfrontier/docker-jenkins


text:
[CI] Clone CircleCI in gitlab-ci file

Feature Request
To have a exemple of functional CI on Gitlab. It should not be usefull to have a gitlab-ci in parallel of Circle CI.
Proposal

[ ] Translate the content of CircleCI file into .gitlab-ci.yml

It should be implemented in a separated repository.

authors: cchaudier jmaupetit

repo: openfun/arnold


text:
Media type

Hi there,
is there already an agreed media type (e.g. for usage in STAC)?
Related issue for parquet: https://issues.apache.org/jira/browse/PARQUET-1889
Maybe something like:  application/vnd.geo+apache.parquet or  application/geo+vnd.apache.parquet?

I suppose having a registered MIME type for Parquet itself (the issue you linked) would be a required first step (since a geoparquet file is technically just a parquet file).

Yes, that's true. But on the other hand if it takes too long to wait for it in Apache, the community will just come up with their own definition.
Microsoft Planetary Computer is currently using application/x-parquet, which I'll also adopt for now.

It would be good to establish a convention that differentiates between parquet and geoparquet. I'm happy to update the media types in the Planetary Computer as needed.
I'm not familiar with the various components of a media type, but COG uses image/tiff; application=geotiff; profile=cloud-optimized. If Apache Parquet used application/apache.parquet would it be appropriate to use application/apache.parquet; application=geoparquet, since geoparquet files are valid parquet files? Maybe it'd be weird to have application in the media type twice.

If Apache Parquet used application/apache.parquet would it be appropriate to use application/apache.parquet; application=geoparquet, since geoparquet files are valid parquet files? Maybe it'd be weird to have application in the media type twice.

Not necessarily a helpful answer, but having followed a bit the discussions regarding COG (and not sure we managed to reach an endorsed conclusion), I find that understanding IANA rules for MIME types tends to require a dedicated expertise. For example https://www.rfc-editor.org/rfc/rfc6838.html#page-13 mentions " Media types MAY elect to use one or more media type parameters[...] the names, values, and meanings of any parameters MUST be fully specified when a media type is registered in the standards tree". So I guess that a provision for allowing application=geoparquet should be made at the time application/apache.parquet is registered (unless other rules just ban it or make it already possible...)

OGC contacted IANA to ask about adding parameters such as profile or application and you can't simply "add" them if the original type is already registered. So you need to discuss that with Apache upfront or otherwise register your own, e.g. application/geo+apache.parquet (although I'm not sure they would use a . in the name...)

Moving to beta.2. I think we should also try to get in touch with the core Parquet people and see if we can help them register something, even if it's just application/vnd.apache.parquet.
There was also general consensus on a recent call that we don't want a geo-specific parquet mimetype, we'd just use the general parquet one, and users would rely on the presence of geo metadata in the parquet file (or they could also guess that it'd be likely if there are shapefiles / geopackages of the same data). Happy for arguments on why we should have our own special geo one, and sounds like the time to do so is when apache applies for theirs. But I think we don't really want geospatial systems that distribute a non-geo parquet and a geoparquet. And we also hope that eventually all parquet readers would at least know to identify the standard geo data.

The same reasons why we want a COG and GeoTiff specific media type over just using "image/tiff" also applies here.
Example: STAC Browser would try to visualize a 100GB non-GeoTiff TIFF on a map, which is not a nice user experience.
So +1 on having at least geo-specific parameter...

I don't feel that strongly about this on either way, except that an optional parameter (like application/parquet; profile=geo or whatever) seems better to me than the geo+parquet, since then a generic parquet reader would have a better chance of actually opening it.
But I do think the time is 'now' to decide what we want, since as pointed out above the only real chance IANA seems to give for optional parameters is on registration. We can likely help the Apache people with the process, since OGC has experience working with IANA, and we can also just point them to the form to do a vnd registration - you just fill out https://www.iana.org/form/media-types. It does seem like there's precedent for a 'project steering committee' to submit for the official IANA types, with Apache Arrow, Thrift and Node all being submitted by the steering committees or a member on them. But @ogcscotts can likely help navigate the process / talk to the right people.
So seems like we should determine which direction we want to go, and if we want an optional parameter we should determine what we'd like, before engaging with them.

Yes, a parameter would be fine.

As @jorisvandenbossche mentioned in the meeting today, there's currently progress on Parquet getting a MIME/Media type: https://issues.apache.org/jira/browse/PARQUET-1889

Great, so application/vnd.apache.parquet is the intention, so I'll update my implementations to support both application/vnd.apache.parquet and application/x-parquet for now. Do we still intend to add a geo profile or so for GeoParquet?

If @cholmes is right in his above comment, we'd have to register a geo profile along with the current parquet registration?

Yes we would, if we go the official route. But we also never did with COG, and everyone just agreed on a de-facto standard of image/tiff; application=geotiff; profile=cloud-optimized

If @cholmes is right in his above comment, we'd have to register a geo profile along with the current parquet registration?

Yeah, if we want something listed in the official IANA then we'd need to do it. Like @m-mohr points out we can just add something on. With COG we wanted to get something registered but it basically wasn't an option.
So now is the time to try to advocate for it, if we want to. But I think we were leaning away from that, as mentioned in https://github.com/opengeospatial/geoparquet/issues/115#issuecomment-1289785839 I can try to bring it up at the next meeting, but if someone feels like we should push for a 'geo' profile then it'd be good to make the case here. I can't think of the use case where it's really essential, and it seems simpler for the file itself be the place to figure out if it's geo or not. And then not risk it being declared geo but not actually. And I don't see a case where it'd be good to have a non-geo parquet and a geoparquet version of the same file.

Same reason as for why COGs have a profile: A client can just detect easily what it is and whether it can render it without actually loading (parts of) the file. Think STAC Browser (and ol-stac, stac-layer, ...) for example... @cholmes

If a Parquet file is served with a media type that indicates that it is GeoParquet, a client cannot blindly try to render it, for example (the same is true for COG, despite what others may believe).  Before deciding what to do with the contents of a Parquet file, a client would need to read the footer - this is true for geo and non-geo Parquet.  After reading the footer, you can see if it has the geo metadata.
@m-mohr - can you provide more specific examples of what a client like STAC Browser would do if it knew that a Parquet file was GeoParquet?  If the answer is that it would display the geo-specific metadata, then this is going to require reading the footer of the file - which you can safely do for a non-geo Parquet file as well (and you might want to do anyway to show the user something about the data).

It's all about giving users the nicest user experience without loading a whole lot of headers upfront. Thinking more about it, it might be more relevant for COGs than it is for GeoParquet files though.
Example:
Having a STAC file with 12 COGs, I'd like to display a default COG for visualization purposes. If I'd just have image/tiff or the geotiff equivalent as media type, I'd need to read all headers of 12 COG files to know which I might be able to render. Having a specific media type I know upfront whether I can or can not render the 12 COG files. I know there are edge cases, but by default STAC Browser indeed tries to blindly render it, which works in many cases.

Looks like there's now a parquet media type, see https://github.com/opengeospatial/geoparquet/issues/115
Search 'parquet' on https://www.iana.org/assignments/media-types/media-types.xhtml
application/vnd.apache.parquet
I think they could have gotten application/parquet pretty easily, but this does seem consistent with the other apache ones.
I'm going to go ahead and make a PR without a 'geo' parquet media type - we can revisit and add it later if there is a lot of value.
I do wonder if there's a 'hint' we could give in STAC, for 'show on map'. I also do think it's not crazy to try to blindly render, as most parquet in STAC will likely be geoparquet.

I'd hope that in STAC people use the https://github.com/stac-extensions/table extension.
STAC Browser could also try to simply infer the "Show on map" from the table:primary_geometry.
So I'm not feeling as strongly anymore. So let's start with application/vnd.apache.parquet, indeed.

authors: TomAugspurger cholmes jorisvandenbossche kylebarron m-mohr rouault tschaub

repo: opengeospatial/geoparquet


text:
Get last timestamp for sensor

Ported a piece of TMPO code to the houseprint, it should be replaced as
soon as the last_timestamp method is pulled there.
Do sensor.last_timestamp() to get the end of the last TMPO-block that
is found for that sensor, or device.last_timestamp() to get the last timestamp for the device.
@WolfBerwouts I have also added a last_timestamp() method on the device level, so you can deal with inactive gas sensors by checking if their 'brothers and sisters' are also inactive or not.

Coverage decreased (-0.7%) to 68.795% when pulling 1b7a070069a2cffb1b8087610f71f4bdc84c1315 on JrtPec:get-last-timestamp-for-sensor into 2c695474e0b3bb6b19fb766eda315471f5fbe719 on opengridcc:develop.

Coverage decreased (-0.7%) to 68.795% when pulling 1b7a070069a2cffb1b8087610f71f4bdc84c1315 on JrtPec:get-last-timestamp-for-sensor into 2c695474e0b3bb6b19fb766eda315471f5fbe719 on opengridcc:develop.

authors: JrtPec coveralls

repo: opengridcc/opengrid


text:
Create Call for Speakers for Regular Meetup

membuat pengumuman + poster untuk orang-orang yg ingin berkontribusi menjadi pembicara di acara meetup bulanan

ini ticket untuk tulisan posternya dari ticket ini https://github.com/orgs/openinfraid/projects/3?pane=issue&itemId=17999467

repo: openinfraid/general


text:
Download JavaFX modules JavaDoc

Can the plugin download by default the JavaDoc for the required JavaFX modules?
I see it already downloads the sources.

You are right, it was IntelliJ. Using command line or NetBeans it doesn't happen.

Can be achieved using appropriate IDE plugin. Here is the example
apply plugin: 'java'
apply plugin: 'idea' // or 'eclipse'

idea {
    module {
        downloadJavadoc = true
        downloadSources = true
    }
}

authors: eugener jperedadnr

repo: openjfx/javafx-gradle-plugin


text:
Proposed template to gather info from projects and maintainers

Questionnaire and Metrics for OpenJS Foundation Projects
Questionnaire for Project Members
Project Details

Capture essential information about each project, including its name and overview. Document key milestones or achievements from the past year to understand development trajectory and successes.

Challenges and Solutions

Detail significant challenges faced and the strategies or solutions implemented to overcome these issues. This helps identify where the foundation can assist.

Future Outlook

Outline short-term and long-term goals. Helps the foundation understand project direction and support their path.

Community Engagement

Discuss methods and channels used for engaging with the community, including forums, social media, and other communication channels.

Contributor Analysis

Provide insights into the number and activity of contributors, tracking trends over the year.

User Feedback

Gather common praises or complaints from users to guide future development.

Foundation Support

Query about the support received from the OpenJS Foundation and additional support needed.

Ecosystem Improvement

Invite insights and suggestions on improving the overall JavaScript ecosystem.

Additional Insights

Allow projects to share other relevant information or experiences not covered in other sections.

Metrics for Measuring Project Status
Contribution Metrics

Monitor pull requests, issues, and commits to quantify project activity and community involvement.

Community Metrics

Track growth in contributors and followers, and subscriber metrics to assess popularity and reach.

Project Activity

Count frequency of updates and releases to gauge project dynamism and responsiveness.

Engagement Levels

Analyze activity in discussions and support channels to measure community health and communication effectiveness.

Adoption Metrics

Measure downloads, installations, and external mentions to quantify project adoption and relevance.

Diversity in Contributions

Evaluate contributions across various areas like documentation, code, and community management.

Survey on Maintainer Engagement
Engagement

Measure maintainers' satisfaction with the engagement level within the community.

Support

Assess the effectiveness of support provided by the foundation and identify areas needing additional resources.

Collaboration

Understand how well team members and external stakeholders collaborate.

Community Spirit

Describe the spirit or atmosphere within the project community, impacting volunteer retention and satisfaction.

cc @openjs-foundation/ecosystem-report
This is first proposal for the questionnaire that we would send to every project, the idea is to to discuss in this issue and finalize (if possible) in the meeting next week.

Hi @edsadr - I'd like to help with this and am wondering if we've picked a mechanism for providing the survey to members (Google Form, GitHub, other?)
GitHub would be interesting as it is open and can be directly correlated with the project, but can be more challenging to correlate the results. Google Form is easy to work with, but it requires that we have up-to-date email addresses for the project maintainers, and it is harder to track whether people received the request...
If the survey mechanism has been decided, let me know and I'm happy to create a draft for collaboration/review-
Thanks!

A Google form can be distributed via GitHub.

From today's call, there were suggestions to organize these metrics into those can be specifically measured / quantified, and those that are more subjective. Also, end user vs. maintainer/team metrics. In addition, provide some thoughts on which metrics would be actionable or lead to action vs. those that are just for information and open sharing.
Providing data vs. providing opinions was discussed, and avoiding becoming prescriptive or interpreting the data before we have consistent results.
The topic of how the metrics could be captured was discussed (e.g. use LFX Insights or create a bespoke metrics capture capability for the OpenJS Foundation using https://github.com/chaoss/grimoirelab which underlies LFX).
Perhaps we can help people understand what metrics really mean. What would 'green yellow red' really mean? Ensure people understand the value in numbers alone vs. trends over time. Creating a baseline would be an important step there.
Should we start small?
Attendees were asked to review and rank the metrics in this issue. These are for the project, not for the users. LFX also has project insights, and @rginn can connect us to the fact sheet. Note that not all projects opt in to metrics collection and we will need to work with projects directly in some (or many) cases.
@edsadr will mark the items here that can be collected directly form GitHub or other source by automation

@edsadr your set of questions were a great starting point for the discussion today. Using those as input these are a few of my thoughts:
Goals
Some of the goals that I think we discused as well as some thoughts I had while writing this down:

Help end users understand how Foundation projects fit into their plans
Share the state of each of the projects within the Foundation (active, emeritus)

For active projects share more detailed info

Include metrics that are important to users of a project, for example release cadence, last release etc.
Include current end user satisfaction, as well as pain points
Include sentiment/health of collaborators  including available resources versus resources available, and how the Foundation is helping projects

Provide a channel for end users to share their requests and pain points on Foundation projects
Share information to help build end user involvement and support of Foundation projects
Capture data to help the Foundation make its projects successful

And having read through the questions above, a the report might look like in terms of what I think would be interesting. It covers some of the same things you had above but as you asked for in the meeting I trimmed it down to the ones I thought were most interesting and tweaked into what I see supporting the goals we discussed.
Summary
Summarize interesting points from the data captured and shared across the projects
Foundation (from user survey)

Percentage of respondents who company are a member of the Foundation

X% of respondents are planning on Joining the foundation this year

Y% of respondents indicates that a project being part of the Foundation increases their likelihood of using a project

Most end users expect the following from the Foundation: a, b,c

End users also support projects by:

Projects
Project 1
Key project details

Created
Collaborators/contributors
Collaborator diversity
Last release
Number of releases this year

Adoption

Downloads or equivalent

Future Outlook

Planned Release cycle
Major plans for next year

Collaborator sentiment and resource needs

Does the project have the resources and people it needs to be successful
How good a place is the project to work in (community Spirit)
Resources currently provided by the Foundation
Satisfaction with support received by the Foundation
Major challenges

User Feedback (From User survey)

User happiness with current state (1-10)
Major Feature requests
Major pain points
Plan to use more, or plan to use less

Help Needed

Help needed by project
Ways to get involved
Ways to support the project financially

Based on our last meeting, I sent a PR to proposing the goals we discussed, please check it here: https://github.com/openjs-foundation/ecosystem-report-collab-space/pull/5

I also created this document specifying what metrics can be collected automatically and what will require manual effort, I would suggest we discuss this in a Google Drive Document here: https://docs.google.com/document/d/1tz1YHM_6NZs0bDfjvtT_ff1XkPBWHb_4otzhht5vi_o/edit#heading=h.u27qs6aj6wpc

authors: PaulaPaul edsadr ljharb mhdawson


text:
rollout support application full-link greyscale

What is full-link greyscale?
The community microservice governance solution MSE, Istio is able to support the full link grey scale capability of the business, as follows:

A new full-link greyscale solution
There are some differences between the full-link greyscale model and the regular Rollout model, the most significant of which is that the traffic greyscale should be triggered even if only application C is published. Due to these characteristics, the traffic grayscale should not be tied to a specific application, but should be shared by all applications.
The shared TrafficRouting yaml
You can start by defining a TrafficRouting resource that describes the rules for full-link greyscale, as follows to import traffic from the Android client into the greyscale environment.
apiVersion: rollouts.kruise.io/v1alpha1
kind: TrafficRouting
metadata:
  name: mse-traffic
spec:
  objectRef:
  - service: spring-cloud-a
    ingress:
      name: spring-cloud-a
      classType: mse
  strategy:
    matches:
    - headers:
      - type: RegularExpression
        name: x-user-agent
        value: 'andriod'

The application's rollouts yaml
For each Application (a, b, c), you can define the corresponding Rollout configuration, and you can add annotations[rollouts.kruise.io/trafficrouting] to statement the associated TrafficRouting.
---
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-a
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: spring-cloud-a
  strategy:
    canary:
      steps:
      - pause: {}
        replicas: 1
---
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-c
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: spring-cloud-c
  strategy:
    canary:
      steps:
      - pause: {}
        replicas: 1
---
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-b
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: Deployment
      name: spring-cloud-b
  strategy:
    canary:
      steps:
      - pause: {}
        replicas: 1

Some other features
1. TrafficRouting add RequestHeaderModifier field
Add grayscale traffic to header[x-mse-tag]=gray, in turn, the microservices governance solutions MSE and Isito can identify which traffic is base traffic and which is grey traffic.
// Set overwrites the request with the given header (name, value)
// before the action.
//
// Input:
//   GET /foo HTTP/1.1
//   my-header: foo
//
// requestHeaderModifier:
//   set:
//   - name: "my-header"
//     value: "bar"
//
// Output:
//   GET /foo HTTP/1.1
//   my-header: bar
//

apiVersion: rollouts.kruise.io/v1alpha1
kind: TrafficRouting
metadata:
  name: mse-traffic
spec:
  objectRef:
    - service: spring-cloud-a
      ingress:
        name: spring-cloud-a
        classType: mse
  strategy:
    ...
    requestHeaderModifier:
      set:
      - name: x-mse-tag
        value: gray

2. Rollout add addPodTemplate field.
Rollout will add labels[alicloud.service.tag]=gray in workload.spec.template, then the microservices governance solutions MSE and Isito can identify which pod is base version and which is grey version.
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-b
  annotations:
    rollouts.kruise.io/trafficrouting: mse-traffic
spec:
  objectRef:
    ...
  strategy:
    canary:
      steps:
      - pause: {}
        replicas: 1
      addPodTemplate:
        labels:
          alicloud.service.tag: gray

3. Rollout patch webhook label on workload
Currently rollout will webhook all workloads update requests, include Deployment, StatefulSet, CloneSet, Advanced StatefulSet. But not all workloads have to be rollout released, and if kruise rollout crashes will lead to workload update failed. So we can add configuration in MutatingWebhookConfiguration yaml.
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
...
  objectSelector:
      matchExpressions:
        - key: rollouts.kruise.io/workload-type
          operator: Exists

And the rollout controller patches the above label to the workload according to the Rollout objectRef.

Codecov Report

Merging #130 (3c1777b) into master (3578b39) will decrease coverage by 0.37%.
The diff coverage is 42.91%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##           master     #130      +/-   ##
==========================================
- Coverage   44.67%   44.31%   -0.37%
==========================================
  Files          48       49       +1
  Lines        5077     5186     +109
==========================================
+ Hits         2268     2298      +30
- Misses       2432     2508      +76
- Partials      377      380       +3

Flag
Coverage Δ

unittests
44.31% <42.91%> (-0.37%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/controller/rollout/rollout_status.go
17.00% <0.00%> (-2.38%)
:arrow_down:

pkg/util/constant.go
0.00% <0.00%> (ø)

pkg/util/controller_finder.go
0.00% <0.00%> (ø)

...ollout/validating/rollout_create_update_handler.go
63.56% <0.00%> (-0.50%)
:arrow_down:

pkg/trafficrouting/network/gateway/gateway.go
52.38% <7.14%> (-0.90%)
:arrow_down:

...chrelease/control/canarystyle/deployment/canary.go
55.95% <20.00%> (-5.39%)
:arrow_down:

pkg/controller/rollout/rollout_progressing.go
46.38% <35.00%> (-4.58%)
:arrow_down:

pkg/controller/rollout/rollout_canary.go
42.95% <40.00%> (+0.20%)
:arrow_up:

pkg/trafficrouting/manager.go
51.07% <60.00%> (+0.26%)
:arrow_up:

pkg/trafficrouting/network/ingress/ingress.go
63.33% <72.72%> (+1.79%)
:arrow_up:

... and 1 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter zmberg

repo: openkruise/rollouts


text:
Web doc improvment

adding wsgi-args https://github.com/openmicroscopy/openmicroscopy/pull/4352
mod_wsgi required version

cc: @manics

As discussed, could mention security issues with mod_wsgi <3.5 and indicate that all versions since 3.5 are supported.

is that better?

@will-moore can you sign off on whether this is good to merge please?

I think this is fine now. Good to merge.

Actually waiting on a code Pr being merged apparently, will hold off for now

just added troubleshooting as stated in source https://github.com/iuscommunity-pkg/python27-mod_wsgi/blob/master/SOURCES/python27-mod_wsgi.conf (due to changes in https://github.com/openmicroscopy/ome-documentation/pull/1348#issuecomment-163579266)

@hflynn as decided in https://github.com/openmicroscopy/openmicroscopy/pull/4352#issuecomment-163583469 this PR is not relaying on anything now.

@joshmoore confirms this is now as discussed, thanks.

authors: aleksandra-tarkowska hflynn will-moore

repo: openmicroscopy/ome-documentation


text:
API "Internal Server Error" on certain resource paths

Certain endpoints such as

https://api.opennem.org.au/station/
https://api.opennem.org.au/facility/

Return Internal Server Error with status code 500

Looks good on dev

repo: opennem/opennem


text:
Disable numba JIT for pytest coverage

JITted code is not coveraged by codecov, which is makes it look like we have a lot of untested code. Better to run the tests without JIT, imho.
Submission type

[X] This submission fixes a bug

Testing

[X] This submission was written on a forked copy of SingleCellOpenProblems
[X] GitHub Actions "Run Benchmark" tests are passing on this base branch of this pull request (include link to passed test: https://github.com/scottgigante-immunai/openproblems/runs/5589920207?check_suite_focus=true)
[X] If this pull request is not ready for review (including passing the "Run Benchmark" tests), I will open this PR as a draft (click on the down arrow next to the "Create Pull Request" button)

Submission guidelines

[X] This submission follows the guidelines in our Contributing document
[X] I have checked to ensure there aren't other open Pull Requests for the same update/change

PR review checklist
This PR will be evaluated on the basis of the following checks:

[X] The latest version of master is merged and tested

Codecov Report

Merging #337 (472c037) into main (5a9323f) will increase coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #337      +/-   ##
==========================================
+ Coverage   86.32%   86.36%   +0.04%
==========================================
  Files          77       77
  Lines        1696     1702       +6
  Branches      100      100
==========================================
+ Hits         1464     1470       +6
  Misses        201      201
  Partials       31       31

Flag
Coverage Δ

unittests
86.36% <ø> (+0.04%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

openproblems/tools/normalize.py
100.00% <0.00%> (ø)

openproblems/tools/decorators.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a9323f...472c037. Read the comment docs.

authors: codecov-commenter scottgigante-immunai

repo: openproblems-bio/openproblems


text:
Add pyrad in Docker installation

Hi everybody,
I was wondering if we could install pyrad on our binder for the course.
The command is
conda install -c conda-forge pyrad_mch (see https://anaconda.org/conda-forge/pyrad_mch)
Thanks a lot,
Cheers,
Daniel

We would need to add it to both the environment.yml files.

https://github.com/openradar/erad2022/blob/main/environment.yml
https://github.com/openradar/erad2022/blob/main/binder/environment.yml

Thanks @kmuehlbauer , should I do a PR?

Yes, that would be OK. You are using this in the pyrad notebooks, right?

@kmuehlbauer , yes, I will show a pyrad example at the end of the pyart notebooks.

Then it should added, otherwise it won't work.

solved by #46

authors: kmuehlbauer wolfidan


text:
Add Control Flow visualization utilities

Add more dataflow tests
Add Control Flow Java printer

Do we agree we can closet his in favor of dot-format emitting markers and punting visualization a bit?

repo: openrewrite/rewrite


text:
First pass at fixing linter errors

This PR tackles the first bunch of linter errors in the code. To tackle the problem I have temporarily disabled the unused stylecheck and whitespace. This lets us focus on the security and programming errors while leaving the style checks for later.
Tests are completely broken, they don'r really work and some of them even panic. I will try to focus on them after this PR merges.
/cc @cpanato

Closed as @cpanato has a more complete version of this. Sorry :\

repo: opensbom-generator/parsers


text:
Advertise that AIO is not supported

Fixes a crash on 5.3.0 with xrootd.async enabled  (default).

OK, that's why I said "optional".
On Thu, 15 Jul 2021, Matev? Tadel wrote:

@osschar commented on this pull request.

@@ -752,6 +753,7 @@ int XrdHdfsFile::Write(XrdSfsAio *aiop)
{

// Execute this request in a synchronous fashion
+// Will not be called as XrdHdfsSys::Features() advertises no AIO support.
Hmmh, I think it will only confuse people ... it does confuse me :)
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/opensciencegrid/xrootd-hdfs/pull/37#discussion_r670769783

repo: opensciencegrid/xrootd-hdfs


text:
use a keystone installed in a container

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
use a keystone installed in a container
Special notes for your reviewer:
Release note:

use a keystone installed in a container

Coverage remained the same at 37.948% when pulling de5ed28216ccb06c9d351ccd1549c961123e3f3d on BaiHuoYu:u_k_c_328 into b9cd3fe65a676e8b6fc1d507bcacfb8f1b242618 on opensds:master.

Coverage remained the same at 37.948% when pulling de5ed28216ccb06c9d351ccd1549c961123e3f3d on BaiHuoYu:u_k_c_328 into b9cd3fe65a676e8b6fc1d507bcacfb8f1b242618 on opensds:master.

Coverage remained the same at 37.948% when pulling de5ed28216ccb06c9d351ccd1549c961123e3f3d on BaiHuoYu:u_k_c_328 into b9cd3fe65a676e8b6fc1d507bcacfb8f1b242618 on opensds:master.

Can u also update it in the opensds-installer?

/retest

authors: BaiHuoYu coveralls leonwanghui satya-gorli

repo: opensds/opensds


text:
Merge development into master branch for publishing Capri RC-03

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

@wisererik , please wait for dashboard merge. Anything else?

authors: skdwriting wisererik


text:
[BUG] org.opensearch.common.settings.SettingTests.testMemorySizeWithFallbackValue is flaky

Describe the bug
org.opensearch.common.settings.SettingTests.testMemorySizeWithFallbackValue is flaky.
Related component
Other
To Reproduce
JvmInfo.jvmInfo().getMem().getHeapMax().getBytes()=8.0gb
./gradlew ':server:test' --tests "org.opensearch.common.settings.SettingTests.testMemorySizeWithFallbackValue" -Dtests.seed=B6AA005F20EE7928 -Dtests.security.manager=true -Dtests.jvms=1 -Dtests.heap.size=1g -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sq -Dtests.timezone=Africa/Asmera -Druntime.java=17
if JvmInfo.jvmInfo().getMem().getHeapMax().getBytes()=8gb, (JvmInfo.jvmInfo().getMem().getHeapMax().getBytes() * 0.3) > Integex.MAX, then (long)((int) value) will lose precision.
https://github.com/opensearch-project/OpenSearch/blob/b4da802a56f6ec9ed6f57104d94a211919504b34/server/src/test/java/org/opensearch/common/settings/SettingTests.java#L222
Expected behavior
Test should pass
Additional Details
Screenshots

[Triage - attendees 1 2 3 4 5 6]
@kkewwei Thanks for filing. Looking forward to seeing the fix merged.

authors: andrross kkewwei

repo: opensearch-project/OpenSearch


text:
Feature Proposal : Pluggable Translog

Overview
The purpose of this document is to propose various mechanisms of achieving durability of uncommitted indexing operations and outlines the advantages of making the implementation pluggable. This is not a complete design document, it is a proposal to seek community feedback on.
Motivation

Provide flexible user controlled durability options
Support for highly durable translogs

Background
What is the role of a translog?
All operations that affect an OpenSearch index (i.e. adding, updating, or removing documents) are added to an in-memory buffer, which is periodically flushed to create segments on disk on a Lucene commit. These operations also need to be durably written to a write-ahead transaction log (aka translog) before they are acknowledged back to the client since Lucene commits are too expensive to be performed per request. In the event of a process crash, recent operations that have been acknowledged but not yet included in the last Lucene commit are recovered from the translog.
The following are the various mechanisms through which the engine interacts with the translog—

Ingestion Interactions
For efficiency, the translog operations are first buffered to an in-memory queue so that not all the indexing threads have to perform the costly fsync operation ** to disk. One of the indexing threads acquire a lock to dequeue and subsequently fsync, the buffered translog operations to disk. Once the fsync operation is complete, for operation consistency, OpenSearch also writes a checkpoint to the disk with the last sequence number that is written to the translog, which helps prevent translog corruption. Once all translog operations for a given request have been durably persisted to disk, the OpenSearch sends an acknowledgment back to the client
When the translog grows beyond a configurable size, in order to prevent recoveries from taking too long, the OpenSearch engine triggers a flush in the background. OpenSearch flush performs a Lucene commit, moving segments to disk also rolling over and starting a new translog file.

Shard Recovery
In the event of a crash, existing segment files are first recovered from the local disk. Translog files are opened and recent operations that have not been committed to Lucene are recovered by replaying all operations from the translog beyond the sequence number from the last Lucene commit checkpoint. A flush is triggered creating Lucene segments on disk and purging all unreferenced translog files.
Recovery mechanisms

Local recovery/ Primary failover : In case of recovery from an existing store(has in-sync translog and segments locally on disk) all uncommitted operations are replayed from local translog on disk.
Peer recovery : Translog doesn’t play a role in peer recovery instead peer shard copy(primary-primary or primary-replica) is orchestrated via shard retention leases by replaying indexing operations that are indexed into Lucene and deleted operations from the preserved soft-deletes on the peer primary.

Cross cluster replication
Cross Cluster Replication in OpenSearch uses a pull-based replication model to pull translog operations into the follower index. The translog in the leader cluster is pulled incrementally from the current sequence number of the follower shard upto the global checkpoint of the shard being replicated from at the leader, once the bootstrap from committed segments are complete.

Downsides of current translog implementation

Less flexible : The engine is tightly coupled with local disk based translogs and forces users to heavily rely on them for durability
Lesser durability options: The current option supported for translog persistence is local disk which makes full-recovery impossible even with faster snapshots in the event of losing a single node in a non-replicated setup.
Slow Ingestion :  The present primary-backup replication model requires translogs to be synchronously written to all shard copies in the replication group before acknowledging a write request. There is a possibility that one degraded node can stall ingestion. With remote store and segment based replication model, we do not need to replicate operations to all copies synchronously
Costly : Users achieve high durability by replicating translogs on additional shard copies, even if that copy isn’t needed for read availability

Use Cases
The pluggable translog support is meant to cater to a broad spectrum of users who would want to customise their durability options based on specific needs.

The translogs are essential for durability but not the only way to achieve the same. Some users might choose not to use a translog at all, as they might want to self-manage durability by periodically checkpointing indexing requests and making calls to commit Lucene segments to disk. In the event of a primary node crash, all uncommitted changes could be re-driven from the previous checkpoint which is what Yelp does today. The commits can alternatively be made at the end of a batch of requests controlled by the user.
Users might want to choose a separate durability semantic based on document search-ability(refresh). So all changes since the last refresh which haven’t been made visible to search is also not guaranteed to persist
Users might need high durability guarantees on their data that can be achieved by configuring a remote storage for persisting translogs in addition to keeping segments.
In future we might want to support streaming APIs which would trigger an explicit commit on events like connection close

High Level Proposal
We propose decoupling translog from the engine by abstracting out translog code from the server code base and moving it to a separate module. The engine would provide knobs to choose the translog option, whether or not translogs need to be maintained by the system. The translog module would further support extension points for a remote store which would be implemented as separate extended plugins. The translog module would support the default store as the local store so that the code is fully backward compatible, while providing users an option to customize their own durability by installing the specific durability extension plugins. All IO interactions across different storage interactions would be checksummed to detect or prevent translog corruption.
Translog durability

Local : This is same as the durability option we have today, translogs will be co-located on the local disk with the segments and would behave the same for both segment and document based replication i.e all indexing operation would require translog to be replicated synchronously within the shard replication group. For a non-replicated index, incase of a node failure, shards cannot be recovered on a different node.

Remote : Translogs can be stored on a remote store as provided by the durability extension plugin, in which case we don’t need translogs to be redundantly written on replicas. The indexing operations in case of segment based replication, need not even be replicated to the replica as primary is responsible for both segment creation and translog persistence to a durable store
Local recovery process will need to pull translogs from the remote store and replay the needed translog operations once segments have been restored.
For a non-replicated index, incase of a node failure, shards can be recovered on a different node if segments are also present on the remote store, by first restoring the segments and then pulling the translogs and re-playing the missing operations

No translog : Users can choose to call commit periodically to make recent changes durable on disk.  In the event of a primary node failure all changes since the previous Lucene commit would be lost and would need to re-driven.

Low-level proposal

Future Work

Support for request checkpointing For users who wouldn’t want to choose translogs for durability, we would need to support additional durability semantics based on checkpoints. We would need to add support for checkpointing requests that might need API changes(backward compatible) to return a monotonically increasing seq no corresponding to the last indexing operation (per shard). The same seqno would be made available via APIs like flush to return seqno corresponding to the last Lucene commit. Indexing requests beyond the last successful commit checkpoint can be safely purged.
Performance benchmarks with remote stores The remote translog durability options could have an indexing performance impact. We would be benchmarking indexing performance with the supported durability extensions

FAQs
What is a Lucene commit?
Lucene commit is a process of flushing all pending changes (added & deleted documents, segment merges, added indexes, etc.) to the index, and syncing all referenced index files. The data on disk is ready to be used for searching. and the index updates will survive an OS or machine crash or power loss.
What are shard retention leases?
Shard retention leases is a mechanism used for peer recoveries, aimed at preventing shard recoveries from having to fallback to expensive file copy operations if shard history is available from a certain sequence number. This lease would be acquired by a recovering replica copy to prevent any operations on the primary beyond this sequence number from being merged away
What is a soft-delete?
Lucene has a functionality to keep deleted documents alive based on time or any other constraint in the index. The soft delete merge policy allows to control when soft deletes are claimed by merges.

3. No translog : Users can choose to call commit periodically to make recent changes durable on disk.  In the event of a primary node failure all changes since the previous Lucene commit would be lost and would need to re-driven.

I'm curious what other folks think about this. I know some are strongly opposed giving the option to disable this level of persistence. I'm not in that camp. I like the idea of giving the option to completely disable the xlog and save that processing storage time for simple fast/unreliable uses cases. Similar to folks choosing UDP over TCP and not caring if certain packets are lost.

As of now translog.add(..) happens in the same flow of indexing the doc - here .
Do we want to provide control over when to add to translog ? If yes, we can provide functionality to register custom listeners which will be called in index method, and it's upto the listener if it want to add translog sync or async.

@jayeshathila I have updated the the sequence diagram for translog interaction, for better understanding of the existing system, would recommend you go through the same

Will this Path mentioned in TranslogDurabilityExtension (LLD diagram above) , work generically accross different Stores. For say, do we want to use the same Path for S3 files as well ?
I know this Path have method to map a URI public static Path of(URI uri) but not sure if  it will work for all the remote filesystems.

For my understanding, who (which class) will be caller to loadExtenstions(Class<T>) ?

Sub-tasks for this Meta Issue
https://github.com/opensearch-project/OpenSearch/issues/3241
https://github.com/opensearch-project/OpenSearch/issues/3600
https://github.com/opensearch-project/OpenSearch/issues/3601
https://github.com/opensearch-project/OpenSearch/issues/3599

@Bukhtawar I saw this didn't have the 2.1 tag. Is this still targeting 2.1?

I'm pretty interested in this feature primarily for the pluggable transport option. I want the capability to encrypt the translog byte stream with different keys (different indexes must have different encryption keys). The different keys per index requirement is what pushes me away from using EBS; mounting different EBS volumes per index doesn't make sense for my use-case either due to volume and variability of indexes/keys.
@Bukhtawar Looking at the diagram and description, it seems like AbstractStore and the remote store idea is more about a backing store that is used to copy the data locally then read it from the local filesystem when needed. Is that a correct interpretation? Can we ensure the interfaces provided for the plugin are able to be used to layer read/write operations from the store? For example: remote-store-to-memory or data encryption purpose (plugin would prevent unencrypted translog data from hitting disk)
+1 to that
I am also interested in extending the translog and potentially filter through all the input/output file system bytestreams.
It would be great if we had the same level of abstraction to input/output byte streams like we have for FSDirectory in Lucene for example.

@Bukhtawar @rramachand21 Are this tracking green for 2.9?

authors: Bukhtawar anasalkouz elfisher jayeshathila nknize samuel-oci


text:
[RFC] Introduce pluggable QueryCollectorContexts and CollectorManagers

TL;DR
I am proposing a way to separate out query preprocessing from query path execution in the existing SearchPlugin's QueryPhaseSearcher extension points. This will allow multiple SearchPlugins to provide Collector/CollectorManager related customization which is a limitation of the QueryPhaseSearcher today.
Background Information
Today the only extension point to plug in any search customization in the query phase is the QueryPhaseSearcher interface defined in the SearchPlugin. This requires any plugins that want to add customization into the query phase or even introduce additional Collectors implement an entire new QueryPhaseSearcher, and only 1 plugin's QueryPhaseSearcher can be used at a time.
Specifically, the main entry point is the searchWith method https://github.com/opensearch-project/OpenSearch/blob/293905af45b8eabb751fe1d89cde3bfcf3d54014/server/src/main/java/org/opensearch/search/query/QueryPhaseSearcher.java#L39-L46
which allows an implementation to [1] customize how the search path is executed using the ContextIndexSearcher (for example the ConcurrentQueryPhaseSearcher is used to enable concurrent search) as well as [2] perform any preprocessing or modifications on the Query, SearchContext, or even the QueryCollectorContexts linked list. However, because of the broad possibility of implementations here, today we are limited to a single QueryPhaseSearcher implementation per cluster. In this past this limitation created a conflict between the ConcurrentQueryPhaseSearcher and the HybridQueryPhaseSearcher which had to be resolved by making the latter extend the (wrapper class of) the former: https://github.com/opensearch-project/neural-search/pull/356
In the first case [1] where a QueryPhaseSearcher is used to modify the search execution path there is not much we can do to support multiple QueryPhaseSearchers as ultimately we need to decide on only a single execution path to perform. For example, we can’t use the DefaultQueryPhaseSearcher and the ConcurrentQueryPhaseSearcher together because we can’t perform both concurrent and non-concurrent search (or at least it doesn’t make sense to).
However, for the second case [2] such as the HybridQueryPhaseSearcher where the searchWith extension point is not used to modify the search execution path but instead modify the Query on which to perform search, multiple preprocessing implementations should be able to be composed together.
Additionally the aggregationProcessor method https://github.com/opensearch-project/OpenSearch/blob/293905af45b8eabb751fe1d89cde3bfcf3d54014/server/src/main/java/org/opensearch/search/query/QueryPhaseSearcher.java#L53-L55
is used to setup and post process aggregation related collectors however, today it is also the only place which can be used to introduce custom CollectorManager implementations. For example, the HybridAggregationProcessor uses this extension point to introduce the HybridCollectorManager without making any modifications to the default AggregationProcessor. Since the AggregationProcessor is introduced in the same QueryPhaseSearcher interface, the same limitation exists today where only a single AggregationProcessor can be used per cluster. However, in cases like this where a plugin strictly wants to add additional CollectorManagers to the queryCollectorManagers map there are no conflicts across plugin implementations so we should be able to support an additional extension point here to allow SearchPlugins to provide custom CollectorManagers.
Describe the solution you'd like
At a high level I am proposing to decouple the query preprocessing logic (performing any processing or modifications with Query, SearchContext, and QueryCollectorContexts) from the search execution path logic. Each of the Query, SearchContext, and QueryCollectorContexts could have their own distinct pluggable extension points that allow multiple plugins to provide implementations. Each of these components deserve their own separate RFC to discuss how to best proceed so this issue will focus on making QueryCollectorContext and associated CollectorManagers pluggable.
Today a LinkedList of QueryCollectorContexts is created in QueryPhase::executeInternal with various default QueryCollectorContexts based on the query parameters (such as terminate_after, min_score, etc.). It is subsequently passed to the previously mentioned QueryPhaseSearcher::searchWith method where any plugin customization can happen and this is also where it is usually transformed from the QueryCollectorContexts into the Collector tree and passed to IndexSearcher::search to perform the query phase.
I am proposing to introduce a QueryCollectorContextProvider construct which can take pluggable QueryCollectorContext constructors to override default contexts.
I see 2 use cases for this today:

As a performance optimization for the hybrid query some custom logic was introduced to allow excluding the TopDocsCollector. This was done by adding an additional SearchWithCollector method which is pretty fragile and still tightly coupled to the QueryPhaseSearcher. This could be done instead by overriding the default TopDocsQueryCollectorContext with an empty context based on whatever condition checks need to be performed. Ref: https://github.com/opensearch-project/OpenSearch/pull/13481.
As described in https://github.com/opensearch-project/OpenSearch/issues/13806, one of the warm index search performance optimizations is to perform prefetch on the remote store backed indices and a key component of this may be injecting prefetch logic into the Collector tree. This could involve taking the Aggregator tree and wrapping it in a PrefetchableCollector type of construct to capture the doc IDs on which to perform prefetch related actions. This could be accomplished by overriding the default search_multi QueryCollectorContext via this new extension point.

Moreover, I am also proposing to add a plugin hook to allow SearchPlugins to add implementations directly to the queryCollectorManagers map. Although custom CollectorManagers could be plugged in with the above proposal of overriding the QueryCollectorContext implementation, this would still have the same limitation of only 1 plugin implementation at a time. CollectorManagers do not need to be bound by this restriction and we can open up an extension point to allow as many pluggable CollectorManagers as needed.
I also see 2 use cases for this today:

As previously mentioned, the HybridAggregationProcessor simply adds a HybridCollectorManager to the queryCollectorManagers map however today is has to implement a whole AggregationProcessor just to do so despite not making any changes to the default AggregationCollectorManagers.
Some users may want a way to plug in custom Lucene Collector implementations: https://opensearch.slack.com/archives/C051JEH8MNU/p1715869554173169

Describe alternatives you've considered
Instead of an extension point for overriding default QueryCollectorContexts as well as an extension point for the queryCollectorManagers map, we could instead introduce a notion of priority to QueryCollectorContext and allow plugins to provide their own QueryCollectorContext implementations with priority, similar to how ActionFilters work today. All QueryCollectorContext implementations would then be chained together based on their priority when transforming the QueryCollectorContext into the Collector tree. Rough POC: https://github.com/opensearch-project/OpenSearch/compare/main...jed326:OpenSearch:ordered-context
This allows plugins to provide an uncapped number of QueryCollectorContext implementations, however since the order of the contexts will determine the structure of the Collector tree this means the context priority will be tightly coupled with it’s functionality and each QueryCollectorContext would likely need to have many instanceof checks to ensure it is performing its intended function, similar to how BucketCollectorProcessor is implemented today: https://github.com/opensearch-project/OpenSearch/blob/618782d8fe8c26298caf912795513b23c33db149/server/src/main/java/org/opensearch/search/aggregations/BucketCollectorProcessor.java#L60-L86

Tagging some individuals for thoughts: @sohami @msfroh @mch2 @kasundra07 @reta
Also, I believe this should at least partly address the concerns raised in https://github.com/opensearch-project/OpenSearch/issues/13320, tagging @conggguan and @martin-gaievski for some thoughts on this as well.

[Triage - attendees 1 2 3 4 5 6 7]
@jed326 Thanks for creating this RFC, look forward to seeing how this pans out.