text:
proxy.golang.org: https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.6.zip: 403 Forbidden

What version of Go are you using (go version)?

go version go1.17.3 windows/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Master\AppData\Local\go-build
set GOENV=C:\Users\Master\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\Master\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Master\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.17.3
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\Master\go\src\mongoAPI\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\Master\AppData\Local\Temp\go-build3367626320=/tmp/go-build -gno-record-gcc-switches

What did you do?

I've tried to download mongoDB golang driver to a project in VS code running this command in the VS code terminal:
go get go.mongodb.org/mongo-driver/mongo
but I got a 403 forbidden error like this:
go: downloading github.com/klauspost/compress v1.13.6 go.mongodb.org/mongo-driver/mongo imports        github.com/klauspost/compress/zstd: github.com/klauspost/compress@v1.13.6: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.6.zip: 403 Forbidden
I've figured maybe my ip address is restriced so I've tried downloading it using several VPNs like psiphon and hotspot shield but I'm still getting the same error.
What did you expect to see?
go: downloading github.com/klauspost/compress v1.13.6
getting the package without any error
What did you see instead?
go: downloading github.com/klauspost/compress v1.13.6 go.mongodb.org/mongo-driver/mongo imports         go.mongodb.org/mongo-driver/x/mongo/driver imports         github.com/klauspost/compress/zstd: github.com/klauspost/compress@v1.13.6: reading https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.6.zip: 403 Forbidden

where are you/what is your IP address?

my IP address is 192.168.15.100 @seankhliao

what does https://ifconfig.co/ say your ip address is?

this is the output of running ipconfig in cmd
`Ethernet adapter Ethernet:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix  . : tishreen.com
Unknown adapter Local Area Connection:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix  . :
Wireless LAN adapter Local Area Connection* 10:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix  . :
Wireless LAN adapter Wi-Fi:
Connection-specific DNS Suffix  . :
Link-local IPv6 Address . . . . . : fe80::a993:2c0:b720:92c6%6
IPv4 Address. . . . . . . . . . . : 192.168.15.100
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.15.1`

Those are private network addresses, not the ones used to reach the proxy servers.
Click the link I gave above and tell me that address

this is my ip address using the link
2001:67c:2660:425:1b::125

@siba36 is it still reproducible?

yes I couldn't figure out the solution to this error @cherrymui

cc @katiehockman @heschi @hyangah

Is this a GCP host? If so you may be seeing #49144. Otherwise, please visit that URL directly and see if the page gives you any more information. You must be behind some kind of proxy or network gateway.

I am having the same problem installing packages from my Linode server.

https://proxy.golang.org/github.com/klauspost/compress/@v/v1.13.4.zip: 403 Forbidden
I using go version go1.18.1 windows/amd64

I have this problem too. The IP addresses of the data center (which I have a VPN tunnel through) are blacklisted. But I don't know why? The data center is located in Germany.

If someone affected is willing to post their IP address I can try to see if there's some kind of blocking on our side that I'm not aware of.

@heschi My IP address is 162.55.230.234

@Lavaei sorry for the late response. The proxy.golang.org service is available in most countries and regions. However, Google restricts access to some of its services in certain countries or regions, such as Crimea, Cuba, the so-called Donetsk People's Republic and Luhansk People's Republic, Iran, North Korea, and Syria.

Set the environment variable GOPROXY=direct and then install it

authors: Lavaei cherrymui gastonbarboza heschi hyangah jamalkaksouri seankhliao siba36 smzm


text:
proposal: x/exp/slices: add First and Last

The pattern s[len(s)-1] appears fairly often.
However, s[len(s)-1] is neither readable nor easy to type,
especially if s is a relatively complex expression.
Thus, I propose the addition of slices.Last that
retrieves the last elements, respectively.
I also propose adding slices.First for consistency.
Supporting evidence
Frequency of constant slice indexes relative to start or end:

Index
Occurrences
Percentage

0
4953854
41.684%

1
2192516
18.449%

2
1135707
 9.556%

3
1370480
11.532%

4
521199
 4.386%

other
1391696
11.710%

-5
196
 0.002%

-4
443
 0.004%

-3
1682
 0.014%

-2
10336
 0.087%

-1
306168
 2.576%

Even though obtaining the last element is an order-of-magnitude
less frequent than getting the first element,
there are still ~300k occurences.
Of the number of cases where s[len(s)-1] appears,
it appears alongside s[0] 78% of the time.
For example:
minTime, _ := extractTimeRange(lastShard.spans[0])
_, maxTime := extractTimeRange(lastShard.spans[len(lastShard.spans)-1])

For this reason, I propose the addition of slices.First,
so that these case will read naturally:
minTime, _ := extractTimeRange(slices.First(lastShard.spans))
_, maxTime := extractTimeRange(slices.Last(lastShard.spans))

Considerations
This API does not help the following statement:
s[len(s)-1] = ...

since the returned value of slices.Last cannot be assigned to.
This occurs 32344 times (10.6%).
It also does not help the following expression:
&s[len(s)-1]

since you cannot take the address of the return value of slice.Last.
This occurs 8102 times (2.6%).
To support these 13% of cases, we could have slices.Last
return a pointer to the element, so that you could do:
*slices.Last(s) = ...

However, that is clunky to use.
Alternatives
Last is unhelpful if you want the penultimate element (i.e., s[len(s)-2]),
but there are an order-of-magnitude fewer use-cases for that compared
to just getting the last element.
We could consider a Python-like indexing operation with negative indexing:
// At retrieves the element at i.
// If i is negative, then it is indexed relative to the end.
func At[S ~[]E, E any](s S, i int) E

This provides a general-purpose way to retrieve elements relative to the end.
Negative indexing has been proposed many times in the past for slicing and was rejected
because it is easy for programming bug to go unnoticed.
Using an explicit API like slices.At could be clear enough signal that we truly want
negative indexing. Perhaps that is an appropriate trade-off.
I personally would like to have At as well,
but the evidence does not seem to indicate that it is as prevalent
compared to the need for Last.
\cc @ianlancetaylor @eliben

Should First and Last panic for an empty slice?

Should First and Last panic for an empty slice?

I would expect so, in the same way that s[0] panics for an empty slice.

Thanks for highlighting #25594. I like the solution that it proposes, but it seems that similar languages changes like it have been proposed multiple times and rejected. Barring a language change, perhaps a library change is more palatable. I would argue that the frequency that similar proposals keep getting filed indicates that this is a real pain point for users that is sufficiently common.

IMHO [-1] would be better.

Not only is it better for readability, but it also resolves the inability for slices.Last to be addressed or assigned to.

What about slices.At(s []T, n int) *T? It can accept -1 for last, it returns an address, and it doesn’t have to panic if the n is out of range. If you know your slice is going to be long enough, you can just dereference without testing for nil first.

What about slices.At(s []T, n int) *T?

'At' would almost always only be used with negative indices, because s[i] is clearer for the other cases, so why not a 'FromEnd' function:
slices.FromEnd(s, 0) // last element
slices.FromEnd(s, 1) // penultimate element
...

This feels like it starts to fight the language.
Surely we don't want to get into arguments about whether to write s[0] or slices.First(s).
And similarly s[len(s)-1] is a clear, common idiom.
I'm not sure that we need a function for every idiom we have, even common ones.
This doesn't seem like a win, and I worry about the slices package getting bloated.
/cc @robpike

The comment that this is turning an idiom into a function is a good one. The addition of slices.First is unnecessary, more verbose than the idiom, except to complement slices.Last. So is Last necessary?
It can help with the slice expression is messy, but many things we do in any programming language get messy when the expression is messy. In other words, it doesn't add functionality, only convenience, and that requires a high bar that Last might not meet.
And once we have First and Last, there will be calls for Index and Nth and Last2 and LastN and so on. It's a can you might not want to open.
The philosophical question is: do we want to turn the language into this? We could have had maps and slices and so on be run with functions or methods, but we didn't. It would be sad to wake up one day and find many uses of slices eschewing the index notation.

When you're dealing with a short local variable, the s[len(s)-1] idiom is totally fine. But it can get out of hand when accessing nested fields.
some.Nested.Values[len(some.Nested.Values)-1]

vs
slices.At(some.Nested.Values, -1)

@icholy You can always assign a long expression to a short local variable.
v := some.Nested.Values
v[len(v)-1]

You can always assign a long expression to a short local variable.

That doesn't work if you're trying to assign into that expression and it isn't addressable.

You can always assign a long expression to a short local variable.

That doesn't work if you're trying to assign into that expression and it isn't addressable.

@dsnet Isn't the basis of this issue that the expression is a slice?

No change in consensus, so declined.
— rsc for the proposal review group

source: gharchive/issue

authors: carlmjohnson dsnet fzipp ianlancetaylor icholy robpike rsc

repo: golang/go

license_type: permissive

license_source: github-api


text:
cmd/link: rare corruption of ELF binaries

What version of Go are you using (go version)?

$ go version
go version go1.18 linux/amd64

Does this issue reproduce with the latest release?
Yes, Go 1.18 is the latest (stable) release.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michael/.cache/go-build"
GOENV="/home/michael/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/michael/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/michael/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/michael/sdk/go1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/michael/sdk/go1.18/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/michael/2022-06-25-squashfs-debug/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1868362460=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I am building and deploying Go software from a cron job every day.
Recently, I noticed that sometimes, some of my executable binary files do not start up because they are corrupt!
The first time I noticed the issue, the init binary of one of my https://gokrazy.org/ installations was affected, resulting in an installation that wouldn’t boot at all.
The other time, it wasn’t the init binary, but a program of mine called regelwerk which is involved in motion sensor/light control in my home, so I noticed that because the lights weren’t working as they should.
It’s possible this happened more times and I just didn’t notice it.
Yesterday, I found someone on twitter who is also running into this issue, but with an entirely different program (not related to gokrazy at all): https://twitter.com/alvs_versteck/status/1546601648532983808
What did you expect to see?
The Go compiler/linker should produce ELF binaries that contain a valid ELF header.
What did you see instead?
The first 4096 bytes of the ELF binary are zeroed out, as well as another block of 4096 bytes at offset 256K.
You can find the files at https://t.zekjur.net/_2022-06-25-init/
In the other occurrence, it was 4096 bytes at the start of the ELF binary, then 4096 bytes at offset 0x9000.
Unfortunately I have no idea how to reproduce this issue.

Are the corrupted binaries deterministic?
If not, I wonder if #51611 is related.

Are the corrupted binaries deterministic?

WDYM by that, if they are always corrupted in the same way? If yes, I think so but currently only have one sample lying around where this happened so would need to wait for it to happen again to be able to tell for sure.

CC @golang/compiler

Is the corruption always on the ELF header?  Is it on the same machine or on multiple machines? What the file system are you running on? Thanks.

Is the corruption always on the ELF header?

I think so, I remember I got a couple of cannot execute binary file: Exec format error errors and rebuilding helped. I only have one lying around right now though and it happens intermittently, so I can't say for sure until it happens again.

Is it on the same machine or on multiple machines?

Multiple, mine and @stapelberg also sees it on his

What the file system are you running on?

xfs

Thanks. Frequent corruption on the ELF header is probably enough information. The ELF header is written by the linker only. The compiler  and (most of) the Go command are unrelated. The one place that the go command touches the binary after linking is stamping the build ID. So it is either  the linker or the that build ID stamping.
Is your program a pure-Go binary or it uses cgo (i.e. whether it is internal linking or external linking)?

Is your program a pure-Go binary or it uses cgo (i.e. whether it is internal linking or external linking)?

Mine is pure go (i.E. CGO_ENABLED=0)

4096 = block size of ext4, file system bug?

4096 = block size of ext4, file system bug?

Again: note that both the source code (generated) and the binary output location are on /tmp, which is a tmpfs mount in my case.
So I don’t think a file system bug is likely here.

Thanks. Pure-Go means that the file is written by the Go linker, not the C linker.
If this is somewhat reproducible, could you try if this patch makes any difference? (You'll need to rebuild the linker,  go install cmd/link)
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
        "syscall"
+       "unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
        if out.buf == nil {
                return
        }
+       syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
        syscall.Munmap(out.buf)
        out.buf = nil
 }

Also, do you know if this is a new bug, or it occurs with old versions of Go as well? Thanks.

@cherrymui I haven't seen this before using go 1.18, so I guess it is new. I did some more experimenting and found out:

I can somewhat reliably reproduce this by go building in a loop, in five attempts it always took less than 200 builds
It is not reproducible in a docker container, so seems specific to my env
It is reproducible with an empty go cache
It is not reproducible with a hello world go program, so seems to either be specific to my project or maybe just the size of the codebase

Regarding your patch, I suppose I need to replace the link binary in the GOTOOLDIR? I did that and the issue still happens. I then looked more at the link binary and it has the same sha256sum regardless of applying your patch or not...?
$ g diff
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
        "syscall"
+       "unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
        if out.buf == nil {
                return
        }
+       syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
        syscall.Munmap(out.buf)
        out.buf = nil
 }
[2022-07-13-23:17]:[app.ci@ci][master][1]~/.../goroot/src
$ go build -o /tmp/link cmd/link/
[2022-07-13-23:17]:[app.ci@ci][master][1]~/.../goroot/src
$ sha256sum /tmp/link
e7e671ba4d83e6f3a820ba3f0269ef21e26f2c477938499d389f58a696ec7e19  /tmp/link
[2022-07-13-23:17]:[app.ci@ci][master][1]~/.../goroot/src
$ g reset --hard
HEAD is now at 558785a0a9 cmd/compile: remove -installsuffix flag
[2022-07-13-23:17]:[app.ci@ci][master][1]~/.../goroot/src
$ go build -o /tmp/link cmd/link/
[2022-07-13-23:17]:[app.ci@ci][master][1]~/.../goroot/src
$ sha256sum /tmp/link
e7e671ba4d83e6f3a820ba3f0269ef21e26f2c477938499d389f58a696ec7e19  /tmp/link

This is weird. The patch does make a difference on my machine
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
$ git log -1 --oneline
558785a0a9 (HEAD -> master, origin/master, origin/HEAD) cmd/compile: remove -installsuffix flag
$ go build -o /tmp/link cmd/link
$ sha256sum /tmp/link
173de0c0efed5ea3c57ef497effe803f20c60603889de4c64cb1ac726b7ed45f  /tmp/link

$ git apply </tmp/p
$ git diff
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
        "syscall"
+       "unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
        if out.buf == nil {
                return
        }
+       syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
        syscall.Munmap(out.buf)
        out.buf = nil
 }
$ go build -o /tmp/link cmd/link
$ sha256sum /tmp/link
3237cd32022b5b6cee6a9286cdec0d10e7e24455d9ee610553e8caecd8754c81  /tmp/link

$ git reset --hard
HEAD is now at 558785a0a9 cmd/compile: remove -installsuffix flag
$ go build -o /tmp/link cmd/link
$ sha256sum /tmp/link
173de0c0efed5ea3c57ef497effe803f20c60603889de4c64cb1ac726b7ed45f  /tmp/link

Maybe I messed up tabs and spaces when I first pasted the patch, so it didn't directly apply. Try this.
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
 	"syscall"
+	"unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
 	if out.buf == nil {
 		return
 	}
+	syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
 	syscall.Munmap(out.buf)
 	out.buf = nil
 }

@alvaroaleman I think the problem you ran into when trying to apply the patch is:

go build -o /tmp/link cmd/link/

Go will not use the local directory, but your installation’s cmd/link source instead. You need to use:

go build -o /tmp/link ./cmd/link/

You can verify the code was updated by using go tool objdump ./pkg/tool/linux_amd64/link. Search the output for outbuf_mmap.go and look for the munmap code. You should see disassembly like:
  outbuf_mmap.go:55     0x5ae5d2                4889442450              MOVQ AX, 0x50(SP)
  outbuf_mmap.go:59     0x5ae5d7                4889542438              MOVQ DX, 0x38(SP)
  outbuf_mmap.go:59     0x5ae5dc                488b4818                MOVQ 0x18(AX), CX
  outbuf_mmap.go:59     0x5ae5e0                48c704241a000000        MOVQ $0x1a, 0(SP)
  outbuf_mmap.go:59     0x5ae5e8                4889542408              MOVQ DX, 0x8(SP)
  outbuf_mmap.go:59     0x5ae5ed                48894c2410              MOVQ CX, 0x10(SP)
  outbuf_mmap.go:59     0x5ae5f2                48c744241804000000      MOVQ $0x4, 0x18(SP)
  outbuf_mmap.go:59     0x5ae5fb                0f1f440000              NOPL 0(AX)(AX*1)
  outbuf_mmap.go:59     0x5ae600                e8bbf0eeff              CALL syscall.Syscall.abi0(SB)
  outbuf_mmap.go:59     0x5ae605                450f57ff                XORPS X15, X15
  outbuf_mmap.go:59     0x5ae609                644c8b3425f8ffffff      MOVQ FS:0xfffffff8, R14
  outbuf_mmap.go:60     0x5ae612                488b442450              MOVQ 0x50(SP), AX
  outbuf_mmap.go:60     0x5ae617                488b5810                MOVQ 0x10(AX), BX
  outbuf_mmap.go:60     0x5ae61b                488b4818                MOVQ 0x18(AX), CX
  outbuf_mmap.go:60     0x5ae61f                488b7820                MOVQ 0x20(AX), DI
  syscall_linux.go:1142 0x5ae623                488b1576be2600          MOVQ syscall.mapper(SB), DX
  syscall_linux.go:1142 0x5ae62a                4889d0                  MOVQ DX, AX
  syscall_linux.go:1142 0x5ae62d                e88eceeeff              CALL syscall.(*mmapper).Munmap(SB)

If you can’t find munmap, look for Close — maybe the code got inlined.
If you only see the syscall.(*mmapper).Munmap call, the patch was not applied correctly. If you see the syscall.Syscall.abi0 call, the patch was applied correctly.
I have applied the patch on my system and will report back if I still encounter the corruption. However, as I can’t trigger it even when building hundreds of times in a loop, it would be better if @alvaroaleman could report back, as it sounds like it happens more frequently for you?
Thanks

We also encountered the same problem (see: #53116),   On some linux/loong64 machines, an "exec format error" error occasionally occurs when executing the make.bash or all.bash. Check these executable files with errors, it is found that the data of 16K(only one pagesize)  at the beginning of the file is missing. The result of hexdump is as follows:

   1  0000000 0000 0000 0000 0000 0000 0000 0000 0000
   2  *
   3  0004000 0020 4c00 3d24 1c01 d084 02fc 2064 29c0
   4  0004010 9004 02c0 4064 29c0 4800 576a 0000 0014
   5  0004020 62d3 28c0 8e73 0012 1260 5c00 0025 0015

This problem is caused by the occasional loss of 16K data at the beginning of the file after the link process calls the syscall.Fallocate function for the second time. However, not all machines can reproduce this error (the file system formats I tested include xfs, ext4, tmfs ).

my kernel version is  5.17.0+, pagesize is 16K。
In the case of ensuring enough disk space, I removed the second fallocate call, the problem no longer occurs.
I tried doing MS_SYNC before munmap, but the problem persists

Thanks for adding to this report, @abner-chenc!
I also just encountered this corruption again today, with @cherrymui’s patch applied — so the extra MSYNC does not help unfortunately.

I tried   using posix_fallocate (the implementation refers to glib's posix_fallocate.c) replaces syscall.fallcate, and this error does not occur on the same machine.
Here is my implementation of posix_fallocate:    https://github.com/abner-chenc/go-abner/commit/c43a4a75a031ac9550eb02697b50032f1dced9ae

@stapelberg thanks for the hint, using go build -o /tmp/link ./cmd/link just resulted in main module (std) does not contain package std/cmd/link and building from the cmd folder as go build -o /tmp/link ./link still ended up using my installations cmd/link. I ended up applying the patch there, but as you already noted, it doesn't fix the issue.

Seems like @abner-chenc patch is indeed fixing this, I couldn't reproduce the issue anymore in 500 builds when I never needed more than 200 for it to occur prior to their patch.

Are all the Exec format error case that the first page is missing? Or (sometimes) just the ELF header?
Does the fallocate syscall not work on your file system? But it also doesn't return an error? The fallocate isn't really necessary unless you're running low on disk space. Is that the case on your system? What happens if you just comment that fallocate call out?
@abner-chenc your patch also reorders  Truncate and fallocate. What happens if you don't reorder them, or just reorder them without doing the rest?

In the case of ensuring enough disk space, I removed the second fallocate call, the problem no longer occurs.

Could you explain what you did exactly? What  is "the second fallocate call"?
@alvaroaleman it looks  like @abner-chenc 's patch only affects  Loong64. Just to make  sure,  did you change  it to apply to your architecture as well?
Thanks.

@abner-chenc are you also running on tmpfs?
@alvaroaleman @stapelberg does it reproduce on a non-tmpfs file system?
Thanks.

@cherrymui I just applied https://github.com/abner-chenc/go-abner/commit/c43a4a75a031ac9550eb02697b50032f1dced9ae.patch without any changes, then built the link binary. While testing it, I output the binary to /tmp so a tmpfs, but when I orginally encountered the error I had targeted an xfs fs. I can check later if it also fixes the issue when targeting the xfs fs.

That patch should not make any difference if you're not running on Loong64. Would be good to try on xfs. Thanks.

Thanks. Could be. Could you try just moving the Truncate call without the rest of the change? Thanks.

@cherrymui yes, reducing the patch to only the changes in outbufs_mmap.go as shown below is enough to make the issue go away. Thanks for taking the time to look into this btw!
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -20,6 +20,10 @@ func (out *OutBuf) Mmap(filesize uint64) (err error) {
                out.munmap()
        }

+       err = out.f.Truncate(int64(filesize))
+       if err != nil {
+               Exitf("resize output file failed: %v", err)
+       }
        for {
                if err = out.fallocate(filesize); err != syscall.EINTR {
                        break
@@ -33,10 +37,6 @@ func (out *OutBuf) Mmap(filesize uint64) (err error) {
                        return err
                }
        }
-       err = out.f.Truncate(int64(filesize))
-       if err != nil {
-               Exitf("resize output file failed: %v", err)
-       }
        out.buf, err = syscall.Mmap(int(out.f.Fd()), 0, int(filesize), syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED|syscall.MAP_FILE)
        if err != nil {
                return err

Thanks.  If that is the fix then it seems that it should be possible to cause the same problem in a fairly simple C program.

Thanks @alvaroaleman !
I couldn't find the requirement of ftruncate before fallocate being documented anywhere. But it is probably a good thing to do to swap the two calls anyway.
cc @aclements in case you know something about the FS.

@abner-chenc your patch also reorders Truncate and fallocate. What happens if you don't reorder them, or just reorder them without doing the rest?

I have tested and found that by exchanging the calling order of 'fallocate' and 'ftruncate',  the occurrence of this error can be reduced,  but it cannot completely solve the problem.

Could you explain what you did exactly? What is "the second fallocate call"?

In the function copyHeap(outbuf.go), if len(out.heap) > 0,   syscall.Fallocate will be called again to allocate more disk space.
I tested and found that it is this Fallocate call that occasionally causes the loss of existing data in the file, all missing data is the beginning of the file 16K,   exactly one page size.
I tried executing MS_SYNC before munmap, but the problem persists.

@abner-chenc are you also running on tmpfs?
Yes, I also tested on tmpfs

I think mixing file IO syscalls and mmaps on the same file is probably not a good idea. So your f.ReadAt may not work as expected. (For the same reason I think we don't want to use the implementation of posix_fallocate.)  Can you observe the same corruption if you read the first 4 bytes from the mmap immediately after the mmap call?
Also, in the patch above the truncate call  is still after the fallocate. Could you try moving it up?
On the other hand, if fallocate causes occasional data lost, it sounds like an OS bug to me. Could you try writing a C program to see if this happens? Thanks.

The problem still happens with Go 1.19 (just happened to me again).

Has anyone tried to reproduce the failure in C?
It is interesting to know why it seems to only fail for you. Like, it doesn't fail on the builders (except loongarch). It would be good to know what is special on those machines, and why it fails. Thanks.

I tried testing with C code  fallocate.c , but the problem doesn't occur.

I tested and found that when fallocate returns the error code EINTR for the first time, and trying to fallocate again will cause the problem to appear (the second time no error code is returned).

I tried preallocate new space at the end of the file and the problem didn't reappear by testing, but I don't know what the root cause is.  this is my patch

Thanks for trying the C code.

when fallocate returns the error code EINTR for the first time, and trying to fallocate again will cause the problem to appear

Could you try in the C code, having a thread (or process) sending some signals to the fallocate thread, so it may return EINTR?

Thanks for trying the C code.

when fallocate returns the error code EINTR for the first time, and trying to fallocate again will cause the problem to appear

Could you try in the C code, having a thread (or process) sending some signals to the fallocate thread, so it may return EINTR?

@cherrymui

I can reproduce the error of fallocate returning EINTR in the C code by sending a signal to another process, and found that the problem of data loss does occur.  My C code:  https://github.com/abner-chenc/abner/blob/master/fallocate.c

I can reproduce the same error as loong64 by executing make.bash in Golang's linux/amd64 (kernel version 6.1.0-rc2), here are my logs and corrupted files：
bad_go_asm_x86.tar.gz
golang-x86.log

Note: You need to mount /tmp as tmpfs when testing (because some Linux distributions don't do this)

This should be a kernel bug(introduced in linux 5.16-rc4), I have reported this bug：https://lore.kernel.org/linux-mm/33b85d82.7764.1842e9ab207.Coremail.chenguoqic@163.com/

This is the fix patch provided by our colleagues： https://lore.kernel.org/linux-mm/20221101032248.819360-1-kernel@hev.cc/T/#u

Thanks very much @abner-chenc for tracking this down!
A fix has landed in the Linux kernel as commit https://github.com/torvalds/linux/commit/44bcabd70cf1425b4243e02251c02b01638a8287, which is included in Linux 6.1 and newer. I’ll update my machine when I get a chance.

authors: abner-chenc alvaroaleman bcmills cherrymui ianlancetaylor mengzhuo mknyszek stapelberg


text:
x/vulndb: compute derived symbols for exported package

Suppose a vulnerable symbol module/internal.v is reachable in a user program and that there is a derived symbol "module/internal.V" in the vulnerability report. A call stack produced by govulncheck could be something like
main.go calls module/internal.V
The vulnerability is in an internal package and that can be confusing to the user. Instead, we should make the derived symbols be the closest exported symbols of the same module that are in a public package.
For instance, the mere fact that the above summarized call stack exists means that there has to be a sequence of calls of the form module/p.Foo -> module/internal.V -> ... -> module/internal.v. We should thus have module/p.Foo as the derived symbol.
This will also help during report creation where we are interested if a unexported vulnerable symbol is reachable by an exported symbol of a public package of the module in question.

We should also not include the private symbol in OSV entry if we are able to find some derived symbols.


text:
runtime: TestCollisions failure on linux-arm64-longtest TryBot

#!watchflakes
post <- pkg == "runtime" && test == "TestCollisions"

https://storage.googleapis.com/go-build-log/eb2e13b8/linux-arm64-longtest_dc236d4a.log:
--- FAIL: TestCollisions (0.68s)
    hash_test.go:779: too many collisions i=3 j=13 outputs=32768 out of 65536
    hash_test.go:779: too many collisions i=13 j=3 outputs=32768 out of 65536
FAIL
FAIL	runtime	154.178s

That was on release-branch.go1.20 at CL 484758.

Weird. Probability says that the number of outputs should be close to 41427 (N balls in N bins, how many bins have a ball? N=65536). And the variance should be quite low, stddev is ~123. That means 32768 is 70 standard deviations away.
amd64 runs seem entirely consistent with that, both with amd64 assembly and with the fallback Go version. I see deviations at most in the small hundreds, not the tens of thousands.
arm64 looks different. The fallback is still ok, but the arm64 assembly has some weird behavior. For some cases it hits every bin exactly once! As a hash function that's great, but there's definitely something nonrandom about it. I have not been able to reproduce the described failure, but given the weird nonrandom behavior I'm less surprised.
Sounds like the arm64 assembly hash function needs some more mixing.


text:
net/http: frequent timeouts in TestIssue4191_InfiniteGetToPutTimeout/h1 on linux-amd64-wsl

#!watchflakes
post <- pkg == "net/http" && builder ~ `.*-wsl` && `test timed out` && `TestIssue4191_InfiniteGetToPutTimeout \(\d+m\d+s\)`

(attn @golang/wsl; CC @qmuntal)

https://build.golang.org/log/269049c7773455ebb4b1fbd7fbafa4a482b7b7e7:

2023/05/17 05:11:56 httptest.Server blocked in Close after 5 seconds, waiting for connections:
  *net.TCPConn 0xc00028c980 127.0.0.1:64282 in state active
panic: test timed out after 3m0s
running tests:
	TestIssue4191_InfiniteGetToPutTimeout (2m59s)
	TestIssue4191_InfiniteGetToPutTimeout/h1 (2m58s)
…
goroutine 7340 [IO wait, 2 minutes]:
runtime.gopark(0xc000c65348?, 0xb?, 0x1?, 0x0?, 0xc?)
	/tmp/workdir-host-linux-amd64-wsl/go/src/runtime/proc.go:387 +0xce fp=0xc000c65330 sp=0xc000c65310 pc=0x43bb2e
runtime.netpollblock(0x47ed7b?, 0x405466?, 0x0?)
	/tmp/workdir-host-linux-amd64-wsl/go/src/runtime/netpoll.go:556 +0xf7 fp=0xc000c65368 sp=0xc000c65330 pc=0x434557
internal/poll.runtime_pollWait(0x7f0b2c59a5b8, 0x77)
	/tmp/workdir-host-linux-amd64-wsl/go/src/runtime/netpoll.go:335 +0x85 fp=0xc000c65388 sp=0xc000c65368 pc=0x467f45
internal/poll.(*pollDesc).wait(0xc000893900?, 0xc000807000?, 0x0)
	/tmp/workdir-host-linux-amd64-wsl/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000c653b0 sp=0xc000c65388 pc=0x49ebe7
internal/poll.(*pollDesc).waitWrite(...)
	/tmp/workdir-host-linux-amd64-wsl/go/src/internal/poll/fd_poll_runtime.go:93
internal/poll.(*FD).Write(0xc000893900, {0xc000807000, 0x1000, 0x1000})
	/tmp/workdir-host-linux-amd64-wsl/go/src/internal/poll/fd_unix.go:391 +0x2d6 fp=0xc000c65460 sp=0xc000c653b0 pc=0x49ff76
net.(*netFD).Write(0xc000893900, {0xc000807000?, 0x4d8516?, 0xc00040c340?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/fd_posix.go:96 +0x25 fp=0xc000c654a8 sp=0xc000c65460 pc=0x5e3585
net.(*conn).Write(0xc00028c980, {0xc000807000?, 0xc7a840?, 0x423045?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/net.go:191 +0x45 fp=0xc000c654f0 sp=0xc000c654a8 pc=0x5ee665
net.(*TCPConn).Write(0x1?, {0xc000807000?, 0x5fbd05?, 0x476c38?})
	<autogenerated>:1 +0x25 fp=0xc000c65520 sp=0xc000c654f0 pc=0x5fbd05
net/http.checkConnErrorWriter.Write({0xc000d2eff8?}, {0xc000807000?, 0xc000c655a8?, 0x4d230c?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:3597 +0x2d fp=0xc000c65570 sp=0xc000c65520 pc=0x6db26d
bufio.(*Writer).Flush(0xc000a61c00)
	/tmp/workdir-host-linux-amd64-wsl/go/src/bufio/bufio.go:642 +0x55 fp=0xc000c655b8 sp=0xc000c65570 pc=0x505575
bufio.(*Writer).Write(0xc000a61c00, {0xc000d2e000?, 0x8c2c13?, 0x4?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/bufio/bufio.go:686 +0xcd fp=0xc000c65618 sp=0xc000c655b8 pc=0x50578d
net/http.(*chunkWriter).Write(0xc0006d43c0, {0xc000d2e000, 0x8000, 0x8000})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:388 +0x106 fp=0xc000c65688 sp=0xc000c65618 pc=0x6ccd66
bufio.(*Writer).Write(0xc000a61c80, {0xc000d2e000?, 0xc000893900?, 0xc000807000?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/bufio/bufio.go:682 +0xea fp=0xc000c656e8 sp=0xc000c65688 pc=0x5057aa
net/http.(*response).write(0xc0006d4380, 0x8000, {0xc000d2e000, 0x8000, 0x8000}, {0x0, 0x0})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:1648 +0x22a fp=0xc000c657f0 sp=0xc000c656e8 pc=0x6d31aa
net/http.(*response).Write(0x850300?, {0xc000d2e000?, 0x1?, 0xc000d95e50?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:1606 +0x2a fp=0xc000c65838 sp=0xc000c657f0 pc=0x6d2eea
net/http.(*writerOnly).Write(0xc000c658d8?, {0xc000d2e000?, 0xc000d95e50?, 0xc000c658e8?})
	<autogenerated>:1 +0x29 fp=0xc000c65868 sp=0xc000c65838 pc=0x71aa89
io.copyBuffer({0x9902a0, 0xc000d95e50}, {0x9905a0, 0xc11fc8}, {0xc000d2e000, 0x8000, 0x8000})
	/tmp/workdir-host-linux-amd64-wsl/go/src/io/io.go:429 +0x1f4 fp=0xc000c658e8 sp=0xc000c65868 pc=0x498e74
io.CopyBuffer({0x9902a0?, 0xc000d95e50?}, {0x9905a0?, 0xc11fc8?}, {0xc000d2e000?, 0xc000198dcc?, 0x4?})
	/tmp/workdir-host-linux-amd64-wsl/go/src/io/io.go:400 +0x36 fp=0xc000c65930 sp=0xc000c658e8 pc=0x498bf6
net/http.(*response).ReadFrom(0xc0006d4380, {0x9905a0?, 0xc11fc8})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:614 +0x2e5 fp=0xc000c65a08 sp=0xc000c65930 pc=0x6cd925
io.copyBuffer({0x9902c0, 0xc0006d4380}, {0x9905a0, 0xc11fc8}, {0x0, 0x0, 0x0})
	/tmp/workdir-host-linux-amd64-wsl/go/src/io/io.go:413 +0x147 fp=0xc000c65a88 sp=0xc000c65a08 pc=0x498dc7
io.Copy(...)
	/tmp/workdir-host-linux-amd64-wsl/go/src/io/io.go:386
net/http_test.testIssue4191_InfiniteGetToPutTimeout.func1({0x992e80?, 0xc0006d4380}, 0xc00056fb18?)
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/transport_test.go:2253 +0x4a fp=0xc000c65ad0 sp=0xc000c65a88 pc=0x7ee66a
net/http.HandlerFunc.ServeHTTP(0x443ce0?, {0x992e80?, 0xc0006d4380?}, 0x6cdde5?)
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:2134 +0x29 fp=0xc000c65af8 sp=0xc000c65ad0 pc=0x6d5529
net/http.(*ServeMux).ServeHTTP(0xc80c80?, {0x992e80, 0xc0006d4380}, 0xc000b26d00)
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:2512 +0x142 fp=0xc000c65b48 sp=0xc000c65af8 pc=0x6d6e42
net/http.serverHandler.ServeHTTP({0xc000bd5470?}, {0x992e80?, 0xc0006d4380?}, 0x6?)
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:2936 +0x8e fp=0xc000c65b78 sp=0xc000c65b48 pc=0x6d826e
net/http.(*conn).serve(0xc000f23e60, {0x993ae0, 0xc000bd5170})
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:2007 +0x5e5 fp=0xc000c65fb8 sp=0xc000c65b78 pc=0x6d4105
net/http.(*Server).Serve.func3()
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:3084 +0x28 fp=0xc000c65fe0 sp=0xc000c65fb8 pc=0x6d89e8
runtime.goexit()
	/tmp/workdir-host-linux-amd64-wsl/go/src/runtime/asm_amd64.s:1622 +0x1 fp=0xc000c65fe8 sp=0xc000c65fe0 pc=0x46d8c1
created by net/http.(*Server).Serve in goroutine 7336
	/tmp/workdir-host-linux-amd64-wsl/go/src/net/http/server.go:3084 +0x5cb

This failure module was not very common till a couple of days ago, and I don't see any commit that could have regressed the net package.. Is the builder healthy/has it enough space on disk?

This failure module was not very common till a couple of days ago, and I don't see any commit that could have regressed the net package.. Is the builder healthy/has it enough space on disk?

Sorry for these wsl related issues.
At first, The wsl-2.farm.mzh.io builder try to update to Win10 22H2 automatically and run out free disk. WSL became read only which triggered #60253
However after I resize disk the ESXi report disk failure so I had to make a new VM and reinstall Windows, which is default to WSL1 instead of WSL2 (causing #60295, #60258 )
Since it's a new VM, I forget to drop it's IPv6 (causing #60257 )
I'll backup all builders in the future and monitor free disk.

authors: bcmills mengzhuo qmuntal


text:
cmd/go: in go 1.21 module, go mod tidy and 'go list -m -u all' disagree over which go.sum lines are needed

What version of Go are you using (go version)?

$ go version
go version go1.21rc3 darwin/arm64

Does this issue reproduce with the latest release?
No(-ish).
I first ran into this in a module whose 'go' line is 1.21, so the latest release today (1.20.6) isn't applicable.
If I change the 'go' line to 1.20, then 'go mod tidy' no longer deletes as many go.sum lines, and the issue doesn't reproduce.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/gopher/Library/Caches/go-build'
GOENV='/Users/gopher/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT='loopvar'
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/gopher/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/gopher/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21rc3'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/_0/h0671fcn4rgb5pn9c745dx2h0000gn/T/go-build1549452710=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?
I ran go mod tidy in a medium sized local module whose go.mod has 'go 1.21' (primarily to help test this language version and its semantics, not because I need it; this go.mod file isn't public).
Then I ran go list -m -u all.
Edit: It turned out trivial to trigger the issue with a very minified top-level go.mod file. Here's a complete reproducer:
$ cd $(mktemp -d)
$ go mod init issue61605.test
$ echo 'package p; import _ "golang.org/x/build/maintner/reclog"' > p.go
$ go get golang.org/x/build@v0.0.0-20230726111318-84c18c56e0b0
$ go mod tidy
$ go list -m -u all

What did you expect to see?
The usual list of modules and available updates, and exit code 0.
What did you see instead?
$ go list -m -u all
go: updates to go.sum needed, disabled by -mod=readonly
$ echo $?
1

Running the go list command with -mod=mod re-adds the go.sum lines and makes the command work.
Or setting go to 1.20 instead of 1.21 and re-running go mod tidy also works around the issue.

Here's a test case version. I have not managed to whittle it down but I think I see a way to make it stop breaking (not the right way, but a way).
[!net:golang.org] skip

env GOPROXY=https://proxy.golang.org/
env GOSUMDB=

! go list
stderr 'missing go.sum entry'
go mod tidy
go list all
cat go.sum

go list -m -u
cat go.sum

go list -m -u all
cat go.sum

go list -mod=mod -m -u
cat go.sum

go list -mod=mod -m -u all
cat go.sum
stdout xxx

-- go.mod --
module m
require golang.org/x/build v0.0.0-20230726111318-84c18c56e0b0

-- x.go --
package p; import _ "golang.org/x/build/maintner/reclog"

After CL 513778 is submitted to the Go 1.21 release branch, we can move this to the Go 1.22 milestone.

Submitted the band-aid to the Go 1.21 release branch. Now a Go 1.22 bug.

I think a proper fix is in the general direction of #40775. The set of checksums to retain should be associated with the modload.Requirements struct in some way rather than tracked as a global variable.

@matloob, @samthanawalla: this is a code cleanup needed in cmd/go. I expect that a proper solution to it will be fairly involved; there is an unfortunate amount of communication through global variables in the code that updates the go.sum file.

authors: bcmills dmitshur rsc


text:
x/pkgsite: package removal request for open.boneffect.de/libs/vcsaccess

What is the path of the package that you would like to have removed?
open.boneffect.de/libs/vcsaccess
Are you the owner of this package?
Yes.
What is the reason that you could not retract this package instead?
I have retracted the package, but it still shows up when searching for it on pkg.go.dev (with a "Retracted" banner).
Please remove the package from search as well, and from the site entirely. Thank you!

This is done, and should take effect within an hour or so.

authors: findleyr lhinderberger


text:
regexp: individual regexp scales poorly under highly concurrent use

version: go1.3rc2

When matching against a regexp that's shared between goroutines, the program spends most
of its time waiting for the locks here:

68 @ 0x413579 0x4135fb 0x424eae 0x425080 0x496096 0x449dd7 0x4474fe 0x44df9e 0x400c6c
0x413810
#	0x425080	sync.runtime_Semacquire+0x30			/go/gc/src/pkg/runtime/sema.goc:199
#	0x496096	sync.(*Mutex).Lock+0xd6				/go/gc/src/pkg/sync/mutex.go:66
#	0x449dd7	regexp.(*Regexp).get+0x37			/go/gc/src/pkg/regexp/regexp.go:192
#	0x4474fe	regexp.(*Regexp).doExecute+0x5e			/go/gc/src/pkg/regexp/exec.go:423
#	0x44df9e	regexp.(*Regexp).FindStringSubmatch+0x9e	/go/gc/src/pkg/regexp/regexp.go:883
#	0x400c6c	main.match+0x6c					experimental/users/jrockway/regex.go:24

56 @ 0x413579 0x4135fb 0x424eae 0x425080 0x496096 0x449f2a 0x4477a4 0x44df9e 0x400c6c
0x413810
#	0x425080	sync.runtime_Semacquire+0x30			/go/gc/src/pkg/runtime/sema.goc:199
#	0x496096	sync.(*Mutex).Lock+0xd6				/go/gc/src/pkg/sync/mutex.go:66
#	0x449f2a	regexp.(*Regexp).put+0x3a			/go/gc/src/pkg/regexp/regexp.go:210
#	0x4477a4	regexp.(*Regexp).doExecute+0x304		/go/gc/src/pkg/regexp/exec.go:450
#	0x44df9e	regexp.(*Regexp).FindStringSubmatch+0x9e	/go/gc/src/pkg/regexp/regexp.go:883
#	0x400c6c	main.match+0x6c					experimental/users/jrockway/regex.go:24

With GOMAXPROCS=32 on a 32 core machine, we end up only using about 700% CPU according
to /usr/bin/time.

If we give each goroutine its own copy of the regexp (created in the goroutine with
regexp.MustCompile), then we use 2500% CPU.  This is much more reasonable.

While my actual problem occurs in a process that is reading data from the network, the
attached test case reproduces the issue rather well.

(Note: time CPU timing is done with the profile printer code commented out.  The test
case is compiled with optimization on.)
Attachments:

regex.go (878 bytes)

Too late for Go 1.5.

@rsc why possibility to use regexp in high-concurrent application was removed from 1.5 release? :(

@maxbaldin If this is a bottleneck in your code, you can work around it today by using a different regexp.Regexp in each goroutine. Based on what @rsc mentioned above, it may be that in future Go versions that becomes slightly easier via a Copy method or something.

I gave the Copy approach a shot in https://golang.org/cl/16110. I also compared benchmarks against @rsc's RegexpForSingleGoroutine; that gave a 7% improvement over the no-contention Copy version using the regex @bradfitz and @dvyukov were using above. (See CL for details.)

Fixed (at least worked around) in CL 16110.

authors: cespare jrockway maxbaldin rsc


text:
implement peerPicker with grpc

this PR implements GRPCPool which follows PeerPicker interface but the implementation is based on google's grpc.
there are some issues which haven't been addressed in this PR, we could have them in a separated PR
==1. interface:
grpc uses protobuf version 3, the generated stub is different with the one generated before. Like
v2:
type GetRequest struct {
        Group            *string `protobuf:"bytes,1,req,name=group" json:"group,omitempty"`
        Key              *string `protobuf:"bytes,2,req,name=key" json:"key,omitempty"`
        XXX_unrecognized []byte  `json:"-"`
}

v3:
type GetRequest struct {
       Group string `protobuf:"bytes,1,opt,name=group" json:"group,omitempty"`
       Key   string `protobuf:"bytes,2,opt,name=key" json:"key,omitempty"`
}

If we switched to protoc 3 directly, we may break something. so we just workaround this. The interface is untouched.
==2. Context:
right now the context used in groupcache is self-defined Context
// ProtoGetter is the interface that must be implemented by a peer.
type ProtoGetter interface {
        Get(context Context, in *pb.GetRequest, out *pb.GetResponse) error
}

and grpc use "golang.org/x/net/context" as default context. so we can pick one.
==3. performance comparison:
we will do that after the implementation sounds good.

I only want to look at a single commit (not a stack of 3) and I think this should live in a new child package, not be part of the existing package.
Also, have you signed a CLA?

Hi
I have squashed the commits into one and also moved them into grpc subpackage.
I also signed a CLA.
thanks

@hsinhoyeh Have you got any benchmark result compared to http? @bradfitz Are you interested in getting this merged into groupcache?

@xiang90 not yet. will do that if I have some bandwidth.

Can this peer-picker live elsewhere on github, outside of this repo?

I'm going to close this for now, assuming it's expired.
I have the googlebot doing CLA checks here now, so I can take contributions if you're still interested in pursuing this. Let me know or just reopen it if you are.

source: gharchive/pull-request

authors: bradfitz hsinhoyeh xiang90

repo: golang/groupcache

license_source: bigquery


text:
tour: [REPLACE WITH SHORT DESCRIPTION]

Context: https://tour.golang.org/moretypes/18
The example doesn't work. Here is the error:
go: finding golang.org/x/tour latest
go: downloading golang.org/x/tour v0.0.0-20191204025012-2e616b842c11
go: extracting golang.org/x/tour v0.0.0-20191204025012-2e616b842c11
./prog.go:6:1: missing return at end of function

Hi,
The linked page is an exercise. The slide title is Exercise: Slices, and the text below says:

Implement Pic. It should return a slice of length dy ...

so yes, it won't compile until you solve the exercise by implementing the requested function, as the error says: "missing return at end of function".
Closing here, since this is working as intended.

Sorry and thank you for your time

authors: ALTree borgogelli

repo: golang/tour


text:
校园网的运营商是CERNET，请问该选用哪个脚本？

如题，不是联通也不是电信...

测试了一下各个脚本的结果，电信稍微快一些。不过Apple music 下载还是不行，时好时不好。总的来说，加速不定，有时候还是不开比较快。

嗯，三个我都跑了一遍，联通电信差不多都是50m，移动最差 100+

校园网下还真是 有时候开了还没有不开快

现在体验最好的方式还是ipv6的dns加速，一直很快，然而只能电脑使用。不过我在寝室通过电脑发Wi-Fi给iPhone iPad使用，都不会卡，十分流畅。如果想尝试某度搜奶齿校园网ipv6 host

在国际出口足够好的情况下，可以尝试 AppleDNS International https://gist.github.com/gongjianhui/823be8fcc23cc178547a5c2dcdd173ba

@gongjianhui 表中的地址还需要先ping一下吗？

authors: 2788897518 HeavenlyKing gongjianhui

repo: gongjianhui/AppleDNS


text:
Control abort conditions

Currently, gosdm aborts after 5 consecutive failures. This is problematic in cases where several meters are connected to the bus, one of the meters fails, but one wants to continue logging data from the remaining meters.
It would therefore be nice if there was a flag/mode that, in case of one failure, would "forward" the scheduler to the next device ID in the hope that this device is still working.

The software has only one error handling method right now: if a meter is not reachable, terminate the software. This simplifies the design significantly, but does not handle single-device failures well. The HTTP interface needs significant changes in order to do this:

For the /last/{id} calls, the HTTP handler should not send invalid data, but signal the current device state using the HTTP error code "503 Service Unavailable".
This will not work for the /last call. Instead, it should reflect the device state using some kind of "health" information. It is the callers responsibility to check the health state before processing the information. All readings of a failed smart meter should be set to null.
As for the firehose: it should send the health information as a part of the datastream. If a meter is healthy, all other readings should be sent. If not, only the health information will be sent, followed by the next meter's readings.
The builtin website needs to evaluate the health information from the firehose.

Internally, there is no abstraction for a meter yet, i.e. there is no place to store the health information. In addition, different types of smart meters cannot be supported by the current code (see also issue #11). A simple list of meter metadata must be added to the code. If a modbus failure is detected, the meter should be marked accordingly.
The scheduler must incorporate the meter state into the request schedule - i.e. run a probe for all inactive meters, but if it fails, don't ask for more measurements.
All in all: This is a reasonable request, but the impact on the code is rather high.

Commit 1eb4f0346 implements the status information over the http api, except for the firehose. The semantic of the initial design has changed a little. More tests and documentation needed.

Now fully implemented and documentation updated. Closing.

authors: TheCount gonium


text:
PCIe Error

Hi Guys,
i'm trying to setup coral PCIe devices in Windows 10 and getting an error as follows:

I'm using a LattePanda Alpha with Windows 10 onside.
As devices i'm using one USB and two PCIe devices..
USB is running fine.
In Python all is working fine and i'm able to run all of this devices and switch between them, but in c++ i didn't have success.
Maybe someone else got this issue also?
thanks a lot

@ESEricWare humn, it is pretty odd that it's working in python but not c++
Are you running this while the python script is also running? Just checking because a single tpu cannot be owned by multiple processes at the same time

@ESEricWare Is this issue still reproducible ?

@ESEricWare Is this issue still reproducible ?

I didn't try current version of this library till now.
At the moment i'm not available but when i'm back at work i will try it again

@ESEricWare Is this issue still reproducible ?

authors: ESEricWare Namburger manoj7410

repo: google-coral/libcoral


text:
Direct pose

What do you think about anchor-free inspired approaches like https://arxiv.org/abs/1911.07451?
Are these model plausible for Coral devices?
Do you plan to have an example here?
If you can talk internally with other TF teams see also HUB:
https://github.com/tensorflow/hub/issues/424
And related:
https://github.com/tensorflow/models/issues/4676

Hi,
This seems certainly an interesting approach though i'm not deeply familiar with it. The convolutional parts of this would likely run on EdgeTPU no problem.
The post processing will have to get done on CPU i suspect.
We don't currently plan to provide such models as pretrained/canned examples.

authors: bhack mtyka

repo: google-coral/project-posenet


text:
How to get bodys' linear and angular velocity(data.cvel?)

Hello,
I'm asking help for the function of getting the bodys' linear and angular velocity. I saw that there is a data.cvel offered angular and linear velocity of bodys, but I don't know which coordinate frame it's in(local or global). Can anyone tell me about this?
Appreciate it for any kind help!

https://mujoco.readthedocs.io/en/latest/APIreference/APIfunctions.html#mj-objectvelocity
Thank you vey much sir!

repo: google-deepmind/mujoco


text:
Frame ids doesn't match with timestamps in action localisation

Hi, thank you for releasing this wonderful dataset.
I noticed that in action localisation annotations the frame_ids don't match with timestamps per each segments.
For example, if you take a look at video_9431 in action_localisation_train.json, the first part of the annotations are:
{'metadata': {'split': 'train',
  'video_id': 'video_9431',
  'frame_rate': 30,
  'num_frames': 1051,
  'resolution': [1080, 1920],
  'audio_samples': 1680754,
  'audio_sample_rate': 48000,
  'is_cup_game': 0,
  'is_camera_moving': 1},
 'action_localisation': [{'id': 0,
   'label_id': 23,
   'label': 'Opening something',
   'parent_objects': [1, 0],
   'timestamps': [0, 2866667],
   'frame_ids': [0, 86]},
..........}

but if you compute the second element of the 'frame_ids' with frame_rate, it doesn't match with the corresponding timestamps.
(e.g. 86 * 30 = 2580 sec but the 2nd element of 'timestamps' is 2866.667 sec)
Could you take a look if you have time? Thank you.

Sorry I made a terrible mistake. I should have 'divided' the frame rate. I'll close this issue


text:
Navigate between screens with Compose: Android Basics with Compose

I have a Problem as "Interface NavHost does not have constructors" even when i have given constructors
Code is following:
`/*

Copyright (C) 2023 The Android Open Source Project

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

 https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package com.example.cupcake

import androidx.compose.foundation.layout.padding
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.ArrowBack
import androidx.compose.material3.Icon
import androidx.compose.material3.IconButton
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Scaffold
import androidx.compose.material3.Text
import androidx.compose.material3.TopAppBar
import androidx.compose.material3.TopAppBarDefaults
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.tooling.preview.Preview
import androidx.lifecycle.viewmodel.compose.viewModel
import androidx.navigation.NavHost
import androidx.navigation.NavHostController
import androidx.navigation.compose.rememberNavController
import com.example.cupcake.ui.OrderViewModel
enum class CupcakeScreen {
Start,
Flavour,
Pickup,
Summary
}
/**

Composable that displays the topBar and displays back button if back navigation is possible.
*/
@Composable
fun CupcakeAppBar(
canNavigateBack: Boolean,
navigateUp: () -> Unit,
modifier: Modifier = Modifier
) {
TopAppBar(
title = { Text(stringResource(id = R.string.app_name)) },
colors = TopAppBarDefaults.mediumTopAppBarColors(
containerColor = MaterialTheme.colorScheme.primaryContainer
),
modifier = modifier,
navigationIcon = {
if (canNavigateBack) {
IconButton(onClick = navigateUp) {
Icon(
imageVector = Icons.Filled.ArrowBack,
contentDescription = stringResource(R.string.back_button)
)
}
}
}
)
}

@Preview(showBackground = true)
@Composable
fun CupcakeApp(
viewModel: OrderViewModel = viewModel(),
navController: NavHostController = rememberNavController()
) {
Scaffold(
    topBar = {
        CupcakeAppBar(
            canNavigateBack = false,
            navigateUp = { /* TODO: implement back navigation */ }
        )
    }
) { innerPadding ->
    val uiState by viewModel.uiState.collectAsState()
    Text("Hi", modifier = Modifier.padding(innerPadding))
    NavHost(
        navController = navController,
        startDestination = CupcakeScreen.Start.name,
        modifier = Modifier.padding(innerPadding),
    ) {}
}

}
`

Used import androidx.navigation.compose.NavHost
Instead of import androidx.navigation.NavHost
Solved My Issue


text:
Create an interactive Dice Roller app: Android Basics with Compose

URL of codelab
Step 3: Create Layout Infrastructure

not clear that this is in DicRollerApp() method

realised this is in whether the method is called not the method header


text:
Write automated tests: Android Basics with Compose

In the code along video, Sagar said to create the first field as Bill amount and the result text field as Tip amount: %s but in this codelab (and previous chapters) they are actually Cost of Service and Tip Amount: %s. Selecting the field with onNodeWithText("Cost of Service") may cause issue.

It's the same issue with #15

@chaoszcat would you consider closing this issue, as it is already tracked on #15

authors: chaoszcat divyanshdxn ifanzalukhu97