text:
cmd/go: apparent freezes on Windows when a 'go generate' subprocess invokes 'go build

What version of Go are you using (go version)?

go version go1.15.4 windows/amd64

Does this issue reproduce with the latest release?
I can reproduce it very regularly at least on my system.
What operating system and processor architecture are you using (go env)?
go env Output
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Dev\Temp\go-build
set GOENV=C:\Users\Alex\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Dev\workdir\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Dev\workdir
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Dev\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=C:\Dev\Temp
set GOTOOLDIR=C:\Dev\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Dev\Temp\go-build995151114=/tmp/go-build -gno-record-gcc-switches

What did you do?
I experienced a lot of freezes for several days now, when using the command:
go generate ./..
in one of my projects.
go generate does trigger four different mockgen commands, thats all.
These mockgen commands in turn trigger some go list and go build commands, which freeze very often.
I don't think it has anything to do with go generate or mockgen. It just seems to trigger the issue more often as when I just do go build. But also go build as a standalone command sometimes triggered this issue.
What did you expect to see?
Successful execution of all go commands that were called from go generate --> mockgen.
What did you see instead?
The main issue here is, that go build and also go list that is executed by mockgen freezes very often.
In windows, by using a process monitor tool, I can see, that go generate called mockgenthat in turn called go build.
It happens very often, that the go build -o prog.bin.exe prog.goprocess is does not return and seems to be stuck.
I also found out, when setting GOMAXPROCS=1, go generate ./... works perfectly without a single one of this issues appearing.
Furthermore when go build was stuck I attached with dlv and got the following list of goroutines:
Goroutine 1 - User: c:/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x756e89)
  Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
* Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 5 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 15 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 19 - User: c:/go/src/runtime/symtab.go:940 runtime.step (0x744a7e) (thread 30604)
  Goroutine 20 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 1267 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
  Goroutine 1268 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
  Goroutine 1269 - User: c:/go/src/cmd/go/internal/work/exec.go:169 cmd/go/internal/work.(*Builder).Do.func3 (0xb6f83f)
  Goroutine 1270 - User: :0 ??? (0x7ffbb80dc144) (thread 21588)

The following goroutines are waiting at the provided code lines:
1    waits at: c:/go/src/cmd/go/internal/work/exec.go:188
1267 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
1268 waits at: c:/go/src/cmd/go/internal/work/exec.go:169
1269 waits at: c:/go/src/cmd/go/internal/work/exec.go:169

I think the other goroutines are not that interesting, they look like this:
the other routines look something like this:
0  0x000000000072be3a in runtime.gopark
   at c:/go/src/runtime/proc.go:307
1  0x0000000000715929 in runtime.goparkunlock
   at c:/go/src/runtime/proc.go:312
2  0x0000000000715929 in runtime.bgsweep
   at c:/go/src/runtime/mgcsweep.go:182
3  0x000000000075b9a1 in runtime.goexit
   at c:/go/src/runtime/asm_amd64.s:1374

or this:
   0  0x000000000072be3a in runtime.gopark
   at c:/go/src/runtime/proc.go:307
1  0x000000000070e174 in runtime.gcBgMarkWorker
   at c:/go/src/runtime/mgc.go:1891
2  0x000000000075b9a1 in runtime.goexit
   at c:/go/src/runtime/asm_amd64.s:1374

As mentioned, this issue does not appear, when using GOMAXPROCS=1.
Otherwise it appears nearly always.
Any idea?

I am not sure about this problem. It "freezes" on many different parts in go, e.g.
go list -e -json -compiled=true -test=true -export=false -deps=false -find=false -- C:\Dev\projects\XXX\pkg\config
With goroutines
Goroutine 1 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 5 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 6 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 28 - User: :0 ??? (0x7ffbb80dc4c4) (thread 5088)
  Goroutine 29 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 30 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 31 - User: c:/go/src/strings/builder.go:68 strings.Join (0x7fc3c7)
  Goroutine 32 - User: :0 ??? (0x7ffbb80dc7e4) (thread 28972)
  Goroutine 33 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 34 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 50 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 66 - User: :0 ??? (0x7ffbb80dc144) (thread 17908)
  Goroutine 67 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 98 - User: c:/go/src/runtime/proc.go:312 sync.runtime_notifyListWait (0x757157)
  Goroutine 99 - User: c:/go/src/strings/builder.go:68 strings.Join (0x7fc3c7)
  Goroutine 100 - User: :0 ??? (0x7ffbb80dc144) (thread 21604)
  Goroutine 101 - User: c:/go/src/os/stat_windows.go:64 os.stat (0x7e2a05)
  Goroutine 102 - User: c:/go/src/cmd/go/internal/imports/read.go:46 cmd/go/internal/imports.(*importReader).readByte (0xbbb20e)

or on command: go  generate ./...
with goroutines
Goroutine 1 - User: c:/go/src/syscall/syscall_windows.go:46 syscall.UTF16FromString (0x7a736e)
  Goroutine 2 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 3 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 4 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 18 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 19 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 43 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 44 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 50 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
* Goroutine 51 - User: :0 ??? (0x7ffbb80dc144) (thread 16904)
  Goroutine 52 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 53 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 54 - User: c:/go/src/runtime/proc.go:307 runtime.gopark (0x72be3a)
  Goroutine 55 - User: c:/go/src/crypto/sha256/sha256.go:214 crypto/sha256.(*digest).Sum (0x92ac0e)
  Goroutine 56 - User: c:/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x756fae)
  Goroutine 57 - User: c:/go/src/sync/map.go:226 sync.(*Map).LoadOrStore (0x77511a)
  Goroutine 58 - User: c:/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x756fae)
  Goroutine 59 - User: :0 ??? (0x7ffbb80dc4c4) (thread 7224)
  Goroutine 60 - User: c:/go/src/bytes/buffer.go:229 bytes.makeSlice (0x80c6da)
  Goroutine 61 - User: c:/go/src/io/io.go:406 io.copyBuffer (0x7a045b)
  Goroutine 62 - User: c:/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x756fae)
  Goroutine 63 - User: c:/go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go:62 cmd/vendor/golang.org/x/mod/sumdb/dirhash.Hash1 (0xaf2c29)

I am quite confused with this behavior.

@ybbus, unfortunately those goroutine traces are not particularly helpful: the goroutines waiting in src/cmd/go/internal/work/exec.go are waiting for something else to complete, but don't tell us what that something else is.
The later traces show a program that does not appear to be deadlocked.

  Goroutine 102 - User: c:/go/src/cmd/go/internal/imports/read.go:46 cmd/go/internal/imports.(*importReader).readByte (0xbbb20e)

is in a conditional after having successfully read a byte from a *bufio.Reader:
https://github.com/golang/go/blob/1f2e58d89b754c28ba7448ead7d8d9ae8957ab49/src/cmd/go/internal/imports/read.go#L46

  Goroutine 63 - User: c:/go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go:62 cmd/vendor/golang.org/x/mod/sumdb/dirhash.Hash1 (0xaf2c29)

is in a loop after having read a file to be hashed:
https://github.com/golang/go/blob/1f2e58d89b754c28ba7448ead7d8d9ae8957ab49/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go#L62

So I'm not sure what to make of this: it looks like a running program, albeit possibly one that is running slowly. (Is it possible that you're seeing slowness due to, say, filesystem contention?)

@bcmills
It is very hard for me to get some useful information out of this issue. :(
What I can tell you is, that the last two go-routines you mentioned are indeed "stuck".
I could leave the process for an hour and they would still be in the exact same position in code.
I checked this with "continue" command in delve and then break again after a while, with the exact same positions of the go routines.
I also cannot explain why this doesn't happen with GOMAXPROCS=1.
I would appreciate if someone has a hint for me how to better approach this problem.
I tried all kind of tools (windows sysinternals, dlv) but did not yet find any useful clue to figure out, why it gets stuck.
Could some anti-virus software be the problem here? blocking file-access / forbid memory allocation...?

Looked a bit more into this.
Still the same: GOMAXPROCS=1  no problems, GOMAXPROCS=8, nearly every invoke freezes
What I also observed, that I cannot really nail it down to some code line. It seems to happen random on any line of code, and also on any part in the execution but mostly when go build is executing (maybe since this takes most of the runtime).
But what seems to be in common is, that the end of the stack on that gorouting is:
0  0x000000000072be3a in runtime.gopark
    at c:/go/src/runtime/proc.go:307
 1  0x00000000007106f1 in runtime.goparkunlock
    at c:/go/src/runtime/proc.go:312
 2  0x00000000007106f1 in runtime.gcParkAssist
    at c:/go/src/runtime/mgcmark.go:716
 3  0x00000000007100dd in runtime.gcAssistAlloc
    at c:/go/src/runtime/mgcmark.go:580
 4  0x00000000006fe59c in runtime.mallocgc
    at c:/go/src/runtime/malloc.go:952
 5  0x00000000006fb7c5 in runtime.convTslice
    at c:/go/src/runtime/iface.go:385
 6  0x0000000000af2c29 in cmd/vendor/golang.org/x/mod/sumdb/dirhash.Hash1
    at c:/go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go:62

To be honest I am not very familiar with this low level stuff, but for me it seems like that a preemtion happens for the goroutine currently running, and it never ever wakes up.
Also not very deterministic:
Sometimes the stuck process has 0% CPU usage, and sometimes it uses at lot of CPU what looks like some busywaiting.
It don't think it is just slow. I've let this command run for over an hour without any progress and after that the goroutines stack still looked the same.
Normally this command should take no more than a couple of seconds.
Using GOMAXPROCS=1, the command does run in under one minute.

Could some anti-virus software be the problem here?

I think the answer to that is always “maybe”. 😅
(Do you have any anti-virus software installed?)

blocking file-access / forbid memory allocation...?

The go command does uses file-locking, but in that case I would expect to see some goroutine blocked in the cmd/go/internal/lockedfile or cmd/go/internal/lockedfile/internal/filelock packages, and that doesn't seem to be the case here — none of the goroutine stacks you've posted appear to be in I/O at all.

A few more diagnostic questions:

Do other Go programs exhibit similar hangs on your system, or just cmd/go?
How did you install the Go toolchain? (From source, from an installer at https://golang.org/dl, something else..?)

Bryan:
I havent experienced this yet on other go programs, but to be honest I do not run a lot of go programs locally, just developing and therefore doing a lot of build, test and generate.
When I run other programs locally there was no issue yet.
For installing I always download the windows archive and extract it to my machine.
Yes, MCAfee is installed, but I was allowed to completely disable it (corporate machine) for testing, and it did not change anything.
Is ther an easy way to check with dlv, what to program is really doing at the moment or why (for what) it is waiting?
It seems like it just stopped doing stuff without any good reason.

try passing the -x flag to go build with both the GOMAXPROCS 1 and 8 scenarios. Please provide the entire output

Hi Dave,
I will try that out.
In the meantime I also checked a bit more using Sysinternals Process explorer.
It seems that even though I disabled most of the McAfee stuff, it still tries to be part of the problem I think.
I did not even know that is is going to directly inject itself in other processes, but that is what you can see on the screenshot.
The stack of all threads of the hanging go processes look the same and seem to "WaitForSingleObject". The only stack that looks different is the one that also seems to have something injected from McAfee.
I know too little of the low level windows os stuff to have some opinion about this, but for me it actually looks like some issue with mcafee.
I just don't get, why there a tons of programs running on my machine that don't habe a similar problem, also it is quite strange that GOMAXPROCS seems to make a huge difference here.

The other thing that may happen is that there is one thread that is running all the time with high CPU, but still the go command does never return:

Dave:
MAXPROCS=8 with go build -x does not get very far :)
C:\XXX\projects\XXX\XXX\dlt-service\pkg\config\gomock_reflect_037446375>go build -x -o prog.bin.exe prog.go
WORK=C:\XXX\Temp\go-build997381258

otherwise with MAXPROCS=1
MAXPROCS=1 Output
go build -x -o prog.bin.exe prog.go
WORK=C:\XXXXX\Temp\go-build585078346
mkdir -p $WORK\b001\
cat >$WORK\b001\_gomod_.go << 'EOF' # internal
package main
import _ "unsafe"
//go:linkname __debug_modinfo__ runtime.modinfo
var __debug_modinfo__ = "0w\xaf\f\x92t\b\x02A\xe1\xc1\a\xe6\xd6\x18\xe6path\tcommand-line-arguments\nmod\tzzzzz/yyyyy/dlt-service\t(devel)\t\ndep\tgithub.com/fsnotify/fsnotify\tv1.4.7\th1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=\ndep\tgithub.com/golang/mock\tv1.4.4\th1:l75CXGRSwbaYNpl/Z2X1XIIAMSCquvXgpVZDhwEIJsc=\ndep\tgithub.com/hashicorp/hcl\tv1.0.0\th1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=\ndep\tgithub.com/magiconair/properties\tv1.8.1\th1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=\ndep\tgithub.com/mitchellh/mapstructure\tv1.1.2\th1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=\ndep\tgithub.com/pelletier/go-toml\tv1.2.0\th1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=\ndep\tgithub.com/spf13/afero\tv1.1.2\th1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=\ndep\tgithub.com/spf13/cast\tv1.3.0\th1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=\ndep\tgithub.com/spf13/jwalterweatherman\tv1.0.0\th1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=\ndep\tgithub.com/spf13/pflag\tv1.0.3\th1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=\ndep\tgithub.com/spf13/viper\tv1.7.1\th1:pM5oEahlgWv/WnHXpgbKz7iLIxRf65tye2Ci+XFK5sk=\ndep\tgithub.com/subosito/gotenv\tv1.2.0\th1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=\ndep\tgolang.org/x/text\tv0.3.2\th1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=\ndep\tgopkg.in/ini.v1\tv1.51.0\th1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=\ndep\tgopkg.in/yaml.v2\tv2.2.4\th1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=\n\xf92C1\x86\x18 r\x00\x82B\x10A\x16\xd8\xf2"
        EOF
cat >$WORK\b001\importcfg << 'EOF' # internal
# import config
packagefile encoding/gob=C:\XXXXX\go\pkg\windows_amd64\encoding\gob.a
packagefile flag=C:\XXXXX\go\pkg\windows_amd64\flag.a
packagefile fmt=C:\XXXXX\go\pkg\windows_amd64\fmt.a
packagefile github.com/golang/mock/mockgen/model=C:\XXXXX\Temp\go-build\10\109e2e69560a3543b230cdf986befcb2e6290529c978a3b0eabb2f18094dab72-d
packagefile zzzzz/yyyyy/dlt-service/pkg/config=C:\XXXXX\Temp\go-build\58\5887bb7485e88fbcc15fb62384273e3026e1e63106530bb84f5c50f386972a1b-d
packagefile os=C:\XXXXX\go\pkg\windows_amd64\os.a
packagefile path=C:\XXXXX\go\pkg\windows_amd64\path.a
packagefile reflect=C:\XXXXX\go\pkg\windows_amd64\reflect.a
packagefile runtime=C:\XXXXX\go\pkg\windows_amd64\runtime.a
EOF
cd C:\XXXXX\projects\zzzzz\yyyyy\dlt-service\pkg\config\gomock_reflect_037446375
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\compile.exe" -o "$WORK\\b001\\_pkg_.a" -trimpath "$WORK\\b001=>" -p main -lang=go1.15 -complete -buildid JJ3Jf0mSmjshQOlN5BMT/JJ3Jf0mSmjshQOlN5BMT -goversion go1.15.4 -D _/C_/XXXXX/projects/zzzzz/yyyyy/dlt-service/pkg/config/gomock_reflect_037446375 -importcfg "$WORK\\b001\\importcfg" -pack -c=4 "C:\\XXXXX\\projects\\zzzzz\\yyyyy\\dlt-service\\pkg\\config\\gomock_reflect_037446375\\prog.go" "$WORK\\b001\\_gomod_.go"
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\buildid.exe" -w "$WORK\\b001\\_pkg_.a" # internal
cp "$WORK\\b001\\_pkg_.a" "C:\\XXXXX\\Temp\\go-build\\4f\\4fa3da2ae4df62905dff257ffe93933cf830e556c79d91891177273d2ff58241-d" # internal
cat >$WORK\b001\importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK\b001\_pkg_.a
packagefile encoding/gob=C:\XXXXX\go\pkg\windows_amd64\encoding\gob.a
packagefile flag=C:\XXXXX\go\pkg\windows_amd64\flag.a
packagefile fmt=C:\XXXXX\go\pkg\windows_amd64\fmt.a
packagefile github.com/golang/mock/mockgen/model=C:\XXXXX\Temp\go-build\10\109e2e69560a3543b230cdf986befcb2e6290529c978a3b0eabb2f18094dab72-d
packagefile zzzzz/yyyyy/dlt-service/pkg/config=C:\XXXXX\Temp\go-build\58\5887bb7485e88fbcc15fb62384273e3026e1e63106530bb84f5c50f386972a1b-d
packagefile os=C:\XXXXX\go\pkg\windows_amd64\os.a
packagefile path=C:\XXXXX\go\pkg\windows_amd64\path.a
packagefile reflect=C:\XXXXX\go\pkg\windows_amd64\reflect.a
packagefile runtime=C:\XXXXX\go\pkg\windows_amd64\runtime.a
packagefile bufio=C:\XXXXX\go\pkg\windows_amd64\bufio.a
packagefile encoding=C:\XXXXX\go\pkg\windows_amd64\encoding.a
packagefile encoding/binary=C:\XXXXX\go\pkg\windows_amd64\encoding\binary.a
packagefile errors=C:\XXXXX\go\pkg\windows_amd64\errors.a
packagefile io=C:\XXXXX\go\pkg\windows_amd64\io.a
packagefile math=C:\XXXXX\go\pkg\windows_amd64\math.a
packagefile math/bits=C:\XXXXX\go\pkg\windows_amd64\math\bits.a
packagefile sync=C:\XXXXX\go\pkg\windows_amd64\sync.a
packagefile sync/atomic=C:\XXXXX\go\pkg\windows_amd64\sync\atomic.a
packagefile unicode=C:\XXXXX\go\pkg\windows_amd64\unicode.a
packagefile unicode/utf8=C:\XXXXX\go\pkg\windows_amd64\unicode\utf8.a
packagefile sort=C:\XXXXX\go\pkg\windows_amd64\sort.a
packagefile strconv=C:\XXXXX\go\pkg\windows_amd64\strconv.a
packagefile strings=C:\XXXXX\go\pkg\windows_amd64\strings.a
packagefile time=C:\XXXXX\go\pkg\windows_amd64\time.a
packagefile internal/fmtsort=C:\XXXXX\go\pkg\windows_amd64\internal\fmtsort.a
packagefile github.com/golang/mock/gomock=C:\XXXXX\Temp\go-build\60\601f44e01f55f73609660ecdc59c862e261d2b809533709aae14e00ea27598b3-d
packagefile github.com/spf13/viper=C:\XXXXX\Temp\go-build\a7\a7b8d2703b14f15e33bab09c30715eda783308d3a003e3056c81f6a79229feac-d
packagefile internal/oserror=C:\XXXXX\go\pkg\windows_amd64\internal\oserror.a
packagefile internal/poll=C:\XXXXX\go\pkg\windows_amd64\internal\poll.a
packagefile internal/syscall/execenv=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\execenv.a
packagefile internal/syscall/windows=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\windows.a
packagefile internal/testlog=C:\XXXXX\go\pkg\windows_amd64\internal\testlog.a
packagefile syscall=C:\XXXXX\go\pkg\windows_amd64\syscall.a
packagefile unicode/utf16=C:\XXXXX\go\pkg\windows_amd64\unicode\utf16.a
packagefile internal/unsafeheader=C:\XXXXX\go\pkg\windows_amd64\internal\unsafeheader.a
packagefile internal/bytealg=C:\XXXXX\go\pkg\windows_amd64\internal\bytealg.a
packagefile internal/cpu=C:\XXXXX\go\pkg\windows_amd64\internal\cpu.a
packagefile runtime/internal/atomic=C:\XXXXX\go\pkg\windows_amd64\runtime\internal\atomic.a
packagefile runtime/internal/math=C:\XXXXX\go\pkg\windows_amd64\runtime\internal\math.a
packagefile runtime/internal/sys=C:\XXXXX\go\pkg\windows_amd64\runtime\internal\sys.a
packagefile bytes=C:\XXXXX\go\pkg\windows_amd64\bytes.a
packagefile internal/reflectlite=C:\XXXXX\go\pkg\windows_amd64\internal\reflectlite.a
packagefile internal/race=C:\XXXXX\go\pkg\windows_amd64\internal\race.a
packagefile internal/syscall/windows/registry=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\windows\registry.a
packagefile context=C:\XXXXX\go\pkg\windows_amd64\context.a
packagefile encoding/csv=C:\XXXXX\go\pkg\windows_amd64\encoding\csv.a
packagefile encoding/json=C:\XXXXX\go\pkg\windows_amd64\encoding\json.a
packagefile github.com/fsnotify/fsnotify=C:\XXXXX\Temp\go-build\1e\1ed0f1c46c880e2c6cc8148dfadead67b4eefa80e7d6cf6afa0004941457ea80-d
packagefile github.com/hashicorp/hcl=C:\XXXXX\Temp\go-build\32\32b7a1ffbacd01fcaa242286bd2bb7cccbc72e3149bd109b95fe93080a2a5ab6-d
packagefile github.com/hashicorp/hcl/hcl/printer=C:\XXXXX\Temp\go-build\01\016804adc7b57984933ab80850831574c5d4a3fae5ccc856216d17941a0487dd-d
packagefile github.com/magiconair/properties=C:\XXXXX\Temp\go-build\23\239a0de4f71fd1234ed89705587cf6cc7bb55785f60d91401ab243cc62d5a92b-d
packagefile github.com/mitchellh/mapstructure=C:\XXXXX\Temp\go-build\ab\ab3935b0e07b7438fb20e119b33efbe700696c812df23b86c536586cb7987071-d
packagefile github.com/pelletier/go-toml=C:\XXXXX\Temp\go-build\5d\5dba111cfc102063714dede76bafc32c66264998beb25d6c7129ebf2cb170a41-d
packagefile github.com/spf13/afero=C:\XXXXX\Temp\go-build\93\931e58331c1b802787f5ff389434883280786eda5634890e6d793dae50229626-d
packagefile github.com/spf13/cast=C:\XXXXX\Temp\go-build\11\11b966d35db068c18238ff0f4e4784c9c865ba958cc28ef3e5297c535ba1b91a-d
packagefile github.com/spf13/jwalterweatherman=C:\XXXXX\Temp\go-build\7f\7fb7dd87189e3144cb19036b43bdac0ebf73d9de0fd355d48d2eaf0ac6b4923d-d
packagefile github.com/spf13/pflag=C:\XXXXX\Temp\go-build\4b\4b57e69250b14274268a29feb79b19458e1c6438f29d8713c479fdddb93f9c0b-d
packagefile github.com/subosito/gotenv=C:\XXXXX\Temp\go-build\ca\ca08d89fb013a51f590f0e5cce4e83b9b863444f9ba7399b1865893e8ecfe20f-d
packagefile gopkg.in/ini.v1=C:\XXXXX\Temp\go-build\0c\0cfc989b448b54874061740e25f59372de99a858b2661ed0c22822e1f2fc7117-d
packagefile gopkg.in/yaml.v2=C:\XXXXX\Temp\go-build\63\63770eb6a562fbcbedc0853b62fe1e376fb059ae340fe38254689a3b646dfeb8-d
packagefile log=C:\XXXXX\go\pkg\windows_amd64\log.a
packagefile path/filepath=C:\XXXXX\go\pkg\windows_amd64\path\filepath.a
packagefile internal/syscall/windows/sysdll=C:\XXXXX\go\pkg\windows_amd64\internal\syscall\windows\sysdll.a
packagefile encoding/base64=C:\XXXXX\go\pkg\windows_amd64\encoding\base64.a
packagefile github.com/hashicorp/hcl/hcl/ast=C:\XXXXX\Temp\go-build\9b\9bf532129a40b204feb1fb47fdb7f4e066a4eb7fd51b1cfe00b6a598202a9cd1-d
packagefile github.com/hashicorp/hcl/hcl/parser=C:\XXXXX\Temp\go-build\19\196933d4932413c85c778a58ce4b5e8f86517cadec2239b1a6a4263d157224cc-d
packagefile github.com/hashicorp/hcl/hcl/token=C:\XXXXX\Temp\go-build\1e\1ec247c5ba207a59d6b55e2ff205c846939f18168ca5d283c5e39087a2638ef8-d
packagefile github.com/hashicorp/hcl/json/parser=C:\XXXXX\Temp\go-build\72\7297703cbb80ca4ca59f806edcdaa73bd3a27f4723cc483ac5f8a0f85ab60aa9-d
packagefile text/tabwriter=C:\XXXXX\go\pkg\windows_amd64\text\tabwriter.a
packagefile io/ioutil=C:\XXXXX\go\pkg\windows_amd64\io\ioutil.a
packagefile net/http=C:\XXXXX\go\pkg\windows_amd64\net\http.a
packagefile regexp=C:\XXXXX\go\pkg\windows_amd64\regexp.a
packagefile net=C:\XXXXX\go\pkg\windows_amd64\net.a
packagefile github.com/spf13/afero/mem=C:\XXXXX\Temp\go-build\a1\a11301268a1e43bd44623a80d8249cdd50058bf05206b0a64c2e3385add04d92-d
packagefile golang.org/x/text/transform=C:\XXXXX\Temp\go-build\ab\ab0f407746c6f50ddb771ed1df9249f300d7be5afcada3a81d3a79297ed9bef7-d
packagefile golang.org/x/text/unicode/norm=C:\XXXXX\Temp\go-build\8e\8e5b7fe9c16570f265818d417ba9a107b37842c4d5e44f0fb2ab76cef19ff992-d
packagefile html/template=C:\XXXXX\go\pkg\windows_amd64\html\template.a
packagefile encoding/hex=C:\XXXXX\go\pkg\windows_amd64\encoding\hex.a
packagefile github.com/hashicorp/hcl/hcl/scanner=C:\XXXXX\Temp\go-build\36\361f78b7d31e4fdea0c92df8f5f7eb4517a8eab090fef6d9c31a5e4d058fdb75-d
packagefile github.com/hashicorp/hcl/hcl/strconv=C:\XXXXX\Temp\go-build\ea\eae329e2ca698227192663f9a745192dbbced4855404c946c1e43a5e87da62fb-d
packagefile github.com/hashicorp/hcl/json/scanner=C:\XXXXX\Temp\go-build\a7\a7464a1a8b7c1b4547b3a71818191b63f77d136a32c1542b80d8f593e8186945-d
packagefile github.com/hashicorp/hcl/json/token=C:\XXXXX\Temp\go-build\44\44180e6fb1d7f7fccbffe65b7a5e64c5c7c92e65e876a6bc38a044e3911146a0-d
packagefile compress/gzip=C:\XXXXX\go\pkg\windows_amd64\compress\gzip.a
packagefile container/list=C:\XXXXX\go\pkg\windows_amd64\container\list.a
packagefile crypto/rand=C:\XXXXX\go\pkg\windows_amd64\crypto\rand.a
packagefile crypto/tls=C:\XXXXX\go\pkg\windows_amd64\crypto\tls.a
packagefile vendor/golang.org/x/net/http/httpguts=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\http\httpguts.a
packagefile vendor/golang.org/x/net/http/httpproxy=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\http\httpproxy.a
packagefile vendor/golang.org/x/net/http2/hpack=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\http2\hpack.a
packagefile vendor/golang.org/x/net/idna=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\idna.a
packagefile math/rand=C:\XXXXX\go\pkg\windows_amd64\math\rand.a
packagefile mime=C:\XXXXX\go\pkg\windows_amd64\mime.a
packagefile mime/multipart=C:\XXXXX\go\pkg\windows_amd64\mime\multipart.a
packagefile net/http/httptrace=C:\XXXXX\go\pkg\windows_amd64\net\http\httptrace.a
packagefile net/http/internal=C:\XXXXX\go\pkg\windows_amd64\net\http\internal.a
packagefile net/textproto=C:\XXXXX\go\pkg\windows_amd64\net\textproto.a
packagefile net/url=C:\XXXXX\go\pkg\windows_amd64\net\url.a
packagefile regexp/syntax=C:\XXXXX\go\pkg\windows_amd64\regexp\syntax.a
packagefile vendor/golang.org/x/net/dns/dnsmessage=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\net\dns\dnsmessage.a
packagefile internal/nettrace=C:\XXXXX\go\pkg\windows_amd64\internal\nettrace.a
packagefile internal/singleflight=C:\XXXXX\go\pkg\windows_amd64\internal\singleflight.a
packagefile html=C:\XXXXX\go\pkg\windows_amd64\html.a
packagefile text/template=C:\XXXXX\go\pkg\windows_amd64\text\template.a
packagefile text/template/parse=C:\XXXXX\go\pkg\windows_amd64\text\template\parse.a
packagefile compress/flate=C:\XXXXX\go\pkg\windows_amd64\compress\flate.a
packagefile hash/crc32=C:\XXXXX\go\pkg\windows_amd64\hash\crc32.a
packagefile math/big=C:\XXXXX\go\pkg\windows_amd64\math\big.a
packagefile crypto=C:\XXXXX\go\pkg\windows_amd64\crypto.a
packagefile crypto/aes=C:\XXXXX\go\pkg\windows_amd64\crypto\aes.a
packagefile crypto/cipher=C:\XXXXX\go\pkg\windows_amd64\crypto\cipher.a
packagefile crypto/des=C:\XXXXX\go\pkg\windows_amd64\crypto\des.a
packagefile crypto/ecdsa=C:\XXXXX\go\pkg\windows_amd64\crypto\ecdsa.a
packagefile crypto/ed25519=C:\XXXXX\go\pkg\windows_amd64\crypto\ed25519.a
packagefile crypto/elliptic=C:\XXXXX\go\pkg\windows_amd64\crypto\elliptic.a
packagefile crypto/hmac=C:\XXXXX\go\pkg\windows_amd64\crypto\hmac.a
packagefile crypto/md5=C:\XXXXX\go\pkg\windows_amd64\crypto\md5.a
packagefile crypto/rc4=C:\XXXXX\go\pkg\windows_amd64\crypto\rc4.a
packagefile crypto/rsa=C:\XXXXX\go\pkg\windows_amd64\crypto\rsa.a
packagefile crypto/sha1=C:\XXXXX\go\pkg\windows_amd64\crypto\sha1.a
packagefile crypto/sha256=C:\XXXXX\go\pkg\windows_amd64\crypto\sha256.a
packagefile crypto/sha512=C:\XXXXX\go\pkg\windows_amd64\crypto\sha512.a
packagefile crypto/subtle=C:\XXXXX\go\pkg\windows_amd64\crypto\subtle.a
packagefile crypto/x509=C:\XXXXX\go\pkg\windows_amd64\crypto\x509.a
packagefile encoding/pem=C:\XXXXX\go\pkg\windows_amd64\encoding\pem.a
packagefile vendor/golang.org/x/crypto/chacha20poly1305=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\chacha20poly1305.a
packagefile vendor/golang.org/x/crypto/cryptobyte=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\cryptobyte.a
packagefile vendor/golang.org/x/crypto/curve25519=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\curve25519.a
packagefile vendor/golang.org/x/crypto/hkdf=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\hkdf.a
packagefile hash=C:\XXXXX\go\pkg\windows_amd64\hash.a
packagefile vendor/golang.org/x/text/secure/bidirule=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\secure\bidirule.a
packagefile vendor/golang.org/x/text/unicode/bidi=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\unicode\bidi.a
packagefile vendor/golang.org/x/text/unicode/norm=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\unicode\norm.a
packagefile mime/quotedprintable=C:\XXXXX\go\pkg\windows_amd64\mime\quotedprintable.a
packagefile crypto/internal/subtle=C:\XXXXX\go\pkg\windows_amd64\crypto\internal\subtle.a
packagefile crypto/internal/randutil=C:\XXXXX\go\pkg\windows_amd64\crypto\internal\randutil.a
packagefile vendor/golang.org/x/crypto/cryptobyte/asn1=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\cryptobyte\asn1.a
packagefile crypto/ed25519/internal/edwards25519=C:\XXXXX\go\pkg\windows_amd64\crypto\ed25519\internal\edwards25519.a
packagefile crypto/dsa=C:\XXXXX\go\pkg\windows_amd64\crypto\dsa.a
packagefile crypto/x509/pkix=C:\XXXXX\go\pkg\windows_amd64\crypto\x509\pkix.a
packagefile encoding/asn1=C:\XXXXX\go\pkg\windows_amd64\encoding\asn1.a
packagefile vendor/golang.org/x/crypto/chacha20=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\chacha20.a
packagefile vendor/golang.org/x/crypto/internal/subtle=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\internal\subtle.a
packagefile vendor/golang.org/x/crypto/poly1305=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\crypto\poly1305.a
packagefile vendor/golang.org/x/sys/cpu=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\sys\cpu.a
packagefile vendor/golang.org/x/text/transform=C:\XXXXX\go\pkg\windows_amd64\vendor\golang.org\x\text\transform.a
EOF
mkdir -p $WORK\b001\exe\
cd .
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\link.exe" -o "$WORK\\b001\\exe\\a.out.exe" -importcfg "$WORK\\b001\\importcfg.link" -buildmode=pie -buildid=cLYmR2vpJ0BXtT-SxR4k/JJ3Jf0mSmjshQOlN5BMT/7qbI6Ur-PSPn5ckbZQJl/cLYmR2vpJ0BXtT-SxR4k -extld=gcc "$WORK\\b001\\_pkg_.a"
"C:\\XXXXX\\go\\pkg\\tool\\windows_amd64\\buildid.exe" -w "$WORK\\b001\\exe\\a.out.exe" # internal
cp $WORK\b001\exe\a.out.exe prog.bin.exe
rm -r $WORK\b001\

Just another thought:
It seems that mcafee acquires one or two threads in the running go process.
Maybe the following could be an explanation:
MAXPROCS=1
mcafee does not even have a chance to start a thread and do strange stuff --> everything works
MAXPROCS=8
enough threads (time) to start up the mcafee hook. since mcafee wants to check the binary before execution, it blocks evrything else but needs some other threads, that it does not get because of the MAXPROCS=8, so it waits forever and also blocks all others forever
go runtime also waits for a thread, since it does not really know that there is some "stranger" also requesting threads and so go also blocks because go runtime is not aware of threads beeing stolen.
That is my guess and would fit to my following observation:
The build again runs smoothly, when setting GOMAXPROCS to a high value like 256.
At least 90% of the time, unfortunately it also gets stuck time by time
Maybe this is enough on the one hand for go itself and on the other hand also for mcafee to do whatever it does...
64 and 128 for example still does not seem to be enough --> again always hanging
256 only works when directly invoking go build.
When trying this with go generate ./.. it again hangs after sometime. At least later in time.
When setting to 1024 , then also the go generate runs without problems (most of the time)
Any thoughts on that?
I can just guess here since I have no real clue.

Seems this is a bug in mcafee av

It is possible that the McAfee DLL is hijacking the threads when the Go runtime makes a system call under the assumption that every thread “probably” makes a system call eventually.
Some Go threads essentially never make system calls, so they would not be hijackable. That may be what leads to the observed deadlock.
If that is correct, then a workaround on the Go side might look like the workaround for #18717 (CL 35494), in which the Go runtime periodically makes an inexpensive but otherwise needless call to an arbitrary function in the C standard library in order to allow hooks embedded in that library to run on the thread.
(CC @aclements @ianlancetaylor @alexbrainman)
That said, even if this hypothesis is correct I would rather have McAfee fix their faulty assumption than us work around it.

(And by the way, @ybbus, thanks for the detailed investigation of what's running on those threads!)

Ah, one other experiment you could try (via #36494). @ybbus, what happens if you set GODEBUG=asyncpreemptoff=1 in the environment, still with GOMAXPROCS=8? (Does that change the frequency of the hangs?)

I'd imagine you can configure McAfee to ignore your program.
This problem is not unique to McAfee; we've heard reports where the Go toolchain runs fine, but the binaries it builds fail due to AV interference.

If that is correct, then a workaround on the Go side might look like the workaround for #18717 (CL 35494)

Sorry I don't know enough to comment.

what happens if you set GODEBUG=asyncpreemptoff=1 in the environment,

I agree. This might help. I have similar unresolved issue #36492. @aclements thinks it is my buggy antivirus program. Mind you, it is different antivirus vendor.
Alex

@networkimprov
I'll try to discuss that, but this seems not to be very easy in a big company. Also go does generate a lot random name binaries that you do not know upfront, and if the problem also exists there (not sure yet), you would need to exclude a lot bunch of folders from the scan. Nothing the security guys like to see, but I will try to address this.
@bcmills
Indeed, setting asyncpreemtoff solves the problem. I was not able to reproduce the issue with MAXPROCS=8 over several runs.
What are the implications of using this flag in my daily development? Is it ok to let this enabled all the time?

You're probably better off setting asyncpreemptoff, which switches the runtime to cooperative scheduling of goroutines on available cores (GOMAXPROCS) -- it was the default until 1.14.
Since that fixes it, maybe McAfee (or your IT dept) has an update planned.

IT dept allowed to uninstall McAfee temporary --> issue disappeard
Unfortunatly after reinstalling McAfee, issue is still gone. I will leave it with that, and if the issue again comes up ask the IT guys to talk to McAfee about that.
Thanks for all your help. I now have at least some good short term workaround options to continue working when the issue comes up again.
From my side, this issue may be closed and hopefully helps other devs to faster get to the root cause of this problem (took me quite a while to be honest)

Thanks for following up.  Closing because it doesn't seem that there is anything we can change in Go to fix this.

Closing because it doesn't seem that there is anything we can change in Go to fix this.

I disagree.
I think Go should have asyncpreemptoff=1 by default on Windows.
None of our Windows users complained about preemption being a problem, but now @ybbus and I are suffering.
So far, we only know about developers are affected. But maybe this also affects users who run our programs.
And, while I agree that the problem is most likely with buggy antivirus software, that is the world we have to live in.
Alex

That's a good point. I wondered why async preemption was rolled out as the default, instead of an option.
To make matters worse, we can't disable it in a build, and my proposal to allow that was quickly rejected, #40870.

Lets take async preemption on Windows to a different issue, not this one.  Thanks.

authors: alexbrainman bcmills davecheney ianlancetaylor networkimprov ybbus


text:
x/tools/cmd/getgo: uses incorrect "latest" url

What version of Go are you using (go version)?

$ go version
1.5.55

But the golang/tools commit that has this problem is the latest as of today 2020/11/17
9036a0f9af11674fea137b794a0a2e6aef6882e1
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
Not relevant but:
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/wizard/.cache/go-build"
GOENV="/home/wizard/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/mnt/extra/gopath/pkg/mod"
GONOPROXY="github.com/tradellama"
GONOSUMDB="github.com/tradellama"
GOOS="linux"
GOPATH="/mnt/extra/gopath"
GOPRIVATE="github.com/tradellama"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/mnt/extra/code/work/tradellama/goat/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build370880754=/tmp/go-build -gno-record-gcc-switches"

What did you do?

curl -LO https://get.golang.org/$(uname)/go_installer && chmod +x go_installer && ./go_installer -i
Got:

Welcome to the Go installer!
Would you like to install Go? Y/n [Y]: y
The latest go version is go1.15.4, install that? Y/n [Y]:
What did you expect to see?
go1.15.5 being offered
What did you see instead?
go.1.15.4
Possible fix:
this line
https://github.com/golang/tools/blob/master/cmd/getgo/download.go#L24
gets the version that the golang.org site "runs" but at this this time, the site doesn't run the latest. We should instead use:
https://golang.org/dl/?mode=json
Which was suggested on the mailing list
https://groups.google.com/g/golang-nuts/c/kweFo4TE4N0/m/XyH-tI0OBQAJ

CC @golang/release.

I agree that switching it to use the https://golang.org/dl/?mode=json API endpoint is the right fix.

@fmpwizard CL 270878 has some outstanding review comments, once addressed it should be good to submit. Are you able to take a look?

Hi @dmitshur yes, I'll take care of the comments, it may be a few more days but I'm still planning on addressing them.
Thank you!

That sounds good, thank you!

source: gharchive/issue

authors: dmitshur fmpwizard

repo: golang/go

license_type: permissive

license_source: github-api


text:
x/build: enable builder-specific GOPROXY settings on reverse builders

Looking at the builder log in https://build.golang.org/log/26ea2fd1426a51a224b5d321915717e906ded84a, I see the following line. Note that it lists GOPROXY three different times and GOROOT_BOOTSTRAP twice:

:: Running /tmp/farm/go/bin/go with args ["/tmp/farm/go/bin/go" "test" "-short" "golang.org/x/crypto/..."] and env ["LANG=zh_CN.UTF-8" "PATH=/tmp/farm/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/bash" "INVOCATION_ID=fe969678216d4cceaebbdc436df708c2" "JOURNAL_STREAM=8:4142335" "GO_BUILDER_ENV=host-linux-amd64-wsl" "WORKDIR=/tmp/farm" "HTTPS_PROXY=http://proxy:8123" "HTTP_PROXY=http://proxy:8123" "GOPROXY=https://goproxy.io,direct" "WSL_DISTRO_NAME=Ubuntu" "WSL_INTEROP=/run/WSL/20_interop" "WSLENV=BASH_ENV/u" "GO_STAGE0_NET_DELAY=700ms" "GO_STAGE0_DL_DELAY=600ms" "GOROOT_BOOTSTRAP=/tmp/farm/go1.4" "GO_BUILDER_NAME=linux-amd64-wsl" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/lib/go" "GOROOT=/tmp/farm/go" "GOPATH=/tmp/farm/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=https://proxy.golang.org" "TMPDIR=/tmp/farm/tmp" "GOCACHE=/tmp/farm/gocache"] in dir /tmp/farm/gopath/src/golang.org/x/crypto

The second GOROOT_BOOTSTRAP seems to come from here, and might be working as intended: https://github.com/golang/build/blob/a9f75ee9992b28221ceec0c56af51265ea857ce2/cmd/buildlet/buildlet.go#L263
The second GOPROXY setting comes from here: https://github.com/golang/build/blob/5f1e306b98a112c236997d393adc5c11405377ca/cmd/coordinator/coordinator.go#L2702
The third GOPROXY setting comes from here: https://github.com/golang/build/blob/b6220de1adc000cad31eadf59558e1d3a8e32205/dashboard/builders.go#L921-L925

Given that one of the tests in that log failed with an EOF error communicating with proxy.golang.org, I suspect that it really should be using the local GOPROXY default (https://goproxy.io,direct) instead of the cmd/go default. I'm not sure what needs to happen to fix that configuration.
CC @mengzhuo @cagedmantis @golang/release

Please assign this issue to me. @bcmills
p.s. Why can't I assign this issue to myself?

To other builders maintainer, linux-amd64-wsl using systemd to set the enviroment.
[Unit]
Description=Go Builder For WSL
After=network-online.target

[Service]
ExecStartPre=/bin/bash -c 'chmod -R u+w /tmp/farm; rm -rf /tmp/farm; /bin/mkdir -p /tmp/farm; true'
ExecStart=/home/gopher/go/bin/stage0
Environment=GO_BUILDER_ENV=host-linux-amd64-wsl
Environment=WORKDIR=/tmp/farm
Environment=HTTPS_PROXY=http://proxy:8123
Environment=HTTP_PROXY=http://proxy:8123
Environment=GOPROXY=https://goproxy.io,direct

User=gopher
WorkingDirectory=/home/gopher

Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

@mengzhuo Take a look at https://golang.org/wiki/GitHubAccess#assignees.

If anything else need to be change, I'd like to push it.

If anything else need to be change, I'd like to push it.

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2021-01-01

2022-02-04T14:01:19-77f5fa5-4afcc9f/linux-amd64-wsl
2022-02-02T23:37:18-6944b10-a9384ee/linux-amd64-wsl
2022-02-01T21:46:52-d1c84af-36b81ac/linux-riscv64-unmatched
2022-02-01T19:09:20-6944b10-e22a14b/linux-amd64-wsl
2022-01-27T16:35:59-03fcf44-6cbcf58/linux-amd64-wsl
2022-01-25T22:04:10-03971e3-38729cf/linux-mips64le-mengzhuo
2022-01-18T16:01:17-e26b9f0-378766a/linux-amd64-wsl
2022-01-05T23:17:27-416c73c-2b39d86/linux-amd64-wsl
2022-01-05T09:50:29-f2171f2-2c58bb2/linux-riscv64-unmatched
2022-01-04T21:07:03-c09c1d3-95b240b/linux-riscv64-unmatched
2022-01-04T16:08:05-cd4a9ba-95b240b/linux-riscv64-unmatched
2021-12-30T12:17:33-53d65af-6178d25/linux-riscv64-unmatched
2021-12-30T12:11:21-084fe85-6178d25/linux-riscv64-unmatched
2021-12-20T22:25:28-30823e0-7c94355/linux-amd64-wsl
2021-12-16T22:15:57-b76863e-d01cd8f/linux-amd64-wsl
2021-12-15T15:54:46-d92f387-de690c2/linux-amd64-wsl
2021-12-04T11:59:35-c33205f-1876b38/linux-amd64-wsl
2021-12-02T05:26:57-ae814b3-1b2930d/linux-amd64-wsl
2021-12-01T15:58:46-ae814b3-029dfbc/linux-amd64-wsl
2021-12-01T15:58:46-6944b10-029dfbc/linux-amd64-wsl
2021-12-01T13:12:25-eb34ac4-b7651e5/linux-amd64-wsl
2021-11-22T03:24:07-03fcf44-a287c4a/linux-amd64-wsl
2021-11-20T08:47:36-ae814b3-91abe4b/linux-amd64-wsl
2021-11-19T21:36:24-b76863e-aec5c2e/linux-amd64-wsl
2021-11-10T02:26:41-03fcf44-02d7eab/linux-riscv64-unmatched
2021-11-02T06:25:39-4a448f8-088bb4b/linux-mips64le-mengzhuo
2021-09-30T01:32:54-22bfdb6-6656269/linux-riscv64-unmatched
2021-09-06T14:02:24-ddb58f2-ecfff58/linux-amd64-wsl
2021-09-02T12:49:05-027fa76-08e2519/linux-amd64-wsl
2021-08-17T04:37:32-b64cdd1-a304273/linux-amd64-wsl
2021-08-16T18:44:48-5d84be3-d1ba047/linux-amd64-wsl
2021-08-16T18:44:48-0a44fdf-d1ba047/linux-amd64-wsl
2021-04-02T17:58:59-cd4b3ce-254948a/linux-amd64-wsl

@golang/release: this is a frequent failure mode on multiple reverse builders, and there has been a fix CL CL 275412 out for review since December 2020. Can we prioritize a review for that change at some point in the near future?

@golang/release, any progress to share on this issue? As far as I can tell the CL from 2020 still hasn't been picked up, and I'm still seeing a lot of errors on the builders that look like they could be resolved by this:
greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2022-02-04
2022-04-18T12:04:50-9780585-91b9915/linux-amd64-wsl
2022-04-18T12:04:50-46bc274-91b9915/linux-amd64-wsl
2022-04-18T12:04:50-39d8c21-91b9915/linux-amd64-wsl
2022-04-18T12:04:50-290c469-91b9915/linux-amd64-wsl
2022-04-16T11:17:39-506dc81-5e4543c/linux-mips64le-mengzhuo
2022-04-15T17:07:27-807ebe9-df2421d/linux-amd64-wsl
2022-04-14T15:54:48-fc48f9f-cb702a2/linux-mips64le-mengzhuo
2022-04-13T18:05:38-7838acb-346b18e/linux-amd64-wsl
2022-04-12T23:50:58-99f80d0-65558a4/linux-amd64-wsl
2022-04-12T23:24:32-fb223f7-9c7f0b1/linux-amd64-wsl
2022-04-12T18:02:22-caaab50-0b0d2fe/linux-amd64-wsl
2022-04-12T16:47:01-64c5db5-5bf6c97/linux-amd64-wsl
2022-04-12T15:47:54-64c5db5-45c3387/linux-amd64-wsl
2022-04-12T14:45:32-99f80d0-84264fc/linux-amd64-wsl
2022-04-12T14:42:58-e5f449a-2116d60/linux-amd64-wsl
2022-04-12T13:54:06-99f80d0-9ccf5b8/linux-amd64-wsl
2022-04-12T12:59:27-99f80d0-d6320f1/linux-amd64-wsl
2022-04-12T03:13:39-e5f449a-f2d9ab2/linux-amd64-wsl
2022-04-12T03:13:28-fc48f9f-e299381/linux-amd64-wsl
2022-04-12T02:13:10-99f80d0-a362d54/linux-amd64-wsl
2022-04-11T22:24:03-e5f449a-11c450f/linux-amd64-wsl
2022-04-11T16:34:30-1b505d1-1930977/linux-mips64le-mengzhuo
2022-04-07T20:39:32-5424468-6f6942e/linux-amd64-wsl
2022-04-06T22:27:40-03fcf44-9a6acc8/linux-amd64-wsl
2022-04-06T03:13:34-5424468-5bb2628/linux-amd64-wsl
2022-04-05T22:09:11-03fcf44-dbb52cc/linux-amd64-wsl
2022-04-05T17:00:25-a8550c1-65153e4/linux-amd64-wsl
2022-04-05T01:54:27-1983b93-07b0c57/linux-mips64le-mengzhuo
2022-04-05T01:32:49-1983b93-4aacb7f/linux-amd64-wsl
2022-04-04T04:01:17-03fcf44-a92ca51/linux-mips64le-mengzhuo
2022-04-03T15:43:15-5424468-85b5f86/linux-mips64le-mengzhuo
2022-03-31T15:59:22-92d54ef-5138401/linux-amd64-wsl
2022-03-19T23:49:55-723b81c-fa8efc1/linux-amd64-wsl
2022-03-19T23:49:55-3147a52-fa8efc1/linux-amd64-wsl
2022-03-19T23:49:55-3147a52-fa8efc1/linux-mips64le-mengzhuo
2022-03-15T00:19:35-b76863e-41a82aa/linux-mips64le-mengzhuo
2022-03-14T17:45:32-9d39a96-88be85f/linux-amd64-wsl
2022-03-08T21:35:38-03fcf44-dbbe4cc/linux-mips64le-mengzhuo
2022-03-07T23:22:26-9b9b3d8-079a027/linux-amd64-wsl
2022-02-15T01:01:15-9d39a96-dd7194b/linux-riscv64-unmatched
2022-02-11T00:30:56-9d39a96-30501bb/linux-amd64-wsl
2022-02-11T00:26:29-9d39a96-ca3fae1/linux-amd64-wsl
2022-02-11T00:26:18-9d39a96-11788aa/linux-amd64-wsl
2022-02-11T00:26:18-6944b10-11788aa/linux-amd64-wsl
2022-02-10T23:04:01-3450d37-cd6e0d7/linux-amd64-wsl
2022-02-10T22:20:18-d5e7b87-c4b87b8/linux-mips64le-mengzhuo
2022-02-10T22:20:18-cd36cc0-c4b87b8/linux-amd64-wsl
2022-02-10T22:20:18-bff2e91-c4b87b8/linux-mips64le-mengzhuo
2022-02-10T22:20:18-9d39a96-c4b87b8/linux-amd64-wsl
2022-02-10T22:20:18-9d39a96-c4b87b8/linux-mips64le-mengzhuo
2022-02-10T22:20:18-1a07ae6-c4b87b8/linux-amd64-wsl
2022-02-10T22:20:18-03fcf44-c4b87b8/linux-amd64-wsl
2022-02-10T21:40:49-9d39a96-99b61be/linux-amd64-wsl
2022-02-10T21:40:49-6944b10-99b61be/linux-amd64-wsl
2022-02-10T21:40:12-f4118a5-3b8c716/linux-amd64-wsl
2022-02-10T21:40:12-03971e3-3b8c716/linux-amd64-wsl
2022-02-10T19:08:19-d5e7b87-8ba3ad9/linux-mips64le-mengzhuo
2022-02-10T19:08:19-c0b9fb5-8ba3ad9/linux-mips64le-mengzhuo
2022-02-10T19:08:19-9d39a96-8ba3ad9/linux-mips64le-mengzhuo
2022-02-10T01:08:52-d3ed0bb-656d3f4/linux-riscv64-unmatched
2022-02-09T16:29:28-9d39a96-0a6cf87/linux-amd64-wsl
2022-02-04T14:01:19-77f5fa5-4afcc9f/linux-amd64-wsl

CL 275412 has now landed. Comparing a recent before/after log shows it's working as intended: there's no more unwanted "GOPROXY=https://proxy.golang.org" entry.
We can check again in a few weeks to make sure the same problem doesn't happen again for another reason, but at least so far there aren't any new hits (other than one yesterday, before the fix):
greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2022-04-28
2022-04-28T16:00:39-dcaea06-17371ee/linux-amd64-wsl

Unfortunately, goproxy.io doesn't seem to be entirely reliable either. 😞
https://build.golang.org/log/58e02729ea56983be383b850ca96f268ba258e4b:
:: Running /tmp/workdir-host-linux-amd64-wsl/go/bin/go with args ["/tmp/workdir-host-linux-amd64-wsl/go/bin/go" "test" "-short" "golang.org/x/website/..."] and env ["LANG=zh_CN.UTF-8" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/bash" "INVOCATION_ID=4b8fac3e13e44f2ca8c6759ce9cfce12" "JOURNAL_STREAM=8:47829246" "GO_BUILDER_ENV=host-linux-amd64-wsl" "HTTPS_PROXY=http://uplink.golangcn.org:8123" "HTTP_PROXY=http://uplink.golangcn.org:8123" "GOPROXY=https://goproxy.io,direct" "WSL_DISTRO_NAME=Ubuntu" "WSL_INTEROP=/run/WSL/20_interop" "WSLENV=BASH_ENV/u" "GO_STAGE0_NET_DELAY=400ms" "GO_STAGE0_DL_DELAY=1.3s" "WORKDIR=/tmp/workdir-host-linux-amd64-wsl" "GO_BUILDER_NAME=linux-amd64-wsl" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/lib/go" "GOROOT=/tmp/workdir-host-linux-amd64-wsl/go" "GOPATH=/tmp/workdir-host-linux-amd64-wsl/gopath" "GO111MODULE=on" "TMPDIR=/tmp/workdir-host-linux-amd64-wsl/tmp" "GOCACHE=/tmp/workdir-host-linux-amd64-wsl/gocache" "PATH=/tmp/workdir-host-linux-amd64-wsl/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/tmp/workdir-host-linux-amd64-wsl/gopath/src/golang.org/x/website"] in dir /tmp/workdir-host-linux-amd64-wsl/gopath/src/golang.org/x/website

…
cmd/events/main.go:16:2: github.com/microcosm-cc/bluemonday@v1.0.2: Get "https://proxy.golang.com.cn/github.com/microcosm-cc/bluemonday/@v/v1.0.2.zip": Service Unavailable

cc @oiooj as goproxy.io maintainer

I tested this request and there is no problem so far.
curl -v https://proxy.golang.com.cn/github.com/microcosm-cc/bluemonday/@v/v1.0.2.zip

and goproxy.io use Tencent Cloud global CDN service. 😢

greplogs -l -e 'https://(?:goproxy\.io|proxy.golang.com.cn).*(?:unexpected EOF|Service Unavailable|connection reset)' --since=2022-04-29
2022-05-03T12:34:17-fb9be63-a41e37f/linux-amd64-wsl
2022-05-02T19:53:56-fb9be63-3b01a80/linux-amd64-wsl
2022-04-29T18:37:11-fb9be63-8c5917c/linux-amd64-wsl

At any rate: it looks like the builder-specific GOPROXY settings are working as intended, so I'm going to close this issue.
I'll file a separate issue for the connection errors on the builders using goproxy.io.

(Filed the goproxy.io connectivity issue as #52793.)

authors: bcmills dmitshur mengzhuo oiooj


text:
embed: FS root directory is always the package directory, while os.DirFS may have another root

What version of Go are you using (go version)?

go version devel +2ff33f5e44 Thu Dec 17 16:03:19 2020 +0000 linux/amd64

(that's 1.16beta1)
Does this issue reproduce with the latest release?
yes
What did you do?
I want to serve static documents in a webserver using the new embed package, but with an option to use local files.
https://play.golang.org/p/hyBgSUBUhf3
There is a subdir ./docroot/ with an index.html file:
myuser@mymachine:/tmp/root$ ls -R
.:
docroot/  go.mod  main.go  root

./docroot:
index.html

When I use the os.DirFS() version (./root -docroot ./docroot) I get the index.html nicely at http://localhost:6602/index.html. If I use the embed version (./root) it ends up at http://localhost:6602/docroot/index.html
What did you expect to see?
I expect either none or both to have the ./docroot/ subdir in there. This way I can't use http.StripPrefix(). But maybe I missed something else.
What did you see instead?
The embed.FS and os.DirFS seem to behave inconsistently.
If embed.FS implemented fs.SubFS I think I could make this work by using docroot.Sub("./docroot/")

You can do something like this:
//go:embed docroot/*
var static embed.FS

// myFS implements fs.FS
type myFS struct {
	content embed.FS
}

func (c myFS) Open(name string) (fs.File, error) {
	return c.content.Open(path.Join("docroot", name))
}

Then:
r.Handle("/", http.FileServer(http.FS(myFS{root})))

It seems to me that embed.FS and os.DirFS are working as intended, you will need to arrange your code to either handle different roots, or have the same root by running your program with ./root -docroot=. instead of ./root -docroot=./docroot.
Since it's a new API, I'll let @rsc consider if anything more should be done.

Thanks for the quick responses.
Is there anything against embed.FS implementing fs.SubFS? That would make it easier to deal with this case.

@changkun solution would work great. I ended up adding a single file inside the docroot:
$ cat docroot/fs.go
package docroot

import "embed"

//go:embed *
var FS embed.FS

and importing that. It's not pretty, but works.
(as far as I'm concerned this issue can be closed)

@alicebob Indeed. I noticed the behavior you reported in this thread while I was trying to convert my static website here.
I didn't report this simply don't know that is the actually expected behavior from these new set of APIs. The design draft is just a draft, as far as I partially skimmed with the ongoing discussion. The current implementation is already apart from the draft design (lack of documentation).
For more instance, I'd like to directly embed everything in a repository (and of course exclude the .git folder):
$ l
total 16
drwxr-xr-x   5 changkun  staff   160B Dec 31 13:56 .
drwxr-xr-x  29 changkun  staff   928B Dec 31 13:49 ..
drwxr-xr-x   9 changkun  staff   288B Dec 31 13:50 .git
-rw-r--r--   1 changkun  staff    18B Dec 31 13:56 go.mod
-rw-r--r--   1 changkun  staff   102B Dec 31 13:54 main.go

$ cat main.go
package main

import "embed"

//go:embed *
var static embed.FS

func main() {
        println(static)
}

With the (undocumented) https://github.com/golang/go/commit/37588ffcb221c12c12882b591a16243ae2799fd1, the .git should be ignored and have a successful build.
But unfortunately, build such a simple program can result in the following compile error:
$ go build
pattern *: cannot embed directory .git: invalid name .git

Am I misunderstanding or does this use of fs.Sub do what you want? https://github.com/carlmjohnson/exembed/blob/main/web/main.go

Am I misunderstanding or does this use of fs.Sub do what you want? https://github.com/carlmjohnson/exembed/blob/main/web/main.go

Oh, how cloud I miss the fs.Sub, thanks for the hint. It would be great to have an example like this in a *_example.go and demonstrate in the document.

@carlmjohnson awesome, yes that solves everything.
I misread the output of go doc: I saw SubFS interface, which embed.FS doesn't implement, but I missed the pure Sub() function.

@carlmjohnson awesome, yes that solves everything.
I misread the output of go doc: I saw SubFS interface, which embed.FS doesn't implement, but I missed the pure Sub() function.

authors: alicebob carlmjohnson changkun dmitshur


text:
cmd/pprof: disassembly support for PIE binaries / shared libraries

go tool pprof does not support disassembly of position independent code (PIE binaries or shared libraries). Notably, on Windows PIE is default, so disassembly doesn't work by default:
gopher@SERVER-2016-V7- c:\workdir>go\bin\go build cpu.go

gopher@SERVER-2016-V7- c:\workdir>cpu.exe -output cpu.pprof

gopher@SERVER-2016-V7- c:\workdir>pprof.exe -top cpu.exe cpu.pprof
File: cpu.exe
Type: cpu
Time: Jun 7, 2021 at 9:19pm (GMT)
Duration: 1.11s, Total samples = 990ms (89.48%)
Showing nodes accounting for 990ms, 100% of 990ms total
      flat  flat%   sum%        cum   cum%
     330ms 33.33% 33.33%      840ms 84.85%  time.Since
     280ms 28.28% 61.62%      280ms 28.28%  time.Time.Sub
     150ms 15.15% 76.77%      990ms   100%  main.main
     130ms 13.13% 89.90%      130ms 13.13%  runtime.nanotime1
     100ms 10.10%   100%      230ms 23.23%  runtime.nanotime
         0     0%   100%      990ms   100%  runtime.main

gopher@SERVER-2016-V7- c:\workdir>pprof.exe -disasm . cpu.exe cpu.pprof
Total: 990ms

Without PIE:
gopher@SERVER-2016-V7- c:\workdir>go\bin\go build -buildmode=exe cpu.go

gopher@SERVER-2016-V7- c:\workdir>cpu.exe -output cpu.pprof

gopher@SERVER-2016-V7- c:\workdir>pprof.exe -disasm . cpu.exe cpu.pprof
Total: 990ms
ROUTINE ======================== time.Time.Sub
     310ms      310ms (flat, cum) 31.31% of Total
...

Upstream pprof's binutils backend supports base address computation to handle position independent code, but our implementation skips that entirely.
The "binutils" implementation doesn't actually seem particularly specific to binutils, so it would likely not be too difficult to support in our implementation.
cc @cherrymui @mknyszek

See also #17883.

authors: ianlancetaylor prattmic


text:
misc/wasm: wasm_exec.js does not enable strict mode

What version of Go are you using (go version)?

$ go version
go version go1.16.4 darwin/amd64

Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rhysd/Library/Caches/go-build"
GOENV="/Users/rhysd/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/rhysd/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/rhysd/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.16.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.16.5/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/rhysd/.go/src/github.com/rhysd/actionlint/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/98/2tcw8j4157578xpvbqfrwdyr0000gn/T/go-build2360158005=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
I'm running Wasm file with browsers which was compiled from Go source in my project. It works fine, but when I read misc/exec_wasm.js, I noticed that it did not enable strict mode. Enabling strict mode means:
What did you expect to see?
The MDN document says:

Strict mode makes several changes to normal JavaScript semantics:

Eliminates some JavaScript silent errors by changing them to throw errors.
Fixes mistakes that make it difficult for JavaScript engines to perform optimizations: strict mode code can sometimes be made to run faster than identical code that's not strict mode.
Prohibits some syntax likely to be defined in future versions of ECMAScript.

Due to historical reasons, browsers don't enable strict mode by default, but it should be enabled to avoid several JS pitfalls. For example, TypeScript recommends enabling strict mode.
If this change looks good to maintainers, I could submit a patch for this following the strict mode migration guide.
What did you see instead?
misc/exec_wasm.js did not enable strict mode.

cc @neelance @cherrymui

Strict mode should probably be fine. :+1:

Can I try to make changes for this issue following the guideline?

Sure :)

@neelance I made change for fixing this issue. Would you take a look?

@neelance How can we proceed the review? If I need to do something, please let me know. I'll do it.

authors: neelance rhysd seankhliao


text:
net: Listen("tcp", "0.0.0.0:1234") binds to IPv6 addresses

What version of Go are you using (go version)?

go version go1.17.1 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ncw/.cache/go-build"
GOENV="/home/ncw/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ncw/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ncw/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/go/go1.17"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/go/go1.17/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.1"
GCCGO="/usr/bin/gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3014442064=/tmp/go-build -gno-record-gcc-switches"

What did you do?
Run this program
package main

import (
	"fmt"
	"log"
	"net"
	"os"
	"os/exec"
	"strings"
)

func main() {
	if len(os.Args) != 2 {
		log.Fatalf("Syntax: %s bind_address", os.Args[0])
	}
	bindAddr := os.Args[1]
	go func() {
		_, err := net.Listen("tcp", bindAddr)
		if err != nil {
			log.Fatalf("net.Listen failed: %v", err)
		}
	}()
	// Read netstat
	cmd := exec.Command("netstat", "-tuanp")
	out, err := cmd.Output()
	if err != nil {
		log.Fatal(err)
	}
	// Print netstat lines with our PID in
	pid := fmt.Sprint(os.Getpid())
	for _, line := range strings.Split(string(out), "\n") {
		if strings.Contains(line, pid) {
			fmt.Println(line)
		}
	}
}

I get this
$ ./listen-bug 127.0.0.1:1234
tcp        0      0 127.0.0.1:1234          0.0.0.0:*               LISTEN      3179765/./listen-bu
$ ./listen-bug 0.0.0.0:1234
tcp6       0      0 :::1234                 :::*                    LISTEN      3179773/./listen-bu

Note the second of these is listening on IPv6 even though we explictly gave it an IPv4 bind address.
What did you expect to see?
I expected to see it not binding to IPv6 addresses, something like this
$ ./listen-bug 0.0.0.0:1234
tcp        0      0 0.0.0.0:1234          0.0.0.0:*               LISTEN      3179765/./listen-bu

What did you see instead?
I see it binding to IPv4 and IPv6 addresses
$ ./listen-bug 0.0.0.0:1234
tcp6       0      0 :::1234                 :::*                    LISTEN      3179773/./listen-bu

This was originally discussed in the rclone forum

https://pkg.go.dev/net#Listen

For TCP networks, if the host in the address parameter is empty or a literal unspecified IP address, Listen listens on all available unicast and anycast IP addresses of the local system. To only use IPv4, use network "tcp4".

As @AlexanderYastrebov points out, this appears to be working as intended. Closing optimistically. Reply if you disagree or some new information arises.
CC @neild

authors: AlexanderYastrebov mknyszek ncw


text:
go/types, types2: nil underlying for selectors on the current type decl

The following code crashes the type checker with a "nil underlying" panic, because we try to check the underlying when selection fails:
type X X.t

https://go.dev/play/p/H3Dv94uxQL9?v=gotip
Marking as a release blocker as this is a crash.
CC @griesemer

It's notoriously difficult to prevent crashes in a compiler. We should fix this of course, but I think we'd be ok even if this made it into the release as this is pretty pathological.

Yes: a compiler crash should be fixed, of course, but in general a compiler crash on invalid code is not a release blocker.  It's just a really really bad error message.

Agree that it may not be a release blocker, though it also crashes gopls in which case it is more than an error message.
One argument for the severity / commonality of this pattern is that it was introduced in https://go.dev/cl/378175, i.e. post beta, and seems to have been encountered pretty quickly inside of Google.

authors: findleyr griesemer ianlancetaylor


text:
x/pkgsite: duplicate drop down button

What is the URL of the page with the issue?
It is https://pkg.go.dev/std, but actually all of the pages like the former (from pkg.go.dev/xxx) represent the same.
What is your user agent?
Mozilla/5.0 (Android 10; Mobile; rv:106.0) Gecko/106.0 Firefox/106.0
btw, my cellphone's screen resolution is	360 X 800
Screenshot

What did you do?
I just visit the website using firefox browser on my phone

What did you expect to see?
Like

without duplicate drop down btns.
What did you see instead?
already showed above

I can not reproduce the issue. This is my browser:
Mozilla/5.0 (Android 12; Mobile; rv:106.0) Gecko/106.0 Firefox/106.0

@ZekeLu result's below

the same.
But I noticed your Android is 12, 2 version above mine(Android 10), so I guess is there anything relatived to this?
Just guess, I have no idea about that.

CC @jamalc
I also cannot reproduce, either on my mobile device or in an emulator with the same resoluition.

Well.. My browser is to blame... I've just re-install the firefox and nothing went wrong 🙂

Thx 4 ur quickly reply :)

It is weird... This had relapsed after I re-installed my browser after several days... well, never mind. It seems it's all about my bowser 😳

authors: ZekeLu findleyr sslime336


text:
cmd/trace: visualize time taken by syscall

Currently (Go 1.20-dev), syscalls as processed by go tool trace from a runtime/trace profile visualize as points in time, being about 1 pixel wide no matter how long they took (see reproducer below). It would be nicer if they were slices, to visualize how much time is spent in (conceptually blocking) syscalls for a given goroutine.

Somewhere in the last release cycle (1.19), I tried making the obvious patch to cmd/trace:
                case trace.EvGoSysCall:
-                       ctx.emitInstant(ev, "syscall", "")
+                       if ev.Link != nil {
+                               ctx.emitSlice(ev, "syscall")
+                       } else {
+                               ctx.emitInstant(ev, "syscall", "")
+                       }

And I remembered it didn't work: most (all?) EvGoSysCall events don't have a link to the finish event. But before filing this proposal I tried patching up cmd/trace from 1.20 and to my surprise it did work. I assume something changed in Go 1.20 and now more  (most?) EvGoSysCall events have a link, which is great. Debugging output of my patched go tool trace:
...
2022/12/14 05:42:10 syscall event with link (6/12):
    101597987   GoSysCall       p=1     g=1     off=2876        stkID=22        seq=0
 -> 101613258   GoSysExit       p=1000003       g=1     off=264 stkID=0 seq=0 { g=1 seq=17 ts=0 }
 -> 101613376   GoStart p=0     g=1     off=270 stkID=0 seq=0 { g=1 seq=0 }
 -> 101619219   GoSysBlock      p=0     g=1     off=283 stkID=0 seq=0
2022/12/14 05:42:10 syscall event with link (6/13):
    101618055   GoSysCall       p=0     g=1     off=279 stkID=22        seq=0
 -> 101633491   GoSysExit       p=1000003       g=1     off=297 stkID=0 seq=0 { g=1 seq=19 ts=0 }
 -> 101633634   GoStart p=0     g=1     off=303 stkID=0 seq=0 { g=1 seq=0 }
 -> 101637671   GoSysBlock      p=0     g=1     off=315 stkID=0 seq=0
...

Resulting trace view comparison:

Reproducer:
The program that generated this trace was:
package main

import (
        "fmt"
        "os"
        "runtime/trace"
        "time"
)

func main() {
        if err := rmain(); err != nil {
                fmt.Fprintf(os.Stderr, "error: %v", err)
                os.Exit(1)
        }
}

func rmain() error {
        if err := trace.Start(os.Stdout); err != nil {
                return err
        }
        defer trace.Stop()
        f, err := os.Open("/dev/urandom")
        if err != nil {
                return err
        }
        var buf [1024 * 1024 * 1024]byte
        for i, last := 0, time.Now(); i < 5; i++ {
                fmt.Fprintf(os.Stderr, "start...")
                n, err := f.Read(buf[:])
                if err != nil {
                        return err
                }
                if n == 0 {
                        return fmt.Errorf("read: returned empty slice (%d, %v)", n, err)
                }
                now := time.Now()
                fmt.Fprintf(os.Stderr, "took %v\n", now.Sub(last))
                last = now
        }

        return nil
}

Running it:
$ go run longsyscall.go > longsyscall.trace
start...took 2.203378744s
start...took 2.207477581s
start...took 2.211924562s
start...took 2.202174787s
start...took 2.219617454s

Here are the notes I took on Go runtime tracing and syscall behavior. At the time I looked at Go 1.19's runtime, but I don't think anything of importance has changed in 1.20. I hope this helps.

When user code invokes a syscall, the runtime emits an EvGoSysCall event. If the syscall returns before sysmon finds
the P that's blocked in the syscall, we'll not see any other events. This is a fast path, avoiding a bunch of
scheduling, allowing the goroutine to resume immediately after the syscall returns, having retained its P and M.
If sysmon does find the P, it retakes the P from the M and the runtime emits GoSysBlock, followed by ProcStop. When a
blocking syscall returns, we get a EvGoSysExit event. The EvGoSysExit event gets emitted right before the G resumes
running. However, to maintain accurate timing, the event carries the original timestamp of when the syscall returned,
and the trace parser uses this to reorder the event. [...]
One implication of sysmon detecting blocked syscalls is that the duration between EvGoSysCall and EvGoSysBlock should
be attributes to the syscall, too. This is the time between starting the syscall and Go figuring out that it's
blocking. [...]
Sysmon runs at most every 20 μs, but it will run considerably less often if it thinks there's nothing to do, up to 10
ms. In testing, even in busy programs (with GOMAXPROCS restricted to low values), sysmon sometimes decides to sleep
for 5 ms. A loop calling syscall.Nanosleep, sleeping 10 ms each time, will have EvGoSysBlock spans that last from ~10
ms to 5 ms, with the other 5 ms being the time between EvGoSysCall and EvGoSysExit.
Another issue is that when a syscall returns fast enough we'll not be provided with information on how long the
syscall took. In the worst case, a fast syscall could've run for 10 ms, and to us it'll look like 10 ms of user code.
One special case is syscall.RawSyscall, which doesn't emit any tracing events whatsoever, and AFAIU isn't handled by
sysmon. That means these syscalls can block Ps and Ms, but we have no insight into it happening.

cc @golang/runtime @mknyszek

I'm not sure this has to be a proposal.  Are there any API changes here, or cases where this might potentially break an existing program?  Thanks.

There aren't. But I didn't know which template would be appropriate. I doubted between "proposal" and "bug". While this is actually more of a feature request, between proposal and bug I'd choose the former. Which one should I have chosen?

The "Bugs" template I guess.  I considered changing it to "Bugs or feature requests", but that would lead too many people away from the "Proposal" template appropriate for API changes.  So not sure how to make this better.

Thanks @dominikh, that's very useful. Your notices explain (I think) exactly what's going on, and why my sample code/trace was not representative of the problems I experienced in production nearly a year ago.

If sysmon does find the P, it retakes the P from the M and the runtime emits GoSysBlock, followed by ProcStop. When a
blocking syscall returns, we get a EvGoSysExit event. The EvGoSysExit event gets emitted right before the G resumes
running. However, to maintain accurate timing, the event carries the original timestamp of when the syscall returned,
and the trace parser uses this to reorder the event. [...]

In the sample trace, I'm artificially issuing long-running read syscalls from /dev/urandom, because I wanted to make the "problem" clear on a human timescale. In the production trace where I couldn't get slices even with the cmd/trace patch above, the situation was different:

The syscall used is also read, but from a socket.
There is very high concurrency (GOMAXPROCS > 200 on a 100+ core machine AFAIK).
The syscalls are much shorter (I don't have exact numbers, as I didn't have text logging and there's no duration in the traces), I estimate it was milliseconds.

All of that likely prevented sysmon from finding those syscalls.
So this issue is not the only one preventing good observability. Ideally Go would:

(this issue) Render syscalls as a slice instead of a point. This improvement can (IMHO) be made already, long-running syscalls will benefit already.
(no issue yet) Syscall duration (EvGoSysExit) even for short syscalls. This would also fix the inability to see how much time is spent in syscalls in the cmd/trace (blocking) syscall profile. Currently it ignores syscalls without a link. https://github.com/golang/go/blob/ea14d1b6e1167159dfc8408073ef411c3cf1d7e0/src/cmd/trace/pprof.go#L224

@aktau can you explain the format of your debug output? That is, what exactly does
2022/12/15 14:19:00 syscall event with link (48510/48640):
                 106527657 GoSysCall      p=0       g=1              stkID=52         seq=0
 ->  2.214s     2320112634 GoSysExit      p=1000003 g=1              stkID=0          seq=0   { g=1 seq=10 ts=0 }
 ->  0.000s     2320113061 GoStart        p=2       g=1              stkID=0          seq=0   { g=1 seq=0 }
 ->  0.006s     2326550570 GoSysBlock     p=2       g=1              stkID=0          seq=0

represent?

It also started on proc 0. This indicates that the long-running syscall was moved to another proc close to when it started, or perhaps immediately.

There's no moving between Ps here. The goroutine was simply running on p=0 all along, until it eventually invoked a syscall. (The corresponding event indicating this would be the last GoStart before the GoSysCall)

I suppose that is proc 1000003 mentioned in GoSysExit which may be some virtual ID for the proc attached (?) to the kernel thread servicing the event poller.

1000003 is a synthetic ID that gets used for "returns from syscalls". There's no reason like a kernel thread or event poller for this, just the fact that when the syscall returns, there is no P associated with that event, but we wanted some number. Other such IDs are
const (
        // Special P identifiers:
        FakeP    = 1000000 + iota
        TimerP   // depicts timer unblocks
        NetpollP // depicts network unblocks
        SyscallP // depicts returns from syscalls
        GCP      // depicts GC state
        ProfileP // depicts recording of CPU profile samples
)

@aktau can you explain the format of your debug output? That is, what exactly does

For every GoSysCall event in the trace, print metadata about it (first line, without the ->) and all of its linked events (iterating over the links). Perhaps easier to show the code I added to cmd/trace to produce this (see bottom of reply).

There's no moving between Ps here. The goroutine was simply running on p=0 all along, until it eventually invoked a syscall. (The corresponding event indicating this would be the last GoStart before the GoSysCall)

I think I've just been misusing terminology here. The main trace view is a proc-centric view, and I wanted to express that the syscall was (apart from the very beginning perhaps) not executing on the proc (0) anymore. When I read "proc" in that view my mind just thinks "thread". Stated differently: some other goroutine (like g7) could occupy proc 0 when the main goroutine (g1) was in the read syscall.

1000003 is a synthetic ID that gets used for "returns from syscalls". There's no reason like a kernel thread or event poller for this, just the fact that when the syscall returns, there is no P associated with that event, but we wanted some number. Other such IDs are

I was wondering why SyscallP was used and not NetpollP given that a cursory look at newFile from os/file_unix.go shows that under normal circumstances files resulting from Open are added to the netpoller. But thinking about it, a regular file is always ready to read and a read is always synchronous (even if the fd is O_NONBLOCK), so it has to be SyscallP.
Debug log code:
+       var sysCalls, sysCallsNoLink int
        for _, ev := range ctx.parsed.Events {
                // Handle state transitions before we filter out events.
                switch ev.Type {
@@ -817,7 +819,29 @@ func generateTrace(params *traceParams, consumer traceConsumer) error {
                case trace.EvGoUnblock:
                        ctx.emitArrow(ev, "unblock")
                case trace.EvGoSysCall:
-                       ctx.emitInstant(ev, "syscall", "")
+                       sysCalls++
+                       if ev.Link != nil {
+                               ctx.emitSlice(ev, "syscall")
+                               var sb strings.Builder
+                               fmt.Fprintf(&sb, "syscall event with link (%d/%d):\n            %+v", sysCallsNoLink, sysCalls, ev)
+                               for prev, itev := ev, ev.Link; itev != nil; prev, itev = itev, itev.Link {
+                                       sb.WriteString("\n -> ")
+                                       fmt.Fprintf(&sb, " %2.3fs ", time.Duration(itev.Ts-prev.Ts).Seconds())
+                                       sb.WriteString(itev.String())
+                               }
+                               log.Println(sb.String())
+                       } else {
+                               sysCallsNoLink++
+                               if (sysCallsNoLink % 100000) == 0 {
+                                       log.Printf("sampled syscall event without link (%d/%d):\n    %+v", sysCallsNoLink, sysCalls, ev)
+                               }
+                               ctx.emitInstant(ev, "syscall", "")
+                       }
                case trace.EvGoSysExit:
                        ctx.emitArrow(ev, "sysexit")
                case trace.EvUserLog:

diff --git a/internal/trace/parser.go b/internal/trace/parser.go
index b091a85..a17db4b 100644
--- a/internal/trace/parser.go
+++ b/internal/trace/parser.go
@@ -974,13 +974,29 @@ func PrintEvent(ev *Event) {
 func (ev *Event) String() string {
        desc := EventDescriptions[ev.Type]
        w := new(strings.Builder)
-       fmt.Fprintf(w, "%v %v p=%v g=%v off=%v", ev.Ts, desc.Name, ev.P, ev.G, ev.Off)
+       fmt.Fprintf(w, "%14d %-14s p=%-7d g=%-14d stkID=%-10d seq=%-3d", ev.Ts, desc.Name, ev.P, ev.G, ev.StkID, ev.seq)
+       if len(desc.Args) > 0 {
+               w.WriteString(" {")
+       }
        for i, a := range desc.Args {
                fmt.Fprintf(w, " %v=%v", a, ev.Args[i])
        }
+       if len(desc.Args) > 0 {
+               w.WriteString(" }")
+       }
+       if len(desc.SArgs) > 0 {
+               w.WriteString(" {")
+       }
        for i, a := range desc.SArgs {
                fmt.Fprintf(w, " %v=%v", a, ev.SArgs[i])
        }
+       if len(desc.SArgs) > 0 {
+               w.WriteString(" }")
+       }
        return w.String()
 }

The first line

This issue is fixed in the new tracer. All system calls for Go 1.22 traces and later have a duration in the trace.

authors: aktau dominikh ianlancetaylor mknyszek prattmic


text:
proposal: affected/package: Normalized reflect function

// Indirect returns the value that v points to.
// If v is a nil pointer, Indirect returns a zero Value.
// If v is not a pointer, Indirect returns v.
// Deprecated: use Value.Indirect
func Indirect(v Value) Value {
	if v.Kind() != Pointer {
		return v
	}
	return v.Elem()
}

// Indirect returns the value that v points to.
// If v is a nil pointer, Indirect returns a zero Value.
// If v is not a pointer, Indirect returns v.
func (v Value) Indirect() Value {
	if v.Kind() != Pointer {
		return v
	}
	return v.Elem()
}

“Indirect()” have a similar function to “Elem()”，
Unified invocation mode is recommended

reflect has operations that the language has. Go does not have such an operation.
See also https://go.dev/doc/faq#x_in_std


text:
proposal: os/signal: Provide a new error type to communicate the signal triggered context cancellation

This comes from the need to fix #60733.
In order to provide the information of which signal triggered context cancellation in signal.NotifyContext using context.Cause, we need a new error type (as that's the only type allowed by context cause) to communicate that info back to the user.
The proposal is to define it as a simple struct that can only be used by pointer as an error:
// CanceledBySignalError represents the error can be retrieved via context.Cause
// on the context returned by NotifyContext to inspect which signal caused the
// cancellation.
type CanceledBySignalError struct {
	// The signal caused the context cancellation.
	Signal os.Signal
}

func (cse *CanceledBySignalError) Error() string {
	return fmt.Sprintf("context canceled by signal %v", cse.Signal)
}

So users can inspect the signal via code like:
select {
case <-ctx.Done():
  cause := context.Cause(ctx)
  var cse *signal.CanceledBySignalError
  if errors.As(cause, &cse) {
    sig := cse.Signal
    // Do something with sig
  }
}

I think this should be named os.SignalError instead of signal.CanceledBySignalError, so that it may be used for cases beyond just signal.Notify. For example, I could envision a caller transforming an exec.ExitError by checking whether its ProcessState indicates termination by signal and returning a SignalError.
Actually, SignalError would be even more useful if exec.ExitError had a convenience Is method for it:
	err := cmd.Wait()
	if errors.Is(err, os.SignalError{Signal: syscall.SIGABRT}) {
		// err indicates termination due to signal SIGABRT — program probably failed an assertion
		…
	}

@bcmills I agree that os.SignalError is probably a better name, but I have a few comments/questions regarding that approach:

The error message of it becomes ambiguous if it's used beyond signal.NotifyContext, e.g. it's no longer necessarily "context canceled by signal ...", I guess we can have a more generic message there and use fmt.Errorf("context canceled by %w", &os.SignalError{...}) in signal.NotifyContext
I feel that using errors.Is on it opens some cans of worms we probably don't want to deal with. os.Signal is an interface so there's no guarantee that it will be comparable, and I think in general we prefer pointer errors when the error is a struct to make sure it's comparable (because in that case we only need to compare the pointer address).

But I think this can work if we use a generic error message, and change the errors.Is to errors.As in your example.

os.Signal is an interface so there's no guarantee that it will be comparable

In practice it always is:
~/go/src$ git grep 'func .* Signal() {' .
cmd/vendor/golang.org/x/sys/plan9/syscall_plan9.go:func (n Note) Signal() {}
cmd/vendor/golang.org/x/sys/windows/syscall_windows.go:func (s Signal) Signal() {}
sync/cond.go:func (c *Cond) Signal() {
syscall/syscall_js.go:func (s Signal) Signal() {}
syscall/syscall_plan9.go:func (n Note) Signal() {}
syscall/syscall_unix.go:func (s Signal) Signal() {}
syscall/syscall_wasip1.go:func (s Signal) Signal() {}
syscall/syscall_windows.go:func (s Signal) Signal() {}

syscall.Signal and windows.Signal are both always an integer type when defined. syscall.Note is a string type on Plan 9.
sync.Cond isn't intended to implement os.Signal at all, and doesn't today because it lacks a String method.
So I'm not worried about somebody stuffing an incomparable os.Signal into a SignalError.

I believe that when you use errors.Is(err, os.SignalError{Signal: syscall.SIGABRT}) and err indeed unwraps to os.SignalError type, you need both the unwrapped error and the target to be comparable to not panic. Here you can control the target to be comparable (as you know syscall.SIGABRT is comparable), you have no control what the err is and it's always comparable, so this is still a hanging knife that could panic the code unexpectly.

errors.Is already handles comparability:
https://cs.opensource.google/go/go/+/refs/tags/go1.20.5:src/errors/wrap.go;l=48;drc=4a0a2b33dfa3c99250efa222439f2c27d6780e4a

hmm you are right. tested on playground: https://go.dev/play/p/qpyJXU05CFl
but another issue with making the nonpointer type to implement error is that people often don't know which type to use with errors.As, for example they can make this mistake:
var se *os.SignalError // they thought it would be the pointer type
if errors.As(err, &se) {
  // they'd never get here
}

the solution to that is to also implement os.SignalError.As to support both pointer and non-pointer types, but is that something we consider worth doing here?

To be clear, I think it would be ok for the type implementing error to be *SignalError.
(exec.ExitError.Is could follow the pointer explicitly.)

But if we use pointer type, errors.Is will no longer work as you expected here, because it will only return true if the pointers are the same, not the contents: https://go.dev/play/p/VNQ-iy4vA_2

The error type can provide an explicit Is method. 🙃

yes but then you need to deal with the comparable check of Signal interface in that Is function (or you could get unexpected panics)

You don't need to check whether the Signal implementation is comparable. Every real implementation is.

That's not really in the "contract" of the API (https://pkg.go.dev/os#Signal definition doesn't guarantee comparability). You are essentially depending on implementation details to not panic.
Although this is very very unlikely to happen in real life, hypothetically someone could define a type implementing os.Signal that wraps some real signals but is uncomparable, and when they try to use it with errors.Is (either as part of the target or trigger the signal into the error) it would cause panic.

@bcmills do you mind if I leave the errors.Is case out of this proposal for now?
Basically the revised proposal would be to add SignalError to os package as:
// TODO: what's a generic doc comment to describe it here that's appropriate?
type SignalError struct {
  Signal Signal
}

func (se *SignalError) Error() string {
  return "received signal " + se.Signal.String() // I think this error message is generic enough, in signal.NotifyContext I can just use wrap the error again with prefix of "context canceled by "
}

@fishy, I started working on this in parallel too. When I saw you worked, I rebased with your changes.
Please take a look at my changes here:
https://go-review.googlesource.com/c/go/+/504215
Regarding using a pointer for SignalError, I think we shouldn't.
My reasoning is that there isn't a hard rule on using pointers for SignalError, and we don't need one here.
Using a pointer in this code (that might have good visibility) when we don't need to might make people think they should do the same regardless, and this can hit them with increased garbage collector pressure when they use code on a hot path.
/cc @ianlancetaylor (on the original implementation of NotifyContext you shared some thoughts/ideas on how this type could be named; you might want to take a look).

@henvic to reiterate https://github.com/golang/go/issues/60756#issuecomment-1591964848, the problem of using allowing non-pointer type to implement error is that when user try to use errors.As it's very error-prone for them to use the wrong type.

I've created a new change request here https://go-review.googlesource.com/c/go/+/579375

authors: bcmills fishy henvic


text:
x/tools/gopls: completion should also provide just the function name

gopls version
v0.14.2
go env
irrelevant

What did you do?
I have an existing expression f(a), and I would like to replace it with f(protocol.FWithLongName(a)). After typing protocol.F, the only useful completion offered is the function snippet, which would complete as
f(protocol.FWithLongName(x protocol.TypeWithLongName, y TypeOfSecondArg)(a))
which requires too much editing.
What did you expect to see?
I would like to have seen both the function snippet and just the function names as suggested completions.
What did you see instead?
The unsyntactic completion described above. gopls is not looking at the context when it proposed a snippet where there is already an argument.
Editor and settings
No response
Logs
No response

Sometimes i want the whole thing, sometimes I want just the name, depending
on my editing need at the moment.
I'd like much more control, actually, but can't envision a suitable user
interface.  I want 'false' for printf-like functions, 'true' for non-varargs
functions if I'm editing at the end of an expression or statement, and
'false' if i'm editing in the middle of an expression. Providing both
versions of the function completion seems easier.
On Thu, Nov 30, 2023 at 3:41 PM findleyr @.***> wrote:

Does "completeFunctionCalls": false do what you want?
—
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/64469#issuecomment-1834531062, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJIAI5P76S6GETHKOX2UZLYHDVQ7AVCNFSM6AAAAABABK6YY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZUGUZTCMBWGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

Turning it on and off with a setting would mean changing setting frequently
while editing. Giving both choices seems better.
On Wed, Dec 6, 2023 at 10:15 PM Hyang-Ah Hana Kim @.***>
wrote:

I have an existing expression f(a), and I would like to replace it with
f(protocol.FWithLongName(a)). After typing protocol.F, the only useful
completion offered is the function snippet, which would complete as
f(protocol.FWithLongName(x protocol.TypeWithLongName, y
TypeOfSecondArg)(a))
which requires too much editing.
I agree that the completion logic needs to be smarter in this case.
Can you share your gopls setting? I disabled usePlaceholder for the
similar issue at some point in the past.
—
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/64469#issuecomment-1844187132, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJIAIYAQXL2ZHO2IA26SRTYIEYENAVCNFSM6AAAAABABK6YY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBUGE4DOMJTGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

perhaps we should have a counter to see what fraction of our users use
'usePlaceholder'. If hardly anyone does, then it could be removed. It seems
we have no principled way of deciding about what sort of settings to have,
other than our individual personal valuing of user suggestions.
On Thu, Dec 7, 2023 at 8:31 AM Hyang-Ah Hana Kim @.***>
wrote:

@pjweinb https://github.com/pjweinb i am not suggesting users have to
turn on/off the settings while editing. I meant usePlaceholder setting
became counter-productive due to this problem and I personally don't use
that setting any more. Giving both option will be confusing unless we
figure out good presentation, and make the suggestion list longer.
—
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/64469#issuecomment-1845348990, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJIAI5XW5ULTFSNR3IOXRLYIHALLAVCNFSM6AAAAABABK6YY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVGM2DQOJZGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I wonder if we could try to make the snippet smarter. In f(a) where you want f(other(a, ...)), if a can be passed to other, maybe we could see that and offer other(a, <nextArg>) as the completion. Or maybe that is too specific.
Anyway, in cases where there is a trailing identifier after the cursor, I feel it is reasonable to also offer the naked function call as a completion candidate since it is hard to tell what the user wants.

authors: muirdm pjweinb


text:
cmd/build: -trimpath not working completely

Go version
go version go1.16.15b7p1 linux/amd64
Output of go env in your module/workspace:
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOENV="/home/user/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/user/go/pkg/mod"
GONOPROXY="gitlab.eng.company.com,github.com/google/martian"
GONOSUMDB="*.eng.company.com"
GOOS="linux"
GOPATH="/home/user/go"
GOPRIVATE="gitlab.eng.company.com,github.com/google/martian"
GOPROXY="https://build-artifactory.eng.company.com/srp-mds-go-remote"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.15b7p1"
GCCGO="gccgo"
AR="ar"
CC="x86_64-vmk-linux-gnu-gcc"
CXX="/build/mts/release/bora-23244397/compcache/cayman_esx_toolchain/ob-16111630/linux-centos72-gc32/usr/bin/x86_64-vmk-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD="/home/user/dev/dev/go/src/project/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2531685105=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I have compiled a Go binary using the below flags and it has a runtime exception and dumped the below Stacktrace.
My GOPATH = /home/hemant/dev/go/src
My compiler flags while building are: -gcflags=-trimpath=$GOPATH -trimpath -asmflags=-trimpath=$GOPATH -trimpath
Stacktrace is below:
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x144efd2]
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: goroutine 1 [running]:`
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: github.com/go-redis/redis.(*cmdable).HGetAll(0xc002cc78b0, 0x0, 0x0, 0x0)
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: github.com/go-redis/redis.(*cmdable).HGetAll(0xc002cc78b0, 0x0, 0x0, 0x0)
Apr 15 09:56:59 10-79-108-22 system-health[3923528]:         github.com/go-redis/redis@v6.13.2+incompatible/commands.go:923 +0xd2
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: controller/systemhealth.someOtherFunc(...)
Apr 15 09:56:59 10-79-108-22 system-health[3923528]:         controller/systemhealth/system_health.go:676
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: controller/systemhealth.someFunction(...)
Apr 15 09:56:59 10-79-108-22 system-health[3923528]:         controller/systemhealth/system_health.go:680
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: controller/systemhealth.ServiceInitialize()
Apr 15 09:56:59 10-79-108-22 system-health[3923528]:         controller/systemhealth/system_health.go:685 +0x6b
Apr 15 09:56:59 10-79-108-22 system-health[3923528]: main.main()
Apr 15 09:56:59 10-79-108-22 system-health[3923528]:         /home/hemant/dev/go/src/controller/go-controller.go:743 +0x231e

What did you see happen?
I have an issue where the -trimpath is not removing the GOPATH /home/hemant/dev/go/src/ from the last line of the stacktrace.
Apr 15 09:56:59 10-79-108-22 system-health[3923528]:         /home/hemant/dev/go/src/controller/go-controller.go:743 +0x231e

What did you expect to see?
GOPATH should have been removed from the last line also.
Also side question: Is it possible to NOT expose directory path also and have mere filename ?

This is a very old version of go, does that works with one of the maintained release (1.22.2 or 1.21.9) ?
It would also help if you give a complete build command, or repro. I guess you might have pointed go build at files instead of a folder ?

Yes @Jorropo, I have ran the go build on a file, but I think this might be same even for the folder as well.
Unfortunately, I cannot upgrade the go version as its a legacy product.
COMMAND cd ${CMAKE_CURRENT_SOURCE_DIR} && env CC=gcc CXX=g++ GOPATH=${CMAKE_SOURCE_DIR}/go go build -mod vendor -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn" -gcflags=-trimpath=$GOPATH -trimpath -asmflags=-trimpath=$GOPATH -trimpath go-controller.go

Ah, the problem is that you set -gcflags=-trimpath=$GOPATH and GOPATH="/home/user/go" while your project is stored at /home/hemant/dev/go I guess user → hemant is not real, but $HOME/go vs $HOME/dev/go is real.
Everything works fine for me when I set -trimpath once (afaik you shouldn't need to set -gcflags nor -asmflags manually, nor you need to set -trimpath twice).

We might be able to error on this (-trimpath set too many times) or interpret it set multiple times as each path to be trimmed, but I don't think that worth breaking the cli here.
Can reopen if someone else feels strongly.

@Jorropo : just plain simple -trimpath worked. Thank you very much.


text:
proposal: encoding/csv add a new constructor with a configurable buffer size

Proposal Details
Currently encoding/csv Writer is currently exposing only one constructor using bufio.NewWriter constructor. This hardcodes the buffer size to 4096 which sometimes is not enough and code panics if a single csv record is longer than this length (assuming we're flushing after every record).
Proposal:
Add one more constructor that allows to control the buffer size, so single row can fit into the buffer:
func NewWriterWithBufferSize(w io.Writer, size int) *[Writer]

Note that you can get the same effect using
csv.NewWriter(bufio.NewWriterSize(w, size))

See https://pkg.go.dev/bufio#NewWriterSize .

Closing as irrelevant then, since the provided standard solution works well.

authors: ianlancetaylor thecoldwine


text:
New PyPI release needed

@goldsmith
See, e.g. https://github.com/goldsmith/Wikipedia/issues/107, which was fixed 4 years ago, but the bug still exists in the version published on PyPI.  It appears that the version on PyPI is nearly 6 years old.

On the other hand, this library seems to work well as an alternative: https://github.com/barrust/mediawiki

repo: goldsmith/Wikipedia

license: mit

license_source: bigquery


text:
Impossible to export image in docx

Hello @gongph , I've used your plugin and it's pretty cool.
I'm facing an issue, i'm building a django app where the user will be able to export his current page into docx file , the issue now is that the css and the images isn't exported into the docx.
What do you suggest for this issue?

You can use inline base64 image not any link, that works in my case.
CSS you need to attach as styleCss property in
saveAsDocx({
selector: '#my-selector',
margins: {
top: 720
},
styleCss: ''
});

You can use inline base64 image in place image-link, that works in my case.
CSS you need to attach as styleCss property in
saveAsDocx({
selector: '#my-selector',
margins: {
top: 720
},
styleCss: ''
});

authors: Im-praveen manutheblacker