text:
spec: requiring identical tags for struct identity seems overly restrictive

The rules for struct identity require that two identical structs have identical tags.
This plays into the rules for assignments and conversions of structs in unfortunate ways:
There appears to be no way to assign or even convert a struct with tags to another struct with different or no tags, wether that other struct is named or unnamed:
http://play.golang.org/p/6FZcWSIc9w
I think we're overly restrictive here. For instance, if I wanted to convert a struct with json tags to a struct with tags for some other encoding (to convert from one encoding to another), I actually have to copy all the fields by hand.
This seems counter-intuitive. I would have expected that I could go via an unnamed struct, or at the very least use an explicit conversion.
I'm thinking that perhaps we should relax the rules here. One might be to remove again the requirement of identical tags for struct identity (simplest); or perhaps to relax the rules for assignments and conversions of structs.

Feedback from @ianlancetaylor (copied from e-mail):
I tend to agree.  FYI, the requirement that struct tags be identical
was added here: https://codereview.appspot.com/1698043 .  It seems to
me that this doesn't make much difference in practice.  You already
can not assign or implicitly convert between named struct types.
Unnamed struct types are rarely used, so I don't see them as an
issue.  If we reverted this CL, I think it would be possible to
explicitly convert between struct types as long as the fields had the
same names and types, because the underlying types would be identical.

PS: I vaguely recall that I voted for this change. But I also don't remember the details. It looks like a case of unintended consequences. Rolling back the change would be a backward-compatible change.

Duplicate of #6858.

authors: griesemer robpike


text:
Usage of variable after if-else scope not detected by complier

go version : go1.4.2 darwin/amd64, Running on Mac OS X Yosemite
Issue : The go compiler reports result, err as unused in the code below
		if len(tableSchema.ReplicaTags) == 0 {
			result, err := rethink.DB(rethinkdbConfig.Database).TableCreate(tableSchema.Name, rethink.TableCreateOpts{
				PrimaryKey: tableSchema.PrimaryKey,
				Durability: tableSchema.Durability,
				Shards:     tableSchema.Shards,
				DataCenter: tableSchema.Replicas,
			}).RunWrite(session)
		} else {
			result, err := rethink.DB(rethinkdbConfig.Database).TableCreate(tableSchema.Name, rethink.TableCreateOpts{
				PrimaryKey: tableSchema.PrimaryKey,
				Durability: tableSchema.Durability,
				Shards:     tableSchema.Shards,
				DataCenter: tableSchema.PrimaryReplicaTag,
			}).RunWrite(session)
		}
		if err != nil {
			fmt.Println(err.Error())
		} else {
			fmt.Println(result)
		}

Compiler output
setup/dbSetup.go:96: result declared and not used
setup/dbSetup.go:96: err declared and not used
setup/dbSetup.go:104: result declared and not used
setup/dbSetup.go:104: err declared and not used

Expected behaviour would be no error as they are being used in their lifetime (i.e. in this case the scope of the function)
The below code compiles in one "go" :)
		if len(tableSchema.ReplicaTags) == 0 {
			result, err := rethink.DB(rethinkdbConfig.Database).TableCreate(tableSchema.Name, rethink.TableCreateOpts{
				PrimaryKey: tableSchema.PrimaryKey,
				Durability: tableSchema.Durability,
				Shards:     tableSchema.Shards,
				DataCenter: tableSchema.Replicas,
			}).RunWrite(session)
			_, _ = result, err
		} else {
			result, err := rethink.DB(rethinkdbConfig.Database).TableCreate(tableSchema.Name, rethink.TableCreateOpts{
				PrimaryKey: tableSchema.PrimaryKey,
				Durability: tableSchema.Durability,
				Shards:     tableSchema.Shards,
				DataCenter: tableSchema.PrimaryReplicaTag,
			}).RunWrite(session)
			_, _ = result, err
		}
		if err != nil {
			fmt.Println(err.Error())
		} else {
			fmt.Println(result)
		}

Go is block scoped, so each {} block introduces its own scope.
The scopes are described in the spec: http://golang.org/ref/spec#Declarations_and_scope

I get go is block scoped, most languages are. Scoping is for type and value deduction of variable not its usage. If we limit the usage as well, then accessing a variable declared inside an if, for block should throw an error too.
Also, as per http://golang.org/ref/spec#Variable_declarations
Implementation restriction: A compiler may make it illegal to declare a variable inside a function body if the variable is never used.
as it says: illegal if its not used inside the function body
I accept that parsing usage across branched scoping throughout the function body will be a pain, but that will be the ideal behaviour. Else, it will result in lot of unnecessary variable assignment here and there in heavily nested code.

Please move this to the golang-nuts@ mailing list.

We can't change the declaration and scoping rules. http://golang.org/doc/go1compat

source: gharchive/issue

authors: adg agaurav bradfitz

repo: golang/go

license_type: permissive

license_source: github-api


text:
x/crypto/ssh: Return better error message for non-zero exit codes

Version
go version go1.5.3 darwin/amd64
What I did
I executed a script on a remote server through SSH.
if err := sesh.Run("/home/jaime/fail.sh"); err != nil {
    log.Println(err)
}

This is fail.sh.
#!/bin/sh
exit 1

What I expected to see
I expected to see a clear and concise error. Maybe something like: Process exited with: 1.
What I saw instead
I saw a slightly confusing and half-useless error: Process exited with: 1. Reason was:  ()
The code in question is here, crypto/ssh/session.go:604. This is happening because it's Sprintfing empty w.msg and w.signal values. This doesn't seem right.

We could slop the printing of the Reason was: () part if w.msg is blank.

Sounds reasonable. I'm willing to submit a PR, if possible.

@variadico Are you still working on this one, or is it free to pick up?

I never heard back from anyone, so I never started. But, it looks like someone else grabbed this already: https://go-review.googlesource.com/#/c/19444. The status says "Needs Code-Review".

@gopherbot
I am still facing the issue where session.Run() returns --> Process exited with: 1. Reason was:  ()
if the scp path is incorrect.
Can you please help?

@udayskyport Which version of Go are you using?

@gopherbot go1.6

authors: davecheney ianlancetaylor sctb udayskyport variadico


text:
runtime_stoptheworld deadlock on ppc64le

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.4.2 gccgo (GCC) 5.3.0 linux/ppc64le

What operating system and processor architecture are you using (go env)?

ppc64le
linux

What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

Unfortunately haven't been able to come up with a simple repro, but core dump and binary are provided (all of the interesting source information is within gccgo's libgo c implementation).

What did you expect to see?

Something other than a hard deadlock across all threads.  If this is an application derived deadlock, perhaps deadlock detection triggering.

What did you see instead?

All threads stuck in runtime_futexsleep without a timeout.
One thread is in runtime_stoptheworld, all other threads are blocked.  On inspection, the stoptheworld thread is waiting because one of the outstanding P's is still in Psyscall (runtime_sched.stop_wait == 1).  Values for that P:
(gdb) p runtime_allp[1][0]
$1 = {{key = 0}, id = 1, status = 2, link = 0x0, schedtick = 88, syscalltick = 10, m = 0x0, mcache = 0x3fff90df0898,
  deferpool = 0x0, goidcache = 37, goidcacheend = 49, runqhead = 21, runqtail = 21, runq = {0xc2094b0000, 0xc2094b0000,
    0xc20a10ba00, 0xc20a10c700, 0xc20a10d400, 0xc2094b0000, 0xc20a10e100, 0xc20a10e100, 0xc20a10e100, 0xc20bd8a000,
    0xc20bd8ad00, 0xc20bd8ba00, 0xc2094b0000, 0xc20d592700, 0xc2094b0000, 0xc2094b0000, 0xc2094b0000, 0xc20d590000,
    0xc20bd82d00, 0xc20d594e00, 0xc20b526100, 0x0 <repeats 235 times>}, gfree = 0x0, gfreecnt = 0,
  pad = '\000' <repeats 63 times>}

All of the other P's are in status 3, I.e. Pgcstop
I'm partly suspicious of the doentersyscall() edge, which seems to do a non-atomic load of runtime_sched.gcwaiting, which feels like it could race with the gc'ing thread:
// doentersyscall()
...
      runtime_atomicstore(&m->p->status, Psyscall);
      if(runtime_sched.gcwaiting) {
              runtime_lock(&runtime_sched);
              if (runtime_sched.stopwait > 0 && runtime_cas(&m->p->status, Psyscall, Pgcstop)) {
                      if(--runtime_sched.stopwait == 0)
                              runtime_notewakeup(&runtime_sched.stopnote);
              }
              runtime_unlock(&runtime_sched);
      }
...

Coredump with binary and goroutine stack output:
https://www.dropbox.com/s/3lht3xs4fgi7fub/hang_info.tgz?dl=0

I've written a smaller piece of code that seems to exhibit the same behavior. Running this with a higher GOMAXPROCS seems to make it more likely to deadlock. On the 80 core machine I'm running this on I run it with GOMAXPROCS=200. For me, it hangs somewhere between 15 and 60 seconds.
package main

import (
        "syscall"
        "fmt"
)

func main() {
        n := 200
        count := make([]int,n)
        for i:=0;i<n;i++ {
                go func(x int) {
                        for {
                                syscall.Nanosleep(&syscall.Timespec{ 0, 100} ,&syscall.Timespec{})
                                count[x]++
                                _ = make([]byte,1024)
                        }
                }(i)
        }
        for i := 0;;i++ {
                syscall.Nanosleep(&syscall.Timespec{ 1, 0} ,&syscall.Timespec{})
                fmt.Printf("%v %#v\n",i, count)
        }
}

With this patch, my reproo program no longer deadlocks.
[ec2-user@ubuntu1404-ppc-2 gabriel]$ diff  -u -r gcc-5.3.0 gcc-5.3.0-fix                                                                                diff -u -r gcc-5.3.0/libgo/runtime/proc.c gcc-5.3.0-fix/libgo/runtime/proc.c
--- gcc-5.3.0/libgo/runtime/proc.c      2015-04-29 21:32:07.000000000 +0000
+++ gcc-5.3.0-fix/libgo/runtime/proc.c  2016-04-01 16:04:01.471160881 +0000
@@ -2042,7 +2042,7 @@
        m->mcache = nil;
        m->p->m = nil;
        runtime_atomicstore(&m->p->status, Psyscall);
-       if(runtime_sched.gcwaiting) {
+       if(runtime_atomicload(&runtime_sched.gcwaiting)) {
                runtime_lock(&runtime_sched);
                if (runtime_sched.stopwait > 0 && runtime_cas(&m->p->status, Psyscall, Pgcstop)) {
                        if(--runtime_sched.stopwait == 0)

Thanks for tracking this down.


text:
net: Addr interface documentation unclear

The Addr Name() method's comment/doc is unclear.
The Name() method as implemented by IPAddr is not the name of a given network, but the name of the protocol of the Addr implementation.
Maybe "Network Name" is the appropriate terminology for this, but it is hard to search. I don't know what it should say but I think more would be helpful.

The net.Addr interface does not have a Name method.  I assume you mean the Network method.

FWIW, I have a draft for #10624 like the following, but it's still too much explanation. Need a wordsmith.
Networks, Hosts and Services

The network, host and service parameters specify the detail of communicaton
facilities for Dial, Listen, Lookup and Resolve functions.

The networks:

The known networks are "tcp", "tcp4", "tcp6", "udp", "udp4", "udp6", "ip",
"ip4", ip6", "unix", "unixgram" and "unixpacket".
A name ending with "4" uses only IPv4 and "6" uses only IPv6 as a network layer
protocol.

        "tcp"           TCP over either IPv4 or IPv6 communication
                        the network protocol will be chosen automatically
        "tcp4"          TCP over IPv4 communication
        "tcp6"          TCP over IPv6 communication
        "udp"           UDP over either IPv4 or IPv6 communication
                        the network protocol will be chosen automatically
        "udp4"          UDP over IPv4 communication
        "udp6"          UDP over IPv6 communication
        "ip"            either raw IPv4 or IPv6 communication
                        the network protocol will be chosen automatically
        "ip4"           raw IPv4 communication
        "ip6"           raw IPv6 communication
        "unix"          host-local connection-oriented stream communication
        "unixgram"      host-local datagram communication
        "unixpacket"    host-local connection-oriented datagram communication

The raw IP network must be "ip", "ip4" or "ip6" followed by a colon and a
protocol number or name, e.g. "ip4:1", "ip6:58", "ip:icmp", "ip:ipv6-icmp".

The hosts:

        IPv4 literal    dotted decimal form, e.g. "192.0.2.1"
        IPv6 literal    colon-separated hexadecimal form, with IPv6 zone
                        identifier separated by a percent sign if required,
                        e.g. "2001:db8::1", "fe80::1%eth0"
        name on IP network
                        opaque string, which is mostly a host name or domain
                        name, e.g. "golang.org"
        name on Unix network
                        opaque string, which is mostly a file system path

The services:

        number literal on IP network
                        decimal digits, e.g. "80"
        name on IP network
                        opaque string, which is mostly a service name,
                        e.g. "http"

The hosts with services:

        IPv4 literal    colon-separated host and service,
                        e.g. "192.0.2.1:80", "192.0.2.1:http"
        IPv6 literal    colon-separated host enclosed in square brackets and
                        service, e.g. "[2001:db8::1]:80", "[2001:db8::1]:http",
                        "[fe80::1%eth0]:80"
        name on IP network
                        colon-separated host and service,
                        e.g. "golang.org:80", "golang.org:http"

TODO:
- describe the detail of IP address and service wildcard stuff

authors: ianlancetaylor mikioh turnage


text:
fmt: formatting of reflect.Value items inconsistent with documentation

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?
1.6.2

What operating system and processor architecture are you using (go env)?
playground

What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

From sbkim in https://groups.google.com/d/topic/golang-nuts/3DofSQIIF2w/discussion:
https://golang.org/pkg/fmt/ says:

If the operand is a reflect.Value, the operand is replaced by the concrete value that it holds, and printing continues with the next rule.

If an operand implements method String() string, that method will be invoked to convert the object to a string, which will then be formatted as required by the verb (if any).

So, in the following program (https://play.golang.org/p/V4H-vkjoE5):
package main

import (
 "fmt"
 "reflect"
)

type SpecialInt int

func (i SpecialInt) String() string { return fmt.Sprintf("SpecialInt(%d)", int(i)) }

func main() {
 v := reflect.ValueOf(SpecialInt(73))
 fmt.Printf("%+v\n", v)
 fmt.Printf("%+v\n", v.Interface())
}

I expect the first fmt.Printf call to substitute the concrete value of v, which is SpecialInt(73), invoke the String() method on it, and print "SpecialInt(73)". However, it just prints "73". On the contrary, the second fmt.Printf call (correctly) prints "SpecialInt(73)".
Is there anything I'm missing here, or maybe something wrong with the Go documentation or implementation?

What did you expect to see?
SpecialInt(73)
SpecialInt(73)

What did you see instead?
73
SpecialInt(73)

There is a trivial fix to this that solves the test case but won't work in general. The code says

We could instead call
  p.PrintArg(f.Interface(), verb)

which would extract the interface and print that. This works but only if the value being printed was not acquired by using Value.Field of an unexported interface.
We could write
if f.CanInterface() {
		p.printValue(f, verb, 0)
} else {
		p.printArg(f, verb)
}

and maybe that's the best we can do. Needs though.

I think the depth > 0 condition in printValue is a relic of a previous design, and was perhaps just an optimization.  It causes the handleMethods check for a reflect.Value argument to Printf to be skipped.  Removing the condition fixes the problem (and the fmt tests pass).
 func (p *pp) printValue(value reflect.Value, verb rune, depth int) {
        // Handle values with special methods if not already handled by printArg (depth == 0).
        if depth > -100 && value.IsValid() && value.CanInterface() {

Added the depth check when refactoring reflect value printing https://go-review.googlesource.com/#/c/20923/.
As far as i recall currently:
It preserves behavior and performance of 1.6 in 1.7 and is an optimization to call handlemethods in printArg at depth 0 instead of in printValue always.
If only the depth check is removed handleMethods is called twice for the default branch of printArg.
If the handleMethods call in printArg is removed to prevent the above there is a hit
in performance for e.g. simple Stringer printing (did a quick benchmark to reconfirm my old testing):
name               old time/op  new time/op  delta
SprintfStringer-4   281ns ±10%   311ns ±20%  +10.98%  (p=0.000 n=18+19)

So the depth check (see comments in CL) address the CL comment about "slows down a very common operation" and to not slow down printValue for depth 0 either.
I will have a look for go 1.8 if we ideally can keep the performance for Stringer and still consistently replace reflect.Value with concrete values.

@martisch Have you had a chance to look at this?

Thanks for bringing this back to my attention. Was on vacation in september and sidetracked by some other patches. (Seems i can not assign myself to golang issues) I will do some measurements again and send a proposal CL with tests within a week so we have this fixed for 1.8.

authors: adonovan martisch mattharden quentinmit robpike


text:
Confirm the visibility of Method Sets

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?
go version go1.6.2 darwin/amd64

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/yangyuqian/gopkg"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GO15VENDOREXPERIMENT="0"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

// main.go
package main

type A struct {}

func (A) F1(){}
func (*A) Fp1(){}

func main(){
  a := A{}
  a.Fp1()
}

What did you expect to see?

According the description in go spec:

The method set of any other type T consists of all methods declared with receiver type T. The method set of the corresponding pointer type *T is the set of all methods declared with receiver *T or T (that is, it also contains the method set of T).

I expect my example above to throw some exceptions or panics because I'm calling a method Fp1 defined on *A.

What did you see instead?

I can access A.Fp1 without errors.

In addition, when I try to access the A.Fp1 with reflection, it panics
package main

import (
	"reflect"
)

type A struct{}

func (A) F1() { println("A.F1") }
func (A) f1() { println("A.f1") }

func (*A) Fp1() { println("*A.Fp1") }
func (*A) fp1() { println("*A.fp1") }

func main() {
	a := A{}
	aa := &a
	println("---------- without reflection -----------------")
	a.Fp1()
	aa.Fp1()
	println("---------- with reflection -----------------")
	reflect.ValueOf(a).MethodByName("F1").Call(nil)
	reflect.ValueOf(a).MethodByName("Fp1").Call(nil)
}

Console output:
---------- without reflection -----------------
*A.Fp1
*A.Fp1
---------- with reflection -----------------
A.F1
panic: reflect: call of reflect.Value.Call on zero Value

goroutine 1 [running]:
panic(0x917c0, 0x82015a2a0)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
reflect.flag.mustBe(0x0, 0x13)
        /usr/local/go/src/reflect/value.go:202 +0xa6
reflect.Value.Call(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:301 +0x48
main.main()
        /Users/yangyuqian/code/example-go/main.go:23 +0x255
exit status 2

We don't use the issue tracker for questions.  Please ask questions in  a forum instead; see https://golang.org/wiki/Questions .
In your first example a.Fp1 works because the address of a is implicitly taken, making this identical to (&a).Fp1.  See https://golang.org/ref/spec#Calls : "If x is addressable and &x's method set contains m, x.m() is shorthand for (&x).m()."  This does not help with your second example, because this shorthand does not apply when using reflection.
Closing as this is not a bug.

I see, thanks for the quick response anyway.

authors: ianlancetaylor yangyuqian


text:
Proposal: Improve dead code ilimination when using CGO

What version of Go are you using (go version)?

$ go version
go version go1.14.1 windows/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env

What did you do?

Build an executable that uses a large C library statically linked.
What did you expect to see?
A smaller executable
What did you see instead?
A kind-of large executable.
I'm not completely sure about this, but when Go links, either statically or dynamically, I think it includes a reference to all C functions wrapped with CGO in the resulting executable, even if those functions are never called.
It would help to only include required C symbols. This could in theory help the C compiler do dead code ilimination with static libraries. Adding the CFLAGS -ffunction-sections -fdata-sections, and LDFLAGS -Wl,-gc-sections could also do something.
Thanks.

When using cgo, the Go linker builds an object file containing all the Go code, and it passes that object file to the C linker.  The Go linker will discard any unused Go code.  So the only references to C functions will be from Go code that might be called.  Everything else is up to the C linker.
I agree with your CFLAGS and LDFLAGS suggestion, but I don't think that Go should do that by default.  I think that the C compilation and linking should use just a few simple options, not options like -ffunction-sections or -Wl,--gc-sections that can cause some programs to break.
So I don't think there is anything to do here.


text:
x/tools/gopls:

What version of Go are you using (go version)?

$ go version

Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
go env Output
$ go env

What did you do?

What did you expect to see?
What did you see instead?

Empty.


text:
cmd/go: spuriously stale dependencies

During compiler development, I periodically take snapshots of my git working tree, copy them to my workstation, and then run a bunch of tests. The script I run on the workstation currently looks like this:
cd $GOROOT/test
git checkout $TRYREV
go install -toolexec=toolstash cmd/compile
go build -a -v -gcflags=-d=inlfuncswithclosures=0 -toolexec='toolstash -cmp' std cmd
go install -v std cmd/cgo cmd/go cmd/link
go install -v -x cmd/cgo cmd/go cmd/link  # why??
go run run.go
go test -short std cmd

I've added the second go install -v -x cmd/cgo cmd/go cmd/link invocation because otherwise tests like cmd/link's TestDWARF fail because cmd/go says cmd/link is stale. I would expect the one go install invocation to be sufficient, but the go install -x output (e.g., https://gist.github.com/mdempsky/bdfc2247df2bba822b833b78309660e9 has the output from my most recent script run) confirms that cmd/cgo, cmd/go, and cmd/link are all being relinked and reinstalled.
The staleness issue bites me quite frequently during my workflow, but I haven't managed to isolate steps that reliably reproduce the failure.
/cc @bcmills

This is my understanding.
Running go install cmd/link will update the installed link tool if the hash code has changed.  The hash code of linking a program includes the hash code of cmd/link itself.  So the first install of cmd/link gives you a link tool with a hash code that records the link tool used to build it.  When you run go install again, the link tool has changed, so the hash code has changed, so the linker gets rebuilt and reinstalled.  This time, it won't actually change, so another go install cmd/link will see that hash code matches and it won't do anything.
This is basically why make.bash builds the tools three times.  See this comment from cmd/dist/build.go:
// Toolchain2 should be semantically equivalent to toolchain1,
// but it was built using the new compilers instead of the Go 1.4 compilers,
// so it should at the least run faster. Also, toolchain1 had no build IDs
// in the binaries, while toolchain2 does. In non-release builds, the
// toolchain's build IDs feed into constructing the build IDs of built targets,
// so in non-release builds, everything now looks out-of-date due to
// toolchain2 having build IDs - that is, due to the go command seeing
// that there are new compilers. In release builds, the toolchain's reported
// version is used in place of the build ID, and the go command does not
// see that change from toolchain1 to toolchain2, so in release builds,
// nothing looks out of date.
// To keep the behavior the same in both non-release and release builds,
// we force-install everything here.
//
//	toolchain3 = mk(new toolchain, toolchain2, go_bootstrap)
//

At least, that is my understanding.

It is interesting to get some insight into this problem.
I run the cmd/link TestDWARF tests by hand a good deal (since I spend a fair amount of time making changes to the dwarf code), and it is frustrating to have the the test inform me that the linker is stale right after I just did "go install cmd/link".
Maybe we need a new "go installthreetimes cmd/link" or something like that.

I see. I knew bootstrapping reinstalls multiple times, but I guess I assumed cmd/go would handle the details after that.
If it stays the user's responsibility to run go install to fixpoint and things are working as intended, then that's surprising, but okay.

Re "the user's responsiblity", this is not something that affects ordinary users. It is something that affects only people who install the tools that the go command uses to build things, namely 'cmd'.
The go command should not go out of its way to add special cases for developing the toolchain itself. That is the job of cmd/dist.
Apologies for needing to run 'go install' multiple times to reach a fixed point, but this is a fundamental consequence of the interaction between (1) cmd/link essentially depending on itself and (2) content-hash-based staleness.

For this reason I use toolstash restore && go install cmd/xxx if I'm hacking on the toolchain itself. This speeds up the iteration by making sure that I always build with the master version (stored with toolstash save), so I don't need to rebuild half of std and cmd again.
If we agree that this is something we should simply document better, maybe we can add it to https://github.com/golang/go/issues/30074.

Re "the user's responsiblity", this is not something that affects ordinary users. It is something that affects only people who install the tools that the go command uses to build things, namely 'cmd'.

Ack, I agree this issue has limited scope of people affected. But as someone who works primarily on cmd/compile, I still consider myself a user of cmd/go. That's all I meant.

That is the job of cmd/dist.

I'm not very familiar with cmd/dist's full functionality. After I make changes to cmd/compile, is there a cmd/dist invocation to bring things back to a fixed point with the minimal amount of redundant work?
I generally avoid running the full make.bash/all.bash script because I like the flexibility of tweaking my test script to reorder tests to prioritize whatever particular corner case I'm actively dealing with. Waiting for the complete triple-bootstrap really slows down my test cycles when I'm trying to figure out why my change broke fixedbugs/issue12345.go.

For this reason I use toolstash restore && go install cmd/xxx if I'm hacking on the toolchain itself.

Yeah, I'm thinking I should use toolstash restore in my script instead of -toolexec=toolstash. The latter is just muscle memory from doing cmd/compile-only hacking.
But does that actually address the fixpoint issue?

My build script is currently:
toolstash restore
go tool compile -V

go install cmd/compile

go build -a -v -gcflags='-d=inlfuncswithclosures=0' -toolexec='toolstash -cmp' std cmd

# golang.org/issue/46377
go install std cmd
go install std cmd
go install std cmd

go run run.go
go test -short std cmd

But I keep running into a situation where the initial go install cmd/compile command seems to try recompiling cmd/compile with a broken compiler instead of the known-good version that I stashed and restored.
For example, right now I have this reliable behavior from cmd/go:
$ go install -a -toolexec=toolstash cmd/compile
[succeeds]

$ toolstash restore
$ go install -a cmd/compile
# math
math/trig_reduce.go:36:19: internal compiler error: math/fma.go:96:27: internal compiler error: math/expm1.go:170:14: internal compiler error: 'trigReduce': addr of canSSA expression:
.   NAME-math.f esc(no) tc(1) Class:PAUTO Offset:0 InlFormal OnStack Used float64 # unsafe.go:23:18,trig_reduce.go:36:19

Why is cmd/go giving me different behavior depending on whether I use -toolexec=toolstash or toolstash restore immediately before?
(I'm not surprised about the compiler error: that's explained by a previous change I made. But I've reverted it in my source tree, and it wasn't in the stashed compiler, so I don't understand why the failure is persisting.)

If you want a complete fixed-point where nothing at all in the tree appears to be stale after modifying the sources to cmd/compile or cmd/link, then the minimum is

go install cmd
go install cmd
go install cmd
go install std

Looking at the inputs and outputs of each (the input to all steps is the same modified source code, so I'm only looking at the toolchain inputs, namely which toolchain is being used):

Inputs = original toolchain.
Outputs = new toolchain as compiled by original.

Inputs = new toolchain as compiled by original.
Outputs = new toolchain as compiled by (new toolchain as compiled by original).

Inputs = new toolchain as compiled by (new toolchain as compiled by original).
Outputs = new toolchain as compiled by (new toolchain as compiled by (new toolchain as compiled by original)).

Because we are using content-based (as opposed to mtime-based) staleness, a particular build result is indexed by the content hashes of the input sources (not changing here) and the binaries being used.
The binaries used as inputs in steps 1 and 2 are certainly different, since the one in step 2 has the modifications compiled into it.
The binaries used as inputs in steps 2 and 3 are different as well, at least when the modifications affect generated code.
Consider the case where the modifications contain an optimization that applies to some part of the compiler.
The output of step 1 = input of step 2 is an unoptimized compilation of the modified compiler sources.
The output of step 2 = input of step 3 is the optimized compilation of the modified compiler sources.
In the absence of compiler bugs, these two toolchains, although different in binary form, should be semantically equivalent.
Assuming those two toolchains really are semantically equivalent, they should produce identical binaries as output, except for the fact that the input hashes recorded in the binaries are different. We can't just rewrite the hashes, because maybe the toolchains aren't semantically equivalent. That is, maybe there's a bug in the compiler (it happens!).
Assuming the two toolchains really behave identically, then step 3 is a semantic repeat of step 2, just using a toolchain with different binaries. So it should produce the same outputs. But these outputs will have been compiled with themselves as their own input hashes, proving that we really have reached a fixed point and that the toolchains really are semantically equivalent, at least as applied to their own source code.
If go list says any part of cmd is stale after step 3, then there is a bug in the toolchain that is causing it not to produce deterministic outputs. This is what the staleness check in cmd/dist looks for.
Finally we need to recompile the standard library with the fixed-point compilers, which is step 4.
All this only matters if you are trying to make the compilers themselves not be reported as stale. The go command has no problem invoking stale compilers, so when I'm working on the toolchain, I just do one go install cmd (or go install cmd/link etc, whatever I'm changing) and then immediately move on to go test -short std. If you trust that the existing compiler works well enough to compile your changes, then there's no problem not forcing the fixed point. You're just trusting instead of verifying. At that point the only tests that fail are the ones that are picky about staleness.

At that point the only tests that fail are the ones that are picky about staleness.

For what it's worth, I just made a lot of those tests a lot less picky about staleness for #46347; see CL 322629.

Should this issue be closed?

authors: bcmills ianlancetaylor mdempsky mvdan rsc thanm


text:
all: Go 1.16.8 release status

Issue tracking the go1.16.8 release by releasebot.

Log
2021/09/09 10:05:27 starting
2021/09/09 10:05:27 working in /usr/local/google/home/thanm/go-releasebot-work/go1.16.8
2021/09/09 10:05:27 $ git clone https://go.googlesource.com/go /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gitmirror
2021/09/09 10:05:49 $ git config gc.auto 0
2021/09/09 10:05:49 $ git fetch origin release-branch.go1.16
2021/09/09 10:05:49 $ git clone --reference /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gitmirror -b release-branch.go1.16 https://go.googlesource.com/go /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gitwork
2021/09/09 10:05:51 $ git codereview change relwork
2021/09/09 10:05:51 $ git config gc.auto 0
2021/09/09 10:05:54 $ git merge-base --is-ancestor e9e0473681e581040b4adcd64b53967e1572fe8d HEAD
2021/09/09 10:05:54 $ git rev-parse go1.16.8
2021/09/09 10:05:54 Release status issue title: "all: Go 1.16.8 release status"
2021/09/09 10:05:54 Release status issue: https://golang.org/issue/48283
2021/09/09 10:05:54 $ git add VERSION
2021/09/09 10:05:55 $ git commit -m go1.16.8

Change-Id: I1926e1506e56fd0484bec71f7b43cee62e41b5b5
 VERSION
2021/09/09 10:05:55 $ git codereview mail -trybot
2021/09/09 10:05:59 $ go clean -modcache
2021/09/09 10:05:59 $ go install golang.org/x/build/cmd/release@latest
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-386 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-386 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-amd64-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target freebsd-386 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target freebsd-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-s390x -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-386-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target src -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-armv6l -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target darwin-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-amd64-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target darwin-arm64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-ppc64le -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:24 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-arm64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:06:42 release -target="src": build succeeded (after 0 retries)
2021/09/09 10:11:11 release -target="linux-ppc64le": build succeeded (after 0 retries)
2021/09/09 10:13:02 release -target="linux-s390x": build succeeded (after 0 retries)
2021/09/09 10:16:52 release -target="linux-arm64": build succeeded (after 0 retries)
2021/09/09 10:16:56 release -target="darwin-arm64": build succeeded (after 0 retries)
2021/09/09 10:18:40 release -target="linux-armv6l": build succeeded (after 0 retries)
2021/09/09 10:19:52 release -target="darwin-amd64": build succeeded (after 0 retries)
2021/09/09 10:22:02 release -target="freebsd-386": build succeeded (after 0 retries)
2021/09/09 10:22:04 release -target="freebsd-amd64": build succeeded (after 0 retries)
2021/09/09 10:22:06 release -target="windows-amd64-longtest" did not produce expected output files [go1.16.8.windows-amd64-longtest.test-only]:
error from cmd/release binary = <nil>
output from cmd/release binary:
2021/09/09 10:06:24 windows-amd64-longtest: Start.
2021/09/09 10:06:24 windows-amd64-longtest: Creating buildlet.
2021/09/09 10:07:01 windows-amd64-longtest: Pushing source to buildlet.
2021/09/09 10:07:07 windows-amd64-longtest: Installing go1.4.
2021/09/09 10:07:10 windows-amd64-longtest: Writing VERSION file.
2021/09/09 10:07:10 windows-amd64-longtest: Cleaning goroot (pre-build).
2021/09/09 10:07:10 windows-amd64-longtest: Building (make.bash only).
2021/09/09 10:09:06 windows-amd64-longtest: Cleaning goroot (post-build).
2021/09/09 10:09:06 windows-amd64-longtest: Pushing and running releaselet.
2021/09/09 10:13:47 windows-amd64-longtest: Cleaning workdir.
2021/09/09 10:13:48 windows-amd64-longtest: Installing go1.4 (second time, for all.bash).
2021/09/09 10:13:51 windows-amd64-longtest: Building (all.bash to ensure tests pass).
2021/09/09 10:22:06 windows-amd64-longtest: Error: Build failed: exit status 1
Output:
Building Go cmd/dist using C:\workdir\go1.4
Building Go toolchain1 using C:\workdir\go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for windows/amd64.

##### Testing packages.
ok  	archive/tar	0.067s
ok  	archive/zip	62.721s
ok  	bufio	0.114s
ok  	bytes	3.008s
ok  	compress/bzip2	0.103s
ok  	compress/flate	14.629s
ok  	compress/gzip	0.229s
ok  	compress/lzw	0.114s
ok  	compress/zlib	1.797s
ok  	container/heap	0.031s
ok  	container/list	0.034s
ok  	container/ring	0.048s
ok  	context	0.275s
ok  	crypto	0.045s
ok  	crypto/aes	0.055s
ok  	crypto/cipher	0.237s
ok  	crypto/des	0.040s
ok  	crypto/dsa	37.577s
ok  	crypto/ecdsa	6.967s
ok  	crypto/ed25519	0.105s
ok  	crypto/elliptic	3.044s
ok  	crypto/hmac	0.029s
ok  	crypto/internal/subtle	0.035s
ok  	crypto/md5	0.028s
ok  	crypto/rand	0.289s
ok  	crypto/rc4	0.063s
ok  	crypto/rsa	0.953s
ok  	crypto/sha1	0.039s
ok  	crypto/sha256	0.035s
ok  	crypto/sha512	0.052s
ok  	crypto/subtle	0.033s
ok  	crypto/tls	7.067s
ok  	crypto/x509	3.374s
ok  	database/sql	1.653s
ok  	database/sql/driver	0.034s
ok  	debug/dwarf	0.051s
ok  	debug/elf	0.055s
ok  	debug/gosym	4.038s
ok  	debug/macho	0.064s
ok  	debug/pe	8.812s
ok  	debug/plan9obj	0.039s
ok  	embed/internal/embedtest	0.048s
ok  	encoding/ascii85	0.046s
ok  	encoding/asn1	0.047s
ok  	encoding/base32	0.075s
ok  	encoding/base64	0.041s
ok  	encoding/binary	0.049s
ok  	encoding/csv	0.080s
ok  	encoding/gob	0.077s
ok  	encoding/hex	0.037s
ok  	encoding/json	2.208s
ok  	encoding/pem	0.049s
ok  	encoding/xml	0.048s
ok  	errors	0.048s
ok  	expvar	0.058s
ok  	flag	0.167s
ok  	fmt	0.148s
ok  	go/ast	0.050s
ok  	go/build	1.795s
ok  	go/build/constraint	0.044s
ok  	go/constant	0.041s
ok  	go/doc	0.088s
ok  	go/format	0.042s
ok  	go/importer	0.299s
ok  	go/internal/gccgoimporter	0.050s
ok  	go/internal/gcimporter	7.441s
ok  	go/internal/srcimporter	8.130s
ok  	go/parser	0.058s
ok  	go/printer	0.518s
ok  	go/scanner	0.048s
ok  	go/token	0.074s
ok  	go/types	24.510s
ok  	hash	0.028s
ok  	hash/adler32	0.058s
ok  	hash/crc32	0.051s
ok  	hash/crc64	0.035s
ok  	hash/fnv	0.040s
ok  	hash/maphash	222.534s
ok  	html	0.050s
ok  	html/template	0.633s
ok  	image	0.403s
ok  	image/color	0.052s
ok  	image/draw	0.096s
ok  	image/gif	0.835s
ok  	image/jpeg	0.289s
ok  	image/png	0.898s
ok  	index/suffixarray	108.965s
ok  	internal/cpu	0.031s
ok  	internal/execabs	0.037s
ok  	internal/fmtsort	0.031s
ok  	internal/poll	1.089s
ok  	internal/profile	0.035s
ok  	internal/reflectlite	0.125s
ok  	internal/singleflight	0.057s
ok  	internal/syscall/windows	0.064s
ok  	internal/syscall/windows/registry	0.034s
ok  	internal/trace	2.534s
ok  	internal/unsafeheader	0.048s
ok  	internal/xcoff	0.069s
ok  	io	0.126s
ok  	io/fs	0.042s
ok  	io/ioutil	0.037s
ok  	log	0.032s
ok  	math	0.081s
ok  	math/big	6.220s
ok  	math/bits	0.036s
ok  	math/cmplx	0.043s
ok  	math/rand	7.887s
ok  	mime	0.071s
ok  	mime/multipart	0.964s
ok  	mime/quotedprintable	0.744s
--- FAIL: TestDialParallel (7.45s)
    dial_test.go:187: dialClosedPort: measured delay 1.015617s
    dial_test.go:332: #6: got 1.2714759s; want <= 1.2187404s
FAIL
FAIL	net	70.139s
ok  	net/http	96.956s
ok  	net/http/cgi	0.417s
ok  	net/http/cookiejar	0.042s
ok  	net/http/fcgi	0.060s
ok  	net/http/httptest	2.337s
ok  	net/http/httptrace	0.029s
ok  	net/http/httputil	1.051s
ok  	net/http/internal	0.042s
ok  	net/http/pprof	5.227s
ok  	net/internal/socktest	0.033s
ok  	net/mail	0.035s
ok  	net/rpc	0.078s
ok  	net/rpc/jsonrpc	0.039s
ok  	net/smtp	0.046s
ok  	net/textproto	0.044s
ok  	net/url	0.029s
ok  	os	131.558s
ok  	os/exec	4.608s
ok  	os/signal	1.927s
ok  	os/user	0.052s
ok  	path	0.035s
ok  	path/filepath	0.731s
ok  	plugin	0.037s
ok  	reflect	4.085s
ok  	regexp	34.623s
ok  	regexp/syntax	0.524s
ok  	runtime	327.281s
ok  	runtime/debug	0.106s
ok  	runtime/internal/atomic	2.937s
ok  	runtime/internal/math	0.036s
ok  	runtime/internal/sys	0.046s
ok  	runtime/metrics	0.027s
ok  	runtime/pprof	57.964s
ok  	runtime/race	0.081s
ok  	runtime/trace	28.370s
ok  	sort	1.020s
ok  	strconv	2.354s
ok  	strings	1.920s
ok  	sync	8.563s
ok  	sync/atomic	3.992s
ok  	syscall	0.031s
ok  	testing	5.346s
ok  	testing/fstest	0.084s
ok  	testing/iotest	0.040s
ok  	testing/quick	0.087s
ok  	text/scanner	0.030s
ok  	text/tabwriter	0.030s
ok  	text/template	1.520s
ok  	text/template/parse	0.135s
ok  	time	9.755s
ok  	unicode	0.044s
ok  	unicode/utf16	0.051s
ok  	unicode/utf8	0.035s
ok  	cmd/addr2line	5.881s
ok  	cmd/api	22.520s
ok  	cmd/asm/internal/asm	1.026s
ok  	cmd/asm/internal/lex	0.071s
ok  	cmd/compile	11.446s
ok  	cmd/compile/internal/gc	36.349s
ok  	cmd/compile/internal/logopt	1.552s
ok  	cmd/compile/internal/ssa	210.663s
ok  	cmd/compile/internal/syntax	8.184s
ok  	cmd/compile/internal/test	0.046s
ok  	cmd/compile/internal/types	0.028s
ok  	cmd/cover	4.196s
ok  	cmd/doc	0.384s
ok  	cmd/fix	6.359s
ok  	cmd/go	182.558s
ok  	cmd/go/internal/auth	0.027s
ok  	cmd/go/internal/cache	154.899s
ok  	cmd/go/internal/fsys	0.362s
ok  	cmd/go/internal/generate	0.055s
ok  	cmd/go/internal/get	0.253s
ok  	cmd/go/internal/imports	0.034s
ok  	cmd/go/internal/load	0.095s
ok  	cmd/go/internal/lockedfile	3.296s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.112s
ok  	cmd/go/internal/modconv	0.057s
ok  	cmd/go/internal/modfetch	0.536s
ok  	cmd/go/internal/modfetch/codehost	0.046s
ok  	cmd/go/internal/modfetch/zip_sum_test	0.111s
ok  	cmd/go/internal/modload	0.065s
ok  	cmd/go/internal/mvs	0.031s
ok  	cmd/go/internal/par	0.065s
ok  	cmd/go/internal/renameio	89.327s
ok  	cmd/go/internal/search	0.047s
ok  	cmd/go/internal/str	0.028s
ok  	cmd/go/internal/test	0.063s
ok  	cmd/go/internal/txtar	0.031s
ok  	cmd/go/internal/vcs	0.414s
ok  	cmd/go/internal/web	0.033s
ok  	cmd/go/internal/work	0.196s
ok  	cmd/gofmt	9.749s
ok  	cmd/internal/archive	2.745s
ok  	cmd/internal/buildid	0.416s
ok  	cmd/internal/dwarf	0.096s
ok  	cmd/internal/edit	0.058s
ok  	cmd/internal/goobj	0.048s
ok  	cmd/internal/moddeps	4.733s
ok  	cmd/internal/obj	0.469s
ok  	cmd/internal/obj/ppc64	1.101s
ok  	cmd/internal/obj/riscv	3.981s
ok  	cmd/internal/obj/s390x	0.057s
ok  	cmd/internal/obj/x86	16.168s
ok  	cmd/internal/objabi	0.059s
ok  	cmd/internal/pkgpath	0.449s
ok  	cmd/internal/src	0.112s
ok  	cmd/internal/sys	0.106s
ok  	cmd/internal/test2json	0.264s
ok  	cmd/link	48.817s
ok  	cmd/link/internal/benchmark	0.104s
ok  	cmd/link/internal/ld	54.007s
ok  	cmd/link/internal/loader	0.094s
ok  	cmd/nm	18.395s
ok  	cmd/objdump	18.005s
ok  	cmd/pack	14.358s
ok  	cmd/trace	0.587s
ok  	cmd/vet	21.854s
FAIL
go tool dist: Failed: exit status 1
2021/09/09 10:22:06 release -target="windows-amd64-longtest": waiting a bit and trying again
2021/09/09 10:22:41 release -target="linux-amd64": build succeeded (after 0 retries)
2021/09/09 10:23:01 release -target="linux-386": build succeeded (after 0 retries)
2021/09/09 10:23:06 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-amd64-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/9a348ce4c329eadd02ffe7e397a5ff77c5f503b8_2613948513 -rev 9a348ce4c329eadd02ffe7e397a5ff77c5f503b8
2021/09/09 10:27:25 release -target="linux-amd64-longtest": build succeeded (after 0 retries)
2021/09/09 10:31:36 release -target="windows-386": build succeeded (after 0 retries)
2021/09/09 10:31:52 release -target="linux-386-longtest": build succeeded (after 0 retries)
2021/09/09 10:34:38 release -target="windows-amd64": build succeeded (after 0 retries)
2021/09/09 10:50:40 release -target="windows-amd64-longtest": build succeeded (after 1 retries)
2021/09/09 10:50:40

The prepare stage has completed.

Please review and submit https://go-review.googlesource.com/q/I1926e1506e56fd0484bec71f7b43cee62e41b5b5
and then run the release stage.

Latest build: go1.16.8

src (tar.gz)
linux-386 (tar.gz)
linux-armv6l (tar.gz)
linux-amd64 (tar.gz)
linux-arm64 (tar.gz)
freebsd-386 (tar.gz)
freebsd-amd64 (tar.gz)
windows-386 (zip) (msi)
windows-amd64 (zip) (msi)
darwin-amd64 (tar.gz)
darwin-arm64 (tar.gz)
linux-s390x (tar.gz)
linux-ppc64le (tar.gz)
linux-386-longtest (test only) - ok
linux-amd64-longtest (test only) - ok
windows-amd64-longtest (test only) - ok

Log
2021/09/09 11:28:48 starting
2021/09/09 11:28:48 working in /usr/local/google/home/thanm/go-releasebot-work/go1.16.8
2021/09/09 11:28:48 $ git fetch origin master
2021/09/09 11:28:50 $ git fetch origin release-branch.go1.16
2021/09/09 11:28:52 $ git clone --reference /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gitmirror -b release-branch.go1.16 https://go.googlesource.com/go /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gitwork
2021/09/09 11:28:54 $ git codereview change relwork
2021/09/09 11:28:55 $ git config gc.auto 0
2021/09/09 11:28:56 $ git merge-base --is-ancestor e9e0473681e581040b4adcd64b53967e1572fe8d HEAD
2021/09/09 11:28:56 Release status issue title: "all: Go 1.16.8 release status"
2021/09/09 11:28:56 Release status issue: https://golang.org/issue/48283
2021/09/09 11:28:56 $ git rev-parse go1.16.8
2021/09/09 11:35:58 $ git tag go1.16.8 170a72e58bd128b421f4b3974fe2a37fd035efdf
2021/09/09 11:35:58 $ git push origin go1.16.8
2021/09/09 11:36:02 $ go clean -modcache
2021/09/09 11:36:03 $ go install golang.org/x/build/cmd/release@latest
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-armv6l -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-amd64-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target freebsd-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target darwin-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-s390x -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-386-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-ppc64le -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-amd64-longtest -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target freebsd-386 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target darwin-arm64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-arm64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target windows-386 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target src -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-amd64 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:13 $ /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/gopath/bin/release -target linux-386 -user thanm -version go1.16.8 -staging_dir /usr/local/google/home/thanm/go-releasebot-work/go1.16.8/release-staging/170a72e58bd128b421f4b3974fe2a37fd035efdf_1980453687 -rev 170a72e58bd128b421f4b3974fe2a37fd035efdf -skip_tests
2021/09/09 11:36:37 release -target="src": build succeeded (after 0 retries)
2021/09/09 11:38:50 release -target="linux-arm64": build succeeded (after 0 retries)
2021/09/09 11:39:02 release -target="darwin-amd64": build succeeded (after 0 retries)
2021/09/09 11:39:14 release -target="linux-armv6l": build succeeded (after 0 retries)
2021/09/09 11:39:24 release -target="freebsd-amd64": build succeeded (after 0 retries)
2021/09/09 11:43:27 release -target="windows-amd64-longtest": build succeeded (after 0 retries)
2021/09/09 11:43:31 release -target="linux-ppc64le": build succeeded (after 0 retries)
2021/09/09 11:43:32 release -target="freebsd-386": build succeeded (after 0 retries)
2021/09/09 11:44:43 release -target="linux-386": build succeeded (after 0 retries)
2021/09/09 11:44:46 release -target="linux-amd64": build succeeded (after 0 retries)
2021/09/09 11:46:14 release -target="darwin-arm64": build succeeded (after 0 retries)
2021/09/09 11:51:00 release -target="windows-amd64": build succeeded (after 0 retries)
2021/09/09 11:51:44 release -target="windows-386": build succeeded (after 0 retries)
2021/09/09 11:53:49 release -target="linux-amd64-longtest": build succeeded (after 0 retries)
2021/09/09 11:55:04 release -target="linux-386-longtest": build succeeded (after 0 retries)
2021/09/09 11:58:51 release -target="linux-s390x": build succeeded (after 0 retries)
2021/09/09 11:58:54 changing milestone of issue 47858 to Go1.16.9
2021/09/09 11:58:55 changing milestone of issue 47997 to Go1.16.9
2021/09/09 11:58:55 changing milestone of issue 48115 to Go1.16.9
2021/09/09 11:58:56 changing milestone of issue 48251 to Go1.16.9
2021/09/09 11:58:56 closing milestone Go1.16.8
2021/09/09 11:58:57

The release stage has completed. Thanks for riding with releasebot today!

Please refer to the playbook for the next steps.

Closing, release complete.


text:
net/http: frequent HTTP2 INTERNAL_ERROR errors during module zip download since 2021-10-06

#!watchflakes
post <- log ~ `golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*:.* stream error: stream ID \d+; INTERNAL_ERROR; received from peer`

The builders are seeing a lot of INTERNAL_ERROR results since 2021-10-06, typically for google.golang.org/protobuf but sometimes for other golang.org paths as well.
This may be related to #50541, which saw an uptick in errors from go.googlesource.com in about the same timeframe.
CC @golang/release
../../../../pkg/mod/cloud.google.com/go/datastore@v1.2.0/client.go:25:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/internal/trace/trace.go:23:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/google.golang.org/grpc@v1.39.0/status/status.go:34:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/datastore@v1.2.0/save.go:26:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/cloudbuild/apiv1/v2/cloud_build_client.go:33:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/longrunning/autogen/operations_client.go:31:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go@v0.88.0/iam/iam.go:30:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer
../../../../pkg/mod/cloud.google.com/go/storage@v1.10.0/iam.go:24:2: google.golang.org/genproto@v0.0.0-20210726143408-b02e89920bf0: stream error: stream ID 3; INTERNAL_ERROR; received from peer

greplogs --dashboard -md -l -e 'golang\.org/[^@]+@v\d+\.\d+\.\d+[a-z0-9-.]*: stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2021-01-01
2022-02-19T16:23:54-3962a08-0261fa6/dragonfly-amd64
2022-02-10T22:36:44-59536be-0a6cf87/linux-amd64-wsl
2022-02-09T18:09:24-9862752-6a70ee2/darwin-arm64-11_0-toothrot
2022-02-09T16:29:28-dad3315-0a6cf87/darwin-arm64-12_0-toothrot
2022-02-09T16:29:28-095f870-0a6cf87/darwin-arm64-12_0-toothrot
2022-02-08T17:01:10-9b156ee-ef06a5f/plan9-amd64-0intro
2022-02-08T16:42:58-ac99473-e2277c8/darwin-arm64-12_0-toothrot
2022-02-03T19:55:40-cd36cc0-896df42/plan9-amd64-0intro
2022-01-25T22:56:45-c20fd7c-6eb58cd/plan9-amd64-0intro
2022-01-24T21:27:33-6944b10-671e115/plan9-amd64-0intro
2022-01-24T21:27:33-5e0467b-671e115/plan9-amd64-0intro
2022-01-14T16:41:18-4bd3f69-b41185c/plan9-386-0intro
2022-01-07T02:32:39-e83268e-2bb7f6b/linux-amd64-wsl
2022-01-05T22:15:55-43762dc-a845a56/darwin-arm64-12_0-toothrot
2022-01-03T23:45:12-be6af36-95b240b/plan9-amd64-0intro
2022-01-02T14:27:43-6944b10-c886143/plan9-amd64-0intro
2021-12-15T20:26:19-be6af36-07ed86c/darwin-arm64-12_0-toothrot
2021-12-14T18:18:45-ecdc095-becaeea/plan9-arm
2021-12-14T01:48:22-6944b10-1afa432/plan9-amd64-0intro
2021-11-10T19:35:55-03971e3-b954f58/darwin-arm64-11_0-toothrot
2021-10-28T14:25:03-103d89b-a3bb28e/darwin-amd64-10_15
2021-10-26T11:58:05-d418f37-1e2820a/openbsd-arm64-jsing
2021-10-14T15:16:55-a66eb64-ad99d88/linux-amd64-wsl
2021-10-14T01:51:22-e13a265-276fb27/windows-arm64-10
2021-10-13T00:11:47-281050f-4fb2e1c/darwin-amd64-11_0
2021-10-11T20:46:14-089bfa5-7023535/darwin-amd64-11_0
2021-10-07T20:37:43-86761ae-ef2ebbe/darwin-arm64-11_0-toothrot
2021-10-07T20:37:43-59d4e92-ef2ebbe/darwin-arm64-11_0-toothrot
2021-10-06T18:44:56-40a54f1-195945a/darwin-amd64-nocgo

From logs, it's also happened for github.com dependencies such as github.com/yuin/goldmark in 2022-02-08T16:42:58-ac99473-e2277c8/darwin-arm64-12_0-toothrot.
Downloading happens with GOPROXY=https://proxy.golang.org set, so perhaps the go.googlesource.com and golang.org servers aren't relevant (they just happen to be where most of dependencies are coming from).
I've learned that INTERNAL_ERROR refers to an HTTP/2 error code that is described as "Implementation fault", but it's not clear to me if the server or client is at fault. CC @neild.

The best explanation I have for this is some kind of infrequent incompatibility between the Go HTTP libraries and the Google front end servers.

It occurs to me that this could be related to #49645 (CC @neild), although the timeline doesn't quite match up because the fix for that landed Nov. 17.
greplogs --dashboard -md -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2022-02-23 --omit=plan9
2022-03-29T15:43:06-e96d8cf-ae9ce82/windows-amd64-longtest
2022-03-23T17:12:00-87a8611-dd211cf/freebsd-arm64-dmgk

Is there any update on this issue! Experiencing too frequently

greplogs -l -e 'stream error: stream ID \d+; INTERNAL_ERROR; received from peer' --since=2022-03-30 --omit=plan9
2022-05-05T16:31:21-15f5199-fdb640b/darwin-arm64-12_0-toothrot
2022-04-26T16:21:18-7b82a4e-e357634/solaris-amd64-oraclerel
2022-04-23T05:38:56-84e58bf-9717e8f/linux-mips64le-mengzhuo
2022-04-12T22:05:26-2897e13-d85694a/windows-amd64-longtest
2022-04-04T04:01:17-a8550c1-a92ca51/linux-mips64le-mengzhuo
2022-03-31T14:57:35-27fe37a-378221b/linux-amd64-longtest
2022-03-31T14:51:17-27fe37a-2ea9376/linux-amd64-longtest

This could possibly be a duplicate of #36759 (reported as a regression Jan. 2020).

Assigning to @neild (net/http owner) on the theory that this may be reproducing a previously-reported net/http client bug (#36759).

stream error: stream ID N; INTERNAL_ERROR; received from peer means an HTTP/2 stream (a request) was closed by the remote endpoint with the error code INTERNAL_ERROR.
There are various other error codes to tell the client that it is misbehaving, such as PROTOCOL_ERROR. INTERNAL_ERROR is supposed to indicate an internal error on the remote side.
So whatever is doing the module zip download is claiming that it's unhealthy. It's not inconceivable that this is somehow the result of a net/http client bug, but if it is the remote side is doing a very bad job of telling us about it.
Maybe if we can reproduce the failure with GODEBUG=http2debug=2 enabled we might see something that helps indicate the source of the problem, but at the moment my suspicion is that the problem lies on the remote end of the connection.

https://github.com/googleapis/google-cloud-go/issues/784 seems like the same issue. From 2017, eventual "fix" was to notice INTERNAL_ERROR and retry the read.

The other end of the connection here is the GFEs; should we try to escalate to them internally?

Yeah, I think so.
The possibilities I see are:

The net/http client is conjuring a peer-originated error out of thin air. This seems incredibly unlikely. I can find only one code path that adds "received from peer" and I don't see any way for it to lie.
The net/http client is misbehaving (very easy to believe) and the GFE is somehow responding with INTERNAL_ERROR. That's still a GFE bug, even if it's our fault for tickling it.
The net/http client is doing everything right, and the problem is entirely at or beyond the GFE.

At the same time, perhaps we should have a retry loop on module downloads. Connections break for various other reasons; doesn't seem like it would hurt to retry when they do.

Any idea how to solve this? Getting the following error when running go install golang.org/x/tools/gopls@latest.
/hpath/to/go/pkg/mod/golang.org/x/tools@v0.1.11-0.20220513164230-dfee1649af67/internal/lsp/source/hover.go:23:2: golang.org/x/text@v0.3.7: stream error: stream ID 9; INTERNAL_ERROR; received from peer

@neild What is the current status here?  This issue is currently in the 1.19 milestone.  Should it move to 1.20?  To Backlog?  Thanks.

Our current belief is that the root cause is the GFE sending an INTERNAL_ERROR, probably due to the backend behind the GFE breaking the response stream.
This is not an net/http bug; net/http seems to be accurately reporting the error code sent by the GFE. (The error message could perhaps be clearer, but that's a secondary issue.)
Whatever tools are using net/http to fetch large files should be more robust in the handling of stream errors. There are many reasons a large fetch might break mid-stream, and it's unlikely we can entirely address this situation entirely by reducing the number of broken streams. This means we should add some form of retry to module zip downloads.
I think it's okay to move this to the 1.20 milestone.

I am facing the below errors
http2: Framer 0xc00029b5e0: read RST_STREAM stream=3 len=4 ErrCode=INTERNAL_ERROR
http2: Transport received RST_STREAM stream=3 len=4 ErrCode=INTERNAL_ERROR
RoundTrip failure: stream error: stream ID 3; INTERNAL_ERROR; received from peer
Any root cause for this?

In stream error: stream ID 3; INTERNAL_ERROR; received from peer, the "received from peer" indicates that the INTERNAL_ERROR code was received from the remote endpoint.
That is, the server you are talking to sent back an INTERNAL_ERROR code, presumably indicating that it has encountered an internal error of some form. net/http is just relaying that fact to you. The root cause is presumably on the server side.

is server sending RST_STREAM incase of any internal error introduced in latest go version or was it existing from beginning?

I'm sorry, but I don't have enough details to attempt to answer that question.
The error is coming from the remote server. Did you update your server to a newer version of x/net, your client, or both? If it's just the client, then you want to ask why your server is producing INTERNAL_ERROR errors, not why the client is surfacing them. If you did update the server, then I'd really need a reproduction case to attempt to debug the issue (and we should do that on a new issue, not this one, which is not about errors being produced by Go servers).
v0.0.0-20211029224645-99673261e6eb to v0.19.0 is two years of changes. I'm afraid I don't remember everything that happened in that time.

authors: Sebastin-cr Tomperez98 bcmills dmitshur heschi ianlancetaylor mintunitish neild


text:
x/tools: timeouts on slow builders due to rebuilding std packages for import

#!watchflakes
post <- builder ~ `(netbsd-arm.*-bsiegert|.*-riscv64.*)` && pkg ~ `^golang.org/x/tools` && `^panic: test timed out`

(Pulled out from https://github.com/golang/go/issues/56180#issuecomment-1329599265.)

(CC @bsiegert @matloob)
I believe that these tests are timing out because they are rebuilding std in order to use its compiled export data in the test. Skipping the tests that time out seems to just shift the test timeout to a later test.
I considered suggesting that cmd/coordinator should run go build std very early on (before the test timeout takes effect), but that seems wasteful for the majority of repos that only need a small fraction of the standard library.
Looking into this further, I think the most appropriate solution is probably to run the x/tools tests respecting the GO_TEST_TIMEOUT_SCALE configured for the builder (filed as #56968).

Looking at the current state of the build dashboard, I believe this has been resolved (for now) by CL 454497 and CL 454499.

The last of the above failures precedes the fix CL 454499.

Old failures are old.

That last one looks like #57999; not clear to me why it wasn't posted there. I'm going to remove this issue from the Test Flakes project to stop watchflakes from reopening it.


text:
runtime: lock ordering problem: gscan, profBlock

https://build.golang.org/log/b20065c3dd85e387e63949a5dcd73cf49f3ab7a5 shows a staticlockranking failure, reproduced below.
This looks real: gscan is held while profBlock is acquired. But the ranking order seems to say profBlock has to be less than STACKGROW, which has to be less than gscan. So it does not appear that adjusting the policy will fix the problem.
/cc @aclements
0 : gscan 39 0x0
1 : profBlock 36 0xa70520
fatal error: lock ordering problem

runtime stack:
runtime.throw({0x77977b?, 0x0?})
	/workdir/go/src/runtime/panic.go:1021 +0x5c fp=0x7f840108a780 sp=0x7f840108a750 pc=0x43d77c
runtime.checkRanks(0xc0000a8380, 0x434079?, 0xc0000a8380?)
	/workdir/go/src/runtime/lockrank_on.go:162 +0x236 fp=0x7f840108a7e0 sp=0x7f840108a780 pc=0x40f136
runtime.lockWithRank.func1()
	/workdir/go/src/runtime/lockrank_on.go:87 +0x85 fp=0x7f840108a810 sp=0x7f840108a7e0 pc=0x40ed85
runtime.lockWithRank(0x3?, 0x0?)
	/workdir/go/src/runtime/lockrank_on.go:76 +0x85 fp=0x7f840108a848 sp=0x7f840108a810 pc=0x40ecc5
runtime.lock(...)
	/workdir/go/src/runtime/lock_futex.go:52 pc=0x435394
runtime.saveBlockEventStack(0xff7, 0x1, {0xc0001f6280?, 0x80c00041bfff?, 0x2030001?}, 0x3)
	/workdir/go/src/runtime/mprof.go:749 +0x48 fp=0x7f840108a890 sp=0x7f840108a848 pc=0x435388
runtime.(*mLockProfile).store(0xc0001f6278)
	/workdir/go/src/runtime/mprof.go:733 +0xa5 fp=0x7f840108a8f0 sp=0x7f840108a890 pc=0x4352a5
runtime.(*mLockProfile).recordUnlock(0xc0001f6278?, 0x0?)
	/workdir/go/src/runtime/mprof.go:673 +0x47 fp=0x7f840108a908 sp=0x7f840108a8f0 pc=0x434fc7
runtime.unlock2(0x2030000?)
	/workdir/go/src/runtime/lock_futex.go:134 +0x5d fp=0x7f840108a930 sp=0x7f840108a908 pc=0x40e6dd
runtime.unlockWithRank.func1()
	/workdir/go/src/runtime/lockrank_on.go:188 +0xed fp=0x7f840108a988 sp=0x7f840108a930 pc=0x40f36d
runtime.unlockWithRank(0xa16f00?)
	/workdir/go/src/runtime/lockrank_on.go:174 +0x77 fp=0x7f840108a9b8 sp=0x7f840108a988 pc=0x40f257
runtime.unlock(...)
	/workdir/go/src/runtime/lock_futex.go:121 pc=0x42dd54
runtime.(*mheap).allocSpan(0xa16f00, 0x4, 0x3, 0x0)
	/workdir/go/src/runtime/mheap.go:1262 +0x227 fp=0x7f840108aa58 sp=0x7f840108a9b8 pc=0x42dd47
runtime.(*mheap).allocManual(0x40f7a0?, 0xc0000a8380?, 0x2c?)
	/workdir/go/src/runtime/mheap.go:990 +0x19 fp=0x7f840108aa80 sp=0x7f840108aa58 pc=0x42d6d9
runtime.getempty.func1()
	/workdir/go/src/runtime/mgcwork.go:378 +0x2d fp=0x7f840108aab0 sp=0x7f840108aa80 pc=0x42c40d
runtime.getempty()
	/workdir/go/src/runtime/mgcwork.go:377 +0xfc fp=0x7f840108ab10 sp=0x7f840108aab0 pc=0x42c29c
runtime.(*stackScanState).addObject(0x7f840108ac38, 0xc000503fb8, 0x7b6c28)
	/workdir/go/src/runtime/mgcstack.go:274 +0x2f fp=0x7f840108ab38 sp=0x7f840108ab10 pc=0x42976f
runtime.scanframeworker(0x7f840108abd8, 0x7f840108ac38, 0xc000038668)
	/workdir/go/src/runtime/mgcmark.go:1095 +0x1fd fp=0x7f840108ab98 sp=0x7f840108ab38 pc=0x4242fd
runtime.scanstack(0xc0004508c0, 0xc000038668)
	/workdir/go/src/runtime/mgcmark.go:898 +0x267 fp=0x7f840108acc8 sp=0x7f840108ab98 pc=0x423be7
runtime.markroot.func1()
	/workdir/go/src/runtime/mgcmark.go:239 +0xb5 fp=0x7f840108ad18 sp=0x7f840108acc8 pc=0x422895
runtime.markroot(0xc000038668, 0x72, 0x0)
	/workdir/go/src/runtime/mgcmark.go:213 +0x1a8 fp=0x7f840108adc0 sp=0x7f840108ad18 pc=0x422528
runtime.gcDrainN(0xc000038668, 0x10000)
	/workdir/go/src/runtime/mgcmark.go:1321 +0x15d fp=0x7f840108adf0 sp=0x7f840108adc0 pc=0x4248dd
runtime.gcAssistAlloc1(0xc000196e00, 0x10000)
	/workdir/go/src/runtime/mgcmark.go:663 +0x112 fp=0x7f840108ae50 sp=0x7f840108adf0 pc=0x423412
runtime.gcAssistAlloc.func1()
	/workdir/go/src/runtime/mgcmark.go:554 +0x1b fp=0x7f840108ae70 sp=0x7f840108ae50 pc=0x4232db
runtime.systemstack(0x800000)
	/workdir/go/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f840108ae80 sp=0x7f840108ae70 pc=0x4750aa

This appears to be related to the addition of runtime lock profiling to the mutex profile.
CC @rhysh @prattmic

@mknyszek , is this the same problem described in https://github.com/golang/go/issues/64706#issuecomment-1861227870 ?
That was tabled in December because 1/ it was test-only and 2/ it was too late in the release cycle to change Gscan users to also note their non-preemptibility in gp.m.locks. Do those still hold?

I continue to see these on the dashboard at a low rate. Marking release-blocker. It sounds like we could remove runtime locks from the mutex profile at the very least.

Looks like more reports are accumulating on #58277.

Is there any status update to report on this issue?

authors: cagedmantis mknyszek rhysh rsc


text:
x/tools/gopls: export LSP types so they can be imported by external consumers

The code gopls uses for LSP types in tools/gopls/internal/protocol looks great and well-maintained. There doesn't appear to be another well-maintained open-source alternative. Is it possible to move this code out from under internal so it can be utilized by other consumers who wish to implement a language server in Go?

This topic has come up before, but for one thing, we have never had the
resources to support a public package. Also, the package contains a bunch
of specializations tuned to the needs of gopls which change (admittedly
slowly) as gopls does.
On Sun, May 26, 2024 at 5:00 PM japem @.***> wrote:

The code gopls uses for LSP types in tools/gopls/internal/protocol
https://github.com/golang/tools/tree/master/gopls/internal/protocol
looks great and well-maintained. There doesn't appear to be another
well-maintained open-source alternative. Is it possible to move this code
out from under internal so it can be utilized by other consumers who wish
to implement a language server in Go?
—
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/67658, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABJIAIYOHV74UUBRSF7TD6LZEJENNAVCNFSM6AAAAABIKC5MECVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMYTOOJTGA4DKOI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

That's a bummer to hear. Is the existing code fit for use to create types to use in a generic, non-gopls language server? How much maintenance is required if I were to fork it, beyond pointing the codegen at the latest version of the spec?

Unfortunately we have strict compatibility rules on the Go team that would make it rather a lot of toils to export a volatile package such as the protocol package. We can and do change the API of that package in breaking ways.
It would be nice to expose the guts of gopls as a 'LSP in a box' toolkit that could be used for other languages, but frankly speaking we do not have the resources to support such an effort. Folks are of course welcome to fork for their own purposes, but currently have no plans to do this ourselves in the near future.

authors: findleyr japem pjweinb


text:
x/telemetry/config: add govulncheck counters

Proposal Details
Counter Names
govulncheck/level:{symbol, package, module}
govulncheck/mode:{source, binary, extract, query, convert}
govulncheck/format:{text, json, sarif, openvex}
govulncheck/show:{none, traces, color, verbose, version}
govulncheck/assumptions:{multi-patterns, no-binary-platform, no-relative-path, no-go-root, local-replace, unknown-pkg-mod-path}
govulncheck/vulns: {reviewed,unreviewed}
Description
See proposed config change.
Rationale
The proposed counters will allow us to better understand how people are using and interacting with govulncheck, which should help us with work prioritization and improvement of services. For instance, govulncheck/show:traces will tell us how often are users inspecting call stacks in detail. This in turn can help us plan for better call stack heuristics down the road.
Do the counters carry sensitive user information?
No.
Version
Starting with govulncheck v1.1.1.
Proposed Graph Config
counter: govulncheck/level:{symbol,package,module}
title: Scan Level Distribution
description: measure govulncheck scan level distribution
type: partition
program: golang.org/x/vuln/cmd/govulncheck
version: [1.1.1-]

counter: govulncheck/mode:{source,binary,extract,query,convert}
title: Scan Mode Distribution
description: measure govulncheck scan mode distribution
type: partition
program: golang.org/x/vuln/cmd/govulncheck
version: [1.1.1-]

counter: govulncheck/format:{text,json,sarif,openvex}
title: Output Format Distribution
description: measure govulncheck output format distribution
type: partition
program: golang.org/x/vuln/cmd/govulncheck
version: [1.1.1-]

counter: govulncheck/show:{none,traces,color,verbose,version}
title: Show Options Distribution
description: measure govulncheck show flag distribution
type: partition
program: golang.org/x/vuln/cmd/govulncheck
version: [1.1.1-]

counter: govulncheck/vulns:{reviewed,unreviewed}
title: Vulnerability Status Distribution
description: measure detected reviewed/unreviewed vulnerability distribution
type: partition
program: golang.org/x/vuln/cmd/govulncheck
version: [1.1.1-]

govulncheck/assumptions:{multi-patterns, no-binary-platform, no-relative-path, no-go-root, local-replace, unknown-pkg-mod-path}
title: Code Invariants Distribution
description: measure distribution of failed govulncheck internal assumptions
type: partition
program: golang.org/x/vuln/cmd/govulncheck
version: [1.1.1-]

Looks good to us. Can you please send a CL implementing the new config? Thanks.

authors: findleyr zpavlinovic


text:
website: add file size on golang.org/dl/

It would be nice if http://golang.org/dl/ shows the size of downloadable files also. (third world country reasons)

This can be closed, https://golang.org/dl/ has had the file sizes starting with Go 1.4.3.

authors: andrewaustin kokizzu


text:
net: add support for FileConn, FilePacketConn, FileListener on windows

Issue for discussion.
FileListener is useful to implement graceful-restart server. ex: https://github.com/lestrrat/go-server-starter
I know windows doesn't have a way to pass FDs to external process like UNIX OSs.
But it's not impossible. WSADuplicateSocket can export intormation of file descriptor as WSAPROTOCOL_INFO to specified process. This structure is possible to be written to a file as byte array.
http://msdn.microsoft.com/ja-jp/library/windows/desktop/ms741565(v=vs.85).aspx
How about implementation of os.File.dup, os.FileListener on windows?
https://codereview.appspot.com/177590043/

@mikioh I've been trying to wrap a net.Listener around the syscall.Handle I get from windows sockets APIs (e.g. WSASocket).  SyscallConn doesn't really help me with that...does it? I'd love to be able to do something like this:
	pi := syscall.WSAProtocolInfo{}
        //fill pi by some means
	fd, err := xwin.WSASocket(pi.AddressFamily, pi.SocketType, pi.Protocol, &pi, 0, 0)
	if err != nil {
		return nil, err
	}
        listener, err := net.FileListener(os.NewFile(uintptr(fd), "listener"))

But of course this doesn't work because of this issue.

authors: evanmoses-okta mattn


text:
新增研討會及資訊社群

加入 Python Taiwan, Google Developers Groups Taiwan, Telegram 資訊群組清單
加入 HITCON, MOPCON

抱歉一忙就忘記了 😝
感謝！🎉

source: gharchive/pull-request

authors: SiriusKoan goodjack

repo: goodjack/awesome-cs-training


text:
can't explore UNet2DConditionModel from diffusers lib

Hi,
I can't run model_explorer on UNet2DConditionModel from diffusers lib.
Could you please help me solve this problem?
Environment
google colab

torch                            2.4.0+cu121
torchvision                      0.19.0+cu121
diffusers                        0.30.1
ai-edge-model-explorer           0.1.10
ai-edge-model-explorer-adapter   0.1.5

Steps to Reproduce:
!pip install ai-edge-model-explorer
!pip install diffusers

import os
import torch
import torchvision
from diffusers import UNet2DConditionModel
import model_explorer

import requests
url = 'https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder/resolve/main/unet/config.json'

response = requests.get(url)
config = response.json()

unet = UNet2DConditionModel(**config)

image_embeds = torch.randn((1, 1280))
latent_model_input = torch.randn((1, 4, 64, 64))
t = torch.tensor(1.).long()
added_cond_kwargs = {"image_embeds": image_embeds}

ep = torch.export.export(
    unet,
    args=(latent_model_input, t, None),
    kwargs={
        'added_cond_kwargs': added_cond_kwargs,
        'return_dict': False
    },
)

model_explorer.visualize_pytorch('unet', exported_program=ep)

Error log
Converting pytorch model to model explorer graphs...
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-8163fd1130fd> in <cell line: 3>()
      1 import model_explorer
      2 # Visualize
----> 3 model_explorer.visualize_pytorch('unet', exported_program=ep)

/usr/local/lib/python3.10/dist-packages/model_explorer/apis.py in visualize_pytorch(name, exported_program, host, port, extensions, colab_height, settings)
     80   # Construct config.
     81   cur_config = config()
---> 82   cur_config.add_model_from_pytorch(
     83       name, exported_program=exported_program, settings=settings
     84   )

/usr/local/lib/python3.10/dist-packages/model_explorer/config.py in add_model_from_pytorch(self, name, exported_program, settings)
     88     print('Converting pytorch model to model explorer graphs...')
     89     adapter = PytorchExportedProgramAdapterImpl(exported_program, settings)
---> 90     graphs = adapter.convert()
     91     graphs_index = len(self.graphs_list)
     92     self.graphs_list.append(graphs)

/usr/local/lib/python3.10/dist-packages/model_explorer/pytorch_exported_program_adater_impl.py in convert(self)
    236
    237   def convert(self) -> ModelExplorerGraphs:
--> 238     return {'graphs': [self.create_graph()]}

/usr/local/lib/python3.10/dist-packages/model_explorer/pytorch_exported_program_adater_impl.py in create_graph(self)
    232     graph = Graph(id='graph', nodes=[])
    233     for node in self.gm.graph.nodes:
--> 234       graph.nodes.append(self.create_node(node))
    235     return graph
    236

/usr/local/lib/python3.10/dist-packages/model_explorer/pytorch_exported_program_adater_impl.py in create_node(self, fx_node)
    225     )
    226     self.add_incoming_edges(fx_node, node)
--> 227     self.add_node_attrs(fx_node, node)
    228     self.add_outputs_metadata(fx_node, node)
    229     return node

/usr/local/lib/python3.10/dist-packages/model_explorer/pytorch_exported_program_adater_impl.py in add_node_attrs(self, fx_node, node)
    183               KeyValue(
    184                   key='__value',
--> 185                   value=self.print_tensor(
    186                       tensor, self.settings['const_element_count_limit']
    187                   ),

/usr/local/lib/python3.10/dist-packages/model_explorer/pytorch_exported_program_adater_impl.py in print_tensor(self, tensor, size_limit)
    152
    153   def print_tensor(self, tensor: torch.Tensor, size_limit: int = 16):
--> 154     shape = tensor.shape
    155     total_size = 1
    156     for dim in shape:

AttributeError: 'bool' object has no attribute 'shape'

Hi @hanruijiang, I was able to replicate exactly as you described ... it seems like model explorer expects all arguments to be tensors: https://github.com/google-ai-edge/model-explorer/blob/main/src/server/package/src/model_explorer/pytorch_exported_program_adater_impl.py#L177
Hi @yijie-yang, can you please take a look?

Hi @hanruijiang, I was able to replicate exactly as you described ... it seems like model explorer expects all arguments to be tensors: https://github.com/google-ai-edge/model-explorer/blob/main/src/server/package/src/model_explorer/pytorch_exported_program_adater_impl.py#L177
Hi @yijie-yang, can you please take a look?

Hi @pkgoogle ,
You're right. I fixed this bug by remove 'return_dict': False in the kwargs .
Thank you for your help.

@hanruijiang, Thanks, yeah that worked for me as well... if you have no more open items, feel free to close, thanks.

authors: hanruijiang pkgoogle

repo: google-ai-edge/model-explorer


text:
Makefile is not compatible with tf 2.16.1

If you try compile using make, shows there isn't  common.c file. In Tf 2.16.1 there is only common.cc .

got the same issue here.

got the same issue here.
I fixed makefile:

new_makefile.patch.txt

I think that TF-2.17.0 replace common_internal.cc for common.cc .

@feranick without the patch provided by @mwprado it won't compile, even with the newest commit of the main branch.
Additionally, it is important to mention, that the documentation is misleading (not to say wrong) in respect to the build dependencies: On ubuntu / debian systems the provided package for libflatbuffers-dev uses a different scheme for version numbers. The current versions are v2.x, but tensorflow requests v23.x.y. So builds will never work.
It would be great to change the documentation and add instructions to build and install the matching flatbuffer version which tensorflow requires.
Please add to makefile_build documentation
For example for tensorflow 2.16.1 you require to build and install libflatbuffers 23.5.26:
git clone https://github.com/google/flatbuffers.git
cd flatbuffers/
git checkout v23.5.26
mkdir build && cd build
cmake .. -DFLATBUFFERS_BUILD_SHAREDLIB=ON -DFLATBUFFERS_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr
make
sudo make install

Please add to main documentation
And in order to make debian packages using the Makefile-only approach you need to build both versions of the library first (std, max) before the packaging routine can successfully been run:
TFROOT=<path to tensorflow source>/ LD_LIBRARY_PATH=/usr/local/lib/ make -f makefile_build/Makefile  libedgetpu-throttled
TFROOT=<path to tensorflow source>/ LD_LIBRARY_PATH=/usr/local/lib/ make -f makefile_build/Makefile  libedgetpu-throttled
debuild -us -uc -tc -b -a amd64 -d

@jagiella I am not sure why you are tagging me on this. I am not a Google developer and have no access to the main repo. Feel free to create an actual patch, make a PR against the main repo and hope that it gets merged.
I maintain a separate fork that is updated:
https://github.com/feranick/libedgetpu/
with a PR to the main repo:
https://github.com/google-coral/libedgetpu/pull/70

authors: FeeJai feranick jagiella mwprado

repo: google-coral/libedgetpu

license_source: bigquery


text:
Text composables: Android Basics with Compose

URL of codelab
**Step 3 **
**The compose function that returns nothing and bears the @Composable annotation MUST be named using pascal case. Pascal case refers to a naming convention in which the first letter of each word in a compound word is capitalized. The difference between pascal case and camel case is that all words in pascal case are capitalized. In camel case, the first word is not capitalized.
I have given camel case but still worked.  As per documentation composable function MUST be names as pascal.
**
Steps to reproduce?

Go to...
Click on...
See error...

Versions
Android Studio version: Android Studio Flamingo
_API version of the emulator: _
Additional information
_
_

Hi @srikanth104
Thank you for writing in with your question.
These instructions are referring to the API Guidelines for Jetpack Compose which are the guidelines to follow for best practices.
The "MUST" in this context refers to what the guidelines require in order to adhere to them. It is a coding convention and not a requirement of Jetpack Compose nor Android Studio.
Android Studio does not enforce this as you have seen.
Hope you are enjoying the course!
Regards.


text:
Create an interactive Dice Roller app: Android Basics with Compose

URL of codelab
Create an interactive Dice Roller app
In which task and step of the codelab can this issue be found?
In step 4 of the codelab, section: Add a modifier
Describe the problem
The code sample incorrectly contains the @Preview annotation.
@Preview(showBackground = true)
@Composable
fun DiceWithButtonAndImage(modifier: Modifier = Modifier) {
}

The annotation actually belongs to the DiceRollerApp function as shown in the previous section:
@Preview
@Composable
fun DiceRollerApp() {

}

@Composable
fun DiceWithButtonAndImage() {

}

Steps to reproduce?

Go to...
Click on...
See error...

Versions
Android Studio version:
API version of the emulator:
Additional information
The issue is not present in the actual solution code:
https://github.com/google-developer-training/basic-android-kotlin-compose-training-dice-roller/blob/cce49feeab07410f33b535c9eac9699768f6f8c9/app/src/main/java/com/example/diceroller/MainActivity.kt#L54-L64

Hello @kazi-kavish,
Thank you for reaching out to us and reporting this issue. The code snippet is fixed:

authors: kazi-kavish osuleymanova