text:
runtime/signal_amd64x.go:76 SIGINT on darwin/amd64 causes segfault under c-shared library

A refinement of the report in https://github.com/golang/go/issues/13028 to focus on the OSX-only segfault that happens when I send SIGINT to program that has a c-shared library loaded.
Details:
jaten@Jasons-MacBook-Pro:~$ go version  # at tip. Also happens in go1.5.1.
go version devel +79a3b56 Thu Oct 22 21:19:43 2015 +0000 darwin/amd64
On OSX Yosemite, 10.10.5, darwin amd64.
I built a c-shared library (source code: https://github.com/glycerine/rmq), loaded it under R (a large C program), then called from C into the c-shared go library. Inside the library we start a web-server and the do ListenAndServe(). I have established signal handlers in the go code for SIGINT. However pressing ctrl-c and sending SIGINT to the process results in a segfault on OSX, but not on linux.
Under gdb on osx, I observe that the segfault is isolated to go1.6-dev/src/runtime/signal_amd64x.go:76, the value of c is out of bounds in the line if c.sigcode() != _SI_USER && flags&_SigPanic != 0 {.
context of the segfault, at signal_amd64x.go:76
...
// May run during STW, so write barriers are not allowed.
//go:nowritebarrier
func sighandler(sig uint32, info *siginfo, ctxt unsafe.Pointer, gp *g) {
    _g_ := getg()
    c := &sigctxt{info, ctxt}

    if sig == _SIGPROF {
        sigprof(uintptr(c.rip()), uintptr(c.rsp()), 0, gp, _g_.m)
        return
    }

    if GOOS == "darwin" {
        // x86-64 has 48-bit virtual addresses. The top 16 bits must echo bit 47.
        // The hardware delivers a different kind of fault for a malformed address
        // than it does for an attempt to access a valid but unmapped address.
        // OS X 10.9.2 mishandles the malformed address case, making it look like
        // a user-generated signal (like someone ran kill -SEGV ourpid).
        // We pass user-generated signals to os/signal, or else ignore them.
        // Doing that here - and returning to the faulting code - results in an
        // infinite loop. It appears the best we can do is rewrite what the kernel
        // delivers into something more like the truth. The address used below
        // has very little chance of being the one that caused the fault, but it is
        // malformed, it is clearly not a real pointer, and if it does get printed
        // in real life, people will probably search for it and find this code.
        // There are no Google hits for b01dfacedebac1e or 0xb01dfacedebac1e
        // as I type this comment.
        if sig == _SIGSEGV && c.sigcode() == _SI_USER {
            c.set_sigcode(_SI_USER + 1)
            c.set_sigaddr(0xb01dfacedebac1e)
        }
    }

    flags := int32(_SigThrow)
    if sig < uint32(len(sigtable)) {
        flags = sigtable[sig].flags
    }
=>  if c.sigcode() != _SI_USER && flags&_SigPanic != 0 {   <<<<< segfaulting here
        // Make it look like a call to the signal func.
        // Have to pass arguments out of band since
        // augmenting the stack frame would break
        // the unwinding code.
        gp.sig = sig
        gp.sigcode0 = uintptr(c.sigcode())
 ...

(gdb) handle SIGINT pass
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) y
Signal        Stop  Print   Pass to program Description
SIGINT        Yes   Yes Yes     Interrupt
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00000001053b89a9 in runtime.sighandler (sig=537083904, info=0x14003b, ctxt=0x38000051901,
    gp=0xffffffff0538a380) at /Users/jaten/pkg/go1.6-dev/src/runtime/signal_amd64x.go:76
(gdb) p c
$1 = (runtime.sigctxt *) 0x53be72f
(gdb) p flags
$2 = 4
(gdb) p _SigPanic
No symbol "_SigPanic" in current context.
(gdb) p _SI_USER
No symbol "_SI_USER" in current context.
(gdb) p *c
Cannot access memory at address 0x53be72f
(gdb)

a backtrace from gdb:
(gdb) bt
#0  0x00000001053b89a9 in runtime.sighandler (sig=537083904, info=0x14003b, ctxt=0x38000051901,
    gp=0xffffffff0538a380) at /Users/jaten/pkg/go1.6-dev/src/runtime/signal_amd64x.go:76
#1  <signal handler called>
#2  0x00000001053d630c in runtime.mach_semaphore_wait ()
    at /Users/jaten/pkg/go1.6-dev/src/runtime/sys_darwin_amd64.s:407
#3  0x00000001053a2a15 in runtime.semasleep1 (ns=-1, ~r1=0)
    at /Users/jaten/pkg/go1.6-dev/src/runtime/os1_darwin.go:385
#4  0x00000001053c9f89 in runtime.semasleep.func1 ()
    at /Users/jaten/pkg/go1.6-dev/src/runtime/os1_darwin.go:401
#5  0x00000001053d2b4b in runtime.systemstack ()
    at /Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:278
#6  0x00000001053a2a96 in runtime.semasleep (ns=-1, ~r1=0)
    at /Users/jaten/pkg/go1.6-dev/src/runtime/os1_darwin.go:402
#7  0x00000001053893b0 in runtime.notesleep (n=0xc820034110)
    at /Users/jaten/pkg/go1.6-dev/src/runtime/lock_sema.go:169
#8  0x00000001053ab37e in runtime.stoplockedm () at /Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:1586
#9  0x00000001053ac38e in runtime.schedule () at /Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:1908
#10 0x00000001053ac80b in runtime.park_m (gp=0xc820000600)
    at /Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:2016
#11 0x00000001053d2a7b in runtime.mcall () at /Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:204
#12 0x00007fff5fbfd2b8 in ?? ()
#13 0x00000001053d469a in runtime.cgocallback ()
    at /Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:712
#14 0x0000000105373f28 in _cgoexp_7e71d05b448d_ListenAndServe (a=0x7fff5fbfd378, n=32)
    at rmq/_obj/_cgo_gotypes.go:490
#15 0x00000001054fa43d in crosscall2 () at /Users/jaten/pkg/go1.6-dev/src/runtime/cgo/asm_amd64.s:36
#16 0x00007fff5fbfd378 in ?? ()
#17 0x0000000000000020 in ?? ()
#18 0x0000000103c9bff8 in ?? ()
#19 0x00007fff5fbfd3b0 in ?? ()
#20 0x0000000104d2ad80 in ?? ()
#21 0x0000000104d2abc0 in ?? ()
#22 0x0000000104d2b6d0 in ?? ()
#23 0x0000000104d241c8 in ?? ()
#24 0x0000000104d2b6d0 in ?? ()
#25 0x0000000104d241c8 in ?? ()
#26 0x00007fff5fbfd3b0 in ?? ()
#27 0x000000010537087e in ListenAndServe (p0=0x2203, p1=0x8, p2=0x9)
    at /var/folders/2x/hm9gp5ys3k9gmm5f_vzm_6wc0000gn/T/go-build514057958/rmq/_obj/_cgo_export.c:21
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

The signal handling Go code for SIGINT consisted of either one or two prior calls to signal.Notify().
A full stack dump from pressing ctrl-c generates a seg-fault with this stack trace not under gdb:

R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin14.5.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> options(STERM='iESS', editor='emacsclient')
> require(rmq)
Loading required package: rmq
> require(testthat)
Loading required package: testthat
> handler = function(x) {
+   print(paste("handler called back with argument x = ", paste(collapse=" ",sep=" ",x)))
+   reply=list()
+   reply$hi = "there!"
+   reply$yum = c(1.1, 2.3)
+   reply$input = x
+   reply
+ }
> options(error=recover)
> r = .Call("ListenAndServe", "127.0.0.1:9090", handler, new.env(), package="rmq")
ListenAndServe listening on address '127.0.0.1:9090'...
  C-c C-cfatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x140043 pc=0x10bb1d9a9]

runtime stack:
runtime.throw(0x10c206480, 0x2a)
	/Users/jaten/pkg/go1.6-dev/src/runtime/panic.go:527 +0x90
runtime.sigpanic()
	/Users/jaten/pkg/go1.6-dev/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.sighandler(0xc820032000, 0x14003b, 0xd800051001, 0xffffffff0bb24260)
	/Users/jaten/pkg/go1.6-dev/src/runtime/signal_amd64x.go:76 +0x139

goroutine 17 [select, locked to thread]:
runtime.gopark(0x10c24d868, 0xc82005de00, 0x10c152430, 0x6, 0xc82017a218, 0x2)
	/Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:202 +0x163 fp=0xc82005dad0 sp=0xc82005daa8
runtime.selectgoImpl(0xc82005de00, 0x0, 0x18)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:392 +0xa64 fp=0xc82005dc78 sp=0xc82005dad0
runtime.selectgo(0xc82005de00)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:212 +0x12 fp=0xc82005dc98 sp=0xc82005dc78
main.ListenAndServe(0x7ffb64cba578, 0x7ffb648dc548, 0x7ffb648e5120, 0x7ffb648e4840)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:178 +0x866 fp=0xc82005ded0 sp=0xc82005dc98
runtime.call32(0x0, 0x7fff5687b3d8, 0x7fff5687b468, 0x20)
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:442 +0x3e fp=0xc82005def8 sp=0xc82005ded0
runtime.cgocallbackg1()
	/Users/jaten/pkg/go1.6-dev/src/runtime/cgocall.go:264 +0x10c fp=0xc82005df30 sp=0xc82005def8
runtime.cgocallbackg()
	/Users/jaten/pkg/go1.6-dev/src/runtime/cgocall.go:177 +0xd7 fp=0xc82005df90 sp=0xc82005df30
runtime.cgocallback_gofunc(0x0, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:799 +0x60 fp=0xc82005dfa0 sp=0xc82005df90
runtime.goexit()
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:1805 +0x1 fp=0xc82005dfa8 sp=0xc82005dfa0

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
	/Users/jaten/pkg/go1.6-dev/src/runtime/sigqueue.go:112 +0x132
os/signal.loop()
	/Users/jaten/pkg/go1.6-dev/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
	/Users/jaten/pkg/go1.6-dev/src/os/signal/signal_unix.go:28 +0x37

goroutine 18 [select, locked to thread]:
runtime.gopark(0x10c24d868, 0xc82002ef28, 0x10c152430, 0x6, 0x10bb0c118, 0x2)
	/Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:202 +0x163
runtime.selectgoImpl(0xc82002ef28, 0x0, 0x18)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82002ef28)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
	/Users/jaten/pkg/go1.6-dev/src/runtime/signal1_unix.go:227 +0x324
runtime.goexit()
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:1805 +0x1

goroutine 19 [chan receive]:
main.init.1.func1(0xc820144180)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:43 +0x44
created by main.init.1
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:48 +0xd3

goroutine 34 [syscall, locked to thread]:
runtime.goexit()
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:1805 +0x1

goroutine 35 [IO wait]:
net.runtime_pollWait(0x10ce01118, 0x72, 0xc820014210)
	/Users/jaten/pkg/go1.6-dev/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820147a30, 0x72, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820147a30, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8201479d0, 0x0, 0x10cab1888, 0xc8200bb1e0)
	/Users/jaten/pkg/go1.6-dev/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc82013a028, 0xc820192d78, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82013a028, 0x0, 0x0, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc820182000, 0x10cd7c070, 0xc82013a028, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/http/server.go:1907 +0x118
vendor/github.com/glycerine/go-tigertonic.(*Server).Serve(0xc820182000, 0x10cd7c070, 0xc82013a028, 0x0, 0x0)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/go-tigertonic/server.go:160 +0xe1
vendor/github.com/glycerine/go-tigertonic.(*Server).ListenAndServe(0xc820182000, 0x0, 0x0)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/go-tigertonic/server.go:144 +0x1e2
main.(*WebServer).Start.func1(0xc820170080)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/web.go:54 +0x25
created by main.(*WebServer).Start
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/web.go:60 +0x4e

Process R exited abnormally with code 2 at Fri Oct 23 01:03:07 2015

R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin14.5.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> options(STERM='iESS', editor='emacsclient')
> require(rmq)
Loading required package: rmq
> require(testthat)
Loading required package: testthat
> handler = function(x) {
+   print(paste("handler called back with argument x = ", paste(collapse=" ",sep=" ",x)))
+   reply=list()
+   reply$hi = "there!"
+   reply$yum = c(1.1, 2.3)
+   reply$input = x
+   reply
+ }
> options(error=recover)
> r = .Call("ListenAndServe", "127.0.0.1:9090", handler, new.env(), package="rmq")
ListenAndServe listening on address '127.0.0.1:9090'...
require(rmq)
  C-c C-cfatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x140043 pc=0x1108849a9]

runtime stack:
runtime.throw(0x110f6d480, 0x2a)
	/Users/jaten/pkg/go1.6-dev/src/runtime/panic.go:527 +0x90
runtime.sigpanic()
	/Users/jaten/pkg/go1.6-dev/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.sighandler(0xc820032000, 0x14003b, 0x23000025101, 0xffffffff00000000)
	/Users/jaten/pkg/go1.6-dev/src/runtime/signal_amd64x.go:76 +0x139

goroutine 17 [select, locked to thread]:
runtime.gopark(0x110fb4868, 0xc820061e00, 0x110eb9430, 0x6, 0xc82001e518, 0x2)
	/Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:202 +0x163 fp=0xc820061ad0 sp=0xc820061aa8
runtime.selectgoImpl(0xc820061e00, 0x0, 0x18)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:392 +0xa64 fp=0xc820061c78 sp=0xc820061ad0
runtime.selectgo(0xc820061e00)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:212 +0x12 fp=0xc820061c98 sp=0xc820061c78
main.ListenAndServe(0x7f9f3ab02578, 0x7f9f39f44748, 0x7f9f39f4a920, 0x7f9f39f4a040)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:178 +0x866 fp=0xc820061ed0 sp=0xc820061c98
runtime.call32(0x0, 0x7fff51b0f3d8, 0x7fff51b0f468, 0x20)
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:442 +0x3e fp=0xc820061ef8 sp=0xc820061ed0
runtime.cgocallbackg1()
	/Users/jaten/pkg/go1.6-dev/src/runtime/cgocall.go:264 +0x10c fp=0xc820061f30 sp=0xc820061ef8
runtime.cgocallbackg()
	/Users/jaten/pkg/go1.6-dev/src/runtime/cgocall.go:177 +0xd7 fp=0xc820061f90 sp=0xc820061f30
runtime.cgocallback_gofunc(0x0, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:799 +0x60 fp=0xc820061fa0 sp=0xc820061f90
runtime.goexit()
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:1805 +0x1 fp=0xc820061fa8 sp=0xc820061fa0

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
	/Users/jaten/pkg/go1.6-dev/src/runtime/sigqueue.go:112 +0x132
os/signal.loop()
	/Users/jaten/pkg/go1.6-dev/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
	/Users/jaten/pkg/go1.6-dev/src/os/signal/signal_unix.go:28 +0x37

goroutine 18 [select, locked to thread]:
runtime.gopark(0x110fb4868, 0xc82002ef28, 0x110eb9430, 0x6, 0x110873118, 0x2)
	/Users/jaten/pkg/go1.6-dev/src/runtime/proc.go:202 +0x163
runtime.selectgoImpl(0xc82002ef28, 0x0, 0x18)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82002ef28)
	/Users/jaten/pkg/go1.6-dev/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
	/Users/jaten/pkg/go1.6-dev/src/runtime/signal1_unix.go:227 +0x324
runtime.goexit()
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:1805 +0x1

goroutine 19 [chan receive]:
main.init.1.func1(0xc820140180)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:43 +0x44
created by main.init.1
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:48 +0xd3

goroutine 34 [syscall, locked to thread]:
runtime.goexit()
	/Users/jaten/pkg/go1.6-dev/src/runtime/asm_amd64.s:1805 +0x1

goroutine 8 [IO wait]:
net.runtime_pollWait(0x111819850, 0x72, 0xc820014210)
	/Users/jaten/pkg/go1.6-dev/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820064220, 0x72, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820064220, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc8200641c0, 0x0, 0x111a60100, 0xc8200bb200)
	/Users/jaten/pkg/go1.6-dev/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820030098, 0xc820040d78, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820030098, 0x0, 0x0, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc82001a0c0, 0x1118199d0, 0xc820030098, 0x0, 0x0)
	/Users/jaten/pkg/go1.6-dev/src/net/http/server.go:1907 +0x118
vendor/github.com/glycerine/go-tigertonic.(*Server).Serve(0xc82001a0c0, 0x1118199d0, 0xc820030098, 0x0, 0x0)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/go-tigertonic/server.go:160 +0xe1
vendor/github.com/glycerine/go-tigertonic.(*Server).ListenAndServe(0xc82001a0c0, 0x0, 0x0)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/go-tigertonic/server.go:144 +0x1e2
main.(*WebServer).Start.func1(0xc820017800)
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/web.go:54 +0x25
created by main.(*WebServer).Start
	/Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/web.go:60 +0x4e

Process R exited abnormally with code 2 at Fri Oct 23 01:06:56 2015

another, earlier, example of the same thing:
$ R
R version 3.2.2 (2015-08-14) -- "Fire Safety"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin14.5.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> options(STERM='iESS', editor='emacsclient')
> require(rmq)
Loading required package: rmq
> require(testthat)
Loading required package: testthat
> handler = function(x) {
+   print(paste("handler called back with argument x = ", paste(collapse=" ",sep=" ",x)))
+   reply=list()
+   reply$hi = "there!"
+   reply$yum = c(1.1, 2.3)
+   reply$input = x
+   reply
+ }
> options(error=recover)
> r = .Call("ListenAndServe", "127.0.0.1:9090", handler, new.env(), package="rmq")
ListenAndServe listening on address '127.0.0.1:9090'...
  C-c C-cfatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x8 pc=0x10dd976d9]

runtime stack:
runtime.throw(0x10e425c20, 0x2a)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
runtime.sighandler(0xc820032000, 0x0, 0x0, 0x3)
    /usr/local/go/src/runtime/signal_amd64x.go:76 +0x139

goroutine 17 [select, locked to thread]:
runtime.gopark(0x10e468a70, 0xc820061e00, 0x10e37e638, 0x6, 0xc82001e718, 0x2)
    /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc820061ad0 sp=0xc820061aa8
runtime.selectgoImpl(0xc820061e00, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa64 fp=0xc820061c78 sp=0xc820061ad0
runtime.selectgo(0xc820061e00)
    /usr/local/go/src/runtime/select.go:212 +0x12 fp=0xc820061c98 sp=0xc820061c78
main.ListenAndServe(0x7ffd733c4178, 0x7ffd730ddd48, 0x7ffd730e3920, 0x7ffd730e3040)
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/rmq.go:161 +0x865 fp=0xc820061ed0 sp=0xc820061c98
runtime.call32(0x0, 0x7fff545f43d8, 0x7fff545f4468, 0x20)
    /usr/local/go/src/runtime/asm_amd64.s:437 +0x3e fp=0xc820061ef8 sp=0xc820061ed0
runtime.cgocallbackg1()
    /usr/local/go/src/runtime/cgocall.go:252 +0x10c fp=0xc820061f30 sp=0xc820061ef8
runtime.cgocallbackg()
    /usr/local/go/src/runtime/cgocall.go:177 +0xd7 fp=0xc820061f90 sp=0xc820061f30
runtime.cgocallback_gofunc(0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/asm_amd64.s:801 +0x60 fp=0xc820061fa0 sp=0xc820061f90
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820061fa8 sp=0xc820061fa0

goroutine 8 [select, locked to thread]:
runtime.gopark(0x10e468a70, 0xc82002f728, 0x10e37e638, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82002f728, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82002f728)
    /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:227 +0x323
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 6 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 9 [IO wait]:
net.runtime_pollWait(0x10edd6950, 0x72, 0xc820014210)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82011c060, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011c060, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82011c000, 0x0, 0x10f061038, 0xc8201200e0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820122000, 0xc82003cd88, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820122000, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
net/http.(*Server).Serve(0xc82008f040, 0x10f061000, 0xc820122000, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
vendor/github.com/glycerine/go-tigertonic.(*Server).Serve(0xc82008f040, 0x10f061000, 0xc820122000, 0x0, 0x0)
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/go-tigertonic/server.go:160 +0xd8
vendor/github.com/glycerine/go-tigertonic.(*Server).ListenAndServe(0xc82008f040, 0x0, 0x0)
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/vendor/github.com/glycerine/go-tigertonic/server.go:144 +0x1e2
main.(*WebServer).Start.func1(0xc820017d80)
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/web.go:54 +0x25
created by main.(*WebServer).Start
    /Users/jaten/pkg/R-3.2.2/src/library/Recommended/rmq/src/rmq/web.go:60 +0x4e

goroutine 18 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

Process R exited abnormally with code 2 at Thu Oct 22 13:47:58 2015
$

I'm getting sporadic crashes (on linux, fedora22 amd64) on my host R process shutdown. It appears that SIGCHLD is occassionally intercepted by the cshared-library-go runtime.  Not sure if this should be addressed along with this ticket or distinct issue. If the fix for this ticket involves allowing the host to avoid guest doing any signal handling at all then the SIGCHLD intercept below would be fixed automatically.
Stack trace from the SIGCHLD interception, is here:
(gdb) bt
#0  runtime.sigaltstack () at /home/jaten/pkg/go1.5.1/src/runtime/sys_linux_amd64.s:355
#1  0x00007ffff14dfc00 in runtime.signalstack (s=0xc820000300)
    at /home/jaten/pkg/go1.5.1/src/runtime/os1_linux.go:337
#2  0x00007ffff14df5d5 in runtime.minit () at /home/jaten/pkg/go1.5.1/src/runtime/os1_linux.go:216
#3  0x00007ffff14e7768 in runtime.needm (x=0 '\000')
    at /home/jaten/pkg/go1.5.1/src/runtime/proc1.go:968
#4  0x00007ffff1513696 in runtime.cgocallback_gofunc ()
    at /home/jaten/pkg/go1.5.1/src/runtime/asm_amd64.s:742
#5  0x00007ffff15135ca in runtime.cgocallback () at /home/jaten/pkg/go1.5.1/src/runtime/asm_amd64.s:714
#6  0x00007ffff14f719b in runtime.badsignal (sig=17)
    at /home/jaten/pkg/go1.5.1/src/runtime/sigqueue.go:169
#7  0x00007ffff14f6212 in runtime.sigtrampgo (sig=17, info=0x672f70, ctx=0x672e40)
    at /home/jaten/pkg/go1.5.1/src/runtime/signal_linux.go:90
#8  0x00007ffff151537b in runtime.sigtramp ()
    at /home/jaten/pkg/go1.5.1/src/runtime/sys_linux_amd64.s:234
#9  0x00007ffff1515380 in runtime.sigtramp ()
    at /home/jaten/pkg/go1.5.1/src/runtime/sys_linux_amd64.s:235
#10 0x0000000000000001 in ?? ()
#11 0x0000000000000000 in ?? ()
(gdb)

repo: golang/go


text:
crypto/tls: TLS handshake swallows errors

What version of Go are you using (go version)?
go version go1.6 darwin/amd64

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"

What did you do?
Runnable example here: https://github.com/tamird/go-conn-repro

What did you expect to see?
I expect that requests will either succeed or fail.

What did you see instead?
The test times out because a connection has hung in TLS handshake (both sides are trying to read). Stack trace:

=== RUN   TestFoo
panic: test timed out after 5s

goroutine 453 [running]:
panic(0x2e2860, 0xc820017920)
	/Users/tamird/src/go1.6/src/runtime/panic.go:464 +0x3e6
testing.startAlarm.func1()
	/Users/tamird/src/go1.6/src/testing/testing.go:725 +0x14b
created by time.goFunc
	/Users/tamird/src/go1.6/src/time/sleep.go:129 +0x3a

goroutine 1 [chan receive]:
testing.RunTests(0x492080, 0x5c78d0, 0x1, 0x1, 0x373001)
	/Users/tamird/src/go1.6/src/testing/testing.go:583 +0x8d2
testing.(*M).Run(0xc820191f08, 0x492388)
	/Users/tamird/src/go1.6/src/testing/testing.go:515 +0x81
main.main()
	_/Users/tamird/Desktop/go-conn-repro/_test/_testmain.go:54 +0x117

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/Users/tamird/src/go1.6/src/runtime/asm_amd64.s:1998 +0x1

goroutine 6 [select]:
net/http.(*Transport).getConn(0xc8200fc0c0, 0xc8201d0e00, 0x0, 0xc82029aae0, 0x5, 0xc82029aae8, 0xf, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/transport.go:707 +0x4ef
net/http.(*Transport).RoundTrip(0xc8200fc0c0, 0xc8201d0e00, 0xc8200fc0c0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/transport.go:307 +0x7e9
net/http.send(0xc8201d0e00, 0x12047f8, 0xc8200fc0c0, 0x0, 0x0, 0x0, 0xc82029ab00, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc82040bf20, 0xc8201d0e00, 0x0, 0x0, 0x0, 0xd8, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc82040bf20, 0xc8201d0e00, 0x492408, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Get(0xc82040bf20, 0xc82029aae0, 0x17, 0xc82014cde0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/client.go:422 +0xaa
_/Users/tamird/Desktop/go-conn-repro.TestFoo(0xc820080090)
	/Users/tamird/Desktop/go-conn-repro/main_test.go:92 +0x611
testing.tRunner(0xc820080090, 0x5c78d0)
	/Users/tamird/src/go1.6/src/testing/testing.go:473 +0x98
created by testing.RunTests
	/Users/tamird/src/go1.6/src/testing/testing.go:582 +0x892

goroutine 7 [IO wait]:
net.runtime_pollWait(0x12059a8, 0x72, 0x1200050)
	/Users/tamird/src/go1.6/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820131fe0, 0x72, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820131fe0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820131f80, 0x0, 0x1240000, 0xc82029aba0)
	/Users/tamird/src/go1.6/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc82002a068, 0xc8203c6020, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82002a068, 0x0, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/tcpsock_posix.go:264 +0x3d
crypto/tls.(*listener).Accept(0xc82013f880, 0x0, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/tls.go:52 +0x60
net/http.(*Server).Serve(0xc82012ee80, 0x1205a90, 0xc82013f880, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/server.go:2117 +0x129
created by _/Users/tamird/Desktop/go-conn-repro.TestFoo
	/Users/tamird/Desktop/go-conn-repro/main_test.go:80 +0x465

goroutine 452 [IO wait]:
net.runtime_pollWait(0x1205768, 0x72, 0xc82022a000)
	/Users/tamird/src/go1.6/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f0c30, 0x72, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f0c30, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201f0bd0, 0xc82022a000, 0x400, 0x400, 0x0, 0x1200050, 0xc820016180)
	/Users/tamird/src/go1.6/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82035a020, 0xc82022a000, 0x400, 0x400, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820359ec0, 0x1240078, 0xc82035a020, 0x5, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc82035e900, 0x1208116, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).readHandshake(0xc82035e900, 0x0, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:800 +0xd3
crypto/tls.(*Conn).clientHandshake(0xc82035e900, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/handshake_client.go:149 +0xfcb
crypto/tls.(*Conn).Handshake(0xc82035e900, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:1033 +0xec
net/http.(*Transport).dialConn.func3(0xc82035e900, 0x0, 0xc82030c840)
	/Users/tamird/src/go1.6/src/net/http/transport.go:824 +0x21
created by net/http.(*Transport).dialConn
	/Users/tamird/src/go1.6/src/net/http/transport.go:829 +0x986

goroutine 413 [chan receive]:
net/http.(*Transport).dialConn(0xc8200fc0c0, 0x0, 0xc82029aae0, 0x5, 0xc82029aae8, 0xf, 0xc820300360, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/http/transport.go:830 +0x9c2
net/http.(*Transport).getConn.func4(0xc8200fc0c0, 0x0, 0xc82029aae0, 0x5, 0xc82029aae8, 0xf, 0xc8202a6480)
	/Users/tamird/src/go1.6/src/net/http/transport.go:702 +0x66
created by net/http.(*Transport).getConn
	/Users/tamird/src/go1.6/src/net/http/transport.go:704 +0x262

goroutine 414 [IO wait]:
net.runtime_pollWait(0x1205828, 0x72, 0xc820202400)
	/Users/tamird/src/go1.6/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8201f0ca0, 0x72, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201f0ca0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201f0c40, 0xc820202400, 0x400, 0x400, 0x0, 0x1200050, 0xc820016180)
	/Users/tamird/src/go1.6/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8203c6020, 0xc820202400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820178480, 0x1240078, 0xc8203c6020, 0x5, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc820330f00, 0xe7c16, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).readHandshake(0xc820330f00, 0x0, 0x0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:800 +0xd3
crypto/tls.(*serverHandshakeState).readClientHello(0xc820285ab8, 0xc820195aa0, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/handshake_server.go:107 +0x5b
crypto/tls.(*Conn).serverHandshake(0xc820330f00, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/handshake_server.go:48 +0xac
crypto/tls.(*Conn).Handshake(0xc820330f00, 0x0, 0x0)
	/Users/tamird/src/go1.6/src/crypto/tls/conn.go:1035 +0x169
net/http.(*conn).serve(0xc8202a4480)
	/Users/tamird/src/go1.6/src/net/http/server.go:1405 +0x382
created by net/http.(*Server).Serve
	/Users/tamird/src/go1.6/src/net/http/server.go:2137 +0x44e

ERROR: exit status 2

This appears to be caused by errors not being propagated from the TLS handshake code.

Fixed by https://golang.org/cl/19990. The rest of the issue is tracked in #14548.

goroutine 26204301 [IO wait]:
net.runtime_pollWait(0x7f8ebcc030e0, 0x77, 0x45a330)
	/opt/work/golang/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc83fc2edf0, 0x77, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc83fc2edf0, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc83fc2ed90, 0x0, 0x0, 0x7f8ec1895c90, 0xc83fb9c2e0, 0xece8190e1, 0x3b3f615e, 0xf82ba0, 0x0, 0x0, ...)
	/opt/work/golang/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc83fc2ed90, 0x7f8ec1895c48, 0x0, 0x7f8ec1895c48, 0xc82037c780, 0xece8190e1, 0xc83b3f615e, 0xf82ba0, 0x0, 0x0, ...)
	/opt/work/golang/go/src/net/sock_posix.go:137 +0x364
net.socket(0xb23770, 0x3, 0x2, 0x1, 0x0, 0xc82037c700, 0x7f8ec1895c48, 0x0, 0x7f8ec1895c48, 0xc82037c780, ...)
	/opt/work/golang/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xb23770, 0x3, 0x7f8ec1895c48, 0x0, 0x7f8ec1895c48, 0xc82037c780, 0xece8190e1, 0xc83b3f615e, 0xf82ba0, 0x1, ...)
	/opt/work/golang/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xb23770, 0x3, 0x0, 0xc82037c780, 0xece8190e1, 0x3b3f615e, 0xf82ba0, 0x0, 0x1, 0x0, ...)
	/opt/work/golang/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc820808f30, 0x7f8ec1895bb8, 0xc82037c780, 0xece8190e1, 0x3b3f615e, 0xf82ba0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0xece8190e1, 0x3b3f615e, 0xf82ba0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/dial.go:343 +0x75
net.dial(0xb23770, 0x3, 0x7f8ec1895bb8, 0xc82037c780, 0xc8219b1538, 0xece8190e1, 0x3b3f615e, 0xf82ba0, 0x0, 0x0, ...)
	/opt/work/golang/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc820808f30, 0xc82074ccc0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/dial.go:345 +0x751
net.(*Dialer).Dial(0xc8219b17b0, 0xb23770, 0x3, 0xc83f00c140, 0x19, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/dial.go:239 +0x512
net.DialTimeout(0xb23770, 0x3, 0xc83f00c140, 0x19, 0x4a817c800, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/dial.go:200 +0xa3
webcrawler/httplib.TimeoutDialer.func1(0xb23770, 0x3, 0xc83f00c140, 0x19, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:552 +0x8d
net/http.(*Transport).dial(0xc82184af00, 0xb23770, 0x3, 0xc83f00c140, 0x19, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:655 +0x96
net/http.(*Transport).dialConn(0xc82184af00, 0x0, 0xc83f00c0c0, 0x5, 0xc83f00c140, 0x19, 0xc8224f3a40, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:758 +0x1e3d
net/http.(*Transport).getConn.func4(0xc82184af00, 0x0, 0xc83f00c0c0, 0x5, 0xc83f00c140, 0x19, 0xc83fbab9e0)
	/opt/work/golang/go/src/net/http/transport.go:702 +0x66
created by net/http.(*Transport).getConn
	/opt/work/golang/go/src/net/http/transport.go:704 +0x262

goroutine 26228283 [IO wait]:
net.runtime_pollWait(0x7f8ec17e2098, 0x72, 0xc82e9e9000)
	/opt/work/golang/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc821476530, 0x72, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc821476530, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8214764d0, 0xc82e9e9000, 0x1000, 0x1000, 0x0, 0x7f8ec1891050, 0xc82000a240)
	/opt/work/golang/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc83d86af90, 0xc82e9e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f8ec1860448, 0xc83d86af90, 0xc83f64c548, 0xc82e9e9000, 0x1000, 0x1000, 0x60, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc8229a9ee0, 0xc82e9e9000, 0x1000, 0x1000, 0xb19df0, 0x0, 0x0)
	<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82ede86c0)
	/opt/work/golang/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82ede86c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc83f64c4e0)
	/opt/work/golang/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
	/opt/work/golang/go/src/net/http/transport.go:853 +0x10a6

goroutine 26227170 [select]:
net/http.(*persistConn).roundTrip(0xc8308be8f0, 0xc83342b9e0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:1469 +0xf1f
net/http.(*Transport).RoundTrip(0xc82fd423c0, 0xc8307f9180, 0xc82fd423c0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:320 +0x9bb
net/http.send(0xc8307f9180, 0x7f8ec1895550, 0xc82fd423c0, 0x0, 0x0, 0x0, 0xc8334314a0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc83f5f1ad0, 0xc8307f9180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc83f5f1ad0, 0xc82fd4a0e0, 0xc4fd10, 0xc833402870, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0xc83f5f1ad0, 0xc82fd4a0e0, 0xa, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:188 +0xff
webcrawler/httplib.(*BeegoHTTPRequest).DoRequest(0xc82fd42300, 0xc82fd21e00, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:463 +0x800
webcrawler/httplib.(*BeegoHTTPRequest).getResponse(0xc82fd42300, 0xa754c0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:380 +0x5f
webcrawler/httplib.(*BeegoHTTPRequest).Response(0xc82fd42300, 0xc82fd313c0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:546 +0x2d
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc82fd3e960, 0xf, 0xb200d8, 0x4, 0x1, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:179 +0x7de
main.(*WebCrawler).concurrentCheck(0xc820c88840, 0xc82fd2f160, 0xb, 0x20fe80)
	/opt/work/golang/project/src/webcrawler/main.go:381 +0x169
created by main.(*WebCrawler).loadRange
	/opt/work/golang/project/src/webcrawler/main.go:417 +0x544

goroutine 26230881 [IO wait]:
net.runtime_pollWait(0x7f8ebcbfd860, 0x72, 0xc830f78000)
	/opt/work/golang/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc83439a290, 0x72, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc83439a290, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc83439a230, 0xc830f78000, 0x1000, 0x1000, 0x0, 0x7f8ec1891050, 0xc82000a240)
	/opt/work/golang/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc833c4bd98, 0xc830f78000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f8ec1860448, 0xc833c4bd98, 0xc8218a7c08, 0xc830f78000, 0x1000, 0x1000, 0x60, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc830f4ca20, 0xc830f78000, 0x1000, 0x1000, 0xb19df0, 0x0, 0x0)
	<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc830f47c20)
	/opt/work/golang/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc830f47c20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8218a7ba0)
	/opt/work/golang/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
	/opt/work/golang/go/src/net/http/transport.go:853 +0x10a6

goroutine 26230788 [IO wait]:
net.runtime_pollWait(0x7f8ec17df898, 0x72, 0xc830b04000)
	/opt/work/golang/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc834eefb10, 0x72, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc834eefb10, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc834eefab0, 0xc830b04000, 0x1000, 0x1000, 0x0, 0x7f8ec1891050, 0xc82000a240)
	/opt/work/golang/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc833c4b5c8, 0xc830b04000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f8ec1860448, 0xc833c4b5c8, 0xc83487a208, 0xc830b04000, 0x1000, 0x1000, 0x60, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:1683 +0x67
net/http.(*noteEOFReader).Read(0xc834ffe9a0, 0xc830b04000, 0x1000, 0x1000, 0xb19df0, 0x0, 0x0)
	<autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc83500e3c0)
	/opt/work/golang/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc83500e3c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc83487a1a0)
	/opt/work/golang/go/src/net/http/transport.go:1069 +0x177
created by net/http.(*Transport).dialConn
	/opt/work/golang/go/src/net/http/transport.go:853 +0x10a6

goroutine 26230547 [IO wait]:
net.runtime_pollWait(0x7f8ebd562ea0, 0x72, 0xc83404b400)
	/opt/work/golang/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8341da060, 0x72, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8341da060, 0x0, 0x0)
	/opt/work/golang/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8341da000, 0xc83404b400, 0x200, 0x200, 0x0, 0x7f8ec1891050, 0xc82000a240)
	/opt/work/golang/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc833ae4728, 0xc83404b400, 0x200, 0x200, 0x200, 0x0, 0x0)
	/opt/work/golang/go/src/net/net.go:172 +0xe4
net.(*UDPConn).readDNSResponse(0xc833ae4728, 0xc834045700, 0x0, 0x0)
	/opt/work/golang/go/src/net/dnsclient_unix.go:52 +0xa5
net.exchange(0xc8341d8090, 0xd, 0xc8341bdc00, 0x12, 0xc8341d0001, 0x12a05f200, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/dnsclient_unix.go:155 +0x4ea
net.tryOneName(0xc82007c500, 0xc8341bdc00, 0x12, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/opt/work/golang/go/src/net/dnsclient_unix.go:181 +0x2f3
net.goLookupIPOrder.func1(0xc82007c500, 0xc8341d21e0, 0xc8341ce600, 0x1)
	/opt/work/golang/go/src/net/dnsclient_unix.go:462 +0x59
created by net.goLookupIPOrder
	/opt/work/golang/go/src/net/dnsclient_unix.go:464 +0x3d0

goroutine 26230440 [select]:
net/http.(*persistConn).roundTrip(0xc833c02340, 0xc833c5ae60, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:1469 +0xf1f
net/http.(*Transport).RoundTrip(0xc833ad0540, 0xc833b3e540, 0xc833ad0540, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:320 +0x9bb
net/http.send(0xc833b3e540, 0x7f8ec1895550, 0xc833ad0540, 0x0, 0x0, 0x0, 0xc833bffb20, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc82232dad0, 0xc833b3e540, 0x0, 0x0, 0x0, 0x16, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc82232dad0, 0xc833b3e540, 0xc4fd10, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0xc82232dad0, 0xc833b3e540, 0xa, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:188 +0xff
webcrawler/httplib.(*BeegoHTTPRequest).DoRequest(0xc833ad0480, 0xc833c57410, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:463 +0x800
webcrawler/httplib.(*BeegoHTTPRequest).getResponse(0xc833ad0480, 0xa754c0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:380 +0x5f
webcrawler/httplib.(*BeegoHTTPRequest).Response(0xc833ad0480, 0xc833bffae0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:546 +0x2d
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc833bffaa0, 0x16, 0xb200d8, 0x4, 0x1, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:179 +0x7de
main.(*WebCrawler).concurrentCheck(0xc820c88840, 0xc833b1d940, 0x12, 0x20ff7b)
	/opt/work/golang/project/src/webcrawler/main.go:381 +0x169
created by main.(*WebCrawler).loadRange
	/opt/work/golang/project/src/webcrawler/main.go:417 +0x544

goroutine 26214952 [select]:
net/http.(*persistConn).roundTrip(0xc82164dd40, 0xc820fbdce0, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:1469 +0xf1f
net/http.(*Transport).RoundTrip(0xc82307c840, 0xc8224ee540, 0xc82307c840, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:320 +0x9bb
net/http.send(0xc8224ee540, 0x7f8ec1895550, 0xc82307c840, 0x0, 0x0, 0x0, 0xc8218d0e40, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc820eedad0, 0xc8224ee540, 0x0, 0x0, 0x0, 0xb, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc820eedad0, 0xc8224ee540, 0xc4fd10, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0xc820eedad0, 0xc8224ee540, 0xa, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:188 +0xff
webcrawler/httplib.(*BeegoHTTPRequest).DoRequest(0xc82307c780, 0xc83f62eed0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:463 +0x800
webcrawler/httplib.(*BeegoHTTPRequest).getResponse(0xc82307c780, 0xa754c0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:380 +0x5f
webcrawler/httplib.(*BeegoHTTPRequest).Response(0xc82307c780, 0xc8218d0de0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:546 +0x2d
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc820fbdbf0, 0xb, 0xb200d8, 0x4, 0x1, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:179 +0x7de
main.(*WebCrawler).concurrentCheck(0xc820c88840, 0xc82027b647, 0x7, 0x20fab0)
	/opt/work/golang/project/src/webcrawler/main.go:381 +0x169
created by main.(*WebCrawler).loadRange
	/opt/work/golang/project/src/webcrawler/main.go:417 +0x544

goroutine 26209595 [select]:
net/http.(*Transport).getConn(0xc83f3d18c0, 0xc82277e7e0, 0x0, 0xc8219d5aa0, 0x5, 0xc8219d5b20, 0x16, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:707 +0x4ef
net/http.(*Transport).RoundTrip(0xc83f3d18c0, 0xc82277e7e0, 0xc83f3d18c0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:307 +0x7e9
net/http.send(0xc82277e7e0, 0x7f8ec1895550, 0xc83f3d18c0, 0x0, 0x0, 0x0, 0xc8219d5b00, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc82093d810, 0xc82277e7e0, 0x0, 0x0, 0x0, 0x12, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc82093d810, 0xc82277e7e0, 0xc4fd10, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0xc82093d810, 0xc82277e7e0, 0xa, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:188 +0xff
webcrawler/httplib.(*BeegoHTTPRequest).DoRequest(0xc83f3d1800, 0xc821e1cc30, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:463 +0x800
webcrawler/httplib.(*BeegoHTTPRequest).getResponse(0xc83f3d1800, 0xa754c0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:380 +0x5f
webcrawler/httplib.(*BeegoHTTPRequest).Response(0xc83f3d1800, 0xc8219d5ac0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:546 +0x2d
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc820a0fac0, 0x12, 0xb200f0, 0x5, 0xc800000002, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:179 +0x7de
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc820a0fac0, 0x12, 0xb200d8, 0x4, 0x1, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:187 +0x8ed
main.(*WebCrawler).concurrentCheck(0xc820c88840, 0xc820f0fcf0, 0xe, 0x20f90a)
	/opt/work/golang/project/src/webcrawler/main.go:381 +0x169
created by main.(*WebCrawler).loadRange
	/opt/work/golang/project/src/webcrawler/main.go:417 +0x544

goroutine 26154807 [select]:
net/http.(*Transport).getConn(0xc8218cc600, 0xc83f6442a0, 0x0, 0xc821305460, 0x5, 0xc821305520, 0x16, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:707 +0x4ef
net/http.(*Transport).RoundTrip(0xc8218cc600, 0xc83f6442a0, 0xc8218cc600, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/transport.go:307 +0x7e9
net/http.send(0xc83f6442a0, 0x7f8ec1895550, 0xc8218cc600, 0x0, 0x0, 0x0, 0xc8213054e0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc8231dd810, 0xc83f6442a0, 0x0, 0x0, 0x0, 0x12, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc8231dd810, 0xc83f6442a0, 0xc4fd10, 0x0, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0xc8231dd810, 0xc83f6442a0, 0xa, 0x0, 0x0)
	/opt/work/golang/go/src/net/http/client.go:188 +0xff
webcrawler/httplib.(*BeegoHTTPRequest).DoRequest(0xc8218cc540, 0x7f8ec18e14b0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:463 +0x800
webcrawler/httplib.(*BeegoHTTPRequest).getResponse(0xc8218cc540, 0xa754c0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:380 +0x5f
webcrawler/httplib.(*BeegoHTTPRequest).Response(0xc8218cc540, 0xc821305480, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/httplib/httplib.go:546 +0x2d
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc83ef7a5e0, 0x12, 0xb200f0, 0x5, 0xc800000002, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:179 +0x7de
main.(*WebCrawler).checkDomain(0xc820c88840, 0xc83ef7a5e0, 0x12, 0xb200d8, 0x4, 0x1, 0x0, 0x0, 0x0, 0x0)
	/opt/work/golang/project/src/webcrawler/main.go:187 +0x8ed
main.(*WebCrawler).concurrentCheck(0xc820c88840, 0xc821b3c060, 0xe, 0x20e7d8)
	/opt/work/golang/project/src/webcrawler/main.go:381 +0x169
created by main.(*WebCrawler).loadRange
	/opt/work/golang/project/src/webcrawler/main.go:417 +0x544

In go version go1.6 linux/amd64,Seems to be the same bug.I wrote a crawler , when 1.5 million urls are fetched, there has been such a problem.How to fix it?

source: gharchive/issue

authors: aegsea tamird

license_type: permissive

license_source: github-api


text:
RWMutex.Unlock() appearing in mutex contention profiles

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go1.8rc3
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/alex/dev/go"
GORACE=""
GOROOT="/Users/alex/sdk/go1.8rc3"
GOTOOLDIR="/Users/alex/sdk/go1.8rc3/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/l5/x43qw4sd10j_459gxsy9h3j80000gn/T/go-build886344651=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?
built a contrived test to profile mutex contention using RWMutex (without realising that RWMutex was not supported).
if required i can create a new version using only public code but for now the library being profiled is closed source.
What did you expect to see?
No contention shown in pprof list output for RWMutex
What did you see instead?
(pprof) list
Total: 1.06s
ROUTINE ======================== code.avct.io/services/lru.(*Map).Get in /Users/alex/dev/go/src/code.avct.io/services/lru/lru.go
         0    34.01us (flat, cum) 0.0032% of Total
         .          .    137:	m.em.RLock()
         .          .    138:	if e, exists := m.exist[k]; exists {
         .          .    139:		m.em.RUnlock()
         .          .    140:		m.vm.Lock()
         .          .    141:		m.values.MoveToFront(e)
         .    34.01us    142:		m.vm.Unlock()
         .          .    143:		return e.Value.(mapElement).V, true
         .          .    144:	}
         .          .    145:	m.em.RUnlock()
         .          .    146:	return nil, false
         .          .    147:}
ROUTINE ======================== code.avct.io/services/lru.(*Map).Set in /Users/alex/dev/go/src/code.avct.io/services/lru/lru.go
         0      1.06s (flat, cum)   100% of Total
         .          .    159:
         .          .    160:	// never panics because we always ensure the list has instances of mapElement
         .          .    161:	me := e.Value.(mapElement)
         .          .    162:	delete(m.exist, me.K)
         .          .    163:	m.exist[k] = m.values.PushFront(mapElement{K: k, V: v})
         .      1.01s    164:	m.em.Unlock()
         .          .    165:
         .          .    166:	m.values.Remove(e)
         .    49.82ms    167:	m.vm.Unlock()
         .          .    168:
         .          .    169:	// Run registered eviction function, if any
         .          .    170:	if me.Fn != nil {
         .          .    171:		me.Fn()
         .          .    172:	}
ROUTINE ======================== main.main.func2 in /Users/alex/dev/go/src/code.avct.io/services/cmd/scratch/main.go
         0      1.06s (flat, cum)   100% of Total
         .          .     46:		ctx := context.Background()
         .          .     47:		r.Wait(ctx)
         .          .     48:		go func() {
         .          .     49:			start := time.Now()
         .          .     50:			id := ids[rand.Intn(150000)]
         .    34.01us     51:			v, ok := m.Get(id)
         .          .     52:			if !ok {
         .      1.06s     53:				m.Set(id, users[id], nil)
         .          .     54:			}
         .          .     55:			if v == -1 {
         .          .     56:				panic("no")
         .          .     57:			}
         .          .     58:			tChan <- time.Since(start)
ROUTINE ======================== runtime.goexit in /Users/alex/sdk/go1.8rc3/src/runtime/asm_amd64.s
         0      1.06s (flat, cum)   100% of Total
         .          .   2192:	RET
         .          .   2193:
         .          .   2194:// The top-most function running on a goroutine
         .          .   2195:// returns to goexit+PCQuantum.
         .          .   2196:TEXT runtime·goexit(SB),NOSPLIT,$0-0
         .      1.06s   2197:	BYTE	$0x90	// NOP
         .          .   2198:	CALL	runtime·goexit1(SB)	// does not return
         .          .   2199:	// traceback from goexit1 must hit code range of goexit
         .          .   2200:	BYTE	$0x90	// NOP
         .          .   2201:
         .          .   2202:TEXT runtime·prefetcht0(SB),NOSPLIT,$0-8
ROUTINE ======================== sync.(*Mutex).Unlock in /Users/alex/sdk/go1.8rc3/src/sync/mutex.go
     1.06s      1.06s (flat, cum)   100% of Total
         .          .    121:			return
         .          .    122:		}
         .          .    123:		// Grab the right to wake someone.
         .          .    124:		new = (old - 1<<mutexWaiterShift) | mutexWoken
         .          .    125:		if atomic.CompareAndSwapInt32(&m.state, old, new) {
     1.06s      1.06s    126:			runtime_Semrelease(&m.sema)
         .          .    127:			return
         .          .    128:		}
         .          .    129:		old = m.state
         .          .    130:	}
         .          .    131:}
ROUTINE ======================== sync.(*RWMutex).Unlock in /Users/alex/sdk/go1.8rc3/src/sync/rwmutex.go
         0      1.06s (flat, cum)   100% of Total
         .          .    120:	// Unblock blocked readers, if any.
         .          .    121:	for i := 0; i < int(r); i++ {
         .          .    122:		runtime_Semrelease(&rw.readerSem)
         .          .    123:	}
         .          .    124:	// Allow other writers to proceed.
         .      1.06s    125:	rw.w.Unlock()
         .          .    126:	if race.Enabled {
         .          .    127:		race.Enable()
         .          .    128:	}
         .          .    129:}

i realised this profiling is just due to the sync.Mutex inside the RWMutex, so this issue is incorrect


text:
Proposal: x/crypto/blake2b: Make blake2b compatible with crypto/hmac

hmac.New() requires a function[1] that returns hash.Hash, which is what sha1,sha256,sha512 do but this thing changes with blake2b as it returns an error along with hash.Hash[2] which makes it incompatible with hmac.New().
Creating a hmac now with blake2b requires manual intervention, it would nice to support hmac out of box. I don't know how this could be done without breaking the existing API as I think just not returning error on blake2b functions would do the job. I don't know why blake2b alone returns error while SHAs don't.
Please advise.
Thanks.
// func New(h func() hash.Hash, key []byte) hash.Hash
[1] https://golang.org/pkg/crypto/hmac/#New
// func New512(key []byte) (hash.Hash, error)
[2] https://godoc.org/golang.org/x/crypto/blake2b#New512

hmac.New takes a func() hash.Hash because it needs to create two separate hash.Hashes. It's not clear to me what is supposed to happen with HMAC-around-Blake2b as far as whether both should be keyed the same. But yes we should try to find a way to make Blake2b work with hmac.New.
/cc @aead

HMAC-BLAKE2 does not make a lot of sense since BLAKE2 can be used itself as a keyed MAC, but I agree that x/crypto/blake2b should expose func() hash.Hash functions.
I wish we called the keyed options NewXXXWithKey, but I guess we can call the un-keyed versions (which don't return an error) NewHashXXX:
func NewHash256() hash.Hash
func NewHash384() hash.Hash
func NewHash512() hash.Hash

👍 ?

HMAC-BLAKE2 does not make a lot of sense since BLAKE2 can be used itself as a keyed MAC, but I agree that x/crypto/blake2b should expose func() hash.Hash functions.
I wish we called the keyed options NewXXXWithKey, but I guess we can call the un-keyed versions (which don't return an error) NewHashXXX:
func NewHash256() hash.Hash
func NewHash384() hash.Hash
func NewHash512() hash.Hash

👍 ?

I would agree with @FiloSottile , BLAKE2 combined with HMAC doesn't make much sense - as far as I can see... However if someone want to use BLAKE2 as a hash function for the HMAC construction I would prefer something like this (works only with 1.9):
import (
	"crypto"
	"crypto/hmac"
	"fmt"

	_ "golang.org/x/crypto/blake2b"
)

func main() {
	key := make([]byte, 32)
	msg := []byte("hello world")
	hash := hmac.New(crypto.BLAKE2b_256.New, key)
	hash.Write(msg)
	fmt.Println(hash.Sum(nil))
}

over adding new functions to the blake2(b/s) packages. They already contain a lot of exported functions and with crypto.Hash there is already a mechanism to create a hash.Hash

Forcing the indirection through package crypto is a bit odd. Do any real-world protocols use HMAC-BLAKE2? Maybe we don't need to do this?

I'm not aware of any protocols using HMAC-BLAKE2. I agree that the indirection is a bit odd compared to code using HMAC-SHA2. On the other hand BLAKE2 can be used as a MAC directly. However, the usage of BLAKE2 as a hash function is a bit different because of the error handling:
hash, err := blake2b.New256(nil)
if err != nil {
      // this will never happen...
}

Of course crypto.Hash solves this but I think many people use the package directly instead of crypto.Hash.New()

It sounds like there are thorny API issues here and on top of that "BLAKE2 combined with HMAC doesn't make much sense". And we know of no protocols that actually need HMAC-BLAKE2. So the fact that it's hard to do (well, a few lines of code) seems OK.

authors: FiloSottile aead demonshreder rsc


text:
x/net/http2: client can hang forever if headers' size exceeds connection's buffer size and server hangs past request time

What version of Go are you using (go version)?
go version go1.9.3 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/<redacted>/Dev/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/cd/_7rcv5812531s2lswhn6kp680000gp/T/go-build436968147=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

Also happened on linux/amd64, see below.
What did you do?
A bug in production showed an http2 client hanging more than 250 seconds without respect to the request context which was set to timeout after 5 sec.
All possible timeouts are set on connection and TLS handshake and I didn't see many dials (they are monitored).
Latency graph in ms:

Clients are using net/http2.Transport directly.
What did you expect to see?
The requests should have timed out after 5s.
What did you see instead?
No or very long timeouts (I believe the server has reset the connection or TCP timed out).
The synchronous write cc.writeHeaders to the connection in ClientConn.roundTrip does not set any deadline on the connection which can block forever (or being timed out by TCP) if the server or network hangs:
https://github.com/golang/net/blob/0ed95abb35c445290478a5348a7b38bb154135fd/http2/transport.go#L833
I wrote a test that demonstrates this:
https://github.com/gwik/net/commit/e4c191ad930256d02228b1daf94aeb4d03614f03
// TestTransportTimeoutServerHangs demonstrates that client can hang forever
// when the server hangs and the headers exceed the conn buffer size (forcing a
// flush). Without respect to the context's deadline.
func TestTransportTimeoutServerHangs(t *testing.T) {
	clientDone := make(chan struct{})
	ct := newClientTester(t)
	ct.client = func() error {
		defer ct.cc.(*net.TCPConn).CloseWrite()
		defer close(clientDone)

		buf := make([]byte, 1<<19)
		_, err := rand.Read(buf)
		if err != nil {
			t.Fatal("fail to gen random data")
		}
		headerVal := hex.EncodeToString(buf)

		req, err := http.NewRequest("PUT", "https://dummy.tld/", nil)
		if err != nil {
			return err
		}

		ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
		defer cancel()
		req = req.WithContext(ctx)
		req.Header.Add("Authorization", headerVal)
		_, err = ct.tr.RoundTrip(req)
		if err == nil {
			return errors.New("error should not be nil")
		}
		if ne, ok := err.(net.Error); !ok || !ne.Timeout() {
			return fmt.Errorf("error should be a net error timeout was: +%v", err)
		}
		return nil
	}
	ct.server = func() error {
		ct.greet()
		select {
		case <-time.After(5 * time.Second):
		case <-clientDone:
		}
		return nil
	}
	ct.run()
}

Setting the write deadline fixes the test:
https://github.com/gwik/net/commit/052de95fd6c4deec2b16b158e4b4b70793fbf678
It seems to fail when the header value exceeds 1MB. I might miss something here, the default buffer size of bufio.Writer is 4096 bytes, I had expected to see it fail around that value, maybe compression and/or TCP buffers...
Also I don't think it sent 1MB headers when it failed in production something else must have fill the buffer.
The buffer is on the connection and shared among streams the buffer can be filled by other requests on the same connection.
Besides this particular call which is synchronous, no write nor read to the connection has a deadline set. Can't this lead to goroutine leaks and http2 streams being stuck in the background ?

Any update here?
Keep in mind that there are 3 goroutines per Transport connection to a server:

the writer (which might block forever)
the reader (blocked by default, waiting for frames from server)
the control one, which never blocks, and owns all state

I think the behavior we want is:

if the user said 5 seconds timeout on a request, the control channel should notice and return an error to the user's RoundTrip. I'd be surprised if this part isn't already working?
don't accept new streams (new RoundTrips) on that ClientConn if the conn has been stuck in a write for "too long", for some definition TBD
if the write appears totally hung, nuke the whole connection and all streams still attached to it

if the user said 5 seconds timeout on a request, the control channel should notice and return an error to the user's RoundTrip. I'd be surprised if this part isn't already working?

There are cases where this isn't working. I believe the graph and tests demonstrates it.

don't accept new streams (new RoundTrips) on that ClientConn if the conn has been stuck in a write for "too long", for some definition TBD

Unfortunately, the writes almost never time out I don't know if this is specific to my use particular use case or if buffering makes it very long to happen.
I run this CL in production for a very specific service that only makes requests to APNS and Firebase  push gateways.
The patch helps to recover from stuck connections and kills it but it's slow to converge, is incomplete and may not work properly for the general use case.
What happens is that there are situations where the APNS gateway is blocking the connection, it behaves like a firewall that drops packet never reset the connection. Most of the time the requests are timing out correctly, the context allow to unlock the client stream goroutine and return to the caller however the connection read loop never times out nor receive any other error causing all requests to time out. (Also not that the APNS gateway doesn't allow pings).
What I see most often is the writes are not timing out. I believe that the only way to guarantee to time out the connection is have a way to unblock the read loop. The reads (ClientConn) are asynchronous to the writes (ClientStream) so we have to find a way to add a deadline on the reads that will follow the writes (requests) or that follow a read (e.g reading the rest of the body) from the client stream side.
The CL takes a simple approach and avoids synchronization between the client stream and the connection read loop. It resets the read deadline and sets a write deadline before every write from the client stream. After the write is done, it sets the read deadline.
This kinda work, I see the that connections are no longer stuck forever but it is slow to time out the connection. Resetting the read deadline before every write continuously pushes back the deadline.
Also there are no read timeout when waiting for subsequent body reads for example.
I wanted to experiment the following algorithm:
Upon waiting for something from the network on the ClientStream / caller side, push a deadline in a heap data structure (probably under ClientConn.wmu lock) keeping track of the stream that pushes it.
If there was no deadline, unlock the read loop by setting one in the past an set the deadline before next read.
Before every read look at the root of the deadline heap and set it.
After every read remove the deadline of the stream that has successfully completed the read. We need some sort of reverse index to find the entry in the heap and remove it (maybe every client stream can track it's unique read deadline entry in the heap).
What do you think ?

This is actually worse than just a single connection hanging, it can cause the entire app to hang.
The blocking behaviour in both the cc.bw.Flush() and cc.fr.WriteHeaders(...) called within cc.writeHeaders(...) is done while holding cc.mu which is required for cc.idleState() called by p.getClientConn() while holding the pool look.
This means a single connection can block all new requests potentially forever.

@stevenh Could you please share a bit more on the context ? What your app does, which kind of services it talks to and when this kind of behavior happens ?

I'd have to defer to a colleague to answer that but the high level basics are its making http2 calls and the entire app hung due to this.
The relevant stacks from the hung app are:
-- goroutine blocked in writeHeaders -> Flush
1 @ 0x43bddc 0x436eea 0x4364e7 0x4a201e 0x4a210d 0x4a3a04 0x577812 0x589dbd 0x5d1887 0x5d1da0 0x5d2cbb 0x662cea 0x6ad1d4 0x52bdce 0x6678dc 0x66680b 0x663215 0x662dda 0x6476b9 0x690a17 0x63b819 0x63b4ad 0x63cb4d 0xb1d000 0xe641df 0xe6213e 0xe8245c 0xe7f42e 0xe7ad28 0xe61e87 0xf598ed 0xf59637
#    0x4364e6    internal/poll.runtime_pollWait+0x56                                                    /usr/local/go/src/runtime/netpoll.go:173
#    0x4a201d    internal/poll.(*pollDesc).wait+0xad                                                    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85
#    0x4a210c    internal/poll.(*pollDesc).waitWrite+0x3c                                                /usr/local/go/src/internal/poll/fd_poll_runtime.go:94
#    0x4a3a03    internal/poll.(*FD).Write+0x243                                                        /usr/local/go/src/internal/poll/fd_unix.go:227
#    0x577811    net.(*netFD).Write+0x51                                                            /usr/local/go/src/net/fd_unix.go:220
#    0x589dbc    net.(*conn).Write+0x6c                                                            /usr/local/go/src/net/net.go:188
#    0x5d1886    crypto/tls.(*Conn).write+0x1a6                                                        /usr/local/go/src/crypto/tls/conn.go:832
#    0x5d1d9f    crypto/tls.(*Conn).writeRecordLocked+0x35f                                                /usr/local/go/src/crypto/tls/conn.go:909
#    0x5d2cba    crypto/tls.(*Conn).Write+0x1ba                                                        /usr/local/go/src/crypto/tls/conn.go:1068
#    0x662ce9    net/http.http2stickyErrWriter.Write+0x79                                                /usr/local/go/src/net/http/h2_bundle.go:6803
#    0x6ad1d3    net/http.(*http2stickyErrWriter).Write+0x73                                                <autogenerated>:1
#    0x52bdcd    bufio.(*Writer).Flush+0x7d                                                        /usr/local/go/src/bufio/bufio.go:567
#    0x6678db    net/http.(*http2ClientConn).writeHeaders+0x1eb                                                /usr/local/go/src/net/http/h2_bundle.go:7436
#    0x66680a    net/http.(*http2ClientConn).RoundTrip+0x46a                                                /usr/local/go/src/net/http/h2_bundle.go:7289
#    0x663214    net/http.(*http2Transport).RoundTripOpt+0x144                                                /usr/local/go/src/net/http/h2_bundle.go:6858
#    0x662dd9    net/http.(*http2Transport).RoundTrip+0x39                                                /usr/local/go/src/net/http/h2_bundle.go:6820
#    0x6476b8    net/http.http2noDialH2RoundTripper.RoundTrip+0x38                                            /usr/local/go/src/net/http/h2_bundle.go:990
#    0x690a16    net/http.(*Transport).RoundTrip+0xd46                                                    /usr/local/go/src/net/http/transport.go:371
#    0x63b818    net/http.send+0x1a8                                                            /usr/local/go/src/net/http/client.go:249
#    0x63b4ac    net/http.(*Client).send+0xfc                                                        /usr/local/go/src/net/http/client.go:173
...

-- goroutine waiting for connection lock, held above, while holding the pool lock
1 @ 0x43bddc 0x43bece 0x44d2c4 0x44cfdd 0x48115e 0x66546f 0x6457d8 0x646dce 0x6631c7 0x662dda 0x6476b9 0x690a17 0x63b819 0x63b4ad 0x63cb4d 0xb1d000 0xe641df 0xe6213e 0xe8245c 0xe7f42e 0xe7ad28 0xe61e87 0xf598ed 0xf59637 0x46b1f1
#    0x44cfdc    sync.runtime_SemacquireMutex+0x3c                                                    /usr/local/go/src/runtime/sema.go:71
#    0x48115d    sync.(*Mutex).Lock+0xed                                                            /usr/local/go/src/sync/mutex.go:134
#    0x66546e    net/http.(*http2ClientConn).CanTakeNewRequest+0x3e                                            /usr/local/go/src/net/http/h2_bundle.go:7083
#    0x6457d7    net/http.(*http2clientConnPool).getClientConn+0xe7                                            /usr/local/go/src/net/http/h2_bundle.go:739
#    0x646dcd    net/http.http2noDialClientConnPool.GetClientConn+0x4d                                            /usr/local/go/src/net/http/h2_bundle.go:927
#    0x6631c6    net/http.(*http2Transport).RoundTripOpt+0xf6                                                /usr/local/go/src/net/http/h2_bundle.go:6852
#    0x662dd9    net/http.(*http2Transport).RoundTrip+0x39                                                /usr/local/go/src/net/http/h2_bundle.go:6820
#    0x6476b8    net/http.http2noDialH2RoundTripper.RoundTrip+0x38                                            /usr/local/go/src/net/http/h2_bundle.go:990
#    0x690a16    net/http.(*Transport).RoundTrip+0xd46
...

-- goroutines blocked waiting for the pool lock head above
13033 @ 0x43bddc 0x43bece 0x44d2c4 0x44cfdd 0x48115e 0x645754 0x646dce 0x6631c7 0x662dda 0x6476b9 0x690a17 0x63b819 0x63b4ad 0x63cb4d 0xb1d000 0xe641df 0xe6213e 0xe8245c 0xe7f42e 0xe7ad28 0xe61e87 0xf598ed 0xf59637 0x46b1f1
#   0x44cfdc    sync.runtime_SemacquireMutex+0x3c                                                   /usr/local/go/src/runtime/sema.go:71
#   0x48115d    sync.(*Mutex).Lock+0xed                                                         /usr/local/go/src/sync/mutex.go:134
#   0x645753    net/http.(*http2clientConnPool).getClientConn+0x63                                          /usr/local/go/src/net/http/h2_bundle.go:737
#   0x646dcd    net/http.http2noDialClientConnPool.GetClientConn+0x4d                                           /usr/local/go/src/net/http/h2_bundle.go:927
#   0x6631c6    net/http.(*http2Transport).RoundTripOpt+0xf6                                                /usr/local/go/src/net/http/h2_bundle.go:6852
#   0x662dd9    net/http.(*http2Transport).RoundTrip+0x39                                               /usr/local/go/src/net/http/h2_bundle.go:6820
#   0x6476b8    net/http.http2noDialH2RoundTripper.RoundTrip+0x38                                           /usr/local/go/src/net/http/h2_bundle.go:990
#   0x690a16    net/http.(*Transport).RoundTrip+0xd46                                                   /usr/local/go/src/net/http/transport.go:371
#   0x63b818    net/http.send+0x1a8                                                         /usr/local/go/src/net/http/client.go:249
#   0x63b4ac    net/http.(*Client).send+0xfc                                                        /usr/local/go/src/net/http/client.go:173
#   0x63cb4c    net/http.(*Client).Do+0x28c                                                     /usr/local/go/src/net/http/client.go:602

I believe we just hit this same issue or extremely similar issue with go1.12.7. I have thread stacks from the "blocked" threads from kill -ABRT below. To be clear: The caller of http.Client.Do is stuck in http2ClientConn.roundTrip on cc.wmu.Lock()  so it can call cc.writeHeaders. The holder of that lock is from scheduleBodyWrite/http2clientStream.writeRequestBody, which is doing an actual network write that is hung.
As a result: Client.Do ignores any context Deadlines in this code path.
Possible fixes:

The RoundTrip thread can't try to lock cc.wmu since it can get stuck on network I/O, so writing headers needs to be done with some Go routines so it can use something like awaitRequestCancel.
Don't attempt to reuse HTTP/2 connections while another request is using it to send

Details
Stripe Veneur periodically sends a set of HTTP2 POSTs and waits for them to complete. When it is getting CPU overloaded, we have observed these calls to Client.Do getting "stuck" for ~10 minutes, even though the HTTP requests set a 10 second deadline. I think the following is happening:

One thread calls http.Client.Do, but then times out waiting for the response. However, it has already called scheduleBodyWrite.
scheduleBodyWrite's goroutine holds cc.wmu while trying to send the POST body. Because the system is overloaded, this is now stuck waiting for some sort of TCP timer.
Another thread starts http.Client.Do. It gets the same connection from the pool, since it is an HTTP2 connection that supports multiple streams.
This eventually calls:

    cc.wmu.Lock()
    endStream := !hasBody && !hasTrailers
    werr := cc.writeHeaders(cs.ID, endStream, int(cc.maxFrameSize), hdrs)
    cc.wmu.Unlock()

The client gets stuck waiting on cc.wmu.Lock(), and will wait until the io operation completes, ignoring the context

Stacks
The first thread is the writer, holding cc.wmu. The second is the OTHER writer, blocked waiting on cc.wmu. The third is the read loop, for completeness (this lets us see that the http2Framer belongs to the connection, so these are in fact the threads involved in this issue.
internal/poll.runtime_pollWait(0x7f2818062bb0, 0x77, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc003c06298, 0x77, 0x100, 0x19f, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitWrite(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:96
internal/poll.(*FD).Write(0xc003c06280, 0xc004ed7000, 0x401d, 0x401d, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:276 +0x26e
net.(*netFD).Write(0xc003c06280, 0xc004ed7000, 0x401d, 0x401d, 0x401d, 0x4018, 0x0)
    /usr/local/go/src/net/fd_unix.go:220 +0x4f
net.(*conn).Write(0xc000013460, 0xc004ed7000, 0x401d, 0x401d, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:189 +0x69
crypto/tls.(*Conn).write(0xc005b38e00, 0xc004ed7000, 0x401d, 0x401d, 0xc009306000, 0x4000, 0x40000)
    /usr/local/go/src/crypto/tls/conn.go:896 +0x160
crypto/tls.(*Conn).writeRecordLocked(0xc005b38e00, 0x15f4217, 0xc009306000, 0x10009, 0x40000, 0x3, 0x3, 0xc000c24daa)
    /usr/local/go/src/crypto/tls/conn.go:945 +0x2a1
crypto/tls.(*Conn).Write(0xc005b38e00, 0xc0092fe000, 0x10009, 0x48000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1114 +0x184
net/http.http2stickyErrWriter.Write(0x7f281c2393f0, 0xc005b38e00, 0xc00427bad8, 0xc0092fe000, 0x10009, 0x48000, 0x1, 0xc01f3b20c0, 0xc000c24d40)
    /usr/local/go/src/net/http/h2_bundle.go:6792 +0x68
bufio.(*Writer).Write(0xc004686480, 0xc0092fe000, 0x10009, 0x48000, 0xc000052a70, 0xc000052a00, 0x0)
    /usr/local/go/src/bufio/bufio.go:622 +0x145
net/http.(*http2Framer).endWrite(0xc00330fb20, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:1605 +0xab
net/http.(*http2Framer).WriteDataPadded(0xc00330fb20, 0x3bccb, 0xc0084ce000, 0x10000, 0x80000, 0x0, 0x0, 0x0, 0x1525493, 0xc000c24f88)
    /usr/local/go/src/net/http/h2_bundle.go:1927 +0x25d
net/http.(*http2Framer).WriteData(...)
    /usr/local/go/src/net/http/h2_bundle.go:1885
net/http.(*http2clientStream).writeRequestBody(0xc003362000, 0x7f281c239488, 0xc000084d70, 0x7f281c2394a8, 0xc000084d70, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:7690 +0x337
net/http.(*http2Transport).getBodyWriterState.func1()
    /usr/local/go/src/net/http/h2_bundle.go:8878 +0xc2
created by net/http.http2bodyWriterState.scheduleBodyWrite
    /usr/local/go/src/net/http/h2_bundle.go:8925 +0xa3

goroutine 3263511 [semacquire, 10 minutes]:
sync.runtime_SemacquireMutex(0xc00427bad4, 0x40df00)
    /usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc00427bad0)
    /usr/local/go/src/sync/mutex.go:134 +0x109
net/http.(*http2ClientConn).roundTrip(0xc00427b980, 0xc005448300, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:7444 +0x467
net/http.(*http2Transport).RoundTripOpt(0xc002094180, 0xc005448300, 0xc01ea24600, 0x6dedb6, 0x0, 0xc000914240)
    /usr/local/go/src/net/http/h2_bundle.go:6867 +0x159
net/http.(*http2Transport).RoundTrip(...)
    /usr/local/go/src/net/http/h2_bundle.go:6829
net/http.http2noDialH2RoundTripper.RoundTrip(0xc002094180, 0xc005448300, 0xc000838000, 0x5, 0xc0009142c8)
    /usr/local/go/src/net/http/h2_bundle.go:8959 +0x3f
net/http.(*Transport).roundTrip(0xc00038c480, 0xc005448300, 0xc02eee6240, 0xc029bcd9bc, 0xc029bcd9c0)
    /usr/local/go/src/net/http/transport.go:430 +0xe90
net/http.(*Transport).RoundTrip(0xc00038c480, 0xc005448300, 0xc00038c480, 0xbf4650ff70c64558, 0x459b8945e32c2)
    /usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc005448200, 0x1791b80, 0xc00038c480, 0xbf4650ff70c64558, 0x459b8945e32c2, 0x23e7c00, 0xc029e7a3a8, 0xbf4650ff70c64558, 0x1, 0x0)
    /usr/local/go/src/net/http/client.go:250 +0x461
net/http.(*Client).send(0xc0003aa660, 0xc005448200, 0xbf4650ff70c64558, 0x459b8945e32c2, 0x23e7c00, 0xc029e7a3a8, 0x0, 0x1, 0x0)
    /usr/local/go/src/net/http/client.go:174 +0xfb
net/http.(*Client).do(0xc0003aa660, 0xc005448200, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:641 +0x279
net/http.(*Client).Do(...)
    /usr/local/go/src/net/http/client.go:509
github.com/stripe/veneur/http.PostHelper(0x17c4ce0, 0xc000492ed0, 0xc0003aa660, 0xc000440400, 0x1515d91, 0x4, 0xc000838000, 0x50, 0x135a860, 0xc000493200, ...)
    /go/src/github.com/stripe/veneur/http/http.go:229 +0xc94
github.com/stripe/veneur/sinks/datadog.(*DatadogMetricSink).flushPart(0xc0001e2240, 0x17c4ce0, 0xc000492ed0, 0xc034f00000, 0x4ebf, 0x4ebf, 0xc00821a850)
    /go/src/github.com/stripe/veneur/sinks/datadog/datadog.go:356 +0x30f
created by github.com/stripe/veneur/sinks/datadog.(*DatadogMetricSink).Flush
    /go/src/github.com/stripe/veneur/sinks/datadog/datadog.go:144 +0x40a

goroutine 1319 [IO wait, 13 minutes]:
internal/poll.runtime_pollWait(0x7f2818062bb0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc003c06298, 0x72, 0x1900, 0x199a, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003c06280, 0xc003c1af80, 0x199a, 0x199a, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc003c06280, 0xc003c1af80, 0x199a, 0x199a, 0x203003, 0x0, 0x198d)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000013460, 0xc003c1af80, 0x199a, 0x199a, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:177 +0x69
crypto/tls.(*atLeastReader).Read(0xc00e6373e0, 0xc003c1af80, 0x199a, 0x199a, 0xc0126b6068, 0x2, 0xc0005929e0)
    /usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc005b39058, 0x178fae0, 0xc00e6373e0, 0x40b805, 0x1397300, 0x14c5ee0)
    /usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc005b38e00, 0x1791ae0, 0xc000013460, 0x5, 0xc000013460, 0x1)
    /usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc005b38e00, 0x15f4200, 0xc005b38f38, 0xc000592d58)
    /usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
    /usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc005b38e00, 0xc006189000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1236 +0x137
bufio.(*Reader).Read(0xc0003f3380, 0xc00330fb58, 0x9, 0x9, 0x407e54, 0xc028298480, 0xc000592d58)
    /usr/local/go/src/bufio/bufio.go:223 +0x23e
io.ReadAtLeast(0x178f920, 0xc0003f3380, 0xc00330fb58, 0x9, 0x9, 0x9, 0x178fcc0, 0xc02e520ad0, 0xc000084060)
    /usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
net/http.http2readFrameHeader(0xc00330fb58, 0x9, 0x9, 0x178f920, 0xc0003f3380, 0x0, 0x0, 0xbf4650d0587ebd97, 0x4598cb66a5319)
    /usr/local/go/src/net/http/h2_bundle.go:1476 +0x88
net/http.(*http2Framer).ReadFrame(0xc00330fb20, 0xc03755a150, 0x0, 0x0, 0x1)
    /usr/local/go/src/net/http/h2_bundle.go:1734 +0xa1
net/http.(*http2clientConnReadLoop).run(0xc000592fb8, 0x15f3a90, 0xc0005427b8)
    /usr/local/go/src/net/http/h2_bundle.go:8102 +0x8f
net/http.(*http2ClientConn).readLoop(0xc00427b980)
    /usr/local/go/src/net/http/h2_bundle.go:8030 +0x76
created by net/http.(*http2Transport).newClientConn
    /usr/local/go/src/net/http/h2_bundle.go:7093 +0x637

It looks like CL https://go-review.googlesource.com/c/net/+/181457/ mentioned on issue #32388 will probably address my issue.

So we have a total deadlock scenario that hasn't been fixed for years now.  This regularly break hundreds of requests for us because a whole process has to be killed because new requests coming in just sit behind the lock and do not get serviced so we kill the process and abandon any in-progress requests in that process in order to stop creating even more locked requests.
I could quickly fix this if sync.Mutex supported TryLock() but that request (only a few lines of code) was rejected because a super strong use case for having it was required.
Perhaps being able to fix a very serious bug that can't be fixed for years is a strong enough use case?  (Or should we argue about whether the connection mutex should be replaced with a channel instead?)
(TryLock proposal: https://github.com/golang/go/issues/6123)
So, if there is a chance this would be accepted, I'm happy to submit a code change proposal that adds sync.Mutex.TryLock() and then uses it to not deadlock when looking for a connection to reuse (by not considering connections that are currently locked and so really can't be reused with any guarantee of timeliness anyway).

FWIW, I concur with the quibble about the name and would be happy to call it LockIfNoWait().

@TyeMcQueen what all is this pertaining with,  I'm dealing with a bug on my PC that I'm not exactly sure how to execute the correct code for the fix

I don't believe the TryLock approach is the right fix here.
In the usual case where the server is responsive and we do not exceed the server per-connection stream limit, we want to create a single HTTP/2 connection and use it for all requests. Creating a new connection whenever we are unable to unlock any existing connection means that we will create new connections much more aggressively. If, for example, we make ten simultaneous requests, we will be very likely to create several new connections as all the requests contend for the connection lock.
The TryLock-based patch above also completely breaks the (http2.Transport).StrictMaxConcurrentStreams configuration knob.
Possibly we should have a way to create multiple HTTP/2 connections to a single server even before we exceed the stream limit on a single connection, but if so, this should be done more deliberately than creating new connections every time two requests briefly contend on a mutex.
The right fix to avoid connection pool deadlocks (#32388) is to avoid holding the connection mutex indefinitely during blocking operations, such as header writes. I have a proposed fix for that in CL 349594.
The right fix to avoid requests hanging past the request deadline when writing headers blocks (this bug, as distinct from #32388) is to watch for cancelation while writing. Unfortunately, this means we need to move header writes to a separate goroutine, since there is no simple way to bound a write to a net.Conn with a context. I have a change which does this in CL 347299.

authors: TyeMcQueen bradfitz evanj gwik king40or neild stevenh


text:
runtime: wrong Unix error code from panic.

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version devel +baf3eb1625 Tue Mar 6 01:11:26 2018 +0000 darwin/amd64
% cat p.go
package main

func main() {
	panic(3)
}
% ./p
panic: 3

goroutine 1 [running]:
main.main()
	/Users/r/p.go:4 +0x39
% echo $?
2
%

Exit code 2 traditionally means 'incorrect arguments', as in flag.Usage.
Panic should be something like 127 or 255 on Unix.
Reported by darren.e.grant@gmail.com on nuts.

or simply 1 meaning 'general error' ?
(http://tldp.org/LDP/abs/html/exitcodes.html)

As has been pointed out in the mail thread, it might be best to leave this alone and say that the error that triggered this is in a program that takes exit code 2 as 'success'. That is certainly nonstandard.

Sure, although a note in the language spec might avoid some similar confusion in the future..
I'll open an issue with Supervisor (http://supervisord.org/)

E-mail thread is at https://groups.google.com/d/msg/golang-nuts/u9NgKibJsKI/XxCdDihFDAAJ .

Thanks!

How's it going @cachvico? Any news from filing the bug with Supervisord?
@ianlancetaylor sure let's document in the spec that a panic returns a non-zero code to the shell.
@robpike @aclements shall we proceed with making a panic return 127 or 255? @robpike if you might have time, a reference to a UNIX manual would be useful for us to include in the commit for posterity.

Thanks - note that the Supervisord issue should be fixed on their side as per the above linked issue.

Exit status 127 is used by the shell to indicate "command not found."
Exit status 126 is used by the shell to indicate "command found but not executable."
Exit status values 128 and up are used by the shell to indicate exiting due to a signal.
If a C++ program exits due to an uncaught throw it raises SIGABRT and therefore exits with status 134 (on GNU/Linux).
If a Python program exits due to raising an exception, it exits with status 1.
I think we should change an unrecovered panic to exit with status 1 rather than 2.  And we should document in the language spec that an uncaught panic will cause the program to exit with a non-zero exit status, but we shouldn't document it further.

The quickest solution to the original problem would simply be a correction to the spec at https://golang.org/pkg/builtin/#panic
I've submitted a change at https://go-review.googlesource.com/c/go/+/167709 - is this correct?

Beg pardon - just realised the previous post said not to document the value explicitly.
May I make the change to  suggested by @ianlancetaylor to change the 2 to 1, and correct my change to the comment appropriately?
This would involve an edit to src/runtime/signal_sighandler.go line 151.

I've submitted a change at https://go-review.googlesource.com/c/go/+/167709 - is this correct?

Thanks for submitting the change @cachvico! I've posted some comments  on it.

May I make the change suggested by @ianlancetaylor to change the process exit code 2 to 1, and correct my change to the comment appropriately?
This would involve an edit to src/runtime/signal_sighandler.go line 151, and a supporting test-case.

Oh yes you can, all yours but we can review and submit those changes piece-meal/independent of the docs update.

@ianlancetaylor Note that only values greater than 128 are used by the shell to indicate exiting due to a signal.

Exit status values 128 and up are used by the shell to indicate exiting due to a signal.

Some documents indicate that 128 is used to indicate an invalid argument to shell builtin exit, but POSIX leave that behavior unspecified.

@Gnouc Thanks for the correction.

Docs updated in https://github.com/golang/go/commit/f947c4dcfe6f22e6c92e6e5a170096b518a18448 to reflect the behaviour of panic()
Thanks all!

authors: Gnouc cachvico ianlancetaylor odeke-em robpike


text:
x/gddo: HSTS for godoc.org

godoc.org uses HTTPS. It would be great to increase protection by implementing HSTS and preloading: https://hstspreload.org/?domain=godoc.org
This is especially valuable for godoc.org, since URLs are designed to be easily constructed (from other URLs) by hand and not everyone might add/keep the HTTPS scheme when they do so.
cc @FiloSottile

It seems the godoc.org server is constructed at
https://github.com/golang/gddo/blob/9ab275bde8fe1bb887642e9250b8d58aba11af61/gddo-server/main.go#L850
but I'm not sure about the best place to add a new header.

If this is just about godoc.org, I believe issues about that are tracked on that repo.
I also checked golang.org which seems to be missing the includeSubDomains directive, but it does have the preload header though.

I suggested opening an issue here so that we can do godoc and golang.org at the same time.

Ah alright. ping @andybons for golang.org.

authors: FiloSottile agnivade lgarron


text:
cmd/gopherbot: race condition in auto-closure of scratch CLs causing gopherbot to stall 30s

When closing scratch CLs, the maintner corpus doesn't get updated fast enough and it attempts to close the same CLs again, resulting in a 409 conflict:
2018/10/19 20:19:32 closing scratch CL https://golang.org/cl/140997 ...
2018/10/19 20:19:32 abandon scratch reviews: HTTP status 409 Conflict; change is abandoned
2018/10/19 20:19:32 HTTP status 409 Conflict; change is abandoned
2018/10/19 20:19:32 gopherbot ran in 1.745149115s
2018/10/19 20:19:32 sleeping 30s after previous error.

This error causes gopherbot to sleep for 30s as a result.
This is related to #28226 /cc @dmitshur

I think I've already seen code somewhere that uses Gerrit API in addition to maintner to confirm the "latest" state before taking some action. @bradfitz will probably remember where that was.
I suspect a fix can use a similar approach.

@dmitshur auto-assignment of reviews and gerritbot both use the Gerrit API so that’s likely where you saw it.

@dmitshur, can you take this one?

authors: andybons bradfitz dmitshur