text:
[0.9.4-nightly-cf58c38] - 2nd day over weekend received, "fatal error: unexpected signal during runtime execution"

I started pushing data in around 2015/08/28 18:55 on Friday and appears it continued okay for ~2 days with 80-200ms writes until 2015/08/30 lunch time where 5 second timeouts hit.  After a few hours of timeouts it appears the server returned fatal error and threw stack.
I have about 162GB of influx data on disk with IO jumping high, but never running steadly capped for more than 20 seconds during the writes. (4 ssds raid 0)  40 cores jumped to 100%, but never stayed high for more than 30 seconds.
Let me know if needing more information.
Sorry for the massive stack trace, but not sure of a better way for you to understand what is going on.
Log starting where problem occurred:
2015/08/28 18:55:44 InfluxDB starting, version 0.9.4-nightly-cf58c38, branch master, commit cf58c38995f64a45bd8f8b985654facee1892e33
...
[http] 2015/08/30 16:21:28 127.0.0.1 - test_db [30/Aug/2015:16:21:23 -0700] POST /write?db=test_db&precision=s HTTP/1.1 500 32 - python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-44-generic d4585661-4f6d-11e5-b4e5-000000000000 5.016726244s
[http] 2015/08/30 16:21:28 127.0.0.1 - test_db [30/Aug/2015:16:21:23 -0700] POST /write?db=test_db&precision=s HTTP/1.1 500 32 - python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-44-generic d46bdffa-4f6d-11e5-b4e6-000000000000 5.016340449s
[http] 2015/08/30 16:21:29 127.0.0.1 - test_db [30/Aug/2015:16:21:24 -0700] POST /write?db=test_db&precision=s HTTP/1.1 500 32 - python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-44-generic d4bba2d5-4f6d-11e5-b4e7-000000000000 5.047049858s
[http] 2015/08/30 16:21:29 127.0.0.1 - test_db [30/Aug/2015:16:21:24 -0700] POST /write?db=test_db&precision=s HTTP/1.1 500 32 - python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-44-generic d4c1a96f-4f6d-11e5-b4e9-000000000000 5.150094094s
[http] 2015/08/30 16:21:30 127.0.0.1 - test_db [30/Aug/2015:16:21:25 -0700] POST /write?db=test_db&precision=s HTTP/1.1 500 32 - python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-44-generic d5626996-4f6d-11e5-b4ea-000000000000 5.00570939s
[http] 2015/08/30 16:21:30 127.0.0.1 - test_db [30/Aug/2015:16:21:25 -0700] POST /write?db=test_db&precision=s HTTP/1.1 500 32 - python-requests/2.7.0 CPython/2.7.6 Linux/3.13.0-44-generic d5866845-4f6d-11e5-b4eb-000000000000 5.068950362s
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xf1 pc=0x46318b]

runtime stack:
runtime.throw(0xc871a0, 0x2a)
        /root/.gvm/gos/go1.5/src/runtime/panic.go:527 +0x90 fp=0x7feea56fbd70 sp=0x7feea56fbd58
runtime.sigpanic()
        /root/.gvm/gos/go1.5/src/runtime/sigpanic_unix.go:12 +0x5a fp=0x7feea56fbdc0 sp=0x7feea56fbd70
runtime.munmap(0x7ff607fdb000, 0x1ac1070, 0x41cd0f, 0x7ff607fdb000, 0x1ac1070, 0xfe94c0, 0x7ff607fdb000, 0x35820e, 0x36bc9a, 0x41c52c, ...)
        /root/.gvm/gos/go1.5/src/runtime/sys_linux_amd64.s:266 +0x1b fp=0x7feea56fbdc8 sp=0x7feea56fbdc0
runtime.sysFree(0x7ff607fdb000, 0x1ac1070, 0xfe94c0)
        /root/.gvm/gos/go1.5/src/runtime/mem_linux.go:103 +0x34 fp=0x7feea56fbde0 sp=0x7feea56fbdc8
runtime.gcCopySpans()
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1599 +0xdf fp=0x7feea56fbe18 sp=0x7feea56fbde0
runtime.gcMark(0x143bb3f6906e11)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1414 +0x5c fp=0x7feea56fbe80 sp=0x7feea56fbe18
runtime.gc.func3()
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1088 +0x23 fp=0x7feea56fbe90 sp=0x7feea56fbe80
runtime.systemstack(0xc820027500)
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:262 +0x79 fp=0x7feea56fbe98 sp=0x7feea56fbe90
runtime.mstart()
        /root/.gvm/gos/go1.5/src/runtime/proc1.go:674 fp=0x7feea56fbea0 sp=0x7feea56fbe98

goroutine 29 [garbage collection]:
runtime.systemstack_switch()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:216 fp=0xc8201c5498 sp=0xc8201c5490
runtime.gc(0x0)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1096 +0x3d0 fp=0xc8201c5790 sp=0xc8201c5498
runtime.backgroundgc()
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:896 +0x3d fp=0xc8201c57c0 sp=0xc8201c5790
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c57c8 sp=0xc8201c57c0
created by runtime.startGC
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:869 +0x186

goroutine 1 [chan receive, 2725 minutes]:
runtime.gopark(0xcea0d8, 0xc8855e6538, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8a4f6d958 sp=0xc8a4f6d930
runtime.goparkunlock(0xc8855e6538, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc8a4f6d990 sp=0xc8a4f6d958
runtime.chanrecv(0x988b80, 0xc8855e64e0, 0x0, 0xc87d358b01, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:505 +0x901 fp=0xc8a4f6da50 sp=0xc8a4f6d990
runtime.chanrecv1(0x988b80, 0xc8855e64e0, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:349 +0x2b fp=0xc8a4f6da80 sp=0xc8a4f6da50
main.(*Main).Run(0xc8a4f6df10, 0xc82000a0b0, 0x4, 0x4, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/main.go:91 +0xa16 fp=0xc8a4f6ddf8 sp=0xc8a4f6da80
main.main()
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/main.go:42 +0x29b fp=0xc8a4f6df50 sp=0xc8a4f6ddf8
runtime.main()
        /root/.gvm/gos/go1.5/src/runtime/proc.go:111 +0x2b0 fp=0xc8a4f6dfa0 sp=0xc8a4f6df50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8a4f6dfa8 sp=0xc8a4f6dfa0

goroutine 17 [syscall, 2726 minutes, locked to thread]:
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820066fb8 sp=0xc820066fb0

goroutine 2 [force gc (idle), 2 minutes]:
runtime.gopark(0xcea0d8, 0xfc1340, 0xbbc690, 0xf, 0x14, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820052758 sp=0xc820052730
runtime.goparkunlock(0xfc1340, 0xbbc690, 0xf, 0xc820000114, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc820052790 sp=0xc820052758
runtime.forcegchelper()
        /root/.gvm/gos/go1.5/src/runtime/proc.go:152 +0xb8 fp=0xc8200527c0 sp=0xc820052790
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200527c8 sp=0xc8200527c0
created by runtime.init.4
        /root/.gvm/gos/go1.5/src/runtime/proc.go:141 +0x2b

goroutine 3 [GC sweep wait, 2 minutes]:
runtime.gopark(0xcea0d8, 0xfc1560, 0xbb2630, 0xd, 0x454c14, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820052f48 sp=0xc820052f20
runtime.goparkunlock(0xfc1560, 0xbb2630, 0xd, 0x14, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc820052f80 sp=0xc820052f48
runtime.bgsweep(0xc82000c070)
        /root/.gvm/gos/go1.5/src/runtime/mgcsweep.go:67 +0x14d fp=0xc820052fb8 sp=0xc820052f80
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820052fc0 sp=0xc820052fb8
created by runtime.gcenable
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:205 +0x53

goroutine 4 [finalizer wait, 2726 minutes]:
runtime.gopark(0xcea0d8, 0xfe71d0, 0xbbc100, 0xe, 0x14, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820053718 sp=0xc8200536f0
runtime.goparkunlock(0xfe71d0, 0xbbc100, 0xe, 0x14, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc820053750 sp=0xc820053718
runtime.runfinq()
        /root/.gvm/gos/go1.5/src/runtime/mfinal.go:154 +0xaa fp=0xc8200537c0 sp=0xc820053750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200537c8 sp=0xc8200537c0
created by runtime.createfing
        /root/.gvm/gos/go1.5/src/runtime/mfinal.go:135 +0x60

goroutine 5 [syscall, 2726 minutes]:
runtime.notetsleepg(0xfe75a0, 0xffffffffffffffff, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/lock_futex.go:202 +0x4e fp=0xc820053f40 sp=0xc820053f18
runtime.signal_recv(0x0)
        /root/.gvm/gos/go1.5/src/runtime/sigqueue.go:111 +0x132 fp=0xc820053f78 sp=0xc820053f40
os/signal.loop()
        /root/.gvm/gos/go1.5/src/os/signal/signal_unix.go:22 +0x18 fp=0xc820053fc0 sp=0xc820053f78
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820053fc8 sp=0xc820053fc0
created by os/signal.init.1
        /root/.gvm/gos/go1.5/src/os/signal/signal_unix.go:28 +0x37

goroutine 50 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82060c480, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820054750 sp=0xc820054728
runtime.gcBgMarkWorker(0xc820023500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8200547b8 sp=0xc820054750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200547c0 sp=0xc8200547b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 19 [IO wait, 2726 minutes]:
runtime.gopark(0xcea098, 0x7ff62ee6f040, 0xba4ef0, 0x7, 0x1b, 0x5)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820067b40 sp=0xc820067b18
runtime.netpollblock(0x7ff62ee6f018, 0x72, 0x7ff630eb1050)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:338 +0x15e fp=0xc820067b88 sp=0xc820067b40
net.runtime_pollWait(0x7ff62ee6f018, 0x72, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60 fp=0xc820067ba8 sp=0xc820067b88
net.(*pollDesc).Wait(0xc8201505a0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a fp=0xc820067bc8 sp=0xc820067ba8
net.(*pollDesc).WaitRead(0xc8201505a0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36 fp=0xc820067bf0 sp=0xc820067bc8
net.(*netFD).accept(0xc820150540, 0x0, 0x7ff630eb5818, 0xc8200110e0)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c fp=0xc820067cf0 sp=0xc820067bf0
net.(*TCPListener).AcceptTCP(0xc82013c0a0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d fp=0xc820067d50 sp=0xc820067cf0
net.(*TCPListener).Accept(0xc82013c0a0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d fp=0xc820067d90 sp=0xc820067d50
github.com/influxdb/influxdb/tcp.(*Mux).Serve(0xc820199140, 0x7ff62ee6e098, 0xc82013c0a0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:45 +0x5d fp=0xc820067f88 sp=0xc820067d90
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820067f90 sp=0xc820067f88
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:313 +0xa81

goroutine 20 [chan receive, 2726 minutes]:
runtime.gopark(0xcea0d8, 0xc820148d78, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc82004ed00 sp=0xc82004ecd8
runtime.goparkunlock(0xc820148d78, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc82004ed38 sp=0xc82004ed00
runtime.chanrecv(0x9888e0, 0xc820148d20, 0xc82004ee80, 0x1, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:448 +0x472 fp=0xc82004edf8 sp=0xc82004ed38
runtime.chanrecv2(0x9888e0, 0xc820148d20, 0xc82004ee80, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:354 +0x2b fp=0xc82004ee28 sp=0xc82004edf8
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82013c0a8, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f fp=0xc82004ee98 sp=0xc82004ee28
github.com/influxdb/influxdb/meta.(*raftLayer).Accept(0xc8201992c0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:2058 +0x51 fp=0xc82004eed8 sp=0xc82004ee98
github.com/hashicorp/raft.(*NetworkTransport).listen(0xc820150620)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:346 +0x79 fp=0xc82004efb8 sp=0xc82004eed8
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82004efc0 sp=0xc82004efb8
created by github.com/hashicorp/raft.NewNetworkTransport
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/net_transport.go:138 +0x284

goroutine 21 [select]:
runtime.gopark(0xcea110, 0xccfa6dfbb0, 0xbaba58, 0x6, 0x118, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xccfa6df810 sp=0xccfa6df7e8
runtime.selectgoImpl(0xccfa6dfbb0, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xccfa6df9b8 sp=0xccfa6df810
runtime.selectgo(0xccfa6dfbb0)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xccfa6df9d8 sp=0xccfa6df9b8
github.com/hashicorp/raft.(*Raft).leaderLoop(0xc8201b2000)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:826 +0xcce fp=0xccfa6dfda0 sp=0xccfa6df9d8
github.com/hashicorp/raft.(*Raft).runLeader(0xc8201b2000)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:798 +0x88e fp=0xccfa6dff38 sp=0xccfa6dfda0
github.com/hashicorp/raft.(*Raft).run(0xc8201b2000)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:595 +0xbb fp=0xccfa6dff60 sp=0xccfa6dff38
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.run)-fm()
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:245 +0x20 fp=0xccfa6dff70 sp=0xccfa6dff60
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8201b2000, 0xc8201ae1b0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:152 +0x44 fp=0xccfa6dff90 sp=0xccfa6dff70
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xccfa6dff98 sp=0xccfa6dff90
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 22 [select, 6 minutes]:
runtime.gopark(0xcea110, 0xc8208efe58, 0xbaba58, 0x6, 0x25dac818, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8208ef9b0 sp=0xc8208ef988
runtime.selectgoImpl(0xc8208efe58, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc8208efb58 sp=0xc8208ef9b0
runtime.selectgo(0xc8208efe58)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc8208efb78 sp=0xc8208efb58
github.com/hashicorp/raft.(*Raft).runFSM(0xc8201b2000)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:502 +0x10eb fp=0xc8208eff60 sp=0xc8208efb78
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runFSM)-fm()
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:246 +0x20 fp=0xc8208eff70 sp=0xc8208eff60
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8201b2000, 0xc8201ae1c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:152 +0x44 fp=0xc8208eff90 sp=0xc8208eff70
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8208eff98 sp=0xc8208eff90
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 23 [select, 4 minutes]:
runtime.gopark(0xcea110, 0xc889720ea0, 0xbaba58, 0x6, 0xcedd778918, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc889720bf0 sp=0xc889720bc8
runtime.selectgoImpl(0xc889720ea0, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc889720d98 sp=0xc889720bf0
runtime.selectgo(0xc889720ea0)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc889720db8 sp=0xc889720d98
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc8201b2000)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:1613 +0x493 fp=0xc889720f70 sp=0xc889720db8
github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/raft.runSnapshots)-fm()
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/raft.go:247 +0x20 fp=0xc889720f80 sp=0xc889720f70
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc8201b2000, 0xc8201ae1d0)
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:152 +0x44 fp=0xc889720fa0 sp=0xc889720f80
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc889720fa8 sp=0xc889720fa0
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.SXplMOFgOF/src/github.com/hashicorp/raft/state.go:153 +0x4d

goroutine 24 [select, 2726 minutes]:
runtime.gopark(0xcea110, 0xc820065f10, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820065c50 sp=0xc820065c28
runtime.selectgoImpl(0xc820065f10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc820065df8 sp=0xc820065c50
runtime.selectgo(0xc820065f10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc820065e18 sp=0xc820065df8
github.com/influxdb/influxdb/meta.(*localRaft).logLeaderChanges(0xc820146f00)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/state.go:187 +0x4f5 fp=0xc820065fa8 sp=0xc820065e18
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820065fb0 sp=0xc820065fa8
created by github.com/influxdb/influxdb/meta.(*localRaft).open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/state.go:177 +0x1344

goroutine 25 [chan receive, 2726 minutes]:
runtime.gopark(0xcea0d8, 0xc820148e38, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820051570 sp=0xc820051548
runtime.goparkunlock(0xc820148e38, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc8200515a8 sp=0xc820051570
runtime.chanrecv(0x9888e0, 0xc820148de0, 0xc8200516f0, 0x1, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:448 +0x472 fp=0xc820051668 sp=0xc8200515a8
runtime.chanrecv2(0x9888e0, 0xc820148de0, 0xc8200516f0, 0xc8200516d0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:354 +0x2b fp=0xc820051698 sp=0xc820051668
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82013c0b0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f fp=0xc820051708 sp=0xc820051698
github.com/influxdb/influxdb/meta.(*Store).serveExecListener(0xc820154280)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:619 +0xa2 fp=0xc8200517b8 sp=0xc820051708
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200517c0 sp=0xc8200517b8
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:239 +0x364

goroutine 26 [chan receive, 2726 minutes]:
runtime.gopark(0xcea0d8, 0xc820148e98, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820051d70 sp=0xc820051d48
runtime.goparkunlock(0xc820148e98, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc820051da8 sp=0xc820051d70
runtime.chanrecv(0x9888e0, 0xc820148e40, 0xc820051ef0, 0x1, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:448 +0x472 fp=0xc820051e68 sp=0xc820051da8
runtime.chanrecv2(0x9888e0, 0xc820148e40, 0xc820051ef0, 0xc820051ed0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:354 +0x2b fp=0xc820051e98 sp=0xc820051e68
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82013c0b8, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f fp=0xc820051f08 sp=0xc820051e98
github.com/influxdb/influxdb/meta.(*Store).serveRPCListener(0xc820154280)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:723 +0xa2 fp=0xc820051fb8 sp=0xc820051f08
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820051fc0 sp=0xc820051fb8
created by github.com/influxdb/influxdb/meta.(*Store).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/meta/store.go:242 +0x3b6

goroutine 6954344 [semacquire, 32 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc946790950 sp=0xc946790928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc946790988 sp=0xc946790950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xc9467909e8 sp=0xc946790988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xc946790a00 sp=0xc9467909e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xc946790a20 sp=0xc946790a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xce6cf52000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xc946790b78 sp=0xc946790a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xce6cf52000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xc946790c28 sp=0xc946790b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xce6cf52000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xc946790e50 sp=0xc946790c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xce6cf52000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xc946790e98 sp=0xc946790e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xccb2f4d1af, 0x7, 0xc82019a7e0, 0x7, 0xc9d4208de0, 0x1, 0x1, 0x1, 0xce6cf52000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xc946790f50 sp=0xc946790e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc946790f58 sp=0xc946790f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 28 [select, 2726 minutes]:
runtime.gopark(0xcea110, 0xc8201c4f18, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c4ce8 sp=0xc8201c4cc0
runtime.selectgoImpl(0xc8201c4f18, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc8201c4e90 sp=0xc8201c4ce8
runtime.selectgo(0xc8201c4f18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc8201c4eb0 sp=0xc8201c4e90
github.com/influxdb/influxdb/cmd/influxd/run.(*Server).monitorErrorChan(0xc820156380, 0xc8201486c0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:442 +0x145 fp=0xc8201c4fb0 sp=0xc8201c4eb0
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c4fb8 sp=0xc8201c4fb0
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Server).Open.func1
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/server.go:319 +0xbff

goroutine 7 [syscall]:
runtime.notetsleepg(0xfc1598, 0x11ec697b, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/lock_futex.go:202 +0x4e fp=0xc82004f738 sp=0xc82004f710
runtime.timerproc()
        /root/.gvm/gos/go1.5/src/runtime/time.go:209 +0xde fp=0xc82004f7c0 sp=0xc82004f738
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82004f7c8 sp=0xc82004f7c0
created by runtime.addtimerLocked
        /root/.gvm/gos/go1.5/src/runtime/time.go:116 +0x11f

goroutine 30 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8205f5b00, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c5f50 sp=0xc8201c5f28
runtime.gcBgMarkWorker(0xc820016000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c5fb8 sp=0xc8201c5f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c5fc0 sp=0xc8201c5fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 34 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82060c000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c0750 sp=0xc8201c0728
runtime.gcBgMarkWorker(0xc820017500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c07b8 sp=0xc8201c0750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c07c0 sp=0xc8201c07b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 35 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8201ccd80, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c0f50 sp=0xc8201c0f28
runtime.gcBgMarkWorker(0xc820018a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c0fb8 sp=0xc8201c0f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c0fc0 sp=0xc8201c0fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 36 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc88c4a8000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c1750 sp=0xc8201c1728
runtime.gcBgMarkWorker(0xc82001a000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c17b8 sp=0xc8201c1750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c17c0 sp=0xc8201c17b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 37 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82704c000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c1f50 sp=0xc8201c1f28
runtime.gcBgMarkWorker(0xc82001b500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c1fb8 sp=0xc8201c1f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c1fc0 sp=0xc8201c1fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 38 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc88c490000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c2750 sp=0xc8201c2728
runtime.gcBgMarkWorker(0xc82001ca00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c27b8 sp=0xc8201c2750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c27c0 sp=0xc8201c27b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 39 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820600d80, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c2f50 sp=0xc8201c2f28
runtime.gcBgMarkWorker(0xc82001e000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c2fb8 sp=0xc8201c2f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c2fc0 sp=0xc8201c2fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 40 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8205f5680, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c3750 sp=0xc8201c3728
runtime.gcBgMarkWorker(0xc82001f500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c37b8 sp=0xc8201c3750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c37c0 sp=0xc8201c37b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 41 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8201e4480, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c3f50 sp=0xc8201c3f28
runtime.gcBgMarkWorker(0xc820020a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c3fb8 sp=0xc8201c3f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c3fc0 sp=0xc8201c3fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 42 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820600480, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205de750 sp=0xc8205de728
runtime.gcBgMarkWorker(0xc820022000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205de7b8 sp=0xc8205de750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205de7c0 sp=0xc8205de7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 31 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc9381e2000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c6750 sp=0xc8201c6728
runtime.gcBgMarkWorker(0xc820024a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c67b8 sp=0xc8201c6750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c67c0 sp=0xc8201c67b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 32 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820059b00, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c6f50 sp=0xc8201c6f28
runtime.gcBgMarkWorker(0xc820026000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c6fb8 sp=0xc8201c6f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c6fc0 sp=0xc8201c6fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 33 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82008c900, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c7750 sp=0xc8201c7728
runtime.gcBgMarkWorker(0xc820027500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c77b8 sp=0xc8201c7750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c77c0 sp=0xc8201c77b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 66 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820601b00, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8201c7f50 sp=0xc8201c7f28
runtime.gcBgMarkWorker(0xc820028a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8201c7fb8 sp=0xc8201c7f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8201c7fc0 sp=0xc8201c7fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 67 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82060d680, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205da750 sp=0xc8205da728
runtime.gcBgMarkWorker(0xc82002a000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205da7b8 sp=0xc8205da750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205da7c0 sp=0xc8205da7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 68 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc88c4b4000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205daf50 sp=0xc8205daf28
runtime.gcBgMarkWorker(0xc82002b500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205dafb8 sp=0xc8205daf50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205dafc0 sp=0xc8205dafb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 69 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8205f4900, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205db750 sp=0xc8205db728
runtime.gcBgMarkWorker(0xc82002ca00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205db7b8 sp=0xc8205db750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205db7c0 sp=0xc8205db7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 51 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8205f4000, 0xc28540, 0x12, 0x100000014, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820050750 sp=0xc820050728
runtime.gcBgMarkWorker(0xc82002e000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8200507b8 sp=0xc820050750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200507c0 sp=0xc8200507b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 52 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8e3de6000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820050f50 sp=0xc820050f28
runtime.gcBgMarkWorker(0xc82002f500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc820050fb8 sp=0xc820050f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820050fc0 sp=0xc820050fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 53 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8201cc480, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e8750 sp=0xc8205e8728
runtime.gcBgMarkWorker(0xc820030a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e87b8 sp=0xc8205e8750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e87c0 sp=0xc8205e87b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 54 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82008c000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e8f50 sp=0xc8205e8f28
runtime.gcBgMarkWorker(0xc820032000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e8fb8 sp=0xc8205e8f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e8fc0 sp=0xc8205e8fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 43 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820601680, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205def50 sp=0xc8205def28
runtime.gcBgMarkWorker(0xc820033500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205defb8 sp=0xc8205def50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205defc0 sp=0xc8205defb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 44 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8205f4480, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205df750 sp=0xc8205df728
runtime.gcBgMarkWorker(0xc820034a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205df7b8 sp=0xc8205df750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205df7c0 sp=0xc8205df7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 45 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8201e4000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205dff50 sp=0xc8205dff28
runtime.gcBgMarkWorker(0xc820036000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205dffb8 sp=0xc8205dff50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205dffc0 sp=0xc8205dffb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 70 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820601200, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205dbf50 sp=0xc8205dbf28
runtime.gcBgMarkWorker(0xc820037500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205dbfb8 sp=0xc8205dbf50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205dbfc0 sp=0xc8205dbfb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 46 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8201cc000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e0750 sp=0xc8205e0728
runtime.gcBgMarkWorker(0xc820038a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e07b8 sp=0xc8205e0750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e07c0 sp=0xc8205e07b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 71 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc88c49c000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205dc750 sp=0xc8205dc728
runtime.gcBgMarkWorker(0xc82003a000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205dc7b8 sp=0xc8205dc750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205dc7c0 sp=0xc8205dc7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 72 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8205f5200, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205dcf50 sp=0xc8205dcf28
runtime.gcBgMarkWorker(0xc82003b500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205dcfb8 sp=0xc8205dcf50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205dcfc0 sp=0xc8205dcfb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 55 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820600000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e9750 sp=0xc8205e9728
runtime.gcBgMarkWorker(0xc82003ca00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e97b8 sp=0xc8205e9750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e97c0 sp=0xc8205e97b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 73 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82060db00, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205dd750 sp=0xc8205dd728
runtime.gcBgMarkWorker(0xc82003e000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205dd7b8 sp=0xc8205dd750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205dd7c0 sp=0xc8205dd7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 74 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8f7be3680, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205ddf50 sp=0xc8205ddf28
runtime.gcBgMarkWorker(0xc82003f500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205ddfb8 sp=0xc8205ddf50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205ddfc0 sp=0xc8205ddfb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 75 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc88c4c0000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e4750 sp=0xc8205e4728
runtime.gcBgMarkWorker(0xc820040a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e47b8 sp=0xc8205e4750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e47c0 sp=0xc8205e47b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 76 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820058d80, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e4f50 sp=0xc8205e4f28
runtime.gcBgMarkWorker(0xc820042000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e4fb8 sp=0xc8205e4f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e4fc0 sp=0xc8205e4fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 77 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc820059680, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e5750 sp=0xc8205e5728
runtime.gcBgMarkWorker(0xc820043500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e57b8 sp=0xc8205e5750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e57c0 sp=0xc8205e57b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 78 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc8dc92c900, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e5f50 sp=0xc8205e5f28
runtime.gcBgMarkWorker(0xc820044a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e5fb8 sp=0xc8205e5f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e5fc0 sp=0xc8205e5fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 56 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82060c900, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e9f50 sp=0xc8205e9f28
runtime.gcBgMarkWorker(0xc820046000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e9fb8 sp=0xc8205e9f50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e9fc0 sp=0xc8205e9fb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 79 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82012e480, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205e6750 sp=0xc8205e6728
runtime.gcBgMarkWorker(0xc820047500)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205e67b8 sp=0xc8205e6750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205e67c0 sp=0xc8205e67b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 57 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82012e000, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205ea750 sp=0xc8205ea728
runtime.gcBgMarkWorker(0xc820048a00)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205ea7b8 sp=0xc8205ea750
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205ea7c0 sp=0xc8205ea7b8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 58 [mark worker (idle)]:
runtime.gopark(0xce9f70, 0xc82060cd80, 0xc28540, 0x12, 0x14, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205eaf50 sp=0xc8205eaf28
runtime.gcBgMarkWorker(0xc82004a000)
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1288 +0xf7 fp=0xc8205eafb8 sp=0xc8205eaf50
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205eafc0 sp=0xc8205eafb8
created by runtime.gcBgMarkStartWorkers
        /root/.gvm/gos/go1.5/src/runtime/mgc.go:1238 +0x93

goroutine 6952318 [semacquire, 40 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcb0ff80950 sp=0xcb0ff80928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xcb0ff80988 sp=0xcb0ff80950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xcb0ff809e8 sp=0xcb0ff80988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xcb0ff80a00 sp=0xcb0ff809e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xcb0ff80a20 sp=0xcb0ff80a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xc89ed48000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xcb0ff80b78 sp=0xcb0ff80a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xc89ed48000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xcb0ff80c28 sp=0xcb0ff80b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xc89ed48000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xcb0ff80e50 sp=0xcb0ff80c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xc89ed48000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xcb0ff80e98 sp=0xcb0ff80e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xd0c52c3ddf, 0x7, 0xc82019a7e0, 0x7, 0xcde973b800, 0x1, 0x1, 0x1, 0xc89ed48000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xcb0ff80f50 sp=0xcb0ff80e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcb0ff80f58 sp=0xcb0ff80f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 6952601 [semacquire, 38 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc9332df950 sp=0xc9332df928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc9332df988 sp=0xc9332df950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xc9332df9e8 sp=0xc9332df988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xc9332dfa00 sp=0xc9332df9e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xc9332dfa20 sp=0xc9332dfa00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xc8f425e000, 0x100c, 0x1200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xc9332dfb78 sp=0xc9332dfa20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xc8f425e000, 0x100c, 0x1200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xc9332dfc28 sp=0xc9332dfb78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xc8f425e000, 0x100c, 0x1200, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xc9332dfe50 sp=0xc9332dfc28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xc8f425e000, 0x100c, 0x1200, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xc9332dfe98 sp=0xc9332dfe50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcb7d8c92cf, 0x7, 0xc82019a7e0, 0x7, 0xcd409f2240, 0x1, 0x1, 0x1, 0xc8f425e000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xc9332dff50 sp=0xc9332dfe98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc9332dff58 sp=0xc9332dff50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 15 [select]:
runtime.gopark(0xcea110, 0xc8e8168f50, 0xbaba58, 0x6, 0xc7bd4dcb18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8e8168c28 sp=0xc8e8168c00
runtime.selectgoImpl(0xc8e8168f50, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc8e8168dd0 sp=0xc8e8168c28
runtime.selectgo(0xc8e8168f50)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc8e8168df0 sp=0xc8e8168dd0
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc820150380, 0xd475c6c580, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x518 fp=0xc8e8168fe8 sp=0xc8e8168df0
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine(0xc8201504d0, 0x7fef007ef690, 0xcd33421ae0, 0xd27ad87260, 0xd476418000, 0x15228, 0x1fe00, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:491 +0xeb0 fp=0xc8e8169228 sp=0xc8e8168fe8
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite(0xc8201504d0, 0x7fef007ef690, 0xcd33421ae0, 0xd27ad87260, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:346 +0x722 fp=0xc8e8169358 sp=0xc8e8169228
github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveWrite)-fm(0x7fef007ef690, 0xcd33421ae0, 0xd27ad87260, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:103 +0x48 fp=0xc8e8169388 sp=0xc8e8169358
github.com/influxdb/influxdb/services/httpd.authenticate.func1(0x7fef007ef690, 0xcd33421ae0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:621 +0x70 fp=0xc8e8169468 sp=0xc8e8169388
net/http.HandlerFunc.ServeHTTP(0xc82019c8c0, 0x7fef007ef690, 0xcd33421ae0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e8169488 sp=0xc8e8169468
github.com/influxdb/influxdb/services/httpd.gzipFilter.func1(0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:679 +0x2af fp=0xc8e8169548 sp=0xc8e8169488
net/http.HandlerFunc.ServeHTTP(0xc82019c8e0, 0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e8169568 sp=0xc8e8169548
github.com/influxdb/influxdb/services/httpd.versionHeader.func1(0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:688 +0xc0 fp=0xc8e81695b0 sp=0xc8e8169568
net/http.HandlerFunc.ServeHTTP(0xc82019c900, 0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e81695d0 sp=0xc8e81695b0
github.com/influxdb/influxdb/services/httpd.cors.func1(0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:721 +0x328 fp=0xc8e8169720 sp=0xc8e81695d0
net/http.HandlerFunc.ServeHTTP(0xc82019c920, 0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e8169740 sp=0xc8e8169720
github.com/influxdb/influxdb/services/httpd.requestID.func1(0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:731 +0x15d fp=0xc8e81697a8 sp=0xc8e8169740
net/http.HandlerFunc.ServeHTTP(0xc82019c940, 0x7fef007ef5b8, 0xcd33421aa0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e81697c8 sp=0xc8e81697a8
github.com/influxdb/influxdb/services/httpd.logging.func1(0x7fef007ef5b8, 0xcd33421a80, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:739 +0x10b fp=0xc8e81698b0 sp=0xc8e81697c8
net/http.HandlerFunc.ServeHTTP(0xc82019c960, 0x7fef007ef5b8, 0xcd33421a80, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e81698d0 sp=0xc8e81698b0
github.com/influxdb/influxdb/services/httpd.recovery.func1(0x7fef007ef580, 0xce5ef4d4a0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:749 +0x114 fp=0xc8e81699e8 sp=0xc8e81698d0
net/http.HandlerFunc.ServeHTTP(0xc82019c980, 0x7fef007ef580, 0xce5ef4d4a0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xc8e8169a08 sp=0xc8e81699e8
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc82013c060, 0x7fef007ef580, 0xce5ef4d4a0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/bmizerany/pat/mux.go:109 +0x244 fp=0xc8e8169b80 sp=0xc8e8169a08
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc8201504d0, 0x7fef007ef580, 0xce5ef4d4a0, 0xd27ad87260)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:167 +0x243 fp=0xc8e8169be0 sp=0xc8e8169b80
net/http.serverHandler.ServeHTTP(0xc885455b00, 0x7fef007ef580, 0xce5ef4d4a0, 0xd27ad87260)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1862 +0x19e fp=0xc8e8169c40 sp=0xc8e8169be0
net/http.(*conn).serve(0xc835220000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1361 +0xbee fp=0xc8e8169f98 sp=0xc8e8169c40
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8e8169fa0 sp=0xc8e8169f98
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6

goroutine 59 [runnable]:
bytes.(*Buffer).Write(0xd32e3cc230, 0xd3aa538b86, 0x9, 0x5381a0, 0x51, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/bytes/buffer.go:129 fp=0xd093c556c0 sp=0xd093c556b8
github.com/influxdb/influxdb/tsdb/engine/wal.marshalWALEntry(0xd32e3cc230, 0xd3aa538b35, 0x51, 0x5381f1, 0x13ff5b62a41af200, 0xd3aa538b86, 0x9, 0x5381a0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:1635 +0x3d0 fp=0xd093c55768 sp=0xd093c556c0
github.com/influxdb/influxdb/tsdb/engine/wal.(*Partition).writeCompactionEntry(0xc8842560f0, 0xd32c561ca0, 0xd32e2b45c0, 0x37, 0xd3da6a8000, 0xaecc3b, 0xca2400, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:1188 +0x160 fp=0xd093c558a0 sp=0xd093c55768
github.com/influxdb/influxdb/tsdb/engine/wal.(*Partition).compactFiles(0xc8842560f0, 0xd088f09cc0, 0x1, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:1152 +0x608 fp=0xd093c55ae0 sp=0xd093c558a0
github.com/influxdb/influxdb/tsdb/engine/wal.(*Partition).flushAndCompact(0xc8842560f0, 0x1, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:1095 +0x798 fp=0xd093c55c60 sp=0xd093c55ae0
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).triggerAutoFlush(0xc82009a270)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:587 +0x15b fp=0xd093c55d58 sp=0xd093c55c60
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc82009a270, 0xc8842b8060)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:608 +0x17d fp=0xd093c55f40 sp=0xd093c55d58
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd093c55f48 sp=0xd093c55f40
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:219 +0x7c3

goroutine 6957497 [semacquire, 17 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xd306b3e950 sp=0xd306b3e928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xd306b3e988 sp=0xd306b3e950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xd306b3e9e8 sp=0xd306b3e988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xd306b3ea00 sp=0xd306b3e9e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xd306b3ea20 sp=0xd306b3ea00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xd1e784c000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xd306b3eb78 sp=0xd306b3ea20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xd1e784c000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xd306b3ec28 sp=0xd306b3eb78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xd1e784c000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xd306b3ee50 sp=0xd306b3ec28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xd1e784c000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xd306b3ee98 sp=0xd306b3ee50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xd1a135633f, 0x7, 0xc82019a7e0, 0x7, 0xccc512eae0, 0x1, 0x1, 0x1, 0xd1e784c000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xd306b3ef50 sp=0xd306b3ee98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd306b3ef58 sp=0xd306b3ef50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab
goroutine 10 [IO wait, 2442 minutes]:
runtime.gopark(0xcea098, 0x7ff62ee6ed40, 0xba4ef0, 0x7, 0x4a011b, 0x5)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8a4f6f4f0 sp=0xc8a4f6f4c8
runtime.netpollblock(0x7ff62ee6ed18, 0xffffff0000000072, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:338 +0x15e fp=0xc8a4f6f538 sp=0xc8a4f6f4f0
net.runtime_pollWait(0x7ff62ee6ed18, 0x72, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60 fp=0xc8a4f6f558 sp=0xc8a4f6f538
net.(*pollDesc).Wait(0xc885590a70, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a fp=0xc8a4f6f578 sp=0xc8a4f6f558
net.(*pollDesc).WaitRead(0xc885590a70, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36 fp=0xc8a4f6f5a0 sp=0xc8a4f6f578
net.(*netFD).Read(0xc885590a10, 0xc882d16000, 0x1000, 0x1000, 0x0, 0x7ff630eb1050, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:232 +0x23a fp=0xc8a4f6f670 sp=0xc8a4f6f5a0
net.(*conn).Read(0xc88493e000, 0xc882d16000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/net.go:172 +0xe4 fp=0xc8a4f6f6d8 sp=0xc8a4f6f670
net/http.(*liveSwitchReader).Read(0xc820104048, 0xc882d16000, 0x1000, 0x1000, 0xc8312a5c80, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:219 +0xa4 fp=0xc8a4f6f728 sp=0xc8a4f6f6d8
io.(*LimitedReader).Read(0xc87df475c0, 0xc882d16000, 0x1000, 0x1000, 0x20f, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/io/io.go:427 +0xbd fp=0xc8a4f6f778 sp=0xc8a4f6f728
bufio.(*Reader).fill(0xc8854d9800)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:97 +0x1e9 fp=0xc8a4f6f838 sp=0xc8a4f6f778
bufio.(*Reader).ReadSlice(0xc8854d9800, 0xc8175a3a0a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:328 +0x21a fp=0xc8a4f6f880 sp=0xc8a4f6f838
bufio.(*Reader).ReadLine(0xc8854d9800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/bufio/bufio.go:357 +0x53 fp=0xc8a4f6f8d0 sp=0xc8a4f6f880
net/textproto.(*Reader).readLineSlice(0xc8e8342db0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/textproto/reader.go:55 +0x81 fp=0xc8a4f6f978 sp=0xc8a4f6f8d0
net/textproto.(*Reader).ReadLine(0xc8e8342db0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/textproto/reader.go:36 +0x40 fp=0xc8a4f6f9d8 sp=0xc8a4f6f978
net/http.ReadRequest(0xc8854d9800, 0xc934b8a000, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/request.go:653 +0xb6 fp=0xc8a4f6fb50 sp=0xc8a4f6f9d8
net/http.(*conn).readRequest(0xc820104000, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:633 +0x32f fp=0xc8a4f6fc40 sp=0xc8a4f6fb50
net/http.(*conn).serve(0xc820104000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1319 +0x727 fp=0xc8a4f6ff98 sp=0xc8a4f6fc40
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8a4f6ffa0 sp=0xc8a4f6ff98
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6

goroutine 6951178 [semacquire, 44 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcc40ba1950 sp=0xcc40ba1928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xcc40ba1988 sp=0xcc40ba1950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xcc40ba19e8 sp=0xcc40ba1988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xcc40ba1a00 sp=0xcc40ba19e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xcc40ba1a20 sp=0xcc40ba1a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xcdd48c0000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xcc40ba1b78 sp=0xcc40ba1a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xcdd48c0000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xcc40ba1c28 sp=0xcc40ba1b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xcdd48c0000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xcc40ba1e50 sp=0xcc40ba1c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xcdd48c0000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xcc40ba1e98 sp=0xcc40ba1e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcbd5e1427f, 0x7, 0xc82019a7e0, 0x7, 0xc89ec6d860, 0x1, 0x1, 0x1, 0xcdd48c0000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xcc40ba1f50 sp=0xcc40ba1e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcc40ba1f58 sp=0xcc40ba1f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 47 [select, 2723 minutes]:
runtime.gopark(0xcea110, 0xc88428ce80, 0xbaba58, 0x6, 0xc81ccc7b18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc88428cc18 sp=0xc88428cbf0
runtime.selectgoImpl(0xc88428ce80, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc88428cdc0 sp=0xc88428cc18
runtime.selectgo(0xc88428ce80)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc88428cde0 sp=0xc88428cdc0
github.com/influxdb/influxdb/tsdb.(*PointBatcher).Start.func2(0xc820148c60, 0xc87d3e6de0, 0xc87d3e6e00, 0xc884ec3fc8, 0xc884ec3fc0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/batcher.go:69 +0x522 fp=0xc88428cf88 sp=0xc88428cde0
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88428cf90 sp=0xc88428cf88
created by github.com/influxdb/influxdb/tsdb.(*PointBatcher).Start
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/batcher.go:104 +0x17e

goroutine 6954951 [semacquire, 27 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xca9720c950 sp=0xca9720c928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xca9720c988 sp=0xca9720c950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xca9720c9e8 sp=0xca9720c988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xca9720ca00 sp=0xca9720c9e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xca9720ca20 sp=0xca9720ca00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xc902f48000, 0x110, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xca9720cb78 sp=0xca9720ca20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xc902f48000, 0x110, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xca9720cc28 sp=0xca9720cb78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xc902f48000, 0x110, 0x200, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xca9720ce50 sp=0xca9720cc28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xc902f48000, 0x110, 0x200, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xca9720ce98 sp=0xca9720ce50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcfd0b71e0f, 0x7, 0xc82019a7e0, 0x7, 0xcd3807b560, 0x1, 0x1, 0x1, 0xc902f48000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xca9720cf50 sp=0xca9720ce98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xca9720cf58 sp=0xca9720cf50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 65 [select]:
runtime.gopark(0xcea110, 0xc8d152ae98, 0xbaba58, 0x6, 0xc820059b18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8d152abf0 sp=0xc8d152abc8
runtime.selectgoImpl(0xc8d152ae98, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc8d152ad98 sp=0xc8d152abf0
runtime.selectgo(0xc8d152ae98)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc8d152adb8 sp=0xc8d152ad98
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).autoflusher(0xc88429c1a0, 0xc8855e6300)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:603 +0x567 fp=0xc8d152afa0 sp=0xc8d152adb8
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8d152afa8 sp=0xc8d152afa0
created by github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:219 +0x7c3

goroutine 98 [select]:
runtime.gopark(0xcea110, 0xcb6a225f10, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcb6a225ca8 sp=0xcb6a225c80
runtime.selectgoImpl(0xcb6a225f10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xcb6a225e50 sp=0xcb6a225ca8
runtime.selectgo(0xcb6a225f10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xcb6a225e70 sp=0xcb6a225e50
github.com/influxdb/influxdb/services/hh.(*Service).retryWrites(0xc820166120)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:103 +0x314 fp=0xcb6a225fa8 sp=0xcb6a225e70
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcb6a225fb0 sp=0xcb6a225fa8
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:67 +0x210

goroutine 99 [select, 25 minutes]:
runtime.gopark(0xcea110, 0xc88560ef10, 0xbaba58, 0x6, 0xc88560ec18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc88560eca8 sp=0xc88560ec80
runtime.selectgoImpl(0xc88560ef10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc88560ee50 sp=0xc88560eca8
runtime.selectgo(0xc88560ef10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc88560ee70 sp=0xc88560ee50
github.com/influxdb/influxdb/services/hh.(*Service).expireWrites(0xc820166120)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:121 +0x2d1 fp=0xc88560efa8 sp=0xc88560ee70
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88560efb0 sp=0xc88560efa8
created by github.com/influxdb/influxdb/services/hh.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/hh/service.go:68 +0x232

goroutine 100 [chan receive, 2725 minutes]:
runtime.gopark(0xcea0d8, 0xc820148ef8, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8205eb560 sp=0xc8205eb538
runtime.goparkunlock(0xc820148ef8, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc8205eb598 sp=0xc8205eb560
runtime.chanrecv(0x9888e0, 0xc820148ea0, 0xc8205eb6e0, 0x1, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:448 +0x472 fp=0xc8205eb658 sp=0xc8205eb598
runtime.chanrecv2(0x9888e0, 0xc820148ea0, 0xc8205eb6e0, 0xc8205eb6c0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:354 +0x2b fp=0xc8205eb688 sp=0xc8205eb658
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82013c0c0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f fp=0xc8205eb6f8 sp=0xc8205eb688
github.com/influxdb/influxdb/cluster.(*Service).serve(0xc8201503f0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/service.go:84 +0xdb fp=0xc8205eb7b8 sp=0xc8205eb6f8
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8205eb7c0 sp=0xc8205eb7b8
created by github.com/influxdb/influxdb/cluster.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/service.go:61 +0x135

goroutine 101 [select, 6 minutes]:
runtime.gopark(0xcea110, 0xce5e1c3f10, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xce5e1c3c60 sp=0xce5e1c3c38
runtime.selectgoImpl(0xce5e1c3f10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xce5e1c3e08 sp=0xce5e1c3c60
runtime.selectgo(0xce5e1c3f10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xce5e1c3e28 sp=0xce5e1c3e08
github.com/influxdb/influxdb/services/precreator.(*Service).runPrecreation(0xc820140bc0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/precreator/service.go:75 +0x44f fp=0xce5e1c3fa8 sp=0xce5e1c3e28
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xce5e1c3fb0 sp=0xce5e1c3fa8
created by github.com/influxdb/influxdb/services/precreator.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/precreator/service.go:53 +0x21f

goroutine 102 [chan receive, 2725 minutes]:
runtime.gopark(0xcea0d8, 0xc820148f58, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc88491dd38 sp=0xc88491dd10
runtime.goparkunlock(0xc820148f58, 0xbb98f0, 0xc, 0x17, 0x3)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc88491dd70 sp=0xc88491dd38
runtime.chanrecv(0x9888e0, 0xc820148f00, 0xc88491deb8, 0x1, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:448 +0x472 fp=0xc88491de30 sp=0xc88491dd70
runtime.chanrecv2(0x9888e0, 0xc820148f00, 0xc88491deb8, 0xc88491de98)
        /root/.gvm/gos/go1.5/src/runtime/chan.go:354 +0x2b fp=0xc88491de60 sp=0xc88491de30
github.com/influxdb/influxdb/tcp.(*listener).Accept(0xc82013c0c8, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tcp/mux.go:117 +0x6f fp=0xc88491ded0 sp=0xc88491de60
github.com/influxdb/influxdb/services/snapshotter.(*Service).serve(0xc820146780)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/snapshotter/service.go:75 +0xa9 fp=0xc88491dfb8 sp=0xc88491ded0
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88491dfc0 sp=0xc88491dfb8
created by github.com/influxdb/influxdb/services/snapshotter.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/snapshotter/service.go:48 +0x130

goroutine 103 [IO wait, 2725 minutes]:
runtime.gopark(0xcea098, 0x7ff62ee6ef80, 0xba4ef0, 0x7, 0x1b, 0x5)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820069b28 sp=0xc820069b00
runtime.netpollblock(0x7ff62ee6ef58, 0x72, 0x7ff630eb1050)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:338 +0x15e fp=0xc820069b70 sp=0xc820069b28
net.runtime_pollWait(0x7ff62ee6ef58, 0x72, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60 fp=0xc820069b90 sp=0xc820069b70
net.(*pollDesc).Wait(0xc8855e24c0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a fp=0xc820069bb0 sp=0xc820069b90
net.(*pollDesc).WaitRead(0xc8855e24c0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36 fp=0xc820069bd8 sp=0xc820069bb0
net.(*netFD).accept(0xc8855e2460, 0x0, 0x7ff630eb5818, 0xc87d3e39a0)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c fp=0xc820069cd8 sp=0xc820069bd8
net.(*TCPListener).AcceptTCP(0xc8854ce0a8, 0x455470, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d fp=0xc820069d38 sp=0xc820069cd8
net.(*TCPListener).Accept(0xc8854ce0a8, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d fp=0xc820069d78 sp=0xc820069d38
net/http.(*Server).Serve(0xc885455aa0, 0x7ff62ee6e098, 0xc8854ce0a8, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1887 +0xb3 fp=0xc820069e60 sp=0xc820069d78
net/http.Serve(0x7ff62ee6e098, 0xc8854ce0a8, 0x7fef000807f8, 0xc87d36cc90, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1765 +0x85 fp=0xc820069e98 sp=0xc820069e60
github.com/influxdb/influxdb/services/admin.(*Service).serve(0xc820146820)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/admin/service.go:107 +0x12f fp=0xc820069fa8 sp=0xc820069e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820069fb0 sp=0xc820069fa8
created by github.com/influxdb/influxdb/services/admin.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/admin/service.go:70 +0x491

goroutine 104 [select]:
runtime.gopark(0xcea110, 0xcfa7672ed8, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcfa7672c68 sp=0xcfa7672c40
runtime.selectgoImpl(0xcfa7672ed8, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xcfa7672e10 sp=0xcfa7672c68
runtime.selectgo(0xcfa7672ed8)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xcfa7672e30 sp=0xcfa7672e10
github.com/influxdb/influxdb/services/continuous_querier.(*Service).backgroundLoop(0xc820150460)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:157 +0x434 fp=0xcfa7672fa8 sp=0xcfa7672e30
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcfa7672fb0 sp=0xcfa7672fa8
created by github.com/influxdb/influxdb/services/continuous_querier.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/continuous_querier/service.go:93 +0x294

goroutine 105 [IO wait, 2719 minutes]:
runtime.gopark(0xcea098, 0x7ff62ee6eec0, 0xba4ef0, 0x7, 0x1b, 0x5)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc820064b60 sp=0xc820064b38
runtime.netpollblock(0x7ff62ee6ee98, 0x72, 0x7ff630eb1050)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:338 +0x15e fp=0xc820064ba8 sp=0xc820064b60
net.runtime_pollWait(0x7ff62ee6ee98, 0x72, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60 fp=0xc820064bc8 sp=0xc820064ba8
net.(*pollDesc).Wait(0xc8855e2530, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a fp=0xc820064be8 sp=0xc820064bc8
net.(*pollDesc).WaitRead(0xc8855e2530, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36 fp=0xc820064c10 sp=0xc820064be8
net.(*netFD).accept(0xc8855e24d0, 0x0, 0x7ff630eb5818, 0xc87deae400)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:408 +0x27c fp=0xc820064d10 sp=0xc820064c10
net.(*TCPListener).AcceptTCP(0xc8854ce0b0, 0xc820064dc0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:254 +0x4d fp=0xc820064d70 sp=0xc820064d10
net.(*TCPListener).Accept(0xc8854ce0b0, 0x0, 0x0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/tcpsock_posix.go:264 +0x3d fp=0xc820064db0 sp=0xc820064d70
net/http.(*Server).Serve(0xc885455b00, 0x7ff62ee6e098, 0xc8854ce0b0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1887 +0xb3 fp=0xc820064e98 sp=0xc820064db0
net/http.Serve(0x7ff62ee6e098, 0xc8854ce0b0, 0x7fef00080820, 0xc8201504d0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1765 +0x85 fp=0xc820064ed0 sp=0xc820064e98
github.com/influxdb/influxdb/services/httpd.(*Service).serve(0xc820146960)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/service.go:108 +0x6e fp=0xc820064fa8 sp=0xc820064ed0
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc820064fb0 sp=0xc820064fa8
created by github.com/influxdb/influxdb/services/httpd.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/service.go:76 +0x6ca

goroutine 106 [IO wait, 2723 minutes]:
runtime.gopark(0xcea098, 0x7ff62ee6ee00, 0xba4ef0, 0x7, 0x7ff630eb101b, 0x5)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc884286be8 sp=0xc884286bc0
runtime.netpollblock(0x7ff62ee6edd8, 0x72, 0x0)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:338 +0x15e fp=0xc884286c30 sp=0xc884286be8
net.runtime_pollWait(0x7ff62ee6edd8, 0x72, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/runtime/netpoll.go:157 +0x60 fp=0xc884286c50 sp=0xc884286c30
net.(*pollDesc).Wait(0xc8855e25a0, 0x72, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:73 +0x3a fp=0xc884286c70 sp=0xc884286c50
net.(*pollDesc).WaitRead(0xc8855e25a0, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/fd_poll_runtime.go:78 +0x36 fp=0xc884286c98 sp=0xc884286c70
net.(*netFD).readFrom(0xc8855e2540, 0xc82bb28000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x7ff630eb1050, 0xc82000e150)
        /root/.gvm/gos/go1.5/src/net/fd_unix.go:259 +0x2a5 fp=0xc884286d80 sp=0xc884286c98
net.(*UDPConn).ReadFromUDP(0xc8854ce0b8, 0xc82bb28000, 0x10000, 0x10000, 0x10000, 0x10000, 0x0, 0x0)
        /root/.gvm/gos/go1.5/src/net/udpsock_posix.go:61 +0x117 fp=0xc884286e60 sp=0xc884286d80
github.com/influxdb/influxdb/services/udp.(*Service).serve(0xc82015ed80)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/udp/service.go:115 +0x149 fp=0xc884286fa8 sp=0xc884286e60
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc884286fb0 sp=0xc884286fa8
created by github.com/influxdb/influxdb/services/udp.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/udp/service.go:72 +0x73b

goroutine 107 [select, 2723 minutes]:
runtime.gopark(0xcea110, 0xc88560ff10, 0xbaba58, 0x6, 0xc8261be018, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc88560fc68 sp=0xc88560fc40
runtime.selectgoImpl(0xc88560ff10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc88560fe10 sp=0xc88560fc68
runtime.selectgo(0xc88560ff10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc88560fe30 sp=0xc88560fe10
github.com/influxdb/influxdb/services/udp.(*Service).writePoints(0xc82015ed80)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/udp/service.go:82 +0x462 fp=0xc88560ffa8 sp=0xc88560fe30
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88560ffb0 sp=0xc88560ffa8
created by github.com/influxdb/influxdb/services/udp.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/udp/service.go:73 +0x760

goroutine 108 [select, 6 minutes]:
runtime.gopark(0xcea110, 0xc884280f10, 0xbaba58, 0x6, 0x2260ff9290018, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc884280ca8 sp=0xc884280c80
runtime.selectgoImpl(0xc884280f10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc884280e50 sp=0xc884280ca8
runtime.selectgo(0xc884280f10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc884280e70 sp=0xc884280e50
github.com/influxdb/influxdb/services/retention.(*Service).deleteShardGroups(0xc820146a50)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:69 +0x351 fp=0xc884280fa8 sp=0xc884280e70
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc884280fb0 sp=0xc884280fa8
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:45 +0x1b4

goroutine 109 [select, 6 minutes]:
runtime.gopark(0xcea110, 0xc88428df10, 0xbaba58, 0x6, 0x2260ff9290018, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc88428dbf0 sp=0xc88428dbc8
runtime.selectgoImpl(0xc88428df10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc88428dd98 sp=0xc88428dbf0
runtime.selectgo(0xc88428df10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc88428ddb8 sp=0xc88428dd98
github.com/influxdb/influxdb/services/retention.(*Service).deleteShards(0xc820146a50)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:102 +0x7a0 fp=0xc88428dfa8 sp=0xc88428ddb8
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88428dfb0 sp=0xc88428dfa8
created by github.com/influxdb/influxdb/services/retention.(*Service).Open
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/retention/service.go:46 +0x1d9

goroutine 110 [select, 2725 minutes]:
runtime.gopark(0xcea110, 0xc8855f3f20, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8855f3c98 sp=0xc8855f3c70
runtime.selectgoImpl(0xc8855f3f20, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc8855f3e40 sp=0xc8855f3c98
runtime.selectgo(0xc8855f3f20)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc8855f3e60 sp=0xc8855f3e40
github.com/influxdb/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0xc82015e100)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:137 +0x2c5 fp=0xc8855f3fb8 sp=0xc8855f3e60
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8855f3fc0 sp=0xc8855f3fb8
created by github.com/influxdb/influxdb/cmd/influxd/run.(*Command).Run
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cmd/influxd/run/command.go:119 +0xe81

goroutine 111 [select, 2725 minutes, locked to thread]:
runtime.gopark(0xcea110, 0xc88491a728, 0xbaba58, 0x6, 0x431018, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc88491a520 sp=0xc88491a4f8
runtime.selectgoImpl(0xc88491a728, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xc88491a6c8 sp=0xc88491a520
runtime.selectgo(0xc88491a728)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xc88491a6e8 sp=0xc88491a6c8
runtime.ensureSigM.func1()
        /root/.gvm/gos/go1.5/src/runtime/signal1_unix.go:227 +0x353 fp=0xc88491a7c0 sp=0xc88491a6e8
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc88491a7c8 sp=0xc88491a7c0
created by runtime.ensureSigM
        /root/.gvm/gos/go1.5/src/runtime/signal1_unix.go:240 +0xd0

goroutine 6948931 [semacquire, 55 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xc8bc2a6950 sp=0xc8bc2a6928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xc8bc2a6988 sp=0xc8bc2a6950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xc8bc2a69e8 sp=0xc8bc2a6988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xc8bc2a6a00 sp=0xc8bc2a69e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xc8bc2a6a20 sp=0xc8bc2a6a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xcb36a2c000, 0x31e, 0x400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xc8bc2a6b78 sp=0xc8bc2a6a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xcb36a2c000, 0x31e, 0x400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xc8bc2a6c28 sp=0xc8bc2a6b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xcb36a2c000, 0x31e, 0x400, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xc8bc2a6e50 sp=0xc8bc2a6c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xcb36a2c000, 0x31e, 0x400, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xc8bc2a6e98 sp=0xc8bc2a6e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcf96d9606f, 0x7, 0xc82019a7e0, 0x7, 0xc822664c00, 0x1, 0x1, 0x1, 0xcb36a2c000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xc8bc2a6f50 sp=0xc8bc2a6e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8bc2a6f58 sp=0xc8bc2a6f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 6949300 [semacquire, 55 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcf82456950 sp=0xcf82456928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xcf82456988 sp=0xcf82456950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xcf824569e8 sp=0xcf82456988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xcf82456a00 sp=0xcf824569e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xcf82456a20 sp=0xcf82456a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xcdc7348000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xcf82456b78 sp=0xcf82456a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xcdc7348000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xcf82456c28 sp=0xcf82456b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xcdc7348000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xcf82456e50 sp=0xcf82456c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xcdc7348000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xcf82456e98 sp=0xcf82456e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcac8a9ce7f, 0x7, 0xc82019a7e0, 0x7, 0xc9841dd920, 0x1, 0x1, 0x1, 0xcdc7348000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xcf82456f50 sp=0xcf82456e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcf82456f58 sp=0xcf82456f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 152 [select]:
runtime.gopark(0xcea110, 0xccfa6e4f50, 0xbaba58, 0x6, 0xc7bd59bf18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xccfa6e4c28 sp=0xccfa6e4c00
runtime.selectgoImpl(0xccfa6e4f50, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xccfa6e4dd0 sp=0xccfa6e4c28
runtime.selectgo(0xccfa6e4f50)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xccfa6e4df0 sp=0xccfa6e4dd0
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc820150380, 0xd473eac080, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x518 fp=0xccfa6e4fe8 sp=0xccfa6e4df0
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine(0xc8201504d0, 0x7fef007ef690, 0xcbf60e2400, 0xd474686000, 0xd47479a000, 0x879e7, 0xffe00, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:491 +0xeb0 fp=0xccfa6e5228 sp=0xccfa6e4fe8
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite(0xc8201504d0, 0x7fef007ef690, 0xcbf60e2400, 0xd474686000, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:346 +0x722 fp=0xccfa6e5358 sp=0xccfa6e5228
github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveWrite)-fm(0x7fef007ef690, 0xcbf60e2400, 0xd474686000, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:103 +0x48 fp=0xccfa6e5388 sp=0xccfa6e5358
github.com/influxdb/influxdb/services/httpd.authenticate.func1(0x7fef007ef690, 0xcbf60e2400, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:621 +0x70 fp=0xccfa6e5468 sp=0xccfa6e5388
net/http.HandlerFunc.ServeHTTP(0xc82019c8c0, 0x7fef007ef690, 0xcbf60e2400, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e5488 sp=0xccfa6e5468
github.com/influxdb/influxdb/services/httpd.gzipFilter.func1(0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:679 +0x2af fp=0xccfa6e5548 sp=0xccfa6e5488
net/http.HandlerFunc.ServeHTTP(0xc82019c8e0, 0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e5568 sp=0xccfa6e5548
github.com/influxdb/influxdb/services/httpd.versionHeader.func1(0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:688 +0xc0 fp=0xccfa6e55b0 sp=0xccfa6e5568
net/http.HandlerFunc.ServeHTTP(0xc82019c900, 0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e55d0 sp=0xccfa6e55b0
github.com/influxdb/influxdb/services/httpd.cors.func1(0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:721 +0x328 fp=0xccfa6e5720 sp=0xccfa6e55d0
net/http.HandlerFunc.ServeHTTP(0xc82019c920, 0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e5740 sp=0xccfa6e5720
github.com/influxdb/influxdb/services/httpd.requestID.func1(0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:731 +0x15d fp=0xccfa6e57a8 sp=0xccfa6e5740
net/http.HandlerFunc.ServeHTTP(0xc82019c940, 0x7fef007ef5b8, 0xcbf60e2300, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e57c8 sp=0xccfa6e57a8
github.com/influxdb/influxdb/services/httpd.logging.func1(0x7fef007ef5b8, 0xcbf60e2280, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:739 +0x10b fp=0xccfa6e58b0 sp=0xccfa6e57c8
net/http.HandlerFunc.ServeHTTP(0xc82019c960, 0x7fef007ef5b8, 0xcbf60e2280, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e58d0 sp=0xccfa6e58b0
github.com/influxdb/influxdb/services/httpd.recovery.func1(0x7fef007ef580, 0xd474690000, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:749 +0x114 fp=0xccfa6e59e8 sp=0xccfa6e58d0
net/http.HandlerFunc.ServeHTTP(0xc82019c980, 0x7fef007ef580, 0xd474690000, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e5a08 sp=0xccfa6e59e8
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc82013c060, 0x7fef007ef580, 0xd474690000, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/bmizerany/pat/mux.go:109 +0x244 fp=0xccfa6e5b80 sp=0xccfa6e5a08
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc8201504d0, 0x7fef007ef580, 0xd474690000, 0xd474686000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:167 +0x243 fp=0xccfa6e5be0 sp=0xccfa6e5b80
net/http.serverHandler.ServeHTTP(0xc885455b00, 0x7fef007ef580, 0xd474690000, 0xd474686000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1862 +0x19e fp=0xccfa6e5c40 sp=0xccfa6e5be0
net/http.(*conn).serve(0xc8201040b0)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1361 +0xbee fp=0xccfa6e5f98 sp=0xccfa6e5c40
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xccfa6e5fa0 sp=0xccfa6e5f98
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6

goroutine 217 [select]:
runtime.gopark(0xcea110, 0xccfa6e0f50, 0xbaba58, 0x6, 0xc7bd488e18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xccfa6e0c28 sp=0xccfa6e0c00
runtime.selectgoImpl(0xccfa6e0f50, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xccfa6e0dd0 sp=0xccfa6e0c28
runtime.selectgo(0xccfa6e0f50)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xccfa6e0df0 sp=0xccfa6e0dd0
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc820150380, 0xd473ead0c0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x518 fp=0xccfa6e0fe8 sp=0xccfa6e0df0
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine(0xc8201504d0, 0x7fef007ef690, 0xcb518356c0, 0xd476c4c000, 0xd476cda000, 0x7bb64, 0x7fe00, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:491 +0xeb0 fp=0xccfa6e1228 sp=0xccfa6e0fe8
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite(0xc8201504d0, 0x7fef007ef690, 0xcb518356c0, 0xd476c4c000, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:346 +0x722 fp=0xccfa6e1358 sp=0xccfa6e1228
github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveWrite)-fm(0x7fef007ef690, 0xcb518356c0, 0xd476c4c000, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:103 +0x48 fp=0xccfa6e1388 sp=0xccfa6e1358
github.com/influxdb/influxdb/services/httpd.authenticate.func1(0x7fef007ef690, 0xcb518356c0, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:621 +0x70 fp=0xccfa6e1468 sp=0xccfa6e1388
net/http.HandlerFunc.ServeHTTP(0xc82019c8c0, 0x7fef007ef690, 0xcb518356c0, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e1488 sp=0xccfa6e1468
github.com/influxdb/influxdb/services/httpd.gzipFilter.func1(0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:679 +0x2af fp=0xccfa6e1548 sp=0xccfa6e1488
net/http.HandlerFunc.ServeHTTP(0xc82019c8e0, 0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e1568 sp=0xccfa6e1548
github.com/influxdb/influxdb/services/httpd.versionHeader.func1(0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:688 +0xc0 fp=0xccfa6e15b0 sp=0xccfa6e1568
net/http.HandlerFunc.ServeHTTP(0xc82019c900, 0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e15d0 sp=0xccfa6e15b0
github.com/influxdb/influxdb/services/httpd.cors.func1(0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:721 +0x328 fp=0xccfa6e1720 sp=0xccfa6e15d0
net/http.HandlerFunc.ServeHTTP(0xc82019c920, 0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e1740 sp=0xccfa6e1720
github.com/influxdb/influxdb/services/httpd.requestID.func1(0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:731 +0x15d fp=0xccfa6e17a8 sp=0xccfa6e1740
net/http.HandlerFunc.ServeHTTP(0xc82019c940, 0x7fef007ef5b8, 0xcb51835640, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e17c8 sp=0xccfa6e17a8
github.com/influxdb/influxdb/services/httpd.logging.func1(0x7fef007ef5b8, 0xcb51835600, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:739 +0x10b fp=0xccfa6e18b0 sp=0xccfa6e17c8
net/http.HandlerFunc.ServeHTTP(0xc82019c960, 0x7fef007ef5b8, 0xcb51835600, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e18d0 sp=0xccfa6e18b0
github.com/influxdb/influxdb/services/httpd.recovery.func1(0x7fef007ef580, 0xd476c52000, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:749 +0x114 fp=0xccfa6e19e8 sp=0xccfa6e18d0
net/http.HandlerFunc.ServeHTTP(0xc82019c980, 0x7fef007ef580, 0xd476c52000, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xccfa6e1a08 sp=0xccfa6e19e8
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc82013c060, 0x7fef007ef580, 0xd476c52000, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/bmizerany/pat/mux.go:109 +0x244 fp=0xccfa6e1b80 sp=0xccfa6e1a08
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc8201504d0, 0x7fef007ef580, 0xd476c52000, 0xd476c4c000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:167 +0x243 fp=0xccfa6e1be0 sp=0xccfa6e1b80
net/http.serverHandler.ServeHTTP(0xc885455b00, 0x7fef007ef580, 0xd476c52000, 0xd476c4c000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1862 +0x19e fp=0xccfa6e1c40 sp=0xccfa6e1be0
net/http.(*conn).serve(0xc835220a50)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1361 +0xbee fp=0xccfa6e1f98 sp=0xccfa6e1c40
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xccfa6e1fa0 sp=0xccfa6e1f98
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6

goroutine 6952273 [semacquire, 40 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcc2e776950 sp=0xcc2e776928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xcc2e776988 sp=0xcc2e776950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xcc2e7769e8 sp=0xcc2e776988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xcc2e776a00 sp=0xcc2e7769e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xcc2e776a20 sp=0xcc2e776a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xd143ea6000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xcc2e776b78 sp=0xcc2e776a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xd143ea6000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xcc2e776c28 sp=0xcc2e776b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xd143ea6000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xcc2e776e50 sp=0xcc2e776c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xd143ea6000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xcc2e776e98 sp=0xcc2e776e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcfb877bd7f, 0x7, 0xc82019a7e0, 0x7, 0xc8b782d620, 0x1, 0x1, 0x1, 0xd143ea6000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xcc2e776f50 sp=0xcc2e776e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcc2e776f58 sp=0xcc2e776f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 220 [select]:
runtime.gopark(0xcea110, 0xcaf5d22f50, 0xbaba58, 0x6, 0xc7bd56b518, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcaf5d22c28 sp=0xcaf5d22c00
runtime.selectgoImpl(0xcaf5d22f50, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xcaf5d22dd0 sp=0xcaf5d22c28
runtime.selectgo(0xcaf5d22f50)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xcaf5d22df0 sp=0xcaf5d22dd0
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints(0xc820150380, 0xd474b3c780, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:210 +0x518 fp=0xcaf5d22fe8 sp=0xcaf5d22df0
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWriteLine(0xc8201504d0, 0x7fef007ef690, 0xc987ffdd60, 0xd474fbe000, 0xd4750e2000, 0x8189b, 0xffe00, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:491 +0xeb0 fp=0xcaf5d23228 sp=0xcaf5d22fe8
github.com/influxdb/influxdb/services/httpd.(*Handler).serveWrite(0xc8201504d0, 0x7fef007ef690, 0xc987ffdd60, 0xd474fbe000, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:346 +0x722 fp=0xcaf5d23358 sp=0xcaf5d23228
github.com/influxdb/influxdb/services/httpd.(*Handler).(github.com/influxdb/influxdb/services/httpd.serveWrite)-fm(0x7fef007ef690, 0xc987ffdd60, 0xd474fbe000, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:103 +0x48 fp=0xcaf5d23388 sp=0xcaf5d23358
github.com/influxdb/influxdb/services/httpd.authenticate.func1(0x7fef007ef690, 0xc987ffdd60, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:621 +0x70 fp=0xcaf5d23468 sp=0xcaf5d23388
net/http.HandlerFunc.ServeHTTP(0xc82019c8c0, 0x7fef007ef690, 0xc987ffdd60, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d23488 sp=0xcaf5d23468
github.com/influxdb/influxdb/services/httpd.gzipFilter.func1(0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:679 +0x2af fp=0xcaf5d23548 sp=0xcaf5d23488
net/http.HandlerFunc.ServeHTTP(0xc82019c8e0, 0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d23568 sp=0xcaf5d23548
github.com/influxdb/influxdb/services/httpd.versionHeader.func1(0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:688 +0xc0 fp=0xcaf5d235b0 sp=0xcaf5d23568
net/http.HandlerFunc.ServeHTTP(0xc82019c900, 0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d235d0 sp=0xcaf5d235b0
github.com/influxdb/influxdb/services/httpd.cors.func1(0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:721 +0x328 fp=0xcaf5d23720 sp=0xcaf5d235d0
net/http.HandlerFunc.ServeHTTP(0xc82019c920, 0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d23740 sp=0xcaf5d23720
github.com/influxdb/influxdb/services/httpd.requestID.func1(0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:731 +0x15d fp=0xcaf5d237a8 sp=0xcaf5d23740
net/http.HandlerFunc.ServeHTTP(0xc82019c940, 0x7fef007ef5b8, 0xc987ffdb60, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d237c8 sp=0xcaf5d237a8
github.com/influxdb/influxdb/services/httpd.logging.func1(0x7fef007ef5b8, 0xc987ffda40, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:739 +0x10b fp=0xcaf5d238b0 sp=0xcaf5d237c8
net/http.HandlerFunc.ServeHTTP(0xc82019c960, 0x7fef007ef5b8, 0xc987ffda40, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d238d0 sp=0xcaf5d238b0
github.com/influxdb/influxdb/services/httpd.recovery.func1(0x7fef007ef580, 0xd474fc8000, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:749 +0x114 fp=0xcaf5d239e8 sp=0xcaf5d238d0
net/http.HandlerFunc.ServeHTTP(0xc82019c980, 0x7fef007ef580, 0xd474fc8000, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1422 +0x3a fp=0xcaf5d23a08 sp=0xcaf5d239e8
github.com/bmizerany/pat.(*PatternServeMux).ServeHTTP(0xc82013c060, 0x7fef007ef580, 0xd474fc8000, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/bmizerany/pat/mux.go:109 +0x244 fp=0xcaf5d23b80 sp=0xcaf5d23a08
github.com/influxdb/influxdb/services/httpd.(*Handler).ServeHTTP(0xc8201504d0, 0x7fef007ef580, 0xd474fc8000, 0xd474fbe000)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/services/httpd/handler.go:167 +0x243 fp=0xcaf5d23be0 sp=0xcaf5d23b80
net/http.serverHandler.ServeHTTP(0xc885455b00, 0x7fef007ef580, 0xd474fc8000, 0xd474fbe000)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1862 +0x19e fp=0xcaf5d23c40 sp=0xcaf5d23be0
net/http.(*conn).serve(0xc835220d10)
        /root/.gvm/gos/go1.5/src/net/http/server.go:1361 +0xbee fp=0xcaf5d23f98 sp=0xcaf5d23c40
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcaf5d23fa0 sp=0xcaf5d23f98
created by net/http.(*Server).Serve
        /root/.gvm/gos/go1.5/src/net/http/server.go:1910 +0x3f6

goroutine 6947924 [semacquire, 62 minutes]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xcb59954950 sp=0xcb59954928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xcb59954988 sp=0xcb59954950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xcb599549e8 sp=0xcb59954988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xcb59954a00 sp=0xcb599549e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xcb59954a20 sp=0xcb59954a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xc8bef18000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xcb59954b78 sp=0xcb59954a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xc8bef18000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xcb59954c28 sp=0xcb59954b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xc8bef18000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xcb59954e50 sp=0xcb59954c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xc8bef18000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xcb59954e98 sp=0xcb59954e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xcb90daf56f, 0x7, 0xc82019a7e0, 0x7, 0xc9c7093200, 0x1, 0x1, 0x1, 0xc8bef18000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xcb59954f50 sp=0xcb59954e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xcb59954f58 sp=0xcb59954f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

End of log looks like:
goroutine 6959339 [select]:
runtime.gopark(0xcea110, 0xd475453e10, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xd475453ad8 sp=0xd475453ab0
runtime.selectgoImpl(0xd475453e10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xd475453c80 sp=0xd475453ad8
runtime.selectgo(0xd475453e10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xd475453ca0 sp=0xd475453c80
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc820150380, 0xcc41243160, 0xd474f9e5df, 0x7, 0xc82019a7e0, 0x7, 0x1, 0xd475b56000, 0xca8, 0xe00, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x844 fp=0xd475453ee0 sp=0xd475453ca0
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints.func1(0xc8977560c0, 0xc820150380, 0xd472498100, 0xcc41243160, 0xd474f9e5df, 0x7, 0xc82019a7e0, 0x7, 0xd475b56000, 0xca8, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0x97 fp=0xd475453f58 sp=0xd475453ee0
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd475453f60 sp=0xd475453f58
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x33d

goroutine 6959340 [semacquire]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xd475454950 sp=0xd475454928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xd475454988 sp=0xd475454950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xd4754549e8 sp=0xd475454988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xd475454a00 sp=0xd4754549e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xd475454a20 sp=0xd475454a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xd475b56000, 0xca8, 0xe00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xd475454b78 sp=0xd475454a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xd475b56000, 0xca8, 0xe00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xd475454c28 sp=0xd475454b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xd475b56000, 0xca8, 0xe00, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xd475454e50 sp=0xd475454c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xd475b56000, 0xca8, 0xe00, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xd475454e98 sp=0xd475454e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xd474f9e5df, 0x7, 0xc82019a7e0, 0x7, 0xc897756180, 0x1, 0x1, 0x1, 0xd475b56000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xd475454f50 sp=0xd475454e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd475454f58 sp=0xd475454f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 6960826 [select]:
runtime.gopark(0xcea110, 0xd4736b7e10, 0xbaba58, 0x6, 0x18, 0x2)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xd4736b7ad8 sp=0xd4736b7ab0
runtime.selectgoImpl(0xd4736b7e10, 0x0, 0x18)
        /root/.gvm/gos/go1.5/src/runtime/select.go:392 +0xa64 fp=0xd4736b7c80 sp=0xd4736b7ad8
runtime.selectgo(0xd4736b7e10)
        /root/.gvm/gos/go1.5/src/runtime/select.go:212 +0x12 fp=0xd4736b7ca0 sp=0xd4736b7c80
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard(0xc820150380, 0xcdf41daf60, 0xd4757e4e4f, 0x7, 0xc82019a7e0, 0x7, 0x1, 0xd4761b0000, 0x1388, 0x1800, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:278 +0x844 fp=0xd4736b7ee0 sp=0xd4736b7ca0
github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints.func1(0xc897755260, 0xc820150380, 0xd474b3c980, 0xcdf41daf60, 0xd4757e4e4f, 0x7, 0xc82019a7e0, 0x7, 0xd4761b0000, 0x1388, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:205 +0x97 fp=0xd4736b7f58 sp=0xd4736b7ee0
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd4736b7f60 sp=0xd4736b7f58
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).WritePoints
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:206 +0x33d

goroutine 6960827 [semacquire]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xd4736b8950 sp=0xd4736b8928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xd4736b8988 sp=0xd4736b8950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xd4736b89e8 sp=0xd4736b8988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xd4736b8a00 sp=0xd4736b89e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xd4736b8a20 sp=0xd4736b8a00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xd4761b0000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xd4736b8b78 sp=0xd4736b8a20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xd4761b0000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xd4736b8c28 sp=0xd4736b8b78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xd4761b0000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xd4736b8e50 sp=0xd4736b8c28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xd4761b0000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xd4736b8e98 sp=0xd4736b8e50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xd4757e4e4f, 0x7, 0xc82019a7e0, 0x7, 0xc897755320, 0x1, 0x1, 0x1, 0xd4761b0000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xd4736b8f50 sp=0xd4736b8e98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd4736b8f58 sp=0xd4736b8f50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

goroutine 6960850 [semacquire]:
runtime.gopark(0xcea0d8, 0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:185 +0x163 fp=0xd2975ae950 sp=0xd2975ae928
runtime.goparkunlock(0xfc8820, 0xbc3880, 0xa, 0x19, 0x4)
        /root/.gvm/gos/go1.5/src/runtime/proc.go:191 +0x54 fp=0xd2975ae988 sp=0xd2975ae950
runtime.semacquire(0xc82009a2d4, 0x1)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:100 +0x2f7 fp=0xd2975ae9e8 sp=0xd2975ae988
sync.runtime_Semacquire(0xc82009a2d4)
        /root/.gvm/gos/go1.5/src/runtime/sema.go:43 +0x26 fp=0xd2975aea00 sp=0xd2975ae9e8
sync.(*RWMutex).RLock(0xc82009a2c8)
        /root/.gvm/gos/go1.5/src/sync/rwmutex.go:36 +0x58 fp=0xd2975aea20 sp=0xd2975aea00
github.com/influxdb/influxdb/tsdb/engine/wal.(*Log).WritePoints(0xc82009a270, 0xd4768a4000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/wal/wal.go:240 +0x27e fp=0xd2975aeb78 sp=0xd2975aea20
github.com/influxdb/influxdb/tsdb/engine/bz1.(*Engine).WritePoints(0xc8201e2300, 0xd4768a4000, 0x1388, 0x1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/engine/bz1/bz1.go:210 +0xba fp=0xd2975aec28 sp=0xd2975aeb78
github.com/influxdb/influxdb/tsdb.(*Shard).WritePoints(0xc82014f9e0, 0xd4768a4000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/shard.go:212 +0x589 fp=0xd2975aee50 sp=0xd2975aec28
github.com/influxdb/influxdb/tsdb.(*Store).WriteToShard(0xc82013a750, 0x1, 0xd4768a4000, 0x1388, 0x1800, 0x0, 0x0)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/tsdb/store.go:303 +0xf7 fp=0xd2975aee98 sp=0xd2975aee50
github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard.func1(0xc820150380, 0xd4764badef, 0x7, 0xc82019a7e0, 0x7, 0xc897745080, 0x1, 0x1, 0x1, 0xd4768a4000, ...)
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:241 +0xd4 fp=0xd2975aef50 sp=0xd2975aee98
runtime.goexit()
        /root/.gvm/gos/go1.5/src/runtime/asm_amd64.s:1696 +0x1 fp=0xd2975aef58 sp=0xd2975aef50
created by github.com/influxdb/influxdb/cluster.(*PointsWriter).writeToShard
        /tmp/tmp.SXplMOFgOF/src/github.com/influxdb/influxdb/cluster/points_writer.go:271 +0x1ab

As a note, config is default with also enabling udp, but using tcp communication as was having trouble with influxdb ingesting on a late night Friday. :)

At first glance, this looks like a runtime bug in go 1.5.

I'm wondering why there are so many calls to WritePoints open. Why wouldn't those be getting freed up? I'm wondering if this is related to some of the other WAL/Storage engine related issues a few people have been reporting.

i just got hit by this on 0.9.5-nightly-b079d20

Closing since this is related to b1/bz1 storage engine which are no longer maintained.

authors: DanielMorsing gerrickw jwilder parisholley pauldix

repo: influxdb/influxdb