text:
x/vgo: missing "get" subcommand without arguments

What did you do?
cd $my_go_module
vgo get

What did you expect to see?
All the dependencies listed in the go.mod file ready for the next build.
What did you see instead?
> vgo get
vgo get: need arguments or -u

System details
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/pierre/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/pierre/code"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/1_/m_98m3w13zddr8ph_1d6lhb40000gp/T/go-build605510730=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""

why do I need vgo get
My workflow contemplates frequent Docker builds. In my Dockerfile, I want to be able to fetch the project dependencies in a step before copying (and building) my actual code from the host, so that Docker can cache the dependency-fetching layer and reuse it if I change the code of my module.

It would be great to have a command that just resolves imports/requires and downloads the dependencies. This is very useful for CI systems that often download package first, cache them and then move on to running tests. With current design, a CI system will not be able to cache dependencies until the tests pass.

Yes, we need to figure out what this means exactly.

authors: owais pierreprinetti rsc


text:
go/doc: lines with double-quotes cannot be headings

What did you do?
Add a package doc like:
// ...
// The behaviour can be customized by annotations on the CRDs.
//
// Annotation "annotation-name"
//
//   example.com/annotation-name: <bool>
//
// If true, lorem ipsum ...
//

What did you expect to see?
The string Annotation "annotation-name" should be a title since it is delimited by blank lines, starts with and uppercase letter and does not end with punctuation.
What did you see instead?
It shows up as a regular text paragraph. I've also tested simply putting the work Annotation there to exclude the quotes.

Actual example here:
https://godoc.org/github.com/googlecloudrobotics/core/src/go/cmd/cr-syncer

See https://godoc.org/go/doc#ToHTML for the rules of what makes a heading:

a span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation other than parentheses and commas is formatted as a heading.

Specifically, that line isn't recognized as a heading because it contains the " character, which is neither parenthesis nor comma.
Here's the check in code:
https://github.com/golang/go/blob/76659e47971b594aa6855ef77e022420e070e5ea/src/go/doc/comment.go#L232-L235

Like I said, I also just tried 'Annotation' in the above example to exclude that it is due to the '"', but it was still not recognized as a heading.

Is there a technical reason why we couldn't use quotes in titles?  I hit this today, and was forced to reword the title, but that did not quite capture the original meaning.  Would love to see quotes (double, and/or single) be included in the title allow-list!

See https://github.com/golang/go/discussions/48305.

source: gharchive/issue

authors: dmitshur ensonic jpap rsc

repo: golang/go

license_type: permissive

license_source: github-api


text:
proposal: gopherbot: add proposal process status box to each proposal issue

Many people filing issues that turn into proposals have no idea what that means.
The proposal process is linked from the code contribution guidelines, but reading those is not necessary to file an issue.
And because the issue template targets bug fixes, adding more to the issue template is not appropriate either.
It would be nice for gopherbot to post a link to the proposal process when an issue
becomes a proposal.
Separately, it would also be nice if there were a clear place for information about the status of a proposal.
GitHub being what it is, with no pinned comments or anything like that, the only place you can put that information and be sure it will be displayed is in the issue description (the top-most comment box).
So probably what gopherbot should do instead of leaving a comment is to insert and then maintain a basic header at the top of the issue description that clearly states the status of the issue and where it is in the proposal process, along with a link to the proposal process description itself.
To start, it would be enough to insert the header with a link to the proposal process doc.

There's also an option to create multiple issue templates. Creating one tailored for proposals makes sense to me.
https://help.github.com/en/articles/about-issue-and-pull-request-templates

When I wrote the post about this issue, someone reached out to suggest GitHub Projects instead.
We've been using that for quite a while now, and it lets us show the status on the right side of the issue.
I've also added a link to the proposal process to the description of both the Proposal label and the Proposals issue project.
That seems to be working well enough that editing the top comment is not worth the trouble.
Closing this issue - completed in another way.

authors: rsc vcabbage


text:
need help with tls and io.Pipe

What version of Go are you using (go version)?
go version go1.13.1 linux/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="~/.cache/go-build"
GOENV="~/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="~/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build376784943=/tmp/go-build -gno-record-gcc-switches"
What did you do?
Trying to redirect pipe from zip tool from CMD to the http browser client. It works without TLS, but when I set tls cert and key, it sends corrupted file randomly, and shows "http2: stream closed" message.
Addition: on mipsle application crash with nil mistake at bufio, only when compiled with flags : -ldflags="-s -w". But without mentioned flags, the application does not crash, but produces corrupt zip file, only with tls enabled.
function :

func serveDownload(c *fb.Context, w http.ResponseWriter, files []string) (err error) {
  // Defines the file name.
  name := ""

  if c.File != nil {
  	name = c.File.Name
  }
  if name == "." || name == "" {
  	name = "archive.zip"
  } else {
  	name += ".zip"
  }
  w.Header().Set("Content-Disposition", "attachment; filename*=utf-8''"+url.PathEscape(name))
  pr, pw := io.Pipe()
  //j - cut path, and store only
  cmd := exec.Command("zip", append([]string{"-0rq", "-"}, files...)...)
  cmd.Stdout = pw
  cmd.Stderr = os.Stderr
  go func() {
  	defer pr.Close()
  	// copy the data written to the PipeReader via the cmd to stdout
  	if _, err := io.Copy(w, pr); err != nil {
  		log.Println(err)
  	}
  }()
  return cmd.Run()

}

This is full code:
https://github.com/browsefile/backend
This is rice box file, you do not need to compile front, just put it into src/lib
https://drive.google.com/file/d/1VFiNFv1_K2DiS2Wx-x03go58yaHUF3ac/view?usp=sharing
What did you expect to see?
zip file that opens
What did you see instead?
corrupted zip file and next message in console : http2: stream closed

Hello @vitaliy-kuzmich, thank you for the question! Unfortunately this page isn't for asking questions, please ask questions on the following resources https://github.com/golang/go/wiki/Questions.
When filing this bugs, please also try to provide standalone reproductions.
Given your description and initial code snippet, I produced for you a full standalone snippet that has a TLS server running, makes an HTTP/2.0 request with the TLS request and streams back the zipped file which you can invoke with go run main.go > foo.zip and it unzips it alright https://play.golang.org/p/gSqhFSAUieG or inlined below
package main

import (
	"context"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
	"net/http/httputil"
	"net/url"
	"os"
	"os/exec"

	"golang.org/x/net/http2"
)

func main() {
	cst := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		files := []string{"main.go"}
		if err := serveDownload(r.Context(), w, files); err != nil {
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}
	}))

	if err := http2.ConfigureServer(cst.Config, new(http2.Server)); err != nil {
		log.Fatal(err)
	}
	cst.TLS = cst.Config.TLSConfig
	cst.StartTLS()
	defer cst.Close()

	tr := &http.Transport{
		TLSClientConfig: cst.Config.TLSConfig,
	}
	tr.TLSClientConfig.InsecureSkipVerify = true
	http2.ConfigureTransport(tr)
	req, _ := http.NewRequest("GET", cst.URL, nil)

	res, err := tr.RoundTrip(req)
	if err != nil {
		log.Fatalf("Failed to make request: %v", err)
	}
	defer res.Body.Close()

	if false { // Uncomment in order to examine the entire response and ensure HTTP/2.0 being used.
		blob, _ := httputil.DumpResponse(res, true)
		println(string(blob))
	} else {
		io.Copy(os.Stdout, res.Body)
	}
}

func serveDownload(ctx context.Context, w http.ResponseWriter, files []string) (err error) {
	// Defines the file name.
	name := "archive.zip"
	w.Header().Set("Content-Disposition", "attachment; filename*=utf-8''"+url.PathEscape(name))
	pr, pw := io.Pipe()
	//j - cut path, and store only
	cmd := exec.Command("zip", append([]string{"-0rq", "-"}, files...)...)
	cmd.Stdout = pw
	cmd.Stderr = os.Stderr
	go func() {
		defer pr.Close()
		// copy the data written to the PipeReader via the cmd to stdout
		if _, err := io.Copy(w, pr); err != nil {
			panic(err)
		}
	}()
	return cmd.Run()
}

Running it
$ go run main.go > rp/foo.zip && cd rp && file foo.zip && unzip foo.zip && cd $OLDPWD
foo.zip: Zip archive data, at least v1.0 to extract
Archive:  foo.zip
replace main.go? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
 extracting: main.go

which works alright, but it isn't using your *fb.Context, just the standard library's context.Context, so perhaps please check and see where/how that *fb.Context is reacting with TLS, and please follow up your question on the resources that I provided up above in https://github.com/golang/go/wiki/Questions.

@odeke-em thank you for quick response, one last question, does this code works at your side ?
`package main
import (
"context"
"fmt"
"golang.org/x/net/http2"
"io"
"io/ioutil"
"log"
"net/http"
"net/http/httptest"
"net/http/httputil"
"net/url"
"os"
"os/exec"
"sync"
"sync/atomic"
)
func main() {
n := 1000
wg := new(sync.WaitGroup)
wg.Add(n)
var rn int64
cst := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
wg.Done()
atomic.AddInt64(&rn, 1)
fmt.Println(" req # ", atomic.LoadInt64(&rn))
files := []string{
<<Insert here at least 10(or more, maybe more then cores count ?) files and folders, with absolute path>>
}
if err := serveDownload(r.Context(), w, files); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}))
if err := http2.ConfigureServer(cst.Config, new(http2.Server)); err != nil {
	log.Fatal(err)
}
cst.TLS = cst.Config.TLSConfig
cst.StartTLS()
defer cst.Close()

tr := &http.Transport{
	TLSClientConfig: cst.Config.TLSConfig,
}
tr.TLSClientConfig.InsecureSkipVerify = true
http2.ConfigureTransport(tr)

for i := 0; i < n; i++ {
	go func() {
		req, _ := http.NewRequest("GET", cst.URL, nil)

		res, err := tr.RoundTrip(req)
		if err != nil {
			log.Fatalf("Failed to make request: %v", err)
		}
		defer res.Body.Close()

		if false { // Uncomment in order to examine the entire response and ensure HTTP/2.0 being used.
			blob, _ := httputil.DumpResponse(res, true)
			println(string(blob))
		} else {
			io.Copy(ioutil.Discard, res.Body)
		}
	}()

}

fmt.Println("hung here")

wg.Wait()

}
func serveDownload(ctx context.Context, w http.ResponseWriter, files []string) (err error) {
// Defines the file name.
name := "archive.zip"
w.Header().Set("Content-Disposition", "attachment; filename*=utf-8''"+url.PathEscape(name))
pr, pw := io.Pipe()
//j - cut path, and store only
cmd := exec.Command("zip", append([]string{"-0rq", "-"}, files...)...)
cmd.Stdout = pw
cmd.Stderr = os.Stderr
go func() {
defer pr.Close()
// copy the data written to the PipeReader via the cmd to stdout
if _, err := io.Copy(w, pr); err != nil {
panic(err)
}
}()
return cmd.Run()
}
`

@odeke-em  thank you! Now I will know about questions page.

authors: odeke-em vitaliy-kuzmich


text:
fmt does not format byte slices nicely

What version of Go are you using (go version)?

go version go1.12.1 windows/amd64

Does this issue reproduce with the latest release?
reproduces on go playground
What operating system and processor architecture are you using (go env)?
go env Output
C:\Users\gs082r>go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\gs082r\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\gs082r\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\gs0
82r\AppData\Local\Temp\go-build177268958=/tmp/go-build -gno-record-gcc-switches

What did you do?
I'm trying to print a slice of bytes as an array of hex values.
What did you expect to see?
[0x0a 0x14 0x1e 0x28 0x32 0x3c 0x46 0x50 0x5a 0x64 0x65]
What did you see instead?
0x0a14
Please see playground link:
https://play.golang.org/p/gQIYJV6ogx4
The first line is ok for a slice of ints.. the %.2v is a sort of workaround, but not what I actually need.
dump from playground:
package main

import (
	"fmt"
)

func main() {
	slcInt8 := [...]int8{10,20,30,40,50,60,70,80,90,100,101}
	fmt.Printf("%#.2x\n", slcInt8)
	slcByte := [...]byte{10,20,30,40,50,60,70,80,90,100,101}
	fmt.Printf("%#.2x\n", slcByte)
	slcUint8 := [...]byte{10,20,30,40,50,60,70,80,90,100,101}
	fmt.Printf("%#.2x\n", slcUint8)

	fmt.Printf("%#x\n", slcInt8)
	fmt.Printf("%#x\n", slcByte)
	fmt.Printf("%#x\n", slcUint8)

	fmt.Printf("%#.2v\n", slcInt8)
	fmt.Printf("%#.2v\n", slcByte)
	fmt.Printf("%#.2v\n", slcUint8)
}

output:
[0x0a 0x14 0x1e 0x28 0x32 0x3c 0x46 0x50 0x5a 0x64 0x65]
0x0a14
0x0a14
[0xa 0x14 0x1e 0x28 0x32 0x3c 0x46 0x50 0x5a 0x64 0x65]
0x0a141e28323c46505a6465
0x0a141e28323c46505a6465
[11]int8{10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 101}
[11]uint8{0x0a, 0x14, 0x1e, 0x28, 0x32, 0x3c, 0x46, 0x50, 0x5a, 0x64, 0x65}
[11]uint8{0x0a, 0x14, 0x1e, 0x28, 0x32, 0x3c, 0x46, 0x50, 0x5a, 0x64, 0x65}

I'm sorry, I don't understand what you are asking for.  It seems that using %#.2x gives you exactly what you want.
Whatever we think of the behavior of the fmt package (and I think it is making a reasonable choice here) we can't change it now.  It would break too many existing programs.

fmt.Printf("%# x\n", slcByte) will output:
0x0a 0x14 0x1e 0x28 0x32 0x3c 0x46 0x50 0x5a 0x64 0x65

@ianlancetaylor
slcByte := [...]byte{10,20,30,40,50,60,70,80,90,100,101}
fmt.Printf("%#.2x\n", slcByte)
gives output:
0x0a14
does this look like expected/reasonable behaviour ?
@martisch  thanks ! this is exactly what I was looking for !

The current behavior looks to me as document. Note that []byte often formats the way a string would be formatted. That is why []int behaves differently. We can not change it without breaking backwards compatibility. Since there is a way to format the slcByte the way wanted I am closing the issue.
If the behavior should be changed please open a new issue with a concrete proposal for Go 2 how the behavior of fmt should be changed and how the gained advantage is worth breaking the backwards compatibility.

authors: guysoffer ianlancetaylor martisch


text:
x/website: https://golang.org/doc/articles/wiki/

The website https://golang.org/doc/articles/wiki/
uses in the section "Other Tasks" this sentence:
"Spruce up the page templates by making them valid HTML and adding some CSS rules"
This sentence uses a word unknown to google translator.
IMHO. This is a typo.
Does this issue reproduce with the latest release?
Seen last on 2020-02-06 at 22:00 CET
What did you do?
View the website.
What did you expect to see?
"Sprout"
What did you see instead?
"Spruce"

"Spruce" is an English word and there is nothing wrong here.
See https://www.merriam-webster.com/dictionary/spruce up .

authors: ianlancetaylor rpoe


text:
math: MaxUint64 doesn't return correct result but an error?

What version of Go are you using (go version)?

$ go@1.12

Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?

$ amd64

What did you do?

I just use the const in math packge: math.MaxUint64

What did you expect to see?

18446744073709551615

What did you see instead?

./main.go:169:29: constant 18446744073709551615 overflows uint

Notice that "1<<64" in "math.MaxUint64 = 1<<64 - 1" may cause overflow, perhas you can use "math.MaxUint64 = ^uint64(0)" instead.

Show us your code. This code works fine:
package main

import "math"

func main() {
	var x uint = math.MaxUint64
	_ = x
}

The definition of MaxUint64 as 1<<64-1 has no issues with overflow. It is a constant expression, which in Go is evaluated with arbitrary precision/range.

Show us your code. This code works fine:
package main

import "math"

func main() {
	var x uint = math.MaxUint64
	_ = x
}

The definition of MaxUint64 as 1<<64-1 has no issues with overflow. It is a constant expression, which in Go is evaluated with arbitrary precision/range.

Your code works well.
Here is my code.

package main

import "math"
func main() {
 	x := math.MaxUint64
	fmt.Println(x)
 }

Is this related to untyped constant?  The untyped constant math.MaxUint64 cannot infer the type from x that has no type. So it is given the default type int, so it overflows.

Yes, the second example is due to the type inference with the untyped constant.
x := math.MaxUint64

infers that x is an int and it overflows.
Also, the first example
var x uint = math.MaxUint64

still overflows on any architecture where uint is 32-bit, e.g. arm.
If you want to assign math.MaxUint64 to a variable you need to store it in a uint64

Closing, this is working as intended.
As @bmkessler says, explicitly type your variable instead of using :=.

See also #19621, #23086, #23087, #23163.

authors: IIvyPy bmkessler ianlancetaylor randall77 zhangfannie


text:
x/pkgsite: standard library packages are not discoverable

What is the URL of the page with the issue?
https://pkg.go.dev/
What is your user agent?

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
Screenshot

What did you do?

I clicked on "Discover Packages".
What did you expect to see?
I expected to have the ability to discover packages, especially from the standard library, as the link promises.
On https://golang.org/pkg/, I am able to see at a glance the entire standard library and discover new packages I haven't used before.
What did you see instead?
Instead, I see a search bar. This indicates that I need to already know what I'm looking for.

On https://golang.org/pkg/, only the packages from the standard library are included, so it is viable to list every single package in an exhaustive list on the front page. On https://pkg.go.dev, there are many more packages that can be discovered, so exhaustively listing them all on the front page is not viable.
Do you think something else can be done? Or is your suggestion to make it easier to understand how to find a list of packages that are only in the standard library?
One more question. Are you already aware of the https://pkg.go.dev/std page, which has a "Packages" tab that lists all packages? I'm asking because if you weren't aware of it, it's possible that making that page more discoverable can be a way to improve the situation.
/cc @julieqiu @jba @jamalc per owners.

Do you think something else can be done? Or is your suggestion to make it easier to understand how to find a list of packages that are only in the standard library?

Yes I was suggesting making it easier to discover standard library packages without needing to know about their existence in the first place.
https://godoc.org lists the top 25 packages by popularity and has links to the standard library, golang.org/x packages, and various search filters on Github. This level of discoverability is nice, in my humble opinion.
Given that a flash message on https://godoc.org is currently offering to "Always use pkg.go.dev" instead of godoc.org, it would be great to keep this level of discoverability in the new offering.

One more question. Are you already aware of the https://pkg.go.dev/std page, which has a "Packages" tab that lists all packages? I'm asking because if you weren't aware of it, it's possible that making that page more discoverable can be a way to improve the situation.

I wasn't aware of this page. Thank you for pointing it out.
For the purposes of making this issue actionable, I second your suggestion of linking to https://pkg.go.dev/std from the "Discover Packages" page.

I'd like to give a strong +1 for this issue. I would really like to see the standard library documentation explicitly linked somewhere instead of hidden behind the search functionality.
Right now, my solution to this problem is having a bookmark that takes me to https://pkg.go.dev/std -- but I'm an experienced Go user, and I knew that that was what I was looking for, and that it's the standard library documentation. From the perspective of a new user of any language, I like to know what packages are available in the standard library, and right now that information is not intuitive to find on pkg.go.dev at all.
In addition to the proposed option of adding some kind of link to the main "Discover" page, I would also recommend adding a "Standard Library" link under "Packages" in the page footer (where the red box is in this screenshot):

While putting the link in the footer isn't ideal, it would be a lot better than not having it anywhere, and I'm honestly surprised every time I go to pkg.go.dev without using my standard library bookmark that it's not there already.

/cc @jinhongy @fflewddur for UX input

Thanks for the feedback. We are in the process of unifying the experience for golang.org and go.dev, with the new design, we will include the standard library link on the secondary navigation on top, which I think will solve this issue.

The new unified experience was launched today, but there still doesn't seem to be a link to the standard library overview https://pkg.go.dev/std in the navigation.

The updated footer seems to only affect go.dev but not pkg.go.dev?

Closing as done

authors: dmitshur fzipp jinhongy julieqiu penmanglewood seankhliao themowski


text:
Why isn't there a heap library in the standard library similar to Python 3's heapq library?

Hi all,
What version of Go are you using (go version)?
Go Version 1.15
Does this issue reproduce with the latest release?
Yes
What did you do?
Minimum heaps allow for insertion and removal in log(N) time. Python 3's heapq library (https://docs.python.org/3/library/heapq.html), provides this functionality.
Creating a heap in golang, however, requires boilerplate code, below is an example implementation of a heap in golang. Why isn't this included as a standard library in golang?
type Heap [][]float64

func (p Heap) Len() int            { return len(p) }
func (p Heap) Less(i, j int) bool  { return p[i][2] > p[j][2] }
func (p Heap) Swap(i, j int)       { p[i], p[j] = p[j], p[i] }
func (p *Heap) Push(i interface{}) { *p = append(*p, i.([]float64)) }
func (p *Heap) Pop() interface{}   { v := (*p)[len(*p)-1]; *p = (*p)[:len(*p)-1]; return v }

Thanks in advance!

Hi, we don't really use our issue tracker for questions. More people will see your question, and you will get better answers, if you ask on a forum rather than on the issue tracker. See https://golang.org/wiki/Questions . Thanks.
The container/heap package may be improved when generics is added (#43651).


text:
x/sys/unix: EpollEvent use void *ptr instead of int fd

What version of Go are you using (go version)?

go version go1.16.2 linux/amd64

I'm using EPOLL(7) from the sys/unix package, since epoll_event struct have a C union epoll_data
typedef union epoll_data {
    void    *ptr;
    int      fd;
    uint32_t u32;
    uint64_t u64;
} epoll_data_t;

struct epoll_event {
    uint32_t     events;    /* Epoll events */
    epoll_data_t data;      /* User data variable */
};

The sys/unix package supporting the struct EpollEvent with Pad
at the end of the struct to fill the union size.
type EpollEvent struct {
	Events uint32
	Fd     int32
	Pad    int32
}

The goal of this issue, is to use EpollEvent with void *ptr  and not int fd
I created a struct epollEvent and called the syscall manually
To try to implement the change.
type epollEvent struct {
	Events uint32
	Ptr    uintptr
}

func epollWait(epfd int, events []epollEvent, msec int) (n int, err error) {
	var _p0 unsafe.Pointer
	if len(events) > 0 {
		_p0 = unsafe.Pointer(&events[0])
	} else {
		_p0 = unsafe.Pointer(&_zero)
	}

	r0, _, e1 := unix.Syscall6(unix.SYS_EPOLL_WAIT, uintptr(epfd), uintptr(_p0), uintptr(len(events)), uintptr(msec), 0, 0)
	n = int(r0)
	if e1 != 0 {
		err = errnoErr(e1)
	}

	return
}

The implementation issue
When I initialize my slice epollEvent and all Ptr uintpr  to a struct instance associated to an epoll event.
And I call epollWait I retrieve my address associated to the event.
But the problem here is the Golang runtime, sometime the runtime move or GC the data associated to my address.
I tried to use unsafe.Pointer insteaf of uintptr but the issue is the same.
In this case what is the best way to create data and associate it to my Ptr
To be retrieved it later by epoll_wait(2) syscall?
Do you think the Idea I would like to realize it possible, if yes I could propose some change to support both in sys/unix package fdmode  and ptr mode.
Thanks a lot

This was previously proposed in #32192, which was merged into #32153. The decision there was to not change EpollEvent: https://github.com/golang/go/issues/32153#issuecomment-523188402
As per https://github.com/golang/go/issues/32192#issuecomment-494989597 having an EpollEventPtr variant would mean that we'd have to introduce another EpollCtl variant. And the kernel might return a mix of the two types, which will probably become messy to deal with.
@pierre-emmanuelJ Could you elaborate a bit on your use case (e.g. with an example using your epollWait implementation), i.e. why you'd want to send pointers to the kernel?
/cc @ianlancetaylor @bradfitz

Would it make sense to expose the u32 and u64 fields of epoll_data instead?
That could be pretty straightforward to use in conjunction with runtime/cgo.Handle (#37033).

Yes, the new runtime/cgo.Handle is a good suggestion.  Note that passing a Go pointer to epoll isn't safe: nothing will keep that pointer alive.  In certain circumstances it's even possible for the pointer to move from Go's point of view, but the copy in epoll will not be updated.

A sample of the event package
package epoll

const (
	// MaxEvents represents the max events for an epoll instance.
	MaxEvents = 32
)

type Event struct {
	Fd     int32
	send   bool
	Cookie interface{}
}

// Send to the handler HandlerOut
func (e *Event) Send() {
	e.send = true
}

// Receive to the handler HandlerIn
func (e *Event) Receive() {
	e.send = false
}

// Handlers represents all handler for incoming events.
type Handlers struct {
	HandlerIn            func(*Event) bool
	HandlerOut           func(*Event) bool
	HandlerErr           func(*Event)
	HandlerNewConnection func(socket int) (int, interface{}, error)
}

// Epoll represents an epoll instance.
type Epoll struct {
	ev       *epollEvent
	epollfd  int
	fd       int
	handlers Handlers

	events [MaxEvents]epollEvent
}

// New returns an epoll instance.
func New(fd int, h Handlers) (*Epoll, error) {
	ptr := C.malloc(C.ulong(unsafe.Sizeof(Event{})))

	(*Event)(ptr).Fd = int32(fd)
	(*Event)(ptr).send = false
	(*Event)(ptr).Cookie = nil

	ev := &epollEvent{
		Ptr: ptr,
	}

	epfd, err := unix.EpollCreate1(0)
	if err != nil {
		return nil, err
	}

	ev.Events = unix.EPOLLIN

	if err := epollCtl(epfd, unix.EPOLL_CTL_ADD, fd, ev); err != nil {
		return nil, err
	}

	return &Epoll{
		ev:       ev,
		epollfd:  epfd,
		fd:       fd,
		handlers: h,
	}, nil
}

// Close epoll File Descriptor.
func (e *Epoll) Close() {
	unix.Close(e.epollfd)
}

func (e *Epoll) Wait() {
	for {
		nfds, err := e.wait()
		if err != nil {
			log.Fatal(err)
		}

		for i := 0; i < int(nfds); i++ {
			if e.incoming(i) {
				fd, data, err := e.handlers.HandlerNewConnection(e.fd)
				if err != nil {
					log.Println(err)
					continue
				}

				if err := e.add(fd, data); err != nil {
					log.Fatal(err)
				}

				continue
			}

			if err := e.executeHandlers(&e.events[i]); err != nil {
				log.Fatal(err)
			}
		}
	}
}

func (e *Epoll) wait() (int, error) {
	var nfds int
	var err error

	for {
		nfds, err = epollWait(e.epollfd, e.events[:], -1)
		if err == unix.EINTR {
			continue
		}

		break
	}
	if err != nil {
		return 0, err
	}

	return nfds, nil
}

func (e *Epoll) incoming(index int) bool {
	return (*(*Event)(e.events[index].Ptr)).Fd == int32(e.fd)
}

func (e *Epoll) add(fd int, data interface{}) error {
	err := unix.SetNonblock(fd, true)
	if err != nil {
		return err
	}

	e.ev.Events = unix.EPOLLIN

	ptr := C.malloc(C.ulong(unsafe.Sizeof(Event{})))
	(*Event)(ptr).Fd = int32(fd)
	(*Event)(ptr).send = false
	(*Event)(ptr).Cookie = data
	e.ev.Ptr = unsafe.Pointer(ptr)

	return epollCtl(e.epollfd, unix.EPOLL_CTL_ADD, fd, e.ev)
}

func (e *Epoll) executeHandlers(ev *epollEvent) error {
	event := (*Event)(ev.Ptr)

	if (ev.Events & (unix.EPOLLIN | unix.EPOLLERR | unix.EPOLLHUP)) != 0 {
		if !e.handlers.HandlerIn(event) {
			return nil
		}

		if event.send {
			if err := e.out(event.Fd); err != nil {
				return err
			}
		}

		return nil
	}

	if (ev.Events & (unix.EPOLLOUT | unix.EPOLLERR | unix.EPOLLHUP)) != 0 {
		if !e.handlers.HandlerOut(event) {
			C.free(ev.Ptr)
			return nil
		}

		if !event.send {
			if err := e.in(event.Fd); err != nil {
				return err
			}
		}

		return nil
	}

	if (ev.Events & (unix.EPOLLERR | unix.EPOLLHUP)) != 0 {
		e.handlers.HandlerErr(event)
		C.free(ev.Ptr)
		return nil
	}

	log.Println("Unknown Epoll event")

	return nil
}

func (e *Epoll) out(fd int32) error {
	event := (*Event)(e.ev.Ptr)

	event.Fd = fd
	e.ev.Events = unix.EPOLLOUT

	return epollCtl(e.epollfd, unix.EPOLL_CTL_MOD, int(fd), e.ev)
}

func (e *Epoll) in(fd int32) error {
	event := (*Event)(e.ev.Ptr)

	event.Fd = fd
	e.ev.Events = unix.EPOLLIN

	return epollCtl(e.epollfd, unix.EPOLL_CTL_MOD, int(fd), e.ev)
}

My file private.go to upstream the unix package:
package epoll

import (
	"syscall"
	"unsafe"

	"golang.org/x/sys/unix"
)

var (
	_zero uintptr
)

type epollEvent struct {
	Events uint32
	Ptr    unsafe.Pointer
}

// errnoErr returns common boxed Errno values, to prevent
// allocations at runtime.
func errnoErr(e syscall.Errno) error {
	switch e {
	case 0:
		return nil
	case unix.EAGAIN:
		return syscall.EAGAIN
	case unix.EINVAL:
		return syscall.EINVAL
	case unix.ENOENT:
		return syscall.ENOENT
	}
	return e
}

func epollWait(epfd int, events []epollEvent, msec int) (n int, err error) {
	var _p0 unsafe.Pointer
	if len(events) > 0 {
		_p0 = unsafe.Pointer(&events[0])
	} else {
		_p0 = unsafe.Pointer(&_zero)
	}

	r0, _, e1 := unix.Syscall6(unix.SYS_EPOLL_WAIT, uintptr(epfd), uintptr(_p0), uintptr(len(events)), uintptr(msec), 0, 0)
	n = int(r0)
	if e1 != 0 {
		err = errnoErr(e1)
	}

	return
}

func epollCtl(epfd int, op int, fd int, event *epollEvent) (err error) {
	_, _, e1 := unix.RawSyscall6(unix.SYS_EPOLL_CTL, uintptr(epfd), uintptr(op), uintptr(fd), uintptr(unsafe.Pointer(event)), 0, 0)
	if e1 != 0 {
		err = errnoErr(e1)
	}

	return
}

Example of usage:
socket, err := newSocket(ip, port)
	if err != nil {
		log.Fatal(err)
	}
	defer unix.Close(socket)

	h := epoll.Handlers{
		HandlerIn:  func(e *epoll.Event) {
			// in stuff
		},
		HandlerOut: func(e *epoll.Event) {
			// out stuff
		},
		HandlerErr: func(e *epoll.Event) {
			// errro stuff
		},
		HandlerNewConnection: func(fd int) (int, interface{}, error) {
                         // call from CGO
			conn, err := C.GotNewConnection(fd, n.c)
			if err != nil {
				return 0, nil, err
			}

			return conn.FD, conn, nil
		},
	}

	epoll, err := epoll.New(socket, h)
	if err != nil {
		log.Fatal(err)
	}
	defer epoll.Close()

	epoll.Wait()

In this sample of code I tried to use pointers from MALLOC(3) instead of Go pointers. to prevent this issue.
(malloc value returns are not checked)

It's very weird @tklauser after calling epoll_wait with my events slice, ptr is populated  with pointers I have not registered in, and that cause segfault so.

Here is a more compact example if you want (quick implementation):
https://gist.github.com/pierre-emmanuelJ/4562322b24020e24ca4389b6aa7f7d3d
Same here epollWait populate me the events slice with unknown pointer in unsafe.pointer Ptr.
Did I miss something in the unix.Syscall6 call?

I found the main reason of the issue mentioned in those examples, if you want to use epoll in ptr mode correctly with the new runtime/cgo.Handlestuff or malloc there are some points to mention about memory alignment.
Since the C struct epoll_event is using attribute packet:
#define __EPOLL_PACKED __attribute__ ((__packed__))

struct epoll_event
{
  uint32_t events;	/* Epoll events */
  epoll_data_t data;	/* User data variable */
} __EPOLL_PACKED;

the sizeof the struct is 12 bytes. In Golang it's not possible to use attribute packed.
This following structure could not be used with epoll:
type epollEvent struct {
	Events uint32
	Ptr    unsafe.Pointer // Or uintptr
}

The sizeof this struct is 16 bytes.
To properly use epoll in ptr mode the solution is to use CGO to instantiate an epoll_event struct or use something like this:
type epollEvent struct {
	Events uint32
	Ptr    [8]byte
}

And then use the binary package to retrieve the pointer address.

@pierre-emmanuelJ
Hello, my version is go1.19, and there is the following definition in $GOROOT/src/syscall/ztypes_linux_amd64.go:
type EpollEvent struct {
    Events uint32
    Fd     int32
    Pad    int32
}

To use the associated data returned by epoll_wait, you can do the following:
ed := new(evData)
ev := syscall.EpollEvent{
    Events: uint32(events),
}
*(**evData)(unsafe.Pointer(&ev.Fd)) = ed
syscall.EpollCtl(ep.efd, syscall.EPOLL_CTL_MOD, fd, &ev)

When syscall.EpollWait returns, you can use the following to obtain the evData pointer data:
ev := &events[i]
ed := *(**evData)(unsafe.Pointer(&ev.Fd))

This should work. Sizeof(syscall.EpollEvent) == 12

authors: alfiver bcmills ianlancetaylor pierre-emmanuelJ tklauser