text:
cmd/go: provide straightforward way to see non-test dependencies

It's important to be able to see the full set of dependencies used by production code exclusive of tests, particularly with different major versions as distinct modules, as inadvertently using an import with a different major version can introduce bugs (also, some external dependencies can be toxic to have in production code).
The go mod why command provides a -vendor flag to exclude tests; perhaps go list could provide a similar flag to exclude testing dependencies.
In addition to the above, it is also useful to be able to be able to see modules used by local production code and tests, but not transitively by tests in external dependencies, as external dependencies are harder to change, and inadvertently forked modules are still an issue for tests too. I'm not sure of a good way to specify that, though.

We should see if it's better to have this information directly in the go.mod file (like // test or separate sections of require). Or if it's better to use a cmd or a web site like godoc.org ?

@flibustenet - just in case you haven't seen https://github.com/golang/go/issues/26913#issuecomment-411976222 yet.
I think it's worth focussing on what people are trying to achieve here... which I agree has some overlap with their workflow and hence the tools they are using, but in the case of using godoc.org, I believe the underlying question that someone is trying to answer is "what dependencies does module M have?" and making some judgement on whether or not to use M using that information. Getting more detail on that will, I think, help to steer us towards the building blocks we then need in place.
As I noted on Slack:

Because all of these sorts of queries are possible using go list.... but the very fact we're having this discussion means that that solution is probably too esoteric. And if the queries are common enough it probably makes sense to provide something more accessible for day-to-day use

That said, we don't need to load everything into the go tool; if it provides the building blocks for some other solution that answers these same questions but via a different UI/UX then that's just as good.

As an example of the issue here, I just noticed a dependency on github.com/jtolds/gls which I wasn't expecting. The go mod why output looked like this:
# github.com/jtolds/gls
github.com/juju/charmstore-client/cmd/charm
github.com/juju/juju/juju/osenv
github.com/juju/juju/juju/osenv.test
github.com/juju/juju/testing
github.com/juju/juju/api/block
github.com/juju/juju/api/block.test
github.com/juju/juju/state
github.com/juju/juju/state.test
github.com/juju/juju/component/all
github.com/juju/juju/cmd/juju/commands
github.com/juju/juju/provider/all
github.com/juju/juju/provider/ec2
gopkg.in/ini.v1
gopkg.in/ini.v1.test
github.com/smartystreets/goconvey/convey
github.com/jtolds/gls

So the shortest path to this dependency takes us through four levels of external test dependencies. In practice I almost never care about test dependencies of external dependencies.

Hi,  I'm redirecting the comment here
Russ's comment about scopes is compelling to have unified go.mod, but not compelling to hide scopes (build tags x test).
I think having go mod why not refer to scopes can be confusing or worse: go mod why is arguably not providing the functionality advertised (i.e. explaining why) if it doesn't also give scope information, at least for the main module.

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module.  In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.
One idea is for go to add comments like github.com/gopherjs/gopherjs v0.0.0-20181004151105-1babbf986f6f // indirect-test for indirect dependencies that are dependencies from tests in the dependencies' modules.  Then one could look at go.mod and see that packages/modules marked as // indirect-test are needed only if you want to run go test all or go test ... but not for the "regular" workflow of go install -v .... && go test -v ./... (to build and test your own code).

Tests dependencies should be visible, even for direct test dependencies. It's a valuable information. I'd like to choose a lib with as few dependency as possible to run but the most tests (and test dependencies if needed) as possible...

In particularly, this is needed if you have a policy that all open-source / "external" (outside of your company control) packages must be mirrored on internal servers.

That's probably a task better suited to GOPROXY than to go.mod. (In particular, I would expect the company to provide a GOPROXY server containing all of the approved/vetted versions of external modules.)

I would like to be able to look at go.mod and find out which packages are needed to build and run tests in the current module.

I think the go tool would be better.  But, in theory the problem of deciding whether or not a file has satisfiable build tags is NP-complete :)

perhaps go list could provide a similar flag to exclude testing dependencies.

I'm curious: does go list -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Per my answer in https://github.com/golang/go/issues/27900#issuecomment-437926158, I think it depends if you expect/want that answer to be irrespective of build constraints.

I'm curious: does go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u produce the list that you want? (It's possible that the same flag can address both this use-case and #27900.)

Yes, this is pretty much exactly what I was after (I wasn't aware of the -deps flag). For the request in my final paragraph, go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u does the job. Not that either of these are particularly easy to type, but I can write a script. It would be nice if go mod why could work with this call graph instead of the regular one though.
PS I'd be happier if the go command never looked at external test dependencies (they can be enormous and it's often impossible or uneconomic to run all tests in all third party modules), but that's another story.

Story
I want to share my frustrating journey to this problem. I recently started a new project and from the get go I wanted dependency management to be done purposefully. Meaning that i want to be careful and conscious about how codes get into my binary.
I'm not building any thing exotic, a basic web app with a database. This is my go.mod file:
module github.com/my/project

go 1.12

require (
	github.com/advanderveer/go-test v1.0.1
	github.com/golang-migrate/migrate/v4 v4.2.4
	github.com/google/wire v0.2.1
	github.com/hashicorp/go-uuid v1.0.1
	github.com/lib/pq v1.0.0
)

As a general policy I wanted all my dependencies to be at version v1 or higher. So unfortunately wire stands out here. I though that wire was only for code generation purposes so I was confused as to why it was here. Ergo, i run go why github.com/google/wire and it gives me no new information as to "why" it is there even though i guarded it from building //+build wireinject flag. Googling around a bit I start to accept that the whole build flag thing is a bit confusing for determining what modules are important here. So i cut my losses and simply add a comment to the mod file for the future, I'm unsure if this comment stays intact why I'm adding more dependencies but I have bigger fish to fry:
Because during my research above I went on to to open my go.sum file. Even though I managed my dependencies carefully I somehow ended up with 200 lines of packages I don't directly seem to use. Some of them quite big:
...
cloud.google.com/go
github.com/aws/aws-sdk-go
github.com/cockroachdb/cockroach-go
github.com/gogo/protobuf
github.com/gopherjs/gopherjs
google.golang.org/grpc
...

So why are they there? go mod why github.com/gogo/protobuf says: # github.com/gogo/protobuf (main module does not need package github.com/gogo/protobuf). Eehm Ok. So why are they listed there? After some googling I run into this reddit thread which refers me to this issues  where it is said: "If you have dependencies only needed for tests, then they will show up in your go.mod, and go get will download their go.mods, but it will not download their code" Even though they are not in my go.mod I think believe this changed 1.12(?) as I used to have his behaviour. At this point I'm just thinking that these dependencies are listed in my go.sum for testing purposes. It is cool that I can test my code and all dependencies but it is impractical for me because when I do this go test all it fails to compile on some deep package github.com/jackc/pgx and starts testing cockroachdb itself (which is huge).
At this point I start googling and find all kinds of commands in issues such as go mod why all go mod why -vendor, go list or event go list -test -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}'| sort -u to figure how how these dependencies end up in my project. None of them seem to give a conclusive answer so I take a step back:
Want I want to know is which code ends up in the binary I ship. So I can also just look at the build and see which packages are included for that, so I run: go build -o /tmp/my-project -v. I was pretty sure that the -v command would show which packages are being build (help says: "print the names of packages as they are compiled." ) but instead I got empty output... Hmm maybe its using some cache? Sure enough, when I add the -a flag I finally get mostly the information i'm looking for.
Solution
If you're looking for the list of packages that:

Are not (indirect) test dependencies
Take into account build flags

Simply build your binary using go build -v -a and see which packages are listed there. It also lists standard library packages so it still convoluted with information that you might not be interested in but at least it gives answer to the question.
Ideas
It is not that i'm new to programming with Go, and have spent considerate amount teaching the language to (new) engineer. I'm a big fan of the design rational (having read all the blog posts, they are awesome!) but experiences like the one above leave a lot to be desired. I have the following ideas that could solve the (problematic) user story above.

have a comment in the go.sum file that indicate whether a dependency is indirect and/or just for testing. A I understand is the .sum file acts as the ultimate list of all external dependencies that are needed in one-way or the other and new comers always like to look a files first. Having the information there makes it more accessible
the go why should always have an answer for each module in .sum file. And specifically without any extra flags. A user user goes like this: "Hey there is a dependency here I didn't expect, why?" and expects to get some sort of answer with "go why "
Have a command that shows which external packages will be used for a build without the binary actually being build. I my usecase I just wanted to know what external was shipping with my application.
Maybe update the FAQ in the wiki with some of the information above?
Maybe have the -v flag of the build command show when packages are not being build because of the cache?

Hopefully this information is usefully and I know that there are probably complex underlying issues that I don't understand as i'm not very knowledgable about how it all works. But go modules is also a part of the tooling  new comers will interact with from the get go (phun intentded), so it makes sense to put some effort in the usability aspects.
Cheers,
Ad

Want I want to know is which code ends up in the binary I ship.

That's what the -deps flag to go list is for:
go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{.end}}' ./my-project | sort -u

the go why should always have an answer for each module in .sum file

go.sum is not intended for human consumption. However, go mod graph should include the information you're looking for. (See also #27900.)

Thank you for your response. I think in general my feedback here is that, besides making sure that dependencies are there for a code base, the module system is also a means for engineers to understand a codebase. Especially in large projects, developers need to understand what parts are included when they open and start working on a project. The module system acts as a BoM (bill of materials) in that sense, it acts as the blueprint of a piece of software. In my experience (having tough this practice) the current tooling is ill suited for this activity. I understand that somewhere between:

go list [....]
go mod why [...]
go mod graph [...]
the go.mod file
the go.sum file

there is a way of answering these questions but as it is now this information and spread out and confusing.

Is there something that prevent to put something like // indirect-test in go.mod ?
It's not clear if it's not because we don't want this design or because of a technical difficulty ?

@flibustenet, not all dependencies of your module are listed in your module's go.mod file. It only includes requirements that satisfy direct imports from your module, or that are needed by a transitive dependency but not required in any dependency's go.mod file.
So a comment in the go.mod file would be actively misleading: it would convey the impression that all indirect dependencies are listed there, and in the long term the vast majority probably won't be.

Sometimes users of a lib will check go.mod files to see how heavy a given dependency for their project is. If the repository lists 5 binary dependencies and 95 test dependencies it is definitely less scary to use than a package that lists 100 dependencies (numbers exaggerated, but you get the point).
Separating the binary deps and test deps into 2 separate categories (build and test) inside the go.mod file would avoid this confusion.
require.build (
	dep/1
	dep/2
)

require.test (
	dep/3
	dep/4
)```

I would like to add that in downstream Linux packaging, we would like to have an easy way to split tests dependencies from non-tests dependencies. It is important for us to be able to disable tests and tests deps to solve cyclic dependencies (since we build iteratively, not everything together, it happens often).
(speaking as a Go-SIG member for Fedora).

To complete what @eclipseo wrote: if go.mod stated something was only needed for tests, it would be possible to configure the CI/CD toolchain to ignore it for builds that are not expected to run tests¹
If the info is not available in go.mod, and you need to re-parse the project tree to get it, why bother with the module system at all? You end up rewriting your own private module-like component handling logic.
And I write go.mod here, but I actually not care that much, if the info can be requested via golang/x/mod and not written in the go.mod file itself, that would work for us, though I suspect many would find it terribly inconvenient.
¹ The general case, since the Go definition of a test is lax enough, to allow dependencies on network calls, private servers, private accounts, root access, etc all things that can not be relied to exist outside the original test writer private env. Trying to run a set of Go tests, written by another org, is an exercise in frustration, just to sift real code failures from differences in environment.

@seebs, the general solution for that case is to load the dependency graph lazily, not to omit dependencies from it altogether.
We're looking at what it would take to facilitate lazy loading of the module graph for 1.15. Hopefully we'll have something to propose not too long after the 1.14 freeze.

Can we kindly get an update on this? This is still critical for downstream packaging. We can't upgrade our tooling to include Go modules if we can't spit the test dependencies from the build dependencies.

@eclipseo Does https://github.com/golang/go/issues/26955#issuecomment-437476523  work for you?

@eclipseo Does #26955 (comment) work for you?

As @nim-nim said:

If the info is not available in go.mod, and you need to re-parse the project tree to get it, why bother with the module system at all? You end up rewriting your own private module-like component handling logic.

So no, it's not a solution that we are currently planning. We would somehow rewrite a tool to map the output of go list to the output of go mod, to determine which modules is tests only.

So, to recap the issue,

we have a CI/CD system
for security reasons, it’s not allowed to trawl the internet, only internal stashes of vetted data
runs (including Go builds) are done in network-less ephemeral containers populated from this internal stash
we need to parse go.mod  files (either via custom code relying on golang.org/x/mod, or via parsing the output of a go mod foo  tool provided by the Go project) to identify the Go artefacts that need to be present in an ephemeral container for a Go build to succeed
as long as test dependencies are not clearly indicated as such in go.mod or some go mod foo command provided by the Go project, we are forced to populate containers with test dependencies too
test dependencies are sufficiently problematic that go get ignores them by default. We just want to replicate this mode when populating containers that build Go software

as long as test dependencies are not clearly indicated as such in go.mod or some go mod foo command provided by the Go project, we are forced to populate containers with test dependencies too

You can discover the package dependency structure using go list with either the -json or -f flags. (Also note that go list populates the module cache on demand as it walks the import graph, so you can also use go list itself to download dependencies — go mod download is a simplified command-line API for simple use-cases.)
So, no: you are not “forced” to populate containers with test dependencies; nor do you “need to parse go.mod files” in order for the build to succeed.

We fill an issue to make module usable in offline context.
You reply "you can just use pre-module commands that download things"
This is not constructive (to be polite)

@nim-nim, the design of modules integrates module operations into existing commands. It is not productive to ignore the possibility of using those commands, since they are part of the supported API for working with modules.

@bcmills then suggest things that do not download things and do not require re-parsing the module code.
The source code has already been parsed. That’s what produced go.mod. If we need to reparse it, because you don’t want to produce an inclusive module desing that addresses the needs or others rm -f go.mod and we’ll stop our work to convert to modules.
If your suggestion is to run commands that will download things for us when our use-case is offline then than is also completely useless.
We know about go list. What do you think we’ve been using all those years? If your suggestion is to continue using go list, except it will now do things that are incompatible with our workflow and download things if we make the mistake to enable modules, then we’ll force disable module everywhere forever (we’re already been doing it, except we didn’t want that to get permanent) . And ask all the projects w work with to drop modules and use go dep or any other system that works for our needs.

@nim-nim I support the general idea of this issue, and have been lurking the thread since it was created. Your latest comments haven't been constructive, and I think that's quickly derailing this thread and lowering the chances that it will get any further attention.
To give everyone the bigger picture here, there are lots of important issues with modules that need to be worked on. And there are just a few people working on them full-time, like @bcmills. Just look at the sheer amount of open issues. They aren't perfect people, but they are doing their best to resolve issues in a way that helps Go in the long run. So, please, be respectful.

@nim-nim, my suggestion is to either:
a. populate a Go module cache by whatever means you like (go mod download is one way, but there are others), and then set GOPROXY to point to that cache during offline use, or
b. use go list -json and/or go list -json -m to enumerate dependencies, and then set up a secondary module that uses replace directives to redirect those dependencies in the local filesystem.
Once you have tried either or both of those options, show us the concrete, actionable problems you encountered (problems, not prescriptive solutions!), and we can figure out how to address them in a way that is consistent with the rest of the design of the go command and Go modules.

@bcmills as we told you repeatedly for more than a year we already perform code analysis with golist and we already set up a cache using GOPATH and the result of the code analysis.  And we’ve done tens of thousands of production builds of hundreds of Go projects over the years using this code. It works and we understand why it works and what are its limitations.
However, we do not like maintaining tooling with behaviour that may differ or not from the behaviour of tooling other people had to write for the same reasons.
When Go modules were announced it looked like the Go project was serious about obviating the need of such custom tooling and provide upstream primitives that worked for everyone.
I personally invested months of work to identify the minimal requirements to switch our build system modules. And weeks more to report them.
You persist in dismissing this work. Your answer is basically “just continue to use custom go list logic, I’m not interested in your use case, report if it does not work” (except when we do report, you just ask us to go back to step 1).
FINE
GO111MODULES=OFF as far as I’m concerned.
Not going to invest more months of migration to modules just to get back to the custom go list logic stage.

@nim-nim once again, be cordial. It is fine to disagree, but being condescending or dramatic (e.g. dismissing modules entirely) is unnecessary. See https://golang.org/conduct.
If you really intend to not participate here anymore, that's your choice.

@nim-nim, the numerous issues that you filed are prescribing specific API changes. Specific API changes should go through the proposal process, and in general must meet a high bar — especially if they address use-cases that can already be handled using the existing API.
I am suggesting that you will likely have more success if you report use-cases instead of specific APIs, and make use of the existing APIs already designed for those use-cases when possible. (#35922 is a good example following that model for the use-case of distro packaging specifically. Note that that issue is an ongoing conversation, and we have not yet decided what — if any — additional API may be needed to address that use-case.)

At any rate, this discussion has clearly wandered far from the topic of viewing non-test dependencies. I'm going to hide some comments to get it back on track.

Hello @nim-nim and @eclipseo, one quick comment is that if you haven't already read this, it is probably worth reading https://github.com/golang/go/issues/29410#issuecomment-515203754 and some of the related discussion in #29410 around Go modules packaging approaches for Debian. Debian is distinct from Fedora, so that might not be useful, but maybe there is some overlap at least in terms of some of the challenges.  There is also a short comment there from @qbit (who I think works on packaging for OpenBSD) at https://github.com/golang/go/issues/29410#issuecomment-527678401. Again, perhaps not useful, but wanted to at least mention it.
I have followed some of the different issues you have submitted, but not all of them, so sorry if this is redundant or otherwise not helpful.

I am laymen.
I simply look for when ater I run go mod tidy, those package apparently used only within *_test.go file will not be external

hubris got the better of me and I created: https://github.com/komuw/ote which adds a // test comment next to test dependencies

It would be useful if there was a way to get an equivalent list with go list as you get from go version -m binary.  If you are planning to distribute the binary, it's valuable to know this list quickly perhaps without having to run a build.

Have any resolution?

Hi @so2bin, does https://github.com/golang/go/issues/26955#issuecomment-437476523 work for you?

authors: advanderveer bcmills blitzprog davidkhala dmitris eclipseo flibustenet komuw mvdan myitcv nim-nim rogpeppe so2bin thepudds tim-gp wsc1


text:
cmd/go: go directive is insufficiently documented for module authors to be able to make a decision about its value

Go 1.12 has added a go directive to the go.mod file. It's documented in the following places.
At https://golang.org/doc/go1.12#modules:

The go directive in a go.mod file now indicates the version of the language used by the files within that module. It will be set to the current release (go 1.12) if no existing version is present. If the go directive for a module specifies a version newer than the toolchain in use, the go command will attempt to build the packages regardless, and will note the mismatch only if that build fails.

Additionally, a relevant paragraph from https://golang.org/doc/go1.12#compiler:

The compiler now accepts a -lang flag to set the Go language version to use. For example, -lang=go1.8 causes the compiler to emit an error if the program uses type aliases, which were added in Go 1.9. Language changes made before Go 1.12 are not consistently enforced.

At https://golang.org/cmd/go/#hdr-The_go_mod_file:
go, to set the expected language version;

There is additional information available in issues, proposals, commit messages and code review comments, which I did not mention above. However, that information is not readily accessible to users.
It is very common for package authors to aim to ensure their Go packages can be used in multiple versions of Go. Sometimes it's just Go 1.12.x and Go 1.11.x (the current and previous releases). Other times the goal is to support even older versions of Go. This often includes running tests in CI with those versions of Go to ensure that the build and tests are successful. When these package authors add a go.mod file file to their repositories, they should be able to make a sensible decision about what go directive should be included.
Problem
I believe the current documentation is not sufficient for module authors to make a well-informed decision on whether to include the go directive in their go.mod files, and if so, what the value of the go directive should be.
I think we should try to improve documentation to resolve that. But first I want to make sure others agree that it's a problem.
(This is based on looking over the discussions that have happened around various PRs/CLs where the go directive is introduced, often by people who are very familiar with modules. Sometimes this happens more often due to #30790.)
/cc @ianlancetaylor @dsnet @bcmills @julieqiu @heschik @matloob

I'm sure we should write better docs.
The basic guideline is fairly simple: a "go" directive 1.N means that the code in that module is permitted to use language features that existed in 1.N even if those features were removed in later releases of the language.
In other words, if the code compiles successfully at version 1.N, then it will continue to compile for all later versions of the language, even if it happens to use language features that were later removed.
To a first approximation, nobody should ever have to worry about the "go" directive.  The only likely time you might need to set it manually is if you are copying some existing code to a new module, that code uses some obsolete language feature, and you don't have time to rewrite it.  In that case you might want to set the "go" directive (using go mod edit -go) to the version used by the original module, thus permitting your new module to use the obsolete features.  When writing new code you will presumably simply avoid the obsolete language features.

a "go" directive 1.N means that the code in that module is permitted to use language features that existed in 1.N even if those features were removed in later releases of the language.

My understanding of #28221 is that it should not only permit the older features, but also prohibit newer features. Otherwise, there is little incentive for folks to upgrade to a language version beyond the removed features: they can just set go 1.11 and get every feature.
That understanding is based on this part of the proposal discussion:

suppose we had this feature for the 1.9 release, when type aliases were introduced to the language. Then passing -lang=go1.8 to the compiler would give a syntax error for any code that used a type alias.

That has an interesting interaction with build tags, though: if I set go 1.11 in my go.mod file, but then have a source file somewhere that specifies // +build go1.13, is that file permitted to use language features that were added in 1.13? Is it permitted to use language features that were removed in 1.13? Or should that file be ignored entirely, since the go 1.11 language did not include a go1.13 tag by default?

I would argue that a // +build go1.13 file in a go 1.11 module should be ignored, since it would have been ignored when the module was originally tested under Go 1.11.

(CC @jayconrod)

Those are good questions. When looking for an answer, we should make sure that authors who wish to create Go modules that work with a wide range of Go versions (e.g., 1.8–1.12 as go-cmp does, or 1.1–1.9 as bluemonday does) have a good way of selecting the value of the go directive. /cc @dsnet @buro9

That understanding is based on this part of the proposal discussion:

suppose we had this feature for the 1.9 release, when type aliases were introduced to the language. Then passing -lang=go1.8 to the compiler would give a syntax error for any code that used a type alias.

I made similar conclusions based on that type alias example. When I asked Ian about it, his response was:

When I added the -lang option to cmd/compile, I had to have some way
to test the way the go tool handled it. So, I changed cmd/compile to
reject type aliases when using a -lang option before Go 1.9. But for
future backward-compatible changes, like for example the support for
0o number prefixes that we just added, I do not expect that
cmd/compile will be changed to reject the new features when building
for an older version.
[...] it is not intended to be taken as guidance for future work.

Which made me understand that the type alias example can be misleading when thinking about how the go directive value should be chosen.

I will write some docs.

See #31747 too.

I would argue that a // +build go1.13 file in a go 1.11 module should be ignored, since it would have been ignored when the module was originally tested under Go 1.11.
Or perhaps it should result in an explicit error, since the user may otherwise be confused that it has no effect.

On the basis of this, it seems like an odd behaviour for go test to attempt to mutate the go.mod file if there is no go directive line. If I have a go mod file that makes no claims about the language version, I would think under the model quoted, that should be respected.
Real world justification (reaching because of lack of docs): if a module is aimed at a number of go versions, how is that specified?

@kortschak,  regarding your question above about the interaction with build tags, also worth reading https://github.com/golang/go/issues/31747#issuecomment-488353818 if you haven’t already.

Thanks, @thepudds.

Although not entirely within the scope of this issue a related piece of feedback I have seen (and heard many times more) is that the meaning of the go directive, without even speaking of its exact semantics, tends to be poorly understood by gophers that are new to modules.
The overwhelming reaction is that people believe that it will enforce a specific Go toolchain to be used. As in running your local Go 1.13 toolchain on a module with a go 1.11 directive will result in the 1.11 toolchain being downloaded and invoked instead.
The only real documentation that can be found on the web appears to be the 1.12 release notes (already referenced in the issue description) which leaves enough space for people to come to such a conclusion.
Source (one amongst many): https://github.com/golangci/golangci-lint/pull/605#discussion_r321258359

FWIW, I have seen a fair a amount of confusion on this topic as well.
Now that the playground is running Go 1.13, here is a trivial example that illustrates one aspect of the expected behavior. Setting the Go language version to go 1.11 in the go.mod here causes a compilation error if you use signed integer shifts, which is a Go 1.13 language change:
https://play.golang.org/p/Mb7imkDHhoz
The current error is:
invalid operation: x << s (signed shift count type int, only supported as of -lang=go1.13)
In contrast, setting the Go language version in the go.mod to go 1.13 allows the use of signed integer shifts, as expected:
https://play.golang.org/p/VokbN-hHq02
That (I think?) illustrates how the go directive in a go.mod controls the version of the Go language that is used when building files within that module.

One particular aspect of confusion is people end up debating each other on random issues on open source projects regarding what the go value should be in the go.mod. (And it often is a "random" issue in the sense that someone is often sending a PR unrelated to modules that happens to also initially set or update the go version in the go.mod).
I think it would be helpful if there was a small bit of advice in the documentation on that piece, and I'm not sure "whatever cmd/go picks" is what most projects should be doing currently, or at least there is some nuance to the choice.
(If someone asked me, and if that someone does not yet want to use newer language features like signed integer shifts, then I'd probably personally recommend go 1.11 as a reasonable default choice currently for an open source project, given versions earlier than Go 1.11 don't even read the value. Alternatively, a recommendation could be to set it to the oldest version of Go that the open source project already knows it supports (e.g., go 1.8 or whatever). All that said, I'm not sure of the opinion of the core Go team).

So just to clarify, the version set in the go directive only changes the expected language version of the source code in the module in question, and does not affect anything regarding dependencies or dependency version selection, correct?

The go directive was indeed advisory earlier on, but as of 1.21 it's no longer advisory but rather sets the minimum version of Go required to use the module. This new-to-1.21 behavior is documented at https://go.dev/doc/toolchain and at https://go.dev/ref/mod#go-mod-file-go.
Closing, since this issue was resolved in Go 1.21.

source: gharchive/issue

authors: Helcaraxan bcmills dmitshur ianlancetaylor kortschak rsc thepudds trevyn

repo: golang/go

license_type: permissive

license_source: github-api


text:
cmd/go: go run . uses wrong executable name

If I am in a directory x/y/z (in module mode, but maybe in GOPATH too)
such that the import path of that directory is m/x/y/z, and there is
a single file foo.go in the directory, and I run go run ., the name of
the binary that is built and run should be z. Right now it is foo.
This matters because it shows up in ps, etc.

Is someone handling this issue ? I think I found the code that is setting the "wrong" name for the temporary executable and would like to tackle it ?

authors: Liberatys rsc


text:
runtime: concurrent object placement on heap causes Linux kernel contention

This applies to newer Go versions on any modern Linux kernel, but in this case:
go version go1.11.5 linux/amd64
Linux 4.16.18, SMP
48-thread 24-core 2-socket Haswell machine
If I allocate lots of small objects in parallel from many concurrent routines, they are all placed on the heap linearly, therefore each of the Go threads waiting for Linux pagefaults - which end up becoming more and more expensive because of spinlocking in that path.
In other cases of highly concurrent programming developers rely on allocators like jemalloc to do scalable placement of objects - that option does not exist with Go heap.
In some cases developers chose to have a background page that touches/cleans the pages before giving them to concurrent routines.
Having some spread in object placement would be a useful feature in the runtime.
A very basic program I have written to illustrate this in Go  spends 50s in kernel while only 3s in userland, mostly from inflated costs in faulting:
2.75user 50.06system 0:01.19elapsed 4405%CPU (0avgtext+0avgdata 522224maxresident)k
0inputs+0outputs (0major+221566minor)pagefaults 0swaps

perf report shows that most of the cost from running my program is in page fault spinlocks:
-   91.23%     0.00%  tt       [kernel.vmlinux]                                [k] page_fault                                                                                ▒
   - 91.22% page_fault                                                                                ▒
      - 91.11% __do_page_fault                                                                                ▒
         - 90.96% handle_mm_fault                                                                                ▒
            - 90.94% __handle_mm_fault                                                                                ▒
               + 62.74% queued_spin_lock_slowpath                                                                                ▒
               - 27.99% do_wp_page                                                                                ▒
                  - 27.98% wp_page_copy                                                                                ▒
                     + 24.77% queued_spin_lock_slowpath                                                                                ▒
                     + 2.91% ptep_clear_flush

Source code for it:
package main
import (
	"strings"
	"sync"
)
func main() {
	var wg sync.WaitGroup
	defer wg.Wait()

	rp := strings.Repeat("a", 100)

	for n := 0; n < 100; n++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			a := make(map[int]map[string]string)
			for z := 0; z < 10000; z++ {
				a[z] = map[string]string{"x": rp + "a",}
			}
		}()
	}
}

Setting GOMAXPROCS=1 shows how little amount of work kernel has to do without contention:
# time ./tt

real	0m1.190s
user	0m3.552s
sys	0m48.070s
# time GOMAXPROCS=1  ./tt

real	0m0.801s
user	0m0.790s
sys	0m0.012s

CC @aclements @mknyszek

Maybe I don't understand what you mean by "spread in object placement", but for a long time Go has placed objects in separate pages depending on which P ("processor") is allocating, and it's able to do this without any locking. This is similar to TCMalloc's design. Allocating pages requires a lock, but small object allocation already scales fairly well.
Furthermore, I'm not actually able to reproduce the exact behavior you're seeing on your machine with your example program. I get:
thing $ GOMAXPROCS=1 /usr/bin/time ./thing
0.67user 0.00system 0:00.67elapsed 100%CPU (0avgtext+0avgdata 39944maxresident)k
0inputs+0outputs (0major+649minor)pagefaults 0swaps
thing $ /usr/bin/time ./thing
3.17user 8.90system 0:00.34elapsed 3532%CPU (0avgtext+0avgdata 476600maxresident)k
0inputs+0outputs (0major+18703minor)pagefaults 0swaps

It's true there are many more page faults, and significantly more time is spent in the kernel, but overall elapsed time still improves.
GODEBUG=gctrace=1 on the other hand indicates that these page faults are coming from the heap growing and touching many new pages.
thing $ GODEBUG=gctrace=1 ./thing
gc 1 @0.006s 8%: 0.020+2.9+0.66 ms clock, 0.96+1.0/9.1/0+32 ms cpu, 5->7->5 MB, 6 MB goal, 48 P
gc 2 @0.013s 14%: 0.22+4.3+1.0 ms clock, 10+4.8/25/0+51 ms cpu, 8->13->11 MB, 10 MB goal, 48 P
gc 3 @0.026s 13%: 0.38+11+1.1 ms clock, 18+1.6/56/0+54 ms cpu, 17->30->26 MB, 22 MB goal, 48 P
gc 4 @0.045s 16%: 0.86+13+1.7 ms clock, 41+2.9/86/0+82 ms cpu, 33->49->46 MB, 53 MB goal, 48 P
gc 5 @0.078s 14%: 0.72+16+0.33 ms clock, 34+9.7/144/0+15 ms cpu, 63->90->86 MB, 93 MB goal, 48 P
gc 6 @0.131s 12%: 0.43+43+0.50 ms clock, 20+5.2/326/0+24 ms cpu, 124->193->187 MB, 173 MB goal, 48 P
gc 7 @0.238s 12%: 1.3+59+1.5 ms clock, 63+15/525/0+73 ms cpu, 258->381->361 MB, 374 MB goal, 48 P
thing $ GODEBUG=gctrace=1 GOMAXPROCS=1 ./thing
gc 1 @0.007s 30%: 0.005+9.8+0.005 ms clock, 0.005+4.1/1.3/0+0.005 ms cpu, 4->5->5 MB, 5 MB goal, 1 P
gc 2 @0.018s 27%: 0.001+5.3+0.001 ms clock, 0.001+1.2/0/0+0.001 ms cpu, 6->9->4 MB, 10 MB goal, 1 P
gc 3 @0.025s 29%: 0.001+8.4+0.001 ms clock, 0.001+3.2/0.088/0+0.001 ms cpu, 5->9->8 MB, 8 MB goal, 1 P
gc 4 @0.034s 29%: 0.001+10+0.002 ms clock, 0.001+1.5/1.9/0+0.002 ms cpu, 9->15->9 MB, 17 MB goal, 1 P
gc 5 @0.046s 29%: 0.001+3.6+0.001 ms clock, 0.001+1.0/0/0+0.001 ms cpu, 11->13->3 MB, 19 MB goal, 1 P
gc 6 @0.051s 28%: 0.001+9.8+0.001 ms clock, 0.001+0.57/2.3/0+0.001 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 7 @0.062s 28%: 0.001+3.8+0.001 ms clock, 0.001+1.1/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 8 @0.067s 27%: 0.001+9.9+0.001 ms clock, 0.001+0.57/2.4/0+0.001 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 9 @0.078s 27%: 0.001+3.8+0.001 ms clock, 0.001+1.1/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 10 @0.084s 27%: 0.001+10+0.002 ms clock, 0.001+0.58/2.5/0+0.002 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 10 @0.084s 27%: 0.001+10+0.002 ms clock, 0.001+0.58/2.5/0+0.002 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 11 @0.095s 27%: 0.001+3.8+0.001 ms clock, 0.001+1.0/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 12 @0.100s 27%: 0.001+10+0.003 ms clock, 0.001+0.57/2.7/0+0.003 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 13 @0.112s 27%: 0.002+3.9+0.001 ms clock, 0.002+1.1/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 14 @0.117s 27%: 0.001+10+0.003 ms clock, 0.001+0.58/2.8/0+0.003 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 15 @0.129s 27%: 0.003+3.8+0.001 ms clock, 0.003+1.0/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 16 @0.134s 26%: 0.001+9.8+0.001 ms clock, 0.001+0.62/2.1/0+0.001 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 17 @0.145s 26%: 0.001+3.7+0.001 ms clock, 0.001+1.0/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 18 @0.150s 26%: 0+9.9+0.002 ms clock, 0+0.58/2.3/0+0.002 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 19 @0.161s 26%: 0.001+3.8+0.001 ms clock, 0.001+1.0/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 20 @0.166s 26%: 0.001+10+0.002 ms clock, 0.001+0.59/2.5/0+0.002 ms cpu, 4->10->9 MB, 7 MB goal, 1 P
gc 21 @0.177s 26%: 0.001+3.5+0.001 ms clock, 0.001+0.98/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 22 @0.182s 26%: 0+8.5+0.001 ms clock, 0+0.57/1.8/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 23 @0.192s 26%: 0.001+3.0+0.001 ms clock, 0.001+0.81/0/0+0.001 ms cpu, 10->13->3 MB, 19 MB goal, 1 P
gc 24 @0.196s 26%: 0+7.8+0.001 ms clock, 0+0.53/1.6/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 25 @0.205s 26%: 0.001+2.9+0.001 ms clock, 0.001+0.77/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 26 @0.209s 26%: 0.001+8.4+0.003 ms clock, 0.001+0.49/2.2/0+0.003 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 27 @0.218s 26%: 0.002+2.7+0.001 ms clock, 0.002+0.75/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 28 @0.222s 25%: 0+7.9+0.001 ms clock, 0+0.52/1.5/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 29 @0.231s 25%: 0.001+2.6+0.001 ms clock, 0.001+0.73/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 30 @0.234s 25%: 0.001+8.3+0.002 ms clock, 0.001+0.57/1.8/0+0.002 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 31 @0.244s 25%: 0.001+2.7+0.001 ms clock, 0.001+0.72/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 32 @0.248s 26%: 0.001+5.8+0.003 ms clock, 0.001+2.3/0.56/0+0.003 ms cpu, 4->7->6 MB, 6 MB goal, 1 P
gc 33 @0.255s 26%: 0.001+3.9+0 ms clock, 0.001+0.88/0.79/0+0 ms cpu, 7->9->5 MB, 13 MB goal, 1 P
gc 34 @0.260s 26%: 0+2.4+0 ms clock, 0+0.70/0/0+0 ms cpu, 6->7->2 MB, 10 MB goal, 1 P
gc 35 @0.264s 26%: 0+8.3+0.001 ms clock, 0+0.73/1.4/0+0.001 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 36 @0.273s 25%: 0+2.7+0 ms clock, 0+0.72/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 37 @0.277s 25%: 0+8.2+0.001 ms clock, 0+0.54/1.7/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 38 @0.286s 25%: 0.001+2.8+0.001 ms clock, 0.001+0.75/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 39 @0.290s 25%: 0+8.1+0.001 ms clock, 0+0.58/1.5/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 40 @0.299s 25%: 0.001+2.6+0.001 ms clock, 0.001+0.71/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 41 @0.303s 25%: 0+8.1+0.002 ms clock, 0+0.55/1.6/0+0.002 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 42 @0.312s 25%: 0.001+2.7+0 ms clock, 0.001+0.71/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 43 @0.316s 25%: 0+7.7+0 ms clock, 0+0.58/1.3/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 44 @0.325s 25%: 0+2.6+0 ms clock, 0+0.71/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 45 @0.328s 25%: 0+7.5+0 ms clock, 0+0.55/1.4/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 46 @0.337s 25%: 0+2.5+0 ms clock, 0+0.67/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 47 @0.341s 25%: 0.001+7.4+0.001 ms clock, 0.001+0.53/1.5/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 48 @0.349s 25%: 0+2.5+0 ms clock, 0+0.71/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 49 @0.352s 25%: 0+7.4+0 ms clock, 0+0.53/1.3/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 50 @0.361s 25%: 0+2.4+0 ms clock, 0+0.66/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 51 @0.364s 25%: 0+8.1+0.001 ms clock, 0+0.58/1.7/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 52 @0.373s 25%: 0.012+2.7+0 ms clock, 0.012+0.74/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 53 @0.377s 24%: 0+8.0+0.013 ms clock, 0+0.53/1.5/0+0.013 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 54 @0.386s 24%: 0.002+2.8+0.001 ms clock, 0.002+0.73/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 55 @0.390s 24%: 0+8.1+0.001 ms clock, 0+0.52/1.6/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 56 @0.399s 24%: 0+2.6+0 ms clock, 0+0.69/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 57 @0.403s 24%: 0+8.0+0.002 ms clock, 0+0.55/1.4/0+0.002 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 58 @0.412s 24%: 0+2.6+0 ms clock, 0+0.69/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 59 @0.416s 24%: 0+7.9+0.001 ms clock, 0+0.53/1.5/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 60 @0.424s 24%: 0+2.6+0 ms clock, 0+0.67/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 61 @0.428s 24%: 0+7.5+0 ms clock, 0+0.51/1.5/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 62 @0.437s 24%: 0+2.5+0 ms clock, 0+0.68/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 63 @0.440s 24%: 0+7.5+0.001 ms clock, 0+0.58/1.3/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 64 @0.449s 24%: 0+2.6+0 ms clock, 0+0.72/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 65 @0.452s 24%: 0+7.5+0 ms clock, 0+0.50/1.5/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 66 @0.461s 24%: 0+2.6+0 ms clock, 0+0.67/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 67 @0.464s 24%: 0+7.7+0.001 ms clock, 0+0.53/1.5/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 68 @0.473s 24%: 0+2.7+0.001 ms clock, 0+0.74/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 69 @0.477s 24%: 0+7.9+0.001 ms clock, 0+0.57/1.4/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 70 @0.486s 24%: 0.001+2.7+0.001 ms clock, 0.001+0.73/0/0+0.001 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 71 @0.490s 24%: 0+7.7+0.001 ms clock, 0+0.54/1.4/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 72 @0.498s 24%: 0+2.6+0 ms clock, 0+0.67/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 73 @0.502s 24%: 0+7.5+0 ms clock, 0+0.46/1.5/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 74 @0.510s 24%: 0+2.5+0 ms clock, 0+0.65/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 75 @0.514s 24%: 0+7.7+0 ms clock, 0+0.52/1.5/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 76 @0.523s 24%: 0.001+2.4+0 ms clock, 0.001+0.64/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 77 @0.526s 24%: 0+7.4+0 ms clock, 0+0.54/1.3/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 78 @0.535s 24%: 0+2.4+0 ms clock, 0+0.64/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 79 @0.538s 24%: 0+7.4+0.001 ms clock, 0+0.53/1.4/0+0.001 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 80 @0.547s 24%: 0+2.3+0 ms clock, 0+0.63/0/0+0 ms cpu, 10->12->3 MB, 19 MB goal, 1 P
gc 81 @0.550s 24%: 0+7.2+0 ms clock, 0+0.49/1.4/0+0 ms cpu, 4->10->9 MB, 6 MB goal, 1 P
gc 82 @0.558s 24%: 0+2.2+0 ms clock, 0+0.60/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 83 @0.562s 24%: 0+7.3+0.001 ms clock, 0+0.56/1.2/0+0.001 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 84 @0.570s 24%: 0+2.3+0 ms clock, 0+0.64/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 85 @0.573s 23%: 0+7.4+0.001 ms clock, 0+0.62/1.2/0+0.001 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 86 @0.582s 23%: 0+2.4+0 ms clock, 0+0.64/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 87 @0.585s 23%: 0+7.7+0.001 ms clock, 0+0.69/1.2/0+0.001 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 88 @0.594s 23%: 0+2.2+0 ms clock, 0+0.59/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 89 @0.597s 23%: 0+7.3+0 ms clock, 0+0.64/1.2/0+0 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 90 @0.606s 23%: 0+2.3+0 ms clock, 0+0.61/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 91 @0.609s 23%: 0+7.3+0 ms clock, 0+0.64/1.2/0+0 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 92 @0.617s 23%: 0.001+2.1+0 ms clock, 0.001+0.59/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 93 @0.621s 23%: 0+7.5+0.002 ms clock, 0+0.76/1.1/0+0.002 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 94 @0.629s 23%: 0.001+2.2+0 ms clock, 0.001+0.57/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 95 @0.633s 23%: 0+7.5+0.001 ms clock, 0+0.71/1.2/0+0.001 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 96 @0.641s 23%: 0.001+2.1+0 ms clock, 0.001+0.53/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 97 @0.645s 23%: 0+7.4+0.002 ms clock, 0+0.68/1.2/0+0.002 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 98 @0.653s 23%: 0.001+2.1+0 ms clock, 0.001+0.57/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 99 @0.657s 23%: 0+7.4+0.001 ms clock, 0+0.73/1.1/0+0.001 ms cpu, 4->10->9 MB, 5 MB goal, 1 P
gc 100 @0.665s 23%: 0+2.1+0 ms clock, 0+0.53/0/0+0 ms cpu, 10->12->2 MB, 19 MB goal, 1 P
gc 101 @0.669s 23%: 0+7.3+0.001 ms clock, 0+1.3/0.77/0+0.001 ms cpu, 4->9->8 MB, 5 MB goal, 1 P

When we don't set GOMAXPROCS=1, the runtime assumes the heap just keeps growing. There are few GCs, but the heap grows to  almost 400 MiB which is the cause of all those page faults (AFAICT).
When we set GOMAXPROCS=1, our heap stays small, and we end up re-using a lot of memory and don't have to fault in new pages quite so often.
For comparison, these are the parameters of the machine I was using:
go version go1.12.5 linux/amd64
Linux 4.19.37-amd64, SMP
48-thread 24-core 2-socket Haswell machine
There may be a legitimate bug here (I need to think more about the sample program), but the bug is likely not allocator scalability.

Weird, to have some common ground I set up a google cloud instance (n1-standard-64), downloaded latest stable golang:
go version go1.12.7 linux/amd64
Linux my-vm 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64 GNU/Linux
Even named the binary 'thing':
domas_mituzas@my-vm:~/thing$ time ./thing
real	0m6.469s
user	0m3.764s
sys	6m32.516s
domas_mituzas@my-vm:~/thing$ time GOMAXPROCS=1  ./thing
real	0m1.038s
user	0m0.916s
sys	0m0.124s
And indeed, allocator scalability itself is fantastic, which is why it slams into kernel scalability. I did not look much at how Go places objects - if really objects are placed into separate blocks, then it is hitting another page table contention that may be harder to work around (multiple mappings could do, though).
I have a very basic C testcase that shows how this breaks inside kernel:
# /usr/bin/time ./x
5.59user 951.16system 0:20.20elapsed 4735%CPU (0avgtext+0avgdata 1050196maxresident)k
0inputs+0outputs (0major+24724446minor)pagefaults 0swaps

#define _GNU_SOURCE

#include <unistd.h>
#include <pthread.h>
#include <stdlib.h>
#include <sys/sysinfo.h>
#include <stdio.h>

char *x;

unsigned long long arena_size = 1L * 1024 * 1024 * 1024;
unsigned long long start;

void * worker(void *p) {
	int n;
	while (n<arena_size) {
		x[n]++;
		n += 4095;
	}
}

int main( int ac, char ** av ) {
	int n, nthr = get_nprocs();
	pthread_t *threads = malloc(sizeof(pthread_t) * nthr);

	x = malloc(arena_size);
	if (x==(caddr_t)-1) {
		exit(1);
	}

	for (n=0; n<nthr; n++)
		pthread_create(&threads[n], NULL, worker, NULL);

	while (n--)
		pthread_join(threads[n], NULL);

	return 0;
}

Let me check the properties of random arena access vs linear.

update with latest stable kernel (5.2.1) and go1.12.7:
gcoff, ncpu1: 0.37user 0.34system 0:00.72elapsed 100%CPU (0avgtext+0avgdata 535076maxresident)k
gcoff: 1.29user 287.80system 0:04.83elapsed 5976%CPU (0avgtext+0avgdata 540384maxresident)k

gcon, ncpu1: 0.62user 0.01system 0:00.64elapsed 99%CPU (0avgtext+0avgdata 21436maxresident)k
gcoff: 3.93user 251.22system 0:04.23elapsed 6022%CPU (0avgtext+0avgdata 516776maxresident)k

random access to same gigabyte does not show same contention signature:
-   95.93%    38.21%  mmapr    mmapr                                                     [.] worker                                                                                ▒
   - 57.71% worker                                                                                ▒
      - 56.39% page_fault                                                                                ◆
         - 56.35% __do_page_fault                                                                                ▒
            - 44.17% handle_mm_fault                                                                                ▒
               - 43.97% __handle_mm_fault                                                                                ▒
                  - 38.50% do_wp_page                                                                                ▒
                     - 38.47% wp_page_copy                                                                                ▒
                        - 35.38% ptep_clear_flush                                                                                ▒
                           - 35.13% flush_tlb_mm_range                                                                                ▒
                              - 34.35% on_each_cpu_cond_mask                                                                                ▒
                                 - 31.15% on_each_cpu_mask                                                                                ▒
                                    - 30.84% smp_call_function_many                                                                                ▒
                                       - 9.27% llist_add_batch                                                                                ▒
                                          + 1.97% call_function_interrupt                                                                                ▒
                                       - 8.88% native_sched_clock                                                                                ▒
                                            2.56% cyc2ns_read_begin.part.2                                                                                ▒
                                          + 1.20% call_function_interrupt                                                                                ▒
                                       + 2.16% call_function_interrupt                                                                                ▒
                                       + 1.73% native_send_call_func_ipi                                                                                ▒
                                       + 0.88% cpumask_next                                                                                ▒
                                 + 1.76% cpumask_next                                                                                ▒
                                   1.06% tlb_is_not_lazy                                                                                ▒
                        + 1.54% alloc_pages_vma                                                                                ▒
                    1.50% queued_spin_lock_slowpath                                                                                ▒
                    1.18% _raw_spin_lock                                                                                ▒
            - 6.60% down_read_trylock                                                                                ▒
               - 1.12% call_function_interrupt                                                                                ▒
                  + 0.91% smp_call_function_interrupt                                                                                ▒
              2.61% up_read                                                                                ▒
            + 1.99% down_read                                                                                ▒
   + 38.21% start_thread

meanwhile, single thread allocs seem to be much much more efficient:
-   88.96%    20.99%  mmapr    mmapr               [.] worker                                                                                ▒
   - 67.97% worker                                                                                ▒
      - 56.31% page_fault                                                                                ▒
         - 55.74% __do_page_fault                                                                                ◆
            - 52.29% handle_mm_fault                                                                                ▒
               - 50.21% __handle_mm_fault                                                                                ▒
                  - 46.11% do_wp_page                                                                                ▒
                     - 45.49% wp_page_copy                                                                                ▒
                        - 23.04% alloc_pages_vma                                                                                ▒
                           - 21.95% __alloc_pages_nodemask                                                                                ▒
                              - get_page_from_freelist                                                                                ▒
                                   16.18% clear_page_erms                                                                                ▒
                        - 8.75% __lru_cache_add                                                                                ▒
                           + pagevec_lru_move_fn                                                                                ▒
                        + 4.73% ptep_clear_flush                                                                                ▒
                        + 3.78% mem_cgroup_try_charge_delay                                                                                ▒
                          0.52% mem_cgroup_commit_charge

(I'm having all the threads randomly walk the same memory, instead of linearly, with some adjustments to the .c above).
So, there's definitely ~25x pagefaulting cost increase in random allocs (very little of that in actual spinlocking), but the linear edge case shows ~1700x pagefaulting cost increase, mostly in spinlocks, which I trigger with my Go testcase too (which indicates that the page gets faulted from multiple threads at once).
By changing my C test to linearly allocate different maps in each thread I can see that the scaling behavior is very similar to random-order allocation - increased costs are from flush_tlb_mm_range, which would be hard to avoid, and down_read_trylock.
I'm really waving my hands here, but from what I see - the profiles of go testcase are indicative of pagefaults hitting same page and a spinlock, rather than TLB flushing.

In some of my other tests I ended up hitting runtime.(*mcentral).cacheSpan contention (simply because I was allocating lots of large objects).
The main way how I hit this pagefault issue is with fast mapassign functions, and this code in concurrent fashion will hit high % in kernel spinlocks:
type Maps struct {
l *Maps
m map[int]int
}
var prev *Maps
for z := 0; z < 100000; z++ {
	prev = &Maps{prev, map[int]int{1:1}}
}

i can reproduce this on my machine: centos 7.3.1611, and the results are weird :

build the same test go program above(the: ./tt) with golang 1.11.1 , result:
[thing]$ time GOMAXPROCS=1 ./thing
real	0m5.026s
user	0m1.103s
sys	0m3.928s
[thing]$ time ./thing
real	0m2.318s
user	0m1.667s
sys	0m5.307s

then i changed GOROOT to go1.9.1 dir on the same machine ,then build , performance is much better:
[thing]$ time ./thing
real	0m0.886s
user	0m2.219s
sys	0m1.114s
[thing]$ time GOMAXPROCS=1 ./thing
real	0m1.928s
user	0m1.544s
sys	0m0.382s

switch back to GOROOT 1.11.1 again, then build, performance is like go 1.9:
[thing]$ time ./thing
real	0m0.700s
user	0m1.904s
sys	0m0.716s

rm -f ~/.cache/*   build with go1.11.1 again, SLOW:
[thing]$ time ./thing
real	0m1.021s
user	0m1.549s
sys	0m2.272s

@bradfitz      runtime: concurrent object placement on heap causes Linux kernel contention

retesting with 1.10, 1.13 and master as of today (2020 feb):
go version go1.10.4 linux/amd64
real	0m26.668s
user	0m2.680s
sys	15m28.629s
go version go1.13.7 linux/amd64
real	0m13.369s
user	0m2.359s
sys	7m52.419s
go version devel +ca8bf63809 Sat Feb 8 16:15:48 2020 +0000 linux/amd64
real	0m0.282s
user	0m1.908s
sys	0m5.713s
singlethreaded perf between master and 1.13 does not change much
this can be closed

@ianlancetaylor should this repro be added to the runtime test sequence?

@networkimprov We could add this as a benchmark. Feel free to submit a CL.
I don't think we want to make it a test. We abhor flaky tests, and I think this one would be hard to make reliable enough to include by default.

Re a CL, cc @mknyszek @tcolgate as they're already tagged "Contributor".

authors: ianlancetaylor midom mknyszek networkimprov randall77 teou


text:
vendor/golang.org/x/xerrors/adaptor_go1_13.go:16:14: undefined: errors.Frame ...

What version of Go are you using (go version)?
1.13.0
Does this issue reproduce with the latest release?
Yes
What did you do?
Trying to build my project after upgrading to Golang 1.13.0
And I get those errors:
vendor/golang.org/x/xerrors/adaptor_go1_13.go:16:14: undefined: errors.Frame
vendor/golang.org/x/xerrors/format_go1_13.go:12:18: undefined: errors.Formatter

I saw a related issue there: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240319
But I don't understand what could help me to fix? It seems on Golang side no?
Thank you,

You need to update your vendored golang.org/x/xerrors package.  It was updated for a preliminary version of Go 1.13, but the final Go 1.13 release changed.  xerrors was corrected for that change in May, 2019, and you are missing that correction.

@ianlancetaylor I have deleted all my cache (/go/pkg/mod/ and also /go/src) but the problem persists.
That's the same when building inside a clean Jenkins pipeline.
In my go.sum file I can see:
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522 h1:bhOzK9QyoD0ogCnFro1m2mz41+Ib0oOhfJnBp5MR4K4=
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=

😢

@sneko 3ee3066db522 is not the latest version of golang.org/x/xerrors. You need version golang/xerrors@a985d3407aa71f30cf86696ee0a2f409709f22e1.
I had the same problem and go get -u golang.org/x/xerrors did the trick.

xerrors is not part of a Go release.  It is from the repo golang.org/x/xerrors.  That repo exists independently and is not updated as part of a Go release.

Thank you both for your answers 😃!
In my case, this is due to a dependency code (https://github.com/google/go-cloud/commit/31e061dd5727ffc5064e16da45f7779b806ff82d) not yet released.

authors: ianlancetaylor mmcloughlin sneko


text:
net/mail: ParseAddress cannot parse email address with name contains chinese character

What version of Go are you using (go version)?

$ go version
go version go1.13.5 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/username/.cache/go-build"
GOENV="/home/username/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/username/dev/go-packages/"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/username/dev/go/go1.13.5"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/username/dev/go/go1.13.5/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build697152646=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
    "fmt"
    "net/mail"
)

func main() {

    // 测试<example@example.com>
    addr1 := "=?GB2312?B?suLK1A==?= <example@example.com>"
    addr2 := "=?GB18030?B?suLK1A==?= <example@example.com>"
    addr3 := "=?GBK?B?suLK1A==?= <example@example.com>"

    var err error

    _, err = mail.ParseAddress(addr1)
    fmt.Println(err) // mail: missing word in phrase: charset not supported: "gb2312"

    _, err = mail.ParseAddress(addr2)
    fmt.Println(err) // mail: missing word in phrase: charset not supported: "gb18030"

    _, err = mail.ParseAddress(addr3)
    fmt.Println(err) // mail: missing word in phrase: charset not supported: "gbk"
}

What did you expect to see?
Chinese name should be parsed correctly.
What did you see instead?
mail: missing word in phrase: charset not supported: "gb2312"

Why not use utf-8? See https://golang.org/pkg/mime/#WordDecoder:

utf-8, iso-8859-1 and us-ascii charsets are handled by default.

I don't think we want to be implementing all possible charsets out there in the standard library. If you really need to support arbitrary charsets, you should probably supply your own word decoder via https://golang.org/pkg/net/mail/#AddressParser.

Why not use utf-8? See https://golang.org/pkg/mime/#WordDecoder:

Because the email is from the sender, I cannot control the charset they use.

I don't think we want to be implementing all possible charsets out there in the standard library. If you really need to support arbitrary charsets, you should probably supply your own word decoder via https://golang.org/pkg/net/mail/#AddressParser.

Thanks for the response, I'll try to make my own word decoder.

Great. I'm closing this for now. If you want to open an issue for the mime package to support many more encodings, you can do that too, but we'd probably need a good argument to add more code to the standard library.


text:
image/color: documentation doesn't include links to relevant color theory resources

What version of Go are you using (go version)?

$ go version
1.13.7

Does this issue reproduce with the latest release?
Yes: https://tip.golang.org/pkg/image/color/
What did you expect to see?
The image/color relies on some color theory. But no pointers are given to learn about that theory.
Here are a few questions for which the documentation should give answers or even just hints:

What is Alpha?
What is the difference between alpha-premultiplied and non-alpha-premultiplied (some answer exist in doc for type RGBA, but more in overview would be helpful) and when should one be used.

Also, no examples. That would be useful to understand Model.
What did you see instead?
The package overview section is a single sentence:

Package color implements a basic color library.

There were some old blog posts on image packages at https://blog.golang.org/go-image-package and https://blog.golang.org/go-imagedraw-package that have additional relevant information. Perhaps linking to those is a good idea.
/cc @robpike per owners. Also /cc @nigeltao.

authors: dmitshur dolmen


text:
encoding/json: Number no longer accepts invalid numbers

What version of Go are you using (go version)?

$ go version
go version go1.14 darwin/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dean/Library/Caches/go-build"
GOENV="/Users/dean/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=" -mod="
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY="github.com/segment*"
GONOSUMDB="github.com/segment*"
GOOS="darwin"
GOPATH="/Users/dean/dev"
GOPRIVATE="github.com/segment*"
GOPROXY="https://proxy.golang.org"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/dean/go/json-number-test/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9f/y8mlmc1n7p9_g_8dkfr3zr400000gn/T/go-build201012045=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
I decoded an empty string into a json.Number
https://play.golang.org/p/BasiDL88-1O
What did you expect to see?
{Num:}
{Num:1.0}
{Num:blah}
{Num:1e3}

What did you see instead?
panic: json: invalid number literal, trying to unmarshal "\"\"" into Number

goroutine 1 [running]:
main.test(0x110164f, 0xa)
        /Users/dean/go/json-number-test/main.go:23 +0x14d
main.main()
        /Users/dean/go/json-number-test/main.go:13 +0x36
exit status 2

I realize that this was done on purpose and released in 1.14 and it technically doesn't break the Go1 compatibility promise, however, it is a big behavioural breaking change and am hoping it can be reverted.
I cannot speak for the rest of the community but I know many instances where json.Number was specifically used for this very behaviour and now would require code changes to get the same behaviour back; code changes to code that we may not even own.
It was also used because it was a std lib type that supported this behaviour that didn't pull in or make another dependency that caused abstraction leakage.

Another instance of #37308 and #14702.
CC @mvdan

It appears that this has been discussed pretty thoroughly, if it is to remain that's ok but if I could request the release notes or somewhere there could be documentation demonstrating the fix or showing the use of an alternative like json.RawMessage just to guide people better during the transition.

Hey @deankarn. As you mentioned, this was discussed in #37308. The release notes contain documentation noting the change. I'm going to close this issue. if you feel like this was closed in error please comment below.

authors: cagedmantis deankarn ianlancetaylor


text:
Segmantation Fault when initializing slice: type []string and initializing one memnber with space in the begining

What version of Go are you using (go version)?

$ go version go1.10.4 linux/amd64

Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GORACE=""
GOROOT="/usr/lib/go-1.10"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build512368729=/tmp/go-build -gno-record-gcc-switches"

What did you do?
initialized slice:
var myslice []string = []string{"abcdefg", " ABCDEFG"} <<-- pay attention to the space in the second element

What did you expect to see?
warning, not crash
What did you see instead?
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x528886]

Please provide a full program that crashes, with reproduction instructions.
That line definitely shouldn't segfault, space or not.

Ok, ill check whats up on my side again,


text:
x/pkgsite: add GetLicenses function to return licenses for a given path

As part of the new data model migration (#39629), we will have a single function get licenses for a given path (regardless of whether that path is a package, module, or directory).
This function will replace the existing LegacyGetPackageLicenses and LegacyGetModuleLicenses.
The function signature will be:
GetLicenses(ctx context.Context, fullpath, modulePath, resolvedVersion string)([]*licenses.License, error)

The function will execute the query below:
SELECT
    l.types,
    l.file_path,
    l.contents,
    l.coverage
FROM
    licenses l
INNER JOIN
    paths p
ON
    p.module_id=l.module_id
INNER JOIN
    modules m
ON
    p.module_id=m.id
WHERE
    p.path = $1
    AND m.module_path = $2
    AND m.version = $3;

Only licenses relevant to the path should be returned. A license applies to the path if the license is in the path directory, or any of its parent directories.

Hi @julieqiu
If allowed, may I pick this up?

Please do! :)

Hi @julieqiu
Thanks for allowing me to work on this issue ))
Just to confirm if I understood it correctly, we need to implement -
func (db *DB) GetLicenses(ctx context.Context, fullPath, modulePath, resolvedVersion string) (_ []*licenses.License, err error) {
in internal/postgres/postgres.go file to fetch the list of licenses at the given fullPath & modulePath.
I am correct?
If yes, then internal.DataSource interface will have GetLicenses( ) as contract.
As a result, DataSource in internal/proxydatasource will also need an implementation of GetLicenses().

Thank you for contributing! :)
That is correct. I would add the function to internal/postgres/licenses.go though, as opposed to internal/postgres/postgres.go.

ohh yeah...
Thanks @julieqiu for correcting me.
Will try to implement tomorrow.

@julieqiu
Is there a way to load data data in my local docker postgres? Any doc you can point to?

Yup! Check out doc/worker.md#populating-data-locally-using-the-worker for instructions.
By the way, if you have other questions, feel free to reach out in the #pkgsite channel on Gophers Slack!

authors: amarjeetanandsingh julieqiu


text:
http request store context in structure, it may cause some bug?

https://github.com/golang/go/blob/fa04d488bd54b8fdd78cc9bcc6d90de4bf5f8efb/src/net/http/request.go#L319
according to official document, do not store Contexts inside a struct type, but request.go did(https://golang.org/pkg/context/)

Thank you for raising this issue. Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions.
The request.ctx field was added in Go 1.7 in this change https://go-review.googlesource.com/c/go/+/21496/ I encourage you to review the history of the change and the associated discussion.
GitHub Issues are used for tracking bugs and proposals only.
For asking questions, see:

Stack Overflow with questions tagged "go"

The Go Forum, a web-based forum

Gophers Slack, use the invite app for access. The #general channel is a good starting point.

Go Community on Hashnode with questions and posts tagged with "go"

The golang-nuts mailing list

IRC channel #go-nuts on Freenode


text:
"Exception 0xc0000005", "runtime: unknown pc" on program exit with loaded windows dll

What version of Go are you using (go version)?

$ go version
go version go1.15.5 windows/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
Windows 10 Pro, 20H2, build 19042.630
go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Marek\AppData\Local\go-build
set GOENV=C:\Users\Marek\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=d:\Go.external\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=d:\Go.external;d:\Go
set GOPRIVATE=
set GOPROXY=direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\Marek\AppData\Local\Temp\go-build143512754=/tmp/go-build -gno-record-gcc-switches

What did you do?
Our IS vendor provides us with DLL to access information system from scripts etc. Recently I recompiled one of our scripts and noticed, that after successful execution it ends with memory dump, registry dump, stacktrace and command window freeze.
When compiling with go 1.14.12 everything works. When compiling with any go from 1.15.x series it throws the error. To reproduce it's enough to load the DLL even without executing any function from it.

package main

import (
	"log"
	"golang.org/x/sys/windows"
)

func main() {
	ec := windows.NewLazyDLL("Econnect.dll")
	err := ec.Load()
	if err != nil {
		log.Println(err)
	}
}

Unfortunately I don't have source code for the library nor info about compilation options (the communication with our IS vendor is a bit difficult), so I'm attaching the binary and a test script. Any other DLL that I have tested on my system loads, works and exits cleanly.
Loading and calling the library from Python works without error too.
xtest.zip
What did you expect to see?
Clean program exit.
What did you see instead?
D:\Temp\test>go build xtest.go   && .\xtest.exe
Exception 0xc0000005 0x0 0x1dbb16305c0 0xefea89
PC=0xefea89
runtime: unknown pc 0xefea89
stack: frame={sp:0x2ac3fff670, fp:0x0} stack=[0x0,0x2ac3fffec0)
0000002ac3fff570:  0000000000ef1390  0000000000f04c2f
0000002ac3fff580:  000001db8a190000  0000000000ef69f1
0000002ac3fff590:  000001dbb0feb770  0000000000000000
0000002ac3fff5a0:  000001dbb0fea9b0  0000000000f1e260
0000002ac3fff5b0:  0000000000000001  0000000000ef0000
0000002ac3fff5c0:  000000000299ec00  000000000277f490
0000002ac3fff5d0:  000001db89fed280  00007ffc28be5d21
0000002ac3fff5e0:  000000000299f260  000001db89fc0000
0000002ac3fff5f0:  0000002ac3fff7c0  0000000000000000
0000002ac3fff600:  0000000000000000  000000000299ed28
0000002ac3fff610:  000000000299ed28  00007ffc26744c2f
0000002ac3fff620:  000001db89fed280  0000002ac3fff9d0
0000002ac3fff630:  000000000299f130  000000000299f130
0000002ac3fff640:  000000000299ef88  00000000b1c1ef01
0000002ac3fff650:  000000000299ed28  0000000000f0b843
0000002ac3fff660:  0000000000000000  0000000002a509c0
0000002ac3fff670: <0000002ac3fff9d0  0000000002a4e3f0
0000002ac3fff680:  000000000299ed28  0000000000f0b7fc
0000002ac3fff690:  000001db89fed280  0000000000f38f78
0000002ac3fff6a0:  ffffffffffffffff  0000002ac3fff9d0
0000002ac3fff6b0:  000000000299ed28  0000000000f0b3c2
0000002ac3fff6c0:  000001dbb16305c0  0000000000f00b26
0000002ac3fff6d0:  0000000002784858  000000000000001a
0000002ac3fff6e0:  0000000000000000  0000000000000000
0000002ac3fff6f0:  000000010eedfade  0000000000000000
0000002ac3fff700:  00007ffc284b847e  0000000000000007
0000002ac3fff710:  00007ffc284b847e  000001dbb16305c0
0000002ac3fff720:  0000000000000000  0000000000000000
0000002ac3fff730:  0000000000000000  0000000000000000
0000002ac3fff740:  0000000000000000  0000002ac3fff9d0
0000002ac3fff750:  0000000002784040  0000000000f0b6db
0000002ac3fff760:  0000000000000000  000001dbb0dc0000
runtime: unknown pc 0xefea89
stack: frame={sp:0x2ac3fff670, fp:0x0} stack=[0x0,0x2ac3fffec0)
0000002ac3fff570:  0000000000ef1390  0000000000f04c2f
0000002ac3fff580:  000001db8a190000  0000000000ef69f1
0000002ac3fff590:  000001dbb0feb770  0000000000000000
0000002ac3fff5a0:  000001dbb0fea9b0  0000000000f1e260
0000002ac3fff5b0:  0000000000000001  0000000000ef0000
0000002ac3fff5c0:  000000000299ec00  000000000277f490
0000002ac3fff5d0:  000001db89fed280  00007ffc28be5d21
0000002ac3fff5e0:  000000000299f260  000001db89fc0000
0000002ac3fff5f0:  0000002ac3fff7c0  0000000000000000
0000002ac3fff600:  0000000000000000  000000000299ed28
0000002ac3fff610:  000000000299ed28  00007ffc26744c2f
0000002ac3fff620:  000001db89fed280  0000002ac3fff9d0
0000002ac3fff630:  000000000299f130  000000000299f130
0000002ac3fff640:  000000000299ef88  00000000b1c1ef01
0000002ac3fff650:  000000000299ed28  0000000000f0b843
0000002ac3fff660:  0000000000000000  0000000002a509c0
0000002ac3fff670: <0000002ac3fff9d0  0000000002a4e3f0
0000002ac3fff680:  000000000299ed28  0000000000f0b7fc
0000002ac3fff690:  000001db89fed280  0000000000f38f78
0000002ac3fff6a0:  ffffffffffffffff  0000002ac3fff9d0
0000002ac3fff6b0:  000000000299ed28  0000000000f0b3c2
0000002ac3fff6c0:  000001dbb16305c0  0000000000f00b26
0000002ac3fff6d0:  0000000002784858  000000000000001a
0000002ac3fff6e0:  0000000000000000  0000000000000000
0000002ac3fff6f0:  000000010eedfade  0000000000000000
0000002ac3fff700:  00007ffc284b847e  0000000000000007
0000002ac3fff710:  00007ffc284b847e  000001dbb16305c0
0000002ac3fff720:  0000000000000000  0000000000000000
0000002ac3fff730:  0000000000000000  0000000000000000
0000002ac3fff740:  0000000000000000  0000002ac3fff9d0
0000002ac3fff750:  0000000002784040  0000000000f0b6db
0000002ac3fff760:  0000000000000000  000001dbb0dc0000
goroutine 1 [running]:
runtime.asmcgocall(0x7f6060, 0x91fbf8)
c:/go/src/runtime/asm_amd64.s:641 +0x47
runtime.stdcall(0x7ffc287ee0a0, 0x84ccd8)
c:/go/src/runtime/os_windows.go:916 +0x93
runtime.stdcall1(0x7ffc287ee0a0, 0x0, 0xc0000180b8)
c:/go/src/runtime/os_windows.go:936 +0x47
runtime.exit(0xc000000000)
c:/go/src/runtime/os_windows.go:579 +0x49
runtime.main()
c:/go/src/runtime/proc.go:226 +0x25d
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:1374 +0x1
rax     0x0
rbx     0x2ac3fff6f0
rcx     0x1dbb16305c0
rdi     0x299ed28
rsi     0x7ffc284b847e
rbp     0x2ac3fff6d0
rsp     0x2ac3fff670
r8      0x1db8a191380
r9      0x1
r10     0x8000
r11     0x2ac3fff5b0
r12     0x277f490
r13     0x299ec00
r14     0xef0000
r15     0x1
rip     0xefea89
rflags  0x10206
cs      0x33
fs      0x53
gs      0x2b

You might try go build -buildmode=exe; the default changed recently.
cc @alexbrainman @mattn
@gopherbot add OS-Windows

go build -buildmode=exe works. Thank you for suggestion.
Closing the issue as my problem is solved.
Mentioning for tracking purposes:
origin https://github.com/golang/go/issues/35192
similar https://github.com/golang/go/issues/41329

authors: altermark networkimprov