text:
undefined reference to `_cgo_sys_thread_start'

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?
go version go1.6 linux/mips64le

What operating system and processor architecture are you using (go env)?
GOARCH="mips64le"
GOBIN=""
GOEXE=""
GOHOSTARCH="mips64le"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/lib/golang"
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_mips64le"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -fmessage-length=0"
CXX="g++"
CGO_ENABLED="0"

What did you do?
download http://dl.fedoraproject.org/pub/fedora/linux/updates/23/SRPMS/r/runc-0.0.8-0.1.git1a124e9.fc23.src.rpm and build it. The build process is exit by ld returned 1 exit status.
The full error msg is as follow:
gcc -I . -fPIC -pthread -fmessage-length=0 -o $WORK/runtime/cgo/_obj/cgo.o $WORK/runtime/cgo/_obj/_cgo_main.o $WORK/runtime/cgo/_obj/_cgo_export.o $WORK/runtime/cgo/_obj/cgo.cgo2.o $WORK/runtime/cgo/_obj/gcc_fatalf.o $WORK/runtime/cgo/_obj/gcc_libinit.o $WORK/runtime/cgo/_obj/gcc_setenv.o $WORK/runtime/cgo/_obj/gcc_util.o -g -O2 -lpthread

runtime/cgo
/tmp/go-build507880994/runtime/cgo/_obj/gcc_util.o: In function x_cgo_thread_start': /usr/lib/golang/src/runtime/cgo/gcc_util.c:42: undefined reference to _cgo_sys_thread_start'
/usr/lib/golang/src/runtime/cgo/gcc_util.c:48: undefined reference to `_cgo_sys_thread_start'
collect2: error: ld returned 1 exit status

What did you expect to see?
build pass and no such error report.

What did you see instead?

I didn't think mips had cgo support at all, sort of surprised it got that far...

I found some one got support for cgo externel link @ https://github.com/golang/go/issues/14126

I think this can be merged into #14126 (cgo with extlink on mips64x)

authors: minux mwhudson vanloswang


text:
Now and Parse set different location UTC objects in the time structure when the system local is UTC

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go version go1.7.3 darwin/amd64
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/george/golang"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7.3/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7.3/libexec/pkg/tool/darwin_amd64"
CC="/usr/local/bin/gcc-5"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/var/folders/f1/17sq245d7639797h5dmqrb4w0000gn/T/go-build700069154=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="/usr/local/bin/c++-5"
CGO_ENABLED="1"
What did you do?
https://play.golang.org/p/FPuDXFSiRz
What did you expect to see?
correct
What did you see instead?
incorrect
Note that this is happening only on systems that have their local set to UTC - everywhere else in the world, this code will print 'correct'.

Go makes no promises about time.Time and ==. See 023556c07166be15a914d1f828d8c7b20a59a804 and 2c5bce1cfa242f27ffece3d30e8b851e3e923be2
But this was also changed for Go 1.8 in 5fbf35dc3fadd29785739fcec061d42157ea7861

It used to, and it still does in 1.7.3 which is the latest release. That is a significant change because not only time but every struct that has directly or indirectly time in it is not comparable anymore.
This is a lot of code that I have to review now - what about of the reset of the world. Thank you for fixing the actual issue, as well.

023556c and 2c5bce1 merely clarified the (lack of) promise that was already implicit and oft-mentioned on the mailing lists and other forums. It's just now explicitly recommended against.

I am not aware of any other language that the standard time type is not comparable. In your place, I would try harder to make/keep it as such.
Anyways, at least you should probably advertise this in a more aggressive way - say, add this to go vet or lint.

source: gharchive/issue

authors: bradfitz ggeorgiev

repo: golang/go

license_type: permissive

license_source: github-api


text:
access: may-start-trybots

I have a small history of patches here, and I would like to continue doing that.
I would like to gain access to may-start-trybots with Gerrit.  For example in changes like https://go-review.googlesource.com/c/go/+/101256
Thank you!

CC @andybons @bradfitz

Done!

authors: andybons ysmolsky


text:
cmd/api: failing on linux-arm builder due to non-devel Go version

https://build.golang.org/log/89d71a5b0369a691899c3435abc7df3d4abd4074
##### API check
Error running API checker: exit status 1
Go version is "builder host-linux-armel-cross", ignoring -next /home/builder/stage0scratch/workdir/go/api/next.txt
[ a lot of +pkg ... lines ]
exit status 1
2018/04/07 13:27:59 Failed: exit status 1
2018/04/07 13:28:04 FAILED

The fact that the version does not contain devel implies that +pkg lines, i.e. new APIs, are a test failure:
fail = !compareAPI(bw, features, required, optional, exception,
        *allowNew && strings.Contains(runtime.Version(), "devel"))

The fact that no other builder hits this tells me that the version in this ARM builder is wrong; it should contain devel.

/cc @zeebo @bradfitz

@bradfitz the builder is still seeing the same failures. Do we require manual action to apply your dashboard fix above?

I haven't deployed it

Ah, apologies for the noise - I thought this was somehow automated.

Ping @rsc. What do you think the VERSION file should contain in these cases?

@bradfitz The reason for the change is, essentially, to avoid having the string "devel" appears in the third field of the output of  go tool compile -V=full.  When the string "devel" appears there, the code in cmd/go/internal/work/buildid.go, function (*Builder).toolID, uses a hash of the compiler binary for the build ID.  When "devel" does not appear, (*Builder).toolID just hashes the version string itself.  The effect is that for released compilers we get the same build ID regardless of whether we are using a cross-compiler or not.
The output of go tool compile -V=full is generated by cmd/internal/objabi/flag.go based on the value of version in cmd/internal/objabi/zbootstrap.go.  That file is generated by mkzbootstrap in cmd/dist/buildruntime.go, where version is set to the result of findgoversion, which is the function you are asking about.
So this all boils down to: findgoversion can return anything that doesn't start with "devel".  That will ensure that the build ID of the packages built on the cross-compiler host will be the same as the build ID of the packages on the target, and will thus avoid rebuilding lots of packages on the target.
Other than not starting with "devel" I don't think it really matters what we put in the VERSION file on the builders.  It could be more or less anything, since the files don't escape the builders anyhow as far as I know.

@ianlancetaylor, one medium-term goal for the builders is to use cmd/go's build+test cache infrastructure across builds, at least for trybots. If we put the git commit hash in the VERSION, doesn't that get mixed in to the action ID / cache keys? Can I just use a static string for all builds, like "golang-x-build"?

I don't understand how we can safely use the cache across builds across CLs.  If a CL changes the compiler, then we definitely do not want to pick up an cached build result built by an earlier version of the compiler.

@ianlancetaylor, I assumed the digest of the compiler/linker/assembler bits were part of the cache. I just don't want some arbitrary string to be part of the digest unnecessarily.
If a given commit only touches a comment in some test file, I want to have an identical compiler/linker/assembler from the previous commit, and then reuse all the built packages & test results.

If go tool compile -V=full prints a string that contains the string "devel", then the digest of the compiler will be part of the cache.  If it does not, then only the -V=full output will be part of the cache.  If there is a VERSION file, then the string "devel" will be printed if and only if the VERSION file contains the string "devel".  So we could perhaps use "devel golang-x-build".
At least, that is my understanding.

authors: bradfitz ianlancetaylor mvdan


text:
[RFC] go2 error handling best reference

I read the https://github.com/golang/proposal/blob/master/design/go2draft-error-handling.md
The new go2 error handling design paper.
Sorry for create this issue, ( i can not find any place to submit my thought and worry about the doc)
This issue is to talk my thought about the go2 error handler.
The go2draft error handling design a ugly error handle proccess way.
I can not agree with the design.
And I would like to suggest some good error handling

C# https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/exceptions/
F# https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/exception-handling/

these two exception handling design , provided us a clean interface, and easy to deal , easy to track problems.
the Go2Draf handler Check seems too complicated and not well structed
And here some other article talk about exception handler, very well document .
https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/exceptions
https://docs.microsoft.com/en-us/dotnet/standard/exceptions/best-practices-for-exceptions
Exception handling has many advantages over return-value-based error reporting.
Good framework design helps the application developer realize the benefits of exceptions.
This section discusses the benefits of exceptions and presents guidelines for using them effectively.

A well-designed app handles exceptions and errors to prevent app crashes.

Discussion about the error handling design is welcome.  But it would help quite a bit if you could be more specific.  What exactly is ugly?  What exactly do you not agree with?
Referring to exception handling is a non-starter.  We are not going to implement C# style exception handling in Go, for the reasons explained at https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling-overview.md .  While alternative suggestions are welcomed, a general pointer to other languages that use exceptions is not going to be adopted.  Sorry.

The handler and check not match , you have to write a lot check , check check, and define lot's of handler.
This stupid idea.
for Exception handler,
you can
Try {

} catch (NullPointException e) {
  //TODO: handler Null Point errors
 } catch (OverflowException e) {
  //TODO handler Overflow errors
}

Very clean and tidy.
did you see it ?

Your check and handler was nosense. Check what ? handler what?
nothing to know, no problem resolved. you just make things worse and worse

( i can not find any place to submit my thought and worry about the doc)

This is the place - https://github.com/golang/go/wiki/Go2ErrorHandlingFeedback.
Please feel free to outline all your thoughts in a post,gist,google doc and add a link in the wiki. It looks like you have more of a feedback, rather than a concrete proposal. So I will close the issue and request you to use the wiki page.
Thank you.

authors: agnivade ianlancetaylor netroby


text:
x/tools/go/packages: can't distinguish generated sources from user-provided ones

When using go list -test, we can distinguish user-provided sources from generated ones based on whether the paths are absolute, as documented here (emphasis mine):

By default, the lists GoFiles, CgoFiles, and so on hold names of files in Dir (that is, paths relative to Dir, not absolute paths). The generated files added when using the -compiled and -test flags are absolute paths referring to cached copies of generated Go source files. Although they are Go source files, the paths may not end in ".go".

Unfortunately, golang.org/x/tools/go/packages removes that distinction by making all of the GoFiles it returns absolute, without apparently preserving enough information to undo that transformation. That makes it very difficult to use go/packages to write tools that transform or otherwise refactor user-provided source code, such as the one in CL 137076.
CC @matloob @alandonovan @ianthehat @rsc

The assumption "absolute <=> generated" doesn't hold for other build systems such as Bazel, in which a package may contain a mixture of sources from different directories. Indeed, it doesn't even hold for go list in the presence of modules, since source files may reside in the module cache.
The go/packages API could in principle record which files are generated (and at one point it did), but we removed on the grounds that refactoring tools should decide which files to edit based on the "DO NOT EDIT" comment convention proposed in github.com/golang/go/issues/13560.

The DO NOT EDIT convention relies on three assumptions:

Generated code is not checked in to the repository.
Users never want refactoring tools to apply even trivial edits to files that say DO NOT EDIT.
All generated files actually follow the convention.

I don't know about (3), but I do not believe that (1) or (2) holds outside of Blaze/Bazel.

In particular, you could imagine a workflow that goes:

Refactor the code using some tool, including arbitrary package moves.
Re-run the generators for all generated files that were touched.

Step 1 should not require additional intervention to prune or exclude generated files, since they may be needed to type-check packages at any intermediate state.

authors: alandonovan bcmills


text:
runtime: performance degradation in high-load long running applications

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/spin/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/.../go"
GOPROXY=""
GORACE=""
GOROOT="/.../1.12.5/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/q5/b4nzm8zj3g5gcjs4rhkpvczh0000gn/T/go-build533400238=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
My company spotted serious performance degradation in one of high load services. Service handles about 100K TCP permanent connections and using complex processing for about 75K partners requests per second. When we tried to upgrade Go from 1.11.6 to 1.12.5 we noted that performance fell by a third. Service begins seriously misbehave after about 15-30 mins after start.
After investigation I found that performance degradation caused by commits 1678b2c5..873bd47d. Commits 1678b2c5..457c8f4f causes slight performance degradation. Performance of service builded with 873bd47d and later is absolutely horrible as well as memory consumption. Also I noted that CPU load from 873bd47d reduced from 100% to 67%. Last commit with acceptable performance is 6e9fb11b3.
Very strange benchmark:
func BenchmarkWrite(b *testing.B) {
	f, err := os.Create("/dev/null")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	buf := make([]byte, 1024*100)
	b.ResetTimer()
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			for i:=0;i<100000;i++ {
				f.SetDeadline(time.Now().Add(5*time.Minute))
				f.Write(buf)
				f.SetDeadline(time.Time{})
				runtime.GC()
			}
		}
	})
	b.StopTimer()
}

Maybe not of all ops makes sense but the results speak for themselves. Benchmarks are executed with Go built from source 6e9fb11b..873bd47d. Here is brief results. You can find details in attached archives.
6e9fb11b (OK):
$ CGO_ENABLED=0 $HOME/go.dev/bin/go test -run=XXX -bench=BenchmarkWrite ./...
goos: linux
goarch: amd64
pkg: ...
BenchmarkWrite-2   	       1	17839662490 ns/op	   93816 B/op	      61 allocs/op
PASS

873bd47d (degraded):
...
goos: linux
goarch: amd64
pkg: ...
BenchmarkWrite-2   	       1	41872237656 ns/op	 6708104 B/op	   68744 allocs/op
PASS

Detailed results:
0-6e9fb11b.zip
1-1678b2c5.zip
2-143b13ae.zip
3-457c8f4f.zip
4-873bd47.zip
Referred commits:

https://github.com/golang/go/commit/1678b2c580584fa1ea8de3c14df0d8d77b6f7387
https://github.com/golang/go/commit/143b13ae82d81020dfa6db40818bef5a1f701c3f
https://github.com/golang/go/commit/457c8f4fe9e4d45f97d0a3f3c4a80789c6616fd6
https://github.com/golang/go/commit/873bd47dfb34ba4416d4df30180905250b91f137

What did you expect to see?
Normal performance like in Go 1.11.6.
What did you see instead?
Big performance degradation with service built with Go 1.12+

CC @aclements @mknyszek

What did you do?
My company spotted serious performance degradation in one of high load services. Service handles about 100K TCP permanent connections and using complex processing for about 75K partners requests per second. When we tried to upgrade Go from 1.11.6 to 1.12.5 we noted that performance fell by a third. Service begins seriously misbehave after about 15-30 mins after start.
After investigation I found that performance degradation caused by commits 1678b2c5..873bd47d. Commits 1678b2c5..457c8f4f causes slight performance degradation. Performance of service builded with 873bd47d and later is absolutely horrible as well as memory consumption. Also I noted that CPU load from 873bd47d reduced from 100% to 67%. Last commit with acceptable performance is 6e9fb11b3.

Just to be totally clear, this investigation was done using your service and not the benchmark below, correct?
Can you comment more on memory consumption? Do you mean RSS?
Also, since it's related to the changes that appear to cause the regression, do you run your service with GODEBUG=gcstoptheworld=1 or GODEBUG=gcstoptheworld=2?

Very strange benchmark:
func BenchmarkWrite(b *testing.B) {
	f, err := os.Create("/dev/null")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	buf := make([]byte, 1024*100)
	b.ResetTimer()
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			for i:=0;i<100000;i++ {
				f.SetDeadline(time.Now().Add(5*time.Minute))
				f.Write(buf)
				f.SetDeadline(time.Time{})
				runtime.GC()
			}
		}
	})
	b.StopTimer()
}

Maybe not of all ops makes sense but the results speak for themselves. Benchmarks are executed with Go built from source 6e9fb11b..873bd47d. Here is brief results. You can find details in attached archives.

That is a strange benchmark indeed. The many calls to runtime.GC() are going to cause this benchmark to run really slowly in the first place. Does this benchmark relate to your service in any way?

@mknyszek This strange benchmark is just a quick and dirty try to simulate service load. Of course there are no runtime.GC() calls in service. This benchmark used with Git bisect to find suspicious commit. After this service was built with corresponding Go commits and deployed. Results of deployment with 873bd47d confirmed that "new" GC is absolutely ineffective under high load.
gcstoptheworld is not used. There are no notes about "new" GC in documentation. Also docs about gcstoptheworld is not changed. In our company we never used gcstoptheworld with any Go version.
I want to clarify again: right now service built with Go 1.11 works absolutely well without any GODEBUG options.

Thanks for the information, and belated thanks for all the detail and the reproducer. My apologies for how long it's taken to get back to you.
I just want to be absolutely clear that 873bd47d and the commits before it are in fact the source of the problem for your service. Did you try the commit before 1678b2c5 (6e9fb11b)  against your service? If you're confident your service is also running just fine with 6e9fb11b, then I'll dig deeper into what could be going wrong with the benchmark.

@mknyszek Thanks for response. Yes. As I wrote earlier I made test deployments with commits from 6e9fb11b to 873bd47d.  I can confirm that 6e9fb11b works well.

Service built with 6e9fb11b works absolutely same as built with Go 1.11.6 (our current production version).
Three commits before 873bd47d gives slight performance degradation.
Service built with 873bd47d gives 30-40% performance degradation.

@mknyszek Just notes which may be useful during investigation (Go 1.11.6):
In service we massively using sync.Pool, sync.Map, large maps e. t. c.
Average stats:

Memory consumption: alloc: 55GB, sys: 90GB.
Heap objects: 450M
Heap memory: idle: 32GB, inuse: 52GB, released: 16GB, sys: 85GB
Memory: mspan: 690MB, stack: 890MB
GC pauses: 10ms
GD per minute: 2
GC CPU fraction: 2.5%
Next GC: 65GB
Heap alloc: 60GB
Goroutines: 100K

Hope this helps.

@akaspin I am doing almost same what you have described above and am facing the same issue. we are doing around the same we are having almost 600 TCP/IP connection connected with another 600 TCP/IP so roughly around we are using 600x600 TCP/IP connection but right now we are having a temporary connection so every-time it will be creating and closing the connection. But it is consuming a huge amount of memory in short period but the Garbage collector is not freeing up in the same time so what do you suggest to have a permanent connection as you have done in you project and di you get the solution for your issue if you have solved it can you please guide us how can we solve this issue. we are using GRPC with protobuf.

@jayeshwebcluesglobal We just fixed Go version on 1.11.10.

@akaspin Thanks! it seems we are running on older version(1.9.3), will check issue by upgrading to 1.11.10

@akaspin Want to take you view on one thing,
We are making 600*600 TCP connections, should we make it permanent or should close and reopen when we want to make a request to each other ?

authors: akaspin ianlancetaylor jayeshwebcluesglobal mknyszek


text:
net/http: http.Client.Do() sometimes throws INTERNAL_ERROR when using high number of parallel HTTP/2 requests (regression between 1.12 and 1.13.4)

What version of Go are you using (go version)?
go version go1.13.4 darwin/amd64

(also reproduced with 1.13.6)
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13.4/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/k7/jx2bs5q170z4lvdzbsdt426r0000gn/T/go-build634112063=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
Create several http.Clients, and call a high number of simultaneous multipart byterange requests on each of them. Make sure the server supports HTTP/2.
What did you expect to see?
No errors being thrown. Which is the case on Go 1.12
What did you see instead?
On Go 1.13.4 and 1.13.6, http.Do() sometimes returns an error of the form stream error: stream ID 523; INTERNAL_ERROR. It does seem to happen more often when there is some network congestion, but I've also observed it on very good networks. Typically, I'd say it happens once every 200 requests or so.

/cc @bradfitz

cc @fraenkel

We are going to need a test case or at the very least some http/2 debug to determine why the INTERNAL_ERROR occurred.
Does this problem also occur with tip?

authors: cagedmantis fraenkel networkimprov rayvbr


text:
proposal: x/crypto/acme: allow setting client KID directly

What version of Go are you using (go version)?

$ go version
go version go1.13.4 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
N/A
What did you do?
I have a service that uses acme.Client for communication with an RFC8555 CA. For technical reasons I need to reconstruct the acme.Client multiple times during the full flow.
Current implementation works overall, but I'd like to be able to specify account KID (equal to Account.URI) explicitly, instead of having the library always call the CA's account registration endpoint to obtain kid to cache: https://github.com/golang/crypto/blob/c07d793c2f9aacf728fe68cbd7acd73adbd04159/acme/acme.go#L152
What did you expect to see?
I'd like to be able to create the client like this:
&acme.Client{
    Key:          signer,
    KID:          accountURI,
    DirectoryURL: directoryUrl,
    HTTPClient:   &http.Client{...}
},

And have the kid field populated from it , e.g.:

KID field could be a string, only used in the first request to pre-populate the private mutex-protected kid value.
The following snippet could be added to accountKID function between lines 151 and 152

if c.KID != "" {
  c.kid = c.KID
  return c.kid
}

What did you see instead?
I have to specify the client like this:
&acme.Client{
    Key:          signer,
    DirectoryURL: directoryUrl,
    HTTPClient: &http.Client{...}
},

And then the library calls the CA's account registration endpoint to obtain kid to cache. This  results in a lot of unnecessary calls to the CA.

(sent a CL for this, but just to articulate my thoughts here) This seems reasonable to me. I don't think adding a duplicate public field is worth the extra complexity, seems fine to just expose the existing kid field.

authors: Stanwise rolandshoemaker


text:
reflect: method ABI and value ABI do not align

What version of Go are you using (go version)?

$ go version
go version go1.17beta1 windows/amd64

Does this issue reproduce with the latest release?
No
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\reus\AppData\Local\go-build
set GOENV=C:\Users\reus\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\reus\go\pkg\mod
set GONOPROXY=github.com/reusee/*
set GONOSUMDB=github.com/reusee/*
set GOOS=windows
set GOPATH=C:\Users\reus\go
set GOPRIVATE=github.com/reusee/*
set GOPROXY=https://goproxy.cn,direct
set GOROOT=C:\Users\reus\sdk\go1.17beta1
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\reus\sdk\go1.17beta1\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.17beta1
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\reus\AppData\Local\Temp\go-build1901699100=/tmp/go-build -gno-record-gcc-switches
GOROOT/bin/go version: go version go1.17beta1 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.17beta1

What did you do?
run

package main

import (
        "reflect"
)

type Foo struct{}

func (_ Foo) Foo(
        _ Bar,
        _ Bar,
        _ Baz,
        _ Baz,
        _ Baz,
        _ Bar,
        _ Qux,
) {
}

type Bar interface{}

type Baz func()

type Qux int

func main() {
        f := reflect.ValueOf(Foo{}).MethodByName("Foo").Interface()
        reflect.ValueOf(f).Call([]reflect.Value{
                reflect.ValueOf(Bar(42)),
                reflect.ValueOf(Bar(42)),
                reflect.ValueOf(Baz(nil)),
                reflect.ValueOf(Baz(nil)),
                reflect.ValueOf(Baz(nil)),
                reflect.ValueOf(Bar(42)),
                reflect.ValueOf(Qux(0)),
        })
}

What did you expect to see?
No panic
What did you see instead?
panic: method ABI and value ABI do not align                                                                                goroutine 1 [running]:
reflect.callMethod(0xc000060040, 0xc00007d640, 0xc00007d518, 0xc00007d520)
        C:/Users/reus/sdk/go1.17beta1/src/reflect/value.go:969 +0x894
reflect.Value.call({0x6b18e0, 0xc000060040, 0x48}, {0x6bd1ee, 0x4}, {0xc00007dec8, 0x7, 0x6b18e0})
        C:/Users/reus/sdk/go1.17beta1/src/reflect/value.go:543 +0x814
reflect.Value.Call({0x6b18e0, 0xc000060040, 0x60}, {0xc00007dec8, 0x7, 0x7})
        C:/Users/reus/sdk/go1.17beta1/src/reflect/value.go:339 +0xc5
main.main()
        C:/Users/reus/a.go:30 +0x885
exit status 2

This panics in 1.17 but runs without error in earlier releases, so marking as a release blocker.
CC @mknyszek

Thanks for the small reproducer. Looking into it.

Slightly smaller reproducer:
package main

import (
	"fmt"
	"reflect"
)

type Foo struct{}

func (_ Foo) Foo(
	_ Bar,
	_ Bar,
	_ Baz,
	_ Baz,
	_ Baz,
	_ Bar,
	z Qux,
) int {
	return int(z)
}

type Bar interface{}

type Baz func()

type Qux int

func main() {
	f := reflect.ValueOf(Foo{}).MethodByName("Foo").Interface().(func(Bar, Bar, Baz, Baz, Baz, Bar, Qux) int)
	fmt.Println(f(Bar(42), Bar(42), Baz(nil), Baz(nil), Baz(nil), Bar(42), Qux(0)))
}

The failure is in the method value call, not the latter reflect call.

I see the problem. In the "value" ABI the Qux-typed argument is stack-allocated, but in the "method" ABI, the argument before it (which is 16 bytes together) ends up on the stack, leaving a register free for the Qux-typed argument. callMethod erroneously assumes that this case can't happen.
Fix incoming.

authors: ianlancetaylor mknyszek reusee


text:
Support for major internet protocols such as IMA, POP3 etc.

Go doesn't support for major internet protocols such as IMAP, POP3 etc.
Why don't we add support for this in standard "net" package?.

See https://go.dev/doc/faq#x_in_std
For questions please refer to https://github.com/golang/go/wiki/Questions

authors: SakthiMahendran seankhliao


text:
cmd/compile: the output messages for some generic functions are confusing when using -gcflags="-m -m"

What version of Go are you using (go version)?

$ go version
go version go1.20rc3 linux/amd64

Does this issue reproduce with the latest release?
Yes
What did you do?

package main

type Signed interface {
	~int | ~int8 | ~int16 | ~int32 | ~int64
}

type Unsigned interface {
	~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr
}

type Integer interface {
	Signed | Unsigned
}

type Float interface {
	~float32 | ~float64
}

type Ordered interface {
	Integer | Float | ~string
}

func Min[T Ordered](x, y T) T {
	if x <= y {
		return x
	}
	return y
}

type ByteSeq interface{ ~string | ~[]byte }

func CommonPrefixes[X, Y ByteSeq](x X, y Y) (X, Y) {
	min := Min(len(x), len(y))
	x2, y2 := x[:min], y[:min]
	if len(x2) != len(y2) { // BCE hint
		panic("len(x2) != len(y2)")
	}
	for i := 0; i < len(x2); i++ {
		if x2[i] != y2[i] {
			return x2[:i], y2[:i]
		}
	}
	return x2, y2
}

var (
	_, _ = CommonPrefixes("", []byte{})
	_, _ = CommonPrefixes("", "")
	_, _ = CommonPrefixes([]byte{}, "")
	_, _ = CommonPrefixes([]byte{}, []byte{})
)

func Compare[X, Y ByteSeq](x X, y Y) int {
	min := Min(len(x), len(y))
	x2, y2 := x[:min], y[:min]
	if len(x2) != len(y2) {
		panic("len(x2) != len(y2)")
	}
	for i := 0; i < len(x2); i++ {
		if v := int(x2[i]) - int(y2[i]); v != 0 {
			return v
		}
	}

	if len(x) == len(y) {
		return 0
	}
	if len(x) < len(y) {
		return -1
	}
	return 1
}

var (
	_ = Compare("", []byte{})
	_ = Compare("", "")
	_ = Compare([]byte{}, "")
	_ = Compare([]byte{}, []byte{})
)

func main() {}

go run -gcflags="-m -m"  main.go 2>&1 | grep inline

What did you expect to see?
Consistent messages for the two generic functions, and cleaner messages.
What did you see instead?
./main.go:82:6: can inline main with cost 0 as: func() {  }
./main.go:23:6: can inline Min[go.shape.int] with cost 8 as: func(*[1]uintptr, go.shape.int, go.shape.int) go.shape.int { if x <= y { return x }; return y }
./main.go:54:6: cannot inline Compare[go.shape.[]uint8,go.shape.[]uint8]: function too complex: cost 92 exceeds budget 80
./main.go:23:6: can inline Min[int] with cost 15 as: func(int, int) int { return Min[go.shape.int](&.dict.Min[int], x, y) }
./main.go:54:6: can inline Compare[[]uint8,[]uint8] with cost 64 as: func([]byte, []byte) int { return Compare[go.shape.[]uint8,go.shape.[]uint8](&.dict.Compare[[]uint8,[]uint8], x, y) }
./main.go:54:6: cannot inline Compare[go.shape.[]uint8,go.shape.string]: function too complex: cost 92 exceeds budget 80
./main.go:54:6: can inline Compare[[]uint8,string] with cost 64 as: func([]byte, string) int { return Compare[go.shape.[]uint8,go.shape.string](&.dict.Compare[[]uint8,string], x, y) }
./main.go:54:6: cannot inline Compare[go.shape.string,go.shape.string]: function too complex: cost 92 exceeds budget 80
./main.go:54:6: can inline Compare[string,string] with cost 64 as: func(string, string) int { return Compare[go.shape.string,go.shape.string](&.dict.Compare[string,string], x, y) }
./main.go:54:6: cannot inline Compare[go.shape.string,go.shape.[]uint8]: function too complex: cost 92 exceeds budget 80
./main.go:54:6: can inline Compare[string,[]uint8] with cost 64 as: func(string, []byte) int { return Compare[go.shape.string,go.shape.[]uint8](&.dict.Compare[string,[]uint8], x, y) }
./main.go:33:6: can inline CommonPrefixes[go.shape.[]uint8,go.shape.[]uint8] with cost 73 as: func(*[2]uintptr, go.shape.[]uint8, go.shape.[]uint8) (go.shape.[]uint8, go.shape.[]uint8) { min := Min[go.shape.int](&.dict.Min[int], len(x), len(y)); x2, y2 := x[:min], y[:min]; if len(x2) != len(y2) { panic("len(x2) != len(y2)") }; for loop; return x2, y2 }
./main.go:33:6: cannot inline CommonPrefixes[[]uint8,[]uint8]: function too complex: cost 89 exceeds budget 80
./main.go:33:6: can inline CommonPrefixes[go.shape.[]uint8,go.shape.string] with cost 73 as: func(*[2]uintptr, go.shape.[]uint8, go.shape.string) (go.shape.[]uint8, go.shape.string) { min := Min[go.shape.int](&.dict.Min[int], len(x), len(y)); x2, y2 := x[:min], y[:min]; if len(x2) != len(y2) { panic("len(x2) != len(y2)") }; for loop; return x2, y2 }
./main.go:33:6: cannot inline CommonPrefixes[[]uint8,string]: function too complex: cost 89 exceeds budget 80
./main.go:33:6: can inline CommonPrefixes[go.shape.string,go.shape.string] with cost 73 as: func(*[2]uintptr, go.shape.string, go.shape.string) (go.shape.string, go.shape.string) { min := Min[go.shape.int](&.dict.Min[int], len(x), len(y)); x2, y2 := x[:min], y[:min]; if len(x2) != len(y2) { panic("len(x2) != len(y2)") }; for loop; return x2, y2 }
./main.go:33:6: cannot inline CommonPrefixes[string,string]: function too complex: cost 89 exceeds budget 80
./main.go:33:6: can inline CommonPrefixes[go.shape.string,go.shape.[]uint8] with cost 73 as: func(*[2]uintptr, go.shape.string, go.shape.[]uint8) (go.shape.string, go.shape.[]uint8) { min := Min[go.shape.int](&.dict.Min[int], len(x), len(y)); x2, y2 := x[:min], y[:min]; if len(x2) != len(y2) { panic("len(x2) != len(y2)") }; for loop; return x2, y2 }
./main.go:33:6: cannot inline CommonPrefixes[string,[]uint8]: function too complex: cost 89 exceeds budget 80

Here is the simplified version:
...
./main.go:54:6: can inline Compare[[]uint8,[]uint8] with cost 64 as: func([]byte, []byte) int { return Compare[go.shape.[]uint8,go.shape.[]uint8](&.dict.Compare[[]uint8,[]uint8], x, y) }
./main.go:54:6: cannot inline Compare[go.shape.[]uint8,go.shape.string]: function too complex: cost 92 exceeds budget 80
...
./main.go:33:6: cannot inline CommonPrefixes[[]uint8,[]uint8]: function too complex: cost 89 exceeds budget 80
./main.go:33:6: can inline CommonPrefixes[go.shape.[]uint8,go.shape.string] with cost 73 as: func(*[2]uintptr, go.shape.[]uint8,
...

It looks, for each of the generic functions, there are two groups of instances: go.shape.xxx group and xxx group.
Why two groups? It is some confusing.
And for the Compare function, the xxx group can be inlined, but the go.shape.xxx group can't. For the CommonPrefixes function, it is the inverse. This makes developers unsure whether or not a generic function can be inlined or not.

CC @golang/compiler

There's a "who's this for?" problem here, since if debugging a problem in the compiler, those ugly names are "the truth" and the requirement for pretty/ugly name translation is just another thing to get wrong in a situation where at least one thing is already wrong (else we would not be debugging).  This complaint is however very much applicable to the output of the -json flag, since that is intended for IDE/normal-programmer consumption.

I don't care much about the names. Now I just not clear whether or not the instances of the two generic functions can be inlined or not?

It looks, for each of the generic functions, there are two groups of instances: go.shape.xxx group and xxx group. Why two groups?

There are two groups because that's how cmd/compile implements generics. See https://go.googlesource.com/proposal/+/refs/heads/master/design/generics-implementation-gcshape.md for more details.
If you read the diagnostics that you posted, you'll notice that the functions are different:
./main.go:23:6: can inline Min[go.shape.int] with cost 8 as: func(*[1]uintptr, go.shape.int, go.shape.int) go.shape.int { if x <= y { return x }; return y }
./main.go:23:6: can inline Min[int] with cost 15 as: func(int, int) int { return Min[go.shape.int](&.dict.Min[int], x, y) }

Min[go.shape.int] is the intarnal GC-shaped instantiation, which takes an extra dictionary parameter (the *[1]uintptr-typed parameter).
Min[int] is a wrapper function with the correct user-type of just func(int, int) int, which calls Min[go.shape.int] with the appropriate dictionary argument for int.

Now I just not clear whether or not the instances of the two generic functions can be inlined or not?

I'm afraid I don't know what additional information you need to answer this question. We report for each instantiation whether it was inlinable.

Thanks for the explanation.
Now, I understand why the inline analysis statuses for the two functions are different.
func foo_wrapper(...) (...) {
   ...
   return foo(...)
}

If a foo call can be inlined but its inline cost is larger than the inline cost of a function call,
then it might make the inline cost of foo_wrapper too larger to be inline-able.
And on the other hand, if foo is not inline-able, then foo_wrapper might be.
Please feel free to close this issue if nothing needs to be done.


text:
proposal: unicode/utf8: improve Encode performance by requiring a fixed 4 bytes width

Proposal Details
Introduction
The function Encode from package unicode/utf8 requires from the provided byte slice only the exact amount of space needed to write the provided rune. This is great for space efficiency, but an alternative approach requiring a fixed 4 spare bytes, regardless if they will use all of them, enables greater runtime optimization. The following benchstat shows the results of such optimizations:
goos: linux
goarch: amd64
pkg: unicode/utf8
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
                               │  variable   │                fixed                │
                               │   sec/op    │   sec/op     vs base                │
Encode/ASCIIRune-8               1.086n ± 1%   1.075n ± 0%   -1.06% (p=0.000 n=20)
Encode/SpanishRune-8             1.523n ± 2%   1.119n ± 2%  -26.58% (p=0.000 n=20)
Encode/JapaneseRune-8            1.933n ± 2%   1.683n ± 1%  -12.93% (p=0.000 n=20)
Encode/MaxRune-8                 2.175n ± 2%   1.741n ± 0%  -19.97% (p=0.000 n=20)
Encode/InvalidRuneMaxPlusOne-8   1.959n ± 1%   1.302n ± 1%  -33.56% (p=0.000 n=20)
Encode/InvalidRuneSurrogate-8    1.938n ± 2%   1.286n ± 0%  -33.67% (p=0.000 n=20)
Encode/InvalidRuneNegative-8     1.954n ± 1%   1.315n ± 1%  -32.73% (p=0.000 n=20)
geomean                          1.757n        1.340n       -23.74%

The benchmarks currently only have an ASCII (a valid 1 byte rune) and a Japanese (a valid 3 bytes rune) cases. Here, the benchmarks were extended to test for every possible rune case that needs to be encoded:

InvalidRuneMaxPlusOne is an erroneous rune overflowing MaxRune, likely the product of rune arithmetic. This is written as RuneError, which is a 3 bytes rune
InvalidRuneSurrogate is an erroneous rune which is in the surrogate range, same treatment as above
InvalidRuneNegative is an erroneous rune that is a negative value, same treatment as above
SpanishRune is a valid 2 bytes rune
MaxRune is a valid 4 bytes rune

Note that the cases for erroneous runes are added for completeness, and improving any implementation for those cases need not be a priority for the standard library.
Proposal alternatives
Add a new function to implement the proposed behavior
This is obviously not ideal, and may even generate confusion in users. Having two functions do mostly the same thing but with slightly different requirements may prove counter productive. It also increases the API surface only to provide a performance improvement, which is something that is to be avoided as much as possible, even if that already happened in the past.
Change Encode to require a fixed length
The doc comment in Encode is:
// EncodeRune writes into p (which must be large enough) the UTF-8 encoding of the rune.
// If the rune is out of range, it writes the encoding of [RuneError].
// It returns the number of bytes written.

Here, "large enough" is never specified to be "the least amount needed to encode the rune", which is what the current implementation does, thus it's permissible to argument that 4 available bytes is also "large enough" because that's large enough to encode any UTF-8 rune. So users shouldn't rely on that, but some may already do. Users rely on internals of implementation details as far as calling runtime.morestack_noctxt, so it's all possible.
But why would someone aware of utf8.Encode not reserve utf8.UTFMax bytes in the first place? The only reasonable explanation the author can think of is micro managing the tail of a buffer to save, say, between 1 and 3 bytes. As a corollary, it should be noted that if this approach was taken, then all buffers written to exclusively with utf8.Encode should have a length multiple of 4 bytes. Potentially wasting 3 bytes in the worst case doesn't sound that terrible, though, if we compare to the benefit of the improved performance.
In the standard library, for example, all buffers are already (sanely) allocating for utf8.UTFMax bytes before calling utf8.Encode, and so should everyone be doing. If the text to be written is always ASCII, then there is no reason to call utf8.Encode in the first place, just call copy for that.
But varying implementations exist, so if the approach of changing the Encode function is taken, an extra cautious approach would be first rolling this change with a notice and only enabled with an environment variable, defaulting to use old behavior, in another release this environment variable could default to the new behavior, and finally the variable removed and this be the only possible behavior.
Appendix: example implementation
Changes to src/unicode/utf8/utf8.go
The name EncodeRune4 is definitely not great, probably something like EncodeRuneFixed could be used, if distinction is going to be made.
// EncodeRune4 writes into p (which must be have a minimum length of 4) the
// UTF-8 encoding of the rune. If the rune is out of range, it writes the
// encoding of [RuneError]. It returns the number of bytes written.
func EncodeRune4(p []byte, r rune) int {
       _ = p[3]
       return utf8Bytes((*[4]byte)(unsafe.Pointer(unsafe.SliceData(p))), r)
}

// utf8Bytes writes into p the UTF-8 encoding of the rune. If the rune is out of
// range, it writes the encoding of [RuneError]. It returns the number of bytes
// written.
func utf8Bytes(p *[4]byte, r rune) int {
       switch u := uint32(r); {
       case u <= rune1Max:
               p[0] = byte(r)
               return 1
       case u <= rune2Max:
               p[0] = t2 | byte(u>>6)
               p[1] = tx | byte(u)&maskx
               return 2
       case surrogateMin <= u && u <= surrogateMax: // RuneError: in surrogate range
               p[0] = 239
               p[1] = 191
               p[2] = 189
               return 3
       case u <= rune3Max:
               p[0] = t3 | byte(u>>12)
               p[1] = tx | byte(u>>6)&maskx
               p[2] = tx | byte(u)&maskx
               return 3
       case u <= MaxRune:
               p[0] = t4 | byte(u>>18)
               p[1] = tx | byte(u>>12)&maskx
               p[2] = tx | byte(u>>6)&maskx
               p[3] = tx | byte(u)&maskx
               return 4
       }
       // RuneError: overflowing MaxRune or negative. Delayed to favor the case of valid runes
       p[0] = 239
       p[1] = 191
       p[2] = 189
       return 3
}

Benchmarking

Setup:mkdir ~/go/src/go.googlesource.com
git clone https://go.googlesource.com/go ~/go/src/go.googlesource.com/go
cd ~/go/src/go.googlesource.com/go/src
./all.bash
cd ..
export GOROOT="$PWD"
export PATH="$PWD/bin:$PATH"

Benchmarks run with: go test -timeout=30m -run=- -count=20 -bench=BenchmarkEncode > results-encode.txt
Results processed with: benchstat -col /implem results-encode.txt

Changes made to src/unicode/utf8/utf8_test.go for the benchmarks

diff --git a/src/unicode/utf8/utf8_test.go b/src/unicode/utf8/utf8_test.go
index 2e6aace766..810166ea39 100644
--- a/src/unicode/utf8/utf8_test.go
+++ b/src/unicode/utf8/utf8_test.go
@@ -638,18 +638,108 @@ func init() {
        longStringJapanese = strings.Repeat(japanese, 100_000/len(japanese))
 }

-func BenchmarkEncodeASCIIRune(b *testing.B) {
-       buf := make([]byte, UTFMax)
-       for i := 0; i < b.N; i++ {
-               EncodeRune(buf, 'a')
-       }
-}
+func BenchmarkEncode(b *testing.B) {
+       b.Run("implem=variable", func(b *testing.B) {
+               b.Run("ASCIIRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, 'a') // 1 byte
+                       }
+               })

-func BenchmarkEncodeJapaneseRune(b *testing.B) {
-       buf := make([]byte, UTFMax)
-       for i := 0; i < b.N; i++ {
-               EncodeRune(buf, '本')
-       }
+               b.Run("SpanishRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, 'Ñ') // 2 bytes
+                       }
+               })
+
+               b.Run("JapaneseRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, '本') // 3 bytes
+                       }
+               })
+
+               b.Run("MaxRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, MaxRune) // 4 bytes
+                       }
+               })
+
+               b.Run("InvalidRuneMaxPlusOne", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, MaxRune+1) // 3 bytes: RuneError
+                       }
+               })
+
+               b.Run("InvalidRuneSurrogate", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, 0xD800) // 3 bytes: RuneError
+                       }
+               })
+
+               b.Run("InvalidRuneNegative", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune(buf, -1) // 3 bytes: RuneError
+                       }
+               })
+       })
+
+       b.Run("implem=fixed", func(b *testing.B) {
+               b.Run("ASCIIRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, 'a') // 1 byte
+                       }
+               })
+
+               b.Run("SpanishRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, 'Ñ') // 2 bytes
+                       }
+               })
+
+               b.Run("JapaneseRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, '本') // 3 bytes
+                       }
+               })
+
+               b.Run("MaxRune", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, MaxRune) // 4 bytes
+                       }
+               })
+
+               b.Run("InvalidRuneMaxPlusOne", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, MaxRune+1) // 3 bytes: RuneError
+                       }
+               })
+
+               b.Run("InvalidRuneSurrogate", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, 0xD800) // 3 bytes: RuneError
+                       }
+               })
+
+               b.Run("InvalidRuneNegative", func(b *testing.B) {
+                       buf := make([]byte, UTFMax)
+                       for i := 0; i < b.N; i++ {
+                               EncodeRune4(buf, -1) // 3 bytes: RuneError
+                       }
+               })
+       })
 }

 func BenchmarkAppendASCIIRune(b *testing.B) {

Similar Issues

unicode/utf8: add AppendRune #47609
unicode/utf8: make DecodeRune and DecodeLastRune inlineable #48195
cmd/compile,unicode/utf8: utf8.EncodeRune has different performance from the equivalent string conversion #48684
proposal: unicode/utf8: distinguish between bona fide encoding error and insufficient bytes #45898
unicode/utf8: RuneError should be a 4-byte value -- something uncreatable via DecodeRune's valid returns #47826
unicode/utf16: add AppendRune #51896
proposal: unicode: improvement of rune-checking funcs #68064
proposal: unicode/utf16: add DecodeRuneBytes #65511
proposal: bufio: add Reader.DiscardRunes(n int) and utf8: add RuneByteLen  #47621
bytes, strings: add ToValidUTF8 #25805

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

There is no function Encode in the unicode/utf8 package.  I assume you are taking about EncodeRune.
I'm confident that changing EncodeRune to require a slice of at least length 4 would break existing working code.  We shouldn't do that.  The performance benefits you measure are big percentages but in actual time they are less than a nanosecond.  I'm skeptical that this is measurable for anything other than micro-benchmarks.
Why is your new code faster?  Does it just have fewer bounds checks?  But the old code should also only have one bounds check.
Note that you don't need to use unsafe to convert, you are permitted to convert from a slice to a pointer to an array.

There is no function Encode in the unicode/utf8 package. I assume you are taking about EncodeRune.

Fixed!

Note that you don't need to use unsafe to convert, you are permitted to convert from a slice to a pointer to an array.

Thank you! Fixed as well. Interestingly, it appears that runtime.panicSliceConvert is more expensive than runtime.panicIndex, so it still helps to do _ = p[3] before doing the conversion.

I'm confident that changing EncodeRune to require a slice of at least length 4 would break existing working code. We shouldn't do that. The performance benefits you measure are big percentages but in actual time they are less than a nanosecond. I'm skeptical that this is measurable for anything other than micro-benchmarks.

Yes, it's possible. I'll try to make more tangible tests and see what would be the result outside of micro-benchmarks.

Why is your new code faster? Does it just have fewer bounds checks? But the old code should also only have one bounds check.

Yes, both perform a single bound check. My intention was that by having the bound check in an inlined function, then the bound check is hoisted to the caller, where I hoped that the runtime may have already proven that the bounds are preserved. Adding go:noinline causes EncodeRune4 to perform far worse, bu adding go:noinline also makes the result undiscernible from what would be otherwise, so I may be wrong there.

I made another synthetic benchmark, this time writing 1024*1024 runes of different widths (so, 1MiB for ASCII). There is no further data processing, just passing through runes to a slice of bytes.
goos: linux
goarch: amd64
pkg: github.com/diegommm/runes
cpu: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
                                    │   stdlib    │             EncodeRune4             │
                                    │   sec/op    │   sec/op     vs base                │
EncodeRuneText/invalid,_negative-8    2.930m ± 3%   2.662m ± 2%   -9.13% (p=0.000 n=20)
EncodeRuneText/ASCII-8                2.115m ± 2%   1.970m ± 3%   -6.86% (p=0.000 n=20)
EncodeRuneText/2_bytes-8              2.542m ± 3%   2.129m ± 3%  -16.22% (p=0.000 n=20)
EncodeRuneText/invalid,_surrogate-8   3.134m ± 3%   2.230m ± 3%  -28.84% (p=0.000 n=20)
EncodeRuneText/3_bytes-8              3.042m ± 3%   2.686m ± 3%  -11.71% (p=0.000 n=20)
EncodeRuneText/4_bytes-8              3.367m ± 1%   2.990m ± 1%  -11.19% (p=0.000 n=20)
EncodeRuneText/invalid,_too_long-8    3.031m ± 3%   2.603m ± 1%  -14.12% (p=0.000 n=20)
geomean                               2.851m        2.444m       -14.29%

In the case of 4-bytes runes, which is one of the most notable, there is a ~0.3ms improvement in a 4MiB test data (yes, for slower machines it will be greater, but I don't think it will be so in a significant way). Adding any interesting processing may probably have this improvement lost in noise.
To your point, I think it's clear that, while there is a reduction in run time, it becomes negligible compared to any meaningful work that could be performed on such volume of data, and the risk of breaking anyone's code would not be worth it. And not to mention increasing the surface of the API. I will be closing this issue as a consequence, but feel free to reopen if you think there would be anything else that would need to be investigated, I would be happy to help.
Thank you for your time and patience!
Benchmarking code (for reference)

var runeCases = []struct {
        r    rune
        name string
        n    int // number of bytes of what it should encode to
}{
        {-1, "invalid, negative", 3},
        {10, "ASCII", 1},
        {209, "2 bytes", 2},
        {surrogateMin, "invalid, surrogate", 3},
        {65535, "3 bytes", 3},
        {maxRune, "4 bytes", 4},
        {maxRune + 11, "invalid, too long", 3},
}

func BenchmarkEncodeRuneText(b *testing.B) {
        const runesToWrite = 1024 * 1024

        for _, c := range runeCases {
                buf := make([]byte, runesToWrite*c.n+4)

                b.Run(c.name, func(b *testing.B) {
                        b.Run("implem=stdlib", func(b *testing.B) {
                                for i := 0; i < b.N; i++ {
                                        dst := buf
                                        for j := 0; j < runesToWrite; j++ {
                                                n := utf8.EncodeRune(dst, c.r)
                                                dst = dst[n:]
                                        }
                                }
                        })

                        b.Run("implem=EncodeRune4", func(b *testing.B) {
                                for i := 0; i < b.N; i++ {
                                        dst := buf
                                        for j := 0; j < runesToWrite; j++ {
                                                n := EncodeRune4(dst, c.r)
                                                dst = dst[n:]
                                        }
                                }
                        })
                })
        }
}

authors: diegommm gabyhelp ianlancetaylor


text:
proposal: net: Add method to get underlying socket file descriptor or method to shutdown the socket

Proposal Details
Summery
Getting the underlying socket file descriptor is required when sending eg. shutdown signals to the socket, while keeping the http server running.
Why
By having a listener with SO_REUSEPORT enabled, there is no good way to tell the kernel to stop queuing requests for the listening service. The only I found to stop queuing requests is to shutdown the socket, but to do that we need the file descriptor for the socket.
By calling listener.Close() or http.Shutdown() will not tell the kernel to stop queuing requests to the service. If this is called while the kernel is queuing requests, all requests in queue (and new requests) will fail until the service stops. This would result in a lot of failed requests.
Example
Code
package main

import (
	"context"
	"fmt"
	"log"
	"net"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"golang.org/x/sys/unix"
)

func main() {
	mux := http.NewServeMux()

	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello world\n")
	})

	svc := http.Server{
		Handler: mux,
		Addr:    ":8000",
	}

	var socketFd uintptr

	listener := net.ListenConfig{}

	listener.Control = func(_, _ string, c syscall.RawConn) error {
		var opErr error

		err := c.Control(func(fd uintptr) {
			// Not recomended to do this, because the file descriptor is not
			// guaranteed to remain valid after this function finishes.
			socketFd = fd
			opErr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
		})
		if err != nil {
			return err
		}

		return opErr
	}

	go func() {
		l, err := listener.Listen(context.Background(), "tcp", ":8000")
		if err != nil {
			panic(err)
		}

		log.Fatal(svc.Serve(l))
	}()

	c := make(chan os.Signal, 1)
	signal.Notify(c, os.Interrupt)
	<-c

	log.Print("Shutting down service...")

	err := syscall.Shutdown(int(socketFd), syscall.SHUT_WR)
	if err != nil {
		panic(err)
	}

	// wait for the queue to drain
	time.Sleep(5 * time.Second)

	ctx, cancelFunc := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancelFunc()

	if err := svc.Shutdown(ctx); err != nil {
		log.Fatalln("Failed to shutdown http service:", err)
	}

	log.Print("Server stopped")
}

In this example, I manually call syscall.Shutdown(int(socketFd), syscall.SHUT_WR) to signal the shutdown signal to the socket.
In this code, we save the socket file descriptor during the Control function, which is not recommended, as the file descriptor is not guaranteed to be valid after the Control function. Ideally, we can use a method to get the underlying socket file descriptor, or have a method to tell the kernel to stop queuing requests for the socket.
Addinital details (as I understand it)
Enabling SO_REUSEPORT allows multiple services to listen on the same port, the kernel will use hash-based load balancing to determine where to route an incoming request. The request then sits in a queue for the specific service, then the service can accept the request when its ready.
Under heavy load, even by calling syscall.Shutdown() still occasional results in few failed requests, that might be because there is still a few requests in queue, or maybe not the right shutdown signal, I'm not completely sure. However, its still many times less failed requests then just shutdowning the listener or http server. Ideally this should have zero failed requests even under heavy load.

Related Issues and Documentation

proposal: Make sysListener public as SysListener to make SO_REUSEPORT available for UDP --- affected/package: net #60250 (closed)
proposal: net/http: context based graceful server shutdown #52805 (closed)

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

This idea only makes sense for a TCP listener.  So you can already do something like (untested)
    c, err := l.(*net.TCPListener).SyscallConn()
    if err != nil { panic("listener already closed?") }
    err = c.Control(func(fd uintptr) {
        if err := syscall.Shutdown(int(fd)); err != nil {
            log.Error(err)
        }
    })

Thank you @ianlancetaylor for that snippet, that seem to mostly work, I'm still getting some failed requests reported by ab under light load, however its overall very similar to my code.
I do wish there was a more reliable way to "shutdown" the socket to stop the kernel from queuing requests, but my original question has been answered by your snippet, thank you very much.

Update: My test code was faulty, the svc.Serve(l) should not exit the main thread on serve error (at least not stopped error, need to add error check in real code). In addition, calling syscall.Close(int(fd)) is sufficient.

authors: WestleyK gabyhelp ianlancetaylor


text:
proposal: container/set: time to re-open discussion

Proposal Details
The discussion at https://github.com/golang/go/discussions/47331 has been closed (and locked) for quite some time, presumably because we lacked generics and/or iterators. We now have both and the conversation can be re-opened and the proposal moved forward.

I don't see a proposal here.
If you want to discuss something, please use the forums.

The proposal is simply to re-open the discussion I referenced.

authors: mdwhatcott seankhliao


text:
x/website/_content/doc/modules: invalid require directive example

What is the URL of the page with the issue?
https://go.dev/doc/modules/managing-source#multiple-module-source
What is your user agent?
n/a
Screenshot

What did you do?
n/a
What did you expect to see?
The doc describes the setup of a mono-repository with multiple modules that each have their own version.
In the example it says that module module1 can be versioned by pushing the version tag module1/v1.2.3, but then incorrectly states the the require directive for that module and that version needs to be example.com/mymodules/module1 module1/v1.2.3, whereas instead it needs to be example.com/mymodules/module1 v1.2.3.
+ Module path as given in a user’s require directive: example.com/mymodules/module1 v1.2.3

What did you see instead?
- Module path as given in a user’s require directive: example.com/mymodules/module1 module1/v1.2.3

Related Issues

doc: cmd/go: document use with modules "from the ground up" #29298 (closed)
x/website: various mistakes in documentation on modules #44241 (closed)
cmd/go: document version tagging for modules not at the repo root #36014 (closed)
cmd/go: perhaps misleading 'require new/thing v2.3.4' example in documentation #28374 (closed)

Related Documentation

Managing module source > Choosing repository scope > Sourcing multiple modules in a single repository

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

Thanks for reporting.

authors: Eynorey findleyr gabyhelp


text:
Create structpb.UnmarshalTo similar to anypb

anypb supports an easy way to to unmarshall and set option using the anypb.UnmarshalTo().
This FR is to extend that to structpb as well.
My usecase maybe specific but its like this:
Google cloud logging's AuditLog returns a some fields as structpb.Struct
Request *structpb.Struct `protobuf:"bytes,16,opt,name=request,proto3" json:"request,omitempty"`

the data returned has a string @type field included that defines the message itself

the way to parse it now is something likethis:
g := &storagepb.GetIamPolicyRequest{} //  storagepb "google.golang.org/genproto/googleapis/storage/v1"
iamRequestStruct, err := audit_data.Request.MarshalJSON()
err = protojson.UnmarshalOptions{
    DiscardUnknown: true,
}.Unmarshal(iamRequestStruct, g)

which work but i think it'd be easier if structpb supported the same format as anypb:
err := structpb.UnmarshalTo(audit_data.Request., &g, proto.UnmarshalOptions{
    DiscardUnknown: true,
})

🤔 Sounds pretty reasonable. I don’t really see any particularly big reason why we should force a JSON round-trip work around.
BUt it seems like there is a disconnect in definitions and generated data. A structpb should not be producing a @type field, that’s an anypb feature. 😕 It just happens that they’re essentially compatible deifnitions.

Usign the structpb here for cloud logging  was done to accomodate arbitrary data (it could send back a protobuf or just standalone json, i think)
field Request *structpb.Struct
The operation request. This may not include all request parameters, such as those that are too large, privacy-sensitive, or duplicated elsewhere in the log record. It should never include user-generated data, such as file contents. When the JSON object represented here has a proto equivalent, the proto name will be indicated in the `@type` property.

the idea there is to inspect the data and then apply specific treatment (its awkward)
if we were just dealign with proto, sending anypb and using its functions  would be the way as you mentioned.
the more i think about this...this FR is to accomodate a specific apis' idiosyncracy so i'm ok with the workaround and closing

for ref, cloud logging also allows for mutiple assignable types to get sent down (eg, logging.LogEntry allows for:
Resource  `protobuf:"bytes,8,opt,name=resource,proto3" json:"resource,omitempty"`
	// The log entry payload, which can be one of multiple types.
	//
	// Types that are assignable to Payload:
	//	*LogEntry_ProtoPayload
	//	*LogEntry_TextPayload
	//	*LogEntry_JsonPayload

which if its a LogEntry.ProtoPayload is an anypb.Any  and i can deal with that easily with the anypb functions...

Hm… it seems that the anypb UnmarshalTo is done via defering the actual unmarshalling until that call… the JSON version skips all of that by loading the resolved message and unmarshalling the JSON directly to that message rather than having at any point the anypb.Any struct instantiated.
I don’t think there’s any standards compliant way we could really ensure that we could treat this idiosyncratic use of structpb.Struct conflated with anypb.Any.
One should be able to write something using protoreflect if one really wants to avoid a round-trip marshal/unmarshal. But yeah, I think this is a kind of one-off case, and specific to this API, so the workaround is more reasonable than trying to write a general purpose protoreflect based already-unmarshaled structpb.Struct into an arbitrary other type like anypb.Any does.

thanks.  i'll stick with the workaround.  for this specific api, i'll ask to document the worksarounds in cloud logging docs (in various languages, if feasible even!).
fyi, envoyproxy (which in in cpp) uses something called typed_config declarations which i think has to be anypb's.

repo: golang/protobuf

license_source: bigquery


text:
Update ru_RU.lang

Лучше, нет?

Да, оно и вправду лучше! Спасибо.

source: gharchive/pull-request

license: mit


text:
update for kotlin 2.1.0

When using this plugin with kotlin 2.1.0 and a js project, my build fails with
... [irrelevant stack]

Caused by: java.lang.NoSuchMethodError: 'org.jetbrains.kotlin.gradle.targets.js.nodejs.NodeJsRootExtension org.jetbrains.kotlin.gradle.targets.js.npm.NpmProject.getNodeJs()'
	at com.goncalossilva.resources.ResourcesPlugin.applyToCompilation(ResourcesPlugin.kt:68)
	at org.jetbrains.kotlin.gradle.plugin.SubpluginEnvironment.addSubpluginOptions(SubpluginEnvironment.kt:47)
	at org.jetbrains.kotlin.gradle.plugin.KotlinJsIrSourceSetProcessor$doTargetSpecificProcessing$2.invoke(KotlinJsIrSourceSetProcessor.kt:50)
	at org.jetbrains.kotlin.gradle.plugin.KotlinJsIrSourceSetProcessor$doTargetSpecificProcessing$2.invoke(KotlinJsIrSourceSetProcessor.kt:46)
	at org.jetbrains.kotlin.gradle.utils.WhenEvaluatedKt$whenEvaluated$4.execute(whenEvaluated.kt:48)
	at org.jetbrains.kotlin.gradle.utils.WhenEvaluatedKt$whenEvaluated$4.execute(whenEvaluated.kt:45)
... [snipped]

I guess this method was removed in Kotlin 2.1.0; which also causes this PR to fail https://github.com/goncalossilva/kotlinx-resources/pull/138.
It seems like this change is breaking for Kotlin 2.0.x so maybe someone else can find a way to make it work in both.

Thanks for this! Reflection can help making it work with both Kotlin versions simultaneously, but I'd rather keep it simple.
We already had a note about Kotlin vs kotlinx-resources versions in the README:

The latest version targets Kotlin 2.0 and is incompatible with Kotlin 1.9. On 2.0 and above, use the latest version available. On 1.9, use the latest version below v0.9.0.

I'll just turn it into a table. 😄

authors: chadselph goncalossilva

repo: goncalossilva/kotlinx-resources

license: MIT


text:
Support for Quantized ELU is missing in TFLite MLIR converter

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): WSL2 Ubuntu
TensorFlow installed from (source or binary): pip
TensorFlow version (or github SHA if from source): Latest

Unlike ReLU for example, currently ELU isn't supported for 8 bit quantization in TFLite:

I think we can confirm this by looking at ELU's definition in tfl_ops.td , it doesn't have the quantizable trait.
While researching upon this issue, I stumbled upon an old stackoverflow issue  implying that quantized ELU was once supported. Maybe it was skipped after the move to MLIR?
Are there plans to support it? If not, will it be straightforward to implement it by my own?

This issue originally reported by @Doomski99 has been moved to this dedicated repository for LiteRT to enhance issue tracking and prioritization. To ensure continuity, we have created this new issue on your behalf.
We appreciate your understanding and look forward to your continued involvement.

repo: google-ai-edge/LiteRT


text:
How can I render a image avatar using face landmarker detection result?

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
Yes
OS Platform and Distribution
Mac M1
MediaPipe version
0.10.12
Bazel version
No response
Solution
face landmarker detection
Programming Language and version
web
Describe the actual behavior
Now I'm using @mediapipe/tasks-vision DrawingUtils to render grid as the codepen example.
Describe the expected behaviour
I wonder other render effect, such as a cartoon avatar.
Standalone code/steps you may have used to try to get what you need
I notice the official page "https://mediapipe-studio.webapps.google.com/studio/demo/face_landmarker" has provide a choice about "display mode", is there any example code to teach me how to set it in my own codes.
I hope more interesting use about face landmarker detection model.
Hoping for your reply!

Other info / Complete Logs
No response

Hi @ly15927086342,
Currently, customization of our Avatar render effect is not possible, as this feature is not open-sourced. Additionally, we do not have examples available on other platforms. If you are interested in this feature, we recommend raising a new feature request with all the relevant details. Based on the discussion and demand, we may implement this in the future.
Thank you!!

repo: google-ai-edge/mediapipe


text:
Sceneform anchor points sway as depth changes.

Sceneform locates a cylindrical model through the anchor point. The deeper the anchor point is, the greater the negative value is. Why can't the position of the cylinder be fixed, but it wobbles?
               Pose pose = Pose.makeTranslation(0, 0, -1.26f);
                Anchor anchor = arSceneView.getSession().createAnchor(pose);
                final AnchorNode anchorNode = new AnchorNode(anchor);
                anchorNode.setParent(arSceneView.getScene());

Are you trying to make an anchor at a fixed distance in front of the camera?

@gstanlo I want to fix an anchor on the bottle in the texture, and build a virtual bottle model through this anchor to cover the bottle in the texture. How do you do this?

Hi, Im unsure how exactly Sceneform is being used here.  Can you clarify "I want to fix an anchor on the bottle in the texture, and build a virtual bottle model through this anchor to cover the bottle in the texture"

@malik-at-work I want to add cards around the cylinder of the camera.
How do we achieve this special effect?

authors: LyngLi gstanlo malik-at-work

repo: google-ar/sceneform-android-sdk


text:
Reduce penalty for missing ignore [silence], [hidden], etc tracks.

What's the problem? How can I reproduce it?
Missing [silence] and [hidden] type tracks skew the similarity so that the
album is incorrectly matched, or not at all.

If it seems relevant, what system are you running on (OS, Python version,
etc.)?

Win7 x64, Python 2.7

Original issue reported on code.google.com by volden....@gmail.com on 30 Jul 2012 at 2:07

Do not comment here, but on sampsyo/beets#144

authors: GoogleCodeExporter untitaker

repo: google-code-export/beets


text:
How can I connect a coral dev board micro to a coral dev board?

Description
What I am trying to achieve
I would like to use a coral dev board as a host which can run some modified python example scripts as a client and receive image data from three USB connected coral dev board microcontrollers and store image data on the client.
Some context
For example, I would like to flash three coral dev board microcontrollers with the camera_streaming.cc example and give them each a unique IP address (192.168.101.10, 192.168.102.10 and 192.168.103.10). Like so:
python3 scripts/flashtool.py -e camera_streaming_rpc --subapp camera_streaming_rpc_usb --usb_ip_address 192.168.101.10
python3 scripts/flashtool.py -e camera_streaming_rpc --subapp camera_streaming_rpc_usb --usb_ip_address 192.168.102.10
python3 scripts/flashtool.py -e camera_streaming_rpc --subapp camera_streaming_rpc_usb --usb_ip_address 192.168.103.10

Then run a python script on a coral dev board which acts as a client to request images from each coral microcontroller.
I have the above functionality working when connecting the three microcontrollers to my ubuntu 20 machine (I can run a python client that makes GET requests to each coral microcontroller and store the response data).
My main problem
When connecting a coral dev board microcontroller to a coral dev board I am unable to see any devices on usb0 or usb1 the same way I can on an ubuntu 20 machine. I tried to follow the instructions here by configuring the /etc/NetworkManager/system-connections/gadget0.nmconnection and /etc/NetworkManager/system-connections/gadget1.nmconnection files to the same domains as the USB ip address that I flashed. E.g
mendel@zippy-tang:~$ sudo cat /etc/NetworkManager/system-connections/gadget1.nmconnection
[connection]
id=gadget1
uuid=6c85437c-ba59-4522-baf0-bf68e006b601
type=ethernet
interface-name=usb1
permissions=

[ethernet]
mac-address-blacklist=

[ipv4]
address1=192.168.103.2/24
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

However the above has no success as I still cannot ping the board and receive a response.
mendel@zippy-tang:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 28:bd:89:bc:a0:96 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.70/24 brd 192.168.178.255 scope global dynamic noprefixroute eth0
       valid_lft 7120sec preferred_lft 7120sec
    inet6 fe80::797c:2a5c:c17b:6afa/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 3000
    link/ether 28:bd:89:bc:a0:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.141/24 brd 192.168.178.255 scope global dynamic noprefixroute wlan0
       valid_lft 7120sec preferred_lft 7120sec
    inet6 fe80::3f3d:7c3e:c48c:a54f/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: p2p0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 3000
    link/ether 00:0a:f5:89:89:81 brd ff:ff:ff:ff:ff:ff
5: usb0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 02:22:78:0d:f6:df brd ff:ff:ff:ff:ff:ff
    inet 192.168.103.100/24 brd 192.168.103.255 scope global noprefixroute usb0
       valid_lft forever preferred_lft forever
    inet6 fe80::2d12:dfdd:6c07:530a/64 scope link tentative noprefixroute
       valid_lft forever preferred_lft forever
6: usb1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 02:22:78:0d:f6:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.102.2/24 brd 192.168.102.255 scope global noprefixroute usb1
       valid_lft forever preferred_lft forever
    inet6 fe80::549d:3a3d:cd32:1636/64 scope link tentative noprefixroute
       valid_lft forever preferred_lft forever
mendel@zippy-tang:~$ ping 192.168.103.10
PING 192.168.103.10 (192.168.103.10) 56(84) bytes of data.
From 192.168.103.100 icmp_seq=1 Destination Host Unreachable
From 192.168.103.100 icmp_seq=2 Destination Host Unreachable
From 192.168.103.100 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.103.10 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 83ms

Here is the output of netstat as well
mendel@zippy-tang:~$ sudo netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.178.1   0.0.0.0         UG        0 0          0 eth0
0.0.0.0         192.168.178.1   0.0.0.0         UG        0 0          0 wlan0
192.168.102.0   0.0.0.0         255.255.255.0   U         0 0          0 usb1
192.168.103.0   0.0.0.0         255.255.255.0   U         0 0          0 usb0
192.168.178.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.178.0   0.0.0.0         255.255.255.0   U         0 0          0 wlan0

Not sure if what I'm doing is completely wrong so I figured it would be best to ask for any help.
Click to expand!
Issue Type
Support
Operating System
Mendel Linux
Coral Device
Dev Board, Dev Board Micro
Other Devices
No response
Programming Language
No response
Relevant Log Output
No response

On dev board please try the below steps to connect to dev board micro. Thanks!!
sudo apt-get install screen
sudo usermod -aG plugdev,dialout $USER
sudo reboot now
screen /dev/ttyACM0 115200

Hi @hjonnala, thanks for your response. Apologies maybe I should've been more specific, but I would specifically like to establish a connection with the microcontroller using the USB network interfaces on the coral dev board. Such as, usb0 usb1 usb2 etc.
screen will help me connect to the board and monitor the serial output but doesn't help me for my problem.
I noticed yesterday some differences between my laptop (running ubuntu 20) and the coral dev board for the USB ethernet interfaces. The main difference being the output of cat /sys/kernel/debug/usb/devices for both devices like follows:
laptop:
I:* If#= 2 Alt= 0 #EPs= 2 Cls=02(comm.) Sub=0c Prot=07 Driver=cdc_eem
coral dev board:
I:* If#= 2 Alt= 0 #EPs= 2 Cls=02(comm.) Sub=0c Prot=07 Driver=(none)
This led me to find this on the coral dev board after running zcat /proc/config.gz | grep -i 'cdc_eem'
# CONFIG_USB_NET_CDC_EEM is not set

Is this what I would need to set to allow the coral dev board to communicate with the coral micro over the usb ethernet interface?
Thanks again for your help.

authors: dylan-leslie hjonnala

repo: google-coral/coralmicro


text:
Updated troubleshooting to add permissions example

Added another case that can cause auth to fail, helpful to users on day one of using the tool and trying to understand what's happening.
Fixes https://github.com/google-github-actions/auth/issues/260#issuecomment-1419662751

@sethvargo this seems to be stuck on some checks, or approvals?

/cc @averikitsch

repo: google-github-actions/auth


text:
FIREBASE_CONFIG environment variable

How do i set FIREBASE_CONFIG environment variable for my python function
I am not able to set the FIREBASE_CONFIG environment variable using the KEY1=VALUE syntax using env_var arguments of the google-github-actions/deploy-cloud-functions@main action.
I get the below error
Env Vars must be in "KEY1=VALUE1,KEY2=VALUE2" format, received "databaseURL":"https://.firebaseio.com"

Hi @arkum
Thanks for the report. Is the env_vars: databaseURL=https://firebaseio.com syntax throwing the error?
Could you show an example config?

Thanks @bharathkkb The expected format is as below
Key is FIREBASE_CONFIG and Value is {"projectId":"project-id","databaseURL":"https://project-id.firebaseio.com","storageBucket":"project-id.appspot.com","locationId":"asia-south1"}
Where project-id will the gcp/firebase projectid
env_vars:FIREBASE_CONFIG={"projectId":"project-id","databaseURL":"https://project-id.firebaseio.com","storageBucket":"project-id.appspot.com","locationId":"asia-south1"}
i tried passing the value inside a single quote and double quote. But did not work

@arkum that makes sense. What is happening is we split KEY=VALUE pairs based on , for env_vars and it is trying to parse by splitting the json configs which also contains ,.  A short term fix might to break up into individual k,v pairs i.e env_vars: projectId= project-id,databaseURL=https://project-id.firebaseio.com,storageBucket=project-id.appspot.com and then in your application, create the FIREBASE_CONFIG. For instance,
const FIREBASE_CONFIG={};
FIREBASE_CONFIG.projectId=process.env.projectId;
FIREBASE_CONFIG.databaseURL=process.env.databaseURL;
...

Longer term I think we should have a better solution, perhaps we use a less common delimiter or have an escape char. @averikitsch I assume this is also relevant for Run, WDYT?

thank you @bharathkkb i  can see gcloud functions deploy has option for reading environment variables from a file. Wondering whether we can bring it to Function deploy action. Another thing i want to try is to use the setup gcloud action and try to run the gcloud function set-env-var command. I will keep you posted

Hi @bharathkkb deploying through gcloud works
`- name: Setup gcloud
uses: google-github-actions/setup-gcloud@master
    with:
      project_id: project-id
      service_account_key: ${{ secrets.FIREBASE_FUNCTION_TEST_SERVICE_ACCOUNT }}
      export_default_credentials: true
  - name: Deploy poincareplot function
    working-directory: ./Python/PythonFunction
    run: gcloud functions deploy functioname --runtime=python38 --entry-point=entrypoint --env-vars-file ../config-test.yml

`
In my config file i defined the environment variable as
FIREBASE_CONFIG: '{"projectId":"project-id","databaseURL":"https://project-id.firebaseio.com","storageBucket":"project-id.appspot.com","locationId":"asia-south1"}'

Thanks for the update @arkum
env-vars-file seems like a reasonable add and would keep the configs cleaner

Longer term I think we should have a better solution, perhaps we use a less common delimiter or have an escape char. @averikitsch I assume this is also relevant for Run, WDYT?

Passing in JSON for the FIREBASE_CONFIG seems like an anti-pattern. I would only use the firebase CLI to interact and create this config. It also looks like this is a special env var functions uses and I wouldn't try to overwrite it. These values should be passed into the action or used with setup-gcloud.
A solution for both would be looking into escaping quotes for the JSON value or being able to add a delimiter. Since env vars may contain sensitive information, I wouldn't  want to commit them to a file.

It also looks like this is a special env var functions uses and I wouldn't try to overwrite it. These values should be passed into the action or used with setup-gcloud.

That makes sense, I did not know FIREBASE_CONFIG was auto populated.

Since env vars may contain sensitive information, I wouldn't want to commit them to a file.

Even with the current approach the env vars would be committed to the action workflow yml? My thoughts with the env-vars-file was that if you have alot of vars to populate, it maybe cleaner to just use a file instead of passing them comma separated in the workflow yml.
For secrets, I think the approach should be using secretmanager and retrieving it at runtime directly from the API.

Hi @averikitsch as of now firebase CLI doesn't support deploying python cloud functions. Neither the firebase deploy action(s). Wondering whether the best approach will be to make the firebase deploy action include support for python functions as well, which sets the FIREBASE_CONFIG variable automatically

Hi @averikitsch as of now firebase CLI doesn't support deploying python cloud functions. Neither the firebase deploy action(s). Wondering whether the best approach will be to make the firebase deploy action include support for python functions as well, which sets the FIREBASE_CONFIG variable automatically

I see, you are using the python Cloud Functions for now. I'm still hesitant about an env var file and would lean towards improving the env var parsing so you can pass in env_vars: FIREBASE_CONFIG=${{ secrets.myConfig }}, .... It seems like an extra step to have to add the env vars to a file and then pass the file into the action.
For @arkum , I would deploy the function the first time manually and set the FIREBASE_CONFIG env var. I don't assume this changing on redeployments. Then you'd use the action to do the redeployments which should maintain the env var. Is this correct @bharathkkb ?

fixed via #34

Best practices for secrets still apply even with the new feature: https://cloud.google.com/functions/docs/env-var#managing_secrets

authors: arkum averikitsch bharathkkb


text:
Multi-GPU training

Hi Kevin,
Thanks for the great work and releasing the codes/models. Was wondering if you have tried multi-GPU training for ELECTRA-base and ELECTRA-large (does your current codes support multi-GPU)? And if you have stats for multi-GPU experiments as well?
Also the stats for single GPU training of ELECTRA-base and ELECTRA-large (how many days needed till they converge to a descent performance?).
Thanks!
-Hamid

I've tried the current starter's code for pretraining a small network. It seems like the model is trained on a single GPU.

Yes. It is working on a single GPU. Looking for multi-gpu support.

Gentle follow up Kevin, any thoughts?
Thanks,
-Hamid

Is there any plans for multi-gpu support @clarkkev

Hope to support multiple GPU and provide detailed configuration. @clarkkev

@008karan @Palang2014 How are you getting the model to run on a GPU? Even with a GPU available, I'm only able to run on CPU. Mostly interested in running the fine-tuning, not the pre-training.

@008karan @Palang2014 How are you getting the model to run on a GPU? Even with a GPU available, I'm only able to run on CPU. Mostly interested in running the fine-tuning, not the pre-training.

I had the same issue and I realised it was because the program could not find the cuda libraries. Check if you get messages like
"Successfully opened dynamic library libcudnn.so.7"
If not, or if you see errors saying it couldn't find some cuda libraries, maybe you have the same problem?

repo: google-research/electra


text:
No module named 'absl'

No module named 'absl'  ,How to solve this

pythton setup.py install

authors: WeigamgHe whhwhhhh

repo: google-research/maxim


text:
ERA5 sea surface temperatures all NaN

Some of the era5 sea surface temperatures are all NaN. You can verify by running:
import numpy as np
import gcsfs
import xarray as xr

fs = gcsfs.GCSFileSystem(token="anon")
for path in fs.ls("weatherbench2/datasets/era5"):
    if ".zarr" in path:
        try:
            mapper = fs.get_mapper(path)
            ds = xr.open_zarr(mapper)
        except FileNotFoundError:
            print(f"issue opening {path}")
        else:
            sst_slice = ds["sea_surface_temperature"].isel(time=-1).values
            if np.all(np.isnan(sst_slice)):
                print(f"all NaN SST: {path}")

which prints the following datasets:
all NaN SST: weatherbench2/datasets/era5/1959-2022-1h-240x121_equiangular_with_poles_conservative.zarr
all NaN SST: weatherbench2/datasets/era5/1959-2022-1h-360x181_equiangular_with_poles_conservative.zarr
all NaN SST: weatherbench2/datasets/era5/1959-2022-6h-128x64_equiangular_conservative.zarr
all NaN SST: weatherbench2/datasets/era5/1959-2022-6h-64x32_equiangular_conservative.zarr
all NaN SST: weatherbench2/datasets/era5/1959-2022-6h-64x33.zarr

It doesn't seem to matter which time slice is checked.
This might be related to #124.

Hi, can I be assigned to this issue?


text:
Source information

There's a mention of

But did you know that 66% of mobile purchases are made through websites rather than apps?

I was wondering what's the source for this claim?

@matijagrcic - what page is this on? Will try to find out who wrote and where it came from.

@petele Sorry, forgot to include the links. It's on payment-request.md which is public on the Bringing Easy and Fast Checkout with Payment Request API page.
Thanks.

@agektmr - can you add a source to this?

I have the PDF by Deutsche Bank but don't know where it came from. @zkoch may know.

Thanks for looking into it @petele and @agektmr.
It would also help If you can link to that PDF if that's possible.

Any update on this @petele @agektmr ? I've saw your talk and it's also referencing Deutsche Bank as a source

@matijagrcic For "66% lower conversion rate" look at this website "April 2016 update - retail conversion rates by device" the Q42015 table 1.43% vs 4.66% conv phone vs traditional.
For "66% of mobile transactions are on web" I'm still looking for where the PDF came from.

@agektmr Thanks for that, looking forward to the PDF info.

authors: agektmr matijagrcic petele

repo: google/WebFundamentals


text:
Fix captions not supporting links.

What's changed, or what was fixed?
We ran into an issue with video captions not supporting inline links. As this is live, I'm going to drop the link while I try to play around with what can render.

[ ] This has been reviewed and approved by (NAME)
[x] I have run npm test locally and all tests pass.
[x] I have added the appropriate type-something label.
[x] I've staged the site and manually verified that my content displays correctly.

CC: @petele

:+1:

:tada: This has been pushed live to https://developers.google.com/web/

authors: WebFundBot addyosmani