text:
go/types: "missing return" error should be "unreachable statement"

Go version
go version go1.22rc2 linux/amd64
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/biget/.cache/go-build'
GOENV='/home/biget/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/edi/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/edi/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go-1.22'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go-1.22/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22rc2'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/user/1300/go-build1494716450=/tmp/go-build -gno-record-gcc-switches'

What did you do?
The following function should either compile without problem or give a "unreachable code", but it gives "missing return".
func bad_return(x bool) int8 {
	switch x {
	case true:
		return 1
		break
	default:
		return 0
	}
}

It might has something to do with the order of validation.
If I add a return to the end of the function, then I get "unreachable code".
Here is the same function in the playground
What did you see happen?
./prog.go:15:1: missing return
What did you expect to see?
./prog.go:11:3: unreachable code

The error about unreachable code is a vet check, not a type checker error.
The type checker implements a rather strict definition of terminating statements, specified here. Per that specification, the type checker is correct.
We could improve this, but I'm not sure it's a common enough mistake, nor confusing enough, to warrant the immediate attention implied by my assignment. I'm moving this into the backlog, though I'll discuss with @griesemer in our next meeting.

The reported error is correct per the spec. I don't think we should do anything here.
I note that the program with an additional return at the end actually runs in the playground, even with the "unreachable code" error.

Agreed, closing this as working as intended.

authors: BigET findleyr griesemer


text:
go compiler: Incorrect results on GOOS=linux GOARCH=arm64 running on Google t2a-standard-8 (Ampere Altra)

Go version
go version go1.22.1 linux/amd64
Output of go env in your module/workspace:
GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/seth.bunce/.cache/go-build'
GOENV='/home/seth.bunce/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/seth.bunce/go/pkg/mod'
GONOPROXY='github.robot.car'
GONOSUMDB='github.robot.car'
GOOS='linux'
GOPATH='/home/seth.bunce/go'
GOPRIVATE='github.robot.car'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/seth.bunce/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.1.linux-amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/seth.bunce/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.1.linux-amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/seth.bunce/go/src/github.robot.car/cruise/rbe/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1436812387=/tmp/go-build -gno-record-gcc-switches'

What did you do?
I cross compiled the following program for linux/arm64 with the following command, "GOARCH=arm64 go install ./cmd/armtest"
package main

import (
	"fmt"
	"hash/crc32"
	"math"
)

func hash() int {
	u32 := crc32.Checksum(nil, crc32.IEEETable)
	return int(u32 % math.MaxInt32)
}

func main() {
	result := hash()
	if result < 0 {
		fmt.Printf("FAIL: %d\n", result)
	} else {
		fmt.Printf("PASS: %d\n", result)
	}
}

I copied the built binary to a t2a-standard-8 running in GCP.
What did you see happen?
I ran the program and observed an incorrect result.
root@gcp-cpu-arm-rbe-test-swfn:/# ./armtest
FAIL: -4294967296

What did you expect to see?
I expected to see the same output I see on linux/amd64 (a google n1-standard-32 GCP instance).
seth.bunce@cs-a2n6x93j-heavy-homedir-946879:~/go/src/github.robot.car/cruise/rbe$ go run ./cmd/armtest/main.go
PASS: 0

This is fixed on tip, see #66076 for backport issue

Duplicate of #66066

source: gharchive/issue

authors: sbunce seankhliao

repo: golang/go

license_type: permissive

license_source: github-api


text:
proposal: text/encoding/charmap: Support for code page 1125 aka cp866u

Proposal Details
👋 Hey team,
I'm encountering issues with the charmap package when attempting to decode/encode using code page cp1125. This encoding is missing. Given that some of our Ukrainian projects rely on CP1125, especially when interacting with legacy systems, addressing this would be greatly beneficial.
IBM CP1125 (aka cp866u) is Ukrainian government standard (RST 2018-91) for DOS, based on common "alternative" encoding, but different from cp866 in 0xF2-0xF9. It is known by GNU iconv as CP1125.

https://public.dhe.ibm.com/software/globalization/gcoc/attachments/CP01125.pdf
https://public.dhe.ibm.com/software/globalization/gcoc/attachments/CP01125.txt
https://github.com/unicode-org/icu/blob/main/icu4c/source/data/mappings/ibm-1125_P100-1997.ucm
https://en.wikipedia.org/wiki/Code_page_866#Ukrainian_and_Belarusian_variants
https://segfault.kiev.ua/cyrillic-encodings/#ruscii

https://github.com/microsoft/vscode/issues/230438
I'd be happy to provide more details or assist with testing if needed.
Thank you for considering this request.

Oh, it seems this issue should be located at https://github.com/golang/text,
sorry guys.

CC @mpvl
I don't see a reason for this to be in the proposal process.  Happy to hear otherwise.

authors: ianlancetaylor yukal


text:
Warnings with Block Comments—Spacing, and Extra Asterisks

If you begin a block comment with a space, golint does not recognize that it begins with the exported function name.
/* TestFunction does ... */
func TestFunction() {}

comment on exported function TestFunction should be of the form "TestFunction ..."

The following passes without any warnings.
/*TestFunction does ...*/

Some people also prefer using multiple asterisks.
/***TestFunction does ...***/

comment on exported function TestFunction should be of the form "TestFunction ..."

+1

Does gofmt adjust block comments in any way which would dictate the "correctness" here?

Just stumbled upon this issue...
Is this really unintended? Where is this specified? Im about 3 days into go and not quite sure where to get the information about specs...
Just had a look into the source:
In my opinion this switch-case is not working properly. There its defined what prefixes to strip from the comment (like " " or "*"...). I think it should also consider "* " (asterisk + whitespace).
		// Remove comment markers.
		// The parser has given us exactly the comment text.
		switch c[1] {
		case '/':
			//-style comment (no newline at the end)
			c = c[2:]
			// strip first space - required for Example tests
			if len(c) > 0 && c[0] == ' ' {
				c = c[1:]
			}
		case '*':
			/*-style comment */
			c = c[2 : len(c)-2]
		}

Also I'm not 100% sure where else this function is used - this might have some unintended implications.

I have an internal fix for this coming that will be exported soon.

I like @voigt 's suggestion to strip asterisk or whitespace.  I format with a leading tab (gasp)  and the linter complains that my comment is formatted incorrectly

@dsnet did your fix for this get completed?

@dsnet This issue appears to be outstanding in Golint 0.0+git20161013.3390df4-1, which is the latest version in Debian Stable (Stretch). It's a little annoying inside Atom; I look forward to the fix!

@dsnet what‘s the status on the internal fix?

I apologize. I really dropped the ball on this one. I'm searching my email inbox and searched through all my CLs or CLs that I reviewed around that time frame, and I'm not sure what I was talking about.

+1 Extremely wierd bug..

Yep, still there 😢

Had this
Problem:
/test function bla bla bla/
func CreateTestContext(....)
and was giving me the strange message: warning: comment on exported function CreateTestContext should be of the form "CreateTestContext ..."
Solution:
Ensure you include the name of the function in the comment itself, like so:
/CreateTestContext bla bla bla/
and that did the trick.
Even you have white-space between /* and function name, you'd still get a warning.
Pretty idiotic I must say, but for the time being, that solution works.

is this project abandoned?

authors: Jesse-V Jessicaward andybons danesparza dsnet freebiesoft jkalandaibm josephholsten kkirsche mileusna prkara theGeekPirate voigt

repo: golang/lint

license_source: bigquery


text:
Why Register in the first place?

We started a discussion here
https://github.com/golang/protobuf/commit/cf10ca0ea91371dd30946defd93a0ed43aa1efe7#commitcomment-20160444

I think Registering is breaking people's setups and I am wondering why this is needed in the first place

@bcmills says

proto.MessageType and proto.MessageName require it. So does extension support, and ptypes.DynamicAny, and there may be other cases I'm not thinking of at the moment.

Also to be concrete here one place it is breaking people's setups
https://github.com/golang/protobuf/issues/178#issuecomment-215356264

Register fixes #179 only because ExtensionsMap was removed, which was another controversial move.

proto.MessageType and proto.MessageName are nice features, but not really necessary.
Extension support: ok yes maybe, but sacrificing some safety can also be used to get rid of Register.
How is register needed for DynamicAny?

Register fixes #179 only because ExtensionsMap was removed,

Register has ~nothing to do with #179.  #179 requires having a copy of the generated Descriptor proto package; the only impact that registration has on that is to require the generated Descriptor package to be the same one that other uses of Descriptor depend on.  (But that's true in general anyway: within a given binary, there should be one and only one copy of each package needed.  To do otherwise leads to bloated binaries.)

proto.MessageType and proto.MessageName are nice features, but not really necessary.

This is a Chesterton's Fence.  If you want to remove these methods, you should start by understanding what purpose they serve (or served).  "Not really necessary" merits further explanation.

Extension support: ok yes maybe, but sacrificing some safety can also be used to get rid of Register.

Extensions are already error-prone; sacrificing safety is unlikely to help most users.  But can you be more specific about the tradeoff you're proposing?  What kinds of errors would it fail to detect?

How is register needed for DynamicAny?

DynamicAny requires the ability to map from a fully-qualified message name to a Go type.  That is, it requires proto.MessageType.

How does having one copy of the generated Descriptor work with go vendoring?
My statement about not really necessary was a little strong, sorry.  Thank you for explaining DynamicAny's usage of of proto.MessageType.
I have not really thought it through, but don't you think there could be a way to do Extensions without Register, since the user is already passing in the E_generatedextensionstruct
Ok I didn't know about the implementation of DynamicAny using proto.MessageType.  Another way might be doing something similar to how extensions work.  Ask the user to pass in the correct type to unmarshal Any into.  But that will probably be a big change.

How does having one copy of the generated Descriptor work with go vendoring?

I honestly don't know.  The proto package was designed well before vendoring support and in the context of predominantly "monorepo" usage, and I personally haven't had much exposure to Go vendoring tools.

I have not really thought it through, but don't you think there could be a way to do Extensions without Register, since the user is already passing in the E_generatedextensionstruct

Yeah, it does seem possible to do extensions without Register.  Thinking about it some more it's likely more of an issue for the (Google-internal) "weak" field feature (the one corresponding to FieldOptions.weak in the descriptor), which is a precursor to extensions.  So it might be possible to remove registration from the open-source version of extensions.

That still leaves the problem of DynamicAny, though.  Given AnyMessageName and UnmarshalAny, it's not obvious to me whether DynamicAny is actually all that useful, but removing it would definitely not be backward-compatible.

I'm not sure I get the context here, but using Any types are not supported in all languages, so having the ability to simulate the same behavior using an envelope message with typename and messagedata, you still need to peek into the registered types if doing cross platform integration.

Thanks for your really cool response @bcmills
I really appreciate it :)
I wonder whether:

MarshalAny could have taken the messageName as input.
UnmarshalAny could have taken in a proto.Message not of type DynamicAny and not do any name checking.

Then it doesn't seem like Any needs to call "home" to the proto package

I wonder whether:

MarshalAny could have taken the messageName as input.

Possible but error-prone.  The message name is proto-path-qualified, so it isn't derivable from the Go name for the type; without registration there's no easy way to detect typo'd names.
However, MarshalAny would be easy to implement (with its current signature) without registration.  Generated message types already include a method to access the file and message descriptors, which contain the path and name information.  So MarshalAny is (fortunately) not a problem.

UnmarshalAny could have taken in a proto.Message not of type DynamicAny and not do any name checking.

It already supports that usage, and the name checking could be implemented without registration using the same technique as for MarshalAny.

The major use-case that DynamicAny addresses is for programs that use type-switches.  With DynamicAny, one can write this code:
var ext ptypes.DynamicAny
if err := ptypes.UnmarshalAny(a, &ext); err := nil {
  …
}
switch m := ext.(type) {
case *somepb.SomeProtoExtension:
  …
case *otherpb.SomeOtherExtension:
  …
default:
  …
}

and the compilation process will ensure that the extensions are valid types and the (implict) names are correct.  Without DynamicAny, the function becomes:
switch ptypes.AnyMessageName(a) {
case "some.proto.path.SomeProtoExtension":
  m := new(somepb.SomeProtoExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

case "other.proto.path.SomeOtherExtension":
  m := new(otherpb.SomeOtherExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

default:
  …
}

and now we've lost the compile-time checking on the paths: we'll have to rely on test coverage to avoid typos in the strings.  (We've also introduced a bunch of redundancy in the user code: the message types already know their paths, so why should the user repeat them?)
Another way to write this function would be:
var (
  someExt somepb.SomeProtoExtension
  otherExt otherpb.SomeOtherExtension
)
switch {
case ptypes.UnmarshalAny(a, &someExt) == nil:
  …

case ptypes.UnmarshalAny(a, &otherExt) == nil:
  …

default:
  …
}

but that's O(N) for what ought to be an O(1) switch and also requires fairly sophisticated compiler optimizations to avoid wasting a huge amount of memory.  (In my experience, one should not expect such optimizations from the gc compiler.)
There is an equivalent (and somewhat more subtle) use-case for programs which use reflection to process protocol messages using a sort of plugin architecture: currently, a package can use DynamicAny even for types that package doesn't depend on or know about.

I assume the name is the same name that is generated as a parameter to proto.RegisterType.  This means that this could also be generated as a method on each type.
So then we chould have
switch ptypes.AnyMessageName(a) {
case (somepb.SomeProtoExtension{}).Name():
  m := new(somepb.SomeProtoExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

case (otherpb.SomeOtherExtension{}).Name():
  m := new(otherpb.SomeOtherExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

default:
  …
}

This is certainly not as simple as the type switch

I don't really understand how the plugin usecase would work exactly, but I can understand the need for it, since I am pretty sure I have needed this in the past.  I planned to do this with extensions instead of any, but go was not supporting plugins yet.

Trying to unmarshal each message in succession is also not an option, its slow and error prone, since protos are so backwards and forwards compatible.

We now have a resolver for Any https://github.com/golang/protobuf/pull/410 , which means there is one less reason to Register in the first place :)

I'm in the process of thinking over the Go API for protobufs. I'm noting this here for future reference. A mechanism for registration makes the assumption that there is a one-to-one mapping between Go types and protobuf message types.
However, my observation is that this not always true. The following proto.Message types exist. Some internally to Google, and some externally. But the general idea of each exists in both places in some form:

There are generated protos, for which we commonly assume a one-to-one relation between Go types and protobuf message types.
There are raw message types (similar to json.RawMessage), which are effectively []byte containing the encoded the form of a proto message. By their very nature, they are a single Go type that can represent every possible protobuf message type.
There are also dynamic message types, which are proto.Message implementations that can dynamically represent any proto message type at runtime based on some proto description generated by the user. As such, they are also a single Go type that can represent every possible protobuf message type.
The existence of the previous three implies that there are more than one Go type that can represent a protobuf message type.

This seems to be evidence for a n-to-m relationship between Go types and protobuf message types, further indicating that registration needs to be rethought. The discussion that has occured thus far in this issue is also valuable.
I have absolutely not fully thought through what would happen if we removed registration.

If I were to redo protobufs for go, which I would love to do, if I was given the time to do it.
I would love to try and think if I could remove all dependencies on the proto library.
If I couldn't do that I would try to inject Unmarshal with the Any and Extensions that could be used in unmarshaling and eliminate reasons for registering.

Proto.Actor https://github.com/AsynkronIT/protoactor-go is heavily dependent on registered types.
We do not use Any as that is too verbose for us, instead we do smart batching of messages and optimize our own lookup of what types are used in each "batch".
e.g. if there are 50 messages of the same type being sent, the type name will only occur once in the batch lookup.
Redesigning the lookup / type information too much would break our project.
(and in extension Oracle Fn Project)

I think that we have run into this, but not 100% sure it's relevant. In our case, we are wrapping concrete events within a generic event via Any. We tried to use DynamicAny, but we are missing something. Here's both:
package util

import (
	"reflect"

	"github.com/golang/protobuf/proto"
	"github.com/golang/protobuf/ptypes"
	"github.com/golang/protobuf/ptypes/any"
)

// UnmarshalPayload uses the internal proto.protoTypes registry to instance a concrete type
func UnmarshalPayload(payload *any.Any) (interface{}, error) {
	// Get the message name which is compatible with the golang proto.protoType registry
	payloadMessageName, err := ptypes.AnyMessageName(payload)
	if err != nil {
		return nil, err
	}

	// Get the type from the registry
	payloadMessageType := proto.MessageType(payloadMessageName)

	// .Elem() gives us the value of the struct type instead of a pointer
	// In other words: thing := MyStruct{}, without Elem() it would be: var thing *MyStruct
	// .Interface() is just so we can type assert when passing into UnmarshalAny
	payloadInstance := reflect.New(payloadMessageType.Elem()).Interface()

	err = ptypes.UnmarshalAny(payload, payloadInstance.(proto.Message))
	if err != nil {
		return nil, err
	}

	return payloadInstance, nil
}

// This should be what we use, but seems broken with errors like:
// mismatched message type: got "monolith.Event" want ""
func unmarshalPayload(payload *any.Any) (*ptypes.DynamicAny, error) {
	var result ptypes.DynamicAny
	err := ptypes.UnmarshalAny(payload, result)
	if err != nil {
		return nil, err
	}

	return &result, nil
}

We discussed the .proto file name piece @jhump brought up in https://github.com/grpc/grpc/issues/19832. The long and short of it is that protos should have a canonical name, otherwise things break in general. https://github.com/protocolbuffers/protobuf/issues/6492 is tracking the issue in protobuf to make the situation more clear to users and to potentially detect when things are awry. It also references an issue where C++ breaks when imports are non-canonical.

@ejona86 I agree. The v2 implementation will be moving towards a world where the set of protobuf declarations (i.e., enums, messages, extensions, etc) linked into a binary must be unique. The current situation where half the register functions fail loudly, and the other half silently fail is not okay.

I guess this is not a problem in Java because the descriptors are loaded (e.g. cross-linked with their dependencies) in static initializers, so if there is a problem linking descriptors, the message classes cannot even be loaded. Maven and gradle plugins for protos also make this much less error-prone than how other repos (such as Go repos) tend to handle managing their proto dependencies.
So I guess this will largely be "fixed" by v2 of the go proto APIs since it looks like they also construct and link descriptors during an init() function. Basically, the runtime will perform the necessary validation during initialization (so things like server reflection will be correct because invalid registrations would have caused the program to fail to start).
However, be prepared for the blast radius once v2 is released/GA: there's a good chance it will be a source of new issues filed and an inhibitor to projects upgrading to v2 because they may have dependencies that have unlinkable descriptors. They've basically been able to "get away with it" up to now because there was nothing in package initialization that was checking the validity of the compiled descriptors.

However, be prepared for the blast radius once v2 is released/GA: there's a good chance it will be a source of new issues filed and an inhibitor to projects upgrading to v2 because they may have dependencies that have unlinkable descriptors.

I agree. For that reason, we'll be easing users into it, but the end goal is to enforce strict unique-ness in the protobuf namespace.
Today (in v1), the behavior is that two of the register functions panic on conflicts, two of them log warnings to stderr on conflicts, and two of them silently allow the conflict. This situation is inconsistent and confusing.
For the initial release of v2, all registration functions will log warnings to stderr on any conflicts. The warning will 1) warn users that a future release will panic on conflicts, and 2) point users to documentation on how to resolve it.
For a later minor release of v2, all registration functions will panic on any conflicts. We may provide an opt-in to go back to the behavior of making them warnings, but that opt-out will not be around forever.

While developing the new protobuf implementation, we considered removing global registration entirely (the Java approach), but decided that it wasn't feasible if we want to a straightforward way for existing code to migrate. As such, we are double-downing on the concept of a global registry. However, almost every top-level function that consults the global registry can have a custom resolver passed in. The v1.20.0 release implements these details.

authors: HakShak awalterschulze bcmills dsnet ejona86 jhump rogeralsing

repo: golang/protobuf


text:
proposal: unwrapped request response

Currently protoc-gen-go generates interfaces that hide the method arguments and responses. It would be clearer if the request/response types were "unwrapped".
For example:
service Database {
  rpc Put(PutRequest) returns (EmptyResponse);
  rpc Get(GetRequest) returns (GetResponse);
}

message PutRequest {
  string path = 1;
  bytes  data = 2;
}

message EmptyResponse {}

message GetRequest {
  string path = 1;
}

message GetResponse {
  bytes  data = 1;
}

It currently generates interfaces:
func (Database) Put(ctx context.Context, req *PutRequest) (*EmptyResponse, error) { ... }
func (Database) Get(ctx context.Context, req *GetRequest) (*GetResponse, error) { ... }

However it would look nicer and clearer if the request/response could be implemented as:
func (Database) Put(ctx context.Context, path string, data []byte) error { ... }
func (Database) Get(ctx context.Context, path string) ([]byte, error) { ... }

I understand that it wouldn't work for all scenarios, however it would be nicer code for common cases.
First requested in https://github.com/gogo/protobuf/issues/517
Proposal
Version "Option". Based on a option instead of generating the "wrapped" interfaces, generate the "unwrapped" versions.
Version "Wrapped". Based on a option additionally generate a Register that takes the unwrapped service implementation as an argument.
Alternatives
These wrappers can be written manually, which adds overhead.
Separate wrapper generator could be used on top of protobuf, which would be somewhat annoying to use.

The only generation we do for services is for gRPC. If you have suggestions for changes to the gRPC generator, they should be filed here:
https://github.com/grpc/grpc-go/issues
(For historical reasons, the gRPC service generator currently lives in this repository and is linked into the protoc-gen-go command, but it's separate from general-purpose protobufs.)

Yeah, the gRPC generator really should be in the gRPC repo. I have hopes of fixing that at some point, but that's a ways off.

authors: egonelbre neild


text:
Fix typos in comments, tests and function names

Golang code usually used marshal instead of marshall.

@googlebot  I signed it!

Thank you

https://en.wikipedia.org/wiki/Marshalling_(computer_science)
https://en.wikipedia.org/wiki/Marshalling (double L) seems to be a correct spelling of the term, sorry.

@tandr https://github.com/golang/go/wiki/Spelling

source: gharchive/pull-request

authors: alexandear dsnet tandr


text:
tour: undefined: math.pi

Context: https://tour.golang.org/basics/3
Running the default code gives the following return:
prog.go:9:14: cannot refer to unexported name math.pi
prog.go:9:14: undefined: math.pi

Please read the text in the left part of the window, that is the point of the tutorial.
To fix the error, rename math.pi to math.Pi and try it again.
Lower case means the property is private, uppercase is public.

Working as expected and documented. Closing.

authors: ALTree TaurusD bgadrian

repo: golang/tour


text:
spanish usage

hi there
is it possible to use RASA in spanish? with the MITIE model in spanish?
if so, could you please point me some resource to do all changes?
thanks
angelo

the spanish MITIE models are here , if you unzip them and find the feature extractor file you should use that as your mitie_file. If you find that the tokenizer isn't working perfectly for spanish we can address that.

I just Download that model and place all that infor in the config file, however I am getting this error:
would you please point me out how to fix it?
thanks

creangel@creangel_hadoop:~/Downloads/mitie/rasa_nlu$ time python -m rasa_nlu.train -c config.json
Training to recognize 4 categories: 'saludo', 'restaurante_busqueda', 'afirmacion', 'despedida'
Train classifier
extracting text features
now do training
num training samples: 63
C: 200   f-score: 0.709677
C: 400   f-score: 0.709677
C: 300   f-score: 0.709677
C: 100   f-score: 0.709677
C: 0.01   f-score: 0.612903
C: 600   f-score: 0.709677
C: 1400   f-score: 0.709677
C: 3000   f-score: 0.709677
C: 5000   f-score: 0.709677
C: 2550   f-score: 0.709677
C: 1325   f-score: 0.709677
C: 712.5   f-score: 0.709677
C: 406.25   f-score: 0.709677
C: 253.125   f-score: 0.709677
C: 176.562   f-score: 0.709677
C: 138.281   f-score: 0.709677
C: 119.141   f-score: 0.709677
C: 109.57   f-score: 0.709677
C: 104.785   f-score: 0.709677
C: 102.393   f-score: 0.709677
C: 101.196   f-score: 0.709677
C: 100.598   f-score: 0.709677
C: 100.299   f-score: 0.709677
best C: 100.598
test on train:
20  0  0  0
0  8  0  0
0  0 21  0
0  0  0 14

overall accuracy: 1
Training time: 429 seconds.
df.number_of_classes(): 4
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "build/bdist.linux-x86_64/egg/rasa_nlu/train.py", line 65, in
File "build/bdist.linux-x86_64/egg/rasa_nlu/train.py", line 59, in do_train
File "build/bdist.linux-x86_64/egg/rasa_nlu/trainers/mitie_trainer.py", line 25, in train
File "build/bdist.linux-x86_64/egg/rasa_nlu/trainers/mitie_trainer.py", line 42, in train_entity_extractor
File "build/bdist.linux-x86_64/egg/rasa_nlu/trainers/mitie_trainer.py", line 31, in start_and_end
IndexError: list index out of range

looks like there's an error picking up one of your entities. I can't tell if this is a bug or a problem with your data without seeing it.
Please try training intents only (e.g. removing any entities from your training data), and then add them back one by one until you trigger this error. Then please post here the training example which causes the error.

hi there
I just try your solutions and work like a charm, i figure out my mistake is that start counting sentences from 1 instead of 0.
now is fix it.
thanks

I have the same problem @angelo337 had.. IndexError: list index out of range
I am using the expressions.json file from wit.ai
is there a problem with training wit data??
expressions.json.zip

thanks for sharing your training data! I'm able to reproduce this error. It's down to the fact that you have entities like 'perth' in the sentence "what is perths weather like next week". MITIE can only handle entities made up of whole tokens. I will handle this edge case in rasa, but it will still return "perths" rather than "perth" as your location. So for now you will have to resolve that entity yourself. It's on the roadmap to come up with a solution to that, though.

although thinking about it we could explicitly insert a whitespace in these cases. I will create a new issue & make a proposal

Hello
I would like to use rasa por spanish texts.
I already download the spanish Mitie model and prepared the config file.
During training, I get the following error:
python -m rasa_nlu.train -c config.json
Training to recognize 8 categories: 'greet', 'restaurant_search', 'affirm', 'goodbye', 'saludo', 'busqueda_restaurante', 'afirmacion', 'despedida'
Train classifier
extracting text features
now do training
num training samples: 44
C: 200   f-score: 0.525
C: 400   f-score: 0.525
C: 300   f-score: 0.525
C: 100   f-score: 0.525
C: 0.01   f-score: 0.575
C: 50.005   f-score: 0.525
C: 25.0075   f-score: 0.525
C: 12.5088   f-score: 0.525
C: 6.25938   f-score: 0.525
C: 3.13469   f-score: 0.525
C: 1.57234   f-score: 0.525
C: 0.791172   f-score: 0.525
C: 0.400586   f-score: 0.525
best C: 0.01
test on train:
5 0 0 0 0 0 0 0
0 8 0 0 0 0 0 0
0 0 6 0 0 0 1 0
0 0 0 5 0 0 0 0
1 0 0 0 1 0 0 0
0 0 0 0 0 8 0 0
0 0 0 0 0 0 5 0
0 0 0 1 0 0 0 3
overall accuracy: 0.931818
Training time: 854 seconds.
df.number_of_classes(): 8
Traceback (most recent call last):
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/site-packages/rasa_nlu/train.py", line 65, in
do_train(config)
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/site-packages/rasa_nlu/train.py", line 59, in do_train
trainer.train(training_data)
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/site-packages/rasa_nlu/trainers/mitie_trainer.py", line 30, in train
self.entity_extractor = self.train_entity_extractor(data.entity_examples)
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/site-packages/rasa_nlu/trainers/mitie_trainer.py", line 53, in train_entity_extractor
start, end = self.find_entity(ent, text)
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/site-packages/rasa_nlu/trainers/mitie_trainer.py", line 35, in find_entity
tokens, offsets = tk.tokenize_with_offsets(text)
File "/home/lisettegarcia/miniconda3/envs/py2/lib/python2.7/site-packages/rasa_nlu/tokenizers/mitie_tokenizer.py", line 24, in tokenize_with_offsets
offset += m.start()
AttributeError: 'NoneType' object has no attribute 'start'
Tracing the error, I found the problem in
(mitie_tokenizer.py)
line 22 m = re.search(re.escape(tok), _text[offset:])
when we work with words with accents.
Any idea ?
Thanks
busq_restaurante_Data.json.zip

I have the same problem than @beeva-lisettegarcia when training with spanish accents. The problem appears to be in the mitie_tokenizer.py script. Any idea or clue to fix it? Thanks!

@beeva-lisettegarcia @frankai I just pushed a change that should fix the encoding issue (unfortunately the test that should have ensured this functionality had a bug on its own 😓 ). Would be great if you could test that to see if it solves your issue.
For the future: Please avoid re-using closed issues. Don't hesitate to create new issues. The only thing you should do is the following: make sure the exact problem is not already an existing issue.

I had the same issue as @beeva-lisettegarcia @frankai, now pulled the changes and it is working.
Thanks

Thanks, Now it is working :-)

authors: amn41 angelo337 beeva-lisettegarcia cbonadio frankai oziee tmbo


text:
Non root-relative URLs

My app is currently hosted in a subfolder of the domain and spent some time debugging why all the urls being fetched by sw-precache were root-relative (ie. /app.js) instead of relative (ie. app.js). I don't believe there is an issue but I'm posting this in case it helps anyone else. Feel free to close.
The root of the issue is that sw-precache-webpack-plugin will use the webpack output path as the stripPrefix param to sw-precache. My webpack config looked something like this
{
  output: {
    filename: 'app.js',
    path: path.resolve(__dirname, 'dist', env)
  },
  plugins: [
    new SWPrecacheWebpackPlugin({
      cacheId: require('./package.json').name
    })
  ]
}

The fix is to set stripPrefix to replace the output path and the leading '/' manually
{
  output: {
    filename: 'app.js',
    path: path.resolve(__dirname, 'dist', env)
  },
  plugins: [
    new SWPrecacheWebpackPlugin({
      cacheId: require('./package.json').name,
      stripPrefix: path.resolve(__dirname, 'dist', env) + '/',
      replacePrefix: ''
    })
  ]
}

@jschr good find, I think it would be better to default to relative paths.

agreed that it might be a better default but would you still be able to allow for root-relative urls (with a proceeding /)?
The source of most of my confusion was just not being aware that output.path was being used automatically as the stripPrefix although it did make sense once I realized. The first thing I tried was
new SWPrecacheWebpackPlugin({
  cacheId: require('./package.json').name,
  stripPrefix: '/'
})

Which results in full file path urls without the proceeding / ie. Users/myuser/myapp/dist/development/app.js. Once I realized output.path was being used as the default stripPrefix I was able to figure out a solution.
Perhaps another solution is to append whatever stripPrefix is provided with the plugin config to the default output.path instead of overriding.

@jschr if we remove the / prefix by default you can can allow for root-relative urls again by specifying the replacePrefix: '/' parameter:
    new SWPrecacheWebpackPlugin({
      cacheId: require('./package.json').name,
      replacePrefix: '/',
    }),

I will update docs about the output.path usage so its clearer.
A good idea but I don't want to prevent user's from skipping the plugins functions and passing their own options into sw-precache.

Updated in 0.3.0 :smile:

authors: goldhand jschr

repo: goldhand/sw-precache-webpack-plugin

license: MIT


text:
Retry a few times when exporting the image in case of timeout

Sometimes we can see timeouts when exporting the image. This is because the Docker daemon can be busy, especially when we execute a lot of builds in parallel. We need to gracefully handle it.

Fixed in 5cde851b173bdbada98675333ee0144d5c7e4cdd.

repo: goldmann/docker-scripts

license: mit


text:
updated doc/Makefile.am to remove dependencies on achbench.html and o…

…ther man2html files

The build is still failing though due to some issues in ach_klinux.c in the 'chan-lock' function:
DKMS make.log for ach-klinux-2.0.0 for kernel 3.19.0-28-generic (x86_64)
Tue Sep 15 03:27:49 UTC 2015
make[4]: Entering directory `/usr/src/linux-headers-3.19.0-28-generic'
LD      /var/lib/dkms/ach-klinux/2.0.0/build/built-in.o
CC [M]  /var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.o
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c: In function ‘chan_lock’:
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:125:2: error: too many arguments to function ‘rt_mutex_lock_interruptible’
int i = rt_mutex_lock_interruptible(&chan->shm->sync.mutex, 0);
^
In file included from /var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:55:0:
include/linux/rtmutex.h:93:12: note: declared here
extern int rt_mutex_lock_interruptible(struct rt_mutex *lock);
            ^
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c: In function ‘ach_ch_close’:
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:490:2: error: too many arguments to function ‘rt_mutex_lock_interruptible’
if (rt_mutex_lock_interruptible(&ctrl_data.lock, 1)) {
^
In file included from /var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:55:0:
include/linux/rtmutex.h:93:12: note: declared here
extern int rt_mutex_lock_interruptible(struct rt_mutex *lock);
            ^
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c: In function ‘ach_ch_open’:
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:511:2: error: too many arguments to function ‘rt_mutex_lock_interruptible’
if (rt_mutex_lock_interruptible(&ctrl_data.lock, 1)) {
^
In file included from /var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:55:0:
include/linux/rtmutex.h:93:12: note: declared here
extern int rt_mutex_lock_interruptible(struct rt_mutex *lock);
            ^
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c: In function ‘ach_ch_ioctl’:
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:584:4: error: too many arguments to function ‘rt_mutex_lock_interruptible’
    if (rt_mutex_lock_interruptible(&ch_file->shm->sync.mutex, 0)) {
    ^
In file included from /var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:55:0:
include/linux/rtmutex.h:93:12: note: declared here
extern int rt_mutex_lock_interruptible(struct rt_mutex *lock);
            ^
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c: In function ‘ach_ctrl_ioctl’:
/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:910:2: error: too many arguments to function ‘rt_mutex_lock_interruptible’
if (rt_mutex_lock_interruptible(&ctrl_data.lock, 1)) {
^

In file included from /var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.c:55:0:
include/linux/rtmutex.h:93:12: note: declared here
extern int rt_mutex_lock_interruptible(struct rt_mutex *lock);
            ^
make[5]: *** [/var/lib/dkms/ach-klinux/2.0.0/build/ach_klinux.o] Error 1
make[4]: *** [_module_/var/lib/dkms/ach-klinux/2.0.0/build] Error 2
make[4]: Leaving directory `/usr/src/linux-headers-3.19.0-28-generic'

Any ideas what I should do to make this work?

Please check if the kernel module changes in ach branch linux-3.18.16-rt compile against your kernel.  Unfortunately, the Linux kernel API changes frequently, so it may be necessary to track multiple API versions.  Perhaps this deserves a mention in the README and/or manual.
Unconditionally disabling man page generation is not a maintainable option because the documentation does sometimes need to be built.  What is the specific error you are still seeing regarding manpage generation?  Did the previous commit and/or configure --disable-documentation not resolve your issue?

94022f78fab4ee697ae349ed4d54fe254baed9f2
I missed the HTML manpage in the earlier commit.  Now fixed.

Nope, even in the linux-3.18.16-rt branch the build is failing with the following error:
Making install in src/klinux/dkms-post
make[2]: Entering directory `/home/varun/golems/ach/src/klinux/dkms-post'
make[3]: Entering directory `/home/varun/golems/ach/src/klinux/dkms-post'
make[3]: Nothing to be done for `install-exec-am'.
if test -n "`dkms status ach-klinux/2.0.0`"; then \
                dkms remove ach-klinux/2.0.0 --all; \
        fi
/bin/bash: dkms: command not found
dkms add /usr/local/src/ach-klinux-2.0.0
/bin/bash: dkms: command not found
make[3]: *** [install-data-local] Error 127
make[3]: Leaving directory `/home/varun/golems/ach/src/klinux/dkms-post'
make[2]: *** [install-am] Error 2
make[2]: Leaving directory `/home/varun/golems/ach/src/klinux/dkms-post'
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory `/home/varun/golems/ach'
make: *** [install] Error 2

I tried configuring the build with
./configure  --enable-dkms=yes --enable-dkms-build=yes

Looks like you need to install dkms.
-ntd
On 09/15/2015 05:12 PM, Varun Agrawal wrote:

Nope, even in the linux-3.18.16-rt branch the build is failing with the
following error:
|Making install in src/klinux/dkms-post
make[2]: Entering directory /home/varun/golems/ach/src/klinux/dkms-post' make[3]: Entering directory /home/varun/golems/ach/src/klinux/dkms-post'
make[3]: Nothing to be done for install-exec-am'. if test -n "dkms status ach-klinux/2.0.0"; then \                  dkms remove ach-klinux/2.0.0 --all; \          fi /bin/bash: dkms: command not found dkms add /usr/local/src/ach-klinux-2.0.0 /bin/bash: dkms: command not found make[3]: *** [install-data-local] Error 127 make[3]: Leaving directory /home/varun/golems/ach/src/klinux/dkms-post'
make[2]: *** [install-am] Error 2
make[2]: Leaving directory /home/varun/golems/ach/src/klinux/dkms-post' make[1]: *** [install-recursive] Error 1 make[1]: Leaving directory /home/varun/golems/ach'
make: *** [install] Error 2
|
—
Reply to this email directly or view it on GitHub
https://github.com/golems/ach/pull/8#issuecomment-140561252.

Closing with 94022f7

authors: ndantam varunagrawal

repo: golems/ach


text:
Which Model I can use ? (Pytorch, Tensorflow , ONNX)

Could you please clarify for me which Model type i can use for GOMLX , Currently I have Pytorch model , also I can convert it to ONNX model and even if I make some effort probably I can use Tensorflow As well.
So if it possible give me the code Example how can I use it.

Sure. So, GoMLX is meant to be a ML framework where you can build your own machine learning models from scratch. The tutorial does that in a very simple model, but the examples/ subdirectory has many models built and trained from scratch -- I recommend taking a look at UCI Adult.
Now, if you want to use GoMLX simply as a backend to execute an existing model, that is, converting a model from another platform, there is no automated way yet. Usually, it's not hard to manually convert a model, if you understand the code (I converted a the Inceptions models from TensorFlow, and the Gemma model from Jax. Under the gomlx/ml/data directory there are some file format conversions/donwload.
In parallel, we (with some collaborators) are working right now on something that will convert an ONNX model and build a GoMLX model, see example here. There are two use cases for that:

Use GoMLX XLA engine (as opposed to onnxruntime for ONNX) which may be faster in some cases (and TPU support), we have yet to benchmark.
Load an ONNX model in GoMLX and use GoMLX to fine-tune it (it does the auto-differentiation) without requiring python+pytorch, and even export the weights back to the ONNX model. This should be coming the next few days.

Thank you so much for prompt response.
I have tried run this code:
`package main
import (
"fmt"
"github.com/janpfeifer/must"

"github.com/gomlx/gomlx/backends"
. "github.com/gomlx/gomlx/graph"
"github.com/gomlx/gomlx/ml/data"
"github.com/gomlx/onnx-gomlx/onnx"

_ "github.com/gomlx/gomlx/backends/xla"

)
var (
_       = Add
backend = backends.New()
)
func main() {
var modelPath = data.ReplaceTildeInDir("category_model.onnx") // all-MiniLM-L6-v2
model := must.M1(onnx.ReadFile(modelPath))
fmt.Printf("%s\n", model)
}
`
But i got this error:
`# github.com/gomlx/onnx-gomlx/onnx
../go/pkg/mod/github.com/gomlx/onnx-gomlx@v0.0.0-20241029113425-03b8b422d41a/onnx/graph.go:247:9: undefined: MatMul
../go/pkg/mod/github.com/gomlx/onnx-gomlx@v0.0.0-20241029113425-03b8b422d41a/onnx/materialize.go:19:15: node.ConstantValue undefined (type *graph.Node has no field or method ConstantValue)
../go/pkg/mod/github.com/gomlx/onnx-gomlx@v0.0.0-20241029113425-03b8b422d41a/onnx/ops.go:140:9: undefined: ExpandAxes
../go/pkg/mod/github.com/gomlx/onnx-gomlx@v0.0.0-20241029113425-03b8b422d41a/onnx/ops.go:170:65: node.IsConstantExpression undefined (type *graph.Node has no field or method IsConstantExpression)
github.com/gomlx/gopjrt/xlabuilder
../go/pkg/mod/github.com/gomlx/gopjrt@v0.4.4/xlabuilder/literal.go:4:10: fatal error: 'gomlx/xlabuilder/literal.h' file not found
4 | #include <gomlx/xlabuilder/literal.h>
|          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.`
Note: I use model that has LSTM layer

hey @akshin18 ,
While the LSTM layer should be fine to be implemented (or converted to) GoMLX ,  the onnx-gomlx repository is not finished yet -- and it is being co-developed with a new release of GoMLX (also in the works) for some things it needs.
You can see how the example is broken for the sentence encoder -- not all Ops conversions are done yet.
I was going to suggest to wait for the ONNX converter ... I'm hopeful that by the end of the week or early next week we'll have something that works for most models. I'll keep updating the repository and I'll add the documentation accordingly. Also GoMLX will have a new release.
Btw, you need to install gopjrt manually: it requires a C/C++ library (it uses CGO) installation to talk to XLA/PJRT (the underlying ML engine for accelearators developed by Google). There is a straight forward script to do that. Hopefully, at some point I'll package it in Homebrew or Snap.
Let me know if you have any questions.

Just an update: https://github.com/gomlx/onnx-gomlx is in a working state already. A sentence encoder has already been fully converted and all seems to work.
Probably there are Ops that are not converted yet, but those are easy. If you try to convert anything that is missing an Op let me know, I'd be happy to fix it.
Currently I'm working on  https://github.com/gomlx/go-huggingface to try to streamline the import of HuggingFace models -- at least those that have an ONNX version.  It's slowly but surely improving :smiley:

Hi @janpfeifer ,
Thank you so much for letting me know , I appreciate  that , will try

I have trie to use next code to try use my model  but it does not work , probably I do not have enough knowledge to understand how to use this library for my model:
package main
// This example illustrates how to use pre-trained vision models.
// model to get the imagenet label for some image.
import (
"fmt"
"os"
"github.com/gomlx/gomlx/backends"
. "github.com/gomlx/gomlx/graph"
"github.com/gomlx/gomlx/ml/context"
"github.com/gomlx/gomlx/ml/data"
"github.com/gomlx/onnx-gomlx/onnx"
"github.com/janpfeifer/must"

_ "github.com/gomlx/gomlx/backends/xla"

)
var (
_       = Add
backend = backends.New()
)
func main() {
var modelPath = data.ReplaceTildeInDir("category_model.onnx") // all-MiniLM-L6-v2
model := must.M1(onnx.ReadFile(modelPath))
fmt.Printf("%s\n", model)
must.M(model.PrintVariables(os.Stdout))
fmt.Println()
must.M(model.PrintGraph(os.Stdout))
fmt.Println()

// Convert ONNX variables to GoMLX context (which stores variables):
ctx := context.New()
must.M(model.VariablesToContext(ctx))
fmt.Printf("Variables loaded from %q:\n", modelPath)

gomlxFn := func(ctx *context.Context, x *Node) *Node {
	return model.CallGraph(ctx, x.Graph(), map[string]*Node{"X": x})[0]
}
fmt.Println(gomlxFn)
x := [][]float32{{101, 2070, 2518, 102}}
fmt.Printf("\tX*A+B=%v\n", context.ExecOnce(backend, ctx, gomlxFn, x).GoStr())

}

Also i tried this code and got that kind of issue :
panic: ONNX node output "" not found as the output of any Op, and not a variable or input either -- could it be a node name, and note a node output name ?
My model:
ONNX Model:
Producer:       pytorch / 2.5.0
# inputs:       1
[#0] input: (Int64) [batch_size, 4]
# outputs:      1
[#0] output: (Float32) [batch_size, 10]
# nodes:        13
# tensors (variables):  6
# sparse tensors (variables):   0
Op types:       []string{"Concat", "Constant", "ConstantOfShape", "Gather", "Gemm", "LSTM", "Shape", "Squeeze", "Transpose", "Unsqueeze"}
IR Version:     6
Operator Sets:  [v11]
   code:
   ```
   package main

import (
"fmt"
"os"
"github.com/janpfeifer/must"

"github.com/gomlx/gomlx/backends"
_ "github.com/gomlx/gomlx/backends/xla"
. "github.com/gomlx/gomlx/graph"
"github.com/gomlx/gomlx/ml/context"
"github.com/gomlx/onnx-gomlx/onnx"

)
func main() {
// Replace with the path to your LSTM ONNX model
modelPath := "lstm_model.onnx"
// Load and parse the ONNX model
model := must.M1(onnx.ReadFile(modelPath))
fmt.Printf("%s\n", model)

// Optionally, print variables and graph for debugging
must.M(model.PrintVariables(os.Stdout))
fmt.Println()
must.M(model.PrintGraph(os.Stdout))
fmt.Println()

// Create a new GoMLX context and load model variables into it
ctx := context.New()
must.M(model.VariablesToContext(ctx))

// Prepare input data
// Assuming batch_size = 2, sequence_length = 5
inputData := [][]int64{
	{1, 2, 3, 4},
}

ou := [][]float32{
	{1, 2, 3, 4},
}

// Create a backend (defaults to XLA)
backend := backends.New()

// Define the computation function
gomlxFn := func(ctx *context.Context, x *Node, y *Node) *Node {
	// Map input names to input nodes
	inputsMap := map[string]*Node{
		"input": x,
	}
	// Convert the ONNX model to a GoMLX computation graph
	outputNodes := model.CallGraph(ctx, x.Graph(), inputsMap)
	// Return the first output node (assuming single output)
	return outputNodes[0]
}

// Execute the model using the GoMLX context and backend
outputTensor := context.ExecOnce(backend, ctx, gomlxFn, inputData,ou)

// Print the output tensor
fmt.Printf("Model output: %s\n", outputTensor)

}
   ```

hi @akshin18 ,
Interesting, thank you for reporting, it helps a lot.
Just yesterday, I manged to get my third model to work -- see the very last cell in my notebook here.
Also I created github.com/gomlx/go-huggingface to download&cache model files from HuggingFace, to make it simpler. Also, I eventually want to add tokenizers support for those doing LLMs.
And you pointed out a couple of new things I need to add support to:

Unnamed inputs and outputs -- I hadn't notice that ... I'll have to think of something.
From the Ops listed, I can already say I haven't yet converted the LSTM op.

Is this a model available in HuggingFace that I can try to make it work ? Tomorrow I'll get back to try to expand its support, and it would be nice to have some target models :)
cheers

Hi @janpfeifer ,
Thank you for your reply.
Unfortunately I can not share the model that I use cause it belongs to company, however you can use as a reference this mode: https://www.kaggle.com/code/mehmetlaudatekman/lstm-text-classification-pytorch/notebook
Please do not hesitate to letting me know If i can somehow help you.
Thanks,
Agshin

Perfect, I'll use that as model! ETA for having something: mid-next week.

hi Agshin,
I just added support for another couple of ONNX models in the list, and I was going to take a look at the LSTM model you pointed to, but there doesn't seem to be a .onnx model file I can download in the Kaggle page. Do you have a version of the model not trained (with random initialization) that you could share, so I can use it as test ?
cheers

hi @janpfeifer ,
I do not have exact file , You can use as example this class:
class TextClassificationModel(nn.Module):
def init(self, vocab_size, embedding_dim, hidden_dim, output_size):
super(TextClassificationModel, self).init()
self.embedding = nn.Embedding(vocab_size, embedding_dim)
self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)
self.fc = nn.Linear(hidden_dim, output_size)
def forward(self, x):
    x = self.embedding(x)
    _, (hn, _) = self.lstm(x)
    hn = hn.squeeze(0)
    out = self.fc(hn)
    return out

Thanks @akshin18 , but how do you save that to an ONNX file ?

I use that code:`
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
Convert to ONNX
dummy_input = torch.randint(0, 30522, (1, 128)).to(device)  # Dummy input tensor
onnx_file_path = "test_.onnx"
torch.onnx.export(
model,
dummy_input,
onnx_file_path,
input_names=["input"],
output_names=["output"],
dynamic_axes={"input": {0: "batch_size", 1: "sequence_length"}, "output": {0: "batch_size"}},
opset_version=11
)`

Thanks again, I'll take a look at it!

hi @akshin18 ,
Just an update: I created the LSTM model, and to convert I had to create an LSTM layer in GoMLX as well -- which is a good thing, I've been meaning to do it.
Most of the code is already there,  but I still need to add tests and stitch together in the converter. Hopefully, I'll get that by tomorrow or Monday if not.

LSTM seems to be working now. The new releases of GoMLX v0.15.2 and onnx-go v0.1.2 includes the update.
See example of model in python created here and the GoMLX converted version here: both seem to return the same result -- search for the section with LSTM in the title.

hi Akshin (@akshin18 ), I'm closing the issue, ok ? But let me know (open another issue or discussion) if you have any questions.
cheers

repo: gomlx/gomlx


text:
🛑 Properstar.ch search is down

In 1de3922, Properstar.ch search (https://www.properstar.ch/suisse/geneve/acheter/appartement-maison) was down:

HTTP code: 404
Response time: 499 ms

Resolved: Properstar.ch search is back up in 6a1199d.

repo: goncharov/powersite-uptime


text:
Hack Session 500 ✼ Remotely

01.12.2021
Wrap-up of our Gōng-fu I/O Weekly · Hack Session #500 ✼ Remotely ✼ meetup.
Projects
• ‹Bouboucle› generative art with Javascript — @andreaskundig and @david-hodgetts
• ‹Distributed Arcade› Game World & Boards with H3 & hex-coordinate systems — @olange and @rudifa
See also

#499 Previous session
#501 Next session

Study of @olange and @rudifa
Worked on our ‹h3-minimap› web component in ObservableHQ:

replaced our implementation of bbox() with d3.geoBounds(this.areasGeom) — to discover that d3-geo was producing the exact same results as our first implementation, which was simply looking for the mix/max ‹lat,lon› of all corners of our areas;
tried to draw a spherical bounding box, instead of a rectangular bounding box one, by using d3.geoCentroid(this.areasGeom) and a radius derived from the d3.geoBounds(…), but we struggled to understand what was the centroid that d3-geo gave us, which did not wrap nicely all of our areas.

repo: gongfuio/sessions


text:
ADT

Needs generics.
type Maybe a = Just a | Nothing
type Either a b = Left a | Right b
type Tree a = Node a | (Tree a, Tree a)

Bonus: monads
interface Monad a {
  func Bind<b>(Monad a, func(a) Monad b) Monad b
  func Return(a) Monad a
}

func Bind<a>(ma Maybe a, f func(a) Maybe b) Maybe b {
  r := match ma {
    case Just a: f(a)
    case Nothing: Nothing
  }
  return r
}

func Return<a>(x a) Maybe a {
  return Just x
}

requires #8

@quetz please fix generics syntax to match #8

authors: kirillDanshin quetz

repo: goodlang/good


text:
Where are internal pullup resistors being enabled?

Description
I am using the coral som with gpio 6,7, and 8, and at it seems as though the internal pullups are being activated at boot, but I cannot find where that is happening in the device tree.  My understanding is that the pulldown is enabled by default, and the pullup would have to be explicitly enabled with IOMUXC_SW_PAD_CTL in the device tree, but I do not see that anywhere.
Click to expand!
Issue Type
Others
Operating System
Linux
Coral Device
No response
Other Devices
No response
Programming Language
Other
Relevant Log Output
No response

I think, here is no control bit for pull down enable/disable for NXP IMX8M Soc. Please check with NXP community regarding this issue. Thanks!!
Ref: https://community.nxp.com/t5/i-MX-Processors/i-MX8M-processor-pins-pull-up-down-configuration/m-p/882224#M133754

Right.  This is what I was referencing also.  "There is internal 90K Ohm pull down resistor on the IOs to avoid excessive current through the resistive voltage"  this is enabled by default.  I'm asking about the pullup resistors not the pull down resistors.  "For the PUE bit field, it only controls the internal 27K Ohm pull up resistor enable/disable.  Customers can use IOMUXC registers, such as for example IOMUXC_SW_PAD_CTL_PAD_GPIO1_IO02 to define what pins have internal 27K Ohm pull up resistor enabled."  So it seems like the pullup is enabled, but I don't see any reference to IOMUXC_SW_PAD_CTL anywhere.

I am not sure, if it helps. I can find those references in im-atf folder in mendel source tree.https://coral.googlesource.com/imx-atf/+/refs/heads/master/plat/imx/common/include/imx_io_mux.h#214

We're getting somewhere.  If I understand correctly the OFFSET defines the location of the bits that would be set, but does not set them.

Could it be set somewhere in python-periphery?

Please check with NXP community if they have any suggestions on this issue. Thanks!

And how you know that internal pull-ups are enabled, and what is the problem if they are?

authors: easy-and-simple hjonnala synaption

repo: google-coral/edgetpu


text:
Update lift.py

Improved code readability and maintainability.
Renamed variable names to be more descriptive.
Applied PEP 8 coding style guidelines.
Added unit tests.

The author has sent 3 spam PRs on TF repo and one PR that got merged. They also sent 3 spam PRs to Keras.

Oops sorry, I will make sure to run them before sending PR, thank you


text:
Android Basics: Load and display images from the Internet

name: Android Basics Unit 4 issue
about: Report problems with the Android Basics Unit 4 codelabs
title: 'Load and display images from the Internet'
labels: ''
assignees: ''

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-training-internet-images?continue=https%3A%2F%2Fdeveloper.android.com%2Fcourses%2Fpathways%2Fandroid-basics-kotlin-unit-4-pathway-2%23codelab-https%3A%2F%2Fdeveloper.android.com%2Fcodelabs%2Fbasic-android-kotlin-training-internet-images#3
In which task and step of the codelab can this issue be found?
Step 4
Describe the problem
There's a typo error in first item of under the block Add Recyclerview. It's written layout/gridview_item.xml instead of layout/grid_view_item.xml
Also in step 3, under the block Add loading and error images, there's a trailing d at the end of the second item of the list. Bellow is the full sentence:

Open res/drawable/loading_animation.xml. This drawable is an animation that rotates an image drawable, loading_img.xml, around the center point. (You don't see the animation in the preview.) d

I hope it'd help
Steps to reproduce?

Go to...
Click on...
See error...

Versions
Android Studio version:
API version of the emulator:
Additional information
Include screenshots if they would be useful in clarifying the problem.

@PatrissolJuns
Thank you for reaching out to us. The typo was fixed and published:

authors: PatrissolJuns osuleymanova


text:
Error: Not a Matter-certified device

I have a Thread Matter device which is not yet certified. I'm able to commission it using:

a Google Nest Hub 2
an android phone with Google Home app
To do so, I have created a project on Google Home developper console. Google Home is not complaining anymore when I commission this non certified device and I'm able to control it.

I'm now trying to do the same with Google Home Sample App for Matter but I'm always getting the error "Not a Matter-certified device".
Can you please point me what I need to do to enable the commissioning of non certified devices? "Google Home Sample App for Matter" and "Google Home" are running on the same smartphone so with the same google account. So I was expecting that the project added on Google Home developper console would work also for GHSAM but that's not the case.
Thanks for your advices
Regards
Olivier

Yes, version 1.4.1 is the latest release.
It should work. Unfortunately your logs are not complete. Please do the following:
$ adb logcat -c
$ adb logcat > logs.txt
... then use GHSAFM normally
When the app fails, you can kill the above command and send us the logs.txt.

Thanks for your help. Here is a new log.
GHSAFM_logcat2.zip
I can see:
05-26 09:25:09.439 22401 22401 E SetupDeviceViewModel: Commissioning failed with state Attestation failure. [CONTEXT service_id=336 ]
05-26 09:25:09.439 22401 22401 E SetupDeviceViewModel: m.epr: Device did not pass attestation (error(s)=[5, 10])

Thanks for your help. Here is the log:
GHSAFM_logcat3.zip
I can see:
05-26 10:46:08.121  4126  4126 E SetupDeviceViewModel: Commissioning failed with state Attestation failure. [CONTEXT service_id=336 ]
05-26 10:46:08.121  4126  4126 E SetupDeviceViewModel: m.epr: Device did not pass attestation (error(s)=[5, 10])

GHSAFM uses the Device Attestation Delegate to properly handle issues with device attestion.
Looking at your logs, I see that the call is properly done:
05-26 10:44:56.880  4041  4041 D CTL     : setDeviceAttestationDelegate() called

Device Attestion should trigger the following callback, and debug messages should show up in the logs
https://github.com/google-home/sample-app-for-matter-android/blob/main/app/src/main/java/com/google/homesampleapp/screens/home/HomeFragment.kt#L436
          if (errorCode == STATUS_PAIRING_SUCCESS) {
            Timber.d("DeviceAttestationDelegate: Success on device attestation.")
            lifecycleScope.launch {
              chipClient.chipDeviceController.continueCommissioning(devicePtr, true)
            }
          } else {
            Timber.d("DeviceAttestationDelegate: Error on device attestation [$errorCode].")

However this is not triggered. Not sure why.
I'll have to try to reproduce.
In the meantime, could you make sure to do a complete reset of your testing environment.
Settings > Apps > GHSAFM > > Storage & cache > Clear storage

Factory reset the device and remove it from GHA if it is showing there.
Also, make sure it does not show up in the Local Android Fabric:
Settings > Google > Devices & sharing > Matter devices
Select "This Android device" in the drop down on the top right.
If any device is shown in "Linked Matter devices", remove them.

Hi Pierre,
Thank you very much for your help.
I have followed your instructions but I still get the same issue. I have put a breakpoint in the callback that you mention and indeed it never gets called.
FYI, I had nothing in "Linked Matter devices" and no devices was shown by Google Home. When I commisison a device with google Home, I remove it before switching to GHSAFM so that should be fine.
I'm attaching the log obtained with GHSAFM after doing the clean up that you recommended and a commissioning. I have put also the traces of the Matter device. This matter device is not certified but it is working when I commission it with Google Home.
traces.zip
Best regards
Olivier

I don't know if it will help but here are the traces (android phone and matter device) when doing a commissioning with GHA.
traces_GHA.zip

When using GHA, after scanning the QRCode, it asks to choose an App (I choose Google Home), then it asks if I want to connect this device to my Google account, then there is the message "Connecting device to Google Home...".
When using GHSAFM , I don't get such questions. After scanning the QRCode, it immediately display "Connecting device with GHSAFM ...".
Is it expected?

Hi Olivier,

When using GHSAFM , I don't get such questions. After scanning the QRCode, it immediately display "Connecting device with GHSAFM ...".
Is it expected?

Yes it is.
GHA commissions to the Google ecosystem and a Google account is required.
With GHSAFM, we use the GPS Commissioning APIs but it commissions first to a Local Android Fabric (not the Google ecosystem fabric), and then to GHSAFM's fabric (which is just a demo fabric for the purpose of our sample app). No Google account is needed.
I'll make some time this week to dig more into this. Hang in there :-)

Hi Pierre,

GHA commissions to the Google ecosystem and a Google account is required.
With GHSAFM, we use the GPS Commissioning APIs but it commissions first to a Local Android Fabric (not the Google ecosystem fabric), and then to GHSAFM's fabric (which is just a demo fabric for the purpose of our sample app). No Google account is needed.

This is here that something is not clear to me:
When using GHA, I have first created a project on Google Home developper console  and this is the reason why Google Home doesn't complain when I commission a non certified device.
When using GHSAFM, you say that no Google account is used so GHSAFM doesn't know that I want to use a developper mode and it looks normal that it refuses a non certified device. What did I miss? How do you configure GHSAFM to accept non certified devices?

I'll make some time this week to dig more into this. Hang in there :-)
Ok thanks.

When using GHA, I have first created a project on Google Home developper console and this is the reason why Google Home doesn't complain when I commission a non certified device.

Correct.

When using GHSAFM, you say that no Google account is used so GHSAFM doesn't know that I want to use a developper mode and it looks normal that it refuses a non certified device. What did I miss? How do you configure GHSAFM to accept non certified devices?

As I mentioned above,
"GHSAFM uses the Device Attestation Delegate to properly handle issues with device attestion."
It should work (as it's definitely been tested). Not sure what's going on, will have a look.

Hi Pierre,
Do you have some news regarding this issue?

Hi Pierre,
Thank you, I'm going to look at that.

Hi Pierre,
Today I was looking at this but I'm facing some problems.
When working with Matter, I use a Wifi hot spot (usually my mobile) and I have some various equipments (Google Nest Hub 2, Raspberry Pi4 + RCP, Thread devices, android phones...etc). I cannot use them on the company Wifi network.
I have thought that I could install MVD on the Raspberry pi4 (that would not be used as a border router in that case) but I see that the installation is done through a Debian Package for 64-bit x86 processor machines. So I'm afraid that it cannot be used on my Raspberry Pi which has an ARM processor.

Hi Pierre,
My Matter device is supporting Matter 1.0 and not Matter 1.1.
Is there a branch to take for GHSA for devices running 1.0?
Thank you

CSA certification is acting like a big barrier for small startups. I have developed a matter device but exhorbitant CSA certification and testing cost is really lot than my device manufacturing setup and investment.
Moreover, Google Home app is not allowing to add non-certified devices. Which again makes CSA certification mandatory.
Apple and Alexa allows non-certified devices to be added but Google Home is not. Developers can add their own devices but he cannot sell un-certified devices because it is not allowed in Google Home.

@sushantr5
Correct, the Google Home App (GHA) currently doesn't allow users to commission unattested Matter devices.
We have been working on a solution for developers and users to commission devices without Matter certification. We will share updates through our regular channels soon.

I hope this will happen soon.  I am waiting for it. Please update us here if possible timeline for it.

@pierredelisle I'm now able to commission a non certified deviced. In June it was not working and I know know that it was due to my device which was running Matter 1.0. Now that my device has moved to Matter 1.1, I'm able to commission it with GHSAM.
Regards

Thanks for the update Olivier. Glad to hear it's finally working!

@pierredelisle I'm now able to commission a non certified device. In June it was not working and I know now that it was due to my device which was running Matter 1.0. Now that my device has moved to Matter 1.1, I'm able to commission it with GHSAM. Regards

Is it because of Matter 1.1 or due to update in Google Home ?

It was due to my device which was running Matter 1.0. The FW has been updated to Matter 1.1 and now it works.
A side comment:
As mentioned in the first message, I had created a project on Google Home developper console in order to be able to commission a non certified device.

It was due to my device which was running Matter 1.0. The FW has been updated to Matter 1.1 and now it works.
A side comment: As mentioned in the first message, I had created a project on Google Home developper console in order to be able to commission a non certified device.

@OlivierGre Thank you for update. I am waiting for the day when I can commission TEST matter device in Google Home WITHOUT Developer Console Project. Amazon Alexa and Apple Homekit can do it.

authors: OlivierGre pierredelisle sushantr5


text:
Error message occured in common.js file

I got the below two errors, Could you help me to fix .

Deprecation Warning and Missing Refresh Token:
(node:92853) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead.
(Use node --trace-deprecation ... to show where the warning was created)
Error retrieving access token: Error: No refresh token or refresh handler callback is set.

TypeError: Cannot read properties of undefined (reading 'match'):
/home/sasithagamage/vigenair/dist/common.js:143
let webAppLink = lastNonEmptyLine.match(/- (.) @./);
^

TypeError: Cannot read properties of undefined (reading 'match')
at UiDeploymentHandler.deployUi (/home/sasithagamage/vigenair/dist/common.js:143:43)
at /home/sasithagamage/vigenair/dist/index.js:87:41
at Generator.next ()
at fulfilled (/home/sasithagamage/vigenair/dist/index.js:20:58)

I have updated the .clasp.json file with the following values
{
"token": {
"access_token": "your-new-access-token",
"refresh_token": "your-new-refresh-token",
"scope": "https://www.googleapis.com/auth/script.projects https://www.googleapis.com/auth/drive.metadata.readonly https://www.googleapis.com/auth/drive.file https://www.googleapis.com/auth/service.management https://www.googleapis.com/auth/logging.read https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/cloud-platform",
"token_type": "Bearer",
"expiry_date": 1629186341  // Use the calculated expiry date
},
"oauth2ClientSettings": {
"clientId": "your-client-id",
"clientSecret": "your-client-secret",
"redirectUri": "urn:ietf:wg:oauth:2.0:oob"
},
"isLocalCreds": true
}

There are several unnecessary scopes in there, and you shouldn't need to manually update your .clasp.json file.
Could you please share your node and npm versions?
Mine are node: v18.19.1 and npm: 10.2.4, and with those versions the issue is not reproducible.
The second "match" problem is a side-effect of the first, because your Apps Script project wasn't created or deployed, and so we couldn't "extract" the deployed Web App URL from the npm logs.
If you're going to upgrade node and npm please make sure to git reset --hard your workspace before rerunning npm start, and make sure you clean up any .*clasp*.json files in <root>/dist/, <root>/ui/dist/ and <root>/ui/ before your run npm start.

Okay Thanks, My node version and npm version are v22.5.1, 10.8.2

authors: SasithaGamage mohabfekry

repo: google-marketing-solutions/vigenair


text:
Different accuracy when predicting the dev set with do_predict

I train my dataset(the same format with CoLA dataset) for sentence classification model and I take the evaluation results in eval mode after training which is
eval_accuracy = 0.79638886
eval_loss = 1.4008523
global_step = 128711
loss = 1.4008523
Afterthat, I copied this dev.tsv into test.tsv by removing the labels and the other columns in order to construct the same format of test.csv of CoLa. After I took the prediction results, I compare the predictions with their own labels. When I calculate the accuracy ((TP + TN) / (TP + TN + FP + FN)), the accuracy is equal to 0.48138888888888887.
In brief, the accuracy scores are different when I predict the same dataset in eval mode and as test.csv predict by using do_predict option. Is it possible? Thanks for all comments.

I also got the similar issue.
I notice that in the run_classifier.py script, when read test data, the label also should be assigned. (eg. line 288, 328, 368). It really makes me confused.
Generally, when we do prediction or test, there is no labels should be given.
When training my own dataset, everything goes well. But, when the same dataset was used to do the test, following the run_classifier.py script, I also give a constant value to all the test data. The test results all are the given constant values.
@atifemreyuksel  do you have any progress？

Which data processor do you use? I am using CoLA data processor and I don't give the labels of the instances in test dataset. I construct test.tsv by giving index and sentence as columns.
In lines 288, 328, 368, the unnecessary label is assigned to test set, you are right. But this label is not used in evaluation when you want to take predictions by using --do_predict option.
I can take the right predictions of the same dataset in test.tsv by using the instruction of taking predictions. After you take the test_results.tsv, you can see the softmax results of each instances. You can find the maximum index of each row in order to reach the prediction of the model.

Hi @atifemreyuksel , thanks for your kind reminder. I got the errors, it is not the issue of the model. It is because when doing the test, I still use the unnecessary label attached in the InputFeatures to calculate the prediction accuracy. That is why I got a very low accuracy. Really a stupied mistake. ╮(╯▽╰)╭

@hainan89
i met the same problem。If the test data has no label, then program  interruption.
Code below:
if set_type == "test":
text_a = tokenization.convert_to_unicode(line[0])
examples.append(
InputExample(guid=guid, text_a=text_a, text_b=None,label=None))
The error is file_based_converter_to example return none

I had the same problem and figured out that, if you give separate output dir while only do eval or test, it doesn't find the model dir. Since;
run_config = tf.contrib.tpu.RunConfig(
cluster=tpu_cluster_resolver,
master=FLAGS.master,
model_dir=FLAGS.output_dir,
save_checkpoints_steps=FLAGS.save_checkpoints_steps,
tpu_config=tf.contrib.tpu.TPUConfig(
iterations_per_loop=FLAGS.iterations_per_loop,
num_shards=FLAGS.num_tpu_cores,
per_host_input_for_training=is_per_host))
above model_dir set to output_dir and if you give a separate dir (i.e. give a dir to output eval/test results), it does't produce same results.
So you either have it save eval/test to the same training/model folder or manually set the model_dir from above code.

authors: NancyLele atifemreyuksel hainan89 oakkas

repo: google-research/bert


text:
Accessing the duplicates and their counts

Hey, thanks for releasing the code!
I'm a bit confused regarding how to use the dups_ and sizes_ files.
I would like to get a mapping between all duplicate strings and their corresponding number of appearances in the data.
From my understanding, this is what you get with the points in those files, but I don't understand how to read these.
Any explanation, would be helpful! (and code snippet / reference even better!)
Thanks

Did you follow the README example? I tried to explain what's going on there with some code that shows processing it. Maybe if you could say what specifically is most unclear so I know what to answer that would be easiest to explain.

I did.
I'm confused about the part describing the pointers and the format of the duplicate files.

Hi Yanai, sorry for the delayed response; this has been a busy month.
We don't have a single function which outputs a mapping between all duplicate strings and their corresponding number of appearances in the data.
However, we do have a script which allows you to query the suffix array for how often each query string occurs. This is scripts/count_occurrences.py. There is an example for how to run it in the README.

Isn't that what's stored in "/cache/dups_$DATASET_A-B" and "/cache/sizes_$DATASET_A-B"
I understand how to get the number for a single query, but I would like to get the top-K occurrences in a specific dataset.
From my understanding the "dups_$DATASET_A-B" file contains the pointers for these duplicates, and the "sizes_$DATASET_A-B" file contains the info on their counts.
Is this correct?
If so, I'm struggling with understanding the format of these files.

Let me try maybe a simpler example.
Create a file and call it "banana.txt" and put into it "this is a bananana" [sic]. Now run cargo run make --data-file banana.txt. This makes a suffix array. Now run cargo run self-similar banana.txt --length-threshold 2 --cache-dir /tmp/cache --num-threads 1.
We should now have two files: dups and sizes. Let's ignore sizes for now, and just look at dups. My file has the following data
0f0d 0b05 0210 0e0c 0603
Each of these are single bytes pointing into our banana file. We said the length threshold is 2 so we'll take slices of length 2. These pointers are:
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f 10 11
 t  h  i  s  _  i  s  _  a  _  b  a  n  a  n  a  n  a

so therefore
0f -> "an"
0d -> "an"
0b -> "an"
05 -> "is"
02 -> "is"
10 -> "na"
0e -> "na"
0c -> "na"
06 -> "s_"
03 -> "s_"

So we can see we've found all the length two duplicates, in order. Now if you're willing to look at the file you can see how these clusters fall. What the sizes file tells you is how many examples are in each cluster, 3, 2, 3, and 2 in this case. Indeed, if we cat the sizes file I get
0302 0302

So if I want to know "which cluster has the most repeats?" then I want to argsort on the sizes file, and then pull those examples out of the dups file.
In this example the code has allocated 1 byte for each pointer because the file length is short. For longer files it will use more bytes. (It will use the smallest pointer size possible.)
Does this make sense?

Thanks for the detailed explanation!!
I think the thing I don't understand is much simpler than that.
What is the format of the files dups_... and sizes_...?
And how did you convert (in the README) from 00000000: daa4 ae05 8c7a 8505 c7a4 ae05 797a 8505  .....z......yz
to 0x05aea4da

These files are raw bytes; there is no file format. Just cat the contents.

cool, thanks!
Do you happen to have a pointer for reading these files?

I don't understand what you mean? open("sizes_...", "rb").read() should do the trick?

I meant on how to parse the files. It's not trivial, but I managed to do it eventually.

Is there something that we could put in the README that would explain whatever you had trouble with so others could do it easier?

Based on @yanaiela's code, we've implemented a script to count the top-k duplicates along with their frequencies. Additionally, we've enhanced this functionality by dynamically calculating the pointer size based on the ratio of sizes between the raw data file and its suffix array.
A specific case study is detailed in this link, showcasing deduplicated spans occurring more than twice within a 1,000-line cc100 file.
We extend our gratitude to the authors of this repository, with special thanks to @carlini. Leveraging this tool, we've built a data cleaning and deduplication tool called SailCraft. We invite you to give it a try!

authors: carlini daphnei longxudou yanaiela


text:
issue about segment in build_pretrain_dataset.py

first_segment = []
second_segment = []
for sentence in self._current_sentences:
  # the sentence goes to the first segment if (1) the first segment is
  # empty, (2) the sentence doesn't put the first segment over length or
  # (3) 50% of the time when it does put the first segment over length
  if (first_segment or
      len(first_segment) + len(sentence) < first_segment_target_length or
      (second_segment and
       len(first_segment) < first_segment_target_length and
       random.random() < 0.5)):
    first_segment += sentence
  else:
    second_segment += sentence

i have two question about above snippet code:

does the above code in “if“ branch should be "not first_segment" and "not second_segment"？
it seems there would be chances like this: suppose sentences are A,B,C, and A,C in first_segment but B in second_segment?   dose the sentence order not matter?

Thank you for pointing out the issue, this is a bug! It should be fixed fixed now.

hey， what about the second question？ i still cannot understand why this situation is allowed. shouldn't sentences in first_segment are all before those in second_segment?
A,B,C three sentences, B goes to second_segment when probability> 0.5, then len(A)+len(C) < first_segment_target_length would make C goes to first_segment again.
which makes A,C in first_segment but B in second_segment.

authors: clarkkev kelvinleen

repo: google-research/electra


text:
Random baseline scores in two subtasks' results plots incorrect

In the result plots at the bottom of the README of logical_deduction, the random baselines are off: for all three subtasks, which ask the model to choose between three, five, and seven options, respectively, the random baseline score is 20%. The overall baseline also seems to be 20%, when it should perhaps be 100 * (1/3 + 1/5 + 1/7) / 3 % ≈ 22.5%. The same thing appears to have happened with tracking_shuffled_objects.
I don't think it's a huge issue, but I thought I'd flag it in case it's a sign of some deeper bug or has an easy fix. Thanks again for organizing this all!

Thanks for identifying this! A fix is in progress. (It's a bug in the plotting code, where we are pulling the random performance value for an entire task, and all subtasks, just from the random performance value for the first subtask -- https://github.com/google/BIG-bench/blob/d00322adcfc5b18cf6ab757911e9c0d68961765f/bigbench/api/results.py#L378 )

This should now be fixed, and all plots have been regenerated to incorporate the fix. Please let us know if you still see a problem, or see any other problems! Thanks again for noticing this.

authors: Sohl-Dickstein james-simon

repo: google/BIG-bench


text:
Auto_categorization Task

This task tries to measure the model's capability of classifying given instances.
(It is the updated version of Similarity Task #125)

Hi @fanyuex2 - Should I close the old PR for Task #125?
Reviewers:
@ibab and @nick11roberts Do you still accept this task?

@chiafullo yep! I still vote for acceptance.

@chiafullo: Accept

Hi @fanyuex2 - Should I close the old PR for Task #125?
Reviewers:
@ibab and @nick11roberts Do you still accept this task?

Yes! Thank you.

I'm the meta-reviewer for this task. I like it and I'm going to merge it. I do have one nit request though: could you add a concrete (rather than abstract) example of what the task entails near the beginning of the README? I found the task difficult to understand from the README, but it became clear as soon as I looked at the examples in the .json file.
Thank you for your contribution!

Hi @Sohl-Dickstein, thank you for your suggestions! I've updated the Readme file to include an example.

LGTM. Thank you for your submission!


text:
scripts/setup: Improve gcc versioning

This change prints the URL of the GCC compiler installation
documentation when GCC is not found, or is an unrecognized version.
Currently, setup attempts to install GCC if one is not found. That is
almost always the wrong thing to do.
Fixes #57

@alanvgreen should we take out the missing+=(gcc-riscv64-unknown-elf) logic earlier in the script?

authors: alanvgreen tcal-x

repo: google/CFU-Playground


text:
GREYUIWebViewDelegate makes all XMLHttpRequests synchronous

The GREYUIWebViewDelegate forces all XMLHttpRequests to be synchronous:
grey_ajaxListener.originalOpen.apply(this, [method, url, false, user, password]);
This causes failures in some scenarios which the sync api doesn't support, for example setting the responseType will result in the following error: "Synchronous HTTP(S) requests made from the window context cannot have XMLHttpRequest.responseType set".
Is it possible not to force XMLHTTPRequests to be sync? (perhaps adding an event listener tot he 'load' event and then doing grey_ajaxListener.updateStatus('completed'))

Thanks for bringing this to our attention. We will have to look into the load event and what guarantee it provides. A quick search revealed https://rainsoft.io/catch-the-xmlhttp-request-in-plain-javascript/ which may be useful.
Would you be willing to take a stab at it and provide a test for the underlying issue that you're running into?

Sure, I'll work on a PR for this.

authors: blankg khandpur

repo: google/EarlGrey


text:
Updated pods for EarlGrey caused GREYActions .perform to error out

At this point, every test is broken because every .perform and grey_tap is no longer functioning.
Cannot invoke 'perform' with an argument list of type '(GREYAction)'

Did you update to the latest EarlGrey.swift?

Is this with Swift or Objective_c? We recently broke our Swift update for Users of Swift 3.0. We'll send an update in a bit to fix this.

I am using swift and everything is up to date.
Thanks @tirodkar , can you post an update here when it's pushed?

Absolutely. Thanks for confirming.

@tirodkar any new update on this issue?

authors: abbasmousavi bootstraponline cbiermannIMVU tirodkar


text:
Update travis run to 10.3 and Xcode 8.3 and fix compatibility for FTRVisibilityTestViewController.xib

…

Branch has conflicts.

authors: khandpur tirodkar