text:
proposal: container: add weak reference maps

This is an outgrowth of #32779, and also of https://mdlayher.com/blog/unsafe-string-interning-in-go/ and ensuing discussion of this on Gopher Slack.
Occasionally, it would be desireable to have weak references. Unfortunately, you can't implement these as a user; you need hooks into the runtime to make a safe weak reference implementation, I think. The string interning hack described above probably works but is not complying with the uintptr conversion rules.
In nearly every case where weak references are desireable, it's desireable to be able to look up specific weak references, which suggests something similar to a map data type. It's also desireable to be able to find out whether the weak reference exists, or to be able to request the reference, getting an actual stable non-weak reference back, without a separation between the query and the retrieval.
Conveniently, a map with a pointer key type already provides exactly this behavior. What it doesn't provide is the potential for GC to remove objects.
Thus, my proposal:
cache := make(weak map[string]*Data)
This is equivalent to make(map[string]*Data), except that things which are in the map but not referenced anywhere else may become deleted from the map at any arbitrary time. However, retrieval from the map is guaranteed to either yield a nil (and a false in the ,ok form) or the pointer last stored for that key, and if it yields a non-nil pointer, that pointer is safe to use.
The internal implementation details aren't 100% obvious to me. You could nearly do it with SetFinalizer, except you need a way to ensure that this doesn't clash with any other finalizers used on the item. (For instance, the same value could be present in two weak maps...)
I'm not totally sure about the syntax, but I'm pretty sure that this wouldn't compile now under any possible circumstances, so it doesn't break any existing code.
Why not sync.WeakMap, parallel to sync.Map and/or sync.Pool? Because type safety is nice, and because the ability to use or not use the ", ok" form of retrieval seems desireable. Plain old maps are easier and friendlier to use, and less error-prone.
That said, sync.WeakMap, which would vaguely combine the behaviors of sync.Map and sync.Pool, would be viable too. The main issue this is intended to address is that "would like to keep this if it's convenient and/or if it's being used, but don't need to retain it if there's memory pressure" seems like a common use case.

I assume when iterating over a map the map would be "locked" in some way? To avoid async reads/writes to the map.

I assume when iterating over a map the map would be "locked" in some way? To avoid async reads/writes to the map.

I would assume it'd be the same as any other map. Up to you to ensure that you are not modifying it while the iterator is running.

I would assume it'd be the same as any other map. Up to you to ensure that you are not modifying it while the iterator is running.

What about when the a weak reference gets removed by the runtime while iterating over the map?

What about when the a weak reference gets removed by the runtime while iterating over the map?

Same thing that would happen with anything else involving pointers in a moving GC: It's up to the runtime to have suitable barriers on or around their accesses.

Same thing that would happen with anything else involving pointers in a moving GC: It's up to the runtime to have suitable barriers on or around their accesses.

CC @bradfitz

CC @bradfitz

Some thoughts:

A detriment of using a string as the key is that it probably requires an allocation forming a key to look something up that may not even exist anymore. I see some benefit to an allocation-free key like [16]byte or [32]byte.
How complete is a sync.WeakMap? Are entries allowed to be removed even if there may be references to it (i.e., similar to sync.Pool)?

Some thoughts:

A detriment of using a string as the key is that it probably requires an allocation forming a key to look something up that may not even exist anymore. I see some benefit to an allocation-free key like [16]byte or [32]byte.
How complete is a sync.WeakMap? Are entries allowed to be removed even if there may be references to it (i.e., similar to sync.Pool)?

You can use any key type you want. It's just a normal map. string is just an example.
No, entries can't be removed if there are references to them. If there's references to them, they still exist, and a weak reference should stay valid. The entire point is to have a reliable lookup for these objects if they still exist, without ensuring that they still exist.

You can use any key type you want. It's just a normal map. string is just an example.
No, entries can't be removed if there are references to them. If there's references to them, they still exist, and a weak reference should stay valid. The entire point is to have a reliable lookup for these objects if they still exist, without ensuring that they still exist.

@seebs, how would weak references interact with finalizers (if at all)?
Specifically: is the weak reference removed from the map before the finalizer is run, or after, or is it forbidden to obtain a weak reference to an object that has a finalizer?

@seebs, how would weak references interact with finalizers (if at all)?
Specifically: is the weak reference removed from the map before the finalizer is run, or after, or is it forbidden to obtain a weak reference to an object that has a finalizer?

That is a fascinating question. On consideration: If we assume that nothing else ever users finalizers, finalizers might be a credible way to implement this, internally. Runtime can skip the storage for a weak map when marking, and set a finalizer for each item to delete it from the map, roughly. So one solution would be "forbidden to set a finalizer on weak references, or obtain a weak reference to an object with a finalizer". Except... that's horribly brittle, and might result in someone developing a time machine so they can go back to before it was done and add "never use this feature" to Effective Go.
But this is a really interestingly-difficult philosophical question. It is permissible for a finalizer to actually do a thing that prevents a thing from being collected. So if we remove things from the map first, the object will potentially still exist but we will report it missing (and thus, in most use cases, create a new one). If we run finalizers first, we can have objects in the map which have had their finalizers run, and which can thus end up referenced again post-finalizer. And you can't tell, from the perspective of the caller of the finalizer, whether that finalizer may have stashed the object somewhere so that it won't really be collected.
And it might be possible to fix this if finalizers had a way to indicate whether or not they were vetoing collection, but that could nonetheless be wrong...
I think the answer is: Finalizers run after deletion from the map, but if your finalizer preserves the object, it becomes possible that the object exists but the map doesn't refer to it. So don't do that. But not in the sense of "runtime is allowed to explode in a shower of demons", just "yes, so, that previous object is around, and the map doesn't refer to it but might now have a new entry for that key."
But philosophically, in general, a finalizer is supposed to run "after the object is dead and never referenced again", and it should not do something that would make the object get referenced again, it's just after-the-fact cleanup, and it seems fine for the object to no longer be in the map, and then get cleaned up, even if a new object for its former key ends up getting created.
(Although if you did, say, "directory named after the key" and the finalizer was "delete the directory" that would be a Bad Idea, and also your own fault, don't do that.)

That is a fascinating question. On consideration: If we assume that nothing else ever users finalizers, finalizers might be a credible way to implement this, internally. Runtime can skip the storage for a weak map when marking, and set a finalizer for each item to delete it from the map, roughly. So one solution would be "forbidden to set a finalizer on weak references, or obtain a weak reference to an object with a finalizer". Except... that's horribly brittle, and might result in someone developing a time machine so they can go back to before it was done and add "never use this feature" to Effective Go.
But this is a really interestingly-difficult philosophical question. It is permissible for a finalizer to actually do a thing that prevents a thing from being collected. So if we remove things from the map first, the object will potentially still exist but we will report it missing (and thus, in most use cases, create a new one). If we run finalizers first, we can have objects in the map which have had their finalizers run, and which can thus end up referenced again post-finalizer. And you can't tell, from the perspective of the caller of the finalizer, whether that finalizer may have stashed the object somewhere so that it won't really be collected.
And it might be possible to fix this if finalizers had a way to indicate whether or not they were vetoing collection, but that could nonetheless be wrong...
I think the answer is: Finalizers run after deletion from the map, but if your finalizer preserves the object, it becomes possible that the object exists but the map doesn't refer to it. So don't do that. But not in the sense of "runtime is allowed to explode in a shower of demons", just "yes, so, that previous object is around, and the map doesn't refer to it but might now have a new entry for that key."
But philosophically, in general, a finalizer is supposed to run "after the object is dead and never referenced again", and it should not do something that would make the object get referenced again, it's just after-the-fact cleanup, and it seems fine for the object to no longer be in the map, and then get cleaned up, even if a new object for its former key ends up getting created.
(Although if you did, say, "directory named after the key" and the finalizer was "delete the directory" that would be a Bad Idea, and also your own fault, don't do that.)

From a purely philosophical view, is it proper to impose "at most one" on weak references to an object?

From a purely philosophical view, is it proper to impose "at most one" on weak references to an object?

This is a major language and implementation change on the garbage collector side.
It is highly unlikely that we are going to add weak references any time soon.
That's an amazing amount of new complexity.
/cc @aclements @mknyszek

This is a major language and implementation change on the garbage collector side.
It is highly unlikely that we are going to add weak references any time soon.
That's an amazing amount of new complexity.
/cc @aclements @mknyszek

The "major change" thing is both why it's a hard sell to put it in runtime, and why it's impossible to put it somewhere else.

The "major change" thing is both why it's a hard sell to put it in runtime, and why it's impossible to put it somewhere else.

This is something I've thought about quite a bit in the past. My past thinking has led to a bit of a counter-proposal: don't allow iteration over weak maps. Right now, the garbage collector is invisible in the language semantics (other than finalizers, which are an unfortunate wart at best). The language specification is built on the illusion of infinite memory. If you have a map that allows lookups and not iteration, we maintain that illusion because the GC can delete unreachable keys from that map without any effect on program execution. It remains invisible. An un-iterable map still provides most functionality that people typically want from a weak map, including string interning and carrying side information for objects without forcing them to remain live. There is also exactly how ECMAScript's WeakMap works.
I would argue that this should at least wait on generics. At that point, we could provide this as a regular type with all the type-safety of the built-in map. I'm not sure where it should live; it's not about synchronization, so it seems odd to put it in sync.
Implementing the above for pointer keys is actually pretty easy. We'd mark the key storage as non-pointer data, use allocation specials to track the set of weak maps containing an object as a key, and delete keys from those weak maps during sweeping. Disallowing iteration simplifies a lot of questions around races with the GC.
Implementing it for non-pointer keys is harder, but probably necessary to make it useful. I think the map would have to find the pointers in a key when it's added and attach specials to each of those allocations. During tracing, if the GC reaches any of these objects, it would have to trace across to everything else referenced by that key as well. Object tracing doesn't currently have to look at specials, so I'm not sure how to do this without a performance penalty to every object scanned during GC.

This is something I've thought about quite a bit in the past. My past thinking has led to a bit of a counter-proposal: don't allow iteration over weak maps. Right now, the garbage collector is invisible in the language semantics (other than finalizers, which are an unfortunate wart at best). The language specification is built on the illusion of infinite memory. If you have a map that allows lookups and not iteration, we maintain that illusion because the GC can delete unreachable keys from that map without any effect on program execution. It remains invisible. An un-iterable map still provides most functionality that people typically want from a weak map, including string interning and carrying side information for objects without forcing them to remain live. There is also exactly how ECMAScript's WeakMap works.
I would argue that this should at least wait on generics. At that point, we could provide this as a regular type with all the type-safety of the built-in map. I'm not sure where it should live; it's not about synchronization, so it seems odd to put it in sync.
Implementing the above for pointer keys is actually pretty easy. We'd mark the key storage as non-pointer data, use allocation specials to track the set of weak maps containing an object as a key, and delete keys from those weak maps during sweeping. Disallowing iteration simplifies a lot of questions around races with the GC.
Implementing it for non-pointer keys is harder, but probably necessary to make it useful. I think the map would have to find the pointers in a key when it's added and attach specials to each of those allocations. During tracing, if the GC reaches any of these objects, it would have to trace across to everything else referenced by that key as well. Object tracing doesn't currently have to look at specials, so I'm not sure how to do this without a performance penalty to every object scanned during GC.

That's a fascinating idea. My vague intuition is there might be cases where you'd want to iterate the map to do something about the data, but in all those cases, I think you'd also want it to be non-weak at that point.
But I think "unreachable keys" feels like a slightly different question; I was assuming ordinary keys, which might be strings or integers, and potentially-unreachable values. So for instance, with a weak map[string]*Node, if someone has associated a given string with a *Node in the map, then we want any attempt to look up that same string -- even at a different address and constructed differently -- to necessarily find that *Node, if it is still referenced anywhere. But if nothing references that *Node except for this map, then we no longer care; there's no way for anyone to detect that a new node we return isn't the one we had for that string before, in theory.
I guess the answer is that either keys or values could in principle be weak references.

That's a fascinating idea. My vague intuition is there might be cases where you'd want to iterate the map to do something about the data, but in all those cases, I think you'd also want it to be non-weak at that point.
But I think "unreachable keys" feels like a slightly different question; I was assuming ordinary keys, which might be strings or integers, and potentially-unreachable values. So for instance, with a weak map[string]*Node, if someone has associated a given string with a *Node in the map, then we want any attempt to look up that same string -- even at a different address and constructed differently -- to necessarily find that *Node, if it is still referenced anywhere. But if nothing references that *Node except for this map, then we no longer care; there's no way for anyone to detect that a new node we return isn't the one we had for that string before, in theory.
I guess the answer is that either keys or values could in principle be weak references.

Implementing it for non-pointer keys is harder

Could the implementation allocate for these and implicitly unbox when returning them? Then the GC could continue to deal only in pointers.

Implementing it for non-pointer keys is harder

Could the implementation allocate for these and implicitly unbox when returning them? Then the GC could continue to deal only in pointers.

A weak map has two types: a key type and a value type.  Only the value type is weak.  That is, if m is a weak map, then m.Set(k, v) stores v with key k.  If m is the only reference to v, then the garbage collector is permitted to release v, after which m.Get(k) will indicate that there is no such key.  Moreover, if v is released, then k can be released.
My point is that with those semantics I think it's fine to restrict the value type of a weak map to be a pointer.  It's hard to even understand what it means to store a non-pointer in a weak map.  What does it mean to say that there are no remaining references to a non-pointer value?  Isn't that equivalent to saying that there are no pointers to the value?
Are people thinking of different semantics?

A weak map has two types: a key type and a value type.  Only the value type is weak.  That is, if m is a weak map, then m.Set(k, v) stores v with key k.  If m is the only reference to v, then the garbage collector is permitted to release v, after which m.Get(k) will indicate that there is no such key.  Moreover, if v is released, then k can be released.
My point is that with those semantics I think it's fine to restrict the value type of a weak map to be a pointer.  It's hard to even understand what it means to store a non-pointer in a weak map.  What does it mean to say that there are no remaining references to a non-pointer value?  Isn't that equivalent to saying that there are no pointers to the value?
Are people thinking of different semantics?

What does it mean to say that there are no remaining references to a non-pointer value? Isn't that equivalent to saying that there are no pointers to the value?

I was thinking of the value being of type (say) [2]*int. There are two semantics possible: (1) Each of the pointers is weak; they can be set to nil independently, when the int that they refer to is GC'd. (2) The entire object is GC'd or not; only when both ints referred to by the pointers are GC'd may both be set to nil.
That said, I'd be perfectly happy to restrict the value type of a weak map to be a pointer. I imagine it'd have the same restrictions as the first argument to runtime.SetFinalizer, which are even stronger than "must be a pointer".

What does it mean to say that there are no remaining references to a non-pointer value? Isn't that equivalent to saying that there are no pointers to the value?

I was thinking of the value being of type (say) [2]*int. There are two semantics possible: (1) Each of the pointers is weak; they can be set to nil independently, when the int that they refer to is GC'd. (2) The entire object is GC'd or not; only when both ints referred to by the pointers are GC'd may both be set to nil.
That said, I'd be perfectly happy to restrict the value type of a weak map to be a pointer. I imagine it'd have the same restrictions as the first argument to runtime.SetFinalizer, which are even stronger than "must be a pointer".

It was pointed out to me that my point about non-pointer keys was really unclear and also wrong. I don't mean pointer-free keys; I'm not even sure what that would mean in a weak-keyed map (probably it would just act like a normal map). I mean keys that contain pointers, like string or struct {a, b *int}. However, in my post above I was thinking this was hard because you couldn't remove the key until all of the pointers were unreachable, but you can delete the key as soon as any of the pointers are unreachable (since you can no longer construct that key). That makes non-pointer keys not much harder to support than pointer-typed keys.

But I think "unreachable keys" feels like a slightly different question; I was assuming ordinary keys, which might be strings or integers, and potentially-unreachable values.

Ah, that is quite different from what I was talking about. I much less on-board with a weak-valued map than a weak-keyed (non-interable) map because it exposes the behavior and timing of the garbage collector, exactly the way finalizers do. Also, I understand concrete use cases of weak-keyed maps, but I don't actually understand concrete uses of weak-valued maps. Your example of a string-to-*Node map was rather abstract. Can you give a concrete example of what you would use that for?

Are people thinking of different semantics?

I was definitely thinking of different semantics, but maybe I'm the only one.

It was pointed out to me that my point about non-pointer keys was really unclear and also wrong. I don't mean pointer-free keys; I'm not even sure what that would mean in a weak-keyed map (probably it would just act like a normal map). I mean keys that contain pointers, like string or struct {a, b *int}. However, in my post above I was thinking this was hard because you couldn't remove the key until all of the pointers were unreachable, but you can delete the key as soon as any of the pointers are unreachable (since you can no longer construct that key). That makes non-pointer keys not much harder to support than pointer-typed keys.

But I think "unreachable keys" feels like a slightly different question; I was assuming ordinary keys, which might be strings or integers, and potentially-unreachable values.

Ah, that is quite different from what I was talking about. I much less on-board with a weak-valued map than a weak-keyed (non-interable) map because it exposes the behavior and timing of the garbage collector, exactly the way finalizers do. Also, I understand concrete use cases of weak-keyed maps, but I don't actually understand concrete uses of weak-valued maps. Your example of a string-to-*Node map was rather abstract. Can you give a concrete example of what you would use that for?

Are people thinking of different semantics?

I was definitely thinking of different semantics, but maybe I'm the only one.

Well, hmm. You can't GC the int pointed to by one of the pointers unless you set that pointer to nil, because otherwise you'd have an easy way to get an invalid pointer unexpectedly. I think my intuition is that as long as the [2]*int exists, it is in fact a valid reference and the things it points to aren't collectible, but if nothing refers to it, then it can be collected, at which point the things it points to are no longer referenced (by it, anyway). But that implies that the value type really ought to be *[2]*int, because otherwise it's not really meaningful to talk about collecting it.
I am definitely leaning towards Ian's interpretation; the keys are not "weak" in any way, they're just keys, if they contain pointers they are references to them. The only difference from a regular map is that, if the values are pointers, they don't count as references, but if the thing they point to gets GCd, the key/value pair is deleted from the map when that happens.
So, a concrete example of utility: Interning strings. You want to have a lot of objects which should contain a string, but you don't want to have multiple different allocations for the same string, and you want comparisons of them for sameness to be cheap. So...
type Intern string
var interned weak map[string]*Intern
var internMu sync.Mutex

// intern a string:
func intern(k string) *Intern {
  internMu.Lock()
  defer internMu.Unlock()
  if i, ok := interned[k]; ok { return i }
  internMe := Intern(k)
  interned[k] = &internMe
  return internMe
}

The result of calling intern(foo) for any given string is a *Intern, with the convenient guarantee that anyone else who interned the same string got, not just a pointer to an equal Intern, but exactly the same pointer. You can use these as map keys, for instance, and get single-word compares instead of string comparisons. If you have a million objects that were created using slices of different strings, and all the slices compared equal, you have a single string that all of them intern to, and the things those were sliced out of aren't being pinned. So you get much better performance than you would using the original string object(s).
So why not just use a normal map? Because if the set of strings is largeish, and changes over time, and sometimes a string will stop being used, a non-weak map[string]*Intern will eventually grow without bound. You can't safely remove anything from it because you can't prove that nothing's still using that value, and having two different *Intern that actually refer to identical strings would break using them as map keys, etcetera.
With the weak map, you can discard any values that are no longer referenced anywhere, and you still have the guarantee that no two users who try to intern the same string will ever see different interned values.
This can also be used for some kinds of caching. For instance, say you were doing a naive cache of fetched URLs. If you do it as weak map[url]*body, then if you fetch a URL, and dump it in the map, another fetch while that body is still around can just grab the copy in the weak-reference map, avoiding re-fetching the body that you already have, but if no one is using the body, it can be dropped silently from the cache. (I am aware that this wouldn't be a great choice for a lot of use cases, for a number of reasons. The first hard problem is called cache invalidation.)
So basically, the use case is: (1) it would make sense to have a map with these things as values, (2) we definitely don't want to throw them away if anyone's using them (3) it's fine to throw them away if no one's using them (4) it would be annoying to try to track who might be using them.
I note that "not iterable" seems fine to me for a weak-valued map as well. It's not really useful to iterate it, generally.
I'm not getting a clear picture of how a weak-keyed map would be used. Like, for interning, what would I be storing as keys? The addresses of strings can't be right, because the whole point is that I want to get the same thing from the map given two distinct strings which happen to compare equal.

Well, hmm. You can't GC the int pointed to by one of the pointers unless you set that pointer to nil, because otherwise you'd have an easy way to get an invalid pointer unexpectedly. I think my intuition is that as long as the [2]*int exists, it is in fact a valid reference and the things it points to aren't collectible, but if nothing refers to it, then it can be collected, at which point the things it points to are no longer referenced (by it, anyway). But that implies that the value type really ought to be *[2]*int, because otherwise it's not really meaningful to talk about collecting it.
I am definitely leaning towards Ian's interpretation; the keys are not "weak" in any way, they're just keys, if they contain pointers they are references to them. The only difference from a regular map is that, if the values are pointers, they don't count as references, but if the thing they point to gets GCd, the key/value pair is deleted from the map when that happens.
So, a concrete example of utility: Interning strings. You want to have a lot of objects which should contain a string, but you don't want to have multiple different allocations for the same string, and you want comparisons of them for sameness to be cheap. So...
type Intern string
var interned weak map[string]*Intern
var internMu sync.Mutex

// intern a string:
func intern(k string) *Intern {
  internMu.Lock()
  defer internMu.Unlock()
  if i, ok := interned[k]; ok { return i }
  internMe := Intern(k)
  interned[k] = &internMe
  return internMe
}

The result of calling intern(foo) for any given string is a *Intern, with the convenient guarantee that anyone else who interned the same string got, not just a pointer to an equal Intern, but exactly the same pointer. You can use these as map keys, for instance, and get single-word compares instead of string comparisons. If you have a million objects that were created using slices of different strings, and all the slices compared equal, you have a single string that all of them intern to, and the things those were sliced out of aren't being pinned. So you get much better performance than you would using the original string object(s).
So why not just use a normal map? Because if the set of strings is largeish, and changes over time, and sometimes a string will stop being used, a non-weak map[string]*Intern will eventually grow without bound. You can't safely remove anything from it because you can't prove that nothing's still using that value, and having two different *Intern that actually refer to identical strings would break using them as map keys, etcetera.
With the weak map, you can discard any values that are no longer referenced anywhere, and you still have the guarantee that no two users who try to intern the same string will ever see different interned values.
This can also be used for some kinds of caching. For instance, say you were doing a naive cache of fetched URLs. If you do it as weak map[url]*body, then if you fetch a URL, and dump it in the map, another fetch while that body is still around can just grab the copy in the weak-reference map, avoiding re-fetching the body that you already have, but if no one is using the body, it can be dropped silently from the cache. (I am aware that this wouldn't be a great choice for a lot of use cases, for a number of reasons. The first hard problem is called cache invalidation.)
So basically, the use case is: (1) it would make sense to have a map with these things as values, (2) we definitely don't want to throw them away if anyone's using them (3) it's fine to throw them away if no one's using them (4) it would be annoying to try to track who might be using them.
I note that "not iterable" seems fine to me for a weak-valued map as well. It's not really useful to iterate it, generally.
I'm not getting a clear picture of how a weak-keyed map would be used. Like, for interning, what would I be storing as keys? The addresses of strings can't be right, because the whole point is that I want to get the same thing from the map given two distinct strings which happen to compare equal.

Okay, I think I understand your interning example (I certainly understand the code and intent; not sure I've completely wrapped my head around the implications :). And I think interning like this does require a structure that exposes GC behavior, since it has to distinguish whether objects still exist. So weak-keyed maps wouldn't work for interning.
But my basic concern with a weak-valued map is that its semantics are deeply tied to GC behavior. It can't be described without explaining what it means for things to become unreachable. Finalizers are the only other thing in the language like that right now, and those are an unfortunate and necessary evil. In my opinion, the bar for adding another is extremely high. (sync.Pool's semantics don't expose GC behavior. It can be described simply as a lossy set. GC behavior is key to Pool's performance characteristics, but that's true for a lot of things.)
Also, I believe weak-valued maps are ultimately no different from weak references, and I'm not sure why we would add weak-valued maps rather than just weak references. You can build a weak reference from a weak map using a single-entry map, and you can build a weak map from a weak reference by using weak references for the values (plus either finalizers or phantom reference queues, depending on the details of the weak reference API).
My other concern is implementation. First, a weak-valued map (or any form of weak reference) cracks open the assumptions Go's current write barrier is based on. It might be possible to work around that by synchronizing carefully with the garbage collector during the "Get" and "Put" operations, but I'd have to work out those details. It's particularly hairy if an unreachable value is actively being swept during a Get. Sweeping would probably have to lock the map. (Weak-keyed maps have none of these problems.)
Second, weak-valued maps have the problem I was (incorrectly) getting at above for values containing multiple pointers, like [2]*int. It doesn't make sense to collect the map entry until all of the pointers are unreachable simultaneously, but I don't know how to do that without slowing down the GC on everything. Maybe we would only allow pointer values, but does that restrict the utility of a weak-valued map? I don't know.

Okay, I think I understand your interning example (I certainly understand the code and intent; not sure I've completely wrapped my head around the implications :). And I think interning like this does require a structure that exposes GC behavior, since it has to distinguish whether objects still exist. So weak-keyed maps wouldn't work for interning.
But my basic concern with a weak-valued map is that its semantics are deeply tied to GC behavior. It can't be described without explaining what it means for things to become unreachable. Finalizers are the only other thing in the language like that right now, and those are an unfortunate and necessary evil. In my opinion, the bar for adding another is extremely high. (sync.Pool's semantics don't expose GC behavior. It can be described simply as a lossy set. GC behavior is key to Pool's performance characteristics, but that's true for a lot of things.)
Also, I believe weak-valued maps are ultimately no different from weak references, and I'm not sure why we would add weak-valued maps rather than just weak references. You can build a weak reference from a weak map using a single-entry map, and you can build a weak map from a weak reference by using weak references for the values (plus either finalizers or phantom reference queues, depending on the details of the weak reference API).
My other concern is implementation. First, a weak-valued map (or any form of weak reference) cracks open the assumptions Go's current write barrier is based on. It might be possible to work around that by synchronizing carefully with the garbage collector during the "Get" and "Put" operations, but I'd have to work out those details. It's particularly hairy if an unreachable value is actively being swept during a Get. Sweeping would probably have to lock the map. (Weak-keyed maps have none of these problems.)
Second, weak-valued maps have the problem I was (incorrectly) getting at above for values containing multiple pointers, like [2]*int. It doesn't make sense to collect the map entry until all of the pointers are unreachable simultaneously, but I don't know how to do that without slowing down the GC on everything. Maybe we would only allow pointer values, but does that restrict the utility of a weak-valued map? I don't know.

So, as a user, I mostly want a thing like this in a case where I don't care whether the object never existed or used to exist but got GCd, I just don't want to make an object if it already exists, but I don't want to pin a zillion objects that I might never use again. So, it sort of exposes a GC behavior, but in a very loose sense; nothing prevents the implementation from continuing to hold things for a long time. (Strictly speaking, existing maps already satisfy the expected usage behavior, in that they meet the constraint that the retrieved pointer is valid as long as the object hasn't been collected, it's just that it always turns out that the object hasn't been collected because the map counts as a reference to it. But the GC would be allowed to never get around to collecting objects anyway...)
The reason I wrote this proposal with a map rather than with references is sort of glossed over in the initial proposal. Basically:
(1) I can't think of a real use case where I want a weak-reference, and I don't want to be able to look it up by some kind of key.
(2) Weak references are naturally well suited to the , ok idiom of maps.
And then, thinking about it, it occurred to me that a lot of the things that make weak references seem like an annoying mess to track are potentially significantly mitigated by grouping them all together. If all the weak references are being kept in one object, then we have one object/region that needs special treatment. If you have a million structs each of which has a weak reference to something, you have to do whatever your special processing is for every one of those weak references, separately from whatever you do for pointers in those structs. If you have a weak map, you can do your special processing for everything in the map and nothing not in the map, and that seems like it should be simpler.
I think a weak-valued map only makes sense if the values in it are themselves pointers to things. Maps aren't addressable; nothing can be pointing-to the map entry, so the map entry can't itself be collectable or pinned. So I think it's only meaningful for a map to be weak when the value type is a pointer type itself.
My vague intuition, which I haven't fully thought through, is that if a weak-valued map didn't mark the things it contains, but did follow pointers within them and mark those, and it were prohibited to use SetFinalizer on things that live in a weak-valued map, and things lived in only one such map, it would be adequate for the runtime to SetFinalizer() on them to delete them from the map. Except, as you note, the sync issue.
I'm still not quite able to make sense of a weak-keyed map. I can describe it but I'm not sure what I'd use it for. I guess it would be a more implicit usage. With the weak-valued map, I have a key and I want to know whether there's an associated value. With the weak-keyed map, if I have a key, I know the value can't have been deleted.

So, as a user, I mostly want a thing like this in a case where I don't care whether the object never existed or used to exist but got GCd, I just don't want to make an object if it already exists, but I don't want to pin a zillion objects that I might never use again. So, it sort of exposes a GC behavior, but in a very loose sense; nothing prevents the implementation from continuing to hold things for a long time. (Strictly speaking, existing maps already satisfy the expected usage behavior, in that they meet the constraint that the retrieved pointer is valid as long as the object hasn't been collected, it's just that it always turns out that the object hasn't been collected because the map counts as a reference to it. But the GC would be allowed to never get around to collecting objects anyway...)
The reason I wrote this proposal with a map rather than with references is sort of glossed over in the initial proposal. Basically:
(1) I can't think of a real use case where I want a weak-reference, and I don't want to be able to look it up by some kind of key.
(2) Weak references are naturally well suited to the , ok idiom of maps.
And then, thinking about it, it occurred to me that a lot of the things that make weak references seem like an annoying mess to track are potentially significantly mitigated by grouping them all together. If all the weak references are being kept in one object, then we have one object/region that needs special treatment. If you have a million structs each of which has a weak reference to something, you have to do whatever your special processing is for every one of those weak references, separately from whatever you do for pointers in those structs. If you have a weak map, you can do your special processing for everything in the map and nothing not in the map, and that seems like it should be simpler.
I think a weak-valued map only makes sense if the values in it are themselves pointers to things. Maps aren't addressable; nothing can be pointing-to the map entry, so the map entry can't itself be collectable or pinned. So I think it's only meaningful for a map to be weak when the value type is a pointer type itself.
My vague intuition, which I haven't fully thought through, is that if a weak-valued map didn't mark the things it contains, but did follow pointers within them and mark those, and it were prohibited to use SetFinalizer on things that live in a weak-valued map, and things lived in only one such map, it would be adequate for the runtime to SetFinalizer() on them to delete them from the map. Except, as you note, the sync issue.
I'm still not quite able to make sense of a weak-keyed map. I can describe it but I'm not sure what I'd use it for. I guess it would be a more implicit usage. With the weak-valued map, I have a key and I want to know whether there's an associated value. With the weak-keyed map, if I have a key, I know the value can't have been deleted.

I can't think of a real use case where I want a weak-reference, and I don't want to be able to look it up by some kind of key.

“Looking up by key” and “stored in a field alongside other data” often end up being interchangeable. Imagine an expensive immutable cached result. You store it with a weak ref alongside the raw data and return it. If no one is using it any more, it returns to nil, and you recalculate again later.
At the end of this conversation, I don’t see the advantage that maps have over plain old weak refs, except possibly in syntax. But with generics, we could have a runtime.WeakRef[T] type.

I can't think of a real use case where I want a weak-reference, and I don't want to be able to look it up by some kind of key.

“Looking up by key” and “stored in a field alongside other data” often end up being interchangeable. Imagine an expensive immutable cached result. You store it with a weak ref alongside the raw data and return it. If no one is using it any more, it returns to nil, and you recalculate again later.
At the end of this conversation, I don’t see the advantage that maps have over plain old weak refs, except possibly in syntax. But with generics, we could have a runtime.WeakRef[T] type.

Weak references are basically isomorphic to finalizers.  It is possible to implement weak references using finalizers, it's just that the result is awkward to use because you can't use ordinary pointers.  So if you don't like finalizers, you won't like weak references.

Weak references are basically isomorphic to finalizers.  It is possible to implement weak references using finalizers, it's just that the result is awkward to use because you can't use ordinary pointers.  So if you don't like finalizers, you won't like weak references.

It is possible to implement weak references using finalizers, it's just that the result is awkward to use because you can't use ordinary pointers.

That was not my experience. Please see the discussion at https://github.com/go4org/intern/blob/3eb7198706b254cddea46d066da7316b2620ba7a/intern.go#L169. I’d love to be wrong about this, but the conclusion I came to was that weak refs are not implementable using finalizers.

It is possible to implement weak references using finalizers, it's just that the result is awkward to use because you can't use ordinary pointers.

That was not my experience. Please see the discussion at https://github.com/go4org/intern/blob/3eb7198706b254cddea46d066da7316b2620ba7a/intern.go#L169. I’d love to be wrong about this, but the conclusion I came to was that weak refs are not implementable using finalizers.

@bcmills made a similar point on Gopher Slack, so cc’ing him here.

@bcmills made a similar point on Gopher Slack, so cc’ing him here.

@josharian I sketched it out at https://github.com/golang/go/issues/41303#issuecomment-717401656.  The problem is that everybody has to use Strong pointers, they can't use ordinary pointers.  And they have to be careful about using runtime.KeepAlive.

@josharian I sketched it out at https://github.com/golang/go/issues/41303#issuecomment-717401656.  The problem is that everybody has to use Strong pointers, they can't use ordinary pointers.  And they have to be careful about using runtime.KeepAlive.

Oh, I see you linked there already.  Perhaps my sketch is not useful for the intern package, but I claim that it does indeed implement weak references.

Oh, I see you linked there already.  Perhaps my sketch is not useful for the intern package, but I claim that it does indeed implement weak references.

it does indeed implement weak references.

Sorry. You're right. However, there are limitations on how those weak pointers can be used. And those limitations make them not usable in the sole case to date in which I have wanted weak pointers.

Weak references are basically isomorphic to finalizers.

I guess this is the statement that I disagree with, because there are tasks that you can accomplish with weak refs that you can't accomplish with finalizers.

it does indeed implement weak references.

Sorry. You're right. However, there are limitations on how those weak pointers can be used. And those limitations make them not usable in the sole case to date in which I have wanted weak pointers.

Weak references are basically isomorphic to finalizers.

I guess this is the statement that I disagree with, because there are tasks that you can accomplish with weak refs that you can't accomplish with finalizers.

Fair enough.  Let me put it this way: any trouble you can get into with finalizers, you can also get into that trouble with weak references.  (Except crashing in the finalizer function or something like that.)
For example, cyclical references don't work with finalizers, and they won't work with weak references either.  Also, the compiler can cause a strong reference to die at times when people don't expect, and that can cause the weak reference to stop working.

Fair enough.  Let me put it this way: any trouble you can get into with finalizers, you can also get into that trouble with weak references.  (Except crashing in the finalizer function or something like that.)
For example, cyclical references don't work with finalizers, and they won't work with weak references either.  Also, the compiler can cause a strong reference to die at times when people don't expect, and that can cause the weak reference to stop working.

If I understand https://github.com/golang/go/issues/41303#issuecomment-717401656 correctly, it does not implement what I would consider to be “weak references”.
In particular, I expect the following properties:

A weak reference can be obtained from any strong reference.
A strong reference can be obtained from any weak reference if the underlying object still exists.
A strong reference obtained from any weak reference to an object keeps all weak references to that object alive.

As far as I can tell, the API in that sketch does not satisfy the third property.

If I understand https://github.com/golang/go/issues/41303#issuecomment-717401656 correctly, it does not implement what I would consider to be “weak references”.
In particular, I expect the following properties:

A weak reference can be obtained from any strong reference.
A strong reference can be obtained from any weak reference if the underlying object still exists.
A strong reference obtained from any weak reference to an object keeps all weak references to that object alive.

As far as I can tell, the API in that sketch does not satisfy the third property.

For example, cyclical references don't work with finalizers, and they won't work with weak references either.

I don't see why that would be the case. Could you elaborate?
In reference-counted languages, weak references are sometimes used to explicitly break strong-reference cycles.

For example, cyclical references don't work with finalizers, and they won't work with weak references either.

I don't see why that would be the case. Could you elaborate?
In reference-counted languages, weak references are sometimes used to explicitly break strong-reference cycles.

Sorry, I meant specifically for Go.

Sorry, I meant specifically for Go.

I don't see why that would be the case for a native, runtime-supported implementation of weak references in Go either.
If you implement weak references in terms of finalizers, then it is certainly true that they share all the same drawbacks. However, that is a conclusion that follows from the assumption that the two are equivalent — it is not evidence for their intrinsic equivalence.

I don't see why that would be the case for a native, runtime-supported implementation of weak references in Go either.
If you implement weak references in terms of finalizers, then it is certainly true that they share all the same drawbacks. However, that is a conclusion that follows from the assumption that the two are equivalent — it is not evidence for their intrinsic equivalence.

In a garbage collected language that doesn't use reference counting, what can you do with weak references that you can't do with finalizers?  And vice-versa?

In a garbage collected language that doesn't use reference counting, what can you do with weak references that you can't do with finalizers?  And vice-versa?

With weak references but not finalizers, you can have structs with weakly-referenced fields without rewriting all transitive references to the values to use a “strong reference” wrapper type.
With weak references but not finalizers, you can implement interning tables with weak values, but you have to manually prune the stale keys as values are collected.
With weak references plus finalizers, you can implement interning tables with weak values with automatic pruning of stale keys.
That is not to say that weak references are essential, but they are not redundant with finalizers.

With weak references but not finalizers, you can have structs with weakly-referenced fields without rewriting all transitive references to the values to use a “strong reference” wrapper type.
With weak references but not finalizers, you can implement interning tables with weak values, but you have to manually prune the stale keys as values are collected.
With weak references plus finalizers, you can implement interning tables with weak values with automatic pruning of stale keys.
That is not to say that weak references are essential, but they are not redundant with finalizers.

Maybe I'm lost in the discussion, but what's the exact relationship between "weak references versus finalizers" and what the issue was originally about, which is weak references maps? Are we suggesting we should provide weak references instead of weak reference maps or are we talking about how a weak reference map would be implemented?
It seems that that weak references alone is insufficient since a key feature of a weak reference map is that the garbage collector takes care of deleting entries that are dead. I'm not sure I know how to implement a weak reference map with weak references alone.

Maybe I'm lost in the discussion, but what's the exact relationship between "weak references versus finalizers" and what the issue was originally about, which is weak references maps? Are we suggesting we should provide weak references instead of weak reference maps or are we talking about how a weak reference map would be implemented?
It seems that that weak references alone is insufficient since a key feature of a weak reference map is that the garbage collector takes care of deleting entries that are dead. I'm not sure I know how to implement a weak reference map with weak references alone.

@dsnet
Given that Go already has finalizers, I claim that we could enable users to implement weak-reference maps by adding only ordinary weak references to the language.
That seems simpler to me, because the semantics of ordinary weak references seem straightforward (just the invariants described in https://github.com/golang/go/issues/43615#issuecomment-760456332), in comparison to the confusion and ambiguity of “weak keys” vs. “weak values”.

@dsnet
Given that Go already has finalizers, I claim that we could enable users to implement weak-reference maps by adding only ordinary weak references to the language.
That seems simpler to me, because the semantics of ordinary weak references seem straightforward (just the invariants described in https://github.com/golang/go/issues/43615#issuecomment-760456332), in comparison to the confusion and ambiguity of “weak keys” vs. “weak values”.

For example, cyclical references don't work with finalizers, and they won't work with weak references either.

@ianlancetaylor, I don't believe this is correct. Cycles through a weak reference can be collected. (There are other forms of finalizers that can also deal with cycles, such as phantoms.)

Maybe I'm lost in the discussion, but what's the exact relationship between "weak references versus finalizers" and what the issue was originally about, which is weak references maps?

@dsnet, that was partly my fault. I brought up weak references vs weak maps here.
There are many variations on weak references. One variation is in how you tell an application that the weak reference has been collected. The runtime can just nil out the reference and leave the application to discover that later, or it can provide a callback or a "reference queue" that lets the user do finalization-style things. Usually when used with weak references, the callback or queue doesn't provide the original object, but some other bit of information you associated with it, for example, the map and key it was stored under. Doing this with weak references and callbacks/queues is nice because it's clear what happens when there are multiple weak references. While there's nothing impossible about having multiple finalizers for an object, we disallow it because it gets messy quickly (what order do they run in? if one of them revives the object, do you run the rest? how do you remove a finalizer?).
One reasons Go finalizers are so tricky in the runtime is that they provide the original object to the callback, which means the runtime has to revive the object just to run the callback. This is what makes objects with finalizers in cycles un-collectable, delays reclaiming of memory for finalized objects, and means that a chain of N objects with finalizers takes N GC cycles to clean up. If weak references had a callback, but didn't provide the original object, they don't have these problems.

For example, cyclical references don't work with finalizers, and they won't work with weak references either.

@ianlancetaylor, I don't believe this is correct. Cycles through a weak reference can be collected. (There are other forms of finalizers that can also deal with cycles, such as phantoms.)

Maybe I'm lost in the discussion, but what's the exact relationship between "weak references versus finalizers" and what the issue was originally about, which is weak references maps?

@dsnet, that was partly my fault. I brought up weak references vs weak maps here.
There are many variations on weak references. One variation is in how you tell an application that the weak reference has been collected. The runtime can just nil out the reference and leave the application to discover that later, or it can provide a callback or a "reference queue" that lets the user do finalization-style things. Usually when used with weak references, the callback or queue doesn't provide the original object, but some other bit of information you associated with it, for example, the map and key it was stored under. Doing this with weak references and callbacks/queues is nice because it's clear what happens when there are multiple weak references. While there's nothing impossible about having multiple finalizers for an object, we disallow it because it gets messy quickly (what order do they run in? if one of them revives the object, do you run the rest? how do you remove a finalizer?).
One reasons Go finalizers are so tricky in the runtime is that they provide the original object to the callback, which means the runtime has to revive the object just to run the callback. This is what makes objects with finalizers in cycles un-collectable, delays reclaiming of memory for finalized objects, and means that a chain of N objects with finalizers takes N GC cycles to clean up. If weak references had a callback, but didn't provide the original object, they don't have these problems.

Part of the appeal, to me, of the map is that you don't have to think about a specific reference. If you have a reference, it's a strong reference. The weak-reference nature is restricted to grabbing things from a map, and that already has the , ok behavior.

Part of the appeal, to me, of the map is that you don't have to think about a specific reference. If you have a reference, it's a strong reference. The weak-reference nature is restricted to grabbing things from a map, and that already has the , ok behavior.

Ephemerons[1] solve some of the problems mentioned and are as close to what
Austin is suggesting as the literature provides. Wikipedia reports use in
Smalltalk, .Net, and a few more languages. Ephemerons aren't in Java since
the Hayes paper didn't come out until a few years later. I doubt an
ephemeron proposal will be accepted but it has a track record and provides
a vocabulary that can be used to discuss the issues.
[1] Hayes, B. Ephemerons: A new finalization mechanism. In Proceedings of
the 12th ACM SIGPLAN Conference on Object-Oriented Programming,
Systems, Languages, and Applications (Atlanta, GA, Oct. 5--9). ACM, New
York, 1997, 176--183.
On Fri, Jan 15, 2021 at 4:36 PM seebs notifications@github.com wrote:

Part of the appeal, to me, of the map is that you don't have to think
about a specific reference. If you have a reference, it's a strong
reference. The weak-reference nature is restricted to grabbing things from
a map, and that already has the , ok behavior.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/43615#issuecomment-761211248, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHNNH6MGW5Y5QJ45LWF7ALS2CYPXANCNFSM4V36V3UA
.

Ephemerons[1] solve some of the problems mentioned and are as close to what
Austin is suggesting as the literature provides. Wikipedia reports use in
Smalltalk, .Net, and a few more languages. Ephemerons aren't in Java since
the Hayes paper didn't come out until a few years later. I doubt an
ephemeron proposal will be accepted but it has a track record and provides
a vocabulary that can be used to discuss the issues.
[1] Hayes, B. Ephemerons: A new finalization mechanism. In Proceedings of
the 12th ACM SIGPLAN Conference on Object-Oriented Programming,
Systems, Languages, and Applications (Atlanta, GA, Oct. 5--9). ACM, New
York, 1997, 176--183.
On Fri, Jan 15, 2021 at 4:36 PM seebs notifications@github.com wrote:

Part of the appeal, to me, of the map is that you don't have to think
about a specific reference. If you have a reference, it's a strong
reference. The weak-reference nature is restricted to grabbing things from
a map, and that already has the , ok behavior.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/43615#issuecomment-761211248, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHNNH6MGW5Y5QJ45LWF7ALS2CYPXANCNFSM4V36V3UA
.

I was looking into a memory leak that stemmed from dynamically created types not being garbage collected (#28783). In that issue, @randall77 comments:

Yep. This is a hard one to fix.
We depend on type uniqueness, so we need to keep track of all the types we've currently made.
To GC the type, we need some way of telling reflect we've done so, so it can remove that type from its list of created types. Probably something akin to Java's weak references.
All the interface tables are allocated in persistent memory, not on the heap, so that would need to change also.
The compiler has a bunch of assumptions about types and interface tables never being GCd (e.g. the first word of an interface isn't a pointer that GC needs to traverse), that would need finding and fixing also.

I don't know much about the implementation of reflect, but there might be overlap in fixing #28783 with this issue.
Even if dynamically constructed types could be garbage collected, we still have the problem where some libraries (e.g., encoding/json) have a global sync.Map that acts like a cache keyed on reflect.Type to some expensively computed data structure. The use of a global sync.Map or even a map[reflect.Type]V would keep dynamically created types alive forever (even if #28783 were fixed) unless there was some weak reference map that allowed for the removal of certain reflect.Type entries when they are GC'd.

This is a really interesting problem, I implemented my own version of a weak-reference using go 1.18 and will try to do the same for a weak map, but theoretically you could use this already for a weak map. I could as well change the implementation so that I do not "reattach" the finalizer, but let the reference die, but this way it works better, at least with the current GC. In any case, interesting topic.

I implemented my own version of a weak-reference using go 1.18

@xeus2001, for the reference, you might be interested to check how GC can crash when a pointer/interface is recreated from uintptr even if pointed object is known to be still alive and referenced via another regular pointer: https://github.com/golang/go/issues/41303#issuecomment-717199573.

I also have my own weakref implementation for go1.18 that seems to be doing fine, based on @xeus2001 's code but with a map object too.
@navytux I've looked at the https://github.com/golang/go/issues/41303 issue and it looks like to be an issue with using direct rematerialization of pointer through transforming uintptr[2] into interface{} which is likely not a good thing. Using unsafe.Pointer to store the object directly seems a better way to do it, and go1.18's generics makes the whole process a lot nicer.
I've done some load testing and so far I haven't had any crash, even with GOGC=0.

I implemented my own version of a weak-reference using go 1.18

@xeus2001, for the reference, you might be interested to check how GC can crash when a pointer/interface is recreated from uintptr even if pointed object is known to be still alive and referenced via another regular pointer: #41303 (comment).

@MagicalTux : I added a comment to the linked issue why I think that the implementation is buggy and what has to be done to fix it (adding a load barrier). However, you could directly try out my version, which uses two atomics where memory ordering is guaranteed (when I get the specs correct).

Tried my hand at a weakmap implementation here that uses a single persistent finalizer per non-empty map to automatically evict old entries based on memory pressure. The approach may solve the weak map for memory caching use case but not others.

I think with https://github.com/golang/go/issues/67552 accepted and implemented this is no longer relevant as anyone can implement those?

authors: DmitriyMV JAicewizard MagicalTux RLH aclements ammario bcmills dsnet ianlancetaylor ingvarm-gr josharian navytux rsc seebs xeus2001


text:
runtime: "fatal error: unexpected signal" 0xC0000005 on Windows during string operations

What version of Go are you using (go version)?

$ go version
go version go1.16.4 windows/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Administrator\AppData\Local\go-build
set GOENV=C:\Users\Administrator\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\Administrator\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Administrator\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.4
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Repos\github.com\VirtualMetric\virtualmetric-go\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\ADMINI~1\AppData\Local\Temp\2\go-build2746031420=/tmp/go-build -gno-record-gcc-switches

What did you do?
Our app makes many syscalls to collect different data for forensic analysis. Everytime when we got panic, it always happens in string operations (usually strconv.Itoa()) and always in "Remote Connection" analysis to detect remote attackers to pc. So right now, we disabled the remote connection analysis and we will test it. This issue usually happens after 12-24 hours. If issue would not happen, we will activate the remote connection analysis and disable Windows Defender to find out if Defender somehow crashing the app. App uses around 30-50 mb memory and there is always around 6 gb of free memory on the machine. Machine is running in Hyper-V virtualization as a VM. There is no dynamic memory or ballooning. All resources are allocated just for this VM. Here is where the issue happens:

// Remote Connection
remoteConn := model.HostRemoteConnection{
	SessionName:   "console-" + strconv.Itoa(session.Session),
	SessionID:     session.Session,
	Type:          session.Type,
	RemoteAddress: session.RemoteAddress,
	LocalPort:     session.LocalPort,
	Username:      session.Username,
	State:         session.State,
	CreationTime:  session.LogonTime,
}

strconv.Itoa(session.Session) panics.
What did you expect to see?
I would expect not to see any issue during the string operation or get an error in return.
What did you see instead?

C:\VirtualMetric>VirtualMetric.exe
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x1 addr=0xc00110b4d0 pc=0x103e77a]

goroutine 16 [running]:
runtime.throw(0x1a2250c, 0x2a)
        C:/Program Files/Go/src/runtime/panic.go:1117 +0x79 fp=0xc0002993c8 sp=0xc000299398 pc=0x106c3b9
runtime.sigpanic()
        C:/Program Files/Go/src/runtime/signal_windows.go:233 +0x317 fp=0xc000299420 sp=0xc0002993c8 pc=0x10823b7
runtime.mallocgc(0x9, 0x0, 0x2002d00, 0x1c1daaa0000)
        C:/Program Files/Go/src/runtime/malloc.go:1048 +0x33a fp=0xc0002994a8 sp=0xc000299420 pc=0x103e77a
runtime.rawstring(0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
        C:/Program Files/Go/src/runtime/string.go:263 +0x56 fp=0xc0002994d8 sp=0xc0002994a8 pc=0x1087396
runtime.rawstringtmp(0x0, 0x9, 0x1082898, 0x0, 0x0, 0xc000299590, 0x10f4b38)
        C:/Program Files/Go/src/runtime/string.go:131 +0x7c fp=0xc000299518 sp=0xc0002994d8 pc=0x1086d3c
runtime.concatstrings(0x0, 0xc0002995f8, 0x2, 0x2, 0x1b2eec0, 0x2002de0)
        C:/Program Files/Go/src/runtime/string.go:50 +0xc5 fp=0xc0002995b0 sp=0xc000299518 pc=0x1086785
runtime.concatstring2(0x0, 0x1a02bc1, 0x8, 0x1a1d12a, 0x1, 0x0, 0x0)
        C:/Program Files/Go/src/runtime/string.go:59 +0x4e fp=0xc0002995f0 sp=0xc0002995b0 pc=0x1086a0e
github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event.processUserRemoteConnection(0xc0000f6fd0, 0x1, 0x1, 0xc000299940, 0x0, 0x0, 0xc0002999c0)
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event/remoteconnection.go:191 +0x13f fp=0xc000299848 sp=0xc0002995f0 pc=0x163b2ff
github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event.remoteconnection.collect(0xc000029600, 0xc000366b40, 0xc000938060)
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event/remoteconnection.go:86 +0xfcd fp=0xc000299d88 sp=0xc000299848 pc=0x163a48d
github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event.(*remoteconnection).collect(0x208fd78, 0xc000029600, 0xc000366b40, 0xc000938060)
        :1 +0x57 fp=0xc000299db0 sp=0xc000299d88 pc=0x1651917
github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event.Collect(0xc000029600, 0xc000366b40, 0x0, 0x0)
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/event/event_controller.go:33 +0xf1 fp=0xc000299e20 sp=0xc000299db0 pc=0x1632d71
github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host.IIntegration.CollectEvent(...)
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host/main.go:28
github.com/VirtualMetric/virtualmetric-go/module/windows/integration/host.(*IIntegration).CollectEvent(0x208fd78, 0xc000029600, 0xc000366b40, 0xc000e84000, 0x18c2960)
        :1 +0x4d fp=0xc000299e50 sp=0xc000299e20 pc=0x16bcded
github.com/VirtualMetric/virtualmetric-go/module/windows/collector.event.processCollector(0xc000029600, 0xc000366b40)
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/collector/event_controller.go:80 +0x92 fp=0xc000299ec0 sp=0xc000299e50 pc=0x1845412
github.com/VirtualMetric/virtualmetric-go/module/windows/collector.event.startCollector(0xc000029600, 0xc000366b40, 0xc00033d7a0, 0xc000302a80)
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/collector/event_controller.go:61 +0x2b4 fp=0xc000299f88 sp=0xc000299ec0 pc=0x18450d4
github.com/VirtualMetric/virtualmetric-go/module/windows/collector.(*event).startCollector(0x208fd78, 0xc000029600, 0xc000366b40, 0xc00033d7a0, 0xc000302a80)
        :1 +0x65 fp=0xc000299fb8 sp=0xc000299f88 pc=0x1849f05
runtime.goexit()
        C:/Program Files/Go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000299fc0 sp=0xc000299fb8 pc=0x10a5361
created by github.com/VirtualMetric/virtualmetric-go/module/windows/collector.startSubCollectorTask
        C:/Repos/github.com/VirtualMetric/virtualmetric-go/module/windows/collector/collector_controller.go:82 +0x1a9

Thank you for raising this issue.  You mentioned that this issue occurs using strconv but the stack trace you supplied does not appear to include strconv.
This may be memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

Thank you so much for your reply @davecheney
We started the app with race detector and there are many issues and now we are solving them. I have one question about the race issues. Do you think memory corruption might happen after many data race issues? I see them in the race detector but app runs without issue. But I understand that, at some point, it will corrupt the memory?
App is still being in the test. If the panic happens, I will also share the latest data race issue.

Hi there,
Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.
For asking questions, see:

Stack Overflow with questions tagged "go"

The Go Forum, a web-based forum

Gophers Slack, use the invite app for access. The #general channel is a good starting point.

Go Community on Hashnode with questions and posts tagged with "go"

The golang-nuts mailing list

IRC channel #go-nuts on Freenode

Please ask the question on one of the above forums.
(Quoted from https://golang.org/wiki/Questions)

source: gharchive/issue

authors: davecheney heschi yusufozturk

repo: golang/go

license_type: permissive

license_source: github-api


text:
debug/gosym: race in LineTable.strings

What version of Go are you using (go version)?

$ go version
1.16.5

Does this issue reproduce with the latest release?
I have not tested 1.17 beta but the issue in the code is still present in the main branch.
What did you do?
The LineTable attribute strings is not guarded against concurent reads and writes nor is it documented. If the same *sym.Table is used by multiple go routines that calls the method PCToLin the process may crash due to a race condition.
Call flow: PCToLin -> go12PCToFile -> stringFrom -> read and write to map without mutex.
It looks like the same can happen to the other map funcNames.
Part of panic:
fatal error: concurrent map read and map write

goroutine 16 [running]:
runtime.throw(0x5a3bb9, 0x21)
	runtime/panic.go:1117 +0x72 fp=0xc0006d8c08 sp=0xc0006d8bd8 pc=0x434bd2
runtime.mapaccess2_fast32(0x56e8a0, 0xc0000faae0, 0x32, 0x3006cd968, 0x402364)
	runtime/map_fast32.go:61 +0x1ac fp=0xc0006d8c30 sp=0xc0006d8c08 pc=0x4115ec
debug/gosym.(*LineTable).stringFrom(0xc00007a000, 0xc0005d2960, 0xf74a0, 0xf74a0, 0x32, 0x4021b0, 0x48)
	debug/gosym/pclntab.go:344 +0x4a fp=0xc0006d8c80 sp=0xc0006d8c30 pc=0x4fc5aa
debug/gosym.(*LineTable).go12PCToFile(0xc00007a000, 0x4021b4, 0x0, 0x0)
	debug/gosym/pclntab.go:483 +0x371 fp=0xc0006d8d00 sp=0xc0006d8c80 pc=0x4fcf11
debug/gosym.(*Table).PCToLine(0xc0000be180, 0x4021b4, 0x0, 0x1, 0x48, 0xc0001803c0)
	debug/gosym/symtab.go:510 +0xc5 fp=0xc0006d8d50 sp=0xc0006d8d00 pc=0x4ff645

What did you expect to see?
No panic
What did you see instead?
Panic

Can you attach a program that reproduces this?
/cc @jeremyfaller @josharian

Here is the location in my project where it is triggered. It's a library for analyzing Go binaries (I use it for malware analysis) so a single program isn't enough. This file triggers it consistantly. If I analyze the binary using this code logic it panics (a tool using it):
% redress -src 03fcfe56348cc1ff272e340ea1f401feee1f4f4b9446fb05d65e9d58ecadbc78
fatal error: concurrent map read and map write

goroutine 12 [running]:
runtime.throw(0x11a367d, 0x21)
	runtime/panic.go:1117 +0x72 fp=0xc000038408 sp=0xc0000383d8 pc=0x1034452
runtime.mapaccess2_fast32(0x116e740, 0xc0000f8bd0, 0x32, 0x3012cf598, 0x402364)
	runtime/map_fast32.go:61 +0x1ac fp=0xc000038430 sp=0xc000038408 pc=0x101144c
debug/gosym.(*LineTable).stringFrom(0xc000078000, 0xc00055c960, 0xf74a0, 0xf74a0, 0x32, 0x4020d4, 0x4f)
	debug/gosym/pclntab.go:344 +0x4a fp=0xc000038480 sp=0xc000038430 pc=0x10fceca
debug/gosym.(*LineTable).go12PCToFile(0xc000078000, 0x4020d8, 0x0, 0x0)
	debug/gosym/pclntab.go:483 +0x371 fp=0xc000038500 sp=0xc000038480 pc=0x10fd831
debug/gosym.(*Table).PCToLine(0xc0000bc180, 0x4020d8, 0x0, 0x1, 0x4f, 0xc00010a3c0)
	debug/gosym/symtab.go:510 +0xc5 fp=0xc000038550 sp=0xc000038500 pc=0x10fff65
github.com/goretk/gore.findSourceLines(0x401d40, 0x402370, 0xc0000bc180, 0x1, 0x12)
	github.com/goretk/gore@v0.9.0-alpha3/file.go:188 +0xa9 fp=0xc0000385c0 sp=0xc000038550 pc=0x1131969
github.com/goretk/gore.(*GoFile).enumPackages.func1(0xc000016540, 0xc00008c180, 0xc00008c1e0, 0xc00008c240)
	github.com/goretk/gore@v0.9.0-alpha3/file.go:242 +0x145 fp=0xc0000387c0 sp=0xc0000385c0 pc=0x1141ca5
....

If I use a different version that use an older logic that doesn't use multiple go routines, it's fine:
% redress-legacy -src 03fcfe56348cc1ff272e340ea1f401feee1f4f4b9446fb05d65e9d58ecadbc78
Package github.com/GeertJohan/go%2erice: .
File: .
	findBox Lines: 28 to 136 (108)
	init Lines: 30 to 30 (0)
	(*File)Close Lines: 30 to 124 (94)
	init0 Lines: 32 to 125 (93)
	(*virtualFile)read Lines: 82 to 117 (35)
	glob.func1 Lines: 117 to 129 (12)
	(*File)Read Lines: 124 to 153 (29)
	(*appendedDirInfo)Name Lines: 125 to 128 (3)
	(*appendedDirInfo)Size Lines: 128 to 131 (3)
	(*appendedDirInfo)Mode Lines: 131 to 137 (6)
	(*Box)resolveAbsolutePathFromCaller Lines: 136 to 146 (10)
	(*appendedDirInfo)IsDir Lines: 137 to 138 (1)
	(*Box)resolveAbsolutePathFromWorkingDirectory Lines: 146 to 182 (36)
	(*File)Seek Lines: 153 to 170 (17)
	(*Box)Open Lines: 182 to 281 (99)
	(*Box)Bytes Lines: 281 to 284 (3)
....

One thing that I can tell is that the file path for all files appears to have been patched to "." which may be why it's crashing consistantly vs randomly.
Here is how it normaly should look like:
% redress -src /usr/local/bin/gofmt
Package main: /usr/local/Cellar/go/1.16.5/libexec/src/cmd/gofmt
File: <autogenerated>
	(*simplifier)Visit Lines: 1 to 1 (0)
File: gofmt.go
	init Lines: 29 to 53 (24)
	usage Lines: 64 to 76 (12)
	isGoFile Lines: 76 to 83 (7)
	processFile Lines: 83 to 166 (83)
	visitFile Lines: 166 to 182 (16)
	main Lines: 182 to 190 (8)
	gofmtMain Lines: 190 to 236 (46)
	diffWithReplaceTempFile Lines: 236 to 253 (17)
	replaceTempFilename Lines: 253 to 278 (25)
	backupFile Lines: 278 to 300 (22)
File: internal.go
	parse Lines: 23 to 94 (71)
	parsefunc1 Lines: 45 to 69 (24)
	parsefunc2 Lines: 69 to 80 (11)
	format Lines: 94 to 175 (81)
File: rewrite.go
	initRewrite Lines: 19 to 38 (19)
	initRewritefunc1 Lines: 31 to 64 (33)
	parseExpr Lines: 38 to 57 (19)
	rewriteFile Lines: 57 to 85 (28)
	rewriteFilefunc1 Lines: 64 to 90 (26)
	set Lines: 85 to 117 (32)
	setfunc1 Lines: 90 to 97 (7)
	apply Lines: 117 to 152 (35)
	isWildcard Lines: 152 to 160 (8)
	match Lines: 160 to 248 (88)
	subst Lines: 248 to 308 (60)
File: simplify.go
	simplifierVisit Lines: 15 to 130 (115)
	simplifiersimplifyLiteral Lines: 102 to 133 (31)
	simplify Lines: 133 to 141 (8)
	removeEmptyDeclGroups Lines: 141 to 152 (11)
	isEmpty Lines: 152 to 164 (12)


text:
Building the go binary with GOEXPERIMENT=boringcrypto shows security vulnerabilities on go 1.19+

What version of Go are you using (go version)?

$ go version
go 1.19+

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.19.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.19.2/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"

What did you do?

Build the go binary with flag GOEXPERIMENT=boringcrypto flag
CGO_ENABLED=0 GOOS=linux GOEXPERIMENT=boringcrypto go build -a -o bin/xxx xxx.go

What did you expect to see?
There should not be High security vulnerabilities on the image with respect to version previous to go 1.19+
What did you see instead?
High Sev security vulnerabilities which were related to prior version of go  1.16.1x and 1.17.x , 1.18.x etc are showing up.
https://nvd.nist.gov/vuln/detail/CVE-2022-23806
https://nvd.nist.gov/vuln/detail/CVE-2021-41772
https://nvd.nist.gov/vuln/detail/CVE-2021-44716
https://nvd.nist.gov/vuln/detail/CVE-2022-1705
https://nvd.nist.gov/vuln/detail/CVE-2022-1962
https://nvd.nist.gov/vuln/detail/CVE-2022-23772
https://nvd.nist.gov/vuln/detail/CVE-2022-23773
https://nvd.nist.gov/vuln/detail/CVE-2022-23806
https://nvd.nist.gov/vuln/detail/CVE-2022-24675
https://nvd.nist.gov/vuln/detail/CVE-2022-24921
https://nvd.nist.gov/vuln/detail/CVE-2022-27664
https://nvd.nist.gov/vuln/detail/CVE-2022-28131
https://nvd.nist.gov/vuln/detail/CVE-2022-28327
https://nvd.nist.gov/vuln/detail/CVE-2022-2879
https://nvd.nist.gov/vuln/detail/CVE-2022-2880
https://nvd.nist.gov/vuln/detail/CVE-2022-30580
https://nvd.nist.gov/vuln/detail/CVE-2022-30630
https://nvd.nist.gov/vuln/detail/CVE-2022-30631
https://nvd.nist.gov/vuln/detail/CVE-2022-30632
https://nvd.nist.gov/vuln/detail/CVE-2022-30633
https://nvd.nist.gov/vuln/detail/CVE-2022-30635
https://nvd.nist.gov/vuln/detail/CVE-2022-32148
https://nvd.nist.gov/vuln/detail/CVE-2022-32189
https://nvd.nist.gov/vuln/detail/CVE-2022-41715

how are you determining that there are these vulnerabilities?

Seems to be issue with our scan. Closing it. Thanks.


text:
access: plan9: add 9front developers to the plan 9 group

Right now, the only plan 9 maintainers are associated with 9legacy, which means that if the builders break, the 9front developers have no visibility into the breakage, which makes keeping up with fixes.
When talking about this at the runtime office hours, I was told that to get notifications, we should get the 9front developers into the 'plan9' group.
I think the people included should be myself (@oridb), moody (@majiru), and if they want, @psilva261.
I believe the groups should be https://github.com/orgs/golang/teams/port-maintainers, https://github.com/orgs/golang/teams/plan9 -- but both of those are 404s for me, so it's hard to say.
(I'm not sure if this is a proposal, but I didn't see a more appropriate option in the list)

cc @golang/release

authors: oridb seankhliao


text:
cmd/go: the -o option does not append .exe suffix to the compiled executable

What version of Go are you using (go version)?

$ go version
go version go1.20.3 windows/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Haoqing Chen\AppData\Local\go-build
set GOENV=C:\Users\Haoqing Chen\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\Haoqing Chen\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Haoqing Chen\go
set GOPRIVATE=
set GOPROXY=https://goproxy.cn,direct
set GOROOT=C:\Users\Haoqing Chen\scoop\apps\go\current
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\Haoqing Chen\scoop\apps\go\current\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.20.3
set GCCGO=gccgo
set GOAMD64=v3
set AR=ar
set CC=clang
set CXX=clang++
set CGO_ENABLED=0
set GOMOD=D:\projects\sing-box\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O3 -march=native -flto=thin
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O3 -march=native -flto=thin
set CGO_FFLAGS=-O2 -g
set CGO_LDFLAGS=-g -O3 -march=native -flto=thin
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\HAOQIN~1\AppData\Local\Temp\go-build2509538014=/tmp/go-build -gno-record-gcc-switches

What did you do?

Tried to compile any go program and specify an executable name by -o option, such as go build -o hello ./hello-world.go
What did you expect to see?
The executable name should be hello.exe. clang and gcc both add the .exe suffix automatically.
What did you see instead?
The executable name is simply hello, which will not be recognized by Windows as an executable file.

CC @bcmills, @matloob.

This was intentional behaviour https://github.com/golang/go/issues/3190#issuecomment-66064023

Yes, this looks like it's working as documented. The -o flag names a file, not a basename of the file.
Is there some reason you cannot pass -o hello.exe when compiling on Windows?

I think automagically appending file suffix in this case would not be ideal. I personally, would prefer to see the -o option output exactly what I provide it without having to account for arbitrary OS specific extension differences in behaviour. I think it's easy enough to just provide -o blah.exe as suggested by bcmills if you're working on Windows.
My two cents anyway.

Yes, this looks like it's working as documented. The -o flag names a file, not a basename of the file.
Is there some reason you cannot pass -o hello.exe when compiling on Windows?

Well, I personally feels a bit awkward to write separate Makefile targets for Windows and Unix-like systems that outputs a binary to a specific path. I feel like if clang and gcc does it on Windows, it is odd to not have the same behavior for go. After all having an executable that does not have an .exe suffix on Windows is pointless.

.exe is not the only possible extension on windows for executables, and it is possible to execute binaries without extensions at all.
As before, I don't think we want to be changing this as it will make the command less consistent and possibly break existing users.

authors: bcmills bshephar dmitshur seankhliao shanoaice


text:
cmd/go: go fmt aborts with operation not permitted when using mounted vmware shared folders file system

What version of Go are you using (go version)?

$ go version
go version go1.20.4 linux/amd64

Does this issue reproduce with the latest release?
Yes, version 1.20.4 is the latest release as of this writing
What operating system and processor architecture are you using (go env)?
Rocky Linux 8.7 (amd64) in a VM, running on top of VMware Workstation 17 Pro, on Windows 10
go env:
$ go env
GOARCH="amd64"
GOCACHE="/home/myuser/.cache/go-build"
GOENV="/home/myuser/.config/go/env"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOMODCACHE="/home/myuser/go/pkg/mod"
GOOS="linux"
GOPATH="/home/myuser/go"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVERSION="go1.20.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/mnt/hgfs/vmshare/golang/myapp/go.mod"
CGO_CFLAGS="-O2 -g"
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2445446007=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ ll
total 52
drwxrwxrwx. 1 root root  4096 May 16 08:56 cmd
-rwxrwxrwx. 1 root root   840 May 15 16:01 go.mod
-rwxrwxrwx. 1 root root 47337 May 15 16:01 go.sum
-rwxrwxrwx. 1 root root     0 May 15 16:01 LICENSE
-rwxrwxrwx. 1 root root   152 May 15 16:01 main.go

$ sudo chown myuser *.go
[sudo] password for myuser:

$ sudo chmod 750 *.go

$ ll
total 52
drwxrwxrwx. 1 root root  4096 May 16 08:56 cmd
-rwxrwxrwx. 1 root root   840 May 15 16:01 go.mod
-rwxrwxrwx. 1 root root 47337 May 15 16:01 go.sum
-rwxrwxrwx. 1 root root     0 May 15 16:01 LICENSE
-rwxrwxrwx. 1 root root   152 May 15 16:01 main.go

$ go fmt main.go
main.go
chmod ./main.go.2356257763: operation not permitted
exit status 2

Comments
The go fmt command above appears to create a temporary file in the current dir, and then apparently runs chmod on
that file.  It would seem the chmod should not be needed to gain access to the file by go fmt, so I presume the
purpose of the chmod was actually to impose access restriction to the temp file by other users/processes during the
formatting operation.
In my case the file system of the working dir is a mounted vmware shared folder.  This is a windows directory that
is shared with the linux VM by means of vmware tools.  From the perspective of linux, the file permissions on such a
mount will always be 777 and owned by root, as demonstrated by the ineffective chown and chmod commands above.  When
the go fmt command tries and fails to do a chmod on the temp file, it completely aborts the code formatting
operation.
Handling for this should be improved so that code formatting can still work when using various (less capable)
mounted file system types.  It may make sense to check the permissions of the temp file, and only call chmod if the
permissions on the temp file are actually less restrictive than the permissions on source file for example.  Another
option might be to attempt the chmod but not completely abort on failure, although the potential security
implications of that option should be considered.

cc @griesemer @mvdan

The chmod isn't to protect the file.  It's simply that go fmt creates a backup file before rewriting the original file.  It creates the backup file by creating a temporary file and then calling chmod to set the permissions of the temporary file to the permissions of the original file.  Then if rewriting the original file fails, it renames the backup file to the original file, so that the file is not lost.
It would be slightly more efficient and slightly more secure to create the backup file with the correct permissions in the first place, but the code just calls os.CreateTemp which doesn't have a way to specify the permissions.  If we did that, somehow, then I assume that this problem would go away.  We could also skip the chmod if the permissions happen to be exactly the same as the temporary file (0o600) but that is not particularly likely and is not the case in your example.

authors: heschi ianlancetaylor mdepot


text:
x/vuln/cmd/govulncheck: output a final result in json mode

What version of Go are you using (go version)?

# govulncheck  --version
Go: go1.21.0
Scanner: govulncheck@v1.0.1
DB: https://vuln.go.dev
DB updated: 2023-08-23 14:38:50 +0000 UTC

What operating system and processor architecture are you using (go env)?
go env Output
# go env
GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://goproxy.cn,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/root/tmp/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build351832300=/tmp/go-build -gno-record-gcc-switches'

What did you do?
执行了以下命令检查漏洞
# govulncheck -json ./...
What did you expect to see?
{
"config": {
"protocol_version": "v1.0.0",
"scanner_name": "govulncheck",
"scanner_version": "v1.0.1",
"db": "https://vuln.go.dev",
"db_last_modified": "2023-08-23T14:38:50Z",
"go_version": "go1.21.0",
"scan_level": "symbol"
}
}
{
"progress": {
"message": "Scanning your code and 48 packages across 1 dependent module for known vulnerabilities..."
}
}
{
"result": {
"status":  true
}
}
What did you see instead?
{
"config": {
"protocol_version": "v1.0.0",
"scanner_name": "govulncheck",
"scanner_version": "v1.0.1",
"db": "https://vuln.go.dev",
"db_last_modified": "2023-08-23T14:38:50Z",
"go_version": "go1.21.0",
"scan_level": "symbol"
}
}
{
"progress": {
"message": "Scanning your code and 48 packages across 1 dependent module for known vulnerabilities..."
}
}

What would be the easiest way for us to reproduce this?

It is also not clear what is being suggested here. The govulncheck command provides streaming json containing the result section. Could it just be that govulncheck did not find anything for your project?

Use the command 'govulncheck-json. /... 'Reproduce.

It is hoped that a key value in the json result of govulncheck can clearly indicate whether there is any vulnerability in the detection result. The current version does not have this key/value, so it cannot be integrated into CI for automation.

govulncheck supports streaming json where typically one will see a config section followed by a progress section, followed by findings. (More info also here). If there are no findings, that means nothing has been found. It is specifically designed for CI automation where the results are communicated as soon as they are computed (hence streaming).
Closing.

Like the following? If the finding property is in, there is a vulnerability; otherwise, there is no vulnerability
root@Ubuntu ~/tmp# govulncheck  -json ./... |jq ".finding" null null null {   "osv": "GO-2021-0113",   "fixed_version": "v0.3.7",   "trace": [     {       "module": "golang.org/x/text",       "version": "v0.3.5",       "package": "golang.org/x/text/language",       "function": "Parse"     },     {       "module": "tmp",       "package": "tmp",       "function": "main",       "position": {         "filename": "/root/tmp/g1.go",         "offset": 189,         "line": 12,         "column": 43       }     }   ] } null {   "osv": "GO-2022-1059",   "fixed_version": "v0.3.8",   "trace": [     {       "module": "golang.org/x/text",       "version": "v0.3.5",       "package": "golang.org/x/text/language"     }   ] }

Yes, and you can have multiple findings.
Each finding will give you the information needed to interpret it. Above, you can see that you have a trace of multiple frames showing you the call stack. That means that the vulnerability was called and the function in the first frame is the vulnerable symbol.
If the trace has only one entry whose function is empty, that means that the vulnerability is only imported.
We plan to write up more detailed documentation about json output.

ok, thanks

authors: andy-github-info zpavlinovic


text:
crypto/x509/pkix does parse "1.2.840.113549.1.9.1" (emailAddress) in Subject/Issuer data

What version of Go are you using (go version)?

$ go version
go version go1.21.1 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/sweh/.cache/go-build'
GOENV='/home/sweh/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/sweh/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/sweh/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go_versions/go1.21.1'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go_versions/go1.21.1/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/gobuild1494015273=/tmp/go-build -gno-record-gcc-switches'

What did you do?

My organisation uses Active Directory Cert Services for cert issuance.   The root CA and intermediate certs have Subject and Issuer entries with emailAddress=support@team type data in it as an extra field
e.g.
C=US,ST=...,L=...,O=...,OU=...,CN=...,emailAddress=support@team
This email address appears to be created with the (deprecated) 1.2.840.113549.1.9.1 oid.
When parsing this information (e.g from a client cert passed to a server as a mTLS session) this attribute is not decoded and is left as a BER encoded oid field.
What did you expect to see?
CN=...,OU=...,O=...,L=...,ST=...,C=...,emailAddress=support@team
What did you see instead?
CN=...,OU=...,O=...,L=...,ST=...,C=...,1.2.840.113549.1.9.1=#berdata
Possible fix
For testing I was able to add
"1.2.840.113549.1.9.1": "emailAddress",
to attributeTypeNames in crypto/x509/pkix.go and that was sufficient to make my certs display properly.  I don't know if this is sufficient, though, so I'm not proposing it with a PR.
It's also possible ADCS is doing the wrong thing, here, but since it's a common tool for generating certificates it probably needs to be handled gracefully.

related to #33093, which causes the unrecognized DN to render as hex instead of text

@FiloSottile @rolandshoemaker @golang/security per owners

authors: rittneje sweharris thanm