text:
RFC for a Rust Memory Model

with thanks to Amanieu, huonw, durka42, aatch, acrichto, nmatsakis, and anyone
else I might have missed.
Fixes, or at least starts the process of fixing, #1447

cc @nikomatsakis @RalfJung @arielb1 @pnkfelix @Amanieu

This opens up the possibility of performing TBAA on types which have implementation-defined bytes since these types are only allowed to be read through a pointer of the correct type. I don't really see this as a big issue since this is only possible from unsafe code and the struct layout isn't guaranteed to match any other type.

It's not that they're only allowed to be read through a pointer of a correct type. It's that if they are then used, it's undefined behavior for them to be aliased.

Enthusiasm is commendable, but I'd prefer this particular work to be done by professionals.

I agree with @petrochenkov, there are several shortfallings in this model.

@Ticki
I'm not talking about concrete shortcomings, but about general approach.
I'd expect this to be a published academic paper by @RalfJung or someone else, with level appropriate for such papers, overview of prior art etc.
Anything less, e.g. "obvious" useful guarantees on struct layout or valid values of primitive types, can be specified on case by case basis.

I'm not talking about concrete shortcomings, but about general approach.

Exactly. It should be formalized (to avoid these ambigiuities). It is very vague for a memory model, as is.

@Ticki @petrochenkov Honestly, this has been simplified and such a lot from how it would appear originally, in order to fit in the RFC format. Being a "professional" also has very little to do with the work here; I've been thinking about this memory model for months, and I'm likely one of the experts on Rust's. People send people to me to explain it. And, I get it; you would be more comfortable if the memory model was written by a professional. But I'm not a professional, and I wrote (part of) a memory model (the important part, the pointer aliasing part). The other things are just a basis for pointer aliasing, which is what really needs to be defined.

@usban Don't get me wrong. I don't doubt that you know a lot about this subject, and have put a lot of effort into it. What I am saying is just that this wasn't really what was inteded with a memory model. What was inteded is a formalized model, instead.

@Ticki well, I am working on formalizing the model now... I just wanted to talk about it first, in public. It's important that we are all able to give feedback on it.

The fact is, anything is better than nothing, and nothing is exactly what Rust has right now. People are clueless as to the precise rules when it comes to pointer aliasing, which aren't currently documented anywhere.

This RFC reads to me as if there are two different operations that are both called ptr::read, but follow different rules (based on whether the argument is a pointer or reference).
In fact, if I see it correctly, replacing ptr::read(int_ref as *const i32) with *int_ref introduces the side-effect of making other pointers derived from that reference non-derived. Which means one can probably build programs where this harmless-looking code change introduces undefined behavior?

I am not gonna weigh in on this RFC very much, but one comment:

The fact is, anything is better than nothing,

When it comes to guarantees that you're making, nothing can often be better. If we were to adopt a poor model, it could inhibit Rust for the rest of its life. You really want something like this to be absolutely foolproof, or as close to it as possible. Adopting something that's not ready because "something is better than nothing" is not a good idea here.

For the moment, then, can we at least have some documentation that pointer aliasing of any form is Undefined Behavior? Because that's what it really is.

@archshift This is, in fact, true currently.

@steveklabnik One thing I found really compelling was something that dikaiosune said in IRC:
"is it just me, or is waiting for 5 or more years for any guarantees about unsafe pointers kinda untenable for rust adoption?"
Currently, this:
let mut x = 0;
let ptr1 = &mut x as *mut i32;
let ptr2 = &mut x as *mut i32;
*ptr1 = 1;
*ptr2 = 2;
*ptr1

is completely undefined. And most unsafe code assumes it is.

I'd expect this to be a published academic paper by @RalfJung or someone else, with level appropriate for such papers, overview of prior art etc.

Just to be clear here, I am not right now working on a memory model for Rust. Nobody should feel blocked by me here. For my Rust formalization I decided to shortcut this discussion, and instead define too many behaviors. This means that at least I can prove the desired libraries to be safe in my model (hopefully), but they may still turn out to rely on behavior that Rust wishes not to define.

@ubsan Even that is unclear, as according to the Rust Doc:

* pointers are allowed to alias, allowing them to be used to write shared-ownership types, and even thread-safe shared memory types

[Raw pointers] have no guarantees about aliasing or mutability other than mutation not being allowed directly through a *const T.

@archshift Exactly. It's completely undefined.

Since @ubsan pinged me here, I'd like to share some of my thoughts on the level of formalism or academic chops present in the RFC.
One of the things that most impresses me about and attracts me to open source in general (and to a greater degree, Rust specifically) is the power that "expert amateurs" can have when they ignore credentials and focus on a substantive problem, emphasizing collaboration and results over social status and signalling. In many cases when one lacks the time, specific knowledge, or some other resource to substantively critique a work, the easiest signal to process can be from a contributor's credentials and prior achievements, or from the format of the presentation. But I think that's a bad trap for an inclusive community like Rust's to fall into.
I have always been impressed by the pragmatism in the other discussions I've read here, and I think it's a distraction to focus (explicitly or implicitly) on ubsan's qualifications or the current level of formalism of the RFC, especially when its author has approached it in this way:

I am working on formalizing the model now... I just wanted to talk about it first, in public. It's important that we are all able to give feedback on it.

I don't see any reason to reject this out of hand. The product of this work will be very important for Rust's community and its future, and I think it deserves concrete feedback and a substantive appraisal.

I have to disagree with @steveklabnik on this one. In this case something is better than nothing, as long as we make sure that the guarantees are not going to get broken in the future.
Personally, I would prefer an access based model over the one presented here. I am using an access based model in my formal verification of Redox. I have outlined my idea below:
(unfortunately, github doesn't support Latex, so I have rendered it as images)

and formalized through:

I have to disagree with @steveklabnik on this one. In this case something is better than nothing, as long as we make sure that the guarantees are not going to get broken in the future.

I think we're on the same page. It's not so much that it has to be absolutely, fully, 100% perfect at first blush, just that we're not painted into a corner.

@steveklabnik, so what you are saying is that you think it should rather make one promise less than one promise more?

@Ticki I'm not familiar with that idiom. But I don't mean "something is worse than nothing" in the sense that it's an all-or-nothing enterprise. But I wouldn't want to rush to get something down just to have some kind of guarantee, only to find out that we're not actually happy with it later.
Anyway, I said I wasn't going to say much, and now I've made three posts, so I'll stop :wink:

Clearly we must aspire to the level of clarity and precision provided by the gold standard of standards, C. [1] [2] [3] [4] [5] [6] [7]

let mut x = 0;
let ptr1 = &mut x as *mut i32;
let ptr2 = &mut x as *mut i32;
*ptr1 = 1;
*ptr2 = 2;
*ptr1

This will actually work on every model that we plan to use. It is undefined in the exact same sense as everything else.
What is more dubious, is for example
#[derive(Debug)] struct Foo<T>(T);
fn main() {
    let five = 5;
    let five_ref = &five;
    let mut x = Some(Foo(five_ref));
    let x_addr = &mut x as *mut Option<_>;
    match x {
        x_copy => {
            if let Some(ref inner) = x_copy {
                unsafe { *x_addr = None; }
                println!("{:?}", inner);
            }
        }
    }
}

This generates code that crashes on current rustc, through I don't think it should.
In any case, I don't like the derived pointer model because it is fundamentally based on "confidentality", while Rust basically tries to guarantee "integrity".

Additionally, LLVM likes attributes on loads, so lvalue-to-rvalue-conversions require the rvalue to be valid.

@arielb1 I don't think I understand what you are saying about confidentiality versus integrity. Or about lvalue-to-rvalue conversion attributes. Which specific attributes are you talking about, which would not be allowed?

@arielb1 I think crashing makes perfect sense though in this case. Moving a value is only semantic, it doesn't necessarily need to be given a new address, and for performance reasons often shouldn't.

Yeah this needs to wait for formal verification. I don't even trust a formal model that doesn't have a proven implementation to go with it. [A formal model is definitely first step to a proven implementation, but absent the proven implementation demonstrating the model's assumptions are valid, there's little use it "ratifying" the formal model.]

I'll note that I specifically find the approach of specifying it in plain English unconvincing - even aside from the compelling  evidence that doing so is a strategy that consistently fails to provide results that are worth the effort[1][2][3], the simple fact of the matter is that English is a lot easier to mistranslate or misconstrue than a formal spec. "Plain English" is, to be quite honest, anything but - and I say this as a native speaker.
As a result, I have a very strong preference for a canonical formal spec paired with an explanatory document, rather than a canonical English spec that will hopefully one day be formalized.
[1]: Java's concurrency model first being unimplementable and then forbidding CSE
[2]: C11's memory model forbidding common optimizations, amongst other issues
[3]: The heroic efforts needed to forumulate a memory model for x86 given prior plain-English descriptions

@eternaleye I understand that. I'm working on a formalization currently, but again, I wanted to talk about it in the open first to allow people to comment on it. This is a "Request For Comments".

@ubsan To clarify, by a formal spec, I mean one suitable for machine-checking - the borrow checker and type checker are essential to safe Rust because humans cannot reliably model all the concurrency present in any given program; as a memory model must be well-defined for all concurrency possible in any program, the full verification of the model is in my opinion at least as crucial as borrow and type checking in safe Rust.

Would it be possible to have a memory model without Undefined Behaviour?

Sure. "Exactly what Rust does right now" is a fully-defined memory model, albeit a rather useless one. You could also get rid of UB systematically.
But ultimately you need undefined behavior to give the optimizer room to maneuver. Rust protects you from undefined behavior within safe code, so it's not as much a problem as it is in other languages. You just need to worry about it when writing unsafe code or doing FFI.

@chrisvest even without "launch the nukes" undefined behavior, there are unavoidably plenty of non-deterministic machine programs due to the nature of multi processor computing. Yes, languages should be defined independent of implementation, but many features of Rust we care about only because they ensure the image of compiling avoids the peskier of those non-deterministic programs. So to prove Rust does we want, we need to model that non-determinism.

@Manishearth There's some room to debate how much UB-based optimization actually benefits optimization by compilers, especially in the broader context of what changes it forces programmers to make in order to cope.

I am quite grateful to @ubsan for writing this up.
I too think that we will require a machine-formalized model of some form.
But we can only benefit from having a variety of well-written documents to draw from in order to understand the design space and trade-offs here.
It is good to have dialogue about the particulars of a model like this.
And it is much easier to read an RFC than to trudge through the various github issues trying to digest the many threads of conversation regarding memory models. (Issue #1447 could be a central repository of knowledge and ideas on the topic, but we will require reference documents that summarize the information there.)

To summarize some discussion that occurred on #rust-offtopic (@ubsan, @eddyb, correct me if I misconstrue or miss something):

There exist formalized 'weak' CPU memory models that we can use to bound us on one side
Can shove libcore and libstd through miri and permute by model-permitted transformations to bound on the other side by seeing if anything goes splat

References:
[1]: A formal hierarchy of weak memory models
[2]: [Other work by the author](http://www0.cs.ucl.ac.uk/staff/J.Alglave/, "herd" and "goto-instrument" likely to be of particular interest (simulating CPU memory models and a similar approach to the miri proposal applied to C, respectively)
[2]: Software verification for weak memory (has source for goto-instrument, along with coq proofs, at the bottom)

@eternaleye The advantage of miri is that with some FFI support, it won't be limited to libcore and libstd but could run even (parts of) Servo (the use of IPC should make that easier).

@eddyb The main reason I'd rather stick to libcore and libstd for now is to try and keep the model small - more guarantees can be added later, but by focusing on something minimally-viable (possibly by also altering libcore and libstd to ask less of the model in tandem) we can have something small and easy to reason about while being able to extend it to be stronger in the future.

@eternaleye
If we have a model, we want to check all unsafe code we can to verify that it follows the model.
@ubsan
Maybe I did not understand LLVM metadata correctly, but we occasionally emit assume/metadata on loads that the loaded rvalues are valid. For example:
pub unsafe fn foo(zero_ref: &usize) -> &'static str {
    let do_ub = *(zero_ref as *const usize as *const &());
    let do_ub_ref : *const &() = &do_ub;
    if *(do_ub_ref as *const usize) == 0 {
        "zero"
    } else {
        "nonzero"
    }
}

fn main() {
    println!("{}", unsafe { foo(&11) });
    println!("{}", unsafe { foo(&0) });
}

@arielb1
Correct. That's a use of an "Invalid" value; when you did do_ub_ref as *const usize, and then loaded it, and then did == 0, you used a value derived from an "Invalid" value, which is undefined behavior.
I need to specify this further. This is all in my head and not on the page.
@eternaleye I'm using coq right now, although I'm not very far.

LLVM's docs are really unclear as for whether loads require that the loaded value is valid. However, they assume that loads of dereferencable things are pure, so I guess that seals the deal.
So we probably can have poison rvalues like LLVM. I don't want to introduce the poison/undef distinction unless I am forced to, because nobody really understands undef, but I'm fine with just copying poison values around.
On the other hand, I am not sure how useful is that - what you want to do with poison values is either to have direct let foo = mem::uninitialized() or to put them in constructors, but clearly you can't put them in enum constructors. It probably makes sense to treat struct constructors as different from enum constructors here.

The reason I don't like derived-based analysis is that if you want to reason in a modular manner you have to play with "can-be-derived", which is very annoying to deal with.

@arielb1 The idea is that there is no can-be-derived. You know exactly which are derived, and which are not derived. It's a strict, at each point in the program, this pointer is either derived from or not derived from another pointer. There is no "can-be-derived", if I understand you.

@ubsan
I understand that at runtime the derived relation is definite. On the other hand, if you want to reason about the validity of optimizations in the presence of external functions, determining whether one pointer is derived from another is non-trivial.
Also, derived pointers are somewhat similar to C++'s consume memory model, which is a thing I want to stay away from (but maybe there is some significant difference?)

@arielb1 Function arguments are one area where derived pointer relations are cut off entirely. A move is counted as a rederivation, and you must move a reference into a function in order to pass the argument; therefore, each new function gets a "clean slate", as it were.
I don't know what you mean by "consume memory model". Could you give an example?

This looks like a very good start, but it is important to note that, so far, this RFC is only a memory model for a single thread. I don't necessarily think that concurrency should be a blocking issue, as even just a single-threaded memory model clears up a lot of questions about the correctness of unsafe code. However, the RFC should explicitly state that it leaves the details with respect to concurrency unspecified, and should eventually be extended to support concurrency.

@ubsan
I meant memory_order_consume.

@ubsan
I meant memory_order_consume.

@arielb1 I don't understand what you are saying. Could you explain it out, please?

I would like to reword the "using a value" paragraph to be more in line with the MIR nomenclature.
I am using pseudo-Rust as my language for data declarations because it is the most familiar for the people reading this.
Type Representation Kinds
At run-time, every type is eqty to a single unique concrete type, that's it, a type without type parameters and projections (FIXME projections within binders). Concrete types can be divided as follows:

A primitive type is a type that is treated as a set of bits. This is integers, floats, and raw pointers.
A scalar type is a type with an atomic representation - primitive types, plus bool, char, references, function pointers, and FnDef markers
A struct-like type is a struct, tuple, or closure.
An array type is a slice or array
A enum-like type is an enum
A universal type is a trait object

NOTE: should C-like enums be scalars?
Values
MIR Rvalues evaluate to a value, which is defined as follows:
pub enum Value {
    Poison,
    Scalar(Vec<u8>),
    Struct(Vec<Value>),
    Array(Vec<Value>),
    Enum(Discr, Vec<Value>),
}

impl Value {
    fn contains_poison(&self) {
        match self {
	    &Poison => true,
	    &Scalar(..) => false, // NOTE: don't look through pointers
	    &Struct(ref vs) | &Array(ref vs) | &Enum(_, ref vs) => {
	        vs.iter().any(|v| v.contains_poison())
	    }
	}
    }
}

NOTE: Do we want to track pointers as separate from integers? This is
required for const-eval and derivation-path.
Non-Poison values must conform to their type representation. This means that:

primitives must be a Scalar of the right size
bool and char must be valid Scalars
pointers and function pointers must be non-null. [0]
enums must have a valid discriminant, and must not contain a Poison.
struct-likes, arrays and enums must have the correct number of fields, and they all must all conform to their representation.
a NonZero struct must contain a non-zero Scalar

If an operation would create a value that does not conform to its representation, a Poison value is created instead.
When a function returns, its return pointer must not contain a Poison. Also, unless otherwise specified, operations on rvalues invoke undefined behavior if one of their inputs contains a Poison. Notable exceptions are:

StatementKind::Assign: writes poison bytes to the memory ranges covered by poison.
Rvalue::Aggregate and Rvalue::Repeat: these operate normally, creating a Value that contains a Poison but that is of the correct constructor. Note that the rule about values having to conform to their representation applies - an enum that contains a poison will be immediately converted to a Poison.

[0] This prohibits using the !dereferenceable or !align metadata on loads of pointers. We currently don't do so but we may want to in the future. In any case, we want this pattern to be a GEPi with no extra requirements:
fn foo(random: *const Foo) -> *const u8 {
    unsafe { &((&*random).1) }
}

or in MIR
let arg0: *const Foo;
let tmp0: &Foo;
let tmp1: &u8;

begin:
    tmp0 = &*arg0;
    tmp1 = &(*tmp0).1;
    return = tmp0 as *const u8 (Misc);
    return;

Hello all. I want to weigh in briefly. I am still digesting the RFC text and the comment thread, but I want to make some kind of "meta" comments.
Generalities
First, like @pnkfelix, I certainly would like to thank @ubsan for putting energy and effort into a "first draft" at some sort of memory model for Rust. Whatever model we eventually adopt, it's great to make forward progress of some kind.
I think the best place for us to focus our efforts at this stage is not on determining the precise model we will use, but rather the process by which we will arrive at a model, and the high-level principles that will guide us in this effort.
I think the process should be as top-down as possible. Basically I'd like to start just by agreeing on the kinds of examples we think are interesting and what makes them interesting. (More on this in a bit.) From there maybe we can come up with high-level rules that "summarize" the key ideas, and then drill down into making them more concrete. One nice part about this is that at each stage of the process we are giving more information to end users about what kinds of code we expect to be legal and illegal. Honestly, I hope that the list of examples alone will be sufficient advice for the most part.
An ideal end-point would be to arrive at something testable, for sure. The idea of an instrumented miri-like interpreter is particularly appealing. I would like to be able to "test" our proposed models by doing "crater runs" across crates.io and seeing what kind of code reports violations.
Now, as far as guiding principles go, I would like to see a model that aims to avoid crashes in the code that naive people write. Put another way, I expect that most programmers will either not read the memory at all, or will only read no more than a few paragraphs. I would like that, given that level of innattention, they still mostly do things right when they write unsafe code. Effectively this means the model must be "intuitive"; that is, whatever model we adapt, it has to permit the kinds of tricks that people pull in practice. I think that many models I have seen fail this criteria dramatically. The problem with an "intuitive" (or perhaps "permissive" is a better word) model is that it naturally runs somewhat counter to the ability to optimize. I think that Rust has a leg up here, though, both because of the nature of our type system and because of the safe/unsafe code division.
It is not clear that we can achieve a model that is both permissive and permits a sufficient level of optimization. But let's posit that we can for a second (and I do believe it is possible). In that case, it's interesting to note that the urgency of having a formal memory model is somewhat reduced, in that we expect most things people do in practice to wind up being sound.
Examples and repo
I mentioned that I thought we ought to start by agreeing upon a set of interesting examples and so forth. I've been making an effort to collect such a set, though I think it's still in relatively early stages (and I haven't had much time for this in the last month or so). You can see my current results here.
There are still a number of things I want to do:

Add some examples about when it is safe to "reuse" a lot; as @arielb1's example earlier demonstrates, this is not always clear).
Consolidate and clean up the examples, which I think contain duplication now, since I mostly just copied things I found on various threads.
Go through unsafe code and extract interesting patterns. I was planning to start with libstd.

I'd very much welcome contributions.
I've been hoping that once we have a pretty good set of examples etc, we can (a) consolidate and clean them up and then (b) use them as a basis for comparing proposals.
Specific responses
Sorry this comment doesn't have much in way of specific responses to this RFC or comments in the thread. It's long enough as is, I guess. I've been taking notes on the RFC as I read as well as other comments on the thread, and I'll try to get back and respond to more specific details there as well.

@nikomatsakis
I've been thinking about this a lot since then. I think this is a good start for a memory model, but I didn't understand optimizations well enough at that point. Yeah, I do believe it's a good idea to use Miri to find undefined behavior.
What I've been thinking about is this:
Unfortunately, I think that any use of an uninitialized value should be undefined behavior. On some architectures, even moving from an uninitialized register to another register will be bad news, due to flags. I think we might have to adopt the C standards ideas of u8/i8 is special; it can be undef, and nothing else. The only thing you can do with an undef variable is write to it, or take a pointer to it.
Our references are dereferenceable, which is good for optimization purposes, but which also means; you can't have a reference to an undef, in case you accidentally read an undef. Perhaps something like, the only thing you can do with a reference to an undef is 1) write through it, or 2) create a pointer from it?

@ubsan: Doesn't that then require the optimizer to be conservative with any reference it cannot prove is to a defined value? I'd much prefer saying that pointer-to-reference coercions are UB if the pointee is an undefined value, and use ptr::write() for the "write through" part.

@eternaleye No, the optimizer assumes that all references it doesn't know are to a defined value, are to a defined value (unless that reference is a reference to a byte, or a byte array). The following are the only times a reference to an undefined is okay, I think:
let mut x = std::mem::uninitialized();
let ptr = &mut x;
write(ptr, ...);
// or
let mut x = std::mem::uninitialized();
let ptr = &mut x as *mut ...;

@ubsan: Hm, I see. Perhaps a combination, by saying that the only valid op on ref-to-undef is immediate conversion to pointer, something trivially lintable. Note that I would also lint returning the ref.
It can't be an error (back-compat), but IMO ref-to-undef is an unequivocal error, and the first form should habe been disallowed. :(

@eternaleye Perhaps. this is far easier to deal with, however; just moving a reference to undef is undefined behavior.

@ubsan: If a reference to undef existing any more than ephemerally is UB, all timelines in which such a move occurs already encountered UB at an earlier point.
Consider:
unsafe fn foo(x: &mut T) {
    *x = mem::uninitialized();
}

This function should lint, IMO, merely by existing. "Escape of ref to undef" permits that.

@eternaleye It makes sense to lint, because you're moving uninitialized data.
let x = std::mem::uninitialized();

is the only way to get uninitialized data, I think... That would be equivalent to something like
unsafe fn foo(x: &mut T) {
  let uninit = mem::uninitialized();
  *x = uninit;
}

Or... = std::mem::uninitialized() could be defined as a no-op.
fn foo(x: &mut T) {
  *x = mem::uninitialized();
}

could be equivalent to
fn foo(x: &mut T) {
}

Mm, right - forgot the initializer/move distinction there. Still, I think losing the dereferenceable attribute on references is a bad trade - and if moving undef is UB, then proving refs to undef are immediately converted to pointers is an entirely tractable analysis AFAICT.

@eternaleye The point is there is no losing the dereferenceable attribute on references. Once you move a reference to an undef, it's completely dereferenceable.

Mm, I think we may actually being violent agreement, but were separated by terms. As 'escape' requires a move one way or another (assign or return). Sorry about that.
I do think the non-escaped ref-of-undef should only permit pointer converstion and not writes, though - we have ptr::write() already.

@eternaleye Yeah, I can agree with that.
I do like the idea of = std::mem::uninitialized() being a no-op in the future. Although... we definitely use undefined behavior with our current implementation if there are any trap representations. If we changed std::mem::uninitialized to pub use std::intrinsics::uninit as uninitialized, and just made any use of intriniscs::uninit on the rhs of an = a no-op.
This would make:
<expr> = intrinsics::uninit(); into <expr>;

Does LLVM dereferencable require that the value behind it is valid? I thought that it only requires that it can be read without triggering a page fault (i.e., it does not even guarantee that the value is not modified by other threads).
The other kinds of load metadata (e.g. nonnull) seem to turn invalid data loaded to undef/poison as I specified. It would be better if that was documented somewhere. LLVM has no problem moving undefs and even storing them to memory - it even performs optimizations that rely on that.

@arielb1 The issue is that, at least in my opinion, we shouldn't be using LLVM as a benchmark. I'm of the opinion that we should allow implementations almost as much wiggle room for trap representations as C does, perhaps excluding integer types. But, for example, if a computer traps on passing sNaN, that should be allowed behavior, or if a computer traps if you read from a register which has NaT set.

I thought you were for allowing the loading of undef rvalues?
The issue here is that we want box mem::uninitialized() and the likes to work, and that desugars into something of the form
        tmp0 = Box::new();
        (*tmp0) = std::mem::uninitialized::<u32>() -> [return: bb3, unwind: bb2];

Should we say that "RVO" writes of undef are ok?
In any case, I heard that people want to create structs with only some of their fields uninitialized, which currently generates e.g.
    start:
        tmp0 = Box::<Big>();
        tmp2 = std::mem::uninitialized::<[u32; 65536]>() uwto unwind0
        (*tmp0) = Big { tag: const 42u8, data: tmp2 };
        return = tmp0;
        return;

    unwind0:
        tmp1 = alloc::heap::box_free::<Big>(tmp0);
        resume; // Scope(0) at <anon>:11:1: 13:2

Source code for ^:
fn make() -> Box<Big> {
    unsafe { box Big { tag: 0, data: mem::uninitialized() } }
}

@arielb1 I thought they were. Then I had a big dose of reality with CPUs that weren't x86 or ARM:
https://blogs.msdn.microsoft.com/oldnewthing/20040119-00/?p=41003/
We may have to use (something like) the C/C++ definition of uninitialized:
An uninitialized variable may either be in a valid, but unspecified state, or a trap representation (and it may change whenever you read it). The only thing that's not allowed to have a trap representation is i8/u8. If you do anything except write to a variable with trap representation, UB happens.

So, for rustc on x86 and ARM, for example, the integer and pointer types would have no trap representations.

@arielb1 It doesn't really matter how we translate it, just how we define it; box mem::uninitialized::<T>() should be equivalent to Box::from_raw(allocate(size_of::<T>(), align_of::<T>()) as *mut T).

@ubsan: And placement box with mem::uninitialized::<T>() would simply skip initialization (i.e. noop)?

@eternaleye That's the idea.

@ubsan I'm interested in hearing your response to the comment I wrote almost a month ago. I wonder what your thoughts are?

@ticki 1) you don't account for shared mutable state, which is allowed for with both raw pointers, and with UnsafeCell, afaict, 2) What can you do with uninitialized variables besides write to them? 3) how is dropping a reference defined? I imagine something like "last use".
Seems good otherwise. Sorry, it kinda got missed.

I do. See point 1.:

mutating a non-aliased value [..]

Or do you mean mutating while having other (passive) pointers accesible?

Well, reading an uninitialized value will always be unsafe, writing or aliasing one is valid of course.

See point 3., basically the "hidden counter" is decremented. From a technical point of view, this has the effect of changing the type state, potentially allowing future mutation.

Fair point, but the idea is the same: the program's behavior is well defined.

Sorry, it kinda got missed.

np.

@ticki 1) Yes, but mutating an aliased value is defined, i.e. in the case of Cell. 2) Reading an uninitialized value may be unsafe, but when is it invalid (for example, i8/u8 should always be valid, other types may be valid/invalid). 3) When is a reference dropped, not what happens when a reference is dropped.

Reading an uninitialized u8 is undefined in LLVM. Despite the types having no inconsistent representation, they cannot be read uninitialized, since the behavior would be defined by how the stack frame was layed out.

Reading an uninitialized u8 is undefined in LLVM. Despite the types having no inconsistent representation, they cannot be read uninitialized, since the behavior would be defined by how the stack frame was layed out.

@ticki It results in an uninitialized value. It does not result in undefined behavior (and in fact, this is true of all integer types in LLVM).

On 3), the reference is dropped when it is inaccesible or dead.

@ticki "Inaccessible" and "dead" are two very different things, and could be defined in multiple ways. Please give examples?

Dead means that it won't be used at some later point in execution. Inaccessible means that it is out of scope or otherwise not possible to read or write.

Got it. I believe we want "dead" as the definition, otherwise our new SEME borrowck would be breaking the rules :P

And I'm still not clear on what aliased writes would mean?

An aliased write means that you write while another pointer is active.

That's the definition of an aliased write. What does it mean for the program? How does one define an aliased write in your model? (for example, when using Cell, or using raw pointers)

https://gist.github.com/ubsan/1fb0aa56dbab7ff2cd6713cbb5ae75f0
@nikomatsakis These are a few more litmus tests, and some thoughts on std::mem::uninitialized.

I wrote a blog post discussing my thoughts on a high-level approach to the memory model:
http://smallcultfollowing.com/babysteps/blog/2016/05/27/the-tootsie-pop-model-for-unsafe-code/
The heart of the proposal is the intution that:

when you enter the unsafe boundary, you can rely that the Rust type system invariants hold;
when you exit the unsafe boundary, you must ensure that the Rust type system invariants are restored;
in the interim, you can break a lot of rules (though not all the rules).

Discuss thread: http://internals.rust-lang.org/t/tootsie-pop-model-for-unsafe-code/3522

@ubsan

These are a few more litmus tests, and some thoughts on std::mem::uninitialized.

Great! I'm looking through now. As always, I apologize for my permanently backlogged state. Most of the initial examples are basically safe code, so it seems indeed pretty hard to imagine them being undefined.

unsafe fn memcpy(dst: *mut u8, mut src: *const u8, len: usize) {
  for el in std::slice::from_raw_parts_mut(dst, len) {
    *el = *src;
    src = src.offset(1);
  }
}

I guess the key point here is the std::slice::from_raw_parts_mut, which then produces an &mut [u8]? Or is the key point that this is applied to a struct with padding?

// dereferencing any non-null pointer to a ZST
fn f(x: *const ()) -> () { if x.is_null() { () } else { unsafe { *x } } }

I definitely think that dereferencing a pointer to a ZST is invalid. Put another way, you have the right to read 0 bytes of data through that pointer, which means dereferencing is invalid.

// Creating a raw pointer to an uninitialized value
let x: u32 = std::mem::uninitialized();
let y = &x as *const u32;

What about this example worries you? Is it the transient &x? It seems like a "pointer to an uninitialized value" will happen every time you call free, so the mere existence of such a pointer had better not be a big problem.

let x: u32 = std::mem::uninitialized();
let y = x;

Yeah, this is a good example to chew on. There are also some fine distinctions to be drawn, around questions like padding and so forth. (Like, just what constitutes uninitialized?) I need to read more into that blog post you cited also.
I'm curious what differentiates the u32 above from the u8 case you cite later on:
// C allows this; should we (I believe so)
let x: u8 = std::mem::uninitialized();
let y = x;

And note, I haven't talked about this, but std::mem::uninitialized is a function, and so we're (semantically, at least) moving uninitialized values when we call it. Why is it special?

Yes, a good question, I agree! Seems to argue that we should permit accesses from uninitialized memory, at least in certain ways.

As you can see from my blog post, I disagree that this ought to be undefined:
// Aliasing writes with &mut
fn f(x: &mut i32, y: &i32) -> i32 {
  *x = *y + 1;
  *y
}
let x = 10;
let y = &mut x as *mut _;
unsafe { f(&mut *ptr, &*ptr) }

I definitely think that dereferencing a pointer to a ZST is invalid. Put another way, you have the right to read 0 bytes of data through that pointer, which means dereferencing is invalid.

(Emphasis mine.)
By dereferencing it, aren't you... doing exactly that? I would expect dereferencing a pointer to type T to always read size_of::<T>() bytes of memory from the address that it points to. In the case of a zero-sized type, that's 0 bytes. A no-op. Safe as heck.

@glaebhoerl Despite being zero sized, it can still be unsafe to deref (e.g., segfaults), unless you want to add a zero-check for unsized types.

@nikomatsakis
Yes, the issue is that you're applying it to a struct with padding, which should be okay with u8s.
"you have the right to read 0 bytes of data through that pointer". Which is what you are doing.
The issue I have with it is that &T should always be safe to deref, I guess? They're dereferenceable in LLVM, so... Pointers, I have no problem with, they're not dereferenceable.
What differentiates u32 from u8 is that C says that char must be allowed to be uninitialized. Therefore, we can assume that, at least, char doesn't have any trap representations.
Assume that f is in a different, completely safe module :P

A problem that has come up in the discussion is assignment temporary elimination.
As you know, assignment of a struct without a destructor creates MIR like tmp = RVAL; LVAL = tmp. The question is when we can optimize that to LVAL = RVAL.
That is significant, because the computation of RVAL can depend on LVAL, For example (this compiles today):
fn must_work() {
    let mut x = S::new();
    x = foo(&x);
}

If we are talking about our optimization, the most simple interesting example is (everywhere Big is a struct without a destructor).
fn example0(b: &mut Big) {
    println!("{:?}", b);
    *b = Big::new();
}

Here, we would like to optimize out the assignment temporary. The problem is, there is no way we can prove the formatting code is not storing a copy of b inside some global, which Big::new will read after it invalidates it.
If we decide to special-case globals in some way, we still are in some trouble:
fn example0(b: &mut Big) {
    let x = foo(&b);
    *b = Big::new(x);
}

Here, there are a few principal possibilities.

foo returns a boolean or something. In that case, we would like to avoid generating a temporary if possible.
foo is the identity function. Then the borrow of b will be extended to contain the call to Big::new, and we have safe code that compiles, and we must generate the temporary.
foo is <*const Big>::as_ptr. In unsafe code, we would like that to work just like the previous case. The problem is that we can't distinguish this case from the first one reliably - every return value that contains a raw pointer is possibly suspect.

There is one pattern that is commonly used that @ubsan's definition seems to make make UB:
fn whatever(a: &mut u32) -> u32 {
     // either order of assignments
    let ptr0: *const u32 = a;
    let ptr1: *mut u32 = a;
    let result = access(&*a_ptr0); // for some `fn access(&u32) -> u32`
    *a_ptr1 = 0;
    result
}

fn whatever_mut(a: &mut u32) -> u32 {
     // either order of assignments
    let ptr0: *const u32 = a;
    let ptr1: *mut u32 = a;
    manipulate(&mut *a_ptr1); // for some `fn manipulate(&mut u32)`
    *a_ptr0
}

For example, ptr::write desugars to the second.

@arielb1 Yes, your second example would be UB under my memory model. I have some ideas for solving it though.

Rust aliasing model
Base Definitions
References are either immutable, mutable, or moving.
Access kinds are read or write.
The aliasing rule
If a reference r with lifetime 'r is created, then afterwards, before 'r ends but at no particular order
* Some memory location l is accessed through a borrow-chain from r - this is the asserting access.
* The same memory location is accessed in any other way - this is the conflicting access.
* One of these accesses is a write.
Then an aliasing access has occured.
Aliasing accesses are UB, unless there exists a reference s to with lifetime 's, such that
* s was created by a borrow-chain from r
* s has sufficient permissions for the conflicting access.
* 's is alive at the point of access.
* s is potentially alive. This means either:
- s is an immutable reference
- s is the reference used for the conflicting access
- s was converted to a raw pointer before the conflicting access
In that case, s is called the guaranteeing reference.
Reference Creation
All references are created by reborrowing, either of a prior reference or of a raw pointer.
A reference is directly reborrowed from another if it is the borrow of some lvalue that came from the original reference. A borrow chain is the reflexive transitive closure of the directly-reborrowed relation.
NOTE: lvalues can contain dereferences of &mut - these have the borrow chain of their parent. However, dereferences of & have the borrow chain of the borrowed reference itself.
NOTE: raw pointers can't take part of a borrow chain.
References created by transmutation ex nihilo should be handled as if they were created by reborrowing a raw pointer.

unless there exists a reference s to with lifetime 's

Missing word or stray "to"?

Additional
Does anyone have any other interesting examples for me to analyse? @ubsan @nikomatsakis

@arielb1 I have a number of questions about the rules you propose here. I think a lot of this has to do with definitions.
For example, when you say "reference", do you mean specifically a value of type &T or &mut T (as opposed to, say, a raw pointer)?
Similarly, when you say "reborrow", what do you mean? I know of at least two definitions for that term (me and @RalfJung seem to have intuited different meanings, for example). I'm not sure how important it is. My definition is basically just a subset of borrows: that is, borrowing a value via some lvalue that involves dereferencing a reference (and thus reaching into content that was already borrowed). So if x: &i32, then &x is a borrow and &*x is a reborrow of the referent.
Certainly the part of this proposal that I find most worrisome was the fact that it relies on specific lifetimes, and in particular the "conservative" rule around reborrows (casts) from &T to *const T. For example, if I have an argument p: &'a u32, and I cast p to *const u32, then saying that this cast extends for all of 'a seems maybe plausible. But what if I cast &*p to *const u32? In that case, the reference I am casting actually has a very short lifetime, but surely we want the same result?
I've been hoping rather for a model where the legal aliasing doesn't consider the lifetime of the references in question at all. Rather, we can define the memory that a fn has permissions to affect and how -- and lifetimes do come into play here, since if you pass in a &'a i32, you are granting that fn permission to read from the reference until the end of 'a (and you are not granting permission to mutate). (And when you declare let x: u32, you are giving permission to read from that stack slot until the end of the enclosing scope.) But within the fn body itself, the optimizer would not consider lifetimes. (Note that you can have this without going to the full Tootsie Pop model I proposed in my blog post.)

@nikomatsakis
Sure. Reference = TyRef = &,&mut,&move. And in my scheme, all borrows are reborrows because locals are lifted into allocas.

Certainly the part of this proposal that I find most worrisome was the fact that it relies on specific lifetimes, and in particular the "conservative" rule around reborrows (casts) from &T to *const T. For example, if I have an argument p: &'a u32, and I cast p to *const u32, then saying that this cast extends for all of 'a seems maybe plausible. But what if I cast &*p to *const u32? In that case, the reference I am casting actually has a very short lifetime, but surely we want the same result?

And it would have the same result. The &*p borrow must also be treated conservatively for UB purposes, therefore it would also be extended. What I meant by being conservative is that if there exists some way of inferring lifetimes that is not UB, then the code is not UB. Basically, we can't infer lifetimes correctly, so we must assume the worst.

permissions

I am quite sure that my model can be stated equivalently in terms of capabilities. In fact, that was the first form I had in mind, and I switched to the access-based form for clarity.

you are granting that fn permission to read from the reference until the end of 'a (and you are not granting permission to mutate).

See how the lifetimes sneak back in? I don't see any way to have a workable model without lifetimes - for example, the only difference between rvo_safe and no_rvo_safe (which I added now) is the lifetime bounds in as_ptr vs. id, yet the functions are supposed to be optimized differently.

@arielb1

See how the lifetimes sneak back in? I don't see any way to have a workable model without lifetimes - for example, the only difference between rvo_safe and no_rvo_safe (which I added now) is the lifetime bounds in as_ptr vs. id, yet the functions are supposed to be optimized differently.

Named lifetimes in the fn signature are a totally different beast than random lifetimes that result from inference. I agree you can't "excise" them completely in some sense. They are the source of the permissions -- and of course if you continue to use memory you obtained from a reference after the fn returns, you need some way (e.g., mentioning the named lifetime in your type) to show that you are still within that lifetime.

@arielb1 This is such an unimportant nit I almost feel bad bringing it up:

let local = &move *alloca();.

I believe you want a scratch pointer since local is uninitialized at lifetime begin, and must be uninitialized at lifetime end.

@Ericson2314 could you explain further?

@ubsan &move is used for moving data out, so at the beginning of the lifetime it must point to an initialized location, and at the end of the lifetime, since the data is moved out, it must point to an uninitialized location.
&uninit or &scratch would be a pointer type where at both the beginning and end of the lifetime, the location pointed to must be uninitialized. The name "scratch" invokes the idea that this pointer gives access to scratch space / empty buffer / playpen etc. It is the proper return type for alloca, and encapsulated in a newtype for Drop to call free, it is the proper return type for malloc.
For more information see https://github.com/rust-lang/rfcs/pull/1617

@Ericson2314 Ah. We've taken to calling it &out.
&move can point to uninitialized memory (i.e., moving in and out, like in a local, would be allowed), and &out's semantics are not well known enough.

mm I thought the name &out referred to the pointer which starts uninitialized and must be initialized before it is dropped / lifetime ends. This one alone of the 4 possible is a linear type, but other than that I think all 4 are pretty understood.

Regardless all for readily fit the proposal of @arielb1 that I was quoting, hence why I felt bad bringing them up.

@Ericson2314 Ah, got it. Yeah, see, &uninit is definitely not well understood, I didn't understand what you were talking about ;P.
Anyways, &move does fill that hole; it has two states, just like a local variable (at least in my model of &move), "moved out", and "moved in". This would just be starting the reference in the "moved out" state.

@ericson2314
This would be a better approximation:
let local = &move *alloca();
mem::forget(*local); // mark `*local` as uninitialized

However, the presence/absence of the mem::forget does not affect the aliasing semantics, so I omitted it.

it has two states, just like a local variable

Well, there needn't be any extra initialized/uninialized "state" because 4 is enough pointer types to make it first-class https://github.com/rust-lang/rfcs/pull/1617#issuecomment-220697500.

This would be a better approximation:

With the above, the forget call wouldn't even be legal, unless we want "rvalue-initalized-ness" and forget to be polymphic over such a thing.

PS: at the MIR level, &move is just as linear as &out - drops do not insert themselves, after all.

Yes indeed! This symmetry is much nicer, and I love that I can say "well, we already have them!" on the next linear types proposal.

Back on topic, I'm still confused on ZST-deref being unsafe. IMO, *x: () should be safe, *Void: () should not. This is one of the reasons I want Void-iso types to have size -∞.

With the above, the forget call wouldn't even be legal, unless we want "rvalue-initalized-ness" and forget to be polymphic over such a thing.

I want Box<T> to be implementable as struct Box<T>(&'static move T) up to it not requiring T: 'static due to WF. You can mem::forget out of a box.

Back on topic, I'm still confused on ZST-deref being unsafe. IMO, *x: () should be safe, *Void: () should not. This is one of the reasons I want Void-iso types to have size -∞.

The unsafety rules take effect before monomorphization, so the size of types is not known yet. I think that deref of a unit type should always be a no-op (never UB) on a non-null pointer/reference, and that a deref of an empty type should be an intrinsics::unreachable().

I want Box to be implementable as struct Box(&'static move T) up to it not requiring T: 'static due to WF. You can mem::forget out of a box.

That looks like killing a fly with a nuke. We just need some control over the internal drop-flag - a shallow_drop - that could be unsafe too.

I suppose we can add &move before the others and then that will make sense. Empty boxes are unsafe, but it's a step in the right direction.

The unsafety rules take effect before monomorphization, so the size of types is not known yet. I think that deref of a unit type should always be a no-op (never UB) on a non-null pointer/reference, and that a deref of an empty type should be an intrinsics::unreachable().

I think that's in agreement with everything I've said? There many (safe) things we can and should do with generic code that assume >0 size, such as allowing match x: Option<A> { Some(x) => ..., ...} even though if A was Void we'd have a dead match arm.

@arielb1 @Ericson2314 The point, at least for me, of &move is that it gets rid of Drop::drop dropping inner fields that it derefs to. See the codegen section of https://gist.github.com/ubsan/9d79ad4299870faf8c7b

@ubsan
So have drop glue call DerefMove to get the interior to drop? That... could work.
OTOH, you ought to be able to do partial drops, which would make DerefMove unsafe:
struct Foo(Box<u32>, Box<u32>);
fn main() {
    let x = Box::new(Foo(Box::new(0), Box::new(1)));
    drop(x.1);
    // drop glue:
    let d = DerefMove::deref_move(&mut x); // unsafe! can access `x.1`!
    drop(d.0);
    mem::forget(d.0);
    Drop::drop(&mut x);
}

@ubsan and I had talked a bit about &move on IRC, but I am wary of extending the language if we are still lift with (different) unsafety/magic. So yes &move alone is still useful (e.g. for drop in place) but we ought to hold of on DerefMove until it can be done right.
My "4 types of unique borrowed pointer" proposal is really less about pointers and more about making "initializedness" first-class---part of the type system instead some extra static analysis tacked onto borrow checking. Once you have that, the generalized pointer types are fairly inevitable.

@arielb1 well, in that case, it's not unsafe :)
Notice how you're deref_moveing in drop(x.1). That would turn into
let inner = x.deref_move();
drop(inner.1);
// drop glue
drop(inner.0);
Drop::drop(&mut x);

So there's gotta be some sort of guarantee about only being called once, I'm thinking... something like that.

I have just opened https://github.com/rust-lang/rfcs/pull/1643, which proposes that rather than adopting one RFC, we adopt a "strike-team-based" approach to work these rules out in a more systematic way.

@ubsan

I think we want the DerefPure unsafe trait, and to not allow using DerefMove unless it is implemented.

@arielb1
But why? DerefMove seems fine if we just... don't call DerefMove multiple times. I think it's actually a really nice guarantee that DerefMove doesn't get called multiple times, just like Drop.

This allows for things like dropping in DerefMove, since it's only called once. If, for example, you have a cache, and you don't need it once you're moved out of, it would be nice to get rid of it once you're no longer useful; and, I don't see the point of requiring DerefPure. It just makes it more annoying to implement DerefMove for users.

@Ericson2314

Back on topic, I'm still confused on ZST-deref being unsafe. IMO, *x: () should be safe, *Void: () should not. This is one of the reasons I want Void-iso types to have size -∞.

Most of the discussion here is well beyond the scope currently covered by my formalization, but this one part is actually already "in scope" and I have definitions which I think make sense. So I will try to share my formal thoughts on this. Funny enough, the outcome is exactly the opposite if what you suggest.
Let me try to explain the relevant parts of my model of types:
A type is a set of lists of values. Values are, for example, integers, addresses (i.e., memory locations), or booleans. These lists describe the layout of the type in memory. This is a very abstract model that does not distinguish integers of different size, or the fact that integers actually overflow. All these issues can be modeled faithfully,  but they are mostly irrelevant for what I am interested in right now, so I am making my life simpler.
For example, the type i32 is the set of singleton lists whose only element is an integer.
The type struct { i32, bool } is the set of two-element lists, the first element being a number and the second being a boolean. The type () is the set consisting only of the empty list. The type Void is the empty set. Every type has a size, which is the length of the lists in the set. (They all have to have the same length -- no unsized types so far.) Clearly, the size of () must be 0. The size of Void can be anything, since there is no list in that set.
The type &mut 'a T is (very roughly speaking) the set of singleton lists whose element is a location l. Furthermore, the locations in [l, l+T.size) (left-inclusive, right-exclusive) have to be valid, allocated addresses that we can access and mutate for the duration of lifetime 'a, and the list of values stored at these addresses must be in T. (I am using mutable borrows here because they are way simpler than shared borrows. I hope to write a blog post some point on why that is, I "just" need to come up with a nice explanation...)
So, if we consider &mut 'a (), what do we have? We have a location l such that [l, l+0) are valid addresses, and the list of values stored at these locations is in the set representing (). This interval is empty. Hence we do actually not know anything about the location, so in particular, we must not dereference it.
Let's consider &mut 'a Void. We have a location l such that [l, l+Void.size) are valid addresses, and the list of values stored at these locations is in the set representing Void. That's the empty set. There cannot be anything in the empty set, so we have a contradiction. From this, we can derive anything, and hence in particular, we are allowed to dereference the pointer.
What I am saying here, essentially, is that &mut 'a Void is actually the same type as Void in the sense that there is no possible value of that type, and hence we can always assume during compilation that no value of type &mut 'a Void exists. (This is actually assuming that 'a is an active lifetime.) Because of this, you can do literally anything if you have a value of type around; that's unreachable code.

@RalfJung But the topic was *mut Void, not &'a mut Void - does the same reasoning apply? If so, wouldn't that make using it for FFI pointers violently invalid?

@RalfJung I believe there's a distinction to be made here that &mut Void is Void but *mut Void isn't.
Specifically, one can create a *mut Void in safe code, but dereferencing it (in unsafe code) creates a value of type Void, which can make reachable code behave as if it wasn't (literally "invoking" UB).
About ZSTs, I believe you're talking about memory reads, not syntactical dereferences, which do not touch memory at all to "read" a ZST.
Sorry if I sound like a broken record, but this distinction in general can lead to a lot of confusion, because *ptr is a memory no-op in C, C++ and Rust (the latter 2 only if not overloaded) in that it converts a pointer rvalue into an lvalue while performing no memory access, but even if, say, &*ptr == ptr can be considered to be trivially true, AFAIK syntactical dereference can be UB in C if ptr == NULL.

@RalfJung still reading the rest of your post, but btw I totally agree that &mut is way easier to understand deeply. In https://internals.rust-lang.org/t/a-stateful-mir-for-rust/3596/15 (which you might be interested in anyways :)) I have a pretty good idea of what I will write for &mut but much less so for &.

Ah OK, so even besides @eddyb's concerns I consider a load from &mut () to be safe because that load must have size 0, so the load is actually a no-op. Totally agree &mut Void is absurd.
Also I'd like to point out that @RalfJung's model is another good reason why the conflation of ()-isomorphic and Void-isomorphic types as both having size 0 is a dangerous thing to do.

@Ericson2314 To be specific, lvalue->rvalue conversion for a () is a no-op, whereas lvalue->rvalue conversion for a ! is undefined behavior (I believe I can start using ! now as the Void type, it seems that will go through).

@eternaleye

But the topic was *mut Void, not &'a mut Void - does the same reasoning apply? If so, wouldn't that make using it for FFI pointers violently invalid?

Oh, I see. As far as I can tell, Rust attaches absolutely no guarantees to raw pointers, so their dereferencability depends entirely on the meaning you assign to them. If your invariants ensure that the given *mut T points to X bytes of valid memory, then sure you can make use of that. Of course, if you put the result into a variable of type T, you have to make sure it is a valid T. So, you better don't dereference the *mut Void, and (naturally) there is a manual proof obligation when creating a *mut Void that this pointer actually satisfies the guarantees you'd like to attach to it.
@eddyb

Sorry if I sound like a broken record, but this distinction in general can lead to a lot of confusion, because *ptr is a memory no-op in C, C++ and Rust (the latter 2 only if not overloaded) in that it converts a pointer rvalue into an lvalue while performing no memory access, but even if, say, &*ptr == ptr can be considered to be trivially true, AFAIK syntactical dereference can be UB in C if ptr == NULL.

Oh, the good ol' lvalue-rvalue confusion. I am entirely sidestepping that issue in my formalization by not having lvalues, which is why I payed no attention to it. ;-) Anyway, when I wrote about dereferencing pointers above, I was talking about actual load operations that happen on the machine.
@Ericson2314

Also I'd like to point out that @RalfJung's model is another good reason why the conflation of ()-isomorphic and Void-isomorphic types as both having size 0 is a dangerous thing to do.

I am not sure that is the case. ;-) Actually, I did not even assume any particular size for ! above -- the reasoning works for any size. Truth is, the size of ! could be literally anything, it just doesn't matter as the type is not inhabited. I doubt it is worth complicating the algebra of sizes for this corner-case. It would be nice to have things like Option<!> having the same representation as (), but that won't fall out of any particular size for ! -- instead, that'd be an extension of the optimizations for enum layout that we already have (e.g., Option<&T> having the pointer size, without an added discriminant).
@ubsan

To be specific, lvalue->rvalue conversion for a () is a no-op, whereas lvalue->rvalue conversion for a ! is undefined behavior (I believe I can start using ! now as the Void type, it seems that will go through).

I would say that lvalue -> rvalue conversion for ! assumes that we have a !, so this cannot even happen. That's not UB, that's just impossible -- the UB happened earlier, when we obtained a value of type !. But the consequence is the same, the compiler is free to emit literally any assembly code when anything is done to a !.

@RalfJung Ah, that comment wasn't supposed to go through :)
The issue is raw pointers, where *mut ! is a valid type, despite being an lvalue to an impossible rvalue.

@ubsan Right, as long as you don't dereference it. Whether &*x is already UB, I am not so sure... I would tend to argue it should not be.

Quoting my comment from #1216 with respect to using *mut ! for things like FFI types:

Given that the contract of *const T/*mut T is that it shall only be dereferenced when it actually points to valid/live data (the type system can't know this, which is why it's unsafe; it's entirely up to the user to determine this): I think *mut UninhabitedType, with the understanding that it will never be dereferenced, and *mut UnitType, with the understanding that it can be dereferenced whenever but there's no point in doing so, make equal amounts of sense.
(In other words, this is unlike &mut UninhabitedType, which is logically equivalent to UninhabitedType itself, because *mut doesn't imply liveness.)

@RalfJung
I am not sure about &*bot, but a use (aka lvalue-to-rvalue conversion) of a Void is UB.

Agreed. I'd argue that &*x is a noop (when translating to an lvalue-free semantics, it compiles to x). But if an actual memory access happens, we get UB.

@RalfJung But the having &! is UB, so &*x is undefined behavior, because it creates an &!.

Here are some of my own thoughts (not an expert, but am working on a garbage collector):

The memory model needs to be formal.  Otherwise you can't formally verify unsafe Rust code, which is a use case that (at least) Redox definitely wants.
We need to avoid falling into the trap that C/C++ fell into with strict aliasing, which as I understand it breaks so much code that major compilers implement -fno-strict-aliasing and major projects rely on this (the OCaml runtime, CPython 2.x, and the Linux kernel, as well as the output of an RPC compiler at a minimum), and yet as I understand it has only minimal performance gains in practice for typical code (i.e. not tight numeric loops).
The memory model needs to be easy for people to understand.  That means that an equally normative, easy to understand prose description of the memory model needs to be present as well – it is, after all, what most people will be programming to.  If the formal model and the prose model diverge, this is a bug in the spec that must be fixed.
It needs to be possible to do what needs to be done without unnecessary copies.  In C++, the only standards-conforming way to pass the buffer stored in an std::string to a function that takes an unsigned char * is to copy the entire buffer!.  This is clearly absurd.
The memory model needs to support writing memory allocators, garbage collectors, etc. without too much boilerplate or pitfalls beyond what C would have.  If the Rust version of some code is twice as long as the C version (with -fno-strict-aliasing) and is much harder to read, then that is a problem with Rust.
Aliasing *mut pointers should be permitted.  Furthermore, it should be possible for an &mut or & pointer to alias a *const or *mut pointer, so long as the pointee does not change (in the case of a &-pointer) or is only modified through the reference (in the case of an &mut) for as long as the reference is live.
In multithreaded code, a common type of data race is two threads racing to write the same value to the same address, with no racing reads.  Such races should be allowed, as the operation of writing a value to an address is idempotent.  Similarly, the case where two threads simultaneously read a value from an address, set and/or clear a bit, and then write the new value back should be allowed – again, no reordering that I can imagine would break the code, and I have read about several algorithms (garbage collectors, if I recall correctly) that rely in this to work.

Nominating for discussion at lang-team meeting - the unsafe guidelines team exists now, so we should consider moving this PR either somewhere else or just the team tag.

I agree with @nrc that, given that we have accepted https://github.com/rust-lang/rfcs/pull/1643, we should close this RFC, since its contents are subsumed by the discussion taking place at the unsafe guidelines repo.
I reviewing the thread and have tried to extract out the most notable questions that were debated. It would make sense to move many of these to issues on the rust-memory-model repo, I think:

how to handle uninitialized data and whether some CPUs will require loading of undef to be UB
some discussion of temporary elimination in MIR
an alternative aliasing model jointly proposed by @arielb1 and @ubsan
some thoughts on ZST interactions

The discussion on the RFC has focused on a number of topics, but surprisingly little on the meat of the rules here.

how much and what to formalize
how well-specified LLVMs docs are (not much)
a reframed proposal by @arielb1 and @ubsan

@rust-lang/lang members, please check off your name to signal agreement. Leave a comment with concerns or objections. Others, please leave comments. Thanks!

[x] @nikomatsakis
[x] @nrc (I took the liberty of pre-checking, given the previous comment)
[ ] @aturon
[ ] @eddyb
[x] @pnkfelix (pre-checked, since he is on vacation)

I'm just going to close it now, since I owns it :P

authors: Amanieu Connorcpu Ericson2314 Manishearth RalfJung Ticki archshift arielb1 chrisvest comex dgrunwald dikaiosune drbo eddyb eternaleye gereeter glaebhoerl nikomatsakis nrc petrochenkov pnkfelix steveklabnik ticki ubsan

repo: rust-lang/rfcs


text:
Fix rust-mode lazy loading

This PR fixes the following error that happends when opening a .rs
file:
File mode specification error: (void-function rust-mode)

It adds a stub function called rust-mode that selects the actual
rust-mode (prog or treesitter) based on the environment. Once the
actual rust-mode is loaded, the stub function gets redefined.
Fixes #528

Thanks, @jroimartin can you see why the tests are failing ?

Sure, I'm working on it.

@psibi PTAL. I think this is the smallest required change. On the flip side, it is a bit tricky. But having two major modes with the same name and loading them conditionally seems problematic. So, I opted for allowing the stub function (called rust-mode) to be redefined when loading the actual mode.

Thanks, LGTM. @condy0919 Can you see if it works in your setup too ? I will test drive this PR too.

Thanks, LGTM. @condy0919 Can you see if it works in your setup too ? I will test drive this PR too.

Sorry, I'm in a touring🥺 can't help that

No rush. There is a valid workaround which is just adding (require 'rust-mode) to init.el. It is just that it is nice to support lazy loading :slightly_smiling_face: This change also make rust-mode just work for people using plain package.el.

No worries, I'm also going to be on travel for the next two days. I will test this out more properly after that.

@psibi @condy0919 what do you think of db7d086? This option does not require messing up with the rust-mode function definition.

@psibi @condy0919 what do you think of db7d086? This option does not require messing up with the rust-mode function definition.

This solution comes to mind at first, and the trick is used in the evil-exchange package.

I did some testing and this seems to work fine. @jroimartin Thanks for the PR. @condy0919 Thanks for testing and review.

source: gharchive/pull-request

authors: condy0919 jroimartin psibi

repo: rust-lang/rust-mode

license_type: permissive

license_source: bigquery


text:
[unstable option] ignore

Tracking issue for ignore

request for stabilisation: https://github.com/rust-lang/rustfmt/issues/3243

This will need to be re-done for 1.x

What's blocking stabilizing this feature? I want this feature a lot. I need to ignore to ignore git submodules which doesn't formatted with rustfmt.

#5365 #5367

I know there are several process but I can't know where this feature is now. In addition, it looks it's stabilized once for 2.0 so I want to know what's missing for 1.x.

@anatawa12 I think the issue is that even the rustfmt team isn't too sure what outstanding issues are blocking stabilization for this one.
I don't know if this is a blocker but I did a quick search through the issue tracker and found #4726. There may be other outstanding issues but I don't have the time to check.
If anyone is interested in moving this option closer to stabilization I'd recommend starting with bullet number 1 listed on https://github.com/rust-lang/rustfmt/discussions/5367#discussioncomment-2874888.

source: gharchive/issue

authors: anatawa12 calebcartwright karyon scampi ytmimi

repo: rust-lang/rustfmt

license_source: github-api


text:
Macro use not formatted - braces, item position, trailing comma

When running rustfmt on https://github.com/rust-lang-nursery/rand/blob/master/src/rand_impls.rs

@nagisa Could you paste the long line here please?

I’m not sure which the one that causes the issue, but removing https://github.com/rust-lang-nursery/rand/blob/master/src/rand_impls.rs#L220 seems to make it not fail anymore.

That's
array_impl!{32, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T,}

So possibly we're doing something silly with the macro use? Given that it fits on one line already, that seems weird.

It doesn't fit within a 100 chars, right? We're not reformatting it yet because

it's an item level macro use;
it uses braces;
it has a trailing comma.

Oh right, it doesn't, that would explain it!
Removing tags because we don't touch it yet and it's a known issue (also we're not making anything worse which is my usual criteria for p-high).

This may be fixed now. I think we format item level macro uses and braces. Only the trailing comma may be an issue since there's no way to know if it's significant. Maybe we could just go ahead and assume it is not.?

As of 8ec0750bb88dfe84ca2d54ddc29b13f5900641d6 it formats as follows:
 fn foo() {
-    array_impl!{32, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T,}
+    array_impl! {32, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T,}
 }

The line is still over the limit, but an error would be displayed only if the error_on_line_overflow option is enabled:
internal error: line formatted, but exceeded maximum width (maximum: 100 (see `max_width` option), found: 117)
 --> /home/yfful/documents/code/rustfmt/438.rs:2
  |
2 |     array_impl! {32, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T, T,}
  |                                                                                ^^^^^^^^^^^^^^^^^

warning: rustfmt may have failed to format. See previous 1 errors.

authors: marcusklaas nagisa nrc scampi


text:
Rustfmt cannot find mod with #[path] on windows

Rustfmt on windows cannot resolve module to file if #[path] used.
Note, #[path] should contain windows-unfriendly relative path.
To Reproduce
Use #[path = "./some-relative-path.rs"] in code.
Pointed file should be and valid rust source or empty.
#[path = "../module_foo.rs"]
mod module_foo;

Error:
> cargo fmt -- --check
error: couldn't read \\?\D:\a\rustfmt-issue-4477\rustfmt-issue-4477\src\..\module_foo.rs: The filename, directory name, or volume label syntax is incorrect. (os error 123)
Error writing files: failed to resolve mod `module_foo`: \\?\D:\a\rustfmt-issue-4477\rustfmt-issue-4477\src\..\module_foo.rs does not exist

Check out demo for this issue. Also there is two cases reproduced on GHA:

this error "file not found"
all is ok, #[path] with just filename

Expected behavior
No rustfmt's IO errors.
All modules resolved.
Meta

rustfmt versions:

rustfmt 1.4.15-stable (530eadf4 2020-06-02)
rustfmt 1.4.15-nightly (aedff61f 2020-05-19)

From where did you install rustfmt?: rustup, crates.io
How do you run rustfmt: rustfmt, cargo-fmt

rustfmt versions:
* rustfmt 1.4.15-stable ([530eadf](https://github.com/rust-lang/rustfmt/commit/530eadf4b42ddf35b209d4f4acd120f3fcc467ce) 2020-06-02)

* rustfmt 1.4.15-nightly ([aedff61](https://github.com/rust-lang/rustfmt/commit/aedff61f7ac4fc2b287ff76d33f2584e1f63a3af) 2020-05-19)

Could you try with a more recent version? There were a lot of improvements on the module resolution in 1.4.20/1.4.22 if I recall correctly

...
Could you try with a more recent version? There were a lot of improvements on the module resolution in 1.4.20/1.4.22 if I recall correctly

@calebcartwright, I'm sorry, versions I previously mentioned is my local. But latest rustfmt on CI, there is rustfmt 1.4.20-stable (48f6c32e 2020-08-09). (proof, see "print versions")

Same problem on rustfmt 1.4.22-nightly (97d03010 2020-10-04) and rustfmt 1.4.22-nightly (97d03010 2020-10-04)

I built the latest rust-fmt from source code and found it has been fixed at #4022.
https://github.com/rust-lang/rustfmt/blob/2ce75b114716cf8900640b8f8e5db44438fc25da/src/config/file_lines.rs#L51-L53
The reason is caused by path across look like this path\..\..\a.rs.
e.g :
\\?\F:\Rust\tikv\tests\failpoints\cases\..\..\integrations\import\util.rs does not exist

This is the description in rust_span.
https://github.com/rust-lang/rust/blob/8cef65fde3f92a84218fc338de6ab967fafd1820/compiler/rustc_span/src/lib.rs#L106-L123

You can build a latest rust-fmt and replace it in toolchains .

Windows:

set CFG_RELEASE=1.45.0-nightly <--(Specify yours)
set export CFG_RELEASE_CHANNEL=nightly <--(Specify yours)

cargo build --bin rustfmt --features="rustfmt"

Just now I did a :
rustup component add rustfmt --toolchain nightly

Says it installed but Windows couldn't find the path.

authors: calebcartwright chadbrewbaker francis-du fzzr-


text:
use_field_init_shorthand is listed as default: false but is enabled by default.

On rust 1.54 when I run cargo fmt -- --help=config it shows use_field_init_shorthand as default: false.
However running cargo fmt will format as if use_field_init_shorthand is true.
I think use_field_init_shorthand should list the default as true to reflect reality.

However running cargo fmt will format as if use_field_init_shorthand is true.

This does not sound accurate. Can you please provide a minimal example that reproduces the behavior you're describing?

Ah, sorry, the issue is not exactly as I described, but it is still an issue regardless.
It seems use_field_init_shorthand: false functions as an off despite the  documentation clearly describing it as forcing longhand formatting. https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#use_field_init_shorthand
consider:
struct Foo {
    a: i32,
}

fn main() {
    let foo = Foo { a };
}

and
struct Foo {
    a: i32,
}

fn main() {
    let foo = Foo { a: a };
}

which both remain unchanged after running cargo fmt

both remain unchanged after running cargo fmt

Can you please clarify what configuration options you are using, and what your expected behavior is?
It seems like in the first snippet you are expecting use_field_init_shorthand to convert shorthand to longhand which is not what the option does. The example snippets shown in the config docs only highlight formatting style that will remain as-is/pass --check with the corresponding option value. They do not imply any reformatting except in cases where the documentation contains a section/comment/etc. that highlights a "before/pre-formatting".
In the second snippet, I'd imagine you would expect that to be converted to:
fn main() {
    let foo = Foo { a };
}

which is precisely what the option is supposed to do, and what it will do so long as you have use_field_init_shorthand overridden with the non-default value of true

I'm not saying to use this exact wording but the documentation would make a lot more sense if we wrote it something like this:

Nope, that wouldn't work either because we actually run automated tests against all the config options and variants to ensure that the formatting of the snippet is idempotent when formatted with the corresponding config. I don't really have much more to offer up that hasn't already said beyond general advice when viewing the docs to not make assumptions about things not explicitly stated nor shown, and always feel free to ask clarifying questions!
As mentioned before, when there's a feeling that showing a formatting diff is helpful, the actual "before" is shown explicitly, e.g.
https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#always, but otherwise don't assume there was some different/specific input which was modified to produce the formatted snippets.

What if we did it like this?

This way we improve both the idempotentency test and the documentation.
I am willing to provide a PR if its just documentation changes and we find something we both agree on.

authors: calebcartwright rukai


text:
Lines containing overly long strings are not formatted correctly

As per the title, lines containing overly long strings are "left alone" and not formatted correctly.
Observing this rust playground and pressing the rustfmt button, line 3 is correctly collapsed while line 4 keeps the weird spacing.
I believe that this should not be the case: Even if the line can't be reduced to a sensible length, it should still be formatted in a "best effort" way.
As for the question of "where would you ever encounter this?", my answer is: Unbroken SQL queries are frequently over the line limit and cause formatting to break in weird ways.
By the way, when this bug happens it seems to "spread" like a virus. Lines near (perhaps inside the same block?) the overly long one are not formatted either, but I'm not sure how to reproduce this in a minimum example.
Thank you all in advance.

Thanks for reaching out but closing as a duplicate of #3863

Ah, sorry about that. The discussion on that issue is surprisingly comprehensive! Sad to see it's been an issue since 2019 however.

authors: PurpleMyst calebcartwright


text:
Fix lost import with Crate granularity

Currently use a::{c as cc, c}; will be formatted to use a::c as cc; when Crate granularity is set.
The regression was introduced in https://github.com/rust-lang/rustfmt/pull/4973

Thank you for your interest and willingness to submit a PR! My apologies for the delay in review, but I think it'd be best to close this in favor of #5209 which I suspect will cover a broader suite of variants


text:
Add test for issue 3937

Closes #3937
It's unclear which change fixed the format_code_in_doc_comments=true
issue brought up in this issue, however I'm unable to reproduce the
error on the current master.
The added test cases should serve to prevent a regression.

As a side note #5171 fixes the first problem listed in the linked issue.


text:
Add support for SO_ORIGINAL_DST and IP6T_SO_ORIGINAL_DST.

Those values contain the original destination IPv4/IPv6 address
of the connection redirected using iptables REDIRECT or TPROXY.
Signed-off-by: Piotr Sikora piotr@aviatrix.com

Could also add a test, even if it's ignored by default.

Could also add a test, even if it's ignored by default.

Done.

Thanks @PiotrSikora.

authors: PiotrSikora Thomasdezeeuw


text:
Add tvOS and watchOS

Follow up to https://github.com/rust-lang/socket2/pull/392.

Thanks!

authors: Thomasdezeeuw thomcc


text:
Document the "rand" feature.

When using this crate I had trouble figuring out why I couldn't use the RandomBits structure. Hopefully future users will be less confused!
Doc locations:

README
RandomBits struct
RandBigInt trait

Thanks!
bors r+

authors: alex-ozdemir cuviper

repo: rust-num/num-bigint


text:
Locale device tree address

I wonder if there is a way to get the address of the device tree by UEFI in aarch64? It's wonderful if the crate can provide an interface to get the address of device tree.

Hello! You might want to look into navigating the EFI Configuration Table structure, accessible with the .config_table() method on the SystemTable. It provides access to a list of descriptors of well-known, predefined "configuration tables", and according to the UEFI spec (page 104), you should be able to find a config table entry for Device Tree:
//
// Devicetree table, in Flattened Devicetree Blob (DTB) format
//
#define EFI_DTB_TABLE_GUID \
{0xb1b621d5, 0xf19c, 0x41a5, \
{0x83, 0x0b, 0xd9, 0x15, 0x2c, 0x69, 0xaa, 0xe0}}

We have some constant GUIDs defined for common tables, but not for DeviceTree (yet). If you're interested in making it easier to access this information for users of the uefi-rs crate, we'd welcome a PR adding it to the list 🙂

repo: rust-osdev/uefi-rs


text:
Some class miss its constructor, Qt5.9.1

I use this wonderful tool to generate rust bindings of Qt5.9.1. The Qt sdk is installed by brew. First I fixed a link err, which is caused by the incorrect is_framework determination. It's works. Then I find some class miss its new() method. One of the class is Widget, link to my output source.

You can run the generator with --debug-logging=save and then look at log/debug_parser_skips.log file in the cache directory. It should explain why the methods are missing.
Can you share what changes related to frameworks did you need to do? Does your Qt installation contain frameworks or libraries?
I'm currently changing implementation to allow supporting all Qt versions at once. Hopefully it will make  parser inconsistency issues less critical. I don't know how long it will take, though.

debug_parser_skips.log
...
Method is removed: Qt::Orientations<Qt::Orientation> QSizePolicy::expandingDirections() const: unknown type: Qt::Orientations
Method is removed: QFlags<QSizePolicy::ControlType> operator|(QFlags::enum_type f1, QFlags::enum_type f2): unknown type: QFlags::enum_type
Method is removed: QFlags<QSizePolicy::ControlType> operator|(QFlags::enum_type f1, QFlags<QSizePolicy::ControlType> f2): unknown type: QFlags::enum_type
Method is removed: QIncompatibleFlag operator|(QFlags::enum_type f1, int f2): unknown type: QIncompatibleFlag
Method is removed: private QWidgetPrivate* QWidget::d_func(): unknown type: QWidgetPrivate
Method is removed: private const QWidgetPrivate* QWidget::d_func() const: unknown type: QWidgetPrivate
Method is removed: [constructor] void QWidget::QWidget(QWidget* parent = ?, Qt::WindowFlags<Qt::WindowType> f = ?): unknown type: Qt::WindowFlags
Method is removed: void QWidget::grabGesture(Qt::GestureType type, Qt::GestureFlags<Qt::GestureFlag> flags = ?): unknown type: Qt::GestureFlags
Method is removed: Qt::WindowStates<Qt::WindowState> QWidget::windowState() const: unknown type: Qt::WindowStates
Method is removed: void QWidget::setWindowState(Qt::WindowStates<Qt::WindowState> state): unknown type: Qt::WindowStates
...

It seems like some complex defines failed to parse.

Can you share what changes related to frameworks did you need to do? Does your Qt installation contain frameworks or libraries?

I just adjust the detect order. Here is my codes.
cpp_to_rust/qt_generator/qt_generator_common/src/lib.rs
...
/// Detects properties of current Qt installation using `qmake` command line utility.
pub fn get_installation_data(sublib_name: &str) -> Result<InstallationData> {
  let qt_version = run_qmake_string_query("QT_VERSION")?;
  log::status(format!("QT_VERSION = \"{}\"", qt_version));
  log::status("Detecting Qt directories");

  let root_include_path = run_qmake_query("QT_INSTALL_HEADERS")?;
  log::status(format!("QT_INSTALL_HEADERS = \"{}\"", root_include_path.display()));
  let lib_path = run_qmake_query("QT_INSTALL_LIBS")?;
  log::status(format!("QT_INSTALL_LIBS = \"{}\"", lib_path.display()));
  let docs_path = run_qmake_query("QT_INSTALL_DOCS")?;
  log::status(format!("QT_INSTALL_DOCS = \"{}\"", docs_path.display()));
  let folder_name = lib_folder_name(sublib_name);
  let dir = lib_path.with_added(format!("{}.framework/Headers", folder_name));
  if dir.exists() {
    Ok(InstallationData {
         root_include_path: root_include_path,
         lib_path: lib_path,
         docs_path: docs_path,
         lib_include_path: dir,
         is_framework: true,
         qt_version: qt_version,
       })
  } else {
    let dir2 = root_include_path.with_added(&folder_name);
    if dir2.exists() {
      Ok(InstallationData {
           root_include_path: root_include_path,
           lib_path: lib_path,
           docs_path: docs_path,
           lib_include_path: dir2,
           is_framework: false,
           qt_version: qt_version,
         })
    } else {
      Err(format!("extra header dir not found (tried: {}, {})",
                  dir.display(),
                  dir2.display())
              .into())
    }
  }
}
...

and my directory struct of the qt5.9.1 sdk library installed by home brew:
/usr/local/Cellar/qt/5.9.1/include
$ ls -l
total 512
lrwxr-xr-x   1 user  admin    38 Jun 29 08:06 Qt3DAnimation -> ../lib/Qt3DAnimation.framework/Headers
lrwxr-xr-x   1 user  admin    33 Jun 29 08:06 Qt3DCore -> ../lib/Qt3DCore.framework/Headers
lrwxr-xr-x   1 user  admin    35 Jun 29 08:06 Qt3DExtras -> ../lib/Qt3DExtras.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 Qt3DInput -> ../lib/Qt3DInput.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 Qt3DLogic -> ../lib/Qt3DLogic.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 Qt3DQuick -> ../lib/Qt3DQuick.framework/Headers
lrwxr-xr-x   1 user  admin    43 Jun 29 08:06 Qt3DQuickAnimation -> ../lib/Qt3DQuickAnimation.framework/Headers
lrwxr-xr-x   1 user  admin    40 Jun 29 08:06 Qt3DQuickExtras -> ../lib/Qt3DQuickExtras.framework/Headers
lrwxr-xr-x   1 user  admin    39 Jun 29 08:06 Qt3DQuickInput -> ../lib/Qt3DQuickInput.framework/Headers
lrwxr-xr-x   1 user  admin    40 Jun 29 08:06 Qt3DQuickRender -> ../lib/Qt3DQuickRender.framework/Headers
lrwxr-xr-x   1 user  admin    41 Jun 29 08:06 Qt3DQuickScene2D -> ../lib/Qt3DQuickScene2D.framework/Headers
lrwxr-xr-x   1 user  admin    35 Jun 29 08:06 Qt3DRender -> ../lib/Qt3DRender.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtAccessibilitySupport
lrwxr-xr-x   1 user  admin    36 Jun 29 08:06 QtBluetooth -> ../lib/QtBluetooth.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtCglSupport
lrwxr-xr-x   1 user  admin    33 Jun 29 08:06 QtCharts -> ../lib/QtCharts.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtClipboardSupport
lrwxr-xr-x   1 user  admin    37 Jun 29 08:06 QtConcurrent -> ../lib/QtConcurrent.framework/Headers
lrwxr-xr-x   1 user  admin    31 Jun 29 08:06 QtCore -> ../lib/QtCore.framework/Headers
lrwxr-xr-x   1 user  admin    31 Jun 29 08:06 QtDBus -> ../lib/QtDBus.framework/Headers
lrwxr-xr-x   1 user  admin    44 Jun 29 08:06 QtDataVisualization -> ../lib/QtDataVisualization.framework/Headers
lrwxr-xr-x   1 user  admin    35 Jun 29 08:06 QtDesigner -> ../lib/QtDesigner.framework/Headers
lrwxr-xr-x   1 user  admin    45 Jun 29 08:06 QtDesignerComponents -> ../lib/QtDesignerComponents.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtDeviceDiscoverySupport
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtEventDispatcherSupport
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtFbSupport
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtFontDatabaseSupport
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 QtGamepad -> ../lib/QtGamepad.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtGraphicsSupport
lrwxr-xr-x   1 user  admin    30 Jun 29 08:06 QtGui -> ../lib/QtGui.framework/Headers
lrwxr-xr-x   1 user  admin    31 Jun 29 08:06 QtHelp -> ../lib/QtHelp.framework/Headers
lrwxr-xr-x   1 user  admin    35 Jun 29 08:06 QtLocation -> ../lib/QtLocation.framework/Headers
lrwxr-xr-x   1 user  admin    36 Jun 29 08:06 QtMacExtras -> ../lib/QtMacExtras.framework/Headers
lrwxr-xr-x   1 user  admin    37 Jun 29 08:06 QtMultimedia -> ../lib/QtMultimedia.framework/Headers
lrwxr-xr-x   1 user  admin    44 Jun 29 08:06 QtMultimediaQuick_p -> ../lib/QtMultimediaQuick_p.framework/Headers
lrwxr-xr-x   1 user  admin    44 Jun 29 08:06 QtMultimediaWidgets -> ../lib/QtMultimediaWidgets.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 QtNetwork -> ../lib/QtNetwork.framework/Headers
lrwxr-xr-x   1 user  admin    38 Jun 29 08:06 QtNetworkAuth -> ../lib/QtNetworkAuth.framework/Headers
lrwxr-xr-x   1 user  admin    30 Jun 29 08:06 QtNfc -> ../lib/QtNfc.framework/Headers
lrwxr-xr-x   1 user  admin    33 Jun 29 08:06 QtOpenGL -> ../lib/QtOpenGL.framework/Headers
drwxr-xr-x   8 user  admin   272 Jun 29 08:06 QtOpenGLExtensions
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtPacketProtocol
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtPlatformCompositorSupport
drwxr-xr-x  30 user  admin  1020 Jun 29 08:06 QtPlatformHeaders
lrwxr-xr-x   1 user  admin    38 Jun 29 08:06 QtPositioning -> ../lib/QtPositioning.framework/Headers
lrwxr-xr-x   1 user  admin    39 Jun 29 08:06 QtPrintSupport -> ../lib/QtPrintSupport.framework/Headers
lrwxr-xr-x   1 user  admin    37 Jun 29 08:06 QtPurchasing -> ../lib/QtPurchasing.framework/Headers
lrwxr-xr-x   1 user  admin    30 Jun 29 08:06 QtQml -> ../lib/QtQml.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtQmlDebug
lrwxr-xr-x   1 user  admin    32 Jun 29 08:06 QtQuick -> ../lib/QtQuick.framework/Headers
lrwxr-xr-x   1 user  admin    41 Jun 29 08:06 QtQuickControls2 -> ../lib/QtQuickControls2.framework/Headers
lrwxr-xr-x   1 user  admin    41 Jun 29 08:06 QtQuickParticles -> ../lib/QtQuickParticles.framework/Headers
lrwxr-xr-x   1 user  admin    42 Jun 29 08:06 QtQuickTemplates2 -> ../lib/QtQuickTemplates2.framework/Headers
lrwxr-xr-x   1 user  admin    36 Jun 29 08:06 QtQuickTest -> ../lib/QtQuickTest.framework/Headers
lrwxr-xr-x   1 user  admin    39 Jun 29 08:06 QtQuickWidgets -> ../lib/QtQuickWidgets.framework/Headers
lrwxr-xr-x   1 user  admin    40 Jun 29 08:06 QtRemoteObjects -> ../lib/QtRemoteObjects.framework/Headers
lrwxr-xr-x   1 user  admin    36 Jun 29 08:06 QtRepParser -> ../lib/QtRepParser.framework/Headers
lrwxr-xr-x   1 user  admin    33 Jun 29 08:06 QtScript -> ../lib/QtScript.framework/Headers
lrwxr-xr-x   1 user  admin    38 Jun 29 08:06 QtScriptTools -> ../lib/QtScriptTools.framework/Headers
lrwxr-xr-x   1 user  admin    32 Jun 29 08:06 QtScxml -> ../lib/QtScxml.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 QtSensors -> ../lib/QtSensors.framework/Headers
lrwxr-xr-x   1 user  admin    36 Jun 29 08:06 QtSerialBus -> ../lib/QtSerialBus.framework/Headers
lrwxr-xr-x   1 user  admin    37 Jun 29 08:06 QtSerialPort -> ../lib/QtSerialPort.framework/Headers
lrwxr-xr-x   1 user  admin    30 Jun 29 08:06 QtSql -> ../lib/QtSql.framework/Headers
lrwxr-xr-x   1 user  admin    30 Jun 29 08:06 QtSvg -> ../lib/QtSvg.framework/Headers
lrwxr-xr-x   1 user  admin    31 Jun 29 08:06 QtTest -> ../lib/QtTest.framework/Headers
lrwxr-xr-x   1 user  admin    39 Jun 29 08:06 QtTextToSpeech -> ../lib/QtTextToSpeech.framework/Headers
drwxr-xr-x   7 user  admin   238 Jun 29 08:06 QtThemeSupport
lrwxr-xr-x   1 user  admin    35 Jun 29 08:06 QtUiPlugin -> ../lib/QtUiPlugin.framework/Headers
drwxr-xr-x   9 user  admin   306 Jun 29 08:06 QtUiTools
lrwxr-xr-x   1 user  admin    37 Jun 29 08:06 QtWebChannel -> ../lib/QtWebChannel.framework/Headers
lrwxr-xr-x   1 user  admin    36 Jun 29 08:06 QtWebEngine -> ../lib/QtWebEngine.framework/Headers
lrwxr-xr-x   1 user  admin    40 Jun 29 08:06 QtWebEngineCore -> ../lib/QtWebEngineCore.framework/Headers
lrwxr-xr-x   1 user  admin    43 Jun 29 08:06 QtWebEngineWidgets -> ../lib/QtWebEngineWidgets.framework/Headers
lrwxr-xr-x   1 user  admin    37 Jun 29 08:06 QtWebSockets -> ../lib/QtWebSockets.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 QtWebView -> ../lib/QtWebView.framework/Headers
lrwxr-xr-x   1 user  admin    34 Jun 29 08:06 QtWidgets -> ../lib/QtWidgets.framework/Headers
lrwxr-xr-x   1 user  admin    30 Jun 29 08:06 QtXml -> ../lib/QtXml.framework/Headers
lrwxr-xr-x   1 user  admin    38 Jun 29 08:06 QtXmlPatterns -> ../lib/QtXmlPatterns.framework/Headers

Same for Qt 5.7.1 with clang 3.9.1 on linux.

Looks like it because of qt_core fails.
qt_core/log/debug_parser.log https://gist.github.com/o01eg/6064fb33abb6914ba0feb770c77c2bbd

Looks like Qt::WindowFlags type is parsed successfully in qt_core, but when the generator works on qt_widgets, it doesn't have this information available. What are the command line arguments to qt_generator that you use when generating qt_core, qt_gui and qt_widgets?

Not sure if it successful:
Failed to tokenize method operator| at SourceRange { start: SourceLocation { file: Some(File { path: "/usr/include/qt5/QtCore/qnamespace.h" }), line: 1758, column: 1, offset: 53192 }, end: SourceLocation { file: Some(File { path: "/usr/include/qt5/QtCore/qnamespace.h" }), line: 1758, column: 47, offset: 53238 } }
The code extracted directly from header: "Q_DECLARE_OPERATORS_FOR_FLAGS(Qt::WindowFlags)"

Args are:
CLANG_SYSTEM_INCLUDE_PATH=/usr/lib64/clang/3.9.1/include/ QT_SELECT=5 cargo run --release -- -c /tmp/_qt_cache -C 0 --debug-logging save -l core gui widgets ui_tools -o /tmp/qt-test001/

What are the command line arguments to qt_generator that you use when generating qt_core, qt_gui and qt_widgets?

export PATH=$PATH:/usr/local/Cellar/qt/5.9.1/bin
CLANG_SYSTEM_INCLUDE_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-migrator/sdks/MacOSX.sdk/usr/include DYLD_LIBRARY_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib ../cpp_to_rust/qt_generator/qt_generator/target/release/qt_generator -c cache -o qt_5.8.0 --libs all --debug-logging=save

The framework detection issue should be fixed now.
Source of the main issue is not clear yet.

Still occurs at Kubuntu 18.04.1, Qt 5.9.5, qt_generator taken from git.

It's correct that there are types we're not going to support, and things like QIntegerForSizeof and QAtomicTraits are among them. Luckily, Qt doesn't use advanced templates heavily, so the majority of Qt's public API will be available.

Is there a way then to improve Rust-Qt to be usable with the recent Qt versions? The generator from master cannot chew the templated types, and the crates version is old and leads to the random runtime segfaults.

The master version does support template types. However, its results are quite unstable across platforms, so some APIs may be missing. The new version is still in development, and the plan is to support newer Qt versions for it.
If you experience segfaults in the crates.io version, I would appreciate an issue that shows concrete code that produces it. Unfortunately, the wrapper is really unsafe in Rust's terms, so it's very easy to produce a segfault by incorrectly using C++-based APIs.

During trying to get the simplest code for segfaults reproduction I have accidentally fixed it. Did not use the create_and_exit method from the tutorial and was dropping args all the time.

@Riateche I see that you updated the "install_qt.py" script with Qt 5.13.0.
Does ritual support Qt 5.13.0? On docs.io it says that the crate was generated against Qt 5.8.0(if it supports Qt 5.13.0, could you please update it against Qt 5.13.0?).
Would really like to use that seamless ETC support in QtQuick :)

Qt 5.13 is currently supported, but new versions are not published on crates.io yet. I'm working on it.

New version supporting Qt 5.13 is published on crates.io.
In the new version, the recommended way is to use generated crates that support a variety of Qt versions and platforms, so lack of some random functions shouldn't be a thing anymore.

license: mit