text:
Fix: Limit the number of generated questions

Description
Limit the number of generated questions to avoid cases where more questions are generated than requested by the parameter num_questions_per_chunk.
Fixes #10694 and #10089
Added the fix to all the modules mentioning num_questions_per_chunk except the following modules that contain deprecated classes that possibly do not have this issue due to their num parameter in _agenerate_dataset method:

llama-index-core/llama_index/core/evaluation/dataset_generation.py

The module implements QueryResponseDataset that is deprecated in favor of LabelledRagDataset

llama-index-legacy/llama_index/legacy/evaluation/dataset_generation.py

The module implements DatasetGenerator that is deprecated in favor of RagDatasetGenerator

New Package?

[ ] Yes
[X] No

Version Bump?

[X] Yes
[ ] No

Type of Change

[X] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?
from llama_index.core.llama_dataset.generator import RagDatasetGenerator
from llama_index.llms.ollama import Ollama
from llama_index.core.schema import TextNode

node = TextNode(
    text="New York, often called New York City or simply NYC, "
    "is the most populous city in the United States, located at the "
    "southern tip of New York State on one of the world's largest "
    "natural harbors. The city comprises five boroughs, each of which "
    "is coextensive with a respective county. New York is a global center "
    "of finance and commerce, culture and technology, entertainment and "
    "media, academics and scientific output, and the arts and fashion, and, "
    "as home to the headquarters of the United Nations, is an important "
    "center for international diplomacy. New York City is the center of "
    "the world's principal metropolitan economy."
)

llm = Ollama(model="gemma:2b", request_timeout=60.0)
num_questions_per_chunk = 1
for i in range(10):
    gen = RagDatasetGenerator(
        nodes=[node],
        llm=llm,
        num_questions_per_chunk=num_questions_per_chunk,
    )
    generated_questions = gen.generate_questions_from_nodes()
    print(
        f"Number of questions expected: {num_questions_per_chunk} - "
        f"Number of questions generated: {len(generated_questions.examples)}"
    )

Suggested Checklist:

[X] I have performed a self-review of my own code
[X] My changes generate no new warnings
[X] I ran make format; make lint to appease the lint gods

@tuomastik that makes sense. Maybe for now, we should at least raise a warning to the user that the LLM call resulted in less than the desired amount of questions per chunk. Similar to here: https://github.com/run-llama/llama_index/blob/8c23e0140ab5b74f1fd8ade22c29b4bd758cec46/llama-index-core/llama_index/core/ingestion/pipeline.py#L713

Maybe for now, we should at least raise a warning to the user that the LLM call resulted in less than the desired amount of questions per chunk.

✅ Done

authors: nerdai tuomastik


text:
Loot tracker plugin - track implings and impling jars

Example:

--
Works with multiple jars per game tick.
Single entry view:

Grouped view:

#8810

Also this needs probably some significant rebase due to the loot tracker metadata stuff.

added in 08359a0df0f31bfeac49d286395c4d0dbcaaadfc and a877c5800546b5c28727224e114f058d473d7894 and 88933be3df096c850a984922ff3a2025671f07b3

source: gharchive/pull-request

repo: runelite/runelite

license_type: permissive

license_source: github-api


text:
Fixed clue scroll typo

Changed "hosptial" to "hospital"

Suppose I could have created an issue instead of PR for this instead.


text:
added new awesome opensource project

Fixes Issue
Closes : #817
Added an awesome open-source project which i personally used and liked it !
Screenshots

please add labels and assignee !

looks good though, but the project is neither active nor accepting contributors..should we still keep it in _open source projects?

Hi @ujjawaltyagii. I just checked out the project. I agree with @rupali-codes. It probably won't be best to put an inactive open source project. Maybe you can replace it with a more active project?

okay ! will update it with an active project

repo: rupali-codes/LinksHub

license: MIT


text:
Poco NetSSL Linkage

I was trying to use Poco to send https requests. I looked through the cmake_args and set the correct flags ON as so:
hunter_config(PocoCpp VERSION 1.7.6-p0 CMAKE_ARGS ENABLE_NETSSL=ON ENABLE_CRYPTO=ON)

(ENABLE_NETSSL=ON only works when ENABLE_CRYPTO=ON)
After the build, I was able to find and add the header files and reference the classes but got linkage errors.
undefined symbols for architecture x86_64:
  "Poco::Net::HTTPSClientSession::HTTPSClientSession(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned short, Poco::AutoPtr<Poco::Net::Context>)", referenced from:

So after some testing I found out that I had to add the NetSSL to the package in the CMakeLists.txt
find_package(Poco REQUIRED Foundation Net NetSSL CONFIG)
target_link_libraries(Foo Poco::Foundation Poco::Net Poco::NetSSL OpenSSL::SSL OpenSSL::Crypto)

(Not sure if the OpenSSL packages are a dependency)
NetSSL (Poco::NetSSL) should be added to available components in the documentation. And it would be better if the Hunter Poco documentation covered all of this a bit better.

NetSSL (Poco::NetSSL) should be added to available components in the documentation. And it would be better if the Hunter Poco documentation covered all of this a bit better.

Page pkg.pococpp is available for editing, feel free to add any useful information.

Closing inactive discussion.

source: gharchive/issue

repo: ruslo/hunter

license_source: bigquery


text:
Add IAM capability

We need to support IAM for various other services.  PR #212 is an example of a service that we can't fully integration test without IAM support in Rusoto.

I'm going to take a look at this.

This is done!

repo: rusoto/rusoto

license: mit


text:
Bad parsing recovery on missing for with impl

rust-analyzer gives errors at the specified locations:
impl<'tcx> rustc::ty::fold::TypeVisitor<'tcx> UI128Visitor {
//                                           ^
    fn visit_ty(&mut self, t: Ty<'tcx>) -> bool {
//  ^ ^        ^^^  ^^           ^    ^^^ ^    ^^
        t.super_visit_with(self)
    }
}

Rustc gives just one error and doesn't abort parsing (proof: adding syntax error in fn body is catched):
error: missing `for` in a trait impl
  --> src/base.rs:75:54
   |
75 | impl<'tcx> rustc::ty::fold::TypeVisitor<'tcx> UI128Visitor {
   |                                              ^ help: add `for` here

I don't think we hardcode any better errors for specific mistakes, but it is always worth adding specific recovery paths. [I'd say this is help-wanted E-easy, but only P-low (if we had such a designation)].
To implement this, if at the impl item parsing, when we check for the FOR keyword we should check if we are at_ts PATH_START, then parse a path and try and parse the inner items then

Filed https://github.com/rust-analyzer/rust-analyzer/pull/925 which should avoid cascading erros in similar cases. #925 will parse the contents of any unrecognized block as an expression.
Adding a specific case for missing for will be useful, but I don't think this would have a good cost/benefit ratio at this stage: we are not yet aiming at producing good error messages.

authors: DJMcNab bjorn3 matklad

repo: rust-analyzer/rust-analyzer


text:
serde trait impls missing for WitnessVersion, WitnessProgram, and ArrayVec

The first two require the third.

We can leave this open but I don't think we should prioritize this until somebody complains.

I'd like to use it in rust-bitcond-json-rpc.

authors: apoelstra tcharding

repo: rust-bitcoin/rust-bitcoin


text:
remove sketchy LikelyFalse error

This error would trigger on l:0 on the basis that this is equivalent to u:0 but always less efficient. There are no other "linting" errors like this in this library, and the C++ implementation doesn't have it, so we should drop it.
Fixes #633

CI is broken because of https://github.com/rust-lang/rust/issues/121684

Added a pile of commits and filed an issue on rustc to fix CI.

I can't properly ack because I don't feel super confident with my knowledge of the repo (talking about b87b4fba8ffb2712e0b5a67cddf77d5dfc690a6b) but
LGTM

cc @sanket1729 can you ACK at least the first commit here?

Sure. Will do in 10 mins

Sure. Will do in 10 mins

authors: RCasatta apoelstra sanket1729

repo: rust-bitcoin/rust-miniscript


text:
cargo-objcopy UI

This issue is about improving the UI of the cargo-objcopy subcommand to make it feel more
integrated with Cargo. Possible additions:

cargo objcopy [--target $T] --example $E [--release] -- -O binary. Builds $E for $T and then
calls llvm-objcopy -O binary target/$T/{debug,release}/examples/$E.

Thoughts? Suggestions?
cc @jcsoo

v0.1.2 implements --bin --example, --lib, --release and --target flags which can be used to build a binary / example / library and transform the build artifact -- no need to pass the path to the artifact as an argument. Examples.
if there's any other UI improvement you'd like to see please open a new issue. I'm closing this one as "done".

repo: rust-embedded/cargo-binutils


text:
Fix missing capture module

cc @eldruin
This must have gotten lost during #298

r? @ryankurte
(rust-highfive has picked a reviewer for you, use r? to override)

repo: rust-embedded/embedded-hal


text:
Newsletter 37 - August 2022

Newsletter 37: August 2022
tags: newsletter
Editors: @AngelOnFira, @17cupsofcoffee, and @ozkriff
Another month has gone by, so it's time to put together the Rust Gamedev newsletter with August's news!
Current Schedule
The soft deadline for all section PRs is the 7th of September.
We may still accept PRs that are submitted later than this, as long as they're ready before the newsletter's release, but this isn't guaranteed. If you want your section to be included, don't leave it till the last minute!
How to Contribute
If you want to help writing the newsletter:

Read CONTRIBUTING.md.
Choose one or more of the "🆓 free" sections listed below, and leave a comment letting us know you want to work on them.

The links in brackets (like "1, 2, 3") are suggestions of links to include in the section. Feel free to add more!
The username listed next to the section (like "@ozkriff?") is a suggestion of who may want to pick up the work (usually the project's developer, or someone who has expressed interest in the past).
You are not obligated to write a section if you're tagged or your project is listed! You're welcome to ask someone else to write the section, or to ask for your project to be excluded from this month's post.
Extra sections not listed in the plan are welcomed - just leave a comment and open a PR!

Write a short overview in the newsletter's Markdown file, making sure to follow the style guidelines (see below).
Send a PR to the source branch (example: #336).
Mention this issue in your PR's description to link it all together.

Style Guidelines
The full style guide is in CONTRIBUTING.md, but here are the most important rules:

Write in third-person perspective.
Each line must be 80 characters or less, for ease of reviewing/diffing.
Only one image per section is allowed.

The maximum size is 300kb for static images and 2.5mb for GIFs.
The image should come before the text, and must have alt text for accessibility.
Prefer static images to GIFs, to keep the page load times down.

Each section should be under 1000 characters, and under 6 paragraphs.

This only applies to the rendered text, not the markup.

Keep formatting minimal - no bold/italics/etc.
Avoid long/nested bullet point lists - no changelogs!

Please use these templates as a starting point:
Games/apps/libraries:
### [Game name]

![alt text](img)
_optional image label_

[Game name] ([GitHub], [Discord], [Twitter]) by [@nickname]
is... {short project description in one sentence}.

{An overview of the recent updates with links to more details}.

_Discussions: [/r/rust_gamedev](link), [Twitter](link), [etc](link)_

[Game name]: http://example.com

Articles/blog posts/videos/etc:
### [Article name]

![alt text](img)
_optional image label_

[@nickname] published an [article] about...
{overview what the resource is about}.

_Discussions: [/r/rust_gamedev](link), [Twiter](link), [etc](link)_

[Article name]: http://example.com

Current Structure & Status
Below is our current planned structure for the newsletter, and the status of each PR (which we'll try to keep updated).
This is not an exhaustive list - if you have your own project that you want to write about, just make a comment on this issue and open a PR!

Announcements

[ ] Rust Gamedev meetup - 🚧 WIP by @AngelOnFira
[ ] Arcade Cabinet (1, 2, 3) - 🚧 WIP by @cdsupina
[ ] bevyjam 2 (1, 2, 3, 4) - 🆓 free (@cart?)
[ ] Bombercrab Challenge (1) - 🆓 free (@PabloMansanet?)

Game Updates

[ ] Country Slice (1, 2) - 🆓 free (@anopara?)
[ ] Veloren (1) - 🚧 WIP by @AngelOnFira
[ ] Hydrofoil Generation (1) - 🚧 WIP by @kunos
[ ] Caveth (1, 2) - 🆓 free (@Dequog?)
[ ] Flesh (1, 2, 3, 4) - 🆓 free (@Im-Oab?)
[ ] Punchy (1, 2, 3) - 🆓 free (@erlend-sh?)
[ ] Spherical Go (1, 2) - 🆓 free (@Dominux?)
[ ] Combine And Conquer (1, 2) - 🆓 free (@I3ck?)
[ ] Rusty Aquarium (1, 2) - 🆓 free (@ollej?)
[ ] Infinite Bunner (1, 2, 3) 🆓 free (@ollej?)
[ ] bevy-cheeseball (1, 2) -  - 🆓 free (@Rust-Ninja-Sabi?)
[ ] Cyber Gate (1) - 🚧 WIP by @cybersoulK

Learning Material Updates

[ ] Bevy Basics video series (1) - 🆓 free (@PhaestusFox?)
[ ] Rust wgpu series (1) - 🆓 free (@jack1232?)
[ ] console 118: rend3 (1) - 🆓 free (@cwfitzgerald?)
[ ] Integrating Rust code into Unity (1, 2) - 🆓 free (@nullvision?)
[ ] Learn OpenGL with Rust Series (1, 2) - 🆓 free (@samkevich?)

Engine Updates

[ ] Fyrox v0.27 (1, 2, 3, 4, 5, 6) - 🚧 WIP by @mrdimas
[ ] Pyxel v0.18 (1, 2) - 🆓 free (@kitao)
[ ] Gamercade (1, 2, 3, 4, 5, 6) - 🆓 free (@RobDavenport?)
[ ] runty8 (1, 2) - 🆓 free (@jjant?)
[ ] Bevy (1, 2, 3) - 🆓 free (@cart?)

Tooling Updates

[ ] Graphite - 🚧 WIP by @Keavon
[ ] noumenal (1) - 🆓 free (@hackerfoo?)
[ ] nbody-wasm-sim (1, 2) - 🆓 free (@simbleau?)
[ ] WGEN (1, 2) - 🆓 free (@jice-nospam?)
[ ] Bevy Shell Template (1, 2) - 🆓 free (@kurbos?)

Library Updates

[ ] bevy_streamdeck (1, 2) - 🆓 free (@vleue?)
[ ] Shipyard 0.6 (1) - 🆓 free (@leudz?)
[ ] kira 0.7 (1) - 🆓 free (@tesselode?)
[ ] bevy_kira_audio 0.11 (1, 2) - 🆓 free (@NiklasEi?)
[ ] edict (1, 2) - 🆓 free (@zakarumych?)
[ ] egui 0.19 (1) - 🆓 free (emilk?)
[ ] Goggles (1) - 🆓 free (@erlend-sh?)
[ ] notan v0.6 (1) - 🆓 free (@Nazariglez?)
[ ] maths (1, 2, 3) - 🆓 free (@polymonster?)
[ ] FunDSP v0.8 (1, 2) - 🆓 free (@SamiPerttu?)
[ ] hot-lib-reloader-rs (1) - 🆓 free (@rksm?)
[ ] bevy_ecs_tilemap (1) - 🆓 free (@StarArawn?)
[ ] grid_pathfinding (1) - 🆓 free (@tbvanderwoude?)

Other News
Discussions
Jobs
Publishing Steps

[ ] Final review - by everyone
[ ] Publish - by @AngelOnFira
[ ] Post on /r/rust, /r/rust_gamedev, /r/gamedev, URLO, twitter.com/rust_gamedev - by @AngelOnFira
[ ] Pin thread on Twitter - by @ozkriff
[ ] Add comment links - by @AngelOnFira
[ ] Add a draft of next month's newsletter - by @AngelOnFira

I'm adding a Game Updates entry for the Catacomb II port :smile:

I'll take Engine Updates - Gamercade again this week.

Not listed, but I'll post a miniquad update tomorrow  :)

I’ll writw something about Infinite Bunner. And maybe Rusty Aquarium as well.

I'll take Bevy's update (and Bevy Jam)

I'll write about Edict

Hey there, I just found out about this newsletter. I'll write something about Runty8 (though it's in very early WIP stages).

Added Notan v0.6, thanks!

PR Up for Gamercade. Thanks!

Country Slice -> Tiny Building Game 😅 figured I'd better call it like we do in twitter/newsletter/etc :3

authors: 64kramsystem AngelOnFira Nazariglez RobDavenport anopara cart jjant not-fl3 ollej zakarumych


text:
Consider relationship to marketing-oriented "explainers"

We're going to want to create very brief explainers about the epoch concept and the various features in Rust 2018. It's not yet clear how we'll want to organize those, but it's obviously related to this guide!
cc @ashleygwilliams @jonathandturner

I did this in https://github.com/rust-lang-nursery/edition-guide/commit/8bb94db1ef7b61689c88b66d35e7237f8d9ba382

authors: aturon steveklabnik

repo: rust-lang-nursery/edition-guide


text:
State 0.3.0-alpha.18

r? @taiki-e @Nemo157 @seanmonstar

cc @taiki-e looks like buffer_unordered::works is flaking :(
will investigate

In any case, it's not new to this release-- only the test is new, so I'm going to go ahead and merge this.

Opened https://github.com/rust-lang-nursery/futures-rs/issues/1790 to track the flake.

repo: rust-lang-nursery/futures-rs


text:
Changing env_logger's log format

What's the recommended way to change env_logger's log format?
I have tried to override the Log trait for Logger to change the log format but can't see how to do it.

env_logger's output format is hard coded at the moment: https://github.com/rust-lang/log/blob/master/env/src/lib.rs#L178-L182

I have had a go at implementing a way to set a custom log format
https://github.com/nicksanders/log/commit/098f9b71c51078c33b1c260ca04ee25ed8898cd2
I have added a function init_with_log_writer (see below for an example). If its along the right lines let me know and I can add tests and documentation as needed.

#[macro_use]
extern crate log;
extern crate env_logger;
extern crate time;

use log::{LogLevel, LogRecord};
use env_logger::LogWriter;

struct CustomLogWriter;

impl LogWriter for CustomLogWriter {
    fn write(&self, record: &LogRecord) {
        let t = time::now();
        println!(
            "{},{:03} - {} - {}",
            time::strftime("%Y-%m-%d %H:%M:%S", &t).unwrap(),
            t.tm_nsec / 1000_000,
            record.level(),
            record.args()
        );
    }
}

fn main() {
    env_logger::init_with_log_writer(CustomLogWriter).unwrap();
    error!("this is printed by default");
}

I have had a another go, this time using a closure for formatting the log.
It also allows you to change the default log level and environmental variable.
I think its a lot nicer, what do you think?
https://github.com/nicksanders/log/commit/381f44e902fcd68aa6b57a2f80287b1b1c0e67b1

#[macro_use]
extern crate log;
extern crate env_logger;
extern crate time;

use log::{LogRecord, LogLevelFilter};
use env_logger::LogConfig;

fn main() {
    let formatter = |record: &LogRecord| {
        let t = time::now();
        format!("{},{:03} - {} - {}",
            time::strftime("%Y-%m-%d %H:%M:%S", &t).unwrap(),
            t.tm_nsec / 1000_000,
            record.level(),
            record.args()
        )
    };

    let config = LogConfig {
        env_var: "RUST_LOGGER",
        format: Box::new(formatter),
        level: LogLevelFilter::Info,
    };

    env_logger::init_with_config(config).unwrap();

    error!("error message");
    info!("info message");
}

+1

can we get this in? i’d really like to see the time without wrapping all the macros with my own…

This has just been merged and released :)
See #48

haha awesome! my story with rust so far:
started translating some python script on thursday, got it done monday (only three days!).
met 3 bugs/feature requests that i required, all of which are fixed by now: this one, alexcrichton/flate2-rs#30 (which i pinned down and the author fixed), shepmaster/sxd-document#27 (which i fixed, but it isn’t merged yet)

authors: flying-sheep nicksanders sfackler softprops

repo: rust-lang-nursery/log


text:
Destructor missing for templated class (like smart pointer)

Input C/C++ Header
template<class T>
class Pointer {
 private:
  T* data;
  char xx;
  char yy;
 public:
  Pointer(T* data);
  ~Pointer();

  int method();

  T& operator*();
  T* operator->();
};

class A {
 public:
  int b, c, d;
  Pointer x;
};

Pointer MakeA();

Bindgen Invocation
$ bindgen test_bind.hpp -o src/bindings.rs -- -x c++ -std=c++11

Actual Results
#[repr(C)]
#[derive(Debug)]
pub struct Pointer<T> {
    pub data: *mut T,
    pub xx: ::std::os::raw::c_char,
    pub yy: ::std::os::raw::c_char,
    pub _phantom_0: ::std::marker::PhantomData<::std::cell::UnsafeCell<T>>,
}
#[repr(C)]
#[derive(Debug)]
pub struct A {
    pub b: ::std::os::raw::c_int,
    pub c: ::std::os::raw::c_int,
    pub d: ::std::os::raw::c_int,
    pub x: Pointer,
}
#[test]
fn bindgen_test_layout_A() {
    assert_eq!(::std::mem::size_of::() , 32usize , concat ! (
               "Size of: " , stringify ! ( A ) ));
    assert_eq! (::std::mem::align_of::() , 8usize , concat ! (
                "Alignment of " , stringify ! ( A ) ));
    assert_eq! (unsafe { & ( * ( 0 as * const A ) ) . b as * const _ as usize
                } , 0usize , concat ! (
                "Alignment of field: " , stringify ! ( A ) , "::" , stringify
                ! ( b ) ));
    assert_eq! (unsafe { & ( * ( 0 as * const A ) ) . c as * const _ as usize
                } , 4usize , concat ! (
                "Alignment of field: " , stringify ! ( A ) , "::" , stringify
                ! ( c ) ));
    assert_eq! (unsafe { & ( * ( 0 as * const A ) ) . d as * const _ as usize
                } , 8usize , concat ! (
                "Alignment of field: " , stringify ! ( A ) , "::" , stringify
                ! ( d ) ));
    assert_eq! (unsafe { & ( * ( 0 as * const A ) ) . x as * const _ as usize
                } , 16usize , concat ! (
                "Alignment of field: " , stringify ! ( A ) , "::" , stringify
                ! ( x ) ));
}
extern "C" {
    #[link_name = "_Z5MakeAv"]
    pub fn MakeA() -> Pointer;
}
#[test]
fn __bindgen_test_layout_Pointer_open0_A_close0_instantiation() {
    assert_eq!(::std::mem::size_of::<Pointer>() , 16usize , concat ! (
               "Size of template specialization: " , stringify ! ( Pointer
               ) ));
    assert_eq!(::std::mem::align_of::<Pointer>() , 8usize , concat ! (
               "Alignment of template specialization: " , stringify ! (
               Pointer ) ));
}
#[test]
fn __bindgen_test_layout_Pointer_open0_A_close0_instantiation_1() {
    assert_eq!(::std::mem::size_of::<Pointer>() , 16usize , concat ! (
               "Size of template specialization: " , stringify ! ( Pointer
               ) ));
    assert_eq!(::std::mem::align_of::<Pointer>() , 8usize , concat ! (
               "Alignment of template specialization: " , stringify ! (
               Pointer ) ));
}

Expected Results
Destructor for Pointer is missing.
Same behaviour happens with ordinary c++ shared_ptr.

Thanks for the bug report! This is, unfortunately, expected for two reasons:

The destructor is inline (I was inspired to start an FAQ to explain this: https://github.com/rust-lang-nursery/rust-bindgen/pull/1020)

We don't know what monomorphizations of the function exist (or will exist!) and we aren't a C++ compiler, so we can't generate new monomorphizations. We do have #492 open to generate bindings to explicitly instantiated templates.

I'm going to close this issue, since I think #492 covers all the actionable bits.

authors: fitzgen usamec

repo: rust-lang-nursery/rust-bindgen


text:
ir: Pass the target to clang if it wasn't explicitly passed already.

Fixes #942
Fixes #947

r? @fitzgen
Let me know if you can think of a test that passes / fails consistently across platforms, couldn't think of one, though I didn't tried extremely hard.

@bors-servo r+

:pushpin: Commit 26da344 has been approved by fitzgen

:hourglass: Testing commit 26da3447a73d0364849f0325f222c4a8c9337550 with merge b209d42444dd3dbe3b57590313d4f47979225492...

:sunny: Test successful - status-travis
Approved by: fitzgen
Pushing b209d42444dd3dbe3b57590313d4f47979225492 to master...

authors: bors-servo emilio fitzgen


text:
Add issue triage docs

cc @alexcrichton
r? @aturon

👍

authors: Mark-Simulacrum alexcrichton

repo: rust-lang-nursery/rust-forge


text:
tracking issue for documenting chalk and the trait system

We have some great docs on the trait system already (if I do say so myself 😉) but there's lots of room for improvement. This issue exists to track ideas and try to stay organized.

[x] Update rustc guide with new rules for GATs (cc @tmandry, @scalexm)

Might be good to link to this talk by @sunjay somewhere: https://www.youtube.com/watch?v=rZqS4bLPL24
It's a great intro :)

authors: mark-i-m nikomatsakis

repo: rust-lang-nursery/rustc-guide


text:
Add to definition of P<T> type to Appendix D: Code Index

And why it's needed.

@eddyb's definition of it is an "immutable Box<T>". I suppose it's needed just due to the nature of tree data structures?

It's not "needed", not to a large extent. The actual code has (potentially-outdated) documentation.

@eddyb Well, heap allocation is surely needed, otherwise we'd have to deal with an immutable tree, and that would be very expensive during construction and later modification.

I mean as a separate concept from Box or anything else.

Sorry, I'm not sure I understand. We use P when we need an owned immutable pointer. We can't use &T because that is not owned and we can't use Box because it is not immutable. Is that correct?

@eddyb Right, fair.
@mark-i-m More or less, yes... @eddyb is saying that P<> doesn't really provide anything new over Box<> though – it's just a new-type with a immutability guarantee.

authors: alexreg eddyb mark-i-m


text:
rustfmt behavior with generated code

I've been using prost recently, and found that rustfmt complains about the generated code (particularly around documentation format). I would have expected rustfmt to ignore the generated code added to the project with include!().
The proto files I'm using are shared so it doesn't make sense to enforce rust-style on the .proto files. Is there a way to exclude certain modules from format checking?

Interesting. Rustfmt runs on pre-macro-expanded code, so it should ignore anything brought in by include!.

Could you try and create a minimal test case please?

Will do!

authors: ameets nrc

repo: rust-lang-nursery/rustfmt


text:
Document having only src/bin/*.rs binaries is valid

Problem
I found the following section of the cargo book could make readers think they must provide src/main.rs.
https://github.com/rust-lang/cargo/blob/59ecb11a298f0c3740216220361024cf05ed5b5a/src/doc/src/reference/cargo-targets.md?plain=1#L30-L34
However, it is also valid to have only src/bin/*.rs binaries and not to have src/main.rs.
I believe clarifying this situation on the book will help readers when arranging codes for binaries.
Proposed Solution
Changing the section like below should work:
Binary targets are executable programs that can be run after being compiled.
The default binary filename is `src/main.rs`, which defaults to the name of
the package. Additionally one or more binaries are stored in the [`src/bin/`
 directory][package layout]. In this case the default binary `src/main.rs`
is not required. The settings for each binary can be [customized] in the
`[[bin]]` tables in `Cargo.toml`.

Notes
No response

@Dnts1 looks like you posted your comment in a wrong issue.

Thanks for providing this feedback on the confusion you had!  It looks like the suggested change is the addition of

n this case the default binary src/main.rs  is not required.

It feels odd to call out that a default is not required, that is generally the case for a default.
For brainstorming purposes, another approach is something like
Binary targets are executable programs that can be run after being compiled.
A binary's source can be `src/main.rs` and/or stored in the  [`src/bin/`
directory][package layout].  For `src/main.rs`, the default binary name is the package name. The settings for each binary can be [customized]
in the `[[bin]]` tables in `Cargo.toml`.

@epage Thank you for the quick response!
I support your fix.
Using and/or make the section much clearer than my take, and also stating *.rs are sources sounds more natural than the original text.
Maybe we need to make sure the text is wrapped at 80 characters though.
I'm considering making a PR after waiting feedbacks for a couple of days.

Since no one seems to submit another proposal, I’m going to make a PR based on the @epage ‘s text to keep this issue from being stale.

authors: epage nekomachi-touge

repo: rust-lang/cargo


text:
cdylib on Windows provides only the .dll without the associated .lib import library

On Windows, runing cargo build on a cdylib crate produces only the .dll in target/{debug,release} while the associated .lib import library is missing.
However, the following files are present under target/{debug,release}/deps:

<name>-<hash>.dll
<name>-<hash>.dll.lib
<name>-<hash>.dll.exp
<name>-<hash>.pdb

The cargo and rustc versions in use are:

cargo-0.17.0-nightly (f9e5481 2017-03-03)
rustc 1.16.0 (30cf806ef 2017-03-10)

(originally opened as https://github.com/rust-lang/rust/issues/41393)

The import library is hardcoded to the name of the dll at the time of linking, so since cargo tells rustc to tell the linker to name the dll <name>-<hash>.dll, that means the import library is hardcoded to <name>-<hash>.dll. Considering the user actually wants to have <name>.dll, this is a major blocker towards being able to use cdylibs. Cargo needs to seriously rethink how it handles naming output artifacts.
cc @alexcrichton


text:
dev-dependencies of package are not included when using cargo test -p

For example, consider a package that has dependency on memchr:
[dependencies]
memchr = "2.0.1"

To validate that dependencies work correctly, I would like to run tests distributed with them. In this case to validate memchr I execute following:
$ cargo test -p memchr
Compiling memchr v2.0.1
error[E0463]: can't find crate for `quickcheck`

I would expect it to work fine, because it works if I were to run tests directly from within memchr. It fails instead. As far as I can see, declaring something as dev dependency is sufficient to reproduct this issue. In this particular case, the quickcheck is a dev dependency in memchr:
[dev-dependencies]
quickcheck = { version = "0.6", default-features = false }

Thanks for the report! Would you be able to gist a sample project exhibiting this behavior?

$ cargo new --bin foo
$ cd foo
$ echo 'memchr = "2.0.1"' >> Cargo.toml
$ cargo test -p memchr
    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling libc v0.2.39
   Compiling memchr v2.0.1
error[E0463]: can't find crate for `quickcheck`

Oh sorry I misinterpreted! This is actually intended behavior in the sense that cargo test -p pointing at crates on crates.io has always been sort of broken. The dev-dependencies for these crates are't included in Cargo.lock, so Cargo should actually be giving you a better error here saying "sorry, but that can't be done"

authors: alexcrichton tmiasko


text:
Support per pkg target for -Zbuild-std

Fixes #9451.
cc @alexcrichton
To review this PR, you can look at the commit summaries and descriptions.

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) soon.
Please see the contribution instructions for more information.

r? @alexcrichton

:umbrella: The latest upstream changes (presumably #9992) made this pull request unmergeable. Please resolve the merge conflicts.

Coming back to this, I don't believe build-std currently has a problem with proc-macro and buildscript builds confusing host-std-that-we-ship with std-that-gets-built-with-build-std. These lines:
https://github.com/rust-lang/cargo/blob/c9a8199e927192d4f03f73a14f97faa29c9df956/src/cargo/core/compiler/unit_dependencies.rs#L159-L168
.. already avoid attaching custom std deps for host compile kinds. This means that if a user did not specify the compile kind to use for a specific unit, it will just use the shipped-std. Only when specifying a target, through per-pkg-target or --target is when build-std will build a std for that specific target and attach std to those crates that requested those.

Hi! @fee1-dea, I'm really looking forward to see this change, we really need it in https://github.com/aya-rs/aya ecosystem. Could you please rebase it?

Hi! @fee1-dead I'm really looking forward to see this change, we really need it in https://github.com/aya-rs/aya ecosystem. Could you please rebase it?

r? @ehuss
@rustbot ready

There seems to be a discussion up at https://github.com/rust-lang/cargo/pull/10330#discussion_r794815070 that doesn't appear to be resolved.  I don't think the --target flag requirement can just be removed without major effort.  There's a bit more discussion at https://github.com/rust-lang/wg-cargo-std-aware/issues/25.
Also, similar to #10405, I don't think this is a complete solution since per-package targets might come from registry dependencies.  Those won't get std built correctly.

:umbrella: The latest upstream changes (presumably #10129) made this pull request unmergeable. Please resolve the merge conflicts.

Okay so I did some testing and found that no --target basically works the same as --target $HOST. I have updated some tests to make sure that this is indeed true for proc macro crates to use sysroot crates instead of the custom built one.
@rustbot ready

:umbrella: The latest upstream changes (presumably #9892) made this pull request unmergeable. Please resolve the merge conflicts.

@ehuss Do you need me to rebase? I am unfamiliar with how artifact dependencies work and I would like this to land. Since these two are both unstable features now, could we note the certain incompatibilities and just merge it for now for further testing?

Hm, I was perhaps not understanding the intent of the per-package-target feature. I did not realize that it doesn't work with dependencies at all. That doesn't match the motivation for the original issue (#7004), so I'm not really clear how per-package-target is intended to be used. If you or others can put some clear use cases in the tracking issue, that would be helpful.
As for moving forward with this, I think that may be possible. There are still issues around resolution, features, and target-info collection. But those aren't new.
Just some general feedback from a quick look:

The restriction on specifying --target can't be removed at this time.
The use of package_set.packages() to fetch a list of root packages of interest seems fragile. I'd like to see a more principled way of being explicit that "these are all of the kinds of the root set".
It isn't clear why there is a change related to --build-plan

Interaction with artifact dependencies can be ruled with requirement for each artifact dep to always have its target specified. Implicit behavior here can always be added later and is unclear now (should it be inherited from dependent/workspace/the package itself???)

authors: alexcrichton bors ehuss fee1-dead rust-highfive tema3210 vadorovsky


text:
cargo add - fix for adding features from repository with multiple packages.

Fixes #13121
As discussed in the issue, when using cargo add to add a package with features from a git repository from one of it's members, the command might fail due to improper target for querying for said features.
This PR adds a test for this edge-case where we expect it to fail with current code. It also adds a fix for this, and updates the test to expect success.
While populating available features, the current code does a Fuzzy search which might lead to searching for features in a wrong member package. If we change it to an Exact query, we get back the proper member to search within.

Thanks for picking this up and moving it along!

have made the changes.
first commit - add a test that expects failure from said command (test passes)
second commit - fixes issue, updates test to expect success + appropriate output (test passes)

Thanks!
@bors r+

:pushpin: Commit e46d80e24ebf8504eec7b98463ff8f909e6cba3c has been approved by epage
It is now in the queue for this repository.

:hourglass: Testing commit e46d80e24ebf8504eec7b98463ff8f909e6cba3c with merge 4f70d1781a2b2d3611087e1ee6e8096903c9b73a...

:sunny: Test successful - checks-actions
Approved by: epage
Pushing 4f70d1781a2b2d3611087e1ee6e8096903c9b73a to master...

authors: bors epage stupendoussuperpowers


text:
Use a ad hoc trie to avoid slow cases

This adds a test for the pure case described here. When added, this test will time out in debug (~20 sec release) with an N of 3.
Looking with a profiler, it is spending most of its time in Vec.contains. Keeping a deduplicated list with contains is O(n^2), so let's change things to be Ord so we can use a btree to deduplicate.
Now the profiler claimed that we are spending our time searching that Vec. So time to work on that "vaporware", the simplest thing I could think of that lets us check for is_active in better then O(n) is nested hashmaps. Each hashmap has keys of a shared Id, and values of hashmaps representing all the conflicts that use that Id. When searching if the key is not active then we need not look at any of its descendants.
There are lots of ways to make this better but even this much gets the test to pass with N of 3. So maybe we can justify the improvements with N of 4? No, eavan in debug N of 4 hits the 50k limit, so the that is as far as we need to go on the conflict_cache.

r? @alexcrichton
(rust_highfive has picked a reviewer for you, use r? to override)

I forgot that I used NLL to test this. I am not seeing how to get it to work without NLL. Thoughts?

Figured out how to remove the reliance on NLL!

Instead of responding in thread, I pushed a commit that added comments to address each of your excellent questions.
On a more general note this is not a heavily optimized and carefully chosen algorithm, it is much closer to the first thing I could think of. I spent some time looking through the data structures I had heard of to see if there is an existing literature, but did not find any good search terms. You menchan "set of sets", do you have links about data structures for that? If anyone has references I am happy to look into them!

@bors: r+
💯

:pushpin: Commit 178ee0b611c97859086a33252dc6f1f9260b602e has been approved by alexcrichton

:hourglass: Testing commit 178ee0b611c97859086a33252dc6f1f9260b602e with merge 1292a6d0d656d6a47b1e81b03313b5b455c5665a...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 1292a6d0d656d6a47b1e81b03313b5b455c5665a to master...

On discord @alexcrichton clarified that the resolver does not have to be deterministic. In most use cases the index will change between any two invocations anyway.

authors: Eh2406 alexcrichton bors rust-highfive


text:
Delete unnecessary 'return'

Unnecessary return keyword deleted as it is also in the title

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @nrc (or someone else) soon.
If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.
Please see the contribution instructions for more information.

:sunny: Test successful - status-appveyor, status-travis
Approved by: ehuss
Pushing f58b34116932ba4c84e5c7e886129a6a8daacd20 to master...

authors: bors k-nasa rust-highfive


text:
Bump version tested on CI to 1.58.1

This is needed because cross-rs increased its MSRV to 1.58.1.
One CI builder is kept at 1.56.1 without using cross-rs to guarantee
that compatibility is maintained.

@bors r+

:pushpin: Commit 2242f9fc0b804f6d36370c784c355986948e11ea has been approved by Amanieu
It is now in the queue for this repository.

:hourglass: Testing commit 2242f9fc0b804f6d36370c784c355986948e11ea with merge 0ea54deb5814d36257372277ee27e89107ad9761...

:sunny: Test successful - checks-actions
Approved by: Amanieu
Pushing 0ea54deb5814d36257372277ee27e89107ad9761 to master...

authors: Amanieu bors

repo: rust-lang/hashbrown


text:
Potential incorrect type for ptrace requests with musl

For some reason normal ptrace requests i.e. PTRACE_TRACEME are c_int whereas the ptrace requests related to program registers (i.e. PTRACE_GETREGS) are unsigned integers musl mod.rs. Now as all these values are used as the request argument to fn ptrace surely the sign should be consistent?
musl is also the only libc environment which has the difference in signedness which also doesn't seem to be mimicking musl itself ptrace.h musl

While technically a breaking change to fix this, my guess is that it's fine to fix in a point release as it's unlikely to break anyone.

I'd only expect it to make a cast unnecessary for some users. I'll submit a pull request later today then unless it's already being covered :+1:

Closing this issue as #745 is merged!

repo: rust-lang/libc


text:
Add swd field to impls for user_fpregs_struct

Addresses https://github.com/rust-lang/libc/pull/2748#discussion_r880853330

r? @Amanieu
(rust-highfive has picked a reviewer for you, use r? to override)

Thanks for finding it! @bors r+

:pushpin: Commit 5e77aa0793aafa1fef67f4597684aaafe2cd051d has been approved by JohnTitor

:hourglass: Testing commit 5e77aa0793aafa1fef67f4597684aaafe2cd051d with merge 187091bae1521f325aa9eab60e469ade5ba6449c...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: JohnTitor
Pushing 187091bae1521f325aa9eab60e469ade5ba6449c to master...

authors: JohnTitor Minoru bors rust-highfive


text:
Roadmap for 2019

This RFC proposes the 2019 Rust Roadmap, in accordance with RFC 1728. The
goal of the roadmap is to lay out a vision for where the Rust project should
be in a year's time.
You can view the rendered version here.
The proposal is based on the 2018 survey, our annual call for blog posts,
direct conversations with individual Rust users, and discussions at the 2019
Rust All Hands. Thanks to everyone who helped with this effort!
In short, 2019 will be a year of rejuvenation and maturation for the Rust
project. Much of the focus is on strengthening our foundations and
paying down debt, both technical and organizational.
Thanks to everyone for being patient while we got the roadmap together; it took a bit longer than we'd like, but I'm very pleased with the result! Furthermore, while I am the one opening this RFC, I'd like to thank @nikomatsakis for putting in a lot of work and revisions, and the teams generally, who put together their plans and gave feedback.

A quick note: the quote attributed to me isn't actually mine.  Having said that, I do agree with the quote, but it's not from my post. :-)

@cessen oops :( sorry about that! At least you know I did read your post, 🤣 😳 I've fixed it now.

There is one thing I was surprised to see not in the roadmap: discussion about the RFC/stabilization processes. Maybe it's just my perception, but it seems like a major theme that came up a few times and was raised by multiple leaders of the project (e.g https://internals.rust-lang.org/t/blog-rust-governance-scaling-empathy/9412, https://boats.gitlab.io/blog/post/rust-2019/). Is the plan to make a separate push independent of the roadmap?

@mark-i-m this is a point @nikomatsakis and I went back and forth with; there's certainly general governance stuff that the core team wants to do this year. We were a bit unsure of the level of granularity we'd want to go into stuff like this, given that we don't know exactly how it will turn out, but maybe it's worth pointing out at a high level. Thanks for calling this out!

Thanks @steveklabnik :) Yeah, I didn't necessarily mean a detailed plan, just a mention that this is something to be investigated this year.
One other thing: are there any plans to do something like an impl Period? Past roadmaps have included timelines for key events (e.g. 2017 impl Period or 2018 edition).

I don't think that was explicitly discussed, but I think that after the big 2018 deadline, people aren't eager for more arbitrary deadlines this year :) at least, that's my rough take. Maybe it's a good idea!

I guess an arbitrary deadline could be helpful for some periodic things like annual planning/roadmapping to help prevent the burden getting delayed or from falling on an individual (like you <3). But I was really thinking more of something like the impl Period: time specifically set aside for a purpose. I have read mixed feelings from different people about whether they thought the impl Period was productive or not, though...

I guess an arbitrary deadline could be helpful for some periodic things like annual planning/roadmapping to help prevent the burden getting delayed or from falling on an individual (like you <3).

Hehe, yeah. That was just life this time though; I was advocating for this to be done by January, and then stuff happened. I think while the push for 2018 was going on, I didn't realize how tired I and everyone else would feel afterward.

But I was really thinking more of something like the impl Period: time specifically set aside for a purpose. I have read mixed feelings from different people about whether they thought the impl Period was productive or not, though...

So, the issue with the impl period (and I did hear this from a lot of people) was that it created pressure to ship RFCs before the cutoff. That's what I meant, more than the actual impl period itself.

@steveklabnik I got some feedback on the community team roadmap that I would like to include later today.

I'm skeptical of ever trying impl-periods again. In particular, I think @steveklabnik is right to say that it created pressure. I think the theme of this year, particularly wrt. "sustainability" suggests that we want to move away from such a development model in which we cram development into one period and proposals into another. Instead, we should have a more balanced, organic, and less forced way of doing things. Also, the impl-period idea might fit our rather serial model today but I think it fits poorly with a the parallel nature of the working group model where they spin up and down and they work at different paces. It also seems to me that impl-periods foster unhealthy and periodical workloads for the language teams and compiler teams respectively.

My sense is that we are not planning an "impl Period"-like push this year.
With respect to text about revamping the RFC process or other such governance changes, I think it's probably a good idea, but couldn't quite figure out where it should fit when I was making changes. Part of the problem is that I don't really view this as such a "top-down effort" any more. I think what we want is individual teams working on their process, and some kind of working group ("governance working group") that is helping to carry lessons between teams and to brainstorm with teams about possible improvements to address their problems. The point here being that a lot of this work is falling on teams, and I do think that many of the individual teams noted their desire to make organizational changes, so in some sense the work is already represented.
That said, it seems like it would add clarify overall to identify that we intend to be actively working on our governance this year across the project. Perhaps @steveklabnik we should add back the Core Team section you originally had, with some text like this?

Core team
A major focus for the core team this year will be reviewing and revising our governance process to better fit our current needs. A number of the teams are already actively changing their structure to better cope with organizational debt, and the core team intends to help with that process where possible. Some possible changes include creating new mechanisms for better inter-team communication, improvements to the RFC process, and creating new teams for areas where we don't currently have coverage.

I like it!
I still plan on filing an RFC for a staged RFC process, personally. I think it'll be really great for us.

This looks great overall! :D I am a little puzzled to see the "Async Ecosystem" section mention Tide but not Tokio though? Shouldn't moving Tokio onto std futures be a major push, since there's so much ecosystem around that already? Whereas, as far as I can tell, Tide isn't really being used yet.

Are there any plans to tackle the remaining proc macros features this year? These (along with the never type, but that could probably be worked around) are the remaining issues blocking Rocket from working on stable. Which, with Rocket planning to go async in their next release, would be fantastic from an Async Ecosystem perspective.
https://github.com/SergioBenitez/Rocket/issues/19

@nicoburns

Are there any plans to tackle the remaining proc macros features this year?

From what I've seen in "All Hands" notes - not as a part of the official roadmap.
I'm going to work on feature(proc_macro_hygiene) occasionally, so parts of it could be stabilized.

@jonhoo tokio is not governed by the rust project, so we cannot control what they do, so it’s not on the roadmap.

@steveklabnik but isn't the point of an "ecosystem" team to work with the ecosystem? Just because a project is not directly under the control of the Rust team doesn't mean that it's not important to Rust users and to the Rust language, or that the Rust team can't be involved. Tokio in particular is an example of something that is widely used in the async ecosystem, and ignoring it seems like an odd choice; it's currently a major part of Rust's async story, and that seems like something worth taking advantage of and building on. Tide on the other hand is currently a minor project that, as far as I know, doesn't have much adoption in practice? I don't think tokio is the only example here either; hyper, actix, and h2 are also big players in this field that it seems reasonable for the ecosystem team to work with.
I don't think it scales for various official Rust teams to have a mandate that only allows them to work on their own stuff, nor do I think it's a good idea. There aren't enough dedicated people working on Rust that they can reasonably own and manage all the popular components in the Rust ecosystem. Which raises the question of why the already-stretched resources of the various Rust teams are being spent on writing something like Tide when the groundwork has already been laid elsewhere?

@jonhoo For the purpose of the roadmap (and specifically, this thread about the roadmap PR), if the ecosystem is planning to do something "official" with regard to Tokio, such as "work more closely with the Tokio team", then that would indeed deserve a bullet point, I think.
That said, I don't think discussion about what a specific team should be working on is appropriate here. It would be much better to discuss Tokio/Tide plans with the ecosystem team directly.

@BatmanAoD Oh, sure, I agree with that. The reason I brought this up in the first place was the explicit mention of Tide. I think either they should list all the parts of the ecosystem they intend to work with, or they should not mention any project by name at all. Singling out Tide strikes me as weird, and also as a little dismissive of the huge ecosystem that is already out there.

I'll import some further discussion from Reddit so that we can continue it here through the "official channels" rather than in a random Reddit sub-thread :)

@steveklabnik: Tide is a project controlled by the WG and so RFCs related to it are actually a thing.

@jonhoo: Hmm, that seems problematic to me... I could be misunderstanding you, but it sounds like the policy is "we can't make goals with regards to the ecosystem, just our own projects"? That approach is troublesome as it encourages a degree of "Not Invented Here", where Rust teams are incentivized to either take over control over existing projects or start their own, instead of working with the ecosystem as-is. I think there is a big question of whether spending significant resources on Tide is even reasonable, given all the infrastructure that already exists out there, and "blessing" it by putting it on the official Rust Roadmap seems premature and somewhat conflict-of-interest-y. To me, it makes sense for the Roadmap to either say what the overall goals are, or to list all the projects that the ecosystem team will focus on; naming just one, even if it is controlled by a Rust team, is basically saying "this is where our focus lies".

@jonhoo thanks for asking questions! Let me try to answer them by one-by-one.
What is the purpose of the Async Ecosystem WG?
The goal of the Async Ecosystem WG is to bolster the async ecosystem. There’s many ways this can be interpreted, but for us this means we work in tandem with the Async Foundations WG to progress the ecosystem around Futures 0.3 and async/await.
I’ve written a blog post that contains more details around the purpose of the working groups. This post is also linked from the roadmap.
Why is Tide important?
It’s nice to have a goal in mind, but in order to turn it into reality we have to make it practical. In order to make progress we’ve found it essential to have a space where we can explore freely. And in order to give our exploration direction we’ve found it greatly beneficial to have deliverable projects.
For us Tide serves both these goals. Building it provides us with direction without being locked into the existing directions that other projects have taken. This means that when we say: “the plan is to focus on the Tide Web Framework and expand the ecosystem around async/await”, we mean that working on Tide will allow us to expand the ecosystem.
Why not mention Tokio in the Roadmap?
The short answer is: we don’t know Tokio’s roadmap, and don’t want to pressure them into upgrading.
The majority of Tokio’s core contributors are employed by Buoyant, who use Tokio to power their enterprise service mesh product. Buoyant’s employees have repeatedly expressed their requirement for stability because their product powers critical code.
However, this does make any commitment to a timeline difficult. For example, Tokio is blocked on upgrading to modern futures until “implementing Future by hand is no longer required for the user” (source). But in turn this is blocked by stabilizing existential types (source), which has an unknown timeframe.
There are too many unknowns for us to meaningfully talk about Tokio's roadmap, and we want to be cautious in making any assumptions. Instead we defer to the Tokio team for any inquiries.
Conclusion
I feel like I've mostly elaborated on the points @steveklabnik already laid out in brief. I hope this helps clarify where the Async WG is coming from, and why we've chosen to highlight the things that we do in the roadmap.

Tokio is blocked on upgrading to modern futures until “implementing Future by hand is no longer required for the user” (source). But in turn this is blocked by stabilizing existential types (source), which has an unknown timeframe.

@yoshuawuyts is there somewhere we can learn more why Tide does not have the same root concerns? Do they simply not apply?

@shepmaster many people have the opinion that async/await means that the vast, vast majority of users will never need to impl Future by hand. So it's generally less of a concern.
It really depends on how you feel about the ratios here.

@yoshuawuyts Thanks for the thoughtful response! That helps clarify things a little, though I still feel like this is a strange direction for the Async Ecosystem WG to go. Specifically, if Tide is simply a space for you to explore potential designs, it doesn't seem warranted to include in the roadmap, since none of the existing and expansive ecosystem actually uses it (again as far as I'm aware -- examples to the contrary would be great). It makes perfect sense to me for such a project to exist for you to play around with ideas internally, but unless this is a serious push for a core piece of stable and production-ready async infrastructure (which will take a long time; tokio arguably just recently started getting there), I don't think it should be a part of the stated goals for the Ecosystem WG. Instead, it makes much more sense to me for the Ecosystem WG to look at ways to improve the existing ecosystem. That may involve prototyping things in Tide, but with an ultimate goal of standardizing and extending what has been tested and proven in real code and workloads, rather than "we're going to do everything our way".
Perhaps a useful question to ask at this point is "is the intention fo Tide to replace Tokio?". If the answer to that question is yes/maybe, then to me that warrants a WG all of its own, since that is a major endeavor, and requires people who have a lot of experience with working low-level network engineering, OS thread scheduling, etc.. Using threads and the network efficiently, especially cross-platform in an async context, is very difficult. And as mentioned below, Tokio is still working out all the kinks there. To fold this into the Ecosystem WG seems unfortunate, since it basically means "we're going to invent our own ecosystem".
If, on the other hand, the answer is no to that question, then I wonder why Tide is even on the roadmap. If it is indeed a tool for experimentation and design space exploration, then it is not one of the deliverables of the Ecosystem WG, and should not be on the official Rust Roadmap.
I'm somewhat surprised by your comments on Tokio. While it's true that many of the maintainers are from Buoyant, the project seems to be run very much in the open, has a wide range of contributors from throughout the ecosystem, and has seen significant investment and thought in nearly all aspects of its design. That seems like something to take advantage of and build upon, rather than ignore under an assertion of "it won't let us innovate". Like @shepmaster observes above, Tokio's commitment to stability and usability is a feature that I believe is vital to something as core to async development as the async execution core. Your statement makes it sound as though that is a drawback that Tide wants to do away with, which seems like a mistake to me. But perhaps you can clarify the WG's position on this?
As for being unable to "meaningfully talk about Tokio's roadmap", that seems like an unproductive position to me. For example, the Ecosystem WG could easily have part of their roadmap be:

Work with the existing actors in the ecosystem (tokio, actix, hyper, etc.) to establish roadmaps for transitioning to std::future and async/await.

That seems like a fantastic goal for the Ecosystem WG to me, and it's one that builds upon what we already have, rather than go "we need to start over". I think dismissing the claims of the existing ecosystem seems entirely contrary to what the Ecosystem WG's mission ought to be.

To clarify a little: from what I can tell, Tide isn't necessarily a replacement for Tokio, but rather a web framework in and of itself. I don't think that changes the questions I raise above though: shouldn't the goal of the WG be to work with and improve existing projects to the greatest extent possible rather than just build things from scratch on their own?

One thing that may not be obvious to everyone following along is that today's Tokio does support the standard libraries Future trait and async/await syntax. As I understand it, the difference is that the fundamental abstraction in Tokio is not std::futures::Future but the one from the stable futures crate; some amount of abstraction/indirection/conversion is performed when interoperating.
While having a unified abstraction does seem a desirable goal, it does not seem to be required for people who wish to use both worlds.

How do I use async/await syntax with Tokio?
Getting started with nightly async/await support (2018-11-26)

@nicoburns

Are there any plans to tackle the remaining proc macros features this year?

Yes and no. One of the major thorny remaining questions is getting a good
handle around the approach to hygiene that we want. Until now, we've been
trying to steer a careful course and enable features that don't really need
to be blocked on hygiene, but that's getting harder and harder to do (and I'm
already a bit concerned that we may have accidentally forced our hand in a few
ways, not sure).
To that end, we had been hoping -- on the compiler team side -- to put in some
work into extracting our currently name resolution and macro expansion algorithm
into a library (we alluded to this in the roadmap, in terms of "library-ification" efforts).
This would hopefully mean that we can better understand what we are
doing now, and lay the groundwork for us to make progress on expansions to hygiene
(which would be more of a "lang team" question, though one with lots of input from
implementors as well).

@djc

it feels like the RFC covers very few of the issues, or in general, the topic of organizational debt

I definitely agree that organzational debt is a major issue for us to tackle this year. But I think it is quite a major theme throughout the RFC. For example, each of the following teams had some text on organizational issues:

Community: "This will bring with it organizational changes to allow the team to grow out of its current locations."

That said, I know that the community team has been doing a variety of other things here too, there is probably room to say a bit more.

Compiler: "Improving our documentation and organization, helping to make it clearer what is going on at any given time, and how people can get involved."
crates.io: "focused on growing itself" (admittedly, I am reading a bit into this comment that may not have been intended, but I think growing generally requires taking a look also at organizational issues)
Dev Tools: "...along with looking for ways to improve their internal processes."
Documentation: "completely reformulating itself"
Lang team: "Introduce working groups with the goal of increasing transparency and lowering the barrier to get involved", "create lang-team repo"
Async: the split in working groups is itself an organizational change
CLI apps: "improving the organizational structure of the WG through meetings, organization, and branding"
And of course the text that we intend to add re: the core team

I feel overall pretty comfortable with this. I'd be interested to know if there are specific organizational challenges that you think are not represented in the quotes above.

I do wish to clarify my current thinking (it has not changed much but I can see how the statements made in the RFC, when combined, could lead to an incorrect conclusion).
For context, relevant statements that I have made in the past.

Because of this, my plan for Tokio will be to stick on futures 0.1 until async / await has reached a point where implementing Future by hand is no longer required for the user.

Tokio will add support for async / await as it stabilizes, but it will not be considered the primary abstraction until it is able to handle all the cases.

The decision to stick with 0.1 for the immediate future was not taken lightly and was discussed at great length with a number of people.

Those are the three statements I made (afaik). As I read the statements, I see now that "we" is ambiguous as it is not clear if I am speaking as a maintainer of Tokio or an engineer at Buoyant (linkerd).
I discussed the roadmap for async / await some here: tokio.rs/blog/2018-12-recap-2018
I don't believe that existential types are required to avoid implementing Future by hand, but I don't know if an in-depth investigation into the matter has been done.

@steveklabnik but isn't the point of an "ecosystem" team to work with the ecosystem? Just because a project is not directly under the control of the Rust team doesn't mean that it's not important to Rust users and to the Rust language, or that the Rust team can't be involved. Tokio in particular is an example of something that is widely used in the async ecosystem, and ignoring it seems like an odd choice; it's currently a major part of Rust's async story, and that seems like something worth taking advantage of and building on. Tide on the other hand is currently a minor project that, as far as I know, doesn't have much adoption in practice? I don't think tokio is the only example here either; hyper, actix, and h2 are also big players in this field that it seems reasonable for the ecosystem team to work with.
I don't think it scales for various official Rust teams to have a mandate that only allows them to work on their own stuff, nor do I think it's a good idea. There aren't enough dedicated people working on Rust that they can reasonably own and manage all the popular components in the Rust ecosystem. Which raises the question of why the already-stretched resources of the various Rust teams are being spent on writing something like Tide when the groundwork has already been laid elsewhere?
Also posted in the Reddit thread.

I agree, Jon. Tide is a project based on noble intentions yet is disruptive to the ecosystem. It represents mission drift. Realistically, though, Tide will die on the vine without real help. It moves at a glacier pace while the rest of the ecosystem rapidly evolves.
If anyone disagrees with Tide's politics, don't contribute to it.

@steveklabnik but isn't the point of an "ecosystem" team to work with the ecosystem? Just because a project is not directly under the control of the Rust team doesn't mean that it's not important to Rust users and to the Rust language, or that the Rust team can't be involved. Tokio in particular is an example of something that is widely used in the async ecosystem, and ignoring it seems like an odd choice; it's currently a major part of Rust's async story, and that seems like something worth taking advantage of and building on. Tide on the other hand is currently a minor project that, as far as I know, doesn't have much adoption in practice? I don't think tokio is the only example here either; hyper, actix, and h2 are also big players in this field that it seems reasonable for the ecosystem team to work with.
I don't think it scales for various official Rust teams to have a mandate that only allows them to work on their own stuff, nor do I think it's a good idea. There aren't enough dedicated people working on Rust that they can reasonably own and manage all the popular components in the Rust ecosystem. Which raises the question of why the already-stretched resources of the various Rust teams are being spent on writing something like Tide when the groundwork has already been laid elsewhere?
Also posted in the Reddit thread.

I agree, @jonhoo . Tide is a project based on noble intentions yet is disruptive to the ecosystem. It represents mission drift. Realistically, though, Tide will die on the vine without real help. It moves at a glacier pace while the rest of the ecosystem rapidly evolves.
If anyone disagrees with Tide's politics, don't contribute to it.

@carllerche

Because of this, my plan for Tokio will be to stick on futures 0.1 until async / await has reached a point where implementing Future by hand is no longer required for the user.

I thought this was already the case on nightly. What is missing to allow this to happen?
Is your position that Tokio will not support async/await until implementing Future by hand is no longer required for the user on the stable compiler. Because from my perspective your position otherwise makes complete sense. No point supporting something that is constantly in flux. But IMO it would seem to make sense to do this while futures are still marked as unstable (even though they are de facto stable). Because that leaves room to change the abstraction if implementing it reveals any issues with the design.
Have you considered releasing Tokio 0.1 as Tokio 1.0 (with the possibility of further 1.x versions supporting futures 0.1), while also developing an experimental Tokio 2.x which supports std futures, but is nightly only until futures stabilise. One could do the same with hyper. Is it simply a lack of resources which precludes this course of action? And if so is there some way in which the community could step in to help with this development?

@jonhoo wrote:

shouldn't the goal of the WG be to work with and improve existing projects to the greatest extent possible rather than just build things from scratch on their own?

I think the primary role of domain working groups is to explore what it takes for Rust to support a given domain. This might include improving existing projects, e.g. by helping them to develop missing features, but I think it typically means something quite different: I would expect a lot of the work to be looking for the places that existing projects are not covering, and looking to fill those gaps. Sometimes those gaps are technical, like implementing a missing piece or building interop layers, but other times they are of a different nature, such as writing documentations and guides.
In the case of the async ecosystem WG, I don't claim to be an expert, but I believe the goal with the Tide framework is to look a bit beyond the runtime layer -- basically, to experiment more with what one might build using a runtime like tokio. In general, I think this is a perfect thing for the WG to be doing.
The tokio project itself seems to be doing very well, and I don't think it really needs direct support. As @shepmaster noted, it already supports std::Future via various shim layers.
Of course, it would be nice for std::Future to become the underlying abstaction for tokio and not a secondary one,  but from I can tell the primarily blockers there have more to do with the design of the trait itself and progress on other related language features, which falls under the lang team's purview (via the async foundations working group). As far as I can tell, @carllerche's concerns have defintely been heard and are under active consideration -- witness the ongoing debate in rust-lang/rust#59119. If the tokio project feels like it needs some additional kind of support,  I'd definitely like to hear about it (but I'm not convinced it would rise to the level of a new roadmap item, so I think reaching out privately or in some other forum would suffice).
So, overall, I feel pretty comfortable with the current status of the async ecosystem WG with respect to the roadmap overall. (To be clear, I'm not actively involved in the async ecosystem WG; this is just my understanding from asking a few questions. I have been getting more involved in the "async foundations" work, particularly from the implementation side.)

@nikomatsakis Oh, sorry, when I said "work with and improve existing projects", I intended that to also cover things like documentation, guides, perhaps API guidelines conformance, etc. Basically things that make the ecosystem work better.
I completely agree with you that filling existing gaps is a reasonable thing to place under the purview of the Ecosystem WG, but what gap is Tide filling? There is already actix-web, rocket, warp, rouille, tower-web, etc. Wouldn't it make more sense for the WG to look at what it would take to move one of those (or all of those) to async/await, rather than re-invent the wheel themselves and add yet another framework to the mix? Essentially to do a survey of how the ecosystem can move to async/await, and what the various blockers and pain-points might be for the projects that people already use.
tower-web, warp, actix-web, rocket, rouille, ???

what gap is Tide filling?

That is https://rustasync.github.io/team/2018/09/11/tide.html

The Network Services Working Group aims to improve the story for web development this year in several respects: by bolstering foundations like async/await, by improving the ecosystem of web-related crates, and by pulling these pieces together into a framework and book called Tide. The name “Tide” refers to “a rising tide lifts all boats”; the intent is to improve sharing, compatibility, and improvements across all web development and frameworks in Rust.

@steveklabnik I think the blog post you linked to sets out a good vision for a "lifts all boats" roadmap, which is great! But then the following blog posts (routing, middleware1, middleware2) all talk about "what Tide does", with no connection to the existing ecosystem. Why doesn't it use warp filters, rocket middleware, or tower services? I'm not involved with Tide (or any of the other web frameworks), but this looks an awful lot like yet another framework that has a different opinion on how things should be done, rather than trying to bring the ecosystem together.

So, it seems to me like this is is very in the weeds; I really want per-repo RFCs, heh. But, it seems like at this point, @jonhoo , your objection is more to a particular project of this particular working group, than it is to the roadmap more generally. Rather than keep having tons of back-and-forth on this specific issue, which pings everyone interested in the entire roadmap, you could chat with @aturon and/or @yoshuawuyts about this, and then one of you can report back with a summary?

@steveklabnik I completely agree with you! In some sense, this is why I objected to having Tide be mentioned in the Rust 2019 Roadmap at all. It is at best an implementation detail of how that WG wants to explore their higher-level goals, but I don't think it makes sense to list as a part of Rust's overall road map for the year :) I would much rather have the Roadmap list the overarching goal and motivation of each WG than specify internal details; those can be ironed out in smaller groups afterwards subject to the high-level goal listed by the 2019 Roadmap.

I would much rather have the Roadmap list the overarching goal and motivation of each WG than specify internal details;

This is not how our roadmap process really works, that is, we don't have this as a current requirement of the process. Describing details around how a group's goals apply to their projects is pretty normal. For example:

The cargo team lists "Xargo, cross, Rustup, and WASM-Pack"
The compiler team talks about RLS 2.0
The CLI WG talks about assert_cli, assert_fs, and assert_cmd
The Wasm WG talks about wasm-pack

The Async Ecosystem WG mentioning tide fits right in with these kinds of things.
This comes out of RFC 1728, which lists both "problem statements" as well as "goals". And it does say that the roadmap should be the problem statements, and then that for each release cycle, the teams would plan out specific goals that refered to a problem statement. In practice though, things didn't play out that way. In our 2018 roadmap, both problems and goals are identified throughout. The guide-level explanation lays out the problems, and then the detailed explanation lays out the goals.
Personally, I believe this happened because an open source project like Rust doesn't work the same way as a company does; trying to model agile too directly doesn't really work in this environment. But I digress. Ultimately, this year's RFC follows in the footsteps of last year's. And regardless, litigating the form of roadmaps generally is not really on-topic for this particular roadmap.
If you'd like to see the Async Ecosystem WG drop Tide as a project, I'd suggest reaching out to the leads as I mentioned before, or attending the WG's open meetings on Thursdays.

That's a good point. I guess the core of my gripe is around the perceived focus of Tide vs the ecosystem writ-large given the name of the team. I feel like the primary focus should be "help expand the ecosystem around async/await" (the second part listed in the RFC), with Tide being a particular way the WG intends to do that. While I personally don't see how focusing on Tide contributes to the ecosystem, I agree with you that the Roadmap mentioning Tide isn't uncharacteristic of past Rust Roadmaps. I'll try to find a way to reach out to the Async Ecosystem WG to discuss directly there.

@jonhoo

I'll try to find a way to reach out to the Async Ecosystem WG...

Here is their Discord channel: https://discordapp.com/channels/442252698964721669/474974025454452766
They have a weekly meeting on Thursdays via that channel, so they may take up your concerns as an item to discuss if you can write them up tonight. (I'm not a member of that WG, though, so I can't speak for them or promise that they'll modify their agenda on your behalf.)

@nikomatsakis

I feel overall pretty comfortable with this. I'd be interested to know if there are specific organizational challenges that you think are not represented in the quotes above.

I suppose if you put it like that, the majority of the issues that boats mentioned are covered somewhat. I feel the money issue is not covered in the roadmap at all, and is something that is top of mind for me; not just the issue of how we can compensate productive contributors, but also how we grow the financial resources that the Rust project can leverage. As far as I understand, two people have left Mozilla Corporation's Rust team with no sign of replacement happening -- does that represent a position that MoCo will scale back their direct contribution definitively?
On the overall RFC, for me it ends up feeling not very coherent and rather fragmented -- this is probably a result of the choice to present everything by team. I feel the RFC would probably read stronger for outside users like myself if it was presented more as (a) problem identified by survey or blog post, (b) proposed solution with (c) SMART goals possibly subdivided by teams. I also feel even with the added section on the core team, the governance theme isn't very well represented.

@djc

On the overall RFC, for me it ends up feeling not very coherent and rather fragmented -- this is probably a result of the choice to present everything by team. I feel the RFC would probably read stronger for outside users like myself if it was presented more as (a) problem identified by survey or blog post, (b) proposed solution with (c) SMART goals possibly subdivided by teams.

This is good feedback, thanks. I too am unsure what I think about the team-based format for the roadmap. In fact, @steveklabnik and I already privately expressed some similar reservations of the format, but we ultimately opted not to try and reorganize everything. Part of the reason for this is time: we are already running later than we'd like to get this RFC posted, and I'd rather not introduce further delay. I definitely think that, when it comes time for next year's roadmap, we should revisit some of these ideas.
That said, once the RFC is merged, we have traditionally made a blog post, which is usually the main "point of entry" for people to read about the roadmap. Our plan was to try and give a more "integrated" presentation there.

I feel the money issue is not covered in the roadmap at all, and is something that is top of mind for me; not just the issue of how we can compensate productive contributors, but also how we grow the financial resources that the Rust project can leverage.

My expectation is that the governance WG (just announced!) -- or a successor, depending on how we scope things -- will tackle some of these topics, but not initially. I think it makes sense to let us figure out the processes that work best for these sorts of discussions first.

two people have left Mozilla Corporation's Rust team with no sign of replacement happening -- does that represent a position that MoCo will scale back their direct contribution definitively?

As the current manager of Mozilla's group, I can speak to that. There has been no change in Mozilla's commitment to Rust. We are working on getting authorization to hire replacements for those who left, but that process takes time.

@rfcbot fcp merge
I think it's time to merge this roadmap, and get started on making it a reality! Thanks to everyone for the discussion and feedback.

By the way, it appears that the rfcbot has not been updated to account for some core team changes. I've checked a few names off on the comment above, and I'm going to manually add one here that is missing:

[ ] @Manishearth

@nikomatsakis Please send a PR against https://github.com/anp/rfcbot-rs/blob/master/rfcbot.toml updating the roster of the core team. :)

🎊 🎊 🎊 🎊 🎊 🎊 🎊 🎊 🎊 🎊
The 2019 Roadmap has been merged!
🎊 🎊 🎊 🎊 🎊 🎊 🎊 🎊 🎊 🎊
As usual, we will be putting out a blog post on the blog, summarizing the roadmap for the community. Expect that early next week.

Rendered version link is broken.

It's merged here: https://github.com/rust-lang/rfcs/blob/master/text/2657-roadmap-2019.md

Updated.

authors: BatmanAoD Centril Dowwie Manishearth andradei carllerche cessen djc jonhoo lzutao mark-i-m nicoburns nikomatsakis petrochenkov shepmaster skade steveklabnik yoshuawuyts

repo: rust-lang/rfcs


text:
Focus the editor on page load.

This was a minor inconvenience that I had forgotten to fix up.

r? @nikomatsakis
(rust_highfive has picked a reviewer for you, use r? to override)

authors: chris-morgan rust-highfive

repo: rust-lang/rust-playpen


text:
support trait bounds const opt out syntax

Closes #4054

Thanks!

authors: calebcartwright topecongiro

repo: rust-lang/rustfmt


text:
Add tests for duplicated attributes on expressions(#4452)

Added tests for #4452.
It covers cast expression , range expression and blocks which described in here

@calebcartwright Thanks for your feedback. I taking a look at cargo-make install failure on i686-pc-windows-msvc.

I taking a look at cargo-make install failure on i686-pc-windows-msvc.

Looks to simply be a transient network issue. Have restarted it

Just to confirm that the updates that include the comments are still coming correct? Asking based on your last commit message while the files are still missing the comments

@calebcartwright Sorry, my previous commit hasn't included the comment update.

Perfect, thanks so much!

@calebcartwright I appreciate your patient help & support!!

authors: calebcartwright chansuke


text:
Implement Euclidean division and modulo

I see we have floor division and the corresponding modulo operations via the Integer trait. However methods for the subtly different Euclidean division and modulo operations were added to the core Rust primitives in RFC 2169 and it might be good to have them on BigInt for consistency. (If I have missed the implementation, please let me know. I am fairly new to Rust.)

I'm open to it. It would be nice to sort out a common trait for this first (rust-num/num-traits#159), then num-bigint can implement that.

authors: betaveros cuviper

repo: rust-num/num-bigint


text:
How could I use uefi-rs and bootloader together?

I'm trying to print some text on the screen using UEFI but my project uses bootloader and AFAIK I don't think you can use a different function for uefi-rs other than efi_main, is there any way I can accomplish this somehow?

No, you can't. The uefi_main/efi_main function is the only way to get access to EFI boot services. The bootloader exits boot services before calling kmain so this is not possible.

So how would I go about printing text on the screen or something like that without the uefi-rs crate

@actuallyexeon You need to use the framebuffer that the bootloader gave you. I'm not too sure on the maths required -- I can't really figure that out myself (but would love to know how) -- and though the OSDev wiki has this article I didn't consider it much help. But maybe you'll find it of use.

authors: actuallyexeon ethindp

repo: rust-osdev/bootloader


text:
2021-10: multiboot2-header

Add info about the multiboot2-header crate

Thanks!

repo: rust-osdev/homepage


text:
impl Sync for UdpSocket

UdpSocket doesn't implement Sync, which makes it hard to build a server that potentially share the socket between futures/threads.

@xofyarg this seems like a reasonable request, and agree we should add a Sync bound to UdpSocket! Would you be interested in filing a patch for that? The relevant code is here.

authors: xofyarg yoshuawuyts

repo: rustasync/runtime


text:
fix: update theme and add customized button/chip

Change

Update theme palette
Add customized button/chip (I didn't add selected state, because MUI button/chip don't have selected state. We might need to change selected state's color in the component when needed)
(Note: The button/chip usage in the individual component is not updated yet. Design team hasn't integrated the new button styling to all of the components yet. I only updated the Chip in the MessageSpace. I also hasn't added customized icon and button group.)

Design

Before

After

https://github.com/rustic-ai/ui-components/assets/103023797/e9d59ce4-2313-4a68-8449-2b7685991792

Something I noticed when trying out one of the buttons- container shifting. This was with button variant rusticPrimary.

Something I noticed when trying out one of the buttons- container shifting. This was with button variant rusticPrimary.

Good catch! I added height: 'fit-content', to button so that button's height is not affected by other buttons

authors: kaseyvee lyjeileen

repo: rustic-ai/ui-components


text:
Large percentage of cpu time in memset when reading with a larger buffer

Reproduced using a slightly modified version of examples/client.rs to allow specifying a path, and reading into a user-provided buffer. The problem was originally noticed via rusoto s3 client, which has a configuration for a read buffer size, which gets mapped to hyper http1_read_buf_exact_size. Since aws s3 recommends fetching large ranges, using a larger buffer seemed like a good idea and was not expected to cause any cpu overhead.

use std::fs::File;
use std::io;
use std::io::BufReader;
use std::net::ToSocketAddrs;
use std::path::PathBuf;
use std::sync::Arc;

use argh::FromArgs;
use tokio::io::{split, AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio_rustls::TlsConnector;

/// Tokio Rustls client example
#[derive(FromArgs)]
struct Options {
    /// host
    #[argh(positional)]
    host: String,

    /// path
    #[argh(positional)]
    path: Option<String>,

    /// port
    #[argh(option, short = 'p', default = "443")]
    port: u16,

    /// domain
    #[argh(option, short = 'd')]
    domain: Option<String>,

    /// cafile
    #[argh(option, short = 'c')]
    cafile: Option<PathBuf>,
}

#[tokio::main]
async fn main() -> io::Result<()> {
    let options: Options = argh::from_env();

    let addr = (options.host.as_str(), options.port)
        .to_socket_addrs()?
        .next()
        .ok_or_else(|| io::Error::from(io::ErrorKind::NotFound))?;
    let domain = options.domain.unwrap_or(options.host);
    let path = options.path.as_ref().map(|p| p.as_str()).unwrap_or("/");
    let content = format!(
        "GET {} HTTP/1.0\r\nConnection: close\r\nHost: {}\r\n\r\n",
        path, domain
    );

    let mut root_cert_store = rustls::RootCertStore::empty();
    if let Some(cafile) = &options.cafile {
        let mut pem = BufReader::new(File::open(cafile)?);
        for cert in rustls_pemfile::certs(&mut pem) {
            root_cert_store.add(cert?).unwrap();
        }
    } else {
        root_cert_store.extend(webpki_roots::TLS_SERVER_ROOTS.iter().cloned());
    }

    let config = rustls::ClientConfig::builder()
        .with_root_certificates(root_cert_store)
        .with_no_client_auth(); // i guess this was previously the default?
    let connector = TlsConnector::from(Arc::new(config));

    let stream = TcpStream::connect(&addr).await?;

    let domain = pki_types::ServerName::try_from(domain.as_str())
        .map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "invalid dnsname"))?
        .to_owned();

    let mut stream = connector.connect(domain, stream).await?;
    stream.write_all(content.as_bytes()).await?;

    let (mut reader, _writer) = split(stream);

    let mut buffer = Vec::with_capacity(4 * 1024 * 1024);
    let mut total_len = 0_usize;

    loop {
        match reader.read_buf(&mut buffer).await {
            Ok(len) => {
                total_len += len;
                buffer.clear();
                if len == 0 {
                    break;
                }
            }
            Err(e) => {
                eprintln!("{:?}", e);
                break;
            }
        }
    }

    println!("Size: {}", total_len);

    Ok(())
}

I believe this is because the poll_read call is passing in a large uninitialized ReadBuf, causing the stream to zeroing the ReadBuf unnecessarily.
https://github.com/rustls/tokio-rustls/blob/main/src/common/mod.rs#L218
We can do some optimizations on read_buf cfg. https://github.com/rustls/rustls/blob/main/rustls/src/conn.rs#L230

This problem can also be fixed if you change Vec::with_capacity(4 * 1024 * 1024) to vec![0; 4 * 1024 * 1024].

I believe this is because the poll_read call is passing in a large uninitialized ReadBuf, causing the stream to zeroing the ReadBuf unnecessarily.
https://github.com/rustls/tokio-rustls/blob/main/src/common/mod.rs#L218
We can do some optimizations on read_buf cfg. https://github.com/rustls/rustls/blob/main/rustls/src/conn.rs#L230

We could limit zero-initialization to the maximum TLS record size. On modern CPUs, a 16KB memset should not have a noticeable impact.

A possible solution could be the following. I think this is sound, since Reader is a concrete type whose behavior we know, and not a trait where adversarial implementations would be possible. But it seems a bit hacky.
I'm profiling this on an aws m6in.8xlarge instance, downloading from s3. In the flamegraph the memset with default buffer sizes was taking about 5% of time. The throughput is varying too much quantify the benefit.
The call stack is coming from hyper h1 poll_read_from_io, which seems  to create a new ReadBuf on every call, so maybe there is also potential improvement on that side.
diff --git a/src/common/mod.rs b/src/common/mod.rs
index fde34c0..a9e3115 100644
--- a/src/common/mod.rs
+++ b/src/common/mod.rs
@@ -248,6 +248,10 @@ where
             }
         }

+        // Safety: We trust `read` to only write initialized bytes to the slice and never read from it.
+        unsafe {
+            buf.assume_init(buf.remaining());
+        }
         match self.session.reader().read(buf.initialize_unfilled()) {
             // If Rustls returns `Ok(0)` (while `buf` is non-empty), the peer closed the
             // connection with a `CloseNotify` message and no more data will be forthcoming.

@seanmonstar do you think it's feasible to avoid recreating the ReadBuf on every call?

I suppose theoretically, but realistically at the moment, that ReadBuf is backed by the unfilled part of a BytesMut, which doesn't expose a way to know how much of it is initialized.

I think best way for now, apart from stabilize read_buf feature in std, is for rustls to provide a ReadBuf of own (or a trait to avoid unsafe in rustls).
like
pub trait ReadBuf {
    fn append(&mut self, buf: &[u8]);
}

pub fn read_buf(&mut self, buf: &mut dyn ReadBuf) {
    //
}

rustls can be asked how much data it would write into an infinite size buffer provided to reader().read() -- self.session.process_new_packets().unwrap().plaintext_bytes_to_read[^1] -- and I think it would be reasonable to expose that quantity as a new method on reader(). I think that would resolve the immediate issue (clearing an 4MB buffer to use the first 4 bytes).
[^1]: unwrap() is for brevity only!

I am considering using unbuffered api refactor, which would also solve the problem.

authors: chen-hongzhi ctz djc jhorstmann quininer seanmonstar

repo: rustls/tokio-rustls


text:
Add doc comments from rust to typescript definitions

💡 Feature description
It would be nice to have documentation comments from rust in corresponding typescript definitions.
💻 Basic example
If I have the following rust code:
#[wasm_bindgen]
/// Test comment
pub fn greet() {
    alert("Hello, test!");
}

I want to see Test comment in .d.ts file similarly to .js file:
/**
 * Test comment
 * @returns {void}
 */
export function greet(): void;

I think this should be relatively simple because we already do this for JS code, we just don't do it for TypeScript yet! In that sense, although I'm not looking at the code, I suspect it's plumbing a few more things to a few more places.

I'd like to take on this issue as my first code contribution! I just need some guidance on where to go and what to look at.

@tyleranton awesome!
You'll be most interested in crates/cli-support. It's not the cleanest but if you poke around there you'll find js/mod.rs where typescript is a variable that's built-up over time which is the TypeScript that we generate.
One example for exports is around here where the JS doc goes to the JS but not the TypeScript

I've been looking over the code trying to piece things together. I'm still not sure how everything works, but would it be enough to take the formatted comments and just push them to the typescript variable on Context?

I think so yeah, I suspect the doc comment syntax is pretty similar across JS/TypeScript, so as long as it's all pushed in the right location it should be good to go!
If it looks wonky though we can debug further from there :)

authors: alexcrichton deaz tyleranton

repo: rustwasm/wasm-bindgen


text:
Wasm In Wasm example does not work with custom wasm

I am tring to use wasm in wasm example for my use case but with or without bundle, console.log always says;
panicked at 'called Result::unwrap()on anErr value: JsValue(TypeError: import object field './home_bg.js' is not an Object
Is there anything i am missing here?
Summary
A clear and concise summary of your question.
Additional Details
Provide any additional details here.

This really depends on the wasm that you're loading, and the error message is that you didn't provide an import. The wasm-in-wasm thing is mostly a toy and I wouldn't recommend trying to copy it exactly since it doesn't easily lend itself to handling the import object.

I send the error.

panicked at 'called Result::unwrap()on anErr value: JsValue(TypeError: import object field './home_bg.js' is not an Object
It cant load the object that wasm file itself tring to load file(like .js files or without bundle "wdm"). I can search for a solution but i do not know which doc i should look.

authors: alexcrichton eribol


text:
I'm not pretty sure is this my problem

❓ Questions & Help
I'm trying to use my own data on the RGCNConv because I find it is able to make use of the relation between nodes. However, my 1-dimension "edge_type" seems not work. In my case, I just use my own Data class to process data, and DataLoader to put them together. But when I feed them into the   RGCNConv, I always get a dimension mismatch error. And Here is my own Data class,  the edge_type was dealt with specially.
class MyData(Data):
    def __init__(self, edge_type, edge_index, x):
        super(MyData, self).__init__()
        self.edge_index = edge_index
        self.x = x
        self.edge_type = edge_type
        # self.y = y

    def __inc__(self, key, value):
      if key == 'edge_type':
        return 0
      else:
        return super(MyData, self).__inc__(key, value)

    def __cat_dim__(self, key, value):
      if key == 'edge_type':
        return 1
      else:
        return super(MyData, self).__cat_dim__(key, value)

And in my case, I'm using a batchsize as 128, and there is some dimensions I get in the message function:
data.edge_index: torch.Size([2, 740])
data.edge_type:torch.Size([1, 740])
x_j:torch.Size([740, 1, 256])
weight:torch.Size([45, 256, 256])
after message:torch.Size([740, 1, 256])

And this is the fault I get, I think it's because somewhere the dimensions mismatch:
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch_scatter/scatter.py", line 12, in scatter_sum
                out: Optional[torch.Tensor] = None,
                dim_size: Optional[int] = None) -> torch.Tensor:
    index = broadcast(index, src, dim)
            ~~~~~~~~~ <--- HERE
    if out is None:
        size = list(src.size())
  File "/usr/local/lib/python3.6/dist-packages/torch_scatter/utils.py", line 13, in broadcast
    for _ in range(src.dim(), other.dim()):
        src = src.unsqueeze(-1)
    src = src.expand_as(other)
          ~~~~~~~~~~~~~ <--- HERE
    return src
RuntimeError: The expanded size of the tensor (1) must match the existing size (740) at non-singleton dimension 0.  Target sizes: [1, 740, 45].  Tensor sizes: [740, 1, 1]

Can you give me some help please?

Can you post the dimensions of the attributes in your data object?

The dimensions in Data in the example mentioned above is here:
data.x:    torch.Size([1664, 256])
data.edge_index:    torch.Size([2, 740])
data.edge_type:    torch.Size([1, 740])

Can you convert data.edge_type to shape [740] and try again?


text:
A mechanism for sync to ignore data prior to mm/dd/yy hh:mm is needed  - enhancement

There needs to be a means to tell station to ignore data prior to a specified date & time.
For Example:
Take the ruuvi to the swamp, on Monday and leave it there.
Return on Wednesday and sync with station leave ruuvi there. Get home export data to laptop.
Return on Friday sync gets data since Monday!
Desired action is to only get data not previously retrieved.
Continuing... retrieve Ruuvi on Friday.
Saturday take Ruuvi to airport.
Sunday sync will retrieve all data from swamp the previous Monday, the data while Ruuvi was not deployed (i.e. in the office) as well as desired data since Saturday.
As second user scenario described  in https://github.com/ruuvi/ruuvi.firmware.c/issues/190
Requires no mechanism to clear log should be provided

This is already implemented, although in the form "only data since last sync is requested."

This is already implemented, although in the form "only data since last sync is requested."

Closing. Implemented - Station remembers last sync datetime

Closing. Implemented - Station remembers last sync datetime

authors: DG12 andreevdenis ojousima