text:
Design & implement system that reduces storage pressure and provides client feedback when PUSH is throttled

This is fork from https://github.com/microsoft/FluidFramework/issues/6596. That issue will continue to track handling of the nack with retryAfter, while this issue will track the following issues:

The lack of feedback to client when PUSH is throttled due to flushing ops. Currently, 99% of that is due to new client joining, as ops flushing is coalesced with new summary commitment, and thus has direct way to provide client feedback in the form of nack.
Optional - reduction or complete elimination of a need to flush ops by PUSH to storage on client joining.
Optional - exposing a way to fetch ops from PUSH (redis).

The key goal should be to push throttling up the stack, such that if the issue is - too high ops rate, then client has enough information to learn about it and possibly adjust its behavior. While long term we may want to even disallow users to make changes, for now we will focus consuming this signal in summarization workflow, i.e. make sure client does not keep piling uncommitted summaries while PUSH has trouble and thus creates positive feedback loop.
The discussion of signals in #6596 is directly related to this problem that we fork into this issue.

Here is the summary of discussion between @GaryWilber, @marcmasmsft  & me:

We commit to a plan for SPO fetch ops API to be single source of truth for ops, and thus fetch ops (on as needed bases) from PUSH / redis

the details needs to be designed. It's likely that client would need to provide some hint for this flow to be engaged, both as a staging mechanism, but also as a way to reduce extra work on the server (later might be indirect, for example we should disambiguate cases where client asked for specific range, vs. client proactively asks for op tail with no specific expectation on how many ops are available).
There is some uncertainty here due to 3 entities involved (and service to service calls), so we need to test feasibility of it.
Ops commit while submitting summary needs more discussion. It's possible that we will stay with existing flow (i.e. push model, not pull) which would require client to ask push to flush ops before client can proceed with submitting summary in single commit flow.

Marcelo to work on design doc, publish it, and figure out execution plan.
This will take substantial time to evolve, so if we need some shorter mechanism to solve immediate scalability problems, then we should address it independently.

Temp solution in this space should be staged in a way where service can retire such temp capability easily, in other words - client (and whole system) should behave correctly when such new capability is missing or become missing in the future.

Here is a proposal from me on how to address this problem (or rather consideration points). I start with just flush() API as consideration, but look into getOps() type of API as well (it has very similar requirements RE implementation).

PUSH to implement flush() API (available via via socket messages) to flush some set (current?) of ops to SPO.
Client to provide indication in connection message if client supports flushing and will use it if server supports it (enlightened clients).

server would not flush ops on connection for such client, otherwise it keeps old behavior.

bto communicate back to client if it supports flushing

Client should keep existing behavior (not calling this API) if server does not support it, such that PUSH can revoke this API in the future without notice.

Client usage of flush:

not flushing on summary (at least for now with 2-phase summary commit), with assumption that ops flushing is bundled with summary upload by PUSH and nack is used for communicating response.
- we can revisit this in the future.
Client will ask to flush ops on hitting gap in ops (after preforming first successful storage ops fetch call but not filling in the gap)

Providing feedback (success / failure) is optional from POV of client that tries to fetch missing ops (as such client can just keep asking SPO for ops for 30 seconds, though feedback would reduce frequency and make telemetry more actionable).
But it's not optional from POV of summarizer client who is blocked by this operation. So there is a need to provide that feedback (at least 429 / any other recoverable failures) to all clients.
instead of flush, we can use getOps() here with similar impact to client. In such case feedback communication is part of API. Perf characteristics are slightly different. It will result in fewer SPO calls, and improved latencies, so it definitely needs to be considered, but flush() is more versatile as it maybe used in the future as a pre-step for summary upload.

Benefits (assuming all enlightened clients):

PUSH does not need to flush ops on connection, substantially reducing number of SPO calls (in some situations, including expiring tokens), as well as provide recovery mechanism for fetching ops that were lost through pipeline.

Gap:

if we have a mix of clients, then enlightened clients need to know about anybody (clients, PUSH) doing flushes and hitting problems. But that's not enough - summarizer may be non-enlightened client and thus not aware why summaries are not coming through and thus continue to post summaries and make things worse (positive feedback loop).

With that, I think there are two possible practical workflows:

flush() and signaling to when PUSH has issues on any flush().

Pros:

has slightly better characteristics in terms of client awareness of what's going on in a mix of enlightened & non-enlightened  clients (if summarizer is enlightened).
possibly reusable for other workflows

Cons: has more implementation nuances (signals need to be re-broadcast to any new clients that joins, signals can be lost, etc.)

getOps()

Pros:

More localized (it's just single client activity)
Better perf profile / latency

Cons: Mix of enlightened & non-enlightened clients will behave worse in terms of scaling down on throttling failures.

My choice would be to go with getOps() kind of API, which I'd guess will look like this:

"getOps" message from client
ops are returned through normal (socket) flow, maybe have some marker that they are result of prior getOps call (it may help client to assert that normal ops have to come sequentially).
some message back to client indicating completion of the call (client can use it to understand if gap was filled in or not). If failures can occur, they are communicated on same message.

I wonder if this is all necessary.
Why does push need to flush on each join?
We support IDocumentDeltaConnection.initialMessages? Does this not include all unflushed ops? Isn't that its purpose? is it not sufficient?
Even with IDocumentDeltaConnection.initialMessages there is still the problem of feedback. There is a server to client feedback that pushing ops to storage is behind so we should hold off summarizing, and maybe in the future take more drastic action like going readonly.  The other feedback is client to server, basically that it cannot find some op gap. If we only have to solve this, and not the above i wonder if other solutions are possible.

Gary would have more precise description here, there is no guarantee on ordering / lack of gaps. I do not remember precise reasoning here, but it was something to do with fetching these ops being async and potentially some ops being added to redis after this call is satisfied, and such ops being not broadcast earlier to that (connecting) client. So they exists only in redis past the point of connection (from client perspective) and are not available to client in any other form.

I'm not super familiar with this layer of the code, but i remember it working like this. The driver connects to the socket, which quickly starts sending ops, which the client should keep. The server then sends connect document success which includes outstanding ops as of that point, but only needs to contain up to the point of when the client started receiving ops.  Older ops come from the server.
It's possible push started flushing on each join, as it wasn't possible to reliably do the above. @GaryWilber and @tanviraumi is this a push only behavior? AFIK r11s writes these directly to mongo, so no need to flush anything. It seems other solutions are possible here, like tracking the first seq on each conn, and ensuring server side, that messages up to that seq are either flushed, or in initial messages.
also, @vladsud  do we envision this all being encapsulated in the driver? Seems like it could be hard to encapsulate with the separation of delta stream and delta storage

@anthony-murphy Yes it's a push only behavior.
For r11s ops are inserted into mongodb. When clients want to get ops it ends up checking mongodb. So there is a single source of truth for that data.
For push, ops are either in SPO or Push. And technically ops could be in 2 separate places within Push.
Clients can access SPO ops via the get ops api and clients get the Push Redis ops via initialMessages. But due to how we are broadcasting messages (we broadcast the op before inserting it into Redis), there is point in time when the op is technically inaccessible. The op could be in a kafka topic, waiting to be consumed and inserted into Redis. For docs with high op rates, clients are likely to not get all the ops when they connect due to this. This is why we always post ops to SPO when clients join.
I have prototyped the solution you described, where we track the join op seq# and send the missed ops to the client. However that solution cannot work for read-only connections because there is no join op for them, so I dropped working on that further because I think we need a more general solution for this.
One other solution is to insert ops into redis before broadcasting them. This would ensure clients joining will always get all the ops. However this would increase latency because we would now be waiting for that redis insert call before broadcasting - we didn't want that to be a thing.. but maybe it would actually be worth doing. I'm conflicted about it.

@vladsud Here's what the "get_ops" and "flush_ops" websocket apis look like right now.
Both apis have a request and a response and take in a client id & and object as params. Both objects contain a client provided nonce.
Get ops - only returns ops from Push's Redis
If all ops in the inclusive range based on the from & to params was found, code is 200. The ops are returned in the messages property.

If no ops were found, code is 404

If only some ops were found in the range, code is 206

Flush ops:

It returns a 200 if it flushed ops and includes the last persisted sequence number.
If no ops are flushed, it returns a 204.
In error cases the apis will return a code & retryAfter property (if available).
I could make the get_ops api broadcast the ops normally instead of making them only be returned in "get_ops_response" if you would prefer that. I would probably rename the event to "request_ops" instead of "get_ops" in that case.

@GaryWilber I don't know enough about how the server code is structured, but ideally, we can know the first op sent to any client, and then the server knows it needs to send the gap from that op down to the last flushed op in initial messages which is part of connect document success. This should be agnostic to write vs read.

@anthony-murphy That is a great idea. Originally the frontends were "dumb". They did not track what it was broadcasting to clients - they simply broadcasted whatever it was told and never inspected the payloads.  This is how r11s works too.
Due to some issues seen in the scale tests, I recently had to make the frontends smarter. They now parse the ops being broadcasted to clients. It tracks the sequence numbers per-document in order to detect issues (op gaps). Now that we have that capability, it should be possible to extend that logic further for per-client tracking too.
@vladsud Would a client recover correctly if this situation occurs:

Client gets initialMessages [1, 2, 3].
Server broadcasts 6.
Server broadcasts 7.
One second later, the server broadcasts 4, 5 to that client.

I think that was supported at some point but I forget if that's still okay or not. Today the per-document op gap recovery logic will not do that. But adding per-client op gap recovery logic may result in that sequence of events.

That's right. I think both scenarios can happen (the one I listed and yours).  The main issue is what you described:

So if PUSH wants to fill in gaps, in needs to detect gap between very first op it sent and last op in initialMessages and attempt to fill it in.

Push should be able to detect that and fetch the gap ops and send them (per-client). I am going to try testing this in spdf this week.

awesome, it would be great it this works, and doesn't necessitate any protocol changes. This still leaves the issue of client feedback in general if push is throttled for whatever reason. I'm really curious if @tanviraumi or someone he knows had to solve a similar issue around their throttling design. Does it just used NACKs? or are there other mechanisms?

I briefly talked to Gary and suggested authoring a document describing current behavior of PUSH (in respect to interactions to redis, kafka, etc., as well as document and client state tracking and processes involved around filling in ops gaps).
As for next steps, client would need to make a change to not immediately hit storage on op gap as PUSH will fill in a gap in next 0-20ms (with this new design), but ops coming out of order today result in immediate request to storage, which will be unnecessary with these changes. We already have enough telemetry to notice when we do unneeded storage calls, as well as cases where client can't fill in op gap in reasonable amount of time (30 seconds), so PUSH changes in this space could be validated by observing client telemetry.
The staging will look like this:

PUSH implements logic to fill in the gaps
Client implements delaying calling storage when gap is detected, in anticipation of PUSH filling in the gap in next N ms.
We validate via telemetry that client never does unneeded requests to storage (i.e. if client does request ops from storage to fill in the gap, then ops are always actually in storage). This tells us that # 1 is actually working and that threshold we chose is reasonable (it's probabilistic, so the goal of # 2 is reduce, but not completely eliminate calls to storage).
PUSH stops flushing ops to storage on client connection.

if there is a way to flight this change, we should, and validate it in our stress tests (likely by client having local change that provides some hint on connection to enable that behavior)
Validation here includes also ensuring that client never fails due to not being able to find ops anywhere for 30 seconds.

Based on discussion at PUSH review meeting:
PUSH has a lot of logic to resolve various issues with reliability of op delivery and ordering guarantees. And even after set of latest fixes, ops are not ordered all the time, resulting in client needing # 2 from above.
It feels like we need to move further to one of the extremes:

PUSH continues on this path and provides strong ordering guarantees, ensuring that all ops are delivered in order at all times, including initial ops on connection (this is almost the case, but not guaranteed, ops can come out of order).
We remove most of the logic that helps with above from PUSH and instead force client (driver) to fetch ops from PUSH through getOps type of API (latest maybe moved to be brokered by SPO).

Solution is described at the beginning of this issue.
The biggest con of that approach is additional network latency. So feasibility of that path depends on how often it happens.

The main telemetry event that is telling is fluid:telemetry:DeltaManager:enqueueMessages event (reason & previousReason properties).
Query:

Office_Fluid_FluidRuntime_Performance
| where Data_eventName == "fluid:telemetry:DeltaManager:enqueueMessages"
| summarize count(), countif(Data_initialGap > 0), countif(Data_gap > 0), countif(Data_duplicate > 0) by Data_eventName, Data_reason, Data_previousReason
| order by count_

First, scale - appropriate set (limiting only to 0.4x loader versions as that's when this event was added) corresponds to

205K container sessions
563K (re)connections.

Terms mean:

ReconnectOps means processing initial ops on some reconnect (not first connection for this container).
InitialOps - same as above, but very first connection in container.
DocumentOpen_pending means we fetched ops from storage on document open, but there was some gap or duplciate
opHandler - op is coming from socket
reason (one of the above) - where op came from when we detected gap or duplicate ops
previousReason - what was the source (reason) of previously successfully processed op (before gap/duplication occured)

What this data tells us:

Ranges do not have that many gaps. I.e. there are cases where initial ops as observed by DeltaManager would have a gap due to establishing early op handler and potential for ops to trickle in while we wait for "connect_document_success" message to bring initial ops - such early ops are merged into initial ops set by driver. So number of gaps for reason == "InitialOps" or reason =="ReconnectOps" is at 300.
There are a lot of duplicates in initial ops (ReconnectOps) and ops coming from web socket (on previous connection), i.e. there are a ton of fast reconnects where initialOps bring same content that client saw before (over 30K cases)
Most interesting is initialGap property when reason == "opHandler", i.e. when we consume an op from socket and realize there is a gap. Here are the top offenders:

previousReason       count
opHandler	     10066
ReconnectOps	     6736
DocumentOpen_fetch   751

So in 10K cases we do see a gap in op stream itself (that's 2% of connections). I believe this is the issue with lost redis connection by front-end.
And in 6K cases (1.2% of connections) we see a gap between first op and initial ops on (presumably) same connection.

And here is Gary's description of current system:
To recap the 2 things causing the gaps:

the fact that Push broadcasts to clients before inserting the op into redis. this results in ops are sometimes not accessible to newly joining clients
redis pubsub issues, so frontends sometimes lose ops

And the workarounds:

Push posting ops to spo whenver a client joins. This results in the client asking SPO for ops after joining and it gets the successfully
Added extra logic to detect these gaps, fetch the ops from redis, and broadcast them to the client.

With my change in SPDF, both are now fixed with the same logic.

The frontend detects gaps on a per document level and sends the ops to the clients
The frontend detects gaps on a per client level and sends the op to the client (the gap ops may be sent up to 5 seconds later after detection, so the client needs to handle non-sequential ops)

@GaryWilber, some observations based on my prototyping (in priority order):

"If no ops are flushed, it returns a 204.". This is not very useful as it does not tell client the latest sequence number that SPO has. So client has no ability to make decision on what to do next (in situation where it needs to ensure that certain op made it to SPO in order to proceed with summarizaiton).
It is very useful for IConnected response to have supportedFeatures property listing all optional features like "api_flush_ops" & "api_get_ops" such that client can make decision on whether to use them, and whether to assert if it does not get proper response from server when such features are supported (i.e. defense in depth). For example, client for most part ensures that it will not have overlapping ops requests, so it could assert that responses indeed come back before next request shows up. Currently I can't do it as I do not know if particular PUSH box supports that feature.
Payloads do not have enough context on multiplexed socket for clients to understand if given response is applicable to given document. Similar, client can't tell if it's response for its request unless it tracks all nonces. In both cases it's possible to format nonce in a way where it has that data (i.e. it is formed in format of documentId-clientId-randomId, but it would be great for responses in both cases to contain documentId & clientId.

@vladsud Great observations.

If no ops are flushed, it would hopefully mean that the latest op the client processed is already in SPO. However we know that ops are broadcasted before being inserted into Redis so that may not be the case.

This is a bit tricky to solve because Push does not explicitly track the last persisted op to SPO. It is tracked by the deli lambda but it's not exposed to clients and it's not exposed in a way that's easy for me to grab during the flush_ops call. Is this a blocker for the single stage post summary work?

That does sound useful. For returning supportedFeatures, would this work:
export type FluidSupportedFeatures = "api_get_ops" | "api_flush_ops"
and add supportedFeatures?: FluidSupportedFeatures[]; to IConnected.

I will add tenantId, documentId & clientId to the responses.

For # 1, I'll go for now with assumption that "all ops are flushed", but if we see SPO rejecting summary due to lack of ops in such cases, we might need to revisit it.
I'd prefer to go with connected.supportedFeatures = { api_get_ops: true, api_flush_ops: true };
Mostly to allow more extensible format (i.e. maybe some future features would want to communicate level of support, or some other properties).

@vladsud

retryAfter will be there. Let me know if it's not.
Yes it's because the flushes are async between the two. The flushes clients run can happen at the same time as the ones Push run. The client initiated flushes do not run through the same queue system so they can happen concurrently.
I looked at the 409s. From what I see, every single conflict was from Push posting ops due to client joins at the same time. So once we stop posting ops when clients joins, you should see a large reduction in 409s

Everything covered in this issue has been implemented, including get_ops & flush_ops flows.
Next step - we should start re-evaluating places where we added complexity on PUSH side that are no longer required with get_ops flow.
Specifically:

We can deprecate initialMessages flow - see https://github.com/microsoft/FluidFramework/issues/7132
PUSH can stop flushing ops to SPO on client connection
We should evaluate any other places where PUSH can be simplified.

@GaryWilber - if you have any PBIs tracking any future work in this domain, it would be great to list them.
Otherwise I'm, not aware of any work on client in this domain, so closing it as fixed.

authors: GaryWilber anthony-murphy vladsud


text:
Refactor setAutoReconnect()

The functionality of setAutoReconnect() will be reworked to give the developer more control over when a container should and should not be reconnected. The new setAutoReconnect() will be defined as follows (the syntax may change in the future).
setAutoReconnect<T extends keyof typeof ReconnectMode>(
    mode: T,
    ...callback: T extends ReconnectMode.Custom ? [() => boolean] : never
): void;

Note about this syntax: If the given mode is ReconnectMode.Custom, then callback is required. If not, then callback not allowed.
We now allow ReconnectMode to be used to set the mode. This was done for a few reasons. First, we want to support the new Custom mode (introduced in https://github.com/microsoft/FluidFramework/issues/9166) which will make use of the callback argument. Secondly, using an enum as the modes will allow us to easily introduce new reconnection modes in the future if we so choose. Lastly, using an enum can help developers more easily understand what they are doing when calling this function.
We are also introducing the callback argument to provide additional control for the developer. This callback function will be required when the mode is set to Custom and will evaluate as a boolean. If the value is true then the container will reconnect, and if false it will not. This allows the developer as mechanism to control when a user is or is not reconnected based on information which the developer has access to, but Fluid does not.
These changes will be reflected in both the Container and ConnectionManager classes.

We are also introducing the callback argument to provide additional control for the developer. This callback function will be required when the mode is set to Custom and will evaluate as a boolean. If the value is true then the container will reconnect, and if false it will not. This allows the developer as mechanism to control when a user is or is not reconnected based on information which the developer has access to, but Fluid does not.

I think what you are trying to say is that developer wants pull model (FF asks it) vs. push model (developer changing auto-reconnect property as new data on his/her side arrives).
Before making API more complex, I'd love to understand why we believe we need to introduce more complexity and allow polling.
I have bigger issues with continuing to introduce these things on Container - I believe things like that should be exposed on Loader object, such that they apply to all containers under same context (here, "context" object is a loader, I do not think we need another context object). That way, it works great for things like guest components, where container points to a component from another container, and thus child container is loaded under same loader - doing operations on one container will miss this kind of scenario).
Developers should have ability to create any number of custom contexts (loader instances) and thus control the scoping of such policies.

This is a good point to consider.
If we do this at a loader level, is it still a natural place to inject callback? (Consider more complex FFX use cases around host apps). Also, maybe elaborate on "information which the developer has access to", so it's clear what those can be.
If the container is disconnected (say for an inactive tab), and you come back to it, what/who initiates reconnections?

I do not understand why callbacks are needed, so I can't answer that question. Access to information does not matter here though, as you can achieve same thing - the question is really why pull model is preferred over push, IMHO, and is this difference warrants more complicated API? Is there an example of scenario demonstrating it? I have not seen one.
Today, in FFX case, I believe OWH code initiates reconnect, I believe they use React capabilities to learn if user is there (any input generated), though I believe browser has similar capability (at global level).

Comments were for @scottn12 to clarify "information which the developer has access to", to understand need for callback. Scott - is this is all around "user inputs"?

Yes, user inputs such as mouse movements or clicks which are not accessible by Fluid. There may also be other relevant information, such as browser tab focus, for example. That said, based on feedback I received we probably won't be refactoring setAutoReconnect() after all, and I'll be updating the main issue shortly.

source: gharchive/issue

repo: microsoft/FluidFramework

license: MIT

license_type: permissive

license_source: github-api


text:
Protocol Definitions Typing Improvements

Description

Change usages of any to unknown. This changes nothing a runtime, but forces typescript users to safely access the value
Add null to ISequencedDocumentMessage.client id, as null is and always has been a valid value at runtime which must be handled. Server generated message currently have null as the value for clientId.
Change references to op or delta in comments to message, as messages do not map one to one with ops in many cases
Fix spelling errors.

Is this the correct branch (main) to target for breaking changes in our common packages? The will need to flow through server to reach client, which i know is only modified in main.

Is this the correct branch (main) to target for breaking changes in our common packages? The will need to flow through server to reach client, which i know is only modified in main.

Yes, it is. There have already been breaking changes to protocol-defs, and it's bumped to 2.0 on main in preparation. @ChumpChief and @ghoshkaj were the folks who made other changes. When you are ready to integrate into server and need a release, let me know.

Should this PR include a major bump in protocol-definitions? I thought one of the reasons we were holding on this change is that it's breaking.

I bumped it in my #15166 and we haven't released since then, so 2.0.0 is still the correct next major to release.

source: gharchive/pull-request

authors: ChumpChief anthony-murphy tylerbutler


text:
Huggingface Demo Error

Hello, it seems that there is a problem with the Huggingface Demo you provided and it cannot run properly.

I'm not sure where in the notebook you get this error, but I did notice than on the demo notebook, the url for downloading the model is not valid anymore. Use https://huggingface.co/GLIPModel/GLIP/tree/main for the weights.

I met with the same demo error problem. Did someone deal with that?

authors: Bowie1026 sarahESL wuyefa

repo: microsoft/GLIP


text:
[c++] Segmentation fault when use parallel_tree_learner method when number of categories of one feature larger than 28 (default max_cat_to_onehot-4).

https://github.com/microsoft/LightGBM/blob/5cd95a5b161d7630731d50e9ac529c6bf3dc809f/src/treelearner/split_info.hpp#L56
This code is used for computing the buffer size of the communicating the split info the in distributed training. But lost the two most recently added parameters(right_sum_gradient_and_hessian,  left_sum_gradient_and_hessian) size.
When use parallel_tree_learner method with number of categories of one feature larger than 28 (default max_cat_to_onehot(32)-2*2), it will resut in an Segmentation fault.
This code must be,
  inline static int Size(int max_cat_threshold) {
    return 2 * sizeof(int) + sizeof(uint32_t) + sizeof(bool) + sizeof(double) * 7 + sizeof(data_size_t) * 2 + max_cat_threshold * sizeof(uint32_t) + sizeof(int8_t) + sizeof(int64_t)*2;
  }

Im also having this issue while trying to use lightgbm ^4.0.0 with a ray cluster, the worker crashes with a segfault in DataParallelTreeLearner. I ended up changing the as stated above and building from source which fixes the issue but it would be great if this could be fixed in master. I tried to write a minimal example to reproduce the issue.
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import ray
from ray.train.lightgbm import LightGBMTrainer
from ray.air.config import ScalingConfig

def generate_synthetic_data(n_rows=10000, n_unique_categories=100):
    np.random.seed(42)
    print(n_rows)
    table = pa.table({
        "var1": np.concatenate((np.arange(n_unique_categories), np.random.randint(0, n_unique_categories, size=n_rows - n_unique_categories))),
        "label": np.random.randint(0, 2, size=n_rows)
    })

    return ray.data.from_arrow(table)

# Load training data
train_set = generate_synthetic_data()

# Train LightGBM
trainer = LightGBMTrainer(
    label_column="label",
    scaling_config=ScalingConfig(num_workers=2),
    params={'categorical_feature': 0, "tree_learner": "data", "verbosity": 2},
    datasets={"train": train_set},

)
result = trainer.fit()
print(result.metrics)

This makes the ray worker crash with the following log. Interestingly a warning is printed that categorical_feature is ignored, while actually it still seems to cause issues.
:job_id:01000000
:actor_name:RayTrainWorker
/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/lightgbm/basic.py:2034: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.
Please use categorical_feature argument of the Dataset constructor to pass this parameter.
  _log_warning(f'{key} keyword has been found in `params` and will be ignored.\n'
*** SIGSEGV received at time=1733493852 on cpu 3 ***
PC: @     0x7fee9b91f274  (unknown)  LightGBM::SerialTreeLearner::SplitInner()
    @     0x7ff1867fe520       1712  (unknown)
    @     0x7fee9b87f826  (unknown)  LightGBM::DataParallelTreeLearner<>::Split()
    @        0x200001388  129571456  (unknown)
    @     0x7fee9b87fca0  (unknown)  (unknown)
    @ 0x4810c08348fb8948  (unknown)  (unknown)
[2024-12-06 15:04:12,161 E 570227 570348] logging.cc:440: *** SIGSEGV received at time=1733493852 on cpu 3 ***
[2024-12-06 15:04:12,161 E 570227 570348] logging.cc:440: PC: @     0x7fee9b91f274  (unknown)  LightGBM::SerialTreeLearner::SplitInner()
[2024-12-06 15:04:12,162 E 570227 570348] logging.cc:440:     @     0x7ff1867fe520       1712  (unknown)
[2024-12-06 15:04:12,162 E 570227 570348] logging.cc:440:     @     0x7fee9b87f826  (unknown)  LightGBM::DataParallelTreeLearner<>::Split()
[2024-12-06 15:04:12,163 E 570227 570348] logging.cc:440:     @        0x200001388  129571456  (unknown)
[2024-12-06 15:04:12,164 E 570227 570348] logging.cc:440:     @     0x7fee9b87fca0  (unknown)  (unknown)
[2024-12-06 15:04:12,165 E 570227 570348] logging.cc:440:     @ 0x4810c08348fb8948  (unknown)  (unknown)
Fatal Python error: Segmentation fault

Stack (most recent call first):
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/lightgbm/basic.py", line 3891 in update
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/lightgbm/engine.py", line 276 in train
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/ray/train/lightgbm/lightgbm_trainer.py", line 65 in _lightgbm_train_fn_per_worker
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/ray/train/_internal/utils.py", line 176 in discard_return_wrapper
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/ray/train/_internal/utils.py", line 197 in train_fn
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/site-packages/ray/air/_internal/util.py", line 104 in run
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/home/melle/miniconda3/envs/thor/lib/python3.11/threading.py", line 1002 in _bootstrap

Extension modules: psutil._psutil_linux, psutil._psutil_posix, msgpack._cmsgpack, google._upb._message, setproctitle, yaml._yaml, _brotli, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, ray._raylet, lz4._version, lz4.frame._frame, zstandard.backend_c, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._parquet, cython.cimports.libc.math, pyarrow._json, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._flinalg, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, PIL._imaging, kiwisolver._cext, sklearn.__check_build._check_build, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, scipy.optimize._minpack2, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats.beta_ufunc, scipy.stats._boost.beta_ufunc, scipy.stats.binom_ufunc, scipy.stats._boost.binom_ufunc, scipy.stats.nbinom_ufunc, scipy.stats._boost.nbinom_ufunc, scipy.stats.hypergeom_ufunc, scipy.stats._boost.hypergeom_ufunc, scipy.stats.ncf_ufunc, scipy.stats._boost.ncf_ufunc, scipy.stats.ncx2_ufunc, scipy.stats._boost.ncx2_ufunc, scipy.stats.nct_ufunc, scipy.stats._boost.nct_ufunc, scipy.stats.skewnorm_ufunc, scipy.stats._boost.skewnorm_ufunc, scipy.stats.invgauss_ufunc, scipy.stats._boost.invgauss_ufunc, scipy.interpolate._fitpack, scipy.interpolate.dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.utils._random, _cffi_backend (total: 202)

yeah,  this error is caused by the above code。https://github.com/microsoft/LightGBM/issues/6491#issue-2362249887 The question was introduced when lightgbm>=4.0。

@moming39 you seem to have a clear idea of where the bug is and what called it. Would you like to submit a pull request with a fix?

Closing this based on the fix in #6738 and @shiyu1994 's approval there. This fix will go out in the next release of LightGBM.
If you build the latest development version and find that it it still not fixed, please comment and we can re-open it.

authors: MelleVessies jameslamb moming39

repo: microsoft/LightGBM

license: mit

license_source: bigquery


text:
fix some shellcheck warnings

Contributes to #6498
Following this attempt https://github.com/microsoft/LightGBM/pull/6548 at contributing to LightGBM and @jameslamb guidance I am now trying to help with fixing the shellcheck warnings for all the bash scripts inside the package (at least I think so).
Screenshot to show that the warnings are gone in my local environment:

Fixing https://www.shellcheck.net/wiki/SC1091 errors (source activate $SOME_CONDA_ENV) by pointing at at /dev/null is quite unsatisfactory but I did not manage to come up with something better.
Welcoming any feedback!

@microsoft-github-policy-service agree

I had to revert some of the changes due to breaking commands. Now we have 3 SC2086 (info): Double quote to prevent globbing and word splitting.  in build-python.sh. Pipelines seem to be passing (except the mac one, let me know if it's a regression please).

@jameslamb Thanks for the extensive review. I have noted all of your points. Apologies for pushing from master, I did not notice I had forgotten to branch out beforehand. I have reversed the files you have asked me too, let's keep the changes for another PR. It think I have adressed all of your points so far.

Thanks for the approval @jameslamb
Yeah I had started pushing from a git config not tied to my github email. One of the reasons I squashed all these commits together. Shouldn't happen again.


text:
[Question]: Why is every head config saved with "vertical_and_slash"?

Describe the issue
Regardless of the pattern observed, the config saves it as "vertical_and_slash" when using the search_patterns function.
https://github.com/microsoft/MInference/blob/b5b8745181c159d99be43f0759eeca3e8311447c/minference/modules/minference_forward.py#L198-L216
The configs saved in the repo appear to only contain this method type ^.
Specifically those lines:
 if best_ty == "stream_llm":
     best_ty = "vertical_and_slash"
 if best_ty == "block_sparse":
     best_ty, best_v, best_s = "vertical_and_slash", 1000, 6096

When doing the forward pass, I think this means that we never route to anything other than the vertical_and_slash impl / kernels.
Is this a bug or intended? The experiment docs cite the use of this search patterns function.

On the other hand:
Search pattern v2 https://github.com/microsoft/MInference/blob/b5b8745181c159d99be43f0759eeca3e8311447c/minference/modules/minference_forward.py#L220
does actually appear to save pattern type with specific names for routing to different pattern impls.

Do we need to use search patterns v2 to replicate the results of the paper? Or are the vertical_and_slash settings actually enough to pull off needle-in-a-haystack for long sequences?

Hi @fmmoret, thanks for your feedback.
The configs provided in the repo can reproduce the results from the paper. This means that the vertical_and_slash settings are sufficient to pass the Needle In A Haystack test for long sequences.
The search_pattern function reroutes to vertical_and_slash because our tests have shown that this setting offers better generalization and efficiency across different context windows and tasks.

Hi @iofu728 @fmmoret,
Why set best_v and best_s to 1000 and 6096 when the search results are block_sparse in ln216?
https://github.com/microsoft/MInference/blob/b5b8745181c159d99be43f0759eeca3e8311447c/minference/modules/minference_forward.py#L216

authors: PengWenChen fmmoret iofu728

repo: microsoft/MInference


text:
Project selection fails if folder names are lower case

Describe the bug
Project selection stage of the tool installer will fail, if the target project has lower case or mixed case folders, it will think its not a unity project and complain that the folders are missing
To reproduce
Make a dummy project with any of the three "Assets", "Packages", "ProjectSettings", with the case of any character in any folder changed from the example.
Selecting the dummy project will fail, with the "needs folders..." type error message.
Expected behavior
The folders are there, its a valid project, this warning should not occur.
Screenshots
Your setup (please complete the following information)

Unity Version N/A
MRTK Version MixedRealityFeatureTool-1.0.2104.4-Beta

Target platform (please complete the following information)

N/A

Additional context
You need to sanity check the comparison when running on windows, since windows does not allow folders to be created using same characters of different casing.
Run a lower-case-pass on all folder strings before comparing, or use a compare function that does so.
For targeting linux; find a suitable folder for the given name and set that as the name to use, or use this method in general.

Hello @Vectrobe thank you for submitting this issue on the Mixed Reality Feature Tool! I have assigned @davidkline-ms to investigate this, and @keveleigh added it to the appropriate Project for the Mixed Reality Feature Tool

I've just opened a PR on the MR Feature Tool repo to fix this. Thanks for the report!

authors: Vectrobe elbuhofantasma keveleigh

repo: microsoft/MixedRealityToolkit-Unity


text:
Add support for "show transparent/opaque settings at edit time" in camera profile

Describe the problem
Currently, the editor reports itself as opaque and our fallback case is also to display as opaque (since non-VR apps probably want a non-clear-flags-black background).
This becomes an issue at edit time, where Unity's game view will display the skybox even if you're primarily creating an AR app.
Describe the solution you'd like
For proper appearance and edit-time configurability, it'd be great if the camera profile had a dropdown where you could select the camera settings you'd like to be applied at edit time, instead of it always defaulting to the opaque settings.
Describe alternatives you've considered
Set the opaque settings to match the transparent ones, but that removes configurability for VR experiences.

We appreciate your feedback and thank you for reporting this issue.
Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.
Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.
Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.
Thank you for your continued support of the Mixed Reality Toolkit!

authors: IssueSyncBot keveleigh


text:
update diagnostic demo to get the service from the registry

Overview
Updates the demo script to use the registry to acquire the diagnostic system instance.
This change builds upon #4099 and is related to #3545

/azp run mrtk_pr


text:
Add daily build with ARM64 build flavour

Overview
Adds a daily build intended to be scheduled once a day. This build allows us to verify additional build flavours without clogging the main CI and PR pipelines. To begin with, I've added an ARM64 build that only runs on the daily build.
Alongside these changes I needed to modify the build scripts to allow us to build ARM64 on CI. This included adding ResolveAssemblyWarnOrErrorOnTargetArchitectureMismatch to the VS project. I believe this issue has been seen elsewhere for ARM64 builds.

Looks like most tests are failing in Unity2019. I confirmed this locally, but didn't look into it further.

Looks like most play mode tests are failing in Unity2019. I confirmed this locally, but didn't look into it further.

I think that @googlan mentioned the same thing a while back when we he was looking at 2019

@MenelvagorMilsom, once the merge conflicts are resolved, is this good to go in?

@davidkline-ms I was holding out on this one until someone's had a look at tests failing for Unity2019. As far as I'm aware no one has had a look yet.

/azp run

@davidkline-ms the usual PR build won't be able to verify if tests are passing in 2019 now. If we want to check that I can rebase this branch and test on the daily build.

authors: MenelvagorMilsom davidkline-ms wiwei


text:
Spell check!

Overview
Ran a quick spell check through our code.
Also fixed an instance of tabs -> spaces.

/azp run

@keveleigh, can we make spellcheck and tabs/spaces part of the PR validation? At least as notes/warnings

can we make spellcheck and tabs/spaces part of the PR validation? At least as notes/warnings

Agreed, yeah. @wiwei and I added https://github.com/apps/prosebot a while back, but it seems to only run on .md files, and inconsistently at that. There's probably a better way.

authors: Troy-Ferrell keveleigh


text:
Add support for motion controller models at runtime in WMR/XRSDK

Overview
Refactors out a WindowsMixedRealityControllerModelProvider to call into the WinRT API and load the corresponding glTF model across both legacy XR and XR SDK.
Changes

Fixes: https://github.com/microsoft/MixedRealityToolkit-Unity/issues/8971

/azp run

/azp run

/azp run

/azp run

/azp run


text:
Treat untyped blank as a scalar rather than a table

Addresses https://github.com/microsoft/Power-Fx/issues/1317 based on excellent suggestions from Carlos.  We will now match Canvas semantics and lean toward an untyped blank being a scalar rather than a table.

✅ No public API change.

✅ No public API change.

✅ No public API change.

✅ No public API change.

✅ No public API change.

How would this work for below expression? Would return type for below expression be Void?
If(false, [1,2], Blank())

This change affects mostly tabular functions - the logic to determine the return type of If is not changed. In this case, since *[Value:n] accepts ObjNull (the type of Blank()), the result will be *[Value:n].

In reply to: 1624171294

✅ No public API change.

✅ No public API change.

authors: CarlosFigueiraMSFT LucGenetier Shpakh-MSFT gregli-msft

repo: microsoft/Power-Fx


text:
[[Power-Fx]] Localization hand-back [CTAS - RunID=20230905-153259-c48tx9kb27]

Translation handback check-in (PR Created by automation).

✅ No public API change.

authors: LucGenetier toshio-msft


text:
Using Ajax inside a custom visual

Hi All,
I am trying to write a custom visual, where I want to make a few REST calls inside the "update" function. But when I try to do this, I get access denied.
Is there a way of doing this?
Thanks!
Edwin

REST calls are not allowed in CV
Closing due to age

authors: AleksSavelev EdwinStraver

repo: microsoft/PowerBI-visuals-tools


text:
Google Books (Independent Publisher)

When submitting a connector, please make sure that you follow the requirements below, otherwise your PR might be rejected. We want to make you have a well-built connector, a smooth certification experience, and your users are happy :)
If this is your first time submitting to GitHub and you need some help, please sign up for this session.

[x] I attest that the connector doesn't exist on the Power Platform today. I've verified by checking the pull requests in GitHub and by searching for the connector on the platform or in the documentation.
[x] I attest that the connector works and I verified by deploying and testing all the operations.
[x] I attest that I have added detailed descriptions for all operations and parameters in the swagger file.
[x] I attest that I have added response schemas to my actions, unless the response schema is dynamic.
[x] I validated the swagger file, apiDefinition.swagger.json, by running paconn validate command.
[ ] If this is a certified connector, I confirm that apiProperties.json has a valid brand color and doesn't use an invalid brand color, #007ee5 or #ffffff. If this is an independent publisher connector, I confirm that I am not submitting a connector icon.

If you are an Independent Publisher, you must also attest to the following to ensure a smooth publishing process:

[x] I have named this PR after the pattern of "Connector Name (Independent Publisher)" ex: HubSpot Marketing (Independent Publisher)
[x] Within this PR markdown file, I have pasted screenshots that show: 3 unique operations (actions/triggers) working within a Flow. This can be in one flow or part of multiple flows. For each one of those flows, I have pasted in screenshots of the Flow succeeding.
[x] Within this PR markdown file, I have pasted in a screenshot from the Test operations section within the Custom Connector UI.
[ ] If the connector uses OAuth, I have provided detailed steps on how to create an app in the readme.md.

Hello @fordosa90,
Thank you very much and congratulations on your new submission. Our Certification Engineers will review it very soon.
Please let us know if you have any questions.
Thank you very much again for working with us.

[[certify-connector]]

Hello @fordosa90,
I hope you are doing well.
Congratulations, your pull request is approved. We will move forward with the certification process. We are preparing your connector for production deployment and will queue up for the next deployment schedule. Once your connector onboards the next deployment schedule, it starts to deploy your connector in our production environments that typically takes 3-4 weeks.
Please let us know if you have any questions.
Thank you very much for working with us.

@Amjed-Ayoub thank you for the very quick process, great improvements!
Looking forward for further collaboration!

Hello @fordosa90 ,
I hope you are doing well.
Your connector is in production deployment and queued up for the next deployment schedule. This process typically takes 3-4 weeks. Please note that the documentation for your connector will not be generated until the deployment process is in the final steps.
Please let us know if you have any questions.
Thank you very much for working with us.

repo: microsoft/PowerPlatformConnectors


text:
Create a tester app for fancy zones that mimics different types of window configs

As we are developing fancy zones, we are encountering the wonders of the windows ecosystem and all the different ways windows can be created. This is great to see all the clever ways people do things.
We should create a mock app that simulates these so we can do automated testing with winappdriver
We when encounter a new app that we didn’t account for, this tester app needs to be updated.
It is ok if we have multiple apps to do all the testing :)

Closing this for now since in the recent months we haven't had issues with regard to properly processing newly created windows.
We still have app-compatibility issue to detect dragged windows but it was never the goal of this test app to cover that scenarios.

Closing this for now since in the recent months we haven't had issues with regard to properly processing newly created windows.
We still have app-compatibility issue to detect dragged windows but it was never the goal of this test app to cover that scenarios.

authors: crutkas enricogior

repo: microsoft/PowerToys


text:
Virtual Desktop Utilities / Recording Utilites

Description of the new feature / enhancement
There are two things I want to propose:
1. Virtual Desktop tooltip
When navigating from a virtual desktop named "Develop" to a virtual desktop named "Test", it would be useful to show a very small tooltip-like popup at the top of the screen containing the name of the current desktop ( i.e. Test). Would this be possible by a new PowerToy ?
Something like this:

2. Keyboard Shortcut popup
When recording tutorials it would be useful to show key combinations like "Ctrl + Shift + C" to help people to follow along. Would this be possible by a new Powertoy ?
Something similar to this:

Or, even better a new simple screen recorder Powertoy? It would be useful when you want to do a quick record, etc. Currently, there is the windows 10 game bar, but it cannot record the desktop or windows explorer and I find it a bit buggy in my computer. There is also an another inbuilt screen recording software. To access it, go to PowerPoint -> Blank Presentation -> Insert -> Screen recording.
This is a very simple and effective tool to record small or large videos, but when recording, the bar at the top (the toolbar like thing when recording, which contains the Play/Pause/Stop buttons) is also included in the recording. So, it would be better if it was fixed in a PowerToy.
Scenario when this would be used?
The first one would be useful when you are working with many virtual desktops.
The second one would be useful when you are recording some tutorials or short videos.
Supporting information
No response

I recommend you to file your suggestions in the
Feedback Hub. PowerToys isn't the right place for adding this.

I recommend you to file your suggestions in the
Feedback Hub. PowerToys isn't the right place for adding this.

Ok. I will use Feedback Hub

@Aaron-Junker wasn't there a concept to build a new, simple screen recorder module? Maybe that tag applies too.

@jay-o-way You're right. Idea 2 is a duplicate of #981

@Aaron-Junker i meant #143 and https://github.com/microsoft/PowerToys/wiki/Video-GIF-Capture

@Aaron-Junker i meant #143 and https://github.com/microsoft/PowerToys/wiki/Video-GIF-Capture

@jay-o-way But this isn't the proposed feature.

authors: Aaron-Junker Jay-o-Way samcodee


text:
Doesn't update properly via WinGet

Microsoft PowerToys version
0.43
Running as admin

[ ] Yes

Area(s) with issue?
General, Installer
Steps to reproduce
Running PT v0.41.4 & winget v1.0.11694
Winget upgrade, lists that there is an update (0.43) available.
But winget upgrade --all, results in "No applicable update found"
✔ Expected Behavior
That PowerToys would acctually be updated via winget.
❌ Actual Behavior
"No applicable Update Found":
C:\Users\USR>winget upgrade
Name                Id                  Version Available Source
----------------------------------------------------------------
PowerToys (Preview) Microsoft.PowerToys 0.41.4  0.43.0    winget

C:\Users\USR>winget upgrade --all
No applicable update found.

This is referenced in an issue on the Winget Github
https://github.com/microsoft/winget-cli/issues/1191#issuecomment-870667044
But I wanted to highlight that it is still present in v0.43
Other Software
winget v1.0.11694

I have the same problem. I think we should delete the "install via WinGet" section in README.

I saw that 0.46 was released so I ran winget search powertoys, and only 0.45 is available. Then I saw in the release notes that 0.46 wasn't published on purpose? Why? It's very useful to install app updates via winget, so this is disappointing...

I saw that 0.46 was released so I ran winget search powertoys, and only 0.45 is available. Then I saw in the release notes that 0.46 wasn't published on purpose? Why? It's very useful to install app updates via winget, so this is disappointing...

0.46 is experimental. So it isn't that entirely crazy that it isn't published on winget. At least not until it becomes stable. And basically the upgrade via winget is not working anyway. So until they fix that, it doesn't really matter much. But hopefully they'll get everything sorted out in due time.

@factormystic .46 was an experimental build and not on WinGet, we directly called that out when we were getting VCM a bit more stable.  No VCM experimental build was on WinGet.
Adding @Camatius  here from the issue he opened.
Currently, I can't repro this.  I did a manually installed 0.53.1 to 0.53.3 and even from an internal build (0.0.2 of a .55 variant) to 0.53.3.  I'm aware this was an issue in the past but I know WinGet has been working on the upgrade scenario for apps like PowerToys.  Wonder if maybe i have a newer winget?
I'm on Windows Package Manager (Preview) v1.2.10132-preview

@crutkas
I'm sorry - didn't catch the notification about your reaction. I've tried it with the recent version after a manual upgrade from an older one and Winget is working correctly. So from my point of view, the problem was solved somewhere between versions.
Thanks.

authors: Camatius ChungZH cloudrootab crutkas factormystic


text:
Version 0.15.1 installer fails to complete installation; error reported

Environment
Windows build number: 10.0.18362.657
PowerToys version: 0.15.1

Steps to reproduce

Start with an existing PowerToys 0.14.1.0 installation. (Don't know if this is important or not.)
Double-click PowerToysSetup-0.15.1-x64.msi to launch it.
Click through the license and the installation location. Click "Install" to begin the installation.
I get an error message saying "The system cannot open the device or file specified."

Expected behavior
I expect the installation to proceed to completion.
Actual behavior
What I actually get is a failed installer.
When I click through the failure dialog, I get a second error saying "The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2755." I'm hoping that error code is meaningful to someone.
I've tried to isolate the issue with Procmon, but can't locate the specific call that's failing.
Screenshots

Hi @ahernsean
thank you for reporting the error.
The 2755 error may happen when installing from some particular folder.
Where did you save the downloaded MSI?

The actual pathname is C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi
Nothing special.

Can you try generate an install log, running this command from the command prompt?
Thanks.
cd C:\Users\sahern\Downloads\
msiexec /i PowerToysSetup-0.15.1-x64.msi /l*v "install.log"

#283 could be same issue.  At the end, it says 1603 is the error code but we do see other errors from the logs.
MSI (s) (BC:24) [12:00:38:290]: Note: 1: 1309 2: 5 3: C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi
The system cannot open the device or file specified.
MSI (s) (BC:4C) [12:00:42:742]: I/O on thread 42596 could not be cancelled. Error: 1168
// lots more
MSI (s) (BC:4C) [12:00:42:743]: I/O on thread 42484 could not be cancelled. Error: 1168
MSI (s) (BC:24) [12:00:42:746]: MainEngineThread is returning 110

MSI (c) (B4:F4) [12:00:42:758]: Note: 1: 2205 2:  3: Error
MSI (c) (B4:F4) [12:00:42:758]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 2755
DEBUG: Error 2755:  Server returned unexpected error 110 attempting to install package C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi.
The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2755. The arguments are: 110, C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi,
MSI (c) (B4:F4) [12:00:43:923]: Note: 1: 2205 2:  3: Error
MSI (c) (B4:F4) [12:00:43:923]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (c) (B4:F4) [12:00:43:923]: Product: PowerToys (Preview) -- The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2755. The arguments are: 110, C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi,

Could be. The error code is different, but there are similarities in the logs.
Could the fact that the "shutdown disable" counter goes negative be part of the issue? A counter with a value of -1 sounds odd. Both of us have that.

@ahernsean
the MSi log isn't helping a lot. The error starts here:
MSI (s) (BC:24) [12:00:38:290]: Note: 1: 1309 2: 5 3: C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi
The system cannot open the device or file specified.

The error is too generic since it doesn't report which file is trying to open.
From the log it seems you have the previous version installed, can you try to re-run
msiexec /i PowerToysSetup-0.15.1-x64.msi /l*v "install.log"

after uninstalling PowerToys?
Go to Windows 10 Settings -> Apps to uninstall it, don't use the legacy control panel because it is known to not work well when uninstalling PowerToys.
Thanks.

@ahernsean
do you have a third party anti-virus? In case can you try to disable it before running the installer?

I do have a 3rd-party anti-virus but am unable to disable it due to corporate policy.
But I will try uninstalling the earlier version of PowerToys and the doing the install. I may also try ProcMon and see if I can capture the specifics of the system call that's failing.

Failed again. Error code 2755 again. I'm not sure this log is any more useful than the last, since the error is still the same: "MSI (s) (24:64) [09:00:28:827]: Note: 1: 1309 2: 5 3: C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi
The system cannot open the device or file specified." But here's the log anyway:
install.log

I found four events with ProcMon that might be worth exploring more. Early on, it gets a "FILE LOCKED WITH ONLY READERS" for the .msi file.
9:06:26.1012395 AM	msiexec.exe	31268	CreateFileMapping	C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi	FILE LOCKED WITH ONLY READERS	SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE
Not sure what that means. Later on, we get the same error again.
And then at the end, we get two similar "ACCESS DENIED" errors:
9:06:26.4428058 AM	msiexec.exe	31268	CreateFile	C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi	ACCESS DENIED	Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Open No Recall, Attributes: n/a, ShareMode: Read, AllocationSize: n/a
9:06:26.4450041 AM	msiexec.exe	31268	CreateFile	C:\Users\sahern\Downloads\PowerToysSetup-0.15.1-x64.msi	ACCESS DENIED	Desired Access: Read Control, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a

As a point of data, the 0.14.1 MSI still installs fine, so this failure doesn't appear to be due to anything that changed on my system.

The ACCESS DENIED error is very strange. Can you try to rename PowerToysSetup-0.15.1-x64.msi to PowerToysSetup.msi? It shouldn't make any difference but it's something I would like to count out as possible cause (the file was signed as PowerToysSetup.msi and then renamed to PowerToysSetup-0.15.1-x64.msi
Thanks.

That worked!
Well, sort of. The installation went all the way through this time. When it was done, Windows Explorer had crashed. I had to manually start it back up again. But once I did and manually start PowerToys up, I was able to confirm it's running the latest version:

this should be fixed then with the pipeline adjustments, right?

this should be fixed then with the pipeline adjustments, right?

Yes, we now rename before signing.

That worked!

Thanks for the time you took to help finding the issue!

Well, sort of. The installation went all the way through this time. When it was done, Windows Explorer had crashed.

This is another issue that has also been reported by other users: the install process is supposed to restart Explorer to reload the PowerRename shell extension. In some cases, Explorer doesn't restart.

Adding a reference to the PR that fixes this issue https://github.com/microsoft/PowerToys/pull/1439

Problem resolved in the 0.15.2 MSI installer.

@ahernsean thank you for the feedback!

Looks like the problem has cropped up again in the 0.16.0 installer:

I guess I shouldn't have reopened the ticket, since it's technically a different installer.

@ahernsean
I suspect your antivirus software needs some time to add the PowerToys binary to the "allowed applications", because this time the msi file has been properly signed and not renamed.
Can you retry in a few hours?

Hm. I'm pretty sure I've got this in a place where developer binaries aren't subject to the scanning.
With that said, sure, I'll try this again in a few hours. Or maybe even tomorrow.

where / how did you get the installer?

Downloaded the MSI from https://github.com/microsoft/PowerToys/releases/tag/v0.16.0

would it be possible to set up a teams call where i could see how you're installing / what other things may be interfering?  the file is signed by Microsoft so any anti-virus software should see it as a trusted installer.
What browser are you using to get the file?

Run or download then run?

legacy edge I think you could run or save.
In chrome / edge, files are downloaded then user executed, not instantly run.

Yeah, we could do that.
I used Chrome to download the file, though I don't think that should matter. The md5sum is 8fd12abbd59bed0de29c1ac6bf4ed957.
I get the same failure when I run it directly from Explorer or the Command shell.
Sean Ahern / Lead Software Developer / Nexus Project Manager
[cid:73ced98c-6dfd-47f1-bfd1-db34f1bff821]
CEI, Inc., an affiliate of ANSYS, Inc.
2166 N. Salem St. Suite 101  /  Apex, NC 27523
C: 865-773-3557  /  W: 919-719-6568
sean.ahern@ansys.com  /   www.ansys.comhttp://www.ansys.com/

From: Clint Rutkas notifications@github.com
Sent: Tuesday, March 31, 2020 4:39 PM
To: microsoft/PowerToys PowerToys@noreply.github.com
Cc: Sean Ahern Sean.Ahern@ansys.com; Mention mention@noreply.github.com
Subject: Re: [microsoft/PowerToys] Version 0.15.1 installer fails to complete installation; error reported (#1444)
would it be possible to set up a teams call where i could see how you're installing / what other things may be interfering? the file is signed by Microsoft so any anti-virus software should see it as a trusted installer.
What browser are you using to get the file?

Run or download then run?
*   legacy edge I think you could run or save.
*   In chrome / edge, files are downloaded then user executed, not instantly run.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/PowerToys/issues/1444#issuecomment-606862270, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMSVFGCCZ4JZVOFSSBGBHTRKJIJVANCNFSM4LBH3BGA.

Interestingly, if I move it to C:\temp, the error message changes:

Monitoring with Procmon.exe doesn't reveal anything useful.

Here's an installer.log, taken with:
msiexec /i PowerToysSetup-0.16.0-x64.msi /l*v "install.log"
install.log

It's now 19 hours since I first ran into the problem. I'm still experiencing the same issue.
The install.log says that the signature is accepted and permits the installation, so it doesn't feel like a signing problem this time.
MSI (c) (EC:50) [17:11:17:046]: SOFTWARE RESTRICTION POLICY: Verifying package --> 'C:\ANSYSDev\home\sahern\tmp\PowerToysSetup-0.16.0-x64.msi' against software restriction policy MSI (c) (EC:50) [17:11:17:048]: SOFTWARE RESTRICTION POLICY: C:\ANSYSDev\home\sahern\tmp\PowerToysSetup-0.16.0-x64.msi has a digital signature MSI (c) (EC:50) [17:11:17:191]: SOFTWARE RESTRICTION POLICY: C:\ANSYSDev\home\sahern\tmp\PowerToysSetup-0.16.0-x64.msi is permitted to run at the 'unrestricted' authorization level.

The identical problem happens with PowerToysSetup-0.16.1-x64.msi.
I'm wondering if others are seeing this bug. If I'm the only one seeing it, then the problem is likely with my local setup.

@ahernsean
so far you are the only user that reported this problem, doesn't necessary mean you are the only one seeing it, but at least it means that is not very common.
I wonder why with 0.15.1, renaming it fixed the problem, we thought it was because we signed it and then rename it, but for the following releases we signed without renaming it.
Just out of curiosity, have you tried to rename it to PowerToysSetup.msi?

Well, I'm guessing it's a different problem this time. The problem last time had to with signing the installer, then renaming it. The 0.16.x installer has been signed with its final name, so that's no longer an issue (and confirmed by the install.log).
Yes, I have tried renaming it to PowerToysSetup.msi. Same error, unfortunately.

Same error, unfortunately.

I was expecting it, but you never know.
Too be honest I'm running out of options.
If you don't mind I'm closing this issue since it was for a different problem and let's keep https://github.com/microsoft/PowerToys/issues/1797 open.

Sounds good.

authors: ahernsean crutkas enricogior


text:
Bug of Always on Top

Microsoft PowerToys version
0.64.0
Installation method
Microsoft Store
Running as admin
Yes
Area(s) with issue?
Always on Top
Steps to reproduce
I'm sory I don't have the screenshoot.
✔ Expected Behavior
Please fix the bug of Always on Top. I use it with Imaging Edge Desktop app from Sony for photograpy.
❌ Actual Behavior
When I use the "Always on Top" with Imaging Edge Desktop from Sony, the folder getting crash and not responding. I use it to photography. I take a photo with my camera Sony a6400 and save automatically in my folder. The folder is set "Always on Top. At first the system running well, after a few minutes the folder becomes not respondingand I have to restart my computer. For your information, I use Windows 10 Home Single Language version 22H2.
Other Software
When I use the "Always on Top" with Imaging Edge Desktop from Sony, the folder getting crash and not responding. I use it to photography. I take a photo with my camera Sony a6400 and save automatically in my folder. The folder is set "Always on Top. At first the system running well, after a few minutes the folder becomes not respondingand I have to restart my computer. For your information, I use Windows 10 Home Single Language version 22H2.

Hi, thank you for the report. Can you please check to be sure that it doesn't behave this way when it's not pinned by AlwaysOnTop?
If it happens again when the window is on top, please attach here a /bugreport.


text:
SVG not rendered correctly.

Microsoft PowerToys version
0.74.0
Installation method
PowerToys auto-update
Running as admin
None
Area(s) with issue?
File Explorer: Thumbnail preview
Steps to reproduce
Some SVG files do not render correctly.
source.zip (svg file zipped)
✔ Expected Behavior
Rendered in browser.

❌ Actual Behavior
Rendered in explorer (Win10).

Missing checkmark. (hole)
Some files with hole works, other not. I cannot reproduce. I played with some styles etc in text editor. Maybe it's only on specific positions of points... I wonder if we could use a more proven/robust web renderer for SVGs (the underlaying lib of browsers).
Other Software
No response

Hi @infacto, thanks for the report.
We do in fact use MS Edge to render previews, though it could have edge cases still that we need to fix.
Note: I've just checked this issue against PowerToys.SvgThumbnailProvider.exe@v0.73 and it's not a regression.

Ok, interesting because the "Rendered in browser" screenshot (last post) is from Edge browser and looks like expected. Why is the result different in explorer thumbnail if both uses the same renderer? ...
Yes, it's no regression. It has never worked correctly in such cases. I updated from 0.6x. Currently I have the problem that the thumbnails are not working anymore. I already tried to toggle the SVG preview enable switches in power toys off and on and restarted the PC. I'll try to uninstall and install again tomorrow. Or is there any other way to fix it or find out what's wrong? How to check if the SVG preview stuff is running?

authors: infacto yuyoyuppe


text:
Powertoys Run causes UWP compile to fail in visual studio

Environment
Windows 2004
Visual Studio 16.6.3
Powertoys 0.19.1
Steps to reproduce
Enable Powertoys Run (I hadn't done this until upgrading to 19.1 last week)
Launch a debug session of a UWP Xamarin app in visual studio
Expected behavior
Expect the debug session to run first time.
Actual behavior
First attempt to compile and debug a UWP app after a code change ends with this error (every time):
DEP1000: Cannot copy file "D:\Work\App\App.UWP\bin\x86\Debug\Core\AppxManifest.xml" to layout "D:\Work\App\App.UWP\bin\x86\Debug\AppX\AppxManifest.xml". IOException - The process cannot access the file 'D:\Work\App\App.UWP\bin\x86\Debug\AppX\AppxManifest.xml' because it is being used by another process. [0x80070020]
Restarting the compile again immediately after the error succeeds (again each time).
Checking in resource monitor on the first run shows Powertoys Launcher has a handle on the file.
Disabling the Powertoys Run features stops the error appearing.
Enabling  it again causes the error to reappear.
The error is consistently reproducible.

@ryanbodrug-microsoft will your package update fix this?   https://github.com/microsoft/PowerToys/pull/4521

@ryanbodrug-microsoft will your package update fix this? #4521

I don't think #4521 will fix this.  It looks like a file handle isn't getting closed properly when we index the UWP applications.   I didn't touch anything that would change parsing locic of the appxmanifest.xml files.

I experienced this same thing, but with the PNG file for my (in-development) UWP app icon.

We just released 0.20 and this should be resolved there.  Please try it out.

authors: Rippletank crutkas jccr ryanbodrug-microsoft


text:
Cannot Install Powertoys 0.20

ℹ Computer information

Windows build number: [run "winver"]
PowerToys version:
PowerToy module:

📝 Provide detailed reproduction steps (if any)

…download powertoys 0.20 from github
…press install
…uninstalls previous version and halts ... see screenshot

✔ Expected result
What is the expected result of the above steps?
❌ Actual result
What is the actual result of the above steps?
📷 Screenshots:

Are there any useful screenshots? WinKey+Shift+S and then just paste them directly into the form

Running into a similar issue, except I don't get the pop up. I can see the PowerToysSetup process in Task Manager but nothing happens. It also prevents me from reinstalling version 0.20 until io kill that preocess.

I cannot install it either. It hangs at 99%.

My notification disappears at around 60% and then nothing.. windows event log shows nothing.

Killing dotnet.exe in taskmanager when installing completes the installation.

@icyhoty2k
do you get the same error installing 0.19.2? https://github.com/microsoft/PowerToys/releases/tag/v0.19.2

authors: PapierenTijger enricogior icyhoty2k lawrencefej mattiaskagstrom


text:
Holding shift key doesn't activate zones while dragging

ℹ Computer information

PowerToys version: b0.21.1
PowerToy Utility: What?
Running PowerToys as Admin: Yes
Windows build number: 20175.1000

📝 Provide detailed reproduction steps (if any)

FancyZones is enabled
Drag a window, hold the shift key
Notice fancyzones doesn't activate, doesn't snap to anything.

✔ Expected result
Holding shift while dragging should show me translucent rectangles of where I can drop the window, and of course, once I let go of the window while dargging, it should snap to the zone.
❌ Actual result
Nothing happens.
📷 Screenshots
Even better: here's a video. https://www.dropbox.com/s/yi6964sllzdl8xe/Untitled Project.mp4?dl=0
Notice I'm holding the shift key. Nothing comes up. I try to select a different zone layout via the editor. Same result.

i have same issue, work without having ''allow zones to span across monitor, single monitor work, and multi monitor doesn't work

@tommymogwais I can't seem to get it to work in either case. However I do keep noticing that "allow zones to span across monitors" won't stay checked. If I leave and come back, it's unchecked even though I know it was checked.

Same here when "allow zones to span across monitors" was selected.
This setting is also not saved. When you click the checkbox, switch to different settings and back or restart powertoys, the checkbox is unchecked.
If the active profile spans multiple monitors, the zones don't seem to activate when pressing shift.

For ''allow zones to span across monitor'' not working the tracker is https://github.com/microsoft/PowerToys/issues/6302
For shift drag not working when ''allow zones to span across monitor'' is not on, the tracker is https://github.com/microsoft/PowerToys/issues/6036

authors: Fonta enricogior samrueby tommymogwais


text:
Enhanced Clipboard history with visualization

it would be fantastic if we could have an enhanced version of clipboard history.
A future item here could be making it modular as well for additional ways to visualize / extend the clipboard items based on the file type.
@xyzzer has a good base.

@crutkas you mean itemas by mime type. Something like https://www.nirsoft.net/utils/inside_clipboard.html

@xmedeko What I have and think of building on looks like this:

When something gets copied to clipboard successfully - you get a little, non-disruptive confirmation tag/button sliding out of a corner of the screen to make you confident you copied something successfully.
When you click the tag - you get the clipboard history UI you normally access with Win+V
When you hover - you get access to a menu of simple contextual commands based on content type and at least in case of text content right now - based on actual content of the copied text using a set of regular expressions that ultimately, a user could configure. If something you copied looks like a GUID - you click to look it up in your source code, documentation, registry or other GUID ID database. If it looks like a name - you could look it up in Facebook, LinkedIn or your address list. Any text can be used to invoke a bing search, a wiki search or source code search. You can also use a keyboard shortcut to strip the format from the text and paste it without formatting even in an app that doesn't normally support "paste as plain text". If you copy an image file - you can click a button to convert it to a bitmap to paste in any app that supports pasting images (e.g. Outlook) and if you copy a bitmap - you can click to use an AI service to get the description of the bitmap as text in the clipboard or save it to an image file and get the file and its path as added formats in the clipboard so you can paste it anywhere that takes a file or a file path but doesn't directly work with an image itself (see below).
You could configure your own text types and URLs or programs that can be opened for them or plug in a PowerShell script to parse+recognize content and/or process it in any way if you want to add your own command to say - resize an image in clipboard before pasting it somewhere or quickly add meme-style text to an image before posting it on social media.
Ultimately, this is a way to speed up work. For example, I use it daily to copy a bug ID and look it up in Azure DevOps with one click instead of - opening the browser, loading ADO, pasting the ID and clicking the search button. We could build a set of built in search patterns and commands and you could also add your own.

@xyzzer Looks very powerful. What I need at the moment is just to see all clipboard formats, like InsideClipboard.
I would welcome just some text (regexp) manipulations. E.g I copy a plain text https://github.com/microsoft/PowerToys/issues/671 and the tool add a HTML clipboard content #671.  So, I can paste it without any click in the tool menu.

Hi, everybody!
Some little suggestions:

to config how many items to save;
to move the used items to top of list;
to keep/remove formatting

Thank you, all!

https://bluemars.org/clipx/

One thing I miss in the clipboard manager is the inability to set time limit for which to keep the clipboard items. Most clipboard managers have this implemented.

authors: crutkas haidar47x nolde robsonsobral xmedeko xyzzer


text:
Standardize Image Resizer naming

Following names are currently in use

Image Resizer: Used in the Enabled field in the global settings file at %userprofile%/appdata/local/microsoft/powertoys/settings.json
ImageResizer: Used in folder name to store settings in %userprofile%/appdata/local/microsoft/powertoys

This change will not be backward compatible but will ensure a consistent naming between module key associated with runner and folder naming on disk, similar to other modules.

This change will not be backward compatible

That's why the old version is uninstalled with every update, not?

@Jay-o-Way
the uninstaller doesn't remove the settings folder, we can add code to detect if the ImageResizer folder is present in the settings folder and rename it to Image Resizer if Image Resizer is not already present and delete it if Image Resizer is already present.
This will only break backwards compatibility if the user uninstalls the current version and installs an older version.

This is fixed in 0.53, get it at https://aka.ms/installPowerToys

authors: Jay-o-Way crutkas enricogior somil55


text:
User/ryanbod/mock settings disk access

Summary of the Pull Request
Dependency Inject SettingsUtils and SystemIOProvider to abstract disk access, and remove the need to write / read from disk during unit tests.
PR Checklist

[x] Applies to #6096
[ ] CLA signed. If not, go over here and sign the CLA
[ ] Tests added/passed
[ ] Requires documentation to be updated
[ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Info on Pull Request
What does this include?

Introduces SystemIOProvider and IIOProvider interface to wrap methods that hit the disk.
Dependency Inject the concrete implementation of SystemIOProvider and SettingsUtils into the viewmodels, so that unit tests can provide stub and mock implementations.
Removes setup and cleanup methods from unit tests that write to disk.
Removed 'enabledbydefault' test for colour picker and added a new test that validates all modules are enabled by default.

Validation Steps Performed
How does someone test & validate?

Runs all unit tests.
Opens Settings for each module,
2.1 Verifies that settings update and save.
2.2 Verifies that modules are able to load modified settings.

All previously observed issues.  Were not a result of these changes.

Keyboard Manager process wasn't launching was fixed in. #6185.
IPC messages that weren't getting picked up on settings change was caused by strings not being localized.  This just required a clean / rebuild.  See #6197 for context.

@laviusmotileng-ms:  ^^ FYI

Validated that all the tests pass and settings get saved/loaded correctly on each page of the Settings app. Apart from the comments I've left, one part I'm confused about is the use of ISettingsUtils. It seems like this interface was required only for the mock done in GetStubSettingsUtils, but I feel like that functionality could be implemented using IIOProvider. The default json could be set in the saveContent and loaded from that. Are there any other areas which motivate the requirement on the two interfaces?
I can approve the PR after these clarifications.

A bit philosophical (shocking coming from me), but IIOProvider, and ISettingsUtils encapsulate different concerns to me.   The SettingsUtils should encapsulate the logic specific to saving and loading settings information, and any patterns that they may include, such as does the file exist (if not create it etc).    The IOProvider encapsulates the logic pertaining to persisting the information, hypothetically by writting to disk, or saving to a sql db, or blob storage.    This is intended to decouple the settings and unit tests from a specific implementation, but doesn't actually define the patterns that we should be using.  Alekhya's change should be doing that, and I may have to modify some of this once I take a look at her PR.


text:
Create workbooks for the katas/tutorials

The katas offer a set of programming exercises on quantum computing and reference solutions to them, but we often hear that it would be helpful to have the solutions explained, with the logic steps necessary to arrive from the problem description to the code spelled out. It is especially important for people who go through the katas on their own, without a study group to support them.
To address this, we came up with the idea of "workbooks" - explanations of the solutions to the tasks offered in the katas/tutorials, written in Jupyter Notebook format. You can see the workbook for Superposition kata here - it is still a work in progress, but it already shows the idea of the workbook, the kind of the content it should have and the way it accompanies the kata.
There are several other katas/tutorials that are a great fit for this kind of learning material:

[x] Complex arithmetic (classical)
[x] Linear algebra (classical)
[ ] Single-qubit gates
[ ] Multi-qubit systems
[ ] Multi-qubit gates
[x] Superposition (work in progress)
[x] Measurements (work in progress)
[ ] Joint measurements
[ ] SolveSATWithGrover
[ ] GraphColoring
[ ] Unitary patterns

Other katas might fit this scheme a bit worse - for example, Superdense Coding kata follows the structure of superdense coding algorithm, so a tutorial on the end-to-end algorithm would work better than independent explanations of individual tasks.

If there is still help needed I could start working on the first two.

Nobody is working on those yet, help will be most definitely appreciated!

I would love to help too, I'll start working on Measurements!

Sounds great!
@vivanwin and @eloiseespel Please don't feel obliged to do the whole workbook at once (it's a lot of work!) - you can do the first couple of tasks and send them for review to get early feedback. Thank you again for volunteering!

Hey @tcNickolas I'd be happy to work on the single qubit gate workbook

@desireevl Sounds great! You can see Superposition workbook as a great example of the kind of explanations we're looking for (although BasicGates ones will be much simpler!) and formatting.

Hi @tcNickolas, I can have a look at the multi-qubit gates.

I am having issues with getting the exercises in SingleQubitGates.ipynb to run. I am using version:
Microsoft.Quantum.Standard::0.10.1911.1607
Microsoft.Quantum.Katas::0.10.1911.1607
I am able to run the demo however get the following error for Exercise 1:
%kata T1_ApplyY_Test

operation ApplyY (q : Qubit) : Unit is Adj+Ctl {
    Y(q);
}

/snippet_.qs(1,85): error QS3001: Syntax does not match any known patterns.
@tcNickolas would you have any insight as to what might be the issue?

Hi @desireevl,
I just tried to reproduce your error, for me this only happens when I have not run the first cell %package Microsoft.Quantum.Katas::0.10.1911.1607.
The issue is fixed when you run that cell and try to run your code again. Note that if you use the Binder  Notebook the kernel shuts down every ~10 minutes so you have to rerun that first cell again after the kernel is up again.
Hopefully this helps.

Hi,
I can pickup the Multi-quibt systems workbook.

@vivanwin Sounds great, thank you!

@desireevl Are you still working on the single qubit gate workbook? Please let us know if you ran into any issues.

@tcNickolas Hi yes, sorry for the delay, been a bit hectic with the situation at the moment. Will be finished shortly!

@desireevl I understand, no rush - just wanted to make sure you're not blocked on something I can help you with.

@tcNickolas I would like to pick up the Measurements (tutorials) if that is available.

@tcNickolas I would like to take up Joint Measurements if no one has started already.

@manav28 Sounds great, thank you!

If no one has started working on it, I would like to call dibs on Unitary Patterns.

@jainvasu631 Sounds great, thank you!

@jainvasu631 The problems in UnitaryPatterns kata were explained in the editorials for the past contest, for the warmup round and for the main contest - here are the LaTeX sources I have for them, in case you want to have a nice starting point for all the formulas and matrices. (They are saved as txt so as to allow uploading them to GitHub)
contest-editorial.txt
warmup-editorial.txt
Hope that helps!

@manav28 are you still interested in writing the workbook for joint measurements? If not, I am willing to work on it. Regardless, I have a short solution for task 5:
`%kata T05_DifferentBasis_Test
operation DifferentBasis (qs : Qubit[]) : Int {
// ...
CNOT(qs[1],qs[0]);
H(qs[1]);

mutable ans=0;

if(M(qs[1]) == One)
{
    set ans=1;
}

H(qs[1]);
CNOT(qs[1],qs[0]); // reset to original state

return ans;

}`

@ArthurConmy I am almost done with it. I just have task 6 pending. I believe task 5 has a clean one liner and I've written some of the math that goes into it
open Microsoft.Quantum.Measurement;

operation DifferentBasis (qs : Qubit[]) : Int {
    // ...
    return Measure([PauliX, PauliX], qs) == Zero ? 0 | 1;
}

@tcNickolas I noticed that Grover's Algorithm though not on this list also doesn't have a Workbook.
Although the problems in it are quite simple. So do you think it needs a Workbook. If yes then I wouldn't mind working on it.

@jainvasu631 I'm a bit on the fence about Grover's algorithm kata - on one hand, if there are tasks, it's good to have a workbook for them; but on the other hand, the tasks follow the algorithm exactly, there's less space for inventing the solution than in some of the other katas (that's why I didn't include this kata in the first wave of workbooks to be developed). I would say it's worth having if you're willing to create it. Thank you!
(Sorry for the delay replying on this and on the PRs, but I'm returning to the open source work now so should be a bit more responsive :-))

@desireevl Hi. Are you still working on the single qubit gate workbook? Please let me know - if you have it in some partial state but don't have time to finish it, you're welcome to send PR with this partial, and somebody else will pick it up to finish. (We have an internal Hackathon approaching so we'll have extra volunteers looking for something to do, and this workbook is a good candidate :-))

Hi @tcNickolas, I am so sorry about this. I won't be able to work on this and have unfortunately lost my progress  for what I have done

@desireevl No worries at all, I know how it can be. Thank you for letting me know!

If there are no applicants for GraphColoring yet, I'd like to work on it. Would that be okay?

@Lukious Absolutely, thank you!

@tcNickolas I wouldn't mind taking over Single Qubit Workbook from @desireevl . Its relatively simpler and I agree the Grover's algorithm Kata would be too directly related to the algorithm so maybe its not a priority right now.

@jainvasu631 Sounds good!

@ArthurConmy I finally finished review of JointMeasurements workbook and merged it to master. Looking at your proposed solution to task 5, it seems to follow quite a different approach, so I think there is value in adding it (together with the math behind it) after the existing solution, as an alternate solution (the way we've done in some Superposition tasks). Let me know if you'd like to do that!

@Lukious How far are you with the graph coloring workbook ? I can help if you want !

@tcNickolas Can I work on the graph coloring workbook then ?

@BrokenDuck Sure, sounds good :-)
(Apologies for the delay replying - I was trying to reach somebody who has expressed interest in the same some time ago, but with no success, so I'm assuming they're not interested any longer.)

@tcNickolas
I was recently going through the QFT Kata Notebook after Section III was added. I felt that while Section I is maybe tutorial-ish,
Section II and especially Section III need a Workbook. None currently exists. The list should be updated with the QFT as well.
PS: I notice that its almost been a year since this Issue was opened (in the last Hacktoberfest). We have almost completed it and it would be extremely cathartic if we could/would be to close it on October 28 (exactly one year after it was created). However, maybe we should expand the list to include new candidate Katas/Tutorial which are ripe and in need of a Workbook.

@jainvasu631
Agreed on both counts - I was hesitant to add new workbooks here, since this would delay the joy of completing the issue for everyone who worked so hard on the first batch of the workbooks :-)
I'm thinking we can file a new issue, "Create workbooks, batch 2", so that we can both complete this one and have more workbooks to work on.
Which reminds me... @BrokenDuck, how are you doing with the workbook for the graph coloring kata? Let us know if you need any help!

@tcNickolas I am working on it, but school has been quite ruthless (4h of French, Maths and Physics exam), so I am a bit late. I would greatly appreciate some help on part 1, I am more interested in part 2. Do I just commit the partially finished workbook, or are there extra steps ?

@BrokenDuck I think it's fine to send a pull request for a partially finished workbook - we've done that with Superposition, which took a join effort of 4 or 5 people and probably around 10 PRs.

@BrokenDuck  I would love to help out on GraphColoring's Workbook. If you can maybe send a draft PR or share whatever you have worked on, I would be glad to help out.

@BrokenDuck @jainvasu631 Here's the history of commits on Superposition workbook, as an example of what we did when different people work on different sections.

@tcNickolas @BrokenDuck
Perhaps we could commit Part 1 and Part 2 independently. I am not sure how much progress @BrokenDuck has made on Part 1 but I could build on that and finish it before end of October.
@BrokenDuck How does that sound?

I didn't really start part 1, so you could do that part and I can continue on part 2 with the oracle

@tcNickolas @BrokenDuck
I have written the first draft of the Workbook Part 1. The Part of the Workbook can then be merged into this. I have right now created an empty coming soon Solution cells for the tasks in Part 2. Later on these can be filled in.

@tcNickolas I finished the graph coloring workbook, I sent a push request.

With the GraphColoring workbook finished, I officially declare this issue complete! Thank you everybody who contributed to it!
I'll go through the remaining katas and put together a plan for the second batch of the workbooks soon.

authors: ArthurConmy BrokenDuck Lukious RankJay desireevl eloiseespel jainvasu631 manav28 tcNickolas vivanwin

repo: microsoft/QuantumKatas


text:
Fix the playground link under Recursive Type Aliases section

This fixes the playground link under the Recursive Type Aliases section in the Typescript 3.7 Release Notes

Duplicate. Removing

repo: microsoft/TypeScript-Handbook


text:
Translate 1 file to ko - Union and Intersection Types

en

Union and Intersection Types.ts

번역 리뷰 요청합니다: @yeonjuan @Bumkeyy
ref #6
감사합니다 😃

LGTM


text:
Update Enums.md: Fix wrong description

in this section
Union enums and enum member types
enum E {
  Foo,
  Bar,
}

function f(x: E) {
  if (x !== E.Foo || x !== E.Bar) {

In that example, we first checked whether x was not E.Foo. If that check succeeds, then our || will short-circuit, and the body of the ‘if’ will run. However, if the check didn’t succeed, then x can only be E.Bar, so it doesn’t make sense to see whether it’s equal to E.Bar.
when the first check didn’t succeed, then x is E.Foo, so it doesn’t make sense to see x !== E.Bar

Presumably, #2868 was in fact wrong?

repo: microsoft/TypeScript-Website