text:
Crash using landscape as acoustic geo (was: Compute node disks full?)

I've successfully baked a small test map and am now attempting to try something a little bit bigger.
I'm looking at a coarse bake, probe count 8799, 17599 tasks for the compute pool, estimated compute cost 13d 10h 56m. I set the bake to use 20 F8S V2 nodes; the quotas on my free starter account limited this to two actual nodes, so I anticipated this taking about a week and kicked it off Friday night. It seemed to be making progress when I left the office Friday.
This morning progress was stalled at 771 tasks complete, two nodes marked "unusable". I opened a quota increase request with Azure support and got 25 FSV2 vCPUs. Progress resumed more rapidly, but it now seems stuck again at 3089/17599 tasks. This time 8 nodes are marked unusable and no others shown.  Digging around a little I found that the nodes are reporting "not enough disk space on the node".
The bake appears to be unable to make progress due to the nodes being out of disk space, but that stall is not indicated in any way in the Unreal Editor UI, which says Status: Running: 3089/17599 tasks complete.
So I have a few questions here:

What's the relationship between nodes and vCPUs in Azure? How many nodes should be represented by my 25 vCPU quota? The pool overview says "Dedicated nodes: 8 -> 20"; what does that mean?
How much disk space, roughly should I expect to need per Acoustics bake task? I now see that F8s is the least-disky VM of those supported by Acoustics, at 64GB per vCPU; I was not expecting to generate many gigabytes of data in this process.
All of the nodes seem to be reporting having completed 385-386 tasks before reaching the unusable state, which seems oddly consistent. Is that expected? Is it expected that temp disk usage progressively increases to many gigabytes as tasks are completed?
Is there any way to recover this bake, or do I need to cancel and start over with more nodes with bigger temp disks?

First, the answer to your easy question :) Quota is defined in CPUs. The F8S V2 machines use 8 CPUs per node, so with a quota of 25 you could expect 3 nodes to be allocated. I'm actually surprised it gave you 8.
We're a little surprised that you ran out of disk space, as we've never hit this problem ourselves. That said, we've never done a bake as large as you're doing, so that could be why. We could use some more information about the space you are baking to help narrow down what the root of the problem is. We'd first like to check that the bake you are attempting is properly configured.

About how large of a space would you say you are baking? With proper configuration, we very rarely see bakes over 10000 probes.
Does your nav mesh properly limit the locations that you'd expect a listener to travel to?

As for recovering this bake, assuming that it is configured properly and you'd like to resume the bake, it should be safe to manually go into each node from Azure portal and delete everything that is saved locally to each node. The important information gets saved off to your Azure storage account, so there's no harm in removing the node-local storage. Let me know if you need more instructions here and I can provide more detailed steps.

The map covers about 20000x20000x2000 Unreal units (i.e. 200x200x20 meters). I didn't use nav mesh to limit the probes for this test; I'll do that and try again with more CPUs & storage and see how that goes.
If the node local storage isn't needed after a particular task is complete, shouldn't the job clean up as it goes?

Ah yes, a 200x200 meter region should have significantly less than 10000 probes. If you'd like, let us know how many probes you end up with after adding the nav mesh, but before submitting the bake to Azure, and we can confirm if it's within expected bounds.
The job should clean up local storage after a task is complete. The fact that it's not indicates something isn't working quite right. We'll look into this part.

Nav mesh isn't yet set up for this level. I tried to use Acoustics Volumes; as soon as I attempt to calculate probes, I get a crash:
UE4Editor-UnrealEd.dll!TMeshAttributeArraySet<FVector2D>::SetNumIndices(const int NumIndices) Line 378
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\MeshDescription\Public\MeshAttributeArray.h(378)
[Inline Frame] UE4Editor-UnrealEd.dll!TMeshAttributesRef<FVertexInstanceID,FVector2D>::SetNumIndices(const int) Line 533
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\MeshDescription\Public\MeshAttributeArray.h(533)
UE4Editor-UnrealEd.dll!GetBrushMesh(ABrush * Brush, UModel * Model, FMeshDescription & MeshDescription, TArray<FStaticMaterial,TSizedDefaultAllocator<32>> & OutMaterials) Line 702
	at C:\proj\phx\Dev-Main\Engine\Source\Editor\UnrealEd\Private\StaticMeshEdit.cpp(702)
UE4Editor-ProjectAcousticsBakeUI.dll!SAcousticsProbesTab::AddVolumeToAcousticMesh(AcousticMesh * acousticMesh, AAcousticsVolume * actor, TArray<unsigned int,TSizedDefaultAllocator<32>> & materialIDsNotFound) Line 744
	at C:\proj\phx\Dev-Main\Phoenix\Plugins\ProjectAcoustics\Source\ProjectAcousticsBakeUI\Private\AcousticsProbesTab.cpp(744)
UE4Editor-ProjectAcousticsBakeUI.dll!SAcousticsProbesTab::ComputePrebake() Line 854
	at C:\proj\phx\Dev-Main\Phoenix\Plugins\ProjectAcoustics\Source\ProjectAcousticsBakeUI\Private\AcousticsProbesTab.cpp(854)
UE4Editor-ProjectAcousticsBakeUI.dll!SAcousticsProbesTab::OnCalculateClearButton() Line 398
	at C:\proj\phx\Dev-Main\Phoenix\Plugins\ProjectAcoustics\Source\ProjectAcousticsBakeUI\Private\AcousticsProbesTab.cpp(398)
UE4Editor-ProjectAcousticsBakeUI.dll!TMemberFunctionCaller<SAcousticsProbesTab,FReply (__cdecl SAcousticsProbesTab::*)(void)>::operator()<>() Line 156
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstanceInterface.h(156)
UE4Editor-ProjectAcousticsBakeUI.dll!TBaseSPMethodDelegateInstance<0,SAcousticsProbesTab,0,FReply __cdecl(void)>::Execute() Line 293
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h(293)
[Inline Frame] UE4Editor-Slate.dll!TBaseDelegate<FReply>::Execute() Line 586
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Core\Public\Delegates\DelegateSignatureImpl.inl(586)
UE4Editor-Slate.dll!SButton::ExecuteOnClick() Line 379
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Widgets\Input\SButton.cpp(379)
UE4Editor-Slate.dll!SButton::OnMouseButtonUp(const FGeometry & MyGeometry, const FPointerEvent & MouseEvent) Line 303
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Widgets\Input\SButton.cpp(303)
[Inline Frame] UE4Editor-Slate.dll!FSlateApplication::RoutePointerUpEvent::__l8::<lambda_cc5aea9e7377b04ce85c30c65ec3889b>::operator()(const FArrangedWidget &) Line 5721
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(5721)
UE4Editor-Slate.dll!FEventRouter::Route<FReply,FEventRouter::FToLeafmostPolicy,FPointerEvent,<lambda_cc5aea9e7377b04ce85c30c65ec3889b>>(FSlateApplication * ThisApplication, FEventRouter::FToLeafmostPolicy RoutingPolicy, FPointerEvent EventCopy, const FSlateApplication::RoutePointerUpEvent::__l8::<lambda_cc5aea9e7377b04ce85c30c65ec3889b> & Lambda) Line 273
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(273)
UE4Editor-Slate.dll!FSlateApplication::RoutePointerUpEvent(const FWidgetPath & WidgetsUnderPointer, const FPointerEvent & PointerEvent) Line 5707
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(5707)
UE4Editor-Slate.dll!FSlateApplication::ProcessMouseButtonUpEvent(const FPointerEvent & MouseEvent) Line 6312
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(6312)
UE4Editor-Slate.dll!FSlateApplication::OnMouseUp(const EMouseButtons::Type Button, const FVector2D CursorPos) Line 6285
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(6285)
UE4Editor-ApplicationCore.dll!FWindowsApplication::ProcessDeferredMessage(const FDeferredWindowsMessage & DeferredMessage) Line 1880
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(1880)
UE4Editor-ApplicationCore.dll!FWindowsApplication::DeferMessage(TSharedPtr<FWindowsWindow,0> & NativeWindow, HWND__ * InHWnd, unsigned int InMessage, unsigned __int64 InWParam, __int64 InLParam, int MouseX, int MouseY, unsigned int RawInputFlags) Line 2332
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(2332)
UE4Editor-ApplicationCore.dll!FWindowsApplication::ProcessMessage(HWND__ * hwnd, unsigned int msg, unsigned __int64 wParam, __int64 lParam) Line 1560
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(1560)
UE4Editor-ApplicationCore.dll!FWindowsApplication::AppWndProc(HWND__ * hwnd, unsigned int msg, unsigned __int64 wParam, __int64 lParam) Line 790
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(790)
[External Code]
[Inline Frame] UE4Editor-ApplicationCore.dll!WinPumpMessages() Line 108
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsPlatformApplicationMisc.cpp(108)
UE4Editor-ApplicationCore.dll!FWindowsPlatformApplicationMisc::PumpMessages(bool bFromMainLoop) Line 130
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsPlatformApplicationMisc.cpp(130)
UE4Editor.exe!FEngineLoop::Tick() Line 4169
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp(4169)
[Inline Frame] UE4Editor.exe!EngineTick() Line 62
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\Launch.cpp(62)
UE4Editor.exe!GuardedMain(const wchar_t * CmdLine, HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, int nCmdShow) Line 173
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\Launch.cpp(173)
UE4Editor.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow) Line 265
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp(265)
[External Code]

this is null in SetNumIndices. The VertexInstanceUVs in the brush mesh appear to be null in GetBrushMesh.

Yikes! That's no good. Thanks for the report - we'll look into the crash.
A quick workaround would be to mark your entire ground/floor plane as both Acoustics Geometry and Acoustics Navigation, assuming your scene has some floor geometry that's easily distinguishable. Aside from that, setting up a rough Nav Mesh Bounds Volume (doesn't have to be perfect) would be the only other way to go.

After selectively tagging floors throughout the level, I get 672 probes, which seems more tractable.

Yes, that is much more inline with our expectations.

@borogove: Regarding the crash you saw with Acoustics Volumes - are you on UE 4.24? There have been some breaking changes in the 4.23->4.24 transition that we are working on accommodating and we have a similar known issue internally for UE 4.24, so just checking.

I'm on 4.23.

I also got a crash using a landscape mesh as acoustic geo, again with VertexInstanceUVs being null, this time in ALandscapeProxy::ExportToRawMesh called from SAcousticsProbesTab::AddLandscapeToAcousticMesh. LMK if you want more callstack than that.

Thanks for filing this bug. For our tracking purposes, I'll note that it's tracked internally with VSO bug # 24509723.

Retitling to match the issue that remains.

Nav mesh isn't yet set up for this level. I tried to use Acoustics Volumes; as soon as I attempt to calculate probes, with an inclusive Acoustics Volume in the world, I get a crash:
UE4Editor-UnrealEd.dll!TMeshAttributeArraySet<FVector2D>::SetNumIndices(const int NumIndices) Line 378
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\MeshDescription\Public\MeshAttributeArray.h(378)
[Inline Frame] UE4Editor-UnrealEd.dll!TMeshAttributesRef<FVertexInstanceID,FVector2D>::SetNumIndices(const int) Line 533
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\MeshDescription\Public\MeshAttributeArray.h(533)
UE4Editor-UnrealEd.dll!GetBrushMesh(ABrush * Brush, UModel * Model, FMeshDescription & MeshDescription, TArray<FStaticMaterial,TSizedDefaultAllocator<32>> & OutMaterials) Line 702
	at C:\proj\phx\Dev-Main\Engine\Source\Editor\UnrealEd\Private\StaticMeshEdit.cpp(702)
UE4Editor-ProjectAcousticsBakeUI.dll!SAcousticsProbesTab::AddVolumeToAcousticMesh(AcousticMesh * acousticMesh, AAcousticsVolume * actor, TArray<unsigned int,TSizedDefaultAllocator<32>> & materialIDsNotFound) Line 744
	at C:\proj\phx\Dev-Main\Phoenix\Plugins\ProjectAcoustics\Source\ProjectAcousticsBakeUI\Private\AcousticsProbesTab.cpp(744)
UE4Editor-ProjectAcousticsBakeUI.dll!SAcousticsProbesTab::ComputePrebake() Line 854
	at C:\proj\phx\Dev-Main\Phoenix\Plugins\ProjectAcoustics\Source\ProjectAcousticsBakeUI\Private\AcousticsProbesTab.cpp(854)
UE4Editor-ProjectAcousticsBakeUI.dll!SAcousticsProbesTab::OnCalculateClearButton() Line 398
	at C:\proj\phx\Dev-Main\Phoenix\Plugins\ProjectAcoustics\Source\ProjectAcousticsBakeUI\Private\AcousticsProbesTab.cpp(398)
UE4Editor-ProjectAcousticsBakeUI.dll!TMemberFunctionCaller<SAcousticsProbesTab,FReply (__cdecl SAcousticsProbesTab::*)(void)>::operator()<>() Line 156
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstanceInterface.h(156)
UE4Editor-ProjectAcousticsBakeUI.dll!TBaseSPMethodDelegateInstance<0,SAcousticsProbesTab,0,FReply __cdecl(void)>::Execute() Line 293
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h(293)
[Inline Frame] UE4Editor-Slate.dll!TBaseDelegate<FReply>::Execute() Line 586
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Core\Public\Delegates\DelegateSignatureImpl.inl(586)
UE4Editor-Slate.dll!SButton::ExecuteOnClick() Line 379
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Widgets\Input\SButton.cpp(379)
UE4Editor-Slate.dll!SButton::OnMouseButtonUp(const FGeometry & MyGeometry, const FPointerEvent & MouseEvent) Line 303
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Widgets\Input\SButton.cpp(303)
[Inline Frame] UE4Editor-Slate.dll!FSlateApplication::RoutePointerUpEvent::__l8::<lambda_cc5aea9e7377b04ce85c30c65ec3889b>::operator()(const FArrangedWidget &) Line 5721
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(5721)
UE4Editor-Slate.dll!FEventRouter::Route<FReply,FEventRouter::FToLeafmostPolicy,FPointerEvent,<lambda_cc5aea9e7377b04ce85c30c65ec3889b>>(FSlateApplication * ThisApplication, FEventRouter::FToLeafmostPolicy RoutingPolicy, FPointerEvent EventCopy, const FSlateApplication::RoutePointerUpEvent::__l8::<lambda_cc5aea9e7377b04ce85c30c65ec3889b> & Lambda) Line 273
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(273)
UE4Editor-Slate.dll!FSlateApplication::RoutePointerUpEvent(const FWidgetPath & WidgetsUnderPointer, const FPointerEvent & PointerEvent) Line 5707
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(5707)
UE4Editor-Slate.dll!FSlateApplication::ProcessMouseButtonUpEvent(const FPointerEvent & MouseEvent) Line 6312
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(6312)
UE4Editor-Slate.dll!FSlateApplication::OnMouseUp(const EMouseButtons::Type Button, const FVector2D CursorPos) Line 6285
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Slate\Private\Framework\Application\SlateApplication.cpp(6285)
UE4Editor-ApplicationCore.dll!FWindowsApplication::ProcessDeferredMessage(const FDeferredWindowsMessage & DeferredMessage) Line 1880
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(1880)
UE4Editor-ApplicationCore.dll!FWindowsApplication::DeferMessage(TSharedPtr<FWindowsWindow,0> & NativeWindow, HWND__ * InHWnd, unsigned int InMessage, unsigned __int64 InWParam, __int64 InLParam, int MouseX, int MouseY, unsigned int RawInputFlags) Line 2332
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(2332)
UE4Editor-ApplicationCore.dll!FWindowsApplication::ProcessMessage(HWND__ * hwnd, unsigned int msg, unsigned __int64 wParam, __int64 lParam) Line 1560
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(1560)
UE4Editor-ApplicationCore.dll!FWindowsApplication::AppWndProc(HWND__ * hwnd, unsigned int msg, unsigned __int64 wParam, __int64 lParam) Line 790
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsApplication.cpp(790)
[External Code]
[Inline Frame] UE4Editor-ApplicationCore.dll!WinPumpMessages() Line 108
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsPlatformApplicationMisc.cpp(108)
UE4Editor-ApplicationCore.dll!FWindowsPlatformApplicationMisc::PumpMessages(bool bFromMainLoop) Line 130
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\ApplicationCore\Private\Windows\WindowsPlatformApplicationMisc.cpp(130)
UE4Editor.exe!FEngineLoop::Tick() Line 4169
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp(4169)
[Inline Frame] UE4Editor.exe!EngineTick() Line 62
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\Launch.cpp(62)
UE4Editor.exe!GuardedMain(const wchar_t * CmdLine, HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, int nCmdShow) Line 173
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\Launch.cpp(173)
UE4Editor.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow) Line 265
	at C:\proj\phx\Dev-Main\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp(265)
[External Code]

this is null in SetNumIndices. The VertexInstanceUVs in the brush mesh appear to be null in GetBrushMesh.

@borogove This GetBrushMesh AV is a bug in the plugin when using UE4.22+. We have a fix internally but it's a one liner that you can easily apply in the plugin code -
In file:AcousticsProbesTab.cpp, function: SAcousticsProbesTab::AddVolumeToAcousticMesh, right after FMeshDescription Mesh; declaration add this line UStaticMesh::RegisterMeshAttributes(Mesh);.

Hope this helps!

authors: MikeChemi ashtat borogove kegodin nikunjragh

repo: microsoft/ProjectAcoustics


text:
Replication Report Not Showing on Main Report Listing when tbl_repl_msmerge_subscriptions table is empty - Remove Unneeded Dependency

Query to determine whether replication report displays on the main listing checks to see if any rows exist in tbl_repl_msmerge_subscriptions. The report itself doesn't have a dependency on this table and the records are likely to be blank. Remove this query from the main report listing.

@JamesFerebee can you open a separate branch for this issue from master and fix it there as this is not an accessibility change and also we may have to do remove the AccessibilityEnhancements branch.

from sql perf main.rdl, remove:
"
INSERT INTO @tbl_reports
(
ReportName,
ReportDisplayName,
ReportDescription,
VersionApplied,
Category,
ValidationObject,
SeqNo,
DataCollection,
Manditory,
CategorySeq
)
VALUES
('Replication_Topology_C', 'Replication Topology', 'Replication Topology in hierarchycal order', 1 | 2 | 4 | 8,
'Replication', 'dbo.tbl_repl_msmerge_subscriptions', 200, 'Replication', 1, 2000);
"

Resolved by PR #370

source: gharchive/issue

authors: JamesFerebee PiJoCoder

repo: microsoft/SqlNexus

license: MIT

license_type: permissive

license_source: github-api


text:
[BUG] LightGBM Bulk Execution mode failures

SynapseML version
0.10.2
System information

Language version (e.g. python 3.8, scala 2.12): python 3.8
Spark Version (e.g. 3.2.3): 3.2.1
Spark Platform (e.g. Synapse, Databricks): Databricks ML 10.4 LTS

Describe the problem
Training LightGBM Regressor in bulk execution mode extremely finnicky with spark configuration.
I mentioned this briefly in https://github.com/microsoft/SynapseML/issues/1761 but wanted to seek additional assistance.
I am training on Databricks runtime on a dataset with approximately 200MM data points and 30 features of DoubleType for simplicity.
If I operate on ~100M data points everything runs great in a reasonable amount of time, however it is the scaling up of the data/cluster that maybe makes default spark configs out of proportion.
I have had to expand spark.driver.maxResultSize to 64g to get around the first set of memory constraints, but quickly run into java Heap OOM issues the more I increase the dataset. I have tried up to r5d.16xlarge (64 cores 512 GB mem) as my worker time to accomodate these issues but still to no avail, additionally when monitoring executor usage in the SparkUI it does not appear I am utilizing nearly max memory even on smaller cluster size leading me to believe there is something in my configuration set up I could change.
I have tried tinkering with numBatches parameter but find that performance significantly declines when using this so have continued with useSingleDatasetMode = true.
This is the most common issue I get after I adjust the maxResultSize, java.lang.OutOfMemoryError: Java heap space. I have tried to increase executor memory and decrease the offHeap memory but this seems like could be headed down the wrong path.
Does lightgbm.fit really require that much more memory that I can't fit in huge executors I am using?
If so, how come I don't see these types of metrics in the Ganglia UI I look at?
Here is part of the common trace I receive.
java.lang.OutOfMemoryError: Java heap space
	at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3995)
	at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3801)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2204)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$unBlockifyObject$4(TorrentBroadcast.scala:432)
	at org.apache.spark.broadcast.TorrentBroadcast$$$Lambda$695/888027523.apply(Unknown Source)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1681)
	at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:434)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$4(TorrentBroadcast.scala:317)
	at org.apache.spark.broadcast.TorrentBroadcast$$Lambda$624/521720090.apply(Unknown Source)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$2(TorrentBroadcast.scala:291)
	at org.apache.spark.broadcast.TorrentBroadcast$$Lambda$622/1138522525.apply(Unknown Source)
	at org.apache.spark.util.KeyLock.withLock(KeyLock.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:286)
	at org.apache.spark.broadcast.TorrentBroadcast$$Lambda$621/1084360586.apply(Unknown Source)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1647)
	at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:286)
	at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:136)
	at org.apache.spark.broadcast.Broadcast.$anonfun$value$1(Broadcast.scala:80)
	at org.apache.spark.broadcast.Broadcast$$Lambda$620/1621883500.apply(Unknown Source)

	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:3029)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2976)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2970)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2970)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:2603)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3235)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3179)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:3167)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1152)
	at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2657)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1036)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:165)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:125)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:419)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:1034)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executePartitionTasks(LightGBMBase.scala:595)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executePartitionTasks$(LightGBMBase.scala:583)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.executePartitionTasks(LightGBMRegressor.scala:39)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executeTraining(LightGBMBase.scala:573)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.executeTraining$(LightGBMBase.scala:545)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.executeTraining(LightGBMRegressor.scala:39)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainOneDataBatch(LightGBMBase.scala:435)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainOneDataBatch$(LightGBMBase.scala:392)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.trainOneDataBatch(LightGBMRegressor.scala:39)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$2(LightGBMBase.scala:61)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:62)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:59)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.logVerb(LightGBMRegressor.scala:39)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain(BasicLogging.scala:48)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain$(BasicLogging.scala:47)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.logTrain(LightGBMRegressor.scala:39)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train(LightGBMBase.scala:42)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train$(LightGBMBase.scala:35)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.train(LightGBMRegressor.scala:39)
	at com.microsoft.azure.synapse.ml.lightgbm.LightGBMRegressor.train(LightGBMRegressor.scala:39)
	at org.apache.spark.ml.Predictor.fit(Predictor.scala:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:750)

I'd be more than happy to provide additional info if you could help point in the right direction. Thanks.
Code to reproduce issue
I cannot share the data, but I have listed above the sizes of the data I work with.
Other info / logs
No response
What component(s) does this bug affect?

[ ] area/cognitive: Cognitive project
[ ] area/core: Core project
[ ] area/deep-learning: DeepLearning project
[X] area/lightgbm: Lightgbm project
[ ] area/opencv: Opencv project
[ ] area/vw: VW project
[ ] area/website: Website
[ ] area/build: Project build system
[ ] area/notebooks: Samples under notebooks folder
[ ] area/docker: Docker usage
[ ] area/models: models related issue

What language(s) does this bug affect?

[ ] language/scala: Scala source code
[X] language/python: Pyspark APIs
[ ] language/r: R APIs
[ ] language/csharp: .NET APIs
[ ] language/new: Proposals for new client languages

What integration(s) does this bug affect?

[X] integrations/synapse: Azure Synapse integrations
[ ] integrations/azureml: Azure ML integrations
[X] integrations/databricks: Databricks integrations

@svotaw / @mhamilton723 Can you please help in looking at the lightgbm.fit out of memory issue?

yes, the current version of SynapseML LightGBM wrapper (bulk mode) suffers from excessive memory usage, and there's not much you can do about it. The fix is to use the newer "streaming" mode which is checked in, but it is not in 10.2. We should be cutting a new version soon (within a month, hopefully sooner). If you'd like to try to get a snapshot version working now, let me know and I'll send you a maven coordinate.

@svotaw that sounds good I would love a snapshot version. Will the snapshot version carry into the new release or could it expire before then?

Here's is the last build I saw from main.
SynapseML Build and Release Information
Maven Coordinates
com.microsoft.azure:synapseml_2.12:0.10.2-86-49d457f8-SNAPSHOT
Maven Resolver
https://mmlspark.azureedge.net/maven

The streaming mode has not been released yet, so it's not seen a lot of public testing. If you hit issues you can let me know. Use setExecutionMode("streaming")

Thanks @svotaw , any other helpful tips for success when using lightgbm with streaming execution mode? or do most of the parameters remain the same as in bulk execution mode?

Still doesn't seem to help memory footprint, but maybe my implementation is wrong.
I have about 80MM records now with 40 columns in the array (double).
I have 8 workers with 128 GB and 16 CPU cores each.
My driver is same spec.
	at java.io.ObjectInputStream$HandleTable.grow(ObjectInputStream.java:3995)
	at java.io.ObjectInputStream$HandleTable.assign(ObjectInputStream.java:3801)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2086)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:2119)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1657)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.broadcast.TorrentBroadcast$.$anonfun$unBlockifyObject$4(TorrentBroadcast.scala:432)
	at org.apache.spark.broadcast.TorrentBroadcast$$$Lambda$763/113317936.apply(Unknown Source)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1681)
	at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:434)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$4(TorrentBroadcast.scala:317)
	at org.apache.spark.broadcast.TorrentBroadcast$$Lambda$692/611180653.apply(Unknown Source)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$2(TorrentBroadcast.scala:291)
	at org.apache.spark.broadcast.TorrentBroadcast$$Lambda$678/1413736270.apply(Unknown Source)
	at org.apache.spark.util.KeyLock.withLock(KeyLock.scala:64)
	at org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:286)
	at org.apache.spark.broadcast.TorrentBroadcast$$Lambda$667/2000516935.apply(Unknown Source)
	at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1647)```

This is issue I run into where each executor in the SparkUI says:

266.9 MiB / 230 (record size/record)

----

Any suggestion?

Hmmm, can you try setting binSampleCount to lower (default is 200K)? This doesn't seem to be the whole error, but it involves broadcasted data, which we do with sample data. I am actually working on a fix for broadcasting sample data, but LightGBM only approved my PR last week, so it'll be a while before we can use it on the SynapseML side.
Also, is this a different memory error than you were seeing before?

Apologies for the delayed response, but yes that worked.
Is my intuition correct that given a size dataset of X increasing the number of partitions may hurt performance for this model?
Since the "batch" size becomes much smaller? i.e. should we try to minimize the number of partitions if all else equal?

You'd have to ask LightGBM developers. We are just a wrapper around it. But we (SynapseML) are wrapping their "distributed" mode, which can work in any distributed environment. We happen to implement it in Spark. But their algorithm requires communication between all nodes repeatedly during the training, so I guess their performance optimizations require this. We actually do a little trick in Spark where we initialize every partition and then literally turn networking over to the native library and then take back control when it is done. This is how we've made it work inside Spark, which is normally more of a fault tolerant environment.

Awesome. Regardless thank you, and congrats on getting streaming execution mode seems to be a major upgrade for the library.

@svotaw
Tried to bump up synapse to 0.11.
Same dataset size, same parameters (I did try to switch off and on again barrierExecution), same cluster spec, as I was using with bulk execution mode successfully and now the job fails.
From looking at the executor logs looks like some task begins to hang.
Any idea what might be the culprit? Happy to provide additional details.

No, I haven't made changes in months, so not sure what would be wrong. Did you try "streaming" mode?
Please provide more details.

Thanks for reporting. This is the first example of streaming not working and bulk working that I've seen, so hopefully we can use it to work a bug out. Are you doing a training? Or a fit?
I suspect it's related to the sample data broadcast we do currently, but would need to confirm. I'm waiting on a new build of LightGBM from someone, and then I can fix the sample data issue. We won't need to broadcast the data anymore. Hopefully within the next few weeks. As soon as I get the build, should only be a day or two to use the new features and check in.
It's working in bulk mode for you now right?

We have released 11.2, which finally has the last of the planned streaming features.

authors: KeerthiYandaOS andrejfd svotaw

repo: microsoft/SynapseML


text:
Ex 4 Tasks Incur Azure Subscription cost

The Microsoft Defender tasks within Exercise 4 instruct the attendee to modify the Microsoft Defender settings for their Azure Subscription. There should be a warning that this will incur cost on their Azure Subscription depending the pricing of the features that are being turned on.
Also, Ex 5 Clean up resources should instruct them to change the Microsoft Defender settings back, as this wont be affected by deleting the Resource Group.
There should be clear indication and rollback of the cost changes that enabling Microsoft Defender features will incur if left enabled after they are done with the lab.

addressed.

authors: crpietschmann tiagocostapt


text:
README

Hey,
In the README should it be this 👇
{
    "presets": [
        "@babel/preset-env",
        "@babel/preset-typescript"
    ],
...
}

rather than this 👇
{
    "presets": [
        "@babel/env",
        "@babel/typescript"
    ],
...
}

The docs at https://babeljs.io/docs/en/options#name-normalization indicate that @babel/env will be normalized to @babel/preset-env, so either will work in Babel's configuration file.
My personal preference is to use the full name @babel/preset-env, so if I search a project for any dependency in package.json then any references to that dependency will show up.

I prefer the abbreviated form. My policy is that if any are abbreviated then all are abbreviated.
More to the point, notice that the plugin name is also abbreviated in the README. (Sans "plugin-".)
Either way, I think this is a good place in the README to point out the equivalence.

This repo isn't really the place for us to teach those kind of things about babel's config resolving.
I don't have a strong opinion either way - if someone makes a PR and the entire thing is consistent then I'll merge it - but if not then we'll keep it as is 👍

authors: PaulieScanlon aplum joebowbeer orta

repo: microsoft/TypeScript-Babel-Starter


text:
Infinite redirect

https://www.typescriptlang.org/docs/home is a redirect to itself.
Perhaps while fixing this, we should start using https://www.gatsbyjs.com/plugins/gatsby-plugin-azure-static-web-app-redirects/ now that we’re on Static Web Apps.

I’m guessing this line: https://github.com/microsoft/TypeScript-Website/blob/b602340194ef7896ea7d1c355245973f3b87cb1b/packages/typescriptlang-org/src/redirects/setupRedirects.ts#L7
used to be necessary but isn’t under Static Web Apps. Later in the file, I left a comment about what else to change after adding the gatsby plugin.

repo: microsoft/TypeScript-Website


text:
Add ability to overload methods in objects

I think TypeScript must allow to define us method overloads in the objects too:
export const ProductManager = {
  async find(id: string): Promise<Product>
  async find(ids: string[]): Promise<Product[]>
  async find(idOrIds: string | string[]): Promise<Product | Product[]> {
  }
}

Such method overloads helps to define really complex cases when it's not possible to properly define everything in a single method signature.

Is there already discussion around this feature and reason why it's declined? I don't suggest to introduce new language constructs, what I suggest is only related to type system.

Sorry, meant to leave a comment here (thought I had actually). There just isn't syntactic room in object literal syntax to allow for this in a way that would be guaranteed to not conflict with either current meanings or future ES features. This is already reasonable achievable today by annotating ProductManager with a type annotation.

That's sad. Imagine some big class like this one, if I'll write a type annotation in there, it just feels... feels like I'm writing a .d.ts code instead of writing a natural code...
We all know it's a tricky question when it comes to bring something new and predict that this feature won't appear in future ES versions, but we do need to bring new features... and since suggested syntax looks same as in class version of what we already have, and we already have an assumption that in class version it won't appear (since we already have implementation in place), plus assumption that same signatures of the same object property (find in this case) unlikely appear in future ES versions makes me think that it's safe to add this feature to the language.
It's what I think, of course you know better where I'm wrong and what to do.

This seems like a pretty reasonable workaround:
interface FooFunc {
    (n:number): number;
    (s:string): string;
    (sn: string | number): string | number;
}
interface Owner {
    foo: FooFunc;
}
const o: Owner = {
    foo: (sn: any) => {
        return sn;
    }
};

// Result is strongly typed
o.foo(1).toFixed(3);
o.foo("two").charAt(2);

The last overload in MyFunc is optional because you're not actually typing the method itself when you assign the object to the interface-typed const, if I understand correctly.

You can always write a separate type definition, it just doesn't feel like its the most convenient and natural way to solve your problem.

I feel the same need, especially for big objects.
In the case of objects with dozens of functions like front-end stores for example and
if we need to overload just one function, we must implement the whole interface just for that on function.
it seems unnecessarily cumbersome and repetitive.
Is there any solution for this case ?
Like partially implement the interface without loosing the types of the others functions ?

repo: microsoft/TypeScript


text:
List @deprecated strikethroughs in Problems tab

The new deprecated strikethroughs in TS code are great. But to make it easier to find and address a series of them, it would be nice to have them also appear in the Problems tab (as an optional view, controlled by a settings flag).

Moving upstream for more feedback
I believe we can already generate these diagnostics on the VS Code side if we want, but perhaps TS should have a compiler flag the enables treating usage of deprecated symbols as an error/warning

Do we already have wraning? If not I'm happy to support that both ts & code side

After thinking about this again, I'm pretty sure that the solution I'd like to see is

VS Code showing suggestions in the problems tab, as suggestions.
A compiler CLI flag to emit all suggestions as errors. (Or similar to errors, with different formatting.)

For those looking for a solution, I found this comment in one of the referenced MR.
https://github.com/microsoft/TypeScript/pull/40607#issuecomment-1369051173
--
However I wish that the deprecated warnings/infos are displayed in the problems tab out of the box

For those looking for a solution, I found this comment in one of the referenced MR.
#40607 (comment)
--
However I wish that the deprecated warnings/infos are displayed in the problems tab out of the box

It's only partial, it just catches the use of a deprecated field (deconstruction) const a = x.deprecated; but does not catch things such as assignations const a: X = { deprecated: 123 }; or destructuring assignment const f: X = ({ deprecated }) => 123; or const { deprecated } = a;, or some more complex cases such as const param = { deprecated: 123 }; callModule(param);. Listing deprecations is quite complex and must target a lot of specific cases. But it's so important that I think it should be handled by TS as warnings during type check and in editor's problems.

+1

authors: Kingwl devi-prsd ericgoe jer-sen mjbvz morganwdavis sandersn


text:
Add insertSpaceAfterOpeningAndBeforeClosingEmptyBraces to tsserver protocol

Looks like #35427 added a new formatter setting: insertSpaceAfterOpeningAndBeforeClosingEmptyBraces
I see this setting in code but it is not exposed in the tsserver protocol at the moment. Should it be added there as well?

@DanielRosenwasser What do you think about this issue?

authors: a-tarasyuk mjbvz


text:
ts v4.5.4 ,the lib.dom  navigator not have  hid.

lib Update Request
ts v4.5.4 ,the lib.dom  navigator not have  hid.  can give me some suggestion ？
Documentation Link
https://wicg.github.io/webhid/

To expedite the triage process, we need everyone to follow the issue template and instructions.
When you clicked "Create New Issue", the issue form was pre-populated with a template and some instructions. We need you to read those instructions completely, follow them, and then fill in all the fields in that template.
We are not able to assist with issues that don't follow the template instructions as they represent a significantly larger amount of work compared to issues which are correctly specified. Thank you for understanding.


text:
"Implicit any" when accessing a remapped key of a mapped type (but no error when accessing the whole mapped type)

Bug Report
🔎 Search Terms
implicitly any, key remapping, filter object properties by type
⏯ Playground Link

Playground link with relevant code
💻 Code

interface Descriptor<X> {
  type: X;
}

// Conditional type required as a workaround when filtering properties by type
// https://github.com/microsoft/TypeScript/issues/38646
type ComposedProperties<T> = T extends any ? {
  [K in keyof T as T[K] extends Descriptor<any> ? K : never]:
    T[K] extends Descriptor<infer X> ? X : never;
} : never;

interface Context<T> {
  props: ComposedProperties<T>;
}

type Composer<T> = {
  <F extends (ctx: Context<T>) => any>(fn: F) : Descriptor<ReturnType<F>>;
}

class Composable {
  compose: Composer<this> = undefined!; // Omitted..
}

class Foo extends Composable {
  a = this.compose(() => 'A');
  b = this.compose(() => 2);
  c = 'c';
}

class Bar extends Foo {
  working = this.compose(ctx => ctx.props)  // <-- IF THIS WORKS...

  notWorking = this.compose(ctx => ctx.props.a) // <-- WHY THIS DOESN'T ??
}

🙁 Actual behavior
'notWorking' implicitly has type 'any' because it does not have a type annotation and is referenced directly or indirectly in its own initializer.(7022)
🙂 Expected behavior
No error

The working initializer works only because it doesn’t reach in far enough to force the constraint of this to be resolved. If you do working = this.compose(ctx => ctx.props).type.a, you hit the same circularity issue.

authors: andrewbranch turolla


text:
Typescript enum suggestion Chinese garbled code

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.88.0
OS Version: Darwin x64 22.6.0

版本: 1.88.0
提交: 5c3e652f63e798a5ac2f31ffd0d863669328dc4c
日期: 2024-04-03T13:28:18.899Z
Electron: 28.2.8
ElectronBuildId: 27744544
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Darwin x64 22.6.0
Steps to Reproduce:

Looks like this comes from the returned displayParts. This is the decoded json object:
  {
                "text": "\"\\u82F9\\u679C\"",
                "kind": "stringLiteral"
            }

Looks like this comes from the returned displayParts. This is the decoded json object:
  {
                "text": "\"\\u82F9\\u679C\"",
                "kind": "stringLiteral"
            }

Is this the intended effect? or a bug

authors: iizyd mjbvz


text:
after wsl --update, can't mount C: windows drives in WSL2

Windows Version
Microsoft Windows [Version 10.0.19045.4529]
WSL Version
2.2.4.0
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
Linux version 5.15.153.1-microsoft-standard-WSL2 (root@941d701f84f1) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Fri Mar 29 23:14:13 UTC 2024
Distro Version
Debian
Other Software
No response
Repro Steps
opening a shell provides
<3>WSL (226) ERROR: CreateProcessParseCommon:711: Failed to translate C:\Users\Tiago <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\WINDOWS\system32 <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\WINDOWS <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\WINDOWS\System32\Wbem <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\WINDOWS\System32\WindowsPowerShell\v1.0\ <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\WINDOWS\System32\OpenSSH\ <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\ProgramData\chocolatey\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\ <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files\dotnet\ <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files (x86)\dotnet\ <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files\Microsoft VS Code\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files\Portugal Identity Card\ <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Users\Tiago\AppData\Local\Microsoft\WindowsApps <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files\JetBrains\PyCharm Community Edition 2019.2.5\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Program Files\JetBrains\PyCharm 2019.2.5\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Users\Tiago\OneDrive\Documents\WindowsPowerShell\Scripts <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Users\Tiago\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Users\Tiago\AppData\Local\GitHubDesktop\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Users\Tiago\AppData\Local\Programs\Lens\resources\cli\bin <3>WSL (226) ERROR: UtilTranslatePathList:2866: Failed to translate C:\Users\Tiago\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-6.0-full_build\bin Processing fstab with mount -a failed. Failed to mount C:\, see dmesg for more details. Failed to mount P:\, see dmesg for more details.
trying to mount provides an error:
root@DESKTOP-FFKGGLD:/home/cusco# mount -t drvfs P: /mnt/ <3>WSL (346) ERROR: UtilCreateProcessAndWait:665: /bin/mount failed with 2 <3>WSL (345) ERROR: UtilCreateProcessAndWait:687: /bin/mount failed with status 0xff00 root@DESKTOP-FFKGGLD:/home/cusco# mount -t drvfs C: /mnt/c <3>WSL (349) ERROR: UtilCreateProcessAndWait:665: /bin/mount failed with 2 <3>WSL (348) ERROR: UtilCreateProcessAndWait:687: /bin/mount failed with status 0xff00
Expected Behavior
normal mounted drives, no errors translating paths from windows
Actual Behavior
dmesg contains
`[Fri Jul 19 13:47:08 2024] scsi 0:0:0:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 5
[Fri Jul 19 13:47:08 2024] sd 0:0:0:0: Attached scsi generic sg0 type 0
[Fri Jul 19 13:47:08 2024] sd 0:0:0:0: [sda] 795856 512-byte logical blocks: (407 MB/389 MiB)
[Fri Jul 19 13:47:08 2024] sd 0:0:0:0: [sda] Write Protect is on
[Fri Jul 19 13:47:08 2024] sd 0:0:0:0: [sda] Mode Sense: 0f 00 80 00
[Fri Jul 19 13:47:08 2024] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[Fri Jul 19 13:47:08 2024] sd 0:0:0:0: [sda] Attached SCSI disk
[Fri Jul 19 13:47:08 2024] EXT4-fs (sda): mounted filesystem without journal. Opts: (null). Quota mode: none.
[Fri Jul 19 13:47:08 2024] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Fri Jul 19 13:47:08 2024] hv_pci 6e11ff5c-e492-4c64-bb40-0c3fb213662d: PCI VMBus probing: Using version 0x10003
[Fri Jul 19 13:47:08 2024] hv_pci 6e11ff5c-e492-4c64-bb40-0c3fb213662d: PCI host bridge to bus e492:00
[Fri Jul 19 13:47:08 2024] pci_bus e492:00: root bus resource [mem 0xc00000000-0xe00001fff window]
[Fri Jul 19 13:47:08 2024] pci_bus e492:00: No busn resource found for root bus, will use [bus 00-ff]
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: [1af4:105a] type 00 class 0x088000
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: reg 0x10: [mem 0xe00000000-0xe00000fff 64bit]
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: reg 0x18: [mem 0xe00001000-0xe00001fff 64bit]
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: reg 0x20: [mem 0xc00000000-0xdffffffff 64bit]
[Fri Jul 19 13:47:08 2024] pci_bus e492:00: busn_res: [bus 00-ff] end is updated to 00
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: BAR 4: assigned [mem 0xc00000000-0xdffffffff 64bit]
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: BAR 0: assigned [mem 0xe00000000-0xe00000fff 64bit]
[Fri Jul 19 13:47:08 2024] pci e492:00:00.0: BAR 2: assigned [mem 0xe00001000-0xe00001fff 64bit]
[Fri Jul 19 13:47:08 2024] virtiofs virtio1: Cache len: 0x200000000 @ 0xc00000000
[Fri Jul 19 13:47:08 2024] memmap_init_zone_device initialised 2097152 pages in 20ms
[Fri Jul 19 13:47:08 2024] FS-Cache: Duplicate cookie detected
[Fri Jul 19 13:47:08 2024] FS-Cache: O-cookie c=00000004 [p=00000002 fl=222 nc=0 na=1]
[Fri Jul 19 13:47:08 2024] FS-Cache: O-cookie d=00000000358687d2{9P.session} n=00000000dd256b37
[Fri Jul 19 13:47:08 2024] FS-Cache: O-key=[10] '34323934393337333534'
[Fri Jul 19 13:47:08 2024] FS-Cache: N-cookie c=00000005 [p=00000002 fl=2 nc=0 na=1]
[Fri Jul 19 13:47:08 2024] FS-Cache: N-cookie d=00000000358687d2{9P.session} n=000000005aab0707
[Fri Jul 19 13:47:08 2024] FS-Cache: N-key=[10] '34323934393337333534'
[Fri Jul 19 13:47:08 2024] FS-Cache: Duplicate cookie detected
[Fri Jul 19 13:47:08 2024] FS-Cache: O-cookie c=00000006 [p=00000002 fl=222 nc=0 na=1]
[Fri Jul 19 13:47:08 2024] FS-Cache: O-cookie d=00000000358687d2{9P.session} n=00000000ad00d342
[Fri Jul 19 13:47:08 2024] FS-Cache: O-key=[10] '34323934393337333535'
[Fri Jul 19 13:47:08 2024] FS-Cache: N-cookie c=00000007 [p=00000002 fl=2 nc=0 na=1]
[Fri Jul 19 13:47:08 2024] FS-Cache: N-cookie d=00000000358687d2{9P.session} n=000000006a8354bd
[Fri Jul 19 13:47:08 2024] FS-Cache: N-key=[10] '34323934393337333535'
[Fri Jul 19 13:47:08 2024] scsi 0:0:0:1: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 5
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: Attached scsi generic sg1 type 0
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: [sdb] 536870912 512-byte logical blocks: (275 GB/256 GiB)
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: [sdb] 4096-byte physical blocks
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: [sdb] Write Protect is off
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: [sdb] Mode Sense: 0f 00 00 00
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[Fri Jul 19 13:47:08 2024] sd 0:0:0:1: [sdb] Attached SCSI disk
[Fri Jul 19 13:47:08 2024] EXT4-fs (sdb): mounted filesystem with ordered data mode. Opts: discard,errors=remount-ro,data=ordered. Quota mode: none.
[Fri Jul 19 13:47:08 2024] FS-Cache: Duplicate cookie detected
[Fri Jul 19 13:47:08 2024] FS-Cache: O-cookie c=0000000b [p=00000002 fl=222 nc=0 na=1]
[Fri Jul 19 13:47:08 2024] FS-Cache: O-cookie d=00000000358687d2{9P.session} n=00000000b29d1f54
[Fri Jul 19 13:47:08 2024] FS-Cache: O-key=[10] '34323934393337343031'
[Fri Jul 19 13:47:08 2024] FS-Cache: N-cookie c=0000000c [p=00000002 fl=2 nc=0 na=1]
[Fri Jul 19 13:47:08 2024] FS-Cache: N-cookie d=00000000358687d2{9P.session} n=000000002d8a6634
[Fri Jul 19 13:47:08 2024] FS-Cache: N-key=[10] '34323934393337343031'
[Fri Jul 19 13:47:08 2024] WSL (2) ERROR: UtilCreateProcessAndWait:665: /bin/mount failed with 2
[Fri Jul 19 13:47:08 2024] WSL (1) ERROR: UtilCreateProcessAndWait:687: /bin/mount failed with status 0xff00
[Fri Jul 19 13:47:08 2024] WSL (1) ERROR: ConfigMountFsTab:2589: Processing fstab with mount -a failed.
[Fri Jul 19 13:47:08 2024] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2e206d05276, max_idle_ns: 440795217445 ns
[Fri Jul 19 13:47:08 2024] clocksource: Switched to clocksource tsc
[Fri Jul 19 13:47:09 2024] WSL (1) WARNING: /etc/resolv.conf updating disabled in /etc/wsl.conf
[Fri Jul 19 13:47:09 2024] WSL (4) ERROR: UtilCreateProcessAndWait:665: /bin/mount failed with 2
[Fri Jul 19 13:47:09 2024] WSL (1) ERROR: UtilCreateProcessAndWait:687: /bin/mount failed with status 0xff00
[Fri Jul 19 13:47:09 2024] WSL (5) ERROR: UtilCreateProcessAndWait:665: /bin/mount failed with 2
[Fri Jul 19 13:47:09 2024] WSL (1) ERROR: UtilCreateProcessAndWait:687: /bin/mount failed with status 0xff00
[Fri Jul 19 13:47:09 2024] WSL (1) WARNING: /etc/hosts updating disabled in /etc/wsl.conf
[Fri Jul 19 13:47:09 2024] WSL (1) WARNING: /etc/resolv.conf updating disabled in /etc/wsl.conf
[Fri Jul 19 13:47:11 2024] misc dxg: dxgk: dxgkio_is_feature_enabled: Ioctl failed: -22
[Fri Jul 19 13:47:11 2024] misc dxg: dxgk: dxgkio_query_adapter_info: Ioctl failed: -22
[Fri Jul 19 13:47:11 2024] misc dxg: dxgk: dxgkio_query_adapter_info: Ioctl failed: -22
[Fri Jul 19 13:47:11 2024] misc dxg: dxgk: dxgkio_query_adapter_info: Ioctl failed: -22
[Fri Jul 19 13:47:11 2024] misc dxg: dxgk: dxgkio_query_adapter_info: Ioctl failed: -2`
Diagnostic Logs
https://ovh.tretas.eu/~cusco/WslLogs-2024-07-19_13-58-16.zip

Thank you for reporting this @cusco. Can you share the content of /etc/wsl.conf ?

Within wsl Debian? Sure. I’m not at that computer right now but should be able to reply back in the next 2 or 3
Days

etc/wsl.conf

[network]
generateResolvConf = false
generateHosts = false

[interop]
enabled = true
appendWindowsPath = true

[automount]
enabled = true
options = "metadata,umask=22,fmask=11"
mountFsTab = true

[wsl2]
virtio9p=false

I was reading stuff online and changed several options within that file... the only think I had, was the [network] stanza

Thank you. Can you also share the content of /etc/fstab ?

It’s empty.
I used to get fstab failed to mount, because it was empty.
But always had /mnt/c and /mnt/p mounted …
On Fri, 26 Jul 2024 at 20:09, Blue @.***> wrote:

Thank you. Can you also share the content of /etc/fstab ?
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/WSL/issues/11812#issuecomment-2253319904,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAW3L73DE3M5GXVBC4K2KE3ZOKNEZAVCNFSM6AAAAABLERZIIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJTGMYTSOJQGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

authors: OneBlue cusco

repo: microsoft/WSL


text:
Accessing Linux files through \wsl$ is unbearably slow

Your Windows build number:  (Type ver at a Windows Command Prompt)

Microsoft Windows [Version 10.0.18362.175]

What you're doing and what's happening:  (Copy&paste the full set of specific command-line steps necessary to reproduce the behavior, and their output. Include screen shots if that helps demonstrate the problem.)

Accessing Linux files via \\wsl$\Ubuntu, and it's super slow.
I did some basic testing with dd, and found the following:

Write in //wsl$/Ubuntu...
dd if=/dev/zero of=//wsl$/Ubuntu/home/glen/testwrite.img bs=100M count=1 oflag=dsync

104857600 bytes (105 MB, 100 MiB) copied, 18.186 s, 5.8 MB/s
104857600 bytes (105 MB, 100 MiB) copied, 17.8413 s, 5.9 MB/s

MCAFEE REMOVED:

104857600 bytes (105 MB, 100 MiB) copied, 16.1272 s, 6.5 MB/s
104857600 bytes (105 MB, 100 MiB) copied, 15.9704 s, 6.6 MB/s

dd if=/dev/zero of=//wsl$/Ubuntu/home/glen/testlatency.img bs=512 count=1000 oflag=dsync

512000 bytes (512 kB, 500 KiB) copied, 3.53898 s, 145 kB/s
512000 bytes (512 kB, 500 KiB) copied, 0.335095 s, 1.5 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.330495 s, 1.5 MB/s

Read in //wsl$/Ubuntu...
time dd if=//wsl$/Ubuntu/home/glen/testwrite.img of=/dev/null bs=8k

104857600 bytes (105 MB, 100 MiB) copied, 14.9655 s, 7.0 MB/s

real    0m15.010s
user    0m0.078s
sys     0m0.265s

104857600 bytes (105 MB, 100 MiB) copied, 4.20551 s, 24.9 MB/s

real    0m4.251s
user    0m0.031s
sys     0m0.124s
MCAFEE REMOVED

104857600 bytes (105 MB, 100 MiB) copied, 3.71398 s, 28.2 MB/s

real    0m3.781s
user    0m0.046s
sys     0m0.125s

104857600 bytes (105 MB, 100 MiB) copied, 3.64852 s, 28.7 MB/s

real    0m3.733s
user    0m0.030s
sys     0m0.140s

Write in ~...
dd if=/dev/zero of=/home/glen/testwrite.img bs=100M count=1 oflag=dsync

104857600 bytes (105 MB, 100 MiB) copied, 0.390321 s, 269 MB/s
104857600 bytes (105 MB, 100 MiB) copied, 0.371199 s, 282 MB/s

dd if=/dev/zero of=/home/glen/testlatency.img bs=512 count=1000 oflag=dsync

512000 bytes (512 kB, 500 KiB) copied, 0.0086376 s, 59.3 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.0088582 s, 57.8 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.0082154 s, 62.3 MB/s

Read in ~...
time dd if=/home/glen/testwrite.img of=/dev/null bs=8k

104857600 bytes (105 MB, 100 MiB) copied, 0.113269 s, 926 MB/s

real    0m0.128s
user    0m0.016s
sys     0m0.094s

104857600 bytes (105 MB, 100 MiB) copied, 0.115235 s, 910 MB/s

real    0m0.132s
user    0m0.031s
sys     0m0.109s

Write in /mnt/c/Users/Glen... (from distro)
dd if=/dev/zero of=/mnt/c/Users/Glen/testwrite.img bs=100M count=1 oflag=dsync

104857600 bytes (105 MB, 100 MiB) copied, 0.384083 s, 273 MB/s
104857600 bytes (105 MB, 100 MiB) copied, 0.252227 s, 416 MB/s

dd if=/dev/zero of=/mnt/c/Users/Glen/testlatency.img bs=512 count=1000 oflag=dsync

512000 bytes (512 kB, 500 KiB) copied, 0.0099631 s, 51.4 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.0116009 s, 44.1 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.0100696 s, 50.8 MB/s

Read in /mnt/c/Users/Glen... (from distro)
time dd if=/mnt/c/Users/Glen/testwrite.img of=/dev/null bs=8k

104857600 bytes (105 MB, 100 MiB) copied, 0.11045 s, 949 MB/s

real    0m0.127s
user    0m0.000s
sys     0m0.125s

104857600 bytes (105 MB, 100 MiB) copied, 0.109702 s, 956 MB/s

real    0m0.122s
user    0m0.000s
sys     0m0.125s

Write in /c/Users/Glen... (from Windows)
dd if=/dev/zero of=/c/Users/Glen/testwrite.img bs=100M count=1 oflag=dsync

104857600 bytes (105 MB, 100 MiB) copied, 0.351462 s, 298 MB/s
104857600 bytes (105 MB, 100 MiB) copied, 0.359245 s, 292 MB/s

dd if=/dev/zero of=/c/Users/Glen/testlatency.img bs=512 count=1000 oflag=dsync

512000 bytes (512 kB, 500 KiB) copied, 0.112353 s, 4.6 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.110369 s, 4.6 MB/s
512000 bytes (512 kB, 500 KiB) copied, 0.113078 s, 4.5 MB/s

Read in /c/Users/Glen... (from Windows)
time dd if=/c/Users/Glen/testwrite.img of=/dev/null bs=8k

104857600 bytes (105 MB, 100 MiB) copied, 0.0635338 s, 1.7 GB/s

real    0m0.105s
user    0m0.000s
sys     0m0.061s

104857600 bytes (105 MB, 100 MiB) copied, 0.0628403 s, 1.7 GB/s

real    0m0.092s
user    0m0.015s
sys     0m0.046s

TL;DR:
\\wsl$\Ubuntu

Writing: ~6 MB/s     !!!
Reading: ~28 MB/s    !!!

/home/glen

Writing: ~275 MB/s
Reading: ~915 MB/s

/mnt/c/Users/Glen

Writing: ~300 MB/s
Reading: ~950 MB/s

/c/Users/Glen (from Windows)

Writing: ~295 MB/s
Reading: ~1.7 GB/s

😞

What's wrong / what should be happening instead:

It needs to be a lot faster in order to be useable.

Strace of the failing command, if applicable:  (If some_command is failing, then run strace -o some_command.strace -f some_command some_args, and link the contents of some_command.strace in a gist here).

n/a

For WSL launch issues, please collect detailed logs.

n/a

#4197

@therealkenc That issue is about WSL2, and has nothing to do with \\wsl$.

One discussion thread for 9p perf, both directions, is plenty.

@therealkenc Doesn't WSL2 have a completely different architecture (using a lightweight VM)? Are you suggesting that accessing \\wsl$ in WSL1 uses exactly the same set of technologies that are used when accessing /mnt/c in WSL2?

@glen-84 - Correct, \wsl$ in WSL1 and WSL2 both use 9p.  In WSL2 9p is also used for Windows drive access (/mnt/c etc).

Okay, I apologize for the duplicate.

This worked for me. for my WSL2 distro
wsl --set-version distro 1
it will take a few minutes
go back to version 2
wsl --set-version distro 2


text:
Allow to use other drives for distro installation

Describe the solution you'd like
Currently, when we install from Microsoft store, we use C:\ by default, however, we have the ability to select whatever drive we can use if we install manually (sacrificing having to do first time user setup however). I think we should have the distro launchers support selecting a host drive instead of relying on just C:,  this is useful for multi-drive laptops where the SSD is only used as a system drive, and are expected to use the hard drive.
Describe alternatives you've considered
wsl --import or wsl --install works fine but it lacks the first-time user install experience.

Packages/Programs from Windows Store runs in a jail/container/sandbox like environment. If Windows Store allows to install in another drive it would be a sweet spot for you-know-who. You can grab the .tar.gz file from Windows Store installed folder and use it to install in another drive.

Packages/Programs from Windows Store runs in a jail/container/sandbox like environment. If Windows Store allows to install in another drive it would be a sweet spot for you-know-who. You can grab the .tar.gz file from Windows Store installed folder and use it to install in another drive.

We already can migrate apps over other drives. Plus DistroLauncher is just an application to unpack or launch the distro installed. I don't see in a security standpoint it would be insecure since we already have the feature set for it.

You can change MS Store install drive in Win10 Settings > System > Storage for Apps or migrate already installed apps to another drive.
If you feel that MS Store is rightfully the worst one there is, there's a feedback button for it.
I don't think this belongs here at all however. You can install your own distro into WSL without relying on MS Store so WSL itself is working as intended.

You can change MS Store install drive in Win10 Settings > System > Storage for Apps or migrate already installed apps to another drive.
If you feel that MS Store is rightfully the worst one there is, there's a feedback button for it.
I don't think this belongs here at all however. You can install your own distro into WSL without relying on MS Store so WSL itself is working as intended.

I don't think this is the desirable response here, while I agree to allowing feedback, I think the WSL experience could be far more improved if the installation path of WSL followed on where the DistroLauncher apps for the respective distros are. So far, DistroLauncher distros are stuck only in the C:\ Drive - that's something we can work on in the long run since we already have the feature for it - we're just not readily using it. This concerns WSL as a whole as well since DistroLauncher is actually a part of WSL - it's primary purpose is to launch the supplied distro. As a user who might be just getting into the Linux ecosystem via WSL, you might want to move your stuff as easy as clicking it over at the settings, we want to have that as well for DistroLauncher distros.

I think this would be a good idea - it allows some flexibility. I do not necessarily disagree with Biswa96, but to me I think this is more a usability question than a security question. You can always provide the information given here, use with caution etc... while allowing it still. (A few years ago, or perhaps ... 2 years ago I think, with WSL, I used to reason that the whole customization/installation should be very flexible, and also inform the user what has happened, in regards to distribution-installations. I am hugely forgetful and literally with all the multitasking I do in general, being flexible is great, while also keeping track of what has happened in files and GUIs etc... - so I think this would be a usability suggestion more than a security-related problem set really. That being said, I am mostly really just neutral on this, I am fine either way, just giving my opinion. :-) )


text:
WSL2 keeps nagging me to install matlab every time I install other tools.

Windows Build Number
10.0.19042.1052
WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version
5.4.72
Distro Version
Ubuntu 20.04
Other Software
No response
Repro Steps
install any tools in the wsl2 system
Expected Behavior
install the valgrind successfully
Actual Behavior
proped out the window to nag me installing matlab

Diagnostic Logs
No response

Is there a way the installed matlab? Or if I want to install anything I will need to install matlab? That totally make no sense to me.

This isn't part of WSL, this is part of your Debian or Ubuntu distribution. dpkg -l *matlab* to figure out which package this is (seems to be possibly matlab-support), and remove it.

Will it told me that I haven't installed matlab yet.
So I tried to install the matlab-support and recieved the same error message which nags me to figure out where I had installed the matlab! But I was just trying to installed matlab-support....... This totally does't make sense!

So should I ask the question in the ubuntu community? I'm afraid it's not appropriate to ask this qustion here?

Will it told me that I haven't installed matlab yet.
So I tried to install the matlab-support and recieved the same error message which nags me to figure out where I had installed the matlab! But I was just trying to installed matlab-support....... This totally does't make sense!

Maybe you should use bash to execute the command. zsh messes with *.

So should I ask the question in the ubuntu community? I'm afraid it's not appropriate to ask this qustion here?

Yes.

@BillRenCN agreed this seems to be something specific to your installation and isn't an actionable item for us here on the WSL team. As others have identified in the thread asking this to the Ubuntu community seems appropriate. I'll close this issue out as it's not a WSL specific issue!

authors: BillRenCN Diablo-D3 NaiveTomcat craigloewen-msft


text:
Cannot execute docker-credential-wincred.exe from within WSL2 filesystem after switching to MS Store

Version
Microsoft Windows [Version 10.0.22000.258]
WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version
5.10.60.1
Distro Version
Ubuntu 20.04
Other Software
https://github.com/docker/docker-credential-helpers/releases/download/v0.6.4/docker-credential-wincred-v0.6.4-amd64.zip
Repro Steps
$ curl -fsSL https://github.com/docker/docker-credential-helpers/releases/download/v0.6.4/docker-credential-wincred-v0.6.4-amd64.zip | zcat | sudo tee /usr/local/bin/docker-credential-wincred.exe

$ sudo chmod +x /usr/local/bin/docker-credential-wincred.exe

$ /usr/local/bin/docker-credential-wincred.exe
/usr/local/bin/docker-credential-wincred.exe: Invalid argument

Expected Behavior
Previously, before I switched to the MS Store version of WSL, it worked fine. Also, if I move the binary to the Windows filesystem, it also works:
$ /mnt/c/docker-credential-wincred.exe
Usage: docker-credential-wincred.exe <store|get|erase|list|version>

Actual Behavior
$ /usr/local/bin/docker-credential-wincred.exe
/usr/local/bin/docker-credential-wincred.exe: Invalid argument

Diagnostic Logs
wsl.etl.zip
I renamed from .etl to .etl.zip. Please rename it back to .etl before opening.

I suspect docker-credential-wincred.exe is not the only with this problem, but it was the only one I found.


text:
WSL --status has no kernal information

Version
Windows 11 22H2 22621.963
WSL Version

[X] WSL 2
[ ] WSL 1

Kernel Version
no information
Distro Version
Debian
Repro Steps
When i exec: wsl.exe --status, The result only displays:
Default Distribution: Debian
Default Version: 2

??? where are the linux kernal information?
Thanks

wsl.exe --version

authors: benhillis jack-liew


text:
Devcon utility OneCore Compliant?

Devcon.exe works only on windows desktop. Any plans to port this to Universal Windows app? Make it OneCore compliant? Should work on windows 10 IOT Core.
Found a way to port SetupAPI to make it OneCore compliant.
https://docs.microsoft.com/en-us/windows-hardware/drivers/install/porting-from-setupapi-to-cfgmgr32

Devcon was meant as a sample to demonstrate how to use various SetupApi APIs.  There is no plan to make it OneCore compliant.  Most of its functionality is supported in the inbox utility pnputil.exe.  Please see https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/devcon-migration

authors: anupdw jasonknichel

repo: microsoft/Windows-driver-samples

license: MS-PL


text:
Add new flag and exclusion for new InfVerif changes

Temporary change to fix InfVerif build errors when using the build script with EEAP WDK and samples in development.

I agree with this change for today.
Feature creep (feel free to ignore): I'd prefer if we can have an EOL at end of exclusion.csv .

source: gharchive/pull-request

authors: JakobL-MSFT NeoAdonis


text:
Latest VisualStudio update as of yesterday breaks solutions which use WindowsAppSDK  1.5.X

Describe the bug
Please see repo below which uses WindowsAppSDK 1.5.X. This no longer builds but throws this error
error CS0433: The type 'TextDecorations' exists in both 'Microsoft.WinUI, Version=3.0.0.0, Culture=neutral, PublicKeyToken=de31ebe4ad15742b' and 'Microsoft.Windows.SDK.NET, Version=10.0.19041.38, Culture=neutral, PublicKeyToken=31bf3856ad364e35'

Minimal.zip
Steps to reproduce the bug
See above
Expected behavior
WindowsAppSDK 1.5.X solutions should build using latest VisualStudio updates.
Screenshots

NuGet package version
None
Packaging type
No response
Windows version
No response
IDE
No response
Additional context
No response

FWIW, we are facing the same issue with our main project.

Hello, I have the same problem

Same for me. But I am getting other errors:

<PackageReference Include="Microsoft.WindowsAppSDK" Version="1.6.240923002" />

// csproj
<TargetFramework>net8.0-windows10.0.22621.0</TargetFramework>
<WindowsSdkPackageVersion>10.0.22621.45</WindowsSdkPackageVersion>
<TargetPlatformMinVersion>10.0.19041.0</TargetPlatformMinVersion>

// The error originates from an obj folder
obj\x64\Debug\net8.0-windows10.0.22621.0\win-x64\intermediatexaml\WinRT.SourceGenerator\Generator.WinRTAotSourceGenerator\WinRTGenericInstantiation.g.cs(3534,72,3534,88): error CS0117: 'HttpProgress' does not contain a definition for 'CreateMarshaler2'
obj\x64\Debug\net8.0-windows10.0.22621.0\win-x64\intermediatexaml\WinRT.SourceGenerator\Generator.WinRTAotSourceGenerator\WinRTGenericInstantiation.g.cs(3535,238,3535,252): error CS1503: Argument 1: cannot convert from 'ABI.Windows.Web.Http.HttpProgress' to 'ABI.Windows.Web.Http.HttpProgress.Marshaler'
obj\x64\Debug\net8.0-windows10.0.22621.0\win-x64\intermediatexaml\WinRT.SourceGenerator\Generator.WinRTAotSourceGenerator\WinRTGenericInstantiation.g.cs(3540,72,3540,86): error CS1503: Argument 1: cannot convert from 'ABI.Windows.Web.Http.HttpProgress' to 'ABI.Windows.Web.Http.HttpProgress.Marshaler'

Everything works fine under visual studio 17.11.4.

Same for me. But I am getting other errors:

<PackageReference Include="Microsoft.WindowsAppSDK" Version="1.6.240923002" />

Everything works fine under visual studio 17.11.4.

@persskog you're on WindowsAppSDK 1.6.X, so your issue is not related to my original report - which is about WindowsAppSDK  1.5.X. I recommend you opened a new issue to not confuse matters. Thanks

The recent Windows SDK projection update did add some Windows namespace types to address another issue.  Most of those overlapping types were duplicate types between the Microsoft and Windows namespaces and the Windows namespace ones were unused in WinUI scenarios.  But there were a couple that are used like TextDecorations which when explicitly used will run into this issue.  We have addressed this in WinAppSDK 1.6, which was built with the new Windows SDK projection update, by declaring those types as forwarded to the Windows SDK projection.
If you are able to move to WinAppSDK 1.6, that should address this issue.  Otherwise, if you want to stay on 1.5, you can use the older Windows SDK projection version (.34) by specifying the below property in your csproj or in a common props file which should keep things working.  Once you are able to move to 1.6, you should remove the property.
<PropertyGroup>
    <WindowsSdkPackageVersion>10.0.19041.34</WindowsSdkPackageVersion>
</PropertyGroup>

Same for me. But I am getting other errors:

<PackageReference Include="Microsoft.WindowsAppSDK" Version="1.6.240923002" />

// csproj
<TargetFramework>net8.0-windows10.0.22621.0</TargetFramework>
<WindowsSdkPackageVersion>10.0.22621.45</WindowsSdkPackageVersion>
<TargetPlatformMinVersion>10.0.19041.0</TargetPlatformMinVersion>

// The error originates from an obj folder
obj\x64\Debug\net8.0-windows10.0.22621.0\win-x64\intermediatexaml\WinRT.SourceGenerator\Generator.WinRTAotSourceGenerator\WinRTGenericInstantiation.g.cs(3534,72,3534,88): error CS0117: 'HttpProgress' does not contain a definition for 'CreateMarshaler2'
obj\x64\Debug\net8.0-windows10.0.22621.0\win-x64\intermediatexaml\WinRT.SourceGenerator\Generator.WinRTAotSourceGenerator\WinRTGenericInstantiation.g.cs(3535,238,3535,252): error CS1503: Argument 1: cannot convert from 'ABI.Windows.Web.Http.HttpProgress' to 'ABI.Windows.Web.Http.HttpProgress.Marshaler'
obj\x64\Debug\net8.0-windows10.0.22621.0\win-x64\intermediatexaml\WinRT.SourceGenerator\Generator.WinRTAotSourceGenerator\WinRTGenericInstantiation.g.cs(3540,72,3540,86): error CS1503: Argument 1: cannot convert from 'ABI.Windows.Web.Http.HttpProgress' to 'ABI.Windows.Web.Http.HttpProgress.Marshaler'

Everything works fine under visual studio 17.11.4.

This has been addressed in the latest Windows SDK projection update which should be available in next month's .NET SDK release.  If you want to try it out earlier, you can specify the WindowsSdkPackageVersion property to the appropriate version based on TFM listed here.

Moved to 10.0.xxxxx.53 and now it works, thanks!

error CS0433: The type 'TextDecorations' exists in both 'Microsoft.WinUI, Version=3.0.0.0, Culture=neutral, PublicKeyToken=de31ebe4ad15742b' and 'Microsoft.Windows.SDK.NET, Version=10.0.19041.38, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
@manodasanW, the above issue occurs in latest VS2022 stable version[17.11.5] and VS2022 preview version[17.12.0 Preview 5.0]. Currently we could not able to update WindowsAppSdk version to latest version.
Is there any specific timeline to fix this issue at your end?

@SubashiniMahendran If you can't yet update to the latest WinAppSDK version, the recommendation is to lock your Windows SDK projection version to the version before those types were added to it.  You just need to add something like the below to your csproj or a common props file assuming you are targeting 19041 in the target framework.  When you move to WinAppSDK 1.6, you can remove it.
<PropertyGroup>
    <WindowsSdkPackageVersion>10.0.19041.34</WindowsSdkPackageVersion>
</PropertyGroup>

Close this issue as solution is provided.

authors: DierkDroth FredoZeLegend LegendaryBlair LucBurlet SubashiniMahendran manodasanW persskog

repo: microsoft/WindowsAppSDK


text:
Controlling the printing output of agents

What happened?
How can we control the output of the code executor? Can we let the executor print only some lines of the output of the code generated? The terminal is crashing when the output of the code is extremely large (~1000 lines or so)
   @staticmethod
    def create_user_proxy_agent() -> autogen.UserProxyAgent:
        x= autogen.UserProxyAgent(
            name="user_proxy",
            human_input_mode="TERMINATE",
            max_consecutive_auto_reply=1,
            code_execution_config={
                "executor": LocalCommandLineCodeExecutor(work_dir="coding"),
            },
        )

What did you expect to happen?
Control code printing on the screen.
How can we reproduce it (as minimally and precisely as possible)?
https://github.com/microsoft/autogen/issues/3896#issue-2606804632
AutoGen version
0.2
Which package was this bug in
AgentChat
Model used
No response
Python version
No response
Operating system
No response
Any additional info you think would be helpful for fixing this bug
No response

Hello,
I don't want a complete blank output, I want it to be controlled upto 1000 lines.

repo: microsoft/autogen


text:
Remove system message functionality from code executors

Why are these changes needed?
@ekzhu as discussed this removes all system message functionality. We agreed that this should be solved a different way, starting with documenting how to create good system messages. Doing this now before code executors become too mature.
Related issue number

Checks

[ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[ ] I've made sure all auto checks have passed.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 20.42%. Comparing base (0a49f2a) to head (c132365).

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1835       +/-   ##
===========================================
- Coverage   35.69%   20.42%   -15.28%
===========================================
  Files          63       63
  Lines        6625     6595       -30
  Branches     1454     1576      +122
===========================================
- Hits         2365     1347     -1018
- Misses       4067     5081     +1014
+ Partials      193      167       -26

Flag
Coverage Δ

unittests
20.42% <ø> (-15.28%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

More context: We are still thinking about a proper approach to solve the compatibility contract that binds the code executor and code writer agents together. Before we have a good design, it was pre-mature to use system message instruction update. Thus, the proposal to remove it. Note: this feature has not been documented anywhere or recommended to use since released. Since many have reviewed it, need to get feedback on whether removing it is going to cause massive downstream rewrites.
Alternatively, we can keep the feature there, but insert explicit warning on possible removal/changes.

@rickyloynd-microsoft @IANTHEREAL since you were in the discussion earlier regarding the user capability, what's your comment on this? Jack and mine rational is that we are not sure if capability is the right abstraction to perform this system message update, so we want to avoid over-engineer this. The current capability only performs a system message update, which is brittle.

Yes, I agree that introducing a UserCapability mechanism (similar to the AgentCapability mechanism) would be overkill for simply updating a system message. Maybe there would be a stronger case for such a mechanism in the future, but I'm a fan of waiting until a feature is actually needed before doing much work to implement it.

@rickyloynd-microsoft @IANTHEREAL since you were in the discussion earlier regarding the user capability, what's your comment on this? Jack and mine rational is that we are not sure if capability is the right abstraction to perform this system message update, so we want to avoid over-engineer this. The current capability only performs a system message update, which is brittle.

I have no objections to first removing it. Moreover, I believe conducting more tests on placing the code executor guide in various structured prompts is essential. Such experiments would yield valuable insights for optimizing our approach.

Thanks @rickyloynd-microsoft and @IANTHEREAL. I made the PR ready for review.

Just one question - Will the code executor work well after this PR merged, or will it affect our developers?

Could you make the PR from the upstream to run the openai test?

See PR: #1933

authors: IANTHEREAL codecov-commenter ekzhu jackgerrits rickyloynd-microsoft sonichi


text:
Updated the visualization and description of Composio - Ecosystem component

Why are these changes needed?
The updated visualisation contains a better attempt to show how Composio can empower platforms like Autogen, to interact with hundreds of external tools.
Related issue number

Checks

[ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[ ] I've made sure all auto checks have passed.

Please feel free to reopen if you'd like to address the comments

authors: jackgerrits sawradip


text:
update the news for v0.2 release

Why are these changes needed?
Make the news more complete.
Related issue number

Checks

[ ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
[ ] I've added tests (if relevant) corresponding to the changes introduced in this PR.
[ ] I've made sure all auto checks have passed.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6258031) 27.66% compared to head (f04c8bb) 27.46%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #844      +/-   ##
==========================================
- Coverage   27.66%   27.46%   -0.20%
==========================================
  Files          27       27
  Lines        3517     3517
  Branches      801      801
==========================================
- Hits          973      966       -7
- Misses       2473     2479       +6
- Partials       71       72       +1

Flag
Coverage Δ

unittests
27.46% <ø> (-0.15%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter sonichi


text:
Selenium sample should clarify applicability to teams using other browser automation test frameworks

We got a report from an internal customer we sent the sample to that, based on the existing selenium-based sample, they had added a Selenium dependency to get a11y testing in place even though they already had Puppeteer-based end to end tests in place.
Suggested resolution:
Update the Selenium sample readme to clarify that Even if a team is not using Selenium and are using a different browser automation framework, many of them already have specific open source libraries for integrating axe scans (puppeteer has axe-puppeteer, webdriverio has axe-webdriverio, cypress has cypress-axe). Most of the concepts from the Selenium sample would be the same, but they would want to use the axe integration library most relevant to the framework they are using.

since this is closed marking it as resolved

authors: adarshrema ferBonnin

repo: microsoft/axe-pipelines-samples


text:
not work for azure devops server

test on:
azure devops server 2019
azure devops server 2019.1
not work

this is by design. Azure devops server does not have the right contribution points for this extension to work. probably looking at 2020

authors: danhellem niyw