text:
Enable the blob storage state system in the runtime

In addition to the CosmosDB state driver, we should also have blob storage.
Can we reuse the same blob storage configuration that is currently being provided for the transcript logger? This would be ideal!
Blob storage is much faster to provision. If we can enable this in the runtime, PERHAPS we can default to using this instead of CosmosDB? This merits discussion!

I raised the same subject with @darrenj - I think his thoughts on this, from experience with the VA, is that Cosmos will avoid some common issues as customers need to scale out / up.  I do see some benefits to enabling blob storage, given the reduced complexity for setup / configuration from a customer perspective.

I'm conflicted about this having seen too many customers ship with defaults - memorystorage being the worst example! My leaning is towards setting people up for success with an optional "step back" for development/testing because of this. Blob Storage is by no means as bad as memorystorage but I've yet to see a customer go live on Blob Storage (happy to be corrected).
I think we should take this change, but be very clear in the UX (@emivers8 ?) that this is for development/testing purposes only and make it super easy to switch up afterwards?

Just to add some colour here - most of the bots I have seen go live have used blob storage.  I think the difference here is likely the size / scale of the bots being deployed and the type of bot.  Blob is likely to be fine for a relatively low load bot.
I think I am leaning towards @darrenj's thinking on balance.  At least for the initial release we could go with just Cosmos and add blob based on feedback.  We could also ensure we have discoverable docs to educate folks how to swap out cosmos for blob if they wanted to? Although, thinking about it, that should already be covered in the BF docs anyway.

Putting aside our default provisioning configuration --
I think having the blob storage state driver loaded and ready to go, enabled only if configured, is something we should include.
The goal is that a developer can choose the right option through configuration only without having to worry about runtime code.
If we default them to blob storage, they can add cosmos later and vice versa because the code knows how to handle both. We can "step" them back or forward however we want.
So back to whether we provision cosmos by default or not ---
My main concern is that the cosmos provision part takes the longest -- it seems to adds minutes to the wait. If we feel strongly that Cosmos is the right place to configure most bots automatically, then we can let it continue to be the default.
We are already also provisioning blob storage for transcripts.

I updated this ticket to more clearly indicate that changing the default is just a potential side effect of this

Yeah to weigh in on this...if I'm making a bot for my wife's choir, I can't afford CosmosDB. It's way too expensive.  Functions+TableStorage is great for light loads.
Having the option is good and having information about the reason for one over the other is super good.
Something like:

TableStorage - Inexpensive storage suitable for light to medium chat loads in one data center.
CosmosDB - Scalable storage suitable for light to heavy work loads which can span multiple data centers.
MemoryStorage - Storage used for testing on a single machine only.

Taking an R10 item to visit this.

old runtime deprecated.

authors: benbrown cwhitten darrenj garypretty luhan2017 tomlm


text:
Renaming Dialogs

Describe the bug
The Framework Composer seems to have issues building the project if you rename the top level dialog. This problem also exists for renaming child dialogs, but doing so only disrupts the UX if it is accessed during the conversation flow. I used the CoreWithLanguage template, made a few child dialogs, then attempted to change the name of CoreWithLanguage to MetricIntent. It did not build. When I then changed it back to CoreWithLanguage to revert the bug, the original name also failed to build. The words in the error message flip flop, suggesting there is something sluggish happening behind the scenes. Closing and reopening the project did not fix things.
Version
Version: 2.1.1
Electron: 8.2.4
Chrome: 80.0.3987.165
NodeJS: 12.13.0
V8: 8.0.426.27-electron.0
OS
Windows 10
To Reproduce
Steps to reproduce the behavior:

Create a CoreWithLanguage bot template (or any template I guess)
Edit the name of the dialog in the properties pane
Build the project
See error. Build failed.

Expected behavior
I expected the Framework Composer to acknowledge the name change and propagate it automatically through the rest of the bot code and the LUIS backend.
Additional context
Output trace (personal information censored):
Error occurred building the bot
Error: Resource MetricIntent.dialog not found. at ConfigurationResourceExporer.loadType (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:195:19) at DialogExpressionConverter.convert (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-adaptive\lib\converters\dialogExpressionConverter.js:34:55) at C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\defaultLoader.js:37:48 at Array.reduce () at DefaultLoader.load (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\defaultLoader.js:31:39) at ConfigurationResourceExporer.load (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:240:23) at Array. (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:214:25) at JSON.parse () at ConfigurationResourceExporer.loadType (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:203:36) at new CoreBot (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-adaptive-runtime\lib\coreBot.js:22:45) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! CoreWithLanguage@ dev: cross-env NODE_ENV=dev node index.js "--port" "3981" "--urls" "http://0.0.0.0:3981" "--luis:endpointKey" "********************************" "--SkillHostEndpoint" "http://127.0.0.1:3981/api/skills" npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the CoreWithLanguage@ dev script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\*****\AppData\Roaming\npm-cache_logs\2021-10-11T18_38_36_577Z-debug.log

Hi @TylerDKnight Any updates or further information you can share to help us reproduce or understand what might be going on?

source: gharchive/issue

authors: TylerDKnight dmvtech

repo: microsoft/BotFramework-Composer

license: MIT

license_type: permissive

license_source: github-api


text:
File or attachment user input action in Composer v2.1.1 not working

I am trying to create an attachment input action in bot framework composer and followed these steps
 but I am getting  the error message - "Object reference not set to an instance of an object" on emulator vesrion 4.14.0. Please find below the screen shot of the implementation and error message:

My Bot framework Composer version details:
Version:  2.1.1
Electron: 8.2.4
Chrome: 80.0.3987.165
NodeJS: 12.13.0
V8: 8.0.426.27-electron.0

Hey @anujprasad, is your intent to attach a file? It looks like your getting the error as the input from the card is a text rather than an attachment file. Run your bot and click on the upload file (paper clip icon) button on bottom left of the chat to attach a file. Please let me know if this helps.

@ramfattah  Yes, my intent is to attach a pdf file. The input text "number" that you see in the screen shot is just a regular expression trigger to trigger the number dialog and the very first step on that dialog is to ask user for file attachment. It throws the error "Object reference not set to an instance of an object" even before prompting the user the message "Please upload your invoice". Ideally user should first see this prompt "Please upload your invoice" and then he/she would click on upload file (paper clip icon) button to attach a file, it should not throw the exception before that.
I have shared the composer authoring screen for "Prompt for a file and attachment" , do you find any anomaly in that implementation that could throw the exception - "Object reference not set to an instance of an object"?

Re-attaching the composer bot authoring screen:

I'm facing the same issue.

In my case the input prompt did not have any text, this is why it errored. Ui should reflect it being necessary or should be possible to not contain any value.

I figured out the root cause and solution, incase anyone facing similar issue:
Exact steps to reproduce the issue:

Select the + icon then select File or attachment from the Ask a question menu
Enter Please send an image. in the Prompt field of the Properties pane.
Switch to the User Input tab ()

Enter dialog.attachments in the Property field.
Enter all in the Output format field.
Enter "application/pdf" in the contentType. Note that contentType field is marked as mandatory

Now run your bot and invoke the dialog that has prompt for file or an attachment - your get the error message - "Object reference not set to an instance of an object"

The root cause for the issue was contentType which is marked as mandatory field with red *, now even if I remove the text "application/pdf" from contentType field I continue to get the error message - "Object reference not set to an instance of an object". So the fix is to remove the control "prompt for file or an attachment" first and then re-add it and make sure you have not added any text in the mandatory field - "contentType " :).

Quick question - Once the file is uploaded by user using the prompt "File or attachment input", does that file stays in the server forever or it's stored in memory within dialog scope? If I have to hand it off to any live agent such as customer helpdesk, what's the best possible way to store the transcript and this file and then send it over an email to the live agent?

Hi @anujprasad , Microsoft Bot Framework is an open platform that allows developers to integrate with a variety of agent engagement platforms. This is the official Microsoft Documentation on Handoff integration models to provide a common language and best practices for bot developers and system integrators building conversational AI systems with human in the loop.
Also, Here is the Bot Framework v4 Handoff Library.

Hi @anujprasad , Microsoft Bot Framework is an open platform that allows developers to integrate with a variety of agent engagement platforms. This is the official Microsoft Documentation on Handoff integration models to provide a common language and best practices for bot developers and system integrators building conversational AI systems with human in the loop.
Also, Here is the Bot Framework v4 Handoff Library.

can i get upload file url so i can store some where in api and the through api i will dowload it

if user send me pdf attachment in bot composer.
How will I get to know file is attach or not?
so i can store it some where and then download from it for example (azure blob storage).

if user send me pdf attachment in bot composer. How I will store that pdf in my code. How it will convert to base 64 or in URL to send via API to store somewhere

My issue is resolved and it uploading in my blob

how to used validation for file ie. file should be in .pdf format only in bot composer

if user send me pdf attachment in bot composer. How I will store that pdf in my code. How it will convert to base 64 or in URL to send via API to store somewhere

My issue is resolved and it uploading in my blob storage and getting the URL too. NOTE : Storing locally only
When I try to run through server its show error Below Error : An attempt was made to access a socket in a way forbidden by its access permissions. An attempt was made to access a socket in a way forbidden by its access permissions.

Hi Hadkar, Can you please explain the process you followed so that whatever the user uploads, you get the URL in blob storage? Right now, after the user uploads any pdf i get ContentURL which is having localhost link. After copy-pasting that localhost link to any website its downloading the file without the extension. Can you also say how did you solve this?
Any help is much appreciated.

See in my case I get the whole name i.e - filename.pdf etc
In your case exactly what link are you getting?
and my issue is still pending

How to store this file to azure blob storage or sharepoint list ?

I'm also searching the possibility of saving attachments to azure storage. My bot as channel is using Teams. What I done up to now - I used "prompt for a file or attachment" and get content url of the file. Im looking for the next steps. From web I found that file should be sent via http request to azure blob. Is it a proper way, should I use this content url? Is it possible to done form Bot Framework Composer or extra coding is necessary? I appreciate all advices.

authors: ParthHadkar1203 RaviKasaudhan anujprasad chinubansal2014 drmanhatin ramfattah surlicki


text:
feat: Added implementation for ResourceConfigurationStep

Description
Added implementation & validation for resource configuration step.
1. Added 'Create new' option for searchable dropdown.
2. Validation for new resource group
3. Persist the configuration to local  storage.
Task Item
refs #7632
refs #7631
Screenshots

Coverage remained the same at 50.044% when pulling 0c3be02d2529832344831bb57a1a0a0bdb0d5236 on vmodem/resource-configuration-step into 4774893bbad77fec2a2fe81f21b05918199e8a72 on main.

source: gharchive/pull-request

authors: VamsiModem coveralls


text:
Test reliability: basic.js "Absolute timestamp" test is timing out

Feature Request
We are seeing the basic.js "absolute timestamp" test is causing a timeout.
FAIL __tests__/basic.js (174.676s)
  ● absolute timestamp

    : Timeout - Async callback was not invoked within the 60000ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 60000ms timeout specified by jest.setTimeout.Error:

      146 | });
      147 |
    > 148 | test('absolute timestamp', async () => {
          | ^
      149 |   const activities = [
      150 |     {
      151 |       type: 'message',

      at new Spec (node_modules/jest-jasmine2/build/jasmine/Spec.js:116:22)
      at Object.<anonymous> (__tests__/basic.js:148:1)

  ● absolute timestamp

    TimeoutError: Waiting Direct Line to connect
    Wait timed out after 15209ms

      at node_modules/selenium-webdriver/lib/webdriver.js:841:17

  ● absolute timestamp

    TimeoutError: Waiting Direct Line to connect
    Wait timed out after 15209ms

      at node_modules/selenium-webdriver/lib/webdriver.js:841:17

  ● avatar background color

    TimeoutError: Waiting Direct Line to connect
    Wait timed out after 15209ms

      at node_modules/selenium-webdriver/lib/webdriver.js:841:17

  ● avatar background color

    TimeoutError: Waiting Direct Line to connect
    Wait timed out after 15209ms

      at node_modules/selenium-webdriver/lib/webdriver.js:841:17

[Enhancement]

Fake negative.

repo: microsoft/BotFramework-WebChat


text:
Unable to add new package 'libdrm'

I'm trying to extend CBL-Mariner to include libdrm in the installation.
For a baseline, I am using 1.0-stable, and without any modifications to the repository, I am able to successfully build an ISO image, and install/run this OS using the following command sequence :
Get the source code
git clone https://github.com/microsoft/CBL-Mariner.git cd CBL-Mariner/toolkit
Checkout the desired release branch. The 1.0-stable tag tracks the most recent successful `release of the 1.0 branch.
git checkout 1.0-stable`
Build the Go tools
sudo make go-tools REBUILD_TOOLS=y
Get the package sources
sudo make input-srpms DOWNLOAD_SRPMS=y
Build an ISO version of ./imageconfigs/core-efi.json entirely from downloaded, pre-built packages
sudo make iso REBUILD_TOOLS=y REBUILD_PACKAGES=n
Now I would like to build a new ISO that allows me to install libdrm, so I've attempted to follow instructions for adding a SPEC file CBL-Mariner/SPECS/libdrm/libdrm.spec. It is unclear to me what additional steps are necessary to build Mariner so that libdrm is supported. I am referencing libdrm packages located at https://dri.freedesktop.org/libdrm/. I have attempted a variety of commands from the documentation in order to build this additional package, but have not been successful.
Any suggestions on how to proceed would be appreciated.

Hi @rradjabi! Could you share the commands and errors you're getting when building the ISO with the new package?
Most instructions should be available here but I see that this section could use some expansion - thank you for pointing this out.

Hi @rradjabi! Glad you were able to build an image on 1.0-stable- always love to hear that our tooling works well for those outside the Mariner team 😄
It sounds like you're trying to add a package to an ISO image- it's a bit hard to suggest how to proceed without knowing what exactly you've read/tried, but (in addition to Pawel's suggestion) I would suggest reading the following:

Documentation on the image generation process
Documentation on the image configuration file format

@PawelWMS , @thcrain-msft ,
Happy to share an example of what I've done and the errors that returned. I've tried many computations, so I wasn't sure which to share, but here is one example.
`

git clone https://github.com/microsoft/CBL-Mariner.git
cd CBL-Mariner/toolkit
git checkout 1.0-stable
sudo make go-tools REBUILD_TOOLS=y
sudo make input-srpms DOWNLOAD_SRPMS=y
sudo make iso REBUILD_TOOLS=y REBUILD_PACKAGES=n
`

Now I want to rebuild and add the SPEC file for libdrm, so I’ve created the file ‘CBL-Mariner/SPECS/libdrm/libdrm.spec’ with the following spec file contents listed below.
The build error I get is:
        spec_file=${spec} && \         srpm_file=$(rpmspec -q ${spec_file} --srpm --define='with_check 1' --define='dist .cm1' --queryformat %{NAME}-%{VERSION}-%{RELEASE}.src.rpm) && \         wget https://packages.microsoft.com/cbl-mariner/1.0/prod/base/srpms/${srpm_file} \                 -O /home/user/Ryan/stable/CBL-Mariner/toolkit/../build/INTERMEDIATE_SRPMS/${srpm_file} \             --no-verbose \              \              \             && \         touch /home/user/Ryan/stable/CBL-Mariner/toolkit/../build/INTERMEDIATE_SRPMS/${srpm_file}  || \                 { echo "Failed to download ${srpm_file}" ; exit 1 ; } ; \ done || { echo "Loop in /home/user/Ryan/stable/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag failed" ; exit 1 ; } ; \ touch /home/user/Ryan/stable/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag Failed to download libdrm-2.4.100-0.src.rpm /home/user/Ryan/stable/CBL-Mariner/toolkit/scripts/srpm_pack.mk:42: recipe for target '/home/user/Ryan/stable/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag' failed
libdrm.spec
#
# spec file for package libdrm
#
# Copyright (c) 2013 SUSE LINUX Products GmbH, Nuernberg, Germany.
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
# upon. The license for this file, and modifications and additions to the
# file, is the same license as for the pristine package itself (unless the
# license for the pristine package is not an Open Source License, in which
# case the license is the MIT License). An "Open Source License" is a
# license that conforms to the Open Source Definition (Version 1.9)
# published by the Open Source Initiative.

# Please submit bugfixes or comments via http://bugs.opensuse.org/
#

Name:           libdrm
Url:            http://dri.freedesktop.org/
Version:        2.4.100
Release:        0
Provides:       libdrm23 = %{version}
Obsoletes:      libdrm23 < %{version}
# bug437293
%ifarch ppc64
Obsoletes:      libdrm-64bit < %{version}
Provides:       libdrm-64bit = %{version}
%endif
#
Summary:        Userspace Interface for Kernel DRM Services
License:        MIT
Group:          System/Libraries

#Git-Clone:     git://anongit.freedesktop.org/git/mesa/drm
#Git-Web:       http://cgit.freedesktop.org/mesa/drm/
Source:         http://dri.freedesktop.org/libdrm/%{name}-%{version}.tar.bz2
#Source:         https://kojipkgs.fedoraproject.org//packages/libdrm/2.4.102/1.fc33/x86_64/libdrm-2.4.102-1.fc33.x86_64.rpm
Source2:        baselibs.conf
# PATCH-FIX-UPSTREAM libdrm-implicit-defs.diff fdo#48599 -- Fix compiler warnings in tests/radeon/radeon_ttm.c
Patch1:         libdrm-implicit-defs.diff
Patch2:         u_Fix-compilation-on-systems-that-don-t-provide-O_CLOE.patch

BuildRoot:      %{_tmppath}/%{name}-%{version}-build
BuildRequires:  autoconf >= 2.63
BuildRequires:  automake >= 1.10
BuildRequires:  libtool >= 2.2
BuildRequires:  pkgconfig
BuildRequires:  pkgconfig(pciaccess) >= 0.10
BuildRequires:  pkgconfig(pthread-stubs)
%ifnarch armv5el armv5tel aarch64
BuildRequires:  pkgconfig(valgrind)
%endif

%description
The package contains the userspace interface to the kernel DRM
services.

%package tools
Summary:        Diagnostic utilities for DRI and DRM
Group:          System/Base
# O/P added in 12.2
Obsoletes:      libdrm < %version-%release
Provides:       libdrm = %version-%release

%description tools
(Upstream has not provided a description)

%package devel
Summary:        Libraries, includes and more to develop libdrm applications
Group:          Development/Libraries/X11
Requires:       libdrm2 = %version
Requires:       libdrm_nouveau2 = %version
Requires:       libdrm_radeon1 = %version
%ifarch %arm
Requires:       libdrm_omap1 = %version
%endif
%ifarch %ix86 x86_64
Requires:       libdrm_intel1 = %version
%endif
# bug437293
%ifarch ppc64
Obsoletes:      libdrm-devel-64bit < %{version}
Provides:       libdrm-devel-64bit = %{version}
%endif

%description devel
libdrm is a library for accessing the Direct Rendering Manager on
Linux, BSD and other operating systes that support the ioctl
interface, and for chipsets with DRM memory manager, support for
tracking relocations and buffers. libdrm is a low-level library,
typically used by graphics drivers such as the Mesa DRI and X
drivers.

This package contains the development headers for the library found
in libdrm2.

%package -n libdrm2
Summary:        Userspace Interface for Kernel DRM Services
Group:          System/Libraries

%description -n libdrm2
libdrm is a library for accessing the Direct Rendering Manager on
Linux, BSD and other operating systes that support the ioctl
interface, and for chipsets with DRM memory manager, support for
tracking relocations and buffers. libdrm is a low-level library,
typically used by graphics drivers such as the Mesa DRI and X
drivers.

%package -n libdrm_intel1
Summary:        Userspace interface for Kernel DRM services for Intel chips
Group:          System/Libraries

%description -n libdrm_intel1
(Upstream has not provided a description)

%package -n libdrm_nouveau2
Summary:        Userspace interface for Kernel DRM services for NVIDIA chips
Group:          System/Libraries

%description -n libdrm_nouveau2
(Upstream has not provided a description)

%package -n libdrm_radeon1
Summary:        Userspace interface for Kernel DRM services for AMD Radeon chips
Group:          System/Libraries

%description -n libdrm_radeon1
(Upstream has not provided a description)

%package -n libkms-devel
Summary:        Development files for the KMS MM abstraction library
Group:          Development/Libraries/C and C++
Requires:       libkms1 = %version

%description -n libkms-devel
This package contains the development headers for the library found
in libkms1.

%package -n libkms1
Summary:        Userspace interface to kernel DRM buffer management
Group:          System/Libraries

%description -n libkms1
The KMS Memory Management abstraction library.

%ifarch %arm
%package -n libdrm_omap1
Summary:        Userspace interface to kernel DRM services for omap chips
Group:          System/Libraries

%description -n libdrm_omap1
(Upstream has not provided a description)
%endif

%prep
%setup -q
%patch1 -p1
%patch2 -p1

%build
export CFLAGS="$RPM_OPT_FLAGS -fno-strict-aliasing"
autoreconf -fi
%configure --with-pic \
%ifarch %arm
           --enable-omap-experimental-api \
%endif
           --enable-udev

make %{?_smp_mflags}
make %{?_smp_mflags} -C tests dristat drmstat

%check
make %{?_smp_mflags} -C tests check-am

%install
make install DESTDIR=$RPM_BUILD_ROOT
make -C tests/modeprint install DESTDIR=$RPM_BUILD_ROOT
make -C tests/modetest install DESTDIR=$RPM_BUILD_ROOT
rm -f "%buildroot/%_libdir"/*.la
%{__mkdir} -p $RPM_BUILD_ROOT/usr/bin
%{__install}  \
    tests/.libs/dristat \
    tests/.libs/drmstat \
    tests/modeprint/.libs/modeprint \
    tests/modetest/.libs/modetest $RPM_BUILD_ROOT/usr/bin

%post   -n libdrm2 -p /sbin/ldconfig

%postun -n libdrm2 -p /sbin/ldconfig

%post   -n libdrm_intel1 -p /sbin/ldconfig

%postun -n libdrm_intel1 -p /sbin/ldconfig

%post   -n libdrm_nouveau2 -p /sbin/ldconfig

%postun -n libdrm_nouveau2 -p /sbin/ldconfig

%post   -n libdrm_radeon1 -p /sbin/ldconfig

%postun -n libdrm_radeon1 -p /sbin/ldconfig

%post   -n libkms1 -p /sbin/ldconfig

%postun -n libkms1 -p /sbin/ldconfig

%ifarch %arm
%post   -n libdrm_omap1 -p /sbin/ldconfig

%postun -n libdrm_omap1 -p /sbin/ldconfig
%endif

%files tools
%defattr(-,root,root)
%_bindir/dristat
%_bindir/drmstat
%_bindir/modeprint
%_bindir/modetest

%files devel
%defattr(-,root,root)
%_includedir/libdrm
%ifarch %arm
%_includedir/omap
%endif
%_includedir/xf86drm.h
%_includedir/xf86drmMode.h
%_libdir/libdrm*.so
%_libdir/pkgconfig/libdrm*.pc
#%_mandir/man3/drm*

%files -n libdrm2
%defattr(-,root,root)
%_libdir/libdrm.so.2*

%ifarch %ix86 x86_64

%files -n libdrm_intel1
%defattr(-,root,root)
%_libdir/libdrm_intel.so.1*
%endif

%files -n libdrm_nouveau2
%defattr(-,root,root)
%_libdir/libdrm_nouveau.so.2*

%files -n libdrm_radeon1
%defattr(-,root,root)
%_libdir/libdrm_radeon.so.1*

%files -n libkms1
%defattr(-,root,root)
%_libdir/libkms.so.1*

%files -n libkms-devel
%defattr(-,root,root)
%_includedir/libkms
%_libdir/libkms.so
%_libdir/pkgconfig/libkms.pc

%ifarch %arm
%files -n libdrm_omap1
%defattr(-,root,root)
%_libdir/libdrm_omap.so.1*
%endif

%changelog

The culprit here is DOWNLOAD_SRPMS=y. Since its not practical to check in the source code for every package, and we want to have a reliable source to pull them from, we host them internally on a 'source server' (See documentation on SOURCE_URL=).  We make the sources available to the public via Source RPMs (SRPMS) which package both the spec files and their associated sources together.
The catch here: DOWNLOAD_SRPMS=y is an all or nothing flag, once it is on the build system looks exclusively to the package server for all its source and spec files (ignoring your new libdrm package). It knows you want a libdrm package, but can't find the SRPM on the server, so fails.
Fix:
Assuming you don't want to setup your own source server, and if you don't mind using pre-built packages:
Delete every package out of ./SPECS except the ones you have modified, then disable DOWNLOAD_SRPMS. By default the build system will reach out to the package server to find any missing packages during both package build and image generation so they aren't needed locally.
Assuming you haven't modified any packages (its fine if you did, just include them as well along with sources), you should have:

./SPECS/libdrm/libdrm.spec,
./SPECS/libdrm/SOURCES/<source files & patches> (Can also be in the ./SPECS/libdrm/ folder, but the SOURCES is recommended)
./toolkit
LICENSE etc...

Validate it worked with sudo make input-srpms  REBUILD_TOOLS=y SRPM_FILE_SIGNATURE_HANDLING=update (DOWNLOAD_SRPMS=n is implied, SRPM_FILE_SIGNATURE_HANDLING=update will generate the source file hashes for you automatically). This will scan the ./SPECS folder for any packages and convert them into SRPMS.
NOTE: You didn't mention if you added libdrm to the image config. By default the system will include all packages in toolkit/imageconfigs/packagelists/core-packages-image.json, as referenced by the default image config toolkit/imageconfigs/core-efi.json. Make sure its added to this list so the build system knows it needs to be built.
Assuming that worked, sudo make iso REBUILD_TOOLS=y REBUILD_PACKAGES=n should now also work.
The build system will figure out the packages needed to build libdrm and download them, build libdrm, then figure out what packages are still missing for the final ISO. It will then download those, and combine them together into the final ISO.
NOTE: You might need to explicitly set the BuildRequires for libdrm, dynamic dependencies are coming soon to 2.0 (#141, #163).
ps: If you want a portable copy of the toolkit you can run sudo make package-toolkit REBUILD_TOOLS=y and setup shop in your own repo. Just extract the toolchain archive into ./toolchain inside the new folder.

Hi @dmcilvaney , I stepped away from needing to extend Mariner for a few months, but now I'm back. I'm trying to use your steps to build in support for 'smartpqi'. Which branch/tag do you recommend I use? Working in 1.0-stable, I used your suggestions above, but hit an issue eventually. I thought this was related to dynamic dependencies as you mentioned in your "NOTE".

`user@user-ProLiant-ML30-Gen10:~/Xilinx-Builds/CBL-Mariner/toolkit$ sudo make input-srpms REBUILD_TOOLS=y SRPM_FILE_SIGNATURE_HANDLING=update -j$(nproc)
[sudo] password for user:
Makefile:13: CONFIG_FILE is empty, defaulting to toolkit's core-efi.json unless CONFIG_FILE="" was set explicitly.
GODEBUG=x509ignoreCN=0 /home/user/Xilinx-Builds/CBL-Mariner/toolkit/out/tools/srpmpacker
--dir=/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../SPECS
--output-dir=/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/INTERMEDIATE_SRPMS
--source-url=
--dist-tag=.cm1
--ca-cert=
--tls-cert=
--tls-key=
--build-dir=/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/SRPM_packaging
--signature-handling=update
WARN[0000] Will update signature files as needed
INFO[0000] Finding all SPEC files
INFO[0000] Calculating SPECs to repack
INFO[0000] Packing 1/1 SPECs
PANI[0000] Unable to hydrate file: kmod-smartpqi-1.1.2-120.rhel7u4.x86_64.tar.bz2
panic: (*logrus.Entry) 0xc00021c230
goroutine 30 [running]:
github.com/sirupsen/logrus.Entry.log(0xc00011e5b0, 0xc0002063f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00021c1c0, 0x0, 0xc000127868, 0x1, 0x1)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Logf(0xc00021c1c0, 0xc000000000, 0x81cee2, 0x1a, 0xc0001279c0, 0x1, 0x1)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:333 +0xe5
github.com/sirupsen/logrus.(*Logger).Logf(0xc00011e5b0, 0x0, 0x81cee2, 0x1a, 0xc0001279c0, 0x1, 0x1)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:146 +0x95
github.com/sirupsen/logrus.(*Logger).Panicf(...)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:187
main.hydrateFiles(0x1, 0xc00001c1e0, 0x4b, 0xc000058310, 0x61, 0xc00001a180, 0x38, 0x7fff8c43159e, 0x0, 0xc000071530, ...)
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:605 +0x3ae
main.packSingleSPEC(0xc00001c1e0, 0x4b, 0xc000058150, 0x65, 0xc00001a180, 0x56, 0x7fff8c4315e8, 0x44, 0xc000058150, 0x3d, ...)
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:484 +0x57c
main.packSRPMWorker(0xc00005cba0, 0xc00005cc00, 0x7fff8c4315b5, 0x4, 0x7fff8c4315e8, 0x44, 0x0, 0x0, 0x7fff8c43159e, 0x0, ...)
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:390 +0x2bf
created by main.packSRPMs
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:344 +0x118
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/scripts/srpm_pack.mk:61: recipe for target '/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag' failed
make: *** [/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag] Error 2
`

How do you suggest I add this package?

Hi @dmcilvaney , I stepped away from needing to extend Mariner for a few months, but now I'm back. I'm trying to use your steps to build in support for 'smartpqi'. Which branch/tag do you recommend I use? Working in 1.0-stable, I used your suggestions above, but hit an issue eventually. I thought this was related to dynamic dependencies as you mentioned in your "NOTE".

`user@user-ProLiant-ML30-Gen10:~/Xilinx-Builds/CBL-Mariner/toolkit$ sudo make input-srpms REBUILD_TOOLS=y SRPM_FILE_SIGNATURE_HANDLING=update -j$(nproc)
[sudo] password for user:
Makefile:13: CONFIG_FILE is empty, defaulting to toolkit's core-efi.json unless CONFIG_FILE="" was set explicitly.
GODEBUG=x509ignoreCN=0 /home/user/Xilinx-Builds/CBL-Mariner/toolkit/out/tools/srpmpacker
--dir=/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../SPECS
--output-dir=/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/INTERMEDIATE_SRPMS
--source-url=
--dist-tag=.cm1
--ca-cert=
--tls-cert=
--tls-key=
--build-dir=/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/SRPM_packaging
--signature-handling=update
WARN[0000] Will update signature files as needed
INFO[0000] Finding all SPEC files
INFO[0000] Calculating SPECs to repack
INFO[0000] Packing 1/1 SPECs
PANI[0000] Unable to hydrate file: kmod-smartpqi-1.1.2-120.rhel7u4.x86_64.tar.bz2
panic: (*logrus.Entry) 0xc00021c230
goroutine 30 [running]:
github.com/sirupsen/logrus.Entry.log(0xc00011e5b0, 0xc0002063f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x345
github.com/sirupsen/logrus.(*Entry).Log(0xc00021c1c0, 0x0, 0xc000127868, 0x1, 0x1)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xf0
github.com/sirupsen/logrus.(*Entry).Logf(0xc00021c1c0, 0xc000000000, 0x81cee2, 0x1a, 0xc0001279c0, 0x1, 0x1)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:333 +0xe5
github.com/sirupsen/logrus.(*Logger).Logf(0xc00011e5b0, 0x0, 0x81cee2, 0x1a, 0xc0001279c0, 0x1, 0x1)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:146 +0x95
github.com/sirupsen/logrus.(*Logger).Panicf(...)
/home/user/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:187
main.hydrateFiles(0x1, 0xc00001c1e0, 0x4b, 0xc000058310, 0x61, 0xc00001a180, 0x38, 0x7fff8c43159e, 0x0, 0xc000071530, ...)
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:605 +0x3ae
main.packSingleSPEC(0xc00001c1e0, 0x4b, 0xc000058150, 0x65, 0xc00001a180, 0x56, 0x7fff8c4315e8, 0x44, 0xc000058150, 0x3d, ...)
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:484 +0x57c
main.packSRPMWorker(0xc00005cba0, 0xc00005cc00, 0x7fff8c4315b5, 0x4, 0x7fff8c4315e8, 0x44, 0x0, 0x0, 0x7fff8c43159e, 0x0, ...)
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:390 +0x2bf
created by main.packSRPMs
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/tools/srpmpacker/srpmpacker.go:344 +0x118
/home/user/Xilinx-Builds/CBL-Mariner/toolkit/scripts/srpm_pack.mk:61: recipe for target '/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag' failed
make: *** [/home/user/Xilinx-Builds/CBL-Mariner/toolkit/../build/make_status/build_srpms.flag] Error 2
`

How do you suggest I add this package?

Where practical please clone the CBL-MarinerDemo repository (https://github.com/microsoft/CBL-MarinerDemo) and follow the associated tutorial.  This is an easier environment to experiment with building new packages.  Once the package builds, and if necessary, you can copy the SPEC file to a clone of CBL-Mariner and follow the standard build instructions for CBL-Mariner.

authors: PawelWMS dmcilvaney jslobodzian rradjabi thcrain-msft

repo: microsoft/CBL-Mariner


text:
Commonize toolchain rpm extract flows

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

[x] The toolchain has been rebuilt successfully (or no changes were made to it)
[x] The toolchain/worker package manifests are up-to-date
[x] Any updated packages successfully build (or no packages were changed)
[x] Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
[x] All package sources are available
[x] cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/tools/cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
[x] LICENSE-MAP files are up-to-date (./SPECS/LICENSES-AND-NOTICES/data/licenses.json, ./SPECS/LICENSES-AND-NOTICES/LICENSES-MAP.md, ./SPECS/LICENSES-AND-NOTICES/LICENSE-EXCEPTIONS.PHOTON)
[x] All source files have up-to-date hashes in the *.signatures.json files
[x] sudo make go-tidy-all and sudo make go-test-coverage pass
[x] Documentation has been updated to match any changes to the build system
[ ] Ready to merge

Summary

The full end-to-end toolchain builds are triggering the fixup extract step for every single toolchain rpm ($(toolchain_rpms): target). The flow which sets REBUILD_TOOLCHAIN=n TOOLCHAIN_ARCHIVE=toolchain.tar.gz however uses a staging mechanism so that we only need to extract the files once (the current flow invokes tar for every rpm which is very slow)
These changes make both paths run through a common set of build targets, with a number of improvements. The archive file should now only be extracted once into a temporary folder. All subsequent operations just copy the toolchain rpms from that temporary staging directory into out/RPMS or the rpm cache.
Any changes to the toolchain archive or manifests etc should trigger a wipe and re-extraction into that folder.
Change Log

Moved extract directory from ./build/toolchain_temp to ./build/toolchain/extract_dir
Converted both toolchain population flows to use the same set of targets by way of setting the variable selected_toolchain_archvie:

$(STATUS_FLAGS_DIR)/toolchain_local_temp.flag: Which populates the extract directory with the contents of selected_toolchain_archive
$(toolchain_rpms): Which goes through and makes sure each rpm is present in its desired location (./out/RPMS or the cache directory)

Each .rpm is copied from the extract directory if it is out of date or missing (much quicker to copy and re-extract from the archive)

Toolchain validate is now a blocking error (if the toolchain manifest doesn't match the archive exactly)
Tooling now watches the REBUILD_TOOLCHAIN=? and TOOLCHAIN_ARCHIVE=? configs for changes and will re-extract the toolchain RPMs if they change

Does this affect the toolchain?

YES

How much will this increase the disk usage during a build?

Added a dependency on the toolchain state for the chroot tar.gz as well, so it gets regenerated if we swap our toolchain around

How much will this increase the disk usage during a build?

I've seen ~1.5 to 1.6 GB

Running pipeline build 191856 to validate

I'm seeing an issue with "make copy-toolchain-rpms". Might need to update this as well
sudo make copy-toolchain-rpms
...
cp: cannot stat '/CBL-Mariner/build/rpm_cache/cache/x86_64/zstd-doc-1.5.0-1.cm2.x86_64.rpm': No such file or directory
toolkit/scripts/toolchain.mk:67: recipe for target 'copy-toolchain-rpms' failed
make: *** [copy-toolchain-rpms] Error 1

Was this from local testing? I wasn't able to repro it on my end. I think this is what it looks like if you run this command after doing a local build rather than a copy from archive/web build.

Added dependencies to catch the edge case where one or more toolchain rpms are missing, but are NOT part of the chroot. The tools would not regenerate those missing rpms, since the only target we actually invoke are the chroot targets.
We need those rpms present to resolve circular dependencies in the graph.
I feel its outside the scope of the Makefile to be responsible for handling copy-toolchain-rpms etc automatically when you are not rebuilding the toolchain from source but are working in the core repo. I've added a warning message for when we find circular dependencies that suggests the most common solution to the issue.

authors: MateuszMalisz dmcilvaney


text:
[AUTOPATCHER-CORE] Upgrade curl to 8.1.2 to address CVE-2023-28322.

[AUTOPATCHER-CORE] Upgrade curl to 8.1.2 Address CVE-2023-28322
Upgrade pipeline run -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=372327&view=results
AMD64 build -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=372328&view=results
ARM64 build -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=372329&view=results

Let's ensure no breaking changes, ABI/API and all is introduce is introduce in between 8.0.1 and 8.1.2: https://curl.se/changes.html
Is a backport to 8.0.1 feasible ?

I believe this is superseded by https://github.com/microsoft/CBL-Mariner/pull/5642?

https://github.com/microsoft/CBL-Mariner/pull/5642 Merged. Marking this as closed.

authors: CBL-Mariner-Bot Sumynwa dmcilvaney eric-desrochers


text:
A question about infinity string

When I execute the testcase below, the returncode is -6 and aborted without other information (exception, warning, OOM etc.) provided. This is not friendly enough for users. Maybe you could enhance it.
version
eaaf7ac
build platform
Ubuntu 16.04.6 LTS (Linux 4.15.0-55-generic x86_64)
build step
./build.sh --debug
testcase
var NISLFuzzingFunc =function(){
    var result = "-";
    while(result !== undefined){
        result += result;
    }
};
NISLFuzzingFunc();

run step
ChakraCore/out/Release/ch testcase.js

The issue here is that you continually extend the string until your computer is out of memory.
If you compile with the flag EnableFatalErrorOnOOM set to false it will turn off this abort and throw a javascript exception in this case.
HOWEVER default behaviour is intentionally to abort when out of memory as there is no safe and reliable way to recover when out of memory - for instance there may not be enough memory to handle the javascript exception.

I have used the command ./build.sh --extra-defines=EnableFatalErrorOnOOM=false to compile the engine and run this testcase again, but it printed Aborted without any exceptions. Did I use the correct compile command?

Hmm curious - if you use a testbuild or a debug build and supply the flag -EnableFatalErrorOnOOM- at runtime you get the Js error for out of memory. However with a release build setting that at compile time doesn't seem to work (release builds don't accept runtime flags).
That aside as already mentioned this behaviour in release builds is by design as there is no reliable safe way to recover from running out of memory.

thanks for your explanation!

authors: YuanWangC rhuanjl

repo: microsoft/ChakraCore


text:
ubuntu16.04 aarch64, not found makefile

Does ChakraCore now support linux arm or arm64 architecture compilation?

Not currently no - but it would be good if it did.
It shouldn't be too large a piece of work to do as it supports linux on x64 and windows on arm/arm64. Probably mostly a case of makefile support - though there will likely be some #ifdefs that needed editing too.
I'm afraid I don't have a suitable arm device to test with - it would be great if someone else wanted to look into this though.

@527395632 can you try adding aarch64 to this condition and re-running the build:
https://github.com/microsoft/ChakraCore/blob/master/CMakeLists.txt#L42

Thank you for your support, but there is currently no ability to modify this code, I look forward to supporting it in the future, I will always pay attention

ubuntu16.04  ARCH  arm32

Looking at the code: _interlockedbittestandreset is a windows API that has an x86 posix implementation on lines 721-730 but assumes if you're on ARM then you're on windows and hence has has no inline definition for ARM - considering that the inline definition is done in ASM I'm going to assume a different implementation will be needed for ARM so, we'll need to add a new #elif case for ARM && not windows - under which we'll need to add an ARM implementation for this.
Looking at the code you will probably hit the same issue with _InterlockedBitTestAndSet, _BitTest and _BitTest and _BitTestAndSet.

好吧! 过程很享受, 结局有点意外.看来在短时间内还是无法实现linux arm或者arm64上移植chakracore了。不知你们是否有这块规划?

Well! The process is very enjoyable, and the ending is a bit unexpected. It seems that it is still impossible to transplant chakracore on linux arm or arm64 in a short time. I wonder if you have this plan?

Well! The process is very enjoyable, and the ending is a bit unexpected. It seems that it is still impossible to transplant chakracore on linux arm or arm64 in a short time. I wonder if you have this plan?

It certainly would be good to support linux on ARM, and the actual work required would probably be quite small, but I think we need a contributor with an ARM device to make it happen. @ppenzin any thoughts?

I really hope to join your team, but I am worried that I will not be qualified for this job because my foundation is too bad. What I learned is .Net Framework, C, C++ will only compose a little bit, simple compilation, nothing more. Code perfection may not be competent. But I know more about arm. I transplanted charakcore in hope that arm-linux can also use react-native-windows to develop applications. This is my purpose. I wonder if my knowledge in this area can help?

In addition, if you have a plan, I will always pay attention to it and strive for the support update of arm in the first time

This is still open.  I can provide an arm64 ubuntu 18.04 device for a month.

@bob-dawson I can take a look.
@rhuanjl to answer your question - it looks like there are going to be PAL issues though, but let's hope it is going to be relatively easy.

xplat arm support is going to become more important in the future with Apple moving to ARM for their new macs.

authors: 527395632 bob-dawson ppenzin rhuanjl


text:
Dxdispatch present separator

Optional argument "--present_separator" which injects a present between each inference iteration. This is helpful for external tools which require a notion of frames to digest the captured data.
The functionality is only compiled into the binary if DX_PRESENT_SEPARATOR is defined.

@microsoft-github-policy-service agree [company="{NVIDIA}"]

Got an internal approval + pushed the agreement to your bot.

@jstoecker, the request should be ready for a merge. Please let me know if there is anything that needs to be adjusted.

Thanks for the merge!

repo: microsoft/DirectML


text:
Question about SpanByte and Async methods

If we use SpanByte it's required to have fixed input, how long does it need to be fixed, according to the comment for the full operation, is that correct? is the full operation until Complete() is called?
As fixed/unsafe code in general does not work with async, how is it supposed (if at all) work with async?
We can work around the async&unsafe problem by using code such as below, method ReadAsync which checks the result of the valueTask for completeness and calls an async method if not. But that breaks the fixed statement as far as I understand, as the pinning will be released at the end of the fixed block and that will be earlier than the Complete() statement in CompleteAsync().
So can we use SpanByte at all with async code, or should we just switch to SpanByteAndMemory for everything?
This means implementing our own SpanByteFunctions just for SpanByteAndMemory like in https://github.com/microsoft/FASTER/blob/9ef252702395ac3d2123044fa89dd54ef17e4aaa/cs/src/core/VarLen/SpanByteFunctions.cs#L74-L76
and probably deriving from FunctionBase directly and copying the current logic from SpanByteFunctions around.
As you're currently adding more async methods it appears that async is the way to go for new implementations?
using System;
using System.Reflection.Metadata.Ecma335;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Threading;
using System.Threading.Tasks;
using FASTER.core;

namespace FasterTest
{
    class Program
    {
        static async Task Main(string[] args)
        {
            using var cache = new Cache();
            cache.Upsert<long, long>(1, 2);
            var value = await cache.ReadAsync<long, long>(1);
        }
    }

    public sealed class Cache : IDisposable
    {
        private readonly IDevice _log;
        private readonly FasterKV<SpanByte, SpanByte> _store;
        private readonly CustomSpanByteFunctions _simpleFunctions = new();

        public Cache()
        {
            _log = Devices.CreateLogDevice("hlog.log", deleteOnClose: true);
            _store = new FasterKV<SpanByte, SpanByte>(1L << 20,
                new LogSettings
                {
                    LogDevice = _log,
                    ReadCacheSettings = new ReadCacheSettings(),
                });
        }

        public void Upsert<TKey, TValue>(TKey key, TValue value) where TKey : unmanaged where TValue : unmanaged
        {
            var session = GetSession();
            unsafe
            {
                var keySpan = MemoryMarshal.AsBytes(MemoryMarshal.CreateSpan(ref Unsafe.AsRef(key), 1));
                fixed (byte* _ = keySpan)
                {
                    SpanByte keySpanByte = SpanByte.FromFixedSpan(keySpan);
                    var dataSpan = MemoryMarshal.AsBytes(MemoryMarshal.CreateSpan(ref Unsafe.AsRef(value), 1));
                    fixed (byte* __ = dataSpan)
                    {
                        var dataSpanByte = SpanByte.FromFixedSpan(dataSpan);
                        session.Upsert(ref keySpanByte, ref dataSpanByte);
                    }

                }
            }
            ReleaseSession(session);
        }

        public ValueTask<TValue?> ReadAsync<TKey, TValue>(TKey key, CancellationToken cancellationToken = default)
            where TKey : unmanaged where TValue : unmanaged
        {
            TValue? result = default;
            var session = GetSession();
            unsafe
            {
                Span<byte> keySpan = MemoryMarshal.AsBytes(MemoryMarshal.CreateSpan(ref Unsafe.AsRef(key), 1));
                fixed (byte* _ = keySpan)
                {
                    SpanByte keySpanByte = SpanByte.FromFixedSpan(keySpan); // how long does it need to be fixed?
                    var tuple = session.ReadAsync(ref keySpanByte, token: cancellationToken);
                    if (tuple.IsCompletedSuccessfully)
                    {
                        var (status, memory) = tuple.Result.Complete();
                        if (status == Status.OK)
                        {
                            result = Deserialize<TValue>(memory);
                        }

                        ReleaseSession(session);
                    }
                    else
                    {
                        return CompleteAsync<TValue>(tuple, session);
                    }
                }
            }

            return new ValueTask<TValue?>(result);
        }

        private TValue? Deserialize<TValue>(SpanByteAndMemory memory) where TValue : struct
        {
            var span = memory.IsSpanByte ? memory.SpanByte.AsSpan() : memory.Memory.Memory.Span;
            if (MemoryMarshal.TryRead<TValue>(span, out var value))
            {
                return value;
            }

            memory.Memory?.Dispose();
            return default;
        }

        private async ValueTask<TValue?> CompleteAsync<TValue>(ValueTask<FasterKV<SpanByte, SpanByte>.ReadAsyncResult<SpanByte, SpanByteAndMemory, Empty>> valueTask,
            ClientSession<SpanByte, SpanByte, SpanByte, SpanByteAndMemory, Empty, CustomSpanByteFunctions> session) where TValue : struct
        {
            try
            {
                var readAsyncResult = await valueTask.ConfigureAwait(false);
                var (status, memory) = readAsyncResult.Complete();
                ReleaseSession(session);
                return status == Status.OK ? Deserialize<TValue>(memory) : default(TValue?);
            }
            catch
            {
                session.Dispose();
                return default;
            }
        }

        private ClientSession<SpanByte, SpanByte, SpanByte, SpanByteAndMemory, Empty, CustomSpanByteFunctions> GetSession()
        {
            var session = _store.For(_simpleFunctions).NewSession<CustomSpanByteFunctions>();
            return session;
        }

        private void ReleaseSession(
            ClientSession<SpanByte, SpanByte, SpanByte, SpanByteAndMemory, Empty, CustomSpanByteFunctions> session)
        {
            session.Dispose();
        }

        public void Dispose()
        {
            _log.Dispose();
            _store.Dispose();
        }

        private sealed class CustomSpanByteFunctions : SpanByteFunctions<Empty>
        {
        }
    }

}

If we use SpanByte it's required to have fixed input, how long does it need to be fixed, according to the comment for the full operation, is that correct? is the full operation until Complete() is called?

Data only needs to be fixed until the operation comes back from the initial call. When an operation goes pending, we copy the data to an internal buffer before returning control to you, so you no longer need it fixed.
Note that if key/value are value-types on the stack (not heap) then you may not need to pin them at all.

So can we use SpanByte at all with async code, or should we just switch to SpanByteAndMemory for everything?

If you are having trouble with fixing and pinning, you can also use Memory<byte> as value and ReadOnlyMemory<byte> as key. See this for a sample: https://github.com/microsoft/FASTER/blob/9ef252702395ac3d2123044fa89dd54ef17e4aaa/cs/samples/StoreVarLenTypes/Program.cs#L37

As you're currently adding more async methods it appears that async is the way to go for new implementations?

Not necessarily. It depends on your performance targets. C# async gives ease of usablity, but comes at some performance cost. We support both, and async methods are using sync methods under the hood so the sync API isn't going away anywhere.

Thank you very much for your detailed answers.

Data only needs to be fixed until the operation comes back from the initial call. When an operation goes pending, we copy the data to an internal buffer before returning control to you, so you no longer need it fixed.

Ah great, then looks like I'm fine with my current approach.

Not necessarily. It depends on your performance targets. C# async gives ease of usablity, but comes at some performance cost. We support both, and async methods are using sync methods under the hood so the sync API isn't going away anywhere.

Thanks, that leaves more options open for me.

authors: Tornhoof badrishc

repo: microsoft/FASTER


text:
Discrepancy between container.serialize and loader.rehydrateDetachedContainerFromSnapshot

Feature
Is your feature request related to a problem? Please describe.

There is a discrepancy between the container.serialize() return type and the loader.rehydrateDetachedContainerFromSnapshot(). The former returns a string while the latter accepts an ISummaryTree.

Describe the solution you'd like

I think both should be ISummaryTree with serialization/deserialization left to the host to handle
Existing work

Nothing I'm aware of.
Describe alternatives you've considered

This doesn't block the e2e flow but requires hosts to depend on an internal implementation detail (that .serialize() returns a JSON.stringify of an ISummaryTree).
Additional context

None

@vladsud @jatgarg fyi

I should add that this should probably be consistent with OdspDocumentServiceFactory.createContainer

@aboktor Do we need to cherrypick this in release/0.27.x

authors: aboktor jatgarg

repo: microsoft/FluidFramework


text:
Unknown errorTypes recorded by our telemetry

Office_Fluid_FluidRuntime_Error
| where Data_loaderVersion contains "0.49"
| where Data_errorType != ""
| summarize count() by Data_errorType
Gives me these error types that I do not recognize - they are not in our repo!
CreateAndCachePromiseError	34
ErrorWithLocalizedString-CreateAndCachePromiseError	7
Based on the errors for first type (like "-Error: Received an error from IPC::{"eventName":"aadGetToken","error":"resourceDisabled","correlationId":"17610973-923d-4a05-a460-52d1b23e3640"}::isOnline-true"), this looks like token callback, which I assume uses same schema. I think that's rare place where we want to overwrite errorType!
Worth taking a look at second case as well
Also I think this one is the same issue (in a sense that it does not have right errorType, but probably can be verified to be coming from token callback and it should set errorType):
Data_eventName == fluid:telemetry:Container:storage_getVersions
Data_error == popup_window_error: Error opening popup window. This can happen if you are using IE or if popups are blocked in the browser. Details: BrowserAuthError: empty_window_error: window.open returned null or undefined window object.
or Data_error == _acquireToken::NetworkOffline::isOnline-false
or Data_error == Failed to acquire auth token for SharePoint
or Data_error == Interactive sign-in is required to get access token

Couple more - these seem all token fetches, so assume your fix is addressing that?
fluid:telemetry:Container:ContainerClose	Teams token retrieval timeout	genericError	1563
fluid:telemetry:Container:ContainerClose	_acquireToken::NetworkOffline::isOnline-false	genericError	843
fluid:telemetry:Container:ContainerClose	Broker response returned error: WAM Error  genericError	398
fluid:telemetry:Container:ContainerClose	post_request_failed:  undefined	genericError	263
This one is more interesting - this is host code calling into close with their error object - not sure if we want to put some errorType on it?
fluid:telemetry:Container:ContainerClose	Current version of the container 'Dogfood'-'20211103005' is no longer supported	genericError	383

This one is more interesting - this is host code calling into close with their error object - not sure if we want to put some errorType on it?
fluid:telemetry:Container:ContainerClose Current version of the container 'Dogfood'-'20211103005' is no longer supported genericError 383

They are setting genericError on it.  Are you suggesting we help them find a different errorType?  This case could be fixed up by #8623, they can just pass this as reason instead of having to craft an error?  Unsure if that's better.

@vladsud, I took a look at these. All the token fetch ones are addressed by #8151 which is in 0.51.  On account of this and my previous comment, closing this issue.

authors: markfields vladsud


text:
Write the sendOps logic for DataObjectWithCounter

Part of AB#1847 and AB#1837
As a part of https://github.com/microsoft/FluidFramework/pull/11928/files#
The goal of this PR is to write the send ops logic by incrementing the dataObjectWithCounter and to allow for the stoppage of sending ops as well. The start method is a way to allow clients to send ops regardless of what the other clients are doing.
This may be useful to put inside the dataObjectWithCounter class.

Closing this PR in favor of just moving all this logic to the datastore


text:
Attachment blobs - Always return local ID and send blob attach op for all blob uploads including de-dupe scenarios

Problem
There are 2 main reasons for adding this change and both are because of GC:

The main reason this change is added because of the way blob de-dupe interacts with GC. If a blob that has been unreferenced for a long time (almost ready for deletion) is de-duped, the server returns the same storage ID for it as the unreferenced blob. GC doesn't know about that the blob was de-duped and will proceed to delete the blob shortly. However, the client that got the de-duped storage ID can still refer to the blob leading to data loss.
The behavior in the case of blob de-dup should be the same as a brand new blob upload since from the users perspective thqat is the case. So, when the blob is de-duped, the time to delete it should be reset giving the user the same amount of time to re-use the blob as with a brand new blob upload.

The other reason is that GC validates all references from a node to another are explicitly notified. In case of blobs, a mapping from local ID to storage ID is essentially a reference and GC should be notified of it.

Solution
The high level solution is that when a blob is de-duped, notify GC so that it resets the unreferenced timestamp. Here is how this PR achieves that:

It always returns local IDs when a blob creation is completed. Currently, local IDs are only returned in detached / disconnected mode. This makes the creation flow consistent and easy to reason about - When a blob creation is complete, a local ID to storage ID mapping is created and the local ID is returned to users.
Whenever a local ID to storage ID mapping is created, notify GC that a reference is added. This will fix problem 2 above and since a mapping is created in blob de-dupe scenario as well, it will fix problem 1 as well.

Reviewer Guidance
Couple of key changes to note:

When container is online and a blob upload is completed, before the local ID is returned, a blob attach op is sent so that all clients can create a local ID to storage ID mapping on receiving it. This will ensure that any op containing this local ID is sequenced after the blob attach op and clients can retrieve the blob successfully.
When a blob is de-duped and the original blob is already ack'd, the blob is returned via the local ID as soon as the upload completes and the blob attach op is sent. Since the blob is already present, we don't need to wait for the op to be ack'd as the server will keep it alive.
When one or more blobs are de-duped and the original blob is not yet ack'd, these ops are added to a local queue (opsInFlight). As soon as one of the ops is ack'd, all the blobs are resolved and returned since this op will keep all of them alive as they have the same storage ID.

AB#2745

How does sending the local ID help? Wouldn't sending the op without it when deduped work?

How does sending the local ID help? Wouldn't sending the op without it when deduped work?

Yes, that will work too.

How does sending the local ID help? Wouldn't sending the op without it when deduped work?

Yes, that will work too. Basically, send an op in de-dupe case irrespective of whether we have a storageId or an op is in flight or not. When receiving the op and creating blobId -> blobId mapping, let GC know that a reference was added.
The reason for sending local ID is more about making the core logic consistent and a little simpler to follow. Basically, the core logic becomes - In all cases, create a local ID -> storage ID mapping, send a blob attach op with this mapping and return the blob with the local ID. On receiving the op, create a local ID -> storage ID mapping.
This is based on my understanding from reading the code and the various cases. Please let me know if you think otherwise.

@wes-carlson Can you please take another look. I made all the changes as per the comments.

Looks correct to me. I'm not sure how much of an effect it will have on performance, since BlobAttach ops in "normal" flow will be larger. Hopefully the guid compression stuff will make this a moot point soon.

Good point. There should be no effect on performance in terms of when we return an uploaded blob. The only concern would be no. of ops. There are multiple optmizations happening in this space like op compression and op virtualization (coming soon) which would help.

authors: agarwal-navin wes-carlson


text:
Converts build-api-markdown-documenter.yml pipeline to 1ES template

Description
This PR converts repo-policy-check.yml to the new 1ES template required by Microsoft with the minimal required set of template changes required. This new template enables security and auditing checks by Microsoft. Read more on how the logic for converting to a 1ES pipeline works on the official 1ES wiki
A temporary 1ES folder was created for converted pipeline templates. This enables other pipelines to remain unchanged and use the original templates while new PR's are opened for individually converted other pipelines to 1ES as required by Microsoft. You can easily view what the template changes are by looking at the diff in the following PR which is a fork of this branch and simply moves the template changes from the 1ES folder to the original so there is a clear diff present on github

Once all pipelines are moved over, we can start moving the files out of the 1ES folder and eventually remove the folder since it will no longer have any files (This is after we convert the rest of the required pipelines).

Breaking Changes
Reviewer Guidance

Confirm the newly converted pipelines run successfully. Look at the completed steps and compare with a recent successful run on main to ensure the expected steps run (plus the new SDL and 1ES security checks)
Make sure all the relevant templates and pipelines for build-client.yml and build-azure.yml have been converted
Make sure there are no unnecessary changes or 1ES template and/or pipeline conversions
Make sure the yml is valid -- a successful pipeline run should suffice for confirming this

For clarity, let's update the description; it's making reference to repo-policy-check.yml, and asks to check build-client and build-azure for review, where it should be the one for api-markdown-documenter.

For clarity, let's update the description; it's making reference to repo-policy-check.yml, and asks to check build-client and build-azure for review, where it should be the one for api-markdown-documenter.

Updated the description.


text:
Use a build service instead of a PAT for pr gating

This pull request mainly focuses on the changes in the src/pr-gating/CIVerify.yml and src/pr-gating/Setup.md files. The changes aim to improve security by replacing the Personal Access Token (PAT) with a system access token in the CIVerify.yml file and removing the steps for generating a new PAT in the Setup.md file. The changes also include adding instructions for granting permissions to the pipeline.

Is it worth documenting both approaches or you are convinced the PAT should be end of life :) Either way good to merge now!

I would argue that using the system token of the build service is strictly better than a PAT. No secret to store, nothing to rotate, etc...

Yeah, I think that there are pluses and minuses to both approaches - a PAT allows for custom permissions, but the system.AccessToken has much better maintainability at the expense of broader permissions radius (need to grant rights to the Build Service Acct, etc.)
That said - I think that most folks will understand that you can swap the System.AccessToken for a PAT.

authors: felickz ncouraud tjcorr

repo: microsoft/GHAzDO-Resources


text:
LightGBM\include\LightGBM/utils/common.h(33,10): fatal error C1083: Cannot open include file: 'fast_double_pa rser.h': No such file or directory

Hi,
The latest version fails to compile. Someone removed a file?
LightGBM\include\LightGBM/utils/common.h(33,10): fatal error C1083: Cannot open include file: 'fast_double_pa
rser.h': No such file or directory

There are access issues
D:\Source\LightGBM>git submodule update --init --recursive
Cloning into 'D:/Source/LightGBM/external_libs/eigen'...
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'https://gitlab.com/libeigen/eigen.git' into submodule path 'D:/Source/LightGBM/external_libs/eigen' failed
Failed to clone 'external_libs/eigen'. Retry scheduled
Cloning into 'D:/Source/LightGBM/external_libs/eigen'...
git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'https://gitlab.com/libeigen/eigen.git' into submodule path 'D:/Source/LightGBM/external_libs/eigen' failed
Failed to clone 'external_libs/eigen' a second time, aborting

Did you make some modifications to LightGBM? I'm surprised to see any errors about SSH (git@gitlab.com) issues cloning Eigen... LightGBM uses an HTTPS remote for Eigen.
https://github.com/microsoft/LightGBM/blob/e9a6c798071632f0e9bdf4c5fa2fdf16bfd5eb36/.gitmodules#L4-L6
And the repository at https://gitlab.com/libeigen/eigen absolutely exists and is publicly available.
What happens if you simply run this?
git clone https://gitlab.com/libeigen/eigen external_libs/eigen

I've edited your post to make the difference between code, output from code, and your own words clearer. Please see https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax if you're not familiar with editing text on GitHub.

Did you make some modifications to LightGBM? I'm surprised to see any errors about SSH (git@gitlab.com) issues cloning Eigen... LightGBM uses an HTTPS remote for Eigen.
https://github.com/microsoft/LightGBM/blob/e9a6c798071632f0e9bdf4c5fa2fdf16bfd5eb36/.gitmodules#L4-L6
And the repository at https://gitlab.com/libeigen/eigen absolutely exists and is publicly available.
What happens if you simply run this?
git clone https://gitlab.com/libeigen/eigen external_libs/eigen

I've edited your post to make the difference between code, output from code, and your own words clearer. Please see https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax if you're not familiar with editing text on GitHub.

This worked and required a password entry for the public key. I got around this by cloning on Ubuntu and moving the files to Windows...

and required a password entry for the public key

There shouldn't be any "public key" or other SSH details used in any part of this process...everything should be using only HTTPS to clone, and shouldn't require any authentication.
Maybe your git client is configured to redirect HTTPS remotes to their SSH equivalents with .insteadOf entries, like this: https://stackoverflow.com/a/11383587/3986677. You could look in ~/.gitconfig (I'm not sure what the equivalent is on Windows) to see if you have any entries like that. Or maybe you are using a managed machine from an employer where that's happening automatically, I'm not sure.
But either way, I don't believe this is something LightGBM can change. If your git client is rewriting HTTPS remotes to SSH ones, you'll need to create an account on GitLab and add SSH keys to it, following https://docs.gitlab.com/ee/user/ssh.html. Or you'll need to get your git client to stop doing that.

and required a password entry for the public key

There shouldn't be any "public key" or other SSH details used in any part of this process...everything should be using only HTTPS to clone, and shouldn't require any authentication.
Maybe your git client is configured to redirect HTTPS remotes to their SSH equivalents with .insteadOf entries, like this: https://stackoverflow.com/a/11383587/3986677. You could look in ~/.gitconfig (I'm not sure what the equivalent is on Windows) to see if you have any entries like that. Or maybe you are using a managed machine from an employer where that's happening automatically, I'm not sure.
But either way, I don't believe this is something LightGBM can change. If your git client is rewriting HTTPS remotes to SSH ones, you'll need to create an account on GitLab and add SSH keys to it, following https://docs.gitlab.com/ee/user/ssh.html. Or you'll need to get your git client to stop doing that.

Alright, thanks so much for all that detailed info!

repo: microsoft/LightGBM

license: mit

license_source: bigquery


text:
changes in the definition in 2-Regression/4-Logistic/README.md

Changed the "false positive" to "false negative" and vice versa as it was wrong definition.
The changes has been made here :
 - If your model predicts a pumpkin as orange and it belongs to category 'not-orange' in reality we call it a false positive, shown by the bottom left number.
 - If your model predicts a pumpkin as not orange and it belongs to category 'orange' in reality we call it a false negative, shown by the top right number.

made some error closing this pull request will open a new one.

repo: microsoft/ML-For-Beginners


text:
[Bug - Partner Center] Unable to publish extension update due to unexpected error

name: Bug report on Partner Center (impacting all developers)
about: Create a report to let us of any breaking issues or bugs that all Edge Addons
developers are facing when publishing their extension to Microsoft Edge Addons
title: "[Bug - Partner Center] Unable to publish extension update due to unexpected error"
labels: Bug, Partner Center
assignees: nagachaitanyalokam

Describe the bug
When attempting to publish a new extension version, after a few minutes of waiting, a vague error messages is shown:

We came across an unexpected error while trying to process your request. Please try again later

Neither the page's JavaScript console nor the network requests that are being made during those few minutes expose any additional information about what kind of error occurred.
To Reproduce
Steps to reproduce the behavior:

Go to https://partner.microsoft.com/en-us/dashboard/microsoftedge/overview
Click on the extension that you want to update
Click on "Update" button
Click on "Replace" link
Upload the new extension update
Click "Continue" button
Click "Publish" button
Click "Publish" button
Wait until the page no longer shows a loading indicator (usually takes a few minutes)
See error: "We came across an unexpected error while trying to process your request. Please try again later"
Go back to "Extension overview" page
Click "Publish" button next to extension version marked as "In draft"
Repeat steps (8) to (13)

Expected behavior

Step (9) should take less than a minute to complete
Step (10) should not show an error message, but instead show a message indicating that the extension was published successfully

Otherwise: The publishing should at least succeed after two or three retries.

Steps (11) to (13) should not be necessary

Screenshots

Desktop (please complete the following information):

OS: Ubuntu 22.04
Browser Chrome
Version 129

Additional context
When publishing new updates for the Adblock Plus for Microsoft Edge extension, we have encountered this problem every once in a while, but it seems to be occurring more frequently recently and it is currently so persistent that we are unable to publish the Adblock Plus 4.8 version to the Microsoft Edge extension store.

@nagachaitanyalokam It looks like the issue meta data wasn't applied and I can't update it after the fact either. I assume I did something wrong there, but I hope it's not a big deal, and thought I'd ping you this way instead.

Hey @ThomasGreiner, thank you for reporting this issue. Could you please try reuploading your extension as this might be an intermediate issue? If you still encounter any problems, feel free to write back to us.

@ManikanthMSFT Thank you for the quick reply and for the suggestion.
I have reuploaded the extension now by replacing the extension file in the "In draft" version, and the problem still persists even after multiple attempts to publish the extension.

Hey @ThomasGreiner, Thank you for your confirmation. We are looking into this and will let you know as soon as we have an update.

Hi @ManikanthMSFT. We're now trying to upload the next version of Adblock Plus and we're still running into this bug. Is there any feedback on getting this resolved?

@ManikanthMSFT Here's a screenshot of the error message we received today.  It appears to be the same issue that Thomas reported 2 weeks ago.

@ManikanthMSFT Sure, no problem. Here's the seller ID that's shown under Account settings > Organization profile > Legal info > Legal business profile > Account details: 26403930

Hi @ThomasGreiner, we appreciate your patience. We see that you submitted an extension, and it got published in the store yesterday. If you have any further questions or concerns, please don’t hesitate to contact us.

authors: JWorthe ManikanthMSFT ThomasGreiner montrosb

repo: microsoft/MicrosoftEdge-Extensions


text:
More options for SDP and video settings

Hi,
I'm currently initiating calls with the SDP i receive from the LocalSdpReadytoSend, but as I understand, it has already been set as local description at that time.
It would be beneficial if I could e.g. modify specific OPUS parameters as Max Average Bitrate, Max Capture Rate, Max Playback Rate and so forth. Same goes for video parameters such as resolution and framerate, but as I understand, that's already in the works. 👍
Maybe CreateOffer() could have an optional input variable for specific SDP options?

There are two concepts here, and I need to dig more into how they interact:

Parameters for the local video/audio capture, which the upcoming video profile changes are part of. This controls the capture only, and I think dictates the default resolution and framerate on connection start before any dynamic adjustment is made.

Parameters for the SDP negotiations for the video/audio transport through the peer connection and the bandwidth management.

SDP packet filtering and adding more options work on the second one. As I understand these constraints are maximum values to optimize network transport.
But for a scenario where you want to reduce the CPU load and/or bandwidth by limiting the video resolution you'd rather constrain directly the video capture module than the SDP offer, so that frames are directly captured in the format you want, and no transformation is needed. Otherwise capturing at 4K to transport at 640 x 480 will waste local CPU cycles to downscale the video stream before sending.
In short you need I think 2 things:

Tell webrtc what are the limits it should work within (SDP)
Tell the video/audio capture module what format to record in, which hopefully match the transport format so that no transform is needed, and at least is no higher than the max spec you expect to transport.

Does that sound correct?

That is correct, and a much better summary of the feature than I managed to write.
As an example of an use case for resolution and framerate:
In my application, a specific setup part between two peers is that I need to specify the resolution and framerate that I intend to transmit, such that the receiver can use those values for internal UI setup or other video related things. Since I'm not in control of the resolution, nor framerate, I have to guess/hardcode these values for now to something generic, say 1280x720x30.
Of course, the CPU utilization savings are a big part of why I would like control of the resolution, but there could also be other reasons, such as the one I mentioned above

Dumping some info from today's investigation:

The OPUS-specific RFC is here : https://tools.ietf.org/html/draft-spittka-payload-rtp-opus-03#section-6.2

The OPUS-specific constants, and all others as well, are in mediaconstants.cc.

The SDP parsing of constants for all codecs is done in webrtcsdp.cc

Encoder code, with actually supported parameters, is in audio_encoder_opus.cc. But it doesn't use the media constants, it's just hard-coded strings.

It seems a priori from those info that it would be possible to support some SDP filtering for OPUS, possibly even a more generic format for other codecs.

Not sure about if it has any effect on performance, it could also be interesting to be able to modify lines 275-276 and 291-291 in peer_connection.cpp
options.offer_to_receive_audio = true;
options.offer_to_receive_video = true;

Now this is a special case for my implementation, but I only either send OR receive video/audio on each peer connection, so previously to this plugin, I modified offer_to_receive_audio/video to false depending on the connection. I also changed sendrecv in the sdp to sendonly or recvonly depending on the use-case.
More information on sendrecv

Re-opening after GitHub's auto-close.

Closing this issue, now fixed with video capture format available via media constraints (a2a98554e59ee3b1757806670c3ccda8fc4d2464). Opened #75 for the offer_to_receive_xxx options, which are deprecated anyway.

authors: djee-ms eirikhollis

repo: microsoft/MixedReality-WebRTC


text:
get teleport target position??

hi,
how can i get teleport target position?
public void OnTeleportCompleted(TeleportEventData eventData) => position = eventData.Hotspot.Position;  ??

You may be able to go off the pointer object itself, something like public void OnTeleportCompleted(TeleportEventData eventData) => position = eventData.Pointer.Result.Details.Point;

You may be able to go off the pointer object itself, something like public void OnTeleportCompleted(TeleportEventData eventData) => position = eventData.Pointer.Result.Details.Point;

thanks for your reply :]
but is this local position or world position?

Should be in world position, as that's what we use to position the teleport cursor, but if you're seeing otherwise let me know!

@davoodkharmanzar did this answer your question? If so, may we close this issue?

Thanks!

authors: RogPodge davoodkharmanzar keveleigh

repo: microsoft/MixedRealityToolkit-Unity


text:
Selecting Input category in MixedRealityToolkit in play mode deletes UIRaycastCamera

Describe the bug
The UIRaycastCamera gets deleted wheh the Input category from the MixedRealityToolkit is selected.
To reproduce
Steps to reproduce the behavior:

Run any scene with the MRTK in it
Select the MRTK
Choose Input in the inspector
See error and that the UIRaycastCamera is missing

Expected behavior
UIRaycastCamera does not get deleted
Your Setup (please complete the following information)

Unity Version 2018.4.2f1
MRTK Version development branch

@Alexees how recently have you pulled the development branch? This sounds like #4944 which was recently fixed by #4991 . Can you please check you pull latest and close this issue if no longers repos? Thanks!

Yes, gone. I was exactly 6 commits behind and the oldest was it.

authors: Alexees Troy-Ferrell


text:
How to use localized speech commands

Describe the issue
It's not clear in the documentation how to use Localized Speech Commands.
Feature area
I've seen that under Input Profile >Speech > Speech Commands there is some LocalizationKey.
By looking at the SpeechCommands struct, I figured out that if my build target a UWP plateform, it will load the string from the resource file instead of the given keyword in the Editor.
But how can I create such a resource file ? Is it possible within Unity, or should I open visual studio after every build to create one ? If so, how can I do it ?
Existing doc link
https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/Input/Speech.html

I could have sworn that I wrote up a similar response on some other issue but now can't find it - in anycase, having speech commands in a different language is definitely something that isn't easy to do out of the box.
Note that MRTK uses Unity's KeywordRecognizer (and DictationRecognizer) which under the covers uses the the UWP speech APIs which are documented here:
https://docs.microsoft.com/en-us/windows/uwp/design/input/specify-the-speech-recognizer-language
A quick google search "KeywordRecognizer language" finds that yeah folks have had some trouble getting this to work on non english (or in general, a language not matching the language of their OS).
In order to make this work I suspect it's possible, you just have to go around Unity APIs and directly invoke the UWP APIs (see link https://docs.microsoft.com/en-us/windows/uwp/design/input/specify-the-speech-recognizer-language above) before the MRTK starts its speech system (or in general, before the first instance of Unity's speech stuff kicking off)

If you end up having some luck here, would definitely love to hear - we can keep this issue open to track some future investigations + results.

I just tested using MRTK 2.7.2 with Spanish Language and works. First, I added the Spanish Language Pack to the OS as shown in the picture, then I configured de MixedRealityInputSystemProfile in the Unity Scene under the speech section, adding in the property LocalizationKey the value es-ES. In fact, I tested speech commands in Spanish and English in the same scene successfully.

authors: amonnot-ah hernancasta wiwei


text:
Better handling for excess proximity lights

Overview
Previously, if the shader's maximum proximity light limit was reached, a warning would be logged. I've changed this to keep all lights around on the CPU side and only send the first two valid ones to the shader. This helps with race conditions where multiple lights are added and removed in a single frame and eliminates a warning that was spamming to the log and degrading performance.

Hey @Cameron-Micka, you've been working on proximity light improvements. Is this something you've considered?

Hey @RogPodge. Sorry, I misspoke in the PR description when I said the lights were discarded. The main problem we were trying to solve was a spike that we were seeing when hands were raised and lowered and this warning was logged several times.
If you're concerned about removing the logging entirely, perhaps it would be better if it were only in the editor since it's not particularly obvious when this warning is triggered on the device?

@Cameron-Micka, can you ensure this gets ported to the v3 shaders?

@RogPodge, can you also follow up with @jonathoncobb

@Cameron-Micka, can you ensure this gets ported to the v3 shaders?

Of course!

Reminder to self, we should be able to get this in for 2.8 :)

/azp run

authors: Cameron-Micka RogPodge davidkline-ms jonathoncobb polar-kev


text:
Progress Indicator Demos, Docs & Fixes

Overview
Adds documentation for progress indicators, adds a demo scene, fixes some bugs.

Changes

Fixes: #7560, #7559

Verification

As a reviewer, it is possible to check out this change locally by using the following
commands (substituting {PR_ID} with the ID of this pull request):
git fetch origin pull/{PR_ID}/head:name_of_local_branch
git checkout name_of_local_branch

/azp run

/azp run

@Railboy, can you review the comments from @Troy-Ferrell? Thanks!

/azp run

authors: Railboy davidkline-ms keveleigh


text:
understanding tensorflow autoencoder

Hello again,
I'm trying to run the Tensorflow implementation for the autoencoder and I have a few questions (excuse me it got too long but i tried to be detailed).
In the repo you have a config file (ae_resnet.yaml) for running a resnet with ".points" tfrecords. I have run this and using the decode_shape gives me pretty good results! However, this is not the case when I try to run an autoencoder with ocnn and ".octree" tfrecords.
Here are the steps I followed:
Step 1. i used python data/completion.py --run generate_dataset which downloaded the ".points" files of the completion dataset and generated:
shape.ply
shape.points
test.scans.ply
test.scans.points
completion_test_points.tfrecords
completion_test_scans_points.tfrecords
completion_train_points.tfrecords
completion_train_points.camera_path.dict
filelist_test.txt
filelist_test_scans.txt
filelist_train.txt

Step 2. under each category folder in shape.points i generated a list.txt that includes the paths of its points and run octree --filenames category/list.txt --output_path shape.octrees/category --depth 6 --split_label 1 --rot_num 6
Here, I used --depth 6 and --split_label 1 because I saw these parameters in ae_resnet.yaml.
Step 3. similarly to filelist_test.txt and filelist_train.txt I generated files that include paths of the corresponding octrees (for each .points file path i included the 6 .octree file paths) and then I made octree tfrecords. Specifically these corresponding files:
completion_test_octrees.tfrecords
completion_test_scans_octrees.tfrecords
completion_train_octrees.tfrecords

Step 4. i run the autoencoder with ae_ocnn.yaml:
SOLVER:
  gpu: 0,
  logdir: /output/ocnn_completion/ae/ocnn_b16
  run: train
  max_iter: 20000
  test_iter: 336
  test_every_iter: 400
  step_size: (80000,)
  ckpt_num: 20

DATA:
  train:
    dtype: octree
    depth: 6
    location: /completion_train_octrees.tfrecords
    batch_size: 16
    distort: False
    offset: 0.0
    node_dis: True
    split_label: True

  test:
    dtype: octree
    depth: 6
    location: /completion_test_octrees.tfrecords
    batch_size: 16
    distort: False
    offset: 0.0
    node_dis: True
    split_label: True

MODEL:
  name: ocnn
  channel: 3
  nout: 32
  depth: 6

LOSS:
  weight_decay: 0.0005

Results I got
What I noticed is that the resnet with points is taking more time to do the same amount of batch iterations (1 hour equals to 2800 iterations in resnet and 12000 iterations in ocnn) and the accuracy and loss of resnet are much smoother than the ones in ocnn. e.g.
resnet:

ocnn:

Also, the decoded octrees from resnet are much better than the ones from ocnn: e.g.
at 6K batch iterations of resnet:

at 20K batch iterations of ocnn:

Afterwards I realized I can run the ocnn model with the ".points" tfrecords by using ae_resnet.yaml and chaning model name from octree to ocnn. This gave reasonable results, but I still dont understand why creating octrees and giving those as input didn't work.
Below I list my questions:
Q1.1 Is my configuration for ocnn autoencoder wrong? Is it expected that ocnn runs much faster because its more efficient than resnet? Do i have to train it for much longer time in order to get as good results as with resnet?
Q1.2 Regarding the input signal, in ae_resnet.yaml the input channel is 4 and I realized after some time that this is due to node_dis: True. Can you explain what this is?
Q1.3 When I run the decode_shape mode, and convert the input octrees into mesh with octree2mesh the original shape i get from ae_resnet.yaml and ae_ocnn.yaml are slightly different. Is this because I used 6 rotations in the octree generation thus octree with suffix "_6_2_000.octree" is not entirely same as octree created from point-cloud?
original image in resnet:

original image in ocnn:

Q2. When I tried to run autoencoder with adaptive points or octrees it happened that loss4 was extremely big resulting in total loss of nan and then training was stuck? Do you have a hint what the problem is?  (note: for adaptive octrees i used the step 2 mentioned above with extra argument --adaptive 4)
Adaptive resnet:
test summaries:

config:
SOLVER:
  gpu: 0,
  logdir: /media/christina/Data/ANFASS_data/O-CNN/output/ocnn_completion/ae/aresnet_b16
  run: train
  max_iter: 20000
  test_iter: 336
  test_every_iter: 400
  step_size: (80000,)
  ckpt_num: 20

DATA:
  train:
    dtype: points
    depth: 6
    location: /media/christina/Data/ANFASS_data/O-CNN/ocnn_completion/completion_train_points.tfrecords
    batch_size: 16
    distort: False
    offset: 0.0
    node_dis: True
    split_label: True
    adaptive: True

  test:
    dtype: points
    depth: 6
    location: /media/christina/Data/ANFASS_data/O-CNN/ocnn_completion/completion_test_points.tfrecords
    batch_size: 16
    distort: False
    offset: 0.0
    node_dis: True
    split_label: True
    adaptive: True

MODEL:
  name: resnet
  channel: 4
  nout: 32   # The channel of the hidden code, the code length is 4*4*4*32 (2048)
  depth: 6

LOSS:
  weight_decay: 0.0005

Adaptive ocnn:
test summaries:

config:
SOLVER:
  gpu: 0,
  logdir: /media/christina/Data/ANFASS_data/O-CNN/output/ocnn_completion/ae/aocnn_b16
  run: train
  max_iter: 20000
  test_iter: 336
  test_every_iter: 400
  step_size: (80000,)
  ckpt_num: 20

DATA:
  train:
    dtype: octree
    depth: 6
    location: /media/christina/Data/ANFASS_data/O-CNN/ocnn_completion/completion_train_aoctrees.tfrecords
    batch_size: 16
    distort: False
    offset: 0.0
    node_dis: True
    split_label: True

  test:
    dtype: octree
    depth: 6
    location: /media/christina/Data/ANFASS_data/O-CNN/ocnn_completion/completion_test_aoctrees.tfrecords
    batch_size: 16
    distort: False
    offset: 0.0
    node_dis: True
    split_label: True

MODEL:
  name: ocnn
  channel: 3
  nout: 32   # The channel of the hidden code, the code length is 4*4*4*32 (2048)
  depth: 6

LOSS:
  weight_decay: 0.0005

Q3. What does the output of check_octree means:
example of an octree generated with octree ... --depth 6 --split_label 1 --rot_num 6:
===============
../ocnn_completion/.../02691156/1a04e3eab45ca15dd86060f189eb133_6_2_000.octree infomation:
This is a valid octree!
magic_str:_OCTREE_1.0_
batch_size: 1
depth: 6
full_layer: 2
adaptive_layer: 4
threshold_distance: 2
threshold_normal: 0.1
is_adaptive: 0
has_displace: 0
nnum: 1 8 64 224 1000 4336 17320 0 0 0 0 0 0 0 0 0
nnum_cum: 0 1 9 73 297 1297 5633 22953 22953 0 0 0 0 0 0 0
nnum_nempty: 1 8 28 125 542 2165 8177 0 0 0 0 0 0 0 0 0
total_nnum: 22953
total_nnum_capacity: 22953
channel: 1 1 0 3 0 1
locations: -1 -1 0 6 0 -1
bbox_max: 128.635 130.831 120.122
bbox_min: 17.2621 19.4578 8.74912
key2xyz: 0
sizeof_octree: 483992
===============

Q3.1. i guess adaptive_layer:4 is dummy because of is_adaptive: 0 ?!
Q3.2. what is the locations parameter showing?
Q3.3 what is the channel parameter showing? I understood it corresponds to split, label, feature, xyz, index and one more property...which is this property and what is their order?

thank you!

repo: microsoft/O-CNN


text:
Speed up OneDrive

Pull Request (PR) description
Reduced calls to Set-SPOTenant
This Pull Request (PR) fixes the following issues
- Fixes #102

@desmay Please update your fork/branch with latest changes. It will help resolve some of the tests that fail.

Codecov Report

Merging #105 into Dev will decrease coverage by 0.21%.
The diff coverage is 93.33%.

@@            Coverage Diff             @@
##              Dev     #105      +/-   ##
==========================================
- Coverage   89.23%   89.01%   -0.22%
==========================================
  Files          36       36
  Lines        2962     2950      -12
  Branches        4        4
==========================================
- Hits         2643     2626      -17
- Misses        315      320       +5
  Partials        4        4

Impacted Files
Coverage Δ

.../DSCResources/MSFT_ODSettings/MSFT_ODSettings.psm1
87.37% <93.33%> (-5.67%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb9e0d6...2486652. Read the comment docs.

authors: NikCharlebois codecov-io desmay


text:
Attempt remove net472 as a targets for packages

Theoretically, we don't need net472 as a target anymore since it could use netstandard2.0 instead, we'll see how it will work :)

If you search the repo for "net472", you can see some other items related to this framework -- references in .props files and a very small code workaround to facilitate use of the framework. Both of these should be possible to remove now with this change.

If you search the repo for "net472", you can see some other items related to this framework -- references in .props files and a very small code workaround to facilitate use of the framework. Both of these should be possible to remove now with this change.

Thank you for noticing that. Unfortunately, some of them are needed but was able to unify some of our targets

authors: AndreyTretyak muiriswoulfe

repo: microsoft/Omex


text:
Where can I download dataset?

Where can I download dataset?

Same question! I would also be interested in the DeBERTa model you trained for the token-level classification.

Same question! I would also be interested in the DeBERTa model you trained for the token-level classification.
Do you have it? I am also interested in it.

Where can I download dataset?

Do you get the paraTag dataset? I also want to research it!

Are you kidding? When did Microsoft do anything well? Maybe Xenix? Their products have more and more bugs with each subsequent version. They do not know how to solve them, or they do not care, and new ones emerge. The only MS product I use, admit, and advocate is MS Lens on a smartphone.
Good luck!

authors: AnnaWegmann ilusm603 tvrbanec

repo: microsoft/ParaTag


text:
Jmstall/fix1511

Fix for race condition in #1511

✅ No public API change.

✅ No public API change.

✅ No public API change.

authors: LucGenetier MikeStall

repo: microsoft/Power-Fx


text:
Initial work on binder refactors for DataflowGraph

Dataflow Graph changes to help reduce duplicate code.

:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Connectors.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Core.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Interpreter.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.LanguageServerProtocol.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Transport.Attributes.dll

:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Connectors.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Core.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Interpreter.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.LanguageServerProtocol.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Transport.Attributes.dll

:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Connectors.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Core.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Interpreter.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.LanguageServerProtocol.dll
:heavy_check_mark: No Binary Compatibility issues for Microsoft.PowerFx.Transport.Attributes.dll

authors: BruceHaley bryancroteau-MSFT


text:
Error unpacking some PowerApps

We're currently unable to unpack a few of our apps on PowerApps v3.22015.22.
When unpacking a few of our PowerApps, we received a few of the following errors which caused unpacking to fail.
PA3001: Internal error. An item with the same key has already been added. Key: groupContainer
We are running PASopa.exe from this repository, not using the Power Apps CLI.
It seems, from our perspective, that the issue is from the new-ish containers but would like some confirmation and advice on getting this working again.

@IC-Jonathan-Roussel - Do you have an MSApp that we can use to reproduce the issue? If so, please send it to tschneider at microsoft.com.

This issue is a duplicate of 317; feel free to move discussions there! https://github.com/microsoft/PowerApps-Language-Tooling/issues/317

authors: IC-Jonathan-Roussel jt000 landonarogers

repo: microsoft/PowerApps-Language-Tooling