text:
Unable to install dependencies

Hello,
I have been trying to reinstall CyberBattleSim without any luck until now.
I am unable to either install dependencies with pip install -r requirements.dev.txt, run bash init.sh or even building the included docker container with the Dockerfile, indicating that this must be a general problem and not a problem with my specific installation of python/conda.
It takes all 3 of these commands about 7 hours to run, with pip taking increasingly long time to determine a compatible version of the requirements, ultimately failing with the following error (also including the last logging information from the installation process):
From pip install -r requirements.dev.txt
Collecting async-lru>=1.0.0
  Using cached async_lru-2.0.3-py3-none-any.whl (6.0 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached async_lru-2.0.2-py3-none-any.whl (5.7 kB)
  Using cached async_lru-2.0.1-py3-none-any.whl (5.4 kB)
  Using cached async_lru-2.0.0-py3-none-any.whl (5.4 kB)
  Using cached async_lru-1.0.3-py3-none-any.whl (4.3 kB)
  Using cached async_lru-1.0.2.tar.gz (5.0 kB)
  Preparing metadata (setup.py) ... done
  Using cached async_lru-1.0.1.tar.gz (4.9 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of async-lru to determine which version is compatible with other requirements. This could take a while.
  Using cached async_lru-1.0.0.tar.gz (4.9 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of asttokens to determine which version is compatible with other requirements. This could take a while.
Collecting asttokens>=2.1.0
  Using cached asttokens-2.4.0-py2.py3-none-any.whl (27 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached asttokens-2.3.0-py2.py3-none-any.whl (27 kB)
  Using cached asttokens-2.2.1-py2.py3-none-any.whl (26 kB)
  Using cached asttokens-2.2.0-py2.py3-none-any.whl (26 kB)
  Using cached asttokens-2.1.0-py2.py3-none-any.whl (26 kB)
INFO: pip is looking at multiple versions of anyio to determine which version is compatible with other requirements. This could take a while.
Collecting anyio>=3.1.0
  Using cached anyio-3.7.1-py3-none-any.whl (80 kB)
INFO: pip is looking at multiple versions of asttokens to determine which version is compatible with other requirements. This could take a while.
  Using cached anyio-3.7.0-py3-none-any.whl (80 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
  Using cached anyio-3.6.1-py3-none-any.whl (80 kB)
  Using cached anyio-3.6.0-py3-none-any.whl (80 kB)
  Using cached anyio-3.5.0-py3-none-any.whl (79 kB)
  Using cached anyio-3.4.0-py3-none-any.whl (78 kB)
INFO: pip is looking at multiple versions of anyio to determine which version is compatible with other requirements. This could take a while.
  Using cached anyio-3.3.4-py3-none-any.whl (78 kB)
  Using cached anyio-3.3.3-py3-none-any.whl (78 kB)
  Using cached anyio-3.3.2-py3-none-any.whl (78 kB)
  Using cached anyio-3.3.1-py3-none-any.whl (77 kB)
  Using cached anyio-3.3.0-py3-none-any.whl (77 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached anyio-3.2.1-py3-none-any.whl (75 kB)
  Using cached anyio-3.2.0-py3-none-any.whl (75 kB)
  Using cached anyio-3.1.0-py3-none-any.whl (74 kB)
INFO: pip is looking at multiple versions of wheel to determine which version is compatible with other requirements. This could take a while.
Collecting wheel
  Using cached wheel-0.41.3-py3-none-any.whl (65 kB)
  Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
  Using cached wheel-0.41.1-py3-none-any.whl (64 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/enzo/miniconda3/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 393, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000

From building Dockerfile:
22328.7 Collecting anyio>=3.1.0
22328.7   Downloading anyio-3.7.1-py3-none-any.whl (80 kB)
22328.7 INFO: pip is looking at multiple versions of asttokens to determine which version is compatible with other requirements. This could take a while.
22328.7   Downloading anyio-3.7.0-py3-none-any.whl (80 kB)
22328.7 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
22328.7   Downloading anyio-3.6.2-py3-none-any.whl (80 kB)
22328.7   Downloading anyio-3.6.1-py3-none-any.whl (80 kB)
22328.7   Downloading anyio-3.6.0-py3-none-any.whl (80 kB)
22328.7   Downloading anyio-3.5.0-py3-none-any.whl (79 kB)
22328.7   Downloading anyio-3.4.0-py3-none-any.whl (78 kB)
22328.7 INFO: pip is looking at multiple versions of anyio to determine which version is compatible with other requirements. This could take a while.
22328.7   Downloading anyio-3.3.4-py3-none-any.whl (78 kB)
22328.7   Downloading anyio-3.3.3-py3-none-any.whl (78 kB)
22328.7   Downloading anyio-3.3.2-py3-none-any.whl (78 kB)
22328.7   Downloading anyio-3.3.1-py3-none-any.whl (77 kB)
22328.7   Downloading anyio-3.3.0-py3-none-any.whl (77 kB)
22328.7 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
22328.7   Downloading anyio-3.2.1-py3-none-any.whl (75 kB)
22328.7   Downloading anyio-3.2.0-py3-none-any.whl (75 kB)
22328.7   Downloading anyio-3.1.0-py3-none-any.whl (74 kB)
22328.7 INFO: pip is looking at multiple versions of tinycss2 to determine which version is compatible with other requirements. This could take a while.
22328.7 Collecting tinycss2
22328.7   Downloading tinycss2-1.2.0-py3-none-any.whl (21 kB)
22328.7   Downloading tinycss2-1.1.1-py3-none-any.whl (21 kB)
22328.7   Downloading tinycss2-1.1.0-py3-none-any.whl (21 kB)
22328.7
22328.7 failed
22328.7
22328.7
22328.7 ==> WARNING: A newer version of conda exists. <==
22328.7   current version: 23.9.0
22328.7   latest version: 23.10.0
22328.7
22328.7 Please update conda by running
22328.7
22328.7     $ conda update -n base -c conda-forge conda
22328.7
22328.7 Or to minimize the number of packages updated during conda update use
22328.7
22328.7      conda install conda=23.10.0
22328.7
22328.7
22328.7 Pip subprocess error:
22328.7 ERROR: Exception:
22328.7 Traceback (most recent call last):
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
22328.7     status = run_func(*args)
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
22328.7     return func(self, options, args)
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 338, in run
22328.7     requirement_set = resolver.resolve(
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
22328.7     result = self._result = resolver.resolve(
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
22328.7     state = resolution.resolve(requirements, max_rounds=max_rounds)
22328.7   File "/opt/miniconda/envs/cybersim/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 393, in resolve
22328.7     raise ResolutionTooDeep(max_rounds)
22328.7 pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2000000
22328.7
22328.7
22328.7 CondaEnvException: Pip failed
22328.7
------
Dockerfile:12
--------------------
  10 |     ADD *.txt ./
  11 |     ADD *.yml ./
  12 | >>> RUN conda env create -f env.yml
  13 |     RUN conda init bash
  14 |     ENV PATH /opt/miniconda/envs/cybersim/bin:$PATH
--------------------
ERROR: failed to solve: process "/bin/sh -c conda env create -f env.yml" did not complete successfully: exit code: 1

@Zone04 Thanks for reporting. Your logs show that your version of conda is 23.9.0. Could you confirm whether you can repro with version 23.5.2?

@blumu The reported version is from the docker building process, done without modifying the provided Dockerfile.
Running bash init.sh on the host with conda 23.9.0 gave the first logs, 23.10.0 gave similar results.
I downgraded conda to 23.5.2 as per your request, bash init.sh is currently running but is now stuck on "Installing pip dependencies" and I suspect that it will crash with the same error in a few hours.
I will let you know when it ends, either with a succes or a failure.

Thank you for trying it. I'm suspecting a regression in the base docker image. A potential workaround, assuming it's still available, is to use the docker image from the last successful github action run: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04@sha256:c3a0158d43baedab356a634312da5ee3135bb99efea4080147e8d0d5dde17425

I've stopped the previous attempt, which was still stuck at "Installing pip dependencies" to try with the image you suggested.
If this can be of any help, here is an export of the conda environment a colleague successfully created few months ago with the init.sh script. When I tried recreating the environment from this file, I ran into the same error as in #100 because the default version of setuptools in a new environment was version 68, so I had to manually downgrade setuptools in this new environment before updating the environment with the file.
name: cybersim
channels:
  - plotly
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - blas=1.0=mkl
  - brotli-python=1.1.0=py39h3d6467e_1
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2023.08.22=h06a4308_0
  - certifi=2023.7.22=py39h06a4308_0
  - charset-normalizer=3.3.1=pyhd8ed1ab_0
  - gmp=6.2.1=h295c915_3
  - gmpy2=2.1.2=py39heeb90bb_0
  - icu=73.2=h59595ed_0
  - idna=3.4=pyhd8ed1ab_0
  - importlib-metadata=6.8.0=pyha770c72_0
  - importlib_metadata=6.8.0=hd8ed1ab_0
  - intel-openmp=2023.1.0=hdb19cb5_46306
  - joblib=1.2.0=py39h06a4308_0
  - jupyter_client=8.6.0=py39h06a4308_0
  - jupyter_core=5.4.0=py39hf3d152e_0
  - kernel-headers_linux-64=2.6.32=he073ed8_16
  - ld_impl_linux-64=2.40=h41732ed_0
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=13.2.0=h807b86a_2
  - libgfortran-ng=11.2.0=h00389a5_1
  - libgfortran5=11.2.0=h1234567_1
  - libgomp=13.2.0=h807b86a_2
  - libnsl=2.0.1=hd590300_0
  - libsodium=1.0.18=h36c2ea0_1
  - libsqlite=3.43.2=h2797004_0
  - libstdcxx-ng=13.2.0=h7e041cc_2
  - libuuid=2.38.1=h0b41bf4_0
  - libuv=1.46.0=hd590300_0
  - libzlib=1.2.13=hd590300_5
  - mkl=2023.1.0=h213fc3f_46344
  - mkl-service=2.4.0=py39h5eee18b_1
  - mkl_fft=1.3.8=py39h5eee18b_0
  - mkl_random=1.2.4=py39hdb19cb5_0
  - mpc=1.1.0=h10f8cd9_1
  - mpfr=4.0.2=hb69a4c5_1
  - mpmath=1.3.0=py39h06a4308_0
  - nb_conda_kernels=2.3.1=py39hf3d152e_2
  - ncurses=6.4=hcb278e6_0
  - nodeenv=1.8.0=pyhd8ed1ab_0
  - nodejs=20.8.1=h1990674_0
  - numpy-base=1.26.0=py39hb5e798b_0
  - openssl=3.1.4=hd590300_0
  - pip=21.3.1=pyhd8ed1ab_0
  - pixman-cos6-x86_64=0.32.8=h9b0a68f_1105
  - platformdirs=3.11.0=pyhd8ed1ab_0
  - plotly-orca=1.3.1=1
  - pyright=1.1.331=py39hd1e30aa_1
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.9.18=h0755675_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=4_cp39
  - pyzmq=25.1.1=py39h8c080ef_2
  - readline=8.2=h8228510_1
  - requests=2.31.0=pyhd8ed1ab_0
  - scikit-learn=1.3.0=py39h1128e8f_0
  - scipy=1.11.3=py39h5f9d8c6_0
  - six=1.16.0=pyh6c4a22f_0
  - sympy=1.11.1=py39h06a4308_0
  - sysroot_linux-64=2.12=he073ed8_16
  - tbb=2021.8.0=hdb19cb5_0
  - tenacity=8.2.3=pyhd8ed1ab_0
  - threadpoolctl=2.2.0=pyh0d69192_0
  - tk=8.6.13=h2797004_0
  - tornado=6.3.3=py39hd1e30aa_1
  - traitlets=5.11.2=pyhd8ed1ab_0
  - typing-extensions=4.8.0=hd8ed1ab_0
  - typing_extensions=4.8.0=pyha770c72_0
  - urllib3=2.0.7=pyhd8ed1ab_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xorg-x11-server-common-cos6-x86_64=1.17.4=h9b0a68f_1105
  - xorg-x11-server-xvfb-cos6-x86_64=1.17.4=h9b0a68f_1105
  - xz=5.2.6=h166bdaf_0
  - zeromq=4.3.5=h59595ed_0
  - zlib=1.2.13=hd590300_5
  - pip:
      - absl-py==2.0.0
      - ale-py==0.7.4
      - ansiwrap==0.8.4
      - anyio==4.0.0
      - argon2-cffi==23.1.0
      - argon2-cffi-bindings==21.2.0
      - arrow==1.3.0
      - asciichart==0.1
      - asciichartpy==1.5.25
      - asttokens==2.4.0
      - async-lru==2.0.4
      - attrs==19.2.0
      - autopep8==2.0.4
      - autorom==0.4.2
      - autorom-accept-rom-license==0.6.1
      - babel==2.13.1
      - backcall==0.2.0
      - beautifulsoup4==4.12.2
      - black==23.10.1
      - bleach==6.1.0
      - boolean-py==4.0
      - cachetools==5.3.2
      - cffi==1.16.0
      - cfgv==3.4.0
      - click==8.1.7
      - cloudpickle==3.0.0
      - comm==0.1.4
      - contourpy==1.1.1
      - cycler==0.12.1
      - debugpy==1.8.0
      - decorator==4.3.0
      - defusedxml==0.7.1
      - distlib==0.3.7
      - entrypoints==0.4
      - exceptiongroup==1.1.3
      - executing==2.0.0
      - farama-notifications==0.0.4
      - fastjsonschema==2.18.1
      - filelock==3.12.4
      - flake8==6.1.0
      - fonttools==4.43.1
      - fqdn==1.5.1
      - google-auth==2.23.4
      - google-auth-oauthlib==1.1.0
      - grpcio==1.59.2
      - gym==0.21.0
      - gymnasium==0.29.1
      - identify==2.5.30
      - importlib-resources==6.1.1
      - iniconfig==2.0.0
      - ipykernel==6.6.1
      - ipython==8.16.1
      - ipython-genutils==0.2.0
      - ipywidgets==8.1.1
      - isoduration==20.11.0
      - jedi==0.19.1
      - jinja2==3.1.2
      - json5==0.9.14
      - jsonpointer==2.4
      - jsonschema==4.17.3
      - jupyter==1.0.0
      - jupyter-client==7.4.9
      - jupyter-console==6.4.4
      - jupyter-events==0.6.3
      - jupyter-lsp==2.2.0
      - jupyter-server==2.9.0
      - jupyter-server-terminals==0.4.4
      - jupyterlab==4.0.7
      - jupyterlab-pygments==0.2.2
      - jupyterlab-server==2.24.0
      - jupyterlab-widgets==3.0.9
      - jupytext==1.6.0
      - keyboard==0.13.5
      - kiwisolver==1.4.5
      - markdown==3.5.1
      - markdown-it-py==0.5.8
      - markupsafe==2.1.3
      - matplotlib==3.6.3
      - matplotlib-inline==0.1.6
      - mccabe==0.7.0
      - mistune==3.0.2
      - mypy-extensions==1.0.0
      - nbclient==0.8.0
      - nbconvert==7.9.2
      - nbformat==5.9.2
      - nest-asyncio==1.5.8
      - networkx==2.8.6
      - notebook==7.0.6
      - notebook-shim==0.2.3
      - numpy==1.23.0
      - nvidia-cublas-cu11==11.10.3.66
      - nvidia-cuda-nvrtc-cu11==11.7.99
      - nvidia-cuda-runtime-cu11==11.7.99
      - nvidia-cudnn-cu11==8.5.0.96
      - oauthlib==3.2.2
      - opencv-python==4.8.1.78
      - ordered-set==4.0.2
      - overrides==7.4.0
      - packaging==23.2
      - pandas==2.1.1
      - pandocfilters==1.5.0
      - papermill==2.3.3
      - parso==0.8.3
      - pathspec==0.11.2
      - pep8-naming==0.13.3
      - pexpect==4.8.0
      - pickleshare==0.7.5
      - pillow==10.1.0
      - plotly==4.11.0
      - pluggy==1.3.0
      - pre-commit==3.5.0
      - progressbar2==3.51.4
      - prometheus-client==0.17.1
      - prompt-toolkit==3.0.39
      - protobuf==3.19.6
      - psutil==5.7.2
      - ptyprocess==0.7.0
      - pure-eval==0.2.2
      - py==1.11.0
      - pyasn1==0.5.0
      - pyasn1-modules==0.3.0
      - pycodestyle==2.11.1
      - pycparser==2.21
      - pyflakes==3.1.0
      - pygments==2.16.1
      - pyparsing==3.1.1
      - pyqt5==5.15.10
      - pyqt5-qt5==5.15.2
      - pyqt5-sip==12.13.0
      - pyqtgraph==0.13.3
      - pyrsistent==0.19.3
      - pytest==6.2.5
      - python-json-logger==2.0.7
      - python-utils==3.8.1
      - pytz==2023.3.post1
      - pyyaml==6.0.1
      - qtconsole==5.4.4
      - qtpy==2.4.1
      - requests-oauthlib==1.3.1
      - retrying==1.3.4
      - rfc3339-validator==0.1.4
      - rfc3986-validator==0.1.1
      - rsa==4.9
      - send2trash==1.8.2
      - setuptools==65.5.1
      - shimmy==1.3.0
      - simpy==4.1.1
      - sniffio==1.3.0
      - soupsieve==2.5
      - stable-baselines3==2.2.1
      - stack-data==0.6.3
      - tabulate==0.8.10
      - tensorboard==2.15.1
      - tensorboard-data-server==0.7.2
      - terminado==0.17.1
      - textwrap3==0.9.2
      - tinycss2==1.2.1
      - toml==0.10.2
      - tomli==2.0.1
      - torch==1.13.1
      - tqdm==4.66.1
      - types-python-dateutil==2.8.19.14
      - tzdata==2023.3
      - uri-template==1.3.0
      - virtualenv==20.24.6
      - wcwidth==0.2.8
      - webcolors==1.13
      - webencodings==0.5.1
      - websocket-client==1.6.4
      - werkzeug==3.0.1
      - widgetsnbextension==4.0.9
      - zipp==3.8.1

The building process from the given image has now been stuck on the pip dependencies for more than 10 minutes, I highly suspect that this will not complete either.

docker_output.txt
As expected, it failed after 7 hours while installing pip dependencies in the conda environment creation step from the Dockerfile. Please find the complete output attached.

@zone04 Just to make sure I understand correctly were you able to make it work using the env.yaml file shared by your colleague? And if yes, did you have to downgrade setuptools in your system python environment prior to running conda env create or by just adding the setuptools depdency to the env.yaml file?

@blumu Using the env.yml file I linked before, creating the conda env fails when pip installs gym. I then have the "cybersim" environment with only part of the dependencies installed, and among these dependcies I have setuptools version 68. I then manually downgrade to version 65, and update the environment using conda env update --file env.yml, which this time succeeds

@Zone04 Thank you for sharing the logs and info. Could you try branch https://github.com/microsoft/CyberBattleSim/tree/wiblum/dep see if it works for you? https://github.com/microsoft/CyberBattleSim/pull/119/

Will do tomorrow when I'll be back on my work computer

The docker container was successfully built and running the docker run command from README worked without any problem.
Manually creating the conda environment with the included requirements was also successful, as well as running the init.sh script. All tests passed.
Thank you very much

@Zone04 Thank you for confirming, I'll go ahead and merge the PR.

repo: microsoft/CyberBattleSim


text:
Changing the length of an VARCHAR or NVARCHAR column that has explicit COLLATE defined that matches the database default COLLATION will trigger a data loss check in the script output

SqlPackage or DacFx Version:  <Sdk Name="Microsoft.Build.Sql" Version="0.1.15-preview" />
.NET Framework (Windows-only) or .NET Core: MSBuild version 17.3.4+a400405ba for .NET
Environment (local platform and source/target platforms): Local SQL 2022 running in docker for testing. This is the same behavior we are seeing on our Windows 2019 servers with SQL Server 2022 Enterprise Edition.

Steps to Reproduce:
Changing the length of an VARCHAR or NVARCHAR column that has explicit COLLATE defined that matches the database default COLLATION will trigger a data loss check in the script output.
Expected Behavior
A VARCHAR or NVARCHAR expansion should be a metadata only operation and should not trigger a data loss warning.
Database Project file (CollationTest.sqlproj)

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build">
  <Sdk Name="Microsoft.Build.Sql" Version="0.1.15-preview" />
  <PropertyGroup>
    <Name>CollationTest</Name>
    <DSP>Microsoft.Data.Tools.Schema.Sql.Sql160DatabaseSchemaProvider</DSP>
    <ModelCollation>1033, CI</ModelCollation>
  </PropertyGroup>
  <ItemGroup>
    <Build Remove="out\**" />
  </ItemGroup>
</Project>

Table Definition (TestTable.sql)

CREATE TABLE [dbo].[TestTable]
(
    [TestTable_ID]   [bigint] IDENTITY (1,1) NOT NULL ,
    -- Test collation that is the same as the DB default
    [SameCollation]  [varchar] (64) COLLATE SQL_Latin1_General_CP1_CI_AS ,
    [SameCollationN] [nvarchar](64) COLLATE SQL_Latin1_General_CP1_CI_AS ,
    -- Test collation that is different than the DB default
    [DiffCollation]  [varchar] (64) COLLATE SQL_Latin1_General_CP1_CS_AS ,
    [DiffCollationN] [nvarchar](64) COLLATE SQL_Latin1_General_CP1_CS_AS ,
    CONSTRAINT [TestTable_PK] PRIMARY KEY CLUSTERED ([TestTable_ID] ASC) ON [PRIMARY]
)
GO

Publish Profile (publish.xml)

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <ProfileVersionNumber>1</ProfileVersionNumber>
    <ScriptDatabaseOptions>false</ScriptDatabaseOptions>
    <DoNotDropCredentials>true</DoNotDropCredentials>
    <DoNotDropDatabaseRoles>true</DoNotDropDatabaseRoles>
    <DoNotDropExtendedProperties>true</DoNotDropExtendedProperties>
    <DoNotDropFilegroups>true</DoNotDropFilegroups>
    <DoNotDropFiles>true</DoNotDropFiles>
    <DoNotDropRoleMembership>true</DoNotDropRoleMembership>
    <DoNotDropUsers>true</DoNotDropUsers>
    <DropExtendedPropertiesNotInSource>false</DropExtendedPropertiesNotInSource>
    <DropObjectsNotInSource>true</DropObjectsNotInSource>
    <DoNotAlterChangeDataCaptureObjects>false</DoNotAlterChangeDataCaptureObjects>
    <DoNotAlterReplicatedObjects>false</DoNotAlterReplicatedObjects>
    <ExcludeObjectTypes>
      Credentials;
      DatabaseRoles;
      ExtendedProperties;
      RoleMembership;
      Users;
      Filegroups;
      Files;
    </ExcludeObjectTypes>
    <IgnoreAuthorizer>true</IgnoreAuthorizer>
    <IgnoreColumnOrder>true</IgnoreColumnOrder>
    <IgnoreComments>true</IgnoreComments>
    <IgnoreExtendedProperties>true</IgnoreExtendedProperties>
    <IgnoreIdentitySeed>true</IgnoreIdentitySeed>
    <IgnoreIncrement>true</IgnoreIncrement>
    <IgnoreIndexOptions>true</IgnoreIndexOptions>
    <IgnorePartitionSchemes>true</IgnorePartitionSchemes>
    <IgnorePermissions>true</IgnorePermissions>
    <IgnoreRoleMembership>true</IgnoreRoleMembership>
    <IgnoreTableOptions>true</IgnoreTableOptions>
    <IgnoreWithNocheckOnCheckConstraints>true</IgnoreWithNocheckOnCheckConstraints>
    <IgnoreWithNocheckOnForeignKeys>true</IgnoreWithNocheckOnForeignKeys>
  </PropertyGroup>
</Project>

Build command
dotnet build
Sqlpackage command
sqlpackage /Action:Script /DeployScriptPath:.\out\CollationTest.dacpac.script.sql /DeployReportPath:.\out\CollationTest.dacpac.report.xml /Profile:.\publish.xml /SourceFile:.\bin\Release\CollationTest.dacpac /TargetServerName:localhost /TargetDatabaseName:CollationTest /TargetTrustServerCertificate:true /p:VerifyDeployment=false /p:BlockOnPossibleDataLoss=true /TargetUser:sqluser /TargetPassword:sqlpass
Change the VARCHAR and NVARCHAR lengths from 64 to 65 to trigger a schema modification

CREATE TABLE [dbo].[TestTable]
(
    [TestTable_ID]   [bigint] IDENTITY (1,1) NOT NULL ,
    -- Test collation that is the same as the DB default
    [SameCollation]  [varchar] (65) COLLATE SQL_Latin1_General_CP1_CI_AS ,
    [SameCollationN] [nvarchar](65) COLLATE SQL_Latin1_General_CP1_CI_AS ,
    -- Test collation that is different than the DB default
    [DiffCollation]  [varchar] (65) COLLATE SQL_Latin1_General_CP1_CS_AS ,
    [DiffCollationN] [nvarchar](65) COLLATE SQL_Latin1_General_CP1_CS_AS ,
    CONSTRAINT [TestTable_PK] PRIMARY KEY CLUSTERED ([TestTable_ID] ASC) ON [PRIMARY]
)
GO

Rerun the build and sqlpackage commands as above and the SQL script output will add a data loss detection message and check into the script for the table. It only adds the message for the 2 columns that have collation matching the database default. It does not have that warning for the 2 columns that have a different collation.
Also to note there is no table rebuild in the SQL script output. It just does a simple ALTER COLUMN as I would expect. So the warning is incorrect.

/*
The type for column SameCollation in table [dbo].[TestTable] is currently  VARCHAR (64) NULL but is being changed to  VARCHAR (65) COLLATE SQL_Latin1_General_CP1_CI_AS NULL. Data loss could occur and deployment may fail if the column contains data that is incompatible with type  VARCHAR (65) COLLATE SQL_Latin1_General_CP1_CI_AS NULL.
The type for column SameCollationN in table [dbo].[TestTable] is currently  NVARCHAR (64) NULL but is being changed to  NVARCHAR (65) COLLATE SQL_Latin1_General_CP1_CI_AS NULL. Data loss could occur and deployment may fail if the column contains data that is incompatible with type  NVARCHAR (65) COLLATE SQL_Latin1_General_CP1_CI_AS NULL.
*/
IF EXISTS (select top 1 1 from [dbo].[TestTable])
    RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127) WITH NOWAIT
GO
PRINT N'Altering Table [dbo].[TestTable]...';

GO
ALTER TABLE [dbo].[TestTable] ALTER COLUMN [DiffCollation] VARCHAR (65) COLLATE SQL_Latin1_General_CP1_CS_AS NULL;
ALTER TABLE [dbo].[TestTable] ALTER COLUMN [DiffCollationN] NVARCHAR (65) COLLATE SQL_Latin1_General_CP1_CS_AS NULL;
ALTER TABLE [dbo].[TestTable] ALTER COLUMN [SameCollation] VARCHAR (65) COLLATE SQL_Latin1_General_CP1_CI_AS NULL;
ALTER TABLE [dbo].[TestTable] ALTER COLUMN [SameCollationN] NVARCHAR (65) COLLATE SQL_Latin1_General_CP1_CI_AS NULL;

Our workaround is to just remove the explicit COLLATE statements from all of the tables where it matches the database default. We understand the explicit COLLATE definition for columns that match the default collation is unnecessary, but this feels like maybe a bug to me.
Did this occur in prior versions?  If not - which version(s) did it work in?
(DacFx/SqlPackage/SSMS/Azure Data Studio)

The data loss warning makes sense if the collation were changing (and, of course, if the length were reduced).
When the collation is explicitly included in the .sql table definition, I think SSDT incorrectly thinks that collation is being changed, even though the actual change is to enbiggen the column length.
For folks who understand what is happening, this is an annoying case of "remember to ignore that error."  But for folks who don't know this bug exists, the warning of data loss warning can cause major confusion that really torpedoes developer velocity.

source: gharchive/issue

authors: AMtwo-olo csommerolo

repo: microsoft/DacFx

license: MIT

license_type: permissive

license_source: github-api


text:
Deploy Databricks in a VNET

Deploy Databricks in a VNET defined by Terraform.
Per Databricks best practices, and usual network isolation requirements in enterprises, deployments of Databricks should be in isolated in virtual networks in order to define network policies and lock down data security.
This will increase the reusability potential of this DataOps template.

Not testing this, but going to go ahead and merge this since we are archiving the repo.

Not testing this, but going to go ahead and merge this since we are archiving the repo.

source: gharchive/pull-request

authors: algattik dtzar

repo: microsoft/DataOps


text:
[BUG] my autocast is not working

Describe the bug
A clear and concise description of what the bug is.
i'm working on https://github.com/YooSungHyun/pytorch-trainer ds_train.py
when i forward deepspeed config fp16, model weight is fp16 but input data is fp32
i know that autocast is make this well, but raised on error like this
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1836, in forward
    loss = self.module(*inputs, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/networks/models.py", line 13, in forward
    hidden, _ = self.lstm1(inputs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
    result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
RuntimeError: Input and parameter tensors are not the same dtype, found input tensor with Float and parameter tensor with Half

What did I do wrong?
To Reproduce
Steps to reproduce the behavior:

run scripts/run_train_deepspeed.sh
raised error

Expected behavior
A clear and concise description of what you expected to happen.
forward well
ds_report output
Please run ds_report to give us details about your setup.
Screenshots
If applicable, add screenshots to help explain your problem.
System info (please complete the following information):

OS: [e.g. Ubuntu 18.04]
GPU count and types [e.g. two machines with x8 A100s each]
Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
Python version
Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
Docker context
Are you using a specific docker image that you can share?
Additional context
my zero1 config like this...
{
    "fp16": {
        "enabled": true,
        "auto_cast": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "zero_optimization": {
        "stage": 1,
        "allgather_partitions": true,
        "allgather_bucket_size": 5e7,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": 5e7,
        "contiguous_gradients": true
    },
    "train_micro_batch_size_per_gpu": "auto",
    "gradient_accumulation_steps": "auto",
    "wall_clock_breakdown": false
}

maybe, that option is working with deepspeed.initialize(training_data=...) only...??
i am not initialize with deepspeed... im using torch.utils.data.Dataset` and torch's dataloader, not deepspeed wrapper

i given argument to model like model(**batch), but, deepspeed auto_cast is only working *args.

Replacing it with model(batch["inputs"]) worked for me, but I got an error in backward(). I'm also using torch optimizer for the optimizer.
Found dtype Float but expected Half
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
    scaled_loss.backward(retain_graph=retain_graph)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 2019, in backward
    self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1958, in backward
    self.optimizer.backward(loss, retain_graph=retain_graph)
  File "/data/bart/temp_workspace/pytorch-trainer/.venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/data/bart/temp_workspace/pytorch-trainer/ds_train.py", line 115, in training_step
    model.backward(loss)
  File "/data/bart/temp_workspace/pytorch-trainer/trainer/deepspeed.py", line 233, in train_loop
    loss = self.training_step(model=model, batch=batch, batch_idx=batch_idx)
  File "/data/bart/temp_workspace/pytorch-trainer/trainer/deepspeed.py", line 155, in fit
    self.train_loop(
  File "/data/bart/temp_workspace/pytorch-trainer/ds_train.py", line 583, in main
    trainer.fit(
  File "/data/bart/temp_workspace/pytorch-trainer/ds_train.py", line 606, in <module>
    main(args)
RuntimeError: Found dtype Float but expected Half

For auto_cast, I'm using torch.cuda.amp, which I'm sure will work, but will that cause any problems when utilizing offload etc?
with autocast(enabled=True, dtype=torch.float16):
    labels = batch.pop("labels")
    output = model(batch["inputs"])
    loss = self.criterion(output, labels)

same issue here. Don't know if torch.autocast can be used together with deepspeed fp16

authors: YooSungHyun npuichigo

repo: microsoft/DeepSpeed


text:
9694 Create new ISummaryTelemetryData

Also consumes ISummaryTelemetryData in relevant places
#9694
AB#161

Looks like my understanding of package consumption was slightly wrong. From what it seems, client packages all move together in terms of versioning. Even though things are separated into different npm packages, they are still tightly linked in terms of automatically using versions of each other.
I previously thought everything was separate in terms of package versioning. As in you would need to manually bump or consume pre-release packages to receive new code. I guess this is only applicable when modifying things in the common directory?

repo: microsoft/FluidFramework


text:
Removed client-api dependency from replay tool

Removed client-api dependency from replay tool.
Added simple code loade, data object factory and runtime factory to replay tool.

Ping!


text:
Inconsistency in the number of sentences due to deleted tweets

Hi, I have observed there were many tweets which were nor downloaded due to the tweets being deleted. Also, for some strange reason for some tasks the number of sentences were higher than the number reported in the paper (specifically for English-Spanish datasets). Please find the table comparing the statistics reported in the paper and the statistics obtained after downloading the data.
English-Hindi

Corpus
Sent (Train)

Sent (Dev)

Sent (Test)

Paper
Downloaded
Paper
Downloaded
Paper
Downloaded

FIRE LID (D)
2631
2098
500
500
406
406

UD POS (D)
1384
1344
215
209
215
225

FG POS (R)
2104
2098
263
261
264
264

IIITH NER (R)
2467
2467
308
308
309
307

SAIL Sentiment   (R)
10080
10080
1260
1260
1261
1261

English-Spanish

Corpus
Sent (Train)

Sent (Dev)

Sent (Test)

Paper
Downloaded
Paper
Downloaded
Paper
Downloaded

EMNLP 2014
10259
7192
1140
824
3014
2981

Bangor POS
2192
2167
274
269
274
270

CALCS NER
27366
28381
3420
3537
3421
3577

Sentiment
1681
1851
211
231
211
232

Could the inconsistency in the dataset lead to unfair comparison of the results?

Hi Rudra, deleted tweets can cause inconsistencies in the numbers and may not lead to direct fair comparison, but unfortunately we cannot do much as we cannot store tweets. Due to this, we reran the benchmark on our models before adding to the leaderboard. So the numbers in the paper and the number of tweets you could download may not be exactly the same, but should be much more comparable to the results on the leaderboard. To slightly alleviate this issue, we will also mention on the leaderboard when the model was run.

Thanks

authors: murthyrudra ssitaram

repo: microsoft/GLUECoS


text:
MT Baseline Score

Can someone provide the baseline bleu score obtained when you ran mBART

bleu on the valid set should be 11.71


text:
when I run the hf-version on the fairseq-dense-125M ,there is error ""TypeError: forward() got an unexpected keyword argument 'prefix_parallel'

when I run the hf-version on the fairseq-dense-125M ,there is error ""model has no attribution of parralel"",when I remove the param --parallel, there is another error  ""TypeError: forward() got an unexpected keyword argument 'prefix_parallel'.

the first error is AttributeError: 'XGLMForCausalLM' object has no attribute 'parallelize'

@2018211801  Could you also post the command?

repo: microsoft/LMOps


text:
[dask] Support asynchronous workflows

Summary

Implementing coroutines for training and computing predictions with an asynchronous dask client.
Motivation

By having an asynchronous interface, LightGBM's distributed training with dask could be used in concurrent applications.
Description

This can be achieved by implementing coroutines for the train and predict functions and then using client.sync on them to get the synchronous variants.
References

XGBoost implementation

Thanks for opening this! Could you please provide a more specific XGBoost link, to the parts of their code that specifically allow async access?
Could you also add details on why someone would want to use LightGBM with Dask asynchronously?

Hi James. I've updated my comment with some cases I can think of. I realize this probably needs to come after some other building blocks but I'd like to work towards this, what do you think would be needed first?

Thanks very much for that. I added a link to XGBoost's docs on this topic as well: https://xgboost.readthedocs.io/en/latest/tutorials/dask.html#working-with-asyncio.

I realize this probably needs to come after some other building blocks but I'd like to work towards this

If you're interested in contributing further, we'd be very grateful!  But to be honest, I think that a lot of fundamental pieces need to be added before we consider supporting asynchronous training / prediction in the Dask interface.
The Dask interface is still very new and is missing big features like init_score (#3807), the other boosting types (#3896), and the other distributed training modes (#3834), to name a few.
Adding support for init_score (#3807) or predict(raw_score=True) (#3793) are good next steps.

I've added this to #2302 , where we keep all feature requests.

authors: jameslamb jmoralez

repo: microsoft/LightGBM

license: mit

license_source: bigquery


text:
[python-package] Training data partially available when loading exported lgb.Dataset

Summary
From a binary file exported using the lgb.Dataset.save_binary() method, it is possible to retrieve the feature names (list of string) and the labels (y, as a numpy array)
It is not however possible to retrieve the data, as a numpy array.
Both the lgb.Dataset.data and lgb.Dataset.get_data() will send back a string with the name of the binary file.
It should, in my opinion, send back the numpy array, just as the label.
Note it seems as well that the group is also not accessible.
The data is effectively contained in the binary file, because we are able to load and train a booster with it.
We should be able to get it from the newly created lgb.Dataset object
Motivation
This would allow us to properly investigate the exported datasets.
Like computing some statistics (mean, stdv etc) to understand our dataset, before using it again (or not) to train a booster.
Description
The .data atttribute and/or .get_data() method send back the proper numpy array.
The group and/or .get_group() send back the proper list of int
References
I executed the following script on the MQ2008, a microsoft open ranking dataset
You should be able to reproduce the same results.
I also provide an output so that you can understand what are the atttibutes and method that should have a modified behaviour
import logging

import lightgbm as lgb
from sklearn.datasets import load_svmlight_file

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s : %(message)s",
)
logging.info(f"lgb.__version__ = {lgb.__version__}")

# Function to load group information
def load_group_info(file_path):
    groups = []
    with open(file_path, "r") as f:
        qid_prev = None
        count = 0
        for line in f:
            qid = line.split()[1].split(":")[1]
            if qid != qid_prev:
                if qid_prev is not None:
                    groups.append(count)
                count = 1
                qid_prev = qid
            else:
                count += 1
        groups.append(count)
    return groups

# Load a small sample dataset from sklearn
data = load_svmlight_file("./MQ2008/Fold5/train.txt")
group = load_group_info("./MQ2008/Fold5/train.txt")
X, y = data[0], data[1]
logging.info(f"X.shape {X.shape}")
logging.info(f"y.shape {y.shape}")
logging.info(f"sum(group) {sum(group)}")
logging.info(f"len(group) {len(group)}")

# Create LightGBM dataset
logging.info("Creating ORIGINAL LightGBM dataset :")
lgb_data = lgb.Dataset(X, label=y, group=group, free_raw_data=False)
logging.info(f"lgb_data.data.shape : {lgb_data.data.shape}")
logging.info(f"lgb_data.label.shape : {lgb_data.label.shape}")
logging.info(f"len(lgb_data.group) : {len(lgb_data.group)}")

logging.info("ORIGINAL dataset training ...")
PARAMS = {
    "objective": "lambdarank",
    "metric": "ndcg",
    "eval_at": 5,
    "num_iterations": 5,
    "random_state": 42,
}
lgb.train(PARAMS, lgb_data, valid_sets=[lgb_data], callbacks=[lgb.log_evaluation()])

# Save lgb Dataset
lgb_data.save_binary("lgb_data.bin")

# Load the dataset from the binary file
lgb_data_loaded = lgb.Dataset("lgb_data.bin", free_raw_data=False)
lgb_data_loaded.construct()

# We show that only feature_name & label is accessible
logging.info("\n\n\t\t\tData is not accessible from the saved binary :\n")
logging.info(f"lgb_data_loaded.feature_name) : {lgb_data_loaded.feature_name}")
logging.info(f"lgb_data_loaded.label) : {lgb_data_loaded.label}")
logging.info(f"lgb_data_loaded.label.shape) : {lgb_data_loaded.label.shape}")
logging.info(f"lgb_data_loaded.group) : {lgb_data_loaded.group}")
logging.info(f"lgb_data_loaded.data) : {lgb_data_loaded.data}")
logging.info(f"type(lgb_data_loaded.data)) : {type(lgb_data_loaded.data)}")
logging.info(f"lgb_data_loaded.get_data()) : {lgb_data_loaded.get_data()}")
logging.info(f"type(lgb_data_loaded.get_data())) : {type(lgb_data_loaded.get_data())}")
logging.info(f"lgb_data_loaded.num_data()) : {lgb_data_loaded.num_data()}")
logging.info(f"lgb_data_loaded.num_feature()) : {lgb_data_loaded.num_feature()}")

# Data is Here because it is able to train... Why can't we access it then ?
logging.info(
    "\n\n\t\t\tData is Here because it is able to train... Why can't we access it then ?\n"
)
lgb.train(
    PARAMS,
    lgb_data_loaded,
    valid_sets=[lgb_data_loaded],
    callbacks=[lgb.log_evaluation()],
)

Which outputs :
2024-07-01 13:41:43,996 - INFO : lgb.__version__ = 4.4.0
2024-07-01 13:41:44,145 - INFO : X.shape (9442, 46)
2024-07-01 13:41:44,146 - INFO : y.shape (9442,)
2024-07-01 13:41:44,146 - INFO : sum(group) 9442
2024-07-01 13:41:44,146 - INFO : len(group) 470
2024-07-01 13:41:44,146 - INFO : Creating ORIGINAL LightGBM dataset :
2024-07-01 13:41:44,146 - INFO : lgb_data.data.shape : (9442, 46)
2024-07-01 13:41:44,146 - INFO : lgb_data.label.shape : (9442,)
2024-07-01 13:41:44,146 - INFO : len(lgb_data.group) : 470
2024-07-01 13:41:44,146 - INFO : ORIGINAL dataset training ...
[LightGBM] [Info] Total Bins 9221
[LightGBM] [Info] Number of data points in the train set: 9442, number of used features: 40
[1]     training's ndcg@5: 0.766816
[2]     training's ndcg@5: 0.8209
[3]     training's ndcg@5: 0.838295
[4]     training's ndcg@5: 0.856369
[5]     training's ndcg@5: 0.865225
[LightGBM] [Info] Saving data to binary file lgb_data.bin
[LightGBM] [Info] Load from binary file lgb_data.bin
2024-07-01 13:41:44,725 - INFO :

                        Data is not accessible from the saved binary :

2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.feature_name) : ['Column_0', 'Column_1', 'Column_2', 'Column_3', 'Column_4', 'Column_5', 'Column_6', 'Column_7', 'Column_8', 'Column_9', 'Column_10', 'Column_11', 'Column_12', 'Column_13', 'Column_14', 'Column_15', 'Column_16', 'Column_17', 'Column_18', 'Column_19', 'Column_20', 'Column_21', 'Column_22', 'Column_23', 'Column_24', 'Column_25', 'Column_26', 'Column_27', 'Column_28', 'Column_29', 'Column_30', 'Column_31', 'Column_32', 'Column_33', 'Column_34', 'Column_35', 'Column_36', 'Column_37', 'Column_38', 'Column_39', 'Column_40', 'Column_41', 'Column_42', 'Column_43', 'Column_44', 'Column_45']
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.label) : [0. 0. 0. ... 0. 2. 0.]
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.label.shape) : (9442,)
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.group) : None
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.data) : lgb_data.bin
2024-07-01 13:41:44,726 - INFO : type(lgb_data_loaded.data)) : <class 'str'>
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.get_data()) : lgb_data.bin
2024-07-01 13:41:44,726 - INFO : type(lgb_data_loaded.get_data())) : <class 'str'>
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.num_data()) : 9442
2024-07-01 13:41:44,726 - INFO : lgb_data_loaded.num_feature()) : 46
2024-07-01 13:41:44,726 - INFO :

                        Data is Here because it is able to train... Why can't we access it then ?

[LightGBM] [Info] Total Bins 9221
[LightGBM] [Info] Number of data points in the train set: 9442, number of used features: 40
[1]     training's ndcg@5: 0.766816
[2]     training's ndcg@5: 0.8209
[3]     training's ndcg@5: 0.838295
[4]     training's ndcg@5: 0.856369
[5]     training's ndcg@5: 0.865225

Thanks for using LightGBM. Someone will help shortly.
I noticed you double-posted this here and to Stack Overflow at the same time (Stack Overflow link). Please do not do that. Maintainers here also monitor the [lightgbm] tag on Stack Overflow. I could have been spending time preparing an answer here while another maintainer was spending time answering your Stack Overflow post, which would have been a waste of maintainers' limited attention that could otherwise have been spent improving this project. Double-posting also makes it less likely that others with a similar question will find the relevant discussion and answer.

Hey @Plenitude-ai, thanks for the thorough description. LightGBM's Dataset doesn't save the data as is contained in the original array, it puts features into bins, so the data that is saved is the bin in which the feature value was put into. Is that what you would like to get from lightgbm.Dataset.get_data()? Once the dataset has been saved there's no way to get the original data back.

100% agree with everything @jmoralez said.
I'll add that there is an open feature request (#5191) for being able to dump out LightGBM's binned representation as an array, which would allow you to at least partially inspect the training data.
You could subscribe to notifications there to be notified if that feature is formally added to the library. And could try some of the workarounds like https://github.com/microsoft/LightGBM/issues/5191#issuecomment-1742263175 mentioned there.
But only do that if there are genuine constraints that lead your application to only having access to a LightGBM Dataset and not the underlying data. If you can store the raw training data alongside the LightGBM Dataset (e.g. in Parquet, pickle, or npy format), you'll find that much easier and more useful than any of the workarounds described in #5191.

I understand.
It seems my comprehension of both the fundamental implementation, and purpose/objective of this class is incomplete (it's more low-level than I thought), thanks for pointing me in the correct direction.
I'll stick with pkl representations then, but I found useful to have all (array, label, group) in the same object.
Yes the issue your referred me to seems to be interesting and I'll definitely subscribe!
May I ask why we can still access the features names'? How is it saved in the bin representation?
Thanks again for your time :)

May I ask why we can still access the features names' & labels ? How is it saved in the bin representation?

LightGBM needs the exact values of the label (after light preprocessing like handling infinite values and NaNs) to calculate the loss, so it's always recoverable as a dense array from the Dataset object.
No preprocessing is done on feature names, so those also are always recoverable in their original form.
These data structures are almost always smaller than the raw features (and often MUCH smaller).
If you're interested in the lower-level details, I encourage you to look at the source code for the Dataset:

Python: https://github.com/microsoft/LightGBM/blob/83df24d23c00e438961f3c06889fff4e4984eac0/python-package/lightgbm/basic.py#L1745
C++: https://github.com/microsoft/LightGBM/blob/master/src/io/dataset.cpp

That is interesting.
I went to look at the source code, but I have to say it's a bit above my coding experience as I don't know C/C++, it is a bit hard to understand how/where the numpy array is converted into bins.
I went back to reading further more the documentation and now realize that I messed things up between "bin" and ".bin"/binary.
I didn't know about the bin representation of the data, for memory optimization
I also found that this phrase in the FAQ was very englightening :
"LightGBM constructs bin mappers to build trees, and train and valid Datasets within one Booster share the same bin mappers, categorical features and feature names etc., the Dataset objects are constructed when constructing a Booster. If you set free_raw_data=True (default), the raw data (with Python data struct) will be freed."
I'm thinking it might be interesting to add this short explanation in the header of the lgb.Dataset documentation, which was my first refering point.
Do you think it could be helpful ?

I'm thinking it might be interesting to add this short explanation in the header of the lgb.Dataset documentation, which was my first refering point.
Do you think it could be helpful?

I think that particular language from the FAQ is a bit too specific and would be confusing there, but yes I'd support adding a bit more documentation to the class docs for Dataset. Something like:
Dataset in LightGBM.

LightGBM does not train on raw data... it discretizes continuous features into histogram bins, tries to opportunistically combine categorical features, and automatically handles missing and infinite values.

The ``Dataset`` class handles that preprocessing, and holds that alternative representation of the input data.

Would you like to contribute that? It (or something similar) could be added here:
https://github.com/microsoft/LightGBM/blob/f8ec57b8eba69b7371324bba1d158f850802c782/python-package/lightgbm/basic.py#L1748
and in the R package as well
https://github.com/microsoft/LightGBM/blob/f8ec57b8eba69b7371324bba1d158f850802c782/R-package/R/lgb.Dataset.R#L761
If you are not familiar with R, you could just open a pull request with the Python changes and once we agree on the language I can push the corresponding R docs. If you want to try the R changes, see https://github.com/microsoft/LightGBM/tree/master/R-package#updating-documentation.

Sure this is a great idea !
Your proposal of addition seems perfectly fine to me, I'll try to create the pull request tonight.
For R, yes I think that I could use your help.

authors: Plenitude-ai jameslamb jmoralez


text:
setup CUDA CI job

Closed #3402.

etetetet

/gha run cuda-builds

Further possible improvements:

shut down machine after build is finished (https://github.com/microsoft/LightGBM/issues/3402#issuecomment-699674718) and turn it on after appropriate trigger;
allow only skipped and succeeded status of CUDA builds to merge PRs (right now I even cannot see it in PR checks)

Unfortunately, I'm not sure I'll be able to work on the items listed above.

@StrikerRUS It seems currently both azure pipeline and github actions cannot support the power-off/turn-on machines.
Even with the VM scale set in azure pipeline, it needs at least one machine stand by.

@guolinke

I know there are some external solutions ...

Yeah, also they are a ton of pain! For instance:
https://github.com/dmlc/xgboost/issues/4958
https://github.com/hcho3/xgboost-devops

... but they seems to require my Azure (Microsoft) account permission, which seems to violate our policy.

Yep, that's true! 😞
https://github.com/hcho3/xgboost-devops/blob/df6c582ac65d237632ec81c0a739ecdb7e9d77e0/.github/workflows/deploy-lambda.yml#L27-L28

Unfortunately, switching to P100 didn't help to get rid of segfault. When I run LightGBM\examples\python-guide\simple_example.py I get the following more informative logs:
Starting training...
Warning: ] [Warning] CUDA currently requires double precision calculations.
Warning: ] [Warning] Using sparse features with CUDA is currently not supported.
Warning: ] [Warning] CUDA currently requires double precision calculations.
Traceback (most recent call last):
  File "/LightGBM/examples/python-guide/simple_example.py", line 35, in <module>
    gbm = lgb.train(params,
  File "/root/.local/lib/python3.8/site-packages/lightgbm/engine.py", line 231, in train
    booster = Booster(params=params, train_set=train_set)
  File "/root/.local/lib/python3.8/site-packages/lightgbm/basic.py", line 1988, in __init__
    _safe_call(_LIB.LGBM_BoosterCreate(
  File "/root/.local/lib/python3.8/site-packages/lightgbm/basic.py", line 55, in _safe_call
    raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
lightgbm.basic.LightGBMError: [CUDA] invalid argument /LightGBM/src/treelearner/cuda_tree_learner.cpp 414

However, I don't think that it should block this PR from merging.

@StrikerRUS so it is hardware problem or ?

@guolinke TBH, I have no idea...
Maybe we can ask @ChipKerchner or fix the following CMake warning (maybe it is important) at first
CMake Warning (dev) in CMakeLists.txt:
  Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
  empty CUDA_ARCHITECTURES not allowed.  Run "cmake --help-policy CMP0104"
  for policy details.  Use the cmake_policy command to set the policy and
  suppress this warning.

  CUDA_ARCHITECTURES is empty for target "histo_16_64_256-allfeats_sp_const".
This warning is for project developers.  Use -Wno-dev to suppress it.

CUDA_ARCHITECTURES

Sorry, this is a new feature in CMake 3.18 and I'm not familiar with it.

@ChipKerchner Thanks for your super fast response!
I'll try to re-run with older CMake.
Will it be possible to adapt CMakeLists.txt code according to the recent CMake changes in the future?

Just tested CMake 3.16.7: https://github.com/microsoft/LightGBM/runs/1199141320?check_suite_focus=true. Unfortunately, the same output.

Will it be possible to adapt CMakeLists.txt code according to the recent CMake changes in the future?

Looks like we could do the same:
https://github.com/dmlc/xgboost/blob/e0e4f15d0e314afcf44d690e0295fa6320fc7f64/cmake/Utils.cmake#L104-L111

I'm not sure that it's related to CUDA_ARCHITECTURES.  It's already compiled for multiple architectures https://github.com/microsoft/LightGBM/blob/master/CMakeLists.txt#L159.
https://github.com/microsoft/LightGBM/blob/master/src/treelearner/cuda_tree_learner.cpp#L414
host memory has to be page-locked host memory. Is it allocated properly?

========= CUDA-MEMCHECK
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Warning] Using sparse features with CUDA is currently not supported.
[LightGBM] [Warning] CUDA currently requires double precision calculations.
[LightGBM] [Info] LightGBM using CUDA trainer with DP float!!
[LightGBM] [Info] Total Bins 22008
[LightGBM] [Info] Number of data points in the train set: 1348045, number of used features: 150
[LightGBM] [Debug] device_bin_size_ = 256
[LightGBM] [Debug] Resized feature masks
[LightGBM] [Debug] Memset pinned_feature_masks_
[LightGBM] [Debug] Allocated device_features_ addr=0x7f3230000000 sz=202206750
[LightGBM] [Debug] Memset device_data_indices_
[LightGBM] [Debug] created device_subhistograms_: 0x7f323e600000
[LightGBM] [Debug] Started copying dense features from CPU to GPU
[LightGBM] [Debug] Started copying dense features from CPU to GPU - 1
[LightGBM] [Debug] Started copying dense features from CPU to GPU - 2
========= Program hit cudaErrorInvalidValue (error 11) due to "invalid argument" on CUDA API call to cudaMemcpyAsync.

Seems not correct pointers or size

@StrikerRUS any updates of this PR?

Great! I thought the cuda job cannot run.

@guolinke

I thought the cuda job cannot run.

It can but unfortunately fails with the following runtime error: https://github.com/microsoft/LightGBM/pull/3424#issuecomment-702189313.
That error can be reproduced by independent CUDA early adopters:

https://github.com/microsoft/LightGBM/pull/3424#issuecomment-704349337;
#3450.

any insights regard to these errors? @ChipKerchner

authors: ChipKerchner StrikerRUS guolinke sh1ng


text:
fix retrain on sequence dataset

https://github.com/microsoft/LightGBM/issues/6413

Hi @eromoe ! Sorry for the delayed response!
Could you please add a test that will fail on the current master branch and will pass with your fixes.
Python tests are stored here: https://github.com/microsoft/LightGBM/tree/master/tests/python_package_test.
Feel free to ask for any help with tests.

@microsoft-github-policy-service agree

@StrikerRUS I just added the test

@eromoe Thank you very much! I'm triggering CI tests to see how it goes.

@jameslamb Updated,

authors: StrikerRUS eromoe


text:
[Question]: ModuleNotFoundError: No module named 'minference.cuda'

Describe the issue
I encountered some issues when using minference in Python.
import minference
The problem is Traceback (most recent call last):   File "<stdin>", line 1, in <module>   File "/workspace/MInference/minference/__init__.py", line 8, in <module>     from .models_patch import MInference   File "/workspace/MInference/minference/models_patch.py", line 7, in <module>     from .patch import minference_patch, minference_patch_vllm, patch_hf   File "/workspace/MInference/minference/patch.py", line 12, in <module>     from .modules.minference_forward import (   File "/workspace/MInference/minference/modules/minference_forward.py", line 20, in <module>     from ..ops.pit_sparse_flash_attention_v2 import vertical_slash_sparse_attention   File "/workspace/MInference/minference/ops/pit_sparse_flash_attention_v2.py", line 10, in <module>     from ..cuda import convert_vertical_slash_indexes ModuleNotFoundError: No module named 'minference.cuda'
enviornment :
Python 3.10.14
minference 0.1.4.post3
triton 2.1.0
torch 2.3.0
CUDA 11.8
vllm 0.4.2+cu118
flash-attn  2.5.8

Hi @lai-serena, thanks for your feedback.
It looks like the build was unsuccessful. You can try installing it using the following method:
pip uninstall minference -y
MINFERENCE_FORCE_BUILD=TRUE pip install minference --no-cache-dir

or build from source,
``bash
git clone https://github.com/microsoft/MInference/
cd MInference
pip install -e .

pip install -e .  works for me. Thanks!

authors: iofu728 lai-serena lepangdan

repo: microsoft/MInference


text:
Hololens 1st gen camera not moving, spatial tracking not working

Describe the bug
When trying to implement an app for Hololens with Vuforia model tracking, the camera seems to be stuck, it doesn't track the world.
To reproduce
Steps to reproduce the behavior:

Go to MixedRealityToolkit
Click on TargetScale and set to World (with Room or seated it doesn't work neither)
In Camera select DefaultHololens1CameraProfile
Place an object away from the camera
Simulation in the editor works fine
Build and test the app, the objects are stuck with the camera view (if spatial awareness is enabled, the world mesh is stuck too)

Expected behavior
The objects should move relative to the camera position
Your setup

Unity Version 2020.3.14f1
MRTK Version 2.6.0
Vuforia 10.4

Target platform

HoloLens

@keveleigh can you take a look at this and advise?

Ahh hmm. I had thought all our profiles were set up for both legacy XR and XR SDK, but this one might have been missed.
As a workaround, you can manually add a Tracked Pose Driver to your camera. I'll investigate the potential misconfiguration in the profile.

We appreciate your feedback and thank you for reporting this issue.
Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.
Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.
Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.
Thank you for your continued support of the Mixed Reality Toolkit!

authors: Aupuma IssueSyncBot davidkline-ms keveleigh

repo: microsoft/MixedRealityToolkit-Unity


text:
Empty "CINEMA_4D_Editor" object inside a DockExample prefab

Describe the bug
Hi there!
The Model_Bucky prefab hold an empty CINEMA_4D_Editor item.
That's all.
Versions

MRTK 2.4
Unity 2019.4.0f1 LTS

Screenshots

Looks like we can get rid of this - marking as Help Wanted

authors: nooxouille wiwei


text:
Fixed articulated hands being classified as HP Motion Controllers

Overview
Adds a check so controllers which are classified as articulated hands will not be named HP Motion Controllers
Changes

Fixes: #9403

/azp run

@CDiaz-MS can I get a quick confirm that these changes have the intended effect?


text:
Support completion in { } in f-strings

Also reported at https://developercommunity.visualstudio.com/content/problem/437333/python-36-new-print-format-is-not-supported-in-vs.html
https://github.com/Microsoft/python-language-server/issues/6

Hello I'd love to +1 this feature. What is the best way to do that @huguesv?

@jakubLangr you can click the + smile icon on on the initial comment of an issue/enhancement to vote for it. Thanks!

Hi, any news on this issue ? Just tested it on the last version (1.43.0) and it still doesn't work.

Tested on 16.5.1 no update

Doesn't work properly for me either, on VS Code 1.43.2.

+1

+1

+1

+1

+1

+1

+1

1.54.3 not working also

Still not working.... 16.11.5

still not working Version: 1.72.2 (Universal)

Still not working, why it was closed? Visual Studio 17.3.6

authors: EwertonES JohnTravolski Pbotsaris SilasPDJ TheWarHam TuTAH1 WeberJulian Zuchis huguesv jakubLangr omrumcetin timwredwards yoel-shapiro

repo: microsoft/PTVS


text:
Fix formatter not formatting special characters

This PR fixes issue #6785
Formatters in PTVS were not able to format special characters such as Chinese and Germany characters correctly. Not sure what encoding/decoding is used on PTVS/formatter end, which might be the cause, so pass in UTF-8 to decode patch returned from formatters.
Test 1:

Test 2 with Black:

Test 3 with autopep8:

Test 4 with yapf:

Great job!


text:
New function to read Power Platform env vars

See here:
https://learn.microsoft.com/en-us/power-apps/maker/data-platform/environmentvariables
Key things there:

Env vars have both display and logical name.
Env vars vars have many types (like json):

Data Type. Select from Decimal number, Text, JSON, Two options, Data source, or Secret.

Given those requirements, looking at something like introducing each variable as a strongly typed constant like this:
Environment.Variable.VarName

VarName is the display name
it's scoped to the Variable object in the Environment namespace.  The extra "Variable" layer helps avoid conflicts with other names on Environment.

We may want this to be plural ".Variables" vs singular ".Variable".  I need to check with the content team before we finalize.

fyi - methods for converting between Display and Logical names:
https://github.com/microsoft/Power-Fx/blob/c70286e724de26c535e2c44386d59340f9b8d6f6/src/libraries/Microsoft.PowerFx.Core/Public/Engine.cs#L498C23-L498C45
Persist in invariant form.
Edit in display name.

authors: MikeStall gregli-msft

repo: microsoft/Power-Fx


text:
Expose CheckResult.GetAnonymizedFormula() for logging

Expose behind a new method like:
CheckResult.GetAnonymizedFormula()
Which returns an anonymized version of the formula (identifiers names removed, etc) so that the structure format can be logged.
Which effectively does:
var sanitizedFormula = StructuralPrint.Print(binding.Top, binding, new NameSanitizer(binding));
This should strip out all identifiers and make something that's safe for logging.

We could actually put this on ParseResult since that has the tree.


text:
Localization of custom connector

I didn't find any document about the localization for custom connector, but I do see that the certified connector is localized if I change the PowerAutomate page to a different language, how it happened? it's auto translated by Bing translator?

any update?

@mamurshe closed without any update?

repo: microsoft/PowerPlatformConnectors


text:
Seismic Configuration (Verified Publisher)

This is an update to the configuration connector.
This will allow us to enable our clients to use Triggers (webhooks) in PA and will allow them to run flows on certain events in power automate.

Hello @gpsarkar, kindly let me know the reason that why are you reviewing this connector here in GitHub?

Hello @vmanoharas, Me and @amit-chaudhari1 are from same organization. I reviewed to ensure the correctness of the changes that are committed in this pr. Hope that is fine.

Hello @gpsarkar,
Thank you very much for the clarification, could you please review files offline and submit the final copy here?

@vmanoharas The files in this pr are final now. Should we close this and create a new pr instead?

Hello @gpsarkar, Yes, please close this PR and create a new PR.

authors: amit-chaudhari1 gpsarkar vmanoharas


text:
[FancyZones] Windows adjusted with FanzyZones not the expected size (e.g. Everything)

Microsoft PowerToys version
0.37.2
Running as admin

[ ] Yes

Area(s) with issue?
FancyZones
Steps to reproduce
Drag the Everything window to a Zone to adjust the size and position
PowerToysReport_2021-05-21-08-58-41.zip
✔ Expected Behavior
Everything window should have the same size as the layout
❌ Actual Behavior
Window is not the same size as the layout and shifted to either left

(On the right side is a small gap)
or right

(Scrollbar on the right is not entirely on the screen and on the left side, there is a gap between the taskbar and the window
Other Software
Everything Version 1.4.1.1005 (x64)

@todesschnitzel
see https://github.com/microsoft/PowerToys/wiki/Application-Compatibility#applications-that-cannot-be-zoned-exactly

authors: enricogior todesschnitzel

repo: microsoft/PowerToys


text:
Allow Microsoft Teams to share a fancy zone in a meeting

Description of the new feature / enhancement
When sharing screen in a Microsoft Teams meeting, it'd be great to have the capability to share a fancy zone only.
Scenario when this would be used?
Here's my use case, I have an 34-inch ultra wide screen of 21:9 ratio. Sharing my entire monitor to a meeting participant with FHD monitor (say 23 inch 16:9 monitor) makes the other party hard to see my content. Sure I can share a Window but it's a workaround, and does not have the same flexibility of sharing a fancy zone.
Supporting information
No response

Great idea, Tracked in #279

This is our how team is viewing this grouping, a “virtual monitor”.

Coming back to this topic: what's the original ticket number (if this is a duplicate)? I've set up my zones but I am not able to share a specific one in teams, just the usual things: screen or program. Was the original idea implemented or still waiting?

I'm sorry, but I read through #279, and don't see that the request is the same thing at all. Its title is "Maximize/Full Screen window within a zone (virtual monitor)" The OP here has asked to be able to share a zone, instead of a window, which means an application, or a screen, which is one monitor. FancyZones lets me create a 1920x1080 zone that I'd like to share in a meeting so that I can drag in different apps into that zone. IOW, I need to be able to share different apps on-the-fly. For example, if I have Azure DevOps open in a browser, and Visual Studio in another, I can look at code with a client but drag the browser into that zone so we can look at work items.

@NancyFolsom  how we are viewing #279 is it would hit that requirement.  Teams and other apps do not have the concept of FancyZones, it does however have it with applications and monitors. We have ideas on how to maybe do treat zones in FZ as an 'app' but then the questions arise of how does the content flow through and is that universal across ALL the apps that do this style screen sharing.
Versus if we trick the OS to think it is a monitor ... stuff just works and you can full screen applications / videos :)
This is why we group these asks against that ask.

@NancyFolsom how we are viewing #279 is it would hit that requirement. Teams and other apps do not have the concept of FancyZones, it does however have it with applications and monitors. We have ideas on how to maybe do treat zones in FZ as an 'app' but then the questions arise of how does the content flow through and is that universal across ALL the apps that do this style screen sharing.
Versus if we trick the OS to think it is a monitor ... stuff just works and you can full screen applications / videos :)
This is why we group these asks against that ask.

Fair enough. It was a big confusing to see the issue stated as being maximize w/i a zone. Maybe a tweak to the title would help? Anyway, thanks, again, for explaining.

I also wait that teams application sharing would work with the zones of FancyZones. But until this is not implemented i use RegionToShare

authors: NancyFolsom Polaquilla crutkas edgarguoms paetzoldf


text:
PowerToys Run doesn't autofocus on first launch

Microsoft PowerToys version
0.53.3
Running as admin

[ ] Yes

Area(s) with issue?
PowerToys Run
Steps to reproduce
After a reboot, PowerToys Run doesn't autofocus on first launch. This means that the quickest way to get it working is to:
Alt + space -> First Launch
Alt + space -> Close
Alt + space -> New launch, with autofocus working
✔ Expected Behavior
PowerToys Run should autofocus on first launch
❌ Actual Behavior
On first launch after booting pc, PowerToys Run doesn't autofocus.
Other Software
No response

Duplicate of #13643

Issue is a duplicate
Thank you for your issue! We've linked your report against another issue to centralize the thread.
Thanks for helping us to make PowerToys a better piece of software.


text:
Editable taskbar quick access

Description of the new feature / enhancement
Love the new taskbar quick access feature!
Just to be greedy, it would be great if one could configure which PowerToys appear in the panel.
Scenario when this would be used?
Quick access to PowerToys that I use less often.
Supporting information
No response

/dup https://github.com/microsoft/PowerToys/issues/24283


text:
Locking Tools

Description of the new feature / enhancement
The ability to password protect application and folders
Scenario when this would be used?
Provides more security to the user
Supporting information
No response

/dup #23410

authors: Omendran davidegiacometti


text:
press and hold to trigger PT Run

Description of the new feature / enhancement
II noticed that shortcut-guide can use long press win, can add long press win key to show powertoys run?
I haven't learned C fully, so I may not be able to help with the code. @.@
Scenario when this would be used?
More ways to activate powertoys run, for people with different habits.
Supporting information

how shortcut guide does it isn't the world's most ideal way.

I understand your ask. I’m saying for us to extend and do more press and holds we need to rethink implementation

authors: crutkas yige-yigeren


text:
Reverse mouse direction, support for more mouse buttons

Reverse the scrolling direction on an OS level so the user doesn't have to deal with often clunky driver software, or diving into the registry.
Reverse scrolling direction with the click of a button in the same way Precision touchpads are handled by Windows, but then for mouse users. Currently, reversing the scrolling direction requires diving into the mouse driver settings, which may or may not have the feature available, or changing entries in the registry. This is not really user friendly, and can easily be handled by the OS/PowerToys.

I don't see the need to re-invent the wheel.

I feel this is tracked by #1475 and #6434

authors: Jay-o-Way crutkas hugobloem


text:
Laptop freezes and blacks out when I type, then get PowerToys error

Microsoft PowerToys version
0.76.0.0
Installation method
Microsoft Store
Running as admin
Yes
Area(s) with issue?
General
Steps to reproduce
When I'm typing, sometimes my laptop will freeze up, go black for a second and then a few seconds later, it finally comes back. Twice now it has come up with this error, but it occurs more often than I get the error.
Version: 0.76.0.0
OS Version: Microsoft Windows NT 10.0.22621.0
IntPtr Length: 8
x64: True
Date: 2/23/2024 11:59:52 AM
Exception:
System.Runtime.InteropServices.COMException (0x80263001): {Desktop composition is disabled} The operation could not be completed because desktop composition is disabled. (0x80263001)
at Standard.NativeMethods.DwmExtendFrameIntoClientArea(IntPtr hwnd, MARGINS& pMarInset)
at System.Windows.Shell.WindowChromeWorker._ExtendGlassFrame()
at System.Windows.Shell.WindowChromeWorker._UpdateFrameState(Boolean force)
at System.Windows.Shell.WindowChromeWorker._HandleDwmCompositionChanged(WM uMsg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at System.Windows.Shell.WindowChromeWorker._WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
✔ Expected Behavior
To type without my computer blacking out.
❌ Actual Behavior
When I'm typing, doesn't matter if it's in an email or word document or in a browser somewhere, randomly it will freeze and cause my screen to go black for a few seconds, then come back up. Seems like a computer aneurism. Then I got this error message.
Other Software
Version: 0.76.0.0
OS Version: Microsoft Windows NT 10.0.22621.0
IntPtr Length: 8
x64: True
Date: 2/23/2024 11:59:52 AM
Exception:
System.Runtime.InteropServices.COMException (0x80263001): {Desktop composition is disabled} The operation could not be completed because desktop composition is disabled. (0x80263001)
at Standard.NativeMethods.DwmExtendFrameIntoClientArea(IntPtr hwnd, MARGINS& pMarInset)
at System.Windows.Shell.WindowChromeWorker._ExtendGlassFrame()
at System.Windows.Shell.WindowChromeWorker._UpdateFrameState(Boolean force)
at System.Windows.Shell.WindowChromeWorker._HandleDwmCompositionChanged(WM uMsg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at System.Windows.Shell.WindowChromeWorker._WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

/dup #31226

authors: davidegiacometti printableaccounting


text:
[enhancement; PowerToys Run] Tooltip for long results and file path

Environment
Windows build number: 10.0.18363
PowerToys version: 0.18.0
PowerToy module for which you are reporting the bug (if applicable): PowerToys Run

Summary of the new feature/enhancement
Implement a tooltip on hovering the results into PowerToys Run to show

long window titles completely. (Interesting to sometimes choose the right window.)
applications paths. (Interesting to understand some results and see what you start.)
some other interesting information.

Screenshots

@crutkas @saahmedm I have a few questions related to tooltip specs :

Do we want to display tooltip for every plugin or a few specific plugins (such as program, indexer)?
What information do we want we display in the tooltip (title, path, etc.)?

@somill55,

I think specific plugins for now (indexer+program)
the full title+ the full Path

We just released 0.20 and this should be resolved there.  Please try it out.

authors: crutkas htcfreek saahmedm somil55


text:
Custom position for PowerToys Run and add more settings to customize the window

Being able to choose custom position for PowerToys run, or making it remember where you placed it last. Also the ability to customize the background/transparency of the run window.
One could use this to place the tool window at the bottom of the screen, or any other position which might be suitable. Also adding transparency would make it see-through, which can be useful in those situations where you still need to see the text behind the run window.

If you'd like to see this feature implemented, add a 👍 reaction to this post.

Related to #1607.

lets dup this against #4576

Could you send the video as is to my business number please ? +1 (463)
867-4946
On Thu, Dec 31, 2020 at 6:57 PM Clint Rutkas notifications@github.com
wrote:

Closed #6970 https://github.com/microsoft/PowerToys/issues/6970.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/PowerToys/issues/6970#event-4160480322, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ASJ2QZPZOUWXBAEYETDD3EDSXUFX7ANCNFSM4SBQH4QQ
.

Could you send the video as is to my business number please ? +1 (463)
867-4946
On Thu, Dec 31, 2020 at 6:57 PM Clint Rutkas notifications@github.com
wrote:

Closed #6970 https://github.com/microsoft/PowerToys/issues/6970.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/PowerToys/issues/6970#event-4160480322, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ASJ2QZPZOUWXBAEYETDD3EDSXUFX7ANCNFSM4SBQH4QQ
.

authors: alekhood317 amirarfan crutkas htcfreek


text:
ZoneWindow Unit test fail because ZoneWindowHost is null

Environment
Windows build number: [Version 10.0.18363.476]
PowerToys version:
PowerToy module for which you are reporting the bug (if applicable): FancyZones

Steps to reproduce
This is issue is being raised retroactively to account for the work that was done in PR #912
Expected behavior
Unit tests should pass.
Actual behavior
TestDeviceId and TestUniqueId are failing because m_host is null the the following method of ZoneWindow.cpp
void ZoneWindow::ChooseDefaultActiveZoneSet() noexcept
{
    // Default zone set can be empty (no fancyzones layout), or it can be layout from virtual
    // desktop from which this virtual desktop is created.
    if (GUID id{ m_host->GetCurrentMonitorZoneSetId(m_monitor) }; id != GUID_NULL) {
        for (const auto& zoneSet : m_zoneSets) {
            if (id == zoneSet->Id()) {
                UpdateActiveZoneSet(zoneSet.get());
                return;
            }
        }
    }
}

Screenshots

Complete in PR #912


text:
[KBM] Support the dual key remap to have different remaps depending on whether another key is pressed together

Summary of the Pull Request
This pull request implements the dual key remap functionality in Keyboard Manager.
What is this about:
The dual key remap allows a single key assigned to different keys depending on whether the key is pressed together with another key or alone. Some products already utilize it.

Space-and-Shift (a.k.a. SandS) to convert Space key to Shift key when other keys are pressed while pressing Space key. euxn23/win-sands-ahk and fiore57/SandSExe are some of those examples.
alt-ime-ahk to convert left / right Alt keys to IME off / on keys when Alt key is pressed alone.

In addition, #5679 has 8 upvotes as well as 6 duplicate issues. There are some demands on the feature worth being supported in Keyboard Manager, I believe.
What is included in the PR:
See also https://github.com/microsoft/PowerToys/issues/5679#issuecomment-1053542523 for the brief spec.

Add a combo box under the arrow symbol for each single key remap in Keyboard Manager Editor to choose one of either "Always", "Alone", and "In combination" (any suggestion on better names is welcome).
Show the remap condition label for each single key remap in Keyboard Settings PowerToys Settings page.
Rename State::GetSingleKeyRemap to State::GetSingleKeyRemapResult which now returns SingleKeyRemapResult having a list of key to handle the case where a dual-remapped key is being pressed.
Convert RemapBufferRow from a tuple to a standard struct with a new field condition.
Split MappingConfiguration::singleKeyReMap into alwaysSingleKeyRemapTable, aloneSingleKeyRemapTable, and combinationSingleKeyReMap tables for "Always", "Alone", and "In Combination" remaps, respectively.
Move EditorHelpers::IsValidShortcut to Shortcut.h for sharing it with KeyboardManagerEngineLibrary.
Pass std::vector<INPUT> instead of LPINPUT to SendVirtualInput because predicting the number of key events beforehand becomes more difficult and C-style arrays are difficult to handle safely.

How does someone test / validate:

[x] All existing test cases pass.
[x] "In Combination" remap works only when the key is pressed together with another key.
[x] "Alone" remap works only when the key is pressed alone.

Quality Checklist

[x] Linked issue: #5679
[x] Communication: I've discussed this with core contributors in the issue.
[x] Tests: Added/updated and all pass
[ ] Installer: Added/updated and all pass
[x] Localization: All end user facing strings can be localized
[x] Docs: Added/updated
[x] Binaries: Any new files are added to WXS / YML

[x] No new binaries
[ ] YML for signing for new binaries
[ ] WXS for installer for new binaries

Contributor License Agreement (CLA)

[x] A CLA must be signed. If not, go over here and sign the CLA.

I'd say it's an expected behavior in edge cases. While I used standard keys (e.g. A) for (over-)simplifying my explanation, I expect most people to dual-map modifier keys only because we usually don't press a standard key together with other keys.

Ok. I thought so. Thanks for clarifying.

Please remember to ask for the next review after you think the requested changes have been pushed. Thank you.

@masaru-iritani when should we look at this again?

@stefansjfw @jaimecbernardo @crutkas
I didn't have enough time to consider and implement the way to indicate remap errors while allowing further edits. My current idea is to show a small error indicator on the array with the hover tooltip. I'm not sure if it is aligned with other PowerToys designs. Any suggesiton is welcome.

@masaru-iritani right now looks like a tooltip is thrown up.
here i did 1 against 1 and it reset the option that was last used.

@crutkas As you explained, the current design resets invalid remaps immediately. I believe we should allow remaps for the same key during editing while saving them should be blocked.

@masaru-iritani , is this ready for reviewing again?
I think it's gotten quite some conflicts since the last time.

@jaimecbernardo I'm sorry for the delay. Let me push my ongoing work and make this pull request draft until I find an appropriate and consistent design to allow tentative mapping errors.

authors: crutkas jaimecbernardo masaru-iritani stefansjfw


text:
Upgrading community toolkit to latest

Summary of the Pull Request
Upgrading toolkit

PR Checklist

[ ] Closes: https://github.com/microsoft/PowerToys/issues/30917, https://github.com/microsoft/PowerToys/issues/29999
[ ] Communication: I've discussed this with core contributors already. If work hasn't been agreed, this work might be rejected
[ ] Tests: Added/updated and all pass
[ ] Localization: All end user facing strings can be localized
[ ] Dev docs: Added/updated
[ ] New binaries: Added on the required places

[ ] JSON for signing for new binaries
[ ] WXS for installer for new binaries and localization folder
[ ] YML for CI pipeline for new test projects
[ ] YML for signed pipeline

[ ] Documentation updated: If checked, please file a pull request on our docs repo and link it here: #xxx

Detailed Description of the Pull Request / Additional comments

Validation Steps Performed

/azp run

authors: crutkas jaimecbernardo


text:
Workspaces replace glyphs by thicker ones

Summary of the Pull Request
replace glyphs by thicker ones

PR Checklist

[ ] Closes: #xxx
[ ] Communication: I've discussed this with core contributors already. If work hasn't been agreed, this work might be rejected
[ ] Tests: Added/updated and all pass
[ ] Localization: All end user facing strings can be localized
[ ] Dev docs: Added/updated
[ ] New binaries: Added on the required places

[ ] JSON for signing for new binaries
[ ] WXS for installer for new binaries and localization folder
[ ] YML for CI pipeline for new test projects
[ ] YML for signed pipeline

[ ] Documentation updated: If checked, please file a pull request on our docs repo and link it here: #xxx

Detailed Description of the Pull Request / Additional comments

Validation Steps Performed


text:
Improve window filter to include start menu and Cortana search box.

Improve window filter to include start menu and Cortana search box.
Also moves GetProcessPath* functions to common, renaming both to get_process_path.

PR Checklist

[x] Closes #274
[x] CLA signed. If not, go over here and sign the CLA

Validation Steps Performed
Manual

Nice work!

authors: bzoz enricogior


text:
Add new workflow to require API review.

This PR adds a new workflow that blocks PRs with the "Status-NeedsAPIReview" label, to prevent accidentally merging PRs that have not yet gone through review.

I added the API review label to test things out quickly, and it looks like it correctly blocked this PR. Note that the new workflow only blocks PRs against master so that new APIs can be incrementally developed on feature branches.

@cgranade is there a formal page yet that explains the API review process that we could link here?

The task for API review failed in this case, but it's still possible to merge. Is this intended?

authors: cgranade guenp msoeken

repo: microsoft/QuantumLibraries


text:
Oct 1 - Build and deploy decentralized apps

Event Information

Category
Details

Reactor Topic
Emerging Technologies

Monthly Topic (if applicable)
Skill Up September

Weekly Topic (if applicable)
Blockchain

Stream Date
October 1, 2020

Stream Time
9am PT

Stream Duration
110 minutes

Title
Build and deploy decentralized apps

Description
Decentralized applications, or Dapps for short, are what really brings smart contracts to life. Creating these UI applications is what allows people to interact with and execute contracts. Join this stream to learn about the popular tools to build Dapps, and walk through the steps to build a Dapp from scratch.

Speaker
@meaghanlewis

Moderator
TBD

SharePoint ID
7969

Additional Resources:

Event Checklist
10 Weeks Prior

[x] Reactor Content Team (RCT) or Speaker opens a new issue for an upcoming stream filling in the Event Information table. Both RCT and Speaker are marked as assignees.
[x] RCT and Speaker discuss details of the event in the comments of this issue. RCT adds a confirmed-event label and global-twitch-event tag to the issue and adds event to the Global Live Streams project. Issue will appear in the Upcoming Streams column
[ ] Speaker adds a ready-for-marketing-edit label so the RCT can review title and description

8 Weeks Prior

[ ] RCT does a review of title and description
[ ] RCT moves the issue to the Ready for SharePoint column
[ ] RCT informs the Reactor Marketing Team of the new event information
[ ] RCT provides speaker with assets (e.g. slides, overlays) and the survey data for the event
[ ] RCT moves the issue to the Stream Preparation In Progress column

2 Weeks Prior (Done in /microsoft/Reactors)

[ ] Speaker opens a PR with the new content for the stream on the customer-facing Reactor repository and adds the content-development and global-twitch-event tag to the PR.
[ ] Speaker adds to the same PR a modification to the Streaming README to include their stream information.
[ ] Speaker adds the RCT assigned to the issue as the reviewer when it is ready for review/merge
[ ] RCT reviews, approves, and merges the changes

1 Week Prior

[ ] Speaker and Moderator test setup for streaming (instructions coming soon)
[ ] RCT checks in with the speaker and moves the issue to the Ready to Stream column

During and After Stream

[ ] Speaker and Moderator run the stream
[ ] RCT submits recording to be uploaded to the Reactor YouTube and to be made available on LearnTV
[ ] RCT moves issue to the Stream Complete column

Will not be doing this stream.

authors: meaghanlewis sguthals

repo: microsoft/ReactorPlanning