text:
[BUG] Job hangs at backward

Describe the bug
The job contains 512 GPUs. All processes are waiting allreduce, but one is waiting at split().
Stack for the waiting point of allreduce
(gdb) py-bt
Traceback (most recent call first):
  <built-in method allreduce of PyCapsule object at remote 0x7fb3b12c3f90>
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1209, in all_reduce
    work = group.allreduce([tensor], opts)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 1326, in allreduce_bucket
    dist.all_reduce(tensor_to_allreduce, group=self.dp_process_group)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 1354, in allreduce_and_copy
    allreduced = self.allreduce_bucket(small_bucket, rank=rank, log=log)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 1374, in allreduce_no_retain
    self.allreduce_and_copy(small_bucket, rank=rank, log=log)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 1642, in buffered_reduce_fallback
    self.averaged_gradients = {}
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 2215, in reduce_ipg_grads

  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 1054, in reduce_independent_p_g_buckets_and_remove_grads
    def set_norm_for_param_grad(self, param):
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 1237, in reduce_ready_partitions_and_remove_grads
    self.reduce_independent_p_g_buckets_and_remove_grads(param, i)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 845, in reduce_gradients

  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1924, in allreduce_gradients
    for event in self.summary_events:  # write_summary_events
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 3020, in backward
    and name not in module._non_persistent_buffers_set):
  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1627, in inner_loop_step
    self.model_engine.backward(loss)
  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1718, in inner_loop

  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1830, in do
    if not self.use_amp and losses != losses:
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 1933, in do_train
    get_model_sub_name(iteration) + '.pt')
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 4630, in train
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 1696, in ensure_train
    log_activation=self.cfg.log_activation,
  File "/tmp/code/quickdetection/src/qd/pipeline.py", line 680, in pipeline_train_eval_multi
    pip.ensure_train()
  File "src/qd/common.py", line 3379, in execute_func
  File "src/qd/common.py", line 4418, in <module>

trace for the split()
raceback (most recent call first):
  <built-in method run_backward of torch._C._EngineBase object at remote 0x7fc106366f80>
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/__init__.py", line 659, in backward
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 511, in backward
    return super(Tensor, self).split(split_size, dim)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 53, in backward
    scaled_loss.backward(retain_graph=retain_graph)
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/zero/stage2.py", line 2409, in backward
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2739, in backward

  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1627, in inner_loop_step
    self.model_engine.backward(loss)
  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1718, in inner_loop

  File "/tmp/code/quickdetection/src/qd/opt/trainer.py", line 1830, in do
    if not self.use_amp and losses != losses:
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 1933, in do_train
    get_model_sub_name(iteration) + '.pt')
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 4630, in train
  File "/tmp/code/quickdetection/src/qd/pipelines/uni_pipeline.py", line 1696, in ensure_train
    log_activation=self.cfg.log_activation,
  File "/tmp/code/quickdetection/src/qd/pipeline.py", line 680, in pipeline_train_eval_multi
    pip.ensure_train()
  File "src/qd/common.py", line 3379, in execute_func
  File "src/qd/common.py", line 4418, in <module>

ds_report output
Please run ds_report to give us details about your setup.
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 1.9.0a0+c3d40fd
torch cuda version ............... 11.3
nvcc version ..................... 11.3
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.5.8, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.9, cuda 11.3

Screenshots
If applicable, add screenshots to help explain your problem.
System info (please complete the following information):

OS: [Ubuntu 20]
GPU count and types [x8 A100s each]
Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
Python version
Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
Docker context
Are you using a specific docker image that you can share?
Additional context
Add any other context about the problem here.

this problem turns out the bad node issue. it can be closed.

bad node issue it turns out to be

bad node issue it turns out to be

Hi @amsword, I wonder how did you identify the bad node in your case? I am also running in a similar situation, but I am not sure how to identify which node is in bad situation. Thank you!

authors: amsword joemzhao

repo: microsoft/DeepSpeed


text:
[BUG] Deepspeed training integration failing with WizardCoder

To Reproduce
I am using the WizardCoder training script to further fine-tune the model on some examples that I have using DeepSpeed integration. I have followed their instructions given here to fine-tune the model and I am getting the following error:
Traceback
datachat_env) root@C.6442427:~/Custom-LLM$ sh train.sh
[2023-06-23 00:36:25,039] [WARNING] [runner.py:191:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-06-23 00:36:25,077] [INFO] [runner.py:541:main] cmd = /root/anaconda3/envs/datachat_env/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None /root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py --model_name_or_path /root/Custom-LLM/WizardCoder-15B-V1.0 --data_path /root/Custom-LLM/data.json --output_dir /root/Custom-LLM/WC-Checkpoint --num_train_epochs 3 --model_max_length 512 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 4 --evaluation_strategy no --save_strategy steps --save_steps 50 --save_total_limit 2 --learning_rate 2e-5 --warmup_steps 30 --logging_steps 2 --lr_scheduler_type cosine --report_to tensorboard --gradient_checkpointing True --deepspeed /root/Custom-LLM/Llama-X/src/configs/deepspeed_config.json --fp16 True
[2023-06-23 00:36:26,992] [INFO] [launch.py:229:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2023-06-23 00:36:26,993] [INFO] [launch.py:235:main] nnodes=1, num_local_procs=4, node_rank=0
[2023-06-23 00:36:26,993] [INFO] [launch.py:246:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2023-06-23 00:36:26,993] [INFO] [launch.py:247:main] dist_world_size=4
[2023-06-23 00:36:26,993] [INFO] [launch.py:249:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
[2023-06-23 00:36:29,650] [INFO] [comm.py:622:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-06-23 00:36:55,124] [INFO] [partition_parameters.py:454:__exit__] finished initializing model with 15.82B parameters
[2023-06-23 00:37:12,845] [WARNING] [cpu_adam.py:84:__init__] FP16 params for CPUAdam may not work on AMD CPUs
[2023-06-23 00:37:12,968] [WARNING] [cpu_adam.py:84:__init__] FP16 params for CPUAdam may not work on AMD CPUs
[2023-06-23 00:37:12,969] [WARNING] [cpu_adam.py:84:__init__] FP16 params for CPUAdam may not work on AMD CPUs
[2023-06-23 00:37:12,970] [WARNING] [cpu_adam.py:84:__init__] FP16 params for CPUAdam may not work on AMD CPUs
Using /root/.cache/torch_extensions/py311_cu118 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py311_cu118/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/TH -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++14 -g -Wno-reorder -L/usr/local/cuda/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -c /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
FAILED: cpu_adam.o
c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/TH -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++14 -g -Wno-reorder -L/usr/local/cuda/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -c /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
In file included from /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes/cpu_adam.h:19,
                 from /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp:6:
/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes/custom_cuda_layers.h:12:10: fatal error: curand_kernel.h: No such file or directory
   12 | #include <curand_kernel.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
Using /root/.cache/torch_extensions/py311_cu118 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu118 as PyTorch extensions root...
Using /root/.cache/torch_extensions/py311_cu118 as PyTorch extensions root...
[2/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/TH -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=compute_80 -c /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
FAILED: custom_cuda_kernel.cuda.o
/usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/TH -isystem /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/anaconda3/envs/datachat_env/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=compute_80 -c /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o
In file included from /root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu:6:
/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/csrc/includes/custom_cuda_layers.h:12:10: fatal error: curand_kernel.h: No such file or directory
   12 | #include <curand_kernel.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
    subprocess.run(
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 247, in <module>
    train()
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 241, in train
    trainer.train()
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1741, in _inner_training_loop
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
                                                ^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/deepspeed.py", line 378, in deepspeed_init
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/__init__.py", line 165, in initialize
    engine = DeepSpeedEngine(args=args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 308, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1218, in _configure_basic_optimizer
    optimizer = DeepSpeedCPUAdam(model_parameters,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
    self.ds_opt_adam = CPUAdamBuilder().load()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 445, in load
    return self.jit_load(verbose)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in jit_load
Loading extension module cpu_adam...
op_module = load(name=self.name,
                ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1284, in load
Traceback (most recent call last):
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 247, in <module>
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
    train()
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 241, in train
    trainer.train()
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1664, in train
    _write_ninja_file_and_build_library(
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1741, in _inner_training_loop
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'cpu_adam'
Loading extension module cpu_adam...
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
                                                ^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/deepspeed.py", line 378, in deepspeed_init
Traceback (most recent call last):
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 247, in <module>
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
                         train()
               File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 241, in train
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    trainer.train()  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/__init__.py", line 165, in initialize

  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1664, in train
    engine = DeepSpeedEngine(args=args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 308, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1741, in _inner_training_loop
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1218, in _configure_basic_optimizer
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
                                                ^^    ^optimizer = DeepSpeedCPUAdam(model_parameters,^
^^ ^ ^ ^ ^ ^ ^ ^ ^ ^
    File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/deepspeed.py", line 378, in deepspeed_init
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
                self.ds_opt_adam = CPUAdamBuilder().load()
                                                       ^ ^ ^ ^ ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 445, in load
^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/__init__.py", line 165, in initialize
    return self.jit_load(verbose)
engine = DeepSpeedEngine(args=args,
                      ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in jit_load
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 308, in __init__
    self._configure_optimizer(optimizer, model_parameters)
    op_module = load(name=self.name,
        File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer
          ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
                      ^^^^^^^^^^^^^    ^return _jit_compile(^
^^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1535, in _jit_compile
^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1218, in _configure_basic_optimizer
    optimizer = DeepSpeedCPUAdam(model_parameters,
       return _import_module_from_library(name, build_directory, is_python_module)
             ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
^^^^^^^^^^^^^^^^^^^^^^^^    ^self.ds_opt_adam = CPUAdamBuilder().load()^
^^^^^^^^^ ^ ^ ^ ^ ^ ^ ^
    File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1929, in _import_module_from_library
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 445, in load
    return self.jit_load(verbose)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in jit_load
    module = importlib.util.module_from_spec(spec)
             ^^^^^^^^^^^^^^^^    ^op_module = load(name=self.name,^
^^^^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
^^  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
^^  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
^^  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
^^ImportError^: ^/root/.cache/torch_extensions/py311_cu118/cpu_adam/cpu_adam.so: cannot open shared object file: No such file or directory^
^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1535, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1929, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py311_cu118/cpu_adam/cpu_adam.so: cannot open shared object file: No such file or directory
Loading extension module cpu_adam...
Traceback (most recent call last):
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 247, in <module>
    train()
  File "/root/Custom-LLM/WizardLM/WizardCoder/src/train_wizardcoder.py", line 241, in train
    trainer.train()
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/trainer.py", line 1741, in _inner_training_loop
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
                                                ^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/transformers/deepspeed.py", line 378, in deepspeed_init
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/__init__.py", line 165, in initialize
    engine = DeepSpeedEngine(args=args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 308, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1162, in _configure_optimizer
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 1218, in _configure_basic_optimizer
    optimizer = DeepSpeedCPUAdam(model_parameters,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 94, in __init__
    self.ds_opt_adam = CPUAdamBuilder().load()
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 445, in load
    return self.jit_load(verbose)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in jit_load
    op_module = load(name=self.name,
                ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1535, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1929, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py311_cu118/cpu_adam/cpu_adam.so: cannot open shared object file: No such file or directory
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7fcaec4a89a0>
Traceback (most recent call last):
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
    ^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7fbf4e6409a0>
Traceback (most recent call last):
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
    ^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f9ce61b09a0>
Traceback (most recent call last):
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
Exception ignored in: <function DeepSpeedCPUAdam.__del__ at 0x7f6c2bf109a0>
Traceback (most recent call last):
  File "/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed/ops/adam/cpu_adam.py", line 102, in __del__
    self.ds_opt_adam.destroy_adam(self.opt_id)
    ^^^^^^^^^^^^^^^^
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'

Expected behavior
Expect the model to train using the deepspeed config file given
ds_report output
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/torch']
torch version .................... 2.0.1
deepspeed install path ........... ['/root/anaconda3/envs/datachat_env/lib/python3.11/site-packages/deepspeed']
deepspeed info ................... 0.9.2, unknown, unknown
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8

System info (please complete the following information):

OS: Ubuntu 20.4
GPU count and types 1 machine with 4xA100 80GB

I did some prodding and found that the curand_kernel.h file was not present in the /usr/local/cuda-11.8/targets/x86_64-linux/include directory thus causing the error in the traceback - fatal error: curand_kernel.h: No such file or directory. To solve this I manually pulled the .deb file associated with curand from here and installed them like so:
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/libcurand-11-8_10.3.0.86-1_amd64.deb
sudo dpkg -i libcurand-11-8_10.3.0.86-1_amd64.deb

This seemed to fix the issue

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
[REQUEST] DeepSpeed-Ulysses with the Pure Deepspeed Zero

Hi DeepSpeed folks. The issue here mentioned there will be a general solution for DeepSpeed-Ulysses with DeepSpeed zero. Can you share when it will be available? Thanks!

I transmit the 'mesh_params' parameter to deepspeed.initialize, which helps

authors: ppengtang yingtongxiong


text:
[BUG] CUDA error: no kernel image is available for execution on the device

Related issue: https://github.com/microsoft/DeepSpeed/issues/5724#issuecomment-2330819411
But I tried the solution and found it didn't work in my setting.
Describe the bug
[rank1]: Traceback (most recent call last):
[rank1]:   File "my_code.py", line 166, in train_model
[rank1]:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 1938, in train
[rank1]:     return inner_training_loop(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 2279, in _inner_training_loop
[rank1]:     tr_loss_step = self.training_step(model, inputs)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/transformers/trainer.py", line 3349, in training_step
[rank1]:     self.accelerator.backward(loss, **kwargs)
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/accelerate/accelerator.py", line 2188, in backward
[rank1]:     self.deepspeed_engine_wrapped.backward(loss, **kwargs)
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 175, in backward
[rank1]:     self.engine.step()
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2204, in step
[rank1]:     self._take_model_step(lr_kwargs)
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2110, in _take_model_step
[rank1]:     self.optimizer.step()
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1910, in step
[rank1]:     self._optimizer_step(i)
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1816, in _optimizer_step
[rank1]:     self.optimizer.step()
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/torch/optim/optimizer.py", line 484, in wrapper
[rank1]:     out = func(*args, **kwargs)
[rank1]:           ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/ops/adam/fused_adam.py", line 191, in step
[rank1]:     multi_tensor_applier(self.multi_tensor_adam, self._dummy_overflow_buf, [g_32, p_32, m_32, v_32],
[rank1]:   File "/opt/conda/lib/python3.11/site-packages/deepspeed/ops/adam/multi_tensor_apply.py", line 17, in call
[rank1]:     return op(self.chunk_size, noop_flag_buffer, tensor_lists, *args)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: RuntimeError: CUDA error: no kernel image is available for execution on the device
[rank1]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
To Reproduce
Steps to reproduce the behavior:
Trainer to train GPT with transformers=4.44.2 using pytorch 2.4 official docker (tried both cuda 12.4 and 12.1). I tried deepspeed 0.14.5 and 0.15.1 but both failed.
Expected behavior
No Error.
ds_report output
[2024-09-18 07:47:07,482] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (override)
[WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
[WARNING]  async_io: please install the libaio-dev package with apt
[WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/zero/linear.py:47: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
@autocast_custom_fwd
/opt/conda/lib/python3.11/site-packages/deepspeed/runtime/zero/linear.py:66: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='cuda') instead.
@autocast_custom_bwd
DeepSpeed C++/CUDA extension op report
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
JIT compiled ops requires ninja
ninja .................. [OKAY]
op name ................ installed .. compatible
[WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
[WARNING]  async_io: please install the libaio-dev package with apt
[WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [YES] ...... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
[WARNING]  FP Quantizer is using an untested triton version (3.0.0), only 2.3.0 and 2.3.1 are known to be compatible with these kernels
fp_quantizer ........... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.4
[WARNING]  using untested triton version (3.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.11/site-packages/torch']
torch version .................... 2.4.0
deepspeed install path ........... ['/opt/conda/lib/python3.11/site-packages/deepspeed']
deepspeed info ................... 0.14.5, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 12.1
deepspeed wheel compiled w. ...... torch 2.4, cuda 12.1
shared memory (/dev/shm) size .... 994.00 GB
System info (please complete the following information):

OS: [e.g. Ubuntu 18.04]
GPU count and types: 8 H100
Python version: 3.11
transformers==4.44.2

Launcher context
deepspeed my_code.py
Docker context
https://hub.docker.com/layers/pytorch/pytorch/2.4.0-cuda12.1-cudnn9-devel/images/sha256-a55ff10111eb11f998884327d37361592e632899edd24fce99886b69289e33e6?context=explore

Hi @getao - if you do build with DS_BUILD_FISED_ADAM=1 pip install deepspeed do you get the same erorr?

Hi @getao - if you do build with DS_BUILD_FISED_ADAM=1 pip install deepspeed do you get the same erorr?

Yes, I built with DS_BUILD_FUSED_ADAM=1.
BTW, I don't get the error in A100 GPUs. Only hopper GPUs will get the error.

Thanks @getao and @SamMicheals for confirming, we will need to take a closer look at this

same issue +1.
OS: Linux
GPU count and types: 8 x H100 80GB
(if applicable) Hugging Face Transformers/Accelerate/etc. versions: transformers@4.40 & 4.42
Python version: 3.11

same issue +1.
OS: Ubuntu20.04
GPU count and types: 8 x H100 80GB
CUDA Version: 11.8
Pytorch: 2.3.1+cu118
Python: 3.10.15
deepspeed: 0.15.3 & 0.14.5
【2024-10-25 13:03:33】[rank0]: File "/nfs/volume-764-2/lirui/envs/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1816, in _optimizer_step
1181
【2024-10-25 13:03:33】[rank0]: self.optimizer.step()
1182
【2024-10-25 13:03:33】[rank0]: File "/nfs/volume-764-2/lirui/envs/lib/python3.10/site-packages/torch/optim/optimizer.py", line 391, in wrapper
1183
【2024-10-25 13:03:33】[rank0]: out = func(*args, **kwargs)
1184
【2024-10-25 13:03:33】[rank0]: File "/nfs/volume-764-2/lirui/envs/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py", line 191, in step
1185
【2024-10-25 13:03:33】[rank0]: multi_tensor_applier(self.multi_tensor_adam, self._dummy_overflow_buf, [g_32, p_32, m_32, v_32],
1186
【2024-10-25 13:03:33】[rank0]: File "/nfs/volume-764-2/lirui/envs/lib/python3.10/site-packages/deepspeed/ops/adam/multi_tensor_apply.py", line 17, in call
1187
【2024-10-25 13:03:33】[rank0]: return op(self.chunk_size, noop_flag_buffer, tensor_lists, *args)
1188
【2024-10-25 13:03:33】[rank0]: RuntimeError: CUDA error: no kernel image is available for execution on the device
1189
【2024-10-25 13:03:33】[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
1190
【2024-10-25 13:03:33】[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
1191
【2024-10-25 13:03:33】[rank0]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Only H800 GPUs can reproduce this bug. A100s GPU cannot reproduce this bug.
When I choose to run with DeepspeedCPUAdam, H100s GPU got no error.

authors: bwnotfound getao loadams mzamini92


text:
Create accelerator for apple silicon GPU Acceleration

May fix #3364 and fix #1580

@NripeshN please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement
Contribution License Agreement
This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”), and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your contributions to Microsoft open source projects. This Agreement is effective as of the latest signature date below.

Definitions.
“Code” means the computer software code, whether in human-readable or machine-executable form,
that is delivered by You to Microsoft under this Agreement.
“Project” means any of the projects owned or managed by Microsoft and offered under a license
approved by the Open Source Initiative (www.opensource.org).
“Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
Project, including but not limited to communication on electronic mailing lists, source code control
systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
discussing and improving that Project, but excluding communication that is conspicuously marked or
otherwise designated in writing by You as “Not a Submission.”
“Submission” means the Code and any other copyrightable material Submitted by You, including any
associated comments and documentation.
Your Submission. You must agree to the terms of this Agreement before making a Submission to any
Project. This Agreement covers any and all Submissions that You, now or in the future (except as
described in Section 4 below), Submit to any Project.
Originality of Work. You represent that each of Your Submissions is entirely Your original work.
Should You wish to Submit materials that are not Your original work, You may Submit them separately
to the Project if You (a) retain all copyright and license information that was in the materials as You
received them, (b) in the description accompanying Your Submission, include the phrase “Submission
containing materials of a third party:” followed by the names of the third party and any licenses or other
restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
guidelines concerning Submissions.
Your Employer. References to “employer” in this Agreement include Your employer or anyone else
for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
Submission is made in the course of Your work for an employer or Your employer has intellectual
property rights in Your Submission by contract or applicable law, You must secure permission from Your
employer to make the Submission before signing this Agreement. In that case, the term “You” in this
Agreement will refer to You and the employer collectively. If You change employers in the future and
desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
and secure permission from the new employer before Submitting those Submissions.
Licenses.

Copyright License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
parties.
Patent License. You grant Microsoft, and those who receive the Submission directly or
indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
Your patent claims that are necessarily infringed by the Submission or the combination of the
Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
import or otherwise dispose of the Submission alone or with the Project.
Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
granted by implication, exhaustion, estoppel or otherwise.

Representations and Warranties. You represent that You are legally entitled to grant the above
licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
have disclosed under Section 3). You represent that You have secured permission from Your employer to
make the Submission in cases where Your Submission is made in the course of Your work for Your
employer or Your employer has intellectual property rights in Your Submission by contract or applicable
law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
You later become aware that would make Your representations in this Agreement inaccurate in any
respect.
Information about Submissions. You agree that contributions to Projects and information about
contributions may be maintained indefinitely and disclosed publicly, including Your name and other
information that You submit with Your Submission.
Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
defenses of lack of personal jurisdiction and forum non-conveniens.
Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
supersedes any and all prior agreements, understandings or communications, written or oral, between
the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.

@microsoft-github-policy-service agree

Hey DeepSpeed team @tjruwase @ShijieZZZZ @jeffra, could you take a look at my code here? I appreciate the feedback.

@NripeshN, thanks for contributing this exciting PR. We are reviewing asap.
Out of curiosity, are you already running DeepSpeed on the mps accelerator?

This is an excellent first step towards supporting apple silicon, thank you so much @NripeshN for initiating this. I've been doing some tests on my own M1 macbook air. I see a handful of other areas that will need fixes outside the abstract accelerator in order to run DeepSpeed training on M1. However, I think we'll need to have some follow-up PRs to address those.
One area I see is even with using a torch.distributed backend of 'gloo' I see errors when using collectives like broadcast or all_reduce. Currently DeepSpeed assumes these are callable even if there's only 1 accelerator being used. We probably need to add guards against calling these collectives in these cases, here's an example:
https://github.com/microsoft/DeepSpeed/blob/aa54dba02532f3f487cc635dfdace8fa95593db0/deepspeed/runtime/engine.py#L1033
After I added a few of these guards I was successfully able to get our simple/small model to run with fp32 weights https://github.com/microsoft/DeepSpeed/blob/master/tests/small_model_debugging/test_model.py I did get a seg fault when trying to use fp16 training, but I am not sure how stable that is on M1 even w/o DeepSpeed currently.

This is an excellent first step towards supporting apple silicon, thank you so much @NripeshN for initiating this. I've been doing some tests on my own M1 macbook air. I see a handful of other areas that will need fixes outside the abstract accelerator in order to run DeepSpeed training on M1. However, I think we'll need to have some follow-up PRs to address those.
One area I see is even with using a torch.distributed backend of 'gloo' I see errors when using collectives like broadcast or all_reduce. Currently DeepSpeed assumes these are callable even if there's only 1 accelerator being used. We probably need to add guards against calling these collectives in these cases, here's an example:
https://github.com/microsoft/DeepSpeed/blob/aa54dba02532f3f487cc635dfdace8fa95593db0/deepspeed/runtime/engine.py#L1033
After I added a few of these guards I was successfully able to get our simple/small model to run with fp32 weights https://github.com/microsoft/DeepSpeed/blob/master/tests/small_model_debugging/test_model.py I did get a seg fault when trying to use fp16 training, but I am not sure how stable that is on M1 even w/o DeepSpeed currently.

This is an excellent first step towards supporting apple silicon, thank you so much @NripeshN for initiating this. I've been doing some tests on my own M1 macbook air. I see a handful of other areas that will need fixes outside the abstract accelerator in order to run DeepSpeed training on M1. However, I think we'll need to have some follow-up PRs to address those.
One area I see is even with using a torch.distributed backend of 'gloo' I see errors when using collectives like broadcast or all_reduce. Currently DeepSpeed assumes these are callable even if there's only 1 accelerator being used. We probably need to add guards against calling these collectives in these cases, here's an example:
https://github.com/microsoft/DeepSpeed/blob/aa54dba02532f3f487cc635dfdace8fa95593db0/deepspeed/runtime/engine.py#L1033
After I added a few of these guards I was successfully able to get our simple/small model to run with fp32 weights https://github.com/microsoft/DeepSpeed/blob/master/tests/small_model_debugging/test_model.py I did get a seg fault when trying to use fp16 training, but I am not sure how stable that is on M1 even w/o DeepSpeed currently.

Hi, Thank you soo much!!

source: gharchive/pull-request

authors: NripeshN jeffra tjruwase


text:
Add document guid to metadata blob

AB#1861
How contribute to this repo.
Guidelines for Pull Requests.
Description
When files processed in MeTA service (for eDiscovery), we have no file identity information, i.e. no itemId/driveId that we can use to for docId for telemetry. Having this info is super useful to understand if problems happening in production are limited to small set of files, or not.

@agarwal-navin There's some failures on snapshot tests because of this new info in the ".metadata" blob. Should I add a runtime option to ignore this new property, or add it to some new blob?

@agarwal-navin There's some failures on snapshot tests because of this new info in the ".metadata" blob. Should I add a runtime option to ignore this new property, or add it to some new blob? https://dev.azure.com/fluidframework/public/_build/results?buildId=140633&view=ms.vss-test-web.build-test-results-tab

You need to update the snapshot tests everytime you change the summary format. You can follow instructions on how to update them here - https://github.com/microsoft/FluidFramework/blob/main/packages/test/snapshots/README.md

authors: agarwal-navin kian-thompson

repo: microsoft/FluidFramework

license: MIT


text:
Enable Synchronous Child Datastore Creation

Overview
This feature introduces a new pattern for creating datastores synchronously within the Fluid Framework. It allows for the synchronous creation of a child datastore from an existing datastore, provided that the child datastore is available synchronously via the existing datastore's registry and that the child's factory supports synchronous creation. This method also ensures strong typing for the consumer.
In this context, "child" refers specifically to the organization of factories and registries, not to any hierarchical or hosting relationship between datastores. The parent datastore does not control the runtime behaviors of the child datastore beyond its creation.
The synchronous creation of child datastores enhances the flexibility of datastore management within the Fluid Framework. It ensures type safety and provides a different way to manage datastores within a container. However, it is important to consider the overhead associated with datastores, as they are stored, summarized, garbage collected, loaded, and referenced independently. This overhead should be justified by the scenario's requirements.
Datastores offer increased capabilities, such as the ability to reference them via handles, allowing multiple references to exist and enabling those references to be moved, swapped, or changed. Additionally, datastores are garbage collected after becoming unreferenced, which can simplify final cleanup across clients. This is in contrast to subdirectories in a shared directory, which do not have native capabilities for referencing or garbage collection but are very low overhead to create.
Synchronous creation relies on both the factory and the datastore to support it. This means that asynchronous operations, such as resolving handles, some browser API calls, consensus-based operations, or other asynchronous tasks, cannot be performed during the creation flow. Therefore, synchronous child datastore creation is best limited to scenarios where the existing asynchronous process cannot be used, such as when a new datastore must be created in direct response to synchronous user input.
Key Benefits

Synchronous Creation: Allows for the immediate creation of child datastores without waiting for asynchronous operations.
Strong Typing: Ensures type safety and better developer experience by leveraging TypeScript's type system.

Use Cases
Example 1: Creating a Child Datastore
In this example, we demonstrate how to support creating a child datastore synchronously from a parent datastore.
/**
 * This is the parent DataObject, which is also a datastore. It has a
 * synchronous method to create child datastores, which could be called
 * in response to synchronous user input, like a key press.
 */
class ParentDataObject extends DataObject {
	get ParentDataObject() {
		return this;
	}
	protected override async initializingFirstTime(): Promise<void> {
		// create synchronously during initialization
		this.createChild("parentCreation");
	}

	createChild(name: string): ChildDataStore {
		assert(
			this.context.createChildDataStore !== undefined,
			"this.context.createChildDataStore",
		);
		// creates a detached context with a factory who's package path is the same
		// as the current datastore, but with another copy of its own type.
		const { entrypoint } = this.context.createChildDataStore(
			ChildDataStoreFactory.instance,
		);
		const dir = this.root.createSubDirectory("children");
		dir.set(name, entrypoint.handle);
		entrypoint.setProperty("childValue", name);

		return entrypoint;
	}

	getChild(name: string): IFluidHandle<ChildDataStore> | undefined {
		const dir = this.root.getSubDirectory("children");
		return dir?.get<IFluidHandle<ChildDataStore>>(name);
	}
}

For a complete example see the follow test:
https://github.com/microsoft/FluidFramework/blob/main/packages/test/local-server-tests/src/test/synchronousDataStoreCreation.spec.ts

This change brings back #22962 which was reverted for a minor incompatibility issue, which fully resolved in this version by splitting the get method on IFluidDataStoreRegistery into two, get and getSync.


text:
forward() got an unexpected keyword argument token_type_ids

I'm trying to run MiniLLM with Llama-2-7b as the teacher model. Not sure why this error is persisting -- my arguments passed into forward() in the PPOModel class do not contain token_type_id and I have included return_token_type_ids=False with the AutoTokenizer class.
All related issues I found online told me to upgrade the transformers version, but that doesn't seem to work because ParallelLlamaForCausalLM is implemented locally.

Nevermind, I just realized the tokenizer mismatch won't work :P

L honestly

repo: microsoft/LMOps


text:
Status of CVE-2024-43598?

Hi,
what is the status of CVE-2024-43598? It claims that all versions below 4.6.0 are vulnerable but the latest version is 4.5.0. So either the information is bogus or it was written by someone who knows that there will be no 4.5.1...
I do not see any commit around November that looks like a fix...
Is this CVE genuine at all or is lightgbm the victim of a bogus CVE?

Fixed by #6752.

authors: StrikerRUS tobiass-sdl

repo: microsoft/LightGBM

license: mit

license_source: bigquery


text:
Scrollbar of dropdown not working on built application

Describe the bug
Used the basic TMP_Dropdown from MRTK Examples and on built version (in hololens2) the scrollbar events not working.
The dropdown open in a right place, but can't click to the scrollbar handler and can't scroll it.
On editor it works well
A clear and concise description of what the bug is.
To reproduce
Steps to reproduce the behavior:

Add extra items to tmp_dropdown (in hand interaction examples scene) [to Dropdown gameobject's TMP_Dropdown items]
Build the application on hololens2
Open and interaction examples and try to scroll down the TMP_Dropdown
See the scrollbar at dropdown not working, only choosing items

Expected behavior
Can scroll on objects of the dropdown
Screenshots

Your setup (please complete the following information)

Unity Version [e.g. 2020.3.35f1]
MRTK Version [e.g. v2.8.2]

Target platform (please complete the following information)

HoloLens 2

Additional context
Add any other context about the problem here.

Are you using OpenXR? If so, there's a bit of a quirk with airtap and the old input system, leading to a "submit" being sent on airtap instead of the previous method of being able to select the dropdown and scroll it.
There's a potential workaround at https://github.com/microsoft/MixedRealityToolkit-Unity/issues/10304#issuecomment-1040503841, if you're able to give it a try! Thanks!

That was the problem, sorry for the late reply.

authors: BalintOXIT keveleigh

repo: microsoft/MixedRealityToolkit-Unity


text:
Scroll View Graduation - Reseting Child Interaction when dragging

Overview
This PR is part of Scroll View graduation and addresses issues related to interaction with children elements after scroll view is engaged into drag:

Pressable Buttons never trigger OnPressCompleted.
When a collection of pressable buttons is added to ScrollingObjectCollection, OnPressCompleted event fire is insconsistent.

Pressable buttons do not reset their state after input is cancelled.
When a pressable button inside a ScrollingObjectCollection is pressed, button reset (retract) is not happening. This happens if user presses the button slowly, if user engages in a scroll right after button press, or if input is cancelled.

Scroll engage should stop any content interaction

Solution proposed
Using the event propagation feature branch, scroll view can register for touch events on trickle down phase allowing it to catch touch events before its children. If scroll view is engaged in drag, it calls eventData.Use() allowing children to handle cancellation if event is used. On Propagation branch event propagation is only stopped by calling eventData.StopPropagation(), so we are using eventData.used as a flag for cancellation.

More performant and clear option could be to create and use an interface for interactable cancellation and raise something like an ICancellable.CancelInteraction  and StopPropagation after cancellation sent.

Changes
The PR is based on Event Propagation feature branch.
Fixes #7444
Fixes #7445
Fixes #7446

Thank you, Vanessa! @vaoliva
@radicalad for visibility.

/azp run

/azp run mrtk_PR

/azp run mrtk_PR

Closing this as the PR was substituted for #8091.


text:
Add OpenXR input data providers

Overview
REQUIRES UNITY 2020.2
To get started with this (real docs will be coming here soon), follow the steps on https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/openxr-getting-started.
Then, in any MRTK example scene (I typically use HandInteractionExamples), switch your active profile to the default OpenXR one provided in this PR.

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run

/azp run


text:
Query Parameters Serialization - Explode don't respect the default value.

When the style: form is present in the specification and explode not, the default value is being treated as false instead of true.
Using the example below:
openapi: 3.0.1
info:
  title: 'Tests v1'
  version: v1
paths:
  '/tests':
    get:
      tags:
        - Test
      summary: 'Test'
      description: |-
        Test Operation
      operationId: GetAsync
      parameters:
        - name: test
          description: "test description"
          in: query
          style: form
          schema:
            type: array
            items:
              type: integer
              format: int32
      responses:
        '200':
          description: 'OK'
          content:
            application/json:
              schema:
                type: string

I just looked around the code and I could see that the object is assuming the default value for a boolean property in .NET
One thing that could be done is to force the default value to be true, what do you think about that?

This is merged into the vnext branch and will release in 1.3

authors: darrelmiller nicolastakashi


text:
python crashed in vs2019 when running kmeans ?

I use visual studio 2019 to open a folder without open any file. However, the above program crashes periodically and a front windows report this crash.

Thanks for the report.
Could you please provide the output of pip freeze for your Python environment so I can recreate an exact copy? Thanks!

Ok, I will try.

I use python version 3.7.3 from official website, visual studio version is 16.0.3, os version is win7 professional.
Below is the output of pip freeze.
aliyun-python-sdk-core==2.13.4
aliyun-python-sdk-core-v3==2.13.3
aliyun-python-sdk-kms==2.6.0
bleach==3.1.0
certifi==2019.3.9
chardet==3.0.4
crcmod==1.7
cycler==0.10.0
decorator==4.4.0
docutils==0.14
easydict==1.9
ete3==3.1.1
graphviz==0.10.1
idna==2.8
jmespath==0.9.4
joblib==0.13.2
kiwisolver==1.1.0
mahotas==1.4.5
matplotlib==3.0.3
networkx==2.3
numpy==1.16.3
oss2==2.6.1
pkginfo==1.5.0.1
pycryptodome==3.8.1
Pygments==2.4.0
pyparsing==2.4.0
PyQt5==5.12.2
PyQt5-sip==4.19.17
python-dateutil==2.8.0
PyYAML==5.1
readme-renderer==24.0
requests==2.21.0
requests-toolbelt==0.9.1
scikit-learn==0.20.3
scipy==1.2.1
six==1.12.0
sklearn==0.0
tabulate==0.8.3
tqdm==4.32.1
twine==1.13.0
urllib3==1.24.3
webencodings==0.5.1
wget==3.2
zss==1.2.0

repo: microsoft/PTVS


text:
Fix bug, add test case for check.GetEvaluator()

Fix bug, add test case for check.GetEvaluator()
Demote AllSymbols  from internal property to private field to protect against misuse.

✅ No public API change.

authors: LucGenetier MikeStall

repo: microsoft/Power-Fx


text:
ServiceClient update does an upsert?

When using the ServiceClient update method using a record ID that does not exist in the system, a new record is being created.
Before, this would throw an error.
Executing the following lines of codes results in a new account record.
ServiceClient service = new ServiceClient(connectionString); Entity updater = new Entity("account", Guid.NewGuid()); updater["name"] = "Updated name"; service.Update(updater);
Is this expected behavior now? Or will this be treated as a bug and fix can be expected in a future release?

Yes, that is expected based on the current design of the WebAPI,

If I execute an update using the WebAPI I get this error, which I was expecting

I can also find the following information
https://docs.microsoft.com/en-us/powerapps/developer/data-platform/webapi/perform-conditional-operations-using-web-api#prevent-create-in-upsert
So, is there any way to overwrite this behavior when using the ServiceClient?

Yes, this is a byproduct of the way the OrganizationService to WebAPI back to Organziation Service translation is happening.
The decision of the API team was to allow it to work this way,  I will discuss it with them in the next meeting as it is a deviation in behavior of the OrganizationService API.

We decided to put the Organization Service Behavior back in place.
should be fixed shortly.

authors: MattB-msft benoitwouter


text:
[D7SMS] Updating outdated brand names

Updated brand names like below:
Microsoft Flow -> Microsoft Power Automate
Flow -> Power Automate (Descriptions & Summaries)
PowerApps -> Power Apps
LogicApps -> Logic Apps

@d7networks Please approve the change and let me know if you have any questions.

repo: microsoft/PowerPlatformConnectors


text:
IN-D Invoice Data Capture

When submitting a connector, please make sure that you follow the requirements below, otherwise your PR might be rejected. We want to make you have a well-built connector, a smooth certification experience, and your users are happy :)
If this is your first time submitting to GitHub and you need some help, please sign up for this session.

[ ] I attest that the connector doesn't exist on the Power Platform today. I've verified by checking the pull requests in GitHub and by searching for the connector on the platform or in the documentation.
[ ] I attest that the connector works and I verified by deploying and testing all the operations.
[ ] I attest that I have added detailed descriptions for all operations and parameters in the swagger file.
[ ] I attest that I have added response schemas to my actions, unless the response schema is dynamic.
[ ] I validated the swagger file, apiDefinition.swagger.json, by running paconn validate command.
[ ] If this is a certified connector, I confirm that apiProperties.json has a valid brand color and doesn't use an invalid brand color, #007ee5 or #ffffff. If this is an independent publisher connector, I confirm that I am not submitting a connector icon.

If you are an Independent Publisher, you must also attest to the following to ensure a smooth publishing process:

[ ] I have named this PR after the pattern of "Connector Name (Independent Publisher)" ex: HubSpot Marketing (Independent Publisher)
[ ] Within this PR markdown file, I have pasted screenshots that show: 3 unique operations (actions/triggers) working within a Flow. This can be in one flow or part of multiple flows. For each one of those flows, I have pasted in screenshots of the Flow succeeding.
[ ] Within this PR markdown file, I have pasted in a screenshot from the Test operations section within the Custom Connector UI.
[ ] If the connector uses OAuth, I have provided detailed steps on how to create an app in the readme.md.

Hello @v-EgorKozhadei,
As per your guidance, I have raised the PR, So I kindly request you to merge it. we need our connector wants to publish in the Powerautomate platform. So, please help me to certificate this connector (In-D Invoice Data Capture)
Thank you so much for supporting us.
Warm Regards,
Gayathri. V

Hello @gayathri35,
There's a conflict between branches. We can't approve and merge the PR at the moment.
Please make sure that your branch is up-to-date and there're no conflicts.
Let us know if you have any questions.

Hello @gayathri35,
Congratulations, your Pull Request is approved and merged.
Please update your submission in ISV Studio by using the latest commit ID.
Let us know if you have any questions.


text:
Web search in powertoys run

Provide a description of the new feature / enhancement
A person could type a query into powertoys run and receive search results. For example, I could type in "python tutorial" and get a google search for "python tutorial".
Thanks for reading ;)

tracked by https://github.com/microsoft/PowerToys/issues/3245

authors: S-Spektrum-M crutkas

repo: microsoft/PowerToys


text:
Enable PDF (.pdf) preview toggle breaks either explorer preview --OR-- outlook preview

Microsoft PowerToys version
0.56.2
Running as admin

[X] Yes

Area(s) with issue?
General
Steps to reproduce
Hours with IT on my work PC:
When I turn Power Toys >> File Explorer Add-ons >> Enable PDF (.pdf) preview >> toggle to "On", the preview window in OUTLOOK says "this file cannot be previewed because there is no previewer installed for it." but I am able to view PDFs in Explorer preview window.
When I turn Power Toys >> File Explorer Add-ons >> Enable PDF (.pdf) preview >> toggle to "OFF", the preview window in OUTLOOK displays the PDF but the preview window Explorer for PDFs preview window says "File Not Found.  It may have been moved or edited"
Toggle back and forth and the problem moves from Outlook to Explorer.  Exclusive NOR.  One or the other can function as intended, not both.
All windows updates are executed.  all power toys updates are in place from general tab.  V0.56.2 last checked 3/9/2022 1:09:14 PM
✔ Expected Behavior
previews to work
❌ Actual Behavior
toggle breaks either outlook or explorer depending on position.
Other Software
outlook and windows explorer

cc: @Aaron-Junker fyi.

/dup #14283

Here's an official helparticle to this topic: https://support.microsoft.com/en-gb/office/outlook-pdf-preview-error-this-file-cannot-be-previewed-because-there-is-no-previewer-installed-for-it-b78bfe24-ba13-44bd-8289-1919ece7ded2

authors: Aaron-Junker AmyMBeaudry franky920920 jaimecbernardo


text:
PowerToys crashing on opening

Microsoft PowerToys version
0.57.2.0
Running as admin

[X] Yes

Area(s) with issue?
General
Steps to reproduce
Simply opening the application.
✔ Expected Behavior
PowerToys opens as usual.
❌ Actual Behavior
The "PowerToys Run ran into an issue" box opens up instead requesting that I file the bug here.

2022-08-02.txt
Other Software
No response

Hi @davidbarrettnyc , 0.57.2 is quite old. Do you have enough space in the disk?
Does this also occur with 0.61? /needinfo

authors: davidbarrettnyc jaimecbernardo


text:
PowerToys Run ran into an issue

Microsoft PowerToys version
0.62.1.0
Running as admin

[ ] Yes

Area(s) with issue?
PowerToys Run
Steps to reproduce
Whenever I start my Windows, the window of Powertoys opens with the title "PowerToys Run ran into an issue"
error statement
Version: 0.62.1.0
OS Version: Microsoft Windows NT 10.0.19044.0
IntPtr Length: 8
x64: True
Date: 11/10/2022 08:13:27
Exception:
System.BadImageFormatException: Could not load file or assembly 'Microsoft.Toolkit.Uwp.Notifications, Version=7.1.0.0, Culture=neutral, PublicKeyToken=4aff67a105548ee2'. Format of the executable (.exe) or library (.dll) is invalid.
File name: 'Microsoft.Toolkit.Uwp.Notifications, Version=7.1.0.0, Culture=neutral, PublicKeyToken=4aff67a105548ee2'
   at Wox.PublicAPIInstance..ctor(SettingWindowViewModel settingsVM, MainViewModel mainVM, ThemeManager themeManager)
   at PowerLauncher.App.<>c__DisplayClass18_0.<OnStartup>b__0()
   at Wox.Infrastructure.Stopwatch.Normal(String message, Action action)
   at PowerLauncher.App.OnStartup(Object sender, StartupEventArgs e)
   at System.Windows.Application.OnStartup(StartupEventArgs e)
   at System.Windows.Application.<.ctor>b__1_0(Object unused)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)

log file generated
2022-10-11.txt
✔ Expected Behavior
I was not expecting anything, I just start the Windows, and this PowerToys should've started without giving any error.
❌ Actual Behavior
after starting my Windows, Powertoys just show error windows with the title "PowerToys Run ran into an issue"
Other Software
No response

/bugreport

I have added the .zip file as you asked
But right after I faced the issue I upgraded the version to 0.63.0. and was not facing any issues after the upgrade.
PowerToysReport_2022-10-19-20-54-33.zip

can you try and update to a new release (.69.1 or newer as the time of writing this)?  This should have been resolved when we went self contained  /needinfo

I don't have the same operating system installed anymore, so I cannot try it.

authors: Ahwar crutkas jaimecbernardo


text:
Top Tabs in file explorer for windows 10

Description of the new feature / enhancement
Having Tabs at the top of File Explorer Like in Windows 11 so that people still on Windows 10 can have the same feature.
Scenario when this would be used?
This would be used when moving files around in file explorer so that the user would not have to have more than one window open.
Supporting information
No response

No plans. Please upgrade to windows 11 to use this feature.


text:
Multiple empty error dialogs thown

Microsoft PowerToys version
0.76.1
Installation method
PowerToys auto-update
Running as admin
No
Area(s) with issue?
General
Steps to reproduce
Keep on having an empty dialog telling that PowerToys ran into an issue, a link to GitHub, but no error description. Seems like it thows them on click away from the active dialog.
✔ Expected Behavior
No response
❌ Actual Behavior
No response
Other Software
No response

@KurtBildeSDU Would you mind posting a screenshot of the error window?

/bugreport

Please upgrade to 0.76.2 to fix this bug.
/dup https://github.com/microsoft/PowerToys/issues/30357

authors: KurtBildeSDU crutkas davidegiacometti niels9001


text:
screen ruler - move toolbar

Description of the new feature / enhancement
add option to move the screen ruler toolbar in the screen.
currently it is alays on the center screen on the top
Scenario when this would be used?
thsi should be used always. the option to move the toolbar should be enabled always
Supporting information

/dup #28335

authors: Aaron-Junker Nitsan-Slovik


text:
Gaming and Powertools

Microsoft PowerToys version
0.81.1
Installation method
Microsoft Store
Running as admin
Yes
Area(s) with issue?
Keyboard Manager
Steps to reproduce
Playing a game that use left alt, wads and tabs keys
✔ Expected Behavior
blocking alt=>w=>tab combo in keyboard tools to block combo
❌ Actual Behavior
i press and hold left alt, then press and hold w then press and hold tab, it bring up window selection screen. seams powertools does not support this key combination. which is an issue for gammers
Other Software
world of warcraft?

hey this is just a little unclear, can you specify the exact remapping pair you want to function correctly? (some of the open or closed similar issues might be reporting the same things? if you do see a duplicate, feel free to close this issue and +1 onto that issue!)
/needinfo

authors: SultonMRP ethanfangg


text:
powertoys run not apering right after install

ℹ Computer information

PowerToys version: 0.27.1
PowerToy Utility:
Running PowerToys as Admin: no
Windows build number: 20h2

📝 Provide detailed reproduction steps (if any)

…install powertoys
…go on desktop
…press alt+spacebar

✔ Expected result
power toys run opens
❌ Actual result
power toys run doesn't open(it opens after like 5 minutes after instalation)
📷 Screenshots
Are there any useful screenshots? WinKey+Shift+S and then just paste them directly into the form

How are you installing PowerToys?  Do you see a nice icon in your systray like this?

authors: ROTOMEGA crutkas


text:
Adds StlThumbnailProvider

Summary of the Pull Request
What is this about:
Adds a new thumbnail provider for stl files (3D model files).
Screenshot of the thumbnails:

Screenshot of the settings page:

What is included in the PR:
A new thumbnail provider and unit tests for it.
How does someone test / validate:

Close PowerToys if installed in your machine
Full build the solution: PowerToys.sln
Start PowerToys from PowerToys\x64\Debug\PowerToys.exe or PowerToys\x64\Release\PowerToys.exe
Toggle the "Gcode thumbnail previewer" setting to enable
Open HelperFiles folder on the tests and check if the icon changes to an image
Check explorer preview to see if image is also shown there

Quality Checklist

[x] Linked issue: #9695
[x] Communication: I've discussed this with core contributors in the issue.
[x] Tests: Added/updated and all pass
[ ] Installer: Added/updated and all pass
[x] Localization: All end user facing strings can be localized
[ ] Docs: Added/ updated
[x] Binaries: Any new files are added to WXS / YML

[ ] No new binaries
[ ] YML for signing for new binaries
[ ] WXS for installer for new binaries

Contributor License Agreement (CLA)
A CLA must be signed. If not, go over here and sign the CLA.

Though it seems to be working mostly fine, I've noticed something very odd: sometimes the thumbnail background goes black!!

For the record, as this is generating transparent images, I made it return pdwAlpha = WTS_ALPHATYPE.WTSAT_ARGB instead of WTS_ALPHATYPE.WTSAT_RGB (no alpha channel) like the other providers have, but I don't see how this would be causing the problem I am seeing here...
@jaimecbernardo @yuyoyuppe do you have any insight on this?

That issue seems to happen inconsistently, I'm trying some things to mitigate it, but this might not be a problem with the extension but an issue in Windows or my drivers...
Anyway, here's a screenshot in dark mode:

Note: this is taking a dependency on Helix Toolkit.
@crutkas if a new full dependency is something one would prefer to avoid, the toolkit is MIT licensed, so there should be no problem to copy in only the required files to make this work.
Having said that, I am not advocating for that (copying the classes in) and would prefer to actually take the dependency as it means someone else with correct knowledge is maintaining them!

By the way, the only reason I have not added a Preview Handler with this is because Microsoft 3D Viewer already has a really good one... only problem is that it requires the app to be installed AND to open STLs in it by default!

@jaimecbernardo @yuyoyuppe do you have any insight on this?

No insight here. I'm assuming the color is being added by the code you've been using.
Thanks for contributing! I'm going to assign some reviewers.

Please don't merge this yet!!
I merged main branch to main and something is now broken, so I'm trying to figure out what it is...

So I traced the issue to ascii based STLs, the HelixToolkit StLReader is failing to parse those after changing the TargetFramework from netcoreapp3.1 to net5.0-windows!
Further tests show that the problem is a string.StartWith("\0") check that is done inside HelixTookit, but caused by different behavior between .netcoreapp31 and .net5
Consider the following line of code:
"test".StartsWith("\0");

The above will return false in .netcoreapp31, but true for .net5 and .net6...
I'm trying to get some further information on why this change of behavior: https://twitter.com/pedrolamas/status/1483769746227146756

The plot thickens! https://github.com/dotnet/runtime/issues/50521#issuecomment-811523668

Moved to draft

Today I submitted a PR to fix the .net5 compatibility issue in Helix Toolkit and that was validated, merged and a new release of the toolkit was pushed to NuGet.
I've now updated said NuGet on the project and confirmed that the issue is now properly fixed.
Given this, I'm putting the PR back to ready for review & merge.

@pedrolamas looks like sln is conflicted

@crutkas merge conflict fixed! 🙂

authors: crutkas jaimecbernardo pedrolamas


text:
Requesting ARM64 version of Procdump

In order to support efficiency scenarios for a large 1P Microsoft cloud platform, would like to request a version of procdump targeting mariner 2.0+ and ARM64. Ideally published to the same PMC
Key scenario is the ability to take on demand dumps of applications.
Have chatted with @MarioHewardt internally about this as well.

Hi - We are actively working on ARM64 support and should have it release ready be end of January.

authors: MarioHewardt suprak

repo: microsoft/ProcDump-for-Linux


text:
Retrieving queue properties for storage account fails: Context Deadline exceeded

Before you open an issue, please check if a similar issue already exists or has been closed before.
You can also find details on Troubleshooting Common Issues. You can use these tools to help gather additional logs and details to include in  your issue.
:warning: Please DO NOT include confidential information in your issue on GitHub. :warning:
Bug Details
Describe the bug
Terraform is unable to contact the storage account when attempting to redeploy resources that are stored in a state file.
Error: retrieving queue properties for Storage Account (Subscription: "<subscripiton_id>"
│ Resource Group Name: "<resource_group>"
│ Storage Account Name: ""): executing request: Get "https://.queue.core.usgovcloudapi.net/?comp=properties&restype=service": context deadline exceeded
│
Steps To Reproduce

Deploy resource with state being stored in a storage account.
Re-deploy resources, deployment will fail trying to get queue information for storage account.
...

What is the expected behavior?
Re-deployments should complete successfully modifying the resources only if the terraform has changed.
Screenshots
If applicable, add screenshots to help explain your problem.
Information Assistant details
Please provide the following details. You can simply include a screenshot of your Info panel as well.

GitHub branch: [e.g. main]
Version or Latest commit: [obtained by running git log -n 1 <branchname>
What region is your Azure Open AI Service in?
What ChatGPT model are you using?
model name: (i.e. gpt-3.5-turbo, gpt-4)
model version: (i.e. 0613)
What embeddings model are you using?

Additional context
Add any other context about the problem here.
If the bug is confirmed, would you be willing to submit a PR?

[ ] Yes
[ ] No

Which version of IA are you deploying?  Is it in secure mode?  Have you tried multiple attempts to redeploy?

Version 1.2
We are using a modified version of secure mode (targeting an existing virtual network and centrally managed private link DNS zones)
Initial deploy completes but on any new pipeline run the deployment fails with that error.

Ensure where you are running the deployment has access to the network restricted assets.  After initial deployment, the Azure resources are network restricted and you must ensure network connectivity (VPN, jump box, etc) to communicate with the resources on redeployment.  In rare circumstances, attempt the deployment again to see if that alleviates the Terraform to Azure issue.
Are you running the version 1.2 deployment on a Government Cloud environment?

This deployment is in Azure US Gov yes. Bonus complications include GitLab and the Runner pool are deployed in AWS, that traffic traverses back to our Datacenter then across our express route to Azure. Checking with the various FW teams to make sure all the services involved can talk to each other.

Updates:
Found an internal block from our Palo Alto Firewalls. There are unique rules for US Gov storage

The GitLab runner IP range does not have these specific rules applied. A request has been made to add them and I'll hopefully test again tomorrow

repo: microsoft/PubSec-Info-Assistant


text:
TeamCloud website cannot allow to login

Describe the bug
According to the documentation, I have registered the AppConfiguration and SignalRService and successfully deploy the TeamCloud instance. However, when I access the TeamCloud website, it doesn't allow to login.
The error is as below:

looks like there is no permission as admin.
To Reproduce
Steps to reproduce the behavior:
1, in App Registration part, input the "api://" as the application ID URI
2, pass the scope during the deployment az tc deploy --scope "api:///user_impersonation"
3，access the TeamCloud website "https://teamcloudapp2dev.azurewebsites.net"
Actual result: The error above will be displayed.
Expected behavior
The page should allow me to login
Additional context

This is something you need to take up with your directory (tenant) admin. Your tenant has restricted users from granting the necessary permissions to aad apps, so your admin has to grant the approval. This is not an issue in TeamCloud.

This is something you need to take up with your directory (tenant) admin. Your tenant has restricted users from granting the necessary permissions to aad apps, so your admin has to grant the approval. This is not an issue in TeamCloud.

Thanks for the reply.
Some customers may have the same permission as mine. Or they have even more restricted rules. So they need to ask tenant admin to approve for Team Cloud try? Any workaround?

Yes, you can use another tenant that does not have these restrictions.

Yes, you can use another tenant that does not have these restrictions.

Thanks Rouke. Now I change to another limited cash subscription. The issue is resolved. However, I found that subscription cannot be input and subscription list is empty, is there any idea to solve this? I will create another thread to address this new issue.

@luxu-ms similar to my comment in issue #334, the TeamCloud's service principal needs contributor permissions on the subscription.  Let me know if that fixes this issue.

Tracking the related documentation update task in issue #336.
For MSFT employees that stumble across this issue, TeamCloud instances do not currently work within the Microsoft tenant due to restrictions to constraints on permissions that can be granted to a service principal.
Closing.

authors: colbylwilliams luxu-ms rouke-broersma

repo: microsoft/TeamCloud


text:
Formatting issue for String type docs in Basic Types.md

Text on lines 60-62 format didn't to appear to be within the template string itself. Found this issue on the typescript website when searching the docs. I will create a PR to fix issue.

Created PR to solve issue.

repo: microsoft/TypeScript-Handbook


text:
Fixes wrong generator reference

Hey there, seems like the last example in the noImplicitAny Errors Apply to Loose yield Expressions section of 4.2 release notes actually meant to refer to the g4 generator function, if I understood this snippet correctly.
I'll also use this tiny PR as an opportunity to express my gratitude to the TypeScript team. You rock!

👍🏻 yep, that fix is right -  thanks!

repo: microsoft/TypeScript-Website


text:
feat: upgrade react-intl

I changes a bunch of unrelated files when I run yarn start so I'm not sure whether I should undo them.
Also prettier autoruns on save

I re-created this locally, but it looks like the update is a breaking change and would need a more thorough look. For now, I'm, going to close this PR as it needs a chunk of work which realistically only I'll end up doing:

authors: longlho orta


text:
Incorrect indent of expression arrow functions with multi-line parameter list

TypeScript Version:  2.4.1 / nightly (2.5.0-dev.201xxxxx)
Code
const exprFunc = (
) => (
  undefined
);

const stmtFunc = (
) => {
  undefined;
};

Expected behavior:
const exprFunc = (
) => (
  undefined
);

Actual behavior:
const exprFunc = (
) => (
    undefined
  );

Works as requested now

authors: RyanCavanaugh magnushiie

repo: microsoft/TypeScript


text:
Extending empty tuple in generic functions causes compiler to infer arrays as tuples

TypeScript Version:  3.4.5

Search Terms:
empty tuple, type inference, generic function
Code
Basically I'm writing a generic function with dynamic type mapping. I think this use case is pretty common for database drivers and JSON validators and I found some related issues. There is a workaround and I want to confirm whether it is intended behavior or a bug.
interface TypeMap { STRING: string; NUMBER: number;}
type MapTypes<T> = {
  [P in keyof T]: T[P] extends keyof TypeMap ? TypeMap[T[P]] : never
};

declare function fromArray<T extends ReadonlyArray<keyof TypeMap>>(
  types: T, params: MapTypes<T>): void;

fromArray(['STRING', 'NUMBER', 'STRING'], ['abc', 123, 'xyz']);  // ok
// the default behavior is to infer types in array as type union
fromArray(['STRING', 'NUMBER', 'STRING'], ['abc', '123', 'xyz']);  // no error
fromArray(['STRING', 'NUMBER', 'STRING'] as const, ['abc', '123', 'xyz']);  // error

declare function fromTuple<T extends([] | ReadonlyArray<keyof TypeMap>)>(
  types: T, params: MapTypes<T>): void;

fromTuple(['STRING', 'NUMBER', 'STRING'], ['abc', 123, 'xyz']);  // ok
// T extends([]|MyType) cases the compiler to infer array as tuple
fromTuple(['STRING', 'NUMBER', 'STRING'], ['abc', '123', 'xyz']);  // error
fromTuple(['STRING', 'NUMBER', 'STRING'] as const, ['abc', '123', 'xyz']);  // error

Expected behavior:
Type of the array literal is inferred as array of union type.
Actual behavior:
Type of the array literal is inferred as tuple when T extends []|MyType is used in generic function.
I'd prefer it to be the expected behavior, but I didn't find any tests for that use case.
Playground Link:
You can see the same behavior in the playground
Related Issues:
https://github.com/microsoft/TypeScript/issues/13151
https://github.com/microsoft/TypeScript/issues/22679
Related PR
This PR includes a similar test, but it there is no test for empty tuple and array combination https://github.com/microsoft/TypeScript/pull/26676

This is working as intended. We only infer tuple types when the contextual type is or contains a tuple-like type.

@ahejlsberg Great, thank you for confirming.
Can I add some tests for this behavior then? I wasn't able to find anything similar by grepping tests directory (grep 'extends \[\]' -r tests/).
There are tests that check a combination of two tuples, but not a combination of a tuple and array.

Yes, it is the intended behavior. See https://github.com/microsoft/TypeScript/issues/27179#issuecomment-422911925 for the recommended pattern.

Thank you for the link. I think I've seen that issue, but didn't go through all comments.

authors: ahejlsberg ipotuzhnov