text:
solrcloud should not unconditionally install tool chain (gcc, patch)

In our environment, we do not allow compilers to be installed on our VMs.  So we package the zookeeper gems in to an RPM and install that before any dependencies (like zk).
Please add an attribute to prevent the following code from executing in tarball.rb:
# Require for zk gem
%w(patch gcc).each do |pkg|
  package pkg do
    action :nothing
  end.run_action(:install)
end

@tjordahl in that case chef_gem 'zk' resource would also become obsolete. correct?

Our strategy is to install the compiled dependencies for the gem in an RPM and not the gem itself, but that is just how we go about it.  That attribute is great, thanks!

authors: tjordahl vkhatri

repo: vkhatri/chef-solrcloud


text:
document is error

document is error：
https://bucket4j.com/7.0.0/toc.html#using-bucket-as-scheduler
2.1.3. Using bucket as scheduler
bucket.asScheduler().consume(1);
interface SchedulingBucket is not existed method  consume(long numTokens)
https://bucket4j.com/7.0.0/toc.html#scheduling-bucket

Indeed, the documentation indicates that you should use: bucket.asScheduler().consume(1); (cf. screen capture or link to documentation).
But in the source, you only have this method available: CompletableFuture<Void> consume(long numTokens, ScheduledExecutorService scheduler); so I don't know how to migrate from 6 to 7. Thx

Hello guys,
The method asScheduler has been renamed in 7.0.0 to asBlocking

Thanks

source: gharchive/issue

license_type: permissive

license_source: bigquery


text:
Support kv_cache_scheme for quantizing KV Cache

FIX #86
A simple fix since kv_cache_scheme just wasn't being piped down to QuantizationConfig
Example script: (I am not sure how to construct the modifier correctly with code, so using recipe)
from llmcompressor.transformers import oneshot

recipe = """
quant_stage:
    quant_modifiers:
        QuantizationModifier:
            ignore: ["lm_head"]
            config_groups:
                group_0:
                    weights:
                        num_bits: 8
                        type: float
                        strategy: tensor
                        dynamic: false
                        symmetric: true
                    input_activations:
                        num_bits: 8
                        type: float
                        strategy: tensor
                        dynamic: false
                        symmetric: true
                    targets: ["Linear"]
            kv_cache_scheme:
                num_bits: 8
                type: float
                strategy: tensor
                dynamic: false
                symmetric: true
"""

oneshot(
    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    dataset="open_platypus",
    recipe=recipe,
    output_dir="TinyLlama-1.1B-Chat-v1.0-FP8-KV",
    num_calibration_samples=16,
)

See the safetensors output has output_scales on k_proj and v_proj:
>>> from safetensors.torch import load_file
>>> state_dict = load_file("TinyLlama-1.1B-Chat-v1.0-FP8-KV/model.safetensors")
>>> [k for k in state_dict.keys() if "output_scale" in k]
['model.layers.0.self_attn.k_proj.output_scale', 'model.layers.0.self_attn.v_proj.output_scale', 'model.layers.1.self_attn.k_proj.output_scale', 'model.layers.1.self_attn.v_proj.output_scale', 'model.layers.10.self_attn.k_proj.output_scale', 'model.layers.10.self_attn.v_proj.output_scale', 'model.layers.11.self_attn.k_proj.output_scale', 'model.layers.11.self_attn.v_proj.output_scale', 'model.layers.12.self_attn.k_proj.output_scale', 'model.layers.12.self_attn.v_proj.output_scale', 'model.layers.13.self_attn.k_proj.output_scale', 'model.layers.13.self_attn.v_proj.output_scale', 'model.layers.14.self_attn.k_proj.output_scale', 'model.layers.14.self_attn.v_proj.output_scale', 'model.layers.15.self_attn.k_proj.output_scale', 'model.layers.15.self_attn.v_proj.output_scale', 'model.layers.16.self_attn.k_proj.output_scale', 'model.layers.16.self_attn.v_proj.output_scale', 'model.layers.17.self_attn.k_proj.output_scale', 'model.layers.17.self_attn.v_proj.output_scale', 'model.layers.18.self_attn.k_proj.output_scale', 'model.layers.18.self_attn.v_proj.output_scale', 'model.layers.19.self_attn.k_proj.output_scale', 'model.layers.19.self_attn.v_proj.output_scale', 'model.layers.2.self_attn.k_proj.output_scale', 'model.layers.2.self_attn.v_proj.output_scale', 'model.layers.20.self_attn.k_proj.output_scale', 'model.layers.20.self_attn.v_proj.output_scale', 'model.layers.21.self_attn.k_proj.output_scale', 'model.layers.21.self_attn.v_proj.output_scale', 'model.layers.3.self_attn.k_proj.output_scale', 'model.layers.3.self_attn.v_proj.output_scale', 'model.layers.4.self_attn.k_proj.output_scale', 'model.layers.4.self_attn.v_proj.output_scale', 'model.layers.5.self_attn.k_proj.output_scale', 'model.layers.5.self_attn.v_proj.output_scale', 'model.layers.6.self_attn.k_proj.output_scale', 'model.layers.6.self_attn.v_proj.output_scale', 'model.layers.7.self_attn.k_proj.output_scale', 'model.layers.7.self_attn.v_proj.output_scale', 'model.layers.8.self_attn.k_proj.output_scale', 'model.layers.8.self_attn.v_proj.output_scale', 'model.layers.9.self_attn.k_proj.output_scale', 'model.layers.9.self_attn.v_proj.output_scale']

Config:
cat TinyLlama-1.1B-Chat-v1.0-FP8-KV/config.json

compression_config output:
  "compression_config": {
    "config_groups": {
      "group_0": {
        "input_activations": {
          "block_structure": null,
          "dynamic": false,
          "group_size": null,
          "num_bits": 8,
          "observer": "minmax",
          "observer_kwargs": {},
          "strategy": "tensor",
          "symmetric": true,
          "type": "float"
        },
        "output_activations": null,
        "targets": [
          "Linear"
        ],
        "weights": {
          "block_structure": null,
          "dynamic": false,
          "group_size": null,
          "num_bits": 8,
          "observer": "minmax",
          "observer_kwargs": {},
          "strategy": "tensor",
          "symmetric": true,
          "type": "float"
        }
      }
    },
    "format": "float-quantized",
    "global_compression_ratio": 1.4556211956659422,
    "ignore": [
      "lm_head"
    ],
    "kv_cache_scheme": {
      "block_structure": null,
      "dynamic": false,
      "group_size": null,
      "num_bits": 8,
      "observer": "minmax",
      "observer_kwargs": {},
      "strategy": "tensor",
      "symmetric": true,
      "type": "float"
    },
    "quant_method": "compressed-tensors",
    "quantization_status": "frozen"
  },

I'll add testing in another PR!

source: gharchive/pull-request

repo: vllm-project/llm-compressor

license_source: github-api


text:
Address family not supported by protocol

env:
-centos 8

python.311
vllm 0.2.3+cu122

env:
-centos 8
python.311
vllm 0.2.7+cu122,

env:
-centos 8
python.311
vllm 0.2.6+cu122,

three version, only 0.2.7+cu122  have error.
ps:
"pip install -e" install from source code.

pytho3.10 is okay. so weird.

repo: vllm-project/vllm


text:
[Bug]: vlm online inference with --enable-prefix-caching: Internal Server Error

Your current environment
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.28.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-3.10.0-1062.9.1.el7.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.3.107
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A30
GPU 1: NVIDIA A30
GPU 2: NVIDIA A30
GPU 3: NVIDIA A30

Nvidia driver version: 525.85.12
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.0.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.0.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 57 bits virtual
Byte Order:                      Little Endian
CPU(s):                          128
On-line CPU(s) list:             0-127
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz
CPU family:                      6
Model:                           106
Thread(s) per core:              2
Core(s) per socket:              32
Socket(s):                       2
Stepping:                        6
Frequency boost:                 enabled
CPU max MHz:                     3500.0000
CPU min MHz:                     800.0000
BogoMIPS:                        5200.00
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities
Virtualization:                  VT-x
L1d cache:                       3 MiB (64 instances)
L1i cache:                       2 MiB (64 instances)
L2 cache:                        80 MiB (64 instances)
L3 cache:                        96 MiB (2 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-127
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: Load fences, __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB
Vulnerability Tsx async abort:   Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] onnx==1.15.0rc2
[pip3] optree==0.10.0
[pip3] pytorch-quantization==2.1.2
[pip3] torch==2.4.0
[pip3] torch-tensorrt==2.3.0a0
[pip3] torchdata==0.7.1a0
[pip3] torchtext==0.17.0a0
[pip3] torchvision==0.19.0
[pip3] transformers==4.43.4
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.4
vLLM Build Flags:
CUDA Archs: 5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity
GPU0	 X 	SYS	SYS	SYS	0-127		N/A
GPU1	SYS	 X 	SYS	SYS	0-127		N/A
GPU2	SYS	SYS	 X 	SYS	0-127		N/A
GPU3	SYS	SYS	SYS	 X 	0-127		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug
pip install vllm==0.5.4
CUDA_VISIBLE_DEVICES=0 vllm serve openbmb/MiniCPM-Llama3-V-2_5 --trust-remote-code --tensor-parallel-size 1 --max-model-len 2048 --gpu-memory-utilization 0.98 --enable-prefix-caching

INFO 08-07 06:33:47 async_llm_engine.py:174] Added request chat-c3004d9e0b7346f898079a1c1a5a217c.
ERROR 08-07 06:33:47 async_llm_engine.py:57] Engine background task failed
ERROR 08-07 06:33:47 async_llm_engine.py:57] Traceback (most recent call last):
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 47, in _log_task_completion
ERROR 08-07 06:33:47 async_llm_engine.py:57]     return_value = task.result()
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 642, in run_engine_loop
ERROR 08-07 06:33:47 async_llm_engine.py:57]     result = task.result()
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 585, in engine_step
ERROR 08-07 06:33:47 async_llm_engine.py:57]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 254, in step_async
ERROR 08-07 06:33:47 async_llm_engine.py:57]     output = await self.model_executor.execute_model_async(
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 159, in execute_model_async
ERROR 08-07 06:33:47 async_llm_engine.py:57]     output = await make_async(self.driver_worker.execute_model
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
ERROR 08-07 06:33:47 async_llm_engine.py:57]     result = self.fn(*self.args, **self.kwargs)
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 273, in execute_model
ERROR 08-07 06:33:47 async_llm_engine.py:57]     output = self.model_runner.execute_model(
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 08-07 06:33:47 async_llm_engine.py:57]     return func(*args, **kwargs)
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1363, in execute_model
ERROR 08-07 06:33:47 async_llm_engine.py:57]     hidden_or_intermediate_states = model_executable(
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
ERROR 08-07 06:33:47 async_llm_engine.py:57]     return self._call_impl(*args, **kwargs)
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
ERROR 08-07 06:33:47 async_llm_engine.py:57]     return forward_call(*args, **kwargs)
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 622, in forward
ERROR 08-07 06:33:47 async_llm_engine.py:57]     image_inputs = self._parse_and_validate_inputs(input_ids, **kwargs)
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 608, in _parse_and_validate_inputs
ERROR 08-07 06:33:47 async_llm_engine.py:57]     image_bounds=self._get_image_bounds(input_ids),
ERROR 08-07 06:33:47 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 566, in _get_image_bounds
ERROR 08-07 06:33:47 async_llm_engine.py:57]     return torch.hstack([
ERROR 08-07 06:33:47 async_llm_engine.py:57] RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 3 for tensor number 1 in the list.
Exception in callback _log_task_completion(error_callback=<bound method...7f97334c4a30>>)(<Task finishe...n the list.')>) at /usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py:37
handle: <Handle _log_task_completion(error_callback=<bound method...7f97334c4a30>>)(<Task finishe...n the list.')>) at /usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py:37>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 47, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 642, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 585, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 254, in step_async
    output = await self.model_executor.execute_model_async(
  File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 159, in execute_model_async
    output = await make_async(self.driver_worker.execute_model
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 273, in execute_model
    output = self.model_runner.execute_model(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1363, in execute_model
    hidden_or_intermediate_states = model_executable(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 622, in forward
    image_inputs = self._parse_and_validate_inputs(input_ids, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 608, in _parse_and_validate_inputs
    image_bounds=self._get_image_bounds(input_ids),
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/minicpmv.py", line 566, in _get_image_bounds
    return torch.hstack([
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 3 for tensor number 1 in the list.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 59, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.
INFO 08-07 06:33:47 async_llm_engine.py:181] Aborted request chat-c3004d9e0b7346f898079a1c1a5a217c.
INFO:     127.0.0.1:19320 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 189, in create_chat_completion
    generator = await openai_serving_chat.create_chat_completion(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 185, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 436, in chat_completion_full_generator
    async for res in result_generator:
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/client.py", line 216, in generate
    raise request_output
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 2 but got size 3 for tensor number 1 in the list.
WARNING 08-07 06:33:48 chat_utils.py:163] Detected image token string in the text prompt. Skipping prompt formatting.

Prefix caching is not supported for multi-modal models yet.

Thanks for reply.


text:
[Usage]: Custom LLM Generate

Your current environment
The output of `python collect_env.py`

How would you like to use vllm
I'm implementating a custom algorithm that requires a custom generate method.
In this method, I need to access and store some of the attention outputs without running a full foward pass whole model as displayed below. But I keep getting errors related to attn_metadata. I tried multiple options such as using some of the abstractions in attn_metadata.py and model_runner.py but with no success.
This very easy to do in transformers and I have a working it but I'm struggling to port it to vLLM.
import torch
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
from typing import List, Tuple, Dict, Any, Optional

model = LLM(model=model_name, dtype="bfloat16", gpu_memory_utilization=gpu_memory_utilization)
model_obj = model.llm_engine.model_executor.driver_worker.model_runner.model
hidden_states = model_obj.model.get_input_embeddings(input_ids)

attention_outputs = []
cache_position = None
past_key_values = None
position_ids = None
n_layers = 2

for layer in model_obj.model.layers[:n_layers]:
    if cache_position is None:
        past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
        cache_position = torch.arange(past_seen_tokens, past_seen_tokens + hidden_states.shape[1]).to(self.device)

    if position_ids is None:
        position_ids = cache_position.unsqueeze(0)

    # Store the input as the residual
    residual = hidden_states

    # Pass attn_metadata and residual to the layer
    layer_output = layer(
        positions=position_ids,
        hidden_states=hidden_states,
        kv_cache=past_key_values,
        attn_metadata=None,
        residual=residual
    )

    # Unpack the layer output
    hidden_states = layer_output[0]

    if attention_output is not None:
        attention_outputs.append(attention_output.detach().to("cpu"))

Traceback
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[1], line 123
    121 model = Model("../workspace/llama_models/llama-8B-Instruct")
    122 messages = [{"role": "user", "content": "Suppose you are provided a 7L bucket. Also, suppose that, afterwards, the floor of that bucket was removed, and its ceiling was sealed. How much water this bucket can hold?"}]
--> 123 output = model.generate(messages, stream=True)
    124 print(output)

Cell In[1], line 89, in Model.generate(self, messages, max_new_tokens, temperature, do_sample, stream)
     86 input_ids = input_ids.input_ids
     88 # Find the context
---> 89 context_indices = self.find_context(input_ids).to(self.device)
     91 if input_ids.device != self.device:
     92     input_ids.to(self.device)

Cell In[1], line 65, in Model.find_context(self, input_ids)
     64 # Get the hidden states and attention matrices
---> 65 attentions, _ = self.get_attention_first_n_layers(input_ids, self.filter_layers)

Cell In[1], line 43, in Model.get_attention_first_n_layers(self, input_ids, n_layers)
     40 residual = hidden_states
     42 # Pass attn_metadata and residual to the layer
---> 43 layer_output = layer(
     44     positions=position_ids,
     45     hidden_states=hidden_states,
     46     kv_cache=past_key_values,
     47     attn_metadata=None,
     48     residual=residual
     49 )
     51 # Unpack the layer output
     52 hidden_states = layer_output[0]

File [/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:1553](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py#line=1552), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:1562](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py#line=1561), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/llama.py:259](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/llama.py#line=258), in LlamaDecoderLayer.forward(self, positions, hidden_states, kv_cache, attn_metadata, residual)
    256 else:
    257     hidden_states, residual = self.input_layernorm(
    258         hidden_states, residual)
--> 259 hidden_states = self.self_attn(positions=positions,
    260                                hidden_states=hidden_states,
    261                                kv_cache=kv_cache,
    262                                attn_metadata=attn_metadata)
    264 # Fully Connected
    265 hidden_states, residual = self.post_attention_layernorm(
    266     hidden_states, residual)

File [/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:1553](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py#line=1552), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:1562](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py#line=1561), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/llama.py:189](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/vllm/model_executor/models/llama.py#line=188), in LlamaAttention.forward(self, positions, hidden_states, kv_cache, attn_metadata)
    187 q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
    188 q, k = self.rotary_emb(positions, q, k)
--> 189 attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
    190 output, _ = self.o_proj(attn_output)
    191 return output

File [/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:1553](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py#line=1552), in Module._wrapped_call_impl(self, *args, **kwargs)
   1551     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1552 else:
-> 1553     return self._call_impl(*args, **kwargs)

File [/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py:1562](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py#line=1561), in Module._call_impl(self, *args, **kwargs)
   1557 # If we don't have any hooks, we want to skip the rest of the logic in
   1558 # this function, and just call forward.
   1559 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1560         or _global_backward_pre_hooks or _global_backward_hooks
   1561         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1562     return forward_call(*args, **kwargs)
   1564 try:
   1565     result = None

File [/usr/local/lib/python3.11/dist-packages/vllm/attention/layer.py:100](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/vllm/attention/layer.py#line=99), in Attention.forward(self, query, key, value, kv_cache, attn_metadata, attn_type)
     90 def forward(
     91     self,
     92     query: torch.Tensor,
   (...)
     97     attn_type: AttentionType = AttentionType.DECODER,
     98 ) -> torch.Tensor:
--> 100     return self.impl.forward(query,
    101                              key,
    102                              value,
    103                              kv_cache,
    104                              attn_metadata,
    105                              self._k_scale,
    106                              self._v_scale,
    107                              attn_type=attn_type)

File [/usr/local/lib/python3.11/dist-packages/vllm/attention/backends/flash_attn.py:584](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/vllm/attention/backends/flash_attn.py#line=583), in FlashAttentionImpl.forward(self, query, key, value, kv_cache, attn_metadata, k_scale, v_scale, attn_type)
    580 # NOTE(woosuk): FlashAttention does not support FP8 KV cache.
    581 assert k_scale == 1.0 and v_scale == 1.0, (
    582     "key/v_scale is not supported in FlashAttention.")
--> 584 output = torch.ops.vllm.unified_flash_attention(
    585     query,
    586     key,
    587     value,
    588     self.num_heads,
    589     self.head_size,
    590     self.num_kv_heads,
    591     kv_cache,
    592     self.kv_cache_dtype,
    593     k_scale,
    594     v_scale,
    595     self.scale,
    596     self.sliding_window,
    597     self.alibi_slopes,
    598     self.logits_soft_cap,
    599 )
    601 return output

File [/usr/local/lib/python3.11/dist-packages/torch/_ops.py:1061](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/_ops.py#line=1060), in OpOverloadPacket.__call__(self_, *args, **kwargs)
   1059 if self_._has_torchbind_op_overload and _must_dispatch_in_python(args, kwargs):
   1060     return _call_overload_packet_from_python(self_, args, kwargs)
-> 1061 return self_._op(*args, **(kwargs or {}))

File [/usr/local/lib/python3.11/dist-packages/torch/_library/autograd.py:98](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/_library/autograd.py#line=97), in make_autograd_impl.<locals>.autograd_impl(keyset, *args, **keyword_only_args)
     97 def autograd_impl(keyset, *args, **keyword_only_args):
---> 98     result = Generated.apply(*args, Metadata(keyset, keyword_only_args))  # type: ignore[attr-defined]
     99     return result

File [/usr/local/lib/python3.11/dist-packages/torch/autograd/function.py:574](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/autograd/function.py#line=573), in Function.apply(cls, *args, **kwargs)
    571 if not torch._C._are_functorch_transforms_active():
    572     # See NOTE: [functorch vjp and autograd interaction]
    573     args = _functorch.utils.unwrap_dead_wrappers(args)
--> 574     return super().apply(*args, **kwargs)  # type: ignore[misc]
    576 if not is_setup_ctx_defined:
    577     raise RuntimeError(
    578         "In order to use an autograd.Function with functorch transforms "
    579         "(vmap, grad, jvp, jacrev, ...), it must override the setup_context "
    580         "staticmethod. For more details, please see "
    581         "https://pytorch.org/docs/main/notes/extending.func.html"
    582     )

File /usr/local/lib/python3.11/dist-packages/torch/_library/autograd.py:40, in make_autograd_impl.<locals>.forward(ctx, *args)
     38 keyset = metadata.keyset
     39 kwargs = metadata.keyword_only_args
---> 40 result = op.redispatch(keyset & _C._after_autograd_keyset, *args, **kwargs)
     41 if info._setup_context_fn:
     42     # The Dispatcher will remove args that are equal to their default
     43     # values from (args, kwargs). We're going to add it back so that
   (...)
     50     # their setup_context (along with the rest of their operator
     51     # registrations)
     52     args, kwargs = utils.fill_defaults(op._schema, args, kwargs)

File [/usr/local/lib/python3.11/dist-packages/torch/_ops.py:672](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/_ops.py#line=671), in OpOverload.redispatch(self_, keyset, *args, **kwargs)
    669 def redispatch(self_, keyset, *args, **kwargs):  # noqa: B902
    670     # use `self_` to avoid naming collide with aten ops arguments that
    671     # are named "self". This way, all the aten ops can be called by kwargs.
--> 672     return self_._handle.redispatch_boxed(keyset, *args, **kwargs)

File [/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py:494](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py#line=493), in CustomOpDef._register_to_dispatcher.<locals>.adinplaceorview_impl(keyset, *args, **kwargs)
    492                 autograd.graph.increment_version(v)
    493 with _C._AutoDispatchBelowADInplaceOrView():
--> 494     return self._opoverload.redispatch(
    495         keyset & _C._after_ADInplaceOrView_keyset, *args, **kwargs
    496     )

File [/usr/local/lib/python3.11/dist-packages/torch/_ops.py:672](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/_ops.py#line=671), in OpOverload.redispatch(self_, keyset, *args, **kwargs)
    669 def redispatch(self_, keyset, *args, **kwargs):  # noqa: B902
    670     # use `self_` to avoid naming collide with aten ops arguments that
    671     # are named "self". This way, all the aten ops can be called by kwargs.
--> 672     return self_._handle.redispatch_boxed(keyset, *args, **kwargs)

File [/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py:236](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py#line=235), in CustomOpDef.register_kernel.<locals>.inner.<locals>.backend_impl(*args, **kwargs)
    228 def backend_impl(*args, **kwargs):
    229     # Checks the assumption that outputs cannot alias
    230     # inputs or other outputs.
    231     storages = {
    232         id(tensor.untyped_storage())
    233         for tensor in iter_tensors(args, kwargs)
    234     }
--> 236     result = self._backend_fns[device_type](*args, **kwargs)
    238     tuple_result = result
    239     if not isinstance(result, tuple):

File [/usr/local/lib/python3.11/dist-packages/vllm/attention/backends/flash_attn.py:624](http://157.157.221.29:17975/lab/tree/workspace/usr/local/lib/python3.11/dist-packages/vllm/attention/backends/flash_attn.py#line=623), in unified_flash_attention(query, key, value, num_heads, head_size, num_kv_heads, kv_cache, kv_cache_dtype, k_scale, v_scale, softmax_scale, window_size, alibi_slopes, logits_soft_cap)
    604 @torch.library.custom_op("vllm::unified_flash_attention",
    605                          mutates_args=["kv_cache"])
    606 def unified_flash_attention(
   (...)
    620     logits_soft_cap: Optional[float] = None,
    621 ) -> torch.Tensor:
    623     current_metadata = get_forward_context()
--> 624     assert current_metadata is not None
    625     assert isinstance(current_metadata, FlashAttentionMetadata)
    626     attn_metadata: FlashAttentionMetadata = current_metadata

AssertionError:

Expected result:
Access and store intermediate results of the model directly without having to run a full forward pass.
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Thanks @DarkLight1337 !
But how do I get the the model_input.attn_metadata??

Thanks @DarkLight1337 !
But how do I get the the model_input.attn_metadata??
It's not very clear how to implementing it on my example.

This is the tricky part, since it's quite integral to how vLLM operates (e.g. KV cache, prefix caching, chunked prefill...). I guess your custom cache most likely interferes with how vLLM does it by default.

I'm not familiar with this part of the code so can't offer suggestions. Perhaps @comaniac could help?

We don't support this use case atm so there might be many unexpected behaviors. I'd suggest cloning a model file in vllm, custom register as a plugin model.

Thanks @comaniac!
Could you provide me an example of how to implement it with my use case?
I want to pass the number of layers and a few other arguments at inference time.
For example:
model = LLM(model=model_name, dtype="bfloat16", gpu_memory_utilization=gpu_memory_utilization)

model.generate(inputs, num_layers, arg1, arg2)

Even high level one would help.

Do your args change per request? Or they will be determined when launching the engine

They change per request.

Then you may try to change llm.generate() (https://github.com/vllm-project/vllm/blob/08075c34483843c75b4420bac92377b59ff9a8ac/vllm/entrypoints/llm.py#L295). One quick way I could think of to make it work is adding your arguments to SamplingParams so that they can be passed all the way to the model runner inputs.

Thanks a lot!
So I change the generate and the model file (i.e., llama.py)?
How can I access just the layers? Is there an object that controls the forward pass?
The pseudo code implementation of what I want to do is something like this

Pass prompt and get attention for the first N layers
Process the prompt using the attention
Generate response normally (full model) using custom prompt

Basically two extra steps before running the normal generate.

authors: Blaizzy DarkLight1337 comaniac


text:
[Bug]: Engine iteration timed out. This should never happen!

Your current environment
hardwark： A800
Driver Version: 535.54.03 CUDA Version: 12.2
vllm commit d3a245138acb358c7e1e5c5dcf4dcb3c2b48c8ff
model qwen72B
Model Input Dumps
No response
🐛 Describe the bug
INFO 10-30 11:46:47 async_llm_engine.py:173] Added request 541ca4832eb9436180e721ef069baedb.
ERROR 10-30 11:47:32 async_llm_engine.py:656] Engine iteration timed out. This should never happen!
ERROR 10-30 11:47:32 async_llm_engine.py:56] Engine background task failed
ERROR 10-30 11:47:32 async_llm_engine.py:56] Traceback (most recent call last):
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 629, in run_engine_loop
ERROR 10-30 11:47:32 async_llm_engine.py:56]     done, _ = await asyncio.wait(
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 413, in wait
ERROR 10-30 11:47:32 async_llm_engine.py:56]     return await _wait(fs, timeout, return_when, loop)
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 525, in _wait
ERROR 10-30 11:47:32 async_llm_engine.py:56]     await waiter
ERROR 10-30 11:47:32 async_llm_engine.py:56] asyncio.exceptions.CancelledError
ERROR 10-30 11:47:32 async_llm_engine.py:56]
ERROR 10-30 11:47:32 async_llm_engine.py:56] During handling of the above exception, another exception occurred:
ERROR 10-30 11:47:32 async_llm_engine.py:56]
ERROR 10-30 11:47:32 async_llm_engine.py:56] Traceback (most recent call last):
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
ERROR 10-30 11:47:32 async_llm_engine.py:56]     return_value = task.result()
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 633, in run_engine_loop
ERROR 10-30 11:47:32 async_llm_engine.py:56]     await asyncio.sleep(0)
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_timeout.py", line 95, in aexit
ERROR 10-30 11:47:32 async_llm_engine.py:56]     self._do_exit(exc_type)
ERROR 10-30 11:47:32 async_llm_engine.py:56]   File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_timeout.py", line 178, in _do_exit
ERROR 10-30 11:47:32 async_llm_engine.py:56]     raise asyncio.TimeoutError
ERROR 10-30 11:47:32 async_llm_engine.py:56] asyncio.exceptions.TimeoutError
2024-10-30 11:47:32,282 - asyncio:default_exception_handler:1753 - ERROR:  Exception in callback _log_task_completion(error_callback=>)(<Task finishe...imeoutError()>) at /usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py:36
handle: <Handle _log_task_completion(error_callback=>)(<Task finishe...imeoutError()>) at /usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py:36>
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 629, in run_engine_loop
done, _ = await asyncio.wait(
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 413, in wait
return await _wait(fs, timeout, return_when, loop)
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 525, in _wait
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
return_value = task.result()
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 633, in run_engine_loop
await asyncio.sleep(0)
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_timeout.py", line 95, in aexit
self._do_exit(exc_type)
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_timeout.py", line 178, in _do_exit
raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 58, in _log_task_completion
raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.
INFO 10-30 11:47:32 async_llm_engine.py:180] Aborted request 6838fbb7076948a7a1f8071d4095c740.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 629, in run_engine_loop
done, _ = await asyncio.wait(
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 413, in wait
return await _wait(fs, timeout, return_when, loop)
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 525, in _wait
await waiter
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/ailab/inference_wrapper/huggingface/lora/nlp/wrapper_vllm.py", line 621, in _process_stream_infence
async for request_output in results_generator:
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 770, in generate
async for output in self._process_request(
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 886, in _process_request
raise e
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 882, in _process_request
async for request_output in stream:
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 93, in anext
raise result
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 46, in _log_task_completion
return_value = task.result()
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 633, in run_engine_loop
await asyncio.sleep(0)
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_timeout.py", line 95, in aexit
self._do_exit(exc_type)
File "/usr/local/lib/python3.9/site-packages/vllm/engine/async_timeout.py", line 178, in _do_exit
raise asyncio.TimeoutError
asyncio.exceptions.TimeoutError
2024-10-30 11:47:32,282 - wrapper:_process_stream_infence:645 - ERROR:  streaming inference exception, 6838fbb7076948a7a1f8071d4095c740
(VllmWorkerProcess pid=198) WARNING 10-30 11:47:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:47:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:47:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:48:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:48:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:48:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:49:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:49:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:49:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:50:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:50:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:50:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:51:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:51:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:51:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:52:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:52:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:52:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:53:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:53:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:53:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:54:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:54:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:54:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:55:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:55:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:55:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=199) WARNING 10-30 11:56:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=198) WARNING 10-30 11:56:32 shm_broadcast.py:386] No available block found in 60 second.
(VllmWorkerProcess pid=200) WARNING 10-30 11:56:32 shm_broadcast.py:386] No available block found in 60 second.
[rank2]:[E ProcessGroupNCCL.cpp:563] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600053 milliseconds before timing out.
[rank3]:[E ProcessGroupNCCL.cpp:563] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600054 milliseconds before timing out.
[rank2]:[E ProcessGroupNCCL.cpp:1537] [PG 2 Rank 2] Timeout at NCCL work: 12471541, last enqueued NCCL work: 12471541, last completed NCCL work: 12471540.
[rank2]:[E ProcessGroupNCCL.cpp:577] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E ProcessGroupNCCL.cpp:583] [Rank 2] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600053 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f3095e6dc62 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f3095e72a80 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3095e73dcc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #5:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
[rank1]:[E ProcessGroupNCCL.cpp:563] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600054 milliseconds before timing out.
terminate called after throwing an instance of 'c10::DistBackendError'
what():  [PG 2 Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600053 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f3095e6dc62 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f3095e72a80 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3095e73dcc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #5:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1:  + 0xe32119 (0x7f3095af7119 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #3:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
[rank3]:[E ProcessGroupNCCL.cpp:1537] [PG 2 Rank 3] Timeout at NCCL work: 12471541, last enqueued NCCL work: 12471541, last completed NCCL work: 12471540.
[rank3]:[E ProcessGroupNCCL.cpp:577] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank3]:[E ProcessGroupNCCL.cpp:583] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
[rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600054 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f3095e6dc62 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f3095e72a80 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3095e73dcc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #5:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
[rank1]:[E ProcessGroupNCCL.cpp:1537] [PG 2 Rank 1] Timeout at NCCL work: 12471541, last enqueued NCCL work: 12471541, last completed NCCL work: 12471540.
[rank1]:[E ProcessGroupNCCL.cpp:577] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E ProcessGroupNCCL.cpp:583] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E ProcessGroupNCCL.cpp:1414] [PG 2 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600054 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f3095e6dc62 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f3095e72a80 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3095e73dcc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #5:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what():  [PG 2 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600054 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f3095e6dc62 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f3095e72a80 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3095e73dcc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #5:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1:  + 0xe32119 (0x7f3095af7119 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #3:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
terminate called after throwing an instance of 'c10::DistBackendError'
what():  [PG 2 Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=12471541, OpType=GATHER, NumelIn=1520640, NumelOut=0, Timeout(ms)=600000) ran for 600054 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x1d2 (0x7f3095e6dc62 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f3095e72a80 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f3095e73dcc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #5:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f318fecf897 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1:  + 0xe32119 (0x7f3095af7119 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2:  + 0x245c0 (0x7f3226ad05c0 in /home/aiges/library/libuds.so)
frame #3:  + 0x94ac3 (0x7f322406bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: clone + 0x44 (0x7f32240fca04 in /lib/x86_64-linux-gnu/libc.so.6)
ERROR 10-30 11:56:48 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 198 died, exit code: -6
INFO 10-30 11:56:48 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[E ProcessGroupNCCL.cpp:1316] [PG 2 Rank 0] Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=9
[rank0]:[E ProcessGroupNCCL.cpp:1153] [PG 2 Rank 0] ProcessGroupNCCL preparing to dump debug info.
[rank0]:[F ProcessGroupNCCL.cpp:1169] [PG 2 Rank 0] [PG 2 Rank 0] ProcessGroupNCCL's watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api, or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. workMetaList_.size() = 9
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

same question but don't know why
同样的问题，但是不知道为什么，很捉急~

请问大家解决了吗，我用tp8也报了这个错误，但是用tp4就正常


text:
[Bugfix]Fix Phi-3 BNB online quantization

Work

Fix Phi-3 BNB config

For current models like Phi-3,Baichuan, ChaGLM, there is a bug in BNB quantization. This is because their merged linear( mlp )weights are already merged in the disk, and the current BNB implementation doesn't account for this scenario,fix this

TODO
For these types of models, there is still a remaining issue regarding TP  weight partitioning during online quantization. This will be addressed in the following  PR.
ping @mgoin

Work

Fix Phi-3 BNB config

For current models like Phi-3,Baichuan, ChaGLM, there is a bug in BNB quantization. This is because their merged linear( mlp )weights are already merged in the disk, and the current BNB implementation doesn't account for this scenario,fix this

TODO
For these types of models, there is still a remaining issue regarding TP weight partitioning during online quantization. This will be addressed in the following PR.

ping @mgoin

FYI, I have implemented the TP support for Phi-3 BNB in my branch, it might be useful for addressing the issue you mentioned: https://github.com/Isotr0py/vllm/blob/79f0b5f725ea74f19d8f4a73b9c9df72fb3ac605/vllm/model_executor/model_loader/loader.py#L909-L934

FYI, I have implemented the TP support for Phi-3 BNB in my branch, it might be useful for addressing the issue you mentioned: https://github.com/Isotr0py/vllm/blob/79f0b5f725ea74f19d8f4a73b9c9df72fb3ac605/vllm/model_executor/model_loader/loader.py#L909-L934

thanks,I will look at this asap


text:
[core] improve cpu offloading implementation

make it friendly with torch.compile

it seems the original cuda tensor is still held alive somewhere. the weights are not offloaded actually.


text:
[TPU] Align worker index with node boundary

Fixes #7485

is it enough to remove the following lines?
https://github.com/vllm-project/vllm/blob/9c71c97ae24ae6f5209a475149808e25554cfe99/vllm/worker/tpu_model_runner.py#L132-L135

@youkaichao I tried it, but still got gibberish results without the patch. I think this is because the rank IDs used in all gather are assigned by XLA runtime, regardless of IPs.

I will not block this pr for that reason, but it would be better to know in the future, what's the rank used by XLA.

Hi, I still get this error due to the mismatched global world size even on this branch(tpu-rank) version.
I use these commands below to build new docker image based on Dockerfile.tpu and use the code of this branch.
# on main node, main vm IP:PORT is 10.130.0.60:6379
sudo docker run -t -d -e HF_TOKEN=INPUT_YOUR_TOKEN --privileged --net host --shm-size=16G -it vllm ray start --head --block
# on other nodes
sudo docker run -t -d -e VLLM_HOST_IP=10.130.0.60 -e HF_TOKEN=YOUR_TOKEN -e  --privileged --net host --shm-size=16G -it vllm ray start --address 10.130.0.60:6379 --block

after checking this ray status -- to confirm all the tpu cores are gathered in ray(32 cores since I use TPUv4-64 for this exp)
======== Autoscaler status: 2024-08-28 04:31:11.442038 ========
Node status
---------------------------------------------------------------
Active:
 1 node_7f47f048ccc1c5455a203b92ef6d2bdd5d0332b18fa6f092d6c6425a
 1 node_be42b0ac8682eacb5da8cee5918de34e28dc90b1041b6522405c1c62
 1 node_13290553e86634224a524ec68c5b3e3f0478e491669681f92723ec49
 1 node_2ee5e9791b60dfaf727341ae8a67910996f0572ba36260df07507668
 1 node_3ad7ae73804a43f3a6acbc4e9b48f92693c8cf9e09e275e49863c056
 1 node_195573b8d1e45ef8e2e27d20cf530f48884e0be3e8ead3bf9c716338
 1 node_a1804b4b12788b7a33e02f0f9c38d18512d6ee4d27a9d9775fce7f5a
 1 node_13d0f73aeba95e5f03f42494a57a9ac1204e273106e7ad109dde11f5
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/1920.0 CPU
 0.0/32.0 TPU
 0.0/1.0 TPU-v4-64-head
 0B/3.00TiB memory
 0B/121.60GiB object_store_memory
 0.0/8.0 v4-64

However when I run this command to run vllm server --
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123 -tp 4 -pp 8 --device tpu --distributed_executor_backend ray

this error comes.
(pid=614, ip=10.130.15.194) INFO 08-28 04:16:28 importing.py:10] Triton not installed; certain GPU-related functions will not be available. [repeated 2x across cluster]
(RayWorkerWrapper pid=376, ip=10.130.0.62) WARNING:root:libtpu.so and TPU device found. Setting PJRT_DEVICE=TPU.
WARNING:root:libtpu.so and TPU device found. Setting PJRT_DEVICE=TPU.
INFO 08-28 04:16:35 selector.py:198] Cannot use _Backend.FLASH_ATTN backend on TPU.
INFO 08-28 04:16:35 selector.py:146] Using Pallas backend.
(RayWorkerWrapper pid=376, ip=10.130.15.193) INFO 08-28 04:16:35 selector.py:198] Cannot use _Backend.FLASH_ATTN backend on TPU.
(RayWorkerWrapper pid=376, ip=10.130.15.193) INFO 08-28 04:16:35 selector.py:146] Using Pallas backend.
(pid=615, ip=10.130.0.59) INFO 08-28 04:16:32 importing.py:10] Triton not installed; certain GPU-related functions will not be available.
ERROR 08-28 04:16:35 worker_base.py:465] Error executing method init_device. This might cause deadlock in distributed execution.
ERROR 08-28 04:16:35 worker_base.py:465] Traceback (most recent call last):
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/worker/worker_base.py", line 457, in execute_method
ERROR 08-28 04:16:35 worker_base.py:465]     return executor(*args, **kwargs)
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/worker/tpu_worker.py", line 83, in init_device
ERROR 08-28 04:16:35 worker_base.py:465]     ensure_model_parallel_initialized(
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/distributed/parallel_state.py", line 965, in ensure_model_parallel_initialized
ERROR 08-28 04:16:35 worker_base.py:465]     initialize_model_parallel(tensor_model_parallel_size,
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/distributed/parallel_state.py", line 931, in initialize_model_parallel
ERROR 08-28 04:16:35 worker_base.py:465]     _TP = init_model_parallel_group(group_ranks,
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/distributed/parallel_state.py", line 773, in init_model_parallel_group
ERROR 08-28 04:16:35 worker_base.py:465]     return GroupCoordinator(
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/distributed/parallel_state.py", line 175, in __init__
ERROR 08-28 04:16:35 worker_base.py:465]     self.tpu_communicator = TpuCommunicator(group=self.cpu_group)
ERROR 08-28 04:16:35 worker_base.py:465]   File "/workspace/vllm/vllm/distributed/device_communicators/tpu_communicator.py", line 29, in __init__
ERROR 08-28 04:16:35 worker_base.py:465]     local_rank = global_rank % local_world_size
ERROR 08-28 04:16:35 worker_base.py:465] ZeroDivisionError: integer division or modulo by zero
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/workspace/vllm/vllm/entrypoints/openai/rpc/server.py", line 230, in run_rpc_server
    server = AsyncEngineRPCServer(async_engine_args, usage_context, rpc_path)
  File "/workspace/vllm/vllm/entrypoints/openai/rpc/server.py", line 31, in __init__
    self.engine = AsyncLLMEngine.from_engine_args(
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 709, in from_engine_args
    engine = cls(
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 600, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 809, in _init_engine
    return engine_class(*args, **kwargs)
  File "/workspace/vllm/vllm/engine/async_llm_engine.py", line 261, in __init__
    super().__init__(*args, **kwargs)
  File "/workspace/vllm/vllm/engine/llm_engine.py", line 288, in __init__
    self.model_executor = executor_class(
  File "/workspace/vllm/vllm/executor/ray_tpu_executor.py", line 304, in __init__
    super().__init__(*args, **kwargs)
  File "/workspace/vllm/vllm/executor/ray_tpu_executor.py", line 36, in __init__
    super().__init__(*args, **kwargs)
  File "/workspace/vllm/vllm/executor/executor_base.py", line 46, in __init__
    self._init_executor()
  File "/workspace/vllm/vllm/executor/ray_tpu_executor.py", line 48, in _init_executor
    self._init_workers_ray(placement_group)
  File "/workspace/vllm/vllm/executor/ray_tpu_executor.py", line 175, in _init_workers_ray
    self._run_workers("init_device")
  File "/workspace/vllm/vllm/executor/ray_tpu_executor.py", line 242, in _run_workers
    driver_worker_output = self.driver_worker.execute_method(
  File "/workspace/vllm/vllm/worker/worker_base.py", line 466, in execute_method
    raise e
  File "/workspace/vllm/vllm/worker/worker_base.py", line 457, in execute_method
    return executor(*args, **kwargs)
  File "/workspace/vllm/vllm/worker/tpu_worker.py", line 83, in init_device
    ensure_model_parallel_initialized(
  File "/workspace/vllm/vllm/distributed/parallel_state.py", line 965, in ensure_model_parallel_initialized
    initialize_model_parallel(tensor_model_parallel_size,
  File "/workspace/vllm/vllm/distributed/parallel_state.py", line 931, in initialize_model_parallel
    _TP = init_model_parallel_group(group_ranks,
  File "/workspace/vllm/vllm/distributed/parallel_state.py", line 773, in init_model_parallel_group
    return GroupCoordinator(
  File "/workspace/vllm/vllm/distributed/parallel_state.py", line 175, in __init__
    self.tpu_communicator = TpuCommunicator(group=self.cpu_group)
  File "/workspace/vllm/vllm/distributed/device_communicators/tpu_communicator.py", line 29, in __init__
    local_rank = global_rank % local_world_size
ZeroDivisionError: integer division or modulo by zero

Because of the code here: https://github.com/vllm-project/vllm/blob/tpu-rank/vllmdistributed/device_communicators/tpu_communicator.py#L28
it causes
class TpuCommunicator:
        ....
        # NOTE(woosuk): When using TP > 1 on TPUs, every TPU on the same node
        # must be used together. Therefore, the local rank and world size can
        # be simply calculated as follows.
        global_rank = dist.get_rank(group) # <- this global rank wil be 0 on main node
        global_world_size = dist.get_world_size(group) # <- this size should be 32 but it is 4(single node's TPU cores)
        num_nodes = len(ray.nodes()) # <- correct value, 8 for TPUv4-32
        local_world_size = global_world_size // num_nodes # this line cases err since 4 // 8 == 0, causing later line ZeroDivision Error.
        local_rank = global_rank % local_world_size
        pjrt.initialize_multiprocess(local_rank, local_world_size)
        xr._init_world_size_ordinal()

I think pytorch distributed dist.get_world_size(group) can't gather all the TPU cores information from ray cluster, rather it uses local TPU xla device only.

@WoosukKwon I think this problem comes from these lines: https://github.com/vllm-project/vllm/blob/c166e7e43e7bb398835d1933a69d106b47f6cc8d/vllm/distributed/parallel_state.py#L121-L132
        for ranks in group_ranks:
            device_group = torch.distributed.new_group(
                ranks, backend=torch_distributed_backend)
            # a group with `gloo` backend, to allow direct coordination between
            # processes through the CPU.
            cpu_group = torch.distributed.new_group(ranks, backend="gloo")
            if self.rank in ranks:
                self.ranks = ranks
                self.world_size = len(ranks)
                self.rank_in_group = ranks.index(self.rank)
                self.device_group = device_group
                self.cpu_group = cpu_group

I think this codes are supposed to create local cpu group for optmized rank (such as TP in 1 node) but I think this cases cpu_group to have only 4 process(=1 node TPU cores), making Zero Division err to the code above I mentioned.
is there suggested way to workaround this issue?

LGTM, please fix the format

#7929 fixes the CI failure related to mypy, please merge from main to resolve it.

@Beomi Thanks for trying out this PR. Currently, vLLM's TPU backend does not support PP. Could you please use smaller TPU pod and retry with the updated main branch?

@WoosukKwon Thanks for clarification! it seems like TP over all nodes works :)
Hope PP comes to TPU as well.
Thank you 😄

@WoosukKwon BTW, there is weird situation related with Multinode situation -- the code works well with TPUv4-8/v4-16/v4-32 but it fails to launch worker on one node if I run on v4-64. I'll open it on the other issue :)

Hope PP comes to TPU as well.

TPU has very good cross-node inter connect. I don't think it needs PP.

authors: Beomi DarkLight1337 WoosukKwon youkaichao


text:
duration filter

Should run on duration_minutes

Reference

repo: vluzko/aukpy

license: MIT


text:
antrea-ovs container restarted multiple times causing antrea-agent container to be forever stuck in "Ready: False" state

Describe the bug
antrea-ovs container restarted multiple times causing antrea-agent container to be forever stuck in "Ready: False" state.
To Reproduce
Not sure how to reproduce but here is what we know happened:

On 06/25/2020 6:17am Pacific Time we receive first alert from Prometheus indicating that antrea-ovs container on the node had restarted 6 times in the past hour.
At 6:42am we started receiving alerts indicating that DNS tests were failing on the node and upon review we realized that container networking wasn't working.
When we looked at the antrea pod we saw that antrea-agent pod was also restarted, but got stuck in "Ready: False" state:

antrea-agent-ghxvn                     1/2     Running   7          40h    10.170.168.35   10.170.168.35   <none>           <none>

  antrea-agent:
    State:          Running
      Started:      Thu, 25 Jun 2020 06:08:49 -0700
    Ready:          False
    Restart Count:  1

  antrea-ovs:
    State:          Running
      Started:      Thu, 25 Jun 2020 06:09:26 -0700
    Ready:          True
    Restart Count:  6

Events:
  Type     Reason     Age                     From                    Message
  ----     ------     ----                    ----                    -------
  Warning  Unhealthy  68s (x2499 over 6h57m)  kubelet, 10.170.168.35  Readiness probe failed: Get https://127.0.0.1:10350/healthz: dial tcp 127.0.0.1:10350: connect: connection refused

Expected
antrea-agent shouldn't get stuck in a "Ready: False" state upon restart, it should be able to recover.
Actual behavior
Sometimes, when antrea-ovs container is restarted multiple times it causes antrea-agent container to be forever stuck in "Ready: False" state, which of course leads to pod networking being down.
Please provide the following information:

Antrea version: v0.7.2
Kubernetes version: 1.15.4
Container runtime: Docker 18.0.6
Linux kernel version on the Kubernetes Nodes: 4.19.43-coreos

Additional context
wdc-prd-decc-002-md-dy-minion020-logs.zip

Attaching a antrea-agent process core file we collected during the troubleshooting session.
antrea-agent-core.zip

@alex-vmw Thanks. I opened an issue with the fix we discussed today.

authors: alex-vmw antoninbas

repo: vmware-tanzu/antrea


text:
Print success message to stderr

The string "Succeeded" confuses tools that parse the output.
For example, you couldn't do eval "$(kapp completion bash)" without the error Succeeded: command not found.
Maybe don't even bother with this message and output "Error" only when unsuccessful?

PR seems reasonable. it seems like there is a failing test.

Hello @jvassev looks like that you just need to correct one test, and from the looks of it you just need to remove the following lines:
https://github.com/vmware-tanzu/carvel-kapp/blob/develop/test/e2e/order_test.go#L121
https://github.com/vmware-tanzu/carvel-kapp/blob/develop/test/e2e/order_test.go#L122
https://github.com/vmware-tanzu/carvel-kapp/blob/develop/test/e2e/order_test.go#L206
https://github.com/vmware-tanzu/carvel-kapp/blob/develop/test/e2e/order_test.go#L207
and everything should be ready to merge

Closing this as we no longer print the Succeeded message for bash completion command. Please feel free to reopen in case you feel that this is still required.

authors: cppforlife joaopapereira jvassev praveenrewar

repo: vmware-tanzu/carvel-kapp


text:
Add multus mtu test

Add mtu setting on macvlan interface
Fix package list on CentOS
What this PR does / why we need it

Add mtu test for multus based interfaces and fix CentOS package list change since Jan. 2022.
Details for the Release Notes (PLEASE PROVIDE)

NONE

Describe testing done for PR

Passed multus-cni package e2e test: ginkgo -v -- --packages="multus-cni" --version="3.7.1" --guest-cluster-name="tce-mycluster1"
Special notes for your reviewer

Hi @wjun - are there other members we can add to the multus GitHub team? Nick and I are apart of it simply to do admin stuff. So if there's someone we should add, let's do that so we don't block on your teams work.

Thank you @jpmcb @seemiller . I know the specific package process now.

authors: jpmcb wjun

repo: vmware-tanzu/community-edition


text:
upgrade linter from 1.58.1 to 1.59.1: no new linter errors/warnings

Release note:
NONE

@joshuatcasey I cancelled a CI job to prevent this PR from running through the whole pipeline, since it can only possibly affect the lint job. If the lint job passes, then I propose that we merge this.

The lint CI job passed while using the new linter version here: https://ci.pinniped.dev/teams/main/pipelines/pull-requests/jobs/lint/builds/646#L66b59442:5

force merging since this doesn't impact anything but lint

authors: cfryanr joshuatcasey

repo: vmware-tanzu/pinniped


text:
Prepare v0.16.3 release

Signed-off-by: Bridget McErlean bmcerlean@vmware.com

Codecov Report

Merging #983 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #983   +/-   ##
=======================================
  Coverage   47.69%   47.69%
=======================================
  Files          76       76
  Lines        5298     5298
=======================================
  Hits         2527     2527
  Misses       2617     2617
  Partials      154      154

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 871ba62...4124540. Read the comment docs.

authors: codecov-io zubron

repo: vmware-tanzu/sonobuoy


text:
SPIFFE/SPIRE

Doing this in episode 94.
The process for producing an episode is here.  All the detail for the steps below is there.
Monday or Tuesday:

[ ] Pick a topic and create/use an issue in the repo
[ ] If applicable coordinate with an upstream project if they're interested in attending the show
[ ] Create the graphic
[ ] Schedule on Youtube
[ ] Add scheduled episode to TGIK playlist on YT
[ ] Create short link in rebrandly
[ ] Tweet about it from your personal account

[ ] Post tweets on internal #tgik slack channel and cc @castrojo and boschem
[ ] Get confirmation from boschem that she's seen the tweet and is scheduling it

Before episode:

[ ] Create HackMD page for live notes.  Start with README template in this repo
[ ] Paste HackMD URL on internal #tgik channel and cc @castrojo or other assistant

[ ] Assistant adds ~5-10 news items from around the web as possible "News of the Week"

[ ] Get everything tested well before episode so you can debug
[ ] Use internal slack channel to ensure that AV setup is good

After show (ideally right away but can wait until Monday):

Create/submit PR for this repo

[ ] Episode directory and README from hackmd
[ ] Any other files from the episode that might be useful
[ ] Update playlist.md with episode
Feel free to self merge

[ ] Edit YT description to point to episode directory here
[ ] OBS/OS Upgrade if applicable (Never on the day of a show)

notes: https://hackmd.io/fLzz6E67QMypqNvwEmYsXg

Installing SPIRE can be pretty complex the first time around. For trying out SPIRE on Kubernetes, we recommend following the detailed walkthrough here. https://spiffe.io/spire/getting-started-k8s/
The tutorial is designed to have as little setup magic as possible, and walks through the components of SPIRE including the Agent and Server and describes briefly how they are configured.
The correct YAML files referenced in the tutorial can be found in the /k8s directory of this repo: https://github.com/spiffe/spire-tutorials

authors: ajessup jbeda

repo: vmware-tanzu/tgik


text:
Validated inputs with type="number" appear corrupted

[x] bug
[ ] feature request
[ ] enhancement

Expected behavior
Up/Down buttons in input[type="number"] should appear at the end of the input field.
Actual behavior
When I enable validation for the numeric text input and when I use the .valid class to style the input, the Up/Down buttons do not look properly aligned - they do not appear at the end of the text input.
The same issues can be observed with normal validated inputs in IE. There is a "x" button in normal text inputs in IE that appears misaligned when Clarity validation styling is added to it.
Reproduction of behavior
https://plnkr.co/1ND8cK
Environment details

Angular version: 2.0.X

Clarity version: 0.8.14

OS and version: Windows 8.1

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

@reddolan : This should probably be added to the broader research into form validation ala #45 and #46.

@tihomir-varbanov  Looking at the CSS for the element, I see a padding-right:28px applied which would push the stepper over from the far right side. This is to create space for the error icon in case of an error and to not cause collision/overlap issues.
@mathisscott noted, will add to the collection. Is it possible to have the padding only be applied when there is an error? Maybe by adding a class when an error is present?

@reddolan : Yeah, I think we could tie the right padding to the CSS that shows the error icon...

Here is the recommended design for validation on a input type="number"

This is different now that validation design has changed for new forms and this will no longer be an issue.

authors: gnomeontherun mathisscott reddolan tihomir-varbanov

repo: vmware/clarity


text:
[NG] Select row by clicking the row in the datagrid

Currently the row can be selected only by clicking on the checkbox, this allows the user to click anywhere in the row to select the row.
Signed-off-by: Sivakumar Annamalai kpa.siva@hotmail.com

Hi @sivaah, thanks for the contribution. This is a request we received several times, but it turns out we cannot provide this functionality if we want to support any kind of content in Datagrid cells.
Many use cases involve having links inside of the datagrid, or action items to perform an action on a specific row. In these cases, we would end up with "conflicts" between click handlers that would result in a very frustrating experience for the end user. Kind of like a button within another button.
That said, I am going to review your code right now to show the way we work, even though this will not be merged in. Please do not lose time fixing what I point out.
I will leave this review open if you want to give us more feedback or share your thoughts on that, and you can close the PR when you want.

On thing I forgot to mention is that this feature is quite easy to implement whenever you need it. It's just one binding in the template and you're done:
<clr-dg-row #row *clrDgItems="let item of items" [clrDgItem]="item"
    (click)="row.selected = !row.selected">

@youdz, I like the workaround you posted which solves my problem. I would have created an issue instead of stepping in and submitting a pull request :)

authors: sivaah youdz


text:
Wrong cloud-init datasource used. Nodes get stuck at bootstrapping phase.

Describe the bug
My understanding is that the bootstrapping nodes in CAPVCD leverages the VMware GuestInfo Cloud-Init datasource.
However, during our testing we see that VMs cloned from our TKG OVA template (ubuntu-2004-kube-v1.21.14+vmware.2) use the OVF Cloud-Init datasource, causing the nodes to get stuck in the bootstrapping phase.
root@ubuntu-2004-001:~# cloud-init status --long
status: done
time: Tue, 18 Oct 2022 09:42:29 +0000
detail:
DataSourceOVF [seed=com.vmware.guestInfo]
root@ubuntu-2004-001:~# DEBUG_LEVEL=2 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
Checking for datasource 'OVF' via 'dscheck_OVF'
Found guestinfo transport.
check for 'OVF' returned found
Checking for datasource 'VMware' via 'dscheck_VMware'
check for 'VMware' returned found
Found 2 datasources found=all: OVF VMware
If we manually disable OVF as a Cloud-Init datasource in the cloned VM and then reboot, we see that the VM is then successfully bootstrapped as expected.
root@demo-work-pool-1-78f97b8f69-2mj82:~# date
Tue 18 Oct 2022 11:49:05 AM UTC
kubectl describe vcdmachines demo-work-pool-1-ptsrt
Last Transition Time:  2022-10-18T11:25:17Z
Status:                True
Type:                  Ready
Last Transition Time:  2022-10-18T11:25:17Z
Status:                True
Type:                  BootstrapExecSucceeded
Last Transition Time:  2022-10-18T09:39:28Z
Status:                True
Type:                  ContainerProvisioned
Reproduction steps

Upload TKG OVA as an OVF template to a Cloud Director Catalog
Deploy a CAPVCD cluster referencing the TKG OVA
Observe that the Cloud-Init datasource used is OVF and not GuestInfo.

Expected behavior

Upload TKG OVA as an OVF template to a Cloud Director Catalog
Deploy a CAPVCD cluster referencing the TKG OVA
Cloned VM uses the GuestInfo Cloud-Init datasource by default so no manual intervention is needed to bootstrap the nodes.

Additional context
No response

This is fixed in CAPVCD 1.0.1.

Closing this issue based on the above comment

authors: Anirudh9794 arunmk jaymzmac


text:
Adding resize disk example

Hello,
I appreciate this package.
However, it is missing an resized disk example.
I based my example off of the 'change_disk_mode.py'
Thank you for your hard work,
Brandon Marick
Application Developer
Intel Corporation

Code fixed to use 1024 instead of 1048

hey @brmarick - can you fix the trailing white space on line 17? that's the reason why CI is failing.
"/home/travis/build/vmware/pyvmomi-community-samples/samples/resize_disk.py:3:17: W291 trailing whitespace"

@mindheist
Thanks for pointing this out.
I have fixed the issue

@brmarick - you are welcome. Hopefully, this gets pulled in.

authors: brmarick mindheist

repo: vmware/pyvmomi-community-samples


text:
vdk-core: add configurable write directory value

what: Added a configurable write directory which can be retrieved from the job_input object with a default value
why: Data jobs running in a cloud environment might have restricted access to the file system depending on the deployment. Data jobs which rely on the file system should have access to a writable folder - which should be configured by the cloud deployment
testing: added unit tests

I'm confused, I thought we already run jobs inside of an ephemeral storage dir which has all the necessary perms; I remember doing that change as part of the security enhancements for VDP certification. Is this change different from that or am I misunderstanding something?

I'm confused, I thought we already run jobs inside of an ephemeral storage dir which has all the necessary perms; I remember doing that change as part of the security enhancements for VDP certification. Is this change different from that or am I misunderstanding something?

We just added the additional ephemeral storage and the option to enable readOnlyRootFilesystem.

repo: vmware/versatile-data-kit


text:
Getting this error

Hey man, sorry i'm bothering you again. What is this?

I tried uninstalling and reinstalling everything, with and without cloudfare support, with and without bugfix version.
If anything then since the last time i had used it i've only installed node.js using its setup from the website and cfscrape using pip install cfscrape. I can work without kissanime so should i uninstall these if thats what is the solution?

9anime changed their api. I'll fix it once I get to my laptop. (Tomorrow, probably).
KissAnime should still be working.
Thanks for understanding.

No problem at all. Take your time.

If anyone out there wants to fix it, I think I can guide you.
Maybe changing variable named DD in 9anime.py in sites to bVZX0bdD might fix the problem.
I'll have to take a decent look at 9anime if not.

Just tested it. Can confirm that changing DD to bVZX0bdD does fix the problem

Yeah, it's a very boring Sunday :P
Just pushed it to my current PR.

Awesome. I'll merge it once ci passes.

I have 0 previous experience about forking and all. I didnt find any DD in anime.py but did find in nineanime.py.

Should i commit to master or create a new branch? After that should i reinstall anime downloaader and try the command again?
In case youre busy tell me to stop and ill leave things be.

Hey! This is my first time forking and contributing to a project, too.
From what I've read online, you should commit all your changes to the master branch in your fork after testing and then start a pull request from your fork
You don't have to do anything for now, though, since I already changed the variable in my own pull request.

Oh. It took me so long to figure out what to do that you'd already done it. So I deleted my comment, but you probably read it :P

@KashTag @Enderlein Sorry. I slept before looking into it yesterday.
@KashTag I'll find some easy issue for you to fix. I find it awesome that new people are contributing to open source through this project.
I'll improve the development instructions too. That should ease it up for newcomers.

Okay it's giving a key error again even though I can see the DD variable has the new bVZX0bdD value. Is it possible that 9anime's API changed yet again? If so is there any way I could find out this new API myself and do the forking and editing and pull request stuff?

authors: Enderlein KashTag vn-ki

repo: vn-ki/anime-downloader

license: Unlicense


text:
Not enter a tag if IME is running

Not enter a tag if IME is running
Fix
before
selected a tag when key down enter even if IME is running

after
Not select a tag when key down enter if IME is running

ref
https://developer.mozilla.org/en-US/docs/Mozilla/IME_handling_guide

Thanks! I would've never thought of this.

authors: AlexMordred Syati

repo: voerro/vue-tagsinput


text:
[Cherry-Pick} Added Configuration to enable/disable predicates

https://github.com/volcano-sh/volcano/issues/185

/lgtm
/approve

/lgtm
/approve

repo: volcano-sh/kube-batch


text:
354 re execute all notebooks with the updated environment

Description
Re-execute and check all old notebooks with the updated environment
See #354
Status

[ ] Ready to go

Hi @mbackenkoehler, thanks a lot for this! I think we will run into issues with notebook changes from this PR https://github.com/volkamerlab/teachopencadd/pull/348 but let's cross that bridge when we get there?

I am leaving out the broken T019

authors: dominiquesydow mbackenkoehler

repo: volkamerlab/teachopencadd


text:
grid yüklendiğinde ilk seferinde verilerin yüklenmesini istemiyorum

grid yüklendiğinde ilk seferinde verilerin yüklenmesini istemiyorum.
getViewOptions den autoLoad  denedim işe yaramadı başka bir yardımcı metod varmı?
        protected getViewOptions(): Slick.RemoteViewOptions {
            var opt = super.getViewOptions();
            opt.autoLoad = false;
            return opt;
        }

Hi oaltay,
if you Need only Volkan's help, then turkish is fine :-) but if you want that all of us potententially can help you, then may I kindly ask that you translate your question / issue into english :-)
With Kind regards,
John

Override initialPopulate(), don't call base method.

authors: JohnRanger oaltay volkanceylan

repo: volkanceylan/Serenity

license: mit


text:
Relax sized-ness constraint on RefCnt

if i try to use the type ArcSwap<[T]> i'd expect it to be able to use a fat pointer the same way it works for Arc<[T]>
instead i get this compiler error:
error[E0277]: the size for values of type `[T]` cannot be known at compilation time
   --> src/proxy/origin.rs:47:24
    |
47  |     live_destinations: Arc<ArcSwap<[T]>>,
    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^ doesn't have a size known at compile-time
    |
   ::: /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/arc-swap-0.4.7/src/lib.rs:686:26
    |
686 | pub struct ArcSwapAny<T: RefCnt, S: LockStorage = Global> {
    |                          ------ required by this bound in `arc_swap::ArcSwapAny`
    |
    = help: the trait `std::marker::Sized` is not implemented for `[T]`
    = note: required because of the requirements on the impl of `arc_swap::RefCnt` for `std::sync::Arc<[T]>`

(note: T is sized)
is there a way to relax this constraint?

oh, dupe of https://github.com/vorner/arc-swap/issues/27

repo: vorner/arc-swap


text:
Investigate Customer CORS and other issues

I was examining this user who used the portal widget generator in 5 Oct:
https://insights.hotjar.com/p?site=294008&recording=319832371&token=068bc123f0a39a105b4096051870a226
He tested his widget at http://codesofmusic.nl/test-webrtc/
He is getting some errors:

There are a few potential issues that require investigation:

[ ] Why is the widget showing when using the http url
[ ] What is causing those CORS issues?

He might be using an old version of the widget generator:

Yesterday we reached to the customer to ask them what they are trying to do with no response (yet).
Can you please confirm that there is no underlying issues here that would impact other users. We plan to start promote the Portal Widget Generator next week. It is important enough to spend a few minutes on by tomorrow.

We might need to ask the user to upgrade his widget code. Let me research a bit more on it

This should be done after the merged all the assets in the the Voxbone CDN. Moving to Icebox

repo: voxbone-workshop/portal-widget