text:
[Bug]: Engine loop has died for Meta-Llama-3.1-8B-Instruct TP=2

Your current environment

The output of `python collect_env.py`
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0
Clang version: Could not collect
CMake version: version 3.27.6
Libc version: glibc-2.35

Python version: 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.5.119
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX 6000 Ada Generation
GPU 1: NVIDIA RTX 6000 Ada Generation

Nvidia driver version: 550.107.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               36
On-line CPU(s) list:                  0-35
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Core(TM) i9-7980XE CPU @ 2.60GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   2
Core(s) per socket:                   18
Socket(s):                            1
Stepping:                             4
CPU max MHz:                          4000.0000
CPU min MHz:                          1200.0000
BogoMIPS:                             5199.98
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti ssbd mba ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts vnmi md_clear flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            576 KiB (18 instances)
L1i cache:                            576 KiB (18 instances)
L2 cache:                             18 MiB (18 instances)
L3 cache:                             24.8 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-35
Vulnerability Gather data sampling:   Mitigation; Microcode
Vulnerability Itlb multihit:          KVM: Mitigation: VMX disabled
Vulnerability L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; Clear CPU buffers; SMT vulnerable

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu124torch2.4
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.20
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.2
[pip3] triton==3.0.0
[conda] No relevant packages
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A (dev)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-35    0               N/A
GPU1    SYS      X      0-35    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Model Input Dumps
No response
🐛 Describe the bug
The error message:
INFO 11-07 19:58:29 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 11-07 19:58:29 engine.py:215] Waiting for new requests in engine loop.
INFO 11-07 19:58:30 logger.py:37] Received request chat-d4bc2e3f9df643d5ae0a7712c781874c: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 07 Nov 2024\n\nYou are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nTell me something about large language models.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8143, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), prompt_token_ids: [128000, 128006, 9125, 128007, 271, 38766, 1303, 33025, 2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696, 25, 220, 2589, 4723, 220, 2366, 19, 271, 2675, 527, 264, 11190, 18328, 13, 128009, 128006, 882, 128007, 271, 41551, 757, 2555, 922, 3544, 4221, 4211, 13, 128009, 128006, 78191, 128007, 271], lora_request: None, prompt_adapter_request: None.
DEBUG 11-07 19:58:30 async_llm_engine.py:525] Building guided decoding logits processor. Params: GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)
DEBUG 11-07 19:58:35 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:35 client.py:170] Waiting for output from MQLLMEngine.
DEBUG 11-07 19:58:35 client.py:154] Heartbeat successful.
INFO 11-07 19:58:35 engine.py:292] Added request chat-d4bc2e3f9df643d5ae0a7712c781874c.
DEBUG 11-07 19:58:35 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:37 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:39 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:41 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:43 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:45 client.py:170] Waiting for output from MQLLMEngine.
DEBUG 11-07 19:58:45 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:47 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:49 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:51 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:53 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:55 client.py:170] Waiting for output from MQLLMEngine.
DEBUG 11-07 19:58:55 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:57 client.py:154] Heartbeat successful.
DEBUG 11-07 19:58:59 client.py:154] Heartbeat successful.
ERROR 11-07 19:59:01 client.py:250] RuntimeError('Engine loop has died')
ERROR 11-07 19:59:01 client.py:250] Traceback (most recent call last):
ERROR 11-07 19:59:01 client.py:250]   File "/home/wanghaoy/.local/lib/python3.10/site-packages/vllm/engine/multiprocessing/client.py", line 150, in run_heartbeat_loop
ERROR 11-07 19:59:01 client.py:250]     await self._check_success(
ERROR 11-07 19:59:01 client.py:250]   File "/home/wanghaoy/.local/lib/python3.10/site-packages/vllm/engine/multiprocessing/client.py", line 314, in _check_success
ERROR 11-07 19:59:01 client.py:250]     raise response
ERROR 11-07 19:59:01 client.py:250] RuntimeError: Engine loop has died
DEBUG 11-07 19:59:05 client.py:170] Waiting for output from MQLLMEngine.
CRITICAL 11-07 19:59:05 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO:     10.239.82.78:57882 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [116760]

My serving script is
CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 --port 11600 \
    --gpu_memory_utilization 0.50 \
    --tensor_parallel_size 2 \
    --model local_ckpt/Meta-Llama-3.1-8B-Instruct --enforce-eager \
    --enable-auto-tool-choice --tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3.1_json.jinja \
    --max_model_len 8192

And my test script is
curl http://<public_ip>:11600/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "local_ckpt/Meta-Llama-3.1-8B-Instruct",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me something about large language models."}
  ],
  "temperature": 0.7,
  "top_p": 0.8,
  "repetition_penalty": 1.05
}'

This setting works well for a single GPU but fails when tp=2. Each time I start the request, the llm_engine just waits for a response and then times out.
My unsuccessful attempts to fix this issue include:

Increasing VLLM_RPC_TIMEOUT by 10 times (setting it to 100000): The same log appears, and it still times out.
change to other model (Qwen-AWQ/GPTQ), the same issue found.

I guess there might be some environment setting to be corrected for me because TP is a basic feature.
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Can you run this with --disable-frontend-multiprocessing? Perhaps the stack trace would be clearer.

Can you run this with --disable-frontend-multiprocessing? Perhaps the stack trace would be clearer.

Thanks for this suggestion! I disable frontend mp and test again, engine doesn't die but runs quite slowly (20 TPS on single GPU, but 0.1 TPS with TP=2).
INFO:     Started server process [136774]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on socket ('0.0.0.0', 11600) (Press CTRL+C to quit)
INFO 11-11 18:18:15 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running:
0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:18:25 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running:
0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:18:31 logger.py:37] Received request chat-27c13943c9f44d9c88a7843c520a418c: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 11 Nov 2024\n\nYou are a helpful
assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nTell me something about large language models.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, r
epetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8143, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_t
okens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), prompt_token_ids: [1280
00, 128006, 9125, 128007, 271, 38766, 1303, 33025, 2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696, 25, 220, 806, 4723, 220, 2366, 19, 271, 2675, 527, 264, 11190, 18328, 13, 128009, 128006, 882, 128007, 271, 41551, 757, 2555, 922, 3544, 4221, 421
1, 13, 128009, 128006, 78191, 128007, 271], lora_request: None, prompt_adapter_request: None.                              INFO 11-11 18:18:31 async_llm_engine.py:209] Added request chat-27c13943c9f44d9c88a7843c520a418c.
DEBUG 11-11 18:18:31 async_llm_engine.py:817] Waiting for new requests...                                                  DEBUG 11-11 18:18:31 async_llm_engine.py:836] Got new requests!
DEBUG 11-11 18:18:31 async_llm_engine.py:525] Building guided decoding logits processor. Params: GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)
INFO 11-11 18:18:36 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:18:46 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:18:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:19:06 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:19:16 metrics.py:345] Avg prompt throughput: 4.9 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:19:24 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running:
1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-11 18:19:31 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running:
1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

Since TP uses NCCL to communicate between GPU, maybe there's some issues in NCCL. I have tested NCCL with the following scripts before and it works well. But I am not familiar with NCCL, would you tell me how to further debug?
import os
import torch
import torch.distributed as dist
import torch.multiprocessing as mp

def init_process(rank, size, fn, backend='nccl'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

def run(rank, size):
    """ Distributed function to be implemented. """
    tensor = torch.ones(8000).cuda(rank)
    dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
    print(f'Rank {rank} has data {tensor}')

if __name__ == "__main__":
    size = 2  # Number of GPUs
    processes = []

    for rank in range(size):
        p = mp.Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

@youkaichao can you help with this?

@HaoyuWang4188 please follow https://docs.vllm.ai/en/latest/getting_started/debugging.html and report more information.

Tested with NCCL_DEBUG=TRACE torchrun --nproc-per-node=2 test.py in https://docs.vllm.ai/en/latest/getting_started/debugging.html, get the output:
(base) wanghaoy@rtx:~/code$ NCCL_DEBUG=TRACE torchrun --nproc-per-node=2 test_nccl.py
W1113 01:24:58.888000 126151159955904 torch/distributed/run.py:779]
W1113 01:24:58.888000 126151159955904 torch/distributed/run.py:779] *****************************************
W1113 01:24:58.888000 126151159955904 torch/distributed/run.py:779] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1113 01:24:58.888000 126151159955904 torch/distributed/run.py:779] *****************************************
rtx:145217:145217 [0] NCCL INFO Bootstrap : Using eth1:10.239.82.78<0>
rtx:145217:145217 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
rtx:145217:145217 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.20.5+cuda12.4
rtx:145218:145218 [1] NCCL INFO cudaDriverVersion 12040
rtx:145218:145218 [1] NCCL INFO Bootstrap : Using eth1:10.239.82.78<0>
rtx:145218:145218 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
rtx:145217:145234 [0] NCCL INFO Failed to open libibverbs.so[.1]
rtx:145217:145234 [0] NCCL INFO NET/Socket : Using [0]eth1:10.239.82.78<0>
rtx:145217:145234 [0] NCCL INFO Using non-device net plugin version 0
rtx:145217:145234 [0] NCCL INFO Using network Socket
rtx:145218:145235 [1] NCCL INFO Failed to open libibverbs.so[.1]
rtx:145218:145235 [1] NCCL INFO NET/Socket : Using [0]eth1:10.239.82.78<0>
rtx:145218:145235 [1] NCCL INFO Using non-device net plugin version 0
rtx:145218:145235 [1] NCCL INFO Using network Socket
rtx:145218:145235 [1] NCCL INFO comm 0x6296967b6a30 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0xb934e4b0ed3d1381 - Init START
rtx:145217:145234 [0] NCCL INFO comm 0x5c89293e5c20 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0xb934e4b0ed3d1381 - Init START
rtx:145217:145234 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ffffffff
rtx:145218:145235 [1] NCCL INFO Setting affinity for GPU 1 to 0f,ffffffff
rtx:145218:145235 [1] NCCL INFO comm 0x6296967b6a30 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
rtx:145217:145234 [0] NCCL INFO comm 0x5c89293e5c20 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
rtx:145217:145234 [0] NCCL INFO Channel 00/02 :    0   1
rtx:145218:145235 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
rtx:145217:145234 [0] NCCL INFO Channel 01/02 :    0   1
rtx:145218:145235 [1] NCCL INFO P2P Chunksize set to 131072
rtx:145217:145234 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
rtx:145217:145234 [0] NCCL INFO P2P Chunksize set to 131072
rtx:145218:145235 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145217:145234 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
rtx:145218:145235 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145217:145234 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
rtx:145218:145235 [1] NCCL INFO Connected all rings
rtx:145218:145235 [1] NCCL INFO Connected all trees
rtx:145218:145235 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:145218:145235 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145217:145234 [0] NCCL INFO Connected all rings
rtx:145217:145234 [0] NCCL INFO Connected all trees
rtx:145217:145234 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:145217:145234 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145217:145234 [0] NCCL INFO comm 0x5c89293e5c20 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0xb934e4b0ed3d1381 - Init COMPLETE
rtx:145218:145235 [1] NCCL INFO comm 0x6296967b6a30 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0xb934e4b0ed3d1381 - Init COMPLETE
PyTorch NCCL is successful!
PyTorch NCCL is successful!
PyTorch GLOO is successful!
PyTorch GLOO is successful!
/home/wanghaoy/.local/lib/python3.10/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
/home/wanghaoy/.local/lib/python3.10/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
INFO 11-13 01:25:03 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-13 01:25:03 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-13 01:25:03 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-13 01:25:03 pynccl.py:63] vLLM is using nccl==2.20.5
rtx:145218:145218 [1] NCCL INFO Using non-device net plugin version 0
rtx:145217:145217 [0] NCCL INFO Using non-device net plugin version 0
rtx:145218:145218 [1] NCCL INFO Using network Socket
rtx:145217:145217 [0] NCCL INFO Using network Socket
rtx:145217:145217 [0] NCCL INFO comm 0x5c892c973870 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0xf790f79c883b322f - Init START
rtx:145218:145218 [1] NCCL INFO comm 0x629699d44800 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0xf790f79c883b322f - Init START
rtx:145218:145218 [1] NCCL INFO Setting affinity for GPU 1 to 0f,ffffffff
rtx:145217:145217 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ffffffff
rtx:145218:145218 [1] NCCL INFO comm 0x629699d44800 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
rtx:145217:145217 [0] NCCL INFO comm 0x5c892c973870 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
rtx:145217:145217 [0] NCCL INFO Channel 00/02 :    0   1
rtx:145218:145218 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
rtx:145217:145217 [0] NCCL INFO Channel 01/02 :    0   1
rtx:145218:145218 [1] NCCL INFO P2P Chunksize set to 131072
rtx:145217:145217 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
rtx:145217:145217 [0] NCCL INFO P2P Chunksize set to 131072
rtx:145218:145218 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145217:145217 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
rtx:145218:145218 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145217:145217 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
rtx:145218:145218 [1] NCCL INFO Connected all rings
rtx:145218:145218 [1] NCCL INFO Connected all trees
rtx:145217:145217 [0] NCCL INFO Connected all rings
rtx:145217:145217 [0] NCCL INFO Connected all trees
rtx:145218:145218 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:145218:145218 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145217:145217 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:145217:145217 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145218:145218 [1] NCCL INFO comm 0x629699d44800 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0xf790f79c883b322f - Init COMPLETE
rtx:145217:145217 [0] NCCL INFO comm 0x5c892c973870 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0xf790f79c883b322f - Init COMPLETE
vLLM NCCL is successful!vLLM NCCL is successful!

vLLM NCCL with cuda graph is successful!vLLM NCCL with cuda graph is successful!

rtx:145218:145236 [1] NCCL INFO [Service thread] Connection closed by localRank 1
rtx:145217:145237 [0] NCCL INFO [Service thread] Connection closed by localRank 0
rtx:145217:145266 [0] NCCL INFO comm 0x5c89293e5c20 rank 0 nranks 2 cudaDev 0 busId 17000 - Abort COMPLETE
rtx:145218:145265 [1] NCCL INFO comm 0x6296967b6a30 rank 1 nranks 2 cudaDev 1 busId b3000 - Abort COMPLETE

And here's the full log of how the server hang/die:
INFO 11-13 01:33:02 api_server.py:528] vLLM API server version dev
INFO 11-13 01:33:02 api_server.py:529] args: Namespace(host='0.0.0.0', port=11600, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template='examples/tool_chat_template_llama3.1_json.jinja', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=True, enable_auto_tool_choice=True, tool_call_parser='llama3_json', tool_parser_plugin='', model='local_ckpt/Meta-Llama-3.1-8B-Instruct', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=8192, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.5, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, override_neuron_config=None, scheduling_policy='fcfs', disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False)
INFO 11-13 01:33:07 config.py:887] Defaulting to use mp for distributed inference
WARNING 11-13 01:33:07 config.py:380] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 11-13 01:33:07 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='local_ckpt/Meta-Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='local_ckpt/Meta-Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=local_ckpt/Meta-Llama-3.1-8B-Instruct, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
WARNING 11-13 01:33:08 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 18 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 11-13 01:33:08 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
WARNING 11-13 01:33:08 logger.py:147] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
INFO 11-13 01:33:08 logger.py:151] Trace frame log is saved to /tmp/vllm/vllm-instance-caddb0870600450383b5d06e167378b7/VLLM_TRACE_FUNCTION_for_process_145986_thread_126635487528512_at_2024-11-13_01:33:08.359088.log
(VllmWorkerProcess pid=146120) WARNING 11-13 01:33:08 logger.py:147] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
(VllmWorkerProcess pid=146120) INFO 11-13 01:33:08 logger.py:151] Trace frame log is saved to /tmp/vllm/vllm-instance-caddb0870600450383b5d06e167378b7/VLLM_TRACE_FUNCTION_for_process_146120_thread_126635487528512_at_2024-11-13_01:33:08.360649.log
(VllmWorkerProcess pid=146120) INFO 11-13 01:33:09 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
DEBUG 11-13 01:33:09 parallel_state.py:929] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41119 backend=nccl
(VllmWorkerProcess pid=146120) DEBUG 11-13 01:33:09 parallel_state.py:929] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41119 backend=nccl
(VllmWorkerProcess pid=146120) INFO 11-13 01:33:09 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=146120) INFO 11-13 01:33:09 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 11-13 01:33:09 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-13 01:33:09 pynccl.py:63] vLLM is using nccl==2.20.5
rtx:145986:146084 [0] NCCL INFO Bootstrap : Using eth1:10.239.82.78<0>
rtx:145986:146084 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
rtx:145986:146084 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.20.5+cuda12.4
rtx:146120:146120 [1] NCCL INFO cudaDriverVersion 12040
rtx:146120:146120 [1] NCCL INFO Bootstrap : Using eth1:10.239.82.78<0>
rtx:146120:146120 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
rtx:146120:146120 [1] NCCL INFO Failed to open libibverbs.so[.1]
rtx:145986:146084 [0] NCCL INFO Failed to open libibverbs.so[.1]
rtx:146120:146120 [1] NCCL INFO NET/Socket : Using [0]eth1:10.239.82.78<0>
rtx:146120:146120 [1] NCCL INFO Using non-device net plugin version 0
rtx:146120:146120 [1] NCCL INFO Using network Socket
rtx:145986:146084 [0] NCCL INFO NET/Socket : Using [0]eth1:10.239.82.78<0>
rtx:145986:146084 [0] NCCL INFO Using non-device net plugin version 0
rtx:145986:146084 [0] NCCL INFO Using network Socket
rtx:145986:146084 [0] NCCL INFO comm 0x732c7183e950 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0xe523c7d5f167c344 - Init START
rtx:146120:146120 [1] NCCL INFO comm 0x732c7183ed70 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0xe523c7d5f167c344 - Init START
rtx:146120:146120 [1] NCCL INFO Setting affinity for GPU 1 to 0f,ffffffff
rtx:145986:146084 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ffffffff
rtx:145986:146084 [0] NCCL INFO comm 0x732c7183e950 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
rtx:146120:146120 [1] NCCL INFO comm 0x732c7183ed70 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
rtx:145986:146084 [0] NCCL INFO Channel 00/02 :    0   1
rtx:146120:146120 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
rtx:145986:146084 [0] NCCL INFO Channel 01/02 :    0   1
rtx:146120:146120 [1] NCCL INFO P2P Chunksize set to 131072
rtx:145986:146084 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
rtx:145986:146084 [0] NCCL INFO P2P Chunksize set to 131072
rtx:146120:146120 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
rtx:145986:146084 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
rtx:146120:146120 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145986:146084 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
rtx:146120:146120 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145986:146084 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
rtx:146120:146120 [1] NCCL INFO Connected all rings
rtx:145986:146084 [0] NCCL INFO Connected all rings
rtx:146120:146120 [1] NCCL INFO Connected all trees
rtx:145986:146084 [0] NCCL INFO Connected all trees
rtx:146120:146120 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:146120:146120 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145986:146084 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:145986:146084 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145986:146084 [0] NCCL INFO comm 0x732c7183e950 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0xe523c7d5f167c344 - Init COMPLETE
rtx:146120:146120 [1] NCCL INFO comm 0x732c7183ed70 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0xe523c7d5f167c344 - Init COMPLETE
INFO 11-13 01:33:10 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/wanghaoy/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=146120) INFO 11-13 01:33:10 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/wanghaoy/.cache/vllm/gpu_p2p_access_cache_for_0,1.json
DEBUG 11-13 01:33:10 shm_broadcast.py:201] Binding to tcp://127.0.0.1:38063
INFO 11-13 01:33:10 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x732cd0ff1090>, local_subscribe_port=38063, remote_subscribe_port=None)
(VllmWorkerProcess pid=146120) DEBUG 11-13 01:33:10 shm_broadcast.py:265] Connecting to tcp://127.0.0.1:38063
INFO 11-13 01:33:10 model_runner.py:1060] Starting to load model local_ckpt/Meta-Llama-3.1-8B-Instruct...
(VllmWorkerProcess pid=146120) INFO 11-13 01:33:10 model_runner.py:1060] Starting to load model local_ckpt/Meta-Llama-3.1-8B-Instruct...
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:03,  1.19s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:02<00:02,  1.12s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.27it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.14it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.09it/s]

(VllmWorkerProcess pid=146120) INFO 11-13 01:33:15 model_runner.py:1071] Loading model weights took 7.5122 GB
INFO 11-13 01:33:15 model_runner.py:1071] Loading model weights took 7.5122 GB
rtx:145986:146185 [0] NCCL INFO Using non-device net plugin version 0
rtx:145986:146185 [0] NCCL INFO Using network Socket
rtx:146120:146186 [1] NCCL INFO Using non-device net plugin version 0
rtx:146120:146186 [1] NCCL INFO Using network Socket
rtx:146120:146186 [1] NCCL INFO comm 0x7328bf325c10 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0x88bda5f578366342 - Init START
rtx:145986:146185 [0] NCCL INFO comm 0x7328a331ec50 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0x88bda5f578366342 - Init START
rtx:146120:146186 [1] NCCL INFO Setting affinity for GPU 1 to 0f,ffffffff
rtx:145986:146185 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ffffffff
rtx:146120:146186 [1] NCCL INFO comm 0x7328bf325c10 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
rtx:145986:146185 [0] NCCL INFO comm 0x7328a331ec50 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
rtx:146120:146186 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
rtx:145986:146185 [0] NCCL INFO Channel 00/02 :    0   1
rtx:146120:146186 [1] NCCL INFO P2P Chunksize set to 131072
rtx:145986:146185 [0] NCCL INFO Channel 01/02 :    0   1
rtx:145986:146185 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
rtx:145986:146185 [0] NCCL INFO P2P Chunksize set to 131072
rtx:145986:146185 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
rtx:146120:146186 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145986:146185 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
rtx:146120:146186 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
rtx:145986:146185 [0] NCCL INFO Connected all rings
rtx:145986:146185 [0] NCCL INFO Connected all trees
rtx:146120:146186 [1] NCCL INFO Connected all rings
rtx:146120:146186 [1] NCCL INFO Connected all trees
rtx:146120:146186 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:146120:146186 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145986:146185 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:145986:146185 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:145986:146185 [0] NCCL INFO comm 0x7328a331ec50 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0x88bda5f578366342 - Init COMPLETE
rtx:146120:146186 [1] NCCL INFO comm 0x7328bf325c10 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0x88bda5f578366342 - Init COMPLETE
rtx:146120:146193 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
rtx:146120:146193 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
INFO 11-13 01:33:19 distributed_gpu_executor.py:57] # GPU blocks: 14616, # CPU blocks: 4096
INFO 11-13 01:33:19 distributed_gpu_executor.py:61] Maximum concurrency for 8192 tokens per request: 28.55x
INFO 11-13 01:33:30 chat_utils.py:352] Using supplied chat template:
... (details of chat_template)
INFO 11-13 01:33:30 chat_utils.py:352] {%- endif %}
INFO 11-13 01:33:30 chat_utils.py:352]
INFO 11-13 01:33:30 launcher.py:19] Available routes are:
INFO 11-13 01:33:30 launcher.py:27] Route: /openapi.json, Methods: HEAD, GET
INFO 11-13 01:33:30 launcher.py:27] Route: /docs, Methods: HEAD, GET
INFO 11-13 01:33:30 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GET
INFO 11-13 01:33:30 launcher.py:27] Route: /redoc, Methods: HEAD, GET
INFO 11-13 01:33:30 launcher.py:27] Route: /health, Methods: GET
INFO 11-13 01:33:30 launcher.py:27] Route: /tokenize, Methods: POST
INFO 11-13 01:33:30 launcher.py:27] Route: /detokenize, Methods: POST
INFO 11-13 01:33:30 launcher.py:27] Route: /v1/models, Methods: GET
INFO 11-13 01:33:30 launcher.py:27] Route: /version, Methods: GET
INFO 11-13 01:33:30 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 11-13 01:33:30 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 11-13 01:33:30 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO:     Started server process [145986]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on socket ('0.0.0.0', 11600) (Press CTRL+C to quit)
INFO 11-13 01:33:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
...
INFO 11-13 01:39:03 logger.py:37] Received request chat-fa706dfef62c402bb8174c9c922e7888: prompt: '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 13 Nov 2024\n\nYou are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nTell me something about large language models.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.05, temperature=0.7, top_p=0.8, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=8143, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), prompt_token_ids: [128000, 128006, 9125, 128007, 271, 38766, 1303, 33025, 2696, 25, 6790, 220, 2366, 18, 198, 15724, 2696, 25, 220, 1032, 4723, 220, 2366, 19, 271, 2675, 527, 264, 11190, 18328, 13, 128009, 128006, 882, 128007, 271, 41551, 757, 2555, 922, 3544, 4221, 4211, 13, 128009, 128006, 78191, 128007, 271], lora_request: None, prompt_adapter_request: None.
INFO 11-13 01:39:03 async_llm_engine.py:209] Added request chat-fa706dfef62c402bb8174c9c922e7888.
DEBUG 11-13 01:39:03 async_llm_engine.py:817] Waiting for new requests...
DEBUG 11-13 01:39:03 async_llm_engine.py:836] Got new requests!
DEBUG 11-13 01:39:03 async_llm_engine.py:525] Building guided decoding logits processor. Params: GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)
INFO 11-13 01:39:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:39:20 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:39:30 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:39:40 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:39:49 metrics.py:345] Avg prompt throughput: 5.5 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:39:56 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:40:03 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 11-13 01:40:10 metrics.py:345] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

My script to run tp is in description, adding --disable-frontend-multiprocessing to make the log clear.

distributed_init_method=tcp://127.0.0.1:41119 backend=nccl is found in log. Is this internal IP expected?
Besides, according to the sec Incorrect network setup, I set the following environment:
export VLLM_HOST_IP="<my_ipv4_address>"
export NCCL_SOCKET_IFNAME="eth1"
export GLOO_SOCKET_IFNAME="eth1"

And get this new log (the issue not solved, and get distributed_init_method=tcp://127.0.0.1:57685 backend=nccl):
(VllmWorkerProcess pid=146919) WARNING 11-13 01:55:42 logger.py:147] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
(VllmWorkerProcess pid=146919) INFO 11-13 01:55:42 logger.py:151] Trace frame log is saved to /tmp/vllm/vllm-instance-3463d543c978466b86e6e2a785d7716e/VLLM_TRACE_FUNCTION_for_process_146919_thread_133442249426496_at_2024-11-13_01:55:42.751765.log
(VllmWorkerProcess pid=146919) INFO 11-13 01:55:43 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
DEBUG 11-13 01:55:44 parallel_state.py:929] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:57685 backend=nccl
(VllmWorkerProcess pid=146919) DEBUG 11-13 01:55:44 parallel_state.py:929] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:57685 backend=nccl
INFO 11-13 01:55:44 utils.py:1008] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=146919) INFO 11-13 01:55:44 utils.py:1008] Found nccl from library libnccl.so.2
INFO 11-13 01:55:44 pynccl.py:63] vLLM is using nccl==2.20.5
(VllmWorkerProcess pid=146919) INFO 11-13 01:55:44 pynccl.py:63] vLLM is using nccl==2.20.5
rtx:146790:146883 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth1
rtx:146790:146883 [0] NCCL INFO Bootstrap : Using eth1:<my_ipv4_address><0>
rtx:146790:146883 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
rtx:146790:146883 [0] NCCL INFO cudaDriverVersion 12040
NCCL version 2.20.5+cuda12.4
rtx:146919:146919 [1] NCCL INFO cudaDriverVersion 12040
rtx:146919:146919 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth1
rtx:146919:146919 [1] NCCL INFO Bootstrap : Using eth1:<my_ipv4_address><0>
rtx:146919:146919 [1] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation
rtx:146919:146919 [1] NCCL INFO Failed to open libibverbs.so[.1]
rtx:146919:146919 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth1
rtx:146919:146919 [1] NCCL INFO NET/Socket : Using [0]eth1:<my_ipv4_address><0>
rtx:146919:146919 [1] NCCL INFO Using non-device net plugin version 0
rtx:146919:146919 [1] NCCL INFO Using network Socket
rtx:146790:146883 [0] NCCL INFO Failed to open libibverbs.so[.1]
rtx:146790:146883 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth1
rtx:146790:146883 [0] NCCL INFO NET/Socket : Using [0]eth1:<my_ipv4_address><0>
rtx:146790:146883 [0] NCCL INFO Using non-device net plugin version 0
rtx:146790:146883 [0] NCCL INFO Using network Socket
rtx:146919:146919 [1] NCCL INFO comm 0x795d4583f560 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0x3fed5c29877812a5 - Init START
rtx:146790:146883 [0] NCCL INFO comm 0x795d45840fd0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0x3fed5c29877812a5 - Init START
rtx:146919:146919 [1] NCCL INFO Setting affinity for GPU 1 to 0f,ffffffff
rtx:146790:146883 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ffffffff
rtx:146790:146883 [0] NCCL INFO comm 0x795d45840fd0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
rtx:146919:146919 [1] NCCL INFO comm 0x795d4583f560 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
rtx:146790:146883 [0] NCCL INFO Channel 00/02 :    0   1
rtx:146919:146919 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
rtx:146790:146883 [0] NCCL INFO Channel 01/02 :    0   1
rtx:146919:146919 [1] NCCL INFO P2P Chunksize set to 131072
rtx:146790:146883 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
rtx:146790:146883 [0] NCCL INFO P2P Chunksize set to 131072
rtx:146919:146919 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
rtx:146790:146883 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
rtx:146919:146919 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
rtx:146790:146883 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
rtx:146919:146919 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
rtx:146790:146883 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
rtx:146790:146883 [0] NCCL INFO Connected all rings
rtx:146790:146883 [0] NCCL INFO Connected all trees
rtx:146919:146919 [1] NCCL INFO Connected all rings
rtx:146919:146919 [1] NCCL INFO Connected all trees
rtx:146919:146919 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:146919:146919 [1] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:146790:146883 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
rtx:146790:146883 [0] NCCL INFO 2 coll channels, 0 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
rtx:146790:146883 [0] NCCL INFO comm 0x795d45840fd0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 17000 commId 0x3fed5c29877812a5 - Init COMPLETE
rtx:146919:146919 [1] NCCL INFO comm 0x795d4583f560 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b3000 commId 0x3fed5c29877812a5 - Init COMPLETE

I got it. In my case, my actual target is to deploy Qwen2.5-72B with as long context as possible and acceptable latency. With single RTX 6000 48G, context can be set to 15K (AWQ-int4 and fp8 kv-cache).
export VLLM_ATTENTION_BACKEND=FLASHINFER
CUDA_VISIBLE_DEVICES=1 python -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 --port 11500 \
    --gpu_memory_utilization 0.98 \
    --model Qwen/Qwen2.5-72B-Instruct-AWQ --enforce-eager \
    --max_model_len 15360 --kv-cache-dtype fp8_e5m2 \
    --quantization awq_marlin \
    --enable-auto-tool-choice --tool-call-parser hermes

So, my question is: For a single node with 2-3 RTX6000 48G, is there more thing I can do to increase the context length if TP is not used? As I know, vllm is designed for high-throughput serving, is there any choice to decrease possible batch_size but increasing the context length (because the request per second in this case is low)?

decrease possible batch_size

you can limit --max-num-seqs

Thanks for all the help! The same setting works well on other GPUs. Close this issue.

repo: vllm-project/vllm


text:
[Bug]: continue_final_message doesn't support strings that end with '\n', '\t', ' ' and so on, for Llama-3.1-8B-Instruct.

Your current environment
vllm 0.6.3.post1
Model Input Dumps
No response
🐛 Describe the bug
I used the following code to call Llama-3.1-8B-Instruct.
import requests

no_bug_messages = [
        {'role': 'user', 'content': """Hello, tell me a joke."""},
        {'role': 'assistant', 'content': """OK!"""},
    ]
bug_messages1 = [
        {'role': 'user', 'content': """Hello, tell me a joke."""},
        {'role': 'assistant', 'content': """OK!\n"""},
    ]
bug_messages2 = [
        {'role': 'user', 'content': """Hello, tell me a joke."""},
        {'role': 'assistant', 'content': """OK! """},
    ]
response = requests.post(url, headers=header,
                             json={
                                 "model": model,
                                 "messages": no_bug_messages,
                                 "continue_final_message": True,
                                 "add_generation_prompt": False})
print(response.json())

When I used no_bug_messages, the response was OK. But when I used bug_messages1, the response was as follows:
{'object': 'error', 'message': 'substring not found', 'type': 'BadRequestError', 'param': None, 'code': 400}

The log is as follows:
ERROR 10-21 15:57:28 serving_chat.py:156] Error in applying chat template from request
ERROR 10-21 15:57:28 serving_chat.py:156] Traceback (most recent call last):
ERROR 10-21 15:57:28 serving_chat.py:156]   File "/aaa/conda_envs/vllm040/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_chat.py", line 145, in create_chat_completion
ERROR 10-21 15:57:28 serving_chat.py:156]     prompt = apply_hf_chat_template(
ERROR 10-21 15:57:28 serving_chat.py:156]   File "/aaa/conda_envs/vllm040/lib/python3.10/site-packages/vllm/entrypoints/chat_utils.py", line 552, in apply_hf_chat_template
ERROR 10-21 15:57:28 serving_chat.py:156]     return tokenizer.apply_chat_template(
ERROR 10-21 15:57:28 serving_chat.py:156]   File "/aaa/conda_envs/vllm040/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1876, in apply_chat_template
ERROR 10-21 15:57:28 serving_chat.py:156]     rendered_chat = rendered_chat[: rendered_chat.rindex(final_message) + len(final_message)].rstrip()
ERROR 10-21 15:57:28 serving_chat.py:156] ValueError: substring not found
INFO:     10.226.155.0:30663 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Same issue, This is a very annoying mistake, why close the issue?

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
How is vlmaps_dataset.zip generated?

Asking this because I wanna to test with a new scene where we have both depth, RGB and pose info of the mobile robot. I got confused on how is the vlmaps_dataset generated.

sorry for the delay. Regarding the data generation, we manually control a robot in the habitat-sim to collect exploration data (RGB, depth, robot pose etc.). The generate_dataset.py code downloads the pre-collected trajectories (only poses) in 10 different scenes in habitat-sim + Matterport3D and places an agent at those poses to generate the observations (RGBD). If you want to collect your own data in habitat-sim, we just update a script to collect your own data: here. Please follow the instruction in readme to collect your own data: Collect your own data in Habitat-Sim.

authors: Tom-Huang seusofthd

repo: vlmaps/vlmaps

license: MIT


text:
Ignore missing file

Hello everyone, first of all thanks for this project.
I'd like to introduce ( as in the Ruby counterpart ) the skip-on-missing-file feature, particularly handy for dev/test environments.
This implementation introduce this feature for load and overload and provides a new force_load method to explicitly retain the current behaviour ( throw on missing ).

@GrahamCampbell thank you for reviewing the code.
I made the required changes!

@GrahamCampbell I refactored the code as you requested.
But reading the load function in Loader ( lines 75 and 78 ) my understanding is that the same race condition exists. Am I wrong?
Thank you.

dupe of https://github.com/vlucas/phpdotenv/pull/224? 👀

@svjacob the main idea behind this PR was to be more similar to the ruby dotenv library.
The final implementation after some reviews by @GrahamCampbell is really similar to the one in the #224. ( Code has been rebased, so unfortunately is not possible to look at that implementation anymore )
Personally I would avoid a flag that changes method behaviour :) but I'm open to alternatives.

@svjacob the main idea behind this PR was to be more similar to the ruby dotenv library.

I'd rather maintain BC than mimic the ruby API.

This PR looks very good now. 👍
One other thing. Is there any chance you could add @throws annotations to the functions that may raise the exception. This is really a documentation bug on our part. :)

Will merge this as is, and make an issue for the other thing.
Thank you! 🍻

Thank you!
I'll follow the related issue and make a PR for it tomorrow ( I'm in CEST
timezone ).

@GrahamCampbell Today I noticed that the 2.4.0 release does not contain this code. Is this expected?

source: gharchive/pull-request

authors: GrahamCampbell endorama svjacob

repo: vlucas/phpdotenv

license_source: bigquery


text:
Fixed warning

Well, since everything went so smoothly with the previous PR, here some more improvements

Warning removed (https://github.com/vmari/DotEnv.jl/issues/5), this fix also duplicates https://github.com/vmari/DotEnv.jl/pull/7
Extended load definition, now it supports kwargs in the same manner as config.
Synchronized override options in kwargs and args versions.
Added Manifest.toml, so this PR also duplicates https://github.com/vmari/DotEnv.jl/pull/6

Thanks again 🥳

authors: Arkoniak vmari


text:
Update beta.kubernetes.io/os nodeSelector to kubernetes.io/os

As per https://v1-14.docs.kubernetes.io/docs/setup/release/notes/ beta.kubernetes.io/os has been deprecated and will be completely removed in k8s 1.18. That said, please update beta.kubernetes.io/os nodeSelector to kubernetes.io/os in the antrea yml file.

Hi all, I would like to complete this issue!
Thanks, Dilan

@dilan-js do you still intend to take a stab at this? Thanks.

No update, so re-assigning the issue to myself to make sure it gets taken care of in time for the 0.5.0 release.

authors: alex-vmw antoninbas dilan-js

repo: vmware-tanzu/antrea


text:
Filter invalid machine types to prevent invalid selection in UI

Feature Request
Currently when installing mgmt cluster on AWS, the UI dropdown list for control plane and worker node machine types includes invalid selections including types that may be below minimum requirements or with invalid hardware such as arm processors.
I am not sure if this is also a problem on Azure or vSphere.
The dropdown list should ideally be filtered to ensure only valid machine types are displayed.
Thanks!
Describe alternatives you've considered

Additional context

We can use this issue to track picking up this from tanzu-framework: https://github.com/vmware-tanzu/tanzu-framework/issues/861

Closing - This issue can be handled via https://github.com/vmware-tanzu/community-edition/issues/2134

authors: afewell seemiller stmcginnis

repo: vmware-tanzu/community-edition


text:
Simplify config handling

In the quest for removing code and eventually the special handling
of the e2e/systemdlogs plugins, I had to start untangling some of the
different configuration management.
It was always more complex than desired but we never had enough
of a reason to spend the time and untangle it. We have that
motivation now since in order to remove those plugins from the core
I have to wire up numerous types and flags differently which impacts
all of the other config 'resolution' code.
In this PR, I took the 4 fields from the genFlags type which were
duplicated in the config (it had a config itself as well as these
four fields for convenience setting from the cmdline). Now it properly
merges the different flags/configs appropriately regardless where they
are set.
Signed-off-by: John Schnake jschnake@vmware.com
What this PR does / why we need it:
Which issue(s) this PR fixes

Fixes #

Special notes for your reviewer:
Release note:
release-note

Some yak shearing to occur here; other PRs to precede this one and ensure we add some extra tests and such

@zubron I tagged you in this even though you dont work primarily on sonobuoy anymore; I just thought that we might benefit from another set of eyes and I know you had tried to untangle some of these things in the past. I might have even fought to keep some of the weird logic. No more!
@vladimirvivien Tiny tiny tiny concession of logic in order to remove 600 LOC and simplify code flows. I hope you'll agree that its worth it.

/approve

authors: johnSchnake vladimirvivien

repo: vmware-tanzu/sonobuoy


text:
Are project configurations supposed to be dropped?

Describe the bug
Submitting a project configuration loses details.
To Reproduce
Steps to reproduce the behavior:

Add a project configuration

curl -X POST "http://127.0.0.1:8090/burp/configuration" -H "accept: */*" -H "Content-Type: application/json" -d "{ \"project_options\":{ \"connections\":{ \"hostname_resolution\":[], \"out_of_scope_requests\":{ \"advanced_mode\":false, \"drop_all_out_of_scope\":false, \"exclude\":[], \"include\":[], \"scope_option\":\"suite\" }, \"platform_authentication\":{ \"credentials\":[], \"do_platform_authentication\":true, \"prompt_on_authentication_failure\":false, \"use_user_options\":true }, \"socks_proxy\":{ \"dns_over_socks\":false, \"host\":\"\", \"password\":\"\", \"port\":0, \"use_proxy\":false, \"use_user_options\":true, \"username\":\"\" }, \"timeouts\":{ \"domain_name_resolution_timeout\":300000, \"failed_domain_name_resolution_timeout\":60000, \"normal_timeout\":120000, \"open_ended_response_timeout\":10000 }, \"upstream_proxy\":{ \"servers\":[], \"use_user_options\":true } }, \"http\":{ \"redirections\":{ \"understand_3xx_status_code\":true, \"understand_any_status_code_with_location_header\":false, \"understand_javascript_driven\":false, \"understand_meta_refresh_tag\":true, \"understand_refresh_header\":true }, \"status_100_responses\":{ \"remove_100_continue_responses\":false, \"understand_100_continue_responses\":true }, \"streaming_responses\":{ \"scope_advanced_mode\":false, \"store\":true, \"strip_chunked_encoding_metadata\":true, \"urls\":[] } }, \"misc\":{ \"collaborator_server\":{ \"location\":\"\", \"poll_over_unencrypted_http\":false, \"polling_location\":\"\", \"type\":\"default\" }, \"logging\":{ \"requests\":{ \"all_tools\":\"\", \"extender\":\"\", \"intruder\":\"\", \"proxy\":\"\", \"repeater\":\"\", \"scanner\":\"\", \"sequencer\":\"\", \"spider\":\"\" }, \"responses\":{ \"all_tools\":\"\", \"extender\":\"\", \"intruder\":\"\", \"proxy\":\"\", \"repeater\":\"\", \"scanner\":\"\", \"sequencer\":\"\", \"spider\":\"\" } }, \"scheduled_tasks\":{ \"tasks\":[] } }, \"sessions\":{ \"cookie_jar\":{ \"monitor_extender\":false, \"monitor_intruder\":false, \"monitor_proxy\":true, \"monitor_repeater\":false, \"monitor_scanner\":false, \"monitor_sequencer\":false, \"monitor_spider\":true }, \"macros\":{ \"macros\":[] }, \"session_handling_rules\":{ \"rules\":[ { \"actions\":[ { \"enabled\":true, \"match_cookies\":\"all_except\", \"type\":\"use_cookies\" } ], \"description\":\"Use cookies from Burp's cookie jar\", \"enabled\":true, \"exclude_from_scope\":[], \"include_in_scope\":[], \"named_params\":[], \"restrict_scope_to_named_params\":false, \"tools_scope\":[ \"Spider\", \"Scanner\" ], \"url_scope\":\"all\", \"url_scope_advanced_mode\":false } ] } }, \"ssl\":{ \"client_certificates\":{ \"certificates\":[], \"use_user_options\":true }, \"negotiation\":{ \"allow_unsafe_renegotiation\":false, \"automatically_select_compatible_ssl_parameters_on_failure\":true, \"disable_ssl_session_resume\":false, \"enabled_ciphers\":[], \"enabled_protocols\":[], \"use_platform_default_protocols_and_ciphers\":true } } }, \"proxy\":{ \"http_history_display_filter\":{ \"by_annotation\":{ \"show_only_commented_items\":false, \"show_only_highlighted_items\":false }, \"by_file_extension\":{ \"hide_items\":[ \"js\", \"gif\", \"jpg\", \"png\", \"css\" ], \"hide_specific\":false, \"show_items\":[ \"asp\", \"aspx\", \"jsp\", \"php\" ], \"show_only_specific\":false }, \"by_listener\":{ \"port\":\"\" }, \"by_mime_type\":{ \"show_css\":false, \"show_flash\":true, \"show_html\":true, \"show_images\":false, \"show_other_binary\":false, \"show_other_text\":true, \"show_script\":true, \"show_xml\":true }, \"by_request_type\":{ \"hide_items_without_responses\":false, \"show_only_in_scope_items\":false, \"show_only_parameterized_requests\":false }, \"by_search\":{ \"case_sensitive\":false, \"negative_search\":false, \"regex\":false, \"term\":\"\" }, \"by_status_code\":{ \"show_2xx\":true, \"show_3xx\":true, \"show_4xx\":true, \"show_5xx\":true } }, \"intercept_client_requests\":{ \"automatically_fix_missing_or_superfluous_new_lines_at_end_of_request\":false, \"automatically_update_content_length_header_when_the_request_is_edited\":true, \"do_intercept\":false, \"rules\":[ { \"boolean_operator\":\"and\", \"enabled\":true, \"match_condition\":\"(^gif$|^jpg$|^png$|^css$|^js$|^ico$)\", \"match_relationship\":\"does_not_match\", \"match_type\":\"file_extension\" }, { \"boolean_operator\":\"or\", \"enabled\":false, \"match_relationship\":\"contains_parameters\", \"match_type\":\"request\" }, { \"boolean_operator\":\"or\", \"enabled\":false, \"match_condition\":\"(get|post)\", \"match_relationship\":\"does_not_match\", \"match_type\":\"http_method\" }, { \"boolean_operator\":\"and\", \"enabled\":false, \"match_relationship\":\"is_in_target_scope\", \"match_type\":\"url\" } ] }, \"intercept_server_responses\":{ \"automatically_update_content_length_header_when_the_response_is_edited\":true, \"do_intercept\":false, \"rules\":[ { \"boolean_operator\":\"or\", \"enabled\":true, \"match_condition\":\"text\", \"match_relationship\":\"matches\", \"match_type\":\"content_type_header\" }, { \"boolean_operator\":\"or\", \"enabled\":false, \"match_relationship\":\"was_modified\", \"match_type\":\"request\" }, { \"boolean_operator\":\"or\", \"enabled\":false, \"match_relationship\":\"was_intercepted\", \"match_type\":\"request\" }, { \"boolean_operator\":\"and\", \"enabled\":false, \"match_condition\":\"^304$\", \"match_relationship\":\"does_not_match\", \"match_type\":\"status_code\" }, { \"boolean_operator\":\"and\", \"enabled\":false, \"match_relationship\":\"is_in_target_scope\", \"match_type\":\"url\" } ] }, \"intercept_web_sockets_messages\":{ \"client_to_server_messages\":true, \"server_to_client_messages\":true }, \"match_replace_rules\":[ { \"comment\":\"Emulate IE\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^User-Agent.*$\", \"string_replace\":\"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)\" }, { \"comment\":\"Emulate iOS\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^User-Agent.*$\", \"string_replace\":\"User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B176 Safari/7534.48.3\" }, { \"comment\":\"Emulate Android\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^User-Agent.*$\", \"string_replace\":\"User-Agent: Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1\" }, { \"comment\":\"Require non-cached response\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^If-Modified-Since.*$\" }, { \"comment\":\"Require non-cached response\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^If-None-Match.*$\" }, { \"comment\":\"Hide Referer header\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^Referer.*$\" }, { \"comment\":\"Require non-compressed responses\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^Accept-Encoding.*$\" }, { \"comment\":\"Ignore cookies\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"response_header\", \"string_match\":\"^Set-Cookie.*$\" }, { \"comment\":\"Rewrite Host header\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"request_header\", \"string_match\":\"^Host: foo.example.org$\", \"string_replace\":\"Host: bar.example.org\" }, { \"comment\":\"Add spoofed CORS origin\", \"enabled\":false, \"is_simple_match\":true, \"rule_type\":\"request_header\", \"string_replace\":\"Origin: foo.example.org\" }, { \"comment\":\"Remove HSTS headers\", \"enabled\":false, \"is_simple_match\":false, \"rule_type\":\"response_header\", \"string_match\":\"^Strict\\\\-Transport\\\\-Security.*$\" }, { \"comment\":\"Disable browser XSS protection\", \"enabled\":false, \"is_simple_match\":true, \"rule_type\":\"response_header\", \"string_replace\":\"X-XSS-Protection: 0\" } ], \"miscellaneous\":{ \"disable_logging_to_history_and_site_map\":false, \"disable_out_of_scope_logging_to_history_and_site_map\":false, \"disable_web_interface\":false, \"remove_unsupported_encodings_from_accept_encoding_headers_in_incoming_requests\":true, \"set_connection_close_header_on_requests\":true, \"set_connection_close_header_on_responses\":false, \"strip_proxy_headers_in_incoming_requests\":true, \"strip_sec_websocket_extensions_headers_in_incoming_requests\":true, \"suppress_burp_error_messages_in_browser\":false, \"unpack_gzip_deflate_in_requests\":false, \"unpack_gzip_deflate_in_responses\":true, \"use_http_10_in_requests_to_server\":false, \"use_http_10_in_responses_to_client\":false }, \"request_listeners\":[ { \"certificate_mode\":\"per_host\", \"listen_mode\":\"loopback_only\", \"listener_port\":8080, \"running\":true } ], \"response_modification\":{ \"convert_https_links_to_http\":false, \"enable_disabled_form_fields\":false, \"highlight_unhidden_fields\":false, \"remove_all_javascript\":false, \"remove_input_field_length_limits\":false, \"remove_javascript_form_validation\":false, \"remove_object_tags\":false, \"remove_secure_flag_from_cookies\":false, \"unhide_hidden_form_fields\":false }, \"ssl_pass_through\":{ \"automatically_add_entries_on_client_ssl_negotiation_failure\":false, \"rules\":[] }, \"web_sockets_history_display_filter\":{ \"by_annotation\":{ \"show_only_commented_items\":false, \"show_only_highlighted_items\":false }, \"by_listener\":{ \"listener_port\":\"\" }, \"by_request_type\":{ \"hide_incoming_messages\":false, \"hide_outgoing_messages\":false, \"show_only_in_scope_items\":false }, \"by_search\":{ \"case_sensitive\":false, \"negative_search\":false, \"regex\":false, \"term\":\"\" } } }, \"repeater\":{ \"follow_redirections\":\"never\", \"process_cookies_in_redirections\":false, \"unpack_gzip_deflate\":true, \"update_content_length\":true }, \"scanner\":{ \"active_scanning_engine\":{ \"concurrent_request_limit\":10, \"do_throttle\":false, \"follow_redirects\":true, \"number_of_retries_on_failure\":3, \"pause_before_retry_on_failure\":2000, \"throttle_interval\":500, \"throttle_random\":false }, \"active_scanning_optimization\":{ \"intelligent_attack_selection\":true, \"scan_accuracy\":\"normal\", \"scan_speed\":\"normal\" }, \"attack_insertion_points\":{ \"change_body_to_cookie\":false, \"change_body_to_url\":false, \"change_cookie_to_body\":false, \"change_cookie_to_url\":false, \"change_url_to_body\":false, \"change_url_to_cookie\":false, \"insert_amf_params\":false, \"insert_body_params\":true, \"insert_cookies\":true, \"insert_entire_body\":true, \"insert_http_headers\":true, \"insert_param_names\":true, \"insert_url_params\":true, \"insert_url_path_filename\":true, \"insert_url_path_folders\":false, \"max_insertion_points\":30, \"skip_all_tests_for_parameters\":[], \"skip_server_side_injection_for_parameters\":[ { \"enabled\":true, \"expression\":\"aspsessionid.*\", \"item\":\"name\", \"match_type\":\"matches_regex\", \"parameter\":\"cookie\" }, { \"enabled\":true, \"expression\":\"asp.net_sessionid\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"cookie\" }, { \"enabled\":true, \"expression\":\"__eventtarget\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"body_parameter\" }, { \"enabled\":true, \"expression\":\"__eventargument\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"body_parameter\" }, { \"enabled\":true, \"expression\":\"__viewstate\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"body_parameter\" }, { \"enabled\":true, \"expression\":\"__eventvalidation\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"body_parameter\" }, { \"enabled\":true, \"expression\":\"jsessionid\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"any_parameter\" }, { \"enabled\":true, \"expression\":\"cfid\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"cookie\" }, { \"enabled\":true, \"expression\":\"cftoken\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"cookie\" }, { \"enabled\":true, \"expression\":\"PHPSESSID\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"cookie\" }, { \"enabled\":true, \"expression\":\"session_id\", \"item\":\"name\", \"match_type\":\"is\", \"parameter\":\"cookie\" } ], \"use_nested_insertion_points\":true }, \"live_active_scanning\":{ \"advanced_mode\":false, \"exclude\":[], \"include\":[], \"scope_option\":\"nothing\" }, \"live_passive_scanning\":{ \"advanced_mode\":false, \"exclude\":[], \"include\":[], \"scope_option\":\"everything\" }, \"scan_issues\":{ \"scan_type_intrusive_active\":true, \"scan_type_light_active\":true, \"scan_type_medium_active\":true, \"scan_type_passive\":true, \"scan_type_static_code_analysis\":true, \"select_individual_issues\":false, \"selected_issues\":[] }, \"scan_queue\":{ \"hide_finished_items\":false }, \"static_code_analysis\":{ \"max_time_per_item\":120, \"mode\":\"active_only\" } }, \"sequencer\":{ \"live_capture\":{ \"ignore_abnormal_length_tokens\":true, \"max_length_deviation\":5, \"num_threads\":5, \"throttle\":0 }, \"token_analysis\":{ \"compression\":true, \"correlation\":true, \"count\":true, \"fips_long_run\":true, \"fips_monobit\":true, \"fips_poker\":true, \"fips_runs\":true, \"spectral\":true, \"transitions\":true }, \"token_handling\":{ \"base_64_decode_before_analyzing\":false, \"pad_short_tokens_at\":\"start\", \"pad_with\":\"0\" } }, \"spider\":{ \"application_login\":{ \"mode\":\"prompt\", \"password\":\"Mike\", \"username\":\"password\" }, \"crawler\":{ \"check_robots_text\":true, \"detect_custom_not_found_responses\":true, \"ignore_links_to_non_text_content\":true, \"make_non_parameterized_request_to_dynamic_pages\":true, \"max_link_depth\":5, \"max_parameterized_requests_per_url\":50, \"request_root_of_all_directories\":true }, \"engine\":{ \"add_random_variation_to_throttle\":false, \"number_of_retries_on_failure\":3, \"number_of_threads\":10, \"pause_before_retry_on_failure\":2000, \"throttle_between_requests\":false, \"throttle_interval\":0 }, \"form_submission\":{ \"default_auto_fill_value\":\"555-555-0199@example.com\", \"individuate_forms_by\":\"action_url_method_and_fields\", \"iterate_all_values_of_submit_fields\":true, \"max_submissions_per_form\":10, \"mode\":\"automatic\", \"param_auto_fill_rules\":[ { \"enabled\":true, \"field_name\":\"mail\", \"field_value\":\"winter@example.com\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"first\", \"field_value\":\"Peter\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"last\", \"field_value\":\"Winter\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"surname\", \"field_value\":\"Winter\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"name\", \"field_value\":\"Peter Winter\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"comp\", \"field_value\":\"Winter Consulting\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"addr\", \"field_value\":\"1 Main Street\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"city\", \"field_value\":\"Winterville\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"state\", \"field_value\":\"WI\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"zip\", \"field_value\":\"36310\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"post\", \"field_value\":\"SW1A 1AA\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"area\", \"field_value\":\"555\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"phone\", \"field_value\":\"555-555-0199\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"tel\", \"field_value\":\"555-555-0199\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"ssn\", \"field_value\":\"123 45 6789\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"social\", \"field_value\":\"123 45 6789\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"age\", \"field_value\":\"30\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"day\", \"field_value\":\"01\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"month\", \"field_value\":\"01\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"year\", \"field_value\":\"1980\", \"match_type\":\"regex\" }, { \"enabled\":true, \"field_name\":\"passport\", \"field_value\":\"0123456789\", \"match_type\":\"regex\" } ], \"set_unmatched_fields\":true }, \"passive_spidering\":{ \"link_depth_to_associate_with_proxy_requests\":0, \"passively_spider_as_you_browse\":true }, \"request_headers\":{ \"custom_headers\":[ \"Accept: */*\", \"Accept-Language: en\", \"User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)\", \"Connection: close\" ], \"use_http_11\":true, \"use_referer\":true }, \"scope\":{ \"advanced_mode\":false, \"exclude\":[], \"include\":[], \"scope_option\":\"suite\" } }, \"target\":{ \"filter\":{ \"by_annotation\":{ \"show_only_commented_items\":false, \"show_only_highlighted_items\":false }, \"by_file_extension\":{ \"hide_items\":[ \"js\", \"gif\", \"jpg\", \"png\", \"css\" ], \"hide_specific\":false, \"show_items\":[ \"asp\", \"aspx\", \"jsp\", \"php\" ], \"show_only_specific\":false }, \"by_folders\":{ \"hide_empty_folders\":true }, \"by_mime_type\":{ \"show_css\":false, \"show_flash\":true, \"show_html\":true, \"show_images\":false, \"show_other_binary\":false, \"show_other_text\":true, \"show_script\":true, \"show_xml\":true }, \"by_request_type\":{ \"hide_not_found_items\":true, \"show_only_in_scope_items\":false, \"show_only_parameterized_requests\":false, \"show_only_requested_items\":false }, \"by_search\":{ \"case_sensitive\":false, \"negative_search\":false, \"regex\":false, \"term\":\"\" }, \"by_status_code\":{ \"show_2xx\":true, \"show_3xx\":true, \"show_4xx\":false, \"show_5xx\":true } }, \"scope\":{ \"advanced_mode\":false, \"exclude\":[], \"include\": [ { \"enabled\":true, \"prefix\":\"https://testwebsitegoeshere.com\" } ] } }}"

Get the project configuration

curl -X GET "http://127.0.0.1:8090/burp/configuration" -H "accept: */*"

Observe the dropped included targets
{"project_options":{"connections":{"hostname_resolution":[],"out_of_scope_requests":{"advanced_mode":false,"drop_all_out_of_scope":false,"exclude":[],"include":[],"scope_option":"suite"},"platform_authentication":{"credentials":[],"do_platform_authentication":true,"prompt_on_authentication_failure":false,"use_user_options":true},"socks_proxy":{"dns_over_socks":false,"host":"","password":"","port":0,"use_proxy":false,"use_user_options":true,"username":""},"timeouts":{"domain_name_resolution_timeout":300000,"failed_domain_name_resolution_timeout":60000,"normal_timeout":120000,"open_ended_response_timeout":10000},"upstream_proxy":{"servers":[],"use_user_options":true}},"http":{"redirections":{"understand_3xx_status_code":true,"understand_any_status_code_with_location_header":false,"understand_javascript_driven":false,"understand_meta_refresh_tag":true,"understand_refresh_header":true},"status_100_responses":{"remove_100_continue_responses":false,"understand_100_continue_responses":true},"streaming_responses":{"scope_advanced_mode":false,"store":true,"strip_chunked_encoding_metadata":true,"urls":[]}},"misc":{"collaborator_server":{"location":"","poll_over_unencrypted_http":false,"polling_location":"","type":"default"},"logging":{"requests":{"all_tools":"","extender":"","intruder":"","proxy":"","repeater":"","scanner":"","sequencer":""},"responses":{"all_tools":"","extender":"","intruder":"","proxy":"","repeater":"","scanner":"","sequencer":""}},"scheduled_tasks":{"tasks":[]}},"sessions":{"cookie_jar":{"monitor_extender":false,"monitor_intruder":false,"monitor_proxy":true,"monitor_repeater":false,"monitor_scanner":false,"monitor_sequencer":false},"macros":{"macros":[]},"session_handling_rules":{"rules":[{"actions":[{"enabled":true,"match_cookies":"all_except","type":"use_cookies"}],"description":"Use cookies from Burp's cookie jar","enabled":true,"exclude_from_scope":[],"include_in_scope":[],"named_params":[],"restrict_scope_to_named_params":false,"tools_scope":["Scanner"],"url_scope":"all","url_scope_advanced_mode":false}]}},"ssl":{"client_certificates":{"certificates":[],"use_user_options":true},"negotiation":{"allow_unsafe_renegotiation":false,"automatically_select_compatible_ssl_parameters_on_failure":true,"disable_ssl_session_resume":false,"enabled_ciphers":[],"enabled_protocols":[],"use_platform_default_protocols_and_ciphers":true}}},"proxy":{"http_history_display_filter":{"by_annotation":{"show_only_commented_items":false,"show_only_highlighted_items":false},"by_file_extension":{"hide_items":["js","gif","jpg","png","css"],"hide_specific":false,"show_items":["asp","aspx","jsp","php"],"show_only_specific":false},"by_listener":{"port":""},"by_mime_type":{"show_css":false,"show_flash":true,"show_html":true,"show_images":false,"show_other_binary":false,"show_other_text":true,"show_script":true,"show_xml":true},"by_request_type":{"hide_items_without_responses":false,"show_only_in_scope_items":false,"show_only_parameterized_requests":false},"by_search":{"case_sensitive":false,"negative_search":false,"regex":false,"term":""},"by_status_code":{"show_2xx":true,"show_3xx":true,"show_4xx":true,"show_5xx":true}},"intercept_client_requests":{"automatically_fix_missing_or_superfluous_new_lines_at_end_of_request":false,"automatically_update_content_length_header_when_the_request_is_edited":true,"do_intercept":false,"rules":[{"boolean_operator":"and","enabled":true,"match_condition":"(^gif$|^jpg$|^png$|^css$|^js$|^ico$)","match_relationship":"does_not_match","match_type":"file_extension"},{"boolean_operator":"or","enabled":false,"match_relationship":"contains_parameters","match_type":"request"},{"boolean_operator":"or","enabled":false,"match_condition":"(get|post)","match_relationship":"does_not_match","match_type":"http_method"},{"boolean_operator":"and","enabled":false,"match_relationship":"is_in_target_scope","match_type":"url"}]},"intercept_server_responses":{"automatically_update_content_length_header_when_the_response_is_edited":true,"do_intercept":false,"rules":[{"boolean_operator":"or","enabled":true,"match_condition":"text","match_relationship":"matches","match_type":"content_type_header"},{"boolean_operator":"or","enabled":false,"match_relationship":"was_modified","match_type":"request"},{"boolean_operator":"or","enabled":false,"match_relationship":"was_intercepted","match_type":"request"},{"boolean_operator":"and","enabled":false,"match_condition":"^304$","match_relationship":"does_not_match","match_type":"status_code"},{"boolean_operator":"and","enabled":false,"match_relationship":"is_in_target_scope","match_type":"url"}]},"intercept_web_sockets_messages":{"client_to_server_messages":true,"server_to_client_messages":true},"match_replace_rules":[{"comment":"Emulate IE","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^User-Agent.*$","string_replace":"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"},{"comment":"Emulate iOS","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^User-Agent.*$","string_replace":"User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B176 Safari/7534.48.3"},{"comment":"Emulate Android","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^User-Agent.*$","string_replace":"User-Agent: Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"},{"comment":"Require non-cached response","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^If-Modified-Since.*$"},{"comment":"Require non-cached response","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^If-None-Match.*$"},{"comment":"Hide Referer header","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^Referer.*$"},{"comment":"Require non-compressed responses","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^Accept-Encoding.*$"},{"comment":"Ignore cookies","enabled":false,"is_simple_match":false,"rule_type":"response_header","string_match":"^Set-Cookie.*$"},{"comment":"Rewrite Host header","enabled":false,"is_simple_match":false,"rule_type":"request_header","string_match":"^Host: foo.example.org$","string_replace":"Host: bar.example.org"},{"comment":"Add spoofed CORS origin","enabled":false,"is_simple_match":true,"rule_type":"request_header","string_replace":"Origin: foo.example.org"},{"comment":"Remove HSTS headers","enabled":false,"is_simple_match":false,"rule_type":"response_header","string_match":"^Strict\\-Transport\\-Security.*$"},{"comment":"Disable browser XSS protection","enabled":false,"is_simple_match":true,"rule_type":"response_header","string_replace":"X-XSS-Protection: 0"}],"miscellaneous":{"disable_logging_to_history_and_site_map":false,"disable_out_of_scope_logging_to_history_and_site_map":false,"disable_web_interface":false,"remove_unsupported_encodings_from_accept_encoding_headers_in_incoming_requests":true,"set_connection_close_header_on_requests":true,"set_connection_close_header_on_responses":false,"strip_proxy_headers_in_incoming_requests":true,"strip_sec_websocket_extensions_headers_in_incoming_requests":true,"suppress_burp_error_messages_in_browser":false,"unpack_gzip_deflate_in_requests":false,"unpack_gzip_deflate_in_responses":true,"use_http_10_in_requests_to_server":false,"use_http_10_in_responses_to_client":false},"request_listeners":[{"certificate_mode":"per_host","listen_mode":"loopback_only","listener_port":8080,"running":true}],"response_modification":{"convert_https_links_to_http":false,"enable_disabled_form_fields":false,"highlight_unhidden_fields":false,"remove_all_javascript":false,"remove_input_field_length_limits":false,"remove_javascript_form_validation":false,"remove_object_tags":false,"remove_secure_flag_from_cookies":false,"unhide_hidden_form_fields":false},"ssl_pass_through":{"automatically_add_entries_on_client_ssl_negotiation_failure":false,"rules":[]},"web_sockets_history_display_filter":{"by_annotation":{"show_only_commented_items":false,"show_only_highlighted_items":false},"by_listener":{"listener_port":""},"by_request_type":{"hide_incoming_messages":false,"hide_outgoing_messages":false,"show_only_in_scope_items":false},"by_search":{"case_sensitive":false,"negative_search":false,"regex":false,"term":""}}},"repeater":{"follow_redirections":"never","process_cookies_in_redirections":false,"unpack_gzip_deflate":true,"update_content_length":true},"sequencer":{"live_capture":{"ignore_abnormal_length_tokens":true,"max_length_deviation":5,"num_threads":5,"throttle":0},"token_analysis":{"compression":true,"correlation":true,"count":true,"fips_long_run":true,"fips_monobit":true,"fips_poker":true,"fips_runs":true,"spectral":true,"transitions":true},"token_handling":{"base_64_decode_before_analyzing":false,"pad_short_tokens_at":"start","pad_with":"0"}},"target":{"filter":{"by_annotation":{"show_only_commented_items":false,"show_only_highlighted_items":false},"by_file_extension":{"hide_items":["js","gif","jpg","png","css"],"hide_specific":false,"show_items":["asp","aspx","jsp","php"],"show_only_specific":false},"by_folders":{"hide_empty_folders":true},"by_mime_type":{"show_css":false,"show_flash":true,"show_html":true,"show_images":false,"show_other_binary":false,"show_other_text":true,"show_script":true,"show_xml":true},"by_request_type":{"hide_not_found_items":true,"show_only_in_scope_items":false,"show_only_parameterized_requests":false,"show_only_requested_items":false},"by_search":{"case_sensitive":false,"negative_search":false,"regex":false,"term":""},"by_status_code":{"show_2xx":true,"show_3xx":true,"show_4xx":false,"show_5xx":true}},"scope":{"advanced_mode":false,"exclude":[],"include":[]}}}

Expected behavior
note the very last field called "include" is empty when it should be https://testwebsitegoeshere.com
Desktop (please complete the following information):

OS: Windows
Burp Suite Professional version : v2 beta 13
Extension burp-rest-api version: v2.0.13beta

Additional context
The burp-rest-api jar outputs the following message to the stdout (when curl -X GET "http://127.0.0.1:8090/burp/configuration" -H "accept: */*"   is invoked) which might be the root cause?

I have just started to use the library, so it is entirely possible that I am doing something incorrect.
Adding targets manually works as expected, however, submitting the entire project configuration drops included/excluded targets; there could be more data loss that I am not aware of.

Just an update that I have been able to reproduce the above observed behaviour against BurpSuite pro v1.7.37.

Hey!
You should use PUT, instead of POST.
I've just tried using:

http://localhost:8090/swagger-ui.html#/burp-controller/getConfigurationUsingPOST
changed target.scope.include
http://localhost:8090/swagger-ui.html#/burp-controller/updateConfigurationUsingPUT
and works. I can see the new target url.

{
  "target": {
    "scope": {
      "include": [
        {
          "enabled": true,
          "prefix": "http://www.doyensec.com"
        }
      ]
    }
  }
}

Please note that POST /burp/configuration accepts partial configuration paths to retrieve the partial config, while PUT /burp/configuration requires the full configuration to save it. If a field is missing, it will revert to use the default configuration which has target.scope.include empty.
Feel free to re-open if you still have problems.
Thanks,
Luca

authors: ikkisoft robocrack

repo: vmware/burp-rest-api


text:
Comfortable spacing Theme

A11y have suggested that a theme with generous spacing and larger targets should be added to Clarity.

This is dependent on theming improvements we are planning for 3.0.

Meeting this requirement is necessary to sell in the EU public sector.  It must be implemented in our PRODUCTS before 9/23/20, which means Clarity really needs to have it by EOY Calendar 19
Success Criterion 1.4.12 Text Spacing
(Level AA)
In content implemented using markup languages that support the following text style properties, no loss of content or functionality occurs by setting all of the following and by changing no other style property:
·       Line height (line spacing) to at least 1.5 times the font size;
·       Spacing following paragraphs to at least 2 times the font size;
·       Letter spacing (tracking) to at least 0.12 times the font size;
·       Word spacing to at least 0.16 times the font size.
·
Target size is AAA – I can’t make you do it, but I really really want you to
Success Criterion 2.5.5 Target Size
(Level AAA)
The size of the target for pointer inputs is at least 44 by 44 CSS pixels except when:
Equivalent
The target is available through an equivalent link or control on the same page that is at least 44 by 44 CSS pixels;
Inline
The target is in a sentence or block of text;
User Agent Control
The size of the target is determined by the user agent and is not modified by the author;
Essential
A particular presentation of the target is essential to the information being conveyed.

(from Sheri)
Our recommendation is to provide the default compact solution that we are currently using with the option for the user to expand to the more spacious version upon request.  Preferably this would be integrated as a permanent login profile item, but I’ll take what I can get out of the gate.

Dev is blocked but design can proceed. Will also need to update design assets.

I'm going to close this for now until we have stronger guidance on what to do here and if we want to tackle it, as its not just a matter of adding padding to make a larger theme.

repo: vmware/clarity


text:
Errors when compiling

Describe the bug
I upgraded to Clarity version 2.0.2 and having some issues in compiling the project.

How to reproduce
Expected behavior
Should be able to compile without errors
Versions
App

Angular: 8.0.0
Node: 10+
Clarity: 2.0.2

Device:

Type: Dell XPS 13
OS: Windows 10 Enterprise Edition
Browser [e.g. chrome, safari]
Version [e.g. 22]

Additional notes
Add any other notes about the problem here.

It seems like when I do an npm install --save @clr/icons@latest @clr/angular@latest @clr/ui@latest downgrades the Clarity reference to 1.1.5.

The workaround is to install the packages with the specific version. I'm not sure if I installed the packages incorrectly or there might be an issue in Clarity's npm registry. I'll just let this ticket open and let you take a look on it. Thanks.

The latest tag got applied to the wrong version with our last release, I've put it back on 2.0.2. This should be resolved.


text:
Master Child capability for Clarity Datagrid

Select one ...  (check one with "x")
[ ] bug
[x ] feature request
[ ] enhancement

Expected behavior
As part of our usecases we have Datagrids where UI represents Master/Detail sort of record in Datagrid using Expand/Collapse option.
When master records are expanded usually we give backend call to fetch all the required information and add it as child record.
This sort of paradigm is used in multiple usecases in our existing UI and it is a priority item for us.
Attaching few references of UI screenshots:

@reddolan @jaffoneh  I'm guessing it's a bit early to ask the question, but roughly when should we expect for this to be done (not only designed, but also included in Clarity)?

@halachliyskid design has started on it. This generally takes a couple weeks. @jaffoneh can give you a better estimate with development included

@halachliyskid and @reddolan : It is still a ways off, development-wise. My understanding is the datagrid will need to be refactored to accommodate master-child views. After design is completed, we are still looking at a non-trivial implementation effort. It won't be constructive to throw out dates at this point in time because the feature is not fully defined and the fluidity of our priorities oftentimes can negate SWAG estimates in real time.
TL;DR It's going to be a while. End of year while? Probably not. But we're probably looking at months away.

Thanks for the quick reply guys!

You're welcome, @halachliyskid. Re-reading my reply, I see I may have not used the best wording for where we have this item prioritized. This is one of our 3-4 top priorities. So when I said a "while", I'm afraid it came off as sounding like a long time off. What is more accurate to say is that this is one of the things we are trying to get into either 0.9 or the minor version after it (0.10).
We are currently on 0.8. I will try not to be so vague in the future.

We have merged your request into Grid Row Detail #547. Please read the documentation and subscribe for updates. Comments and questions can be posted there.
Thank you

authors: halachliyskid mathisscott reddolan vkpavan26


text:
Test should accept "Driver type" as an input to reuse existing E2E automated tests

existing test target should be enhanced to accept "driver type" as an user input.
The benefit of doing this to leverage existing test to extend test coverage for "shared" plugin.
e.g. docker volume create -d <driver_name> -name=abc
Currently driver name is controlled through test constants, it should be configurable through command line.
current code: https://github.com/vmware/docker-volume-vsphere/blob/master/tests/constants/dockercli/cmd.go#L46
	// VDVSPluginName name of vDVS plugin
	VDVSPluginName = "vsphere "

/CC @luomiao @tusharnt @shivanshu21

Need to explore if this is needed.

adding mustfix kw to finish investigation.

Closed by https://github.com/vmware/docker-volume-vsphere/pull/1886

repo: vmware/docker-volume-vsphere