text:
defineConfig type not check non-exist keys with UserConfigFnObject

Describe the bug
When I use UserConfigFnObject type, it not check the non-exist key. As following picture shows:

On the other hand, when directly using UserConfig it works fine. Show below:

This is a typescript related issue.
Reproduction
https://stackblitz.com/edit/vitejs-vite-xbs979?file=vite.config.js
Steps to reproduce
Run with vscode and see if shows errors
System Info
System:
    OS: macOS 12.4
    CPU: (4) x64 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
    Memory: 653.14 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.20.0 - ~/.nvm/versions/node/v16.20.0/bin/node
    Yarn: 1.22.19 - ~/.nvm/versions/node/v16.20.0/bin/yarn
    npm: 8.19.4 - ~/.nvm/versions/node/v16.20.0/bin/npm
    pnpm: 8.2.0 - ~/.nvm/versions/node/v16.20.0/bin/pnpm
  Browsers:
    Chrome: 89.0.4389.90
    Safari: 15.1
  npmPackages:
    @vitejs/plugin-legacy: ^4.0.2 => 4.0.2
    @vitejs/plugin-vue: ^4.0.0 => 4.1.0
    vite: ^4.2.1 => 4.2.1

Used Package Manager
npm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

This behavior is TypeScript's "Excess Property Checks".
https://www.typescriptlang.org/docs/handbook/2/objects.html#excess-property-checks
I don't think we can fix this on Vite side in an ideal way.

Closing as this doesn't have much interest. Feel free to send a PR if it's possible without dirty hacks.

authors: mzvast sapphi-red


text:
Change JetBrains Logo on Website

Documentation is

[ ] Missing
[X] Outdated
[X] Confusing
[X] Not sure?

Explain in Detail
Hi there, this is not really a bug, we just noticed that the JetBrains logo is quite hard to identify on the WebSite so if possible we would like to update the logo. I am more than happy to file a pr, I was just not able to locate https://sponsors.vuejs.org/images/jetbrains.png in the codebase, so if you have any pointer for me I am more than happy to resolve this myself
Your Suggestion for Changes

Here is the logo we would like to use
Reproduction
No response
Steps to reproduce
No response

Updated! (also adjusted CSS to make it appropriately sized)

looks great, thanks a lot for taking care of it, very much appreciate it!

source: gharchive/issue

repo: vitejs/vite

license: MIT

license_type: permissive

license_source: github-api


text:
isn't working in a fresh vite project (dynamic component isn't working)

Describe the bug
dynamic component isn't working
 isn't working in a fresh vite project,
so in the html rendered page i get just a tag like so ;
empty tag and the component isn't rendered

<template>

      <HelloWorld msg="Hello Vue 3 + TypeScript + Vite" />

      <component is="Door" />
</template>

Reproduction
https://github.com/modex98/vite-issue
System Info
System:
    OS: Linux 5.13 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (4) x64 Intel(R) Core(TM) i3-6006U CPU @ 2.00GHz
    Memory: 812.91 MB / 3.72 GB
    Container: Yes
    Shell: 5.0.17 - /bin/bash
  Browsers:
    Chrome: 97.0.4692.99
    Firefox: 97.0.2
  npmPackages:
    @vitejs/plugin-vue: ^2.3.0 => 2.3.1
    vite: ^2.9.0 => 2.9.1

Used Package Manager
npm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

It should be <component :is="Door" />
See https://sfc.vuejs.org/#eyJBcHAudnVlIjoiPHNjcmlwdCBzZXR1cD5cbmltcG9ydCB7IHJlZiB9IGZyb20gJ3Z1ZSdcbmltcG9ydCBDb21wIGZyb20gJy4vQ29tcC52dWUnXG4gIFxuY29uc3QgbXNnID0gcmVmKCdIZWxsbyBXb3JsZCEnKVxuPC9zY3JpcHQ+XG5cbjx0ZW1wbGF0ZT5cbiAgPGgxPnt7IG1zZyB9fTwvaDE+XG4gIDxpbnB1dCB2LW1vZGVsPVwibXNnXCI+XG4gIDxjb21wb25lbnQgOmlzPVwiQ29tcFwiPjwvY29tcG9uZW50PlxuPC90ZW1wbGF0ZT4iLCJpbXBvcnQtbWFwLmpzb24iOiJ7XG4gIFwiaW1wb3J0c1wiOiB7XG4gICAgXCJ2dWVcIjogXCJodHRwczovL3NmYy52dWVqcy5vcmcvdnVlLnJ1bnRpbWUuZXNtLWJyb3dzZXIuanNcIlxuICB9XG59IiwiQ29tcC52dWUiOiI8dGVtcGxhdGU+XG4gIDxwPlxuICAgIEhleSFcbiAgPC9wPlxuPC90ZW1wbGF0ZT4ifQ==


text:
docs: Update deployment documentation for Vercel

Description
Updates the deployment docs for Vercel.
What is the purpose of this pull request?

[ ] Bug fix
[ ] New Feature
[x] Documentation update
[ ] Other

Before submitting the PR, please make sure you do the following

[x] Read the Contributing Guidelines.
[x] Read the Pull Request Guidelines and follow the Commit Convention.
[x] Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
[x] Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
[ ] Ideally, include relevant tests that fail without this PR but pass with it.

Thanks for the PR @leerob!

source: gharchive/pull-request

authors: leerob patak-dev


text:
feat: add configureBuild hook

This is another approach to the proposed configureBuild hook. (See #870 for the first approach)
In this PR, the configureBuild hooks are called before anything is done by the build function (except initializing the build config). When initializing the build config, it gets deep-cloned and filled with default values, so that configureBuild hooks can more easily inspect & mutate it (without checking for undefined everywhere).
Whereas #870 only exposes the Rollup input options to configureBuild, this PR exposes the entire Vite build config, which allows Vite plugins to better manipulate the build process as needed. This means more powerful Vite plugins will be possible.

I rebased this on top of the latest master - can you apply the changes in #878 to this branch instead?

Done 👍


text:
Add doNotFake for vi.useFakeTimers

Clear and concise description of the problem
Currently, jest useFakeTimers has the option to pass { doNotFake: <SomeTimerFacet>} to prevent fake timers from faking specific apis/functions like setTimeOut etc.
vitest provides the toFake option where you can explicitly decide which apis and functions to mock, but for things like usage with msw v2 where we cannot fake queueMicrotask, but would like to fake everything else, this means I have to explicitly specify every other FakeMethod in toFake when using vi.useFakeTimers.
Suggested solution
Implement the doNotFake option so one can simply write vi.useFakeTimers({ doNotFake: queueMicrotask }) and get the desired result of faking every other FakeMethod
Alternative
Currently we just create a const with every other FakeMethod and pass it to toFake. This is a fine workaround, but not a satisfactory one as it's a large list.
Even just exporting the FakeMethod type would be a great help.
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

In hindsight, this may not apply to vitest at all.
Please close if irrelevant.

Vitest exposes fakeTimersDefaults from vitest/config. In the future, we won't specify toFake values, they will be implied from the environment: https://github.com/vitest-dev/vitest/pull/6288 Then it will have doNotFake option.

authors: ailandini-accenture sheremet-va

repo: vitest-dev/vitest


text:
-t not matching full name

Describe the bug
Jest defines the test name as: "the full name, which is a combination of the test name and all its surrounding describe blocks". Vite only uses the the test name. This results in problems e.g. when using VSCode extension https://github.com/firsttris/vscode-jest-runner
Reproduction
Use the standard example and run vitest -t "suite name foo". It would work with Jest.
System Info
System:
    OS: macOS 12.2.1
    CPU: (8) arm64 Apple M1
    Memory: 92.28 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 17.4.0 - /opt/homebrew/bin/node
    Yarn: 1.22.17 - /opt/homebrew/bin/yarn
    npm: 8.4.1 - /opt/homebrew/bin/npm
  Browsers:
    Edge: 97.0.1072.69
    Firefox: 97.0.1
    Safari: 15.3
    Safari Technology Preview: 15.4

Used Package Manager
pnpm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

It works! Thanks so much @patak-dev that was the last brick missing eliminating Jest completely from some of my projects ;)


text:
Add support for custom WebdriverIO capabilities

The aim of this PR is to add support for passing custom browser capability arguments for WebdriverIO. This was done rather quickly so improvements and suggestions are welcome, I only saw tests for playwiright.

Pushed changed in the spirit of the requested modifications 👍

Will take another look in the coming two weeks


text:
After scrolling, when clicking a row the page scrolls up

Hi guys, i have this problem and cant solve it.
I have a lot of information in the grid, when I open the rows and scroll down and try to click the last rows, the page scrolls up.
This only happens if I use the left click, since with the right click it works normally expanding the rows correctly. Any idea what it might be? Thank you!

Resolved, the grid had a mousedown event that jumped the page up, calling the function mousestart. What I did was the following:
            $("#grid").on('mousedown', function(event){
		    	event.preventDefault();
            });

I know it's silly but I find it very difficult to find the event that made the page jump up. I hope if someone ever happens to you, stop by and do not have to set fire to your brain like I hahaha!!

license: mit

license_source: bigquery


text:
Security: Cross-site Scripting (XSS) Attack

w2ui grid control is subject of XSS attack. "records" fields values are insert into HTML as is, without proper escaping.
To reproduce try http://w2ui.com/web/demos/#!grid/grid-21 demo, and insert as a value following text:
<script>alert(123)</script>

If I understand correctly, the main remedy for XSS attacks is to sanitize user input when it is saved on the server. The security issue is not when user can execute code on the local machine (which you can do any time opening console and executing js), but when a js script is saved into db and then executed on another users machine, allowing hacker to still the session.
I will evaluate what can be done to help developer deal with this issues.

The check should be server-sided before data from the grid saved to the the db. It makes no sense doing this client-sided. At most, in addition to server-sided check.

You absolutely right in case of HTML generation on server-side. But imagine if you have generic REST service returning JSON. Such service should return original data as is without any assumption where and how data will be used, so it can't do encoding ahead (on server side).
That is actually my case - I have static HTML, it has w2ui grid which consumes the service. The original "reproduce" scenario in this issue report was just to easily show the general problem.
In my case an user is not able to enter any data by himself, but only see what REST service returned. The user should be able to see and search (with proper highlighting!) whatever data returned from the server (even if data contains tags, scripts or whatever). But at the same time the data from server should not be "executable" in any sense - only visible, searchable and (if needed) editable. So, I don't see how this could be achievable without proper data encoding on client side.
My current workaround is specify "render" function for a column, and there I do something like this:
columns: [{ field: 'name', caption: 'Name', size: '35%', sortable: true,
    render: function (record) {
        return w2utils.encodeTags(record.name);
    }

It works (prevents data "execution"), but search highlighting completely goes wrong as soon as you have < or > symbols is your data. In such case entering "lt", ";" text in search breaks data visualization in the grid.

It is a common practice for render function (similar as in your example) to return HTML that should be inserted as HTML. It is much more common then inserting it with encoded tags. As such, I think it is proper behavior. If you want to display tags, then doing what you doing is the way to go and as you see, it is just with a few lines of code. As for highlight, you are right, there are some bugs with it and it needs to be fixed.
I do not see here a security issue. I am of the opinion that data needs to be sanitized when it is saved and it should be server side sanitation. I think it is logical to assume that whatever your server returns is  trusted data. If you assume that your server data is not trusted, then you have a bigger security concern that all the data (which might include sensitive), on the server is compromised and that need to be addressed.

Again, you absolutely right about sanitizing HTML rendering on server side, and trust to your server, but... In case of GENERIC service how can you do any sanitizing? Imagine a server has "Items" service, and an item has weird name like "<script..." - why not?! As generic service it should return data as is, and NOT as "<script..." or "script..." because it is generic and not intended only for HTML - may be I will consume the service in C# code.
Speaking about trusted server I would like to add that it's quite common to AJAX 3rd party public  data feeds instead of your own server, so in such case you can't trust to these data.
OK, generally I understood that it is not a BUG - it is a FEATURE of the grid :) My main point that I currently see in the grid strong coupling between rendering and business logic (like searching data for highlighting). With such coupling I don't surprise that "...there are some bugs..." - the w2marker() function is a "good" example. IMHO better choice if grid will provide facilities for clean separation of final rendering from original data and logic. In such case it's quite possible that rendering will be simpler, business logic without bugs and ready for any arbitrary data, and there will be no (potential) security issues.
BTW, I would like to thank you for the great library! I very like it, and I'm just trying to do constructive critics ;) Thanks!

Appreciate your comments, will definitely keep this in mind to come with some solution.

I also consider this as a major issue. It is not only about XSS attacks. This also makes it impossible to edit inline text starting with angle bracket. For example: <item.
This library is great and beautiful, but it blindly renders everything.

So for security reasons make sure you use row => w2utils.encodeTags(row.field) for every text field, and do not allow inline editing for text fields.


text:
Section our program

Закончил свою секцию

Модификатор есть, а блока нету) Забыл section-title

Да) Спасибо) Добавил)


text:
Add MediaBlock

Add combined media block, and remove other media blocks.
The MediaBlock itself can be used in composite blocks, the aspect ratio should be handled.
The StandaloneMediaBlock is the standalone solution with aspect ratio selection.

[ ] remove site implementation of YouTubeVideoBlock and DamVideoBlock and replace with comet blocks (https://github.com/vivid-planet/comet/pull/2223)

can you explain the sizes string?

can you explain the sizes string?

@nsams here it is explained in detail: https://nextjs.org/docs/pages/api-reference/components/image#sizes
we use it to load the images in the expected resolution.

closed due to next migration #296

For the next time: Rebasing onto next would also have been an option. Then the previous conversations wouldn't be lost.

I thought merging is the lesser evil, but that did not work out either, as you see the changes here. Maybe the merge main make too much conflicts.

authors: SebiVPS johnnyomair nsams

repo: vivid-planet/comet-starter


text:
Cannot read channel messages if channel is marked NSFW?

I could not get this working in a channel, so I started testing on a personal server and realized that this error I was getting was because the server channel I was trying to run it in was marked NSFW. I'm not 100% this is the issue but it seems like it. The error log below is what I get when I try and run it in an NSFW channel that has already been accepter by me manually. Otherwise in normal channels it works fine. What I want to know is there any possible fix someone could direct me to? In DisCum documentation I saw a includeNsfw (Optional[bool]) Parameter but I'm not very proficient with this so I don't really know if that's helpful. Thanks.

0|waifu  | Exception ignored in thread started by
0|waifu  | :
0|waifu  | <bound method GatewayServer._response_loop of <discum.gateway.gateway.GatewayServer object at 0x7f0dec7ab3d0>>
0|waifu  | Traceback (most recent call last):
0|waifu  |   File "/root/.local/lib/python3.9/site-packages/discum/gateway/gateway.py", line 299, in _response_loop
0|waifu  |     func(resp)
0|waifu  |   File "/root/waifu/MudaeAutoBot.py", line 763, in on_message
0|waifu  |     msg = get_server_settings(gid,matched_channel)
0|waifu  |   File "/root/waifu/MudaeAutoBot.py", line 180, in get_server_settings
0|waifu  |     FsResults = bot.filterSearchResults(FsMsgs)
0|waifu  |   File "/root/.local/lib/python3.9/site-packages/discum/discum.py", line 313, in filterSearchResults
0|waifu  |     return imports.Messages(self.discord, self.s, self.log).filterSearchResults(searchResponse)
0|waifu  |   File "/root/.local/lib/python3.9/site-packages/discum/messages/messages.py", line 245, in filterSearchResults
0|waifu  |     jsonresponse = searchResponse.json()['messages']
0|waifu  | KeyError: 'messages'

That just searching for settings but I can see if I can try to add it in Sent from my iPhoneOn Jan 21, 2023, at 4:55 AM, veetus1 @.***> wrote:
I could not get this working in a channel, so I started testing on a personal server and realized that this error I was getting was because the server channel I was trying to run it in was marked NSFW. I'm not 100% this is the issue but it seems like it. The error log below is what I get when I try and run it in an NSFW channel that has already been accepter by me manually. Otherwise in normal channels it works fine. What I want to know is there any possible fix someone could direct me to? In DisCum documentation I saw a includeNsfw (Optional[bool]) Parameter but I'm not very proficient with this so I don't really know if that's helpful. Thanks.
0|waifu  | Exception ignored in thread started by 0|waifu  | : 0|waifu  | <bound method GatewayServer._response_loop of <discum.gateway.gateway.GatewayServer object at 0x7f0dec7ab3d0>> 0|waifu  | Traceback (most recent call last): 0|waifu  |   File "/root/.local/lib/python3.9/site-packages/discum/gateway/gateway.py", line 299, in _response_loop 0|waifu  |     func(resp) 0|waifu  |   File "/root/waifu/MudaeAutoBot.py", line 763, in on_message 0|waifu  |     msg = get_server_settings(gid,matched_channel) 0|waifu  |   File "/root/waifu/MudaeAutoBot.py", line 180, in get_server_settings 0|waifu  |     FsResults = bot.filterSearchResults(FsMsgs) 0|waifu  |   File "/root/.local/lib/python3.9/site-packages/discum/discum.py", line 313, in filterSearchResults 0|waifu  |     return imports.Messages(self.discord, self.s, self.log).filterSearchResults(searchResponse) 0|waifu  |   File "/root/.local/lib/python3.9/site-packages/discum/messages/messages.py", line 245, in filterSearchResults 0|waifu  |     jsonresponse = searchResponse.json()['messages'] 0|waifu  | KeyError: 'messages'
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

Any luck with this? I was looking for a solution to this as well.

Ive added it in to the bot you guys can test it now and see if it works

repo: vivinano/MudaeAutoBot


text:
After 360 upload, image disappears & requires page reload to display

This has happened to me several times today, on 2 separate machines, in Chrome & Safari.

fixed

authors: kschzt mattatgit

repo: vizorvr/vizor


text:
Database abstraction?

Not sure if we want it, but take a brief look about database abstractions to support Postgres in addition to MariaDB.
My personal preference is PostgreSQL because of features and performance, but not forcing the project into it.

Opted to just go with sqlite to keep things minimal.

authors: vkoskiv zouppen


text:
Add lockById function to the BaseHibernateRepository

As explained in this article.

Fixed.

repo: vladmihalcea/hibernate-types


text:
BatchSequenceGenerator should use PostgreSQL generate_series to speed up ID generation

Replacing the recursive CTE
WITH RECURSIVE t(n) AS (
    SELECT 1
    UNION ALL
    SELECT n + 1
    FROM t
    WHERE n < ?)
    SELECT nextval('your_sequence') FROM t

with
SELECT nextval('your_sequence') FROM generate_series(1, ?)
halves execution time on PostgreSQL.
Implemented in https://github.com/vladmihalcea/hypersistence-utils/pull/612.
Not a big improvement, but a cheap change.

Fixed.

authors: eazrael vladmihalcea

repo: vladmihalcea/hypersistence-utils


text:
[feat] Delete the automatically generated binary file after running

v run calculator.v && rm -rf calculator

v run <whatever.v> automatically deletes the binary after it is run.  That was added a little while ago.

v run <whatever.v> automatically deletes the binary after it is run. That was added a little while ago.

where is it?

Get the latest version of V.

Closed. See https://github.com/vlang/vscode-vlang/pull/196#issuecomment-751408395.

authors: JalonSolov d1y nedpals

repo: vlang/vscode-vlang


text:
ESP266 file removal failure

Thanks for all the work @vlasovskikh on this project have just started using it and I'm impressed how easy to use it is.
I've encountered a bug while using the ESP8266MOD removing files with the Remove all files from the micropython device call it returns a Unknown Error removing file on call. I've tracked down the issue to the microcleanfs.py script specifically here:
line 51: for name in files.ls():
 ...

at the moment this returns a verbose output of the file including the file in same string e.g. ['./main.py file size 443 bytes'] leading to the error above.
Changing this too.
for name in files.ls((long_format=False)
fixed it for me. Thoughts into putting into production?

The fix is in master. I'll release the plugin for PyCharm / IntelliJ 2018.3 shortly.

repo: vlasovskikh/intellij-micropython


text:
Honor passed seq

This keeps the original seq from the insert in crsql_changes. Gaps are allowed.

weird that it doesn't run the whole set of test suites for your pr 🤔 Let me see if I can kick those off.

ah, it ran them in your repo since it runs on commit not on pr -- https://github.com/jeromegn/cr-sqlite/actions. Will change to run on both actions

working on fixing some tests based on your changes here: https://github.com/vlcn-io/cr-sqlite/compare/main...honor-seq

authors: jeromegn tantaman

repo: vlcn-io/cr-sqlite


text:
ConnectionResetError: [Errno 104] Connection reset by peer

Occasionally encounter errors
+ python3 -m vllm.entrypoints.openai.api_server --host xxxxx --port 8003 --served-model-name qwen1.5-72b-chat-int4 --model /home/vllm/model/Qwen1.5-72B-Chat-GPTQ-Int4 --trust-remote-code --tokenizer-mode auto --max-num-batched-tokens 32768 --tensor-parallel-size 4
INFO 02-29 14:28:09 api_server.py:228] args: Namespace(host='xxxxx', port=8003, allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, served_model_name='qwen1.5-72b-chat-int4', lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, root_path=None, middleware=[], model='/home/vllm/model/Qwen1.5-72B-Chat-GPTQ-Int4', tokenizer=None, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='auto', dtype='auto', kv_cache_dtype='auto', max_model_len=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, block_size=16, seed=0, swap_space=4, gpu_memory_utilization=0.9, max_num_batched_tokens=32768, max_num_seqs=256, max_paddings=256, disable_log_stats=False, quantization=None, enforce_eager=False, max_context_len_to_capture=8192, disable_custom_all_reduce=False, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', max_cpu_loras=None, device='auto', engine_use_ray=False, disable_log_requests=False, max_log_len=None)
WARNING 02-29 14:28:09 config.py:186] gptq quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 02-29 14:28:09 config.py:421] Custom all-reduce kernels are temporarily disabled due to stability issues. We will re-enable them once the issues are resolved.
2024-02-29 14:28:12,795 INFO worker.py:1724 -- Started a local Ray instance.
INFO 02-29 14:28:15 llm_engine.py:87] Initializing an LLM engine with config: model='/home/vllm/model/Qwen1.5-72B-Chat-GPTQ-Int4', tokenizer='/home/vllm/model/Qwen1.5-72B-Chat-GPTQ-Int4', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=4, disable_custom_all_reduce=True, quantization=gptq, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, seed=0)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/vllm/entrypoints/openai/api_server.py", line 236, in <module>
    engine = AsyncLLMEngine.from_engine_args(engine_args)
  File "/workspace/vllm/engine/async_llm_engine.py", line 625, in from_engine_args
    engine = cls(parallel_config.worker_use_ray,
  File "/workspace/vllm/engine/async_llm_engine.py", line 321, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/workspace/vllm/engine/async_llm_engine.py", line 366, in _init_engine
    return engine_class(*args, **kwargs)
  File "/workspace/vllm/engine/llm_engine.py", line 126, in __init__
    self._init_workers_ray(placement_group)
  File "/workspace/vllm/engine/llm_engine.py", line 303, in _init_workers_ray
    self._run_workers("init_model",
  File "/workspace/vllm/engine/llm_engine.py", line 1036, in _run_workers
    driver_worker_output = getattr(self.driver_worker,
  File "/workspace/vllm/worker/worker.py", line 94, in init_model
    init_distributed_environment(self.parallel_config, self.rank,
  File "/workspace/vllm/worker/worker.py", line 275, in init_distributed_environment
    cupy_utils.init_process_group(
  File "/workspace/vllm/model_executor/parallel_utils/cupy_utils.py", line 90, in init_process_group
    _NCCL_BACKEND = NCCLBackendWithBFloat16(world_size, rank, host, port)
  File "/usr/local/lib/python3.10/dist-packages/cupyx/distributed/_nccl_comm.py", line 70, in __init__
    self._init_with_tcp_store(n_devices, rank, host, port)
  File "/usr/local/lib/python3.10/dist-packages/cupyx/distributed/_nccl_comm.py", line 94, in _init_with_tcp_store
    self._store_proxy['nccl_id'] = shifted_nccl_id
  File "/usr/local/lib/python3.10/dist-packages/cupyx/distributed/_store.py", line 148, in __setitem__
    self._send_recv(_store_actions.Set(key, value))
  File "/usr/local/lib/python3.10/dist-packages/cupyx/distributed/_store.py", line 130, in _send_recv
    result_bytes = s.recv(sizeof(
ConnectionResetError: [Errno 104] Connection reset by peer

Hi,
Do you solve this problem? I encounter the same problem too.

Hi,
Do you solve this problem? I encounter the same problem too.

This issue hasn't been resolved, I encounter it occasionally.

I think there's an issue with cupy backend that's used for tensor parallelism.
If you use enforce_eager=True, it is likely resolved (though it will affect performance). Regarding error itself, I think https://github.com/cupy/cupy is probably a better place to report an error.

I think there's an issue with cupy backend that's used for tensor parallelism.
If you use enforce_eager=True, it is likely resolved (though it will affect performance). Regarding error itself, I think https://github.com/cupy/cupy is probably a better place to report an error.

Hi,
Thanks for your suggestion. I will report on cupy. However, since the vllm must run successfully somehow in multi-node environment before release while I fail to deploy every time, I think there must be something that is related to environment hinder the deployment.

I think the cupy backend has been introduced lately for cuda graph (which is disabled by force_eager=True). My guess is this backend is not working well in some environments, but it is pretty difficult to troubleshoot without reproducing the issue. If you can tell me your instance details, I can try repro

repo: vllm-project/vllm


text:
Docker Issue

dockerfile:
FROM nvcr.io/nvidia/pytorch:22.12-py3
RUN pip uninstall torch -y
RUN pip install vllm
RUN pip install pydantic==1.10.11
RUN pip install fschat

The native api server was running fine --
python -m vllm.entrypoints.api_server \
--model stabilityai/stablelm-tuned-alpha-7b \
--tensor-parallel-size 2 \
--host 0.0.0.0

Openapi server comes with this issue --
python -m vllm.entrypoints.openai.api_server \
--model stabilityai/stablelm-tuned-alpha-7b \
--tensor-parallel-size 2 \
--host 0.0.0.0

transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so error ^

I believe this is an issue introduced by the accelerate package as a dependency of fastchat. Can you double-check your trasnformer/accelerate installation?

@zhuohan123
i didnt mod any transformer/accelerate installation.
By the way even without
RUN pip install fschat

Its causing the same problem with transformer_engine_extensions.cpython-38-x86_64-linux-gnu.so

when I use the api to do inference it raises an error: "openai.error.APIConnectionError: Error communicating with OpenAI: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /v1/completions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe7d8208df0>: Failed to establish a new connection: [Errno 111] Connection refused'))"
Does anyone have the solution for this bug ? Thank you so much

@doudoujay did you solve the issue?

@trannhatquy have you found any solution?

@trannhatquy have you found any solution?

@Doohae you should add -p 8888:1950 when build docker

Thanks sincerely, I'll try this!
What does exactly happen if I add -p 8888:1950?


text:
[Bug]: can not clean up the memory usage after instantiating the LLM class.

Your current environment
created a new conda environment and run
pip install vllm==0.4.2
> python collect_env.py

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Fedora Linux 37 (Workstation Edition) (x86_64)
GCC version: (GCC) 12.3.1 20230508 (Red Hat 12.3.1-1)
Clang version: Could not collect
CMake version: version 3.27.7
Libc version: glibc-2.36

Python version: 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:49:32)  [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-6.5.12-100.fc37.x86_64-x86_64-with-glibc2.36
Is CUDA available: N/A
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU
Nvidia driver version: 545.29.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      39 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             20
On-line CPU(s) list:                0-19
Vendor ID:                          GenuineIntel
Model name:                         12th Gen Intel(R) Core(TM) i9-12900HK
CPU family:                         6
Model:                              154
Thread(s) per core:                 2
Core(s) per socket:                 14
Socket(s):                          1
Stepping:                           3
CPU(s) scaling MHz:                 45%
CPU max MHz:                        5000.0000
CPU min MHz:                        400.0000
BogoMIPS:                           5836.80
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          544 KiB (14 instances)
L1i cache:                          704 KiB (14 instances)
L2 cache:                           11.5 MiB (8 instances)
L3 cache:                           24 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-19
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] No relevant packages
[conda] No relevant packagesROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-19	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug
Running f() for the second time raises GPU OOM.

def f():
	import vllm
	llm = vllm.LLM(model="facebook/opt-125m")

def cleanup():
	from vllm.distributed import destroy_model_parallel
	import gc
	import torch
	import ray
	import contextlib
	destroy_model_parallel()
	with contextlib.suppress(AssertionError):
	    torch.distributed.destroy_process_group()
	gc.collect()
	torch.cuda.empty_cache()
	ray.shutdown()

f()
cleanup()
f()

/vllm/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 05-29 10:26:04 llm_engine.py:100] Initializing an LLM engine (v0.4.2) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=facebook/opt-125m)
INFO 05-29 10:26:04 utils.py:660] Found nccl from library /home/alipanahi/.config/vllm/nccl/cu12/libnccl.so.2.18.1
INFO 05-29 10:26:04 selector.py:81] Cannot use FlashAttention-2 backend because the flash_attn package is not found. Please install it for better performance.
INFO 05-29 10:26:04 selector.py:32] Using XFormers backend.
INFO 05-29 10:26:05 weight_utils.py:199] Using model weights format ['*.bin']
INFO 05-29 10:26:05 model_runner.py:175] Loading model weights took 0.2389 GB
INFO 05-29 10:26:06 gpu_executor.py:114] # GPU blocks: 4700, # CPU blocks: 7281
INFO 05-29 10:26:07 model_runner.py:937] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 05-29 10:26:07 model_runner.py:941] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 05-29 10:26:09 model_runner.py:1017] Graph capturing finished in 2 secs.
/vllm/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO 05-29 10:26:09 llm_engine.py:100] Initializing an LLM engine (v0.4.2) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=facebook/opt-125m)
INFO 05-29 10:26:10 weight_utils.py:199] Using model weights format ['*.bin']
INFO 05-29 10:26:10 model_runner.py:175] Loading model weights took 0.2424 GB
---------------------------------------------------------------------------
OutOfMemoryError                          Traceback (most recent call last)
Cell In[1], line 20
     18 f()
     19 cleanup()
---> 20 f()

Cell In[1], line 3, in f()
      1 def f():
      2 	import vllm
----> 3 	llm = vllm.LLM(model="facebook/opt-125m")

File /vllm/lib/python3.9/site-packages/vllm/entrypoints/llm.py:123, in LLM.__init__(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, enforce_eager, max_context_len_to_capture, max_seq_len_to_capture, disable_custom_all_reduce, **kwargs)
    102     kwargs["disable_log_stats"] = True
    103 engine_args = EngineArgs(
    104     model=model,
    105     tokenizer=tokenizer,
   (...)
    121     **kwargs,
    122 )
--> 123 self.llm_engine = LLMEngine.from_engine_args(
    124     engine_args, usage_context=UsageContext.LLM_CLASS)
    125 self.request_counter = Counter()

File /vllm/lib/python3.9/site-packages/vllm/engine/llm_engine.py:292, in LLMEngine.from_engine_args(cls, engine_args, usage_context)
    289     executor_class = GPUExecutor
    291 # Create the LLM engine.
--> 292 engine = cls(
    293     **engine_config.to_dict(),
    294     executor_class=executor_class,
    295     log_stats=not engine_args.disable_log_stats,
    296     usage_context=usage_context,
    297 )
    298 return engine

File /vllm/lib/python3.9/site-packages/vllm/engine/llm_engine.py:172, in LLMEngine.__init__(self, model_config, cache_config, parallel_config, scheduler_config, device_config, load_config, lora_config, vision_language_config, speculative_config, decoding_config, executor_class, log_stats, usage_context)
    157 self.generation_config_fields = _load_generation_config_dict(
    158     model_config)
    160 self.model_executor = executor_class(
    161     model_config=model_config,
    162     cache_config=cache_config,
   (...)
    169     load_config=load_config,
    170 )
--> 172 self._initialize_kv_caches()
    174 # If usage stat is enabled, collect relevant info.
    175 if is_usage_stats_enabled():

File /vllm/lib/python3.9/site-packages/vllm/engine/llm_engine.py:249, in LLMEngine._initialize_kv_caches(self)
    242 def _initialize_kv_caches(self) -> None:
    243     """Initialize the KV cache in the worker(s).
    244
    245     The workers will determine the number of blocks in both the GPU cache
    246     and the swap CPU cache.
    247     """
    248     num_gpu_blocks, num_cpu_blocks = (
--> 249         self.model_executor.determine_num_available_blocks())
    251     if self.cache_config.num_gpu_blocks_override is not None:
    252         num_gpu_blocks_override = self.cache_config.num_gpu_blocks_override

File /vllm/lib/python3.9/site-packages/vllm/executor/gpu_executor.py:106, in GPUExecutor.determine_num_available_blocks(self)
    102 def determine_num_available_blocks(self) -> Tuple[int, int]:
    103     """Determine the number of available KV blocks by invoking the
    104     underlying worker.
    105     """
--> 106     return self.driver_worker.determine_num_available_blocks()

File /vllm/lib/python3.9/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/vllm/worker/worker.py:139, in Worker.determine_num_available_blocks(self)
    135 torch.cuda.empty_cache()
    137 # Execute a forward pass with dummy inputs to profile the memory usage
    138 # of the model.
--> 139 self.model_runner.profile_run()
    141 # Calculate the number of blocks that can be allocated with the
    142 # profiled peak memory.
    143 torch.cuda.synchronize()

File /vllm/lib/python3.9/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/vllm/worker/model_runner.py:888, in ModelRunner.profile_run(self)
    886 num_layers = self.model_config.get_num_layers(self.parallel_config)
    887 kv_caches = [None] * num_layers
--> 888 self.execute_model(seqs, kv_caches)
    889 torch.cuda.synchronize()
    890 return

File /vllm/lib/python3.9/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/vllm/worker/model_runner.py:808, in ModelRunner.execute_model(self, seq_group_metadata_list, kv_caches)
    806 if self.vision_language_config:
    807     execute_model_kwargs.update({"image_input": multi_modal_input})
--> 808 hidden_states = model_executable(**execute_model_kwargs)
    810 # Compute the logits.
    811 logits = self.model.compute_logits(hidden_states, sampling_metadata)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /vllm/lib/python3.9/site-packages/vllm/model_executor/models/opt.py:299, in OPTForCausalLM.forward(self, input_ids, positions, kv_caches, attn_metadata)
    292 def forward(
    293     self,
    294     input_ids: torch.Tensor,
   (...)
    297     attn_metadata: AttentionMetadata,
    298 ) -> torch.Tensor:
--> 299     hidden_states = self.model(input_ids, positions, kv_caches,
    300                                attn_metadata)
    301     return hidden_states

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /vllm/lib/python3.9/site-packages/vllm/model_executor/models/opt.py:274, in OPTModel.forward(self, input_ids, positions, kv_caches, attn_metadata)
    267 def forward(
    268     self,
    269     input_ids: torch.Tensor,
   (...)
    272     attn_metadata: AttentionMetadata,
    273 ) -> torch.Tensor:
--> 274     return self.decoder(input_ids, positions, kv_caches, attn_metadata)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /vllm/lib/python3.9/site-packages/vllm/model_executor/models/opt.py:248, in OPTDecoder.forward(self, input_ids, positions, kv_caches, attn_metadata)
    246 for i in range(len(self.layers)):
    247     layer = self.layers[i]
--> 248     hidden_states = layer(hidden_states, kv_caches[i], attn_metadata)
    250 if self.final_layer_norm is not None:
    251     hidden_states = self.final_layer_norm(hidden_states)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /vllm/lib/python3.9/site-packages/vllm/model_executor/models/opt.py:170, in OPTDecoderLayer.forward(self, hidden_states, kv_cache, attn_metadata)
    168     hidden_states = self.final_layer_norm(hidden_states)
    169 hidden_states, _ = self.fc1(hidden_states)
--> 170 hidden_states = self.activation_fn(hidden_states)
    171 hidden_states, _ = self.fc2(hidden_states)
    172 hidden_states = residual + hidden_states

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /vllm/lib/python3.9/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /vllm/lib/python3.9/site-packages/torch/nn/modules/activation.py:103, in ReLU.forward(self, input)
    102 def forward(self, input: Tensor) -> Tensor:
--> 103     return F.relu(input, inplace=self.inplace)

File /vllm/lib/python3.9/site-packages/torch/nn/functional.py:1500, in relu(input, inplace)
   1498     result = torch.relu_(input)
   1499 else:
-> 1500     result = torch.relu(input)
   1501 return result

OutOfMemoryError: CUDA out of memory. Tried to allocate 12.00 MiB. GPU

Interesting issue. I've already tried reproducing it  using:
...
f()
for _ in range(100):
    cleanup()
    f()

I can't reproduce this issue. The LLM model is Llama-7B,  GPU is A800, and  vllm was recently built from source. Perhaps you can clone the 'main' branch and build from source to test if you can reproduce this issue again

+1 can not work

authors: c3-ali jeejeelee wellcasa


text:
[Bug]: RuntimeError: CHECK_EQ(paged_kv_indptr.size(0), batch_size + 1) failed. 1 vs 257. When load gemma-2-9b-it using vllm

Your current environment
Collecting environment information...
PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.31

Python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-117-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 535.183.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        46 bits physical, 48 bits virtual
CPU(s):                               72
On-line CPU(s) list:                  0-71
Thread(s) per core:                   2
Core(s) per socket:                   18
Socket(s):                            2
NUMA node(s):                         2
Vendor ID:                            GenuineIntel
CPU family:                           6
Model:                                85
Model name:                           Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
Stepping:                             7
CPU MHz:                              2200.000
CPU max MHz:                          3900.0000
CPU min MHz:                          1000.0000
BogoMIPS:                             4400.00
Virtualization:                       VT-x
L1d cache:                            1.1 MiB
L1i cache:                            1.1 MiB
L2 cache:                             36 MiB
L3 cache:                             49.5 MiB
NUMA node0 CPU(s):                    0-17,36-53
NUMA node1 CPU(s):                    18-35,54-71
Vulnerability Gather data sampling:   Mitigation; Microcode
Vulnerability Itlb multihit:          KVM: Mitigation: VMX disabled
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; TSX disabled
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] flashinfer==0.1.3+cu121torch2.3
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu11==2.20.5
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] onnxruntime==1.18.0
[pip3] optree==0.11.0
[pip3] pytorch_revgrad==0.2.0
[pip3] sentence-transformers==3.0.1
[pip3] torch==2.3.1
[pip3] torchaudio==2.3.1
[pip3] torchvision==0.18.1
[pip3] transformers==4.43.1
[pip3] transformers-stream-generator==0.0.5
[pip3] triton==2.3.1
[conda] flashinfer                0.1.3+cu121torch2.3          pypi_0    pypi
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-nccl-cu11          2.20.5                   pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] optree                    0.11.0                   pypi_0    pypi
[conda] pytorch-cuda              11.8                 h7e8668a_5    pytorch
[conda] pytorch-revgrad           0.2.0                    pypi_0    pypi
[conda] sentence-transformers     3.0.1                    pypi_0    pypi
[conda] torch                     2.3.1                    pypi_0    pypi
[conda] torchaudio                2.3.1                    pypi_0    pypi
[conda] torchvision               0.18.1                   pypi_0    pypi
[conda] transformers              4.43.1                   pypi_0    pypi
[conda] transformers-stream-generator 0.0.5                    pypi_0    pypi
[conda] triton                    2.3.1                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.3.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    SYS     SYS     0-17,36-53      0               N/A
GPU1    NODE     X      SYS     SYS     0-17,36-53      0               N/A
GPU2    SYS     SYS      X      NODE    18-35,54-71     1               N/A
GPU3    SYS     SYS     NODE     X      18-35,54-71     1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug
from langchain_community.llms import VLLM

llm = VLLM(
    model="/media/user/datadisk/LLM_models/ko-gemma-2-9b-it",
     # model received from huggingface with the git clone command
    trust_remote_code=True,
    max_new_tokens=4096,
    top_k=3,
    top_p=0.9,
    temperature=0.7,

)

Hi, I found some bugs when loading the gemma-2-9b fine tuning model using the vLLM library.
1. Please use Flashinfer backend for models with logits_soft_cap (i.e., Gemma-2). Otherwise, the output might be wrong. Set Flashinfer backend by export VLLM_ATTENTION_BACKEND=FLASHINFER. (type=value_error)

The above error was resolved by setting the environment variable as shown below.
import os
os.environ['VLLM_ATTENTION_BACKEND'] = 'FLASHINFER'

2. 'NoneType' object is not callable (type=type_error)

The above error was resolved by installing the flashinfer library, see #6445 .
Since then, I've encountered the following issues
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/user/anaconda3/envs/llm-api/bin/uvicorn", line 8, in <module>
[rank0]:     sys.exit(main())
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
[rank0]:     return self.main(*args, **kwargs)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/click/core.py", line 1078, in main
[rank0]:     rv = self.invoke(ctx)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
[rank0]:     return ctx.invoke(self.callback, **ctx.params)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/click/core.py", line 783, in invoke
[rank0]:     return __callback(*args, **kwargs)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/main.py", line 409, in main
[rank0]:     run(
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/main.py", line 575, in run
[rank0]:     server.run()
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run
[rank0]:     return asyncio.run(self.serve(sockets=sockets))
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/asyncio/runners.py", line 44, in run
[rank0]:     return loop.run_until_complete(main)
[rank0]:   File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve
[rank0]:     await self._serve(sockets)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve
[rank0]:     config.load()
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/config.py", line 433, in load
[rank0]:     self.loaded_app = import_from_string(self.app)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string
[rank0]:     module = importlib.import_module(module_str)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/importlib/__init__.py", line 126, in import_module
[rank0]:     return _bootstrap._gcd_import(name[level:], package, level)
[rank0]:   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
[rank0]:   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
[rank0]:   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
[rank0]:   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
[rank0]:   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
[rank0]:   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
[rank0]:   File "/home/user/Patent-LLM/main.py", line 6, in <module>
[rank0]:     from routers import chat_manager, clear_memory
[rank0]:   File "/home/user/Patent-LLM/routers/clear_memory.py", line 5, in <module>
[rank0]:     from core.get_session import get_user_id
[rank0]:   File "/home/user/Patent-LLM/core/get_session.py", line 8, in <module>
[rank0]:     from core.llm import llm
[rank0]:   File "/home/user/Patent-LLM/core/llm.py", line 16, in <module>
[rank0]:     llm = VLLM(
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/pydantic/v1/main.py", line 339, in __init__
[rank0]:     values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/pydantic/v1/main.py", line 1048, in validate_model
[rank0]:     input_data = validator(cls_, input_data)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/langchain_core/utils/pydantic.py", line 149, in wrapper
[rank0]:     return func(cls, values)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/langchain_community/llms/vllm.py", line 89, in validate_environment
[rank0]:     values["client"] = VLLModel(
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/entrypoints/llm.py", line 155, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/engine/llm_engine.py", line 441, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/engine/llm_engine.py", line 265, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/engine/llm_engine.py", line 364, in _initialize_kv_caches
[rank0]:     self.model_executor.determine_num_available_blocks())
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/executor/gpu_executor.py", line 94, in determine_num_available_blocks
[rank0]:     return self.driver_worker.determine_num_available_blocks()
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/worker/worker.py", line 179, in determine_num_available_blocks
[rank0]:     self.model_runner.profile_run()
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/worker/model_runner.py", line 896, in profile_run
[rank0]:     self.execute_model(model_input, kv_caches, intermediate_tensors)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/worker/model_runner.py", line 1292, in execute_model
[rank0]:     model_input.attn_metadata.begin_forward()
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/vllm/vllm/attention/backends/flashinfer.py", line 146, in begin_forward
[rank0]:     self.prefill_wrapper.begin_forward(
[rank0]:   File "/home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/flashinfer/prefill.py", line 791, in begin_forward
[rank0]:     self._wrapper.begin_forward(
[rank0]: RuntimeError: CHECK_EQ(paged_kv_indptr.size(0), batch_size + 1) failed. 1 vs 257

I've loaded and used many other models via vLLM libraries, but this is the first time I've encountered this, and I haven't found any documentation of the issue.
I tried (ignorantly) to force the size by setting the batch_size variable to 256 or 0 directly in the /home/user/anaconda3/envs/llm-api/lib/python3.10/site-packages/flashinfer/prefill.py source file, but it only changed the number on both sides of the vs.
Is there a way to fix this?

+1 Seeing the same thing.

Seems like a bug in flash infer 0.1.3
Managed to fix same issue with
pip install flashinfer==0.1.2 -i https://flashinfer.ai/whl/cu121/torch2.3

Seems like a bug in flash infer 0.1.3 Managed to fix same issue with pip install flashinfer==0.1.2 -i https://flashinfer.ai/whl/cu121/torch2.3

I plan to try it out tomorrow, and judging by the number of likes, it seems to be the solution! If it resolves the issue, I will close it.
Thank you 😄

It works!

Seems like a bug in flash infer 0.1.3 Managed to fix same issue with pip install flashinfer==0.1.2 -i https://flashinfer.ai/whl/cu121/torch2.3

I plan to try it out tomorrow, and judging by the number of likes, it seems to be the solution! If it resolves the issue, I will close it. Thank you 😄

Can we re-open it as this isn't really a solution for the Docker image right?

@Noxoomo this is not a bug with flashinfer:
We found that vllm v0.5.3 didn't integrate flashinfer correctly: https://github.com/flashinfer-ai/flashinfer/issues/362#issuecomment-2260213082
To avoid such issues, we add a check at flashinfer-side(https://github.com/flashinfer-ai/flashinfer/pull/413), which was reflected in flashinfer v0.1.3, and it will break vllm v0.5.3.
As you said, downgrade flashinfer to v0.1.2 is a temporary solution to be compatible with vllm v0.5.3. But you are encouraged to use new wheels once vllm integration was fixed.

@yzh119 is vllm integration fixed? I'm having a hella of a time running Gemma 2 with vllm

@RylanSchaeffer yes, it's fixed in main branch and you can either wait for next vllm release or install vllm from source in main branch.

authors: Noxoomo RylanSchaeffer kvikk seongjiko xfalcox yzh119


text:
[Bug]: Loading a model with bitsandbytes 8bit quantization

Your current environment
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Debian GNU/Linux 12 (bookworm) (x86_64)
GCC version: (Debian 12.2.0-14) 12.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.36
Python version: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.1.0-25-cloud-amd64-x86_64-with-glibc2.36
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB
Nvidia driver version: 530.30.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               12
On-line CPU(s) list:                  0-11
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   2
Core(s) per socket:                   6
Socket(s):                            1
Stepping:                             7
BogoMIPS:                             4400.41
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            192 KiB (6 instances)
L1i cache:                            192 KiB (6 instances)
L2 cache:                             6 MiB (6 instances)
L3 cache:                             38.5 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-11
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.68
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.44.2
[pip3] triton==3.0.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.6.68                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.44.2                   pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1.post2@9ba0817ff1eb514f51cc6de9cb8e16c98d6ee44f
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity
GPU0     X      0-11            N/A
Legend:
X    = Self
SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX  = Connection traversing at most a single PCIe bridge
NV#  = Connection traversing a bonded set of # NVLinks
Model Input Dumps
No response
🐛 Describe the bug
How can I load a model using bitsandbytes quantization in 8-bit format? I'm currently loading the model with the following code:
model_id = "path/to/model"
llm = LLM(model=model_id, dtype=torch.bfloat16, trust_remote_code=True,
quantization="bitsandbytes", load_format="bitsandbytes")

This loads the model in 4-bit format, but I can't figure out how to load it in 8-bit. What should I change to load the model in 8-bit instead?
Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

add quantization type for argument
eg:
llm = LLM(model="/path/to/model_bnb_int8",  tensor_parallel_size=1, quantization="bitsandbytes", load_format="bitsandbytes")

What is the error you see? It should be supported as of https://github.com/vllm-project/vllm/pull/7445

I think they are trying to do inflight quantisation. The docs only mention inflight quantisation in 4-bit.
The bitsandbytes tests also indicate that we only support 4-bit inflight quantisation.
Perhaps we need to expand load_format to have bitsandbytes-8-bit and bitsandbytes-4-bit?

@hmellor do you know if this is supported now? It was also mentioned in https://github.com/vllm-project/vllm/issues/8720

I'm not currently aware of any changes (the docs haven't changed either).
The issue you linked appears to be from the same author with duplicated content.

Same question, there is many q4 models, but no q8_0 models (llama 3.3 for example) for use with vllm, and bitsandbytes uses q4, is there any way to load llama 3.3 70b in q8_0 in vllm ?

authors: IkhlasAlhussien hmellor mgoin molereddy qinghon rstanislav


text:
[Model][Jamba] Mamba cache single buffer

By carefully allocating the Mamba cache at the first "n" slots in the mamba cache before FWD pass ,
We can now remove the redundant CG Mamba buffer.
This PR saves memory, simplifies the Jamba inner state management code and accelerates latency (by removing redundant data copies).
This PR is also applicable to #6484 @tlrmchlsmth .
PLUS Added another test for parallel sampling.

/ready

PR is ready, CI failures are not related to this PR.

Thank you for the review! Sorry for the long delay.
Most shuffling occurs during the transition from prefill steps to decoding steps. However, shuffling between sequential decoding steps ( which populate the majority of the steps distribution under a regular load ) doesn't happen very often since the cache is already in place.
And regarding end-to-end perf - so yeah, we benchmark prefill and decoding forward passes independently. We've seen 1-2 ms speed up in decoding, and no change in prefill steps. However, the major purpose of this PR is to reduce the memory usage.

Red line is the previous implementation, blue line is this PR implementation.
RE - Parallel sampling test. Yeah, I've intended to add it but the tiny Jamba model we use for unittest behaves differently on different devices. So I've left it out for now until we have a trained tiny model for tests.

I left a few comments in-line. Generally, I think the approach makes sense and don't see any specific problems, but I think we should get somebody working on multi-step scheduling to review in case any conflicts might arise there. @alexm-neuralmagic could you look into that and suggest other reviewers as well?
I think the functions that manage the mamba cache might be better organized if they were factored out and encapsulated in their own class. I was thinking we could try to make it behave similarly to the BlockManager in terms of interface. Goal would be to incrementally make the mamba cache fit into vLLM's native systems. Doesn't have to be in this PR but curious to hear your thoughts on this.
One last question: A lot of this would be simpler if the two mamba cache update functions took a list of indices rather than requiring contiguous tensors. Have you looked into this at all? To me it looks like it wouldn't be too technically difficult to do, but would require a pair of PRs on https://github.com/Dao-AILab/causal-conv1d and https://github.com/state-spaces/mamba. Might be worth it just to avoid the state management.

Thank you for the review! @alexm-neuralmagic would love to hear your opinion.
RE I totally agree that an ideal solution would be a BlockManager subtype that would also be able to deal space states cache.
This would definitely will require us to add a functionality to the mamba/casual-conv1d kernels to make them take list of indices in addition to the mamba cache. However, We haven't had any work in this area yet, mostly since it doesn't improve performance.
We would love to see future improvements that would allow us to use vLLM's native systems.

@mzusman FYI I am working on modifying the kernels to take a tensor of indices for the batch coordinates. I think this branch gives us the interface we'd need to avoid all of the state copying for causal_conv1d_update:
https://github.com/Dao-AILab/causal-conv1d/compare/main...neuralmagic:causal-conv1d:tms/list_causal_conv1d_update
Going to try to do the same thing to selective_state_update as well. I think it would make sense to go for this approach instead if we can make the kernel updates usable quickly enough, but landing this PR is obviously more expedient. Curious to hear your urgency on getting this improvement in

That's really great! Cache management will be easier to handle.
That's right, landing this PR is quite urgent for us at the moment and does not block future improvements. I think it would be better to split those improvements/PRs.
I suggest we land this PR first, then add the adaptations from the Mamba kernel adjustments you will add in the next PR.

FYI I just restarted the failed jobs

I think it makes sense to add unittests that test the cache management utils,
I guess we will add them in the future in a following PR ,
since at the moment we've already got unittests that test the cache mechanism indirectly, like this one that verifies that generations are correct ,and that batching is also correct . and this one that verifies state cleanup and more
RE CI - I'll rebase, maybe it will help, failures doesn't seems to relate to this PR.

Going to merge this one, and then try to simplify with updated kernels :)
Thanks!

authors: mzusman tlrmchlsmth


text:
[Misc] benchmark: Add option to set max concurrency

Add a new flag to benchmark_serving.py that allows you to specify
the maximum number of concurrent requests. If not specified, it
defaults to the current behavior of unbounded concurrency.
Closes #3127
Signed-off-by: Russell Bryant rbryant@redhat.com

In general LGTM. One question I have is how should this be used together with request rate (QPS)? Are we expecting only one of them being set at a time?

For me, maximum concurrency is mainly used to avoid request buffer overflow at inference engine side (IIRC if we send >1000 requests to vLLM, TGI or other inference engines, there will be request failure due to request buffer overflow). This should always be set when benchmarking many request under very high QPS to prevent request failure.

The message looks good to me. In addition, please also update other parts such as logging and dumped file name. Please search request_rate in this file and make sure max_concurrency is reflected to these places as well.

The message looks good to me. In addition, please also update other parts such as logging and dumped file name. Please search request_rate in this file and make sure max_concurrency is reflected to these places as well.

I added some logging and added max_concurrency to the output json data.
I wasn't sure about the default filename, though. request-rate always has a value (defaults to inf), while max-concurrency defaults to None. At some point, it seems reasonable to just fall back to the ability to set custom filenames. If you feel strongly about this though, I'll update this to conditionally include the concurrency setting when it is set.

The message looks good to me. In addition, please also update other parts such as logging and dumped file name. Please search request_rate in this file and make sure max_concurrency is reflected to these places as well.

I added some logging and added max_concurrency to the output json data.
I wasn't sure about the default filename, though. request-rate always has a value (defaults to inf), while max-concurrency defaults to None. At some point, it seems reasonable to just fall back to the ability to set custom filenames. If you feel strongly about this though, I'll update this to conditionally include the concurrency setting when it is set.

Just make it simple like the following?
max_concurrency_str = f"-concurrency{args.max_concurrency}" if args.max_concurrency is not None else ""
file_name = f"{backend}-{args.request_rate}qps{max_concurrency_str}-{base_model_id}-{current_dt}.json"  #noqa

Just make it simple like the following?

sure, that works.  done!

CI is currently failing on the use of contextlib.nullcontext(). It looks like it worked for me because I was using a newer version of Python, while it's failing with 3.9 in CI.
asyncio support was added to nullcontext in 3.10: https://github.com/python/cpython/issues/85715

authors: KuntaiDu comaniac russellb


text:
Pass current field value to custom getter

Modification which allows to use custom getter with typehinting, ex.:
public function getCreatedAt(\DateTime $createdAt)
{
    return $createdAt->format('Y-m-d H:i:s');
}

Looks like if any field value type is changed via custom getter then it breaks the conversion to database values before creating/updating entity. This also happens without my changes.

I do not really understand purpose of this? Above example would be better named formatDate(). Custom getters are supposed to return value related to the Entity itself. Not to work on given parameter.
public function formatDate(\DateTime $createdAt)
{
    return $createdAt->format('Y-m-d H:i:s');
}

If you want custom getter to format the createdAt property of Entity then you could do something like:
public function getCreatedAt()
{
    return $this->createdAt->format('Y-m-d H:i:s');
}

@tuupola it's called data mutators. It provides a convenient way to convert data without using helper methods. See an example here http://laravel.com/docs/5.0/eloquent#accessors-and-mutators
Also as i said in my previous comment, currently if you define a custom getter which converts the value to other type than it breaks the data conversion to database values, because if the field is datetime so DBAL expects an instance of DateTime but gets a string.

I still fail to see the problem the first commit solves. Accessors which transform the value are already possible. Why does accessor need to be passed the value it returns? Value itself is already accessible via $this.
public function getCreatedAt()
{
    return $this->created_at->format("Y-m-d");
}

Second commit, if I understand correctly, fixes the data conversion issue you described. It probably should be a separate PR.

Why does accessor need to be passed the value it returns (or transforms from)?

Ok if you don't see any value in this, i close this

authors: norkunas tuupola


text:
Remove unneeded message on command completion. Set error output to Stderr

Fixes #426
Basically I removed the "Succeeded" message and redirect error output to Stderr. Do you see any problem with this approach? Is there any case that we want to print that "Succeeded" message?
Signed-off-by: Víctor Martínez Bevià vicmarbev@gmail.com

Hey @vicmarbev the removal of Succeed text might have some impact on the tools that consume imgpkg as a binary, like vendir or even imgpkg. Is there any particular reason for you to want to remove it?
When you run imgpkg ... | cat you will see that the message does not show up. Would that be enough? another option would be to do imgpkg --tty=false this should also remove messages that are meant for humans to read.

What about that last commit? It's the same approach they used in Kapp about this issue: https://github.com/vmware-tanzu/carvel-kapp/pull/592

some work is happening in kapp to clean up similar functionality. will want to port over those changes here. stay tuned.

updated kapp change: https://github.com/vmware-tanzu/carvel-kapp/commit/ed8c3f208aaabfac0c678d50cfee120fbd8377d9

Hi @cppforlife could you help me with the IsCobraInternalCommand not declared by package  error? I see no difference with the carvel-kapp version

I was looking at the PR and to be fair I am not sure I agree with the name of the function that was added to cobrautil. help is not a cobra internal command. It looks like we are overreaching here with the naming.

I was looking at the PR and to be fair I am not sure I agree with the name of the function that was added to cobrautil. help is not a cobra internal command. It looks like we are overreaching here with the naming.

Yeah, I agree with you on this. help is set by cobra by default but it can also be set manually by us, also it is a little bit tricky in the sense that it's set at runtime unlike other commands. I was trying to describe this behaviour and the best thing I could come up with was internal (as the other commands being checked in that function are cobra internal commands). I am definitely open to changing this up if we can think of a better name.

I think what we want is maybe a function that let us know that we should not provide extra output to them because in some sense this is the only use of this particular function.
What do you think of IsCobraManagedCommand?

What do you think of IsCobraManagedCommand?

💯
Makes sense to me.

could you help me with the IsCobraInternalCommand not declared by package error? I see no difference with the carvel-kapp version

@vicmarbev I don't see any diff in the vendor directory, you might wanna run go mod vendor or the ./hack/build.sh script.

I think the problem might be as simple as the vendored packaged was not added to the commit. So if there is a change in the vendor we need to dad the file and commit it.
If this does not solve the problem I can take a look at it later

What do you think of IsCobraManagedCommand?

💯 Makes sense to me.

could you help me with the IsCobraInternalCommand not declared by package error? I see no difference with the carvel-kapp version

@vicmarbev I don't see any diff in the vendor directory, you might wanna run go mod vendor or the ./hack/build.sh script.

That did it. Thanks!

Anything else needed for this PR?

@vicmarbev whenever you can please bump cobrautil to the latest version so that we can use the new function. After that I think we will be ready to merge this PR

authors: cppforlife joaopapereira praveenrewar vicmarbev

repo: vmware-tanzu/carvel-imgpkg


text:
Controller tools now automatically removes metadata descriptions

This overlay currently fails when using vendored controller-tools 0.7.0 because the metadata description has already been removed.
It is no longer generated in CRDs since 0.5.0. See release notes: https://github.com/kubernetes-sigs/controller-tools/releases/tag/v0.5.0
Showing no change in CRD generation:
git clone git@github.com:Samze/carvel-secretgen-controller.git -b description_fails
Cloning into 'carvel-secretgen-controller'...
remote: Enumerating objects: 17998, done.
remote: Counting objects: 100% (256/256), done.
remote: Compressing objects: 100% (96/96), done.
remote: Total 17998 (delta 199), reused 161 (delta 160), pack-reused 17742
Receiving objects: 100% (17998/17998), 27.14 MiB | 9.96 MiB/s, done.
Resolving deltas: 100% (7215/7215), done.
sgunaratne@sg-dev:/tmp$ cd carvel-secretgen-controller/
sgunaratne@sg-dev:/tmp/carvel-secretgen-controller$ ./hack/gen-crds.sh
sgunaratne@sg-dev:/tmp/carvel-secretgen-controller$ echo $?
0
sgunaratne@sg-dev:/tmp/carvel-secretgen-controller$ git diff
sgunaratne@sg-dev:/tmp/carvel-secretgen-controller$

Thanks for the PR @Samze ! Appreciate it 🙏

authors: Samze neil-hickey


text:
Packages for trial/free editions of Tanzu SaaS products.

Feature Request
As a developer I would love if there were also packages for that would put trial/free editions of our SaaS products onto a cluster/namespace.  I know this is usually more of a configuration change, but some might require a daemonset or such. Even if it is just configuration, I would like to set it up by just giving the usersname and password (and URL if needed) to the SaaS Package and it wires it all up for me.
Describe alternatives you've considered
The alternative is a user finding the documentation or product page, reading through the docs, and then applying YAML to their cluster. This seems more complicated and more prone to error. An application developer does not really care about the YAML and such, the whole point of the SaaS is it should be the least amount of work possible for them.

We can pass it along but it is in no way in our control. That said, TMC Starter is in the works.

@joshrosso has any work for the core team been identified on the road to TMC Starter?

repo: vmware-tanzu/community-edition


text:
Fix hover on collapsed navigation

Fixes:

Signed-off-by: GuessWhoSamFoo foos@vmware.com

The light theme behavior could be related to #1084

Works well for me in both dark and light mode.

The icon is not changing color on hover in light theme the same way it's doing in the dark theme. You can see that in the above recording from Wayne. It's probably just not the correct color applied in light theme.

Yeah, that's covered by #1084 that's not a blocker for this.

authors: GuessWhoSamFoo mklanjsek wwitzel3

repo: vmware-tanzu/octant


text:
Adds Microsoft/go-winio vendor package

This commit adds the Microsoft/go-winio vendor package, which will be used for implementing the npipe based Docker plugin on windows.
I'm skipping the test here as there is no direct code change, and this addition was already validated in PR #1447.

@msterin that's right.

repo: vmware/docker-volume-vsphere