text:
problem compiling with the new HAL library release.

most of the functions used in the program are not recognized .
but when i use the scr and Inc files given in the project folder it works.
shouldnt the HAL library releases be compatible.

Its hard to say. Which newer version do you use? Which function calls are not recognized?

void DMA_TransferCompleteHandler(DMA_HandleTypeDef *DmaHandle);
void DMA_TransferHalfHandler(DMA_HandleTypeDef *DmaHandle);
compiler does not recognize DMA_HandleTypeDef??

i think the problem is with the pointer  *DmaHandle
i referred the STM32F103xB document and found that they have used *hdma.
ca this be the issue?
There is another warning showing unused parameter *DmaHandle

even if change the *DmaHandle to hdma the compiler still shows this warning:
../src/ws2812b.c:269:49: warning: unused parameter 'hdma' [-Wunused-parameter]
void DMA_TransferHalfHandler(DMA_HandleTypeDef *hdma)
there is also another Warning:
n file included from ../src/visEffect.c:18:0:
../include/ws2812b.h:90:13: warning: 'ws2812b_set_pixel' declared 'static' but never defined [-Wunused-function]
static void ws2812b_set_pixel(uint8_t row, uint16_t column, uint8_t red, uint8_t green, uint8_t blue);
this is the last warning:
../src/ws2812b.c: In function 'ws2812b_set_pixel':
../include/ws2812b.h:84:36: warning: initialization discards 'volatile' qualifier from pointer target type [-Wdiscarded-qualifiers]
#define BITBAND_SRAM(address, bit) ( (__IO uint32_t *) (RAM_BB_BASE + (((uint32_t)address) - RAM_BASE) * 32 + (bit) * 4))
^
../src/ws2812b.c:458:22: note: in expansion of macro 'BITBAND_SRAM'
uint32_t *bitBand = BITBAND_SRAM(&ws2812bDmaBitBuffer[(calcCol)], row);
As these are warnings the compiler still compiles and give an output .hex file. when i upload it to stm32F103c8t6 nothing happens to the connected Led strip.
I have the strips data pin connected to PC14
i have changed the necessary #defines
#define WS2812B_PORT GPIOC
#define WS2812B_PINS GPIO_PIN_14
#define WS2812B_NUMBER_OF_LEDS 30
// Number of output LED strips. Each has its own buffer.
#define WS2812_BUFFER_COUNT 2
had a doubt what does the  "WS2812_BUFFER_COUNT 2" this do ? what is this define for?

There will be warnings, the code is not clean.
WS2812_BUFFER_COUNT 2
defines how many GPIO pins on a SINGLE port is connected to the WS2812B LEDs.
If you connect one LED strip on PA0 and other on PA1, then the value should be 2

But u have used 4 strips in parallel and then too used a WS2812_BUFFER_COUNT 2.  ????
WHY??

i found the problem i guess.
stm32f103xx does not support mem2peripheral dma transfer.
instead i will have to use mem2mem DMA with destination address that of GPIOx ODR register.
how can i achieve this??

The code outputs data on 2 output pins for 2 LED strips. Where do you see mention of 4 led strips? Maybe the readme was taken from the F4 project.
WS2812_BUFFER_COUNT defines how many outputs you use. You can even assign many outputs the same framebuffer and it will output the same data to all of them.
https://github.com/hubmartin/ws2812b_stm32F3/blob/master/src/ws2812b/ws2812b.h#L69
F103 supports all combinations of memory & peripheral transfer
http://www.st.com/content/ccc/resource/technical/document/application_note/47/41/32/e8/6f/42/43/bd/CD00160362.pdf/files/CD00160362.pdf/jcr:content/translations/en.CD00160362.pdf
As I said. You can send me your code/project and I check that with oscilloscope/logic analyzer. I've troubleshooted a lot of WS2812B glitches so we can quickly find what is wrong.

Ok I'll snet u the code.
I'm current at my native place I'll sent it once I reach home.
Thank you for the help.

i have attached my code.
its created in eclipse IDE
WS2812b_with_given.zip

hey!!

hey,
I do not have time right now, I'll take a look at the code later. Thanks.

I've checked the most important bits which are:
see Table 78. Summary of DMA1 requests for each channel
The channels 2 (up), 5(cc1) and 7(cc2) are ok.
Here do you don't enable GPIOB clock int the ws2812b.h file
// GPIO enable command
#define WS2812B_GPIO_CLK_ENABLE() __HAL_RCC_GPIOC_CLK_ENABLE()
// LED output port
#define WS2812B_PORT GPIOB
// LED output pins
#define WS2812B_PINS (GPIO_PIN_0 | GPIO_PIN_1)
Do you have debugger and oscilloscope?
You have to check that DMA half and full interrupts are working fine. Try to define GPIO debug outputs  LED4_PORT and LED5_PORT, or put a breakpoint to the DMA half/full handlers.
Other issue could be ,that F103 DMA1 could not write to GPIO. For example if you look on STM32F4 reference manual "Figure 23. System implementation of the two DMA controllers
STM32F401xB/C and STM32F401xD/E) then you see that only DMA2 can write to AHB2, where the GPIO is placed.
But in F103 **Figure 1. System architecture (low-, medium-, XL-density devices)
** it seems that both DMAs can access APB2, thus all the GPIOs.
Try that GPIO clock enable and let me know the results.

Thank you.
the clock was not enabled.
sorry for the whole hassle.
I will try to implement it and if any problem comes up ill ask.
again thank you

does changing the clock setting effect the timer events
tim_period =  SystemCoreClock / 800000; // 0,125us period (10 times lower the 1,25us period to have
fixed math below)
uint32_t cc1 = (10 * tim_period) / 36;
uint32_t cc2 = (10 * tim_period) / 15;
my clock speed is 72Mhz as I am using USB CDC to transfer LED data from PC.

can u explain to me in a bit detail about the above doubt.

As long as the SystemCoreClock is 72000000 then everything should be fine. If not, make sure you call System Core Clock update function.

I am very sorry.
but correcting the port to PORTB did not make it work.
it was some random color that was being displayed when i moved the data cable and not the stm32 was driving it.
can u check the code again for me.

tried running the debug in eclipse
and found that the in the function  visHandle() the if(ws2812b.transferComplete) is never turned true it just keep looping over and over.
void visHandle()
{
if(ws2812b.transferComplete)
{
// Update your framebuffer here or swap buffers
visHandle2();
// Signal that buffer is changed and transfer new data
ws2812b.startTransfer = 1;
ws2812b_handle();
}
}
which means
void HAL_TIM_PeriodElapsedCallback(TIM_HandleTypeDef *htim) is never  called

It is initially set in
https://github.com/hubmartin/ws2812b_stm32F3/blob/master/src/ws2812b/ws2812b.c#L545
which is called from visInit()
and then enabled again after DMA transfer is done.
I try to test it, hovewer I have no F103 CPU other than deubggers od ST-link so it will take a week until I find a time to find and prepare the hardware.

Thank you for ur precious time. : )
I'll wait for another week.
Just to let u know , I am done with the whole processing part which sends the led frame data to stm32 through USB CDC.
Another problem was that I put a breakpoint in the DMAtransfercomplete  callback function and it is never triggered while debugging. Could that be a problem.
In the file ws2812b.c the DMA IRQ handlers for channel 2 and 5 were commented out
Only IRQ for channel 7 was enabled.

any luck

aything??

Got it running.

The issue was that the F1xx HAL library, or the new version HAL libraries at general clear the user function callbacks when you call HAL_DMA_Init(). So i just swaped those lines to:
	// TIM2 CC2 event
        ......
	dmaCC2.Init.Priority = DMA_PRIORITY_VERY_HIGH;
	dmaCC2.Instance = DMA1_Channel7;

	// DMA_INIT clears the callbacks, call it first
	HAL_DMA_Init(&dmaCC2);

	// Then set the callbacks
	dmaCC2.XferCpltCallback  = DMA_TransferCompleteHandler;
	dmaCC2.XferHalfCpltCallback = DMA_TransferHalfHandler;
	//dmaUpdate.XferErrorCallback = TransferError;

	HAL_NVIC_SetPriority(DMA1_Channel7_IRQn, 0, 0);
	HAL_NVIC_EnableIRQ(DMA1_Channel7_IRQn);
	HAL_DMA_Start_IT(&dmaCC2, (uint32_t)WS2812_IO_Low, (uint32_t)&WS2812B_PORT->BSRR, BUFFER_SIZE);

....

I changed the output pins in the project to PA2, PA3 so I could test that on the ST-link hardware. Don't forget to change it back to you pins (WS2812B_PORT, WS2812B_PINS), set the WS2812B_GPIO_CLK_ENABLE() define correctly and in visInit() set channels to the pin numbers ws2812b.item[0].channel = 2; // for PA2.
Could I use your example code and create a new repository with F103 support? Are you ok with that? So others can use that. Now the lib could support F0,F1 (+F2), F3, F4 lines. Great.
WS2812b_STM32F103_fixed_jacobrejin.zip

Thank you so much 💯 👍

Could I use your example code and create a new repository with F103 support

yes please.
i struggled a lot to get it working.
let others get it easily

I am working a processing application which would allow us to display videos just like octoWs2812b
once done ill post it

Thanks for details. I wanted to ask later about the usage of my library.
I studied other libraries how they work before I wrote mine. However I did not tested them. Can you please in short tell me what libraries did you try, some comparison and why did you choosed LowMEM ws2812b library?
Thanks a lot.

one of my main issue was that, if  I want  get a large WS2812B display working with OctoWS2811 we would need a Teensy 3.0(65K SRAM)  board which costs almost 11.65$ which can only drive 1000 leds.
on the contrary STM32F103(20K SRAM) which had almost the same spec costs 1.85$ with shipping on aliexpress.
my goal was to create a library similar to OctoWS2811 but using cheap STM32F103 uC which can drive as many leds as possible.
**DMA > SPI/I2S peripheral : ** The limitation of this method, as u said is that it can only support upto 4
parallel strips.
**DMA > TIMer Output compare register & TIMer > DMA > GPIO : ** although it can drive multiple LED's but the memory requirement is very heavy.
lets just say i want to drive 2400 leds using your TIMer > DMA > GPIO in which each new frame data is received after the DMA transfer, I would  require 14.4K of SRAM. Of which 7.2K would be for led Frame data and another 7.2K for pre-computed data. As You Said in your website
This is where your LowMEM ws2812b library shines.
I can use the entire 14.4K of SRAM for new LED data ie 4800 leds  using a STM32F103 uC which only has 20K of SRAM.
But the limiting factor while using your LowMEM ws2812b is not the SRAM size. its the Frame rate that can be achieved .
if we are driving 2400 leds ( by a method in which each new frame data is received after the DMA transfer) it would take almost 50ms to transfer the LED data from processing to STM32F103 using USB CDC and another 10ms to transfer the data to the LEDs using DMA.
That is a total of 60ms which is ----> 16fps( in theory )
if we drive 1200 LEDs we could 32fps and so on.
even if we are able to drive 500 leds it would be great since
we are saving a lot as compared to the price of a single teensy

Thanks for details. I did some basic calculations comparing other libraries, but I never pushed my lowMEM library to the edge. Sounds like I should investigate that not just on the paper but also in reality and find out the real memory usage.

Good day. Have bad english, sorry.
I think I have a problem with void DMA_TransferHalfHandler(DMA_HandleTypeDef *DmaHandle);
Then I fliash project without cubemix code generator its ok working. After I generate code with cubemix,  PORTC generate  periodes  1.25us, but reset pulse no and always ws2812b.transferComplete = 0;
Then I debug DmaHandles:
void DMA_TransferCompleteHandler(DMA_HandleTypeDef *DmaHandle);
void DMA_TransferHalfHandler(DMA_HandleTypeDef *DmaHandle);
In it leds not on. Can you help?

I experence write *hdma
void DMA_TransferCompleteHandler(DMA_HandleTypeDef *hdma);
void DMA_TransferHalfHandler(DMA_HandleTypeDef *hdma);
Compiling ok, but problem does not improve.

CPU STM32F407ZG

Have you tried F4 port which is specifically for F407? Please start a new thread there.
https://github.com/hubmartin/WS2812B_STM32F4

authors: Jeriomas hubmartin jacobrejin


text:
多轮对话能力

请问基于上传文档的知识问答，具有多轮对话能能力吗，如果有请问是如何实现的

目前在提供了History的传递，实现了基本的上下文对话，但是仅仅只基于需要和客户端交互的地方才使用History,对于LLM在后台运行时的输出，是不采用history的，根据业务发现中间过程不需要进行记忆，但是如果你的业务需要的话可以自己改造
对于如何实现记忆能力多轮对话的，看llms/llm_base_model和llm_gpt_model，这两个llm类文件中有注释

注意：不同的LLM的History类型是不一样的，你可以仔细看看llm_base_model和llm_gpt_model

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Minor Documentation tweaks

Just noticed some lint.

@chrisgilbert I lightly tweaked the documentation, and then updated per your recent PR.

👍  looks good to me, thanks @StevenACoffman!

sorry clicked the wrong button

Thanks @StevenACoffman, we appreciate the contribution 😄

source: gharchive/pull-request

authors: StevenACoffman chrisgilbert wliao008

repo: hudl/registrator

license: mit

license_source: bigquery


text:
🛑 Akkoma is down

In 0972e34, Akkoma (https://akkoma.chez.chat) was down:

HTTP code: 404
Response time: 341 ms

Resolved: Akkoma is back up in f480e6b after 17 minutes.

repo: hufman/upptime

license: MIT


text:
При запуске вешает контейнер в proxmox

Добрый. Контейнер создан на базе Ubuntu 24.04 LTS (GNU/Linux 6.8.12-1-pve x86_64). Promox на базе NUC. Архив положен в /opt/byedpi - распакован.
-rw-r--r-- 1 root root  70915 Jul 29 20:34 byedpi-12-x86_64.tar.gz
-rwxr-xr-x 1 root root 133776 Jul 29 20:24 ciadpi-x86_64

При запуске с  ./ciadpi-x86_64 --disorder 1 --fake 0 --ttl 1 -A torst --tlsrec 1+s --debug 1  - да в принципе с любыми ключами вешает контейнер.
Что я делаю не так ?

вешает контейнер.

Как это проявляется? Программа игнорирует ctrl+c?  top показывает какую либо нагрузку? Какая реакция на попытку поключения (есть что-либо в логах)?

вешает контейнер.

Как это проявляется? Программа игнорирует ctrl+c? top показывает какую либо нагрузку? Какая реакция на попытку поключения (есть что-либо в логах)?

В топе ни какой нагрузки нет

ctrl+c - снимает задачу.
Если задачу не снять переподключиться к контенеру нельзя

Ну я даже не знаю, а чего вы ожидаете, запуская программу не в фоне и никак её не используя?

Ну я даже не знаю, а чего вы ожидаете, запуская программу не в фоне и никак её не используя?

Поясните мне не умному. Как запустить в фоне ?

В идеале через systemd unit
В крайнем случае, или для быстрого теста, добавить в конец команды амперсанд:
 ./ciadpi-x86_64 --disorder 1 --fake 0 --ttl 1 -A torst --tlsrec 1+s --debug 1 &

В идеале через systemd unit
В крайнем случае, или для быстрого теста, добавить в конец команды амперсанд:
 ./ciadpi-x86_64 --disorder 1 --fake 0 --ttl 1 -A torst --tlsrec 1+s --debug 1 &

Так запустилось. Спасибо.

./ciadpi-x86_64 --disorder 1 --fake 0 --ttl 1 -A torst --tlsrec 1+s --debug 1 & - вот так все заработало

authors: ProMix0 dumbodumbos hufrea

repo: hufrea/byedpi


text:
Add intrinsic debugging tools for MatchFinder

We should be able to toggle visualisation of:

Raw features
Filtered matches

As such.


text:
[WIP] Translation for _toctree, Chapter 0 and Chapter 1 in Bengali

Partially Fixes #47

_toctree.yml and Chapter 0 is done till now, currently working on Chapter 1

Hi @MKhalusova, this PR is ready for review!

Chapter 2 Added

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

CC @mishig25 Can you please take a look to see why the preview isn't working for another language? Thanks!

@susnato thanks a lot for the translation! For the language names, we use 2-letter format. Therefore, according to this table, instead of ben, it should be bn. After this change, the preview will work 😊

@susnato Please address the comments by @mishig25 so that the course is built correctly, and then we'll be able to merge

Hi @mishig25 thanks for the suggestion, the PR preview is working now. :hugs:

Hi @MKhalusova , I have pushed the changes and the PR is correctly building now!

authors: HuggingFaceDocBuilderDev MKhalusova mishig25 susnato

repo: huggingface/audio-transformers-course


text:
In /valid, associate an enum to every dataset

Instead of a list of datasets:
{
  "valid": [
    "0-hero/OIG-small-chip2",
    "000alen/semantic",
    "04-07-22/wep-probes",
    "0721boy/nva-pic",
    ...
  ]
}

we could return an object, where the datasets are the keys, and the values are lists of available features: "preview" | "parquet" | "parquet-sample"
{
  "0-hero/OIG-small-chip2": ["preview"],
  "000alen/semantic": ["preview", "parquet"],
  "04-07-22/wep-probes": ["preview", "parquet-sample"],
  "0721boy/nva-pic": [],
  ...
}

The list of features for each dataset helps the Hub show the appropriate text/icons/UI.
Note that we might want to change the name of the endpoint to avoid breaking change, and to be more descriptive: /features-by-dataset (I don't like it, because features has another meaning in datasets).
Also: the response could take too long to be computed online as it's currently done (it already takes about 300ms to compute), so we should precompute it regularly with a cronjob and cache the result. See #891

to avoid breaking change
are you sure anyone extrernal is using that endpoint (or the API for that matter :) )

"04-07-22/wep-probes": ["preview", "parquet-sample"],
on the Hub side at least, we won't really care about the "lower-level" attribute, just the "best" one

if we don't need multiple values for one dataset, we could use the following format instead (similar to the current one):
{
  "preview": [list of datasets with highest level == preview],
  "parquet-sample": [list of datasets with highest level == parquet-sample, ie. parquets for the first 5G],
  "parquet": [list of datasets with full conversion to parquet]
}

Done in #1353

authors: julien-c severo

repo: huggingface/datasets-server


text:
Inconsistent argument names.

Just find it a wee bit odd that in the transformers library predictions are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61
While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40
Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.

Also for the Accuracy metric the accuracy_score method should have its args in the opposite order so accuracy_score(predictions, references,,,).

Thanks for pointing this out ! 🕵🏻
Predictions and references should indeed be swapped in the docstring.
However, the call to accuracy_score should not be changed, it signature being:
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)

Feel free to open a PR if you want to fix this :)

authors: Fraser-Greenlee SBrandeis

repo: huggingface/datasets


text:
Downloading/caching only a part of a datasets' dataset.

Hi,
I want to use the validation data only (of natural question).
I don't want to have the whole dataset cached in my machine, just the dev set.
Is this possible? I can't find a way to do it in the docs.
Thank you,
Sapir

Not at the moment but we could likely support this feature.

?

I think it would be a very helpful feature, because sometimes one only wants to evaluate models on the dev set, and the whole training data may be many times bigger.
This makes the task impossible with limited memory resources.

authors: SapirWeissbuch jman1973 thomwolf


text:
Remove ppo visualize redundent imgs

It was confusing to see multiple illustrations of the same graph; there is the graph:

at the top
not under case 1 and 2
under case 3 and 4
under case 5 and 6

Was a bit confused if they are different graph, and then realized they are actually the same.

Hey there 👋 thank you for your PR 🤗
But we prefer to let the image 3x so that's it's easier for most readers to follow the different explanation without having to scrolling back every time.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

repo: huggingface/deep-rl-class


text:
MSE not accepting (n_samples, n_ouptuts) despite docs stating so

The evaluation card for MSE states:

Mandatory inputs:

predictions: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the estimated target values.
references: numeric array-like of shape (n_samples,) or (n_samples, n_outputs), representing the ground truth (correct) target values.

So it should be usable similar to pytorch's MSELoss with multiple dimensions.
But using it with (batch_size, multiple_outputs) doesn't work:
import evaluate
import numpy as np

m = evaluate.load('mse')
print(m.compute(predictions=np.random.rand(16,2), references=np.random.rand(16,2)))

⬇
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='float32', id=None), 'references': Value(dtype='float32', id=None)},
Input predictions: ...
Input references: ...

Version: evaluate==0.4.1

Hi @bauersimon , the quick answer for your question is:
import evaluate
import numpy as np

- m = evaluate.load('mse')
+ m = evaluate.load('mse', 'multilist')
print(m.compute(predictions=np.random.rand(16,2), references=np.random.rand(16,2)))

And, the error message is also too vague to address the problem.

Thanks! Should probably be mentioned in the docs then 😅.

authors: bauersimon shenxiangzhuang

repo: huggingface/evaluate


text:
Update spaces-gpus.md

add of L40S pricing

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

authors: HuggingFaceDocBuilderDev Michellehbn

repo: huggingface/hub-docs


text:
Document the filter argument

This documents the filter argument.

also cc'ing @osanseviero @thomwolf and @Pierrci

LGTM! :) if possible I would take the opportunity to also document the other arguments (not sure what full is or how to use it)

authors: LysandreJik julien-c osanseviero


text:
Inference stuck！

System Info
onnxoptimizer           0.2.7
optimum                 1.12.0
system: centos7

Problem: Convert llama2 to onnx, and then inferencing stuck！

inference code:

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("./hugg_face_with_past/")
# if with past: use_cache=True, use_io_binding=True
model = ORTModelForCausalLM.from_pretrained("./hugg_face_with_past/", use_cache=True, use_io_binding=True)
inputs = tokenizer("My name is Arthur and I live in", return_tensors="pt")
inputs.pop('token_type_ids')

gen_tokens = model.generate(**inputs)
print(tokenizer.batch_decode(gen_tokens))

log:
2023-09-07 19:37:07.305776: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
[2023-09-07 19:37:08,463] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/home/centos/.conda/envs/llms/lib/python3.9/site-packages/transformers/generation/utils.py:1338: UserWarning: Using `max_length`'s default (4096) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
  warnings.warn(

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction (minimal, reproducible, runnable)

..

### Expected behavior

Inference success!

Hi @empty2enrich, which model are you using? Do you face the same issue when loading with transformers AutoModelForCausalLM?
Something you could try is to add for example max_new_tokens=3 to the generate call to check whether it is not simply the model generating a very long sequence.

repo: huggingface/optimum


text:
Is it possible to use transformers.js to implement audio source separation tasks?

Question
Hello, I have a beginner's question.
I want to implement the task of removing the human voice from the audio in the video and retaining the background sound in the browser. The idea is to load the model for audio source separation related to transformers.js to achieve the separation of the background sound and human voice, and then only return the background sound.
But I couldn't find relevant examples in the documentation, so I was wondering if this can be implemented? If so, what are the learning or research paths?
Looking forward to your reply

Yes, speechbrain looks good, but I think demucs would be the best. However, I'm a beginner in machine learning and Python, so I'm looking for the simplest solution that can achieve my goal

Hi there 👋 This library serves as a JavaScript port of the Python transformers library, so if you know of a model where you can do this, we can certainly look into it! Is something like https://huggingface.co/speechbrain/sepformer-wham what you're looking for?

我也需要


text:
Add option to check dataset labels in SFTTrainer

Hi everyone!
This PR introduces a new option check_dataset_labels in SFTTrainer. When enabled, the trainer calls the collator on the first sample from the training set and logs token_id, decoded token_id, and the corresponding label. This helps to uncover mistakes early, such as setting the tokenizer's pad_token toeos_token. The idea is taken from axolotl where such an option exists already.
Problem(s):
It's common practice to set the tokenizer's pad_token to eos_token. This is problematic because DataCollatorForCompletionOnlyLM sets the label for all occurrences of pad_token to -100. If pad=eos, then the model will never learn to output eos. This issue can be hard to debug and many people struggle with it.
Additionally, when using instruction_template and response_template, logging the tokens and labels would be helpful to ensure that all the labels are correctly set to only train on output and ignore the instruction. Furthermore, tokenizers can be complex and the output would aid in spotting tokenization issues like handling of special tokens quickly
Solution:

Add a new option check_dataset_labels to the trainer.
When check_dataset_labels is enabled, the trainer calls the collator on the first sample from the training set and logs token_id, decoded token_id, and the corresponding label.
This creates transparency and helps to uncover mistakes early.

Usage:
# load model and tokenizer
...

messages = [
    {"role": "user", "content": "Hello who are you?"},
    {"role": "assistant", "content": "Luke, I am your father"},
    {"role": "user", "content": "WTF"},
]
dataset = Dataset.from_list([dict(messages=messages)])

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    data_collator = DataCollatorForCompletionOnlyLM(
        instruction_template = "<|im_start|>user",
        response_template = "<|im_start|>assistant",
        tokenizer = tokenizer,
        mlm = False),
    check_dataset_labels = True,
    dataset_kwargs=dict(add_special_tokens=False),
    args = TrainingArguments(output_dir = "out")
)

output:
check_dataset_labels:
<|im_start|> user
Hello who are you? <|im_end|>
 <|im_start|> assistant
Luke, I am your father <|im_end|>
 <|im_start|> user
WTF <|im_end|>

32000 '<|im_start|>' -100
1792 'user' -100
13 '<0x0A>' -100
10994 'Hello' -100
1058 'who' -100
526 'are' -100
366 'you' -100
29973 '?' -100
32001 '<|im_end|>' -100
13 '<0x0A>' -100
32000 '<|im_start|>' -100
465 'ass' -100
22137 'istant' -100
13 '<0x0A>' 13
24126 'Lu' 24126
446 'ke' 446
29892 ',' 29892
306 'I' 306
626 'am' 626
596 'your' 596
4783 'father' 4783
32001 '<|im_end|>' 32001
13 '<0x0A>' 13
32000 '<|im_start|>' -100
1792 'user' -100
13 '<0x0A>' -100
29956 'W' -100
8969 'TF' -100
32001 '<|im_end|>' -100
13 '<0x0A>' -100

Related issues:

https://github.com/huggingface/trl/issues/1384
https://github.com/huggingface/trl/issues/976
https://github.com/huggingface/trl/issues/1191
https://github.com/huggingface/trl/issues/916
https://github.com/huggingface/transformers/issues/22794
https://stackoverflow.com/questions/76446228/setting-padding-token-as-eos-token-when-using-datacollatorforlanguagemodeling-fr

Thanks @geronimi73 the changes look great ! can you just run the styling checks? make precommit , then we can merge

what about the print statements I mentioned? leave it like this?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(to fix the failing tests you just have to rebase with main)

you will need to import logging at the top of the file and import the logger properly

i tried. it works but not sure if this is the way to do it. please check my comment on the code

Hi @geronimi73 sorry for all the iteration ! can you re-run the styling checks again? 🙏

Hi @geronimi73 sorry for all the iteration ! can you re-run the styling checks again? 🙏

sure! but i'm still wondering whether it works. I never see the output of logger.info()!
how am I supposed to set the log level (as a user of SFTTrainer) ?  I tried this, doesnt work:
import transformers
transformers.logging.set_verbosity_info()

Hi @geronimi73 sorry for all the iteration ! can you re-run the styling checks again? 🙏

sure! but i'm still wondering whether it works. I never see the output of logger.info()!
how am I supposed to set the log level (as a user of SFTTrainer) ? I tried this, doesnt work:
import transformers
transformers.logging.set_verbosity_info()

also this one does not enable it
args = TrainingArguments(
    output_dir = "out",
    log_level = "info"
    )

sorry to keep bothering you @younesbelkada but I think the problem ist that we are using logging from transformers.utils and obtaining a logger with __name__ (=trl.trainer.sft_trainer). This means that setting the log_level in TrainingArguments does not have an effect on the logger we obtained because this line in trainer.py affects the root logger transformers and does not have any effect on our logger (trl.trainer.sft_trainer).
To do this correctly I think, we would have to either

add a trl.utils.logger analogous to transformers.utils.logger (too much effort?)
OR use logger.setLevel(log_level) in SFTTrainer.init(). Otherwise we will not see the output of logging.info().

what do you think? option 2?

repo: huggingface/trl


text:
Can you provides a english README

This package is great, but most peoples(like me) don't understand the README language(look like Korean),
can you provides english README?

hello @lucky-xiaohu. Thanks for your interest.
we're sorry for the inconvenience about using react-native-play-services. translation is currently working on, so we'll quickly reply to you when it's completed.

thanks very much.

We have changed default documentation language to English for this repository. 😄
See documentation at https://github.com/humanscape/react-native-play-services/blob/master/README.md

We have changed default documentation language to English for this repository. 😄
See documentation at https://github.com/humanscape/react-native-play-services/blob/master/README.md

authors: humanscape-henry lucky-xiaohu

repo: humanscape/react-native-play-services


text:
Errors when I try to add the consentcookie to my Laravel project

Description

When I try to add the script to my Laravel project I'm getting multiple JS errors: http://joxi.ru/EA4BYJ9UD39Pvm
[Error] [vuex] already installed. Vue.use(Vuex) should be called only once.
[Error] TypeError: undefined is not an object (evaluating 'D.$store.state') — viewService.js:93
[Error] TypeError: undefined is not an object (evaluating 'M.$services') — consentService.js:119
[Error] TypeError: undefined is not an object (evaluating 'D.$store.commit') — viewService.js:158
[Error] TypeError: undefined is not an object (evaluating 'this.$store.state') — consentcookie.min.js:4473
[Error] TypeError: undefined is not an object (evaluating 'D.$store.state')
Expected Behavior

The script is working as expected
Actual Behavior

multiple JS errors (see above)
Possible Fix

Steps to Reproduce

Build your own ConsentCookie or use downloaded version
Include to a Laravel project using
Initialize ConsentCookie
Open your site and check the js console

Context

I can't use ConsentCookie
Your Environment

Laravel 5.4

Reproduced the issue using the downloaded version:
Issue is only when Vue (Standard in every new Laravel project) is used on the page. Without Vue on a site it works perfectly.
The full list of error messages:
[Error] [Vue warn]: Cannot find element: #app
	warn (app.js:36514)
	query (app.js:41221)
	$mount (app.js:46799)
	_init (app.js:40561)
	Vue (app.js:40650)
	(anonieme functie) (app.js:13898)
	__webpack_require__ (app.js:20)
	(anonieme functie) (app.js:13869)
	__webpack_require__ (app.js:20)
	(anonieme functie) (app.js:63)
	Globale code (app.js:64)
[Error] [vuex] already installed. Vue.use(Vuex) should be called only once.
	y (consentcookie.min.js:9:26761)
	use (consentcookie.min.js:8:16590)
	exports (consentcookie.min.js:2:16844)
	r (consentcookie.min.js:1:975)
	e (consentcookie.min.js:1:822)
	Globale code (127.0.0.1:89)
[Error] TypeError: undefined is not an object (evaluating 'C.$store.state') — consentcookie.min.js:2:23530
	at (consentcookie.min.js:7:30434)
	rt (consentcookie.min.js:7:30345)
	it (consentcookie.min.js:7:30211)
	Ut (consentcookie.min.js:8:6169)
	_init (consentcookie.min.js:8:15335)
	a (consentcookie.min.js:8:16905)
	r (consentcookie.min.js:1:1128)
	e (consentcookie.min.js:1:822)
	Globale code (127.0.0.1:89)
[Error] TypeError: null is not an object (evaluating 'k.$services') — consentcookie.min.js:2:21355
	at (consentcookie.min.js:7:30434)
	rt (consentcookie.min.js:7:30345)
	it (consentcookie.min.js:7:30211)
	Ut (consentcookie.min.js:8:6169)
	_init (consentcookie.min.js:8:15374)
	a (consentcookie.min.js:8:16905)
	r (consentcookie.min.js:1:1128)
	e (consentcookie.min.js:1:822)
	Globale code (127.0.0.1:89)
[Error] TypeError: undefined is not an object (evaluating 'C.$store.commit') — consentcookie.min.js:2:24030
	at (consentcookie.min.js:7:30434)
	rt (consentcookie.min.js:7:30345)
	it (consentcookie.min.js:7:30211)
	Ut (consentcookie.min.js:8:6169)
	Ft (consentcookie.min.js:8:4919)
	r (consentcookie.min.js:1:1195)
	e (consentcookie.min.js:1:822)
	Globale code (127.0.0.1:89)
[Error] TypeError: undefined is not an object (evaluating 'this.$store.state') — consentcookie.min.js:2:13862
	at (consentcookie.min.js:7:30434)
	rt (consentcookie.min.js:7:30345)
	it (consentcookie.min.js:7:30211)
	_render (consentcookie.min.js:8:15069)
	o (consentcookie.min.js:8:4982)
	get (consentcookie.min.js:9:9772)
	Yi (consentcookie.min.js:9:9694)
	Ft (consentcookie.min.js:8:4995)
	r (consentcookie.min.js:1:1195)
	e (consentcookie.min.js:1:822)
	Globale code (127.0.0.1:89)```

@stillfinder @timvaniersel I think it has to do with Laravel using Vue as its front-end framework.
Vuex is already initialized so the bootstrapping of ConsentCookie is failing.
Vue in ConsentCookie is isolated and not polluting the global scope, but seeing the error Vuex isn`t.
After searching, I found for example: https://github.com/vuejs/vuex/issues/731 .
Similar error message.
I dont have a Lavarel project atm so I cant definitly confirm it.
Solutions would be, not using Vuex for state handling or make sure (the init of) Vuex is isolated from the global scope.

To see the error: https://jsfiddle.net/StevenChooNL/0jchLx6w/

authors: StevenChoo stillfinder timvaniersel

repo: humanswitch/consentcookie


text:
Superfluid

Hi! Viscosity quantifies the internal frictional force between adjacent layers of fluid that are in relative motion. So, Viscosity is a characteristic of the substance. So in this application it should be a parameter of the substance, i.e. the colour. So each colour should have its own viscosity. If a user wants to define red particles as superfluid they can do so by setting the red viscosity to zero.
I'd like to do that, but I have no idea how to define the velocity calculation on each color. I have defined the viscosity for each color, but I don't know how I could insert it into the p1.vx and p2.vy velocity calculation (how to make p1.vx and p2.vy refer to each color separately). Any ideas? Can you help me?
ofxFloatSlider viscoSliderR;
ofxFloatSlider viscoSliderG;
ofxFloatSlider viscoSliderW;
ofxFloatSlider viscoSliderB;
ofxFloatSlider viscoSliderO;
ofxFloatSlider viscoSliderK;
ofxFloatSlider viscoSliderC;
ofxFloatSlider viscoSliderD;
and
float viscosityR = 0.3F;
float viscosityG = 0.4F;
float viscosityW = 0.1F;
float viscosityB = 0.6F;
float viscosityO = 0.5F;
float viscosityK = 0.7F;
float viscosityC = 0.8F;
float viscosityD = 0.0F;
and
viscosityR = viscoSliderR;
viscosityG = viscoSliderG;
viscosityW = viscoSliderW;
viscosityB = viscoSliderB;
viscosityO = viscoSliderO;
viscosityK = viscoSliderK;
viscosityC = viscoSliderC;
viscosityD = viscoSliderD;

If you have defined the viscosities as global variables in ofApp.h, then you should be able to use them in ofApp.cpp like this

add the viscosity parameter to the function definition
void ofApp::interaction(std::vector* Group1, const std::vector* Group2, const float G, const float radius, float viscosity){ .....}

In the update section pass the paramter for each color like this:
if (numberSliderR > 0)
{
interaction(&red, &red, powerSliderRR, vSliderRR);
if (numberSliderG > 0) interaction(&red, &green, powerSliderRG, vSliderRG, viscosityR );
if (numberSliderB > 0) interaction(&red, &blue, powerSliderRB, vSliderRB, viscosityR);
if (numberSliderW > 0) interaction(&red, &white, powerSliderRW, vSliderRW, viscosityR);
}

Reassign the gui sliders to the variables so that it can capture the changes life this.
viscosityR = viscoSliderR;

You may need to make some changes in other places so that you can export and import this parameter

Hope that helps

Thank you very much. For saving all parameters I had problems on load. I tried it for saving the number of particles per colors but this saved file load negative numbers for them. The same problem I faced when I want to save viscosity, probability, evolutionary parameters.

I put it here:

Can you show me the link to your code?
To be honest, the way we save the parameters in this software is very primitive and probably not portable across different devices. A better way is to use an external C++ library like boost serialization or cereal https://uscilab.github.io/cereal/

Can you show me the link to your code? To be honest, the way we save the parameters in this software is very primitive and probably not portable across different devices. A better way is to use an external C++ library like boost serialization or cereal https://uscilab.github.io/cereal/
OpenFrameworks also has it's own serialization method but it's documentation is lacking : https://openframeworks.cc/documentation/types/ofBaseSerializer/
My code is here:
https://github.com/KhadrasWellun/particle-life/tree/main/particle_life/src

I peeked through your code, I couldn't find a mistake. These errors are hard to track and they are usually due to some casting errors or other hidden logical mistakes. I'll try a deeper look again when have time.  Great job adding the additional colors!

I peeked through your code, I couldn't find a mistake. These errors are hard to track and they are usually due to some casting errors or other hidden logical mistakes. I'll try a deeper look again when have time. Great job adding the additional colors!

Thank you! Now I add probability for every type of interaction. So, an interaction will have; intensity, range, viscosity and probability of happening.

yeah we should switch to some xml/json/yml file for import/export. I'm busy learning oneAPI/DPC++ (still tempted to make a DPC++ version of particle-life :D)

If you will write this code in another programming language, please insert more comments in it. Please also make it easy to insert new colours. Please use the full matrices of interaction strengths as well as their probabilities and viscosity per colour.

yes, I will :)
It's still fully standard C++, it's part of the intel oneAPI.

DPC++: oneAPI’s core language for programming accelerators and multiprocessors. DPC++ allows developers to reuse code across hardware targets (CPUs and accelerators such as GPUs and FPGAs) and tune for a specific architecture
oneDPL: A companion to the DPC++ Compiler for programming oneAPI devices with APIs from C++ standard library, Parallel STL, and extensions
oneDNN: High-performance implementations of primitives for deep learning frameworks
oneCCL: Communication primitives for scaling deep learning frameworks across multiple devices
Level Zero: A system interface for oneAPI languages and libraries
oneDAL: Algorithms for accelerated data science
oneTBB: A library for adding thread-based parallelism to complex applications on multiprocessors
oneVPL: Algorithms for accelerated video processing
oneMKL: High-performance math routines for science, engineering, and financial applications

it's a bit like "cuda" (except that it's not). you can already compile particle-life with it without any change. Except that it won't really show much improvement.
It also come with Intel VTune and Intel Advisor (both are profiler).
All for free :)
particle life need to be fully rewritten :

use SOA (structure of array) instead of AOS (array of structure).
use vertex buffer (VBO) to replace tens of thousands of openGL call per frame with just a dozen call.
That would show the biggest performance improvement and allow much better gpu acceleration.

I peeked through your code, I couldn't find a mistake. These errors are hard to track and they are usually due to some casting errors or other hidden logical mistakes. I'll try a deeper look again when have time. Great job adding the additional colors!

I solved the problem with saving and loading. There was a mistake in my code.


text:
key : on Azerty keyboard launches / to find

Hi,
Congratulations for this software and your work.
I use the latest version via a gentle git pull.
i meet this issue on every azerty keyboard (windows 10 or Ubuntu 18.04.02), when i type on the : key, Orca reacts as y type on the / character which is on a azerty keyboard the same key.
it's annoying to play just a little midi note.

An idea ?
Thanks

Just pushed an update that should fix that :)

Perfect ! Merci for french users !

authors: neauoire nlebellier

repo: hundredrabbits/Orca


text:
fix(cli): Use pathe instead of node:path

Fixes testing issues for the CLI for windows users.

Replaces node:path with pathe

Not sure if this is really important especially since it's an extra dependency for the CLI. But I tried it in one of my side projects and it worked without any issues.
The registry build has the exact same issue so we may be able to do the same there.
P.S.
Didn't add a changeset since there are no user facing changes lmk if I should add one anyways.

Thank you, but I would rather avoid adding another dependency if it's possible! The issue was that the aliases for hooks and ui weren't being normalized, so they were causing the tests to fail on windows. Opened #1516 as an alternate PR

package.json police 👮♂ strikes again! 😃
Gonna close this in favor of #1516 - thanks for bringing it up @ieedan

authors: AdrianGonz97 huntabyte ieedan

repo: huntabyte/shadcn-svelte


text:
🛑 recipes is down

In 47c1922, recipes (https://recipes.thekor.eu/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: recipes is back up in ded21dd after 34 minutes.

repo: hupratt/upptime


text:
🛑 retro is down

In 3038ad6, retro (https://retro.thekor.eu/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: retro is back up in 285a827 after 21 minutes.


text:
Docs are unclear to me

I've read the docs and viewed the demo code.
If I understand this correctly, the times that I need rendered are wrapped in  tags as in this example:
<time class="strong single">when you opened the page</time>
And to render timeago on a single element you call
timeago().render($('.single'));
What I fail to understand is how to get timeago to update on all elements of type  and especially those that are dynamically added as you would find in any feed. Do you have to append the new element to the nodes array?
function start_timeago(native) {
  var nodes = [];
	nodes = $('.timeago time');
	timeago().render(nodes);
}

Also, it doesn't look as though it can calculate timeago for divs with varying times (again, as you would find in a feed).

You should call timeago().render(element) for the new elements those are dynamically added.
The element must conrtains attributes named datetime.

authors: hustcc ppetree


text:
train.py gives me "loss is nan" as soon as it starts training

Thank you for your great work!
The NaN happens with both dnerf and hypernerf dataset.
It gives nan at the first loss.backward(), so I cannot do anything further🥹

Optimizing
Output folder: ./output/hypernerf/3dprinter [12/06 16:21:54]
feature_dim: 48 [12/06 16:21:54]
load finished [12/06 16:21:54]
207it [00:00, 172231.88it/s]
format finished [12/06 16:21:54]
100%|██████████████████████████████████████████████████████████████████████████████████████████████| 207/207 [00:02<00:00, 99.95it/s]
findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans.
findfont: Generic family 'sans-serif' not found because none of the following families were found: Times New Roman
Loading Training Cameras [12/06 16:21:56]
Loading Test Cameras [12/06 16:21:56]
Loading Video Cameras [12/06 16:21:56]
Deformation Net Set aabb [14.06674004 19.73840332 36.85457611] [-12.12202358 -15.1347599    4.79065514] [12/06 16:21:56]
Voxel Plane: set aabb= Parameter containing:
tensor([[ 14.0667,  19.7384,  36.8546],
[-12.1220, -15.1348,   4.7907]]) [12/06 16:21:56]
Number of points at initialisation :  96466 [12/06 16:21:57]
Training progress:   0%|                                                                                | 0/3000 [00:00<?, ?it/s]data loading done [12/06 16:21:57]
loss is nan,end training, reexecv program now. [12/06 16:21:57]
Optimizing
Output folder: ./output/hypernerf/3dprinter [12/06 16:21:59]
feature_dim: 48 [12/06 16:21:59]
load finished [12/06 16:21:59]
207it [00:00, 176827.07it/s]
format finished [12/06 16:21:59]
100%|██████████████████████████████████████████████████████████████████████████████████████████████| 207/207 [00:02<00:00, 98.22it/s]
findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans.
findfont: Generic family 'sans-serif' not found because none of the following families were found: Times New Roman
Loading Training Cameras [12/06 16:22:01]
Loading Test Cameras [12/06 16:22:01]
Loading Video Cameras [12/06 16:22:01]
Deformation Net Set aabb [14.06674004 19.73840332 36.85457611] [-12.12202358 -15.1347599    4.79065514] [12/06 16:22:01]
Voxel Plane: set aabb= Parameter containing:
tensor([[ 14.0667,  19.7384,  36.8546],
[-12.1220, -15.1348,   4.7907]]) [12/06 16:22:01]
Number of points at initialisation :  96466 [12/06 16:22:02]
Training progress:   0%|                                                                                | 0/3000 [00:00<?, ?it/s]data loading done [12/06 16:22:02]
loss is nan,end training, reexecv program now. [12/06 16:22:02]
Optimizing
Output folder: ./output/hypernerf/3dprinter [12/06 16:22:04]
feature_dim: 48 [12/06 16:22:04]
load finished [12/06 16:22:04]
207it [00:00, 173713.67it/s]
format finished [12/06 16:22:04]
100%|██████████████████████████████████████████████████████████████████████████████████████████████| 207/207 [00:02<00:00, 99.28it/s]
findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans.
findfont: Generic family 'sans-serif' not found because none of the following families were found: Times New Roman
Loading Training Cameras [12/06 16:22:06]
Loading Test Cameras [12/06 16:22:06]
Loading Video Cameras [12/06 16:22:06]
Deformation Net Set aabb [14.06674004 19.73840332 36.85457611] [-12.12202358 -15.1347599    4.79065514] [12/06 16:22:06]
Voxel Plane: set aabb= Parameter containing:
tensor([[ 14.0667,  19.7384,  36.8546],
[-12.1220, -15.1348,   4.7907]]) [12/06 16:22:06]
Number of points at initialisation :  96466 [12/06 16:22:06]

Hi,
Can you check your cuda version and pytorch version?
Most problems happens because of above issue :(

Yesss!!!
I was python 3.7, pytorch 1.13, cuda 11.7, and it produced NaN.
But when I changed into python 3.7, pytorch 1.13, cuda 11.6, it worked. Amazing.
Thanks for your help and it would be even great if you could put this information in the README to help the future users.
Thank you a lot.

authors: ChaerinMin guanjunwu


text:
安装 hwameistor 发现 sdb 磁盘未创建 pv, vg, 无可用容量

Output of lsblk:

Output of blkid:

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
DCE 5.0

local-storage logs:

The disk has GPT partition information, use the command to clean up the disk: wipefs -a /dev/sdb, and finally restart local-storage.

a GPT partition exist on disk may cause failure when vgcreate, the LocalDisk object should show Bound status.
i think we should fix this.

one suggestion: when discover a disk, check if there is any partition on the disk, if yes, mark the disk/LD as unavailable with the reason. After clean up it manually, make it available then.

authors: Archimedes-Dear FloatXD SSmallMonster sun7927

repo: hwameistor/hwameistor


text:
Applications cannot be scheduled normally in a namespace other than hwameistor

@FloatXD It looks like missing ns field when search a namespacd resource, may be caused by different version of kubenetes,.

authors: FloatXD SSmallMonster


text:
SQLDatabaseChain runs under the GPT4 model and reports an error

System Info
my python code
from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
from langchain.chains import SQLDatabaseSequentialChain
db = SQLDatabase.from_uri("clickhouse://xx:xx@ip/db",
include_tables=include_tables,
custom_table_info=custom_table_schemas,
sample_rows_in_table_info=2)
llm = OpenAI(temperature=0, model_name="gpt-4-0613", verbose=True, streaming=True, openai_api_base="https://xxx.cn/v1")
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True, use_query_checker=True, return_intermediate_steps=True, top_k=3)
instruction = "Statistics EL_C1 device uptime today"
result = db_chain(instruction)
result["intermediate_steps"]
run result

DatabaseException: Orig exception: Code: 62. DB::Exception: Syntax error: failed at position 1 ('The') (line 1, col 1): The original query seems to be correct as it doesn't have any of the common mistakes mentioned. Here is the reproduction of the original query:
I don't know why these strings are being run as SQL
if set use_query_checker = False

There is an extra double quote in SQL
DatabaseException: Orig exception: Code: 62. DB::Exception: Syntax error: failed at position 1 ('"SELECT SUM(value) FROM idap_asset.EL_MODEL_Run_Time WHERE asset_code = 'EL_C1' AND toDate(CAST(window_end / 1000, 'DateTime')) = today()"'): "SELECT SUM(value) FROM idap_asset.EL_MODEL_Run_Time WHERE asset_code = 'EL_C1' AND toDate(CAST(window_end / 1000, 'DateTime')) = today()".
All these errors are only generated under the GPT4 model. If the default model is used, no errors are generated
Who can help?
No response
Information

[ ] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[X] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
llm = OpenAI(temperature=0, model_name="gpt-4-0613", verbose=True, streaming=True, openai_api_base="https://xxx.cn/v1")
db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True, use_query_checker=False, return_intermediate_steps=True, top_k=3)
As long as the GPT4 model is used, there will be problems through the db chain, and the string returned by the model will be executed as SQL
Expected behavior

The hope is to generate correct SQL execution, do not treat the string returned by the model as SQL execution

@nickscut , Can you please post the Traceback of your Error.

Entering new  chain...
统计EL_C1设备今天正常运行时间
SQLQuery:"SELECT SUM(value) FROM idap_asset.EL_MODEL_Run_Time WHERE asset_code = 'EL_C1' AND toDate(CAST(window_end / 1000, 'DateTime')) = today()"

DatabaseException                         Traceback (most recent call last)
Cell In[332], line 19
13 instruction = "统计EL_C1设备今天正常运行时间"
---> 19 result = db_chain(instruction)
20 result["intermediate_steps"]
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py:166, in Chain.call(self, inputs, return_only_outputs, callbacks, tags, include_run_info)
164 except (KeyboardInterrupt, Exception) as e:
165     run_manager.on_chain_error(e)
--> 166     raise e
167 run_manager.on_chain_end(outputs)
168 final_outputs: Dict[str, Any] = self.prep_outputs(
169     inputs, outputs, return_only_outputs
170 )
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py:160, in Chain.call(self, inputs, return_only_outputs, callbacks, tags, include_run_info)
154 run_manager = callback_manager.on_chain_start(
155     dumpd(self),
156     inputs,
157 )
158 try:
159     outputs = (
--> 160         self._call(inputs, run_manager=run_manager)
161         if new_arg_supported
162         else self._call(inputs)
163     )
164 except (KeyboardInterrupt, Exception) as e:
165     run_manager.on_chain_error(e)
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/sql_database/base.py:181, in SQLDatabaseChain._call(self, inputs, run_manager)
177 except Exception as exc:
178     # Append intermediate steps to exception, to aid in logging and later
179     # improvement of few shot prompt seeds
180     exc.intermediate_steps = intermediate_steps  # type: ignore
--> 181     raise exc
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/sql_database/base.py:126, in SQLDatabaseChain._call(self, inputs, run_manager)
122     intermediate_steps.append(
123         sql_cmd
124     )  # output: sql generation (no checker)
125     intermediate_steps.append({"sql_cmd": sql_cmd})  # input: sql exec
--> 126     result = self.database.run(sql_cmd)
127     intermediate_steps.append(str(result))  # output: sql exec
128 else:
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/sql_database.py:348, in SQLDatabase.run(self, command, fetch)
346     else:
347         connection.exec_driver_sql(f"SET search_path TO {self._schema}")
--> 348 cursor = connection.execute(text(command))
349 if cursor.returns_rows:
350     if fetch == "all":
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1306, in Connection.execute(self, statement, *multiparams, **params)
1302     util.raise_(
1303         exc.ObjectNotExecutableError(statement), replace_context=err
1304     )
1305 else:
-> 1306     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/sql/elements.py:332, in ClauseElement._execute_on_connection(self, connection, multiparams, params, execution_options, _force)
328 def _execute_on_connection(
329     self, connection, multiparams, params, execution_options, _force=False
330 ):
331     if _force or self.supports_execution:
--> 332         return connection._execute_clauseelement(
333             self, multiparams, params, execution_options
334         )
335     else:
336         raise exc.ObjectNotExecutableError(self)
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1498, in Connection._execute_clauseelement(self, elem, multiparams, params, execution_options)
1486 compiled_cache = execution_options.get(
1487     "compiled_cache", self.engine._compiled_cache
1488 )
1490 compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
1491     dialect=dialect,
1492     compiled_cache=compiled_cache,
(...)
1496     linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
1497 )
-> 1498 ret = self._execute_context(
1499     dialect,
1500     dialect.execution_ctx_cls._init_compiled,
1501     compiled_sql,
1502     distilled_params,
1503     execution_options,
1504     compiled_sql,
1505     distilled_params,
1506     elem,
1507     extracted_params,
1508     cache_hit=cache_hit,
1509 )
1510 if has_events:
1511     self.dispatch.after_execute(
1512         self,
1513         elem,
(...)
1517         ret,
1518     )
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1862, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
1859             branched.close()
1861 except BaseException as e:
-> 1862     self._handle_dbapi_exception(
1863         e, statement, parameters, cursor, context
1864     )
1866 return result
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2047, in Connection.handle_dbapi_exception(self, e, statement, parameters, cursor, context)
2043         util.raise(
2044             sqlalchemy_exception, with_traceback=exc_info[2], from_=e
2045         )
2046     else:
-> 2047         util.raise_(exc_info[1], with_traceback=exc_info[2])
2049 finally:
2050     del self._reentrant_error
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py:208, in raise_(failed resolving arguments)
205     exception.cause = replace_context
207 try:
--> 208     raise exception
209 finally:
210     # credit to
211     # https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
212     # as the traceback object creates a cycle
213     del exception, replace_context, from_, with_traceback
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1819, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
1817                 break
1818     if not evt_handled:
-> 1819         self.dialect.do_execute(
1820             cursor, statement, parameters, context
1821         )
1823 if self._has_events or self.engine._has_events:
1824     self.dispatch.after_cursor_execute(
1825         self,
1826         cursor,
(...)
1830         context.executemany,
1831     )
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/base.py:416, in ClickHouseDialect.do_execute(self, cursor, statement, parameters, context)
415 def do_execute(self, cursor, statement, parameters, context=None):
--> 416     cursor.execute(statement, parameters, context=context)
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/connector.py:117, in Cursor.execute(self, operation, parameters, context)
114 params = {'query_id': self._query_id}
115 response_gen = transport.execute(raw_sql, params=params)
--> 117 self._process_response(response_gen)
118 self._end_query()
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/connector.py:216, in Cursor._process_response(self, response)
213 def _process_response(self, response):
214     response = iter(response)
--> 216     self._columns = next(response, None)
217     self._types = next(response, None)
218     self._response = response
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/transport.py:133, in RequestsTransport.execute(self, query, params)
128 def execute(self, query, params=None):
129     """
130     Query is returning rows and these rows should be parsed or
131     there is nothing to return.
132     """
--> 133     r = self._send(query, params=params, stream=True)
134     lines = r.iter_lines()
135     try:
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/transport.py:179, in RequestsTransport._send(self, data, params, stream)
177     orig = HTTPException(r.text)
178     orig.code = r.status_code
--> 179     raise DatabaseException(orig)
180 return r
DatabaseException: Orig exception: Code: 62. DB::Exception: Syntax error: failed at position 1 ('"SELECT SUM(value) FROM idap_asset.EL_MODEL_Run_Time WHERE asset_code = 'EL_C1' AND toDate(CAST(window_end / 1000, 'DateTime')) = today()"'): "SELECT SUM(value) FROM idap_asset.EL_MODEL_Run_Time WHERE asset_code = 'EL_C1' AND toDate(CAST(window_end / 1000, 'DateTime')) = today()". Expected one of: Query, Query with output, EXPLAIN, SELECT query, possibly with UNION, list of union elements, SELECT query, subquery, possibly with UNION, SELECT subquery, SELECT query, WITH, SELECT, SHOW CREATE QUOTA query, SHOW CREATE, SHOW [TEMPORARY] TABLES|DATABASES|CLUSTERS|CLUSTER 'name' [[NOT] [I]LIKE 'str'] [LIMIT expr], SHOW, EXISTS or SHOW CREATE query, EXISTS, DESCRIBE query, DESCRIBE, DESC, SHOW PROCESSLIST query, SHOW PROCESSLIST, CREATE TABLE or ATTACH TABLE query, CREATE, ATTACH, REPLACE, CREATE DATABASE query, CREATE VIEW query, CREATE DICTIONARY, CREATE LIVE VIEW query, CREATE WINDOW VIEW query, ALTER query, ALTER TABLE, ALTER LIVE VIEW, ALTER DATABASE, RENAME query, RENAME TABLE, EXCHANGE TABLES, RENAME DICTIONARY, EXCHANGE DICTIONARIES, RENAME DATABASE, DROP query, DROP, DETACH, TRUNCATE, CHECK TABLE, KILL QUERY query, KILL, OPTIMIZE query, OPTIMIZE TABLE, WATCH query, WATCH, SHOW ACCESS query, SHOW ACCESS, ShowAccessEntitiesQuery, SHOW GRANTS query, SHOW GRANTS, SHOW PRIVILEGES query, SHOW PRIVILEGES, INSERT query, INSERT INTO, USE query, USE, SET ROLE or SET DEFAULT ROLE query, SET ROLE DEFAULT, SET ROLE, SET DEFAULT ROLE, SET query, SET, SYSTEM query, SYSTEM, CREATE USER or ALTER USER query, ALTER USER, CREATE USER, CREATE ROLE or ALTER ROLE query, ALTER ROLE, CREATE ROLE, CREATE QUOTA or ALTER QUOTA query, ALTER QUOTA, CREATE QUOTA, CREATE ROW POLICY or ALTER ROW POLICY query, ALTER POLICY, ALTER ROW POLICY, CREATE POLICY, CREATE ROW POLICY, CREATE SETTINGS PROFILE or ALTER SETTINGS PROFILE query, ALTER SETTINGS PROFILE, ALTER PROFILE, CREATE SETTINGS PROFILE, CREATE PROFILE, CREATE FUNCTION query, DROP FUNCTION query, DROP access entity query, GRANT or REVOKE query, REVOKE, GRANT, EXTERNAL DDL query, EXTERNAL DDL FROM, TCL query, BEGIN TRANSACTION, COMMIT, ROLLBACK, SET TRANSACTION SNAPSHOT, BACKUP or RESTORE query, BACKUP, RESTORE. (SYNTAX_ERROR) (version 22.5.1.2079 (official build))

@keenborder786

db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True, use_query_checker=True, return_intermediate_steps=True, top_k=3)
if set use_query_checker=True
then Traceback of my Error  @keenborder786

Entering new  chain...
统计EL_C1设备今天正常运行时间
SQLQuery:The original query seems to be correct as it doesn't contain any of the common mistakes listed. Here is the reproduction of the original query:

"SELECT asset_code, value FROM idap_asset.EL_MODEL_Run_Time WHERE asset_code = 'EL_C1' AND toDate(CAST(window_end / 1000, 'DateTime')) = today() LIMIT 3"
DatabaseException                         Traceback (most recent call last)
Cell In[334], line 19
13 instruction = "统计EL_C1设备今天正常运行时间"
---> 19 result = db_chain(instruction)
20 result["intermediate_steps"]
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py:166, in Chain.call(self, inputs, return_only_outputs, callbacks, tags, include_run_info)
164 except (KeyboardInterrupt, Exception) as e:
165     run_manager.on_chain_error(e)
--> 166     raise e
167 run_manager.on_chain_end(outputs)
168 final_outputs: Dict[str, Any] = self.prep_outputs(
169     inputs, outputs, return_only_outputs
170 )
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/base.py:160, in Chain.call(self, inputs, return_only_outputs, callbacks, tags, include_run_info)
154 run_manager = callback_manager.on_chain_start(
155     dumpd(self),
156     inputs,
157 )
158 try:
159     outputs = (
--> 160         self._call(inputs, run_manager=run_manager)
161         if new_arg_supported
162         else self._call(inputs)
163     )
164 except (KeyboardInterrupt, Exception) as e:
165     run_manager.on_chain_error(e)
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/sql_database/base.py:181, in SQLDatabaseChain._call(self, inputs, run_manager)
177 except Exception as exc:
178     # Append intermediate steps to exception, to aid in logging and later
179     # improvement of few shot prompt seeds
180     exc.intermediate_steps = intermediate_steps  # type: ignore
--> 181     raise exc
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/chains/sql_database/base.py:151, in SQLDatabaseChain._call(self, inputs, run_manager)
145 _run_manager.on_text(
146     checked_sql_command, color="green", verbose=self.verbose
147 )
148 intermediate_steps.append(
149     {"sql_cmd": checked_sql_command}
150 )  # input: sql exec
--> 151 result = self.database.run(checked_sql_command)
152 intermediate_steps.append(str(result))  # output: sql exec
153 sql_cmd = checked_sql_command
File ~/opt/anaconda3/lib/python3.9/site-packages/langchain/sql_database.py:348, in SQLDatabase.run(self, command, fetch)
346     else:
347         connection.exec_driver_sql(f"SET search_path TO {self._schema}")
--> 348 cursor = connection.execute(text(command))
349 if cursor.returns_rows:
350     if fetch == "all":
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1306, in Connection.execute(self, statement, *multiparams, **params)
1302     util.raise_(
1303         exc.ObjectNotExecutableError(statement), replace_context=err
1304     )
1305 else:
-> 1306     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/sql/elements.py:332, in ClauseElement._execute_on_connection(self, connection, multiparams, params, execution_options, _force)
328 def _execute_on_connection(
329     self, connection, multiparams, params, execution_options, _force=False
330 ):
331     if _force or self.supports_execution:
--> 332         return connection._execute_clauseelement(
333             self, multiparams, params, execution_options
334         )
335     else:
336         raise exc.ObjectNotExecutableError(self)
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1498, in Connection._execute_clauseelement(self, elem, multiparams, params, execution_options)
1486 compiled_cache = execution_options.get(
1487     "compiled_cache", self.engine._compiled_cache
1488 )
1490 compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
1491     dialect=dialect,
1492     compiled_cache=compiled_cache,
(...)
1496     linting=self.dialect.compiler_linting | compiler.WARN_LINTING,
1497 )
-> 1498 ret = self._execute_context(
1499     dialect,
1500     dialect.execution_ctx_cls._init_compiled,
1501     compiled_sql,
1502     distilled_params,
1503     execution_options,
1504     compiled_sql,
1505     distilled_params,
1506     elem,
1507     extracted_params,
1508     cache_hit=cache_hit,
1509 )
1510 if has_events:
1511     self.dispatch.after_execute(
1512         self,
1513         elem,
(...)
1517         ret,
1518     )
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1862, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
1859             branched.close()
1861 except BaseException as e:
-> 1862     self._handle_dbapi_exception(
1863         e, statement, parameters, cursor, context
1864     )
1866 return result
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:2047, in Connection.handle_dbapi_exception(self, e, statement, parameters, cursor, context)
2043         util.raise(
2044             sqlalchemy_exception, with_traceback=exc_info[2], from_=e
2045         )
2046     else:
-> 2047         util.raise_(exc_info[1], with_traceback=exc_info[2])
2049 finally:
2050     del self._reentrant_error
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/util/compat.py:208, in raise_(failed resolving arguments)
205     exception.cause = replace_context
207 try:
--> 208     raise exception
209 finally:
210     # credit to
211     # https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/
212     # as the traceback object creates a cycle
213     del exception, replace_context, from_, with_traceback
File ~/opt/anaconda3/lib/python3.9/site-packages/sqlalchemy/engine/base.py:1819, in Connection._execute_context(self, dialect, constructor, statement, parameters, execution_options, *args, **kw)
1817                 break
1818     if not evt_handled:
-> 1819         self.dialect.do_execute(
1820             cursor, statement, parameters, context
1821         )
1823 if self._has_events or self.engine._has_events:
1824     self.dispatch.after_cursor_execute(
1825         self,
1826         cursor,
(...)
1830         context.executemany,
1831     )
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/base.py:416, in ClickHouseDialect.do_execute(self, cursor, statement, parameters, context)
415 def do_execute(self, cursor, statement, parameters, context=None):
--> 416     cursor.execute(statement, parameters, context=context)
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/connector.py:117, in Cursor.execute(self, operation, parameters, context)
114 params = {'query_id': self._query_id}
115 response_gen = transport.execute(raw_sql, params=params)
--> 117 self._process_response(response_gen)
118 self._end_query()
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/connector.py:216, in Cursor._process_response(self, response)
213 def _process_response(self, response):
214     response = iter(response)
--> 216     self._columns = next(response, None)
217     self._types = next(response, None)
218     self._response = response
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/transport.py:133, in RequestsTransport.execute(self, query, params)
128 def execute(self, query, params=None):
129     """
130     Query is returning rows and these rows should be parsed or
131     there is nothing to return.
132     """
--> 133     r = self._send(query, params=params, stream=True)
134     lines = r.iter_lines()
135     try:
File ~/opt/anaconda3/lib/python3.9/site-packages/clickhouse_sqlalchemy/drivers/http/transport.py:179, in RequestsTransport._send(self, data, params, stream)
177     orig = HTTPException(r.text)
178     orig.code = r.status_code
--> 179     raise DatabaseException(orig)
180 return r
DatabaseException: Orig exception: Code: 62. DB::Exception: Syntax error: failed at position 1 ('The') (line 1, col 1): The original query seems to be correct as it doesn't contain any of the common mistakes listed. Here is the reproduction of the original query:
"SELECT `asset_. Expected one of: Query, Query with output, EXPLAIN, SELECT query, possibly with UNION, list of union elements, SELECT query, subquery, possibly with UNION, SELECT subquery, SELECT query, WITH, SELECT, SHOW CREATE QUOTA query, SHOW CREATE, SHOW [TEMPORARY] TABLES|DATABASES|CLUSTERS|CLUSTER 'name' [[NOT] [I]LIKE 'str'] [LIMIT expr], SHOW, EXISTS or SHOW CREATE query, EXISTS, DESCRIBE query, DESCRIBE, DESC, SHOW PROCESSLIST query, SHOW PROCESSLIST, CREATE TABLE or ATTACH TABLE query, CREATE, ATTACH, REPLACE, CREATE DATABASE query, CREATE VIEW query, CREATE DICTIONARY, CREATE LIVE VIEW query, CREATE WINDOW VIEW query, ALTER query, ALTER TABLE, ALTER LIVE VIEW, ALTER DATABASE, RENAME query, RENAME TABLE, EXCHANGE TABLES, RENAME DICTIONARY, EXCHANGE DICTIONARIES, RENAME DATABASE, DROP query, DROP, DETACH, TRUNCATE, CHECK TABLE, KILL QUERY query, KILL, OPTIMIZE query, OPTIMIZE TABLE, WATCH query, WATCH, SHOW ACCESS query, SHOW ACCESS, ShowAccessEntitiesQuery, SHOW GRANTS query, SHOW GRANTS, SHOW PRIVILEGES query, SHOW PRIVILEGES, INSERT query, INSERT INTO, USE query, USE, SET ROLE or SET DEFAULT ROLE query, SET ROLE DEFAULT, SET ROLE, SET DEFAULT ROLE, SET query, SET, SYSTEM query, SYSTEM, CREATE USER or ALTER USER query, ALTER USER, CREATE USER, CREATE ROLE or ALTER ROLE query, ALTER ROLE, CREATE ROLE, CREATE QUOTA or ALTER QUOTA query, ALTER QUOTA, CREATE QUOTA, CREATE ROW POLICY or ALTER ROW POLICY query, ALTER POLICY, ALTER ROW POLICY, CREATE POLICY, CREATE ROW POLICY, CREATE SETTINGS PROFILE or ALTER SETTINGS PROFILE query, ALTER SETTINGS PROFILE, ALTER PROFILE, CREATE SETTINGS PROFILE, CREATE PROFILE, CREATE FUNCTION query, DROP FUNCTION query, DROP access entity query, GRANT or REVOKE query, REVOKE, GRANT, EXTERNAL DDL query, EXTERNAL DDL FROM, TCL query, BEGIN TRANSACTION, COMMIT, ROLLBACK, SET TRANSACTION SNAPSHOT, BACKUP or RESTORE query, BACKUP, RESTORE. (SYNTAX_ERROR) (version 22.5.1.2079 (official build))

I have tried to solve this problem
modify：langchain/chains/sql_database/base.py
@staticmethod
def extract_sql_command(text: str) -> str:
    # Create a regular expression pattern that matches the SQL command
    pattern = r"(SELECT[\s\S]*LIMIT \d+)"

    # Use the re.search function to search the text for the pattern
    match = re.search(pattern, text, re.IGNORECASE)

    # If a match was found, return the matched string
    if match:
        return match.group(1).strip()

    # If no match was found, return the original text
    return text

checked_sql_command: str = self.extract_sql_command(query_checker_chain.predict(
callbacks=_run_manager.get_child(), **query_checker_inputs
).strip())
it works fine for gpt4 model

@nickscut thank you will have a look

@nickscut thank you will have a look

Had a look in the past few days. Basically it's the disparity between completion models and chat models. Whilst completion models such as da-vinci simply reproduces the SQL query following instruction, chat models like gpt-3.5 or gpt-4 tend to add comments upfront to explain the result.
I have tried to tune tools/sqldatabase/prompt.py to suppress the excessive output, but in vain. Regardless how I changed, chat models always add something before the SQL.
So to me, it seems one of the following options:

Code a pattern recognition same as OP, assuming the models wouldn't add any trailing message after SQL
For chat models, add a system prompt to tell them only respond with SQL
Modify the QUERY_CHECKER prompt, so the models just 'say Yes the query is correct', without reproducing the original SQL
Lastly, if someone can workout a prompt that both works for completion model and suppresses the excessive output for chat models.

I wrote a robust version, which can even support with clause or other ddl. It can handle various kind of text result no matter how model/prompt changes
#!pip install -q sqlfluff
#!pip install -q sqlparse
import re
import sqlfluff
import sqlparse

@staticmethod
def extract_sql_command(text: str) -> str:
    res = []
    T = sqlparse.tokens._TokenType()
    error_line_pattern = r"Line (\d+), Position (\d+): (.*)"
    for i in range(len(text)-1):
        # check the text character by character
        current_statement = text[i:]
        parsed = sqlparse.parse(current_statement)[0]
        # filter based on the token type only allow DML, DDL, CTE to continue
        if parsed.tokens[0].ttype in (T.Keyword.DML, T.Keyword.DDL, T.Keyword.CTE):
            # try to use sqlfluff to check the syntax
            candidate_sql = ""
            current_lines = current_statement.splitlines()
            try:
                sqlfluff.parse(current_statement)
                candidate_sql = current_statement
            except sqlfluff.api.simple.APIParsingError as e:
                # find out first violation's line and position
                # parse the error message
                msg_lines = str(e.msg).split("\n")
                for j in range(len(msg_lines)):
                    if re.search(error_line_pattern, msg_lines[j]):
                        line_no, position = str(msg_lines[j]).split(":")[0].split(",")
                        line_no, position = int(line_no.strip().split(' ')[1]), int(position.strip().split(' ')[1])
                        # extract the candidate sql
                        for k in range(line_no-1):
                            candidate_sql += current_lines[k] + "\n"
                        candidate_sql += current_lines[line_no-1][:position-1]
            # for shorter statement, use length filter
            try:
                parsed = sqlparse.parse(candidate_sql)[0]
                if len(candidate_sql.split()) <= 3:
                    if parsed.tokens[0].ttype in (T.Keyword.DML):
                        continue
                if len(candidate_sql.split()) <= 2:
                    continue
                res.append(candidate_sql)
            except:
                pass
    if len(res)==0:
        return ""
    else: # return the longest
        return max(res, key=len)

I have a robust version of extract_sql_command(...) for SQL extraction from any sql, text mixture. If you modify the code and directly return res, you can get all valid sql found. This method support wide ranges of sql statement including DML, DDL etc.
#!pip install -q sqlfluff>=2.1.3
#!pip install -q sqlparse>=0.4.4
import re
import sqlfluff
import sqlparse

@staticmethod
def extract_sql_command(text: str) -> str:
    # concatenate result from sqlfluff.parse
    def concat_dict(dict_data):
        result = ""
        for value in dict_data.values():
            if isinstance(value, dict):
                result += concat_dict(value)
            elif isinstance(value, list) or isinstance(value, tuple):
                for i in range(len(value)):
                    result += concat_dict(value[i])
            else:
                result += str(value)
        return result
    text = text.replace("\r\n","\n")
    res = []
    T = sqlparse.tokens._TokenType()
    error_line_pattern = r"Line (\d+), Position (\d+): (.*)"
    i = 0
    while i < len(text)-2:
        # check the text character by character
        current_statement = text[i:]
        parsed = sqlparse.parse(current_statement)[0]
        # filter based on the token type only allow DML, DDL, CTE to continue
        if parsed.tokens[0].ttype in (T.Keyword.DML, T.Keyword.DDL, T.Keyword.CTE):
            # try to use sqlfluff to check the syntax
            candidate_sql = ""
            current_lines = current_statement.splitlines()
            try:
                sqlfluff.parse(current_statement, dialect='ansi')
                candidate_sql = current_statement
            except sqlfluff.api.simple.APIParsingError as e:
                # find out first violation's line and position
                # parse the error message
                msg_lines = str(e.msg).split("\n")
                for j in range(len(msg_lines)):
                    if re.search(error_line_pattern, msg_lines[j]):
                        line_no, position = str(msg_lines[j]).split(":")[0].split(",")
                        line_no, position = int(line_no.strip().split(' ')[1]), int(position.strip().split(' ')[1])
                        # extract the candidate sql
                        for k in range(line_no-1):
                            candidate_sql += current_lines[k] + "\n"
                        candidate_sql += current_lines[line_no-1][:position-1]
                        break
            # for shorter statement, use length filter
            try:
                parsed_res = sqlfluff.parse(candidate_sql, dialect='ansi')
                # retrieve the first sql, sometimes it returns multiple sql separated by ;
                if isinstance(parsed_res['file'], list):
                    candidate_sql = concat_dict(parsed_res['file'][0])
                parsed = sqlparse.parse(candidate_sql)[0]
                if len(candidate_sql.split()) <= 3 and parsed.tokens[0].ttype in (T.Keyword.DML):
                    pass
                elif len(candidate_sql.split()) <= 2:
                    pass
                else:
                    i += len(candidate_sql) - 1
                    res.append(candidate_sql)
            except Exception as e:
                pass
        i += 1
    if len(res)==0:
        return ""
    else: # return the longest
        return max(res, key=len)

Hi @nickscut, I hope it's not to late to share my methods. I'm also doing a projet using SQLDatabaseChain to create a chatbot to execute SQL query with the model gpt4.
I faced these problems, too. The way how I solved these problems is just specify the prompt.
Actually, SQLDatabaseChain does no longer accept the parameter of 'prompt'. But we can still define a template and use format.() or just use f-string to form the question. In the prompt of SQLDatabaseChain, You can ask the model to generate a SQL Query without any decoration (for example, Markdown format).
And for the problem that the model generate the general text, which will pass as a SQL query. You can give the description of table (description of each column). And ask the model to generate the SQL query only base on this description. And then the model will generate a correct query tothe chain, which can return correct results.
This is the way I found easiest. I hope it can help you and others who may have the similar problem.

I had a similar issue with backticks appearing in the SQL generated,
I have adjusted the system prompt as below and it is working now.
Before:
def parse_final_answer(output: str) -> str:
return output.split("Final answer: ")[1]
execute_query = QuerySQLDataBaseTool(db=db,use_query_checker=True,response_format='content')
write_query = create_sql_query_chain(llm, db)
chain = full_chain | execute_query
After:
def parse_final_answer(output: str) -> str:
return output.split("Final answer: ")[1]
full_chain = create_sql_query_chain(llm, db, prompt=prompt) | parse_final_answer
execute_query = QuerySQLDataBaseTool(db=db,use_query_checker=True,response_format='content')
write_query = full_chain
chain = write_query | execute_query
---Below is the full system prompt used------
from langchain_community.tools.sql_database.tool import QuerySQLDataBaseTool
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
system = """You are a {dialect} expert. Given an input question, create a syntactically correct {dialect} query to run.
Unless the user specifies in the question a specific number of examples to obtain, query for at most {top_k} results using the LIMIT clause as per {dialect}. You can order the results to return the most informative data in the database.
Never query for all columns from a table. You must query only the columns that are needed to answer the question. Wrap each column name in double quotes (") to denote them as delimited identifiers.
Pay attention to use only the column names you can see in the tables below. Be careful to not query for columns that do not exist. Also, pay attention to which column is in which table.
Pay attention to use date('now') function to get the current date, if the question involves "today".
Only use the following tables:
{table_info}
Write an initial draft of the query. Then double check the {dialect} query for common mistakes, including:

Using NOT IN with NULL values
Using UNION when UNION ALL should have been used
Using BETWEEN for exclusive ranges
Data type mismatch in predicates
Properly quoting identifiers
Using the correct number of arguments for functions
Casting to the correct data type
Using the proper columns for joins

If there are any of the above mistakes, rewrite the query.Do not include characters  in the beginning and end of the sql. Remove any unwanted characters like back ticks :,;, in the beginning and end.
If there are no mistakes, just reproduce the original query with no further commentary.
Use format:
First draft: <<FIRST_DRAFT_QUERY>>
Final answer: <<FINAL_ANSWER_QUERY>>
"""
prompt = ChatPromptTemplate.from_messages(
[("system", system), ("human", "{input}")]
).partial(dialect=db.dialect)
def parse_final_answer(output: str) -> str:
return output.split("Final answer: ")[1]
full_chain = create_sql_query_chain(llm, db, prompt=prompt) | parse_final_answer
execute_query = QuerySQLDataBaseTool(db=db,use_query_checker=True,response_format='content')
write_query = full_chain
chain = write_query | execute_query

authors: Erickrus JUNNIEJUN keenborder786 nickscut praveen-prog yuan-thomas


text:
Training accuracy is coming 0.

While training I am getting 0 accuracy, is there a bug or I am misinterpreting something??

Could you give more info?

There was some problem when the accuracy was being returned by the _eval function in builder.py. Found out that it was because 'correct_n' was of type integer and while finding accuracy correct_n[0]/n was becoming 0. Just changed the type of correct_n as follows "correct_n = correct_n.float()".

Do you use python2?

No, I am using python 3.6

authors: hxbai vikas-baghel


text:
Can not select other points?

This is what I want! Great work, thanks!
Code as document demo, but can not select other point, I don't know why.

My demo

@zerosoul I am looking at your demo now and selecting points fine. Maybe its browser specific? Could you please comment with your machine, browser etc.

@zerosoul sorry for the delay. Are you still seeing this behavior?

Closing as stale


text:
Failed Build: Error while merging dex archives

I experienced failed build due package name. Here the error I get when using v1.0.12:
ERROR
> Task :app:mergeDexStagsecondStagingsec FAILED
~/project-name/node_modules/react-native-disable-battery-optimizations-android/android/build/.transforms/2f8d9ac22f329154eba2ca6aec913408/classes/classes.dex: D8: Type com.reactlibrary.BuildConfig is defined multiple times: ~/project-name/node_modules/react-native-disable-battery-optimizations-android/android/build/.transforms/2f8d9ac22f329154eba2ca6aec913408/classes/classes.dex, ~/project-name/node_modules/react-native-floating-bubble/android/build/.transforms/4f404a63be8db34e91eef9a57a90a10d/classes/classes.dex
com.android.builder.dexing.DexArchiveMergerException: Error while merging dex archives:
Learn how to resolve the issue at https://developer.android.com/studio/build/dependencies#duplicate_classes.
Type com.reactlibrary.BuildConfig is defined multiple times: ~/project-name/node_modules/react-native-disable-battery-optimizations-android/android/build/.transforms/2f8d9ac22f329154eba2ca6aec913408/classes/classes.dex, ~/project-name/node_modules/react-native-floating-bubble/android/build/.transforms/4f404a63be8db34e91eef9a57a90a10d/classes/classes.dex

Related error: https://stackoverflow.com/questions/60507686/type-buildconfig-is-defined-multiple-times
CAUSE
Because the package using default name com.reactlibrary which conflict with other library.
SOLUTION
I manage to make it work by changing 3 files: https://github.com/huedaya/react-native-floating-bubble/commit/c6d98b6eb8593d42c631eef8f799d59b56389d41

AndroidManifest.xml
RNFloatingBubbleModule.java
RNFloatingBubblePackage.java

TEMPORARY SOLUTION
For who need quick solution, you can use my fork by changing the package.json:
{
    "dependencies": {
        ...
        "react-native-floating-bubble": "git+https://github.com/huedaya/react-native-floating-bubble",
    }
 }

@janakact

I experienced failed build due package name. Here the error I get when using v1.0.12:
ERROR
> Task :app:mergeDexStagsecondStagingsec FAILED
~/project-name/node_modules/react-native-disable-battery-optimizations-android/android/build/.transforms/2f8d9ac22f329154eba2ca6aec913408/classes/classes.dex: D8: Type com.reactlibrary.BuildConfig is defined multiple times: ~/project-name/node_modules/react-native-disable-battery-optimizations-android/android/build/.transforms/2f8d9ac22f329154eba2ca6aec913408/classes/classes.dex, ~/project-name/node_modules/react-native-floating-bubble/android/build/.transforms/4f404a63be8db34e91eef9a57a90a10d/classes/classes.dex
com.android.builder.dexing.DexArchiveMergerException: Error while merging dex archives:
Learn how to resolve the issue at https://developer.android.com/studio/build/dependencies#duplicate_classes.
Type com.reactlibrary.BuildConfig is defined multiple times: ~/project-name/node_modules/react-native-disable-battery-optimizations-android/android/build/.transforms/2f8d9ac22f329154eba2ca6aec913408/classes/classes.dex, ~/project-name/node_modules/react-native-floating-bubble/android/build/.transforms/4f404a63be8db34e91eef9a57a90a10d/classes/classes.dex

Related error: https://stackoverflow.com/questions/60507686/type-buildconfig-is-defined-multiple-times
CAUSE Because the package using default name com.reactlibrary which conflict with other library.
SOLUTION I manage to make it work by changing 3 files: huedaya@c6d98b6

AndroidManifest.xml
RNFloatingBubbleModule.java
RNFloatingBubblePackage.java

TEMPORARY SOLUTION For who need quick solution, you can use my fork by changing the package.json:
{
    "dependencies": {
        ...
        "react-native-floating-bubble": "git+https://github.com/huedaya/react-native-floating-bubble",
    }
 }

Thanks , That Working nice

authors: HosamCS huedaya

repo: hybriteq/react-native-floating-bubble


text:
[HELP] Cant find sources

Is your feature request related to a problem? Please describe.
no
Describe the solution you'd like
I cant find where to get the sources to add to the launcher and when i clicked the discord link it said that it had expired.
Describe alternatives you've considered
No response
Additional context
No response

Hi, it works for me:
Go to https://hydralinks.cloud/
copy to clipboard all the sources and in Settings > Download Source, import all the files from hydra website

repo: hydralauncher/hydra


text:
Question: Make HFollow with custom coordinates instead of mouse

I want to make an HFollow and a HRect to follow it. But instead of following the mouse I want to build the  HFollow with custom coordinates ( e.g. mouseX + 50, mouseY + 50).
Is there a way i can achieve this? In the docs I see a reference to a HLocatable. Do I need to build a custom HLocatable with my coordinates and pass it to the HFollow?

import hype.*;
import hype.extended.behavior.HFollow;

HFollow mf;
HRect   r;

void setup(){
	size(900,900);
	H.init(this).background(#242424);

	H.add( r = new HRect(100) ).noStroke().fill(#FF3300).anchor(-50,-50);
	mf = new HFollow().target(r).ease(0.05).spring(0.95);
}

void draw(){
	H.drawStage();
}

the trick is changing the anchor position of the rect / .anchor(-50,-50)

BTW what HLocatable allows you to do is this...
println( mf.goal().x() );
so you could also...
import hype.*;
import hype.extended.behavior.HFollow;

HFollow mf;
HRect   r;
float   dx, dy;

void setup(){
	size(900,900);
	H.init(this).background(#242424);

	H.add( r = new HRect(100) ).noStroke().fill(#FF3300);
	mf = new HFollow().ease(0.05).spring(0.95);
}

void draw(){
	H.drawStage();

	float xPos = mf.goal().x() + 50;
	float yPos = mf.goal().y() + 50;

	dx = dx*mf.spring() + (xPos-r.x()) * mf.ease();
	dy = dy*mf.spring() + (yPos-r.y()) * mf.ease();

	r.x( r.x()+dx );
	r.y( r.y()+dy );
}

only this option would be handy if you wanted to track the mouse... use ease... use spring... but apply different offsets to other objects on screen - notice the rect has no anchor - and the offset of 50 is done in xPos and yPos

Thanks man, i'm planning to track finger movements with the leap motion and draw a Hshape near each finger so the second example suits me better.

rocking!

authors: danieltorrer hype


text:
[BUG] HandlerStackFactory方式使用httpClient导致内存溢出等一些问题

$factory = new HandlerStackFactory();
$stack = $factory->create();
$httpClient = make(Client::class,[
'config'=>[
'handler'=>$stack
]
]);
file_put_contents('/tmp/bbb.log',var_export($httpClient,true));
爆了几个错
PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 121638912 bytes) in /Users/Documents/www/YLHD/hyperf-demo/app/Job/CutPanoJob.php on line 49 (file_put_contents('/tmp/bbb.log',var_export($httpClient,true));就是这一行)
[ERROR] Hyperf\Process\Exception\SocketAcceptException:Socket is closed(0) in /Users/Documents/www/YLHD/hyperf-demo/vendor/hyperf/process/src/AbstractProcess.php:185
[CRITICAL] Socket of process is unavailable, please restart the server

更新到最新版本 2.0.*

Swoole 也更新到 4.5.2

Swoole 也更新到 4.5.2

都是最新的了,composer里的设置:
    "hyperf/async-queue": "~2.0.0",

嗯，那接下来的问题就是，不要把整个对象都打出来，里面涉及到了container。
你试试 symfony/var-dumper 这个组件

repo: hyperf/hyperf


text:
[QUESTION] 各位大佬知道hyperf如何处理图片么?涉及到 下载、本地存储、文件系统、图片操作、云存储

Before you submit an issue, please be sure to search through existing issues as well as search through the documentation

[] I've searched all existing issues
[] I've read all relevant documentation I could find

Describe your question
比如我想要将电商的一张超级长的大图切割成N张小图片，协程上需要注意什么吗？随便composer一个扩展能用吗？
流程如下：
1、判断图片是否过长
2、过长就下载下来存在项目中等待切割
3、切割图片，并上传七牛云
这些步骤，用到了 下载、本地存储、图片操作、云存储。请问使用hyperf需要注意什么，往各位大佬解惑！

https://image.intervention.io/v3
可以用这个库来操作、切割图片
至于七牛云，直接用官方的qiniu/php-sdk即可

authors: Moxyu aaron-code-life


text:
get right response of request

1,the vm state machine doesn't handle unexpected event(request), there will be no response to this request, this will cause the caller blocked to waiting for response.
first patch fixs this by sending unexpected response to request if request needs
2, right now we don't check if exec event is handled correctly. the second patch add this check, so we don't try to read stream callback if exec event failed. And other events get response through response chan, this patch makes exec event consist with other events. This patch also introduces Replyto field to
Vmresponse, with this field, request can check if the response is what it expected.
3, Fix possible getting incorrect response for read/write file event
TODO: Fix possible getting incorrect response for RunPod,StopPod events

LGTM

authors: gao-feng laijs

repo: hyperhq/runv


text:
Empty message in incident / recovery notification

Describe the bug
In the incident / recovery email sent via SMTP and in Slack, the message is empty which makes it confusing what the incident or recovery was.
To Reproduce
Steps to reproduce the behavior:

Use config below.
Run monika -c monika.yml
See the email notification.

# monika.yml
notifications:
  - id: unique-id-smtp
    type: smtp
    data:
      recipients:
        - <someemail>
      hostname: smtp.gmail.com
      port: 587
      username: <gmailusername>
      password:  <gmailpassword>
probes:
  - id: 'monika-localhost-2'
    incidentThreshold: 5
    recoveryThreshold: 5
    requests:
      - url: http://localhost:3000
    alerts:
      - query: response.status < 200 or response.status > 299
      - query: response.time > 2000

Expected behavior
The message should indicates which alert causes the incident and recoveries.
Screenshots

Desktop (please complete the following information):

OS: macOS
Version 1.6.3 and 1.6.2

Notes
Haven't checked in other notification channels, but probably it's the same case?

In the config above, no message is defined for the alerts. You might want to add message to alert like:
    alerts:
      - query: response.status < 200 or response.status > 299
        message: Response is not OK
      - query: response.time > 2000
        message: Response time too slow

Can you suggest the message format for when the configuration is empty? @nicnocquee

authors: armedi nicnocquee

repo: hyperjumptech/monika


text:
Translation-Docs » Contributions Welcome!

I have finished the 'Docs » Contributions Welcome!' translation.

Please assign it to me,thanks!

finished the job,please review.

repo: hyperledger-labs/fabric-docs-cn


text:
Update README.md with link to DCO

Signed-off-by: Jeremiah C. Foster jeremiah@jeremiahfoster.com

LGTM

For the sake of clarification, every commit in our repo that is proposed to Hyperledger has to have the signed-off-by line? Otherwise we have to squash the commits into a new commit with a signed-off-by line? Isn't it enough, at least for DCO purposes, to have a single sign-off per developer? If a developer has signed off on the DCO once, isn't that sufficient?

Every incoming commit has to have DCO.

Okay. Understood. I think the plan then is for us to squash and have me sign.

Something weird happening with the status check here. I am not sure what else it is waiting on. I would like to merge this PR.

authors: jeremiah tkuhrt


text:
ensure generated changes for circuit updates are checked in for PRs

for https://github.com/hyperledger-labs/zeto/pull/105#discussion_r1867217684

failures proved the workflow works.
it will need #102 and #105 to have a green build

Good idea!

One thing that would be useful is documenting the circom level, or even better check the required level in gen.js

authors: Chengxuan EnriqueL8 jimthematrix

repo: hyperledger-labs/zeto


text:
Add an optional parameter '--ledger-socks-proxy'

Add an optional parameter '--ledger-socks-proxy' to specify the socks proxy and pass it on to pool configuration
implementation for issue #1341
Attention! This pull request requires the newest versions of INDY-SDK and INDY-VDR (s. issue)
Signed-off-by: Vitalij Reicherdt vitalij.reicherdt@commerzbank.com

Codecov Report

Merging #1342 (55dd7c8) into main (33d7d9c) will decrease coverage by 0.00%.
The diff coverage is 69.23%.

:exclamation: Current head 55dd7c8 differs from pull request most recent head 29e32a9. Consider uploading reports for the commit 29e32a9 to get more accurate results

@@            Coverage Diff             @@
##             main    #1342      +/-   ##
==========================================
- Coverage   95.43%   95.43%   -0.01%
==========================================
  Files         476      476
  Lines       28966    28977      +11
==========================================
+ Hits        27645    27653       +8
- Misses       1321     1324       +3

authors: codecov-commenter vitalrev

repo: hyperledger/aries-cloudagent-python


text:
Alias connection

This pull request adds an optional alias parameter on the create and receive connection invitation endpoints.
It also adds the ability to query for connections by alias.

Codecov Report

Merging #149 into master will decrease coverage by 0.02%.
The diff coverage is 28.57%.

@@            Coverage Diff             @@
##           master     #149      +/-   ##
==========================================
- Coverage   73.48%   73.45%   -0.03%
==========================================
  Files         188      188
  Lines        8224     8230       +6
==========================================
+ Hits         6043     6045       +2
- Misses       2181     2185       +4

authors: codecov-io nrempel


text:
Use ping pong to keep web socket connections alive

As described here: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#pings_and_pongs_the_heartbeat_of_websockets
This means we don't have to reconnect the websocket each time, but we can keep it alive. Websockets are closed after a while of not receiving messages

After some troubleshooting on my end, the functionality here is that the server needs to implement the ping-pong intervals/config, with no change on the client-side AFAIK. This is something we're testing in ACA-Py now, with the intention of contributing back up stream after our testing.

Cool, I'll be interested to have this in ACA-Py!

Were you able to resolve this issue on ACA-Py's side @JamesKEbert?

authors: JamesKEbert TimoGlastra

repo: hyperledger/aries-framework-javascript