text:
roberta-base pretraining using trainer and accelerate on TPU help

I am trying to pretrain roberta-base on a number of DNA sequences. The data files have one sequence per line. I have the tokenizer trained for my purpose. This is the main code (pretrain.py) I am trying to run:
import sys
sys.path.append('/kaggle/working/florabert')
import torch
import os
from module.florabert import config, utils, training, dataio
from module.florabert import transformers as tr

# Import torch_xla for TPU support
import torch_xla.core.xla_model as xm

DATA_DIR = config.data_final / "transformer" / "seq"
TOKENIZER_DIR = config.models / "byte-level-bpe-tokenizer"

OUTPUT_DIR = config.models / "transformer" / "language-model"
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

args = utils.get_args(
    data_dir=DATA_DIR,
    train_data="all_seqs_train.txt",
    test_data="all_seqs_test.txt",
    output_dir=OUTPUT_DIR,
    model_name="roberta-lm",
)

settings = utils.get_model_settings(config.settings, args, args.model_name)

config_obj, tokenizer, model = tr.load_model(
    args.model_name,
    TOKENIZER_DIR,
    pretrained_model=args.pretrained_model,
    **settings,
)

num_params = utils.count_model_parameters(model, trainable_only=True)
print(f"Loaded {args.model_name} model with {num_params:,} trainable parameters")

datasets = dataio.load_datasets(
    tokenizer,
    args.train_data,
    test_data=args.test_data,
    file_type="text",
    seq_key="text",
)

def main():
    config_obj, tokenizer, model = tr.load_model(
        args.model_name,
        TOKENIZER_DIR,
        pretrained_model=args.pretrained_model,
        **settings,
    )
    device = xm.xla_device()
    model = model.to(device)
    dataset_train = datasets["train"]
    dataset_test = datasets["test"]
    print(f"Loaded training data with {len(dataset_train):,} examples")
    data_collator = dataio.load_data_collator(
        "language-model",
        tokenizer=tokenizer,
    )

    training_settings = config.settings["training"]["pretrain"]

    trainer = training.make_trainer(
        model,
        data_collator,
        dataset_train,
        dataset_test,
        args.output_dir,
        **training_settings,
    )

    print(f"Starting training on TPU")
    training.do_training(trainer, args, args.output_dir)

    print("Saving model")

    trainer.save_model(str(args.output_dir))

def _mp_fn(index):
    # For xla_spawn (TPUs)
    main()

if __name__ == "__main__":
    main()

The corresponding kaggle notebook and dataset are as follows:

Kaggle Notebook (Version 105)
Kaggle Dataset (Version 62)

I am launching this script in my notebook using the command:
!accelerate launch --tpu --main_training_function main /kaggle/working/florabert/scripts/1-modeling/pretrain.py

This is the (main part of the) output log (also available in the Notebook):
Starting training on TPU
Starting training on TPU
Starting training on TPU
Starting training on TPU
Starting training on TPU
Starting training on TPU
Starting training on TPU
Starting training on TPU
wandb: Currently logged in as: gurveervirk-15517. Use `wandb login --relogin` to force relogin
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
wandb: Tracking run with wandb version 0.16.1
wandb: Run data is saved locally in /kaggle/working/florabert/wandb/run-20231227_185309-mjx6thua
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run autumn-bee-119
wandb: ⭐ View project at https://wandb.ai/gurveervirk-15517/huggingface
wandb: 🚀 View run at https://wandb.ai/gurveervirk-15517/huggingface/runs/mjx6thua
  0%|                                                 | 0/56178 [00:00<?, ?it/s]You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
You're using a RobertaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
https://symbolize.stripped_domain/r/?trace=7ff61fdf067d,7ff776543fcf,7ff61fdf0754,7ff61fdf0a6c,7ff61fa4605a,7ff61fa60d6f,7ff61fce4862,7ff739b9b361&map=2bdca697f10000c651e27dbf91628d1bb3a7aa8d:7ff61b1cb000-7ff629a0e3a0
*** SIGSEGV (@0x1), see go/stacktraces#s15 received by PID 1231 (TID 2819) on cpu 6; stack trace: ***
PC: @     0x7ff61fdf067d  (unknown)  std::_Hashtable<>::_M_find_before_node()
    @     0x7ff53e7aa53a       1152  (unknown)
    @     0x7ff776543fd0       3376  (unknown)
    @     0x7ff61fdf0755         80  std::__detail::_Map_base<>::operator[]()
https://symbolize.stripped_domain/r/?trace=7ec77ebf068d,7ec8d5228fcf,7ec77ebf0754,7ec77ebf0a6c,7ec77e84605a,7ec77e860d6f,7ec77eae4862,7ec89899b361&map=2bdca697f10000c651e27dbf91628d1bb3a7aa8d:7ec779fcb000-7ec78880e3a0
*** SIGSEGV (@0x18), see go/stacktraces#s15 received by PID 1230 (TID 2820) on cpu 75; stack trace: ***
    @     0x7ff61fdf0a6d        496  torch_xla::xla_cpu_fallback()
PC: @     0x7ec77ebf068d  (unknown)  std::_Hashtable<>::_M_find_before_node()
    @     0x7ec69e7aa53a       1152  (unknown)
    @     0x7ec8d5228fd0       3376  (unknown)
    @     0x7ff61fa4605b        112  at::native::_call_fallback_fn<>::call()
    @     0x7ec77ebf0755         80  std::__detail::_Map_base<>::operator[]()
    @     0x7ff61fa60d70        112  torch_xla::XLANativeFunctions::_local_scalar_dense()
    @     0x7ec77ebf0a6d        496  torch_xla::xla_cpu_fallback()
    @     0x7ff61fce4863         96  c10::impl::make_boxed_from_unboxed_functor<>::call()
    @     0x7ff739b9b362  (unknown)  c10::Dispatcher::callBoxed()
https://symbolize.stripped_domain/r/?trace=7ff61fdf067d,7ff53e7aa539,7ff776543fcf,7ff61fdf0754,7ff61fdf0a6c,7ff61fa4605a,7ff61fa60d6f,7ff61fce4862,7ff739b9b361&map=2bdca697f10000c651e27dbf91628d1bb3a7aa8d:7ff61b1cb000-7ff629a0e3a0,abbd016d9542b8098892badc0b19ea68:7ff531600000-7ff53e9becf0
E1227 18:53:11.056009    2819 coredump_hook.cc:447] RAW: Remote crash data gathering hook invoked.
E1227 18:53:11.056021    2819 client.cc:272] RAW: Coroner client retries enabled (b/136286901), will retry for up to 30 sec.
E1227 18:53:11.056025    2819 coredump_hook.cc:542] RAW: Sending fingerprint to remote end.
E1227 18:53:11.056068    2819 coredump_hook.cc:551] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E1227 18:53:11.056075    2819 coredump_hook.cc:603] RAW: Dumping core locally.
    @     0x7ec77e84605b        112  at::native::_call_fallback_fn<>::call()
    @     0x7ec77e860d70        112  torch_xla::XLANativeFunctions::_local_scalar_dense()
    @     0x7ec77eae4863         96  c10::impl::make_boxed_from_unboxed_functor<>::call()
    @     0x7ec89899b362  (unknown)  c10::Dispatcher::callBoxed()
https://symbolize.stripped_domain/r/?trace=7ec77ebf068d,7ec69e7aa539,7ec8d5228fcf,7ec77ebf0754,7ec77ebf0a6c,7ec77e84605a,7ec77e860d6f,7ec77eae4862,7ec89899b361&map=2bdca697f10000c651e27dbf91628d1bb3a7aa8d:7ec779fcb000-7ec78880e3a0,abbd016d9542b8098892badc0b19ea68:7ec691600000-7ec69e9becf0
E1227 18:53:11.152067    2820 coredump_hook.cc:447] RAW: Remote crash data gathering hook invoked.
E1227 18:53:11.152081    2820 client.cc:272] RAW: Coroner client retries enabled (b/136286901), will retry for up to 30 sec.
E1227 18:53:11.152086    2820 coredump_hook.cc:542] RAW: Sending fingerprint to remote end.
E1227 18:53:11.152146    2820 coredump_hook.cc:551] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E1227 18:53:11.152156    2820 coredump_hook.cc:603] RAW: Dumping core locally.
E1227 19:01:13.847483    2484 coredump_hook.cc:447] RAW: Remote crash data gathering hook invoked.
E1227 19:01:13.847530    2484 coredump_hook.cc:486] RAW: Called via ReportEvent and disabled coredump
E1227 19:01:13.847541    2484 client.cc:272] RAW: Coroner client retries enabled (b/136286901), will retry for up to 30 sec.
E1227 19:01:13.847548    2484 coredump_hook.cc:542] RAW: Sending fingerprint to remote end.
E1227 19:01:13.847603    2484 coredump_hook.cc:551] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] stat failed on crash reporting socket /var/google/services/logmanagerd/remote_coredump.socket (Is the listener running?): No such file or directory
E1227 19:01:13.847617    2484 coredump_hook.cc:603] RAW: Dumping core locally.
https://symbolize.stripped_domain/r/?trace=786f34017da6,786f33fcdfcf&map=
*** SIGTERM received by PID 1226 (TID 1226) on cpu 0 from PID 296; stack trace: ***
https://symbolize.stripped_domain/r/?trace=7a2077870da6,7a2077826fcf&map=
*** SIGTERM received by PID 1229 (TID 1229) on cpu 23 from PID 296; stack trace: ***
PC: @     0x786f34017da6  (unknown)  (unknown)
PC: @     0x7a2077870da6  (unknown)  (unknown)
    @     0x7a1e427aa53a       1152  (unknown)
    @     0x786cfe7aa53a       1152  (unknown)
    @     0x7a2077826fd0  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7a2077870da6,    @     0x786f33fcdfd0  (unknown)  (unknown)
7a1e427aa539,https://symbolize.stripped_domain/r/?trace=7a2077826fcf&map=786f34017da6,786cfe7aa539,786f33fcdfcf&map=abbd016d9542b8098892badc0b19ea68:7a1e35600000-7a1e429becf0
abbd016d9542b8098892badc0b19ea68:786cf1600000-786cfe9becf0
E1227 19:01:14.510789    1229 coredump_hook.cc:393] RAW: Remote crash gathering disabled for SIGTERM.
E1227 19:01:14.510792    1226 coredump_hook.cc:393] RAW: Remote crash gathering disabled for SIGTERM.
E1227 19:01:14.999856    1226 process_state.cc:783] RAW: Raising signal 15 with default behavior
E1227 19:01:15.029555    1229 process_state.cc:783] RAW: Raising signal 15 with default behavior
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1019, in launch_command
    tpu_launcher(args)
  File "/usr/local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 762, in tpu_launcher
    xmp.spawn(PrepareForLaunch(main_function), args=(), nprocs=args.num_processes)
  File "/usr/local/lib/python3.10/site-packages/torch_xla/runtime.py", line 82, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 38, in spawn
    return pjrt.spawn(fn, nprocs, start_method, args)
  File "/usr/local/lib/python3.10/site-packages/torch_xla/_internal/pjrt.py", line 202, in spawn
    run_multiprocess(spawn_fn, start_method=start_method)
  File "/usr/local/lib/python3.10/site-packages/torch_xla/runtime.py", line 82, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch_xla/_internal/pjrt.py", line 159, in run_multiprocess
    replica_results = list(
  File "/usr/local/lib/python3.10/site-packages/torch_xla/_internal/pjrt.py", line 160, in <genexpr>
    itertools.chain.from_iterable(
  File "/usr/local/lib/python3.10/concurrent/futures/process.py", line 575, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Why is it failing? What changes do I make in my code and approach to resolve this issue? Is this the appropriate way to launch the script with trainer for TPU usage?

It's the appropriate way, however there's not enough code here to see what's going on. Does running the nlp_example.py script work fine?

Will try and let you know. Is it supposed to be launching the main function on all the cores? Can it lead to RAM running out?

authors: gurveervirk muellerzr


text:
The number of started jobs is a lot bigger than the number of workers

We currently have nearly 2000 started jobs, and "only" 200 workers

My guess: the jobs raise an uncaught exception and are never finished properly in the database. They will be deleted afterwards anyway (zombie killer) but possibly we must fix something here.

Note that the three affected steps are dataset-opt-in-out-urls-count, config-opt-in-out-urls-count, split-opt-in-out-urls-count, which at least seems coherent in some way.
maybe @AndreaFrancis you want to check this?

Sure, I will check

Interestingly, there are jobs with "started" status but have a  "finished_at" value and a cache record in the DB. I think only the status is not being set correctly, will put here my findings.

I found some other records in the queue for other steps (Different than dataset-opt-in-out-urls-count, config-opt-in-out-urls-count, split-opt-in-out-urls-count) using filter {status:"started", finished_at:{$exists:true}}. But I was not able to reproduce the issue.
I think setting the TTL index to drop only for those status success/error would help us identify the problem.

Hmmm. No need to investigate too much. If we don't find the root reason quickly, we can let it unchanged: anyway, the cron backfill will fix the remaining issues once a day

Hmmm. No need to investigate too much. If we don't find the root reason quickly, we can let it unchanged: anyway, the cron backfill will fix the remaining issues once a day

Sure, anyway I think we should not delete the inconsistent records, those are not being finished by zombie killer, and looks like queue's finish method is not letting us know if there is a wrong combination of state+dates https://github.com/huggingface/datasets-server/blob/main/libs/libcommon/src/libcommon/queue.py#LL632C15-L632C15 it returns a boolean but nowhere is being evaluated.
I opened https://github.com/huggingface/datasets-server/pull/1325, and would like to understand what is happening and if there is a pattern for this behavior.

Note that since we change the TTL index, we now have "waiting" jobs with a non-null finished_at field, which shows up in the metrics. Maybe we should change the way we compute the metrics, if this issue is not solved soon

source: gharchive/issue

authors: AndreaFrancis severo

repo: huggingface/datasets-server

license_type: permissive

license_source: github-api


text:
Get an IterableDataset from a map-style Dataset

This is useful to leverage iterable datasets specific features like:

fast approximate shuffling
lazy map, filter etc.

Iterating over the resulting iterable dataset should be at least as fast at iterating over the map-style dataset.
Here are some ideas regarding the API:
# 1.
# - consistency with load_dataset(..., streaming=True)
# - gives intuition that map/filter/etc. are done on-the-fly
ids = ds.stream()

# 2.
# - more explicit on the output type
# - but maybe sounds like a conversion tool rather than a step in a processing pipeline
ids = ds.as_iterable_dataset()

I think stream could be misleading since the data is not being streamed from remote endpoints (one could think that's the case when they see load_dataset followed by stream). Hence, I prefer the second option.
PS: When we resolve https://github.com/huggingface/datasets/issues/4542, we could add as_tf_dataset to the API for consistency and deprecate to_tf_dataset.

authors: lhoestq mariosasko

repo: huggingface/datasets


text:
Allow concatenation of datasets with mixed structs

Fixes #6466
The idea is to do a recursive check for structs. PyArrow handles it well enough.
For a demo you can do:
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())

Now both the features and the rows are fixed.
I note that Sequence suffers from the same problem, so I can fix that in a future PR once this one is merged.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

friendly bump

source: gharchive/pull-request


text:
Feature: add community metrics

We would like a way for community members to easily add custom or new metrics. These custom metrics could be hosted on the hub either in a dedicated repository type (like datasets or models) or on Spaces.
Spaces comes with a few advantages, especially in short term:

No dedicated repo type needed
Infrastructure already exists
We can easily setup a demo without needing a custom widget (e.g. with Gradio)

@osanseviero made a PoC here: https://huggingface.co/spaces/osanseviero/accuracy_metric
A few things that we should add to make this as user friendly and useful as possible:

create a template to add a metric space

template README.md (like model/dataset card) with basic structure
template my_metric.py file (similar to datasets loading scripts)
gradio app.py that loads all information from README.md and my_metric.py to create a default widget. This does probably not need to be changed.

debugging tools

instructions on how to clone the repository directly on spaces
maybe an evaluate-cli command to test if everything is fine evaluate-cli test-metric PATH_TO_REPO

dedicated tag: if we ever want to migrate the metrics to a dedicated place like datasets and models it is good if we had a tag to find them all.  Similarly, we could then easily do a list_metrics() function in the evaluate without needing to look at the content of each repo.

The goal is to enable behaviour (example with the space above):
accuracy_metric = load_metric("osanseviero/accuracy_metric")

Closed with #25 and #14.

repo: huggingface/evaluate


text:
Widget for Visual Question Answering

This request for a widget/inference API for the hub for multimodal models for VQA tasks:
Would have input as

Image
Question
Would output
Answer

This should be used for VisualBERT/LXMERT models. And might need detectron or something similar to the FasterRCNN model here : https://github.com/huggingface/transformers/tree/master/examples/research_projects/lxmert
@LysandreJik

cc @mishig25

closed by https://github.com/huggingface/hub-docs/pull/263

authors: NielsRogge gchhablani mishig25

repo: huggingface/hub-docs


text:
More robust uploads

This PR aims to make the upload process more robust to transient errors:

Retry onConnectionError -e.g. connection temporarily lost or connection reset by peer-. Before this was the case only for ProxyError (not sure why :thinking:).
Retry on HTTP 500. It seems that sometimes S3 raises a transient HTTP 500 error (see https://github.com/huggingface/datasets/issues/6392#issuecomment-1806059794). Not sure why but retrying when it's the case doesn't cost mush.

The retry mechanism is the same as before, implemented in http_backoff (up to 5 retries, backoff after 1s, 2s , 4s and 8s). Once this is merged, the upload process will retry on:

ConnectionError
TimeoutError
HTTP 500
HTTP 503

Note: this  "retry on error" mechanism does not apply to uploads using hf_transfer (and is not planned to be).

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Thanks for the reviews!

authors: HuggingFaceDocBuilderDev Wauplin

license_source: bigquery


text:
Add text to image pipeline to generic inference

Similar to #244, this PR adds support for text-to-image pipeline.
Here is an example repo that uses BigGAN-deep-128 (the return of pytorch-pretrained-BigGAN). As can be seen, users specify their own requirements.txt and pipeline.py. I struggled a bit with the output, but it's a base64 string of the image. This PR should unblock #131.
Here is a corresponding repo template https://huggingface.co/templates/text-to-image

lgtm! 🚀

FYI @nateraw

looks great, thanks for adding this! could you share a screenshot of what the model output looks like?
The Inference pipeline returns a base 64 string with the image. I'm not fully satisfied with the output and if you look at the example repo it got quite complicated (which I'm sure it's due to my ignorance on some image stuff).

I won't merge this until @Narsil takes a look, since he probably has good insights on the output for this (not high priority so it can wait for a couple of extra days).

The Inference pipeline returns a base 64 string with the image. I'm not fully satisfied with the output and if you look at the example repo it got quite complicated (which I'm sure it's due to my ignorance on some image stuff).

seems good enough for stack overflow 😄

Another thing, is that we can send raw bytes directly with proper mimecode if we're sending only 1 image per inference (which I guess is the case here).
api_inference_community/routes.py (call_pipe) should be the place to look at.

Looks fine, but the multple dummy inheritance withing generic looks very bloated, why not have a single class handle all pipelines ?

Copied from the comment "That was the initial plan, but unfortunately it can be the case that different pipelines require somewhat different behaviour. For example, the audio pipelines also required specifying sampling rate (this is done in #266). Maybe we could have a DefaultPipeline, AudioPipeline, etc. WDYT?"

Another thing, is that we can send raw bytes directly with proper mimecode if we're sending only 1 image per inference (which I guess is the case here).

Yes, the case here is to return one image at a time. Thanks for this pointer, I'll look into this.

I sent #272 based on @Narsil feedback, which allows users to just return the PIL.Image and we take care of the encoding.
Before

img = transforms.ToPILImage()(output[0])
buffer = io.BytesIO()
img.save(buffer, format="JPEG")
img_str = base64.b64encode(buffer.getvalue()).decode('utf-8')
return img_str

Now

return transforms.ToPILImage()(output[0]) # or any other PIL.Image

I'll merge this now to unblock @mishig25. The refactoring can follow in another PR.

authors: Narsil lewtun mishig25 osanseviero


text:
Fix the progress bars when cloning a repository

The environment changes that had to happen for the GIT_LFS_SKIP_SMUDGE ended up modifying the progress bars for that specific command.
A test should be setup to ensure this doesn't happen.

Will merge this to have the progress bars on main and work on a test as soon as I have a bit of time.


text:
disable slack_sender during unittests

Is there a way to temporarily disable slack sender during unittests?
I want to add knock knock to my training code but don't want it to slack me every time I run the tests.

It'd be great to also have same functionality while debugging code (always forget to comment it out while doing this).

authors: sshleifer valeria-io

repo: huggingface/knockknock

license: MIT


text:
Support NNCF 2.14 and OV 2024.5

What does this PR do?

Fixes # (issue)
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

OpenVINO slow tests on a stable NNCF and OV versions failing with huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error should not block us.

repo: huggingface/optimum-intel


text:
QuantizedModelForCausalLM lost repr, cannot see model structure

Is this intentional?

The repr method is indeed not implemented for this class. It shouldn't be too difficult to add it, calling the inner model repr.

authors: dacorvo imba-tjd

repo: huggingface/optimum-quanto


text:
The exported ONNX model of Qwen/Qwen1.5-0.5B-Chat does not produce a cache-enabled model.

System Info
transformers-4.38.2
optimum-1.17.1

Who can help?
Hi @michaelbenayoun,
I have exported Qwen/Qwen1.5-0.5B-Chat model with text-generation-with-past.  When running the exported ONNX model with the ORTModelForCausalLM class, the following error is observed.
File "/proj/mldata/users/anilm/repos/qwen/run.py", line 11, in
model = ORTModelForCausalLM.from_pretrained("Qwen1.5-0.5B-Chat")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/mldata/users/anilm/workspace/AIE/miniconda/envs/py311/lib/python3.11/site-packages/optimum/onnxruntime/modeling_ort.py", line 662, in from_pretrained
return super().from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/mldata/users/anilm/workspace/AIE/miniconda/envs/py311/lib/python3.11/site-packages/optimum/modeling_base.py", line 399, in from_pretrained
return from_pretrained_method(
^^^^^^^^^^^^^^^^^^^^^^^
File "/proj/mldata/users/anilm/workspace/AIE/miniconda/envs/py311/lib/python3.11/site-packages/optimum/onnxruntime/modeling_decoder.py", line 559, in _from_pretrained
return init_cls(
^^^^^^^^^
File "/proj/mldata/users/anilm/workspace/AIE/miniconda/envs/py311/lib/python3.11/site-packages/optimum/onnxruntime/modeling_decoder.py", line 169, in init
raise ValueError(
ValueError: use_cache was set to True but the loaded model only supports use_cache=False. Please load your current model with use_cache=False or export the original model once again with use_cache=True when calling the from_pretrained method. To export your model, simply set export=True
I have added the custom export script below.
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)
from optimum.exporters.onnx import main_export

from transformers import AutoConfig

from optimum.exporters.onnx.config import TextDecoderOnnxConfig,TextDecoderWithPositionIdsOnnxConfig
from optimum.exporters.onnx.base import ConfigBehavior
from optimum.utils import NormalizedTextConfig, DummyPastKeyValuesGenerator
from typing import Dict
import os
import shutil

class QwenDummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator):

    def generate(self, input_name: str, framework: str = "pt"):
        past_key_shape = (
            self.batch_size,
            self.num_attention_heads,
            self.hidden_size // self.num_attention_heads,
            self.sequence_length,
        )
        past_value_shape = (
            self.batch_size,
            self.num_attention_heads,
            self.sequence_length,
            self.hidden_size // self.num_attention_heads,
        )
        return [
            (
                self.random_float_tensor(past_key_shape, framework=framework),
                self.random_float_tensor(past_value_shape, framework=framework),
            )
            for _ in range(self.num_layers)
        ]

class CustomQwenOnnxConfig(TextDecoderOnnxConfig):
    DUMMY_INPUT_GENERATOR_CLASSES = (QwenDummyPastKeyValuesGenerator,) + TextDecoderOnnxConfig.DUMMY_INPUT_GENERATOR_CLASSES
    DUMMY_PKV_GENERATOR_CLASS = QwenDummyPastKeyValuesGenerator

    DEFAULT_ONNX_OPSET = 15  # aten::tril operator requires opset>=14
    NORMALIZED_CONFIG_CLASS = NormalizedTextConfig

    def add_past_key_values(self, inputs_or_outputs: Dict[str, Dict[int, str]], direction: str):

         if direction not in ["inputs", "outputs"]:
             raise ValueError(f'direction must either be "inputs" or "outputs", but {direction} was given')

         if direction == "inputs":
             decoder_sequence_name = "past_sequence_length"
             name = "past_key_values"
         else:
             decoder_sequence_name = "past_sequence_length + 1"
             name = "present"

         for i in range(self._normalized_config.num_layers):
             inputs_or_outputs[f"{name}.{i}.key"] = {0: "batch_size", 3: decoder_sequence_name}
             inputs_or_outputs[f"{name}.{i}.value"] = {0: "batch_size", 2: decoder_sequence_name}

model_id = "Qwen/Qwen1.5-0.5B-Chat"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)

onnx_config = CustomQwenOnnxConfig(
    config=config,
    task="text-generation",
    use_past=True,
    use_past_in_inputs=False,
)
onnx_config_with_past = CustomQwenOnnxConfig(config, task="text-generation", use_past=True)

custom_onnx_configs = {
    "model": onnx_config_with_past,
}

main_export(
    model_id,
    output="Qwen1.5-0.5B-Chat",
    task="text-generation-with-past",
    trust_remote_code=True,
    custom_onnx_configs=custom_onnx_configs,
    no_post_process=True,
    opset=15
)
### Running
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForCausalLM
from optimum.utils import NormalizedTextConfig, NormalizedConfigManager
NormalizedConfigManager._conf['qwen2'] = NormalizedTextConfig

import torch
model_id = "Qwen/Qwen1.5-0.5B-Chat"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = ORTModelForCausalLM.from_pretrained("Qwen1.5-0.5B-Chat")

Expected behavior
I am generating the model with text-generation-with-past, and it should work seamlessly.

H @anilmartha, thank you for the issue. https://github.com/huggingface/optimum/pull/1746 should be merged today which should make it as straightforward as:
optimum-cli export onnx --model Qwen/Qwen1.5-0.5B-Chat qwen_onnx

Now regarding your code: Qwen/Qwen1.5-0.5B-Chat does not seem to use a custom modeling anymore (maybe it was in the past), as qwen2 model type is now supported in Transformers natively.
So you do not need to use trust_remote_code=True. Also, the PKV generator is not correct. Qwen2 is similar to llama and:
from optimum.exporters.onnx import main_export
from transformers import AutoConfig
from optimum.exporters.onnx.model_configs import LlamaOnnxConfig

class CustomQwenOnnxConfig(LlamaOnnxConfig):
    pass

model_id = "fxmarty/tiny-dummy-qwen2"
config = AutoConfig.from_pretrained(model_id)

onnx_config_with_past = CustomQwenOnnxConfig(config, task="text-generation", use_past=True)

custom_onnx_configs = {
    "model": onnx_config_with_past,
}

main_export(
    model_id,
    output="Qwen1.5-0.5B-Chat",
    task="text-generation-with-past",
    custom_onnx_configs=custom_onnx_configs,
)

just works.
Note that https://github.com/huggingface/optimum/pull/1746 is needed to have the exported model work with ORTModelForCausalLM.

Hi @MrRace these files are an artifact of a step in the ONNX export where all external data are saved under the same model.onnx_data file. These files are not needed, should be deleted but are not. You can simply use model.onnx.
Basically, torch.onnx.export for models > 2GB export weights data as many independent files (onnx__MatMul_5747, onnx__MatMul_5748), which are fused in a single file (and the Constant_attr__value files as well), but these are leftovers.
This will be fixed in https://github.com/huggingface/optimum/pull/1808.

authors: anilmartha fxmarty

repo: huggingface/optimum


text:
Seek help for a more efficient way to use caching to train my model

When I was fine-tuning llama2 model using lora, I came across a problem.
The instruction dataset goes something like this:
"Here's the background to the problem... (1000 identical words)... Now answer the questions in context... (Different question, about 100 words)..." .
Each piece of data has the same very long prefix.
As we know before, during inference, we can pre-computed and cache kv cache, and then pass the valuepast_key_value to speed up the reasoning.
like this

part0 = {}
for k, v in inputs.items():
    part0[k] = v[:, :-1]

output_part0 = model(**part0)

outputs = model.generate(
    **inputs, past_key_values=output_part0.past_key_values, max_new_tokens=5
)
print(tokenizer.batch_decode(outputs[:, inputs["input_ids"].shape[1] :]))

I figure there must be a more efficient way to use caching to train my model
. Can anyone give me a suggestion？Thanks a lot.

hi @whyiug
thanks for the issue! IIUC caching is effective for inference and not for training, if you pre-compute KV cache offline for training how can you propagate the gradients into them ?

@younesbelkada Thanks for your reply. My training method is lora, where all linear layers in the base model are frozen, and for my input training set they are not trainable, but are double-counted at predict time.

Is it my misunderstanding of lora and backpropagation ? Or maybe people don't have a need for it. @younesbelkada thanks for your advice.

authors: whyiug younesbelkada

repo: huggingface/peft


text:
modules_to_save incompatible with load_in_4bit / load_in_8bit ?

System Info
Python version: 3.10.8
torch==2.0.1+cu118
transformers @ git+https://github.com/huggingface/transformers.git@17a55534f5e5df10ac4804d4270bf6b8cc24998d
peft @ git+https://github.com/huggingface/peft.git@3714aa2fff158fdfa637b2b65952580801d890b2
accelerate @ git+https://github.com/huggingface/accelerate.git@7d24bdefb5b3252505151d8c1ac0efbed3574857
bitsandbytes==0.39.0
Who can help?
@pacman100
Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder
[ ] My own task or dataset (give details below)

Reproduction
I'm trying to train a LoRA sequence classification model and found that gradient updates were applied to original_module instead of modules_to_save.default. This causes issues when loading the trained adapter for inference.
I've tried different combinations of load_in_{4,8}bit, device_map, resize_token_embeddings, prepare_model_for_kbit_training, etc. The only combination that behaves as expected was load_in_4bit=False, device_map=None, should_resize_token_embeddings=True, should_prepare_model_for_kbit_training=False (i.e. removing device_map='auto'). However this is not an option for load_in_{4,8}bit.
Thanks for your work and would appreciate any suggestions.
import fire

import torch
from peft import LoraConfig, TaskType, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoTokenizer, AutoModelForSequenceClassification, set_seed

set_seed(123)

PRETRAIN = 'bigscience/bloomz-560m'
tokenizer = AutoTokenizer.from_pretrained(PRETRAIN)

def main(load_in_4bit=False, device_map='auto', should_resize_token_embeddings=False, should_prepare_model_for_kbit_training=False):
    model = AutoModelForSequenceClassification.from_pretrained(
        PRETRAIN,
        load_in_4bit=load_in_4bit,
        torch_dtype=torch.float32,
        device_map=device_map,
    )
    if should_prepare_model_for_kbit_training: model = prepare_model_for_kbit_training(model)

    config = LoraConfig(
        r=16,
        lora_alpha=16,
        lora_dropout=0.05,
        bias="none",
        task_type=TaskType.SEQ_CLS,
    )

    peft_model = get_peft_model(model, config)
    if should_resize_token_embeddings: model.resize_token_embeddings(len(tokenizer))

    lm_head = peft_model.base_model.model.score
    original_module = lm_head.original_module
    modules_to_save = lm_head.modules_to_save.default

    inputs = torch.randn((1024))
    o1 = lm_head(inputs)
    o1.mean().backward()
    original_module_has_grad = original_module.weight.grad is not None
    modules_to_save_has_grad = modules_to_save.weight.grad is not None
    print(f'{load_in_4bit=}, {device_map=}, {should_resize_token_embeddings=}, {should_prepare_model_for_kbit_training=} --> {original_module.weight.requires_grad=}, {modules_to_save.weight.requires_grad=}, {original_module_has_grad=}, {modules_to_save_has_grad=}')

if __name__ == '__main__':
    fire.Fire(main)

# load_in_4bit=False, device_map='auto', should_resize_token_embeddings=False, should_prepare_model_for_kbit_training=False --> original_module.weight.requires_grad=True, modules_to_save.weight.requires_grad=True, original_module_has_grad=True, modules_to_save_has_grad=False
# load_in_4bit=False, device_map=None, should_resize_token_embeddings=False, should_prepare_model_for_kbit_training=False --> original_module.weight.requires_grad=True, modules_to_save.weight.requires_grad=True, original_module_has_grad=False, modules_to_save_has_grad=True ***ONLY THIS IS AS EXPECTED***
# load_in_4bit=False, device_map='auto', should_resize_token_embeddings=True, should_prepare_model_for_kbit_training=False --> original_module.weight.requires_grad=True, modules_to_save.weight.requires_grad=True, original_module_has_grad=True, modules_to_save_has_grad=False
# load_in_4bit=True, device_map='auto', should_resize_token_embeddings=False, should_prepare_model_for_kbit_training=True --> original_module.weight.requires_grad=True, modules_to_save.weight.requires_grad=True, original_module_has_grad=True, modules_to_save_has_grad=False
# load_in_4bit=True, device_map='auto', should_resize_token_embeddings=True, should_prepare_model_for_kbit_training=True --> original_module.weight.requires_grad=True, modules_to_save.weight.requires_grad=True, original_module_has_grad=True, modules_to_save_has_grad=False
# load_in_4bit=True, device_map=None, should_resize_token_embeddings=False, should_prepare_model_for_kbit_training=True --> ValueError: A device map needs to be passed to run convert models into 8-bit and 4-bit formats. Please run`.from_pretrained` with `device_map='auto'`

Expected behavior
modules_to_save.default should get updated instead of original_module

I'm also experiencing something similar!
Until this is fixed, I'm using the following workaround, which seems to work for me (will need to retrain the classifier again though):

Along with the model, save the classification head's state dict separately

..........
model.save_pretrained("my-awesome-model")
torch.save(model.score.state_dict(), 'score-params.pt')

During inference, after loading the peft model, load the saved params into the model's classification head

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel, PeftConfig

device_map = "auto"
peft_model_id = "my-awesome-model"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForSequenceClassification.from_pretrained(
    config.base_model_name_or_path,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map=device_map,
    num_labels=3
)
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
score_params = torch.load('score-params.pt')
model.score.load_state_dict(score_params)

Thanks @00ber for sharing your workaround.
I wonder what is the purpose of ModulesToSaveWrapper splitting into original_module and modules_to_save? Don't you always want to retrain the head i.e. no need for original_module?
Aside from this issue, I've also ran into a weird modules_to_save error with DeepSpeed zero3. Seems if you initialize DeepSpeed before creating the model (e.g. by using HfArgumentParser):

from_pretrained will partition model params: https://github.com/huggingface/transformers/blob/v4.30.2/src/transformers/modeling_utils.py#L2666
ModulesToSaveWrapper will copy the (now empty) original_module: https://github.com/huggingface/peft/blob/main/src/peft/utils/other.py#L136
The forward pass will RuntimeError: size mismatch, got XX, XXxYYY,0 as the head is empty

Seems this module might need more testing?

@linhdvu14 I'm not sure of the original purpose behind splitting into original_module and modules_to_save either....But yeah, this module definitely needs a bit more testing it seems. I thought it was an issue just with the sequence classification model, but recently when I tried saving the LM Head for a CausalLM model, I faced a similar issue of the weights not being saved correctly.

@linhdvu14 I'm not sure of the original purpose behind splitting into original_module and modules_to_save either....But yeah, this module definitely needs a bit more testing it seems. I thought it was an issue just with the sequence classification model, but recently when I tried saving the LM Head for a CausalLM model, I faced a similar issue of the weights not being saved correctly.

@00ber @younesbelkada  I met the same weird problem and had a rough conclusion: it is caused by dispatch_model function triggered by load_in_8bit=True or 'device_map is not None'. A AlignDeviceHook may be attached to that causes unexpected module forwarding.

I'm also running into the same issue. Thanks for the workaround @00ber, I'll try that out today.

Hey, thanks everyone here for investigating this bug and providing code to replicate it. We could confirm the error and could pinpoint the issue, the fix is provided in #755. Since this is quite a tricky bug, it would be great if you could help us by checking if the fix solves your original problem.

@BenjaminBossan Thanks for looking into this issue. For my understanding, could you explain if this fix requires a retraining of our models, or if we can use the adapters from our original training with the old code?

I just tried running the following code from @younesbelkada new fix branch:
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer

peft_model_id = "vincentmin/llama-2-7b-reward-oasst1"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForSequenceClassification.from_pretrained(
    config.base_model_name_or_path,
    num_labels=1,
    load_in_8bit=True,
    torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path, use_auth_token=True)
model.eval()
with torch.no_grad():
  test = model(**tokenizer("hello world", return_tensors='pt')).logits
test

If I run it twice, I get different results.
Note that the adapters that I load here were created from a training run on the old branch.
I guess that answers my question above: "We cannot use adapters from our original training with the old code".
I can try again tomorrow to see if the issue is fixed if we also train on the new branch.

@BenjaminBossan I managed to confirm that #755 fixes my issue. I trained in 4bit on the branch #755 and can now consistently load the model with the same model output. Thanks @BenjaminBossan  and @younesbelkada !

I experienced the same problem today. I had 92% accuracy during training but after loading the saved model, my model's accuracy dropped to like 30%. I retrained the model and checked the before-training and after-training weights of both model.score.modules_to_save and model.score.original_module and found that the model.score.original_module was being updated rather than model.score.modules_to_save which is causing the model to use the non-optimized weights after it is loaded.
Until this is fixed, I'm using the following workaround, which seems to work for me (will need to retrain the classifier again though):

Along with the model, save the classification head's state dict separately

..........
model.save_pretrained("my-awesome-model")
torch.save(model.score.state_dict(), 'score-params.pt')

During inference, after loading the peft model, load the saved params into the model's classification head

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel, PeftConfig

device_map = "auto"
peft_model_id = "my-awesome-model"
config = PeftConfig.from_pretrained(peft_model_id)
model = AutoModelForSequenceClassification.from_pretrained(
    config.base_model_name_or_path,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map=device_map,
    num_labels=3
)
model = PeftModel.from_pretrained(model, peft_model_id)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
score_params = torch.load('score-params.pt')
model.score.load_state_dict(score_params)

it's a good method.

Thanks @00ber for sharing your workaround.
I wonder what is the purpose of ModulesToSaveWrapper splitting into original_module and modules_to_save? Don't you always want to retrain the head i.e. no need for original_module?
Aside from this issue, I've also ran into a weird modules_to_save error with DeepSpeed zero3. Seems if you initialize DeepSpeed before creating the model (e.g. by using HfArgumentParser):

from_pretrained will partition model params: https://github.com/huggingface/transformers/blob/v4.30.2/src/transformers/modeling_utils.py#L2666
ModulesToSaveWrapper will copy the (now empty) original_module: https://github.com/huggingface/peft/blob/main/src/peft/utils/other.py#L136
The forward pass will RuntimeError: size mismatch, got XX, XXxYYY,0 as the head is empty

Seems this module might need more testing?

Hi @linhdvu14 how did u fix this issue?
I'm having this same issue when tuning a LLaVA model with LoRA. I added a customize layer with config LoraConfig(target_modules=['xxx'], modules_to_save=['customi_layer']). It failes on peft==0.4.0 as specified by LLaVA and upgrading to peft==0.6.2 or perf==0.8.0 wouldn't fix this issue either. Is https://github.com/huggingface/peft/pull/755 also fixing this issue or some other workaround is still necessary?


text:
Fix docs typo

This PR fixes the guide that explains saving embedding layers besides adapter parameters. The save_adapter method doesn't exist, should be save_pretrained.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

authors: HuggingFaceDocBuilderDev NielsRogge


text:
Fix bug in run_squad.py

I fired an issue in Google's repo, https://github.com/google-research/bert/issues/540#issue-428344784
After testing, I found it is indeed a bug.
We should not put these chunks with start_position==0 and start_position==0 into training set.
Thanks for code review.

Hi @MottoX, sorry for the delay on reviewing this.
It seems to make sense to me.
What kind of testing are you referring to? Could you share a bit more about them?

Hi, @thomwolf
I was trying to train a BERT-based model on NewsQA, a document-level QA dataset, using run_squad.py. I used 384 for max_seq_length and 128 for doc_stride. The model got an extremely bad result. I found there were numerous chunks with start_position and end_position as zeros.
Then I checked the implementation of run_squad.py as well as BertForQuestionAnswering, which is basically BERT plus a fully connection layer. I suspected that such out-of-span chunks are harmful for model to converge.
So I modified the script and got an acceptable result. I am sorry that I do not remember the exact numbers before and after doing this.
I think the reason why the problem is not identified before is that context length (token-level) in SQuAD is not that long, which means the model can still easily converge anyway, with or without out-of-span chunks as they are really rare. However, when there are numerous out-of-span chunks, there will be a problem for model to converge.

I looked at the code again, and found it is now quite confusing. Since chunks with start_position==0 and end_position==0 are also used for the cased where we have examples that do not have answer. So probably we should investigate this problem deeper. Now I am not exactly sure whether we need to train out-of-span chunks. However, from my last experience, there should be a problem when we have too many out-of-span chunks in the training set and the model is biased to predict the result on the first token, despite the fact that write_predictions filters out those invalid predictions.

authors: MottoX thomwolf

repo: huggingface/pytorch-pretrained-BERT


text:
num_epochs range

Hi there!
I was wondering whether you can provide a range for typically "good" values to use/test for the argument num_epochs both in the single label classification case and the multi label classification case. Of course, the best performing number depends on the classes to be predicted and the dataset, but in non-FSL settings, typically one uses a range between 2-5 (whereas many researchers may also stick to common defaults such as 3). I'm asking because I noticed that you use rather num_epochs = 20 in your example scripts, so perhaps in general in setfit num_epochs should be higher than in non-FSL settings?

Thanks a lot for the clarification! Looking at the code, isn't invoking the sentence pair generation methods num_iterations times the same as invoking it just once and then repeating the resulting list of pairs num_iterations times?

repo: huggingface/setfit


text:
Add sd3 pipeline and models

This PR supports SD3 models via https://github.com/apple/ml-stable-diffusion/pull/329
It includes two new models exported via DiffusionKit

coreml-stable-diffusion-3-medium

Ideal for quick generations
Does not include T5

coreml-stable-diffusion-3-medium-1024-t5

Includes all the bells and whistles: T5 text encoder and 1024x1024 resolution
Better quality generations at higher resolution
Slower than the 512 model and requires more memory, so use with caution

Note:

SD3 medium prefers a guidance scale of 5.0, different from the app default of 7.5

It also includes UserDefaults memory for both prompt and generation config.
More quantizations and speedups to come, so stay tuned!

Worth noting that SD3 is having some issue on macOS 15 with memory & speed, will try to drill down on this issue before the the OS comes out.

repo: huggingface/swift-coreml-diffusers


text:
Is there a way to defines "bad_words"?

I want to define tokens that I don't to be ever generated by the model. Is there a simple way to do it using the API?

@tonylek maybe you might be interested in the logit_bias parameter, which seems not supported in TGI yet. (I might be wrong).

authors: kozistr tonylek

repo: huggingface/text-generation-inference


text:
C++11 & HPP_POSTCONDITION with  BOOST_SCOPE_EXIT

Hi,
If we try to compile this project in c++11, the tests do not pass, because at
https://github.com/humanoid-path-planner/hpp-util/blob/master/include/hpp/util/assertion.hh#L71 if an exception is raised, it can't be catched there:
https://github.com/humanoid-path-planner/hpp-util/blob/master/tests/assertion.cc#L101
Boost's doc on BOOST_SCOPE_EXIT provides the following warning since at least 1.50 (released in 2012):

Warning: The implementation executes the scope exit body within a destructor thus the scope exit body must never throw in order to comply with STL exception safety requirements.

I don't think that this HPP_POSTCONDITION is used anywhere, so I think that we should remove it, otherwise we would have to add CMAKE_CXX_STANDARD=98 to the CMakeLists (and to the packages that use this HPP_POSTCONDITION), to allow compilation of this package on compilers which have c++11 as the default standard, which is the case for the those provided with ubuntu 18.04.

PS: in the last HPP-Meeting, I recall that we agreed on the idea that "HPP should compile on both C++98 & C++11" :)

repo: humanoid-path-planner/hpp-util


text:
Feat/leaderboard datagrid

task HDM-20

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.2 out of 3 committers have signed the CLA.:white_check_mark: rozekmichal:white_check_mark: KacperKoza343:x: migaceYou have signed the CLA already but the status is still pending? Let us recheck it.

Do we need this PR? It seems to be part of https://github.com/humanprotocol/human-protocol/pull/2661

"View All" is not available anymore, is this expected?
Also font size is changed from 16 to 14, not sure if it's an intended change

should be ok :)

authors: CLAassistant KacperKoza343 dnechay mgacek-blockydevs

repo: humanprotocol/human-protocol


text:
convert to symbol in MyAnimeListScrapeWorker

This will fix a lot of our failed Sidekiq workers

Wow where do we even screw this one up but otherwise :+1: though I guess we should fix whatever is making the issues

Yeah, we do pass it as a symbol but I assume Sidekiq doesn't care.

authors: NuckChorris vevix

repo: hummingbird-me/hummingbird


text:
:arrow_up: deps

:speech_balloon: :arrow_up:

:+1:


text:
Binance perpetual - Unhandled error in background task: 'NoneType' object has no attribute 'amount'

Describe the bug
We ran a long term bot running on a modified v2_market-making_dman_v3_multiple_pairs.py with 3 market pairs on Binance perp and reviewed the log files, we observed that there are some occurrences that when it was placing take profit orders to close a position, we are getting the ERROR - Unhandled error in background task: 'NoneType' object has no attribute 'amount'
2023-10-11 02:34:00,543 - 15 - hummingbot.core.utils.async_utils - ERROR - Unhandled error in background task: 'NoneType' object has no attribute 'amount'
Traceback (most recent call last):
  File "/home/hummingbot/hummingbot/core/utils/async_utils.py", line 9, in safe_wrapper
    return await c
  File "/home/hummingbot/hummingbot/smart_components/smart_component_base.py", line 72, in control_loop
    await self.control_task()
  File "/home/hummingbot/hummingbot/smart_components/executors/position_executor/position_executor.py", line 222, in control_task
    self.control_barriers()
  File "/home/hummingbot/hummingbot/smart_components/executors/position_executor/position_executor.py", line 262, in control_barriers
    self.control_take_profit()
  File "/home/hummingbot/hummingbot/smart_components/executors/position_executor/position_executor.py", line 291, in control_take_profit
    elif not math.isclose(self.take_profit_order.order.amount, self.open_order.executed_amount_base):
AttributeError: 'NoneType' object has no attribute 'amount'

It seems that the code is trying to access an attribute called 'amount' on an object, but the object is None. While the error happened, it was able to create the take profit order and eventually filled (see sample clientID x-3QreWesySTBUT607597bc0ddc21b13)

Note: opened this ticket for monitoring:
- Noticed that this happened when placing order for take profit (checking with stop loss)
- Order status was filled and not expired
- No stuck orders ✅

Steps to reproduce

Setup a long term bot on cloud for more than 24hours
Review log file and look for instances of placing orders

Release version
dev-1.21.0
Type of installation
Source
Attach required files
logs_test_mm_dman_v3_multipairs.log added # comments

@rapcmia this issue should be solved in order_level_builder branch

Fixed

authors: cardosofede rapcmia

repo: hummingbot/hummingbot


text:
Feat/polkadex v2

Before submitting this PR, please make sure:

[X] Your code builds clean without any errors or warnings
[X] You are using approved title ("feat/", "fix/", "docs/", "refactor/")

A description of the changes proposed in the pull request:
This PR introduces an update to the Polkadex connector to use the V2 API.
Tests performed by the developer:

Unit tests.

Tips for QA testing:
Note that every now and then, the periodic order status update may fail because it happened immediately after submitting an order. Since Polkadex is a decentralized exchange, it takes a couple of seconds for the order to make it onto the chain and to become available for order status updates.
[ch-43708]

PR update:

Test latest commit 061d77b6ec42a7
Request CA QA team to share account for mainnet and successfully import ✅
Polkadex available on connect command
Successfully connected using mnemonic passphrase with balance displayed correct ✅
Setup simple PMM on DOT/USDT

Observe order fetch status failed which is expected for the connector
Orders are created and cancelled successfully, no stuck order observed
Set tight spreads with longer refresh time but unable to have a opportunity for trade with running for 4hrs

Build docker image and ran tests, all ok

authors: petioptrv rapcmia


text:
Fix #121

This adds a BlobUtils.js module to manage a cache for BLOB Object URLs, keyed on path and url.  It can be used to synchronously look up a URL or filename (e.g., by a LiveDev server implementing pathToUrl() or urlToPath()).  I'm thinking we can use this in #120 and #119.  I've also exposed the extensionToMime method here, so we can re-use that in extension server code if necessary.
This also removes the SampleFileSystem.js file we aren't using.
r? @K-LV, @gideonthomas

We've already fix this.

authors: K-LV humphd

repo: humphd/brackets

license: mit


text:
🛑 bitwarden is down

In a9fc2c4, bitwarden (https://vault.thekor.eu/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: bitwarden is back up in 136396e after 3 days, 22 hours, 55 minutes.

repo: hupratt/upptime


text:
🛑 bitwarden is down

In e5aecf8, bitwarden (https://bitwarden.craftstudios.shop/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: bitwarden is back up in b3a2cb3 after 10 minutes.


text:
Using pretrained Word2vec for sentiment dataset

Salam En. Husein.
Im having problem to use the pre-trained word2vec for the sentiment corpus in your malaya project.
How can i fill the pretrained vectors to the word index from the dataset of size 3600.
Label        text
negative   lebih-lebih lagi dengan kemudahan internet dan laman sosial
positive    boleh memberi teguran kepada parti
negative   adalah memningungkan mengapa masyarakat cina bersorak gembira apabila
positive     kami menurunkan defisit daripada 6.7 peratus pada tahun 2009
embedding_matrix = np.zeros((token_count, EMBEDDING_DIM))
for word, i in word_index.items():
embedding_vector = word2vec.get(word)  _#error here_i can't apply word_vector_news
if embedding_vector is not None:
embedding_matrix[i,:] = embedding_vector
The target is to get the weights=[embedding_matrix] for the model that i build
embedding_layer = Embedding(
token_count,
EMBEDDING_DIM,
weights=[embedding_matrix]
input_length=MAX_SEQUENCE_LENGTH,
trainable=False)
Please help! i'm tryin to learn text analysis, and quite new. Hoping you help. tq

Lebih baik encik menggunakan https://malaya.readthedocs.io/en/latest/load-sentiment.html#Sentence-level sebab outputnya (EMBEDDING_DIM) dari pretrained word2vec sebab nanti encik kena aggregate (token_count, EMBEDDING_DIM) untuk jadi (EMBEDDING_DIM)

hmm, tak berapa faham ni.. hmm,  maksudnye kene aggregate (token_count, EMBEDDING_DIM) untuk jadi embedding_matrix ke embedding_dim? saya nak cari mcm mana nak dapatkan embedding_matrix, x boleh nak map pretrained vector kepada word_index saya
Suggestion en husein tu ialah existing model untuk predict sentiment ya..?

Join discord untuk diskus lebih lanjut, https://discord.gg/aNzbnRqt3A


text:
TransfuserDP Baseline

Hi!
I want to reproduce the TransfuserDP baseline in your paper. Is the baseline model already in the current repo? If so, what is the training command? Thanks!

No, we have not provided the baseline model at this time. While we may consider implementing it in the future, it is not currently part of our plans.

authors: BlueCat-de LegendBC

repo: hustvl/DiffusionDrive


text:
Artifacts with Tall waves and Foam Bubbles

Hey!  We've been noticing a weird artifact for a while now, and we thought it was something with our fog.  However, we just realised what we're seeing is actually being caused by the bubble foam.  Basically, when you look through a tall wave at a low angle (like from our gameplay view), you see this odd infinite plane stretching off into the distance.  I recorded a video demonstrating the issue (I set the bubble foam to be bright red to make it easier to see):
https://drive.google.com/open?id=1gATnANV2aGEeXzBCGCtQRmOxtgotIA4Q
I vaguely remember you making a change some time back to make the bubble foam have a sort of paralax offset or something.  Could that be related?

Yeah its exactly that. The parallax is done by dividing by the view y - this implicitly assumes that the normal vector is up, i.e. a flat horizontal surface.
half bubbleFoamTexValue = tex2Dlod(_FoamTexture, float4(.74 * foamUVBubbles - .05*i_view.xz / i_view.y, 0., 5.)).r;

I think  i_view.xz needs to be the view vector projected onto the tangent frame at that surface pos, and i_view.y needs to be dot(i_view,normal) or something. I didnt do this to save the extra instructions (i thought it added "character" :)) but it sounds like you're finding it disturbing?

Sure, do you want to try it locally before we commit to it and make sure thats will work for you? You can kill the view component on that line - change the .05 to a 0. for example.

Thanks, just tried that out.  Yeah, that solves the issue with tall waves.  Either a toggle for the parrallax or even a slider to scale the effect would be a good, versatile solution. :)

Added now, seems to work for me!

@Midda-C found a fix that should only add one instruction and seems to work pretty well so maybe this is usable now?
I made a vid but google docs seems to be acting funky so not sure if you'll be able to access this or not:
https://drive.google.com/open?id=1p4CTVU6Cj_xTkpPGsAHhfQjFNz5RPzNo

Yeah video works fine.  Looks nice, I'll have a play with it and let you know!
Thanks! :D

Just tried this in our project.  It works well, happy to have the foam parallax back without any compromises!  Thanks for revisiting this! :D

authors: Midda-C huwb

repo: huwb/crest-oceanrender


text:
Use git cat-file --batch and git log --name-status to avoid repeated fork+exec

This significantly improves performance, especially for a Git repository stored on a network filesystem with many loose objects, by launching fewer copies of git and hence reading the repository fewer times.

Good work, thanks!

@andersk Could you maybe take a look at http://trac-hacks.org/ticket/9560 this commit seems to have broken Python 2.5 support

authors: andersk hvr

repo: hvr/trac-git-plugin


text:
add more attributes to localdisk

What this PR does / why we need it:
issue: #984
Special notes for your reviewer:
Does this PR introduce a user-facing change?

add more attributes to localdisk as shown below:

- major
- minor
- devLinks

Codecov Report

Merging #1032 (e85c8be) into main (f124e4c) will increase coverage by 13.22%.
Report is 30 commits behind head on main.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@             Coverage Diff             @@
##             main    #1032       +/-   ##
===========================================
+ Coverage   29.80%   43.02%   +13.22%
===========================================
  Files          24       24
  Lines        1570     1571        +1
===========================================
+ Hits          468      676      +208
+ Misses       1040      823      -217
- Partials       62       72       +10

Files Changed
Coverage Δ

.../local-disk-manager/builder/localdisk/localdisk.go
63.88% <100.00%> (+1.57%)
:arrow_up:

... and 7 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

lgtm

repo: hwameistor/hwameistor


text:
handle single arg to and/or

@ryderwishart @eyurtsev thoughts on handling this in the parser itself? related to #5570

@dev2049 yeah that works -- if it gets complex in the future we can separate out to another post-processing step, but looks good where it is now


text:
Fix Lint: Bittensor llm

I fixed the lint so it passes checks for pull #7602
I didn't remove the bearer key in the headers on line 58 in case it's there on purpose for whatever reason that could be. However, @Kunj-2206 if that's left in on purpose, either reply here to remove it or git checkout this pull req and make a new one.

Hey @kdcokenny , that Bearer Key is on purpose. Thanks for updating the code in required format

Hey @kdcokenny , that Bearer Key is on purpose. Thanks for updating the code in required format So now it is ready for merge?

No, you will need to git checkout my pr, then add notebooks showing how it's used as well as some tests.

Hey @kdcokenny , I added test case and notebook and I commited everything in my PR can you please check that

Hey @kdcokenny , I added test case and notebook and I commited everything in my PR can you please check that

It lgtm, but your PR has been closed. Open it so the project maintainers can review & merge it


text:
Source on main is at version 3, crates.io is still at version 2.0.2?

@complexspaces I'm finally getting around to the next release of the keyring-rs crate, and I'm trying to figure out which released version of secret-service it should use.  I am going to stay with dbus (rather than zbus) because I don't want to force my clients to use an async runtime, so in that sense I could stay with 2.0.2, but then again I'd like to include the unlock improvements of a269ee7 by @jkhsjdhjs, which are only in 3.0.  Are you going to release 3.0 to crates.io anytime soon?  And will I be able to use feature flags to stay with the dbus interface in 3.0?
cc @Sytten, @landhb

Thing is v2 still uses zbus just the sync api which spawns a single thread tokio executor to call the async api. Even worst it pulls in by default both tokio and async-io runtimes...
Last time I looked at it, my conclusion was that the secret service crate should have two implementations, one sync that maybe uses the dbus crate (C wrapper) and one async interface that uses zbus (or dbus-tokio).
I just wish zbus would have decoupled the transport from the dbus implementation (similar to what diesel did with diesel-async).
Right now even if you dont want to impose an async runtime @brotskydotcom you kinda do because it will use tokio under the hood. Though I agree that keyring should remain sync. One could really wish for an STD runtime that would just block on async code 😑

Wow, thanks @Sytten for laying that out for me. As you may recall, I don't spend a lot of time in Linux-land so this is really helpful education for me.
Does this mean that the current keyring-rs release is also causing its clients to pull in asynchronous runtimes ("under the hood", as we say in the US)? So whether I use v2 or v3 of secret service in v2 or keyring makes no difference in that respect?

I am not super versed in Linux either but glad I can help with the async /sync stuff.
Exactly it pulls in async runtimes under the hood (zbus does not this crate) since its the only way to run async from sync code.
I believe that v3 is fully async so would have to do this yourself if you want to keep a async api for keyring.

@complexspaces Are you open to a contribution (under feature flag, I would expect) that uses dbus rather than zbus?  (I seem to remember you suggested as much in some other thread.)

Unsure if it is a good idea re-reading the code since literally all the api is async now so that would basically mean forking from v2 and writing another lib...

I think a possible course of action is:

Add a vendored feature to the libdbus-sys crate that compiles libdbus from source and links it statically
Add a vendored feature to the dbus crate that uses the new vendored feature in libdbus-sys
Update the secret-service crate to use the dbus crate for its blocking interface. And feature gate it.
Update the keyring-rs crate to use the new secret-service blocking interface.

That allows downstream users of kerying-rs to not need an async runtime or have libdbus-1-dev installed to compile keyring-rs.
I started the dbus/libdbus vendoring work in a branch here. Just to see if it's viable.

Hi everyone, sorry for the delay. Let me try and answer everyone's questions:

Are you going to release 3.0 to crates.io anytime soon? And will I be able to use feature flags to stay with the dbus interface in 3.0?

As I think was discovered above, no, there are not currently a set of feature flags that will let downstream users use the libdbus C library (without any Rust async runtimes). Regarding the 3.0 release, everything is ready to do so. There aren't any technical blockers, I just got cold feet (sorry 😞) and then ran out of time to balance with.
If there are no breaking changes required/planned for the proposal to add another backend, then my plan is to release 3.0, merge some of the cryptography updates (at work, we're stuck on some older versions and I would prefer to have a 3.0 release for use internally that doesn't duplicate a bunch of dependencies) and then release 3.1 with those.
I am currently out of town until January for vacation, so I won't be able to do much more then comment on GitHub until then sorry. But once I'm back, a 3.0 release would take 1-2 days at most.

Are you open to a contribution (under feature flag, I would expect) that uses dbus rather than zbus? (I seem to remember you suggested as much in some other thread.)

re-reading the code since literally all the api is async now so that would basically mean forking from v2 and writing another lib...

Yes, I am still open to contributions adding this. Given that the crate now has a blocking module, I imagine that this would be the best place to plug in the alternate backend. @Sytten this should also hopefully resolve your concern about there being no synchronous API remaining, right?
If possible, it would be nice to give users of the keyring-rs crate a way to opt-out of the C backend, primary so that having the crate in your tree doesn't force any independent use of secret-service-rs to use libdbus too. That may be a nicety for anyone who already has Tokio or async-std in-tree and wants to avoid a C dependency.

Hi all,
Thanks so much for the education and context.  I think I have a pretty good picture of the situation right now, and here's what I'm planning.  All feedback welcome!

As soon as I finish testing, I'm going to release a new alpha of keyring-rs that uses the blocking API of the current main branch of secret-service.  There will be features, analogous to those in secret-service, that control which async runtime gets used and which encryption mechanism gets used, with the default being async-io and crypto-rust.
The new alpha will have a feature to disable the use of secret-service completely (so that none of zbus or its dependencies are linked at all).  Clients who disable secret-service can use the linux keyring as their credential store (or bring their own).
Once secret-service 3.0 is released, I will release an rc of keyring-rs 2.0 that uses it.
If anyone gets around to adding a secret-service feature that links the blocking API (or both APIs) to dbus rather than zbus, I will expose that feature in a single-dot release of keyring-rs (assuming the dbus-version is blocking-API-compatible with the zbus version).

This plan seems to me to balance backward-compatibility for keyring-rs (for async-naive clients) with sufficient controls over zbus behavior (for async-aware clients) with sufficient controls over dependencies (for embedded or otherwise zbus-averse clients).  And it leaves the door open for an API-compatible release of keyring-rs if anyone ever moves forward with a dbus-based secret-service.
Your thoughts?

After merging https://github.com/diwic/dbus-rs/pull/408, I began trying to get a start on updating the secret-service crate to use the dbus crate for its blocking interface.
It's looking like a larger refactor than I originally anticipated. Since most of the core library is tightly coupled with zbus.
Ideally the secret-service Cargo.toml would end up looking something like the following, to make zbus fully optional:
[features]
# Default to using rust crypto backend, force user to choose a Mode
default = ["crypto-rust"]
# Cryptography backends
crypto-rust = ["dep:aes", "dep:block-modes", "dep:sha2", "dep:hkdf"]
crypto-openssl = ["dep:openssl"]
# Modes (Blocking or specifying a runtime)
# The async runtime features mirror those of `zbus` for compatibility.
blocking = ["dep:dbus"]
rt-async-io = ["zbus/async-io"]
rt-tokio = ["zbus/tokio"]

[dependencies]
# Crypto: Rust backend
# TODO: Update these when Rust 1.56 isn't so new.
aes = { version = "0.7.0", optional = true }
block-modes = { version = "0.8.0", optional = true }
hkdf = { version = "0.12.0", optional = true }
sha2 = { version = "0.10.0", optional = true }
# Crypto: Openssl backend
openssl = { version = "^0.10.40", optional = true }
# Mode: Blocking/sync backend
dbus = { git = "https://github.com/diwic/dbus-rs.git", features = ["vendored"], optional = true}
# Mode: Async backend (zbus will pull in the relevant run time)
zbus = { version = "3", default-features = false, optional = true}
...

And then the top-level lib.rs looking something like:
/// Error when a runtime has not been selected
#[cfg(not(any(
    feature = "rt-async-io",
    feature = "rt-tokio",
    feature = "blocking",
)))]
mod error_message {
    compile_error!(
        "A Mode must be selected, either \"blocking\", \
        \"rt-async-io\" or \"rt-tokio\" must be enabled."
    );
}

/// Error when a crypto backend has not been selected
#[cfg(not(any(
    feature = "crypto-rust",
    feature = "openssl",
)))]
mod error_message {
    compile_error!(
        "A crypto backend must be selected, either \
        \"crypto-rust\" or \"openssl\" must be enabled."
    );
}

// Generic modules
mod error;

/// Async mode
#[cfg(any(feature = "rt-async-io", feature = "rt-tokio"))]
pub mod runtime;

/// Sync mode
#[cfg(feature = "blocking")]
pub mod blocking;

So I think that your point number 4 may run into issues, since I anticipate potential breaking changes to occur within secret-service when moving all the async code into a separate module. Since all the zbus dependent code will need to be feature gated.

Actually it may not affect users of keyring aside from removing the runtime features you're exposing.

Thanks @landhb for the feedback on number 4.  I am planning to make introducing a new credential store into keyring-rs fully backwards compatible (with respect to the client API) and thus something that can be done in a dot release.  Assuming I succeed, then: if the secret-service blocking API were to change a lot with the move to dbus, or if we needed to split dbus support into a separate sub-module completely, I could handle it in keyring-rs simply by introducing a new credential store that used the new API.
What this does make me realize, however, is that I probably should make the error enum returned by keyring be explicitly non-exhaustive, so that new credential stores can introduce new error types without breaking the clients.

I have published keyring-rs v2 alpha 4 to GitHub so folks can try it out.  It can't be published to crates.io until after secret-service v3 is published.

Hi @complexspaces, Happy New Year!  Just checking in on this.  It would be great to have a 3.0 release; that will allow me to release keyring 2.0.  If you think it will be a while before you do the 3.0 release, that's fine!  Just let me know and I will release keyring 2.0 against secret-service 2.0.2 and then update it in a dot once 3.0 comes out.

I've reverted back to secret service 2.0.2 so I can release keyring-rs v2.  I just pushed beta.1 to crates.io.  I know exactly what's needed to move back to secret service 3.0 so I can do a dot whenever that happens.

Hey @brotskydotcom, I'm really sorry. This month has been far rougher then I expected. I've had to slow down on many non-primary obligations.
Given some of the research done above (thank you everyone), I think that we should be good to cut v3 and keep it for a while. The dbus crate offers a non-blocking backend as well so I still believe adding it as an alternative to zbus should be possible without breaking changes. Even if that turns out to be wrong, hopefully a v4 release won't be that painful for downstream users.
I will do my best to have v3 out by the weekend. Please speak up in the next day or so if you have any issues with the current API.

Hi @complexspaces, wonderful news and thanks for all your effort!  If you can do a release in the next week I will happily use it in keyring 2.0.

I've now published 3.0 on crates.io. Please let me know if anything about it ended up not working (hopefully nothing! 😅).
Regarding the dbus-powered backend, I'm still open to accepting a PR for this. I'm going to close this issue, since crates.io now matches master, and point discussion of that topic to https://github.com/hwchen/secret-service-rs/issues/64 instead.

Thanks much @complexspaces for doing this release!  I will move keyring-rs onto it and release this week.

authors: Sytten brotskydotcom complexspaces landhb

repo: hwchen/secret-service-rs


text:
《Vim基础命令》最后一行的链接格式错了

http://hweining.com/2016/12/01/Vim/
估计是因为*后面没有留空格的原因。

谢谢指正

authors: hweining polunzh


text:
4주차 과제 실행화면

1번 실행화면입니다.

2번실행화면 // 범위를 define함수로 정의를 할때 구간을 어떻게 깔끔히 나눌까 고민이 됐었는데 4개로 범위를 따로따로 정하여 나타내었다.

![4주 3번 실행화면 - 덧셈](https://user-images.githubusercontent.com/112994126/192735076-41e30407-ea47-48ca-ad12-7bcd81d1a2eb.p
![4주 3번 실행화면

뺄

셈](https://user-images.githubusercontent.com/112994126/192735095-338d56e0-99eb-45aa-a350-636be8f8ab02.png)
ng)

3번 실행화면 // 차례로 덧셈, 뺄셈, 곱셈, 나눗셈 //

3번 실행화면 //덧셈 연산식

3번 실행화면 //뺄셈 연산식

3번 실행화면//곱셈 연산식

3번실행화면 // 나눗셈 연산식
@swcho84


text:
👩💻Integration specs for views and ⚙ fix n+1 problems

Hello , 👋 in this milestone we :
✅ Solved N+1 problem is solved when fetching all posts and their comments for a users

✅Use Capybara to write integration tests for each view in the project for the user and the posts .
Kindly ♻ Thank .

Hi @hydaline-code
Can you kindly add a professional ReadMe to you project using this template ASAP!
Standing by..cheers!

Readme updated

Hi @hydaline-code
Again, lets improve the ReadMe by removing or updating the underlined:

Hello @rloterh
Update done 👍

Hi @hydaline-code
Still showing, under the Key Features, can you check once more?

Hi @hydaline-code Kindly ensure to commit and push the changes so that it reflects on your repository..cheers

authors: hydaline-code rloterh

repo: hydaline-code/myblogapp


text:
[columnar] PostgreSQL 16 compatibility

PG16 compatibility

The upgrade test is failing as it needs an initial image to pull, I believe the workaround is to build and push one manually for that to pass.

mm, new failures here because parquet s3 fdw doesnt exist for pg16.

authors: mkaruza wuputah

repo: hydradatabase/hydra


text:
Empty string passed Check validation.

Hi! Thanks for this project. I'm using it to validate json files and an empty string passes validation and I think it should not.

Hi,
Thanks for your feedback.
See, empty values are valid as in {"k":""}.
Also, empty keys are valid like this: {"":"v"}.
See JSON Lint as reference.
Regards,
Marcelo.

Thanks you for your quick response. I know that empty strings are valid but, what I was trying to say is that, if you pass an empty string to validate ( i.e. check('') ) it returns true and it should return false, just like JSON Lint does if you simply click on the [Validate JSON] button without any text in the textarea.
Best regards.

Oh, Now I got it.
It is simple to implement.
Meanwhile, you can check just not empty strings, right? :-)
I'll share it in next update.
Best.

Done!
Sorry for the delay.

authors: hydrobyte mnabarro

repo: hydrobyte/McJSON


text:
Retrieving password

When I try to retrieve a password for a previously-registered (or perhaps partially registered) email, I am told 'invalid user/email.' However, if I try to register a new account with that email I am told that the email is not unique (i.e., appears to be logged somewhere in the system).

@wzell-usgs - response sent via email

Resolved via email

There is still a systemic problem here that needs to be addressed.  An account can be in one of two states:
A.  Active after a user has validated their email.
B.  Pending.  After a user has requested an account an email to validate their account is sent to the email provided, but the user has not yet clicked on the link in that email.

If a user requests an account and provides an email for which an account is active they should be presented the following message:

An account already exists for this email address.  If you have lost your password, reset your password.

If a user attempts to log in to an account that is pending (they enter correct username/email and password pair, account has been requested but validation email not clicked) they should be presented with a page that says:

This account has not been validated.  Check your spam folder for the verification email sent to you.  Resend the validation email.

If a uses requests an account and provides an email for which an account is pending they should be presented the following message.

An account with this email address has been requested but not validated.  Check your spam folder for the verification email sent to you.  Resend the validation email.

If a user requests a password reset for an email address of an account that is pending they should be presented the following message.

An account with this email address has been requested but not validated so the password cannot be reset.  Check your spam folder for the verification email sent to you.  Resend the validation email.

If a user requests an account and provides a username for an account that is active, but an email that is unique they should be presented with the message:

The username you requested has already been taken by a different user, please enter a different username.

Initial user issue is resolved, subsequent work opened as new issue [#966]

authors: dtarb mjstealey wzell-usgs

repo: hydroshare/hydroshare


text:
关于路由的多种访问方式

这样是不是不利于统一？？

？

你是说路由的定义方式太多了？
代码规范的问题，你们可以在内部使用的时候来定，比如我司这里只用路由文件不去使用路由相关的注解。

你是说路由的定义方式太多了？
代码规范的问题，你们可以在内部使用的时候来定，比如我司这里只用路由文件不去使用路由相关的注解。

你是说路由的定义方式太多了？
代码规范的问题，你们可以在内部使用的时候来定，比如我司这里只用路由文件不去使用路由相关的注解。

奥，你们这个 控制器::方法 和控制器@方法有什么区别吗?

没区别

没区别

再多问一下： 难道::不是不用实例化控制器；直接静态调用； 而@是实例化控制器？？？

不是的。。都会实例化控制器，只不过写法不同而已。
实例化控制器也只会实例化一次

authors: limingxinleo wotemenige

repo: hyperf/hyperf


text:
.env里的参数必须用引号引起来，不然创建model就报错[BUG]

部署在 centos 系统
原始 .env

报错截图

修改后的 .env 就不报错了

你的密码里面包含了特殊字符(! #)，需要使用引号包裹起来的。

在env 文件中 #被认为是注释，导致你的密码最后一位没有取到，所以提示密码错误

还是建议官方给示例加上引号，这个坑了我两小时

@whtcgh 这个坑自己知道就好了，添加引号会显得不够简洁，我觉得目前这个没啥问题😁

/switch-to question

authors: hesunfly huangzhhui whtcgh


text:
Add gRPC API for exitcode

reopen #339
Add gRPC API for exitcode
part of #278
Signed-off-by: Crazykev crazykev@zju.edu.cn

LGTM

authors: Crazykev feiskyer

repo: hyperhq/hyperd


text:
JSON RPC/Hyperion: Enable/Disable components during runtime

Hyperion

[ ] smoothing (activate bypass as it is already used if enable false?)
[ ] blackborder
[x] kodicheck
[ ] forwarder
[x] udplistener
[x] boblight server
[ ] grabbers (this should be handled though a platform depedent selection. To keep the command easy/platform independent)

JSON RPC

[ ] smoothing
[ ] blackborder
[ ] kodicheck
[ ] forwarder
[ ] udplistener
[ ] boblight server
[ ] grabbers

So close to cose! :D
Very good, can't wait for testing :)

I can tell you, it's amazing.
A side effect ... I had to make big grabber refactoring ... that's why it take longer as expected. Last thing the bb detection is also tricky, because it is hidden in templates and factory ...

bb was easier as I thought, because of my prior grabber refactoring all needed code was just there. Only a few additions were necessary.

authors: brindosch redPanther


text:
Add Google chat notification channel

Monika Pull Request (PR)
What feature/issue does this PR add

Closes issue #494
Adds google chat as a notification channel

How did you implement / how did you fix it

Add handling for type 'google-chat'
Add  json formatting specific for google chat.

How to test

Create a new notification of type 'google-chat'
Add the spaces/google chat bot and generate the url webhook like in the google instruction
Run monika and observe notifications on Google Chat space

startup message

incident message

Also please check why the tests is failing :grinning:

authors: dennypradipta sapiderman

repo: hyperjumptech/monika


text:
Hyperledger Besu re-deployment fails

Describe the bug
The BAF Hyperledger Besu deployment is not idempotent, it fails when running the deployment again without resetting.
To Reproduce
Steps to reproduce the behavior:

Deploy a hyperledger besu network successfully using BAF.
After all the nodes are up, redeploy using the same network.yaml
The deployment fails with the error (UPDATE required)

Expected behavior
The deployment should not fail. It should succeed even when there is nothing to change.

This is not fixed. Please check the demo pipelines. Redeployment failing with following error:
TASK [helm_component : create value file for supplychainvalidator1ibft] ********
[WARNING]: Unable to find '/jenkins/workspace/BAF_Demo_multi_besu-
VAHPN2FP7GGJMY3ZV4HIPANXNZ7UBA3OIX52A2VXHYHD2DRF3WDA/build/besu_genesis' in
expected paths (use -vvvvv to see paths)
fatal: [localhost]: FAILED! => {"changed": false, "msg": "AnsibleError: An unhandled exception occurred while templating '{{ lookup('file', '{{ network.config.genesis }}') }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /jenkins/workspace/BAF_Demo_multi_besu-VAHPN2FP7GGJMY3ZV4HIPANXNZ7UBA3OIX52A2VXHYHD2DRF3WDA/build/besu_genesis"}

authors: jagpreetsinghsasan sownak


text:
Create charts for Quorum Express API

Description
As a developer I want to deploy Quorum expressAPI on Kubernetes so that I can run applications on Quorum.
Acceptance Criteria

Use the docker image for express api
Get the node address and docker address from Vault
The pods should be able to communicate with Quorum network.

@suvajit-sarkar The new charts have been checked in as per #600  These needs an update so as to get the node_identity and product_contract_address from Vault rather than values.yaml


text:
Archive fabric-private-chaincode which has transitioned to Fabric

Signed-off-by: Arnaud J Le Hors lehors@us.ibm.com

@trbs @vipinsun : Can you please review?

@lehors @tkuhrt I propose deleting this repo: https://github.com/hyperledger-labs/fabric-private-chaincode-wasm since it was never used (as a part of this process)

Sounds good to me.

authors: lehors ryjones tkuhrt