text:
Sonarr doesn't work after update using -U

I'm on raspbian. It worked perfectly fine before update. It won't write any logs and I can't figure out what the actual error is. Perhaps you can help me trace it?
Feb 28 10:44:50 osmc systemd[1]: Started Sonarr.
Feb 28 10:44:52 osmc mono[16309]: Press enter to exit...
Feb 28 10:44:53 osmc systemd[1]: sonarr.service: Main process exited, code=exited, status=1/FAILURE
Feb 28 10:44:53 osmc systemd[1]: sonarr.service: Unit entered failed state.
Feb 28 10:44:53 osmc systemd[1]: sonarr.service: Failed with result 'exit-code'.
Feb 28 10:44:53 osmc systemd[1]: sonarr.service: Service hold-off time over, scheduling restart.
Feb 28 10:44:53 osmc systemd[1]: Stopped Sonarr.

This is most likely related to the issue logged here Sonarr Wont start#2431
We currently use the Stable version of Mono which has been updated and appears to have broken Sonarr
It seems they have a fix that they've pushed to the Sonarr Dev branch but are just testing it at the moment...
I would hold tight for the minute and lets see what happens. ATK Should be able to cope with automatically fixing it once its been merged up.
I think this could have possibly affected a few services which use Mono:
Radarr
Jackett

Ah thanks. I'll just wait for an update then.

If and only if you know what your doing :)
https://download.sonarr.tv/hotfixes/regression-mono-5.10/NLog.dll.gz
replace (or just rename the current Nlog.dll to something like NLog.dll.old) NLog.dll with the above file.
Note: This file is from the dev section. (majority of sonarr will work).

Not sure if this is at all related to this issue. I reinstalled my whole server yesterday. After installing ATK I was trying to get SABnzbd, Sonarr, and Radarr. SAB was fine but Sonarr and Radarr both had issues. The both seem to install fine but they stay in "inactive" state and both claim that the config.xml does not exist (somewhere under the home dir).
I can link screenshots later in the evening today.
Just want to know if that would have to be a new issue or if it may be related to this ongoing Mono issue.

When it will be fix and available as an update, are we talking about days? Weeks?
Thanks

Thanks @Mohammed-JS-Khan for the explanation.
As mentioned above, this isn't a problem with ATK. Sonarr, Radarr and Jackett all have issues logged on their github sites related to this, so a fix is completely dependant on when they decide to push the new NLog.dll into the master branches. When they do a simple update in ATK will pick it up and apply it automatically.
I have no immediate plans to fix this in ATK as I imagine that they will get a fix out soon but obviously I have no idea when!
Please remember that I do this pretty much completely on my own, in my own time for fun, for free. I only took over the admin rights to this github when it became easier when i was pushing a lot of fixes up when all the old devs faded away. I'm struggling to keep up with the demand for fixes and features now on my own and fear that without other people stepping in to help, ATK is going to eventually die.
If anyone wants to add a temporary fix to ATK until all the relevant services have been fixed I'll be happy to consider it.
Thanks
Tom

Sonarr should be fixed at this point. If anyone is able to confirm that using ATK's update menu fixes Sonarr this issue can be closed. Mine was already updated before the issue hit so I never had a problem or I'd confirm myself :)

I updated mine with Atomic Toolkit (ATK) manual update script and it is now up and running.

Super!
Thanks @campbc for letting us know.
I recon we are ok with this now but I'll keep it open just a bit longer so anyone with issues can see it should be ok now.
If anyone wants to also confirm, it would be handy to know.
Thanks everyone for being patient but this one was slightly out of our hands!
Tom

@vrokolos are you working OK now? If so i'll close this off.

Yes thanks

authors: Mohammed-JS-Khan TommyE123 campbc dutch2005 lordraiden nemchik vrokolos

repo: htpcBeginner/AtoMiC-ToolKit


text:
Improve error reporting from parboiled parsers

Our existing delivery scheme discards too much information from the parse error.  Nearly every header parser that fails only returns "entry" as the detail.  Now we get:
scala> HttpHeaderParser.CONTENT_LENGTH("blech").leftMap(_.message)
res4: String \/ org.http4s.headers.Content-Length =
-\/(Invalid header: Invalid input 'b', expected entry (line 1, column 1):
blech
^)

All the leftMapping is so we can bring the ParseError back to where we have the input.
This will make some of the headers work @wedens is doing on #1143 more useful.

Alright, let's do this. Anything that breaks, needs to.

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
Upgrade dependencies

Round of patch upgrades for 0.15.13.  Blaze is the important one.

👍 if Travis is happy.

authors: bryce-anderson rossabaker


text:
Update okhttp to 4.9.0

Updates com.squareup.okhttp3:okhttp from 4.8.1 to 4.9.0.
Changelog
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.squareup.okhttp3", artifactId = "okhttp" } ]

labels: library-update, semver-minor

I don't know enough Kotlin to know the ramifications of a series/0.21 backport.

authors: rossabaker scala-steward


text:
unable to CONTRIBUTING

Checklist

[x] I've searched for similar feature requests.
Yes

Enhancement request
…

Problem it solves
$ make test
\n\n\033[0;32m### Running tests ### \033[0m\n
venv/bin/python -m pytest
process_begin: CreateProcess(NULL, venv/bin/python -m pytest, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [Makefile:107: test] Error 2
(venv)
Admin@Hellbat MINGW64 /d/Open Source Contribution/cli (fixing/prasanna)
$ make all
\n\n\033[0;32m### Uninstalling httpie ### \033[0m\n
venv/bin/pip3 uninstall --yes httpie &2>/dev/null
/usr/bin/sh: line 1: venv/bin/pip3: No such file or directory
Verifyingâ€¦
cd .. && ! venv/bin/python -m httpie --version &2>/dev/null
/usr/bin/sh: line 1: venv/bin/python: No such file or directory
Done

\n\n\033[0;32m### Updating package tools ### \033[0m\n
venv/bin/pip3 install --upgrade pip wheel build
process_begin: CreateProcess(NULL, venv/bin/pip3 install --upgrade pip wheel build, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [Makefile:53: install-reqs] Error 2

Additional information, screenshots, or code examples
$ python --version
Python 3.12.2
…

You may have missed a step in preparing your development environment.
Try the following commands:
$ python -m pip install --upgrade pip wheel
$ python -m pip install --upgrade '.[dev]'
$ python -m pytest --verbose ./httpie ./tests

If this does not help, try to start off a clean environment and redo the provided steps.
I can confirm that HTTPie pipelines work as of today.

You may have missed a step in preparing your development environment.
Try the following commands:
$ python -m pip install --upgrade pip wheel
$ python -m pip install --upgrade '.[dev]'
$ python -m pytest --verbose ./httpie ./tests

If this does not help, try to start off a clean environment and redo the provided steps.
I can confirm that HTTPie pipelines work as of today.

thank for that. but already fix that.

source: gharchive/issue

repo: httpie/cli


text:
chore (setup.cfg): update request to  2.32.1

Related Issue
#1597

Newer versions of Requests have an issue with the way we pass our custom SSL context. That needs to be fixed before we can upgrade.
See https://github.com/httpie/cli/issues/1583 https://github.com/httpie/cli/issues/1581
Example:
$ https google.com

https: error: SSLError: HTTPSConnectionPool(host='google.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) while doing a GET request to URL: https://google.com/

Fixed in v3.2.4 via #1596


text:
断言

问题描述
请对遇到的 bug 进行简要描述。
版本信息
请提供如下版本信息：

操作系统类型: [e.g. macos, Linux, Windows]
Python 版本 [e.g. 3.6]
HttpRunner 版本 [e.g. 2.1.2]

项目文件内容（非必须）
如果可能，提供项目测试用例文件原始内容可加快 bug 定位和修复速度。
提示：请注意在去除项目敏感信息（IP、账号密码、密钥等）后再进行上传。
运行命令 && 堆栈信息
请提供在命令行中运行测试时所在的目录和命令，以及报错时的详细堆栈内容。
$ pwd
/Users/debugtalk/MyProjects/HttpRunner-dev/httprunner/tests
$ hrun testcases/setup.yml
INFO     Loading environment variables from /Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/tests/.env
ERROR    !!!!!!!!!! exception stage: load tests !!!!!!!!!!
ModuleNotFoundError: No module named 'tests.api_server'

请给出具体项目文件内容（包括但不限于 /Users/debugtalk/MyProjects/HttpRunner-dev/httprunner/tests/setup.yml、/Users/debugtalk/MyProjects/HttpRunner-dev/HttpRunner/tests/.env），以及详细的版本信息。
仅凭这一段短短的运行错误，线索太少，无法判定问题。

repo: httprunner/httprunner


text:
在部分Wayland的DM上有黑色遮罩窗口无法使用

archlinux系统，在x11上使用正常。在wayland下，gnome也正常。
但在wayland下的sway或hyprland下，启动docker后，登陆微信的窗口前面被黑色遮罩覆盖。这个问题困扰很久，导致至今无法使用这两个窗口管理器，有遇到的朋友或者解决思路吗？

同样的问题

A quick fix from code below
https://github.com/ChisBread/wechat-box/blob/master/root/bin/wechat-monitor

authors: LiinXemmon chagel cnsvc

repo: huan/docker-wechat


text:
七天预报的折线Y轴计算有问题？

七天预报的折线Y轴计算有问题。布局居中之后两条折线的位置还是和居中之前是一样的。在WeatherView中具体计算我也看不太懂。

好像是因为item中的那条线导致。

repo: huashengzzz/SmartChart

license_source: bigquery


text:
Rename "assay/" to "assays/"

People expect that when they shorten a URL they get a higher level page. Right now we have "docs/assay" and "docs/assays/af", etc. Suggest retitling either the directory or the file to be consistent.

@mccalluc I don't see /assays. Where is that?

Sorry, other way around. File is assays.md; directory is assay; Better to have them consistent.

repo: hubmapconsortium/portal-docs

license: MIT


text:
Luma Array Bug Tracker

Describe the bug
This is a tracker for https://github.com/uber/luma.gl/issues/1345.  We need to be able to pass in an array of textures to make viv scale.

@ilan-gold : Does the work you described this morning impinge on this? Or are they distinct, but happen to share some keywords?

This is distinct for tracking: https://github.com/visgl/luma.gl/issues/1345

authors: ilan-gold mccalluc

repo: hubmapconsortium/vitessce-image-viewer


text:
Upgrade from 7 to 9 causes: "Incorrect number of arguments for middleware callback (expected 1, got 3)"

The code I'm using from the bot is:

 robot.receiveMiddleware((context, next, done) => {
        var text = context.response.message.text;
        var newText = mapping.process(text);
        if (text != newText) {
            context.response.message.text = newText;
            if (robot.logger) {
                robot.logger.info(`Routing '${text}' to '${newText}'.`);
            }
        }
        next(done);
    });

Yes, there was an API change between 7 and v9. Specifically, the move to async/await style code. The code above would look like the following for v9:
 robot.receiveMiddleware(async context => {
        var text = context.response.message.text;
        var newText = mapping.process(text);
        if (text != newText) {
            context.response.message.text = newText;
            if (robot.logger) {
                robot.logger.info(`Routing '${text}' to '${newText}'.`);
            }
        }
        return true;
    });

Returning true will tell the code to continue the pipeline. Returning false will halt the pipeline. Looking at robot.receive, you'll see the infrastructure that does this.

Awesome! Let me upgrade some code. Looks like we also need to update  hubot-mock-adapter.

Fortunately the upgrade was not very hard: https://github.com/KeesCBakker/hubot-command-mapper/commit/2504fd5fa1fbe5c27b04044e0fdceb1a514e8e51

@KeesCBakker I'm really glad to hear that. I've been making some big moves in the codebase with very little feedback to guide decisions.
What are your thoughts about closing this issue?

Yeah, we can close it. Should I open a new ticket for the mock adapter peer dependency?

@KeesCBakker  yes please. That way it stands out as I continue to monitor issues from the main repo page.

authors: KeesCBakker joeyguerra

repo: hubotio/hubot

license: mit


text:
🛑 VGMdb API is down

In 3e911bd, VGMdb API (https://vgmdb.info) was down:

HTTP code: 500
Response time: 23783 ms

Resolved: VGMdb API is back up in 79e902e after 15 minutes.

repo: hufman/upptime


text:
incorrect gradient accumulation with deepspeed

System Info
Copy-and-paste the text below in your GitHub issue

- `Accelerate` version: 0.22.0
- Platform: Linux-5.15.0-1023-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Numpy version: 1.24.3
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 123.22 GB
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
import torch
import copy
from accelerate import Accelerator
from accelerate.utils import set_seed
from torch.utils.data import TensorDataset, DataLoader

# seed
set_seed(0)

# define toy inputs and labels
x = torch.tensor([1., 2., 3., 4., 5., 6., 7., 8.])
y = torch.tensor([2., 4., 6., 8., 10., 12., 14., 16.])
gradient_accumulation_steps = 4
batch_size = len(x) // gradient_accumulation_steps

# define dataset and dataloader
dataset = TensorDataset(x, y)
dataloader = DataLoader(dataset, batch_size=batch_size)

# define model, optimizer and loss function
model = torch.nn.Linear(1, 1)
torch.nn.init.zeros_(model.weight)  # Initialize the weight with zeros
model_clone = copy.deepcopy(model)
criterion = torch.nn.MSELoss()
model_optimizer = torch.optim.SGD(model.parameters(), lr=0.02)
accelerator = Accelerator(gradient_accumulation_steps=gradient_accumulation_steps)
model, model_optimizer, dataloader = accelerator.prepare(model, model_optimizer, dataloader)
model_clone_optimizer = torch.optim.SGD(model_clone.parameters(), lr=0.02)

print(f"initial model weight is {accelerator.unwrap_model(model).weight.item():.5f}")
print(f"initial model weight is {model_clone.weight.item():.5f}")

for i, (inputs, labels) in enumerate(dataloader):
    with accelerator.accumulate(model):
        inputs = inputs.view(-1, 1)
        print(i, inputs.flatten())
        labels = labels.view(-1, 1)
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        accelerator.backward(loss)
        model_optimizer.step()
        model_optimizer.zero_grad()

loss = criterion(model_clone(x.view(-1, 1)), y.view(-1, 1))
model_clone_optimizer.zero_grad()
loss.backward()
model_clone_optimizer.step()

print(f"w/ accumulation, the final model weight is {accelerator.unwrap_model(model).weight.item():.5f}")
print(f"w/o accumulation, the final model weight is {model_clone.weight.item():.5f}")

Things work as expected without deepspeed.
$ poetry run accelerate launch --num_processes 1  grad_accu.py
[2023-09-13 23:01:01,987] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-09-13 23:01:10,720] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
initial model weight is 0.00000
initial model weight is 0.00000
0 tensor([1., 2.], device='cuda:0')
1 tensor([3., 4.], device='cuda:0')
2 tensor([5., 6.], device='cuda:0')
3 tensor([7., 8.], device='cuda:0')
w/ accumulation, the final model weight is 1.94344
w/o accumulation, the final model weight is 1.94344

Things do not work as expected with deepspeed.
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
  gradient_accumulation_steps: 1
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: false
  zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: 'no'
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

$ poetry run accelerate launch  --config_file deepspeed.yaml grad_accu.py
[2023-09-13 22:57:28,874] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-09-13 22:57:47,448] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-09-13 22:57:51,957] [INFO] [comm.py:637:init_distributed] cdb=None
[2023-09-13 22:57:51,957] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-09-13 22:57:52,024] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.10.2, git-hash=unknown, git-branch=unknown
[2023-09-13 22:57:55,684] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2023-09-13 22:57:55,684] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer
[2023-09-13 22:57:55,684] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2023-09-13 22:57:55,684] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = SGD
[2023-09-13 22:57:55,684] [INFO] [utils.py:54:is_zero_supported_optimizer] Checking ZeRO support for optimizer=SGD type=<class 'torch.optim.sgd.SGD'>
[2023-09-13 22:57:55,684] [WARNING] [engine.py:1149:_do_optimizer_sanity_check] **** You are using ZeRO with an untested optimizer, proceed with caution *****
[2023-09-13 22:57:55,684] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 1 optimizer
[2023-09-13 22:57:55,685] [INFO] [stage_1_and_2.py:146:__init__] Reduce bucket size 500,000,000
[2023-09-13 22:57:55,685] [INFO] [stage_1_and_2.py:147:__init__] Allgather bucket size 500,000,000
[2023-09-13 22:57:55,685] [INFO] [stage_1_and_2.py:148:__init__] CPU Offload: False
[2023-09-13 22:57:55,685] [INFO] [stage_1_and_2.py:149:__init__] Round robin gradient partitioning: False
Rank: 0 partition count [1] and sizes[(2, False)]
[2023-09-13 22:57:55,913] [INFO] [utils.py:803:see_memory_usage] Before initializing optimizer states
[2023-09-13 22:57:55,914] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-09-13 22:57:55,914] [INFO] [utils.py:811:see_memory_usage] CPU Virtual Memory:  used = 84.74 GB, percent = 7.6%
[2023-09-13 22:57:56,005] [INFO] [utils.py:803:see_memory_usage] After initializing optimizer states
[2023-09-13 22:57:56,006] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-09-13 22:57:56,006] [INFO] [utils.py:811:see_memory_usage] CPU Virtual Memory:  used = 84.74 GB, percent = 7.6%
[2023-09-13 22:57:56,006] [INFO] [stage_1_and_2.py:520:__init__] optimizer state initialized
[2023-09-13 22:57:56,088] [INFO] [utils.py:803:see_memory_usage] After initializing ZeRO optimizer
[2023-09-13 22:57:56,088] [INFO] [utils.py:804:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB
[2023-09-13 22:57:56,088] [INFO] [utils.py:811:see_memory_usage] CPU Virtual Memory:  used = 84.79 GB, percent = 7.6%
[2023-09-13 22:57:56,093] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = SGD
[2023-09-13 22:57:56,094] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2023-09-13 22:57:56,094] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2023-09-13 22:57:56,094] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.02], mom=[0]
[2023-09-13 22:57:56,094] [INFO] [config.py:963:print] DeepSpeedEngine configuration:
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   activation_checkpointing_config  {
    "partition_activations": false,
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "synchronize_checkpoint_boundary": false,
    "profile": false
}
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   amp_enabled .................. False
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   amp_params ................... False
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   autotuning_config ............ {
    "enabled": false,
    "start_step": null,
    "end_step": null,
    "metric_path": null,
    "arg_mappings": null,
    "metric": "throughput",
    "model_info": null,
    "results_dir": "autotuning_results",
    "exps_dir": "autotuning_exps",
    "overwrite": true,
    "fast": true,
    "start_profile_step": 3,
    "end_profile_step": 5,
    "tuner_type": "gridsearch",
    "tuner_early_stopping": 5,
    "tuner_num_trials": 50,
    "model_info_path": null,
    "mp_size": 1,
    "max_train_batch_size": null,
    "min_train_batch_size": 1,
    "max_train_micro_batch_size_per_gpu": 1.024000e+03,
    "min_train_micro_batch_size_per_gpu": 1,
    "num_tuning_micro_batch_sizes": 3
}
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   bfloat16_enabled ............. False
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   checkpoint_parallel_write_pipeline  False
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   checkpoint_tag_validation_enabled  True
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   checkpoint_tag_validation_fail  False
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7efff803ff40>
[2023-09-13 22:57:56,094] [INFO] [config.py:967:print]   communication_data_type ...... None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   curriculum_enabled_legacy .... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   curriculum_params_legacy ..... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   data_efficiency_enabled ...... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   dataloader_drop_last ......... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   disable_allgather ............ False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   dump_state ................... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   dynamic_loss_scale_args ...... None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_enabled ........... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_gas_boundary_resolution  1
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_layer_num ......... 0
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_max_iter .......... 100
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_stability ......... 1e-06
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_tol ............... 0.01
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   eigenvalue_verbose ........... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   elasticity_enabled ........... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   flops_profiler_config ........ {
    "enabled": false,
    "recompute_fwd_factor": 0.0,
    "profile_step": 1,
    "module_depth": -1,
    "top_modules": 1,
    "detailed": true,
    "output_file": null
}
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   fp16_auto_cast ............... None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   fp16_enabled ................. False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   fp16_master_weights_and_gradients  False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   global_rank .................. 0
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   grad_accum_dtype ............. None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   gradient_accumulation_steps .. 1
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   gradient_clipping ............ 0.0
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   gradient_predivide_factor .... 1.0
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   initial_dynamic_scale ........ 65536
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   load_universal_checkpoint .... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   loss_scale ................... 0
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   memory_breakdown ............. False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   mics_hierarchial_params_gather  False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   mics_shard_size .............. -1
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   nebula_config ................ {
    "enabled": false,
    "persistent_storage_path": null,
    "persistent_time_interval": 100,
    "num_of_version_in_retention": 2,
    "enable_nebula_load": true,
    "load_path": null
}
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   optimizer_legacy_fusion ...... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   optimizer_name ............... None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   optimizer_params ............. None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   pld_enabled .................. False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   pld_params ................... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   prescale_gradients ........... False
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   scheduler_name ............... None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   scheduler_params ............. None
[2023-09-13 22:57:56,095] [INFO] [config.py:967:print]   sparse_attention ............. None
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   sparse_gradients_enabled ..... False
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   steps_per_print .............. inf
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   train_batch_size ............. 2
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   train_micro_batch_size_per_gpu  2
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   use_node_local_storage ....... False
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   wall_clock_breakdown ......... False
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   world_size ................... 1
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   zero_allow_untested_optimizer  True
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   zero_config .................. stage=1 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=False, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   zero_enabled ................. True
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   zero_force_ds_cpu_optimizer .. True
[2023-09-13 22:57:56,096] [INFO] [config.py:967:print]   zero_optimization_stage ...... 1
[2023-09-13 22:57:56,096] [INFO] [config.py:953:print_user_config]   json = {
    "train_batch_size": 2,
    "train_micro_batch_size_per_gpu": 2,
    "gradient_accumulation_steps": 1,
    "zero_optimization": {
        "stage": 1,
        "offload_optimizer": {
            "device": "none",
            "nvme_path": null
        },
        "offload_param": {
            "device": "none",
            "nvme_path": null
        },
        "stage3_gather_16bit_weights_on_model_save": false
    },
    "steps_per_print": inf,
    "fp16": {
        "enabled": false
    },
    "bf16": {
        "enabled": false
    },
    "zero_allow_untested_optimizer": true
}
initial model weight is 0.00000
initial model weight is 0.00000
0 tensor([1., 2.], device='cuda:0')
1 tensor([3., 4.], device='cuda:0')
2 tensor([5., 6.], device='cuda:0')
3 tensor([7., 8.], device='cuda:0')
w/ accumulation, the final model weight is 1.64573
w/o accumulation, the final model weight is 1.94344

Expected behavior
The gradient accumulation result of deepspeed should be
w/o accumulation, the final model weight is 1.94344

@vwxyzjn try installing via main? I think this may have been fixed

@muellerzr thanks for the prompt reply! I just tried, and the results still appear incorrect:
pip install git+https://github.com/huggingface/accelerate.git
Collecting git+https://github.com/huggingface/accelerate.git
  Cloning https://github.com/huggingface/accelerate.git to /tmp/pip-req-build-v7rf5ncz
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/accelerate.git /tmp/pip-req-build-v7rf5ncz
  Resolved https://github.com/huggingface/accelerate.git to commit 40a73e0ae0dad0f5b9c0cdcc1b49165fcf08caf9
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done

initial model weight is 0.00000
initial model weight is 0.00000
0 tensor([1., 2.], device='cuda:0')
1 tensor([3., 4.], device='cuda:0')
2 tensor([5., 6.], device='cuda:0')
3 tensor([7., 8.], device='cuda:0')
w/ accumulation, the final model weight is 1.64573
w/o accumulation, the final model weight is 1.94344

cc @pacman100

Could the discrepancy be tied to the fact that the deepspeed plugin reads the number of gradient accumulation steps from the config and this is overriding the value passed to the accelerator?
What happens if you change this part of your config as follows:
deepspeed_config:
  gradient_accumulation_steps: 4

Hello @vwxyzjn and @lewtun,
the value passed to Accelerator object is only used if the value in deepspeed config for gradient_accumulation_steps is auto. This is only possible when using a DeepSpeed json config file with auto for gradient_accumulation_steps. In other cases, please specify it correctly when creating deepspeed config via accelerate config command as Lewis suggested.

See the tests at https://github.com/huggingface/accelerate/blob/main/tests/deepspeed/test_deepspeed.py#L610 for clarity on this

authors: lewtun muellerzr pacman100 vwxyzjn

repo: huggingface/accelerate


text:
Should get_linear_schedule_with_warmup consider gradient_accumulation_steps?

I think num_training_steps should be len(train_dataloader) * num_epochs // gradient_accumulation_steps

Sorry, I mean accelerate/examples/nlp_example.py line 128, num_training_steps should be len(train_dataloader) * num_epochs // gradient_accumulation_steps. Now it doesn't consider gradient_accumulation_steps.

Thanks for the precision! That's completely correct, would you like to make a PR to fix this?

authors: avsdfirst sgugger


text:
Duplicate elements in split_between_processes

System Info
accelerate==git+https://github.com/huggingface/accelerate.git

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction

Test example test_accelerate.py

from accelerate import PartialState  # Can also be Accelerator or AcceleratorState

state = PartialState()
input_list = list(range(17))

with state.split_between_processes(input_list) as splitted_input_list:
    print(f"{state.device}, {splitted_input_list}")

Run accelerate launch --num_processes 8 test_accelerate.py
Output

cuda:2, [6, 7, 8]
cuda:7, [16]
cuda:1, [3, 4, 5]
cuda:4, [12, 13, 14]
cuda:3, [9, 10, 11]
cuda:6, [16]
cuda:0, [0, 1, 2]
cuda:5, [15, 16]

As we can see, there are three 16 among three different processes. It is caused by https://github.com/huggingface/accelerate/blob/7ac153f4040ce04c3be0c26736577783d14e42b3/src/accelerate/state.py#L440-L442
We should modify above codes as following to get the expected output
num_samples_per_process = length // self.num_processes
num_extras = length % self.num_processes
start_index = self.process_index * num_samples_per_process + min(self.process_index, num_extras)
end_index = start_index + num_samples_per_process + (1 if self.process_index < num_extras else 0)

Expected behavior

Expected output:

cuda:2, [5, 6]
cuda:7, [15, 16]
cuda:1, [3, 4]
cuda:4, [9, 10]
cuda:3, [7, 8]
cuda:6, [13, 14]
cuda:0, [0, 1, 2]
cuda:5, [11, 12]

@muellerzr , Could you check it out?

Thanks for the flag! Would you like to make a PR with your fix? :)

Thanks for the flag! Would you like to make a PR with your fix? :)

Sure!

authors: hkunzhe muellerzr


text:
"Illegal Instruction" on Older CPUS

Hey there! I'm reaching out from a machine running the Intel i3-2120 processor.
The llm-rs library (https://github.com/rustformers/llm) can handle inference just fine on this processor (previously failed on older CPUs, works fine nowadays), but candle fails. I tried:
$ cargo run --example quantized-phi --release
    Finished release [optimized] target(s) in 3.59s
     Running `/home/jett/devel/candle/target/release/examples/quantized-phi`
avx: true, neon: false, simd128: false, f16c: false
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Running on CPU, to run on GPU, build this example with `--features cuda`
loaded 325 tensors (1.79GB) in 0.16s
model built
[1]    13404 illegal hardware instruction  cargo run --example quantized-phi --release

You can see the failure output. An ML/AI library throwing "Illegal instruction" usually means your program needs features that your CPU doesn't offer, so I tried recompiling without AVX2 instructions enabled (https://github.com/huggingface/candle/issues/622#issuecomment-1694726330), but it still failed:
$ RUSTFLAGS="-Ctarget-cpu=native -Ctarget-feature=-avx2" cargo run --example quantized-phi
    Finished dev [unoptimized + debuginfo] target(s) in 9.35s
     Running `/home/jett/devel/candle/target/debug/examples/quantized-phi`
avx: true, neon: false, simd128: false, f16c: false
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Running on CPU, to run on GPU, build this example with `--features cuda`
loaded 325 tensors (1.79GB) in 0.21s
model built
[1]    18202 illegal hardware instruction  RUSTFLAGS="-Ctarget-cpu=native -Ctarget-feature=-avx2" cargo run --example

Any ideas? I realize it might be a code-specific issue with the way candle is written, but I'm hoping it would have a simple fix (with something like a feature flag to check for CPU feature presence that enables certain code so this doesn't throw "Illegal instruction"). Thanks in advance for any reply, and thanks to all the maintainers for maintaining such an awesome ML/AI library!
TL;DR: Candle uses unsupported CPU features in its quantized LLM inference that is encountered when testing the prewritten examples. This causes execution to fail on older CPUs, such as the Intel i3-2120.

Could you try removing the target-cpu=native flag entirely and remove the .cargo/config.toml file if you're running within the main candle repo? Would be interesting to see if it still get some illegal instructions there.

Could you try removing the target-cpu=native flag entirely and remove the .cargo/config.toml file if you're running within the main candle repo? Would be interesting to see if it still get some illegal instructions there.

Sure! I can't believe I didn't think of removing .cargo/config.toml. It runs fine now:
$ cargo run --example quantized-phi --release
# [...]
Finished release [optimized] target(s) in 5m 36s
     Running `target/release/examples/quantized-phi`
avx: false, neon: false, simd128: false, f16c: false
temp: 0.80 repeat-penalty: 1.10 repeat-last-n: 64
Running on CPU, to run on GPU, build this example with `--features cuda`
loaded 325 tensors (1.79GB) in 0.20s
model built
Write a function to count prime numbers up to N.
# Here, N = 20
# Output: 8
'''

def prime_numbers(n):
    flag = [True for i in range(n+1)]
    primes = []
    for i in range(2, n + 1):
        if flag[i^C

I'm confused why this implies that avx won't work on my machine (avx: true becomes avx: false and is the only noticeable difference between the -Ctarget-cpu=native and not builds' outputs), though? It looks like my machine supports avx:
grep "avx" /proc/cpuinfo
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx lahf_lm epb pti tpr_shadow flexpriority ept vpid xsaveopt dtherm arat pln pts vnmi
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx lahf_lm epb pti tpr_shadow flexpriority ept vpid xsaveopt dtherm arat pln pts vnmi
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx lahf_lm epb pti tpr_shadow flexpriority ept vpid xsaveopt dtherm arat pln pts vnmi
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx lahf_lm epb pti tpr_shadow flexpriority ept vpid xsaveopt dtherm arat pln pts vnmi

Unless avx: true in candle's output means both avx and avx2 features are called, which is why it throws Illegal instruction, since my CPU does not support avx2?

repo: huggingface/candle


text:
Fix chapter 5 links

Fixing chapter 5 reference links that were previously broken.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

authors: HuggingFaceDocBuilderDev richardachen

repo: huggingface/course


text:
Support # in config names

https://huggingface.co/datasets/bigcode/commitpackft/viewer/c%23

Could not read the parquet files: https://huggingface.co/datasets/bigcode/commitpackft/resolve/refs%2Fconvert%2Fparquet/c#/train/0000.parquet
Error code:   FileSystemError

Reported here: https://huggingface.co/datasets/bigcode/commitpackft/discussions/4#65b5ca59e02a17f0f86da0b4

same error with https://huggingface.co/datasets/bigcode/the-stack-v2-dedup (C# config)

See https://github.com/huggingface/datasets-server/pull/2527

repo: huggingface/datasets-server


text:
Adding Europarl Bilingual dataset

Implementation of Europarl bilingual dataset from described here.
This dataset allows to use every language pair detailed in the original dataset. The loading script manages also the small errors contained in the original dataset (in very rare cases (1 over 10M) there are some keys that references to inexistent sentences).
I chose to follow the the style of a similar dataset available in this repository: multi_para_crawl.

is there a way to check errors without subscribing to CircleCI? Because they want access to private repositories when logging.

I think you need to be logged in to check the errors unfortunately. Feel free to create an account with bitbucket maybe if you don't want it to access your private github repos

I've resolved some requirements, but I cannot create dummy data. The dataset works as follows: for each language pair <lang1>-<lang2> 3 files are downloaded:

dataset for <lang1>
dataset for <lang2>
alignments between <lang1> and <lang2>

Suppose we work with the bg-cs language pair. Then, the dataset will download three gzip files which should be decompressed. I do not understand the relation between the folders created by the script to create dummy data and the original data provided by the download manager.

Hi ! Indeed the data files structure of this dataset looks very specific.
The command datasets-cli dummy_data ./datasets/europarl_bilingual shows some instructions for each split but let me add more details.
First things to know is that the dummy data files need to be uncompressed data, so for example for the file bg.zip you should actually have one folder with all the xml files in it instead. In the same way, bg-cs.xml.gz must be replaced by an actual uncompressed xml file.
Let's take the bg-cs config as an example. To make the dummy data you need to:

go to ./datasets/europarl_bilingual/dummy/bg-cs/8.0.0 and create a folder named dummy_data. Then go inside this folder
create a text file named bg-cs.xml.gz containing xml content (so without .gz compression). The xml content must have the same structure as the original bg-cs.zml but only include 1 linkGrp entry. You can pick one entry from the original bg-cs.xml file. Let's say this entry is about this file: ep-06-01-16-003.xml
create a folder named bg.zip and inside this folder add one file Europarl/raw/bg/ep-06-01-16-003.xml. You can pick the xml file from the original bg.zip archive.
create a folder named cs.zip and inside this folder add one file Europarl/raw/cs/ep-06-01-16-003.xml. You can pick the xml file from the original cs.zip archive.
zip the dummy_data into dummy_data.zip

At this point you have dummy data files to generate 1 example which is what we want to be able to test the dataset script europarl_bilingual.py with pytest.
In particular this will make this test pass:
pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_europarl_bilingual

Ideally it would be awesome to have dummy data for all the different configs so if we manage to make a script that generates all of it automatically that would be perfect. However since the structure is not trivial, another option would be to only have the dummy data for only 1 or 2 configs, like what we do for bible_para for example. In bible_para only a few configurations are tested. As you can see there is only 6 configs in the BUILDER_CONFIGS attribute. All the other configs can still be used, here is what is said inside the dataset card of bible_para:
To load a language pair which isn't part of the config, all you need to do is specify the language code as pairs.
You can find the valid pairs in Homepage section of Dataset Description: http://opus.nlpl.eu/bible-uedin.php
E.g.

`dataset = load_dataset("bible_para", lang1="fi", lang2="hi")`

In this case the configuration "fi-hi" is simply created on the fly, instead of being picked from the BUILDER_CONFIGS list.
I hope this helps, let me know if you have questions or if I can help

I already created the scripts to create reduced versions of the data. What I didn't understand was how to put files in the dummy_data folder because, as you noticed, some file decompress to a nested tree structure. I will not try again with your suggestions!

Is there something else I should do? If not can this be integrated?

Thanks a lot !!
Since the set of all the dummy data files is quite big I only kept a few of them. If we had kept them all the size of the datasets repo would have increased to much :/
So I did the same as for bible_para: only keep a few configurations in BUILDER_CONFIGS and have all the other pairs loadable with the lang1 and lang2 parameters like this:
dataset = load_dataset("europarl_bilingual", lang1="fi", lang2="fr")

authors: lhoestq lucadiliello

repo: huggingface/datasets


text:
Fix Inference API status check, allow modelLoadInfo to be passed as prop

To reduce the number of calls made to the inference API status check, we let modeLoadInfo to be passed in as a prop - if not provided as a prop, it makes the api call directly as it always has. This allows the Model Page in moon-landing, which needs to make the status check request, reuse the model status instead of making another call. See discussion here: https://github.com/huggingface/moon-landing/pull/6975
Also I believe the status API got out of sync at some point with the actual API returned by api-inference - fixed now. This is why currently every model says This model can be loaded on the Inference API on-demand even when that isn't the case.

Hi there! This part of the code now lives in huggingface.js, so I'll close this PR.

repo: huggingface/hub-docs


text:
Use proper logging in auth module

This is a small PR to use proper logging in the authentication module.
Changes made

replacing print statements by proper logging with the level that reflects the severity of the messages.
replacing some warnings.warn() by logger.warning() following logging best practices : Python Logging HOWTO
setting verbosity level to INFO for authentication commands.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks @Wauplin for the review! your comment is addressed in 87ab6d4. Let's merge :)

authors: HuggingFaceDocBuilderDev hanouticelina


text:
If IPUConfig contains inference_layers_per_ipu, use it in pipeline instead of layers_per_ipu

What does this PR do?
Allows us to define IPU pipeline splits that only work in inference and the Hugging Face pipeline will use these by default if they are present.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Not sure why the mode property isn't appearing in the docs, they should be supported https://github.com/huggingface/doc-builder/pull/87

do we need to establish what is more common to determine what the default should be?

Currently the default mode is training because:

The default mode for torch.nn.Module is also training
IPU pipelines made for training will always work for inference, but not the other way around
Therefore, the performance will either be the same or better than it is now:
If you are using the IPUTrainer or Pipeline
Depending on the advanced developers knowledge of the IPUConfig

what the most appropriate place would be
Yeah, unfortunately to_options() is too late for the mode to be set because it needs to be set to the correct value before parallelize() is called

maybe the logic in IPUConfig.to_options should also be adapted eventually as it currently does the for_inference checks explicitly.

Yeah I agree with this. We would need to change the if for_inference to if self.mode == "inference" and turn replication_factor and device_iterations into ManagedAttributes

Could you app nn.Module and also have IPUConfig.train() and IPUConfig.eval()?

Could you ape nn.Module and have ipu_config.train() and ipu_config.eval() instead of ipu_config.mode = "train"

Good suggestion, done

repo: huggingface/optimum-graphcore


text:
Tests for LogitsProcessor updated with transformers 4.38.2

What does this PR do?

Updates LogitsProcessorTest to match tests from transformer == 4.38.2.
Assert that outputs are in the expected device,  HPU.

Fixes # (issue)
Aside from new tests, the only issue was related to using different torch_devices. The solution here was to import the torch_device from test_modeling_common which uses the HPU.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[x] Did you write any new necessary tests?

@regisss is this PR still useful? or do you want me to update to 4.40+? https://github.com/huggingface/optimum-habana/issues/865#issuecomment-2079006079

@malkomes That would be great if you can update it to v4.40!

authors: malkomes regisss

repo: huggingface/optimum-habana


text:
fix variable name

The variable name of distillation should be distiller, otherwise the distiller parameter in optimizer would be a none parameter.
@echarlaix

Good catch @jiqing-feng, thanks !

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

repo: huggingface/optimum-intel


text:
Unable to resume training after saving checkpoint, while using Zero-1 Optimization

System Info
Platform:

- Platform: Linux-5.15.0-1056-aws-x86_64-with-glibc2.29
- Python version: 3.8.10

Python packages:

- `optimum-neuron` version: 0.0.24
- `neuron-sdk` version: 2.19.1
- `optimum` version: 1.20.0
- `transformers` version: 4.41.1
- `huggingface_hub` version: 0.24.5
- `torch` version: 2.1.2+cu121
- `aws-neuronx-runtime-discovery` version: 2.9
- `libneuronxla` version: 2.0.2335
- `neuronx-cc` version: 2.14.227.0+2d4f85be
- `neuronx-distributed` version: 0.8.0
- `neuronx-hwm` version: NA
- `torch-neuronx` version: 2.1.2.2.2.0
- `torch-xla` version: 2.1.3
- `transformers-neuronx` version: 0.11.351

Neuron Driver:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

aws-neuronx-collectives/now 2.20.22.0-c101c322e amd64 [installed,local]
aws-neuronx-dkms/now 2.16.7.0 amd64 [installed,local]
aws-neuronx-oci-hook/now 2.3.0.0 amd64 [installed,local]
aws-neuronx-runtime-lib/now 2.20.22.0-1b3ca6425 amd64 [installed,local]
aws-neuronx-tools/now 2.17.1.0 amd64 [installed,local]

Who can help?
@michaelbenayoun
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction (minimal, reproducible, runnable)
Pretraining a TinyLLama like model, using the same tokenizer as TinyLlama on the wikitext dataset.
Create a Trainium instance following the steps here
Get the official training script:
wget https://raw.githubusercontent.com/huggingface/optimum-neuron/main/examples/language-modeling/run_clm.py

Export BF16 var
export XLA_USE_BF16=true

Compile:
neuron_parallel_compile torchrun --nproc_per_node=2 run_clm.py --config_name dhruv-huq/tiny-tinyllama --tokenizer_name TinyLlama/TinyLlama-1.1B-Chat-v1.0 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --per_device_train_batch_size 1 --block_size 1024 --bf16 --zero_1 --overwrite_output_dir --output_dir my_training/ --preprocessing_num_workers 64

Train:
torchrun --nproc_per_node=2 run_clm.py --config_name dhruv-huq/tiny-tinyllama --tokenizer_name TinyLlama/TinyLlama-1.1B-Chat-v1.0 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --per_device_train_batch_size 1 --block_size 1024 --bf16 --zero_1 --overwrite_output_dir --output_dir my_training/ --preprocessing_num_workers 64

Error with stack trace:
[INFO|trainers.py:832] 2024-09-09 15:03:21,372 >> ***** Running training *****
[INFO|trainers.py:833] 2024-09-09 15:03:21,372 >>   Num examples = 1,380
[INFO|trainers.py:834] 2024-09-09 15:03:21,372 >>   Num Epochs = 3
[INFO|trainers.py:835] 2024-09-09 15:03:21,372 >>   Instantaneous batch size per device = 1
[INFO|trainers.py:840] 2024-09-09 15:03:21,373 >>   Total train batch size (w. parallel, distributed & accumulation) = 2
[INFO|trainers.py:843] 2024-09-09 15:03:21,373 >>   Gradient Accumulation steps = 1
[INFO|trainers.py:844] 2024-09-09 15:03:21,373 >>   Total optimization steps = 4,140
[INFO|trainers.py:845] 2024-09-09 15:03:21,373 >>   Number of trainable parameters = 336,611,328
 12%|████████████████████████████                                                                                | 500/4140 [00:57<06:32,  9.27it/s][INFO|trainers.py:504] 2024-09-09 15:04:19,002 >> Saving model checkpoint to my_training/checkpoint-500
[INFO|configuration_utils.py:472] 2024-09-09 15:04:20,099 >> Configuration saved in my_training/checkpoint-500/config.json
[INFO|configuration_utils.py:731] 2024-09-09 15:04:20,099 >> Configuration saved in my_training/checkpoint-500/generation_config.json
[INFO|modeling_utils.py:2618] 2024-09-09 15:04:25,503 >> Model weights saved in my_training/checkpoint-500/model.safetensors
[INFO|tokenization_utils_base.py:2513] 2024-09-09 15:04:25,739 >> tokenizer config file saved in my_training/checkpoint-500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2522] 2024-09-09 15:04:25,739 >> Special tokens file saved in my_training/checkpoint-500/special_tokens_map.json
Traceback (most recent call last):
  File "run_clm.py", line 673, in <module>
    main()
  File "run_clm.py", line 621, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/trainers.py", line 1414, in train
    result = super().train(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/transformers/trainer.py", line 1885, in train
    return inner_training_loop(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/utils/require_utils.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/trainers.py", line 1054, in _inner_training_loop
    self.optimizer.step()
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/utils/require_utils.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/optimum/neuron/accelerate/optimizer.py", line 104, in step
    self.optimizer.step(closure)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/optim/optimizer.py", line 373, in wrapper
    out = func(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch_xla/distributed/zero_redundancy_optimizer.py", line 390, in step
    self._clip_grad_norm(max_norm=self.max_norm)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/neuronx_distributed/optimizer/zero_redundancy_optimizer.py", line 74, in _clip_grad_norm
    self._grad_norm = clip_grad_norm(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/neuronx_distributed/parallel_layers/grads.py", line 172, in clip_grad_norm
    total_norm = get_grad_norm(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/neuronx_distributed/parallel_layers/grads.py", line 90, in get_grad_norm
    dtype = parameters[0].dtype
IndexError: list index out of range
[2024-09-09 15:04:29,914] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1768 closing signal SIGTERM
[2024-09-09 15:04:31,079] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 1769) of binary: /opt/aws_neuron_venv_pytorch/bin/python3.8
Traceback (most recent call last):
  File "/opt/aws_neuron_venv_pytorch/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
run_clm.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-09-09_15:04:29
  host      : ip-172-31-54-170.us-west-2.compute.internal
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1769)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

There is no error if zero1 isn't used, training runs as expected
Expected behavior
Model resumes training correctly after saving the checkpoint

+1 similar error on optimum-neuron version: 0.0.25 and Neuron SDK 2.20
Traceback (most recent call last):
  File "/home/ubuntu/ml-specialized-hardware/purpose-built-accelerators/notebooks/src/train.py", line 101, in <module>
    trainer.train()
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/optimum/neuron/trainers.py", line 1456, in train
    result = super().train(
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/transformers/trainer.py", line 1938, in train
    return inner_training_loop(
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/optimum/neuron/utils/require_utils.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/optimum/neuron/trainers.py", line 1096, in _inner_training_loop
    self.optimizer.step()
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/optimum/neuron/utils/require_utils.py", line 51, in wrapper
    return func(*args, **kwargs)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/optimum/neuron/accelerate/optimizer.py", line 104, in step
    self.optimizer.step(closure)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/optim/optimizer.py", line 373, in wrapper
    out = func(*args, **kwargs)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch_xla/distributed/zero_redundancy_optimizer.py", line 336, in step
    self._clip_grad_norm(max_norm=self.max_norm)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/neuronx_distributed/optimizer/zero_redundancy_optimizer.py", line 98, in _clip_grad_norm
    all_parameters, self._grad_norm = self._get_params_and_grad_norm(norm_type)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/neuronx_distributed/optimizer/zero_redundancy_optimizer.py", line 83, in _get_params_and_grad_norm
    grad_norm = get_grad_norm(
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/neuronx_distributed/parallel_layers/grads.py", line 116, in get_grad_norm
    device = parameters[0].device
IndexError: list index out of range
 98%|█████████▊| 1000/1022 [02:22<00:03,  7.01it/s]
[2024-10-15 14:16:00,812] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 7613) of binary: /opt/aws_neuronx_venv_pytorch_2_1/bin/python3
Traceback (most recent call last):
  File "/opt/aws_neuronx_venv_pytorch_2_1/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/opt/aws_neuronx_venv_pytorch_2_1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

It looks like this is fixed by the latest torch-neuronx release: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/torch/torch-neuronx/index.html#release-2-1-2-2-3-2

authors: industrialeaf unography wolanlu

repo: huggingface/optimum-neuron


text:
Use peft for RLHF

Feature request
We should leverage trl: https://github.com/lvwerra/trl - the recent library from Hugging Face for RLHF, to apply PPO using peft and LoRA
I think peft should just work out of the box, the first step could be trying to adapt gpt2-sentiment.py script: https://github.com/lvwerra/trl/blob/main/examples/sentiment/scripts/gpt2-sentiment.py to use peft

@pacman100 @younesbelkada  I could work on this if there is any interest. What would be the first steps?

Thanks for your interest @BirgerMoell !! That would be definitely helpful
We need to check first the compatbility of peft and trl (i.e. it might not directly work out of the box but we will need to double check), so we will most likely adapt the gpt2 script ourselves just to make sure we fix things that needs to be fixed, then you can help us maybe converting the t5 script? Let me know what do you think!
If you feel you want to take the challenge and start converting the gpt2 script feel free to do it and we'll be happy to review your Pull Request / code changes ! 🔥

I think it sounds like a good idea if you start out with the gpt2 script and then I can do the t5 script. I think that would be a nice way to do it and I can understand much more by following along with your work.
Btw I'm on the huggingface discord if you want to messag me my username is birger

The PR https://github.com/lvwerra/trl/pull/163 should add peft support on trl, we may need to convert the other scripts such as t5 summarization etc as well once this PR gets merged!

authors: BirgerMoell younesbelkada

repo: huggingface/peft


text:
DOC: Update & improve docstrings and type annotations for common methods and classes

The docstrings of the most user-exposed methods and classes have been updated, or added if not already present. Furthermore, type annotations have been updated or added for those methods and classes.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

authors: BenjaminBossan HuggingFaceDocBuilderDev


text:
FIX Use different doc builder docker image

Same as in:
https://github.com/huggingface/transformers/blob/c712d05aa8fc8ba3ebe465079bd377d2dc9c2e07/.github/workflows/build_documentation.yml#L19

Doc built is currently failing, as e.g. here: https://github.com/huggingface/peft/actions/runs/8893174861/job/24418810402

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


text:
Not able to start the server using make server-dev

I replaced the server Makefile to serve gpt2 without sharding like
SAFETENSORS_FAST_GPU=1 python -m torch.distributed.run --nproc_per_node=2 text_generation_server/cli.py serve gpt2
Yesterday, the server and router worked fine, but the next time when I restarted the server, it doesn't work.
The issue I'm facing

You don't need to launch using torch.distributed.run when you are not sharding your model.
You can directly use text-generation-server serve gpt2.

authors: Codehackerone OlivierDehaene

repo: huggingface/text-generation-inference


text:
Falcon 7b inference not working

System Info
I get this when I run falcon 7b instruct. Any reason why?
I then tried the gpt2 model and it worked fine.

Information

[ ] Docker
[ ] The CLI directly

Tasks

[ ] An officially supported command
[ ] My own modifications

Reproduction
I follow the same instructions and code snippet on github and huggingface
Expected behavior
It should print this I think
{'generated_text': '\n\nDeep Learning is a subset of Machine Learning that is concerned with the development of algorithms that can'}

I think the bug is because you'r enot using our flash attention implementation and there's a bug on the --trust-remote-code method.

@Narsil Im running it with --trust-remote-code and how to I fix the flash attention error
docker run --rm --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:1.0.2 --model-id $model --trust-remote-code

These are the warning messages I get
2023-08-25T13:41:21.775108Z  WARN text_generation_launcher: `trust_remote_code` is set. Trusting that model `tiiuae/falcon-7b-instruct` do not contain malicious code.
2023-08-25T13:44:16.631496Z  WARN text_generation_launcher: No safetensors weights found for model tiiuae/falcon-7b-instruct at revision None. Converting PyTorch weights to safetensors.
2023-08-25T13:45:38.046335Z  WARN text_generation_launcher: Could not import Flash Attention enabled models: GPU with CUDA capability 7 0 is not supported
2023-08-25T13:46:19.502087Z  WARN text_generation_router: router/src/main.rs:345: `--revision` is not set
2023-08-25T13:46:19.502117Z  WARN text_generation_router: router/src/main.rs:346: We strongly advise to set it to a known supported commit.
2023-08-25T13:46:21.531860Z  WARN text_generation_router: router/src/main.rs:221: Model does not support automatic max batch total tokens
2023-08-25T13:46:21.531893Z  WARN text_generation_router: router/src/main.rs:249: Invalid hostname, defaulting to 0.0.0.0

Could not import Flash Attention enabled models: GPU with CUDA capability 7 0 is not supportexd

You need a more recent card for flash to work :(

authors: Narsil david-hung


text:
(node:20964) DeprecationWarning: Using Buffer without new will soon stop working. Use new Buffer(), or preferably Buffer.from(), Buffer.allocUnsafe() or Buffer.alloc() instead.

These deprecation warnings have recently started appearing.

Closed by #7

authors: phated tiefling

repo: hughsk/vinyl-buffer


text:
Website Agent Parse error

I'm trying to get (my first) Website Agent to handle data from an event. Sample event data looks like so:
{
  "data": {
    "expand": "schema,names",
    "startAt": 0,
    "maxResults": 2,
    "total": 12,
    "issues": [
      {
        "expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
        "id": "123456",
        "self": "https://jira.corp.company.com/rest/api/2/issue/123456",
        "key": "AOS-1448",
        "fields": {
          "summary": "kill A with fire"
        }
      },
      {
        "expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
        "id": "1234567",
        "self": "https://jira.corp.company.com/rest/api/2/issue/1234567",
        "key": "AOS-1447",
        "fields": {
          "summary": "Fix problemB"
        }
      }
    ]
  }
}

The code for the Website Agent: (I tried to make it as simple as possible for debugging. Once the full JSON is returned I'll worry about correct extract path settings):
{
  "expected_update_period_in_days": "8",
  "type": "json",
  "data_from_event": "{{data}}"
}

Returned error:
[00:00:00] INFO -- : Dry Run started
[00:00:00] ERROR -- : Error when handling event data: 784: unexpected token at '{"expand"=>"schema,names", "startAt"=>0, "maxResults"=>2, "total"=>12, "issues"=>[{"expand"=>"operations,versionedRepresentations,editmeta,changelog,renderedFields", "id"=>"123456", "self"=>"https://jira.corp.company.com/rest/api/2/issue/123456", "key"=>"AOS-1448", "fields"=>{"summary"=>"kill A with fire"}}, {"expand"=>"operations,versionedRepresentations,editmeta,changelog,renderedFields", "id"=>"1234567", "self"=>"https://jira.corp.company.com/rest/api/2/issue/1234567", "key"=>"AOS-1447", "fields"=>{"summary"=>"Fix problemB"}}]}'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/json-1.8.6/lib/json/common.rb:155:in `parse'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/json-1.8.6/lib/json/common.rb:155:in `parse'
/home/huginn/huginn/app/models/agents/website_agent.rb:640:in `parse'
/home/huginn/huginn/app/models/agents/website_agent.rb:415:in `handle_data'
/home/huginn/huginn/app/models/agents/website_agent.rb:492:in `block in handle_event_data'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/liquid-4.0.0/lib/liquid/context.rb:123:in `stack'
/home/huginn/huginn/app/models/agents/website_agent.rb:490:in `handle_event_data'
/home/huginn/huginn/app/models/agents/website_agent.rb:464:in `block (2 levels) in receive'
/home/huginn/huginn/app/concerns/liquid_interpolatable.rb:62:in `interpolate_with'
/home/huginn/huginn/app/models/agents/website_agent.rb:458:in `block in receive'
/home/huginn/huginn/app/models/agents/website_agent.rb:457:in `each'
/home/huginn/huginn/app/models/agents/website_agent.rb:457:in `receive'
/home/huginn/huginn/app/concerns/sortable_events.rb:99:in `receive'
/home/huginn/huginn/app/concerns/dry_runnable.rb:29:in `dry_run!'
/home/huginn/huginn/app/controllers/agents/dry_runs_controller.rb:41:in `create'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/abstract_controller/base.rb:186:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/callbacks.rb:131:in `run_callbacks'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal/rescue.rb:20:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/notifications.rb:166:in `block in instrument'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/notifications.rb:166:in `instrument'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/abstract_controller/base.rb:124:in `process'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionview-5.1.1/lib/action_view/rendering.rb:30:in `process'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal.rb:189:in `dispatch'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_controller/metal.rb:253:in `dispatch'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/routing/route_set.rb:31:in `serve'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/journey/router.rb:46:in `block in serve'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/journey/router.rb:33:in `each'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/journey/router.rb:33:in `serve'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/routing/route_set.rb:832:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/callbacks.rb:97:in `run_callbacks'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/railties-5.1.1/lib/rails/rack/logger.rb:36:in `call_app'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/railties-5.1.1/lib/rails/rack/logger.rb:24:in `block in call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/tagged_logging.rb:69:in `block in tagged'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/tagged_logging.rb:26:in `tagged'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.1/lib/active_support/tagged_logging.rb:69:in `tagged'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/railties-5.1.1/lib/rails/rack/logger.rb:24:in `call'
/home/huginn/huginn/config/initializers/silence_worker_status_logger.rb:3:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/request_id.rb:25:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.1/lib/action_dispatch/middleware/ssl.rb:66:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/railties-5.1.1/lib/rails/engine.rb:522:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/railties-5.1.1/lib/rails/railtie.rb:185:in `public_send'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/railties-5.1.1/lib/rails/railtie.rb:185:in `method_missing'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/common_logger.rb:33:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:132:in `start'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `load'
/home/huginn/huginn/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `<top (required)>'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74:in `load'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:74:in `kernel_load'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli/exec.rb:27:in `run'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:362:in `exec'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:22:in `dispatch'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/cli.rb:13:in `start'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/exe/bundle:30:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/usr/local/lib/ruby/gems/2.4.0/gems/bundler-1.15.4/exe/bundle:22:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
[00:00:00] INFO -- : Dry Run finished

I tried it with and without "extract" settings. I tried all possible "mode" settings. If the type is "text" it extracts the entire content when using the "match all" regex, so the agent itself is functioning. I validated the supplied event data is valid JSON.  For that matter, I get the same 'unexpected token' error just dry running the agent on the sample event:
{
  "data": "extractThis"
}

Anyone have any ideas what I'm doing wrong? I'm new to Huginn, but I've vainly gone through every possible Google search result that seemed even vaguely likely to be related to this error.

nm, figured it out. In case anyone else ran into something similar and was puzzled, I had to change my data_from_event line from
 "data_from_event": "{{data}}"

to
  "data_from_event": "{{data | replace: '=>', ':'}}",

after which I could perform extractions on it.

Nice! Sorry for the late response {{ data | as_object }} should work as well.

Also for another solution @dsander also recommended {{ data | json }} in another post and that helped my situation more.

authors: GChester dsander lindabutlerDS

repo: huginn/huginn


text:
Update README.md

Add functional component example for Search component

thank you

My pleasure, thank you for sharing this project!

authors: deemaagog dlford

repo: humanseelabs/gatsby-plugin-lunr


text:
next: Dialog.Content children snippet show open props in docs, but does not exist.

Change Type
Correction
Proposed Changes
Creating this under docs, as the code looks like it intentionally omits the open prop.
Documention mentions the children snippet provides a { open:boolean } prop but it does not exist in the types or at runtime. the open prop is only available on the child snippet.

Oh yeah, that needs to be removed. open is only available on the child snippet :)

repo: huntabyte/bits-ui


text:
Speed up dev servers; lucide-svelte issue

Current Behavior
Given https://github.com/lucide-icons/lucide/pull/1707, which you also briefly touched over in the stream, I wanted to volunteer to import all lucide-svelte icons in the new/faster format.
Expected Behavior
import Circle from "lucide-svelte/icons/circle";

Steps To Reproduce
N/A
Link to Reproduction / Stackblitz (reproduction template)
No response
More Information
No response

Go for it!

Thanks for taking this on! 🙂
I just want to note down here (before I forget) that we'll have to validate that changing the import paths to use deep imports doesn't break our registry builder script. There's a good chance that it will, so we'll have to modify it to accommodate. I can handle that part right before we merge your PR.

Really looking fwd to this! The slow dev server reloads are killing my DevX.

If the docs could be updated, that would be good for new users.
For example I've just been trying it out today. I created a "hello world" app with some basic pages, tried adding a dark mode toggle based on https://www.shadcn-svelte.com/docs/dark-mode and noticed the build took much longer. Sure enough it was looping over every icon as part of the build, despite only needing the sun and moon icon.
Making this change:
//import { Sun, Moon } from 'lucide-svelte';
import SunIcon from 'lucide-svelte/icons/sun';
import MoonIcon from 'lucide-svelte/icons/moon';

... and updating the page to match the names resulted in going from ...
vite v5.1.3 building for production...
✓ 2023 modules transformed.

... to ...
vite v5.1.3 building for production...
✓ 595 modules transformed.

Way faster.

We're tracking @gregmsanderson. There's an issue with a dependency that is holding us back at the moment but should be resolved soon.

authors: AdrianGonz97 coryvirok gregmsanderson huntabyte shyakadavis

repo: huntabyte/shadcn-svelte


text:
检测后识别

用这个检测成功后，一般用哪个算法做识别比较好？

你的检测结果是好的吗，为什么我的检测结果很差，可以邮件交流吗？1079158605@qq.com

你的检测结果是好的吗，为什么我的检测结果很差，可以邮件交流吗？1079158605@qq.com

我的检测很好，看你的场景吧，yahuuu@163.com

CRNN

authors: GUWjj terrynx wjy3326 yahuuu

repo: huoyijie/AdvancedEAST


text:
🛑 Split a bill is down

In 0ae77e0, Split a bill (https://bill.craftstudios.shop/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Split a bill is back up in 65154fe after 20 minutes.

repo: hupratt/upptime


text:
Міграція bb_flags

Стосується #7/#46 - в старій версії таблиця bb_flags є, в новій немає

Відповідно до цього коментаря можна дропати.

repo: hurtom/toloka


text:
added docker file and requ.txt

hi

Thank you snedan🙌


text:
How do you look up accounts with email addresses?

I know you can do it using a iPhone or Android, but is there a way to replicate this process so you don't need a phone. Thank you.

I do you do that with your iphone / adnroid device?

authors: huttarichard joshuafechter

repo: huttarichard/instagram-private-api


text:
Update data.sql

US #1: add horse names in database

/tina - can you have a look at this?

/tina - done

repo: hvegav/pet-clinic


text:
update imgs in blog/livestream

What this PR does / why we need it:
Special notes for your reviewer:
Does this PR introduce a user-facing change?

LGTM

lgtm, thanks.

authors: Michelle951 SSmallMonster windsonsea

repo: hwameistor/hwameistor


text:
fix false alarm in sql string check

Signed-off-by: Kuan Feng fengkuan@gmail.com

/lgtm

authors: kuanf rbmateescu

repo: hybridapp-io/ham-deployable-operator


text:
presenceUpdate event - old and new presences are always identical

Both old and new presences always appear to be identical.
bot.on('presenceUpdate', (oldMember, newMember) => {
  console.log('old:');
  console.log(oldMember.user.presence);
  console.log('new:');
  console.log(newMember.user.presence);
});

gives this:

You need oldMember.presence, not oldMember.user.presence


text:
Adds new benchmarks

The MobileNet benchmark runs a 3-stage pipeline and uses the MobileNet model
The centralized average benchmark runs a scatter-gather algorithm to compare against Lambda over Redis & Dynamo

Codecov Report

Merging #11 into master will increase coverage by 0.01%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master      #11      +/-   ##
==========================================
+ Coverage    89.9%   89.91%   +0.01%
==========================================
  Files          13       13
  Lines         872      873       +1
==========================================
+ Hits          784      785       +1
  Misses         88       88

Impacted Files
Coverage Δ

droplet/server/executor/user_library.py
87.5% <0%> (ø)
:arrow_up:

droplet/server/executor/call.py
87.61% <100%> (+0.05%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 558281a...1e0a85e. Read the comment docs.

authors: codecov-io vsreekanti

repo: hydro-project/droplet


text:
REST API Resource Creation Question

Hello,
I'm trying to implement project export functionality from Model My Watershed into HydroShare, and plan to use the REST API to create and modify resources. As part of this exercise I read the documentation both in Swagger and the source, which lead me to believe that the only required parameter for creating a resource is resource_type, all else being optional. However, when I try to POST to the /resource/ endpoint with that, I get the following message:
> http --print HhBb https://dev.hydroshare.org/hsapi/resource/ Authorization:"Bearer $MMW_HYDROSHARE_ACCESS_TOKEN" resource_type=GenericResource

POST /hsapi/resource/ HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Authorization: Bearer $MMW_HYDROSHARE_ACCESS_TOKEN
Connection: keep-alive
Content-Length: 36
Content-Type: application/json
Host: dev.hydroshare.org
User-Agent: HTTPie/0.9.9

{
    "resource_type": "GenericResource"
}

HTTP/1.1 400 BAD REQUEST
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Language: en
Content-Type: application/json
Date: Mon, 18 Dec 2017 19:22:34 GMT
Server: nginx/1.11.10
Transfer-Encoding: chunked
Vary: Accept, Accept-Language, Cookie

{
    "resource": "Resource creation failed. 'NoneType' object is not iterable"
}

I believe this error is coming from this line, although I am unsure what is causing it. I tried adding title and subject to the request, but the endpoint continued to return the same message.
I was also unable to find an example payload for this endpoint to copy from. I would appreciate any guidance to amend this request.

Hi @rajadain Can you please try the same on beta.hydroshare.org? We discussed this this morning and dev.hydroshare.org is out of date currently.

Trying it now.

Yes I get the same result:
> http --print HhBb https://beta.hydroshare.org/hsapi/resource/ Authorization:"Bearer $MMW_HYDROSHARE_ACCESS_TOKEN" resource_type=GenericResource

POST /hsapi/resource/ HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Authorization: Bearer $MMW_HYDROSHARE_ACCESS_TOKEN
Connection: keep-alive
Content-Length: 36
Content-Type: application/json
Host: beta.hydroshare.org
User-Agent: HTTPie/0.9.9

{
    "resource_type": "GenericResource"
}

HTTP/1.1 400 BAD REQUEST
Allow: GET, POST, HEAD, OPTIONS
Connection: keep-alive
Content-Language: en
Content-Type: application/json
Date: Mon, 18 Dec 2017 19:46:55 GMT
Server: nginx/1.11.13
Transfer-Encoding: chunked
Vary: Accept, Accept-Language, Cookie

{
    "resource": "Resource creation failed. 'NoneType' object is not iterable"
}

Do beta and www share the same backend? Or does it simply have a recent copy of www? dev seemed to be a different environment, which made it a good candidate for connecting to the staging site and experimenting in a sandbox.

@rajadain Ok, this is a bug then. beta and www are clones of each other and are generally in sync as far as versioning, etc

If I create a test resource in beta will it also show up in www?

Negative

@pkdash or @Castronova may be able to provide guidance here.  I know that the Jupyter App platform is able to successfully create resources in HydroShare using the API.  I am not sure how that is different from from what is being tried here.

I'll try using the hs_restclient library which may work better.

@rajadain Here is some code that should help you get started:
from hs_restclient import HydroShare, HydroShareAuthBasic
auth = HydroShareAuthBasic(username='myusername', password='mypassword')
hs = HydroShare(auth=auth)

hs.createResource("genericresource",
                  "my title",
                  resource_file="my file",
                  abstract="my abstract",
                  keywords=["keyword 1", "keyword 2"])

Thank you, the above is working for me.
@aphelionz, this is no longer blocking my work, so maybe doesn't need the High Priority tag anymore. Perhaps this is more a documentation issue at this point.
Thank you all for your support.

Thank you @rajadain

closed, implemented or stale

closed, implemented or stale

authors: Castronova aphelionz dtarb martinseul rajadain

repo: hydroshare/hydroshare


text:
how to suppress output lines works

trying to supress output lines from my EXPECT script. Have echo(false) statement as an first line of script (as in examples) but it still printing output lines. What is the logic of echo()statement? if it once set up to false, does it supress output until its set to true? have it false before any send command, but it does not help.
Also tried @expect.exe "%~f0" >NUL or @expect.exe "%~f0 | break" but does not work for some reason

How about @expect.exe "%~f0" >NUL 2>NUL ?

How about @expect.exe "%~f0" >NUL 2>NUL ?

This syntax is invalid, valid redirection both STDOUT and STDERR to null is >NUL 2>&1
But my problem is, the script does not work, when discard STDOUT. It produce serie of errors:
expect.lua for Windows v0.6.2
The handle is invalid.
The handle is invalid.
The handle is invalid.
The handle is invalid.
The handle is invalid.
The handle is invalid.
The handle is invalid.
The handle is invalid.

and hang until CTRL+C wasnt pressed. But inside of LUA script execution, the echo(true/false)should control output lines echoing, or am I miss something?

The echo function does not control all of output.
When echo(true) is called, the spawn , send , and sendln functions print its arguments to STDOUT.
echo(false) disables them.
It does not control the output of child processes and displaying the software version.

This syntax is invalid, valid redirection both STDOUT and STDERR to null is >NUL 2>&1

It has no problems on the syntax of CMD.EXE because NUL is not a plain file to need to join the two outputs. To drop outputs, it works as we expected.

The handle is invalid.

Sorry, it is the error when expect function is called. It reads the terminal information linked to STDOUT.
Therefore, when STDOUT is not a terminal, its fails.

I will try to create single batch file as an EXPECT scripting function, call it from main batch by
call expect.cmd "%ip_addr%" >NUL
this could be a way to make EXPECT silent

Differently from /usr/bin/expect of Linux, Expect.lua does not reads STDOUT and STDERR of child processes via pipelines. It reads the console buffer of the terminal directly.
Therefore, if child processes outputs nothing to console by redirecting , the expect function never works as we hope because they can read nothing from console.
It is difficult for Expect.lua to satisfy your requests.

authors: hymkor lazna

repo: hymkor/expect


text:
multi-tenancy support for lumen-framework

Hi,
Would it be possible / do you plan to add support for hyn/multi-tenant for lumen framework?
I am interested to create an application having multi-tenancy features without using the entire laravel framework (currently each hyn/multi-tenant package is bounded it's corresponding laravel version).
Are any workarounds available to use multi-tenancy in a lumen setup ?
Thank you,
Lucian

Hello Lucian,
This would most likely need some investigation. I'm not saying no, I'm just not sure it's feasible..
Thanks,
Daniël.

I was considering requiring a deeper level of requirement here, eg using laravel/framework. But Lumen doesn't use the framework at all. The only valid way of doing this then would be either to drop the requirement completely or put the constraint on any of the packages used by both Laravel and Lumen 🤔

Not pursuing.

I have a small fork where I added support for lumen but I haven`t tested it fully (main features works fine) https://github.com/Marko298/multi-tenant

authors: Marko298 luceos lucian-dragomir

repo: hyn/multi-tenant


text:
Recommended GPU for Inference?

Hi, thanks for the great work and all the detailed notes. I've successfully run the code. Now I want to perform some inference on my own dataset, yet it seems the code is only using the CPU, even though I do assign a GPU for the task. Is it because the GPU is not large enough and thus the code automatically falls back into the CPU version? Is there a way to check if it is running on the CPU or the GPU?
Thx in advance!

Ok I just realized it's because the cuda is not available for some reason... I just did the "conda env create" as in your readme.md. Any suggestion?

Yes yes, it was not available in the beginning for some reason. I fixed it by reinstalling the environment. Thx!