text:
Cast Integration Cleanup

Description:
Clean up the cast integration.
Most notably the pychromecast.Chromecast objects are now managed by the entities, not globally. This makes handling "moved" hosts, unavailable chromecasts and disconnecting much cleaner.

[x] setup_platform and internal discovery now only create "ChromecastInfo" objects (with attrs 👌), not Chromecast objects. Missing information is automatically added via HTTP dial.
[x] The CastDevice entities now take care of moved hosts themselves (using dispatcher callbacks).
[x] Added available property.
[x] Enabled polling mode if an app is active, because we sometimes don't receive media status callbacks from certain apps.
[x] Added a ton of comments to hopefully make understanding the source code a bit easier.

Todo:

[x] Some more local testing.
[x] Fixing tests.
[x] Add tests

Out of scope:

[ ] Move some of the logic into pychromecast (would be good to test the logic with this user base first; but pychromecast could benefit from some logic, like availability, info objects, or half-polling mode; the last one is a bit controversial though)

fixes #13359, fixes #13319, fixes #13090
Example entry for configuration.yaml (if applicable):
media_player:
  - platform: cast

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

Tested extensively with chromecast, google home, cast audio and audio groups now. On to the tests!

Once ready let me know and I can test if you'd like.

FYI, regarding core guarding from calling services for unavailable entities: Components that uses async_extract_from_service handles this. If a platform registers its own services it has to handle that on its own, in the service handler.

Awesome work 🐬

@OttoWinter Is it already merged?
I just got a

Platform cast not ready yet. Retrying in 30 seconds.

See #13359

Yes it is in the beta. And that error message is expected now, it signals that one of Chromecast devices defined in your configuration.yaml can't be reached and therefore can't be setup.

You mean the automatically created entity_registry.yaml?
Yesterday the same setup worked as expected but today not. All devices for the group are connected to power and the network (while not all devices of the list in the above mentioned file are connected to power).
Could the TTS and delay command be a problem?
    action:
      - service: media_player.volume_set
        data_template:
          entity_id: media_player.schlafzimmer
          volume_level: 0.1
      - service: tts.google_say
        data_template:
          entity_id: media_player.schlafzimmer
          message: Guten Morgen. Es ist {{now().hour}} Uhr {{now().minute}}
          cache: false
      - delay: 00:00:15
      - service: media_player.play_media
        data_template:
          entity_id: media_player.wecker
          media_content_id: 'http://fluxfm.hoerradar.de/fluxfmberlin-live-aac-mq?sABC=59s45n23%230%23no8617362n29q2o435p17n54928n16s5%23gharva&amsparams=playerid:tunein;skey:1509186083'
          media_content_type: 'audio/mp3'

What are you using to discover your cast devices? discovery: in your configuration.yaml, or media_player: platform: cast or manually by host:?
In what way are they not working? Is it just the error message or do they not turn on/execute the command?
If you have some Chromecast devices in your configuration.yaml that are not connected to power that error message is expected; the other devices should work correctly though. (I hope, at least they did during my testing)
The TTS and delay shouldn't really affect anything here. The entity_regitry.yaml neither.
I don't think this discussion should be done in this PR, as it's not a comment on the code. Please open a separate issue and I will help you out there.

authors: MaluNoPeleke MartinHjelmare OttoWinter balloob edif30


text:
WIP: Add config entry and device registry entry for SimpliSafe

Description:
Related issue (if applicable): fixes #
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):

Checklist:

[ ] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] New dependencies have been added to the REQUIREMENTS variable (example).
[ ] New dependencies are only imported inside functions that use them (example).
[ ] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

@balloob Pushed this up before I was ready. Will close and re-open when I'm ready.

source: gharchive/pull-request

repo: home-assistant/home-assistant

license_type: permissive

license_source: github-api


text:
Added Trafikverket train component

Description:
New component for Trafikverket train information.
This component will enable retreival of the following information:

Next departure for specific train line.
Canceled status.
Number of minutes delayed.
Planned time.
Estimated time.
Actual time.
Other information / additional texts
Deviations

Related issue (if applicable): fixes N/A
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#9333
Example entry for configuration.yaml (if applicable):
sensor:
   - platform: trafikverket_train
     api_key: !secret trafikverket_api
     trains:
     - name: "Morning train to Malmö"
       from: "Stockholm Central"
       to: "Malmö C"
       time: "9:25"
       weekday: [mon, tue, wed, thu, fri]

     - name: "Next train to Uppsala"
       from: "Stockholm Central"
       to: "Uppsala C"

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[x] The manifest file has all fields filled out correctly (example).
[x] New dependencies have been added to requirements in the manifest (example).
[x] New dependencies are only imported inside functions that use them (example).
[x] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[x] New files were added to .coveragerc.

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

@fabaff Can you please support me if i need to take any other action for this PR?

@martinhjelmare i think i am missing something here, could you support me here and maybe double check the code suggested?


text:
Add support for new mqtt_json light platform.

Description:
Add a new light platform, based on the existing MQTT light platform, to handle a JSON payload. This reduces the round trips for setting attributes and simplifies the interaction.
The new platform supports on/off, brightness, RGB, transitions, and flashing.
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#760
Example entry for configuration.yaml (if applicable):
# Some optional entries omitted
light:
  - platform: mqtt_json
    name: mqtt_json_light_1
    state_topic: "home/rgb1"
    command_topic: "home/rgb1/set"
    brightness: true
    rgb: true

Checklist:
If user exposed functionality or configuration variables are added/changed:

[X] Documentation added/updated in home-assistant.io

If code communicates with devices, web services, or a:

[X] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[X] New dependencies have been added to the REQUIREMENTS variable (example).
[X] New dependencies are only imported inside functions that use them (example).
[X] New dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[X] New files were added to .coveragerc.

If the code does not interact with devices:

[X] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[X] Tests have been added to verify that the new code works.

Is it odd that the W503 errors weren't reported when I ran tox?

pep257 does not run for the test directory. This is because pep257ing our tests is still a work in progress. Farcy does check the test directory and will only report on newly added code.

@balloob Ah, thanks. Good to know.

You're very close to 100% test coverage

Looks good 🐬  can merge when tests pass

@balloob I just noticed that I added a few lines that don't have coverage. I will fix those ones tomorrow.

Ok, let me know when it is ready to be merged! 👍

Well, I was going to sleep, but I just couldn't resist.
I think I'm happy with it now, assuming the tests pass. Thanks again for the quick feedback.

Damn it. I want 100%.

Alright, @balloob. I'm happy and going to bed for real this time. 👍

Awesome! 🐬 🍺 💃

I'm using this component however I think it's too opinionated on the format of the data payload. Is there a way to make it more customizable? I don't know HA enough to know the proper way of implementing this. I think templating is the way. Do you have any pointers?
To give more context, my light does RGB but also more as the leds are individually addressable, I can do some nice effects.

@Diaoul I see your point, and it is pretty opinionated. The goal was to make a straight-forward way to develop new lights for systems that can consume JSON payloads.
I don't know how you want to use individually addressable LEDs with this.
I can understand wanting to change the format of the payload, but what additional information do you want to send from Home Assistant? I think it currently supports all of the light component's features.

Thanks for taking the time to answer me.
I've already added my custom logic for individually addressable LEDs
outside the light component. Individually addressable LEDs are out of scope
for HA's light component so we agree on that.
However, because my light can do more, my JSON isn't exactly the same as
yours. I'd like the ability to have control over the way the color is
extracted from the JSON and control over the way it is written in JSON.
Here is an example of JSON:
"brighten_rate": 10,"fade_rate":
5,"density":80},"glitter":{"rate":20,"chance":0}}```
And for fixed color, the difference would be:
```{"pattern":{"name":"solid","color":16777198}}```
With the color being 0xRRGGBB in decimal.

That's just my use case, maybe someone else would have some other JSON
format.
I suggest to use templating like so:

color_read_template would be a jinja2 template that gets the JSON as input
and has to return an array of [r, g, b].
So for me :
`color_read_template: {{ [the_json['pattern']['color'] >> 16  & 0xff,
the_json['pattern']['color'] >> 8  & 0xff, the_json['pattern']['color'] &
0xff] }}`
The color_write_template would be the other way around : you're given the
array or [r, g, b] and you have to output a JSON string of what you want
sent.

I'd recommend keeping your settings by default. I think in home automation,
the software should adapt to the hardware and not enforce specific format.
I'd be happy to contribute a PR for this if you agree.

authors: Diaoul balloob corbanmailloux


text:
Use voluptuous for SqueezeBox

Description:
Migration of the configuration check to voluptuous.
Related issue (if applicable): fixes 127528299
Example entry for configuration.yaml (if applicable):
media_player:
  platform: squeezebox
  host: IP_ADDRESS
  port: 9090
  username: USERNAME
  password: PASSWORD

@persandstrom @jaharkes, would be nice if you could take a look at the changes and run a quick test. Thanks.

I have replaced my squeezeplug with chromecast audio so t I can' t help with testing. If I remember correcly username and password can be optional. Besides that I think it looks good.

Thanks. Yes, according to the docs those can be optional.

I approve this - if you want to organize the voluptuous args, please do - otherwise, merge away!

The usual order I used is Required then Optional, both alphabetical.

That's fine then
On Mon, Sep 5, 2016 at 10:21 AM, Fabian Affolter notifications@github.com
wrote:

The usual order I used is Required then Optional, both alphabetical.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/home-assistant/pull/3212#issuecomment-244781038,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC2fZVyKm-cvShpMRquBpHmbOJBAZbatks5qnEGmgaJpZM4J1Kd2
.

I don't remember that there should be any name parameter. Copy paste error?

It has no trouble validating my config, LGTM

Thanks for your feedback. Name will be removed.

@fabaff Merge once built

authors: Teagan42 fabaff jaharkes persandstrom


text:
Build failed with /bin/bash: $'\r': command not found WSL Ubuntu 20.04

System information

OS Platform and Distribution (e.g., Linux Ubuntu 20.04, WSL2):Linux Ubuntu 20.04, WSL2
Target (e.g. desktop cpu, android arm64): python3.9 build.py build --desktop gpu -v
Bazel version: 4.0.0
Python version: 3.9.5
GCC/G++ version: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Unity version:
Android NDK version (if building for Android):
Xcode version (if building for iOS):

Describe the problem
Build failed
Steps to reproduce the issue
-Install WSL 2 with Ubuntu 20.04 distribution
-installed required library
(python 3.9, pip, numpy, bazel, nuget, gcc)
-run python3.9 build.py build --desktop gpu -v
Full logs
nolkeg@DESKTOP-71OR9NF:/mnt/d/UnityProjects/MediaPipeUnityPlugin$ python3.9 build.py build --desktop gpu -v
INFO (build.py): Building protobuf sources...
DEBUG (build.py): Running bazel build -c opt //mediapipe_api:mediapipe_proto_srcs
Starting local Bazel server and connecting to it...
WARNING: Download from http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'com_google_absl' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'com_google_protobuf' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'com_google_googletest' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'com_github_gflags_gflags' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'rules_cc' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'rules_python' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'build_bazel_rules_apple' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'build_bazel_rules_swift' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'build_bazel_apple_support' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'bazel_skylib' because it already exists.
DEBUG: /home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/org_tensorflow/third_party/repo.bzl:108:14:
Warning: skipping import of repository 'pybind11' because it already exists.
DEBUG: Rule 'rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1564776078 -0400"
DEBUG: Repository rules_python instantiated at:
/mnt/d/UnityProjects/MediaPipeUnityPlugin/WORKSPACE:34:23: in
/home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/rules_pkg/deps.bzl:33:10: in rules_pkg_dependencies
/home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule git_repository defined at:
/home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in
INFO: SHA256 (https://github.com/bazelbuild/rules_cc/archive/master.zip) = 1a42099a527cbef96f5148f4a11d95ddb2d7d8781dd68a664430c1c36b2221cb
DEBUG: Rule 'rules_cc' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "1a42099a527cbef96f5148f4a11d95ddb2d7d8781dd68a664430c1c36b2221cb"
DEBUG: Repository rules_cc instantiated at:
/mnt/d/UnityProjects/MediaPipeUnityPlugin/WORKSPACE:78:13: in
Repository rule http_archive defined at:
/home/nolkeg/.cache/bazel/_bazel_nolkeg/2c6909a90362eaccd06b96ea6b50d910/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in
INFO: Analyzed target //mediapipe_api:mediapipe_proto_srcs (43 packages loaded, 666 targets configured).
INFO: Found 1 target...
ERROR: /mnt/d/UnityProjects/MediaPipeUnityPlugin/mediapipe_api/framework/formats/BUILD:98:17: Executing genrule //mediapipe_api/framework/formats:location_data_cs failed: (Exit 127): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
Use --sandbox_debug to see verbose messages from the sandbox
/bin/bash: $'\r': command not found
Target //mediapipe_api:mediapipe_proto_srcs failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 319.508s, Critical Path: 79.02s
INFO: 202 processes: 25 internal, 177 linux-sandbox.
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
File "/mnt/d/UnityProjects/MediaPipeUnityPlugin/build.py", line 392, in
Argument().command().run()
File "/mnt/d/UnityProjects/MediaPipeUnityPlugin/build.py", line 109, in run
self._run_command(self._build_proto_srcs_commands())
File "/mnt/d/UnityProjects/MediaPipeUnityPlugin/build.py", line 51, in _run_command
return subprocess.run(command_list, check=True)
File "/usr/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bazel', 'build', '-c', 'opt', '//mediapipe_api:mediapipe_proto_srcs']' returned non-zero exit status 1.
Additional context

Please make sure that no files under mediapipe_api directory are changed.
git diff mediapipe_api

# if there are diffs, checkout them
# git checkout mediapipe_api

After that, please run python3.9 build.py build --desktop gpu -vv and attach the log (more verbose log).
By the way, is your target correct?
If you'd like to run UnityEditor on Windows, you need to build native libraries on Windows, not WSL2.

Oh, I didn't know that, I thought I could just use WSL2 to build for all platforms.
I originally want to run the project on Android but I want to test the library so I decided to build for windows first.
I'll try building for Android and I'll report back.

I have built for android successfully.
I have struggled a lot with how to install Android SDK and JDK with WSL2 so I use the official setup_adroid_ndk_sdk.sh(something like that)from mediapipe to install both SDK and JDK. That still result in no matching toolchain error and I had to explicitly set up path to SDK and NDK in WORKSPACE
android_ndk_repository(
name = "androidndk",
path = "/home/nolkeg/Android/Sdk/ndk-bundle/android-ndk-r19c"
)
android_sdk_repository(
name = "androidsdk",
path = "/home/nolkeg/Android/Sdk"
)
Even with that I still got an error with the build and have to install zip and unzip.
just gonna post it here in case someone got problems building on wsl2 ubuntu 20.04 like me.

I recommend you use Docker for Windows to build for Android on Windows 10.

I recommend you use Docker for Windows to build for Android on Windows 10.

Unfotunately, I can't use docker because I have windows10 home.

You can use Docker if your OS is Windows 10 Home, because WSL2 backend is supported (see https://docs.docker.com/docker-for-windows/install/).
You cant' use Docker only when you need to run it with Hyper-V backend.

source: gharchive/issue

authors: Nolkeg homuler

repo: homuler/MediaPipeUnityPlugin

license: MIT


text:
MacOS 14 上 报错： sudo: /dev/disk4s1: command not found

MacOS 14 使用时会出现下面的情况

然后硬盘就被卸载了，重新挂载上硬盘又会反复出现上面的情况，还是没法写入。

#22

authors: hoochanlon qinzhenlove

repo: hoochanlon/Free-NTFS-for-Mac

license: Zlib


text:
🛑 FWAw is down

In e0db6c9, FWAw ($FWA_SITE) was down:

HTTP code: 0
Response time: 0 ms

Resolved: FWAw is back up in b03df57 after 20 minutes.

repo: hoopybe/uptime


text:
🛑 FWAw is down

In bc7b94e, FWAw ($FWA_SITE) was down:

HTTP code: 0
Response time: 0 ms

Resolved: FWAw is back up in 3856b29 after 39 minutes.


text:
ghc-mod stderr: EXCEPTION: types: parse error on input ‘]’

the output of channel "ghc-mod server"is
ghc-mod stderr: EXCEPTION: info:     Not in scope: ‘Name’ ghc-mod stderr: EXCEPTION: types: parse error on input ‘]’
ghc-mod 5.8.0
ghc 8.0.2

I open vscode today, it works!
However, I do not change anything.

repo: hoovercj/vscode-ghc-mod

license: mit

license_source: bigquery


text:
package.json has extra comma which leads to an error.

Line 15.

https://github.com/hora-search/hora-wasm/commit/9bf81aff71d9431c600ff68178bd4195754f971b
fix it.

authors: fenghaolinroix salamer

repo: hora-search/hora-wasm


text:
CDPAM-1531 | Add bash script for ccm debugging information collection

This PR contains a bash script that will be used to collect the information required for ccm debugging and it will also perform record the output of few connectivity checks.
This will be placed along with autossh process launcher script and it is part of the ccmv1 debugging improvement tasks. Please provide your review.

@pdarvasi , just note for you: we have discussed this already on slack, the steps will be implemented in cdp-doctor (most of all these checks have been implemented already in cdp-doctor, but not all of these...like collecting specific logs by the cdp-doctor command itself etc.)

closing this based on the comment from @oleewere

authors: lacikaaa meetrajsinhvala oleewere

repo: hortonworks/cloudbreak-images


text:
[CB-7635] Moving RAZ deployment to main node

Moving RAZ deployment to main node
Closes https://jira.cloudera.com/browse/CB-7635

lgtm, tested it.

authors: atxmemo orngejaket

repo: hortonworks/cloudbreak


text:
Resolve inconsistency in editing custom runtime after initial custom runtime configuration

Current behavior
Users encounter varying behavior when attempting to change the custom runtime after the initial configuration. Sometimes, users can successfully change the custom runtime, while in other cases, they are unable to open the date time modal to change it.
Expected behavior
Users should be able to consistently change the custom runtime after it has been initially configured.
Steps to reproduce the issue

Can you handle fixing this bug by yourself?

[ ] YES
[ ] NO

Environment details

Browser:
OS:
Code Version:

Additional information

Might be fixed with https://github.com/hotwax/job-manager/pull/593

repo: hotwax/job-manager


text:
Add option to opt out of Turbo for forms by default and require manual opt in

Anecdotally, one of the biggest deterrents for people upgrading from Turbolinks to Turbo has been the way forms are handled. For Rails users heavily invested in UJS, it's a mammoth task to rework all the forms to work with Turbo.
I've worked around this in the past in Rails by monkey patching form_with to disable Turbo with data-turbo=false; but I think it'd be a great idea to allow a global setting within Turbo to navigate only on forms that have opted in with data-turbo=true.
I'm thinking it could be set at initialization time like: Turbo.start({ forms_opt_out_by_default: true })
I'm happy to spend some time working on this if this is something maintainers would like to support so just opening this issue to get people's thoughts!

@ayushn21 have you tried setting <html data-turbo="false"> or <body data-turbo="false">, and then incrementally opting into Turbo on an individual <a> and <form> basis?
If it's <form> elements that are the issue, It might be more straightforward to disable all elements, and then opt back in <a> elements:
addEventListener("turbo:load", () => {
  for (const link of document.querySelectorAll("a")) {
    link.setAttribute("data-turbo", "true")
  }
})

Hey @seanpdoyle that is a good solution in the short term. Thanks!
Bigger picture, it seems like if the UJS remote form behaviour was managed via the config.action_view.form_with_generates_remote_forms in your environment, there should also be an equivalent config.action_view.form_with_generates_turbo_forms or similar.
Am I totally out to lunch or does this make some sense?

config.action_view.form_with_generates_turbo_forms

I think that's totally reasonable. At the moment, Action View is completely unaware of Turbo, and I'm not aware of any Rails mechanism to set default options for helper methods.
At a high level, what would you like to see happen when config.action_view.form_with_generates_turbo_forms is true?

Not coincidentally, we were doing a fair bit of digging on this last night in the StimulusReflex Discord.
Ultimately, the token orm_with_generates_remote_forms shows up in just six files, most specifically in form_helper.rb.
The way I would approach this is that right now, it's almost as if there's already an invisible form_with_generates_turbo_forms setting that defaults to true. 😉
If this value was set to false, we'd add a data-turbo="false" attribute to the form helper.

Thanks @seanpdoyle. That's a great stopgap solution on the JS side of things.
I do agree with @leastbad that it'd be great to have a cleaner way to prevent Turbo from navigating on forms by default though. The chat we were having in the StimulusReflex Discord was also centered on what level should this setting be configured (i.e. within Turbo or through a server side helper like config.action_view.form_with_generates_turbo_forms)
If you believe the right way to go about this would be to handle the exclusion of forms on the server rather than within Turbo, I reckon it makes sense to close this issue and we can take the discussion to rails/rails.
Thanks for your input!

Addressed in #419

authors: ayushn21 brunoprietog leastbad seanpdoyle

repo: hotwired/turbo


text:
donate in crypto

if you will provide some (any) crypto donate button will donate equivalent of 21 USD.
if app will use 2x less resources  will donate USD 21 more (make it run of poor children laptops, right now it memory hog for such simple interface).

Hi @dzmitry-lahoda , I have no coins yet :) Not even sure how would I put it into my taxes :)
Is there some easy way how to allow for this? Never really looked into this...

Hi @dzmitry-lahoda , I have no coins yet :) Not even sure how would I put it into my taxes :)
Is there some easy way how to allow for this? Never really looked into this...

@dzmitry-lahoda I've thrown together a quick foundation for a future Stretchlylite here.  Maybe try it out and see how it performs on your laptop.
It doesn't actually work yet, it's just the interface.

authors: GirkovArpa dzmitry-lahoda hovancik

repo: hovancik/stretchly


text:
File uploading examples

I can't get the file upload functionality working this is my latest attempt:
            var filepath = path.join(__dirname, '..', '/screenshots/'+ name + '.png');
            fs.readFile(filepath, function (err, data) {
                if (err) throw err;

                console.log(message.channel);

                bot.api.files.upload({
                    content: data,
                    channel: message.channel
                },function(err,res) {
                    if (err) {
                        bot.botkit.log("Failed to add screenshot :(",err);
                        bot.botkit.log(data);
                    }
                });

But I don't get anything back, the data is there I also tried replacing content: with file and I get an error from the api saying not file data. Any help is highly appreciated.

@amacgregor : Did you find the solution?

I could never make the botkit method work so I ended up using a curl wrapper for the fileuploads that worked like a charm:
            var filepath = path.join(__dirname, '..', '/screenshots/'+ name + '.png');
            child = exec('curl -F file=@' + filepath  + ' -F channels=' + message.channel + ' -F token=' + process.env.token + ' https://slack.com/api/files.upload',
                function (err, stdout, stderr) {
                    if (err !== null) {
                        bot.botkit.log("Failed to add screenshot :(",err);
                    }
                }
            );

I could never make the botkit method work so I ended up using a curl wrapper for the fileuploads that worked like a charm:
            var filepath = path.join(__dirname, '..', '/screenshots/'+ name + '.png');
            child = exec('curl -F file=@' + filepath  + ' -F channels=' + message.channel + ' -F token=' + process.env.token + ' https://slack.com/api/files.upload',
                function (err, stdout, stderr) {
                    if (err !== null) {
                        bot.botkit.log("Failed to add screenshot :(",err);
                    }
                }
            );

Your method worked like a charm.
With https://www.npmjs.com/package/node-slack-upload, you can also add comment to your uploaded file.

@amacgregor, I'm sorry that the Botkit method didn't work for you! I've labeled this as a bug.

@amacgregor - did @mlallinder 's suggestion fix your issue? I am closing this for now, but please reopen if this is still an issue.

Still having problems...
bot.api.files.upload({
                    file: fs.createReadStream("file.xlsx"),
                    filename: "file"+randomNum+".xlsx",
                    filetype: "xlsx",
                    channels: message.channel
                },function(err,res) {
                    if (err) {
                        console.log("Failed to add file :(",err)
                        bot.reply(message, 'Sorry, there has been an error: '+err)
                    }
                })

does not work (invalid_array_arg) but
  /*   var r = request.post('https://slack.com/api/files.upload', function (err, res, body) {
                    // this works
                })

                        var form = r.form()
                        form.append('token', token)
                        form.append('filename', "file"+randomNum+".xlsx")
                        form.append('file', fs.createReadStream("file.xlsx"))
                        form.append('channels', message.channel)
             */

does work

authors: RafaelCosman ZuzooVn amacgregor demac-rd peterswimm shawngiese

repo: howdyai/botkit


text:
Fix function getGlobalScoreList into setGlobalScore

Pull Request Template
Type of PR

[ ] Feature - Platform
[ ] Feature - Game
[x] Fix - Bug

Description of PR

wrong function naming which posts new score is written as a 'getGlobalScoreList'

Changes Made

file name change
unused import and vars erased

Testing Method

Related Issues

Checklist

[ ] Prepared for code review (Assignees set, Labels applied, etc.)
[ ] Checked that changes do not negatively affect existing features
[ ] Documentation completed for new features
[ ] All unit tests passed
[ ] Testing in the test environment completed
[ ] Linked this PR in the related issue
[ ] Communicated sufficiently with the relevant team or person before submitting PR

needs to be merged after #58

There is a commit identical to #58. I think you have to be careful not to get it twist the branch

#58 approved. Merging can proceed as outlined in the PR description.
Thanks for addressing the incorrect naming issue.
Also, #58 #59 both modify global-score-board-provider file.
Ensure to check for conflicts to avoid a branch crash during merging.

There is a commit identical to #58. I think you have to be careful not to get it twist the branch

Understood

authors: Ooyd XPECTER junman95

repo: hp-potion/vsc-gameboy


text:
H5Fget_access_plist does not return a valid faplid

Summary
When using Cache Vol and Async Vol, it seems that H5Fget_access_plist does not return a valid faplid.
The returned id is non-negative but seems not a property list.
Error Details
% echo $HDF5_VOL_CONNECTOR
cache_ext config=cache_1.cfg;under_vol=512;under_info={under_vol=0;under_info={}}

% mpirun -n 1 ./test
HDF5-DIAG: Error detected in HDF5 (1.13.3-1) MPI-process 0:
  #000: ../../hdf5-dev/src/H5Pfapl.c line 1487 in H5Pget_driver(): can't get driver
    major: Property lists
    minor: Can't get value
  #001: ../../hdf5-dev/src/H5Pfapl.c line 1444 in H5P_peek_driver(): not a file access property list
    major: Property lists
    minor: Inappropriate type
  #002: ../../hdf5-dev/src/H5Pint.c line 4067 in H5P_isa_class(): not a property list
    major: Invalid arguments to routine
    minor: Inappropriate type

Test Program
 Click here to see the test program:
#include <hdf5.h>
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

#define N 10

#define CHECK_ERR(A) {if (A < 0) { printf("Error at line %d: code %d\n", __LINE__, A); }}

int main(int argc, char **argv) {
  herr_t err = 0;

  int mpi_required;
  const char *file_name = "test.h5";

  hid_t fid = -1;          // File ID
  hid_t faplid = -1;       // File Access Property List
  hid_t plist_id = -1;
  hid_t faplid2 = -1;

  // init MPI
  err = MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &mpi_required);
  CHECK_ERR(err);

  // create file
  faplid = H5Pcreate(H5P_FILE_ACCESS);
  CHECK_ERR(faplid);
  H5Pset_fapl_mpio(faplid, MPI_COMM_WORLD, MPI_INFO_NULL);
  fid = H5Fcreate(file_name, H5F_ACC_TRUNC, H5P_DEFAULT, faplid);
  CHECK_ERR(fid);

  // get faplid
  faplid2 = H5Fget_access_plist (fid);
  CHECK_ERR (faplid2);
  plist_id = H5Pget_driver (faplid2);  // Error occurs here

  if (fid >= 0)
    H5Fclose(fid);
  if (faplid >= 0)
    H5Pclose(faplid);

  MPI_Finalize();

  return 0;
}

Libraries Versions (commit number)
 Click here to see the details

HDF5 develop branch: HDFGroup/hdf5@b5598575bb8a2495d6f306233b00d612258ad718
Argobots main branch: pmodels/argobots@dce6e727ffc4ca5b3ffc04cb9517c6689be51ec5
AsyncVol develop branch: hpc-io/vol-async@0a92d232ed01ecbb6ab59fbfa4807458c88922a7
Cache Vol develop branch: hpc-io/vol-cache@f453900b64cfbc5d3197acb5292e6e379ce2ac20

Will this issue be addressed soon?

Hi @wkliao @yzanhua, this is an issue of the HDF5 library. I encountered this when I was running E3SM-IO. I have to comment out H5Fget_access_plist in the code to make it running. I mentioned it to Neil before. Maybe report this to HDF5?

I am not sure whether this is HDF5's issue.
@yzanhua testes the small program he provided in this PR using the followings.
It failed only when using Cache+Async VOLs.
Cache+Async VOL: fail
Cache VOL only: success
Passthrough VOL only: success
Log VOL only: success
using:
HDF5: 1.13.3,
Cache VOL: master branch
Async VOL: v1.4

It also fails when using Async only. It seems like Async VOL (instead of Cache VOL) is not handling faplid correctly.

Yes, it is with Async + HDF5.
@houjun , did you encounter this issue before?

Yes, I remember it is related to future ID when async is used, I'll take another look and check with HDF people.

The provided test program failed in H5Pget_driver (the line where the invalid faplid2 is first used). I also tested using other H5Pget_xxxxs to replace H5Pget_driver to see if the program still fails. The results might be helpful to debugging.
H5Pget_driver_info, H5Pget_fapl_mpio andH5Pget_fapl_core fail with the same error messages, complaining about "not a property list".
However, using H5Pget_fclose_degree and H5Pget_evict_on_close can run without a problem.

authors: houjun wkliao yzanhua zhenghh04

repo: hpc-io/vol-cache


text:
Display multiple pins for given geo coordinates

As a user I want to be able to display two markers on the map based on two hardcoded locations in order to get a visual aid for the end points of my route.
Estimated Effort: 5

Actual effort: 4.5h

repo: hpi-swa-teaching/SqueakMaps


text:
Calling driver.initled with wrong pins causes crash

Tested on Esp32-S3
This is the output
pins[, 26, 33, 32, 35, 34, 26]
Start driveroffset initiationsemaphore initPins initiationESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x8 (TG1WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x403743c0
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x3ac
load:0x403c9700,len:0x9a8
load:0x403cc700,len:0x2920
entry 0x403c98b8
COMMODORE BASIC V2
WARNING - possible crash: (code 5) Reset (software or hardware) due to interrupt watchdog. Core#0 (code 8) watchdog; Core#1 (code 8) watchdog.
Possible solution: test on valid pins before allocating

List of valid pins in StarLight for S3:

Created using this code:

This is using a valid pin, but it is crashing with this error message:

But pin 2 is a valid pin isn't it?


text:
请问2X版本还修复吗

let res = Easewechat.Factory.Payment({
      debug: true,
})

res.setSubMerchant (appid,mch_id)

无效

还维护的，不过假期出去玩了，暂时改不了

假期愉快 记录一下 这样都无效
let payment = new Easewechat.Factory.Payment({
  sub_appid: appid,
  sub_mch_id: mch_id,
});

res.setSubMerchant (appid,mch_id)

以下为有效
payment.order.unify({sub_mch_id: mch_id, sub_appid :appid})

发布了v2.12.5

authors: ValueLan hpyer

repo: hpyer/node-easywechat


text:
Col out of range when there is no match between typed word and LSP suggestion (?)

FAQ

[X] I have checked the FAQ and it didn't resolve my problem.

Announcement

[X] I have checked Breaking change announcement.

Minimal reproducible full config
Sorry, I tried to reproduce with a minimal config and failed :(
I've included the stack trace that I get and some debug prints I put together, which I think should give some intuition to the underlying error (see below). If you could provide some pointers for more debugging, I'd appreciate that.
Description
I'm using pyright LSP completion as usual, when I get the following error:
Error in decoration provider cmp.view.custom_entries_view.win:
Error executing lua: .../nvim/lazy/nvim-cmp/lua/cmp/view/custom_entries_view.lua:84: Invalid 'col': out of range
stack traceback:
	[C]: in function 'nvim_buf_set_extmark'
	.../nvim/lazy/nvim-cmp/lua/cmp/view/custom_entries_view.lua:84: in function <.../nvim/lazy/nvim-cmp/lua/cmp/view/custom_entries_view.lua:57>
	[C]: in function 'nvim_exec2'
	vim/_editor.lua: in function 'cmd'
	.../share/nvim/lazy/nvim-notify/lua/notify/service/init.lua:63: in function 'push'
	...ocal/share/nvim/lazy/nvim-notify/lua/notify/instance.lua:77: in function <...ocal/share/nvim/lazy/nvim-notify/lua/notify/instance.lua:41>
	...e/nvim/lazy/noice.nvim/lua/noice/view/backend/notify.lua:169: in function '_notify'
	...e/nvim/lazy/noice.nvim/lua/noice/view/backend/notify.lua:199: in function <...e/nvim/lazy/noice.nvim/lua/noice/view/backend/notify.lua:176>
	[C]: in function 'xpcall'
	...local/share/nvim/lazy/noice.nvim/lua/noice/util/call.lua:149: in function 'try'
	...local/sh
la

The error is repeated many-many times, once every redraw I believe.
It happens right here. I added some debug prints, like so:
            local ok, _ = pcall(vim.api.nvim_buf_set_extmark, buf, custom_entries_view.ns, i, a + m.word_match_start - 1, {
              end_line = i,
              end_col = a + m.word_match_end,
              hl_group = m.fuzzy and 'CmpItemAbbrMatchFuzzy' or 'CmpItemAbbrMatch',
              hl_mode = 'combine',
              ephemeral = true,
            })
            if not ok then
              vim.api.nvim_notify(vim.inspect({ lines = vim.api.nvim_buf_get_lines(buf, i, i + 1, false), a = a, start = m.word_match_start, field = field }), vim.log.levels.INFO, {})
            end

And this prints
{
  a = 0,
  lines = { "RepeatedScalarFieldContainer~          Class    google.protobuf.internal.containers " },
  start = -1
}

With these 2 values, it means that a + m.word_match_start - 1 is -2, which is what is triggering the error. However, I don't understand what is the special case here: what does it mean if m.word_match_start is -1? The word I entered when I saw this error was recons (trying to type reconstructed), does it mean there is no match between what I entered and the suggested words?
Either way, I would really appreciate some pointers - what I did so far is just avoid that line if m.word_match_start == -1, that makes the plugin usable again for me
Steps to reproduce
As I said, I don't have a great repro, sorry. I think it just happens when there is no match between what I typed and what LSP is suggesting, but I can't reliably reproduce this.
Expected behavior
No error messages, just completion.
Actual behavior
A ginormous wall of error messages, 1 error message every redraw, CPU explodes redrawing all these notifications, laptop gets very laggy.
Additional context
I'm using latest master (as of today).
NVIM v0.10.0-dev-737+gdf2f5e391
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Run "nvim -V1 -v" for more info

The issue happened with pyright 1.1.316.
Having made the dummy fix (check for -1 before calling nvim_api_set_extmark), the plugin is usable again for me, but I think there's a deeper underlying issue (matching is incorrect or missing match is not accounted for). I would appreciate any pointers in figuring it out - sorry again that I can't create a good repro, it's hard to trigger the LSP server into giving suggestions that don't match input.

Hi @alfaix
In my case this settings caused issue
window = {         completion = {           side_padding = 0,         },         documentation = {           side_padding = 0,         },       },
probably error happens, when scrollbar appears/disappears.

authors: alfaix chshrct


text:
Latex: Inserting { instead of ( when completing functions

Is there a way to do this? It's rather annoying having to replace () manually every time a function is inserted.

I don't familiar with latex so I can't understand what you mean.

Are you using nvim-autopairs plugin?

And I think nvim-cmp does not provides { and ( so I think it is not related to nvim-cmp.

I don't think this is related to nvim-cmp
See this https://github.com/LunarVim/LunarVim/issues/1257

Oh ok, ok. Sorry 😅

See https://github.com/windwp/nvim-autopairs#mapping-cr, expand nvim-cmp section and copy the setup function.
I think you can close this issue now

authors: Joakker hrsh7th rhcher xeluxee


text:
image_downloader_web >=2.0.0 depends on http ^0.13.5

Seems project depends on a very old pinned version of http. Could you please update it?
In projects which use up-to-date http package, attempt to install image_downloader_web results in:
Because image_downloader_web >=2.0.0 depends on http ^0.13.5 and XXX depends on http ^1.1.0, image_downloader_web >=2.0.0 is forbidden.
So, because XXX depends on image_downloader_web ^2.0.2, version solving failed.

I sent a pr: https://github.com/hrvojecukman/image_downloader_web/pull/9

Thanks for the PR!

authors: alexVinarskis hrvojecukman wyyadd


text:
【layui版问题】getChecked方法取值异常

版本：
layui版本：v2.6.8
eleTree插件版本：hsiangleev-layui-eleTree
问题描述：
初始化eleTree，参数为checkStrictly为默认【即父子节点联动的情况下】
如果只是勾选父节点，然后直接通过getChecked方法获取只是其父节点，而相关的子节点没法获取

但是展开子节点之后，再次通过getChecked方法就可以正常获取所有选中的节点

并且这种情况出现在每次加载相关页面的时候，在展开后获取正常。
分析原因：
在未展开子节点的情况下，父节点相关的子节点dom并没有生成。而getChecked则是直接通过选择器选择的Dom，所以就无法获得相关子节点的数据。
if(leafOnly){
                el=options.elem.find(".layui-icon.leaf-icon").parent(".eleTree-node-content-icon")
                    .siblings("input.eleTree-hideen[eletree-status='1']");
            }else if(includeHalfChecked){
                el=options.elem.find("input.eleTree-hideen[eletree-status='1'],input.eleTree-hideen[eletree-status='2']");
            }else{
                el=options.elem.find("input.eleTree-hideen[eletree-status='1']");
            }
            el.each(function(index,item) {
                var obj={};
                var id=$(item).parent(".eleTree-node-content").parent(".eleTree-node").data(options.request.key);
                var label=$(item).siblings(".eleTree-node-content-label").text();
                obj[options.request.key]=id;
                obj[options.request.name]=label;
                obj.elem=item;
                obj.othis=$(item).siblings(".eleTree-checkbox").get(0)
                arr.push(obj);
            })

抱歉我的问题，没有看到renderAfterExpand这个属性

repo: hsiangleev/eleTree


text:
Add stats page

Closes #66

This is an even better implementation of my feature request than I imagined!
Very well done, thank you @hsjoberg! :black_heart:

authors: MaxHillebrand hsjoberg

repo: hsjoberg/fork-explorer


text:
Cookie expiration

Have you given any thought to cookie expiration?
key=value; Expires=Thu, 01-Jan-1970 00:00:01 GMT

It didn't handle any cookie expiration at the moment, on the other hand, it's killing the cookie if u are trying to set with the expiration date.
if (parts.length === 2) {
key = parts[0], value = parts[1];
} else {
value = parts[0];
key = '';
}

I would like to see some form of cookie expiration done as well.

authors: Mercieral loming sean3z

repo: hstove/electron-cookies


text:
Update CONTRIBUTING.md

corrected command for installing development version in windows

Thank you @hosseingt! :pie:

authors: BoboTiG hosseingt

repo: httpie/httpie


text:
Calculate and export weather data for simulation

We have two different data sets for HTW 2015:

[ ] htw-wetter (measurements, 1 min)

[x] with gaps
[ ] filled data

[x] open_FRED (reanalysis, 15 min)

Calculate and export data that can be used in the simulation software.
There are different parameters that need to be matched and converted.

[x] HTW: G_hor + G_gen (cmp+si)
[x] FRED: dhi, dirhi, dni.

Polysun: Gh, Dh und Lh (1s - 1M)
PVsol: Ta Gh FF RH

filled weatherdata might not be necessary as hourly average weatherdata is used anyways

authors: Ludee nailend


text:
fix: 日期范围选择器取消联动

目前范围日期选择未取消联动导致无法跨年份选择

可以跨年选择

好像只能跨 1 年吧

可以跨任意年

为什么我这里使用只能选择跨一年呢！

你没弄明白怎么用

渍渍渍，选择的过程中取消了之前选中的了

渍渍渍，我这边选择的时候操作错误，选择的过程中取消了之前选中的了

repo: huajian-pro/resume-design


text:
reflect.StructField.IsExported only support go 1.17 or above

I was using sqlbuilder 1.14.0 in project. My golang version of env is 1.16 and can not build project because  the IsExported using in the sqlbuilder pkg. Should upgrade go.mod file in sqlbuilder 1.14.0

The implementation of IsExported is extremely simple -- just check f.PkgPath == "". It's not worth to drop go1.16 support for this method. I'll fix it as a bug.

Fix it in 20e19b9.

authors: huandu ranhaoliuLeo

repo: huandu/go-sqlbuilder


text:
🛑 TW591 is down

In cbcabfe, TW591 (https://www.591.com.tw) was down:

HTTP code: 502
Response time: 619 ms

Resolved: TW591 is back up in 433e772.


text:
Integrate with osqueryd

This will give us a lot more realtime data, and we won't miss out on short-lived processes between our nebula queries. We may even be able to use it for FIM.
Will take a pretty big architectural rewrite of the daemon, however.

https://jira.corp.adobe.com/browse/CSTTEAM-43899

This is (mostly) done.

repo: hubblestack/hubble


text:
Cde landing page updates

#470

@edlu77 pull from develop and fix the footer conflicts (the old cde-ui footer component should be removed)

Once @LibbyUX signs off, the code looks fine, so I'll merge once she approves.

@bherr2 & @edlu77 - This is great! Thanks so much!

Hey @bherr2 & @edlu77 - I just noticed something that caught my eye:
The footer is not centered in this preview: https://667ec9f250c3bc30798726e9--hra-ui.netlify.app/apps/cde-ui/

I updated the +1920px node in Figma so it stays centered when the width is larger than 1920px: https://www.figma.com/design/G0xVFFsAAnBwzHd8PtQEeb/Cell-Distance-Explorer?node-id=2147-11964&t=6a8Dh3qYfZTLUTYa-4
I will work on a more intuitive/responsive solution for our design system footer, as well, so we can move away from 9 different breakpoints for all the components

repo: hubmapconsortium/hra-ui


text:
Race condition? Object.keys(this.props.images)

Describe the bug
This line:
const imageNames = Object.keys(this.props.images);

produced this error
TypeError: "can't convert undefined to object"
vitessce.min.js:47
vitessce.min.js:31
...

The spatial pane never filled in, and the please wait never cleared.
... but it didn't repeat when I reloaded.
To Reproduce
no reproducer
Environment:

Release or git hash:
Browser: FF
Browser version:

I am going to close this, I don't think this line exists any more (maybe Viv handles this part of the image rendering now?)

authors: keller-mark mccalluc

repo: hubmapconsortium/vitessce


text:
budapestkornyeke.hu, hvg.hu

Closes https://github.com/hufilter/hufilter-dev/issues/367

@Alex-302 what do you think, should we block ex1tp.com generally? i mean
||ex1tp.com^$third-party

I think so. Also added to our filter.

@ehrhardt06 please check this pr and if you think it's good, we'll merge it

repo: hufilter/hufilter-dev


text:
Compatibility with Sanic 19.6.2

Hi,
Describe the bug
I just bumped into an error when upgrading to newest version of Sanic using ASGI interface.
Since ASGI lifespan protocol has support for only two server events, we no longer can make usage of before_server_start and after_server_stop server events.
Sanic OpenAPI uses before_server_start server event and this give us the following warning:
UserWarning: You have set a listener for "before_server_start" in ASGI mode. It will be executed as early as possible, but not before the ASGI server is started.

To Reproduce
Run Sanic using an ASGI interface
Expected behavior
Make sure this warning doesn't happen.
Environment:

ubuntu 18.04
Version [0.5.3]

On the decorator of the method build_spec at the file swagger.py I suggest the following change:
before
@blueprint.listener('before_server_start')

after
@blueprint.listener('after_server_start')

What do you think about it?

Sounds good. Would you wanna submit a PR for this?

Yes, I'll do it.

@chenjr0719 I just created PR #130 for it

:+1:  Nice catch.

repo: huge-success/sanic-openapi


text:
[Bug] Failed to load, cause: Failed to deserialize: "property_key"

Bug Type (问题类型)
No response
Before submit

[X] 我已经确认现有的 Server-Issues 与 Loader-Issues 中没有相同 / 重复问题

Environment (环境信息)

Server Version: v0.12.0
Loader Version: v0.12.0
Data Size:  xx vertices, xx edges

Expected & Actual behavior (期望与实际表现)
2022-04-02 17:44:59 [main] [ERROR] c.b.h.l.u.Printer [] - Failed to load, cause: Failed to deserialize: {"property_key": {"id":1,"name":"id","data_type":"LONG","cardinality":"SINGLE","aggregate_type":"NONE","write_type":"OLTP","properties":[],"status":"CREATED","user_data":{"~create_time":"2022-04-02 17:44:59.867"}}, "task_id": 0}
com.baidu.hugegraph.rest.SerializeException: Failed to deserialize: {"property_key": {"id":1,"name":"id","data_type":"LONG","cardinality":"SINGLE","aggregate_type":"NONE","write_type":"OLTP","properties":[],"status":"CREATED","user_data":{"~create_time":"2022-04-02 17:44:59.867"}}, "task_id": 0}
	at com.baidu.hugegraph.rest.RestResult.readObject(RestResult.java:64) ~[hugegraph-common-2.0.1.jar:2.0.1.0]
	at com.baidu.hugegraph.api.schema.PropertyKeyAPI.create(PropertyKeyAPI.java:52) ~[hugegraph-client-2.0.1.jar:2.0.1.0]
	at com.baidu.hugegraph.driver.SchemaManager.addPropertyKey(SchemaManager.java:89) ~[hugegraph-client-2.0.1.jar:2.0.1.0]
	at com.baidu.hugegraph.driver.SchemaManager.addPropertyKey(SchemaManager.java:84) ~[hugegraph-client-2.0.1.jar:2.0.1.0]
	at com.baidu.hugegraph.structure.schema.PropertyKey$BuilderImpl.create(PropertyKey.java:158) ~[hugegraph-client-2.0.1.jar:2.0.1.0]
	at com.baidu.hugegraph.structure.schema.PropertyKey$BuilderImpl.create(PropertyKey.java:141) ~[hugegraph-client-2.0.1.jar:2.0.1.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
	at com.baidu.hugegraph.structure.schema.BuilderProxy.invoke(BuilderProxy.java:56) ~[hugegraph-client-2.0.1.jar:2.0.1.0]
	at com.sun.proxy.$Proxy49.create(Unknown Source) ~[?:?]
	at com.baidu.hugegraph.structure.schema.SchemaBuilder$create.call(Unknown Source) ~[?:?]
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) ~[groovy-all-2.4.6.jar:2.4.6]
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) ~[groovy-all-2.4.6.jar:2.4.6]
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117) ~[groovy-all-2.4.6.jar:2.4.6]
	at Script1.run(Script1.groovy:2) ~[?:?]
	at com.baidu.hugegraph.loader.executor.GroovyExecutor.execute(GroovyExecutor.java:58) ~[hugegraph-loader-0.12.0.jar:?]
	at com.baidu.hugegraph.loader.HugeGraphLoader.createSchema(HugeGraphLoader.java:158) ~[hugegraph-loader-0.12.0.jar:?]
	at com.baidu.hugegraph.loader.HugeGraphLoader.load(HugeGraphLoader.java:109) [hugegraph-loader-0.12.0.jar:?]
	at com.baidu.hugegraph.loader.HugeGraphLoader.main(HugeGraphLoader.java:73) [hugegraph-loader-0.12.0.jar:?]
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "property_key" (class com.baidu.hugegraph.structure.schema.PropertyKey), not marked as ignorable (10 known properties: "check_exist", "status", "properties", "aggregate_type", "name", "user_data", "data_type", "id", "cardinality", "write_type"])
 at [Source: (String)"{"property_key": {"id":1,"name":"id","data_type":"LONG","cardinality":"SINGLE","aggregate_type":"NONE","write_type":"OLTP","properties":[],"status":"CREATED","user_data":{"~create_time":"2022-04-02 17:44:59.867"}}, "task_id": 0}"; line: 1, column: 229] (through reference chain: com.baidu.hugegraph.structure.schema.PropertyKey["property_key"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:987) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1974) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1686) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperties(BeanDeserializerBase.java:1635) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:541) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1390) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3548) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3516) ~[jackson-databind-2.12.1.jar:2.12.1]
	at com.baidu.hugegraph.rest.RestResult.readObject(RestResult.java:62) ~[hugegraph-common-2.0.1.jar:2.0.1.0]
	... 20 more

PropertyKeyAPI.create:
https://github.com/hugegraph/hugegraph-client/blob/6f4876f9911362698dfbf2a652f0843df2103bdd/src/main/java/com/baidu/hugegraph/api/schema/PropertyKeyAPI.java#L52
Vertex/Edge example (问题点 / 边数据举例)
No response
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
No response

The reason is server version updated

repo: hugegraph/hugegraph-loader


text:
The more nodes I use, the slower the training speed，the more time accelarate.backward(loss) take

System Info
- `Accelerate` version: 0.12.0
- Platform: Linux-4.19.159.mizar-x86_64-with-glibc2.2.5
- Python version: 3.8.13
- Numpy version: 1.22.4
- PyTorch version (GPU?): 1.8.0 (True)
- `Accelerate` default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: DEEPSPEED
- mixed_precision: no
- use_cpu: False
- num_processes: 12
- machine_rank: 0
- num_machines: 3
- main_process_ip: 10.233.66.86
- main_process_port: 9011
- main_training_function: main
- deepspeed_config: {'deepspeed_multinode_launcher': 'standard', 'gradient_accumulation_steps': 1, 'gradient_clipping': 1.0, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': True, 'zero_stage': 2}
- fsdp_config: {}
- downcast_bf16: no

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
[ ] My own task or dataset (give details below)

Reproduction
with accelerator.accumulate(model):
     outputs = model(**batch)
     torch.cuda.synchronize()
     loss = output[0]
     t1 = time.time()
     accelerator.backward(loss)
     t2 = time.time()
     optimizer.step()
     lr_scheduler.step()
     optimizer.zero_grad()

Expected behavior
Hi, everyone.  I use accelerate to  Distributed train. Get the questions:The more nodes I use, the slower the training speed，the more time accelarate.backward(loss) takes.
1 node, the function of accelerator.backward(loss) takes 500 ms
2 node, the function of accelerator.backward(loss) takes 5000 ms
3 node, the function of accelerator.backward(loss) takes 7500 ms

Yes, the more you distribute your training, the more communication cost you pay.

Yes, the more you distribute your training, the more communication cost you pay.

thanks to your reply, is there other ways to solve this problem

You can use gradient accumulation with steps > 1 so the communication happens less frequently.

You can use gradient accumulation with steps > 1 so the communication happens less frequently, or find a way to increase throughput between the nodes

according to your advice , I set  gradient accumulation with steps = 4
accelerator = Accelerator(gradient_accumulation_steps=4)
with accelerator.accumulate(model):
     outputs = model(**batch)
     torch.cuda.synchronize()
     loss = output[0]
     t1 = time.time()
     accelerator.backward(loss)
     t2 = time.time()
     optimizer.step()
     lr_scheduler.step()
     optimizer.zero_grad()

but  I find every batch I run  the time(t2 -t1) of the backward(loss)  is still  the same , It doesn't seem the communication   less frequently? Is there something wrong with my code ?

@Roysky here's my test results running on two GPUs and a single GPU (note that in a single GPU, I doubled the batch size so they're equivalent):
Note that the test was ran on the nlp_example script, strictly the training portion for a single epoch
Gradient accumulation step size of 1, two GPUs, "real" batch size of 32:

Average time / step: 0.027s
Smallest: 0.023s
Largest: 0.0378s

Gradient accumulation step size of 4, two GPUs, "real" batch size of 32 (counting grad accum):

Average time / step: 0.022s
Smallest: 0.017s
Largest: 0.042s

Gradient accumulation step size of 1, single GPU, "real" batch size of 32:

Average time / step: 0.107s
Smallest: 0.017s
Largest: 0.499s

Gradient accumulation step size of 4, "real" batch size of 32 (counting grad accum):

Average time / step: 0.016s
Smallest: 0.013s
Largest: 0.046s

You can see that gradient accumulation is faster than no gradient accumulation (even multiplying by the number of extra steps being done), and time is reduced some when doing multi-GPU, but the slowdowns are also not immense (0.007s at the most).
You can see the script I used here, you should change batch_size and gradient_accumulation_steps depending on the combination for the "real batch size". Note that Accelerate uses the stated batch size on each GPU, not in total:
# coding=utf-8
# Copyright 2021 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse

import evaluate
import torch
from datasets import load_dataset
from torch.optim import AdamW
from torch.utils.data import DataLoader
from transformers import AutoModelForSequenceClassification, AutoTokenizer, get_linear_schedule_with_warmup, set_seed

from accelerate import Accelerator, DistributedType

MAX_GPU_BATCH_SIZE = 16
EVAL_BATCH_SIZE = 32

def get_dataloaders(accelerator: Accelerator, batch_size: int = 16):
    """
    Creates a set of `DataLoader`s for the `glue` dataset,
    using "bert-base-cased" as the tokenizer.

    Args:
        accelerator (`Accelerator`):
            An `Accelerator` object
        batch_size (`int`, *optional*):
            The batch size for the train and validation DataLoaders.
    """
    tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
    datasets = load_dataset("glue", "mrpc")

    def tokenize_function(examples):
        # max_length=None => use the model max length (it's actually the default)
        outputs = tokenizer(examples["sentence1"], examples["sentence2"], truncation=True, max_length=None)
        return outputs

    # Apply the method we just defined to all the examples in all the splits of the dataset
    # starting with the main process first:
    with accelerator.main_process_first():
        tokenized_datasets = datasets.map(
            tokenize_function,
            batched=True,
            remove_columns=["idx", "sentence1", "sentence2"],
        )

    # We also rename the 'label' column to 'labels' which is the expected name for labels by the models of the
    # transformers library
    tokenized_datasets = tokenized_datasets.rename_column("label", "labels")

    def collate_fn(examples):
        # On TPU it's best to pad everything to the same length or training will be very slow.
        max_length = 128 if accelerator.distributed_type == DistributedType.TPU else None
        # When using mixed precision we want round multiples of 8/16
        if accelerator.mixed_precision == "fp8":
            pad_to_multiple_of = 16
        elif accelerator.mixed_precision != "no":
            pad_to_multiple_of = 8
        else:
            pad_to_multiple_of = None

        return tokenizer.pad(
            examples,
            padding="longest",
            max_length=max_length,
            pad_to_multiple_of=pad_to_multiple_of,
            return_tensors="pt",
        )

    # Instantiate dataloaders.
    train_dataloader = DataLoader(
        tokenized_datasets["train"], shuffle=True, collate_fn=collate_fn, batch_size=batch_size, drop_last=True
    )
    eval_dataloader = DataLoader(
        tokenized_datasets["validation"],
        shuffle=False,
        collate_fn=collate_fn,
        batch_size=EVAL_BATCH_SIZE,
        drop_last=(accelerator.mixed_precision == "fp8"),
    )

    return train_dataloader, eval_dataloader

def training_function(config, args):
    accelerator = Accelerator(
        cpu=args.cpu,
        mixed_precision=args.mixed_precision,
        gradient_accumulation_steps=int(config["gradient_accumulation_steps"])
    )
    lr = config["lr"]
    num_epochs = int(config["num_epochs"])
    seed = int(config["seed"])
    batch_size = int(config["batch_size"])

    set_seed(seed)
    train_dataloader, eval_dataloader = get_dataloaders(accelerator, batch_size)
    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", return_dict=True)
    optimizer = AdamW(params=model.parameters(), lr=lr)
    lr_scheduler = get_linear_schedule_with_warmup(
        optimizer=optimizer,
        num_warmup_steps=100,
        num_training_steps=(len(train_dataloader) * num_epochs),
    )
    model, optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
        model, optimizer, train_dataloader, eval_dataloader, lr_scheduler
    )
    import time
    times = []
    for _ in range(num_epochs):
        model.train()
        for step, batch in enumerate(train_dataloader):
            # We could avoid this line since we set the accelerator with `device_placement=True`.
            with accelerator.accumulate(model):
                outputs = model(**batch)
                loss = outputs.loss
                t1 = time.time()
                accelerator.backward(loss)
                t2 = time.time()
                times.append(t2-t1)
                optimizer.step()
                lr_scheduler.step()
                optimizer.zero_grad()
    accelerator.print(f'Average time per step: {sum(times)/len(times)}\nSmallest: {min(times)}\nLargest: {max(times)}')

def main():
    parser = argparse.ArgumentParser(description="Simple example of training script.")
    parser.add_argument(
        "--mixed_precision",
        type=str,
        default=None,
        choices=["no", "fp16", "bf16", "fp8"],
        help="Whether to use mixed precision. Choose"
        "between fp16 and bf16 (bfloat16). Bf16 requires PyTorch >= 1.10."
        "and an Nvidia Ampere GPU.",
    )
    parser.add_argument("--cpu", action="store_true", help="If passed, will train on the CPU.")
    args = parser.parse_args()
    config = {"lr": 2e-5, "num_epochs": 1, "seed": 42, "batch_size": 8, "gradient_accumulation_steps":4}
    training_function(config, args)

if __name__ == "__main__":
    main()

tl;dr, I see the intended behavior of the more nodes, the faster it is. Times were done on two T4 GPUs.
And I did not notice a behavior adding torch.cuda.synchronize() before time.time in terms of speed

authors: Roysky muellerzr sgugger

repo: huggingface/accelerate


text:
How do we use the detailed Parameters for the api?

there are detailed parameters: https://huggingface.co/docs/api-inference/detailed_parameters
However there are no usage examples, or parameters etc, how do we add them in?
what i'm trying so far(not working):
const response = await fetch(
            "https://api-inference.huggingface.co/models/tiiuae/falcon-7b-instruct",
            {
                headers: {
                    "Content-Type": "application/json",
                    Authorization: `Bearer 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`
                },
                method: "POST",
                body: JSON.stringify({"inputs": data}),
                parameters: {
                    max_new_tokens: 250
                }
            }
        );

Put the parameters directly in the body.

authors: Narsil VatsaDev

repo: huggingface/api-inference-community


text:
Remove or increase the 5GB limit?

The dataset viewer shows statistics and provides filter + sort + search only for the first 5GB of each split. We are also unable to provide the exact number of rows for bigger splits.
Note that we "show" all the rows for parquet-native datasets (i.e., we can access the rows randomly, i.e., we have pagination).
Should we provide a way to increase or remove this limit?

Please do this. If not possible, at least for the special datasets on request. For example now, special datasets can have a Python file loader with a data viewer at the same time. This can also be a process for some.

only for Parquet-native datasets maybe?

That works.

only for Parquet-native datasets maybe?

I improved the description, because we already allow to "view" all the rows for parquet-native datasets. What we miss is the rest: stats, filter, sort, search, SQL queries in general, because they are run on the 5GB DuckDB export.

only for Parquet-native datasets maybe?

I improved the description, because we already allow to "view" all the rows for parquet-native datasets. What we miss is the rest: stats, filter, sort, search, SQL queries in general, because they are run on the 5GB DuckDB export.

Can we open this issue again? HF already allow to "view" all the rows for parquet-native datasets. But the stats are wrong as they are run on the 5GB DuckDB export.

We already have an (internal) issue to fix the display (show in the UI if the stats are partial).
I don't think we plan to ever compute the stats on the complete data instead of the first 5GB. wdyt @huggingface/dataset-viewer?

computing on full datasets is too expensive, we should focus on making the UI clearer IMO

authors: julien-c kargaranamir lhoestq severo

repo: huggingface/dataset-viewer


text:
Dataloader getitem not working for DreamboothDatasets

Describe the bug
Dataloader getitem is not working as before (see example of DreamboothDatasets)
moving to 2.8.0 solved the issue.
Steps to reproduce the bug
1- using DreamBoothDataset to load some images
2- error after loading when trying to visualise the images
Expected behavior
I was expecting a numpy array of the image
Environment info

Platform: Linux-5.10.147+-x86_64-with-glibc2.29
Python version: 3.8.10
PyArrow version: 9.0.0
Pandas version: 1.3.5

Hi!

(see example of DreamboothDatasets)

Could you please provide a link to it?  If you are referring to the example in the diffusers repo, your issue is unrelated to datasets as that example uses Dataset from PyTorch to load data.

authors: mariosasko salahiguiliz

repo: huggingface/datasets


text:
Thoughts: streaming for conversational

I am implementing a Hugging Face Inference API wrapper for LlamaIndex (repo). I am subclassing its LLM object, and need to implement methods like chat and stream_chat: https://github.com/run-llama/llama_index/blob/v0.8.42/llama_index/llms/base.py#L363
It seems like InferenceClient.conversational is the function to use in chat, but I can't use it in stream_chat.
Any thoughts on exposing stream as an argument in conversational?  Does it make sense to have a streaming endpoint for conversational?

Hey @jamesbraza, InferenceClient.conversational is designed to support the conversational pipeline from transformers (see https://huggingface.co/tasks/conversational). This is not to use with LLMs hosted with TGI for example. As far as I understand, for a chat-based task you will have to encode the messages into a prompt before sending the request with InferenceClient.text_generation. Then you will have to decode the generated message for you. This can be done using chat-templates and especially tokenizer.apply_chat_template. AFAIK, chat templates are not meant to be directly applied by TGI.
cc @Rocketknight1 who worked on chat templates (https://github.com/huggingface/transformers/pull/25323) (cc @LysandreJik as well).

I don't think chat templates are necessary to decode the output! Chat templates encode the input conversation, but generally the output message is just generated with a standard generate() method. cc @gante - do we have any clean way to stream outputs from generate?

Yes sorry my mistake! So @jamesbraza you would have to use tokenizer.apply_chat_template to encode the message and then InferenceClient.text_generation(.., stream=True) to stream tokens generated by the server!

@gante - do we have any clean way to stream outputs from generate?

We are talking about InferenceClient + TGI integration here so streaming is already handled @Rocketknight1 , thanks!

InferenceClient.conversational is designed to support the conversational pipeline from transformers (see huggingface.co/tasks/conversational). This is not to use with LLMs hosted with TGI for example.

Of the tasks in https://huggingface.co/tasks/, which are supported with TGI?  It's not clear to me from looking at their pages.
Also, how would text_generation support chat history (it seems to just be one-shot)?  Is that what you were talking about with tokenizer.apply_chat_template?

Of the tasks in https://huggingface.co/tasks/, which are supported with TGI? It's not clear to me from looking at their pages.

TGI (text-generation-inference) is only meant for the text-generation task.

Also, how would text_generation support chat history (it seems to just be one-shot)? Is that what you were talking about with tokenizer.apply_chat_template?

Chat-templating is the process to turn all the chat history into a single prompt. So yes for each "new message" from the user you need to encode all the history using tokenizer.apply_chat_template, and send to text inference to generate the "response".

Alright thanks for answering.  So my takeaways so far are, starting with backends:

Text Generation Inference (TGI) backend: supports streaming
transformers backend: doesn't support streaming

Then for the tasks:

Text generation task can be backed by TGI, thus it can support streaming
Conversational task is only backed by transformers, thus it does not support streaming

Is there some way the conversational task can support streaming?  Or is that just totally unsupported

@jamesbraza Yes, that's a good summary! About your last question, it is very likely that the conversational task in transformers will never support streaming as it is not designed in that way (compared to TGI that is inference-focused).
It is good for you to close this issue then?

Yep, all good, cheers!

authors: Rocketknight1 Wauplin jamesbraza


text:
Add support for Grammar/Tools + TGI-based specs in InferenceClient

Sorry for the huge PR :grimacing: It goes hand-in-hand with https://github.com/huggingface/huggingface.js/pull/629 (and to a lesser extent https://github.com/huggingface/text-generation-inference/pull/1798). Most of the changes to review are hopefully documentation / auto-generated stuff.
What's in this PR?

Inference types for text_generation and chat_completion have been updated based on TGI specs (see associated PR https://github.com/huggingface/huggingface.js/pull/629)

💔 there are breaking changes for people importing the return types. Otherwise, there shouldn't be  breaking changes in the usage of return values.

Add support for grammar in text_generation task
Add support for tools in chat_completion task
Add support more parameters to those 2 tasks. All parameters supported by TGI are not handled.
When a model is not TGI-served, we now parse the error message to check which parameters are not supported. This makes it more robust instead of manually maintaining a list of incompatible parameters. See MODEL_KWARGS_NOT_USED_REGEX.
In text_generation, only send non-None parameters in payload.
Added a script check_inference_input_params.py that checks task parameters are correctly used and documented, to be consistent with the generated types. For now, raises an error when not consistent but doesn't provide an auto-fix (see TODOs in script).

for now, only for chat_completion/text_generation + do not check everything

Adapted generate_async_inference_client.py to generate KO package reference correctly.
Fixed a bunch of tests, mainly related to new types names

What to review
Mostly:

src/huggingface_hub/inference/_client.py => lots of docs update (not interesting) + few tweaks in code (to review)
src/huggingface_hub/inference/_common.py => only small tweaks
tests/test_inference_client.py => to check "it works"
tests/test_inference_text_generation.py  => to check "it works"
utils/check_inference_input_params.py => new script to check input parameters

The rest is either auto-generated stuff or low-level scripts.
Generated docs:

inference types
chat_completion
text_generation

What's not this PR?

https://github.com/huggingface/huggingface_hub/issues/2225 is not fixed
logic to switch between TGI/transformers backend is still clunky

=> will be handled in a follow-up PR

Thanks for the review @LysandreJik! Agree with you about "I'm of the opinion of ship it and eventually fix if issues arise," :smile:


text:
Remove sharded_ddp

What does this PR do?

sharded_ddp is not used in Transformers anymore (see https://github.com/huggingface/transformers/pull/25702) and was never used on Gaudi anyway (since it's linked to Fairscale). Removing all occurrences.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

authors: HuggingFaceDocBuilderDev regisss

repo: huggingface/optimum-habana


text:
Support specific device id for ORTModel

What does this PR do?
Support to pass a specific device id (e.g. as an int) for ORTModel, as transformers does for pipelines.
For example, model.to(1) will use the GPU with id 1.
Fixes #426
Before submitting

[x] Did you make sure to update the documentation with your changes?
[x] Did you write any new necessary tests? ==> yes but disabled by default

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

I disabled the dummy image classification benchmark parity test where the expected accuracy was hardcoded, that was failing due to an issue in datasets https://github.com/huggingface/datasets/issues/5145
Their is also the parity test for token classification failing, due to an issue with downloading conll2003 from deepai.org , whose data.deepai.org SSL certificate expired today. It's unrelated so I will merge anyway.

authors: HuggingFaceDocBuilderDev fxmarty

repo: huggingface/optimum


text:
[Stable Diffusion] Wrap the UNet under a for-loop in the ONNX graph

What does this PR do?
This a ONNX graph transformation that wraps the Unet ONNX graph in a for-loop to perform the iteration steps during generation directly in the ONNX graph.
Sample code:

First export the Unet model:

optimum-cli export onnx --model hf-internal-testing/tiny-stable-diffusion-torch --task stable-diffusion stable_diffusion

The tiny, non-representative model,  hf-internal-testing/tiny-stable-diffusion-torch was used to iterate fast during development.

Then perform the transformation:

import numpy as np
import onnx
from onnxruntime import InferenceSession
from optimum.onnx.graph_transformations import embedd_loop_in_unet

path_unet = "stable_diffusion/unet/model.onnx"
path_unet_with_loop = "unet_with_loop.onnx"

unet_model = onnx.load(path_unet)
unet_with_loop = embedd_loop_in_unet(unet_model)
onnx.save(unet_with_loop, path_unet_with_loop)

unet_sess = InferenceSession(path_unet, providers=["CPUExecutionProvider"])
unet_with_loop_sess = InferenceSession("unet_with_loop.onnx", providers=["CPUExecutionProvider"])

batch_size = 16
initial_sample = np.random.uniform(size=(batch_size, 4, 32, 32)).astype(np.float32)
hidden_states = np.random.uniform(size=(batch_size, 25, 32)).astype(np.float32)

def outside_loop(iterations):
    sample = initial_sample
    for i in range(iterations):
        timestep = np.array([i] * batch_size)
        out  = unet_sess.run(
            ["out_sample"],
            {
                "sample": sample,
                "timestep": timestep,
                "encoder_hidden_states": hidden_states
            }
        )
        sample = out[0]
    return sample

def embedded_loop(iterations):
    num_iterations = np.array(iterations)
    outputs = unet_with_loop_sess.run(
        ["sample"],
        {
            "num_iterations": num_iterations,
            "initial_sample": initial_sample,
            "encoder_hidden_states": hidden_states
        }
    )
    outputs = outputs[0]
    return outputs

outputs = outside_loop(10)
outputs_with_loop = embedded_loop(10)

print(np.max(np.abs(outputs - outputs_with_loop)))

TODO

[ ] Outputs are different with CUDAExecutionProvider
[ ] Lots of removed initializers for the transformed graph by ONNX Runtime, why??
[ ] Benchmark => On CPU and GPU with the tiny model, no speedup, need to try on real size models.
[ ] Support for schedulers: currently the transformed model iterates linearly on the steps, but the timesteps are usually computed using a scheduler, this needs to be handled.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


text:
This regex is confusing

https://github.com/huggingface/peft/blob/c9df262d69b41d36c7e199219605420be03dfbca/src/peft/tuners/tuners_utils.py#L502
It works now, but seems has a wrong format.
r".*\.{pattern}\.(\d+)\..*".format(pattern=pattern) this is a better format, if only layer_index is needed r".*\.{pattern}\.(\d+)".format(pattern=pattern) is enough.
module name examples:
...
model.layers.0.post_attention_layernorm
model.layers.1  # first expression will escape this
model.layers.1.self_attn
model.layers.1.self_attn.q_proj
model.layers.1.self_attn.k_proj
...

Do you mean specifically that the 2nd . is not escaped and the * at the end? I agree that those looks suspicious. We have an extensive test suite here:
https://github.com/huggingface/peft/blob/main/tests/test_tuners_utils.py
Some test cases have dubious results (see the comments). If you want to, feel free to create a PR with a better regex and we can check how the tests are affected.

ok, i've made a PR, and  this test case should be false?  https://github.com/huggingface/peft/blob/0f1e9091cc975eb5458cc163bf1843a34fb42b76/tests/test_tuners_utils.py#L65

Fixed via #1263

authors: BenjaminBossan s-k-yx

repo: huggingface/peft


text:
Error using sampling_strategy="unique"

Python version: 3.11.4
Dependencies installed:
setfit==1.0.1
torch==2.1.2
torchvision==0.16.2

I am using example code from the README file, the only thing sampling_strategy is set to unique:
from datasets import load_dataset
from setfit import SetFitModel, Trainer, TrainingArguments, sample_dataset

# Load a dataset from the Hugging Face Hub
dataset = load_dataset("sst2")

# Simulate the few-shot regime by sampling 8 examples per class
train_dataset = sample_dataset(dataset["train"], label_column="label", num_samples=8)
eval_dataset = dataset["validation"].select(range(100))
test_dataset = dataset["validation"].select(range(100, len(dataset["validation"])))

# Load a SetFit model from Hub
model = SetFitModel.from_pretrained(
    "sentence-transformers/paraphrase-mpnet-base-v2",
    labels=["negative", "positive"],
)

args = TrainingArguments(
    batch_size=16,
    num_epochs=1,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    sampling_strategy="unique",
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    metric="accuracy",
    column_mapping={
        "sentence": "text",
        "label": "label",
    },  # Map dataset columns to text/label expected by trainer
)

# Train and evaluate
trainer.train()
metrics = trainer.evaluate(test_dataset)
print(metrics)

This code raises the following exception:
Traceback (most recent call last):
  File "/home/ubuntu/example.py", line 40, in <module>
    trainer.train()
  File "/opt/conda/envs/setfit/lib/python3.11/site-packages/setfit/trainer.py", line 410, in train
    self.train_embeddings(*full_parameters, args=args)
  File "/opt/conda/envs/setfit/lib/python3.11/site-packages/setfit/trainer.py", line 443, in train_embeddings
    train_dataloader, loss_func, batch_size = self.get_dataloader(
                                              ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/setfit/lib/python3.11/site-packages/setfit/trainer.py", line 513, in get_dataloader
    dataloader = DataLoader(data_sampler, batch_size=batch_size, shuffle=shuffle_sampler, drop_last=False)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/setfit/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 307, in __init__
    raise ValueError(
ValueError: DataLoader with IterableDataset: expected unspecified shuffle option, but got shuffle=True

In File "/opt/conda/envs/setfit/lib/python3.11/site-packages/setfit/trainer.py", lines 511-513 I see that shuffle is always True for sampling_strategy="unique":
shuffle_sampler = True if args.sampling_strategy == "unique" else False
batch_size = min(args.embedding_batch_size, len(data_sampler))
dataloader = DataLoader(data_sampler, batch_size=batch_size, shuffle=shuffle_sampler, drop_last=False)

At the same time in 'File "/opt/conda/envs/setfit/lib/python3.11/site-packages/torch/utils/data/dataloader.py", lines 302-308' I see that shuffle can be True only for datasets inherited from IterDataPipe:
if isinstance(dataset, IterDataPipe):
    if shuffle is not None:
        dataset = torch.utils.data.graph_settings.apply_shuffle_settings(dataset, shuffle=shuffle)
# We cannot check `shuffle is not None` here, since previously `shuffle=False` was the default.
elif shuffle not in {False, None}:
    raise ValueError(
        f"DataLoader with IterableDataset: expected unspecified shuffle option, but got shuffle={shuffle}")

But setfit uses ContrastiveDataset which is not inherited from IterDataPipe:
from torch.utils.data import IterDataPipe
from setfit.sampler import ContrastiveDataset

isinstance(ContrastiveDataset([], True), IterDataPipe)
>>> False

Hello!
Thanks a bunch for the detailed issue! I'll remove the shuffle_sampler = ... lines, which should solve this problem.

Tom Aarsen

authors: tomaarsen traumgedanken

repo: huggingface/setfit


text:
microsoft/Phi-3-mini-128k-instruct  text_generation_launcher: Could not import Flash Attention enabled models: System cpu doesn't support flash/paged attention

System Info
text-generation-inference: 2.3.0
OS: Ubuntu 24.04
CPU: Xeon E5-1650 v4
Information

[X] Docker
[ ] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction
docker run --gpus all --shm-size 1g -p 8080:80 -v $VOLUME:/data ghcr.io/huggingface/text-generation-inference:2.3.0 --model-id microsoft/Phi-3-mini-128k-instruct
Expected behavior
Not an error about the CPU not supporting flash attention. I have no idea what that's meant to mean.

This was a red herring, the real issue seems to be that switching off the HF Transfer resulted in the correct data being appended to the safetensors, which then broke them. I found the earliest exception I could, rather than the header error at the end.

repo: huggingface/text-generation-inference


text:
Add support for Moonshine ASR

This PR adds support for Moonshine, a family of speech-to-text models optimized for fast and accurate automatic speech recognition (ASR) on resource-constrained devices. They are well-suited to real-time, on-device applications like live transcription and voice command recognition, and will be perfect for in-browser usage. This PR is using a dev branch of transformers by @eustlb (https://github.com/huggingface/transformers/pull/34784), and a dev branch of Optimum for ONNX conversion.
Example usage:
With pipeline API:
import { pipeline } from "@huggingface/transformers";

const transcriber = await pipeline("automatic-speech-recognition", "onnx-community/moonshine-tiny-ONNX");
const output = await transcriber("https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav");
console.log(output);
// { text: 'And so my fellow Americans ask not what your country can do for you as what you can do for your country.' }

Without pipeline API:
import { MoonshineForConditionalGeneration, AutoProcessor, read_audio } from "@huggingface/transformers";

// Load model and processor
const model_id = "onnx-community/moonshine-tiny-ONNX";
const model = await MoonshineForConditionalGeneration.from_pretrained(model_id, {
    dtype: "q4",
});
const processor = await AutoProcessor.from_pretrained(model_id);

// Load audio and prepare inputs
const audio = await read_audio("https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav", 16000);
const inputs = await processor(audio);

// Generate outputs
const outputs = await model.generate({ ...inputs, max_new_tokens: 100 });

// Decode outputs
const decoded = processor.batch_decode(outputs, { skip_special_tokens: true });
console.log(decoded[0]);
// And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

closes #990

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Model works with WebGPU too, and I've adapted this real-time demo to work with model. Significantly faster than the whisper version. 🔥

authors: HuggingFaceDocBuilderDev xenova


text:
Can't figure out how to use DelayAgent

I'm in a situation where I have an event occur, and I want to process that event but only 1 day after the event occurs.  So, DelayAgent seems like it would be perfect for this.  I would have DelayAgent observe the initial event, then wait a day before forwarding the event to another handler.
However, after creating a DelayAgent I don't understand how to specify the delay?  I don't see a field for this.

After reading through the specs, it seems that this agent should re-emit all delayed events at a regularly scheduled interval(default is 12 hours).
I believe what you're looking for is to re-emit the event X hours after the event was received. I don't believe this result can be achieved with this agent alone, at this time.

Oh ok that makes sense.  I was reading the "Schedule" parameter to be when the delay agent ran, not when it fired the events again.  I will try setting it to 12 hours and see if that works.  Thanks!

You could also look at using a DelayAgent + a SchedulerAgent, if you need very fine-grained controls.  To emit an event after a specific amount of time relative to when it was created is a new concept, I think, and might require a JavaScriptAgent + a SchedulerAgent, or a modification and pull request for the DelayAgent.

There's also this need, need to emit an event ASAP, after 24h relative to when it was created.
For reference, I used 2 delay agents in series, the schedule is both 1d, which will ensure the final event is emitted, between 1 and 2 days later, relative to when it was created.
It's inaccurate and not elegant, but better than nothing...

The issue with delaying events for some time is not solved at the moment, right?
Could you point the way, what to change in DelayAgent to reach this? I am new to Huginn, but I think this is very useful functionality.

@Aracon Do you want to delay events individually for a given time? I think you could introduce a mode to the DelayAgent which switches the behavior. If it is in the "delay_by_fixed_duration" mode iterate over the received events in check and only emit them if they are older then the configured duration.

authors: Aracon brianpetro cantino dsander svermeulen waldens

repo: huginn/huginn


text:
Message shows source name not

Hello all,
I'm doing my first steps with Huginn and despair.
I have an HTTP agent that monitors a specific IP address to see if it is still alive.
This also works so far and I can trigger a trigger agent via HTTP agent, which in turn triggers the notification via pushover.
Now I would like to get the name of the HTTP agent in the message of the trigger agent...
But unfortunately this does not work and the output remains empty...
{
"expected_receive_period_in_days": "2",
"keep_event": "false",
"rules": [
{
}, "type": "regex",
}, "value": "false",
"path": "response_received"
}
],
{ "message": "IP: {{url}} offline. Name: {{sources}}"
}
Anybody have an idea???

{{ sources }} is not an included field from the HTTP status agent. Try the special object key {{ agent.name }} instead, it should give you the name of the upstream agent which created the event.

Thank you. This solution works well.

authors: TsAbsinthe Unending


text:
Email optional tweak & Site Description fix

Hi,
I added logic to make the email optional and also a fix to site description. Should be none breaking.

Hi @humboldtux,
I'll do as you say. I think I understand what has happened here.
The documents are out of date... They say "All Params are only accessible using all lowercase characters.". Here:
https://gohugo.io/templates/variables/
In version 0.18, these page parameters should be accessible as case insensitive. But there are some bugs, which means this is not working correctly at the moment, "Description" (capitalised) would not work for me. Here: https://github.com/spf13/hugo/issues/2907 (and others).
So. There is a work around which uses "with" this works for all cases. I'll push that later. It has an advantage, in that it works for cases where description is "", Description and description.

Thanks for the clarification. If you can squash all the commits regarding each fix (disqus and description) so that there are only 2 commits, that would be cool, i would like to keep commits atomic and focused.
Thank you for your time and work.

No problem. Will seperate shortly.

repo: humboldtux/startbootstrap-clean-blog


text:
Why hidden files like .env excluded from box if it specified via finder section?

Hi there! Thanks a lot for such useful composer package!
Question:
Why hidden files like .env excluded from box if it specified via finder section?
    "finder": [
        {
            "in": ".",
            "name": ".env"
        },
    ],

The purpose is to optionally include .env into box if it defined for applications that can be executed with and without .env file

I suppose because behind it is using the Symfony Finder component and is configured to ignore dot files and directories.
I think it's a case where using files or directories is more appropriate

Just have checked additional finder options - yes, you are correct!
Thanks again!

authors: hanovruslan theofidry

repo: humbug/box


text:
Limit humbug_get_contents dependency to 1.0.4

Temporary limitation of dependency version until we handle the deprecation of global namespaced functions in the near future. On merge, the branch will be tagged for release.

Travis failure is expected (PHP 5.3.3 error over openssl support).

repo: humbug/phar-updater


text:
Group search & refactoring

Serverside changes:

Extract the presenters in serverside SearchController into methods which are shared between instant and full searches
Fix sorting (by sorting on the pg_search_rank column in Rails)
Rank everything onto a common scale for all searchable items (0..1 via tsearch normalization mode 32 + 8 + 2)
Add a nasty raw-sql hack to rank user results and maintain email searching without exposing too much info about email

If we're willing to drop email search (does anyone actually use it?) we can just replace this with just a pg_search on name

Rename fuzzy_search_by_title and simple_search_by_title to instant_search and full_search, respectively
Modify the full_search algorithm to just include trigram in the rankings initially
Add searching for Groups
Updated all indices to match the new search queries

Super Sekrit Internal Unsupported API™ Changes:

Change type parameter to separate depth and scope parameters.

Maintain type=full since it's 3 lines of code and it keeps Tenpenchii and Deadman working for now.  I know we don't officially support third party apps using the internal API, but users blame us anyways if they break so I'd rather not.  Plus Tenpenchii is off Google Play right now so updating might be slower (idk how this stuff works tbh)

Clientside Changes

Switch to the modified search endpoints and results
Add groups to dropdown
Move some of the stuff from the autocomplete to use {{#link-to}}
Use transitionToRoute to get to the /search page if you hit enter in the search box
Replace var self = this with () => {} functions where it was easy.

The clientside changes should reduce reboots of the app (searching no longer requires it to reboot) and the search changes should greatly improve result relevance.
I did some rough benchmarks on the search methods too, in development:
Cold Cache: 40ms instant, 600ms full
Hot Cache: 1.1ms instant, 1.5ms full

I just want to note that the results in some cases are somewhat... wrong (I think it might be related to the 8 or 2 normalizations).  So I'd like a dump of the anime, manga, and groups tables (cc @vikhyat) if possible, to tweak the ranking algorithm against :dango:

We should have a client-side SearchResult model or something, currently logic for executing search is distributed all over our routes. It doesn't need to inherit from DS.Model, Ember.Object would also be fine if we don't need ember-data functionality.

I actually mentioned that in the devchat, that we should create some central object to handle the search coordination.  I was leaning towards a service object which would wrap the bloodhound crap and expose the API as simple as setting options for depth/scope and then passing strings in to get a response.
I also was thinking about better options for the serialization/deserialization stuff, like AMS etc. but didn't find any good leads that direction.
Either way, I don't want the scope on this to creep too much, so I say we put those off.  Just calibrate the results on this and call it done.

Cool, let's put it on the todo list then.

authors: NuckChorris vikhyat

repo: hummingbird-me/hummingbird