text:
[BUG]: RuntimeError: rank<size. 0 vs 0

🐛 Describe the bug
I'm running ./applications/ChatGPT/examples/train_dummy.py but changed backend from "NCCL" to "GLOO"( in ColossalAI initialize.py) and set a bunch of environment variables
import os
os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"
os.environ["RANK"] = '0'
os.environ["LOCAL_RANK"] = '0'
os.environ["WORLD_SIZE"] = '0'
os.environ["MASTER_ADDR"] = '127.0.0.1'
os.environ["MASTER_PORT"] = '80'

I run this script with --strategy ddp to see if it could run on a 4 GPU machine. But the following error came up.
C:\Users\abcd\Desktop\ColossalAI\applications\ChatGPT\examples>python train_dummy.py --strategy colossalai_zero2
2023-02-24 09:53:34.324233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\library.py:130: UserWarning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::index.Tensor(Tensor self, Tensor?[] indices) -> Tensor
    registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\build\aten\src\ATen\RegisterSchema.cpp:6
  dispatch key: Meta
  previous kernel: registered at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\functorch\BatchRulesScatterOps.cpp:1053
       new kernel: registered at /dev/null:219 (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\core\dispatch\OperatorEntry.cpp:156.)
  self.m.impl(name, dispatch_key, fn)
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [WIN-MEUAL5THNML]:80 (system error: 10049 - 在其上下文中，该请求的地址无效。).
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\socket.cpp:601] [c10d] The client socket has failed to connect to [WIN-MEUAL5THNML]:80 (system error: 10049 - 在其上下文中，该请求的地址无效。).
Traceback (most recent call last):
  File "train_dummy.py", line 125, in <module>
    main(args)
  File "train_dummy.py", line 38, in main
    strategy = ColossalAIStrategy(stage=2, placement_policy='cuda')
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\chatgpt\trainer\strategies\colossalai.py", line 77, in __init__
    super().__init__(seed)
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\chatgpt\trainer\strategies\ddp.py", line 25, in __init__
    super().__init__()
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\chatgpt\trainer\strategies\base.py", line 23, in __init__
    self.setup_distributed()
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\chatgpt\trainer\strategies\colossalai.py", line 110, in setup_distributed
    colossalai.launch_from_torch({}, seed=self.seed)
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\colossalai-0.2.5-py3.7.egg\colossalai\initialize.py", line 227, in launch_from_torch
    verbose=verbose)
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\colossalai-0.2.5-py3.7.egg\colossalai\initialize.py", line 99, in launch
    gpc.init_global_dist(rank, world_size, backend, host, port)
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\colossalai-0.2.5-py3.7.egg\colossalai\context\parallel_context.py", line 374, in init_global_dist
    dist.init_process_group(rank=rank, world_size=world_size, backend=backend, init_method=init_method)
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\distributed\distributed_c10d.py", line 769, in init_process_group
    timeout=timeout,
  File "C:\Users\abcd\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\distributed\distributed_c10d.py", line 862, in _new_process_group_helper
    pg = ProcessGroupGloo(prefix_store, group_rank, group_size, timeout=timeout)
RuntimeError: [enforce fail at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\third_party\gloo\gloo\context.cc:27] rank < size. 0 vs 0

How can I fix this?
Environment
Windows
Pytorch1.13.1+cu117
Python37
gloo backend
multigpu, CUDA 11.6

I also want to know what master address/port should I use to avoid system error: 10049

Hi, these environment variables should be set carefully. We generally recommend that users can launch by torchrun

Hi @nameless0704 Please check the installation instructions, we are not support Windows currently.
https://github.com/hpcaitech/ColossalAI#Installation
The Chat part also has been updated a lot.
https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat
This issue was closed due to inactivity. Thanks.


text:
Cant install on kubernetes v1.22.3

Hi,
One of the CustomResreouces is using a deprecated kind.
https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122
Error: INSTALLATION FAILED: failed to install CRD crds/hpe-replicated-device-info-crd.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

Hey @v0l this is a know issue with the Helm chart at the moment. It will be fixed in the next version of the upstream chart. We expect it to drop towards end of January. There might be a beta chart before that. In the meantime, deploy using the YAML manifest method.

Thanks @datamattsson I installed directly from the manifest, I have volumes created in TrueNAS but it cant seem to mount them over ISCSI, how does mount these if the portal doesnt have any targets/luns?

I switched to csi-driver edge and that seems to get me a few steps further, i have the targets set on TrueNAS now but mount claims they are not found:
MountVolume.MountDevice failed for volume "pvc-98909345-bcb3-4300-ba4a-e8f68743da2a" : rpc error: code = Internal desc = Failed to stage volume BLK_hpe-csi_pvc-98909345-bcb3-4300-ba4a-e8f68743da2a, err: rpc error: code = Internal desc = Error creating device for volume BLK_hpe-csi_pvc-98909345-bcb3-4300-ba4a-e8f68743da2a, err: device not found with serial 6589cfc00000098cb404eaaaa4195ac3 or target

system logs show this repeatedly on the k8s node:
Nov 23 22:42:56 k8-02 kernel: [39991.966089] sd 7:0:0:0: [sdb] Synchronizing SCSI cache
Nov 23 22:42:56 k8-02 kernel: [39992.045023] scsi 7:0:0:0: alua: Detached
Nov 23 22:42:56 k8-02 kernel: [39992.046190] scsi 7:0:0:0: Direct-Access     TrueNAS  iSCSI Disk       0123 PQ: 0 ANSI: 7
Nov 23 22:42:56 k8-02 kernel: [39992.047022] scsi 7:0:0:0: alua: supports implicit TPGS
Nov 23 22:42:56 k8-02 kernel: [39992.047025] scsi 7:0:0:0: alua: device naa.6589cfc00000098cb404eaaaa4195ac3 port group 1 rel port 4
Nov 23 22:42:56 k8-02 kernel: [39992.047172] sd 7:0:0:0: Attached scsi generic sg2 type 0
Nov 23 22:42:56 k8-02 kernel: [39992.048529] sd 7:0:0:0: [sdb] 1048576000 512-byte logical blocks: (537 GB/500 GiB)
Nov 23 22:42:56 k8-02 kernel: [39992.048530] sd 7:0:0:0: [sdb] 8192-byte physical blocks
Nov 23 22:42:56 k8-02 kernel: [39992.048679] sd 7:0:0:0: [sdb] Write Protect is off
Nov 23 22:42:56 k8-02 kernel: [39992.048681] sd 7:0:0:0: [sdb] Mode Sense: 7f 00 10 08
Nov 23 22:42:56 k8-02 kernel: [39992.048991] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Nov 23 22:42:56 k8-02 kernel: [39992.049455] sd 7:0:0:0: [sdb] Optimal transfer size 1048576 bytes
Nov 23 22:42:56 k8-02 kernel: [39992.065424] sd 7:0:0:0: alua: transition timeout set to 60 seconds
Nov 23 22:42:56 k8-02 kernel: [39992.065426] sd 7:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
Nov 23 22:42:56 k8-02 kernel: [39992.087376] sd 7:0:0:0: [sdb] Attached SCSI disk

Hey! Thanks for digging in here. What you're seeing here is that the CSI driver can't discover the disk on the host. It only supports multipath devices today, even if you only have a single IP interface. Do you have any idea what the worker node OS is? You might be missing a couple of mandatory stanzas in your /etc/multipath.conf or the multipathing service isn't running properly.

I couldnt see a problem with my multipath setup, it seems to mount the disk fine on the OS but its not mounted by kubelet on the container.
OS: Ubuntu 20.04.03

There's an issue with Ubuntu with no fix in sight. The multipath.conf you have is provided by the stock package and not the HPE CSI Driver. Remove /etc/multipath.conf from the worker nodes and restart the driver:
kubectl rollout restart -n hpe-storage ds/hpe-csi-node

@datamattsson any news on that fresh version?

@ishioni yes! Version 2.1.0 of the HPE CSI Driver will be released next week. I'm hoping to have a fresh cut of the TrueNAS CSP built for this version the following weekend. The multipath issue mentioned in this issue has been addressed too and that will simplify deployment on stock Ubuntu.

Please give v2.1.0 a whirl and reopen this bug if problems persist.

source: gharchive/issue

authors: datamattsson ishioni v0l

repo: hpe-storage/truenas-csp

license: MIT

license_type: permissive

license_source: github-api


text:
update poetry.lock

いつもこのテンプレートに助けられています。ありがとうございます！
poetry.lock の依存パッケージが古いという警告が出るので、アップデートしたものを送らせていただきました 🙏

反応が遅くなり申し訳ないですが、大変助かりました、ありがとうございます！！！！

source: gharchive/pull-request

authors: hppRC kaisugi

repo: hppRC/bert-classification-tutorial


text:
获取穿梭框的所有数据？

现在好像没有提供一个可以获取所有fromdata/所有todata的函数，能否提供一个呢

现在只能在事件add、remove的参数中获取，感觉有些不方便，谢谢。


text:
Starting the server on Ubuntu 20.04 results in "Cannot bind to IP port: Address already in use"

Hi,
I created a dockerized version of your project for deployment to Amazon AWS. Unfortunately, I cannot get the server to run. I followed the exact instructions in the README and the port is definitively not in use (in fact, changing the port number to a random value always yields the same result), so I suspect that the implementation is broken in some way.
As the error message is wrong (port is free, verified by netstat) I cannot see any way to resolve this issue myself. Could you consider reviewing your code and maybe improving the error handling at this point to further track down the root cause?
You can find the docker image repository here and a deployed version in the linked DockerHub (See README).
https://github.com/oxytu/hpsjam-docker
Best Regards

I think this commit will fix the issue:
https://github.com/hselasky/hpsjam/commit/0d207a47efa57c05201d1306c23d98dba62a3a49
--HPS

Hi @hselasky - thank you that looks good, I just updated the latest-Image. I could start the server now. Unfortunately, I cannot connect to it. Can I send you the URL and you give it a try? I would do so by private message if possible.

You can reach me at hps@selasky.org

Is fixed with @hselasky commit.

authors: hselasky oxytu

repo: hselasky/hpsjam


text:
搜索内容为空时，会出现多条文本提示

@ruiifly 谢谢提醒，已经改了

@ruiifly 上次没注意到，不能直接用html，不然会把所有的节点都删除了，即第一次未搜索到，然后第二次搜索到之后所有的节点都没了

然后第二次搜索到之后所有的节点都没了
Hahahaha

authors: hsiangleev ruiifly weathon

repo: hsiangleev/layuiExtend


text:
能否增加单个链接是否公开设置，目前只能分组 公开还是隐藏

能否增加单个链接是否公开设置，目前只能分组 公开还是隐藏

分组也没法设置吧，只能统一账号公开还是隐藏

暂时无计划，分组已经实现相关场景需求，做的过多会导致程序臃肿和易用性变得复杂

暂时无计划，分组已经实现相关场景需求，做的过多会导致程序臃肿和易用性变得复杂

authors: chaomarks chrisdong hslr-s

repo: hslr-s/sun-panel


text:
ihsrb not working

I successfully installed and launched the simulator, but I cannot use the usual python api (ihsrb). When I try, I get the a KeyError: u'/hsrb/omni_base_controller/base_coordinates'.
Could you please let me know if the python interface can be used (and therefore I have an issue on my side) or we should use another api?

Hi @lgarattoni ,
We are sorry but ihsrb is only for HSR community members.
By default, it doesn't include ihsrb in this OSS simulator.

Hi @lgarattoni ,
We are sorry but ihsrb is only for HSR community members.
By default, it doesn't include ihsrb in this OSS simulator.

I see, thank you @yuka-hashiguchi for your quick reply.

I see, thank you @yuka-hashiguchi for your quick reply.

authors: lgarattoni yuka-hashiguchi


text:
The package need updated dependencies

Steps to reproduce the issue, if applicable. Include the actual command and output and/or stack trace.
git clone https://github.com/http-party/http-server.git
cd http-server/
npm i

What did you expect to happen?
Expected npm to build
Paste the error and / or stack trace here:
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated common-style@3.1.0: common-style is now eslint-config-populist. Consider upgrading!
npm WARN deprecated jscs-preset-wikimedia@1.0.1: No longer maintained. We recomment migrating to ESLint with eslint-config-wikimedia.
npm WARN deprecated core-js@1.2.7: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated nomnom@1.8.1: Package no longer supported. Contact support@npmjs.com for more info.
npm ERR! Linux 4.19.104-microsoft-standard
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "i"
npm ERR! node v8.10.0
npm ERR! npm  v3.5.2
npm ERR! code EMISSINGARG

npm ERR! typeerror Error: Missing required argument #1
npm ERR! typeerror     at andLogAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:31:3)
npm ERR! typeerror     at fetchPackageMetadata (/usr/share/npm/lib/fetch-package-metadata.js:51:22)
npm ERR! typeerror     at resolveWithNewModule (/usr/share/npm/lib/install/deps.js:456:12)
npm ERR! typeerror     at /usr/share/npm/lib/install/deps.js:457:7
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror     at /usr/share/npm/lib/fetch-package-metadata.js:37:12
npm ERR! typeerror     at addRequestedAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:82:5)
npm ERR! typeerror     at returnAndAddMetadata (/usr/share/npm/lib/fetch-package-metadata.js:117:7)
npm ERR! typeerror     at pickVersionFromRegistryDocument (/usr/share/npm/lib/fetch-package-metadata.js:134:20)
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror This is an error with npm itself. Please report this error at:
npm ERR! typeerror     <http://github.com/npm/npm/issues>
npm ERR! Linux 4.19.104-microsoft-standard
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "i"
npm ERR! node v8.10.0
npm ERR! npm  v3.5.2
npm ERR! code EMISSINGARG

Tell us about your environment
Ubuntu 18.04.4 LTS on WSL2

exact http-server version:
v0.12.3
Node version:
v8.10.0
Platform:
Ubuntu wls2 on Windows 10 build 19624
Other information (related issues, suggestions for a fix, etc):
Update deprecated npm dependencies

This should be rechecked with v13.0.0. Also the second half of that output is a bug in npm itself

authors: saadanerdetbare thornjad

repo: http-party/http-server


text:
Support custom mime types (content-type)

Just pass the mimeTypes option to node-ecstatic.

👍 lgtm

Why has this not been merged? Such a simple change which solves so many problems. I'll be hand-copying these changes so I can use http-server to serve wasm modules.

Even with this patch I wasn't able to use wasm modules -- probably due to https://github.com/indexzero/http-server/issues/480 . As noted there, with ecstatic 4.0.0 wasm was served with the correct content-type out-of-the-box.

@nickolay #565 solves the wasm problem along with supporting mime types.

Closing in favor of #693, since we'll be able to support it directly

authors: 599316527 J-Rojas Smilebags nickolay thornjad


text:
Use v3 with Tide?

It seems that the removal of async-std in v3 has broken the use of async-session with tide. If you try a simple tide program like the one in the async-mongodb-session/examples but with async-sqlx-session v0.4.0 you get build errors like
the trait `SessionStore` is not implemented for `SqliteSessionStore`
Moving async-sqlx-session back to async-session v2.0.1 from v3.0.0 makes the build error go away. I think this is a bug in async-session not async-sqlx-session because moving async-mongodb-session to async-session v3.0.0 from v2.0.1 results in the same build errors but for MongodbSessionStore, and I just don't see anything else in the latest async-sqlx-session that seems like it could be the cause.
Thoughts? Anything more I can do to help troubleshoot?

Thanks, I thought it might be a roadmap thing when I saw the commits putting things behind an async-std feature. What is a better forum than here next time I see something like this that might just be WIP?

This is a totally reasonable place to file an issue. When I bumped async-session, I expected tide would be having more frequent releases, but there hasn't been a tide release in quite a while. Hopefully there will be one soon, and there is reason to expect that this may be the case.

Moving async-sqlx-session back to async-session v2.0.1 from v3.0.0 makes the build error go away
@jgreenbaum could you elaborate on how exactly to do this? I'm quite new to Tide and Rust in general and I've just run into this issue.

authors: jbr jgreenbaum xyzshantaram

repo: http-rs/async-session


text:
Expose Router Debug information

I'd say expose the router itself, but I suppose we may want the option to swap it out.
I think Debug-printing a server in that case should list all registered routes and middleware, if possible.

This is partly addressed in #802 — the router is Debug

repo: http-rs/tide


text:
Document behaviour of nested overlapping routes

I was surprised by this behavior, so I thought it useful to add it to the documentation

Rebased to version with clippy fixes

Thanks a lot for fixing my grammar ^^


text:
🛑 Manhuagui is down

In c684d0b, Manhuagui (https://www.manhuagui.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Manhuagui is back up in 300cabe after 7 minutes.


text:
🛑 Manhuagui is down

In 4232441, Manhuagui (https://www.manhuagui.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Manhuagui is back up in 1a70f47 after 13 minutes.


text:
Adaptive EntityDecoder[Json] for circe

After doing some benchmarks I've decided to make an instance that combines performance of decoding via ByteBuffer for small payloads and via jawn-fs2 for big payloads.
Default cutoff value of 100000 was chosen by doing benchmarks with different cutoff value. Mesages with Content-Length lower than cutoff are handled by ByteBuffer parser. Everything higher than cutoff or when Content-Length is unknown is handled by incremental decoder.
[info] Benchmark                          (approxContentLength)  (cutoff)  Mode  Cnt        Score       Error  Units
[info] CirceJsonBench.decode_adaptive                      1100    100000  avgt   20    28218.335 ±   147.533  ns/op
[info] CirceJsonBench.decode_adaptive                      5000    100000  avgt   20    76649.070 ±  1259.016  ns/op
[info] CirceJsonBench.decode_adaptive                     10000    100000  avgt   20   123584.282 ±   704.567  ns/op
[info] CirceJsonBench.decode_adaptive                    100000    100000  avgt   20  1013073.677 ± 24022.673  ns/op
[info] CirceJsonBench.decode_adaptive                    500000    100000  avgt   20  5211421.634 ± 68338.766  ns/op
[info] CirceJsonBench.decode_byte_buffer                   1100    100000  avgt   20    28265.613 ±    94.894  ns/op
[info] CirceJsonBench.decode_byte_buffer                   5000    100000  avgt   20    75760.460 ±   887.846  ns/op
[info] CirceJsonBench.decode_byte_buffer                  10000    100000  avgt   20   123096.025 ±   667.088  ns/op
[info] CirceJsonBench.decode_byte_buffer                 100000    100000  avgt   20  1088689.815 ± 11907.722  ns/op
[info] CirceJsonBench.decode_byte_buffer                 500000    100000  avgt   20  5534396.999 ± 42943.154  ns/op
[info] CirceJsonBench.decode_incremental                   1100    100000  avgt   20    55218.931 ±   736.221  ns/op
[info] CirceJsonBench.decode_incremental                   5000    100000  avgt   20    95511.884 ±  1571.320  ns/op
[info] CirceJsonBench.decode_incremental                  10000    100000  avgt   20   140359.831 ±  2344.243  ns/op
[info] CirceJsonBench.decode_incremental                 100000    100000  avgt   20  1008533.778 ± 18852.809  ns/op
[info] CirceJsonBench.decode_incremental                 500000    100000  avgt   20  5153683.990 ± 49192.662  ns/op

jsonOf refers to jsonDecoder, which means users of it are stuck with adaptive decoding at our default cutoff

how about making jsonDecoder abstract and specialize it to jsonDecoderAdaptive(cutoff = 100000) in package object (and CirceInstances object?)

Yeah, the abstract idea is similar to how we handled the default pretty printer.
Summoning the implicit EntityDecoder[Json] wouldn't work well with the jsonOf[A] syntax.

I'll do the backporting a bit later this week.

I've made jsonDecoder abstract

authors: rossabaker wedens


text:
add Monoid instance for UrlForm

The tests I added are pretty rudimentary. I'm open to suggestions or even just a pointer to another Monoid instance's tests that would be good to emulate.

What are the expected semantics when both have the same key? Overwrite or combine? I would expect combine, which I think your implementation would do. Just double checking intent.

I expected the keys to be combined as well.
There is an Arbitrary[UrlForm], and using cats-laws looks like it's revealed some bugs, so I'm working through those. Thanks for the suggestion.

I agree about reconsidering Seq later on.
I think it would be good to backport this. I can prepare another PR for that if it's helpful—it looks like the tests pass if I cherry-pick f2af07d onto release-0.18.x.

That sounds great, thanks.

Is there anything left here we won't get from merging #2095?

I’m not sure if the cats wildcard import in the test file would make it via #2095, but I think that’s the only difference.

Superseded by backport in #2095.

authors: bpholt rossabaker


text:
More detail in non-literal error msg in URI macro

I think that @rossabaker wants to put some work into Uri such that this
message may become irrelevant. However, I've seen enough people get
tripped up by literals being special-cased that I thought this small
change might be a nice interim solution.

Current coverage is 61.02% (diff: 0.00%)

Merging #709 into master will increase coverage by 0.42%

@@             master       #709   diff @@
==========================================
  Files           209        209
  Lines          4513       4513
  Methods        4208       4208
  Messages          0          0
  Branches        300        300
==========================================
+ Hits           2735       2754    +19
+ Misses         1778       1759    -19
  Partials          0          0

Powered by Codecov. Last update dbe426e...06bcb15

LGTM.  Could go into 0.14.x, in case there's another one.

👍 Also looks good to me.

cherry-picked onto 0.14.x.

authors: bryce-anderson ceedubs codecov-io rossabaker


text:
Fix typos

Found via codespell -L datas.

:partying_face: :partying_face: :partying_face: :partying_face: :partying_face:

repo: httpie/httpie


text:
Add Request::Body#source

This makes it possible to access the original body object, which httplog needs to log request bodies.
Related to #437.

This is actually not necessary, and isn't applicable (e.g. when using deflating Request#body will not be a Request::Body instance). I think a different approach would be better.

This is now ready for review.
I modified HTTP::Request to create a deflated body only when it's about to stream. The reason for that change is that httplog is currently retrieving the HTTP::Request#body, and I wanted it to be able to call #source to retrieve the original body object, but that wouldn't currently work with :auto_deflate, because HTTP::Request#body wouldn't return a HTTP::Request::Body object but rather the DeflatedBody wrapper.
I think this change is beneficial in general in terms of consistency, because with this PR HTTP::Request#body will always return a HTTP::Request::Body object, even when :auto_deflate is turned on.

Looks good to me. Any thoughts @ixti?

LGTM as well! Thanks!

authors: ixti janko-m tarcieri

repo: httprb/http

license: mit

license_source: bigquery


text:
hrp run 嵌套case，report生成失败

问题描述
hrp 执行嵌套用例，生成报告失败
hrp 是下载的二进制文件，V4.3.0
通过脚手架生成项目，命令如下：
hrp run -s -g testcases/ref_testcase.yml

版本信息

操作系统类型:
mac M1

Python 版本:
v3.10.9

HttpRunner 版本
v4.3.0

同样问题啊。.

@jiwzh @jonzha9527  这是由于引用case的时候，在 (*StepTestCaseWithOptionalArgs).Run 中 的差异行为

和 其他单个step 的操作记录不同

authors: jiwzh jonzha9527 ywanbing

repo: httprunner/httprunner


text:
可否支援繁体中文

在香港，台湾，澳门市场，都是用繁体中文。calendar.getLunar() 是否可回返繁体中文？谢谢。

可以的，我加上一份繁体配置

谢谢。

...这里其实有个建议, 最好是从应用层设置框架的语言 , 因为很多应用语言是自切换不随系统的.

...这里其实有个建议, 最好是从应用层设置框架的语言 , 因为很多应用语言是自切换不随系统的.

authors: huanghaibin-dev lhalcyon yccheok

repo: huanghaibin-dev/CalendarView


text:
scenario traffic specifies flow with invalid route arrival point

BUG REPORT
High Level Description
While performing an online training with multi-scenario-v0 env, TraCI error occurs. The code tries to recover. But the smarts/core/local_traffic_provider.py::_resolve_flow_pos() throws an error stating the scenario traffic specifies an invalid route.
The traffic routes in SMARTS/smarts/scenarios/intersection/1_to_2lane_left_turn_c and SMARTS/smarts/scenarios/intersection/1_to_2lane_left_turn_t are properly described and do not contain any lane index 1 for road id -E0.
SMARTS version
$ git checkout comp-1
The comp-1 branch is up-to-date with the latest develop branch.
Previous associated issues
None
Steps to reproduce the bug
$ cd <path>/SMARTS
$ git checkout comp-1
$ cd ./competition/track1/train
$ python3.8 -m venv ./.venv
$ source ./.venv/bin/activate
$ pip install --upgrade pip
$ pip install -e .
$ python3.8 run.py

Resulting and expected behaviour
Expected for simulation to recover seamlessly after a TraCI error, but it fails with an invalid route error.
Error logs and screenshots

System information

Operating system & Environment [Please note if using Docker]: Ubuntu 18.04
Python version [Please note if using Jupyter]: 3.8
SUMO version: 1.10
Date when bug occurred: 16 August 2022

Impact [If known]
Unable to perform online training, as training stops due to the above error.

@sah-huawei

repo: huawei-noah/SMARTS


text:
StreamDM-97

Summary of the changes
Function train of StreamKM needs to be combined with the assign method, since data are lost between the calls.  Function trainStreamKM updates the bucketManager with the incoming input, then runs kmeans.cluster and assigns the input to the nearest cluster's index. To call trainStreamKM  "ClusteringTrainEvaluate" needs some changes as well.
Classes that are affected
StreamKM, ClusteringTrainEvaluate
Tests
Run with command:
./spark.sh "ClusteringTrainEvaluate -c (StreamKM) -s (SocketTextStreamReader)" 1> result_iris_streamKM.txt 2> log_iris_streamKM.log

Hi @ioanna-ki ,
Many thanks for your contribution! I have tested its functionality and it is now assigning entries to clusters. There are just some small adjustments to make the code compliant with the API. Concretely, you shall have the implementation inside the assign method instead of creating a new one and this will allow you to rollback the changes made to class ClusteringTrainEvaluate.scala.
Best Regards,
Heitor

Hi @ioanna-ki ,
Thanks for applying the requested changes, let's now wait for the other members' reviews
Best Regards,
Heitor

authors: hmgomes ioanna-ki

repo: huawei-noah/streamDM


text:
Error reporting for yaml errors in hubblestack_data

Ref https://github.com/hubblestack/hubble/issues/180
YAML errors, such as the one caused by the \d inside of double quotes as shown below, are not being logged by the hubble loader. We need to log these errors or we can't know what's wrong with our profiles.
    sshd_idle_timeout:
      data:
        CentOS Linux-7:
        - /etc/ssh/sshd_config:
            match_output: "^ClientAliveInterval +[1-3]{0,1}\d{1,2}$"
            match_output_regex: True
            pattern: ^ClientAliveInterval
            tag: CIS-5.2.13

cc @whackyhack

Ha! I already did this and forgot about it...... https://github.com/hubblestack/hubble/pull/153/files

Fantastic!  Since they are Python, can I copy the 5 changed files into my RPM deployed servers? (Or, just one server for testing.)

It might work, though if you're using our RPMs then they're built with pyinstaller, and the pyinstaller magic could potentially break the normal modularity of python -- I'm not sure.
In any case, the only file you need is nova_loader.py. The others are unrelated to this fix.

authors: basepi whackyhack

repo: hubblestack/hubble


text:
why codebook_size=4096?

I'm wondering why SNAC uses codebook_size=4096
(https://github.com/hubertsiuzdak/snac/blob/5e5ebadd1dc1273648464ad2ce9020be95041e12/snac/snac.py#L22)
Doesn't it mean bitrates are higher than DAC which used codebook size of 1024?
(https://github.com/descriptinc/descript-audio-codec/blob/c7cfc5d2647e26471dc394f95846a0830e7bec34/dac/model/dac.py#L156)

bitrate is calculated as log2(codebook_size) * number_of_codebooks * frame_rate
The number of bits needed to store the codebook size grows very slowly, so even if DAC has 4x smaller codebook size, number of codebooks and frame rate are what really determine the bitrate in the end.

repo: hubertsiuzdak/snac


text:
Mccalluc/add portal visualization

The goal is to move vis-conf build to index-time so that problems are encountered earlier, rather than deferred until render-time. This is the start of alternative to:

https://github.com/hubmapconsortium/portal-ui/pull/2313

Vis config code is incorporated as a git submodule so that it can be shared between this this repo and portal-ui. As of

https://github.com/hubmapconsortium/portal-ui/pull/2386

the submodule is being used at render time... once the view conf is available in the index, the plan is to just keep the submodule in portal-ui around for developer use.
Filing as draft until everything works... but early feedback welcome.

Asking Zhou for help on Slack:

What is the right way to access the TypeClient at index-time? We are moving away from accessing it in the portal at render-time: In that context we have access to a TYPE_SERVICE_ENDPOINT from the Flask context… what’s the right way to do this in search-api?

From Joel:

I believe if you are running in the context of search-api you can probably instantiate libs.assay_type.AssayType(type_name) directly.

At the meeting this morning, concluded that this is not a viable solution. Closing.

repo: hubmapconsortium/search-api


text:
A mobilarénán a cikkek eleje meg van vágva

Helló!
Úgy tűnik valami gebasz van mert a reklámmal együtt a cikk/hír eleje is eltűnik.
Pl.:
https://mobilarena.hu/hir/nem_a_gamereknek_szol_a_xiaomi_play.html
(Gondolom esetleg a PH lapcsalád többi oldala is érintett lehet.)

Szia,
én most összenéztem 2 monitoron a küldött linket és nem látok különbséget. Küldenél egy screenshotot? Esetleg tudod hogy mi (melyik szűrőszabály) okozhatja a problémát?

@szpeter80 commented on 2019. aug. 4. 21:33 CEST:

Szia,
én most összenéztem 2 monitoron a küldött linket és nem látok különbséget. Küldenél egy screenshotot? Esetleg tudod hogy mi (melyik szűrőszabály) okozhatja a problémát?

Így néz ki:

Ublock origint használva egyébként. Fogalmam sincs a szűrőről. UBO-val nem is nagyon lehet próbálgatni deaktiválni ezeket.

Az ublock az lehet baj. Meg tudod esetleg nézni Adblock Plus-sal hogy van-e különbség?

Meg, de azt már évek óta nem használtam. Ki tudja sikerül-e mindent jól beállítani.

Köszi! Igazából a plugint kell telepíteni (next-next-finish) és feliratkozni a hufilterre, majd megnyitni az oldalt.

Ezzel más baja van. Nem tűnik el a reklám (jelenleg Honor 20 Lite reklám) és az oldal töltése sem fejeződik be.

Az a baj, hogy a cikk előtti részen van egy reklám, amire a rákövetkező div "rácsúszik". Nem a HUFILTER szűri ezt a reklámot, hanem valamelyik alap UBO filter.
Kép link: https://i.imgur.com/WdO4V84.png

@simkoG commented on 2019. aug. 11. 08:58 CEST:

Az a baj, hogy a cikk előtti részen van egy reklám, amire a rákövetkező div "rácsúszik". Nem a HUFILTER szűri ezt a reklámot, hanem valamelyik alap UBO filter.
Kép link: https://i.imgur.com/WdO4V84.png

Nem ezt sugallja az a tény, hogy ha kikapcsolom a hufiltert UBO-ban akkor nem esik össze az oldal:

A reklámok továbbra is eltűnnek valamilyen szűrő által.

@simkoG commented on 2019. aug. 11. 08:58 CEST:

Az a baj, hogy a cikk előtti részen van egy reklám, amire a rákövetkező div "rácsúszik". Nem a HUFILTER szűri ezt a reklámot, hanem valamelyik alap UBO filter.
Kép link: https://i.imgur.com/WdO4V84.png

Nem ezt sugallja az a tény, hogy ha kikapcsolom a hufiltert UBO-ban akkor nem esik össze az oldal:
A reklámok továbbra is eltűnnek valamilyen szűrő által.

Igaz, most megnéztem, tényleg a hufilter tűnteti el. Viszont ha kikapcsolom a hufilter-t, nekem a fenti huawei reklám látszik.

@simkoG commented on 2019. aug. 11. 10:07 CEST:

Igaz, most megnéztem, tényleg a hufilter tűnteti el. Viszont ha kikapcsolom a hufilter-t, nekem a fenti huawei reklám látszik.

Nyilván attól függ milyen egyéb szűrők, szűrőlisták vannak aktiválva. A teszt profilomban nekem is megmaradnak a reklámok hufilter nélkül.

A hufilter valószínűleg szintaktikailag nem teljesen kompatibilis az UBlock-kal. Ha van valakinek kedve egy hufilter-ublock verziót karbantartani, szóljon.

authors: mzso simkoG szpeter80

repo: hufilter/hufilter


text:
Remove global variables

An attempt to remove the global variables #117.
Any feedback or ideas?
I've been able to remove one pytest.mark.skip with these changes.

Codecov Report

Merging #140 into master will increase coverage by 0.07%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
+ Coverage   95.15%   95.22%   +0.07%
==========================================
  Files           4        5       +1
  Lines         454      461       +7
  Branches       96       94       -2
==========================================
+ Hits          432      439       +7
  Misses          7        7
  Partials       15       15

Impacted Files
Coverage Δ

sanic_openapi/spec.py
100% <100%> (ø)

sanic_openapi/swagger.py
94.69% <100%> (-0.4%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e729e9...fda0c0d. Read the comment docs.

Hi @AtomsForPeace
Thanks for your contribution. I think this PR fixes the swagger._spec global variable very well. And, do you have any plans to fix other global variables? Should I approve and merge this PR first or waiting for other global variable fixes?

@chenjr0719 Thanks. I think merging this now would be good. I've had a look at the other global variables and they're not as easy. I was thinking maybe doing something along the lines of this: https://flask.palletsprojects.com/en/1.1.x/patterns/appfactories/#factories-extensions
The issue would be backwards compatibility though.
What's your opinion on it?

@AtomsForPeace Sorry for the delay. I agree with using the factory pattern to fix all global variables. And about backward compatibility, I would say maybe we can have a PoC first and this might help us to decide what is the better way to keep the compatibility.

authors: AtomsForPeace chenjr0719 codecov-io

repo: huge-success/sanic-openapi


text:
Support schedulefree optimizer

What does this PR do?
Support schedulefree optimizer
https://github.com/facebookresearch/schedule_free
it used optimizer.train()

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi! Thanks for your PR, I already have https://github.com/huggingface/accelerate/pull/2631 which has a bit more of a robust solution and an example

Hi! Thanks for your PR, I already have #2631 which has a bit more of a robust solution and an example

Thank you for great work~

repo: huggingface/accelerate


text:
fix accelerate test failure with cpu config

*failure occurs when testing FP16
*autocast fail to work for cpu bf16 in some gpu+cpu platform, no need to use is_bf16_available logic. because native_amp already contains such logic.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

authors: HuggingFaceDocBuilderDev sywangyi


text:
[BUG] With Pandas 2.0.0, load_dataset raises TypeError: read_csv() got an unexpected keyword argument 'mangle_dupe_cols'

The load_dataset function with Pandas 1.5.3 has no issue (just a FutureWarning) but crashes with Pandas 2.0.0.
For your convenience, I opened a draft Pull Request to fix it quickly: https://github.com/huggingface/datasets/pull/5745

The FutureWarning mentioned above:

FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'

Thanks for reporting, @keyboardAnt.
We haven't noticed any crash in our CI tests. Could you please indicate specifically the load_dataset command that crashes in your side, so that we can reproduce it?

This has been fixed in datasets 2.11

I am still getting this bug with the latest pandas and datasets lib installed. Anyone else?

Feel free to update datasets to fix this issue
pip install -U datasets

I get this with pip when doing pip install -U datasets
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf 23.8.0 requires cubinlinker, which is not installed.
cudf 23.8.0 requires cupy-cuda11x>=12.0.0, which is not installed.
cudf 23.8.0 requires ptxcompiler, which is not installed.
cuml 23.8.0 requires cupy-cuda11x>=12.0.0, which is not installed.
dask-cudf 23.8.0 requires cupy-cuda11x>=12.0.0, which is not installed.
apache-beam 2.46.0 requires dill<0.3.2,>=0.3.1.1, but you have dill 0.3.8 which is incompatible.
apache-beam 2.46.0 requires numpy<1.25.0,>=1.14.3, but you have numpy 1.26.4 which is incompatible.
apache-beam 2.46.0 requires pyarrow<10.0.0,>=3.0.0, but you have pyarrow 15.0.1 which is incompatible.
beatrix-jupyterlab 2023.128.151533 requires jupyterlab~=3.6.0, but you have jupyterlab 4.1.2 which is incompatible.
cudf 23.8.0 requires cuda-python<12.0a0,>=11.7.1, but you have cuda-python 12.3.0 which is incompatible.
cudf 23.8.0 requires pandas<1.6.0dev0,>=1.3, but you have pandas 2.1.4 which is incompatible.
cudf 23.8.0 requires protobuf<5,>=4.21, but you have protobuf 3.20.3 which is incompatible.
cudf 23.8.0 requires pyarrow==11.*, but you have pyarrow 15.0.1 which is incompatible.
cuml 23.8.0 requires dask==2023.7.1, but you have dask 2024.2.0 which is incompatible.
dask-cudf 23.8.0 requires dask==2023.7.1, but you have dask 2024.2.0 which is incompatible.
dask-cudf 23.8.0 requires pandas<1.6.0dev0,>=1.3, but you have pandas 2.1.4 which is incompatible.

I am still having the same issue with the version >= 2.14

This issue still exists as you can see in version 2.14:
https://github.com/huggingface/datasets/blob/bf02cff8d70180a9e89328961ded9e3d8510fd22/src/datasets/packaged_modules/csv/csv.py#L35
that "mangle_dupe_cols" still exists in the arguments.
And this error occurs at this line:
https://github.com/huggingface/datasets/blob/bf02cff8d70180a9e89328961ded9e3d8510fd22/src/datasets/packaged_modules/csv/csv.py#L185
where
self.config.pd_read_csv_kwargs = {'sep': ',', 'header': 'infer', 'index_col': None, 'usecols': None, 'mangle_dupe_cols': True, 'engine': None, 'true_values': None, 'false_values': None, 'skipinitialspace': False, 'skiprows': None, 'nrows': None, 'na_values': None, 'keep_default_na': True, 'na_filter': True, 'verbose': False, 'skip_blank_lines': True, 'thousands': None, 'decimal': '.', 'lineterminator': None, 'quotechar': '"', 'quoting': 0, 'escapechar': None, 'comment': None, 'encoding': None, 'dialect': None, 'skipfooter': 0, 'doublequote': True, 'memory_map': False, 'float_precision': None, 'chunksize': 10000}

for me.
Here is where we got the error: https://github.com/meta-llama/llama-recipes/issues/426

repo: huggingface/datasets


text:
Dataset infos in yaml

To simplify the addition of new datasets, we'd like to have the dataset infos in the YAML and deprecate the dataset_infos.json file. YAML is readable and easy to edit, and the YAML metadata of the readme already contain dataset metadata so we would have everything in one place.
To be more specific, I moved these fields from DatasetInfo to the YAML:

config_name (if there are several configs)
download_size
dataset_size
features
splits

Here is what I ended up with for squad:
dataset_info:
  features:
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: answer_start
      dtype: int32
  splits:
  - name: train
    num_bytes: 79346360
    num_examples: 87599
  - name: validation
    num_bytes: 10473040
    num_examples: 10570
  download_size: 35142551
  dataset_size: 89819400

and it can be a list if there are several configs
I already did the change for conll2000 and crime_and_punish as an example.
Implementation details
Load/Read
This is done via DatasetInfosDict.write_to_directory/from_directory
I had to implement a custom the YAML export logic for SplitDict, Version and Features.
The first two are trivial, but the logic for Features is more complicated, because I added a simplification step (or the YAML would be too long and less readable): it's just a formatting step to remove unnecessary nesting of YAML data.
Other changes
I had to update the DatasetModule factories to also download the README.md alongside the dataset scripts/data files, and not just the dataset_infos.json
YAML validation
I removed the old validation code that was in metadata.py, now we can just use the Hub YAML validation
Datasets-cli
The datasets-cli test --save_infos command now creates a README.md file with the dataset_infos in it, instead of a datasets_infos.json file
Backward compatibility
dataset_infos.json files are still supported and loaded if they exist to have full backward compatibility.
Though I removed the unnecessary keys when the value is the default (like all the id: null from the Value feature types) to make them easier to read.
TODO

[x] add comments
[x] tests
[x] document the new YAML fields
[x] try to reload the new dataset_infos.json file content with an old version of datasets

EDITS

removed "config_name" when there's only one config
removed "version" for now (?), because it's not useful in general
renamed the yaml field "dataset_info" instead of "dataset_infos", since it only has one by default (and because "infos" is not english)

Fix https://github.com/huggingface/datasets/issues/4876

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Alright this is ready for review :)
I mostly would like your opinion on the YAML structure and what we can do in the docs (IMO we can add the docs about those fields in the Hub docs). Other than that let me know if the changes in info.py and features.py look good to you

LGTM and looking forward to having this merged!! ❤

We plan to do a release today, we'll merge this after the release :)

Created https://github.com/huggingface/datasets/pull/5018 where I added the YAML dataset_info of every single dataset in this repo

Tooks your comments into account and updated push_to_hub to push the dataset_info to the README.md instead of json :) Let me know if it sounds good to you now !


text:
Format model/dataset cards "a la wikipedia" (enlarge to the right)

Is your feature request related to a problem? Please describe.
Currently, model/dataset cards README.md takes about only half the screen space after the sections "Hosted inference API", "Dataset used to train", "Spaces using", etc. See

This is not ideal, especially for large tables (i.e. tables with more than 4-5 columns) and for some code as above that doesn't get wrapped and is truncated, lowering readability. Also, the text gets wrapped to a new line in only a few words, which also lowers readability.
See for example https://huggingface.co/deepset/roberta-base-squad2 and https://huggingface.co/fxmarty/donotdelete
Describe the solution you'd like
Extend to the right model cards after the top right section ("Hosted inference API", "Dataset used to train", "Spaces using", etc.) finishes. Wikipedia does a good job at it, see for example: https://en.wikipedia.org/wiki/Progressive_rock

tagging @gary149 for visibility

Visually speaking I'm not sure it will work really well. If we really want to have more horizontal space we'll need to tweak the whole layout, it is something we'll keep in mind when re-working on it (not short-term).

authors: fxmarty gary149 julien-c

repo: huggingface/hub-docs


text:
Shape mismatch error

Hello everyone,
i'm trying to train neuralcoref for french language.
After training the model, i add it to the pipeline but when i execute this command:
doc = nlp(u"je suis firas")

I get the following error
---------------------------------------------------------------------------
ShapeMismatchError                        Traceback (most recent call last)
<ipython-input-31-0862bb3d76d3> in <module>
----> 1 doc = nlp(u"je suis firas")

~/.conda/envs/coref2/lib/python3.8/site-packages/spacy/language.py in __call__(self, text, disable, component_cfg)
    437             if not hasattr(proc, "__call__"):
    438                 raise ValueError(Errors.E003.format(component=type(proc), name=name))
--> 439             doc = proc(doc, **component_cfg.get(name, {}))
    440             if doc is None:
    441                 raise ValueError(Errors.E005.format(name=name))

neuralcoref.pyx in neuralcoref.neuralcoref.NeuralCoref.__call__()

neuralcoref.pyx in neuralcoref.neuralcoref.NeuralCoref.predict()

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/neural/_classes/model.py in __call__(self, x)
    165             Must match expected shape
    166         """
--> 167         return self.predict(x)
    168
    169     def pipe(self, stream, batch_size=128):

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/neural/_classes/feed_forward.py in predict(self, X)
     38     def predict(self, X):
     39         for layer in self._layers:
---> 40             X = layer(X)
     41         return X
     42

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/neural/_classes/model.py in __call__(self, x)
    165             Must match expected shape
    166         """
--> 167         return self.predict(x)
    168
    169     def pipe(self, stream, batch_size=128):

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/check.py in checked_function(wrapped, instance, args, kwargs)
    153                 if not isinstance(check, Callable):
    154                     raise ExpectedTypeError(check, ["Callable"])
--> 155                 check(arg_id, fix_args, kwargs)
    156         return wrapped(*args, **kwargs)
    157

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/check.py in has_shape_inner(arg_id, args, kwargs)
     72             # Allow underspecified dimensions
     73             if dim is not None and arg.shape[i] != dim:
---> 74                 raise ShapeMismatchError(arg.shape, shape_values, shape)
     75
     76     return has_shape_inner

ShapeMismatchError:

  Shape mismatch: input (0, 668) not compatible with [None, 3924].

  Traceback:
  ├─ __call__ in /home/equalios/.conda/envs/coref2/lib/python3.8/site-packages/spacy/language.py:439
  ├─── __call__ in neural/_classes/model.py:167
  └───── predict in neural/_classes/feed_forward.py:40
         >>> X = layer(X)

```

I'am using an embedding vector of size 300
Can any one please help me

Hello everyone,
i'm trying to train neuralcoref for french language.
After training the model, i add it to the pipeline but when i execute this command:
doc = nlp(u"je suis firas")

I get the following error
---------------------------------------------------------------------------
ShapeMismatchError                        Traceback (most recent call last)
<ipython-input-31-0862bb3d76d3> in <module>
----> 1 doc = nlp(u"je suis firas")

~/.conda/envs/coref2/lib/python3.8/site-packages/spacy/language.py in __call__(self, text, disable, component_cfg)
    437             if not hasattr(proc, "__call__"):
    438                 raise ValueError(Errors.E003.format(component=type(proc), name=name))
--> 439             doc = proc(doc, **component_cfg.get(name, {}))
    440             if doc is None:
    441                 raise ValueError(Errors.E005.format(name=name))

neuralcoref.pyx in neuralcoref.neuralcoref.NeuralCoref.__call__()

neuralcoref.pyx in neuralcoref.neuralcoref.NeuralCoref.predict()

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/neural/_classes/model.py in __call__(self, x)
    165             Must match expected shape
    166         """
--> 167         return self.predict(x)
    168
    169     def pipe(self, stream, batch_size=128):

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/neural/_classes/feed_forward.py in predict(self, X)
     38     def predict(self, X):
     39         for layer in self._layers:
---> 40             X = layer(X)
     41         return X
     42

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/neural/_classes/model.py in __call__(self, x)
    165             Must match expected shape
    166         """
--> 167         return self.predict(x)
    168
    169     def pipe(self, stream, batch_size=128):

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/check.py in checked_function(wrapped, instance, args, kwargs)
    153                 if not isinstance(check, Callable):
    154                     raise ExpectedTypeError(check, ["Callable"])
--> 155                 check(arg_id, fix_args, kwargs)
    156         return wrapped(*args, **kwargs)
    157

~/.conda/envs/coref2/lib/python3.8/site-packages/thinc/check.py in has_shape_inner(arg_id, args, kwargs)
     72             # Allow underspecified dimensions
     73             if dim is not None and arg.shape[i] != dim:
---> 74                 raise ShapeMismatchError(arg.shape, shape_values, shape)
     75
     76     return has_shape_inner

ShapeMismatchError:

  Shape mismatch: input (0, 668) not compatible with [None, 3924].

  Traceback:
  ├─ __call__ in /home/equalios/.conda/envs/coref2/lib/python3.8/site-packages/spacy/language.py:439
  ├─── __call__ in neural/_classes/model.py:167
  └───── predict in neural/_classes/feed_forward.py:40
         >>> X = layer(X)

```
I'am using an embedding vector of size 300
Can any one please help me

repo: huggingface/neuralcoref


text:
Add trust_remote_code to audio classification CI tests and README.

Add trust_remote_code to README and CI tests.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
x Did you make sure to update the documentation with your changes?
[x ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@emascarenhas
I was looking at this PR.

There README file for ac needs another --trust_remote in inference section.
for ac baseline, the test is failing on my end due to perf assertation.

diff --git a/examples/audio-classification/README.md b/examples/audio-classification/README.md
index 17935cf1..197bf9ff 100644
--- a/examples/audio-classification/README.md
+++ b/examples/audio-classification/README.md
@@ -174,7 +174,8 @@ python run_audio_classification.py \
     --use_lazy_mode \
     --use_hpu_graphs_for_inference \
     --gaudi_config_name Habana/wav2vec2 \
-    --bf16
+    --bf16 \
+    --trust_remote_code True

diff --git a/tests/baselines/ast_finetuned_speech_commands_v2.json b/tests/baselines/ast_finetuned_speech_commands_v2.json
index f518c929..c5c5cb3d 100644
--- a/tests/baselines/ast_finetuned_speech_commands_v2.json
+++ b/tests/baselines/ast_finetuned_speech_commands_v2.json
@@ -8,8 +8,8 @@
                     "learning_rate": 5e-4,
                     "train_batch_size": 32,
                     "eval_accuracy": 0.1871,
-                    "train_runtime": 139.9477,
-                    "train_samples_per_second": 1955.74,
+                    "train_runtime": 141.4154,
+                    "train_samples_per_second": 1751.404,
                     "eval_samples_per_second": 2301.088,
                     "extra_arguments": [
                         "--audio_column_name audio",

we could adjust these a bit for this to actually pass. I can't find a RD backed config that matches the our settings for #epocs, bs sizes, etc for this case. The public data are taken from RD backend too, so shouldn't be too bad to adjust here.
https://developer.habana.ai/get-started/habana-models-performance/
should we make these changes here now, or wait for later? WDYT @regisss ?
With above changes, I get the test to pass
GAUDI2_CI=1 RUN_SLOW=true python -m pytest  tests/test_examples.py -s -v -k test_run_audio_classification_ast-finetuned-speech-commands-v2_multi_card
.
.

***** train metrics *****
  epoch                       =         10.0
  max_memory_allocated (GB)   =         3.85
  memory_allocated (GB)       =         3.52
  total_flos                  = 1741424192GF
  total_memory_available (GB) =        94.62
  train_loss                  =        1.021
  train_runtime               =   0:02:21.92
  train_samples_per_second    =     1745.801
  train_steps_per_second      =        6.844
[INFO|trainer.py:1812] 2024-08-16 17:12:58,741 >> Using HPU graphs for inference.
[INFO|trainer.py:1830] 2024-08-16 17:12:58,741 >>
***** Running Evaluation *****
[INFO|trainer.py:1832] 2024-08-16 17:12:58,741 >>   Num examples = 5888
[INFO|trainer.py:1835] 2024-08-16 17:12:58,741 >>   Batch size = 64
100%|██████████| 12/12 [00:02<00:00,  5.89it/s]
***** eval metrics *****
  epoch                       =       10.0
  eval_accuracy               =     0.1921
  eval_loss                   =     4.8415
  eval_runtime                = 0:00:04.57
  eval_samples_per_second     =   2535.925
  eval_steps_per_second       =      5.244
  max_memory_allocated (GB)   =       3.85
  memory_allocated (GB)       =       3.73
  total_memory_available (GB) =      94.62
PASSED

============================================================================================ 1 passed, 55 deselected in 210.54s (0:03:30) ===============

@emascarenhas I was looking at this PR.

There README file for ac needs another --trust_remote in inference section.
for ac baseline, the test is failing on my end due to perf assertation.

@imangohari1 ,
Yes lets take this change.
...

we could adjust these a bit for this to actually pass. I can't find a RD backed config that matches the our settings for #epocs, bs sizes, etc for this case. The public data are taken from RD backend too, so shouldn't be too bad to adjust here. https://developer.habana.ai/get-started/habana-models-performance/

This change isn't required, as I could get a result that would pass with the current setting. See results below.

should we make these changes here now, or wait for later? WDYT @regisss ?

My results were:
***** train metrics *****
epoch                       =         10.0
max_memory_allocated (GB)   =         6.61
memory_allocated (GB)       =         3.52
total_flos                  = 1741424192GF
total_memory_available (GB) =        94.62
train_loss                  =        1.021
train_runtime               =   0:02:16.22
train_samples_per_second    =      1963.21
train_steps_per_second      =        7.696
[INFO|trainer.py:1812] 2024-08-15 05:10:08,396 >> Using HPU graphs for inference.
[INFO|trainer.py:1830] 2024-08-15 05:10:08,396 >>
***** Running Evaluation *****
[INFO|trainer.py:1832] 2024-08-15 05:10:08,396 >>   Num examples = 5888
[INFO|trainer.py:1835] 2024-08-15 05:10:08,396 >>   Batch size = 64
100%|██████████| 12/12 [00:01<00:00,  7.21it/s]
***** eval metrics *****
epoch                       =       10.0
eval_accuracy               =     0.1921
eval_loss                   =     4.8415
eval_runtime                = 0:00:04.28
eval_samples_per_second     =   3084.876
eval_steps_per_second       =       6.38
max_memory_allocated (GB)   =       6.61
memory_allocated (GB)       =       3.73
total_memory_available (GB) =      94.62
The README change is required, so we should take that change. The other adjustment isn't required I think as our CI systems can get the test to pass without that change.

@emascarenhas I was looking at this PR.

There README file for ac needs another --trust_remote in inference section.
for ac baseline, the test is failing on my end due to perf assertation.

@imangohari1 , Yes lets take this change. ...

we could adjust these a bit for this to actually pass. I can't find a RD backed config that matches the our settings for #epocs, bs sizes, etc for this case. The public data are taken from RD backend too, so shouldn't be too bad to adjust here. https://developer.habana.ai/get-started/habana-models-performance/

This change isn't required, as I could get a result that would pass with the current setting. See results below.

should we make these changes here now, or wait for later? WDYT @regisss ?

My results were: ***** train metrics ***** epoch = 10.0 max_memory_allocated (GB) = 6.61 memory_allocated (GB) = 3.52 total_flos = 1741424192GF total_memory_available (GB) = 94.62 train_loss = 1.021 train_runtime = 0:02:16.22 train_samples_per_second = 1963.21 train_steps_per_second = 7.696 [INFO|trainer.py:1812] 2024-08-15 05:10:08,396 >> Using HPU graphs for inference. [INFO|trainer.py:1830] 2024-08-15 05:10:08,396 >> ***** Running Evaluation ***** [INFO|trainer.py:1832] 2024-08-15 05:10:08,396 >> Num examples = 5888 [INFO|trainer.py:1835] 2024-08-15 05:10:08,396 >> Batch size = 64 100%|██████████| 12/12 [00:01<00:00, 7.21it/s] ***** eval metrics ***** epoch = 10.0 eval_accuracy = 0.1921 eval_loss = 4.8415 eval_runtime = 0:00:04.28 eval_samples_per_second = 3084.876 eval_steps_per_second = 6.38 max_memory_allocated (GB) = 6.61 memory_allocated (GB) = 3.73 total_memory_available (GB) = 94.62
The README change is required, so we should take that change. The other adjustment isn't required I think as our CI systems can get the test to pass without that change.

Done: https://github.com/huggingface/optimum-habana/pull/1268

repo: huggingface/optimum-habana


text:
FIX Allow same layer adapters on different devices

Resolves #1639
The issue is that so far, we made the assumption in PEFT that all adapter weights of a specific layer are on the same device. There can be cases where it is useful to have adapters on different devices. E.g. when a user loads a lot of LoRA adapters and wants to offload those not currently in use to CPU, they would not currently be able to do so.
With this PR, we add this possibility. To achieve this, when we update an adapter layer with a new adapter, we only move that specific adapter to the device of the base layer, will not touching the other loaded adapters.
While working on this, I discovered a small bug in VeRA when adding multiple adapters, which is now also fixed.
This PR has the potential to lead to unforeseen issues, so careful review is required. After merging this, let's keep it out of releases for a while to ensure it doesn't break anything.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the reviews. I'll wait a bit in case @iuliaturc has time to check if this PR fixes the initial issue. If we don't hear back, I'll merge in a few days.
We should let this PR "simmer" for a bit since there is a small probability that this will break some edge case we haven't thought of.

Thanks so much for the PR! I left a comment here.
TL;DR is that, indeed, only one LoRA seems to be loaded at a time, but the fix doesn't seem to address the original problem (that latency keeps creeping up the more calls we make).

Thanks for the confirmation.

repo: huggingface/peft


text:
[bug fix] ensure base_model is correctly set in model card

TLDR:
When we train LoRA's on top of a model loaded from disk, SFTTrainer.push_to_hub fails because of the model card created by Trainer.create_model_card. The Trainer.create_model_card calls the peft library's create_or_update_model_card which overrides the base_model to the path of the model on disk even if the model card created by Trainer.create_model_card might have contained the value of the base_model.
Long Form Context :
The model card's parsing is an essential when pushing the model to hub. If the model card doesn't contain information that HF finds valid, then HF hub raises a ValueError. One of the fields in the model_card injected by the peft library is the base_model field. This field is set earlier using the model_config["_name_or_path"]. A problem occurs when the model_config["_name_or_path"] is not a model name on HuggingfaceHub but rather just a path on local file system. At this point the hf_hub's folder_upload method crashes. Now if the card already contained a base_modelset then uploading the model from disk is still possible. One way to set the base_model's value is to pass the SFTTrainer.push_to_hub/Trainer.push_to_hub kwargs which are passed down to the create_model_card function.  These kwargs can contain a field called finetuned_from which allow the model card creation properly.

Thanks for this PR. Your suggestion and explanation sound reasonable.
Before proceeding, do you have a small code example to reproduce the current error? I'm asking because I wonder if we can craft a unit test based on that. It's always good to have a unit test to accompany a bug fix so that the same bug cannot later be re-introduced by accident.

Okay, I think this issue got resolved for me as I explained over here https://github.com/huggingface/transformers/issues/33922. Overall since TRL changed the create_model_card function, I was able to make this work without needing the change in PEFT.

authors: BenjaminBossan valayDave


text:
add: example on semantic segmentation.

Closes #45
There are two minor changes to the image classification example. I know this is not a good PR practice but the changes were extremely small.

wrt semantic segmentation example:

can we remove the cell ouputs of trainer.train() as they are huge and will take up considerable space
I think the example notebook has bug because Accuracy Wall is nan  for full finetuning too but in stevhliu/my_awesome_seg_model · Hugging Face it's very high. I tried full fine-tuning using 50 images as the task page (40 train + 10 eval)

I will address the first point in the follow-up commits. Regarding the second point, I think there are also quite a few nan values for the other classes in the repo you mentioned.
If you take a look at the training loss, it looks similar, though. I think the category that might be impacted because of the low number of training examples depends on their quantity in the training set as well. Not sure what you mean by the bug.

By bug I mean I'm unable to reproduce the same results for full fine-tuning using same train samples and eval samples along with same trainer arguments as that task page repo

this is what I see for full fine-tuning: https://github.com/pacman100/temp/blob/main/Copy_of_semantic_segmentation_peft_lora (1).ipynb

@pacman100 addressed the things we discussed during the call. PTAL.

Thank you for iterating! LGTM 🤗


text:
ResNet models on HF hub, multi-weight support

add torchvision v2 weights, new 12k pretrained and fine-tuned timm anti-aliased weights, all previous timm resnet weights and RSB variants (a1, a2, a3)

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

authors: HuggingFaceDocBuilderDev rwightman

repo: huggingface/pytorch-image-models


text:
Improve type hints and docs in timm/models

Part of #1989.
Improves type hints and docs in the following files:

timm/models/mobilenetv3.py
timm/models/resnet.py

Fixes minor typo in CONTRIBUTING.md.
@rwightman I would like to know your thoughts on the changes below, and what you do and don't prefer. Will make sure to follow them for subsequent PRs.
I would also like to say that your implementations here have helped me in understanding a lot about going from paper to code as mentioned in the issue. It is really well written and intuitively understandable in terms of code. Thanks for this!

Also noticed that the formatting command black --skip-string-normalization --line-length 120 <path-to-file> (as mentioned in the contributing guide) has not been run on many files. When executed on the above two files, a long list of unrelated indent changes occur, and so I've not committed that so far.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@a-r-r-o-w thanks! looking overall good so far. Lettting the CI run, annotations shouldn't impact eager but can impact torchscript tests.
A few things:
Black settings were put in there by another Hugging Face dev, it's a common setup for HF repos but I'm opinionated, black is very opinionated and I have some issues so have not made the leap to turn that on. So let's not run it. Still looking for the right balance for formatting rules, black, autopep8, yapf...
The layer typing as you've noticed isn't super consistent, some models support just nn.Module types, some strings, some both, etc. Looks like you're capturing that, but the 'type' of a non-instance of nn.Module is not nn.Module, it's type[nn.Module] (or typing.Type[nn.Module] for pre python 3.9, which we need to support still, officially 3.7 but can bump up to 3.8 now I feel).
Some of the MobileNetV3 and EfficientNet types are very specific to just those two model classes (BlockArgs especially). Should probably constrain those types to _efficientnet_blocks.py as any other model won't use them. Having the typing module for other common types makes sense but should contain types that are broadly common. Also might want to consider moving that to timm/layers/typing.py as may find some of them propagate to layers with an eventual goal to cover them as well.
Similarly, some other model instances might have some non-trivial types specific to them, unless you see it being common to other models, should probably define at the top of the model module.

Makes sense about the formatter part so will put that on hold.
I've added a few changes based on my interpretation about the typing part, and it makes sense for when I open subsequent PRs for more files. Is this what you meant? If not, could you add suggestions to the relevant parts while reviewing?
Also, don't know how I confused the nn.Module as class instances instead of class objects initially :sweat_smile: There are some instances in resnet.py where I thought it was better to put Type[nn.Module] instead of LayerType as the latter is more general (as in Union of types that is then processed by some internal function to yield a nn.Module) and only applies to ResNet (where some preprocessing on the activations is performed) and not BasicBlock (where activations are only instantiated). If this is too convoluted and since typing is not really strict in python, we could just stick to everything being LayerType maybe?

@rwightman Any updates on this?

@a-r-r-o-w I've had hands full pushing some releases out, handling some other things, on quick look it was looking good but will aim to do a more detailed check and merge in a few days.

Sure, take your time as there's no hurry at all. Eventually it should get faster once I understand the format you expect changes in.

@rwightman Added changes from your suggestions. Let me know if this is ready to go.


text:
NSCocoaErrorDomain Code = 3840

I use exporters to convert gpt2-xl, gpt2-large and MBZUAI/LaMini-GPT-1.5B with —feature=text-generation , then loaded into swift-chat and try to generate text, gpt2-large and gpt2-xl are fine,
But received error while trying out MBZUAI/LaMini-GPT-1.5B.
“””
Error: Error Domain=NSCocoaErrorDomain Code=3840 "JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0." UserInfo={NSDebugDescription=JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0., NSJSONSerializationErrorIndex=0}
“””

Interesting, I took a quick look at the json files in the model configuration but did not notice anything wrong.
I'll perform the conversion to try to reproduce. Did you happen to push your converted model to the Hub so I could use it instead?

Seems to be the problem of exporters .
I've tried to convert the same model (MBZUAI/LaMini-GPT-1.5B)again with same method, and somehow there is no error,  everything is fine.
But still, I converted a different model Aira-Instruct-774M four times, all passing the validation in exporters, but still error in swift-transformer
pushed the broken model to HuggingFace

@Leung-Kam-Ho Would you mind uploading the succeeded converted model of LaMini-GPT-1.5B to HuggingFace too? Would greatly appreciate it

I uploaded the success conversion LaMini-GPT-1.5B-coreML to HF, check out if you are interested

authors: Leung-Kam-Ho pcuenca teknoalex

repo: huggingface/swift-transformers


text:
Fixed demo by moving sample audio into repo (avoids CORS)

Soundcloud doesn't serve audio with the CORS header, so it's not possible to run it through an AnalyserNode. To remedy this I've included a audio sample in the repository itself, in order to avoid the CORS problems. The audio is licensed CC-BY-NC 3.0, available at http://freemusicarchive.org/music/Kai_Engel/Irsens_Tale/Kai_Engel_-Irsens_Tale-_04_Moonlight_Reprise I've included attribution on the demo.html page.

this is a great fix, I hope it gets merged in!

repo: hughsk/web-audio-analyser


text:
Does Hugo Apero support interactive elements?

Hi!
I want to create a new blog and I'm considering using Hugo Apero.
Thing is, I'm a heavy user of plotly, reactable and crosstalk. Does Hugo Apero works well with these packages?
Thanks!

I have used Plotly, DT and Shiny on my website. They work great on laptops, but the view is sometimes distorted in mobile view.

Thanks, that's helpful!
I'll give it a try and share the results.

authors: harshvardhaniimi uriahf

repo: hugo-apero/hugo-apero


text:
这个能改改吗

[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: type 'String' is not a subtype of type 'Map<String, dynamic>'
#0      Client.init. (package:flutter_oss_aliyun/src/client.dart:46:45)

#1      AuthMixin.getAuth (package:flutter_oss_aliyun/src/auth_mixin.dart:13:14)

#2      Client.putObject (package:flutter_oss_aliyun/src/client.dart:256:23)

可以提供一个简单的可以复现上述错误的代码吗？

我已经从外部解决了


text:
Modaal instagram, poping up problem

You can check it out on my website https://test.globus-furniture.com/#gallery
Click on any image in "Furniture Heaven" block, modaal frame appears, but when you close and open another one, it can not to appear, although background gets darker and "close button" also works. I don't know why it happens, but that's a big problem.

It looks like that, when it doesn't work. It should appear after loading, but it never happens.

Thanks for raising the issue @faradaytrs, I spent a little time this morning looking at this (and the quality issue in #95) a little more closely and have rolled out a fix for this. Grab the latest copy if you want to fix your build.

Thank you very much

authors: danhumaan faradaytrs

repo: humaan/Modaal


text:
A reviewed dimension should not have the "please review" tooltip

Even when the dot is grey, the tooltip says "please review"

I'm not able to recreate, unfortunately. This is on the latest version on the website?

I see. We actually shouldn't be displaying the grey dot at all in this scenario.

Ok, I now recall that I actually don't remember if we are supposed to be showing the grey dot or not in this scenario. I've changed the tooltip text to "Verified" (see commit) and left it visible. This dot is visible only in the situation where the dimension has special characters but they have all been verified.

authors: elizawe esjewett

repo: humanitiesplusdesign/palladio


text:
Links to PDFs on the tutorial/FAQ page are broken

Page at http://hdlab.stanford.edu/palladio/help/ links to http://hdlab.stanford.edu/palladio/doc/scenario-simple-map.pdf but should be http://hdlab.stanford.edu/doc/scenario-simple-map.pdf (same for point-to-point tutorial).

In faq.md these point to - ({{ site.baseurl }}/doc/scenario-simple-map.pdf)
(site.baseurl is '/palladio' )
Updated the links in faq.md (there are three) on the documentation-updates branch.

authors: esjewett jsuttonmorse


text:
Allow adding of upgraders to channel initialisers

Instead of requiring the channel initialiser to be replaced.
Will allow us to  fix issue where adding websocket upgrade would remove http2 upgrade

I disagree with the breaking changes check here as we add a default version of addProtocolUpgrader

repo: hummingbird-project/hummingbird-core


text:
Website UX Bug: Examples route scroll resets to the top on tab navigation

Current Behavior
Navigating through the tabs (Dashboard/Cards/Playground/Music/Authentication/Forms in the examples route resets the scroll to the top each time a new tab is accessed.
Expected Behavior
The scroll height should be kept when switching through tabs.
Steps To Reproduce

Go to https://www.shadcn-svelte.com/examples/dashboard
Scroll down a bit to see the contents of the Dashboard tab
Click on any of the tabs (Dashboard/Cards/Playground/Music/Authentication/Forms

Link to Reproduction / Stackblitz
No response
More Information
Thank you for your efforts on the port, been using it frequently in my projects. Just wanted to inform about this bug as it has been a bit annoying when switching through the different tabs when viewing examples. Keep up the great work!

https://github.com/huntabyte/shadcn-svelte/assets/102677971/8331a388-88e6-4a32-b389-c80460f3fef5

@huntabyte  I could take a look at this later today if i get some time over after work.

Closed by #355

authors: HubbeDev akifisitan huntabyte

repo: huntabyte/shadcn-svelte


text:
🛑 retro is down

In e5378f5, retro (https://retro.thekor.eu/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: retro is back up in e411c2b after 7 days, 19 hours, 18 minutes.

repo: hupratt/upptime


text:
BUG: table of contents in callouts does not update #77

https://github.com/Aidurber/obsidian-plugin-dynamic-toc/issues/77

@dbarenholz

This should now be working (without any changes to the plugin itself)

repo: husjon/obsidian-plugin-dynamic-toc


text:
Error while training

I tried to run "CUDA_VISIBLE_DEVICES=0 sh train.sh" for training but i get this error.
Traceback (most recent call last):
File "train.py", line 22, in
from misc.rewards import init_scorer, get_self_critical_reward
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/misc/rewards.py", line 13, in
from pyciderevalcap.ciderD.ciderD import CiderD
ModuleNotFoundError: No module named 'pyciderevalcap'
Traceback (most recent call last):
File "train.py", line 22, in
from misc.rewards import init_scorer, get_self_critical_reward
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/misc/rewards.py", line 13, in
from pyciderevalcap.ciderD.ciderD import CiderD
ModuleNotFoundError: No module named 'pyciderevalcap'
i have tried installing "conda install -c hcc pycocotools" this package but i am unable to install and this command gives this error
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:

pycocotools -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0']

Your python: python=3.7
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
The following specifications were found to be incompatible with each other:
Package python-dateutil conflicts for:
pycocotools -> matplotlib[version='>=2.1.0'] -> python-dateutil
Package ca-certificates conflicts for:
pycocotools -> python[version='>=3.6,<3.7.0a0'] -> ca-certificates
python=3.7 -> openssl[version='>=1.0.2p,<1.0.3a'] -> ca-certificates
Package pip conflicts for:
pycocotools -> python[version='>=3.6,<3.7.0a0'] -> pip
python=3.7 -> pip
Package certifi conflicts for:
python=3.7 -> pip -> setuptools -> certifi[version='>=2016.09']
pycocotools -> cython[version='>=0.27.3'] -> setuptools -> certifi[version='>=2016.09']
Package setuptools conflicts for:
python=3.7 -> pip -> setuptools
pycocotools -> cython[version='>=0.27.3'] -> setuptools
Package wheel conflicts for:
python=3.7 -> pip -> wheel
pycocotools -> python[version='>=3.6,<3.7.0a0'] -> pip -> wheel
any suggestions on resolving this error. please help.

You probably don't have the required sub-modules cider and coco-caption in your workspace. Checkout them ranther than install via conda, and follow the instructions to setup.

@husthuaan thanks i have forget to download these sub modules now it work but giving this error now
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocobu_fc data/cocobu_att data/cocobu_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
warnings.warn(warning.format(ret))
Read data: 0.6928925514221191
iter 0 (epoch 0), train_loss = 6.829, time/batch = 0.954
Read data: 0.11802101135253906
Save ckpt on exception ...
model saved to log/log_aoanet/model.pth
Save ckpt done.
Traceback (most recent call last):
File "train.py", line 169, in train
model_out = dp_lw_model(fc_feats, att_feats, labels, masks, att_masks, data['gts'], torch.arange(0, len(data['gts'])), sc_flag)
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/misc/loss_wrapper.py", line 20, in forward
loss = self.crit(self.model(fc_feats, att_feats, labels, att_masks), labels[:,1:], masks[:,1:])
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/CaptionModel.py", line 34, in forward
return getattr(self, ''+mode)(*args, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/AttModel.py", line 126, in _forward
p_fc_feats, p_att_feats, pp_att_feats, p_att_masks = self._prepare_feature(fc_feats, att_feats, att_masks)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/AoAModel.py", line 211, in _prepare_feature
att_feats = self.refiner(att_feats, att_masks)
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/AoAModel.py", line 124, in forward
x = layer(x, mask)
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/AoAModel.py", line 111, in forward
x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/TransformerModel.py", line 101, in forward
return x + self.dropout(sublayer(self.norm(x)))
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/AoAModel.py", line 111, in
x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))
File "/home/faaiz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/models/AoAModel.py", line 86, in forward
x = x.transpose(1, 2).contiguous()
RuntimeError: CUDA out of memory. Tried to allocate 10.62 MiB (GPU 0; 2.95 GiB total capacity; 1.92 GiB already allocated; 18.19 MiB free; 120.47 MiB cached)
Terminating BlobFetcher
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocobu_fc data/cocobu_att data/cocobu_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Traceback (most recent call last):
File "train.py", line 275, in
train(opt)
File "train.py", line 146, in train
init_scorer(opt.cached_tokens)
File "/home/faaiz/Downloads/AoANet-master(Bottom up)/misc/rewards.py", line 23, in init_scorer
CiderD_scorer = CiderD_scorer or CiderD(df=cached_tokens)
File "cider/pyciderevalcap/ciderD/ciderD.py", line 28, in init
self.cider_scorer = CiderScorer(n=self._n, df_mode=self._df)
File "cider/pyciderevalcap/ciderD/ciderD_scorer.py", line 80, in init
pkl_file = cPickle.load(open(os.path.join('data', df_mode + '.p'),'rb'), **(dict(encoding='latin1') if six.PY3 else {}))
FileNotFoundError: [Errno 2] No such file or directory: 'data/coco-train-idxs.p'
Terminating BlobFetcher
before using bottom-up approach we have tried resnet approach but this same error occur when we were trying to train and with bottom-up approach we get the same error of CUDA out of memory. we have NVIDIA GTX 1060 (3GB memory). thus it required GPU with more memory?

Yes, it requires more memory because of the refining module. train-wo-refining.sh requires less. You can also make --rnn_size, --input_encoding_size smaller so as to train the model on your GPU.

@husthuaan i have set --rnn_size, --input_encoding_size to 512 but it gives this error:
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocobu_fc data/cocobu_att data/cocobu_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Traceback (most recent call last):
File "train.py", line 275, in
train(opt)
File "train.py", line 57, in train
assert vars(saved_model_opt)[checkme] == vars(opt)[checkme], "Command line argument and saved model disagree on '%s' " % checkme
AssertionError: Command line argument and saved model disagree on 'rnn_size'
Terminating BlobFetcher
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocobu_fc data/cocobu_att data/cocobu_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Traceback (most recent call last):
File "train.py", line 275, in
train(opt)
File "train.py", line 57, in train
assert vars(saved_model_opt)[checkme] == vars(opt)[checkme], "Command line argument and saved model disagree on '%s' " % checkme
AssertionError: Command line argument and saved model disagree on 'rnn_size'
Terminating BlobFetcher
i have tried doing this before when i was trying to train using resnet approach but this gives the same error. so what should be the size of these two?
by the way how much GPU memory thus its require?

@husthuaan I have change the size of rnn and input_encoding size to 512 from 1024. It has started training but then stop after some time because of CUDA out of memory error. Can you tell me what should be size of rnn and input_encoding according to our GPU(3GB) and how much GPU memory would be sufficient for this training to complete?

According to my experience, using the training script train.sh requires about 8GB GPU memory, while train-wo-refining.sh requires about 3GB (maybe a little more). 3GB GPU memory should be sufficient for  train-wo-refining.sh  if you set  --rnn_size and  --input_encoding_size to 512 (decrease them if still not works).

@husthuaan  as we have changed sizes 1024 to 512 and now we are using train_wo_refining. Will it affect the result?

@faaizhashmi It will.

@husthuaan training successfully completes 1 epoch on 6000 iterations.   Then starts evaluations. When completed, gives this error.
image 296759: a teddy bear is holding a teddy bear
evaluating validation preformance... -1/5000 (2.549190)
loading annotations into memory...
0:00:00.501229
creating index...
index created!
using 5000/5000 predictions
Loading and preparing results...
DONE (t=0.02s)
creating index...
index created!
tokenization...
PTBTokenizer tokenized 307821 tokens at 759930.85 tokens per second.
PTBTokenizer tokenized 55094 tokens at 545484.58 tokens per second.
setting up scorers...
/home/faaiz/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
Traceback (most recent call last):
  File "train.py", line 275, in <module>
    train(opt)
  File "train.py", line 223, in train
    dp_model, lw_model.crit, loader, eval_kwargs)
  File "/home/faaiz/Downloads/AoANet-master(Bottom up)/eval_utils.py", line 161, in eval_split
    lang_stats = language_eval(dataset, predictions, eval_kwargs['id'], split)
  File "/home/faaiz/Downloads/AoANet-master(Bottom up)/eval_utils.py", line 55, in language_eval
    cocoEval.evaluate()
  File "coco-caption/pycocoevalcap/eval.py", line 51, in evaluate
    (WMD(),   "WMD"),
  File "coco-caption/pycocoevalcap/wmd/wmd.py", line 17, in __init__
    self.model = gensim.models.KeyedVectors.load_word2vec_format(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data', 'GoogleNews-vectors-negative300.bin'), binary=True)
  File "/home/faaiz/anaconda3/lib/python3.7/site-packages/gensim/models/keyedvectors.py", line 1119, in load_word2vec_format
    limit=limit, datatype=datatype)
  File "/home/faaiz/anaconda3/lib/python3.7/site-packages/gensim/models/utils_any2vec.py", line 173, in _load_word2vec_format
    with utils.smart_open(fname) as fin:
  File "/home/faaiz/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py", line 453, in smart_open
    return open(uri, mode, ignore_ext=ignore_extension, transport_params=transport_params, **scrubbed_kwargs)
  File "/home/faaiz/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py", line 307, in open
    errors=errors,
  File "/home/faaiz/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py", line 512, in _shortcut_open
    return _builtin_open(parsed_uri.uri_path, mode, buffering=buffering, **open_kwargs)
FileNotFoundError: [Errno 2] No such file or directory: '/home/faaiz/Downloads/AoANet-master(Bottom up)/coco-caption/pycocoevalcap/wmd/data/GoogleNews-vectors-negative300.bin'
Terminating BlobFetcher
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocobu_fc data/cocobu_att data/cocobu_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Traceback (most recent call last):
  File "train.py", line 275, in <module>
    train(opt)
  File "train.py", line 52, in train
    with open(os.path.join(opt.start_from, 'infos_'+opt.id+'.pkl'), 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'log/log_aoanet-wo-refinng/infos_aoanet-wo-refinng.pkl'
Terminating BlobFetcher

@husthuaan We used train_wo_refining. It successfully runs 1 epoch over 6000 iterations. Then it starts evaluation. Once completed, gives this error
image 203138: a man in a kitchen with a refrigerator in the kitchen
image 296759: a teddy bear with a teddy bear and a stuffed teddy bear
evaluating validation preformance... -1/5000 (2.559475)
loading annotations into memory...
0:00:00.579296
creating index...
index created!
using 5000/5000 predictions
Loading and preparing results...
DONE (t=0.02s)
creating index...
index created!
tokenization...
PTBTokenizer tokenized 307821 tokens at 728546.95 tokens per second.
PTBTokenizer tokenized 53019 tokens at 514423.09 tokens per second.
setting up scorers...
/home/faaiz/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
computing Bleu score...
{'testlen': 48020, 'reflen': 47751, 'guess': [48020, 43020, 38020, 33020], 'correct': [33280, 16787, 7241, 2886]}
ratio: 1.0056333898766308
Bleu_1: 0.693
Bleu_2: 0.520
Bleu_3: 0.372
Bleu_4: 0.259
computing METEOR score...
Traceback (most recent call last):
  File "train.py", line 275, in <module>
    train(opt)
  File "train.py", line 223, in train
    dp_model, lw_model.crit, loader, eval_kwargs)
  File "/home/faaiz/Downloads/AoANet-master(Bottom up)/eval_utils.py", line 161, in eval_split
    lang_stats = language_eval(dataset, predictions, eval_kwargs['id'], split)
  File "/home/faaiz/Downloads/AoANet-master(Bottom up)/eval_utils.py", line 55, in language_eval
    cocoEval.evaluate()
  File "coco-caption/pycocoevalcap/eval.py", line 59, in evaluate
    score, scores = scorer.compute_score(gts, res)
  File "coco-caption/pycocoevalcap/meteor/meteor.py", line 43, in compute_score
    stat = self._stat(res[i][0], gts[i])
  File "coco-caption/pycocoevalcap/meteor/meteor.py", line 67, in _stat
    self.meteor_p.stdin.write(score_line+'\n')
BrokenPipeError: [Errno 32] Broken pipe
Terminating BlobFetcher

@faaizhashmi  This error occurs on some machines. Have a look at https://github.com/tylin/coco-caption/issues/17.

@husthuaan From the Issue given above tylin/coco-caption#17. We tried the first suggestion
> It might be a problem of java.
> Try this:
>
>     Install java on your linux system, if it wasn't installed.
>     In the initialization function of meteor.py:
>
> def __init__(self):
>     self.meteor_cmd = ['java', '-jar', '-Xmx2G', METEOR_JAR, '-', '-', '-stdio', '-l', 'en', '-norm']
>     self.meteor_p = subprocess.Popen(' '.join(self.meteor_cmd),\     # change this line
>         cwd=os.path.dirname(os.path.abspath(__file__)), \
>         stdin=subprocess.PIPE, \
>         stdout=subprocess.PIPE, \
>         stderr=subprocess.PIPE,\
>         shell=True)
>         # Used to guarantee thread safety
>         self.lock = threading.Lock()

It did not give broken pipe error but gave this instead
image 203138: a man standing in front of a refrigerator in a kitchen
image 296759: a teddy bear with a teddy bear and a teddy bear
evaluating validation preformance... -1/5000 (2.585238)
loading annotations into memory...
0:00:00.447847
creating index...
index created!
using 5000/5000 predictions
Loading and preparing results...
DONE (t=0.02s)
creating index...
index created!
tokenization...
PTBTokenizer tokenized 307821 tokens at 917630.90 tokens per second.
PTBTokenizer tokenized 54337 tokens at 494168.22 tokens per second.
setting up scorers...
/home/faaiz/anaconda3/lib/python3.7/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function
  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL
computing Bleu score...
{'testlen': 49338, 'reflen': 48664, 'guess': [49338, 44338, 39338, 34338], 'correct': [33896, 17323, 7532, 3038]}
ratio: 1.0138500739766354
Bleu_1: 0.687
Bleu_2: 0.518
Bleu_3: 0.372
Bleu_4: 0.260
computing METEOR score...
Traceback (most recent call last):
  File "train.py", line 275, in <module>
    train(opt)
  File "train.py", line 223, in train
    dp_model, lw_model.crit, loader, eval_kwargs)
  File "/home/faaiz/Downloads/AoANet-master(Bottom up)/eval_utils.py", line 161, in eval_split
    lang_stats = language_eval(dataset, predictions, eval_kwargs['id'], split)
  File "/home/faaiz/Downloads/AoANet-master(Bottom up)/eval_utils.py", line 55, in language_eval
    cocoEval.evaluate()
  File "coco-caption/pycocoevalcap/eval.py", line 59, in evaluate
    score, scores = scorer.compute_score(gts, res)
  File "coco-caption/pycocoevalcap/meteor/meteor.py", line 40, in compute_score
    stat = self._stat(res[i][0], gts[i])
  File "coco-caption/pycocoevalcap/meteor/meteor.py", line 64, in _stat
    self.meteor_p.stdin.write(score_line+'\n')
TypeError: a bytes-like object is required, not 'str'
Terminating BlobFetcher

i tried another suggestion which was just to replace the line in initialization function of meteor.py:

I had the same problem before as well (but I was using Python 3 version from this repo).
My fix was adding the data file when running java cmd. Hope this helps.
self.meteor_cmd = ['java', '-jar', '-Xmx2G', METEOR_JAR, '-', '-', '-stdio', '-l', 'en', '-norm', '-a', 'data/paraphrase-en.gz']

This is weird. You can temporarily avoid computing these metrics by commenting these lines in coco-caption/pycocoevalcap/eval.py, line 45-52:
        scorers = [
            (Bleu(4), ["Bleu_1", "Bleu_2", "Bleu_3", "Bleu_4"]),
            # (Meteor(),"METEOR"),
            (Rouge(), "ROUGE_L"),
            (Cider(), "CIDEr"),
            (self.Spice, "SPICE"),
            # (WMD(),   "WMD"),
        ]

@husthuaan  Thank you for all the cooperation. We have successfully been able to run 34 epochs. Now our programs gives this error.
Read data: 0.37630248069763184
iter 396504 (epoch 34), train_loss = 3.060, time/batch = 0.125
Terminating BlobFetcher
DataLoader loading json file:  data/cocotalk.json
vocab size is  9487
DataLoader loading h5 file:  data/cocobu_fc data/cocobu_att data/cocobu_box data/cocotalk_label.h5
max sequence length in data is 16
read 123287 image features
assigned 113287 images to split train
assigned 5000 images to split val
assigned 5000 images to split test
Traceback (most recent call last):
  File "train.py", line 275, in <module>
    train(opt)
  File "train.py", line 146, in train
    init_scorer(opt.cached_tokens)
  File "/home/faaiz/Downloads/AoANet-master_Bottom up/misc/rewards.py", line 23, in init_scorer
    CiderD_scorer = CiderD_scorer or CiderD(df=cached_tokens)
  File "cider/pyciderevalcap/ciderD/ciderD.py", line 28, in __init__
    self.cider_scorer = CiderScorer(n=self._n, df_mode=self._df)
  File "cider/pyciderevalcap/ciderD/ciderD_scorer.py", line 80, in __init__
    pkl_file = cPickle.load(open(os.path.join('data', df_mode + '.p'),'rb'), **(dict(encoding='latin1') if six.PY3 else {}))
FileNotFoundError: [Errno 2] No such file or directory: 'data/coco-train-idxs.p'
Terminating BlobFetcher

We tried to find the file named

coco-train-idxs.p

. But we keep on ending up on the different coco datasets directories and pages. No file identical to this is found. Do you have any idea regarding this problem?

Please execute the following command to generate coco-train-idxs.p.
$ python scripts/prepro_ngrams.py --input_json data/dataset_coco.json --dict_json data/cocotalk.json --output_pkl data/coco-train --split train

Sorry for my negligence and I will update README. Thank you.

@husthuaan we were able to run 34 epochs and there was no error but only above file was missing. Does this means that training is done or we do we have to start over again after adding the file coco-train-idxs.p

The xe training stage is done. You don't need to start over again. The training script will load the history checkpoint.

authors: faaizhashmi husthuaan

repo: husthuaan/AoANet