text:
The self-test fails and the message "fixture'xxx_file' not found" is displayed

I meet l problem about the self-test fails and the message "fixture'xxx_file' not found" is displayed.
See:
[root@centos8 brotlicffi]#ls
build  HISTORY.rst  libbrotli  LICENSE  MANIFEST.in  PKG-INFO  README.rst  setup.cfg  setup.py  src  test
[root@centos8 brotlicffi]#pytest -v
======================================================================================== test session starts =========================================================================================
platform linux -- Python 3.7.7, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/spack/spack/opt/spack/linux-centos8-aarch64/gcc-8.3.1/python-3.7.7-q43i5iy6mcmisbvj2szzkhehzuunjm5b/bin/python3.7
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/.hypothesis/examples')
rootdir: /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src
plugins: hypothesis-6.3.4
collected 23 items

test/test_compatibility.py::test_compatible_names PASSED                                                                                [  4%]
test/test_simple_compression.py::test_roundtrip_compression_with_files ERROR                                                                                [  8%]
test/test_simple_compression.py::test_streaming_compression ERROR                                                                                [ 13%]
test/test_simple_compression.py::test_streaming_compression_flush ERROR                                                                                [ 17%]
test/test_simple_compression.py::test_compressed_data_roundtrips PASSED                                                                                [ 21%]
test/test_simple_compression.py::test_compressed_data_with_dictionaries PASSED                                                                                [ 26%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params0] PASSED                                                                                [ 30%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params1] PASSED                                                                                [ 34%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params2] PASSED                                                                                [ 39%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params3] PASSED                                                                                [ 43%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params0] PASSED                                                                                [ 47%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params1] PASSED                                                                                [ 52%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params2] PASSED                                                                                [ 56%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params3] PASSED                                                                                [ 60%]
test/test_simple_decompression.py::test_decompression ERROR                                                                                [ 65%]
test/test_simple_decompression.py::test_decompressobj ERROR                                                                                [ 69%]
test/test_simple_decompression.py::test_drip_feed ERROR                                                                                [ 73%]
test/test_simple_decompression.py::test_streaming_decompression_fails_properly_on_garbage[Error0] PASSED                                                                                [ 78%]
test/test_simple_decompression.py::test_streaming_decompression_fails_properly_on_garbage[Error1] PASSED                                                                                [ 82%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[some random garbage-Error0] PASSED                                                                             [ 86%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[some random garbage-Error1] PASSED                                                                             [ 91%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[bogus-Error0] PASSED                                                                                [ 95%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[bogus-Error1] PASSED                                                                                [100%]

=============================================================================================== ERRORS ===============================================================================================
______________________________________________________________________ ERROR at setup of test_roundtrip_compression_with_files _______________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py, line 16
  def test_roundtrip_compression_with_files(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py:16
____________________________________________________________________________ ERROR at setup of test_streaming_compression ____________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py, line 28
  @given(
      chunk_size=integers(min_value=1, max_value=2**12),
      mode=sampled_from(list(brotli.BrotliEncoderMode)),
      quality=integers(min_value=0, max_value=11),
      lgwin=integers(min_value=10, max_value=24),
      lgblock=one_of(
          integers(min_value=0, max_value=0),
          integers(min_value=16, max_value=24)
      ),
  )
  def test_streaming_compression(one_compressed_file,
E       fixture 'one_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py:28
_________________________________________________________________________ ERROR at setup of test_streaming_compression_flush _________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py, line 65
  @given(
      chunk_size=integers(min_value=1, max_value=2**12),
      mode=sampled_from(list(brotli.BrotliEncoderMode)),
      quality=integers(min_value=0, max_value=11),
      lgwin=integers(min_value=10, max_value=24),
      lgblock=one_of(
          integers(min_value=0, max_value=0),
          integers(min_value=16, max_value=24)
      ),
  )
  def test_streaming_compression_flush(one_compressed_file,
E       fixture 'one_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py:65
________________________________________________________________________________ ERROR at setup of test_decompression ________________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py, line 13
  def test_decompression(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py:13
________________________________________________________________________________ ERROR at setup of test_decompressobj ________________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py, line 26
  def test_decompressobj(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py:26
__________________________________________________________________________________ ERROR at setup of test_drip_feed __________________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py, line 41
  def test_drip_feed(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py:41
====================================================================================== short test summary info =======================================================================================
ERROR test/test_simple_compression.py::test_roundtrip_compression_with_files
ERROR test/test_simple_compression.py::test_streaming_compression
ERROR test/test_simple_compression.py::test_streaming_compression_flush
ERROR test/test_simple_decompression.py::test_decompression
ERROR test/test_simple_decompression.py::test_decompressobj
ERROR test/test_simple_decompression.py::test_drip_feed
==================================================================================== 17 passed, 6 errors in 0.95s ====================================================================================
[root@centos8 spack-src]# ls
build  HISTORY.rst  libbrotli  LICENSE  MANIFEST.in  PKG-INFO  README.rst  setup.cfg  setup.py  src  test
[root@centos8 spack-src]# pytest -v
======================================================================================== test session starts =========================================================================================
platform linux -- Python 3.7.7, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/spack/spack/opt/spack/linux-centos8-aarch64/gcc-8.3.1/python-3.7.7-q43i5iy6mcmisbvj2szzkhehzuunjm5b/bin/python3.7
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/.hypothesis/examples')
rootdir: /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src
plugins: hypothesis-6.3.4
collected 23 items

test/test_compatibility.py::test_compatible_names PASSED                                                                                [  4%]
test/test_simple_compression.py::test_roundtrip_compression_with_files ERROR                                                                                [  8%]
test/test_simple_compression.py::test_streaming_compression ERROR                                                                                [ 13%]
test/test_simple_compression.py::test_streaming_compression_flush ERROR                                                                                [ 17%]
test/test_simple_compression.py::test_compressed_data_roundtrips PASSED                                                                                [ 21%]
test/test_simple_compression.py::test_compressed_data_with_dictionaries PASSED                                                                                [ 26%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params0] PASSED                                                                                [ 30%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params1] PASSED                                                                                [ 34%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params2] PASSED                                                                                [ 39%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error0-params3] PASSED                                                                                [ 43%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params0] PASSED                                                                                [ 47%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params1] PASSED                                                                                [ 52%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params2] PASSED                                                                                [ 56%]
test/test_simple_compression.py::test_bad_compressor_parameters[Error1-params3] PASSED                                                                                [ 60%]
test/test_simple_decompression.py::test_decompression ERROR                                                                                [ 65%]
test/test_simple_decompression.py::test_decompressobj ERROR                                                                                [ 69%]
test/test_simple_decompression.py::test_drip_feed ERROR                                                                                [ 73%]
test/test_simple_decompression.py::test_streaming_decompression_fails_properly_on_garbage[Error0] PASSED                                                                                [ 78%]
test/test_simple_decompression.py::test_streaming_decompression_fails_properly_on_garbage[Error1] PASSED                                                                                [ 82%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[some random garbage-Error0] PASSED                                                                             [ 86%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[some random garbage-Error1] PASSED                                                                             [ 91%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[bogus-Error0] PASSED                                                                                [ 95%]
test/test_simple_decompression.py::test_decompression_fails_properly_on_garbage[bogus-Error1] PASSED                                                                                [100%]

=============================================================================================== ERRORS ===============================================================================================
______________________________________________________________________ ERROR at setup of test_roundtrip_compression_with_files _______________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py, line 16
  def test_roundtrip_compression_with_files(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py:16
____________________________________________________________________________ ERROR at setup of test_streaming_compression ____________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py, line 28
  @given(
      chunk_size=integers(min_value=1, max_value=2**12),
      mode=sampled_from(list(brotli.BrotliEncoderMode)),
      quality=integers(min_value=0, max_value=11),
      lgwin=integers(min_value=10, max_value=24),
      lgblock=one_of(
          integers(min_value=0, max_value=0),
          integers(min_value=16, max_value=24)
      ),
  )
  def test_streaming_compression(one_compressed_file,
E       fixture 'one_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py:28
_________________________________________________________________________ ERROR at setup of test_streaming_compression_flush _________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py, line 65
  @given(
      chunk_size=integers(min_value=1, max_value=2**12),
      mode=sampled_from(list(brotli.BrotliEncoderMode)),
      quality=integers(min_value=0, max_value=11),
      lgwin=integers(min_value=10, max_value=24),
      lgblock=one_of(
          integers(min_value=0, max_value=0),
          integers(min_value=16, max_value=24)
      ),
  )
  def test_streaming_compression_flush(one_compressed_file,
E       fixture 'one_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_compression.py:65
________________________________________________________________________________ ERROR at setup of test_decompression ________________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py, line 13
  def test_decompression(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py:13
________________________________________________________________________________ ERROR at setup of test_decompressobj ________________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py, line 26
  def test_decompressobj(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py:26
__________________________________________________________________________________ ERROR at setup of test_drip_feed __________________________________________________________________________________
file /tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py, line 41
  def test_drip_feed(simple_compressed_file):
E       fixture 'simple_compressed_file' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/root/spack-stage/spack-stage-py-brotlipy-0.7.0-4vumgh3ckr6pcfwy6ffd7cqp22hbglfx/spack-src/test/test_simple_decompression.py:41
====================================================================================== short test summary info =======================================================================================
ERROR test/test_simple_compression.py::test_roundtrip_compression_with_files
ERROR test/test_simple_compression.py::test_streaming_compression
ERROR test/test_simple_compression.py::test_streaming_compression_flush
ERROR test/test_simple_decompression.py::test_decompression
ERROR test/test_simple_decompression.py::test_decompressobj
ERROR test/test_simple_decompression.py::test_drip_feed
==================================================================================== 17 passed, 6 errors in 0.93s ====================================================================================

I want to know how to solve the problem of "fixture'xxx_file' not found. Could you help analyze it?

This is an issue tracker for brotlicffi, I cannot provide support for how our test suite runs. I recommend doing more investigation yourself.

repo: python-hyper/brotlicffi


text:
Documentation!

Now that this is its own module we should really add some docs for it.

This is done.

source: gharchive/issue

repo: python-hyper/hyperframe

license: mit

license_type: permissive

license_source: bigquery


text:
Optimization of Frame parsing/serialization.

Hello, I am here to provide some ideas of potential optimizations to Frame parsing and serialization at the suggestion of @Lukasa .  All of the following optimizations have exactly the same behavior (given a few axioms) but are much faster than their counterparts.
These optimizations are on the microsecond level, but because this is a low-level library intended to be used within tight loops every cycle counts. Luckily these optimizations (in my opinion) do not degrade readability and if they do could be justified with a single comment.
All optimizations are tested using the timeit module, if you would like you may test them yourself and see the results. I have categorized the optimizations into Less aggressive and Aggressive.
If this sort of optimization is welcome, I would also like to perhaps optimize the use of flags within this library and it's dependents (hpack, h2) as the current system for flagging is quite slow.
Less aggressive optimization:

True if x else False is faster than bool(x) (318% speedup)
Frames with a defined length can be assigned that length immediately after parsing rather than checking the length again. (290% speedup)
True if x >> 31 else False is faster than bool(x & 0x80000000) as long as the integer is masked. (204% speedup)
struct.Struct("!XX").[un]pack(x) is faster than calling struct.[un]pack("!XX", x) repeatedly as long as struct format stays the same. (52% speedup)
x + y is faster than x | y as long as both integers are masked. (27% speedup)
struct.Struct(">XX") [un]packs faster than struct.Struct("!XX") and means the same thing. (14% speedup)
x & 0xFF is faster than x & 0x0000FF because of how Python bit logic works. (13% speedup)

Aggressive optimization:

Stream association per-frame can be checked faster if the same string is used rather than a new string for each frame. (2% speedup)
(x >> 8) & 0xFFFF is faster than (x & 0xFFFF00) >> 8 (2% speedup)

@Lukasa No thank you for this awesome project! I will be hanging around (if I'm not being too much) and picking up things to do. If proper benchmarks haven't been performed yet I would be okay with setting something up as well.

You're not being too much at all, any and all assistance is welcomed! In fact, I was about to comment to mention to you that, per our one of the team policy, you should find an invite to the Python Hyper organisation in your email inbox. Feel free to dive in and lend your assistance on code reviews and generally make yourself as useful as you can!
In general, performance improvements will be welcomed so long as the penalty they cause on readability is paid for in their gain in performance. If you're going to be excited to write benchmarks then you should totally go ahead and do that, I'd love to be able to track performance and perf regressions and I never really got around to doing it: pedantically reading RFCs and writing protocol stacks is more my jam, I'm not the greatest perf person in the world.
So once again, thanks for contributing!

@Lukasa Of course, I always keep readability in mind, but if I go off the deep end make sure you call me out! :wink: I'll give it a read, thanks again!

Of course! And IMO the best work comes out of people with different priorities working together, so it'll be nice to have someone looking over my shoulder who has an eye on performance. =D

@Lukasa According to a preliminary benchmark I just ran, serializing 1,000,000 random DataFrame's is about 3.5% faster with the minimal changes to DataFrame. Performance increases will be even more impactful with other frames. I'll submit a PR with all my benchmarks once I make them for all frame types and both serializing and parsing.

source: gharchive/pull-request

authors: Lukasa SethMichaelLarson


text:
adds support for reading/writing Run text color

The patch adds a 'color' attribute to a Run so that you can change the foreground text color.

Hi Erik, this feature was added in version 0.8.3 today, Feb 19, 2015. We took a different approach to accommodate a broader API, but thanks for the PR all the same :)
The user docs on the color API are here:
http://python-docx.readthedocs.org/en/latest/user/text.html#font-color

authors: scanny smartt

repo: python-openxml/python-docx


text:
Set GIT_DIR env when checking for git

This prevents poetry from traversing beyond project directory
Currently 2 tests are failing, because they expect to be in a git repo (in this case, poetry-core); TODO: make them have their own repo, which should have been the case from the start
Resolves: python-poetry/poetry#1946

[ ] Added tests for changed code.
[ ] Updated documentation for changed code.

I'm not fully convinced this is the right solution -- can you further explain why this is advantageous?

Note that this will be ripped out soon. We should really replace git ignore parsing with a pure python solution.

the current behaviour is in fact correct; the reason i thought this should be fixed is because when you try to run poetry build in a directory that's ignored by git entirely, it just outputs an empty wheel file with no python files inside it. if the gitignore parsing is getting reworked anyway, i hope this will be fixed

authors: abn neersighted ptrcnull

repo: python-poetry/poetry-core


text:
maintenance tasks

The purerpc repo was transferred to python-trio.  Here are the short-term tasks I'm intending, with the goal of pushing a maintenance release to pypi for active dependees that may be having problems with the old anyio 1.x dependency of this package:

[ ] note on README crediting original author, explain that the project is in maintenance mode (keep things working for active dependees), point to https://github.com/grpc/grpc for people needing asyncio only
[x] misc: doc updates, TODO's I noted while walking through the code, prune unused imports
[x] set up CI (travis-ci.org is defunct, use github actions), pin requirements for reproducible tests
[ ] upgrade to anyio 3.x.  It does imply that curio support will be dropped.  Related to the upgrade, Server.serve() will be strongly discouraged in favor of a newly-added Server.serve_async().  PR #33
[ ] remove runtime patching of anyio.run().  This was to support backend="uvloop" (run() already supports use_uvloop backend-specific option, use the correct way) and the PURERPC_BACKEND environment variable (introduce something like purerpc_run()).
[ ] tag release and push to pypi

I also seem to have used private APIs like anyio._networking and anyio._get_async_lib() back then to get around some issues. I guess we should get rid of this to maintain better compatibility.
Also, maybe there is a way we can get rid of hyper-h2 monkey patches?

additionally:  drop Python 3.5 and 3.6 support, since they are past end of life, and it's a large burden to the implementation verbosity and performance (async_generator shim, etc.)

released v0.7.0

repo: python-trio/purerpc

license_source: github-api


text:
loop.run_in_executor should be a coroutine

Since base_events.BaseEventLoop.run_in_executor returns a Future object, a caller can call it with yield from/await. However, the result of the call is not a coroutine since asyncio.iscoroutine(loop.run_in_executor(...)) returns False.
It matters when one wants to use run_in_executor() in a task, such as:
loop.create_task(loop.run_in_executor(...))
In this case, an exception is raised immediatly (AssertionError), while the task is effectively running in the executor.
This patch propose to make loop.run_in_executor() be an actual coroutine function (and be tested).
I believe that returning a Future and document the function as a coroutine used to be done quite often, maybe this should be fixed elsewhere too.
An alternative solution would be to change the documentation, explain why it can not be used with loop.create_task() and should be used with loop.ensure_future() instead.

There are too many places inside asyncio where coroutine is technically a regular function returning Future instance.
I think it's perfectly fine.
If you want to make doc update -- you are welcome.

I don't think it's perfectly fine: it creates an inconsistency between coroutines and other awaitable objects which leads to confusion for the user and subtle bugs.
Most of the time, loop.create_task() and ensure_future()/async() are presented as similar, but loop.create_task() can handle coroutines and not other awaitable objects. If this difference of semantics is enforced on purpose (coroutines and tasks imply work to be done, while a future only represents a result), I think the implementation of asyncio has to be consistent with this semantic.
There are subtle differences between a function returning a future and a coroutine. For instance, a coroutine is executed only when awaited on (i.e. next(coro)), while a function returning a future is executed when invoked. In the case of loop.run_in_executor(), work starts on a thread (that's a pretty significant side effect) even if the loop never runs.

You've really got this backwards. The most general concept here is not the Task but the Future (which is a base class of Task). If you have either a coroutine or a Future and you want a unified API, you should call ensure_future(), not create_task(). If you already have a Future there's no point in trying to wrap it into a Task, nor to try to make a coroutine into it -- that's backwards.
There is no philosophical difference between Task/coroutine and Future in the way you assume.
If you really have a use case where it matters to you that the action doesn't get started until the loop runs (which seems suspicious to me) you will have to implement this in some other way.

authors: Martiusweb asvetlov gvanrossum

repo: python/asyncio


text:
🛑 Account Info is down

In ac8d9f3, Account Info (https://accountinformation.roblox.com/) was down:

HTTP code: 404
Response time: 207 ms

Resolved: Account Info is back up in 16c9a1b after 3 hours, 33 minutes.

repo: pythoniaweb/roblox-status

license: MIT


text:
🛑 Billing Api is down

In 9fc7ff9, Billing Api (https://billing.roblox.com/) was down:

HTTP code: 404
Response time: 207 ms

Resolved: Billing Api is back up in b89909f after 21 minutes.

repo: pythoniaweb/statusboom


text:
🛑 Games Api is down

In 4fbe219, Games Api (https://games.roblox.com/) was down:

HTTP code: 404
Response time: 84 ms

Resolved: Games Api is back up in 056809d after 2 hours, 20 minutes.


text:
🛑 Dev Forum is down

In 3d548c6, Dev Forum (https://devforum.roblox.com/) was down:

HTTP code: 403
Response time: 170 ms

Resolved: Dev Forum is back up in b2db2dd after 11 minutes.


text:
Dev

yolo

Dockerfile is intentionally broken. Missing a -y here

license: Unlicense


text:
chore(main): release 1.6.5

:robot: I have created a release beep boop
1.6.5 (2024-10-17)
Documentation

fix add-record example (a28d359)
update readme (2e5fd83)

Miscellaneous Chores

release 1.6.4a0 (595c962)
release 1.6.4a1 (52fe9ba)
release 1.6.4a2 (20853fa)
release 1.6.5 (670a4fc)
release 1.7.0 (5e18182)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/pythoninthegrass/gopappy/releases/tag/v1.6.5 :sunflower:

repo: pythoninthegrass/gopappy


text:
❓ [Question] How can I install the latest version of python API? Torch and Tensorrt's CUDA dependencies conflict with each other.

❓ Question

What you have already tried

I have already create a python=3.9 env, when I use the command 'pip install torch-tensorrt', I find that the torch version that the latest torch-tensorrt needs is 2.0.1 and the tensorrt version it needs is 8.6.1, but these two packages need different cuda versions(which one is cu11 and another is cu12). When I run a simple model(input) example python code, torch can't resolve the environment.
Environment

Build information about Torch-TensorRT can be found by turning on debug messages

PyTorch Version (e.g., 1.0): 2.0.1
CPU Architecture: x86_64
OS (e.g., Linux): linux
How you installed PyTorch (conda, pip, libtorch, source): pip
Build command you used (if compiling from source): pip install torch-tensorrt
Are you using local sources or building from archives: archives
Python version: 3.9
tensorrt version: 8.6.1

Additional context
example python code:
import torch
#import torch_tensorrt #No tensorrt and torch_tensorrt installed, this code will run successfully.
conv=torch.nn.Conv2d(3,32,3,1,0,bias=False)
input=torch.randn(1,3,224,224)
conv.cuda()
input.cuda()
print(conv(input).shape)

Can you post the error you are seeing?

before I install the tensorrt, the example code can be run succsessfully, but after installing, here will report an error:
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Can you post the error you are seeing?

This problem maybe has been solved. I firstly install tensorrt==8.6.1, and then install torch==2.0.1, which is ok.

There is another problem, when I use function torch_tensorrt.compile(*) to compile a torch ts model,  it shows WARNINGs and auto exits running. WARNING info is as follows,
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Unable to determine GPU memory usage
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Unable to determine GPU memory usage
WARNING: [Torch-TensorRT TorchScript Conversion Context] - CUDA initialization failure with error: 35. Please check your CUDA installation:  http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

The env packages and versions are as follows,
nvidia-cublas-cu11       11.10.3.66
nvidia-cublas-cu117      11.10.1.25
nvidia-cublas-cu12       12.1.3.1
nvidia-cuda-cupti-cu11   11.7.101
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu11        8.5.0.96
nvidia-cudnn-cu12        8.9.2.26
nvidia-cufft-cu11        10.9.0.58
nvidia-curand-cu11       10.2.10.91
nvidia-cusolver-cu11     11.4.0.1
nvidia-cusparse-cu11     11.7.4.91
nvidia-nccl-cu11         2.14.3
nvidia-nvtx-cu11         11.7.91
packaging                23.1
safetensors              0.3.1
tensorrt                 8.6.1
tensorrt-bindings        8.6.1
tensorrt-libs            8.6.1
torch                    2.0.1
torch-tensorrt           1.4.0
torchvision              0.15.2

What driver are you running? Can you share the output of nvidia-smi?

What driver are you running? Can you share the output of nvidia-smi?
The driver and cuda version are as follows,
NVIDIA-SMI 470.42.01    Driver Version: 470.42.01    CUDA Version: 11.4

The GPU is a Tesla T4.

repo: pytorch/TensorRT


text:
Levenshtien Transformer ideal Hyperparameters

Hey can you please suggest appropriate hyperparameters for training on 4 GPUs,with both normal and FP-16 Version.
Also what would be the parameters if I am using SIngle GPU and FP-16 to achieve same accuracy,should they be same as Base or should they be changed.
@myleott @MultiPath @kahne

Hyperparameters typically need to be tuned yourself, depending on the dataset.
FP16 shouldn't require changing any hyperparameters compared to FP32.
When training on fewer GPUs you need to use --update-freq to match the batch size.

Yeah but even that is giving CUDA : Memory error
This is what I am getting
Hi I am getting CUDA ERROR :OUT OF MEMORY.
I am using 4 V100 GPUS.KIndly please suggest appropriate parameters
(base) ubuntu@ip-172-31-8-60:~/fairseq$ fairseq-train data-bin/org --save-dir checkpoints/lv --ddp-backend=no_c10d --task translation_lev --criterion nat_loss --arch levenshtein_transformer --noise random_delete --share-all-embeddings --optimizer adam --adam-betas '(0.9,0.98)' --lr 0.0005 --lr-scheduler inverse_sqrt --min-lr '1e-09' --warmup-updates 10000 --warmup-init-lr '1e-07' --label-smoothing 0.1 --dropout 0.3 --weight-decay 0.01 --decoder-learned-pos --encoder-learned-pos --apply-bert-init --log-format 'simple' --log-interval 100 --fixed-validation-seed 7 --max-tokens 8000 --save-interval-updates 10000
| distributed init (rank 3): tcp://localhost:12081
| distributed init (rank 2): tcp://localhost:12081
| distributed init (rank 1): tcp://localhost:12081
| distributed init (rank 0): tcp://localhost:12081
| initialized host ip-172-31-8-60 as rank 3
| initialized host ip-172-31-8-60 as rank 2
| initialized host ip-172-31-8-60 as rank 1
| initialized host ip-172-31-8-60 as rank 0
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/bin/fairseq-train", line 11, in
load_entry_point('fairseq', 'console_scripts', 'fairseq-train')()
File "/home/ubuntu/fairseq/fairseq_cli/train.py", line 329, in cli_main
nprocs=args.distributed_world_size,
File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 3 terminated with the following error:
Traceback (most recent call last):
File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/ubuntu/fairseq/fairseq_cli/train.py", line 296, in distributed_main
main(args, init_distributed=True)
File "/home/ubuntu/fairseq/fairseq_cli/train.py", line 35, in main
args.distributed_rank = distributed_utils.distributed_init(args)
File "/home/ubuntu/fairseq/fairseq/distributed_utils.py", line 91, in distributed_init
dist.all_reduce(torch.zeros(1).cuda())
RuntimeError: CUDA error: out of memory

Please run nvidia-smi. You probably have other zombie processes using your GPU memory.

@spprabhu You should use --fp16 in order to run on --max-tokens 8000
In order to increase the batch-size, you can set --update-freq larger than 1 to enable gradient accumulation. It will simulate a bigger batch size.
For example, if you set --max-tokens 8000 --update-freq 8 on single GPU, then you get batch-size as 64K which is the number we used in the paper,

Closing due to inactivity.

authors: MultiPath lematt1991 myleott spprabhu

repo: pytorch/fairseq


text:
Test functorch transform behavior on BatchNorm and other weird in-place operations

Most in-place operations look the same: the first argument is mutable and no other arguments are mutable.
F.batch_norm is an exception to that rule: it has multiple mutable arguments. We should check that our transforms work correctly on it.

Hi @zou3519 , is this error related to the current issue? It seems all models with batchnorm can't be used for jacobian calculation.
import torch
import torch.nn as nn
import functorch
from torch.nn.utils import _stateless
import functools

model = nn.Sequential(nn.Linear(3, 4), nn.BatchNorm1d(4), nn.Linear(4, 5), nn.Softmax(dim=-1))
_input = torch.randn(32, 3)
weight = model[0].weight.clone()
weight.requires_grad_()

def func(param: torch.Tensor, _input: torch.Tensor = None):
    _output: torch.Tensor = _stateless.functional_call(
        model, {'0.weight': param}, _input)
    return _output  # (N, C)

jac: torch.Tensor = functorch.jacrev(functools.partial(func, _input=_input))(weight)

Traceback (most recent call last):
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 33, in <module>
    jac: torch.Tensor = functorch.jacrev(functools.partial(func, _input=_input))(weight)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 441, in wrapper_fn
    vjp_out = vjp(f_wrapper, *primals, has_aux=has_aux)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 270, in vjp
    primals_out = func(*diff_primals)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 624, in f_wrapper
    return f(*replaced_args)
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 28, in func
    _output: torch.Tensor = _stateless.functional_call(
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\utils\stateless.py", line 141, in functional_call
    out = module(args, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
    input = module(input)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\batchnorm.py", line 148, in forward
    self.num_batches_tracked.add_(1)  # type: ignore[has-type]
RuntimeError: During a grad (vjp, jvp, grad, etc) transform, the function provided attempted to call in-place operation (aten::add_.Tensor) that would mutate a captured Tensor. This is not supported; please rewrite the function being transformed to explicitly accept the mutated Tensor(s) as inputs.

Additionally, even if I set the model.eval() to avoid previous issue, I still can't apply vmap onto it.

def vmap_func(_input: torch.Tensor):
    return functorch.jacrev(functools.partial(func, _input=_input.unsqueeze(0)))(weight)

jac = functorch.vmap(vmap_func)(_input)

Traceback (most recent call last):
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 41, in <module>
    jac = functorch.vmap(vmap_func)(_input)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\vmap.py", line 365, in wrapped
    batched_outputs = func(*batched_inputs, **kwargs)
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 38, in vmap_func
    return functorch.jacrev(functools.partial(func, _input=_input.unsqueeze(0)))(weight)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 441, in wrapper_fn
    vjp_out = vjp(f_wrapper, *primals, has_aux=has_aux)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 270, in vjp
    primals_out = func(*diff_primals)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 624, in f_wrapper
    return f(*replaced_args)
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 31, in func
    _output: torch.Tensor = _stateless.functional_call(
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\utils\stateless.py", line 141, in functional_call
    out = module(args, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
    input = module(input)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\batchnorm.py", line 168, in forward
    return F.batch_norm(
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\functional.py", line 2438, in batch_norm
    return torch.batch_norm(
RuntimeError: Batch norm got a batched tensor as input while the running_mean or running_var, which will be updated in place, were not batched.
If you are using a module and do not need eval mode, please set `track_running_stats` to be False.If you are using a prebuilt module and do not need eval mode, please see the functorch website for resources on how to patch your module to work with vmap

cc @samdow if there's more you want to say about batchnorm.

It seems all models with batchnorm can't be used for jacobian calculation.

The error message is complaining that the buffers to batch_norm are not being passed into the function func that is being transformed (func only accepts param and _input). Please rewrite func to also accept and use the buffers.

Additionally, even if I set the model.eval() to avoid previous issue, I still can't apply vmap onto it.

Setting the model to eval() works for me:
import torch
import torch.nn as nn
import functorch
from torch.nn.utils import _stateless
import functools

model = nn.Sequential(nn.Linear(3, 4), nn.BatchNorm1d(4), nn.Linear(4, 5), nn.Softmax(dim=-
1))
model.eval()

_input = torch.randn(32, 3)
weight = model[0].weight.clone()
weight.requires_grad_()

def func(param: torch.Tensor, _input: torch.Tensor = None):
    _output: torch.Tensor = _stateless.functional_call(
        model, {'0.weight': param}, _input)
    return _output  # (N, C)

jac: torch.Tensor = functorch.jacrev(functools.partial(func, _input=_input))(weight)

def vmap_func(_input: torch.Tensor):
    return functorch.jacrev(functools.partial(func, _input=_input.unsqueeze(0)))(weight)

jac = functorch.vmap(vmap_func)(_input)

Btw, @ain-soph -- I noticed you're using stateless.functional_call rather than functorch.make_functional_with_buffers. We're trying to figure out how to consolidate those two APIs, given that they do almost the same thing. Is there one that you prefer over the other?

@zou3519 Thanks for you kind explanation.

Btw, @ain-soph -- I noticed you're using stateless.functional_call rather than functorch.make_functional_with_buffers. We're trying to figure out how to consolidate those two APIs, given that they do almost the same thing. Is there one that you prefer over the other?

Personally, I prefer stateless.functional_call because they allow me to do a dict match, while make_functional requires me to match by list index for params. When there is a huge model and I only want to modify one layer weight, it'll be difficult to use make_functional.
However, stateless.functional_call has a big problem because it can't use nn.DataParallel to utilize multiple GPUs. I don't know if there is any solution yet because it requires the input module as nn.Module.

@zou3519 And I just checked the previously mentioned vmap snippet in release v0.2.0, it still raises the issue I posted.
Not quite sure if it's get solved in master though.

@samdow I’m currently calculating NTK matrix for each input, and making average over the batch. Then I use the largest rig value divided by the smallest one of The averaged NTK matrix as a score function to optimize model (arch) parameters.
This is somewhat motivated by the TENAS paper.
I’ll provide a minimal snippet here.

I use a trick to disable track_running_stats for all batchnorm modules. However, applying vmap is still problematic.
for a in model.modules():
    if isinstance(a, nn.modules.batchnorm._BatchNorm):
        if a.track_running_stats:
            a.track_running_stats = False
            a.register_buffer("running_mean", None)
            a.register_buffer("running_var", None)
            a.register_buffer("num_batches_tracked", None)

The snippet becomes now
import torch
import torch.nn as nn
import functorch
from torch.nn.utils import _stateless
import functools

model = nn.Sequential(nn.Linear(3, 4), nn.BatchNorm1d(4), nn.Linear(4, 5), nn.Softmax(dim=-
1))
model.eval()

for a in model.modules():
    if isinstance(a, nn.modules.batchnorm._BatchNorm):
        if a.track_running_stats:
            a.track_running_stats = False
            a.register_buffer("running_mean", None)
            a.register_buffer("running_var", None)
            a.register_buffer("num_batches_tracked", None)

_input = torch.randn(32, 3)
weight = model[0].weight.clone()
weight.requires_grad_()

def func(param: torch.Tensor, _input: torch.Tensor = None):
    _output: torch.Tensor = _stateless.functional_call(
        model, {'0.weight': param}, _input)
    return _output  # (N, C)

jac: torch.Tensor = functorch.jacrev(functools.partial(func, _input=_input))(weight)

def vmap_func(_input: torch.Tensor):
    return functorch.jacrev(functools.partial(func, _input=_input.unsqueeze(0)))(weight)

jac = functorch.vmap(vmap_func)(_input)

It still raise the error even though model.eval() is called previously. I guess it's the _stateless.functional_call that makes those layers self.training==True again.
Traceback (most recent call last):
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 37, in <module>
    jac = functorch.vmap(vmap_func)(_input)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\vmap.py", line 365, in wrapped
    batched_outputs = func(*batched_inputs, **kwargs)
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 34, in vmap_func
    return functorch.jacrev(functools.partial(func, _input=_input.unsqueeze(0)))(weight)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 441, in wrapper_fn
    vjp_out = vjp(f_wrapper, *primals, has_aux=has_aux)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 270, in vjp
    primals_out = func(*diff_primals)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\functorch\_src\eager_transforms.py", line 624, in f_wrapper
    return f(*replaced_args)
  File "C:\Users\ain-s\OneDrive\workspace\trojanzoo\test.py", line 25, in func
    _output: torch.Tensor = _stateless.functional_call(
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\utils\stateless.py", line 141, in functional_call
    out = module(args, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\container.py", line 139, in forward
    input = module(input)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\modules\batchnorm.py", line 168, in forward
    return F.batch_norm(
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\functional.py", line 2436, in batch_norm
    _verify_batch_size(input.size())
  File "C:\Users\ain-s\miniconda3\lib\site-packages\torch\nn\functional.py", line 2404, in _verify_batch_size
    raise ValueError("Expected more than 1 value per channel when training, got input size {}".format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 4])

@ain-soph, setting the model to eval should work on master https://gist.github.com/zou3519/f581a5de3bbd4721a49923752973c938 . You're correct that it doesn't work on 0.2.0 or earlier releases.

repo: pytorch/functorch


text:
Add USPS and MNIST-M dataset

Hi, I am interested in adding USPS Dataset for digit classification, which commonly used in domain adaption.
I find this repo already implement it, and I modify it slightly here referring to the format of MNIST dataset. Could I make a pull request to add it in torchvision ?
BTW, I write a class for MNIST-M dataset here which consists of MNIST digits blended with random color patches from the BSDS500 dataset. That's also frequently used in domain adaption. But its code is heavily overlapping the original MNIST dataset except the downloading and dataset-making process. I'm wondering whether it's appropriate to add it directly to torchvision, or how to modify it.
Look forward to your suggestions and guidance.

Hi @corenel sorry for the late response. Please send a PR with these and we can review in the PR.
Thanks

authors: alykhantejani corenel

repo: pytorch/vision


text:
Standardize Python Documentation styles

📚 Documentation
~Currently~ TorchVision uses different styles for documenting functions. Here are a few:
https://github.com/pytorch/vision/blob/973db14523d338fa4e772574fe4d49763fb25245/torchvision/io/image.py#L116-L130
https://github.com/pytorch/vision/blob/973db14523d338fa4e772574fe4d49763fb25245/torchvision/io/image.py#L135-L144
https://github.com/pytorch/vision/blob/973db14523d338fa4e772574fe4d49763fb25245/torchvision/models/mobilenet.py#L15-L25
https://github.com/pytorch/vision/blob/973db14523d338fa4e772574fe4d49763fb25245/torchvision/transforms/transforms.py#L195-L210
It's worth selecting one style, standardizing all pydocs and putting a CI test to ensure new PR follows the standard.

Edit: A series of PRs have manually resolved many of the inconsistencies described here. What's missing is adding a CI test to verify that new PRs follow the same standard.
PyTorch seems to have done work towards this direction, so it's worth investigating if we should adopt their solution. See comment: https://github.com/pytorch/vision/pull/3259#issuecomment-765315036
Another alternative is to adopt an open-source docstring linter such as darglint. It seems that it is capable of validating google-style docstrings, it's configurable via setup.cfg and available via pip.
pip install darglint
# setup.cfg
[darglint]
strictness=short
ignore=DAR003,DAR004,DAR005,DAR103,DAR104,DAR105,DAR203,DAR301,DAR302,DAR401,DAR402,DAR501
enable=DAR001,DAR002,DAR101,DAR102,DAR201,DAR202

Unfortunately it's quite slow at the moment:
Not parallelised run:
darglint torchvision

325.77 sec

Parallelised run:
SECONDS=0 ; find torchvision -name "*.py" | xargs -n 1 -P 8 darglint ; echo $SECONDS

119 sec

cc @seemethere @vfdev-5 @fmassa

Good catch!
We should follow the documentation style from PyTorch, which uses Args:, although even PyTorch is not consistent there

cc @brianjo

I'll see if I can find any guidance we have for this and I'll write some up, if I can't find anything. Thanks!

I've been investigating this a bit and here's what I'm thinking. We want to:

make sure that the Args: format is used (i.e. google style)
make sure that a function's signature is in sync with its documented parameters (i.e. that all parameters are properly documented)

can be done by adding

napoleon_numpy_docstring = False
napoleon_google_docstring = True

in docs/conf.py. I made some experiments in https://github.com/pytorch/vision/pull/3278 and sphinx will properly issue a warning if we use the following syntax:
Parameters
----------

We can turn sphinx warnings into error as suggested in https://github.com/pytorch/vision/pull/3259#issuecomment-765315036 so that the CI can enforce such checks. Note that turning sphinx warnings into errors means that the CI will start failing with other kinds of doc-related issues: this isn't a big deal IMHO because there are few of these: only 37 sphinx warnings are raised as-of now, so they don't appear too often.

can be solved by using pydocstyle and by adding the following line in setup.cfg:

[pydocstyle]
ignore=D100,D101,D102,D103,D104,D105,D106,D107,D200,D201,D202,D203,D204,D205,D206,D207,D208,D209,D210,D211,D212,D213,D214,D215,D300,D301,D302,D400,D401,D401,D402,D403,D404,D405,D406,D407,D408,D409,D410,D411,D412,D413,D414,D415,D416

This deactivate all checks but this one: D417 | Missing argument descriptions in the docstring. Shoudl we want to, we may want to stop ignoring other stuff in the future (?)
pydocstyle takes <= 5 seconds to run on torchvision and if it ever becomes too slow we can always just run it on the diff on each PR.
May I suggest to proceed with the following:

Fix current sphinx warnings and update conf.py as above
Use sphinx picky mode and start raising errors on warnings
Fix all of the pydocstyle D417 issues (there are just 20 of them so one PR should be enough) and introduce a new CI check using pydocstyle that would then run on all PRs

Thoughts @datumbox @fmassa ? I'm happy to follow up with PRs if you're happy with this solution

I've been investigating this a bit and here's what I'm thinking. We want to:

make sure that the Args: format is used (i.e. google style)
make sure that a function's signature is in sync with its documented parameters (i.e. that all parameters are properly documented)

can be done by adding

napoleon_numpy_docstring = False
napoleon_google_docstring = True

in docs/conf.py. I made some experiments in https://github.com/pytorch/vision/pull/3278 and sphinx will properly issue a warning if we use the following syntax:
Parameters
----------

We can turn sphinx warnings into error as suggested in https://github.com/pytorch/vision/pull/3259#issuecomment-765315036 so that the CI can enforce such checks. Note that turning sphinx warnings into errors means that the CI will start failing with other kinds of doc-related issues: this isn't a big deal IMHO because there are few of these: only 37 sphinx warnings are raised as-of now, so they don't appear too often.

can be solved by using pydocstyle and by adding the following line in setup.cfg:

[pydocstyle]
ignore=D100,D101,D102,D103,D104,D105,D106,D107,D200,D201,D202,D203,D204,D205,D206,D207,D208,D209,D210,D211,D212,D213,D214,D215,D300,D301,D302,D400,D401,D401,D402,D403,D404,D405,D406,D407,D408,D409,D410,D411,D412,D413,D414,D415,D416

This deactivate all checks but this one: D417 | Missing argument descriptions in the docstring. Shoudl we want to, we may want to stop ignoring other stuff in the future (?)
pydocstyle takes <= 5 seconds to run on torchvision and if it ever becomes too slow we can always just run it on the diff on each PR.
May I suggest to proceed with the following:

Fix current sphinx warnings and update conf.py as above
Use sphinx picky mode and start raising errors on warnings
Fix all of the pydocstyle D417 issues (there are just 20 of them so one PR should be enough) and introduce a new CI check using pydocstyle that would then run on all PRs

Thoughts @datumbox @fmassa ? I'm happy to follow up with PRs if you're happy with this solution

@NicolasHug Great analysis and proposals, thanks for looking into it.
@mattip Any thoughts on the above?

@NicolasHug Great analysis and proposals, thanks for looking into it.
@mattip Any thoughts on the above?

+1 for changing the conf.py and fixing the sphinx errors, then tun on the nitpicky mode. What is  the overlap between the warnings from sphinx and the warnings from pydocstyle? If sphinx makes the pydocstyle tool redundant, then maybe it is not needed?

+1 for changing the conf.py and fixing the sphinx errors, then tun on the nitpicky mode. What is  the overlap between the warnings from sphinx and the warnings from pydocstyle? If sphinx makes the pydocstyle tool redundant, then maybe it is not needed?

There are some redundancy (like checking for unexpected indentation), but as far as I can tell sphinx doesn't issue a warning when a parameter is missing from a function's docstring (item 2 above), which is what pydocstyle can be useful for

There are some redundancy (like checking for unexpected indentation), but as far as I can tell sphinx doesn't issue a warning when a parameter is missing from a function's docstring (item 2 above), which is what pydocstyle can be useful for

flake8 (which we are already using) has an max-doc-length option, which works similar as max-line-length but for docstrings.

flake8 (which we are already using) has an max-doc-length option, which works similar as max-line-length but for docstrings.

This all sounds great to me

This all sounds great to me

authors: NicolasHug brianjo cpuhrsch datumbox fmassa mattip pmeier


text:
Using any torchvision pretrained model as backbone for FasterRcnn

🚀 The feature
Adding only the name of the backbone network, loads the pretrained model in the FasterRcnn model as backbone.
Motivation, pitch
Its possible to use any backbone, but we have to specify many things like feature size, feature_name etc., wont be it easy for the user to just give the name of the backbone and we internally do everything.
Alternatives
No response
Additional context
No response
cc @datumbox @YosuaMichael

Need reviews for this, and thanks for listening to my idea
CC @datumbox

Thanks for the recommendation. If I understand you correctly, you would like to make seamless the combination of different backbones with different detection heads, is that right?
The current API, allows you to provide a backbone on the constructor of the Detection models. This is the first argument that all Object Detection classes receive. Using it is a rather advanced API because one has to pass quite a few configurations to it such as AnchorGenerators. How one will configure these settings is non-trivial and often backbone and dataset specific. Some models even need to adjust the anchors depending on the input size. Here is an investigation we did previously around this which shows how sensitive this is: https://github.com/pytorch/vision/issues/3819#issuecomment-848040195. As a result it's rather difficult to provide one generic method that can attach any backbone to any head and expect it to work (at least that's true for Anchor-based techniques with FPNs etc).
This is the reason we only offer a limited number of models for Detection paired with specific backbones. Typically those are the backbones that their original paper used but sometimes we bended this rule (for example fasterrcnn_mobilenet_v3_large_fpn) to provide better support to for mobile specific applications. Having said that, there is interesting work that researchers are doing in the field to simplify the detection task. For example see this paper by @lyttonhao et al.
So your proposal makes sense, but it's quite hard to achieve with the current models. But it's something that there is active research on and might become possible sooner than later. If I misunderstood the question, please ignore the above and let me know what I missed. Thanks! :)

Yes, exactly I was talking about the same thing. Like you said its not easy to like know some internal config. I think if we can solve this issue, it will be really helpful for the users.
If any work related to this comes I would definitely want to contribute.
Thanks for making it know why its a complex process and giving the link of the papers, I will try to study them.

Nice repo!
I too made something similar https://github.com/mrinath123/timmFasterRcnn I used timm backbone models rather than pytorch vision, but as you can see in the model config, there is a lot of manual work to do.
I hope there will be a easier solution to do this.


text:
Safety Authentication in the shell

safety version: 3.2.0
Python version: 3.9.6
Operating System: macOS

Description
I want to be able to use safety in my CI/CD pipeline but if safety auth is going to return a clickable browser authentication url, it will not be feasible. Is there any way to authenticate in the shell itself?

The SAFETY_API_KEY environment variable seems to work, or using the --key= CLI option: https://docs.safetycli.com/safety-docs/support/invalid-api-key-error#using-the-safety-api-key-in-safety-cli

--key works with safety check not safety scan if I'm not mistaken.

Here is an example in the docs where it is used with safety scan: https://docs.safetycli.com/safety-docs/output/detecting-vulnerabilities-and-sharing-results-via-email#command
Also the variable must work, sine it is used in the official GitHub Action with safety scan: https://github.com/pyupio/safety-action/blob/main/entrypoint.sh
--stage cicd likely makes sense as well, but I could not find a hint about exact effect of this. Probably it ignores the test environment and concentrates on the project (source code) and declared dependencies only, while otherwise it checks the current environment as well, interpreting it as the one the project is finally running in. Just an idea which somehow would make sense to me.

Thanks it's working now (I used --key=API_KEY not --key API_KEY). I had to add --stage cicd or I would get "Invalid value: 'api_key' auth type isn't allowed with the 'Stage.development' stage."

Okay, whatever this "development" stage shall be: Who is doing manual dependency vulnerability checks while coding? Documentation about all of this is missing, or not prominent enough, e.g. searching for stage in the docs does not lead to any other results than examples for --stage cicd usage.
So it is currently not possible to run safety scan without --stage cicd from a non-GUI console?

Thank you so much for the help, forgot to reply promptly, but it worked 🥳

authors: MichaIng akshitadixit yacine-harbi

repo: pyupio/safety


text:
Error while upgrading with 0.3 pre-release

I've tried to upgrade the statuspage with the 0.3 pre release but stumbled upon this error:
$ ./statuspage upgrade --name=StatusPage --token=...
Editing files:  50%|█████████████████████████████████████████▌                                         | 1/2 [00:01<00:01,  1.85s/it]Traceback (most recent call last):
  File "<string>", line 193, in <module>
  File "site-packages/click/core.py", line 716, in __call__
  File "site-packages/click/core.py", line 696, in main
  File "site-packages/click/core.py", line 1060, in invoke
  File "site-packages/click/core.py", line 889, in invoke
  File "site-packages/click/core.py", line 534, in invoke
  File "<string>", line 61, in upgrade
  File "<string>", line 87, in run_upgrade
  File "github/Repository.py", line 1218, in get_file_contents
  File "github/Requester.py", line 171, in requestJsonAndCheck
  File "github/Requester.py", line 179, in __check
github.GithubException.UnknownObjectException: 404 {'documentation_url': 'https://developer.github.com/v3', 'message': 'Not Found'}
statuspage returned -1

Have you moved style.css from the root directory?

Yes, you are right. I moved the files back to root and the upgrade works fine. Thank you.

repo: pyupio/statuspage


text:
Bokeh + HV - SystemError: tile cannot extend outside image

Cross-reference to https://github.com/bokeh/bokeh/issues/8819.

Going to post this as a HoloViews issue too, but this is eerily similar to #8391, so I bet this is a Bokeh problem. In short, exporting a PNG of a HoloViews +-concatenated layout results in a Pillow tile error. Exporting a layout with only one HV plot, on the other hand, works. It's possible that #8085 ended up only solving a subset of the tile problem in Bokeh 1.1.

ALL software version info (bokeh, python, notebook, OS, browser, any other relevant packages)
CentOS 7, Chromium 65.0.3325.181 (Developer Build) Fedora Project (64-bit)
# Name                    Version                   Build  Channel
bokeh                     1.1.0rc1                   py_0    pyviz/label/dev
colorama                  0.4.1                     <pip>
colorcet                  1.0.1                      py_0    pyviz/label/dev
cython                    0.29.5           py37he6710b0_0
datashader                0.7.0                      py_0    pyviz/label/dev
futures-compat            1.0                       py3_0
holoviews                 1.12.0a4                   py_0    pyviz/label/dev
ipykernel                 5.1.0            py37h39e3cac_0
ipyparallel               6.2.3                    py37_0
ipython                   7.1.1            py37h39e3cac_0
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.4.2                    py37_0
jpeg                      9b                   h024ee3a_2
jsonschema                2.6.0                    py37_0
jupyter                   1.0.0                    py37_7
jupyter_client            5.2.4                    py37_0
jupyter_console           6.0.0                    py37_0
jupyter_core              4.4.0                    py37_0
jupyterlab                0.35.4           py37hf63ae98_0
jupyterlab_server         0.2.0                    py37_0
libpng                    1.6.36               hbc83047_0
libsodium                 1.0.16               h1bed415_0
libtiff                   4.0.10               h2733197_2
matplotlib                3.0.3            py37h5429711_0
nbconvert                 5.4.1                      py_2
nbformat                  4.4.0                    py37_0
nodejs                    10.13.0              he6710b0_0
notebook                  5.7.6                    py37_0
numba                     0.43.0           py37h962f231_0
numpy                     1.16.2           py37h7e9f1db_0
numpy-base                1.16.2           py37hde5b4d6_0
numpydoc                  0.8.0                    py37_0
openssl                   1.1.1b               h7b6447c_1
param                     1.8.2                      py_0
phantomjs                 2.1.1                         1    conda-forge
pickleshare               0.7.5                    py37_0
pillow                    5.4.1            py37h34e0f95_0
pims                      0.4.1                      py_1    conda-forge
pixman                    0.36.0               h7b6447c_0
PyNaCl                    1.3.0                     <pip>
pyopenssl                 19.0.0                   py37_0
python                    3.7.2                h0371630_0
python-daemon             2.1.2                     <pip>
pytz                      2018.9                   py37_0
pyviz_comms               0.7.0                    py37_0
requests                  2.21.0                   py37_0
selenium                  3.141.0          py37h7b6447c_0
terminado                 0.8.1                    py37_1
tornado                   6.0.2            py37h7b6447c_0
traitlets                 4.3.2                    py37_0
webencodings              0.5.1                    py37_1
widgetsnbextension        3.4.2                    py37_0
zeromq                    4.3.1                he6710b0_3
zlib                      1.2.11               h7b6447c_3
zstd                      1.3.7                h0b5b093_0

Description of expected behavior and the observed behavior
Expected behavior: save a side-by-side layout of two HV Image elements to a PNG.
Observed behavior: tile error from Pillow. Plot would otherwise be displayed fine in browser / JupyterLab.
Complete, minimal, self-contained example code that reproduces the issue
import holoviews as hv
import numpy as np

from holoviews import opts

hv.extension('bokeh')

im = np.zeros((512, 512))

image_opts = opts.Image(width=750, height=600)

layout = hv.Image(im) + hv.Image(im)

hv.save(layout.opts(toolbar=None), 'asdf.png')

Remove + hv.Image(im) from layout line, and the plot will save correctly.
Stack traceback and/or browser JavaScript console output
WARNING:bokeh.io.export:The webdriver raised a TimeoutException while waiting for                      a 'bokeh:idle' event to signify that the layout has rendered.                      Something may have gone wrong.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/PIL/ImageFile.py in _save(im, fp, tile, bufsize)
    486     try:
--> 487         fh = fp.fileno()
    488         fp.flush()

AttributeError: '_idat' object has no attribute 'fileno'

During handling of the above exception, another exception occurred:

SystemError                               Traceback (most recent call last)
<ipython-input-5-88181649290a> in <module>
      5 layout = hv.Image(im) + hv.Image(im)
      6
----> 7 hv.save(layout.opts(toolbar=None), 'asdf.png')

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/holoviews/util/__init__.py in save(obj, filename, fmt, backend, **kwargs)
    747         if formats[-1] in supported:
    748             filename = '.'.join(formats[:-1])
--> 749     return renderer_obj.save(obj, filename, fmt=fmt)
    750
    751

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/holoviews/plotting/renderer.py in save(self_or_cls, obj, basename, fmt, key, info, options, **kwargs)
    556             return
    557
--> 558         rendered = self_or_cls(plot, fmt)
    559         if rendered is None: return
    560         (data, info) = rendered

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/holoviews/plotting/bokeh/renderer.py in __call__(self, obj, fmt, doc)
    102             return plot(), info
    103         elif fmt == 'png':
--> 104             png = self._figure_data(plot, fmt=fmt, doc=doc)
    105             return png, info
    106         elif fmt == 'html':

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/holoviews/plotting/bokeh/renderer.py in _figure_data(self, plot, fmt, doc, as_script, **kwargs)
    277             img = get_screenshot_as_png(plot.state, None)
    278             imgByteArr = BytesIO()
--> 279             img.save(imgByteArr, format='PNG')
    280             data = imgByteArr.getvalue()
    281             if as_script:

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/PIL/Image.py in save(self, fp, format, **params)
   1992
   1993         try:
-> 1994             save_handler(self, fp, filename)
   1995         finally:
   1996             # do what we can to clean up

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/PIL/PngImagePlugin.py in _save(im, fp, filename, chunk)
    866
    867     ImageFile._save(im, _idat(fp, chunk),
--> 868                     [("zip", (0, 0)+im.size, 0, rawmode)])
    869
    870     chunk(fp, b"IEND", b"")

/scratch/miniconda3/envs/jupyter/lib/python3.7/site-packages/PIL/ImageFile.py in _save(im, fp, tile, bufsize)
    493             if o > 0:
    494                 fp.seek(o, 0)
--> 495             e.setimage(im.im, b)
    496             if e.pushes_fd:
    497                 e.setfd(fp)

SystemError: tile cannot extend outside image

@mattpap informed me that this was a Bokeh problem solved in https://github.com/bokeh/bokeh/pull/8811, which will be integrated into the conda package when Bokeh 1.1 is fully released this coming week.

Yes, this is our fault because due to various constraints we had to release against a bokeh rc release.

authors: philippjfr zbarry

repo: pyviz/holoviews


text:
🛑 nitter.projectsegfau.lt is down

In 9f808d7, nitter.projectsegfau.lt (https://nitter.projectsegfau.lt) was down:

HTTP code: 503
Response time: 115 ms

Resolved: nitter.projectsegfau.lt is back up in f61cca8 after 48 minutes.


text:
🛑 nitter.domain.glass is down

In 82dad8c, nitter.domain.glass (https://nitter.domain.glass) was down:

HTTP code: 503
Response time: 428 ms

Resolved: nitter.domain.glass is back up in edd7250 after 12 minutes.


text:
🛑 nitter.domain.glass is down

In 65249d4, nitter.domain.glass (https://nitter.domain.glass) was down:

HTTP code: 503
Response time: 293 ms

Resolved: nitter.domain.glass is back up in 2e3fd8c after 46 minutes.


text:
🛑 nitter.us.projectsegfau.lt is down

In 4a1f5cd, nitter.us.projectsegfau.lt (https://nitter.us.projectsegfau.lt) was down:

HTTP code: 503
Response time: 369 ms

Resolved: nitter.us.projectsegfau.lt is back up in 6d69ea7 after 17 minutes.


text:
🛑 nitter.us.projectsegfau.lt is down

In 3cf56f0, nitter.us.projectsegfau.lt (https://nitter.us.projectsegfau.lt) was down:

HTTP code: 503
Response time: 333 ms

Resolved: nitter.us.projectsegfau.lt is back up in 3a4e0e6 after 12 minutes.


text:
🛑 nitter.projectsegfau.lt is down

In 565e307, nitter.projectsegfau.lt (https://nitter.projectsegfau.lt) was down:

HTTP code: 503
Response time: 350 ms

Resolved: nitter.projectsegfau.lt is back up in 336d578 after 12 minutes.


text:
🛑 nitter.qwik.space is down

In 8ace613, nitter.qwik.space (https://nitter.qwik.space) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.qwik.space is back up in 114b051 after 4 minutes.


text:
🛑 notabird.site is down

In b5fd011, notabird.site (https://notabird.site) was down:

HTTP code: 520
Response time: 68 ms

Resolved: notabird.site is back up in 93119e4 after 10 minutes.


text:
🛑 read.whatever.social is down

In 9981578, read.whatever.social (https://read.whatever.social) was down:

HTTP code: 504
Response time: 11013 ms

Resolved: read.whatever.social is back up in edde80c.


text:
🛑 notabird.site is down

In 1da0264, notabird.site (https://notabird.site) was down:

HTTP code: 520
Response time: 386 ms

Resolved: notabird.site is back up in ca421d5 after 1 hour, 29 minutes.


text:
switch to lightweight jenkins-slave:latest-alpine in default setup

For api and web jobs it is enough to use latest-alpine images:
qaprosoft/jenkins-slave:latest-alpine
ubuntu "latest" is twice bigger and has android sdk components installed for android mcloud slave

fix: https://github.com/qaprosoft/qps-infra/commit/dcec6b50c23c3de028464ac6fec3e93e1f96a2c1
@okamara test scenario:
inside qps-infra

sudo ./clear.sh
git pull
./start.sh
... register organization, repo and run web/api jobs
-> "docker images" command should display "qaprosoft/jenkins-slave:latest-alpine" image on the system
-> api/web tests could run as usual

@vdelendik I see qaprosoft/jenkins-slave:latest ->

is it correct?

Verified .

authors: okamara vdelendik

repo: qaprosoft/qps-infra


text:
handle NPE exception for zc.getTestRunResults() call

From time to time we receive NPE on below line. The most common way when Zafira is overloaded (mass queue registration is in progress).
ZafiraListener.java:171
List testRunResults = Arrays.asList(zc.getTestRunResults(run.getId()).getObject());
According to my understanding 171-181 lines are needed to recalculate test based on need_rerun flag. I'd recommend to skip those steps if test run is in QUEUED status at all. For queued all tests.
Or put 171-181 line inside if (ZAFIRA_RERUN_FAILURES) { operator
12:31:20  [com.qaprosoft.zafira.listener.ZafiraListener] ERROR - Undefined error during test run registration!
12:31:20  java.lang.NullPointerException: null
12:31:20  	at com.qaprosoft.zafira.listener.ZafiraListener.onStart(ZafiraListener.java:171) ~[zafira-client-4.0.57.jar:na]
12:31:20  	at org.testng.SuiteRunner.invokeListeners(SuiteRunner.java:301) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.SuiteRunner.run(SuiteRunner.java:362) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.TestNG.runSuites(TestNG.java:1049) [testng-6.14.3.jar:na]
12:31:20  	at org.testng.TestNG.run(TestNG.java:1017) [testng-6.14.3.jar:na]
12:31:20  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283) [surefire-testng-2.22.1.jar:2.22.1]
12:31:20  	at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75) [surefire-testng-2.22.1.jar:2.22.1]
12:31:20  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120) [surefire-testng-2.22.1.jar:2.22.1]
12:31:20  	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) [surefire-booter-2.22.1.jar:2.22.1]
12:31:20  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) [surefire-booter-2.22.1.jar:2.22.1]
12:31:20  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) [surefire-booter-2.22.1.jar:2.22.1]
12:31:20  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) [surefire-booter-2.22.1.jar:2.22.1]

Fixed.

authors: hursevich vdelendik

repo: qaprosoft/zafira


text:
[Improvement] Add possibility to set date filter to 'Today'

As a user I'd like to have a possibility to create filter which show me test runs that were conducted today

done

authors: Boniara Litsvin


text:
Bad uuid.NewV4 call

$ go get github.com/qasaur/gremgo
# github.com/qasaur/gremgo
../../github.com/qasaur/gremgo/request.go:32:17: multiple-value uuid.NewV4() in single-value context

Fixed in #39

authors: AgentZombie qasaur

repo: qasaur/gremgo


text:
Unreachable resource not handled

Following along with ackzell's video series
https://goo.gl/8p3msR
it turns out that this library does not seem to handle unreachable resources.
If an unreachable URI is pinged, the network tab returns "status: pending" after after a certain timeout "304".
I don't know, if this package is still maintained, but if so, would it be possible to implement such a feature?
Thank you!

Hi @rowild , thank you for raising this issue. I'm currently swamped with work but will attend to it before the end of the week. Thank you 👍

Fantastic! Looking forward to your solution! Have a good week!

Hi @rowild, I've just pushed a new version(v0.2.0): https://www.npmjs.com/package/ping-monitor#change.
Please let me know this fixes the issue.
Thank you 👍

This is so great that you took care of that!
I just tested it with ackzell's app and it seems to work just fine :-)
Thank you very, very much, @qawemlilo !!!

authors: qawemlilo rowild

repo: qawemlilo/node-monitor


text:
Feature/example advance2

Description

Added new code example Advance 2 and README documentation for multiples VM creation with for_each block.
Change overview (tick true):

[ ] This introduces backward incompatible changes
[X] This adds a new backward compatible Feature
[ ] This fixes a Bug

Version information:

Previous Version: 2.1.0

Next Version based on Semantic Versioning (see above): 2.1.0

How Has This Been Tested?

[X] Apply of all examples was successfull
[X] Apply of all test was successfull

Checklist:

[X] I have run tests and documented them above
[X] I have performed a self-review of my own code
[X] I have updated the documentation
[X] I have updated the CHANGELOG

I don't see the point of the advanced2 example. It doesn't show any features of the module. It just showcases how to use a for_each. I don't think examples of a module is the right place to showcase a terraform function.
In my opinion examples should only showcase features of the module.

So then ?

repo: qbeyond/terraform-azurerm-windows-vm


text:
Provide a notification system

resolve #682
Moved code to new branch to avoid dealing with git history

as I understand this PR, it is the notification system, which is not sufficient to resolve #682 . Correct?

@KochTobi in order for @Shraddha0903 to verify the UI/UX of this feature, please also provide a screen record / screen shot

Something is wrong with the implementation, when I add an experimental variable, the confirmation behaviour is the one of the cancel button.
I also think, that the notification shall be submitted only when the application server returns a successful operation response, and not when the confirm button is clicked.
https://github.com/user-attachments/assets/7e246520-2b03-4bcf-9b73-b2dc2fc48ea3

Thank you @sven1103 for your review I will think about the points you made here. I think the aspect of immutability is important but I am not entirely sure whether we want to do this here.
I will get back to you

repo: qbicsoftware/data-manager-app


text:
Rename GitHub Actions

Makes the name selectable for status checks

Doesn't work


text:
QGIS 3.2.2 Crashes

h2. User Feedback
h2. Report Details
Crash ID: e21fe39b6cd75f2d0325117dfbfdbfd90d98c7ca
Stack Trace

QDockWidget::toggleViewAction :
QgisApp::removeDockWidget :
QgsPresetSchemeColorRamp::clone :
PyCFunction_FastCallDict :
PyObject_GenericGetAttr :
PyEval_EvalFrameDefault :
PyObject_GenericGetAttr :
PyEval_EvalFrameDefault :
PyObject_GenericGetAttr :
PyEval_EvalFrameDefault :
PyErr_Occurred :
PyEval_EvalCode :
PyArena_Free :
PyRun_StringFlags :
instance :
instance :
QgsPluginRegistry::unloadPythonPlugin :
QgsLayoutLocatorFilter::triggerResult :
QgsLayoutLocatorFilter::triggerResult :
QMetaObject::activate :
QStandardItemModel::`default constructor closure' :
QMetaObject::activate :
QAbstractItemModel::dataChanged :
QStandardItemModel::itemChanged :
QStandardItem::setData :
QStandardItemModel::setData :
QSortFilterProxyModel::setData :
QStyledItemDelegate::editorEvent :
QAbstractItemView::selectionModel :
QAbstractItemView::edit :
QAbstractItemView::mouseReleaseEvent :
QListView::mouseReleaseEvent :
QWidget::event :
QFrame::event :
QAbstractItemView::viewportEvent :
QCoreApplicationPrivate::sendThroughObjectEventFilters :
QApplicationPrivate::notify_helper :
QApplication::notify :
QgsApplication::notify :
QCoreApplication::notifyInternal2 :
QApplicationPrivate::sendMouseEvent :
QSizePolicy::QSizePolicy :
QSizePolicy::QSizePolicy :
QApplicationPrivate::notify_helper :
QApplication::notify :
QgsApplication::notify :
QCoreApplication::notifyInternal2 :
QGuiApplicationPrivate::processMouseEvent :
QWindowSystemInterface::sendWindowSystemEvents :
QEventDispatcherWin32::processEvents :
CallWindowProcW :
DispatchMessageW :
QEventDispatcherWin32::processEvents :
qt_plugin_query_metadata :
QEventLoop::exec :
QDialog::exec :
PyInit__gui :
PyCFunction_FastCallDict :
PyObject_GenericGetAttr :
PyEval_EvalFrameDefault :
PyErr_Occurred :
PyObject_GenericGetAttr :
PyEval_EvalFrameDefault :
PyErr_Occurred :
PyEval_EvalCode :
PyArena_Free :
PyRun_StringFlags :
instance :
instance :
QgisApp::rotatePointSymbols :
QgsPythonRunner::run :
QgisApp::showPluginManager :
QMetaObject::activate :
QAction::activate :
QMenu::actionGeometry :
QMenu::actionGeometry :
QMenu::mouseReleaseEvent :
QWidget::event :
QMenu::event :
QApplicationPrivate::notify_helper :
QApplication::notify :
QgsApplication::notify :
QCoreApplication::notifyInternal2 :
QApplicationPrivate::sendMouseEvent :
QSizePolicy::QSizePolicy :
QSizePolicy::QSizePolicy :
QApplicationPrivate::notify_helper :
QApplication::notify :
QgsApplication::notify :
QCoreApplication::notifyInternal2 :
QGuiApplicationPrivate::processMouseEvent :
QWindowSystemInterface::sendWindowSystemEvents :
QEventDispatcherWin32::processEvents :
CallWindowProcW :
DispatchMessageW :
QEventDispatcherWin32::processEvents :
qt_plugin_query_metadata :
QEventLoop::exec :
QCoreApplication::exec :
main :
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info
QGIS Version: 3.2.2-Bonn
QGIS code revision: commit:26842169e9
Compiled against Qt: 5.9.2
Running against Qt: 5.9.2
Compiled against GDAL: 2.2.4
Running against GDAL: 2.2.4
System Info
CPU Type: x86_64
Kernel Type: winnt
Kernel Version: 10.0.17134

Hi,
This repo is about the QGIS website - not about QGIS problems. Please use https://isssues.qgis.org/ instead.
Also - you have to describe what you did before the crash. Otherwise this report is not useful for developers. More details about how to reproduce the crash are necessary.
Closing this now,
Andreas

repo: qgis/QGIS-Website


text:
add compat entry for DelimitedFiles

In Julia v1.9 DelimitedFiles is no longer a stdlib, and so I found it needs a compat entry in order to run the tests.

Thanks a lot, Lorenzo!

authors: lxvm qiaojunfeng


text:
Error when trying to view data

I freshly downloaded some BAS data (and confirmed it downloaded to my device). When I try to view the flight path radar, this error pops up:

I also tried to close and reopen the program but the error still occurs.

Huh. This is obviously a bug somewhere that I need to fix, but I can't figure it out from this alone.

Can you try:

restart QGIS
use the Plugin Reloader to reload QIceRadar
then try viewing data again?
(If this fixes it, it means there's some logic in my handling of the reload signal that doesn't get execute on the initial load)

While you're testing that, can you also scroll up on your layers panel and see if you have a "Radar Viewer" group?


text:
中文报错

看样子使用了R17版本，R17的中文处理方式有所变化，应该这样使用：
?ERROR_MSG("~s", [<<"中文信息"/utf8>]).
不需要ts，字符串需要/utf8定义

authors: liuweiccy qingliangcn

repo: qingliangcn/mslog


text:
Installation from source on linux - process killed

Information

Qiskit Metal version - latest from the source as per 22.10.2024:
Python version - the one that comes along as specified in environment.yml:
Operating system - linux: Linux ThinkPad-X1-Carbon-2nd 5.19.0-46-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 13:30:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux:

What is the current behavior?
I follow recommended installation from source (Option 1, create new environment) but each time I try it (and I have been trying for the last three days), there is the same outcome:
dragana@ThinkPad-X1-Carbon-2nd:~/qiskit-workspace/workshop_SCQHW_fall_2024/qiskit-metal$ conda env create -n qiskitmetal -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: | Killed
(attached option_1_install_new_environment.png screnshot)
dmesg:
[ 2810.246062] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-8c6f265e-a98d-43cd-8cef-bb86c7f858fe.scope,task=conda-env,pid=6308,uid=1000
[ 2810.246093] Out of memory: Killed process 6308 (conda-env) total-vm:6556824kB, anon-rss:5371692kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12164kB oom_score_adj:200
[ 2810.257458] Purging GPU memory, 0 pages freed, 0 pages still pinned, 5440 pages left available.
(attached dmesg_1.png screnshot)
However, the memory does not seem to be that low:
dragana@ThinkPad-X1-Carbon-2nd:~/qiskit-workspace/workshop_SCQHW_fall_2024/qiskit-metal$ free
total        used        free      shared  buff/cache   available
Mem:         7761456     2034620     4449856      412128     1276980     4986912
Swap:        2097148     1167976      929172
(attached memory_snapshot.png screenshot)
I am experienced with working with conda environments and installation of sw and I have never experienced a case like this.
I also tried creating a new clean environment then installing one by one package from environment.yml but at fourth the same kind of kill process happens (oom).
I have succeeded in installing qiskit-metal on macos by using the same commands.
Steps to reproduce the problem
git clone https://github.com/Qiskit/qiskit-metal.git
cd qiskit-metal
conda env create -n qiskitmetal -f environment.yml
What is the expected behavior?
It is expected to have new conda environment named qiskitmetal (in my case the name I gave to new environment) created.
Suggested solutions
I would suggest the team inspect how installation process is realised regarding depndencies as that it is the only thing I could think of which might cause OOM situation. Although it seems very unlikely to need that much memory. Can it be some recurrent link or circular link chain?

Sounds like something very specific to your Lennox environment. You could try the virtual environment set up instead of conda Sent from my iPhoneOn Oct 22, 2024, at 4:26 AM, Dragana @.***> wrote:
Information
Qiskit Metal version - latest from the source as per 22.10.2024:
Python version - the one that comes along as specified in environment.yml:
Operating system - linux: Linux ThinkPad-X1-Carbon-2nd 5.19.0-46-generic #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 13:30:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux:
What is the current behavior?
I follow recommended installation from source (Option 1, create new environment) but each time I try it (and I have been trying for the last three days), there is the same outcome:
@.:~/qiskit-workspace/workshop_SCQHW_fall_2024/qiskit-metal$ conda env create -n qiskitmetal -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: | Killed
(attached option_1_install_new_environment.png screnshot)
dmesg:
[ 2810.246062] @./app.slice/app-org.gnome.Terminal.slice/vte-spawn-8c6f265e-a98d-43cd-8cef-bb86c7f858fe.scope,task=conda-env,pid=6308,uid=1000
[ 2810.246093] Out of memory: Killed process 6308 (conda-env) total-vm:6556824kB, anon-rss:5371692kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:12164kB oom_score_adj:200
[ 2810.257458] Purging GPU memory, 0 pages freed, 0 pages still pinned, 5440 pages left available.
(attached dmesg_1.png screnshot)
However, the memory does not seem to be that low:
@.***:~/qiskit-workspace/workshop_SCQHW_fall_2024/qiskit-metal$ free
total        used        free      shared  buff/cache   available
Mem:         7761456     2034620     4449856      412128     1276980     4986912
Swap:        2097148     1167976      929172
(attached memory_snapshot.png screenshot)
I am experienced with working with conda environments and installation of sw and I have never experienced a case like this.
I also tried creating a new clean environment then installing one by one package from environment.yml but at fourth the same kind of kill process happens (oom).
I have succeeded in installing qiskit-metal on macos by using the same commands.
Steps to reproduce the problem
git clone https://github.com/Qiskit/qiskit-metal.git
cd qiskit-metal
conda env create -n qiskitmetal -f environment.yml
What is the expected behavior?
It is expected to have new conda environment named qiskitmetal (in my case the name I gave to new environment) created.
Suggested solutions
I would suggest the team inspect how installation process is realised regarding depndencies as that it is the only thing I could think of which might cause OOM situation. Although it seems very unlikely to need that much memory. Can it be some recurrent link or circular link chain?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

So, you'd think it is linux environment. I can give it a try by increasing oom score from 200 (visible from dmesg output) to 1000 and run the same  command again. I have installation on macos, though, so I'll test it in a little while back on linux and if I find the cause for such behaviour, I'll share it.
Thank you for a quick response.

repo: qiskit-community/qiskit-metal


text:
Unable to import R extensions

Description
I get an error message in Qlik Sense Enterprise version September 2018 that "the zip file doesn't contain any valid extensions".
Steps to Reproduce

Download sse-r-plugin-1.2.1-qlik-oss.zip
Open the extension section in the QMC
Import extension sse-r-plugin-1.2.1-qlik-oss.zip

Expected behavior
Extension should be imported succesfully and added to the list of extensions.
Actual behavior
Error message apears: "the zip file doesn't contain any valid extensions".
Environment
Operating system where the SSE plugin is deployed
[x ] Windows
[ ] Linux

Qlik Sense
[ ] Desktop
[x ] Server

Versions

sse-r-plugin:1.2.1
Qlik Sense: September 2018 patch 1
Operating system: Windows server 2016

Hi Tobias,
Thanks for your assistance and for clearing up the misunderstanding ;-).
Regards,
Bram

repo: qlik-oss/sse-r-plugin


text:
Test Issue - vjduy

Test Body - bzcpvqwssw

Test Comment - ptxbz

repo: qlikqaa/dummyrepo


text:
Test Issue - fxtzr

Test Body - yajlqbvsaa

Test Comment - pazwj


text:
Test Issue - pdsdt

Test Body - iazbwgygla

Test Comment - beowc


text:
Test Issue - yxngs

Test Body - htgoubfegf

Test Comment - hbnwq


text:
Test Issue - paayp

Test Body - lepbbpyelw

Test Comment - ttfqa


text:
Test Issue - bowhs

Test Body - yczahllfoa

Test Comment - nkupn


text:
Test Issue - kkdif

Test Body - weiygownxt

Test Comment - ezvgh


text:
Test Issue - zgxom

Test Body - lfxmxijzcd

Test Comment - raxvs


text:
Test Issue - uqhgr

Test Body - izrflsabmz

Test Comment - talgi


text:
Test Issue - ttezg

Test Body - flkrffrjjf

Test Comment - ruszl


text:
Test Issue - rwunk

Test Body - ttnecxuzih

Test Comment - allqf


text:
Test Issue - efhnm

Test Body - qkojrpznvp

Test Comment - kitri


text:
Test Issue - otnij

Test Body - dbyazrqnzj

Test Comment - lulcr


text:
Test Issue - lbasp

Test Body - yvsuhsinxq

Test Comment - iwomx