text:
Fix flakey GenericViewTest.toString

makeMapVectorFromJson doesn't guarantee the order of the keys in the map.
It would cause flakey in GenericViewTest.toString.

Reproduced locally,
[==========] Running 1 test from 1 test suite.
[----------] 1 test from GenericViewTest
[ RUN      ] GenericViewTest.toString
E20240318 16:08:45.426143 757481 Exceptions.h:69] Line: /Users/macduan/work/native/velox-3/./velox/expression/ComplexViewTypes.h:358, Function:value, Expression: has_value() , Source: RUNTIME, ErrorCode: INVALID_STATE
unknown file: Failure
C++ exception with description "Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: has_value()
Function: value
File: /Users/macduan/work/native/velox-3/./velox/expression/ComplexViewTypes.h
Line: 358
Stack trace:
# 0
# 1
# 2
# 3
# 4
# 5
# 6
# 7
# 8
# 9
# 10
# 11
# 12
# 13
# 14
# 15
# 16
# 17
# 18
# 19
# 20
" thrown in the test body.
[  FAILED  ] GenericViewTest.toString (0 ms)
[----------] 1 test from GenericViewTest (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (11 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] GenericViewTest.toString

 1 FAILED TEST

@mbasmanova Hi Masha, I found a flakey in a PR's CI. I reproduced it locally and created this PR to fix it. I've run repeatedly for 200 times to verify it. Could you please help take a look? Thanks.

Did #9085 not fix this?

@mbasmanova No, it did not fix this as the order may be different. I can reproduced it with the latest code.

I added a log to debug and found the keys may be ["3", "1", "2]
log diff
diff --git a/velox/expression/tests/GenericViewTest.cpp b/velox/expression/tests/GenericViewTest.cpp
index 93a667033..bfd862a61 100644
--- a/velox/expression/tests/GenericViewTest.cpp
+++ b/velox/expression/tests/GenericViewTest.cpp
@@ -224,6 +224,7 @@ TEST_F(GenericViewTest, toString) {

     auto keys = data->mapKeys();
     auto values = data->mapValues();
+    LOG(ERROR) << "keys: " << keys->toString(0) << " " << keys->toString(1) << " " << keys->toString(2);

### logs
```C++
Note: Google Test filter = GenericViewTest.toString:GenericViewTest/*.toString:GenericViewTest.toString/*:*/GenericViewTest.toString/*:*/GenericViewTest/*.toString
[==========] Running 1 test from 1 test suite.
[----------] 1 test from GenericViewTest
[ RUN      ] GenericViewTest.toString
E20240318 16:29:04.446543 785369 GenericViewTest.cpp:227] keys: 3 1 2
E20240318 16:29:04.446624 785369 Exceptions.h:69] Line: /Users/macduan/work/native/velox-3/./velox/expression/ComplexViewTypes.h:358, Function:value, Expression: has_value() , Source: RUNTIME, ErrorCode: INVALID_STATE
unknown file: Failure
C++ exception with description "Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: has_value()
Function: value
File: /Users/macduan/work/native/velox-3/./velox/expression/ComplexViewTypes.h
Line: 358
Stack trace:
# 0
# 1
# 2
# 3
# 4
# 5
# 6
# 7
# 8
# 9
# 10
# 11
# 12
# 13
# 14
# 15
# 16
# 17
# 18
# 19
# 20
" thrown in the test body.
[  FAILED  ] GenericViewTest.toString (0 ms)
[----------] 1 test from GenericViewTest (0 ms total)

[==========] 1 test from 1 test suite ran. (12 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] GenericViewTest.toString

 1 FAILED TEST

Repeating all tests (iteration 4) . . .

@duanmeng It is indeed the case that the order of keys in the result of makeMapVectorFromJson can be different. However, we are comparing with data->mapKeys() whose order should match the order of rows in decoded vector, no?
    auto data = makeMapVectorFromJson<int32_t, int64_t>({
        "{1: 10, 2: null, 3: 30}",
    });
    auto keys = data->mapKeys();
    auto values = data->mapValues();

@duanmeng It is indeed the case that the order of keys in the result of makeMapVectorFromJson can be different. However, we are comparing with data->mapKeys() whose order should match the order of rows in decoded vector, no?
    auto data = makeMapVectorFromJson<int32_t, int64_t>({
        "{1: 10, 2: null, 3: 30}",
    });
    auto keys = data->mapKeys();
    auto values = data->mapValues();

@mbasmanova It may not match the order as  makeMapVectorFromJson usesfolly::parseJson, which may change the order. I found mostly the keys are 3 2 1, sometimes they are *3 1 2 and failed.

    auto data = makeMapVectorFromJson<int32_t, int64_t>({
        "{1: 10, 2: null, 3: 30}",
    });
    auto keys = data->mapKeys();
    auto values = data->mapValues();
    DecodedVector decoded(*data);
    exec::VectorReader<Map<Any, Any>> reader(&decoded);

    auto mapView = reader[0];

Here, we have a MapVector 'data' and we use exec::VectorReader to read from it. I expect the order of keys in 'data' to match the order of keys in 'exec::VectorReader'. Is this not the case?
The order of keys in 'data' should be:

data->offsetAt(0), data->offsetAt(0) + 1, data->offsetAt(0) + 2

I expect the order of keys in 'data' to match the order of keys in 'exec::VectorReader'.

Yes, they are matched. The problem should be that the UT assumes that the first and the last values are not null. The it->second.value() throws if the last value is null. The MapVector layout may be

keys: [3, 2, 1], values: [30, null, 10]
keys: [3, 1, 2], values: [30, 10, null]
For layout 2, although, the order of keys in 'data' matches the order of keys in 'exec::VectorReader', it would throw in the last EXPECT_EQ(values->toString(2), it->second.value().toString());.

although the order of keys in 'data' matches the order of keys in 'exec::VectorReader', it would throw in the last EXPECT_EQ(values->toString(2), it->second.value().toString());.

I assume that the order of keys and values should match, no?
auto offset = data->offsetAt(0);
row 0: keys->toString(offset), values->toString(offset) -> 1: 10
row 1: keys->toString(offset + 1), values->toString(offset + 1) -> 2: null
row 2: keys->toString(offset + 2), values->toString(offset + 2) -> 3: 30
Are you seeing something different?

although the order of keys in 'data' matches the order of keys in 'exec::VectorReader', it would throw in the last EXPECT_EQ(values->toString(2), it->second.value().toString());.

I assume that the order of keys and values should match, no?
auto offset = data->offsetAt(0); row 0: keys->toString(offset), values->toString(offset) -> 1: 10 row 1: keys->toString(offset + 1), values->toString(offset + 1) -> 2: null row 2: keys->toString(offset + 2), values->toString(offset + 2) -> 3: 30
Are you seeing something different?

Yes, I saw something different as follows
SUCCESS
E20240318 17:22:38.700711 855310 GenericViewTest.cpp:230] row0: 3->30
E20240318 17:22:38.700726 855310 GenericViewTest.cpp:231] row1: 2->null
E20240318 17:22:38.700738 855310 GenericViewTest.cpp:232] row2: 1->10

FAIL
E20240318 17:22:38.711611 855310 GenericViewTest.cpp:230] row0: 3->30
E20240318 17:22:38.711624 855310 GenericViewTest.cpp:231] row1: 1->10
E20240318 17:22:38.711638 855310 GenericViewTest.cpp:232] row2: 2->null

log code
auto offset = data->offsetAt(0);
    LOG(ERROR) << "row0: " << keys->toString(offset) << "->" << values->toString(offset);
    LOG(ERROR) << "row1: " << keys->toString(offset + 1) << "->" << values->toString(offset + 1);
    LOG(ERROR) << "row2: " << keys->toString(offset + 2) << "->" << values->toString(offset + 2);

@duanmeng I think the logging above confirms that order of keys can be different on different runs, but it doesn't demonstrate that the order of keys in 'data' doesn't match the order of keys in 'reader'.

Is this because I ran it in debug mode and triggered the DCHECK failure?

The following code assumes that the second value is always null

Got it.
How about
    auto it = mapView.begin();
    for (auto i = 0 ; i < 3; ++i) {
      EXPECT_EQ(keys->toString(i), it->first.toString());
      if (values->isNullAt(i)) {
        EXPECT_FALSE(it->second.has_value());
      } else {
        EXPECT_EQ(values->toString(i), it->second.value().toString());
      }
      ++it;
    }

@mbasmanova Got it, done. Thanks.

authors: duanmeng mbasmanova


text:
python setup.py install error

(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$ python setup.py install
Traceback (most recent call last):
File "setup.py", line 15, in
name = f"bitsandbytes-cuda{os.environ['CUDA_VERSION']}",
File "/home/chenxin/disk1/anaconda3/envs/bitsandbytes/lib/python3.8/os.py", line 675, in getitem
raise KeyError(key) from None
KeyError: 'CUDA_VERSION'
(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$ conda list | grep cudatoolkit
cudatoolkit               11.1.1               h6406543_8    conda-forge

Sorry, this is currently not well documented and a bit finicky. What you have to do to install is set the CUDA_VERSION variable, for example like this: CUDA_VERSION=113 python setup.py install.

@TimDettmers  thanks
(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$ CUDA_VERSION=111 python setup.py install
running install
running bdist_egg
running egg_info
creating bitsandbytes_cuda111.egg-info
writing bitsandbytes_cuda111.egg-info/PKG-INFO
writing dependency_links to bitsandbytes_cuda111.egg-info/dependency_links.txt
writing top-level names to bitsandbytes_cuda111.egg-info/top_level.txt
writing manifest file 'bitsandbytes_cuda111.egg-info/SOURCES.txt'
reading manifest file 'bitsandbytes_cuda111.egg-info/SOURCES.txt'
adding license file 'LICENSE'
adding license file 'NOTICE.md'
writing manifest file 'bitsandbytes_cuda111.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/bitsandbytes
copying bitsandbytes/functional.py -> build/lib/bitsandbytes
copying bitsandbytes/init.py -> build/lib/bitsandbytes
creating build/lib/bitsandbytes/nn
copying bitsandbytes/nn/modules.py -> build/lib/bitsandbytes/nn
copying bitsandbytes/nn/init.py -> build/lib/bitsandbytes/nn
creating build/lib/bitsandbytes/optim
copying bitsandbytes/optim/adam.py -> build/lib/bitsandbytes/optim
copying bitsandbytes/optim/lamb.py -> build/lib/bitsandbytes/optim
copying bitsandbytes/optim/lars.py -> build/lib/bitsandbytes/optim
copying bitsandbytes/optim/optimizer.py -> build/lib/bitsandbytes/optim
copying bitsandbytes/optim/rmsprop.py -> build/lib/bitsandbytes/optim
copying bitsandbytes/optim/sgd.py -> build/lib/bitsandbytes/optim
copying bitsandbytes/optim/init.py -> build/lib/bitsandbytes/optim
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/bitsandbytes
copying build/lib/bitsandbytes/functional.py -> build/bdist.linux-x86_64/egg/bitsandbytes
creating build/bdist.linux-x86_64/egg/bitsandbytes/nn
copying build/lib/bitsandbytes/nn/modules.py -> build/bdist.linux-x86_64/egg/bitsandbytes/nn
copying build/lib/bitsandbytes/nn/init.py -> build/bdist.linux-x86_64/egg/bitsandbytes/nn
creating build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/adam.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/lamb.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/lars.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/optimizer.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/rmsprop.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/sgd.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/optim/init.py -> build/bdist.linux-x86_64/egg/bitsandbytes/optim
copying build/lib/bitsandbytes/init.py -> build/bdist.linux-x86_64/egg/bitsandbytes
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/functional.py to functional.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/nn/modules.py to modules.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/nn/init.py to init.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/adam.py to adam.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/lamb.py to lamb.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/lars.py to lars.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/optimizer.py to optimizer.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/rmsprop.py to rmsprop.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/sgd.py to sgd.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/optim/init.py to init.cpython-38.pyc
byte-compiling build/bdist.linux-x86_64/egg/bitsandbytes/init.py to init.cpython-38.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes_cuda111.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes_cuda111.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes_cuda111.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bitsandbytes_cuda111.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
bitsandbytes.pycache.functional.cpython-38: module references file
creating dist
creating 'dist/bitsandbytes_cuda111-0.25.0-py3.8.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing bitsandbytes_cuda111-0.25.0-py3.8.egg
creating /home/chenxin/disk1/anaconda3/envs/bitsandbytes/lib/python3.8/site-packages/bitsandbytes_cuda111-0.25.0-py3.8.egg
Extracting bitsandbytes_cuda111-0.25.0-py3.8.egg to /home/chenxin/disk1/anaconda3/envs/bitsandbytes/lib/python3.8/site-packages
Adding bitsandbytes-cuda111 0.25.0 to easy-install.pth file
Installed /home/chenxin/disk1/anaconda3/envs/bitsandbytes/lib/python3.8/site-packages/bitsandbytes_cuda111-0.25.0-py3.8.egg
Processing dependencies for bitsandbytes-cuda111==0.25.0
Finished processing dependencies for bitsandbytes-cuda111==0.25.0
(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0
(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$

@TimDettmers
(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$ python ./check_bnb_install.py
Traceback (most recent call last):
File "./check_bnb_install.py", line 1, in
import bitsandbytes as bnb
File "/home/chenxin/disk1/github/bitsandbytes/bitsandbytes/init.py", line 5, in
from .optim import adam
File "/home/chenxin/disk1/github/bitsandbytes/bitsandbytes/optim/init.py", line 5, in
from .adam import Adam, Adam8bit, Adam32bit
File "/home/chenxin/disk1/github/bitsandbytes/bitsandbytes/optim/adam.py", line 11, in
from bitsandbytes.optim.optimizer import Optimizer2State
File "/home/chenxin/disk1/github/bitsandbytes/bitsandbytes/optim/optimizer.py", line 6, in
import bitsandbytes.functional as F
File "/home/chenxin/disk1/github/bitsandbytes/bitsandbytes/functional.py", line 13, in
lib = ct.cdll.LoadLibrary(os.path.dirname(file) + '/libbitsandbytes.so')
File "/home/chenxin/disk1/anaconda3/envs/bitsandbytes/lib/python3.8/ctypes/init.py", line 451, in LoadLibrary
return self._dlltype(name)
File "/home/chenxin/disk1/anaconda3/envs/bitsandbytes/lib/python3.8/ctypes/init.py", line 373, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/chenxin/disk1/github/bitsandbytes/bitsandbytes/libbitsandbytes.so: cannot open shared object file: No such file or directory

@TimDettmers  but i download all

Sorry that this is all a bit unintuitive. I did not have the time yet to document this. By default, currently, there is no compilation step when you install via python setup.py install. So an additional step to get the library working is to compile from source. Please have a look here for instructions to compile from source.

@TimDettmers  ok
(bitsandbytes) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/bitsandbytes$ python check_bnb_install.py
SUCCESS!
Installation was successful!

1535  make cuda11x
1536  CUDA_VERSION=11x python setup.py install
1537  python check_bnb_install.py
1538  history

@TimDettmers Sorry , if i use g++5,say not support C++,But i see makefile include =std++11

i put nvcc compile --std==c++11,not show this ,but show as bel

ow

thrust?

OSError: /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes.so: cannot open shared object file:
No such file or directory
What Should I Do To Get Rid of This...

source: gharchive/issue

authors: TheFaheem TimDettmers mathpopo

repo: facebookresearch/bitsandbytes

license: MIT

license_type: permissive

license_source: github-api


text:
Memory refactor tracker

Type of Issue
Select the type of issue:

[ ] Bug report (to report a bug)
[ ] Feature request (to request an additional feature)
[x] Tracker (I am just using this as a tracker)
[ ] Refactor request
[ ] Documentation Ask

Checklist
The branch is: kavya/modular_memory_refactor and is based off this issue: https://github.com/facebookresearch/droidlet/issues/337
Following is the checklist:

[x] sql_memory.py, remove:

[x] from droidlet.interpreter.task import Task

[x] memory_values.py:

[x] from droidlet.base_util import TICKS_PER_SEC, TICKS_PER_MINUTE, TICKS_PER_HOUR

[x] memory_nodes.py:

[x] from droidlet.base_util import XYZ, POINT_AT_TARGET, to_player_struct

[x] craftassist/sql_script delete
[x] craftassist/base_sql_mock_environment.py delete
[x] craftassist/mc_memory:

[x] from ...lowlevel.minecraft import specs as minecraft_specs
[x] from ...interpreter.craftassist import dance
[x] from droidlet.lowlevel.minecraft.build_utils import npy_to_blocks_list
[x] from droidlet.base_util import XYZ, Block
[x] from droidlet.interpreter.craftassist.word_maps import SPAWN_OBJECTS

[x] documents/ -> droidlet/documents

[x] fix all references

[x] loco_memory_nodes:

[x] from droidlet.base_util import XYZ, Pos

[x] loco_memory:

[x] from droidlet.interpreter.robot.dance import add_default_dances

[ ] Fix all os.path.joins
[x] Fix minecraft_specs imports in (remove as minecraft_specs):

[x] mc_memory.py
[x] mc_memory_nodes.py
[x] heuristic_perception.py
[x] block_helpers.py

[x] create shared_data_struct folder and move : base_util, shared_data_structs and minecraft_specs
[x] cleanup base_utils
[x] remove shared_data_struct folder
[x] remove classes from base_utils
[x] remove imports from other modules in base_utils
[x] Move out craftassist_specs.py from shared folder into lowlevel
[x] Move SPAWN_OBJECTS into lowlevel minecraft
[x] Add unit tests for functions in base_util.
[x] memory_test update
[x] Shared exception classes in shared_data_structs.py - clean
[x] Clean up tests in memory folder -> some dependencies still exist there
[x] write unit tests for every function in sql_memory.py
[x] unit tests for every function in mc_memory.py
[x] unit tests for every function in loco_memory.py
[x] unit tests for base_utils
[ ] https://github.com/facebookresearch/droidlet/issues/431 : this unit test doesn't yet work

This is now done

repo: facebookresearch/droidlet


text:
Adding support for env variables.

Description
Please include a summary of the change and link to an issue this PR fixes. Please also include relevant context. List any downstream components that might be affected by this change.
Fixes # (issue)
Type of change
Please check the options that are relevant.

[ ] Bug fix (non-breaking change that fixes an issue)
[ ] Proposes a change (non-breaking change that isn't necessarily a bug)
[ ] Refactor
[ ] New feature (non-breaking change that adds a new functionality)
[ ] Breaking change (fix or feature that would break some existing functionality downstream)
[ ] This is a unit test
[ ] Documentation only change
[ ] Datasets Release
[ ] Models Release

Type of requested review

[ ] I want a thorough review of the implementation.
[ ] I want a high level review.
[ ] I want a deep design review.

Before and After
If this PR introduces a change or fixes a bug, please add before and after screenshots (if this is causes a UX change) or before and after examples(this could be plain text, videos etc ) to demonstrate the change.
If this PR introduces a new feature, please also add examples or screenshots demonstrating the feature.
Testing
Please describe the tests that you ran to verify your changes and how we can reproduce.
Checklist:

[ ] I have performed manual end-to-end testing of the feature in my environment.
[ ] I have added Docstrings and comments to the code.
[ ] I have made changes to existing documentation where needed.
[ ] I have added tests that show that the PR is functional.
[ ] New and existing unit tests pass locally with my changes.
[ ] I have added relevant collaborators to review the PR before merge.
[ ] [Polymetis only] I have verified that all scripts in tests/scripts runs on hardware.

Codecov Report

Merging #808 (c38872b) into main (6a358a1) will decrease coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #808      +/-   ##
==========================================
- Coverage   33.11%   33.03%   -0.08%
==========================================
  Files         337      336       -1
  Lines       33060    32825     -235
==========================================
- Hits        10949    10845     -104
+ Misses      22111    21980     -131

Impacted Files
Coverage Δ

interpreter/craftassist/tasks.py
74.45% <0.00%> (-0.19%)
:arrow_down:

agents/droidlet_agent.py

agents/craftassist/tests/fake_agent.py
73.67% <0.00%> (+0.69%)
:arrow_up:

interpreter/robot/tests/world.py
89.05% <0.00%> (+0.72%)
:arrow_up:

Starting a new one, based on discussions with Leo. env will be made an argument of Process

source: gharchive/pull-request

authors: codecov-commenter girifb

repo: facebookresearch/fairo


text:
nan loss / ppl when training blstm model

I'm training a set of translation models using the suggested fconv parameters (but the model switched to blstm):
fairseq train -sourcelang en -targetlang fr -datadir data/fairseq/en-fr -model blstm -nenclayer 4 -nlayer 3 -dropout 0.2 -optim nag -lr 0.25 -clip 0.1 -momentum 0.99 -timeavg -bptt 0 -savedir data/fairseq/en-fr.blstm -batchsize 16 | tee train..blstm.log

I'm seeing loss and perplexity become nan after a few epochs:
| epoch 000 | 0001000 updates | words/s    4328| trainloss     8.72 | train ppl   420.34
| epoch 000 | 0002000 updates | words/s    4559| trainloss     6.91 | train ppl   120.29
| checkpoint 001 | epoch 001 | 0002645 updates | s/checkpnt     767 | words/s    4461 | lr 0.250000
| checkpoint 001 | epoch 001 | 0002645 updates | trainloss     7.40 | train ppl   169.38
| checkpoint 001 | epoch 001 | 0002645 updates | validloss     5.87 | valid ppl    58.37 | testloss     5.82 | test ppl    56.55
| epoch 001 | 0003645 updates | words/s    4371| trainloss     5.85 | train ppl    57.84
| epoch 001 | 0004645 updates | words/s    4373| trainloss     5.58 | train ppl    47.91
| checkpoint 002 | epoch 002 | 0005290 updates | s/checkpnt     783 | words/s    4373 | lr 0.250000
| checkpoint 002 | epoch 002 | 0005290 updates | trainloss     5.65 | train ppl    50.15
| checkpoint 002 | epoch 002 | 0005290 updates | validloss     5.25 | valid ppl    38.13 | testloss     5.21 | test ppl    36.96
| epoch 002 | 0006290 updates | words/s    4327| trainloss     5.33 | train ppl    40.15
| epoch 002 | 0007290 updates | words/s    4274| trainloss     5.24 | train ppl    37.82
| checkpoint 003 | epoch 003 | 0007935 updates | s/checkpnt     800 | words/s    4281 | lr 0.250000
| checkpoint 003 | epoch 003 | 0007935 updates | trainloss     5.25 | train ppl    38.07
| checkpoint 003 | epoch 003 | 0007935 updates | validloss     4.99 | valid ppl    31.81 | testloss     4.95 | test ppl    30.86
| epoch 003 | 0008935 updates | words/s    4235| trainloss      nan | train ppl      nan
| epoch 003 | 0009935 updates | words/s    4341| trainloss      nan | train ppl      nan
| checkpoint 004 | epoch 004 | 0010580 updates | s/checkpnt     791 | words/s    4325 | lr 0.250000
| checkpoint 004 | epoch 004 | 0010580 updates | trainloss      nan | train ppl      nan
| checkpoint 004 | epoch 004 | 0010580 updates | validloss      nan | valid ppl      nan | testloss      nan | test ppl      nan
| epoch 004 | 0011580 updates | words/s    4341| trainloss      nan | train ppl      nan
| epoch 004 | 0012580 updates | words/s    4347| trainloss      nan | train ppl      nan
| checkpoint 005 | epoch 005 | 0013225 updates | s/checkpnt     791 | words/s    4328 | lr 0.250000
| checkpoint 005 | epoch 005 | 0013225 updates | trainloss      nan | train ppl      nan
| checkpoint 005 | epoch 005 | 0013225 updates | validloss      nan | valid ppl      nan | testloss      nan | test ppl      nan

Is this something I should expect? Would you guess this is a parameter configuration issue (eg the optimizer is being too aggressive and overflowing) or does this suggest a bug (eg an overflow in the loss or perplexity code)?

try to remove the -timeavg argument for your command.

Try to decrease the log resolution (-log_interval), it will help you to see if gradients are in fact exploding. If so, try to use a smaller learning rate and/or more strict clipping.

I switched to the adam optimizer (following the readme example) and thing appear to be progressing much more reasonably.

The README contains good basic hyper-parameter settings for all supported models. Here, you're training a BLSTM model with nesterov accelerated gradient and hyper-parameters that were tuned for the fconv model. I recommend starting from the hyper-parameters mentioned in the readme (-optim adam -lr 0.0003125 -bptt 25 -clip 25) or doing a grid search to find good hyper-parameters for the nag optimizer.

authors: 1nadequacy jgehring nuance

repo: facebookresearch/fairseq


text:
data layout

Arrayfire's data layout is WHCN, so I was assuming flashlight would have the same layout but I found some code like this in CudnnUtils.cpp
  // reverse the arrays and cast to int type
  std::array<int, 4> strides = {(int)afstrides[3],
                                (int)afstrides[2],
                                (int)afstrides[1],
                                (int)afstrides[0]};

What is the data layout of flashlight?
why do you reverse the layout at the last moment before calling cuDNN?
does that mean you also flip padding, stride, and dilation?

Hi,
Arrayfire uses column major layout. So, WHCN layout in arrayfire is same as cuDNN NCHW layout once you get the data pointer for the array. Hope that answers the remaining questions.

@vineelpratap
A couple of followup questions:

what do you mean by column major layout? I am used to thinking of memory as a one dimensional array on the hardware and column-major and row-major is just how we chose to interpret the 1D array. So when we are talking about how elements of a tensor are laid out on the memory, I don't understand what you mean by row or column.

I am trying to read an array from Pytorch into flashlight. Pytorch and cuDNN uses NCHW layout where elements along W dimension are stored consecutively, then you have consecutive rows of W stored along the H dimension, and so on. How would I change the data layout of a Pytorch array to correctly read it into an array fire array?

Array fire is HWCN, not WHCN. I had problems porting layers from Pytorch to Flashlight because we thought Arrayfire was WHCN.
In the comments (source):

tensor is in WHCN order and layout in COLUMN-MAJOR

However, this is not true. Although within flashlight WHCN is equivalent to HWCN as long as you keep it consistent, this becomes a problem when we need to port from another framework such as Pytorch or Tensorflow into Flashlight. We should fix the comment

row-major NCHW is equivalent to column-major WHCN in terms of data layout
one can just simply copy weights from Pytorch's tensor into Flashlight without rearranging the dimensions.

@ThisIsIsaac  - yes, these are both true.

Array fire is HWCN, not WHCN. I had problems porting layers from Pytorch to Flashlight because we thought Arrayfire was WHCN.

I think the original comment is correct - input is WHCN for column-major layout. This actually follows your analysis from above two statements -->
one can simply reuse the Pytorch's input tensor in NCHW format for convolution
Can you give an example for this, otherwise ?

I've been asking for clarifications from AF dev via slack, and this is one of his responses:

Say an image of size WxH and C channels is loaded by ArrayFire, the data on arrayfire's array would be of size [ H W C 1 ] , scanlines sit as rows in ArrayFire array

If you want, I am going to ask my junior dev -- @minty99 -- about a specific example led us to thinking that AF is HWCN

@ThisIsIsaac
Check this example http://arrayfire.org/docs/machine_learning_2kmeans_8cpp-example.htm#a13, seems even loadImage function loads image in WHC format.
array img = loadImage(ASSETS_DIR"/examples/images/vegetable-woman.jpg", true) / 255; // [0-255]
int w = img.dims(0), h = img.dims(1), c = img.dims(2);

My reason to say arrayfire is HWCN is this -
af::array a = af::randn(2, 3);
af::array b = af::randn(2, 5);
af::print("2x3", a);
af::print("2x5", b);

prints
2x3
[2 3 1 1]
   -0.9247     2.5441    -0.3452
    0.1808     0.3516     0.2191

2x5
[2 5 1 1]
   -0.7687    -1.1948    -0.5378     0.9354     0.5787
    0.2413     0.8927     0.2270    -0.7613    -0.6174

If arrayfire is WHCN, this test should print 3-by-2 matrix and 5-by-2 matrix.

@minty99 - We shouldn't use print output to conclude anything about the data layout, since it doesn't show how the underlying raw data is stored in memory. You can try using .host operator to get the raw pointer for the array.
As @tlikhomanenko mentioned, you can try to load a rectangular image and see the dimension of the output. Also, note that height and width makes sense only when you are operating on images, while convolutions can be applies on any other domain.

one can just simply copy weights from Pytorch's tensor into Flashlight without rearranging the dimensions.

Based on this statement above, you can also try using the same input, weight in PyTorch 2D convolution and it should produce the same output. This should also confirm that flashlight using WHCN.

@vineelpratap  Thanks. We are going to copy weights from Pytorch and see if it actually works.

authors: ThisIsIsaac minty99 tlikhomanenko vineelpratap

repo: facebookresearch/flashlight


text:
Add attribution for demo scenes to README.md license section

Motivation and Context
Add attribution for demo scene assets to README.md license section
How Has This Been Tested
Local preview of README.md update
Types of changes

[x] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

@aclegg3 -- Agreed. Manolis doesn't have access to those so I'll merge this and perhaps you can add a text file in the zip the next time you update it.

authors: dhruvbatra msavva

repo: facebookresearch/habitat-sim


text:
Release of MyoSim

Thank you for the great platform you built. May I ask if you are planning to open-source MyoSim as written in your paper? Thank you

Thanks for your interest. Yes, open-source the MyoSim models, as well as the automatic model converting pipeline, is our next focus in the following half-year.

@sherwin0402: Currently available models can be found here -- https://github.com/facebookresearch/myosuite/tree/main/myosuite/envs/myo/assets

@vikashplus @HuaweiWang Thanks for the information. Are lower limb models like gait 2392, 2354 etc part of your plans for MyoSim as well? Or is upper limb the focus now?

@sherwin0402 We officially released the model conversion pipeline today and it contains a few leg models converted as examples. Please have a look!

authors: HuaweiWang sherwin0402 vikashplus

repo: facebookresearch/myosuite


text:
Independent ask and tell

Steps to reproduce

I have a function to minimize, which is extremely slow.
Due to the fact, that function has several steps inside, I could detect bad outcome early and return some big constant, say 10e5 (possible optima is 0).
Also, for some parameters combination it's impossible to get any result, so I also return 10e5.
To optimize such a function with nevergrad I need an ability to call ask and tell completely independent. Is it possible?

Relevant Code
What do I want could be coded like:
for param, value in somehow_before_obtained_results:
  optimizer.tell(param, value)

for _ in range(optimizer.budget):
  x = optimizer.ask()
  value = square(*x.args, **x.kwargs)
  if value<10e5:
      optimizer.tell(x, value)
recommendation = optimizer.provide_recommendation()

Something like the following should work:
for param, value in somehow_before_obtained_results:
  candidate = optimizer.create_candidate.from_call(*params)
  # may fail for some optimizers who do not support tell without ask (but most are OK)
  optimizer.tell(candidate, value)

for _ in range(optimizer.budget):
  x = optimizer.ask()
  value = square(*x.args, **x.kwargs)
  optimizer.tell(x, value)
recommendation = optimizer.provide_recommendation()

I suppose your question mostly concerned the if condition. I removed it because while some algorithms could deal with it, all population based algorithm need to have a tell for each ask (otherwise the population does not get updated, and the algorithm blocks at some point). However if you detect the value early, why dont you just send a large value instead of skipping the tell? This is also more informative for the optimizer.

Well, I actually send =).
I tried it with BayesianOptimization library, and it leads to stuck the process: optimizer starts to return params only from the edges (min/max for every dimension).
I will try it with nevergrad.
Thanks a lot!

All BO based algorithms in nevergrad use BayesianOptimization under the hood, so you will probably have the same problem if you use them, but most other optimizers should be fine. TwoPointsDE and OnePlusOne are usually safe bets.

So, I've updated nevergrad to 0.4.0.
Suggested way of creating candidates optimizer.create_candidate.from_call() was removed from code, but still present in tell method description.
As I could find from changelog and repo code I should use something like instrum.spawn_child().set_standardized_data() now. Could I ask someone to provide an example?

Hi,
Sorry this is proving difficult, I tried to fix and improve the documentation regarding inoculation in #595.
I merged it to ease the visualization in the documentation, it should be live:

https://facebookresearch.github.io/nevergrad/optimizers_ref.html#nevergrad.optimizers.base.Optimizer.tell
https://facebookresearch.github.io/nevergrad/optimization.html#telling-non-asked-points-or-suggesting-points

Does that answer your question?
In any case, thank you for flagging the erroneous documentation, and let us know if you have ideas of how to make it clearer ;)

Thanks a lot, now it's clear.
As I could guess from changelog, there is some change in points storage mechanic in recent times also: #571. If this is right, there is one more uncertain part.

Let's assume I've done complete nevergrad run and got some output - let's call it best_y and best_args.
After some time I've got some more data and want to make new nevergrad run.

Could you please say am I right or not for the following statements:

Doing one hand-made tell call from last (best_args, best_y) is better than none.
Doing several (2-10-50-100?) hand-made tell calls from previous runs is better than one.
Doing more than 1000 calls is useless.
Doing hand-made calls for bad outcomes (bad parameter combinations, early stops) is somewhat useless too.
?

Doing one hand-made tell call from last (best_args, best_y) is better than none.

Yes, but not all optimizers support tells that where not asked. (I'll provide another option at the end)

Doing several (2-10-50-100?) hand-made tell calls from previous runs is better than one.

Somehow: some optimizers (like most standard variants of OnePlusOne) only use the one best point so far, so more than 1 may not be necessary, but wont hurt. Other optimizers use a population of points (DE variants, TBPSA, CMA), usually in our implementations this is around 40 points (though that may come to change, 40 is basically a good enough default usually, but hopefully we can find something more adaptative eventually). Those optimizers will keep the 40 bests or so if you tell them more.

Doing more than 1000 calls is useless.

Indeed, we wont keep track of them. Some non-standard variants of OnePlusOne do crossovers with points in the archive, which size I recently decreased as you mentioned, but that's nearly the only one that could maybe make some use of it, and it's not sure it would be a good use.

Doing hand-made calls for bad outcomes (bad parameter combinations, early stops) is somewhat useless too.?

Indeed, they are dropped for most optimizers. However now that you mention it, BO optimizers may be able to make use of all the information you provide since they have some underlying model. In this case you the more tell you do the better I would assume. I've never been a fan of BO though.
Another option to consider: setup your parametrization to have the initialization you want. With the new parametrization you can set the value manually, and it will be used as starting/reference point. You should also update the mutation sigma (param.set_mutation(sigma=x)), which is basically equivalent to a step size. It is initialized to 1 by default.
Does that clarify it for you? If you see specific areas of the documentation I could clarify don't hesitate to point it out. I don't know how to integrate all these information in a clear way :s

A lot of useful info, thank you!

setup your parametrization to have the initialization you want. With the new parametrization you can set the value manually

I'm doing it in the following manner:
instr_dict = {
    'param1':
    ng.p.Scalar(init=best_args['param1'],
                lower=param_bounds['param1'][0],
                upper=param_bounds['param1'][1]),
    'param2':
    ng.p.Log(init=best_args['param2'],
             lower=param_bounds['param2'][0],
             upper=param_bounds['param2'][1]),
    'param3':
    ng.p.TransitionChoice(
        np.arange(param_bounds['param3'][0], param_bounds['param3'][1]))
}

instrum = ng.p.Instrumentation(**instr_dict)

for args, val in previous_outputs:
    cand = instrum.spawn_child(new_value=(tuple(), args))
    optimizer.tell(cand, val)

Do you advice me to do
instrum.value = (tuple(), best_args)

After? It throws RuntimeError: Cannot modify frozen Parameter. Does it really neded if I set init arguments in instrum?

I don't know how to integrate all these information in a clear way

I found your last answer https://github.com/facebookresearch/nevergrad/issues/263#issuecomment-605868169 is a candidate to be completely transferred to https://facebookresearch.github.io/nevergrad/optimization.html#telling-non-asked-points-or-suggesting-points

After? It throws RuntimeError: Cannot modify frozen Parameter. Does it really needed if I set init arguments in instrum?

Parameters are "frozen" as soon as they interact with the optimizer, to avoid side effects. Setting the value of the parametrization does not raise the exception if you do it before instantiating the optimizer.
If you have provided init values then you are all fine though. Note that by default, when you set lower and upper bounds, sigma (= the step size) is set to 1/6th of the full range. You may want to update it if you have a better prior.

authors: Dronablo jrapin

repo: facebookresearch/nevergrad


text:
Expose fork method on python API and sync windows state with backend

Description
This PR will add the following features:

expose a .fork() method on the client python API with the same functionality as the UI.
sync layout state with the backend. We can resize the windows and reload the page that the windows will keep the layout state.

Motivation and Context
#323
How Has This Been Tested?
Installing visdom directly from my repo and testing by hand.
Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] For JavaScript changes, I have re-generated the minified JavaScript code.

cc @JackUrb

I just need some guidance on what should the server reply back to the client on the following cases:

https://github.com/facebookresearch/visdom/pull/609/files#diff-ddbe57b92511a8307bcf5999eb519e6cR860
https://github.com/facebookresearch/visdom/pull/609/files#diff-ddbe57b92511a8307bcf5999eb519e6cR863

@JackUrb I thank you for guiding me through this feature :)
I've fixed what you requested and I also noticed I coded a bug on handling layout_item_update on backend.
It was overwriting the layout state of the env with a single window. I fixed that as well and now the feature works perfectly in any scenario of a fork.
Before that fix if I resized a window and if I immediately forked that env, without saving, the new env wouldn't pick up those changes.

Any ETA for this @JackUrb ?

Sorry for the delay @ricardofbarros - I've been split across a number of different tasks. I'll get to it this week in my conference downtime

Thanks for the reply :)

authors: JackUrb ricardofbarros

repo: facebookresearch/visdom


text:
Update deploy to build wheels for each supported python version

Adds wheels for #146

@gir-bot lgtm

@gir-bot lgtm

I would recommend building all the parts, using actions/upload-artifact@v2, then then doing the submit once in a download job with needs. Also 1.4.1 is the current version of the pypa action, IIRC.

I would recommend building all the parts, using actions/upload-artifact@v2, then then doing the submit once in a download job with needs. Also 1.4.1 is the current version of the pypa action, IIRC.

That way, if there was a problem building a wheel, it would stop before submitting the files.

That way, if there was a problem building a wheel, it would stop before submitting the files.

@henryiii, thanks for the info. I'll definitely upgrade to the latest pypa action. It is one of those things that wasn't broke, so it kept slipping my mind. As for artifacts, it is something to consider. It hasn't been needed for most of my projects as both sdist and bdist_wheel have been built in one step. This is the first project where I am uploading through multiple steps.
I'll take a look in the future. It definitely sounds like a good idea.

@henryiii, thanks for the info. I'll definitely upgrade to the latest pypa action. It is one of those things that wasn't broke, so it kept slipping my mind. As for artifacts, it is something to consider. It hasn't been needed for most of my projects as both sdist and bdist_wheel have been built in one step. This is the first project where I am uploading through multiple steps.
I'll take a look in the future. It definitely sounds like a good idea.

authors: facelessuser henryiii

repo: facelessuser/backrefs

license: mit

license_source: bigquery


text:
The link for "Directional Movement Index (DMI)" in indicators doc page is incorrectly pointing to ADX

What happened?
In https://python.stockindicators.dev/indicators/#content the URL link for Directional Movement Index (DMI) is acrtually pointing to https://python.stockindicators.dev/indicators/Adx/#content
Code usage
No response
Log output
No response

The Directional Movement Index plus/minus components are included as a component of Average Directional Movement Index (ADX), and are part of the same indicator, which is why the links points to the same page and not really a bug.

On inspection, I can see DMI (shown as dx below) is calculated as part of ADX, but am unsure as to why I didn't add it to the output result.  You can calculate this yourself from the ADX output as a workaround.
https://github.com/DaveSkender/Stock.Indicators/blob/7c362fffed60a274058f2ed28259b9f73cae347a/src/a-d/Adx/Adx.Series.cs#L110-L114
I'm converting this to an enhancement in the backlog, to add DMI to the output result.

authors: DaveSkender carusyte

repo: facioquo/stock-indicators-python


text:
[MSVC] Drop constexpr on absl::string_view to avoid build break

When building ros_type_introspection on Windows by MSVC, the usage of constexpr on absl::string_view caused the following build break:
ros_type_introspection\include\ros_type_introspection/stringtree_leaf.hpp(102): error C2131: expression did not evaluate to a constant
abseil-cpp\abseil_cpp\absl/strings/string_view.h(186): note: a non-constant (sub-)expression was encountered

A similar issue was also reported on https://github.com/abseil/abseil-cpp/issues/352. For now, before a better solution comes out, I recommend to drop constexpr to avoid build break.

Thanks!

authors: facontidavide seanyen


text:
Derive Clone for MiniCocoon

#29  and #30

@Deaths-Door Published in the version 0.4.3

authors: Deaths-Door fadeevab

repo: fadeevab/cocoon


text:
SIGSEGV: Pinging...panic: runtime error: invalid memory address or nil pointer dereference

Hi @alfiyansys,
Thanks for reporting the error! I'm aware of the issue and working on a fix. I'll keep you updated on the progress and let you know once it's resolved.
Feel free to reach out if you have any questions or need assistance.

Hi @alfiyansys ,
I've already made a pull request (PR) to address the problem. You can find the PR https://github.com/fadhilyori/subping/pull/8 for review.
Please take a moment to check if the proposed changes resolve the issue. Feel free to test it out and provide feedback. I appreciate your help in verifying the fix.

let me check

confirmed to be working

PR #8 has already been merged, thank you

authors: alfiyansys fadhilyori

repo: fadhilyori/subping


text:
Nokia Lumia 625H (RM-943) problems after upgrade to 10.0.10586.107 and 10.0.10549.4.

Procedures:

Install Stock FFU over thor2.
Upgrade to 8.10.14234.375 by OTC Updater.
Run the app under Administrator.
There will be 2 Updates: 10.0.10549.4 and 10.0.10586.107.

Chooses 10.0.10549.4

Yells: Updates are downloaded but not installed (800b010a).

Chooses 10.0.10586.107

Goes to Recovery runs over 2/3 progress bar.
BSOD: ":-("
Proceeds to bootloop in recovery.

I managed to upgrade it to 10.0.10586.107 with HCTSW_WXMSOUP_V5.3beta6.update.lzma2.7z
Difference betweeen CAB files applied between hikari_calyx's one and yours was 1 CAB file.
So hopefully you can fix this soon.

About the build 10.0.10549.4, use this patch to get it installed.
Can you send me the DeviceLog.cab from the Universal Updater, it can help me to fix it.
And which of the CAB is different?


text:
Fix typo

Ночные незаметочки)

👍

authors: faizrr mrexox

repo: faizrr/use-api-polling


text:
CLI hangs when user name contains special characters

When user's full name contains special characters, such as Ö or ş, some fal commands just hang, e.g. fal key list. My suspicion is that it's a serialization issue.

Wow, great catch!

Though UserKeyInfo doesn't even contain user name, so not super clear right away where specifically it fails. Will need to take a closer look.

authors: efiop mederka

repo: fal-ai/fal


text:
Go SDK

We would like to offer a simple Go consumer of a gRPC output stream based on the gRPC output stream defined in #528
The go library should be easily consumable by other Go programs and should have clear guarantees defined regarding sync/async behavior.
This should effectively be a mapping of the *.h output methods used for our other Falco output streams.

FYI @fntlnz

repo: falcosecurity/falco


text:
Fix base64 flag

For the people who uses the  -b or --print-base64 command argument,
"s_inspector->set_buffer_format(sinsp_evt::PF_NORMAL);" will restore the state to normal dot escaped string instead of base64 (that they chose....).

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

If contributing rules or changes to rules, please make sure to also uncomment one of the following line:

/kind rule-update

/kind rule-create

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area engine

/area examples

/area rules

/area integrations

/area tests

/area proposals

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #987
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

@realgam3: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign kris-nova
You can assign the PR to them by writing /assign @kris-nova in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

dbc981c Fix issue #987

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Welcome @realgam3! It looks like this is your first PR to falcosecurity/falco 🎉

/kind bug

I'm still waiting to merge this because I'm trying to understand how this is related to https://github.com/falcosecurity/falco/pull/581

That line was initially removed to support base64 but then was re-added because json was broken https://github.com/falcosecurity/falco/pull/410/files#diff-619b66cc1dd5ce6c64d2aa45a121ad63L127

Here's what I get after testing this patch:
Output with this change:
{"output":"{\"evt.buffer\":\"IBtbMG0bWzAxOzM0bURlc2t0b3AbWzBtICAgG1swMTszNG1Eb2N1bWVudHMbWzBtICAgG1swMTszNG1Eb3dubG9hZHMbWzBtICAgG1swMTszNG1nbxtbMG0gICAbWzAxOzM0bVBpY3R1cmVzG1swbSAgIBtbMDE7MzRtUHJvamVjdHMbWzBtICAgG1swMTszNG1WaWRlb3MbWzBtICAbWzAxOzM0bSdWaXJ0dWFsQm94IFZNcycbWzBtCg==\",\"evt.time\":1580811468970787674}","priority":"Notice","rule":"Test renzo","time":"2020-02-04T10:17:48.970787674Z", "output_fields": {"evt.buffer":"IBtbMG0bWzAxOzM0bURlc2t0b3AbWzBtICAgG1swMTszNG1Eb2N1bWVudHMbWzBtICAgG1swMTszNG1Eb3dubG9hZHMbWzBtICAgG1swMTszNG1nbxtbMG0gICAbWzAxOzM0bVBpY3R1cmVzG1swbSAgIBtbMDE7MzRtUHJvamVjdHMbWzBtICAgG1swMTszNG1WaWRlb3MbWzBtICAbWzAxOzM0bSdWaXJ0dWFsQm94IFZNcycbWzBtCg==","evt.time":1580811468970787674}}

Output without this change:
{"output":"11:22:08.168296981: Notice Yo (error= .[0m.[01;34mDesktop.[0m   .[01;34mDocuments.[0m   .[01;34mDownloads.[0m   .[01;34mgo.[0m   .[01;34mPictures.[0m   .[01;34mProjects.[0m   .[01;34mVideos.[0m  .[01;34m'VirtualBox VMs'.[0m.)","priority":"Notice","rule":"Test renzo","time":"2020-02-04T10:22:08.168296981Z", "output_fields": {"evt.buffer":" .[0m.[01;34mDesktop.[0m   .[01;34mDocuments.[0m   .[01;34mDownloads.[0m   .[01;34mgo.[0m   .[01;34mPictures.[0m   .[01;34mProjects.[0m   .[01;34mVideos.[0m  .[01;34m'VirtualBox VMs'.[0m.","evt.time":1580811728168296981}}

/close in favor of #1033

This was broken since 0.13.0

authors: fntlnz leodido poiana realgam3


text:
chore(userspace/libsinsp): expose plugin schema in sinsp_plugin::info vector returned by plugin_infos()

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area driver-kmod

/area driver-ebpf

/area libscap

/area libsinsp

/area tests

/area proposals

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

As a self documentation, perhaps it would be useful to call list_open_params() for source plugins too, and returning the list of possible open params?

I think list_open_params is a bit different, because unlike infos it has to be called after init().

Yeah, and while schema is already retrieved, list_open_params is never called (ie: i'd have to call it for each plugin in place in get_plugin_infos, and it can lead to terrible performance).
That is exactly why i avoided adding it for now.

This can be closed IMO; it gives no value considering that you can call "sinsp::get_plugins()" API and have access to full plugin API.

authors: FedeDP jasondellaluce poiana

repo: falcosecurity/libs


text:
update: avoid owning event pointer offset in scap test engine

What type of PR is this?
/kind cleanup
Any specific area of the project related to this PR?
/area libscap
/area libsinsp
/area tests
Does this PR require a change in the driver versions?
What this PR does / why we need it:
Which issue(s) this PR fixes:
The libscap's test engine currently accepts a pointer to an event array, and then stores an internal offset index of that array to proceed with the event generation. This approach limits the degrees of freedom of the engine users, such as the sinsp tests, because they need to provide an increasingly-larger array every time they want to change/add events at runtime between one next call and another.
This PR fixes this by moving the pointer ownership to the caller. The test engine will iterate through the event array using pointer arithmetic and always dereferencing the head left-most element, and by decrementing the size count. Both the pointer and the size count can be manipulated from the caller.
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
NONE

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: jasondellaluce
Once this PR has been reviewed and has the lgtm label, please assign molter73 for approval by writing /assign @molter73 in a comment. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 46cc931f85aadbd4fb48d1b001e9ea2405e85a85

authors: jasondellaluce poiana


text:
🛑 Archive is down

In 2887e66, Archive (https://archive.stsci.edu) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Archive is back up in 0d6d619 after 28 minutes.

repo: falkben/mast-status


text:
Do VRFs count as fancy?

Perhaps verifiable random functions should be tracked here as well. For whatsapp key transparency we use ECVRF from RFC9381, and I think there are likely a bunch of blockchain applications for VRFs in leader selection
Edit: sorry for the shameless plugs 😅

I agree.

authors: bwesterb kevinlewi

repo: fancy-cryptography/fancy-cryptography


text:
stage for exec

go mod
	golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586
	golang.org/x/crypto v0.0.0-20200220183623-bac4c82f6975
	gopkg.in/yaml.v2 v2.2.8
	k8s.io/api v0.17.3
	k8s.io/apimachinery v0.17.3
	k8s.io/client-go v0.17.3
	k8s.io/api v0.18.0
	k8s.io/apimachinery v0.18.0
	k8s.io/client-go v0.18.0

这是个什么功能？

我把vendor 去掉 你重新同步一下吧

好的，这个是exec命令。我记得有个issue的

ok 刚看到。你同步一下吧 后面不用vendor了

build 相关脚本也需要改一下. 去掉vendor了. @cuisongliu

好像搞错了, 我重新pr吧...

authors: cuisongliu oldthreefeng

repo: fanux/sealos


text:
关于unimodal1和unimodal2的问题

请问能否提供Table 1中unimodal1和unimodal2的代码呢？或者你能否告诉我你是如何训练得到这些结果的呢？我自己写的代码在CREMAD中训练出来的结果只有visual 30% ; audio 60%。性能更好的unimodal对我目前的工作很重要。邮箱：1425821304@qq.com

Hi,
I would like to request the same, I tried the unimodal ones for AVE with the given code by running first mp4_to_wav and then video_preprocessing and the results I got from the unimodal networks VClassifier and AClassifier are almost random. I believe there must be something in the unimodal training pipeline that we are missing. Could you please upload the rest of the code?
Thanks!

请问能否提供Table 1中unimodal1和unimodal2的代码呢？或者你能否告诉我你是如何训练得到这些结果的呢？我自己写的代码在CREMAD中训练出来的结果只有visual 30% ; audio 60%。性能更好的unimodal对我目前的工作很重要。邮箱：1425821304@qq.com

I don't know the specific reasons for why you cannot get the uni-modal results. I use 3 frames for each video clips for CREMA-D. Do you just use one frame for that?

Hi,
I can replicate and even surpass the unimodal in audio but the 3 frames per clip do not really get improved while training. I have tried several things, changing the model, optimizers and hyperparams, and the transformations but nothing flourished till now. Could you possibly provide a notebook with dependencies on your code to show its working?
Thanks!

I am coming back to this one because I noticed something strange. I trained the unimodal vision model with 3 frames a bit further till it reached 100 epochs without improvement (~ 250 epochs in total) and the accuracy went up to ~74%. That confused me a bit since the rest of the numbers match the ones from the paper. Any intuition?

Could you upload also the script tuning_uniclassifier that appears in your bash file?

authors: fanyunfeng-bit kkontras yuquanqingyin

repo: fanyunfeng-bit/Modal-Imbalance-PMR


text:
bug(examples/replicate-data-postgres): Duplicate Key Violation in onLinkAdd function

https://github.com/farcasterxyz/hub-monorepo/blob/65a4faff6656dba276468f92bd9efde49e2dbcca/packages/hub-nodejs/examples/replicate-data-postgres/hubReplicator.ts#L639
Description:
I encountered an error in the onLinkAdd function of the codebase. The error message suggests that there is a violation of a unique constraint, specifically "links_fid_target_fid_type_unique," when inserting new records into the "links" table.
Postgres will fail, breaking the Promise. This will cause the application to fail and exit with status code 1.
Error Message:
db-postgres-1  | 2023-08-05 12:51:33.130 UTC [36] ERROR:  duplicate key value violates unique constraint "links_fid_target_fid_type_unique"
db-postgres-1  | 2023-08-05 12:51:33.130 UTC [36] DETAIL:  Key (fid, target_fid, type)=(976, 251, follow) already exists.
db-postgres-1  | 2023-08-05 12:51:33.130 UTC [36] STATEMENT:  insert into "links" ("hash", "timestamp", "target_fid", "type", "fid") values ($1, $2, $3, $4, $5) on conflict ("hash") do nothing

...
PostgresError: duplicate key value violates unique constraint "links_fid_target_fid_type_unique"

Expected Behavior:
The onLinkAdd function should gracefully handle conflicts by retaining the existing record in the "links" table when encountering duplicate entries for the "hash" column. In case of a conflict, the function should take no action and proceed with inserting the next record without throwing an error or modifying the existing data.
Proposed Solution:

Modify the onLinkAdd function to gracefully log out the failure.
try { } catch { } finally { }, however none of the other functions are written this way.

Environment Details:
Using docker-compose.yml provided.
Node.js version: 20
PostgreSQL version: 15
Operating System: macOS M1/ARM
Steps to Reproduce:

Deploy the provided Docker Compose configuration.
Trigger the onLinkAdd function with a set of messages that include duplicate values for "hash," "fid," "target_fid," and "type" columns.
Observe the logs for the Postgres container to encounter the duplicate key violation error.

+1 Also running into this

repo: farcasterxyz/hub-monorepo


text:
Feature Request: Month / year picker

Request new feature that picker can only max view in month or year
like a initialValue props, but if we click month that will be get data of the month

@devcodemada I recently used this library and could be nice if when I select the month get a onChange function for month and year, and not select the month and then the day.

Would be nice to be added, as in my case, i only need to select a month

react-native-ui-datepicker+2.0.4.patch
You can try adding this patch. I added it with a simple implementation.
import Moment from 'moment';
import DateTimePicker from 'react-native-ui-datepicker';
<DateTimePicker
calendarTextStyle={styles.calenderText}
headerTextStyle={styles.headerText}
selectedItemColor={COLOR_PRIMARY}
selectedTextStyle={styles.selectedText}
mode={'single'}
locale={'tr'}
initialView={'month'}
onlyMonth
buttonPrevIcon={()}
buttonNextIcon={()}
firstDayOfWeek={1}
date={start_date}
minDate={this.state.first_date}
maxDate={Moment()}
onChange={({date}) => _onChangeFunction()}
/>


text:
Improve the resolution of relative URLs.

Currently, absolute URLs are being generated by simply joining the relative part for the hostName which is not always true.
https://godoc.org/net/url#URL.ResolveReference can be used to systematically resolve relative URLs.

@farkaskid Should I start on this?

Sorry, just raised a PR for this one. 😅  #18

#18 merged.

repo: farkaskid/WebCrawler


text:
feat(env): adapter vite env

Description:

Add SSR env
Add base_url env

Resolved: #1525

Thanks, please update test snapshot via npm run test and add a patch changeset via npx changeset

repo: farm-fe/farm


text:
Added _read_mp4box_stem_titles function

Basically, for stems which have been created by the NI creator have the titles of the streams in the udta metadata.
I added the functionality to try to read the stream titles if MP4Box is available, else fall back to original default numbered streams.
This is useful if you want to extract wav files using the stem2files, and instead of getting filenames like "stem_0.wav", you can get "Drums.wav" or "Vox.wav" if this information is available in the stem files.
The code is taken almost verbatim from the StemMetadataViewer class in https://github.com/axeldelafosse/stemgen/blob/master/ni-stem/_internal.py

I added sudo apt-get update to fix the previous issue of 404 errors while installing apt-get packages.
But still, the CI failed with some errors in FFmpeg unable to find x264 libraries.
ffprobe: error while loading shared libraries: libx264.so.138: cannot open shared object file: No such file or directory

@faroit sorry for the ping -- can you have a look when you have some free time please? 🙏

repo: faroit/stempeg


text:
Using fastpages github action on other blogs

The problem: When I use the fastpages github action on my own blog with the flag BOOL_SAVE_MARKDOWN set to true I end up stuck in a loop as the Bot is commiting to master.

What would be the best way to stop this from happening?
I tried using if: "!contains(github.event.commits.message, 'Bot')" and variations of the same idea but i can't seem to be able to skip triggering my build jekyll action. Is there anything specific about the bot updates i can use to filter out those commits? using the head_commit or commits.message doesn't seem to work.
Any help would be greatly appreciated, thanks!

Yes this is exactly why this option is turned off by default.  You have to pull the changes locally each time GitHub makes a change on your behalf.  It is not recommended, and is only used in a special case where you need to troubleshoot.
Otherwhise, I would suggest you only do this conversion locally (see the development guide in the README), without using GitHub Actions.

authors: chekos hamelsmu

repo: fastai/fastpages


text:
Only one of my jupyter notebooks shows up on blog

Required Prerequisites for filing a bug

You must follow ALL the steps in the [troubleshooting guide]
(https://github.com/fastai/fastpages/blob/master/_fastpages_docs/TROUBLESHOOTING.md).  Not doing so may result in automatic
closure of the issue.

I carefully checked the README.md of the fastpages repo as well as the troubleshooting guide and I still can't find the reason why the site doesn't render all my notebooks (only the first one appears there); I apologize in advance if the mistake is on my part - but I really have no idea what the problem is.

Required information

Steps to reproduce the problem
A link to the notebook or markdown file where the error is occurring
If the error is happening in GitHub Actions, a link to the specific error along with how you are able to reproduce this error.  You
must provide this in addition to the link to the notebook or markdown file.
A screenshot / dump of relevant logs or error messages you are receiving from your local development environment. Instructions
of running a local development server is provided in the [development guide] >
(https://github.com/fastai/fastpages/blob/master/_fastpages_docs/DEVELOPMENT.md).

Proceeding by parts in order to better elaborate the problem that is happening with my repository:

Even adjusting all the front matter according to the tutorial, only the first notebook I added on the _notebooks page appears on my blog, all the others don't appear.

I tried to access the actions tab of the repository in search of an error in the construction of the page, however when accessing it apparently everything is being built as expected:

If I'm making some weird mistake, my repository can be accessed through the following link:

https://github.com/REsteche/REsteche_blog

and the specific noteeboks page via:

https://github.com/REsteche/REsteche_blog/tree/master/_notebooks

I have updates to give!
Unfortunately they just make the situation weirder for me 😅
If I copy + paste a page from fastai repo I can make it appear on the screen - however if I change a single letter from Front Matter,  lets say in its title, it just disappears from the blog as before.
If I copy the body of one of my repositories and post it with Front Matter from a post from fastai, the post appears on the blog normally:

but I find the following screen when I click on it :

I don't know if it's a problem with my notebooks itself that is causing this behavior... I'm using simple notebooks generating through Jupyter in the anaconda system. (and I'm still super confused)

Conclusion
Well, after hours of documentation and lines of HTML read, I found that the problem was the format and extension of the images that were inside my notebooks. If eventually someone finds themselves in a similar and apparently inexplicable situation, I can show you what I did to get the blog to work naturally again, just get in touch,


text:
Online demo is down

both demos return 404 page not found

Cheers

repo: fastapi-admin/fastapi-admin


text:
📝 Update includes in docs/en/docs/advanced/additional-responses.md

📝 Update includes in docs/en/docs/advanced/additional-responses.md
Part of #12554

Thank you! @krishnamadhavan :nerd_face:

authors: alejsdev krishnamadhavan

repo: fastapi/fastapi


text:
Enhancement: ResponseValidationError instead of ValidationError

Closes #3520
Sometimes it's hard to reproduce response validation error, especially if your app proxies or collects data from several external sources.
With new ResponseValidationError and it's request_body and response_body fields, users can install their own exception handlers to log/sentry both request and failed response.
This change is backward compatible with previously recommended ValidationError approach.

#3520

Thanks for the interest! This was handled in another PR at some point, now there's a ResponseValidationError that you can catch and inspect, it's not documented yet, but you can already use it. :nerd_face:
Given that, for now, I'll close this one. But thanks for the effort! :cake:

@tiangolo
It doesn't seem we can catch the ResponseValidationError . In my case for example, the response object doesn't fit the response schema.
I get:
ipdb> type(exc)
<class 'pydantic_core._pydantic_core.ValidationError'>
ipdb> isinstance(exc, ResponseValidationError)
False

fastapi==0.115.5
pydantic==2.9.2
pydantic_core==2.23.4

@gistart Haha, I didn't realize this PR was from 3 years ago. Yeah that would be great. Thanks.

authors: gistart seperman tiangolo


text:
update gitignore with test data

Update .gitignore to include the following

data_test/*
data_test_cached/*
test.wav

:tada: This PR is included in version 0.1.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

repo: fastaudio/fastaudio


text:
Support for multi-value query parameters in API Gateway

Closes #29

thx, landed in v1.1.0

@adrai Thank you :)

authors: adrai hakimio

repo: fastify/aws-lambda-fastify


text:
Unknown request body when using onRequest hook

Prerequisites

[X] I have written a descriptive issue title
[X] I have searched existing issues to ensure the issue has not already been raised

Issue
When using a hook like onRequest, the schema validation doesn't work anymore. If e.g. a body schema is specified in MySchema, the request body will be of type unknown:
const appWithTypeProvider = app.withTypeProvider<JsonSchemaToTsProvider>();
appWithTypeProvider.post('/', {schema: MySchema, onRequest: randomMethod}, async (request) => {
  request.body // unknown
});

Is this wanted behaviour?

Notice: in the onRequest hook, request.body will always be undefined, because the body parsing happens before the preValidation hook.

https://www.fastify.io/docs/latest/Reference/Hooks/#onrequest
Checkout the hooks execution order here: https://www.fastify.io/docs/latest/Reference/Lifecycle/#lifecycle

In this case, the body is also of type unknown when using a preHandler hook. According to https://www.fastify.io/docs/latest/Reference/Lifecycle/#lifecycle, the preHandler hook is called after the validation, so it should be defined right?

the preHandler hook is called after the validation, so it should be defined right?

Yes, the body is set on that time

In this case, the body is also of type unknown when using a preHandler hook. According to https://www.fastify.io/docs/latest/Reference/Lifecycle/#lifecycle, the preHandler hook is called after the validation, so it should be defined right?

I stumbled on to the same issue with the preHandler hook. So if I define the schema and use an inline hook, the request body is inferred correctly in the actual handler:
server.post(
  "/items",
  {
    schema: {
      body: {
        type: "object",
        properties: {
          name: {
            type: "string",
          },
        },
        required: ["name"],
      } as const,
    },
    // Inline dummy hook
    preHandler: async (request, reply) => {},
  },
  async (request, reply) => {
    // No problems here, request.body.name is a string like it should be
    console.log(request.body.name);
  }
);

However, if I extract the hook into its own function (for reuse purposes and whatnot), then request.body is unknown:
const dummyHook = async (request: FastifyRequest, reply: FastifyReply) => {};

server.post(
  "/items",
  {
    schema: {
      body: {
        type: "object",
        properties: {
          name: {
            type: "string",
          },
        },
        required: ["name"],
      } as const,
    },
    // Hook is now a dedicated function
    preHandler: dummyHook
  },
  async (request, reply) => {
    // TS2571: Object is of type 'unknown'.
    console.log(request.body.name);
  }
);

Is this expected behaviour? Reusing hooks sounds like a common scenario so I might be missing something. One way to work around this is to call the shared hook from the inline hook but it comes off as being a bit redundant:
  {
    schema: {
        ...
    },
    preHandler: async (request, reply) => {
      await dummyHook(request, reply);
    },
  },
  async (request, reply) => {
    // No problems here, request.body.name is a string like it should be
    console.log(request.body.name);
  }

I'm also facing issue described in last comment.
@Eomm maybe it worth to reopen this for visibility that specifying preHandler with dedicated function breaks typing?

Please re-open. I've been hitting this issue as well and haven't found a feel-good work-around. @mcollina.

Ran into this issue as well with type information being lost when using a prehandler that is an external method

authors: Eomm SkeLLLa coocos gurumaxi james-gardner johanbook


text:
Manage global environment variables in the new frontend

Basically the view of /environment_variables_erb ideally listing, adding, editing and deleting global ENV variables

Do we want a whole different page for all the ENV vars, or would we want to have the ability to edit them on each level (project level and CI top level)?

I think we can follow what we wrote up in the our use cases with

edit_global_env_variables.md
edit_project_env_variables.md

Basically the project level ENV variables would be on a project settings level, while the other one is global, somewhere in the fastlane.ci global settings (or whatever we decide to call it or how we implement it)
Let me know if those docs have enough information, or if I should update them to describe things in more detail 👍

authors: KrauseFx nakhbari

repo: fastlane/ci