text:
Velox doesnt build on MacOS 14.2 sdk

Problem description
When building Velox on a MacOs with 14.2 sdk, we get the following error :
[630/1927] Building CXX object velox/common/base/CMakeFiles/velox_exception.dir/Exceptions.cpp.o
FAILED: velox/common/base/CMakeFiles/velox_exception.dir/Exceptions.cpp.o
/Users/kpai/homebrew/bin/ccache /Library/Developer/CommandLineTools/usr/bin/c++ -DBOOST_ATOMIC_DYN_LINK -DBOOST_ATOMIC_NO_LIB -DBOOST_CONTEXT_DYN_LINK -DBOOST_CONTEXT_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_FILESYSTEM_NO_LIB -DBOOST_PROGRAM_OPTIONS_DYN_LINK -DBOOST_PROGRAM_OPTIONS_NO_LIB -DBOOST_REGEX_DYN_LINK -DBOOST_REGEX_NO_LIB -DBOOST_SYSTEM_DYN_LINK -DBOOST_SYSTEM_NO_LIB -DBOOST_THREAD_DYN_LINK -DBOOST_THREAD_NO_LIB -DFMT_LOCALE -DFOLLY_HAVE_INT128_T=1 -DGFLAGS_IS_A_DLL=0 -I/Users/kpai/src/velox/_build/debug/_deps/protobuf-src/src -I/Users/kpai/src/velox/. -I/Users/kpai/src/velox/velox/external/xxhash -isystem /Users/kpai/src/velox/velox -isystem /Users/kpai/src/velox/velox/external -isystem /Users/kpai/homebrew/include -isystem /usr/local/include -isystem /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/include -isystem /Users/kpai/homebrew/opt/openssl@1.1/include -mcpu=apple-m1+crc -std=c++17 -fvisibility=hidden -fvisibility=hidden -fvisibility-inlines-hidden -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-range-loop-analysis          -Wno-mismatched-tags          -Wno-nullability-completeness -Werror -g -std=gnu++17 -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk -fPIC -fcolor-diagnostics -MD -MT velox/common/base/CMakeFiles/velox_exception.dir/Exceptions.cpp.o -MF velox/common/base/CMakeFiles/velox_exception.dir/Exceptions.cpp.o.d -o velox/common/base/CMakeFiles/velox_exception.dir/Exceptions.cpp.o -c /Users/kpai/src/velox/velox/common/base/Exceptions.cpp
In file included from /Users/kpai/src/velox/velox/common/base/Exceptions.cpp:17:
In file included from /Users/kpai/src/velox/./velox/common/base/Exceptions.h:19:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/memory:883:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__memory/align.h:13:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/cstddef:46:5: error: <cstddef> tried including <stddef.h> but didn't find libc++'s <stddef.h> header.           This usually means that your header search paths are not configured properly.           The header search paths should contain the C++ Standard Library headers before           any C Standard Library, and you are probably using compiler flags that make that           not be the case.
#   error <cstddef> tried including <stddef.h> but didn't find libc++'s <stddef.h> header. \
    ^
/Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/cstddef:59:9: error: no member named 'nullptr_t' in the global namespace
using ::nullptr_t;
      ~~^
In file included from /Users/kpai/src/velox/velox/common/base/Exceptions.cpp:17:
In file included from /Users/kpai/src/velox/./velox/common/base/Exceptions.h:19:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/memory:884:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__memory/allocate_at_least.h:13:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__memory/allocator_traits.h:14:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/__memory/construct_at.h:23:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/usr/include/c++/v1/new:96:

System information
MacOS M1.
CMake log
N/A

Deleting because duplicate of #8423


text:
Add IO Counters

Adds a counter for total bytes of IO avoided by hitting RAM cache. Adds the per thread IO wait counter to runTimeStats. Adds a process-wide counter for tracking table scan IO wait.

Looks we have two metrics for counting IO wait time, ioWaitNanos_  for CacheBufferedInput, totalScanTime for BufferedInput. Can we use define a actual variable to count all the IO wait time?
cc @Yuhta

source: gharchive/pull-request

authors: Yohahaha oerling

repo: facebookincubator/velox

license_type: permissive

license_source: github-api


text:
Register memory reclaimers for parquet writer mem pools

The parquet writer memory pools did not have associated memory reclaimers which prevented the worker from being suspended. To resolve the issue the default memory reclaimer is registered with all memory pools used by the parquet writer.

Resolves issue https://github.com/facebookincubator/velox/issues/7475

The initial pipeline was green. Second run encountered:
296: [ RUN      ] SharedArbitrationTest.reclaimFromJoinBuilderWithMultiDrivers
296: ../../velox/exec/tests/SharedArbitratorTest.cpp:1867: Failure
296: Expected: (planStats.spilledBytes) > (0), actual: 0 vs 0
296: [  FAILED  ] SharedArbitrationTest.reclaimFromJoinBuilderWithMultiDrivers (465 ms)

I've seen it pass and fail locally. Re-ran on main and it can fail there too.
I'm opening a new issue about this flay test.
What is the linter warning? Something I can address?

CC: @assignUser @kgpai @laithsakka @pedroerp Looks like conbench reported a false negative.

@mbasmanova I don't think it's a false negative, looking at the results after this one it seems to be a pretty clear outlier, so actually a false positive on the post-merge check:

@assignUser Jacob, thank you for looking into this... but the changes in this PR are completely unrelated. How can this be a valid result? Am I missing something?

but the changes in this PR are completely unrelated. How can this be a valid result?

I am not sure I understand? This is likely just bad luck in runner allocation or some other random factor (as that benchmark seems pretty stable otherwise).

@assignUser Got it. Thank you for explaining, Jacob.

authors: assignUser czentgr mbasmanova


text:
Fix ExchangeBenchmark and PrefixSortAlgorithmBenchmark

terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: The memory manager is not set
The reason for the error is that the benchmark object is initialized before the main function is called. We need to ensure that the memory manager instance is initialized before the benchmark object.

@skadilover Would you rebase?

@skadilover Would you rebase?

done

authors: mbasmanova skadilover


text:
AttributeError: 'NoneType' object has no attribute 'eglGetCurrentContext'

I got this error when run the code:
from animated_drawings import render
render.start('./examples/config/mvc/interactive_window_example.yaml')

Ubuntu 18.04

I fixed this issue by run:
sudo apt-get install libosmesa6-dev freeglut3-dev
But now i got another error:
OpenGL.error.Error: Attempt to retrieve context when no valid context and this window

@justinduynguyen I got the exact same issue running this in Ubuntu on WSL windows 11.

Resolved this by first getting the OpenGL running locally first.

see: https://stackabuse.com/brief-introduction-to-opengl-in-python-with-pyopengl/

import OpenGL import OpenGL.GL import OpenGL.GLUT import OpenGL.GLU print("Imports successful!") from animated_drawings import render render.start('interactive_window_example.yaml')
Followed your install + added some extras.

sudo apt-get install libosmesa6-dev freeglut3-dev
sudo apt-get install libglfw3-dev libgles2-mesa-dev
sudo apt-get install libosmesa6
export PYOPENGL_PLATFORM=osmesa - think this is what fixed the GLUT OpenGL issue

@justinduynguyen , @odalton thank you for sharing what you've found here, much appreciated.
@justinduynguyen, please close this issue if odalton's steps resolved things for you.

@justinduynguyen I got the exact same issue running this in Ubuntu on WSL windows 11. FYI i downgraded the python version also to 3.8.10

Resolved this by first getting the OpenGL running locally first.

see: https://stackabuse.com/brief-introduction-to-opengl-in-python-with-pyopengl/

import OpenGL
import OpenGL.GL
import OpenGL.GLUT
import OpenGL.GLU
print("Imports successful!")
from animated_drawings import render
render.start('interactive_window_example.yaml')

Followed your install + added some extras.

sudo apt-get install libosmesa6-dev freeglut3-dev
sudo apt-get install libglfw3-dev libgles2-mesa-dev
sudo apt-get install libosmesa6
export PYOPENGL_PLATFORM=osmesa - think this is what fixed the GLUT OpenGL issue
4.1.  see: https://pyrender.readthedocs.io/en/latest/examples/offscreen.html

from setuptools import find_packages, setup

setup(
    name='animated_drawings',
    description="Companion code for `A Method For Automatically Animating Children's Drawings of the Human Form.`",
    author='FAIR',
    author_email='jesse.smith@meta.com',
    python_requires='>=3.8.10',
    install_requires=[
        'numpy== 1.23.3',
        'scipy==1.9.3',
        'scikit-image==0.19.3',
        'scikit-learn==1.1.2',
        'shapely==1.8.5.post1',
        'opencv-python==4.6.0.66',
        'Pillow==9.3.0',
        'glfw==2.5.5',
        'PyOpenGL==3.1.6',
        'PyYAML==6.0',
        'requests==2.28.1',
        'torchserve==0.7.0',
        'tqdm==4.64.1',
        'Flask==2.2.3'
    ],
    packages=find_packages(),
)

Thanks for your solution.

source: gharchive/issue

authors: hjessmith justinduynguyen liudan193 odalton

repo: facebookresearch/AnimatedDrawings

license: MIT


text:
Marginal 2D labels

Feature Description
Add point statistic labels to the marginal 2D tool, just like the marginal 1D tool has. See image below as reference.

@horizon-blue I think I will work on this after #1819 has been merged. That way we can include both the labels as well as allowing the user to choose which point statistic they want to display; the mean or median of the distributions.

repo: facebookresearch/beanmachine


text:
Create SelfNode instead of PlayerNode for droidlet agents

Type of Issue
Right now all droidlet agents create a PlayerNode to register themselves in memory, I propose we instead use SelfNode that is meant exactly for this purpose. Link here
Select the type of issue:

[ ] Bug report (to report a bug)
[x] Feature request (to request an additional feature)
[ ] Tracker (I am just using this as a tracker)
[ ] Refactor request
[ ] Documentation Ask

closed in https://github.com/facebookresearch/fairo/pull/1200

authors: aszlam kavyasrinet

repo: facebookresearch/fairo


text:
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

🐛 Bug
After following the Roberta pretraining documentation with the exact same steps and data.
When running fairseq-train I get an error
To Reproduce
TOTAL_UPDATES=125000    # Total number of training steps
WARMUP_UPDATES=10000    # Warmup the learning rate over this many updates
PEAK_LR=0.0005          # Peak learning rate, adjust as needed
TOKENS_PER_SAMPLE=512   # Max sequence length
MAX_POSITIONS=512       # Num. positional embeddings (usually same as above)
MAX_SENTENCES=16        # Number of sequences per batch (batch size)
UPDATE_FREQ=16          # Increase the batch size 16x

DATA_DIR=data-bin/wikitext-103

fairseq-train --fp16 $DATA_DIR \
    --task masked_lm --criterion masked_lm \
    --arch roberta_base --sample-break-mode complete --tokens-per-sample $TOKENS_PER_SAMPLE \
    --optimizer adam --adam-betas '(0.9,0.98)' --adam-eps 1e-6 --clip-norm 0.0 \
    --lr-scheduler polynomial_decay --lr $PEAK_LR --warmup-updates $WARMUP_UPDATES --total-num-update $TOTAL_UPDATES \
    --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
    --max-sentences $MAX_SENTENCES --update-freq $UPDATE_FREQ \
    --max-update $TOTAL_UPDATES --log-format simple --log-interval 1

Then I get the following stack trace:

2020-08-03 09:13:25 | INFO | fairseq_cli.train | begin training epoch 1
Traceback (most recent call last):
  File "/usr/local/bin/fairseq-train", line 33, in <module>
    sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-train')())
  File "/fairseq/fairseq_cli/train.py", line 350, in cli_main
    distributed_utils.call_main(args, main)
  File "/fairseq/fairseq/distributed_utils.py", line 189, in call_main
    main(args, **kwargs)
  File "/fairseq/fairseq_cli/train.py", line 121, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/fairseq/fairseq_cli/train.py", line 217, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/fairseq/fairseq/trainer.py", line 457, in train_step
    raise e
  File "/fairseq/fairseq/trainer.py", line 431, in train_step
    ignore_grad=is_dummy_batch,
  File "/fairseq/fairseq/tasks/fairseq_task.py", line 347, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/criterions/masked_lm.py", line 52, in forward
    logits = model(**sample['net_input'], masked_tokens=masked_tokens)[0]
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/models/roberta/model.py", line 119, in forward
    x, extra = self.encoder(src_tokens, features_only, return_all_hiddens, **kwargs)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/models/roberta/model.py", line 337, in forward
    x, extra = self.extract_features(src_tokens, return_all_hiddens=return_all_hiddens)
  File "/fairseq/fairseq/models/roberta/model.py", line 345, in extract_features
    last_state_only=not return_all_hiddens,
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/modules/transformer_sentence_encoder.py", line 250, in forward
    x = self.emb_layer_norm(x)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/normalization.py", line 170, in forward
    input, self.normalized_shape, self.weight, self.bias, self.eps)
  File "/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2049, in layer_norm
    torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

Environment

fairseq Version : master
PyTorch Version : 1.6.0
OS : Linux (Ubuntu)
How you installed fairseq (pip, source): source
Build command you used (if compiling from source):

git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable .

Python version: 3.6.9
CUDA/cuDNN version: 11.0

Solved after reinstalling CUDA and NCCL

got the same error, restarting the notebook solved it for me
notebook - 39_tutorial.transformers.ipynb

I get the same issue even when running on cpu. Do you know why?

@yuvalkirstain Did you find the solution?

@yuvalkirstain Did you find the solution?

yes @syedasara-angelium , you shouldn't run half on cpu. Probably best to condition the use of half by running on cuda.

yes @syedasara-angelium , you shouldn't run half on cpu. Probably best to condition the use of half by running on cuda.

I have a similar issue, as I run on my CPU I get the error:
2021-01-08 21:43:05 | INFO | fairseq.trainer | begin training epoch 1
Traceback (most recent call last):
  File "/home/wouter/Documents/same-side-classification/venv/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 352, in cli_main
    distributed_utils.call_main(args, main)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/distributed_utils.py", line 301, in call_main
    main(args, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 125, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 208, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 512, in train_step
    raise e
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 480, in train_step
    loss, sample_size_i, logging_output = self.task.train_step(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 416, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/criterions/sentence_prediction.py", line 42, in forward
    logits, _ = model(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/models/roberta/model.py", line 190, in forward
    x, extra = self.encoder(src_tokens, features_only, return_all_hiddens, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/models/roberta/model.py", line 458, in forward
    x, extra = self.extract_features(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/models/roberta/model.py", line 466, in extract_features
    inner_states, _ = self.sentence_encoder(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/modules/transformer_sentence_encoder.py", line 258, in forward
    x = self.emb_layer_norm(x)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 169, in forward
    return F.layer_norm(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2094, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps,
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

If I run on my GPU, I get:
2021-01-08 21:41:13 | INFO | fairseq.trainer | begin training epoch 1
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=115 error=710 : device-side assert triggered
Traceback (most recent call last):
  File "/home/wouter/Documents/same-side-classification/venv/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 352, in cli_main
    distributed_utils.call_main(args, main)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/distributed_utils.py", line 301, in call_main
    main(args, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 125, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 208, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 512, in train_step
    raise e
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 480, in train_step
    loss, sample_size_i, logging_output = self.task.train_step(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 416, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/criterions/sentence_prediction.py", line 52, in forward
    loss = F.nll_loss(lprobs, targets, reduction="sum")
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:115

Any ideas?

I have a similar issue, as I run on my CPU I get the error:
2021-01-08 21:43:05 | INFO | fairseq.trainer | begin training epoch 1
Traceback (most recent call last):
  File "/home/wouter/Documents/same-side-classification/venv/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 352, in cli_main
    distributed_utils.call_main(args, main)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/distributed_utils.py", line 301, in call_main
    main(args, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 125, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 208, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 512, in train_step
    raise e
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 480, in train_step
    loss, sample_size_i, logging_output = self.task.train_step(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 416, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/criterions/sentence_prediction.py", line 42, in forward
    logits, _ = model(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/models/roberta/model.py", line 190, in forward
    x, extra = self.encoder(src_tokens, features_only, return_all_hiddens, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/models/roberta/model.py", line 458, in forward
    x, extra = self.extract_features(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/models/roberta/model.py", line 466, in extract_features
    inner_states, _ = self.sentence_encoder(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/modules/transformer_sentence_encoder.py", line 258, in forward
    x = self.emb_layer_norm(x)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/normalization.py", line 169, in forward
    return F.layer_norm(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2094, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps,
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

If I run on my GPU, I get:
2021-01-08 21:41:13 | INFO | fairseq.trainer | begin training epoch 1
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [0,0,0] Assertion `t >= 0 && t < n_classes` failed.
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [1,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=115 error=710 : device-side assert triggered
Traceback (most recent call last):
  File "/home/wouter/Documents/same-side-classification/venv/bin/fairseq-train", line 8, in <module>
    sys.exit(cli_main())
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 352, in cli_main
    distributed_utils.call_main(args, main)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/distributed_utils.py", line 301, in call_main
    main(args, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 125, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq_cli/train.py", line 208, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 512, in train_step
    raise e
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/trainer.py", line 480, in train_step
    loss, sample_size_i, logging_output = self.task.train_step(
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 416, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/fairseq/criterions/sentence_prediction.py", line 52, in forward
    loss = F.nll_loss(lprobs, targets, reduction="sum")
  File "/home/wouter/Documents/same-side-classification/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (710) : device-side assert triggered at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:115

Any ideas?

Okay, I found my issue. I preprocessed my labels using the argument --srcdict dict.txt.. That made it fail.

Okay, I found my issue. I preprocessed my labels using the argument --srcdict dict.txt.. That made it fail.

Hey, guys, I encounter this problem as well. I follow wzorgdrager's suggestion to comment --srcditc dict.txt when doing binarize data. Still not working. Not sure why? Any more comments? Where can I select not to use Half?

Namely, how can condition the use of half?

I think that I found solution. In fact, what we need to do is to remove the parameter: --fp16. Then, the whole command line will look like:
python3.7 train.py Quora/data-bin --restore-file bart.large/model.pt --max-tokens 1024 --task translation --source-lang source --target-lang target --truncate-source --share-decoder-input-output-embed --reset-optimizer --reset-dataloader --reset-meters --required-batch-size-multiple 1 --arch bart_large --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 --optimizer adam --adam-betas "(0.9, 0.999)" --adam-eps 1e-08 --clip-norm 0.1 --lr-scheduler polynomial_decay --lr 3e-05 --total-num-update 200000 --warmup-updates 500 --update-freq 4 --skip-invalid-size-inputs-valid-test --find-unused-parameters --save-interval-updates 20000 --cpu --device-id 0 --distributed-world-size 1 --distributed-no-spawn --pipeline-chunks 1 --dataset-impl mmap

Try setting env variable
NVIDIA_VISIBLE_DEVICES='all' or
NVIDIA_VISIBLE_DEVICES=all
this worked for me, I am running code through docker

this still exists for me on google collab and i can't reinstall there

Im not using Nvidia so i cannot reinstall CUDA, any solution?

TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.1.1' python launch.py --precision full --no-half
try？

try using these arguments:
set COMMANDLINE_ARGS=--precision full --no-half

try using these arguments in the webui-user.bat:
set COMMANDLINE_ARGS=--precision full --no-half

try this command if u use jupyter or Kaggle notebook or colab
!pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.1.1

!python3 launch.py --precision full --no-half

If you want to run it on gpu you have to set the default floating point to float32 by adding
torch.set_default_dtype(torch.float64) in your code.
I was running the stable diffusion model, I was facing the same error, The issue is fixed when I set the floating point to float32.

🐛 Bug
After following the Roberta pretraining documentation with the exact same steps and data. When running fairseq-train I get an error
To Reproduce
TOTAL_UPDATES=125000    # Total number of training steps
WARMUP_UPDATES=10000    # Warmup the learning rate over this many updates
PEAK_LR=0.0005          # Peak learning rate, adjust as needed
TOKENS_PER_SAMPLE=512   # Max sequence length
MAX_POSITIONS=512       # Num. positional embeddings (usually same as above)
MAX_SENTENCES=16        # Number of sequences per batch (batch size)
UPDATE_FREQ=16          # Increase the batch size 16x

DATA_DIR=data-bin/wikitext-103

fairseq-train --fp16 $DATA_DIR \
    --task masked_lm --criterion masked_lm \
    --arch roberta_base --sample-break-mode complete --tokens-per-sample $TOKENS_PER_SAMPLE \
    --optimizer adam --adam-betas '(0.9,0.98)' --adam-eps 1e-6 --clip-norm 0.0 \
    --lr-scheduler polynomial_decay --lr $PEAK_LR --warmup-updates $WARMUP_UPDATES --total-num-update $TOTAL_UPDATES \
    --dropout 0.1 --attention-dropout 0.1 --weight-decay 0.01 \
    --max-sentences $MAX_SENTENCES --update-freq $UPDATE_FREQ \
    --max-update $TOTAL_UPDATES --log-format simple --log-interval 1

Then I get the following stack trace:

2020-08-03 09:13:25 | INFO | fairseq_cli.train | begin training epoch 1
Traceback (most recent call last):
  File "/usr/local/bin/fairseq-train", line 33, in <module>
    sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-train')())
  File "/fairseq/fairseq_cli/train.py", line 350, in cli_main
    distributed_utils.call_main(args, main)
  File "/fairseq/fairseq/distributed_utils.py", line 189, in call_main
    main(args, **kwargs)
  File "/fairseq/fairseq_cli/train.py", line 121, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/fairseq/fairseq_cli/train.py", line 217, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/fairseq/fairseq/trainer.py", line 457, in train_step
    raise e
  File "/fairseq/fairseq/trainer.py", line 431, in train_step
    ignore_grad=is_dummy_batch,
  File "/fairseq/fairseq/tasks/fairseq_task.py", line 347, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/criterions/masked_lm.py", line 52, in forward
    logits = model(**sample['net_input'], masked_tokens=masked_tokens)[0]
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/models/roberta/model.py", line 119, in forward
    x, extra = self.encoder(src_tokens, features_only, return_all_hiddens, **kwargs)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/models/roberta/model.py", line 337, in forward
    x, extra = self.extract_features(src_tokens, return_all_hiddens=return_all_hiddens)
  File "/fairseq/fairseq/models/roberta/model.py", line 345, in extract_features
    last_state_only=not return_all_hiddens,
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/fairseq/fairseq/modules/transformer_sentence_encoder.py", line 250, in forward
    x = self.emb_layer_norm(x)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/.local/lib/python3.6/site-packages/torch/nn/modules/normalization.py", line 170, in forward
    input, self.normalized_shape, self.weight, self.bias, self.eps)
  File "/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 2049, in layer_norm
    torch.backends.cudnn.enabled)
RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

Environment

fairseq Version : master
PyTorch Version : 1.6.0
OS : Linux (Ubuntu)
How you installed fairseq (pip, source): source
Build command you used (if compiling from source):

git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable .

Python version: 3.6.9
CUDA/cuDNN version: 11.0

run, like this : sh webui.sh --skip-torch-cuda-test --no-half --use-cpu all

running like this: sh webui.sh --skip-torch-cuda-test --no-half --use-cpu all

像这样运行：sh webui.sh --skip-torch-cuda-test --no-half --use-cpu all
这样可以 多谢大佬

I'm using a Macbook Pro running Windows 11 and these COMMANDLINE_ARGS worked for me.
--autolaunch --skip-torch-cuda-test --disable-nan-check --no-half --use-cpu all

so pls how did you fix it??

window

mac
打开终端，cd到指定目录，sh webui.sh --skip-torch-cuda-test --no-half --use-cpu all

这里是参数列表
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Command-Line-Arguments-and-Settings

it works in my work.
 --skip-torch-cuda-test --no-half --share --use-cpu all

good luck

sh webui.sh --skip-torch-cuda-test --no-half --use-cpu all
It's ok.

TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.1.1' python launch.py --precision full --no-half try？

i use on ubantu 22.04   run on cpu only . I use this solution and  its work  thank U so much ^_^

While using gradio app hosting to space, how would you control this?
--skip-torch-cuda-test --no-half --use-cpu all

RuntimeError: “LayerNormKernelImpl” not implemented for ‘Half’
I have successfully solved this problem by way of
Start the webUI with the following Mac OS Terminal command.
cd stable-diffusion-webui
./webui.sh --precision full --no-half
Special Thank >>
https://stable-diffusion-art.com/install-mac/comment-page-1/

RuntimeError: “LayerNormKernelImpl” not implemented for ‘Half’
I have successfully solved this problem by way of
Start the webUI with the following Mac OS Terminal command.
cd stable-diffusion-webui ./webui.sh --precision full --no-half
Special Thank >> https://stable-diffusion-art.com/install-mac/comment-page-1/

this worked for me.

for my case, a reinstall of the nvdia driver solved the problem.

pipe = pipe.to("cuda")

Change
"
torch_dtype = torch.float16
"
to
"
torch_dtype = torch.float32
"
Half becomes whole.

authors: 0tist AbdelrahmanShahrour Archemike Charmve MrWuzy1994 Nxthin Pollakritk Skylixia alexbourg arbazgormani dabao-zhao gauravshegokar iaverypadberg innat-asj leonleeldc maximus-sallam moqinchanyuan pchavers26 sQcrates shillow ssszZ-TH syedasara-angelium wzorgdrager yuruizhi yuvalkirstain zzm88

repo: facebookresearch/fairseq


text:
run error

hi, guys, I try to run the demo, and met some error here.
Traceback (most recent call last):
  File "/data/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/data/generative-recommenders/train.py", line 466, in mp_train_fn
    train_fn(rank, world_size, master_port)
  File "/data/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/data/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/data/anaconda3/envs/pytorch3d/lib/python3.8/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/data/generative-recommenders/train.py", line 300, in train_fn
    eval_dict = eval_metrics_v2_from_tensors(
TypeError: __call__() got an unexpected keyword argument 'target_ids'

my python version is 3.8.17 , pytorch version is torch==2.0.1+cu117,

@a6802739 met the same problem in torch 2.0.0, comment the @torch.inference_mode in data/eval.py worked.

@makefile  thanks, it works in indeed.

Please check https://github.com/facebookresearch/generative-recommenders/issues/1. You may need to upgrade your pytorch version.

Please check https://github.com/facebookresearch/generative-recommenders/issues/1; you may need to upgrade your pytorch version.

authors: Lunarmony a6802739 jiaqizhai makefile


text:
[baselines] Refactor Observations Encoders

🚀 Feature

Motivation

Currently, all the observation heads for various observation types are hard coded in a list of if statements in the ResNet encoder and elsewhere.
We should factor these out and make them useful outside of the policy.
We should also standardize where/when RGB images are normalized and the channel orders are changed in our baselines pipeline.

Pitch

Doing so would make it easier to integrate new observation types and try out new architecture without a massive amount of code reuse from ResNet.

Alternatives

Move these multi-head encoders into an abstract class that can be extended by multiple policies through multi-inheritance? Seems a bit messy.
Make all the observation transforms and such as forward hooks onto the nn.Module

Additional context

@rpartsey, you can share your experience or code snippets here how you done it. Thank you!

@rpartsey, you can share your experience or code snippets here how you done it. Thank you!

@mathfac, thanks for letting me know about this issue
We were also trying to solve a similar problem: simple yet flexible code (to run experiments and iterate fast).
It was important for us to have a model with configurable encoders and input dimensions (number of channels) with FC layers on top.
We also considered implementing each encoder type as part of our project, but after investigating available libraries decided to add segmentation_models_pytorch as a dependency. It already has a wide range of encoders implemented and it is actively supported by the community (+ some useful functionality like patching first Conv layer/loading pre-trained weights without FC layers/... ).
Our model code looks smth like this (we use from_config classmethod to instantiate objects):
# model.py

import torch
import torch.nn as nn
from segmentation_models_pytorch.encoders import get_encoder

class Net(nn.Module):
    def __init__(self, encoder, fc):
        super().__init__()
        self.encoder = encoder
        self.fc = fc

    def forward(self, x):
        x = self.encoder(x)[-1]  # get last stage output
        x = self.fc(x)

        return x

    @classmethod
    def from_config(cls, model_config):
        model_params = model_config.params
        encoder_params = model_params.encoder.params
        fc_params = model_params.fc.params

        encoder = get_encoder(
            name=model_params.encoder.type,
            in_channels=encoder_params.in_channels,
            depth=encoder_params.depth,
            weights=encoder_params.weights
        )

        fc = cls.create_fc_layers(
            input_size=cls.compute_output_size(encoder, encoder_params),
            hidden_size=fc_params.hidden_size,
            output_size=fc_params.output_size,
            p_dropout=fc_params.p_dropout
        )

        return cls(encoder, fc)

    @staticmethod
    def create_fc_layers(input_size: int, hidden_size: list, output_size: int, p_dropout: float = 0.0):
        fc_layers = ...  # FC layers code

        return fc_layers

    @staticmethod
    def compute_output_size(encoder, config):
        input_size = (1, config.in_channels, config.in_height, config.in_width)

        encoder_input = torch.randn(*input_size)
        with torch.no_grad():
            output = encoder(encoder_input)

        return output[-1].view(-1).size(0)

Swapping encoders is as easy as changing the encoder name in the config.yaml file:
# config.yaml

model:
  type: Net
  save: True
  params:
    encoder:
      type: resnet18  # change to any other encoder name from segmentation_models_pytorch
      params:
        depth: 5
        weights: imagenet
        in_channels: 8
        in_height: 360
        in_width: 640
    fc:
      params:
        hidden_size: [512, 512]
        output_size: 4
        p_dropout: 0

@mathfac, thanks for letting me know about this issue
We were also trying to solve a similar problem: simple yet flexible code (to run experiments and iterate fast).
It was important for us to have a model with configurable encoders and input dimensions (number of channels) with FC layers on top.
We also considered implementing each encoder type as part of our project, but after investigating available libraries decided to add segmentation_models_pytorch as a dependency. It already has a wide range of encoders implemented and it is actively supported by the community (+ some useful functionality like patching first Conv layer/loading pre-trained weights without FC layers/... ).
Our model code looks smth like this (we use from_config classmethod to instantiate objects):
# model.py

import torch
import torch.nn as nn
from segmentation_models_pytorch.encoders import get_encoder

class Net(nn.Module):
    def __init__(self, encoder, fc):
        super().__init__()
        self.encoder = encoder
        self.fc = fc

    def forward(self, x):
        x = self.encoder(x)[-1]  # get last stage output
        x = self.fc(x)

        return x

    @classmethod
    def from_config(cls, model_config):
        model_params = model_config.params
        encoder_params = model_params.encoder.params
        fc_params = model_params.fc.params

        encoder = get_encoder(
            name=model_params.encoder.type,
            in_channels=encoder_params.in_channels,
            depth=encoder_params.depth,
            weights=encoder_params.weights
        )

        fc = cls.create_fc_layers(
            input_size=cls.compute_output_size(encoder, encoder_params),
            hidden_size=fc_params.hidden_size,
            output_size=fc_params.output_size,
            p_dropout=fc_params.p_dropout
        )

        return cls(encoder, fc)

    @staticmethod
    def create_fc_layers(input_size: int, hidden_size: list, output_size: int, p_dropout: float = 0.0):
        fc_layers = ...  # FC layers code

        return fc_layers

    @staticmethod
    def compute_output_size(encoder, config):
        input_size = (1, config.in_channels, config.in_height, config.in_width)

        encoder_input = torch.randn(*input_size)
        with torch.no_grad():
            output = encoder(encoder_input)

        return output[-1].view(-1).size(0)

Swapping encoders is as easy as changing the encoder name in the config.yaml file:
# config.yaml

model:
  type: Net
  save: True
  params:
    encoder:
      type: resnet18  # change to any other encoder name from segmentation_models_pytorch
      params:
        depth: 5
        weights: imagenet
        in_channels: 8
        in_height: 360
        in_width: 640
    fc:
      params:
        hidden_size: [512, 512]
        output_size: 4
        p_dropout: 0

One word of warning:  If you are training things, BatchNorm can be problematic due to the highly correlated data seen in RL and IL

One word of warning:  If you are training things, BatchNorm can be problematic due to the highly correlated data seen in RL and IL

authors: Skylion007 erikwijmans mathfac rpartsey

repo: facebookresearch/habitat-lab


text:
[gala_kinematic] Masking the loss for the first observation

Motivation and Context
Masking the losses using the mask in the rollout.
This mask corresponds to the first observation after a reset which needs to be discarded
How Has This Been Tested

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have read the CONTRIBUTING document.
[ ] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Hmm, all the tests including the easily fixable ones like Black are failing on these tests though. 👎

Hmm, all the tests including the easily fixable ones like Black are failing on these tests though. 👎

They are failing on the target branch. This change is not responsible


text:
Re-training possible?

❓ Questions and Help
Hi!
My question is whether retraining on new data is possible? Lets assume that I trained a model with satisfying accuracies but I want to further improve it by adding new data containing the features I want to add. Do I simply set the old model as pre-trained model (MODEL.WEIGHT) and train only on those new data?
Thank you in advance!

Hi,
That's one of the solutions. The potential problem is your model may forget what it learns in the previous dataset. Another straightforward method is to concatenate these two data sets and re-train on it.  You can do it by modifying DATASETS.TRAIN. This could make sure your modal does well on both datasets but it may take longer training time.
This is also an open research problem, you can also check the following research paper for a better solution.

You can also check the following research paper.

Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights, ECCV 2018
Learning Without Forgetting, ECCV 2016

Closing following @chengyangfu comments. Thanks Cheng-Yang!

authors: chengyangfu fmassa maxsenh

repo: facebookresearch/maskrcnn-benchmark


text:
Add MyPy to CI

Patch Description
PR for issue 14.
Here we add MyPy to CI.
Testing steps
Run mypy at local laptop & run the check via an actual pull request.

@stephenroller
I run the CI in the PR in my fork repo.
But there are a lot of mypy type checking errors.
How do we do about them?

Thanks for the contribution.

repo: facebookresearch/metaseq


text:
Possibility to remove numerical suffixes

Types of changes

[ ] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue
This is a frequent case (typically for Mathurin and me): in many cases we want to plot results after moving
the numerical parameters out of the optimizer name, so that results show robustness.
How Has This Been Tested (if it applies)

Checklist

[x] The documentation is up-to-date with the changes I made.
[x] I have read the CONTRIBUTING document and completed the CLA (see CLA).
[x] All tests passed, and additional code has been covered with new tests.

Associated to dfoptim's PR 566.

repo: facebookresearch/nevergrad


text:
Add foreach to optimizers

I rewrote adamw_schedulefree to support using torch._foreach operations as an option.  According to my profiler, this made the optimizer step go from taking ~85ms to ~65ms (the model used was the SD1.5 Unet).  In theory, this should also increase peak memory usage during the optimizer step (according to Pytorch's documentation on the built-in optimizers) -- I have left it as enabled by default regardless since this is the default behavior for built-in optimizers.
I think it would obviously be best to add this to all included optimizers, but before I do that I would like to hear some feedback on code style to make sure everything I add is up to standards.

Nice! I will take a look on Monday.

Have you tried running https://github.com/facebookresearch/schedule_free/blob/main/schedulefree/test_schedulefree.py
to make sure it gives identical outputs to the closure version? That will help verify if it's correct.

I implemented it in all of the main optimizers, both closure and non-closure versions.  I skipped some of the renaming in the foreach versions for the sake of simplicity on the in-place operations.  The code passes the test suite just fine in its current state.

Ok, we will look this over during the next few days, sorry for the delay things are hectic at the moment.

This looks to be working correctly, thanks!

authors: adefazio drhead


text:
torch compile tensorrt error

🐛 Describe the bug
When compiling segment_anything with torch_tensorrt,
I got the error We don't have an op for aten::floor_divide but it isn't a special case.
My reproducible code is:
from segment_anything import sam_model_registry, SamPredictor
import torch_tensorrt

sam = sam_model_registry["vit_b"](checkpoint="sam_vit_b_01ec64.pth")
input_signature = ([torch_tensorrt.Input(shape=frame.shape, dtype=torch.half)])
enabled_precisions = {torch.half,}
sam.image_encoder = torch_tensorrt.compile(sam.image_encoder, input_signature=input_signature, enabled_precisions=enabled_precisions)

The error message is:
Traceback (most recent call last):
  File "realtime.py", line 144, in <module>
    main()
  File "realtime.py", line 116, in main
    sam = newSam(frame)
  File "realtime.py", line 86, in newSam
    sam.image_encoder = torch_tensorrt.compile(sam.image_encoder, input_signature=input_signature, enabled_precisions=enabled_precisions)
  File "/home/topunion/.local/lib/python3.8/site-packages/torch_tensorrt/_compile.py", line 133, in compile
    return torch_tensorrt.ts.compile(
  File "/home/topunion/.local/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 139, in compile
    compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: 0 INTERNAL ASSERT FAILED at "../torch/csrc/jit/ir/alias_analysis.cpp":615, please report a bug to PyTorch. We don't have an op for aten::floor_divide but it isn't a special case.  Argument types: int, int,

Candidates:
	aten::floor_divide(Tensor self, Tensor other) -> Tensor
	aten::floor_divide.Scalar(Tensor self, Scalar other) -> Tensor
	aten::floor_divide.out(Tensor self, Tensor other, *, Tensor(a!) out) -> Tensor(a!)

Versions
Collecting environment information...
PyTorch version: 2.0.1+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.31
Python version: 3.8.10 (default, May 26 2023, 14:05:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060
Nvidia driver version: 530.41.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          12
On-line CPU(s) list:             0-11
Thread(s) per core:              2
Core(s) per socket:              6
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           113
Model name:                      AMD Ryzen 5 3600 6-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         2200.000
CPU max MHz:                     3600.0000
CPU min MHz:                     2200.0000
BogoMIPS:                        7200.08
Virtualization:                  AMD-V
L1d cache:                       192 KiB
L1i cache:                       192 KiB
L2 cache:                        3 MiB
L3 cache:                        32 MiB
NUMA node0 CPU(s):               0-11
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.1
[pip3] torch==2.0.1+cu118
[pip3] torch-tensorrt==1.4.0
[pip3] torchaudio==2.0.2+cu118
[pip3] torchvision==0.15.2+cu118
[pip3] triton==2.0.0
[conda] Could not collect

I also met the same problem, it seems that attention module in ImageEncoderViT has some trouble

It is because torch-trt dont support % op, you need to change % to int(a - b * torch.floor(torch.div(a, b))), the compute is the same, but can convert by torch-trt. Function window_partition in segment_anything/modeling/image_encoder.py contain % op.

hi @Mythos-Rudy have been able to compile the encoder part to tensorrt?

It is because torch-trt dont support % op, you need to change % to int(a - b * torch.floor(torch.div(a, b))), the compute is the same, but can convert by torch-trt. Function window_partition in segment_anything/modeling/image_encoder.py contain % op.

This is helpful for me, thank you.
I use this code below to replace % op:
def TakeRemainder(x: int, y: int) -> int:
    return x - y * int(x / y)

authors: Mythos-Rudy Youngluc adithya-Avataar demuxin fumin

repo: facebookresearch/segment-anything


text:
Materials issue with HM3D

Hello! I am having an issue using SS2.0 with materials enabled in HM3D environments. For some reason, when I run the audio sensor observation, I get these errors:
[13:07:34:776730]:[Assets] ResourceManager.cpp(2941)::joinSemanticHierarchy : Could not get the GenericSemanticMeshData
File: arvr/libraries/audio/AudioSDK/Research/Source/Wrapper/PropagationWrapper.cpp
Function: ovrResult PropagationWrapper::AddMeshVertices(const float *, ovra::Size), Line 926
Mesh vertices were NULL
[13:07:34:777420]:[Sensor] AudioSensor.cpp(394)::loadSemanticMesh : Couldn't add vertices to audio mesh, error:  2001

I am assuming this is something to do with the configurations of the hm3d dataset. However, I saw a previous issue where it was mentioned that materials seems to work fine for hm3d, along with mp3d and gibson.
The hm3d scene I am trying to run is :
hm3d/minival/00802-wcojb4TFT35/wcojb4TFT35.basis.glb
With the scene dataset config:
hm3d/minival/hm3d_annotated_minival_basis.scene_dataset_config.json
Any help would be greatly appreciated.

I got the same issue when I tried to use the HM3D dataset.
Python file: ~/sound-spaces/examples/minimal_example.py
scene_id: data/scene_datasets/hm3d/minival/00800-TEEsavR23oF/TEEsavR23oF.basis.glb
scene_dataset_config_file: data/scene_datasets/hm3d/hm3d_annotated_basis.scene_dataset_config.json
File: arvr/libraries/audio/AudioSDK/Research/Source/Wrapper/PropagationWrapper.cpp
Function: ovrResult PropagationWrapper::AddMeshVertices(const float *, ovra::Size), Line 926
Mesh vertices were NULL
[11:24:37:833446]:[Sensor] AudioSensor.cpp(394)::loadSemanticMesh : Couldn't add vertices to audio mesh, error:  2001
[11:24:38:581026]:[Physics] PhysicsManager.cpp(50)::~PhysicsManager : Deconstructing PhysicsManager
[11:24:38:581103]:[Scene] SceneManager.h(25)::~SceneManager : Deconstructing SceneManager
[11:24:38:581113]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[11:24:38:581203]:[Sensor] AudioSensor.cpp(64)::~AudioSensor : [Audio]  Destroying the audio sensor
[11:24:38:581503]:[Sensor] Sensor.cpp(69)::~Sensor : Deconstructing Sensor
[11:24:38:581525]:[Scene] SceneGraph.h(25)::~SceneGraph : Deconstructing SceneGraph
[11:24:38:581613]:[Scene] SemanticScene.h(47)::~SemanticScene : Deconstructing SemanticScene
[11:24:38:583345]:[Sim] Simulator.cpp(66)::~Simulator : Deconstructing Simulator
# The length of obtained IR when sampling_rate=16000 : (1, 2067)

When I set the enableMaterials=False, it works fine but I need the absorption rate for each material.
@ChanganVR , do you have any specific settings for HM3D? HM3D dataset already used in the paper SS2.0.

@denfed @iyeon915 Sorry for the late reply. we recently found out there is some issue with loading the acoustic materials for HM3D in the simulator. This is still under investigation and due to the lack of developers, the progress has been slow. We will get back to you as soon as possible when this is fixed!
But the acoustic material loading for MP3D and Gibson should be fully functioning.

Have there been any updates on getting the material properties working for HM3D? Thank you!

@denfed @iyeon915 Sorry for the late reply. we recently found out there is some issue with loading the acoustic materials for HM3D in the simulator. This is still under investigation and due to the lack of developers, the progress has been slow. We will get back to you as soon as possible when this is fixed!
But the acoustic material loading for MP3D and Gibson should be fully functioning.

Same issue. Pls~

authors: ChanganVR ZetaRing denfed iyeon915

repo: facebookresearch/sound-spaces


text:
created circulalr_queue.md

[ ] Read the contribution guidelines
[ ] Added unit tests (or unit tests have already been added)
[ ] Updated the Contents

Thanks for your interest in the repository, @saiprasadnayak6858. Please read our contribution guidelines. It explains how you can integrate your code into the project.
I'm going to close this PR for now since it doesn't fit our standards, but please feel free to ask questions below and to open a new PR once you've read the guidelines and adapted your code to the standards. 🙂

repo: faheel/Algos

license: mit

license_source: bigquery


text:
Add functionality to see the details of a feature set.

Proposed Changes

Add functionality to visualize a feature set clicking the button of singular feature set in the table of feature set list.
Pass the data to the visualization component.
Display the current data of the feature set.

Pull request checklist
Please check if your PR fulfills the following requirements:

[ ] Tests
[x] Build locally
[ ] Code format / lint
[ ] Docs have been reviewed and added

Pull request type
Please check the type of change your PR introduces:

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?

Issue Number: N/A

Fixes #79
What is the new behavior?

Enabled the update button in the list of feature sets, this button now redirect to onSelectFeatureSet(selectedFeatureSet) method.
In the onSelectFeatureSet(selectedFeatureSet) method, the component redirect to "/fsdetails" and passing the object of the feature set using router.navigate .
The component redirect to "/fsdetails" but is using the FeatureSetCreationComponent, for this I added in the app-route component another direction, now there are 2 urls to redirect to FeatureSetCreationComponent.
On FeatureSetCreationComponent, the ngOnInit() component checks if is a new feature set or is an existent to edit, there checks with an if conditional if history.state.selectedFeatureSet exists, if exists, fill the form fields and the table with the variables.

Please, remove testing lines like ' test = 'hola mundo.'; ' .

authors: alexanderBruno miquinti


text:
Falco Installation with certificates

Describe the bug
Getting bug while installation. I was following this guide https://falco.org/blog/falco-kind-prometheus-grafana/#install-falco and it gave me following error
Error: INSTALLATION FAILED: template: falco/templates/daemonset.yaml:14:8: executing "falco/templates/daemonset.yaml" at <include "falco.podTemplate" .>: error calling include: template: falco/templates/pod-template.tpl:13:23: executing "falco.podTemplate" at <include (print $.Template.BasePath "/certs-secret.yaml") .>: error calling include: template: falco/templates/certs-secret.yaml:6:11: executing "falco/templates/certs-secret.yaml" at <include "falco.fullname" .>: error calling include: template: falco/templates/_helpers.tpl:14:14: executing "falco.fullname" at <.Values.fullnameOverride>: nil pointer evaluating interface {}.fullnameOverride

How to reproduce it
helm install falco ./falco     --set-file \
certs.server.key=/tmp/certs/server.key,certs.server.crt=/tmp/certs/server.crt,certs.ca.crt=/tmp/certs/ca.crt    \
 --set falco.grpc.enabled=true,falco.grpcOutput.enabled=true

Expected behaviour

Screenshots

Environment

Falco version:

System info:

Cloud provider or hardware configuration:
OS:
Windows WSL2 Linux distro

Kernel:
WSL 2

Installation method:

Additional context

Hi @xi2817-aajgaonkar, thank you for reporting the issue.
I took a quick look and it seems that the problem is here: https://github.com/falcosecurity/charts/blob/c3ea76f4a1556c699fa421116d225cf73f93e907/falco/templates/certs-secret.yaml#L1-L2
We are changing the scope using the with keyword causing the subsequent falco.fullname helper to fail.
Hopefully today I will open PR with the fix.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Fixed by #388
/close

authors: alacuku leogr poiana xi2817-aajgaonkar

repo: falcosecurity/charts


text:
chore(deploy): cleanup folder and add notice

Signed-off-by: Leonardo Grasso me@leonardograsso.com

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup
/kind documentation

/kind feature

/kind sandbox

/kind incubating

/kind officialsupport

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area deploy

/area examples

/area integrations

/area proposals

What this PR does / why we need it:
This PR is part of #144.
Once other tasks are completed, I'll update also the status of the repos.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: leogr poiana

repo: falcosecurity/evolution


text:
Trim repeating alerts which flooding log files and outputs triggered by same action

Hi,
Motivation
In some cases, one action repeating will generate same alerts continuously, and in contents only difference is the timestamp, then alerts will flood log files or other outputs locations, so other alerts may be hidden inside the flood.
e.g. use event-gernerater to simulate a "Write below /etc"
/bin/event-generator run --loop syscall.WriteBelowEtc
by default it will happen every 250ms, and in output, it will show same number of alerts there.
Feature
Probably we need a feature to trim such logs, e.g. it will only show the first log, then transfer other repeating logs to something like " happen xx times in last xx mins" or some json keys/values to indicate this
Thanks

A rate-limit mechanism for alert notifications is already in place. Though sometimes it's not a good thing to have, for example, see https://github.com/falcosecurity/falco/issues/1333.
But your suggestion is going further and makes sense. Moreover, aggregating notifications could be a problem in some use cases. For example, how would one reconstruct the history of a security incident if they don't have the complete list of all separate notifications with precise timestamps?
It would be a good topic for our community call btw.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

authors: TroyPang leogr poiana

repo: falcosecurity/falco


text:
cleanup(driver, userspace): remove the udig engine

What type of PR is this?
/kind cleanup
Any specific area of the project related to this PR?
/area driver-kmod
/area libscap-engine-udig
Does this PR require a change in the driver versions?
No
What this PR does / why we need it:
This PR removes the udig engine and addressed this issue  https://github.com/falcosecurity/libs/issues/1368
Which issue(s) this PR fixes:
Fixes #1368
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: f5153ee69f07eeed9c53b0d39fc751f971d50a98

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/unhold

repo: falcosecurity/libs


text:
Plugin ID blocks

What type of PR is this?
/kind design
/kind documentation
Any specific area of the project related to this PR?
/area registry
/area build
/area documentation

What this PR does / why we need it:
This PR aims to introduce precise guidelines for plugin ID usage. In particular, it mandates specific rules for reserved ID blocks. Also, it handles the 0 value exceptional cases, which libs may use in the future for particular purposes.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jasondellaluce, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jasondellaluce,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

repo: falcosecurity/plugins


text:
WIP master <- develop

Merge the develop branch into the master branch and deploy

Build log
Preview build with this PR (commit): https://output.circle-artifacts.com/output/job/7ca07402-9d91-4d0c-80fc-8496dfac5dd6/artifacts/0/index.html

authors: falgon kiirotori

repo: falgon/roki-web


text:
Support for 3 letter country codes?

Hello,
Thanks for this module. Would you have any interest in also supporting the "ISO 3166-1 alpha 3" codes? (The three letter country codes)?
I find multiple modules that can deal with the 2 letter codes, but currently no modules on npmjs.org that appear to be able to convert the 3 letter codes.

Thanks for the interest in this module and sorry for a such late reply.
The idea for this module was to provide access to the data from http://data.okfn.org/data/core/country-list, and I've decided to keep it that way.
So sorry, no support for other codes.

authors: fannarsh markstos

repo: fannarsh/country-list


text:
UCharPointer is not enough for a single UTF-8 character

UCharPointer points to an array of uint16_t which is generally not enough for a single UTF-8 character since the code point range is [0..0x10FFFF].
After unpack U* and write, the higher bits just vanished.
unpack S* is also not helpful at all. The array can't be packed again.

UCharPointer should use u_strFromUTF8 or u_strFromUTF8WithSub to convert to UChar string.
http://userguide.icu-project.org/strings/utf-8

@tyler-nguyen Unfortunately, it's not easy to do so with ffi code. I feel like C code is required in this case.

@fantasticfears You can add it with ffi lke this

Thanks, I made some snippets earlier in this way. But u_strFromUTF8 may yield its own error. That does requires extra work in Ruby end. And some bindings are purely made for UChar which made me wonder that a C binding sounds much directly as of Ruby's Code Set Indepedent model for string

authors: fantasticfears tyler-nguyen

repo: fantasticfears/ffi-icu


text:
quote individual parts of bigquery table locators, not the whole thing together

what dbcrossbar currently generates, which is invalid:
SELECT `id` FROM `myproject-123.tmp.mytable` WHERE mytable.batch_id = '75c600c4-6da3-47f2-8e6f-893be683ef76'

what dbcrossbar could generate, which is valid:
SELECT `id` FROM `myproject-123`.`tmp`.`mytable` WHERE mytable.batch_id = '75c600c4-6da3-47f2-8e6f-893be683ef76'

in other words, please quote the 3 parts of the table locator separately

Might be worth noting that . still needs to be backwhacked inside quoted identifiers or BQ will error.

Huh, fun. We have a whole bunch of machinery for dealing with this garbage but I guess we missed some places? Fun.

authors: emk n1ywb seamusabshere

repo: faradayio/dbcrossbar


text:
Installation Error on Node 20: spawn node-gyp ENOENT

What is the bug?
When trying to add the @farcaster/hub-nodejs package using Yarn using Node v20.5.1, I encountered the following error:
yarn add @farcaster/hub-nodejs

yarn add v1.22.19
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
[1/3] ⠄ ref-napi
[-/3] ⠄ waiting...
error /Users/danielpyrathon/Desktop/Projects/farcaster-stuff/node_modules/ref-napi: Command failed.
Exit code: 1
Command: node-gyp-build
Arguments:
Directory: /Users/danielpyrathon/Desktop/Projects/farcaster-stuff/node_modules/ref-napi
Output:
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: spawn node-gyp ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:292:12)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn node-gyp',
  path: 'node-gyp',
  spawnargs: [ 'rebuild' ]
}

Node.js v20.5.1

Upon switching to v18.17.1 (via nvm) the issue disappears.
I am on MacOS Ventura (Apple M1 Pro). I installed my node via nvm
To reproduce:
Run yarn add @farcaster/hub-nodejs on v20.5.1
Expected Behavior:
The package should be added successfully.
Actual Behavior:
Encountered an error related to node-gyp.
Let me know how I can help! and thanks for your support. I'm not blocked in any way since i can simply develop using Node 18.

FWIW I'm not able to reproduce this with Node 20.5.1 on my machine (macOS Ventura 13.5.2).
However, based on the error it looks like you might not have Python installed on your system, or NPM is unable to find it. You can find some suggestions of how to resolve in this thread, but I'm going to close since I'm unable to reproduce.

repo: farcasterxyz/hub-monorepo


text:
bug(replicator): user_data table values are missing an update

What is the bug?
I'm running my replicator node (v0.2.5) and encountered an issue with updating user data
In this API, the USER_DATA_TYPE_DISPLAY for fid 3103 is 'Aunthomie'.
https://nemes.farcaster.xyz:2281/v1/userDataByFid?fid=3103
but the display name in the replicator database table is still 'Homie'.
I can locate the correct event update in the message table, just like the screenshot below

as shown in the screenshot below, it seems that the 'updated_at' time is quite close to the message event above

I am also experiencing this issue.
https://nemes.farcaster.xyz:2281/v1/userDataByFid?fid=4085 returns the accurate and true data, but my replicator hasn't processed any jobs to update user_data:

yesterday it actually had this data, per this cast:
https://warpcast.com/christopherw.eth/0x97fd5979

Sorry for the delay in response. We no longer recommend using the replicator. Check out the Shuttle project if you'd like to sync data to a DB.

authors: christopherwxyz masato25 sds


text:
ADDING TO MAJOR WALLETS

TRUST WALLET
--Easy to use many of investor use this wallet for holding token with fast phase swapping less fees.
META MASK
-- META wallet is a secure and easy connection for any web txs safety for connecting sites too.
OTHER SUGGESTION WALLET:

KIWI WALLET
BITGET WALLET
WEB3 BINANCE WALLET

What exactly are you asking for? Farcaster is a social app, not a token.
I'm going to close this as irrelevant, but if we've misunderstood the ask, let us know.

authors: iandareen sds


text:
feat: Implement diff sync

Motivation
See https://github.com/farcasterxyz/hub/issues/66. Allows diff sync between two hubs.
TODO:

[x] A lot more tests

Change Summary
Implements the RPC calls and SyncEngine logic to determine which messages are missing and sync only those messages between two hubs.
Merge Checklist
Choose all relevant options below by adding an x now or at any time before submitting for review

[x] The title of this PR adheres to the conventional commits standard
[x] The PR has been tagged with the appropriate change type label(s) (i.e. documentation, feature, bugfix, or chore)
[ ] Changes to the protocol specification have been merged

Additional Context
Some differences between the original design and the implementation:

Instead of sending the root hash with every message, we only send it in the ContactInfo message. This was originally because I couldn't find a good way to determine the peer address we received a gossip message from. But it turned out to be a better approach (no need to send redundant data on every message).
We have a "sync threshold", and send the excluded hashes for the most recent timestamp that was a multiple of the sync threshold. This helps prevent "trashing" during high traffic.
Also removed simple sync entirely, because diff sync can completely replace it's behavior right now and there's no reason to keep both modes of sync around

@sagar-a16z i suspect this is probably worth a code review from you as well, given that it touches on some of the p2p sync stuff

@varunsrin Addressed most of your comments. The rest, in particular, the message remove tests and custom error class for user not found, I'll address in a followup. Would like to get this merged (unless there are other blockers?) and not have such a large branch hanging around too long.

authors: sanjayprabhu varunsrin

repo: farcasterxyz/hub


text:
🛑 E-solat is down

In 33ec76a, E-solat (https://www.e-solat.gov.my/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: E-solat is back up in 4e8f4bb after 12 minutes.


text:
FAI-12550 Add option to always full sync tasks and skip costly project-tasks streams

Description
There are cases where the project-tasks relationship is so costly that it accounts for the majority of the records pulled in a sync.
This PR introduces an optimization where, if the number of tasks pulled in a full sync is << number of records in the project-tasks relationship (always full sync), it could be more efficient to pull tasks in full sync (the tasks records would include the associated projects) VS tasks incrementally + projects(full) + project-tasks(full).
If the optimize_fetching_projects_and_tasks_with_full_tasks_sync is enabled, we simply remove the unwanted streams from the catalog and add tasks_full. A new stream (pretty much an alias) is needed so that we make sure that the destination receives the appropriate destination sync mode (OVERWRITE)
Another wrinkle that I had to deal with is that the destination doesn't reset models unless ALL the streams that write the model have succeeded. In this case there's an intersection of models between tasks_full and (projects, project_tasks). I marked these two as safe for reset (they'd never succeed, since they are not synced at all) so that the destination frees the respective models for reset.
Type of change

[ ] Bug fix
[X] New feature
[ ] Breaking change

Related issues

Fix #1

Migration notes

Describe migration notes if any

Extra info

Add any additional information

Another wrinkle that I had to deal with is that the destination doesn't reset models unless ALL the streams that write the model have succeeded

@ypc-faros I'm curious, what's the behavior if one of those streams is incremental?

Another wrinkle that I had to deal with is that the destination doesn't reset models unless ALL the streams that write the model have succeeded

@ypc-faros I'm curious, what's the behavior if one of those streams is incremental? Do the models get reset as well?

What I described only applies to full refresh (overwrite on the destination side) streams. Relevant code:
https://github.com/faros-ai/airbyte-connectors/blob/89e3b891b36af9498283919f757d03bdb585b9b2/destinations/airbyte-faros-destination/src/destination.ts#L1137
https://github.com/faros-ai/airbyte-connectors/blob/89e3b891b36af9498283919f757d03bdb585b9b2/destinations/airbyte-faros-destination/src/destination.ts#L559
https://github.com/faros-ai/airbyte-connectors/blob/89e3b891b36af9498283919f757d03bdb585b9b2/destinations/airbyte-faros-destination/src/converters/converter.ts#L136-L172

repo: faros-ai/airbyte-connectors


text:
Fix circular dependency check

Description

Provide description here

Type of change

[x] Bug fix
[ ] New feature
[ ] Breaking change

Related issues

Fix #589

Migration notes

Describe migration notes if any

Extra info

Add any additional information

💥 💥 💥

authors: cjwooo tovbinm


text:
Crashing after a few recursive lookups

The command I'm running is:
mxlrc.exe --token 2307233d031af73b331a2206ba7e49b793c15c613a48d523e42653 -d 5 -t 60 -s "H:\AUDIO\FLAC\CD-res\ROCK\NEED_LYRICS"
And after a couple of songs finish, I get this error. If I re-run the command sometimes it will work, other times it might repeatedly spit the error out several times but then decide to start working again. It seems very random so I am unable to find a specific cause. It is sometimes able to traverse a few directories and work on several of my artist folders, other times it can barely make it past a few songs of a single album.
[+] Song found: Linkin Park - LIES GREED MISERY
[+] Searching lyrics: Linkin Park - LIES GREED MISERY
[+] Formatting lyrics
Lyrics saved: H:\AUDIO\FLAC\CD-res\ROCK\NEED_LYRICS\Linkin Park\2012. Living Things\D1T04 Lies Greed Misery.lrc
    Please wait... 0s

[+] Searching song: Linkin Park - I'll Be Gone
Traceback (most recent call last):
  File "C:\Users\AZOTHY~1\AppData\Local\Temp\ON441D~1\mxlrc.py", line 360, in <module>
  File "C:\Users\AZOTHY~1\AppData\Local\Temp\ON441D~1\mxlrc.py", line 299, in main
  File "C:\Users\AZOTHY~1\AppData\Local\Temp\ON441D~1\mxlrc.py", line 277, in get_lrc
  File "C:\Users\AZOTHY~1\AppData\Local\Temp\ON441D~1\mxlrc.py", line 52, in find_lyrics
KeyError: 'matcher.track.get'

After this started happening repeatedly with a couple of directories, I generated a new API token to use and the error went away for a while. Maybe song lookups are rate limited as well besides just fetching lyrics and this is causing an ungraceful exit of the program?

repo: fashni/MxLRC


text:
Validation does not check parentheses

Validation does not check if parentheses closes or is open and return true, tho evaluate throws emptystack or mismatched parenthesis error.
Function used : (0.14_1.5)/(((I/600)^0.02)-1)
Incorrect use: (0.14_1.5)/((I/600)^0.02)-1) ; 0.14*1.5)/(((I/600)^0.02)-1)

@Fadelis Currently exp4j does not do bracket matching during validation. But this of course makes sense and I think this should be fairly easy to implement.
So give me some time and Ill come back to you.

The issue here seem to be that, the unmatched brackets throw an error even before validation takes place.
So e.g.:
Expression exp = new ExpressionBuilder("(1))")
	.build();

throws an EmtpyStackException in the ShuntingYard. But I know that the Shunting Yard algorithm is able to detect unmatched brackets.
I guess it's possible to throw a more detailed Exception here. But validation of such an invalid expression is not possible since the error is created earlier in the control flow, without allowing invalid expression to be built via the ExpressionBuilder.

Maybe you could create a static validation method of a string, which would check all the validation errors?

There is a bug in the implementation of ShuntingYard in the handling of closing parentheses. I suggest this fix:
            case Token.TOKEN_PARENTHESES_CLOSE:
                try {
                    while (stack.peek().getType() != Token.TOKEN_PARENTHESES_OPEN) {
                        output.add(stack.pop());
                    }
                } catch (EmptyStackException e) {
                    throw new IllegalArgumentException("Mismatched parentheses detected. Please check the expression");
                }
                stack.pop();
                if (!stack.isEmpty() && stack.peek().getType() == Token.TOKEN_FUNCTION) {
                    output.add(stack.pop());
                }
                break;

authors: Fadelis fasseg secondmanonmars


text:
Result collapse

While scrapping for large quantities of images the results are pretty big and not very important for viewer.
Describe the solution you'd like
I would like for feature to collapse only the result and not code along with. The viewer can expand the result if required.
Describe alternatives you've considered
I tried to collapse or hide but that is not helpful
Additional context
I was building a image classifier using images scrapped from the internet. I build a large dataset of few hundred images. I feel scrolling through all those image results while scrapping will be unnecessary for the viewer.

Do you mind opening an issue in fastai/nbdev about this?  This is where the change needs to be made, and I can help.
Thank you

@sharan19 Going ahead and closing this, please add the the issue to nbdev so we can work on this

repo: fastai/fastpages


text:
📝 Improve documentation on using dataclasses in FastAPI

I was reading the FastAPI documentation this morning and came across the following sentence: 'FastAPI is built on top of Pydantic, and I have been showing you how to use Pydantic models to declare requests and responses. But FastAPI also supports using dataclasses the same way.' While I don't disagree with the content, it is widely known that FastAPI was initially maintained by a single person. Therefore, I decided to rephrase the sentence to remove the personal tone.
Revised the documentation to provide a clearer explanation of how FastAPI integrates with Pydantic and supports dataclasses. The previous language was instructional and personal, which may not be appropriate for all readers. The updated text maintains a neutral tone and emphasizes the functionality of FastAPI in handling dataclasses similarly to Pydantic models.

Thanks for the interest @AdamMusa, but I think it's fine as it is. So I'll pass on this one. But thanks! :coffee:

authors: AdamMusa tiangolo

repo: fastapi/fastapi


text:
LU - compact method

New Feature
This is just thought really - a different LU factorisation.
Interested in your thoughts and suggestions for benchmarks / tests / whether it's got legs and so on.

[x] Code conforms to style guide
[ ] Doctests
[x] Unit-tests
[ ] Documentation
[ ] New line item in releases.md
[ ] Info added to literature directory

Bug Report

[ ] Minimal repro in a test function

Approved... @rjenc29 did you want to merge this for the first release?

Yep, that would be fine with me.  I have a couple of other LU methods in the works (not ready yet).
I appreciate 'lu_compact' isn't a great name - will think of something better in due course perhaps.

LGTM, merging

authors: dwillmer pawroman rjenc29

repo: fastats/fastats


text:
Add first training tutorial notebook and configure mkdocs

TODO: update the colab notebook link on the index.md to use the new tutorial

In #4 there's more details about the rendered docs and why this extension was chosen.

repo: fastaudio/fastaudio


text:
Behavior is different for content type

🐛 Bug Report
Not sure if this is a bug or is it planned.
Default fastify behavior is to encode objects to JSON and set content type to 'application/json'; set content type to 'text/plain' for strings.
With fastify-funky response is always encoded as JSON.
To Reproduce
Steps to reproduce the behavior:
import { fastify } from 'fastify';
import funky from 'fastify-funky';
import * as T from 'fp-ts/Task';
import * as E from 'fp-ts/Either';
import * as TE from 'fp-ts/Task';

const text = "text";
const obj = { json: true };

const server = fastify().register(funky);

// type: application/json; body: {"json":true}
server.get("/simple-json", async () => obj);
// type: text/plain; body: text
server.get("/simple-text", async () => text);
// type: application/json; body: {"json":true}
server.get("/task-json", async () => T.of(obj));
// type: application/json; body: "text" (json string!)
server.get("/task-text", async () => T.of(text));
// type: application/json; body: {"json":true}
server.get("/either-json", async () => E.of(obj));
// type: application/json; body: "text"
server.get("/either-text", async () => E.of(text));
// type: application/json; body: {"json":true}
server.get("/taskeither-json", async () => TE.of(obj));
// type: application/json; body: "text"
server.get("/taskeither-text", async () => TE.of(text));

server.listen(3000);

Expected behavior
Same behavior as with simple handler return types. For example, if final type after "unwrapping" is string, return string (not json encoded string) and set content-type to 'plain/text''. Same for buffer, etc.
Related code: https://github.com/fastify/fastify/blob/v3.10.1/lib/reply.js#L127-L182
Your Environment

node version: 12
fastify version: 3.10.1
fastify-funky version: 1.0.1
os: Linux

@knopki Somehow missed it the first time around. Thanks for the report, will fix it.

authors: kibertoad knopki

repo: fastify/fastify-funky


text:
add Procfile for foreman

Now that fastlane.ci requires multiple processes to run, we can take advantage of tools like https://github.com/ddollar/foreman to start or manage the services for us.
Not putting foreman in the Gemfile because @ddollar says so.

@snatchev so is the idea that we need to install foreman outside of the gemfile?

Yes, that's the author's recommendation.

authors: snatchev taquitos

repo: fastlane/ci