text:
Incorrect result from dwrf::Writer or TableScan for Timestamp data

Bug description
I happened to come across an error when testing aggregation functions with TableScan. Testing data was first written to a file and then read back by TableScan. I noticed the output data from TableScan is incorrect. The error can be reproduced via the following unit test.
void writeToFile(
    const std::string& path,
    const VectorPtr& vector,
    memory::MemoryPool* pool) {
  dwrf::WriterOptions options;
  options.schema = vector->type();
  options.memoryPool = pool;
  auto writeFile = std::make_unique<LocalWriteFile>(path, true, false);
  auto sink =
      std::make_unique<dwio::common::WriteFileSink>(std::move(writeFile), path);
  dwrf::Writer writer(std::move(sink), options);
  writer.write(vector);
  writer.close();
}

TEST_F(MinMaxByAggregationTestBase, writeReadRoundTripError) {
  auto vector = makeRowVector(
      {makeNullableFlatVector<Timestamp>(
           {std::nullopt,
            dataAt<Timestamp>(2),
            std::nullopt,
            dataAt<Timestamp>(1),
            std::nullopt,
            dataAt<Timestamp>(0)}),
       makeNullableFlatVector<Timestamp>(
           {dataAt<Timestamp>(4),
            dataAt<Timestamp>(4),
            std::nullopt,
            dataAt<Timestamp>(1),
            dataAt<Timestamp>(1),
            dataAt<Timestamp>(3)}),
       makeNullableFlatVector<int32_t>(
           {dataAt<int32_t>(0),
            dataAt<int32_t>(1),
            dataAt<int32_t>(2),
            dataAt<int32_t>(2),
            dataAt<int32_t>(1),
            dataAt<int32_t>(0)})});
  std::vector<std::shared_ptr<exec::test::TempFilePath>> files;
  auto file = exec::test::TempFilePath::create();
  auto writerPool = rootPool_->addAggregateChild("writer");
  writeToFile(file->path, vector, writerPool.get());

  std::vector<exec::Split> splits;
  splits.emplace_back(std::make_shared<connector::hive::HiveConnectorSplit>(
      "test-hive", file->path, dwio::common::FileFormat::DWRF));

  PlanBuilder builder(pool());
  builder.tableScan(asRowType(vector->type()));

  AssertQueryBuilder queryBuilder(builder.planNode());
  queryBuilder.splits(splits).assertResults(vector);
}

System information
N/A
Relevant logs
Failed
Expected 6, got 6
1 extra rows, 1 missing rows
1 of extra rows:
	"1970-01-01T00:00:00.123000000" | "1970-01-01T00:00:01.123000000" | -2

1 of missing rows:
	"1969-12-31T23:59:59.123000000" | "1970-01-01T00:00:01.123000000" | -2

Unexpected results

This is a known problematic behavior that we put in intentionally to match behavior with Java DWRF, i.e. "the second before UTC epoch with non zero nanos will always be converted to second after UTC epoch with same nanos."  We should avoid using timestamp from this period (between 1969-12-31T23:59:59.000000001 and 1969-12-31T23:59:59.999999999) because it is not representable in DWRF.
See details in:
https://github.com/facebookincubator/velox/blob/b1e9d8a0c17bb321dbed8c43d22c294787ac94e2/velox/dwio/dwrf/writer/ColumnWriter.cpp#L706

authors: Yuhta kagamiori

repo: facebookincubator/velox


text:
Azure adapter not building on CI

Problem description
https://app.circleci.com/pipelines/github/facebookincubator/velox/34302/workflows/9ad76106-de16-4b09-9e87-da5587eb9d44/jobs/223901
cc: @gaoyangxiaozhu
System information
https://app.circleci.com/pipelines/github/facebookincubator/velox/34302/workflows/9ad76106-de16-4b09-9e87-da5587eb9d44/jobs/223901
CMake log
No response

cc: @gaoyangxiaozhu

@kgpai Curious how the offending PR landed in the first place? Did CI not run on it or did it pass somehow?

cc: @gaoyangxiaozhu

hey @kgpai  it works when I run do CI.
Does retry works and did you check the log file  /root/adapter-deps/azure-sdk-for-cpp/sdk/storage/azure-storage-common/_build/vcpkg-manifest-install.log for details ?
CMake Error at _build/_deps/vcpkg-src/scripts/buildsystems/vcpkg.cmake:895 (message):
vcpkg install failed.  See logs for more information:
/root/adapter-deps/azure-sdk-for-cpp/sdk/storage/azure-storage-common/_build/vcpkg-manifest-install.log

Hi @gaoyangxiaozhu , Can you open a PR setting ABFS to true - so we can revalidate on circle ci.

Hi @gaoyangxiaozhu , Can you open a PR setting ABFS to true - so we can revalidate on circle ci.

hey @kgpai  sure, this is the PR https://github.com/facebookincubator/velox/pull/6791 let use it for revalidate.
meanwhile, i actually also raise another follow PR https://github.com/facebookincubator/velox/pull/6675 to add abfs read support,  the CI run on that PR works with abfs enabled.

Hi @gaoyangxiaozhu , Can you open a PR setting ABFS to true - so we can revalidate on circle ci.

hey @kgpai you can see the CI passed https://github.com/facebookincubator/velox/pull/6791

source: gharchive/issue

authors: gaoyangxiaozhu kgpai mbasmanova

license_type: permissive

license_source: github-api


text:
Add support for (UT/UTC/GMT)(+/-)H[H] time zone IDs in from_unixtime and at_timezone Presto UDFs

Summary:
Presto Java and Joda support time zone IDs that look like UT+1, UTC-12, GMT+3, Etc/UT-1,
Etc/UTC+7, Etc/GMT-2.  Of these Velox only supports the Etc/GMT prefix.  This diff adds support for
all 6.
They all map to the offset number of hours in the suffix (except Etc/GMT which flips the sign
for some reason).  E.g. UT+1 maps to +01:00, while Etc/GMT-2 maps to +02:00.
Differential Revision: D65366170

@kevinwilfong Can you take a look at this PR https://github.com/facebookincubator/velox/pull/11114? I am currently developing timezone normalization for Spark, which includes all the timezones supported by Spark (In the description). Could you help to see which of these are supported by Presto? If they are also supported by Presto, I can add them to the common call section for both Presto and Spark. cc @rui-mo @PHILO-HE @pedroerp.

The part you added is also being called by Spark. I noticed some inconsistencies, for example, UTC+1 is supported by Spark, but utc+1 is considered an invalid timezone by Spark. Spark is case-sensitive with timezones; does Presto ignore case sensitivity?
spark-sql (default)> select cast('2022-02-02 12:00:00 utc+1' as timestamp);
NULL
Time taken: 0.6 seconds, Fetched 1 row(s)
spark-sql (default)> select cast('2022-02-02 12:00:00 UTC+1' as timestamp);
2022-02-02 19:00:00
Time taken: 0.053 seconds, Fetched 1 row(s)

Hi @kevinwilfong, Could you take a look at this PR #11114? I am currently developing timezone normalization for Spark, which includes all the timezones supported by Spark (In the description). Could you help to see which of these are supported by Presto? If they are also supported by Presto, I can add them to the common call section for both Presto and Spark.
If there are timezone formats that Presto does not support, I think they can also be added to the normalization rules. The timezone formats supported by Velox can be the union of those supported by both Presto and Spark, which would not impact user usage.
cc @rui-mo @PHILO-HE @pedroerp.

I agree. The current timezone normalization in TimezoneMap can be extended for both Spark and Presto, assuming there is no  semantic conflict case.

source: gharchive/pull-request

authors: PHILO-HE kevinwilfong liujiayi771


text:
Add support for allocations that exceed the PageRun limit

This change enables an allocation to exceed the PageRun limit. We do this by splitting an allocation
across multiple PageRuns.
Non-contiguous allocations in the MallocAllocator use malloc to allocate and store the address
to later free the allocation. Since an allocation can now span across multiple PageRuns, we need to
store the PageRun count for each allocation to identify allocation boundaries within the PageRuns.
Since an address only requires 48 bits, we use the remaining 16 bits to store the PageRun count.

Closing this performant implementation in favor of a simpler implementation https://github.com/facebookincubator/velox/pull/8298
We can revert to this PR if performance is a bottleneck.


text:
Switch to Centos 8 Stream, add curl-devel to the linux CI image

This should resolve #956.

The following test failed
The following tests FAILED:
	113 - velox_functions_test (Failed)

Particularly
111: ../../velox/functions/prestosql/tests/DateTimeFunctionsTest.cpp:319: Failure
111: Expected equality of these values:
111:   21
111:   hour(Timestamp(4000000000, 0))
111:     Which is: 16-byte object <14-00 00-00 00-00 00-00 01-C5 56-80 FE-7F 00-00>
\111: ../../velox/functions/prestosql/tests/DateTimeFunctionsTest.cpp:320: Failure
111: Expected equality of these values:
111:   21
111:   hour(Timestamp(4000000000, 123000000))
111:     Which is: 16-byte object <14-00 00-00 00-00 00-00 01-C5 56-80 FE-7F 00-00>

This test currently always fails on macOS. It now fails on the latest CentOS.
The particular test is
setQueryTimeZone("Pacific/Apia");
319:  EXPECT_EQ(21, hour(Timestamp(4000000000, 0)));
319:  EXPECT_EQ(21, hour(Timestamp(4000000000, 123000000)));

However, the correct answer seems to be 20 for both. I assume there is a timezone library dependency here.
But changing to 20 seems to be the right fix as both macOS and CentOS now behave the same.

I assume there is a timezone library dependency here.
But changing to 20 seems to be the right fix as both macOS and CentOS now behave the same.

date tells the correct hour is 21?
$ TZ=Pacific/Apia date -d '@4000000000'
Tue Oct  2 21:06:40 +14 2096

Am I doing something wrong here? I'd rather disable the test for now, if we're not sure.

I assume there is a timezone library dependency here.
But changing to 20 seems to be the right fix as both macOS and CentOS now behave the same.

date tells me the correct hour is 21?
$ TZ=Pacific/Apia date -d '@4000000000'
Tue Oct  2 21:06:40 +14 2096

Am I doing something wrong here? I'd rather disable the test for now, if we're not sure.

Ah yes, the timezone database on my Mac disagrees with the one here on Fedora 34. Great. I'm just going to comment this test out for now.

On CentOS Stream 8, it is 20. Good call on disabling for now.
sh-4.4# TZ=Pacific/Apia date -d '@4000000000'
Tue Oct  2 20:06:40 +13 2096

authors: funrollloops majetideepak


text:
CLI missing --features flag

First of all, thanks for the dataset!
On your website, you mention a --features flag for the CLI. However, there is no such flag implemented in config.py. Seems like one should use --dataset slowfast8x8_r101_k400. So I guess you should either add the --features flag as a shortcut for this or adjust the website.

Hi Leonard, Exactly right.  We'll update the docs, and please use --dataset with the particular feature in its place.

authors: ebyrne lbaermann

license: MIT


text:
Error when using generate-lines with CPU model

Hi,
I am trying to test the translation for the English to German CPU model. I get the following error when trying to load the model

Hi @shahbazsyed, try updating your nn installation, e.g. via luarocks install nn.

I'm having the same problem.  However when I run the "luarocks install nn", it's giving a compile error which may be due to my environment, but I've only done what the torch installation instructions said to do...
`
.....
-- Generating done
-- Build files have been written to: /home/ddurham/tmp/luarocks_nn-scm-1-7697/nn/build
Scanning dependencies of target THNN
[ 50%] Building C object lib/THNN/CMakeFiles/THNN.dir/init.c.o
In file included from /home/ddurham/torch/install/include/TH/TH.h:4:0,
from /home/ddurham/tmp/luarocks_nn-scm-1-7697/nn/lib/THNN/init.c:1:
/home/ddurham/tmp/luarocks_nn-scm-1-7697/nn/lib/THNN/generic/SparseLinear.c: In function ‘THNN_FloatSparseLinear_accGradParameters’:
/home/ddurham/torch/install/include/TH/THTensor.h:8:39: error: too many arguments to function ‘THFloatTensor_sum’
#define THTensor_(NAME)   TH_CONCAT_4(TH,Real,Tensor_,NAME)
^
/home/ddurham/torch/install/include/TH/THGeneral.h:108:37: note: in definition of macro ‘TH_CONCAT_4_EXPAND’
#define TH_CONCAT_4_EXPAND(x,y,z,w) x ## y ## z ## w
^
/home/ddurham/torch/install/include/TH/THTensor.h:8:27: note: in expansion of macro ‘TH_CONCAT_4’
#define THTensor_(NAME)   TH_CONCAT_4(TH,Real,Tensor_,NAME)
^
/home/ddurham/tmp/luarocks_nn-scm-1-7697/nn/lib/THNN/generic/SparseLinear.c:237:3: note: in expansion of macro ‘THTensor_’
THTensor_(sum)(buf, gradOutput, 0, 1);
^
/home/ddurham/torch/install/include/TH/THTensor.h:8:39: note: declared here
#define THTensor_(NAME)   TH_CONCAT_4(TH,Real,Tensor_,NAME)
^
/home/ddurham/torch/install/include/TH/THGeneral.h:108:37: note: in definition of macro ‘TH_CONCAT_4_EXPAND’
#define TH_CONCAT_4_EXPAND(x,y,z,w) x ## y ## z ## w
^
/home/ddurham/torch/install/include/TH/THTensor.h:8:27: note: in expansion of macro ‘TH_CONCAT_4’
#define THTensor_(NAME)   TH_CONCAT_4(TH,Real,Tensor_,NAME)
^
/home/ddurham/torch/install/include/TH/generic/THTensorMath.h:77:13: note: in expansion of macro ‘THTensor_’
TH_API void THTensor_(sum)(THTensor *r_, THTensor *t, int dimension);
^
....
`
other similar errors follow.  Ever run into this?

Same here. Fresh Torch install on Ubuntu 16 intel64. Unable to upgrade nn with the same error as @ddurham2.

@ddurham2 and @mainka, I don't really know what's wrong here. I suggest heading over to https://github.com/torch/nn for searching prior issues and then filing a new one if necessary.

@ddurham2 @mainka Try doing a luarocks install torch once before luarocks install nn . Worked for me. I am not sure what was wrong as I also did a fresh install of torch before starting with fairseq.
Anyhow, hope it helps. I could load the CPU model.

@jgehring Is there a cmdline argument to specify the gpuid to use for loading the CUDA model for testing? I have 8 GPUs(8GB memory each) and 6 of them are in use by other process. I don't understand how the model is being loaded but it always gives me out of memory error after loading the dictionaries though I have 2 GPUs free.
If it helps, I am running this inside a docker container on the host with 8 GPUs.

No there is no such flag; I suggest using the CUDA_VISIBLE_DEVICES environment variable to control which GPUs are available to the program.

Please see the discussion at #18

Yes it's working for me.  Over at nn, another issue that installing torch from luarocks works.  I think nn simply has a problem compiling against the HEAD of torch (i.e. when you follow their directions of cloning from git and building).

Ok, thanks for the context!

Got the same error: looks like torch nn does not compile anymore with torch master:
luarocks install nn
Installing https://raw.githubusercontent.com/torch/rocks/master/nn-scm-1.rockspec...
Using https://raw.githubusercontent.com/torch/rocks/master/nn-scm-1.rockspec... switching to 'build' mode
Cloning into 'nn'...
remote: Counting objects: 319, done.
remote: Compressing objects: 100% (278/278), done.
remote: Total 319 (delta 76), reused 140 (delta 38), pack-reused 0
Receiving objects: 100% (319/319), 601.91 KiB | 0 bytes/s, done.
Resolving deltas: 100% (76/76), done.
Checking connectivity... done.
cmake -E make_directory build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="/home/wtambellini/torch/install/bin/.." -DCMAKE_INSTALL_PREFIX="/home/wtambellini/torch/install/lib/luarocks/rocks/nn/scm-1" && make
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Torch7 in /home/wtambellini/torch/install
-- TH_LIBRARIES: TH
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp
-- Compiling with OpenMP support
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/luarocks_nn-scm-1-9722/nn/build
Scanning dependencies of target THNN
[ 50%] Building C object lib/THNN/CMakeFiles/THNN.dir/init.c.o
In file included from /home/wtambellini/torch/install/include/TH/TH.h:4:0,
from /tmp/luarocks_nn-scm-1-9722/nn/lib/THNN/init.c:1:
/tmp/luarocks_nn-scm-1-9722/nn/lib/THNN/generic/SparseLinear.c: In function ‘THNN_FloatSparseLinear_accGradParameters’:
/home/wtambellini/torch/install/include/TH/THTensor.h:8:39: error: too many arguments to function ‘THFloatTensor_sum’
#define THTensor_(NAME)   TH_CONCAT_4(TH,Real,Tensor_,NAME)                                       ^
/home/wtambellini/torch/install/include/TH/THGeneral.h:108:37: note: in definition of macro ‘TH_CONCAT_4_EXPAND’
#define TH_CONCAT_4_EXPAND(x,y,z,w) x ## y ## z ## w
even if I do a
luarocks install torch
before
luarocks install nn
Cheers
W.

authors: WilliamTambellini ddurham2 jgehring mainka shahbazsyed

repo: facebookresearch/fairseq


text:
Uninitialised bias parameters in RelPositionMultiHeadedAttention

🐛 Bug
The (u, v) bias terms in the MHA layer with relative position encodings are defined using torch.Tensors:
https://github.com/facebookresearch/fairseq/blob/4fe8583396191c22011350248119db98ec1b5cb8/fairseq/modules/espnet_multihead_attention.py#L127-L128
Since torch.Tensor creates an un-initialised tensor, its use results in non-deterministic behaviour, poor initialisation, and nans if you have an unlucky init:
>>>import torch
>>> torch.Tensor(1, 1).sum()
tensor(0.)
>>> torch.Tensor(10, 10).sum()
tensor(4.9312e-38)
>>> torch.Tensor(100, 100).sum()
tensor(nan)

Section 3.3 of the referenced paper Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context does not specify the initialisation for the bias terms. Since torch.Tensor is typically populated with zeros, setting the initialisation to torch.zeros should provide a fix that is close to the intended behaviour.

Hi Sanchit, thanks for identifying the issue and fixing it.


text:
Remove retention of nightly packages

Motivation and Context
Given our average binary size of ~150 MB and building 12 different binaries on the CI, we can't retain any and stay under the 3 GB limit.
How Has This Been Tested
Just changing a number

Hmmm, no retention has the issue that, for the ~2 hours it takes to build everything there are no conda packages in the nightly channel

Hmmm, no retention has the issue that, for the ~2 hours it takes to build everything there are no conda packages in the nightly channel

Exactly why we need to enable LTO for Conda-binaries.

@erikwijmans According to the nightly channel, we only do 8 builds a day: https://anaconda.org/aihabitat-nightly/habitat-sim/files

Ah that's right, we aren't building bullet == true headless == false on the CI for some reason.  Although we do need to fix that because that is gonna cause a ton of issues for hab2

@erikwijmans thank you for fixing the freshness of nightly builds.  Looks like we can increase at least to 2 versions.

Yep, only after enabling LTO. Without it we don't have room for even 2.
On Thu, May 27, 2021 at 10:53 PM Oleksandr @.***> wrote:

@erikwijmans https://github.com/erikwijmans thank you for fixing the
freshness of nightly builds. Looks like we can increase at least to 2
versions.
[image: Screen Shot 2021-05-27 at 7 49 56 PM]
https://user-images.githubusercontent.com/877440/119922590-f3815780-bf24-11eb-99ed-9a8221d65d42.png
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/habitat-sim/pull/1262#issuecomment-850073321,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAPVMXZM3BJ3BN73KP5CTZ3TP4ARRANCNFSM45NYWOAQ
.

authors: Skylion007 erikwijmans mathfac

repo: facebookresearch/habitat-sim


text:
Add conversion code to the BOP format

This PR contains 3 scripts to convert the HOT3D-Clips dataset from its native format to the BOP format. The BOP format had to be extended because the original format only included RGB and depth modalities. This extension made it possible to include a varying number of modalities (RGB, gray1, gray2) with separate annotations for each modality. The dataset has already been converted using these scripts and uploaded to the BOP website. These scripts went through several revisions, as the conversion process required multiple changes to adapt the dataset to the new standard format, which was initially not strictly defined and required frequent adjustments. The scripts in this PR are isolated and do not make any changes to the existing code in the repository.
The 3 Scripts are:

hot3d_models_to_bop.py : Script that converts the 3D object (full) models from the GLTF format to the PLY format
hot3d_models_eval_to_bop.py: Script to convert the eval object model
hot3d_clips_to_bop_scenewise.py: Script to convert the dataset itself

Thank you @AnasIbrahim,

can you start the process with signing the Facebook CLA and add a description to the text of your PR?

@SeaOtocinclus I just made a trivial change to the argparse input parameters. To make it easier to run the conversion on the BOP server.

@AnasIbrahim Look good to me, I will merge.

authors: AnasIbrahim SeaOtocinclus


text:
ModuleNotFoundError: No module named 'maskrcnn_benchmark.config'

❓ Questions and Help
$ python webcam.py --min-image-size 800
Traceback (most recent call last):
File "webcam.py", line 5, in
from maskrcnn_benchmark.config import cfg
ModuleNotFoundError: No module named 'maskrcnn_benchmark.config'

Hi, are you using python 2?
Also, did you follow the installation instructions in INSTALL.md?

I've added a section in the README with instructions that should help you fix your problem. It is probably related to https://github.com/facebookresearch/maskrcnn-benchmark/issues/22
I'm closing this issue, but let me know if you don't manage to solve your problem.

Hi I still have this problem using this setup, do you know what's wrong with it?@fmassa Thanks!

You didn't install maskrcnn-benchmark properly.
Note that now you need to have a nightly version of PyTorch. 1.0.0 or 1.0.1 won't work

Same error.
$ pwd
/home/emblab/satyam/maskrcnn/maskrcnn-benchmark/demo
$ python -V
Python 3.6.8 :: Anaconda custom (64-bit)

import torch
print(torch.version)
1.2.0.dev20190704

$ nvcc --version
The pinstarogram 'nvcc' is currently not installed. You can install it by typing:
sudo apt ll nvidia-cuda-toolkit
$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
No GPU on the system
$ python webcam.py --min-image-size 300 MODEL.DEVICE cpu
Traceback (most recent call last):
File "webcam.py", line 5, in
from maskrcnn_benchmark.config import cfg
ModuleNotFoundError: No module named 'maskrcnn_benchmark'
Can you help me out @fmassa

Same error.
$ pwd
      /home/emblab/satyam/maskrcnn/maskrcnn-benchmark/demo

$ python -V
Python 3.6.8 :: Anaconda custom (64-bit)

>>> import torch
>>> print(torch.__version__)
1.2.0.dev20190704

$ nvcc --version
      The pinstarogram 'nvcc' is currently not installed. You can install it by typing:
      sudo apt ll nvidia-cuda-toolkit

$ gcc --version
       gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
       Copyright (C) 2015 Free Software Foundation, Inc.
       This is free software; see the source for copying conditions.  There is NO
       warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

GPU-No (Only CPU)
$ python webcam.py --min-image-size 300 MODEL.DEVICE cpu
       Traceback (most recent call last):
           File "webcam.py", line 5, in <module>
        from maskrcnn_benchmark.config import cfg
        ModuleNotFoundError: No module named 'maskrcnn_benchmark'

Can you help me out @fmassa

i have the same problem with u , have u solved it?

@AnnZhn, yeah the torchvision-nightly version was conflicting with torchvision. I uninstalled torchvision using: pip uninstall torchvision.

repo: facebookresearch/maskrcnn-benchmark


text:
BoxCoder have

❓ Questions and Help

In your code the  proposal and reference_boxes positiong wrong?
def encode(self, reference_boxes, proposals):
    """
    Encode a set of proposals with respect to some
    reference boxes
    Arguments:
        reference_boxes (Tensor): reference boxes
        proposals (Tensor): boxes to be encoded
    """

    TO_REMOVE = 1  # TODO remove
    ex_widths = proposals[:, 2] - proposals[:, 0] + TO_REMOVE
    ex_heights = proposals[:, 3] - proposals[:, 1] + TO_REMOVE
    ex_ctr_x = proposals[:, 0] + 0.5 * ex_widths
    ex_ctr_y = proposals[:, 1] + 0.5 * ex_heights

    gt_widths = reference_boxes[:, 2] - reference_boxes[:, 0] + TO_REMOVE
    gt_heights = reference_boxes[:, 3] - reference_boxes[:, 1] + TO_REMOVE
    gt_ctr_x = reference_boxes[:, 0] + 0.5 * gt_widths
    gt_ctr_y = reference_boxes[:, 1] + 0.5 * gt_heights

    wx, wy, ww, wh = self.weights
    targets_dx = wx * (gt_ctr_x - ex_ctr_x) / ex_widths
    targets_dy = wy * (gt_ctr_y - ex_ctr_y) / ex_heights
    targets_dw = ww * torch.log(gt_widths / ex_widths)
    targets_dh = wh * torch.log(gt_heights / ex_heights)

    targets = torch.stack((targets_dx, targets_dy, targets_dw, targets_dh), dim=1)

return targets

I means actually target is equal to
for example , gt_ctr_x is the anchor label? right ?
targets_dx = wx * (ex_ctr_x - gt_ctr_x) / gt_widths

OK, I find your code is right ,but the para name is awesome.
regression_targets_per_image = self.box_coder.encode(
matched_targets.bbox, anchors_per_image.bbox
)


text:
one question about the implementation

Hi,
Thanks for the code.
While q and k are used for positive pairs, q and queue are used for negative pairs, what if earlier samples of q are contained in queue ?
Since a mini batch is sampled randomly, an earlier version of q is possibly in the queue. As are the same instances, they should be positive pairs. However, the are treated negative pairs in the implementation code.

Hello,
The mini-batch is not exactly sampled randomly. It is sampled from the dataset shuffled at every epoch, without replacement. When you call enumerate(dataloader), an iterator of a DataLoader is created which stops when it has used once all the samples of the dataset.
As long as the queue size is smaller than the dataset size - batch_size, you cannot encounter the same samples in the batch and in the queue.

authors: howard-mahe shuuchen

repo: facebookresearch/moco


text:
Update to use FetchContent_MakeAvailable, and avoid build errors due to using the same directory and now use unique directory

Pull Request Template
Summary

Briefly describe the changes made in this pull request:

FetchContent_Populate is deprecated, so migrate to use FetchContent_MakeAvailable
I got errors when buildling for the first time, regarding the directory, this change fixed it for me.

Changes

Describe the changes made in this pull request:

Few changes in Cmake file

Motivation

Explain the motivation behind making these changes:

Improve experience when onboarding and building

Testing

Describe the testing performed to verify the changes:

[Insert testing details]
Include information about:

Test cases added or modified
Results of testing

Checklist

Confirm that the following have been completed:

[ ] Code compiles and runs successfully on all supported platforms
[ ] All tests pass
[ ] Documentation is updated (if applicable)
[ ] Changes follow the project's coding standards

Additional Comments

Any additional comments or information that may be helpful:

[Insert additional comments]

What was the build error you were getting?

Hi @salmanmkc, thank you for reporting this issue sorry for the delay in processing your pull request.
We would like to repro the build error you were seeing so that we can test that this change fixes the issue.
Could you share with us information that you think might help us repro the error you were seeing?  Please find example information that may help us below:

Type of system you are running build on

e.g. x86-64 Ubuntu 24.04 LTS)

How the build was performed

Command used

e.g. Ocean's build script or direct call to cmake?

Any guesses about what is different about your system that may contribute to this behavior

CMake version
Environment variables that modify CMake behavior

Commit https://github.com/facebookresearch/ocean/commit/8a75b0e386e924e835c0a2a89285ef48f76c2c98 should resolve the warning about deprecated usage of FetchContent_Populate(). It was addressed by the upstream provider and we updated to their latest release.

Hey, yeah sorry I did not get round to finishing this PR, I realised that there was more to be done but have been busy with other work/projects. though I see in commit https://github.com/facebookresearch/ocean/commit/8a75b0e386e924e835c0a2a89285ef48f76c2c98 @leapally that FetchContent_Populate is still used, would that not be a problem when it becomes depreciated? Sorry typing this from a train so I don't have much insight right now and just remmebered I need to reply to this.

Only the single-argument version of FetchContent_Populate is deprecated: https://cmake.org/cmake/help/latest/module/FetchContent.html#:~:text=The command supports,use FetchContent_MakeAvailable() instead.
You mentioned another build error, related to needing a unique directory name. Could you provide more detail?

that was when trying to make a workaround, but I think the fix that you have mentioned is better in this case, I can close this I guess in this case. I will test out the fix mentioned though, thanks! Sorry for the extra work.

authors: ASchneiderMeta leapally salmanmkc

repo: facebookresearch/ocean


text:
Cannot Build SAM v2 i have pytorch installed on GPU but build scrypt seem to create a venv  pip-build-env-n0may61s that does not have the required dependencies ?

D:\Applfich\SAM2\segment-anything-2>pip install -e ".[demo]"
Looking in indexes: https://artifactory.hydro.qc.ca/artifactory/api/pypi/global-pypi-virtual/simple
Obtaining file:///D:/Applfich/SAM2/segment-anything-2
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... error
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> [32 lines of output]
C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\torch_subclasses\functional_tensor.py:258: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\utils\tensor_numpy.cpp:84.)
cpu = _conversion_method_template(device=torch.device("cpu"))
Traceback (most recent call last):
File "C:\Users\dd1968\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 353, in
main()
File "C:\Users\dd1968\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dd1968\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip_vendor\pyproject_hooks_in_process_in_process.py", line 132, in get_requires_for_build_editable
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\setuptools\build_meta.py", line 458, in get_requires_for_build_editable
return self.get_requires_for_build_wheel(config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\setuptools\build_meta.py", line 327, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=[])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\setuptools\build_meta.py", line 297, in _get_build_requires
self.run_setup()
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\setuptools\build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 70, in
File "", line 51, in get_extensions
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\torch\utils\cpp_extension.py", line 1076, in CUDAExtension
library_dirs += library_paths(cuda=True)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\torch\utils\cpp_extension.py", line 1214, in library_paths
paths.append(_join_cuda_home(lib_dir))
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\LOCAL\TEMP\pip-build-env-n0may61s\overlay\Lib\site-packages\torch\utils\cpp_extension.py", line 2416, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build editable did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
CUDA is enable pytorch is using GPU ?
D:\Applfich\SAM2\segment-anything-2>pip install torch Looking in indexes: https://artifactory.hydro.qc.ca/artifactory/api/pypi/global-pypi-virtual/simple Requirement already satisfied: torch in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (2.4.0+cu124) Requirement already satisfied: filelock in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (3.13.1) Requirement already satisfied: typing-extensions>=4.8.0 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (4.9.0) Requirement already satisfied: sympy in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (1.12) Requirement already satisfied: networkx in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (3.2.1) Requirement already satisfied: jinja2 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (3.1.3) Requirement already satisfied: fsspec in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (2023.10.0) Requirement already satisfied: setuptools in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (72.1.0) Requirement already satisfied: MarkupSafe>=2.0 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from jinja2->torch) (2.1.4) Requirement already satisfied: mpmath>=0.19 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from sympy->torch) (1.3.0)
`D:\Applfich\SAM2\segment-anything-2>python
Python 3.12.4 (tags/v3.12.4:8e8a4ba, Jun  6 2024, 19:30:16) [MSC v.1940 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import torch
if torch.cuda.is_available():
...     print("CUDA is available. PyTorch can use the GPU.")
...     device = torch.device("cuda")
...     device_name = torch.cuda.get_device_name(device)
...     print(f"Using GPU: {device_name}")
... else:
...     print("CUDA is not available. PyTorch will use the CPU.")
...     device = torch.device("cpu")
...
CUDA is available. PyTorch can use the GPU.
Using GPU: NVIDIA RTX A2000 Laptop GPU
Example tensor on CPU
tensor = torch.randn(3, 3)
print(f"Tensor is on device: {tensor.device}")
Tensor is on device: cpu
Move tensor to GPU if available
tensor_gpu = tensor.to(device)
print(f"Tensor is now on device: {tensor_gpu.device}")
Tensor is now on device: cuda:0
`

Hi, this is likely due to an error in building the SAM 2 CUDA extension. We have recently made the CUDA extension step optional (in https://github.com/facebookresearch/segment-anything-2/pull/155 and https://github.com/facebookresearch/segment-anything-2/pull/175) as a workaround to this problem.
You can pull the latest code and reinstall via
# run the line below inside the SAM 2 repo
git pull;
pip uninstall -y SAM-2;
rm -f sam2/*.so;
pip install -e ".[demo]"

which allows using SAM 2 without CUDA extension (the results should stay the same in most cases, see INSTALL.md for details).

I'm having the same problem as you, so can you solve it?

Buy the way NumPy IS installed the submodule that crash refer as numpy not found and I am on windows CUDA driver for GPU is Installed and CUDA 12.6 toolkit also. Pytorch found the device and GPU capable.
Failed to initialize NumPy: No module named 'numpy'
D:\Applfich>pip install numpy
Looking in indexes: https://artifactory.hydro.qc.ca/artifactory/api/pypi/global-pypi-virtual/simple
Requirement already satisfied: numpy in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (1.26.4)
And the other requirement also meet Pytorch
D:\Applfich>pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
Looking in indexes: https://download.pytorch.org/whl/cu124
Requirement already satisfied: torch in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (2.4.0+cu124)
Requirement already satisfied: torchvision in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (0.19.0)
Requirement already satisfied: torchaudio in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (2.4.0)
Requirement already satisfied: filelock in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (3.13.1)
Requirement already satisfied: typing-extensions>=4.8.0 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (4.9.0)
Requirement already satisfied: sympy in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (1.12)
Requirement already satisfied: networkx in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (3.2.1)
Requirement already satisfied: jinja2 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (3.1.3)
Requirement already satisfied: fsspec in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (2023.10.0)
Requirement already satisfied: setuptools in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torch) (72.1.0)
Requirement already satisfied: numpy<2 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torchvision) (1.26.4)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from torchvision) (10.2.0)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from jinja2->torch) (2.1.4)
Requirement already satisfied: mpmath>=0.19 in c:\users\dd1968\appdata\local\programs\python\python312\lib\site-packages (from sympy->torch) (1.3.0)

Hi @mcousine @fffionayou It seems that the CUDA extension failed to build on Windows. We have recently made it optional to build the SAM 2 CUDA extension. Could you try pulling the latest code and reinstall via
# run the line below inside the SAM 2 repo
git pull;
pip uninstall -y SAM-2;
rm -f sam2/*.so;
pip install -e ".[demo]"

which allows using SAM 2 without CUDA extension (the results should stay the same in most cases, see INSTALL.md for details).

Same, and my environment is torch=2.3.1 cuda=12.1, the output show that it installs torch=2.4.0 in the venv

ch=2.3.1 cuda=12.1

Exactly the same situation. Have you solved this problem by now, @YachiZhang ? Thanks.

authors: YachiZhang fffionayou jiawei151 mcousine ronghanghu


text:
Can anybody share the dataset?

Hi, is there anybody can share the dataset?  It seems the server is down. I can not download it. I do not need the whole dataset but just the .xml files in ground truth. It should be within several hundreds MB.

We are aware that the SUMO challenge data is not available right now, but we cannot comment as to the reason at this time.

repo: facebookresearch/sumo-challenge


text:
Size class label and semantic class label

Hi,
According to the codes below, it seems that size regression is coupled with semantic classification. The size class label is the same as the semantic class label. I guess this is under the assumption that templates (average size) can provide a strong proposal.
I just wanna check if I misunderstand something. Please correct me if I'm wrong.
https://github.com/facebookresearch/votenet/blob/257b8d8369511c6b4e0cea9c33ef0da01dd10f77/sunrgbd/sunrgbd_detection_dataset.py#L169
https://github.com/facebookresearch/votenet/blob/257b8d8369511c6b4e0cea9c33ef0da01dd10f77/sunrgbd/model_util_sunrgbd.py#L38-L42
Best,
Jianyuan

HI @jytime
Your understanding is right. We assumed that the size class is the same as the object class. It could be wrong in some cases (like a huge table, whose size may be closer to the bed template) but is in general okay. You can also decouple this relationship and experiment with more size templates. Let us know if it helps significantly!

Hi,
Currently, I have not found a significant boost. I will update this if I find it later.
Best

repo: facebookresearch/votenet


text:
fix: missing return to top link

Done when:

[ ] #76

This isn't ready, returning to backlog.

repo: facioquo/uswds-design-patterns


text:
🛑 POST to /claim_detection is down

In dd720fd, POST to /claim_detection (https://dev.factiverse.no/v1/claim_detection) was down:

HTTP code: 502
Response time: 162 ms

Resolved: POST to /claim_detection is back up in 33564d6.

repo: factiverse/upptime-dev


text:
IP Address Sorting Result Bug

Describe the bug
I got an issue on your subping IP Address Result that not sorted in a correct way as shown below :

Expected behavior
It Should be appeard in a sequence of IP Address based on the IP Ranges.
please notice this if you find the same issue as mine

Thank you for bringing up this issue in my GitHub project.
I sincerely appreciate your effort in identifying and highlighting it. I kindly request your assistance in resolving this matter, and I would greatly appreciate it if you could lend a hand in fixing this. 😁

Alright, I already came up with the solution for this bug :

Description
So the problem with this bug is that the original sorting you implement on the code is based on String which means by lexicographic order as a string of an IP Address was read from left to right.
Solution
on the code, I change the extracted slice into a bytes of IP Address, so it came up with the sorting of the slice with an Insertion Sorting Algorithm, then after that, I change the byte into a String on the for-each loop

I'll create the Pull Request of the solution

authors: fadhilyori izzatbey

repo: fadhilyori/subping


text:
Find typos and correct

I have found typos on the website and corrected them.

I think a lot of the files in this pull request haven’t actually changed.  Can you please re-submit a new pull request with only the changed files?

Hi Shamal,
I will make a new pull request just for changed files.
Best regards,
Khang

From: Shamal Faily notifications@github.com
Sent: Monday, March 26, 2018 1:57:09 PM
To: failys/cairis
Cc: Huynh Le (s4830295); Author
Subject: Re: [failys/cairis] Find typos and correct (#195)
I think a lot of the files in this pull request haven’t actually changed. Can you please re-submit a new pull request with only the changed files?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/failys/cairis/pull/195#issuecomment-376156060, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AVxyEwwaG_Py-M0sAP9uguZz67Cv3c3bks5tiOWlgaJpZM4S7Ae3.

authors: failys khanghuynhle

repo: failys/cairis


text:
feature: Show spinner when login is submitted

Feature Request
Describe the Feature Request

Show a spinner while the login is being done
Is your feature request related to a problem? Please describe

Describe Preferred Solution

Angular material spinner driven by a boolean flag in the component/suscription.
Additional Context

Done byusing keycloak login thems

authors: mariorguezari miquinti


text:
Simplified and deduplicated the parsing code a bit

Commit messages contain some explanations, feel free to drop some or all of them

Thanks for the pr. bpaf is a nice crate and I appreciate your work on it.

That was fast :)
Any feedback would be nice - what works, what doesn't, what are the main issues you had when using it. Maybe drop a comment here: https://github.com/pacak/bpaf/issues/50

authors: fakeshadow pacak

repo: fakeshadow/dns-proxy


text:
docs: Add a warning to the Request.path docstring re url encoding

If the original path requested by the client was url-encoded, it will be decoded before being returned by Request's path attribute. Therefore, if this attribute is to be used by the app for any upstream requests (say, when implementing a proxy), any non-url-safe characters in the path must be url-encoded back, before making the request. Not all HTTP client libraries handle this automatically.

Hello! Was looking to contribute to this issue, but I believe it's already been addressed here. Should this issue be closed?

Indeed, this has already been tackled by @brooksryba in https://github.com/falconry/falcon/pull/1374 !

repo: falconry/falcon


text:
working backwards from images.py example

Hi there, I'm working backwards on images.py example from
http://falcon.readthedocs.org/en/latest/user/tutorial.html#creating-resources
and I'm writting my own REST resource
POST http://uri/message/ -- to create a message
GET http://uri/message/{id} -- to retreive a message. Here's my code
app.py
import falcon
import messages

#POST /message
#params
# subject
# body
#return message id

#GET /message/{id}
#return message status

api = falcon.API()

message_collection = messages.MessageCollection
message = messages.Message

api.add_route('/message', message_collection())
api.add_route('/message/{id}', message)

messages.py
import json

class MessageCollection(object):
    def on_post(self, req, resp):
        """Handles POST requests"""
        quote = {
            'response' : 'success'
        }

        resp.body = json.dumps(quote)

class Message(object):
    def on_get(self, req, resp, id):
        """Handles GET requests"""
        quote = {
            'status' : 'queued'
        }

        resp.body = json.dumps(quote)

I'm getting this error when I run gunicorn and make a request to http://localhost:8000/messages/123
[2015-08-03 10:25:17 -0700] [25151] [ERROR] Error handling request
Traceback (most recent call last):
  File "/Users/bafshin/workspace/expeditions/falcon-spike/venv/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 130, in handle
    self.handle_request(listener, req, client, addr)
  File "/Users/bafshin/workspace/expeditions/falcon-spike/venv/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 171, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/Users/bafshin/workspace/expeditions/falcon-spike/venv/lib/python2.7/site-packages/falcon/api.py", line 182, in __call__
    responder(req, resp, **params)
TypeError: unbound method on_get() must be called with Message instance as first argument (got Request instance instead)

When I google, there's no apparent distinct result of what's going on, allot of the results on top poing to the WSGI module, and that's shared across more frameworks than just Falcon. Help?

This is your problem:
api.add_route('/message/{id}', message)

You have to instantiate the classes before passing them to add_route:
api.add_route('/message/{id}', messages.Message())

(You also don't have to assign them to temporary variables.)

wow. thanks. you nailed the problem.

authors: BardiaAfshin swizzard


text:
fix(uri): change encode to escape percent

Summary of Changes
Changed the default behavior of uri.encode and uri.envode_value to escape all escapable characters even if they look like they've already been escaped. For example, '%26' would be escaped as '%2526' to conform with behavior of urllib.parse.quote. This ensures that decoding an encoded value returns the same original value. Previously, encoding '%26' returned '%26' which, decoded, become '&'.
A check_is_escaped flag has been added to the encoder method to allow the ability to retain the original behavior of ignoring previously escaped values (currently being used in location, content_location, and append_link()).
Related Issues
Address bug in #1872 while retaining fix for #68.
Pull Request Checklist
This is just a reminder about the most common mistakes.  Please make sure that you tick all appropriate boxes.  But please read our contribution guide at least once; it will save you a few review cycles!
If an item doesn't apply to your pull request, check it anyway to make it apparent that there's nothing to do.

[X] Applied changes to both WSGI and ASGI code paths and interfaces (where applicable).
[X] Added tests for changed code.
[X] Prefixed code comments with GitHub nick and an appropriate prefix.
[X] Coding style is consistent with the rest of the framework.
[X] Updated documentation for changed code.

[X] Added docstrings for any new classes, functions, or modules.
[X] Updated docstrings for any modifications to existing code.
[X] Updated both WSGI and ASGI docs (where applicable).
[X] Added references to new classes, functions, or modules to the relevant RST file under docs/.
[X] Updated all relevant supporting documentation files under docs/.
[X] A copyright notice is included at the top of any new modules (using your own name or the name of your organization).
[X] Changed/added classes/methods/functions have appropriate versionadded, versionchanged, or deprecated directives.

[X] Changes (and possible deprecations) have towncrier news fragments under docs/_newsfragments/, with the file name format {issue_number}.{fragment_type}.rst. (Run towncrier --draft to ensure it renders correctly.)

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!
PR template inspired by the attrs project.

@MinesJA Thanks again for your help on this. We would like to include this fix in 3.0.0rc1 if possible. Do you have time to address the feedback items today? If not, no problem, just let me know and I can put up some patches on your feature branch.

n for your help on this. We would like to include this fix in 3.0.0rc1 if possible. Do you have time to address the feedback it

For sure! I'll push something up in the next couple hours if that's ok.

Ready for review @kgriffs @vytas7

authors: MinesJA kgriffs


text:
chore: remove pinning of importlib-resources for Mailman tests

We had to pin importlib-resources for some CI gates because its very aggressive deprecation policy.
Now the dependency in question has been dropped upstream, so we don't need to pin anymore (in fact, pinning breaks the gate):  https://gitlab.com/mailman/mailman/-/merge_requests/1130.

Apparently it is still broken, maybe the Docker image needs to be updated :thinking: I'll need to check with @maxking.

Closing, it is not as simple as that unfortunately.


text:
Document image.pullSecrets input required format in charts/falco/README.md

What to document
The Documentation for image.pullSecrets  in charts/falco/README.md can it please include a reference to needing the image pull secret in the form of [{name: <pullsecret>}] and not [<pullsecret>] this will save a bunch of time for someone else in the future.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

authors: Aurelian-Shuttleworth poiana

repo: falcosecurity/charts


text:
vote: Applying for Maintainer

What type of PR is this?
/kind documentation
Any specific area of the project related to this PR?
/area docs
What this PR does / why we need it:
As proposed in #256, I'm quite exited to propose myself as Driverkit Maintainers, mainly to continue to support and improve the Kubernetes part in addition of improving automated usability, reviews MR and more if needed.
Whatever the result is, I really appreciate the proposal, it's always a pleasure to get our work recognize!

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Lowaiz
Once this PR has been reviewed and has the lgtm label, please assign dwindsor for approval. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the falcosecurity org. You can then trigger verification by writing /verify-owners in a comment.

lowaiz

User is not a member of the org. Satisfy at least one of these conditions to make the user trusted.

/hold to host the vote!

Thank you for this! You are very active and have great proficiency on a large chunk of driverkit.
For me, welcome aboard! :D
:+1:

+1 NB

+1 from me 🙂

/cc @falcosecurity/driverkit-maintainers

+1

So, we reached majority!
@Lowaiz welcome!
Please, make sure to add yourself in the test-infra repo. You need to add your gh handle to config/org.yaml under:

members
driverkit-maintainers

sections. See for example https://github.com/falcosecurity/test-infra/commit/56a5878f1439ee531c06d8c77e4f9ace2b595e27 :)

Hey everyone,
Thank you for all these votes and the acceptation.
I'm going the PRs for evolution and test-infra repo right now.

LGTM label has been added.  Git tree hash: 7d40aa2ceafa14433496cc8288e7d432550a3e59

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, Lowaiz
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/verify-owners

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the falcosecurity org. You can then trigger verification by writing /verify-owners in a comment.

lowaiz

User is not a member of the org. Satisfy at least one of these conditions to make the user trusted.

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the falcosecurity org. You can then trigger verification by writing /verify-owners in a comment.

lowaiz

User is not a member of the org. Satisfy at least one of these conditions to make the user trusted.

@Lowaiz feel free to re-trigger verify-owners once you accept the invite to the falcosecurity org :)

/verify-owners

@FedeDP It seems ok !

/unhold

Yay! @Lowaiz you can now open the second PR on test-infra ;)

authors: EXONER4TED FedeDP Lowaiz dwindsor leogr poiana

repo: falcosecurity/driverkit


text:
error opening device /host/dev/falco0 for aws ECS

I am trying to run the falco container in ECS and it fails below
command : docker run -i -t --name falco --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro  falcosecurity/falco:latest
Error Message:
Status: Downloaded newer image for falcosecurity/falco:latest
* Setting up /usr/src links from host
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module
* Running dkms build failed, couldn't find /var/lib/dkms/falco/96bd9bc560f67742738eb7255aeb4d03046b8045/build/make.log
* Trying to load a system falco driver, if present
* Trying to find locally a prebuilt falco module for kernel 4.14.173-137.229.amzn2.x86_64, if present
Detected an unsupported target system, please get in touch with the Falco community
2020-05-29T17:27:28+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2020-05-29T17:27:28+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2020-05-29T17:27:28+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2020-05-29T17:27:28+0000: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
2020-05-29T17:27:29+0000: Unable to load the driver. Exiting.
2020-05-29T17:27:29+0000: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

OS Detail:
[ec2-user@ip-172-xxxx-xxx-xxx ~]$ cat /etc/system-release
Amazon Linux release 2 (Karoo)
[ec2-user@ip-172-xxxx-xxx-xxx  ~]$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Seems to be them same issue: https://github.com/falcosecurity/falco/issues/1239

figured it out it look like it failed becasued I did not add -v /etc:/host/etc:ro. Once this was added it started running

How you resolved this issue?

authors: axsc sathya0803 tkumark

repo: falcosecurity/falco


text:
K8s env, best practice, container.info over container.id

What type of PR is this?
/kind rule-update
/area rules
What this PR does / why we need it:
In my case, I log to ES, I want all informations about running pods available within the alert/audit record. Output field of rule, frequently miss important informations (ie, namespace). container.info macro instead can provide bunch of usefull details if k8s is configured (otherwise it has fallback).
It is then better, to log all details and have them either available on alerts.
(Here I miss some clarification, havent found it documented, but it seems to me, that only details about an event, that are part of falcosidekick Outputfields (https://github.com/falcosecurity/falcosidekick/blob/master/outputs/alertmanager.go#L25) are the ones key=value between "()" of the output field of rule. Sorry I am lazy to find it again in src code - it might be bit different, but the truth is that alert does not have all event details.
Which issue(s) this PR fixes:
n/a
Special notes for your reviewer:
racionale here is as I cant get more details in alert, I do have need use flexible %container.info macro to give me all important (namespace, podname, ...)
Does this PR introduce a user-facing change?:
rule(Disallowed SSH Connection): use container.info in output fields
rule(Unexpected outbound connection destination): use container.info in output fields
rule(Unexpected inbound connection source): use container.info in output fields
rule(Modify Shell Configuration File): use container.info in output fields
rule(Read Shell Configuration File): use container.info in output fields
rule(Schedule Cron Jobs): use container.info in output fields
rule(Update Package Repository): use container.info in output fields
rule(Read ssh information): use container.info in output fields
rule(Write below etc): use container.info in output fields
rule(Read sensitive file trusted after startup): use container.info in output fields
rule(Write below rpm database): use container.info in output fields
rule(Modify binary dirs): use container.info in output fields
rule(Mkdir binary dirs): use container.info in output fields
rule(System user interactive): use container.info in output fields
rule(Terminal shell in container): use container.info in output fields
rule(System procs network activity): use container.info in output fields
rule(Program run with disallowed http proxy env): use container.info in output fields
rule(Interpreted procs inbound network activity): use container.info in output fields
rule(Interpreted procs outbound network activity): use container.info in output fields
rule(Unexpected UDP Traffic): use container.info in output fields
rule(User mgmt binaries): use container.info in output fields
rule(Create files below dev): use container.info in output fields
rule(Unexpected K8s NodePort Connection): use container.info in output fields
rule(Netcat Remote Code Execution in Container): use container.info in output fields
rule(Launch Suspicious Network Tool in Container): use container.info in output fields
rule(Launch Suspicious Network Tool on Host): use container.info in output fields
rule(Search Private Keys or Passwords): use container.info in output fields
rule(Clear Log Activities): use container.info in output fields
rule(Remove Bulk Data from Disk): use container.info in output fields
rule(Create Hidden Files or Directories): use container.info in output fields
rule(Launch Remote File Copy Tools in Container): use container.info in output fields
rule(Create Symlink Over Sensitive Files): use container.info in output fields
rule(Detect outbound connections to common miner pool ports): use container.info in output fields
rule(Detect crypto miners using the Stratum protocol): use container.info in output fields
rule(Packet socket created in container): use container.info in output fields
rule(Network Connection outside Local Subnet): use container.info in output fields
rule(Redirect STDOUT/STDIN to Network Connection in Container): use container.info in output fields

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

55c38e5 K8s, best practice, log container.info

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign mstemm
You can assign the PR to them by writing /assign @mstemm in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

rules/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Welcome @epcim! It looks like this is your first PR to falcosecurity/falco 🎉

To change the rules, I have used these few SED cmds..https://github.com/epcim/falco/blob/container-info-dockerfile/docker/falco/Dockerfile#L113 (the targed branch is example, I do have different build pipeline)

LGTM label has been added.  Git tree hash: 09a0428db9f9efc78c828eecd9b3d55f61e6b5b5

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Thansk @epcim for sending this.
Could you please adjust the release-note block as per contributing guidelines (it should theoretically list all the rules when you made changes) and rebase this on top of the master?
Anyways, I tend to agree with your changes but there's at least a point that we need to take into consideration.
When container.info is not available the output will contain its fallback. Is such fallback something like what we have now (eg., container_id=<ID>)?
Could you provide a fallback example, please?

ok will rebase
technically it possibly touch many rules, so release-note this time needs to be "generic statement" but I will list few If catch them.
as far as I remember, container.info is makro has container_id=<ID> as fallback

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign leogr
You can assign the PR to them by writing /assign @leogr in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

rules/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign leogr
You can assign the PR to them by writing /assign @leogr in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

rules/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@leodido so I have rebased + updated commit message.
Each rule change must be on its own commit...
For obvious reason I want to avoid split this into series of 37 commits.
Modified rules are:

Disallowed SSH Connection
Unexpected outbound connection destination
Unexpected inbound connection source
Modify Shell Configuration File
Read Shell Configuration File
Schedule Cron Jobs
Update Package Repository
Read ssh information
Write below etc
Read sensitive file trusted after startup
Write below rpm database
Modify binary dirs
Mkdir binary dirs
System user interactive
Terminal shell in container
System procs network activity
Program run with disallowed http proxy env
Interpreted procs inbound network activity
Interpreted procs outbound network activity
Unexpected UDP Traffic
User mgmt binaries
Create files below dev
Unexpected K8s NodePort Connection
Netcat Remote Code Execution in Container
Launch Suspicious Network Tool in Container
Launch Suspicious Network Tool on Host
Search Private Keys or Passwords
Clear Log Activities
Remove Bulk Data from Disk
Create Hidden Files or Directories
Launch Remote File Copy Tools in Container
Create Symlink Over Sensitive Files
Detect outbound connections to common miner pool ports
Detect crypto miners using the Stratum protocol
Packet socket created in container
Network Connection outside Local Subnet
Redirect STDOUT/STDIN to Network Connection in Container

@leodido all ok?

I think image=%container.image.repository will be redundant if we have container.info
I have some concerns over replacing an output field, as it may breaks the consumer code who may parse by container.id

@epcim I was referring to put them in the release-note block only in this case, not every rule change in its own commit clearly 😊
I'm okay with the changes. But I also get the @Kaizhe point.
It's definitely something to discuss in the community call!
Thanks for your awesome contribution!

LGTM label has been added.  Git tree hash: 732888e29226bb7053c578b1ca574df7959330ab

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leodido
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

rules/OWNERS [leodido]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Putting this on hold until we have two positive reviews, feel free to hold cancel once this is ready. I need to unblock the merge queue.
/hold

Two questions from @ldegio  (during our Falco Community Calls) today:

why we can't just retain container_id=%container.id field and just append %container.info after it?
what is the typical format of a %container.info? does it contains a JSON object, tuples, what?

Because maybe, disregarding eventual container ID duplication, this could be the way to go to also take care of the @Kaizhe's legit concerns.

Two questions from @ldegio (during our Falco Community Calls) today:

why we can't just retain container_id=%container.id field and just append %container.info after it?
what is the typical format of a %container.info? does it contains a JSON object, tuples, what?

Because maybe, disregarding eventual container ID duplication, this could be the way to go to also take care of the @Kaizhe's legit concerns.

Appending container.info in the output looks good to me

container.info

vs container.id vs. id=...
So do we need any other default than? https://github.com/falcosecurity/falco/blob/3693b16c917d6920adffac03e80d38c1b64317ea/userspace/engine/lua/rule_loader.lua#L694

I have some concerns over replacing an output field, as it may breaks the consumer code who may parse by container.id

As long as container.info has fallback to container.id and as long as it containse container.id, thats not an issue:

container.info

vs container.id vs. id=...
So do we need any other default than?
https://github.com/falcosecurity/falco/blob/3693b16c917d6920adffac03e80d38c1b64317ea/userspace/engine/lua/rule_loader.lua#L694

I have some concerns over replacing an output field, as it may breaks the consumer code who may parse by container.id

As long as container.info has fallback to container.id and as long as it containse container.id, thats not an issue:

Can you confirm I can use the same json pointer(I don't need to change anything, e.g. pointer name, level) to get the container ID if you use container.info?

container.info

vs container.id vs. id=...
So do we need any other default than?
https://github.com/falcosecurity/falco/blob/3693b16c917d6920adffac03e80d38c1b64317ea/userspace/engine/lua/rule_loader.lua#L694

I have some concerns over replacing an output field, as it may breaks the consumer code who may parse by container.id

As long as container.info has fallback to container.id and as long as it contains container.id, thats not an issue:

Can you confirm I can use the same json pointer(I don't need to change anything, e.g. pointer name, level) to get the container ID if you use container.info?

Seems like, but somebody else better to confirm. I am not Falco developer, just contributor. Macro container.info need some ownership and as you suggested it req. to be backward compatible, future support and notify existing users early if any change in behaviour.
Also image=%container.image.repository will be redundant is still not fixed. Having some filter that would uniq. filter keys of extra_ would be handy.

@epcim: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@epcim: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I wont do rebase anymore.On 20 Jan 2021 17:06, poiana notifications@github.com wrote:
@epcim: PR needs rebase.
Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

I wont do rebase anymore.On 20 Jan 2021 17:06, poiana notifications@github.com wrote:
@epcim: PR needs rebase.
Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

/milestone 0.28.1

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

@poiana: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

authors: Kaizhe epcim fntlnz leodido poiana


text:
🛑 Archive is down

In 8db1a52, Archive (https://archive.stsci.edu) was down:

HTTP code: 404
Response time: 11208 ms

Resolved: Archive is back up in 3ebbc66 after 8 minutes.

repo: falkben/mast-status


text:
Is this a react native library?

Is this a react native library if yes, how can i use it in my android+ ios app..if not, than what is it nd how can i covert it into react native..Thanks in advance✌

https://github.com/tom555my/react-native-avataaars

authors: aaravchaudhary rafaelvinicius

repo: fangpenlin/avataaars


text:
chore: add peerid @bhgomes

Motivation
Add a peerid for farcaster mainnet hubs.
Change Summary

Added a new allowed peer to the allowedPeers.mainnet.ts file in the apps/hubble/src directory.
The new peer has the following address: 12D3KooWNLshuXk4x2gacFTMEJjMGGFxPjpHg5KcV88KoqJP7dDR.
No other changes were made.

Merge Checklist
Choose all relevant options below by adding an x now or at any time before submitting for review

[x] PR title adheres to the conventional commits standard
[ ] PR has a changeset
[x] PR has been tagged with a change label(s) (i.e. documentation, feature, bugfix, or chore)
[x] PR does not require changes to the protocol
[x] All commits have been signed

PR-Codex overview
This PR adds a new allowed peer to the list in allowedPeers.mainnet.ts.
Detailed summary

Added a new allowed peer to the list in allowedPeers.mainnet.ts
The new peer's public key is 12D3KooWNLshuXk4x2gacFTMEJjMGGFxPjpHg5KcV88KoqJP7dDR
The peer is identified as @bhgomes

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

this looks good to go, you'll just need to rebase to fix the conflict @bhgomes

this looks good to go, you'll just need to rebase to fix the conflict @bhgomes

done!

authors: bhgomes varunsrin

repo: farcasterxyz/hub-monorepo


text:
feat: one custody address per fid

Motivation
Implements https://github.com/farcasterxyz/hub/issues/87
Change Summary

Update SignerSet to store a single custodyEvent that custodyAddress is derived from
Store an adds and removes set of delegate signers for each custody address
Handle conflicts between signer messages from the same custody address using message timestamp, type (i.e. remove >add), and hash

Merge Checklist
Choose all relevant options below by adding an x now or at any time before submitting for review

[x] The title of this PR adheres to the conventional commits standard
[x] The PR has been tagged with the appropriate change type label(s) (i.e. documentation, feature, bugfix, or chore)

Codecov Report

Merging #90 (8025c38) into main (f9c5cf6) will increase coverage by 0.01%.
The diff coverage is 91.17%.

@@            Coverage Diff             @@
##             main      #90      +/-   ##
==========================================
+ Coverage   89.33%   89.34%   +0.01%
==========================================
  Files          13       13
  Lines         928      854      -74
  Branches      225      203      -22
==========================================
- Hits          829      763      -66
+ Misses         93       85       -8
  Partials        6        6

Impacted Files
Coverage Δ

src/types/index.ts
100.00% <ø> (ø)

src/factories.ts
96.15% <80.00%> (-1.60%)
:arrow_down:

src/engine/index.ts
88.20% <86.66%> (+1.62%)
:arrow_up:

src/sets/signerSet.ts
93.75% <92.30%> (+0.64%)
:arrow_up:

src/client.ts
96.36% <100.00%> (-0.19%)
:arrow_down:

src/types/typeguards.ts
98.30% <100.00%> (-0.25%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter pfletcherhill

repo: farcasterxyz/hub


text:
🛑 E-solat is down

In 58f6190, E-solat (https://www.e-solat.gov.my/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: E-solat is back up in 33a9af5 after 10 minutes.


text:
Airbyte GitHub Sync Fails: Something went wrong within the destination connector

Describe the bug
I am experiencing an issue were GitHub Sync always fails. It happens at vcs_PullRequestComment stage. Other data gets synced properly and is available via Metabase and Hasura.
I tried it 5 times, sync always fails after 25 minutes to 1h.
It appears as if newly synced items violate unique constraints but I have no idea why that happens.
To Reproduce
Steps to reproduce the behavior:

Go to http://localhost:8000
Click on 'Launch' next to GitHub connections
Go to details and see failed logs

Expected behavior
Sync succeeds and data is available via Metabase and Hasura.
Screenshots and logs

0 Bytesno recordsno records22m 55sSync
Failure Origin: destination, Message: Something went wrong within the destination connector

2023-08-22 10:46:17 - Additional Failure Information: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed

I'm not able to provide unreacted logs because they contain pull request comments with AWS deployment details (from pull request comments).
023-08-22 10:45:46 destination > Error while saving batch: [{"extensions":{"internal":{"statement":"WITH \"vcs_PullRequestComment__mutation_result\" AS (INSERT INTO \"public\".\"vcs_PullRequestComment\" ( \"origin\"
 \"createdAt\"
 \"uid\"
 \"pullRequest\"
 \"author\"
 \"id\"
 \"updatedAt\"
 \"number\"
 \"comment\"
 \"refreshedAt\" ) VALUES (('github_source')::text
 ('2023-02-08T15:26:36Z')::timestamptz
 ('1100296384')::text

')::text
 DEFAULT) ON CONFLICT ON CONSTRAINT "vcs_PullRequestComment_pkey" DO UPDATE SET "origin" = EXCLUDED."origin"
"createdAt" = EXCLUDED."createdAt"
"author" = EXCLUDED."author"
"updatedAt" = EXCLUDED."updatedAt"
"number" = EXCLUDED."number"
"comment" = EXCLUDED."comment" WHERE (('true') AND ('true')) RETURNING *
 ('true')::boolean AS "check__constraint") SELECT  json_build_object('affected_rows'
 (SELECT  COUNT(*)  FROM "vcs_PullRequestComment__mutation_result"      )
 'returning_columns'
 (SELECT  coalesce(json_agg("root" )
 '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_1_e"  FROM  (SELECT  ("_0_root.base"."origin")::text AS "origin"
 ("_0_root.base"."createdAt")::text AS "createdAt"
 ("_0_root.base"."uid")::text AS "uid"
 ("_0_root.base"."pullRequest")::text AS "pullRequest"
 ("_0_root.base"."author")::text AS "author"
 ("_0_root.base"."id")::text AS "id"
 ("_0_root.base"."updatedAt")::text AS "updatedAt"
 ("_0_root.base"."number")::text AS "number"
 ("_0_root.base"."comment")::text AS "comment"
 ("_0_root.base"."refreshedAt")::text AS "refreshedAt"       ) AS "_1_e"      ) ) AS "root" FROM  (SELECT  *  FROM "vcs_PullRequestComment__mutation_result" WHERE ('true')     ) AS "_0_root.base"      ) AS "_2_root"      ) )
 (SELECT  coalesce(bool_and("check__constraint" )
 'true' )  FROM "vcs_PullRequestComment__mutation_result"      )        \"
\"prepared\":false
\"error\":{\"exec_status\":\"FatalError\"
\"hint\":null
\"message\":\"index row requires 9352 bytes
 maximum size is 8191\"
\"status_code\":\"54000\"
\"description\":null}
\"arguments\":[]}
\"path\":\"$.selectionSet.insert_vcs_PullRequestComment_one.args.object[0]\"
\"code\":\"unexpected\"}
\"message\":\"database query error\"}] - {\"jse_shortmsg\":\"Failed to write vcs_PullRequestComment record with query 'mutation { insert_vcs_PullRequestComment_one (object: {number: 1100296384
 uid: "1100296384"
 comment: "

2023-08-22 10:47:17 WARN i.a.c.i.LineGobbler(voidCall):119 - airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
2023-08-22 10:47:17 ERROR i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:181) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
	Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.
		at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at java.lang.Thread.run(Thread.java:833) ~[?:?]
	Suppressed: java.io.IOException: Stream closed
		at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:445) ~[?:?]
		at java.io.OutputStream.write(OutputStream.java:162) ~[?:?]
		at java.io.BufferedOutputStream.write(BufferedOutputStream.java:123) ~[?:?]
		at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) ~[?:?]
		at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313) ~[?:?]
		at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:318) ~[?:?]
		at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:160) ~[?:?]
		at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:248) ~[?:?]
		at java.io.BufferedWriter.flush(BufferedWriter.java:257) ~[?:?]
		at io.airbyte.workers.internal.DefaultAirbyteDestination.notifyEndOfInput(DefaultAirbyteDestination.java:98) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:111) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more
2023-08-22 10:47:17 WARN i.a.w.g.DefaultReplicationWorker(run):220 - Could not reliably determine committed record counts, committed record stats will be set to null
2023-08-22 10:47:17 INFO i.a.w.g.DefaultReplicationWorker(run):279 - Source output at least one state message
2023-08-22 10:47:18 INFO i.a.w.g.DefaultReplicationWorker(run):285 - State capture: Updated state to: Optional[io.airbyte.config.State@38bf5de4[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"review_comments"},"stream_state":
2023-08-22 10:47:18 INFO i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: {
  "status" : "failed",
  "recordsSynced" : 35177,
  "bytesSynced" : 39556741,
  "startTime" : 1692699895332,
  "endTime" : 1692701237993,
  "totalStats" : {
    "recordsEmitted" : 35177,
    "bytesEmitted" : 39556741,
    "sourceStateMessagesEmitted" : 1739,
    "destinationStateMessagesEmitted" : 1717,
    "meanSecondsBeforeSourceStateMessageEmitted" : 3,
    "maxSecondsBeforeSourceStateMessageEmitted" : 250,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 5,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0
  },
  "streamStats" : [ {
    "streamName" : "github_source__github__pull_requests",
    "stats" : {
      "recordsEmitted" : 194,
      "bytesEmitted" : 2803965
    }
  }, {
    "streamName" : "github_source__github__collaborators",
    "stats" : {
      "recordsEmitted" : 14080,
      "bytesEmitted" : 15532594
    }
  }, {
    "streamName" : "github_source__github__repositories",
    "stats" : {
      "recordsEmitted" : 228,
      "bytesEmitted" : 1253144
    }
  }, {
    "streamName" : "github_source__github__releases",
    "stats" : {
      "recordsEmitted" : 16,
      "bytesEmitted" : 35073
    }
  }, {
    "streamName" : "github_source__github__issues",
    "stats" : {
      "recordsEmitted" : 195,
      "bytesEmitted" : 1168345
    }
  }, {
    "streamName" : "github_source__github__commits",
    "stats" : {
      "recordsEmitted" : 524,
      "bytesEmitted" : 1986803
    }
  }, {
    "streamName" : "github_source__github__review_comments",
    "stats" : {
      "recordsEmitted" : 266,
      "bytesEmitted" : 1116811
    }
  }, {
    "streamName" : "github_source__github__organizations",
    "stats" : {
      "recordsEmitted" : 1,
      "bytesEmitted" : 1775
    }
  }, {
    "streamName" : "github_source__github__branches",
    "stats" : {
      "recordsEmitted" : 2608,
      "bytesEmitted" : 774681
    }
  }, {
    "streamName" : "github_source__github__issue_labels",
    "stats" : {
      "recordsEmitted" : 2985,
      "bytesEmitted" : 826570
    }
  }, {
    "streamName" : "github_source__github__assignees",
    "stats" : {
      "recordsEmitted" : 14080,
      "bytesEmitted" : 14056980
    }
  } ]
}
2023-08-22 10:47:18 INFO i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ {
  "failureOrigin" : "destination",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed",
  "externalMessage" : "Something went wrong within the destination connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 8,
    "connector_command" : "write"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:346)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: java.io.IOException: Broken pipe\n\tat java.base/java.io.FileOutputStream.writeBytes(Native Method)\n\tat java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)\n\tat java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)\n\tat java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)\n\tat java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132)\n\tat java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:205)\n\tat java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:120)\n\tat java.base/java.io.BufferedWriter.write(BufferedWriter.java:233)\n\tat java.base/java.io.Writer.write(Writer.java:249)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.accept(DefaultAirbyteDestination.java:90)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:343)\n\t... 4 more\n",
  "timestamp" : 1692701177835
}, {
  "failureOrigin" : "destination",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1",
  "externalMessage" : "Something went wrong within the destination connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 8,
    "connector_command" : "write"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\n",
  "timestamp" : 1692701177835
} ]
2023-08-22 10:47:18 INFO i.a.c.i.LineGobbler(voidCall):114 -
2023-08-22 10:47:18 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION -----
2023-08-22 10:47:18 INFO i.a.c.i.LineGobbler(voidCall):114 -
2023-08-22 10:47:18 INFO i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling...
2023-08-22 10:47:18 INFO i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):149 - sync summary: io.airbyte.config.StandardSyncOutput@59107209[standardSyncSummary=io.airbyte.config.StandardSyncSummary@608888a8[status=failed,recordsSynced=35177,bytesSynced=39556741,startTime=1692699895332,endTime=1692701237993,totalStats=io.airbyte.config.SyncStats@16490e94[recordsEmitted=35177,bytesEmitted=39556741,sourceStateMessagesEmitted=1739,destinationStateMessagesEmitted=1717,recordsCommitted=<null>,meanSecondsBeforeSourceStateMessageEmitted=3,maxSecondsBeforeSourceStateMessageEmitted=250,maxSecondsBetweenStateMessageEmittedandCommitted=5,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@72431d5f[streamName=github_source__github__pull_requests,stats=io.airbyte.config.SyncStats@2899b924[recordsEmitted=194,bytesEmitted=2803965,sourceStateMessagesEmitted=<null>,destinationStateMessagesEmitted=<null>,recordsCommitted=<null>,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,additionalProperties={}]],
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:346)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 3 more
Caused by: java.io.IOException: Broken pipe
	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)
	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
	at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304)
	at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
	at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132)
	at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:205)
	at java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:120)
	at java.base/java.io.BufferedWriter.write(BufferedWriter.java:233)
	at java.base/java.io.Writer.write(Writer.java:249)
	at io.airbyte.workers.internal.DefaultAirbyteDestination.accept(DefaultAirbyteDestination.java:90)
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:343)
	... 4 more
,retryable=<null>,timestamp=1692701177835], io.airbyte.config.FailureReason@32058b7[failureOrigin=destination,failureType=<null>,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1,externalMessage=Something went wrong within the destination connector,metadata=io.airbyte.config.Metadata@3c77b2c0[additionalProperties={attemptNumber=0, jobId=8, connector_command=write}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 3 more
,retryable=<null>,timestamp=1692701177835]]]
2023-08-22 10:47:18 INFO i.a.w.t.TemporalUtils(withBackgroundHeartbeat):279 - Stopping temporal heartbeating...

Deployment

Local
MacOS 13.4 (22F66), MacBook Air M1
Docker version 23.0.5, build bc4487a
Docker Compose version v2.17.3

Thank you for providing us with a great tool and help with this bug!

Fixed

@horosin you can go to Airbyte (localhost:8000) > Settings > destinations and set the Faros Destination version to 0.6.11. Alternatively, just stop and start Faros, and it will pick it up.

@thomas-gerber Thank you, I really appreciate the help. Manual change helped!
@tovbinm thank you as well

authors: horosin thomas-gerber tovbinm

repo: faros-ai/faros-community-edition


text:
Enhancement request: Exclude folder and sub-folders

"Add Excluded Folder" works for the specified folder but not for nested sub-folders.  Maybe there's a way to do this with regex?

Yeah this would be super useful. I assumed it applied to all sub folders but recently have realised that it does not :/

@penalba I figured out how to do this.

Enable the "use regular expressions to check for excluded folder" option.
Use regex to define excluded folder paths. I.e. instead of:

Notes/Subject1/Topic1
Write:
Notes/Subject1/Topic1/.*
This works for me.

Works for me too -- thanks!

repo: farux/obsidian-auto-note-mover