text:
CI: Add make testp

Validates that "make testp" does not error out.

Any thoughts on whether P=1 or P=2 is preferable?

why run this for buildold?

Any thoughts on whether P=1 or P=2 is preferable?

I would say P=1 if we're just going to validate that the testp target works (to minimize the execution time).

why run this for buildold?

I picked the build with the shortest execution time, currently.

Any thoughts on whether P=1 or P=2 is preferable?

I would say P=1 if we're just going to validate that the testp target works (to minimize the execution time).

Do the tests increase in size as P increases or are they divided up into smaller sections? If the latter then maybe higher values of P will go faster than lower.

Github CI (and all others) run with 2 smt cores per VM.

I'd go for P=2 if that's the case.

2020-07-23T22:12:15.3515170Z make -C partitions testp OPTS='-optimize -production  -g -optimize -production  -g' TESTOPTS='++local'
2020-07-23T22:12:15.3536949Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/tests/charm++/partitions'
2020-07-23T22:12:15.3537825Z ../../../bin/testrun  ./hello +p2 $(( 2 * 4 )) $(( 2 / 2 )) +partitions $(( 2 / 2 ))  ++local
2020-07-23T22:12:15.3676817Z Charmrun> scalable start enabled.
2020-07-23T22:12:15.3677241Z Charmrun> started all node programs in 0.007 seconds.
2020-07-23T22:12:15.3678627Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-23T22:12:15.3678930Z Converse/Charm++ Commit ID: v6.11.0-devel-328-gf89f53f6d
2020-07-23T22:12:15.3728662Z Isomalloc> Synchronized global address space.
2020-07-23T22:12:15.3733109Z Charm++> scheduler running in netpoll mode.
2020-07-23T22:12:15.3733468Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-23T22:12:15.3746012Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-23T22:12:15.3746597Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-23T22:12:15.3748234Z hello: hello.C:26: Main::Main(CkArgMsg*): Assertion `CkNumPes() == expectedNumPes' failed.
2020-07-23T22:12:15.3749425Z ------------- Processor 0 Exiting: Caught Signal ------------
2020-07-23T22:12:15.3749968Z Reason: unknown signal
2020-07-23T22:12:15.3761655Z [0] Stack Traceback:
2020-07-23T22:12:15.3762807Z   [0:0] libc.so.6 0x7ffff6c77fd0
2020-07-23T22:12:15.3764388Z   [0:1] libc.so.6 0x7ffff6c77f47 gsignal
2020-07-23T22:12:15.3765711Z   [0:2] libc.so.6 0x7ffff6c798b1 abort
2020-07-23T22:12:15.3766707Z   [0:3] libc.so.6 0x7ffff6c6942a
2020-07-23T22:12:15.3767688Z   [0:4] libc.so.6 0x7ffff6c694a2
2020-07-23T22:12:15.3769099Z   [0:5] hello 0x5555555fdc54 CkIndex_Main::_call_Main_CkArgMsg(void*, void*)
2020-07-23T22:12:15.3770231Z   [0:6] hello 0x5555556011ae _initCharm(int, char**)
2020-07-23T22:12:15.3771318Z   [0:7] hello 0x55555570476e ConverseInit
2020-07-23T22:12:15.3772374Z   [0:8] hello 0x555555600069 charm_main
2020-07-23T22:12:15.3773467Z   [0:9] libc.so.6 0x7ffff6c5ab97 __libc_start_main
2020-07-23T22:12:15.3774927Z   [0:10] hello 0x5555555fbf6a _start
2020-07-23T22:12:15.3776744Z Fatal error on PE 0> unknown signal

2020-07-23T22:12:15.3515170Z make -C partitions testp OPTS='-optimize -production  -g -optimize -production  -g' TESTOPTS='++local'
2020-07-23T22:12:15.3536949Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/tests/charm++/partitions'
2020-07-23T22:12:15.3537825Z ../../../bin/testrun  ./hello +p2 $(( 2 * 4 )) $(( 2 / 2 )) +partitions $(( 2 / 2 ))  ++local
2020-07-23T22:12:15.3676817Z Charmrun> scalable start enabled.
2020-07-23T22:12:15.3677241Z Charmrun> started all node programs in 0.007 seconds.
2020-07-23T22:12:15.3678627Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-23T22:12:15.3678930Z Converse/Charm++ Commit ID: v6.11.0-devel-328-gf89f53f6d
2020-07-23T22:12:15.3728662Z Isomalloc> Synchronized global address space.
2020-07-23T22:12:15.3733109Z Charm++> scheduler running in netpoll mode.
2020-07-23T22:12:15.3733468Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-23T22:12:15.3746012Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-23T22:12:15.3746597Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-23T22:12:15.3748234Z hello: hello.C:26: Main::Main(CkArgMsg*): Assertion `CkNumPes() == expectedNumPes' failed.
2020-07-23T22:12:15.3749425Z ------------- Processor 0 Exiting: Caught Signal ------------
2020-07-23T22:12:15.3749968Z Reason: unknown signal
2020-07-23T22:12:15.3761655Z [0] Stack Traceback:
2020-07-23T22:12:15.3762807Z   [0:0] libc.so.6 0x7ffff6c77fd0
2020-07-23T22:12:15.3764388Z   [0:1] libc.so.6 0x7ffff6c77f47 gsignal
2020-07-23T22:12:15.3765711Z   [0:2] libc.so.6 0x7ffff6c798b1 abort
2020-07-23T22:12:15.3766707Z   [0:3] libc.so.6 0x7ffff6c6942a
2020-07-23T22:12:15.3767688Z   [0:4] libc.so.6 0x7ffff6c694a2
2020-07-23T22:12:15.3769099Z   [0:5] hello 0x5555555fdc54 CkIndex_Main::_call_Main_CkArgMsg(void*, void*)
2020-07-23T22:12:15.3770231Z   [0:6] hello 0x5555556011ae _initCharm(int, char**)
2020-07-23T22:12:15.3771318Z   [0:7] hello 0x55555570476e ConverseInit
2020-07-23T22:12:15.3772374Z   [0:8] hello 0x555555600069 charm_main
2020-07-23T22:12:15.3773467Z   [0:9] libc.so.6 0x7ffff6c5ab97 __libc_start_main
2020-07-23T22:12:15.3774927Z   [0:10] hello 0x5555555fbf6a _start
2020-07-23T22:12:15.3776744Z Fatal error on PE 0> unknown signal

I'll look into this issue.

I'll look into this issue.

Sounds good. Feel free to push to this branch.

I'll look into this issue.

Sounds good. Feel free to push to this branch.

The previously added command was incorrect breaking an assert in the test. I fixed it in the latest commit.

@nitbhat Looks like examples/ampi/Cjacobi3D is now failing due to parameters being incorrect.

@nitbhat Looks like examples/ampi/Cjacobi3D is now failing due to parameters being incorrect.

The error was because there was no testp target inside benchmarks. I fixed it in the latest commit.

2020-07-27T19:54:42.0374002Z make[1]: Leaving directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/examples'
2020-07-27T19:54:42.0374417Z make -C ../benchmarks testp
2020-07-27T19:54:42.0476437Z make[1]: *** No rule to make target 'testp'.  Stop.
2020-07-27T19:54:42.0476743Z make[1]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks'
2020-07-27T19:54:42.0480297Z make: *** [testp] Error 2
2020-07-27T19:54:42.0480709Z make[1]: Leaving directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks'

Most recent failure was:

make[3]: Leaving directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks/ampi/alltoall'
make -C isomalloc testp OPTS='-optimize -production  -g -optimize -production  -g' TESTOPTS='++local' P='2'
make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/benchmarks/ampi/isomalloc'
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  ++local
Charmrun> scalable start enabled.
Charmrun> started all node programs in 0.019 seconds.
Charm++> Running in non-SMP mode: 2 processes (PEs)
Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.001 seconds.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Random
malloc stress test: ./gnu-isomalloc (...) +vp2 randomseed 171084924
Charmrun> error on request socket to node 0 '127.0.0.1'--
Socket closed before recv.

real	1m25.849s
user	0m0.004s
sys	0m0.025s
Makefile:65: recipe for target 'testp-gnu-isomalloc' failed

The error was because there was no testp target inside benchmarks. I fixed it in the latest commit.

Ah, I see. I guess the jacobi tests don't call MPI_Abort or return a non-zero exit code when parameters are wrong, so that they fail silently. That is worth noting but we don't strictly need to address it as part of this PR.
2020-07-27T21:09:43.2531055Z make -C Cjacobi3D testp OPTS='-optimize -production  -g' TESTOPTS='++local' P='2'
2020-07-27T21:09:43.2573436Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/examples/ampi/Cjacobi3D'
2020-07-27T21:09:43.2573873Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp2 +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.2839599Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.2840044Z Charmrun> started all node programs in 0.019 seconds.
2020-07-27T21:09:43.2841033Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.2841596Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.2854173Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.2854600Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.2869814Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.2870252Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.2919470Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.2921654Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.2924153Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.2925636Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3129097Z 2 x 2 x 2 != 2
2020-07-27T21:09:43.3131717Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3140607Z
2020-07-27T21:09:43.3140955Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp$(( 2 * 2 )) +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.3141270Z real	0m0.052s
2020-07-27T21:09:43.3141416Z user	0m0.004s
2020-07-27T21:09:43.3141550Z sys	0m0.000s
2020-07-27T21:09:43.3287253Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.3288302Z Charmrun> started all node programs in 0.007 seconds.
2020-07-27T21:09:43.3289162Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.3289636Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.3310763Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.3314070Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.3314684Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.3315155Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.3342955Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.3344227Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.3345263Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.3346498Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3573805Z 2 x 2 x 2 != 4
2020-07-27T21:09:43.3839164Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3844789Z
2020-07-27T21:09:43.3848045Z real	0m0.065s
2020-07-27T21:09:43.3848502Z user	0m0.003s
2020-07-27T21:09:43.3848653Z sys	0m0.000s

Most recent failure was:
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  ++local

I'll look at this one.

The error was because there was no testp target inside benchmarks. I fixed it in the latest commit.

Ah, I see. I guess the jacobi tests don't call MPI_Abort or return a non-zero exit code when parameters are wrong, so that they fail silently. That is worth noting but we don't strictly need to address it as part of this PR.
2020-07-27T21:09:43.2531055Z make -C Cjacobi3D testp OPTS='-optimize -production  -g' TESTOPTS='++local' P='2'
2020-07-27T21:09:43.2573436Z make[3]: Entering directory '/home/runner/work/charm/charm/netlrts-linux-x86_64/examples/ampi/Cjacobi3D'
2020-07-27T21:09:43.2573873Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp2 +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.2839599Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.2840044Z Charmrun> started all node programs in 0.019 seconds.
2020-07-27T21:09:43.2841033Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.2841596Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.2854173Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.2854600Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.2869814Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.2870252Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.2919470Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.2921654Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.2924153Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.2925636Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3129097Z 2 x 2 x 2 != 2
2020-07-27T21:09:43.3131717Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3140607Z
2020-07-27T21:09:43.3140955Z ../../../bin/testrun  +p2 ./jacobi.pup 2 2 2 40 +vp$(( 2 * 2 )) +balancer RotateLB +LBDebug 1  ++local
2020-07-27T21:09:43.3141270Z real	0m0.052s
2020-07-27T21:09:43.3141416Z user	0m0.004s
2020-07-27T21:09:43.3141550Z sys	0m0.000s
2020-07-27T21:09:43.3287253Z Charmrun> scalable start enabled.
2020-07-27T21:09:43.3288302Z Charmrun> started all node programs in 0.007 seconds.
2020-07-27T21:09:43.3289162Z Charm++> Running in non-SMP mode: 2 processes (PEs)
2020-07-27T21:09:43.3289636Z Converse/Charm++ Commit ID: v6.11.0-devel-337-g4ee763eff
2020-07-27T21:09:43.3310763Z Isomalloc> Synchronized global address space.
2020-07-27T21:09:43.3314070Z Charm++> scheduler running in netpoll mode.
2020-07-27T21:09:43.3314684Z CharmLB> Verbose level 1, load balancing period: -1 seconds
2020-07-27T21:09:43.3315155Z CharmLB> Load balancer assumes all CPUs are same.
2020-07-27T21:09:43.3342955Z Charm++> Running on 1 hosts (1 sockets x 2 cores x 1 PUs = 2-way SMP)
2020-07-27T21:09:43.3344227Z Charm++> cpu topology info is gathered in 0.001 seconds.
2020-07-27T21:09:43.3345263Z [0] TreeLB in LEGACY MODE support
2020-07-27T21:09:43.3346498Z [0] TreeLB: Using PE_Root tree with strategy Rotate
2020-07-27T21:09:43.3573805Z 2 x 2 x 2 != 4
2020-07-27T21:09:43.3839164Z [Partition 0][Node 0] End of program
2020-07-27T21:09:43.3844789Z
2020-07-27T21:09:43.3848045Z real	0m0.065s
2020-07-27T21:09:43.3848502Z user	0m0.003s
2020-07-27T21:09:43.3848653Z sys	0m0.000s

Most recent failure was:
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  ++local

I'll look at this one.

I missed the Jacobi silent failure. I'll fix that.
For the gnu-isomalloc, when I locally tried it, there was an issue with CMA (https://github.com/UIUC-PPL/charm/issues/2962). But I think CMA is not enabled in the ci test environment, so this issue seems to be different.
On trying locally with +noCMAForZC, I am able to reproduce the issue.
../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  +noCMAForZC
Charmrun> scalable start enabled.
Charmrun> started all node programs in 3.976 seconds.
Charm++> Running in non-SMP mode: 2 processes (PEs)
Converse/Charm++ Commit ID: v6.11.0-devel-335-g7a48f012c
Charm++ built without optimization.
Do not use for performance benchmarking (build with --with-production to do so).
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.000 seconds.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Random
malloc stress test: ./gnu-isomalloc (...) +vp2 randomseed 398988888
Charmrun> error on request socket to node 1 '127.0.0.1'--
Socket closed before recv.

real	3m57.642s
user	0m0.027s
sys	0m0.013s
Makefile:65: recipe for target 'test-gnu-isomalloc' failed
make: *** [test-gnu-isomalloc] Error 1

I think the test takes really long because it uses the copy based approach for transferring (and maybe charmrun times out with the usual socket closed for recv error).

../../../bin/testrun  ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB  +noCMAForZC ++timeout 500
Even with a large timeout, I see this error. So, I'm wondering if this is a segfault.

../../../bin/testrun ./gnu-isomalloc +p2 +vp2 +balancer RandCentLB +noCMAForZC ++timeout 500
Even with a large timeout, I see this error. So, I'm wondering if this is a segfault.

Autobuild is blown up with crashes in the isomalloc benchmark and they look like segfaults.

Running locally reveals the issue: Charm++> [1] Isomalloc tried to mmap(0x2aaaa8000000, 1832706048), but encountered error 12, which means ENOMEM. In a two-process, two-rank run, each process consumes more than 10 GB of memory and eventually uses all of the machine's RAM and swap space.
I'm not sure why this would be the case, because IIRC I designed this benchmark to only use about 1 GB of memory total across the entire distributed job.

@nitbhat I've found that there is a memory leak in the pup_buffer API, at least with +noCMAForZC. Here is a unit test case that demonstrates the problem by migrating in a loop with a large stack size. If you observe memory usage in a tool like htop it will continually increase with each migration until the system runs out of space. If the test calls AMPI_Register_pup to disable pup_buffer, the leak does not occur and memory usage stays constant.
// ampiCC simple-migrate.cpp -o simple-migrate -module TreeLB -memory default -g
// ./charmrun ./simple-migrate +p2 +vp2 ++local +balancer RotateLB +noCMAForZC +tcharm_stacksize 256M
// Then, observe memory usage in `htop`.
// The bug does not occur when compiled with -DTEST_PUP, which disables pup_buffer.

// With valgrind:
// ./charmrun `which valgrind` --leak-check=full --show-leak-kinds=all -- ./simple-migrate +p2 +vp2 ++local +balancer RotateLB +tcharm_stacksize 256M +noCMAForZC ++in-xterm

#include <stdio.h>
#include "mpi.h"

#ifdef TEST_PUP
void null_pup(pup_er p, void *d)
{
}
#endif

int main(int argc, char** argv)
{
  MPI_Init(&argc, &argv);

  int rank, flag;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

#ifdef TEST_PUP
  void * ptr;
  int idx;
  AMPI_Register_pup(null_pup, &ptr, &idx);
#endif

  for (int i = 0; i < 1000; ++i)
  {
    {
      int wth, process;
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_WTH, &wth, &flag);
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_PROCESS, &process, &flag);
      printf("[%d](%d)(%d) Migrating...\n", rank, process, wth);
    }

    AMPI_Migrate(AMPI_INFO_LB_SYNC);

    {
      int wth, process;
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_WTH, &wth, &flag);
      MPI_Comm_get_attr(MPI_COMM_WORLD, AMPI_MY_PROCESS, &process, &flag);
      printf("[%d](%d)(%d) Migrated.\n", rank, process, wth);
    }
  }

  MPI_Finalize();

  return 0;
}

With the valgrind command and the loop changed to 10 iterations I get the following:
==20526== 1,671,544 bytes in 29,849 blocks are still reachable in loss record 631 of 633
==20526==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20526==    by 0x5BEA3A: EnqueueOutgoingDgram(OutgoingMsgStruct*, char*, int, OtherNodeStruct*, int, int) (machine-eth.C:312)
==20526==    by 0x5BEBD1: DeliverViaNetwork(OutgoingMsgStruct*, OtherNodeStruct*, int, unsigned int, int) (machine-eth.C:352)
==20526==    by 0x5BFDC6: DeliverOutgoingMessage(OutgoingMsgStruct*) (machine.C:1640)
==20526==    by 0x5BFE93: LrtsSendFunc(int, int, int, char*, int) (machine.C:1688)
==20526==    by 0x5B9927: CmiInterSendNetworkFunc(int, int, int, char*, int) (machine-common-core.C:692)
==20526==    by 0x5B99E2: CmiInterFreeSendFn (machine-common-core.C:729)
==20526==    by 0x5B9959: CmiFreeSendFn (machine-common-core.C:698)
==20526==    by 0x5C234B: CmiIssueRput(ncpystruct*) (conv-rdma.C:119)
==20526==    by 0x5C2153: getRequestHandler(_converseRdmaMsg*) (conv-rdma.C:50)
==20526==    by 0x5C585C: CmiHandleMessage (convcore.C:1702)
==20526==    by 0x5C5B13: CsdScheduleForever (convcore.C:1953)
==20526==
==20526== 1,342,526,280 bytes in 5 blocks are possibly lost in loss record 632 of 633
==20526==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20526==    by 0x5B68E8: malloc_nomigrate (memory.C:1083)
==20526==    by 0x5C8CD1: CmiAlloc (convcore.C:3209)
==20526==    by 0x5BECE3: AssembleDatagram(OtherNodeStruct*, ExplicitDgramStruct*) (machine-eth.C:392)
==20526==    by 0x5BEF1E: AssembleReceivedDatagrams(OtherNodeStruct*) (machine-eth.C:443)
==20526==    by 0x5BF0BF: IntegrateMessageDatagram(ExplicitDgramStruct*) (machine-eth.C:487)
==20526==    by 0x5BF694: ReceiveDatagram() (machine-eth.C:672)
==20526==    by 0x5BF81B: CommunicationServerNet(int, int) (machine-eth.C:725)
==20526==    by 0x5BFEDA: LrtsAdvanceCommunication(int) (machine.C:1760)
==20526==    by 0x5BAF48: AdvanceCommunication(int) (machine-common-core.C:1629)
==20526==    by 0x5BB1E1: CmiGetNonLocal (machine-common-core.C:1816)
==20526==    by 0x5C5976: CsdNextMessage (convcore.C:1814)
==20526==
==20526== 1,342,526,280 bytes in 5 blocks are possibly lost in loss record 633 of 633
==20526==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20526==    by 0x5B68E8: malloc_nomigrate (memory.C:1083)
==20526==    by 0x5C8CD1: CmiAlloc (convcore.C:3209)
==20526==    by 0x5BECE3: AssembleDatagram(OtherNodeStruct*, ExplicitDgramStruct*) (machine-eth.C:392)
==20526==    by 0x5BEF1E: AssembleReceivedDatagrams(OtherNodeStruct*) (machine-eth.C:443)
==20526==    by 0x5BF0BF: IntegrateMessageDatagram(ExplicitDgramStruct*) (machine-eth.C:487)
==20526==    by 0x5BF694: ReceiveDatagram() (machine-eth.C:672)
==20526==    by 0x5BF81B: CommunicationServerNet(int, int) (machine-eth.C:725)
==20526==    by 0x5BFEDA: LrtsAdvanceCommunication(int) (machine.C:1760)
==20526==    by 0x5BAF48: AdvanceCommunication(int) (machine-common-core.C:1629)
==20526==    by 0x5BB2A8: CmiNotifyStillIdle(CmiIdleState*) (machine-common-core.C:1903)
==20526==    by 0x5C0F07: call_cblist_keep(_ccd_cblist*, double) (conv-conds.C:166)
==20526==
==20526== LEAK SUMMARY:
==20526==    definitely lost: 22,636 bytes in 70 blocks
==20526==    indirectly lost: 0 bytes in 0 blocks
==20526==      possibly lost: 2,685,055,224 bytes in 49 blocks
==20526==    still reachable: 1,994,223 bytes in 31,587 blocks
==20526==                       of which reachable via heuristic:
==20526==                         newarray           : 36,008 bytes in 1 blocks
==20526==         suppressed: 0 bytes in 0 blocks

With -DTEST_PUP the multi-GB leaks disappear but there is still this leak of about 1.5 MB:
==20682==
==20682== 1,672,048 bytes in 29,858 blocks are still reachable in loss record 598 of 598
==20682==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==20682==    by 0x5BEA9A: EnqueueOutgoingDgram(OutgoingMsgStruct*, char*, int, OtherNodeStruct*, int, int) (machine-eth.C:312)
==20682==    by 0x5BEC31: DeliverViaNetwork(OutgoingMsgStruct*, OtherNodeStruct*, int, unsigned int, int) (machine-eth.C:352)
==20682==    by 0x5BFE26: DeliverOutgoingMessage(OutgoingMsgStruct*) (machine.C:1640)
==20682==    by 0x5BFEF3: LrtsSendFunc(int, int, int, char*, int) (machine.C:1688)
==20682==    by 0x5B9987: CmiInterSendNetworkFunc(int, int, int, char*, int) (machine-common-core.C:692)
==20682==    by 0x5B9A42: CmiInterFreeSendFn (machine-common-core.C:729)
==20682==    by 0x5B99B9: CmiFreeSendFn (machine-common-core.C:698)
==20682==    by 0x4E5EFC: Converse::CmiSyncSendAndFree(int, int, char*) (middle-conv.h:65)
==20682==    by 0x4EB5BB: _noCldEnqueue(int, envelope*) (ck.C:1508)
==20682==    by 0x4EBCE8: _sendMsgBranch(int, void*, _ckGroupID, int, int) (ck.C:1745)
==20682==    by 0x4EC494: CkSendMsgBranch (ck.C:1848)
==20682==
==20682== LEAK SUMMARY:
==20682==    definitely lost: 20,608 bytes in 51 blocks
==20682==    indirectly lost: 0 bytes in 0 blocks
==20682==      possibly lost: 2,664 bytes in 39 blocks
==20682==    still reachable: 1,994,811 bytes in 31,560 blocks
==20682==                       of which reachable via heuristic:
==20682==                         newarray           : 36,008 bytes in 1 blocks
==20682==         suppressed: 0 bytes in 0 blocks

Thanks @evan-charmworks for the test-case and the diagnosis. The bug was caused because of not freeing the received message in the putDataHandler (in the non onesided implementation of the ZC API, which was exercised only on netlrts). I've posted a fix for it in this PR (https://github.com/UIUC-PPL/charm/pull/2998).

Ready for review.
The Isomalloc benchmark will require #2188 to work correctly so I've opted to disable the test for now.

Does this not work with the new build system?

It does. Which CI would you suggest we run make testp on? I don't think we can run both make test and make testp without it timing out.

It seems make testp takes only 4 minutes, or am I seeing it wrong? Imho, we could also add this to at least one other cmake build, say Verbs Linux SMP.

Thanks for pointing that out. I think the Isomalloc benchmark was taking a disproportionate amount of time. I've added testp to various CIs that complete in <= 40 minutes, including some +p4 ++ppn 2 variants in SMP builds.
I'm not sure if we can enable tests for the Verbs CI if there is no Infiniband hardware.

Maximum test time on GitHub AFAIR is 6 hours, so we should be able to run it even if it takes a bit longer.
Edit:
See https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#usage-limits for maximum job time.

Good to know. We should still avoid increasing test length too much so the CI is not a burden on the merge master.

netlrts smp +p4 ++ppn 2 failure in lb_test:
make -C lb_test testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='++local +setcpuaffinity +CmiSleepOnIdle ++ppn 2'
make[3]: Entering directory '/home/circleci/project/netlrts-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
../../../../bin/testrun  +p4 ./lb_test $(( 25 * 4)) 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1  ++local +setcpuaffinity +CmiSleepOnIdle ++ppn 2
Charmrun> scalable start enabled.
Charmrun> started all node programs in 0.008 seconds.
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-366-gc2f0768cb
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
traceprojections was off at initial time.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.106 seconds.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
Running lb_test on 4 processors with 100 elements
Print every 10 steps
Sync every 40 steps
First node busywaits 10 usec; last node busywaits 1000 usec

Selecting Topology Ring
Generating topology 0 for 100 elements
[0] Total work/step = 0.020908 sec
calibrated iterations 12721987
TIME PER STEP	10	0.639974	0.104232
TIME PER STEP	20	0.744706	0.104731
TIME PER STEP	30	0.839714	0.095009
--------- Started LB step 0 ---------
[0] num_procs=4 num_objs=100
[0] obj loads: min=0.000222 mean=0.013601 max=0.175276
[0] proc loads: min=0.233311 mean=0.365779 max=0.509561
[0] strategy Greedy time=0.000012 secs, maxLoad after strategy=0.365835, num_migrations=73 migrations_sum_hops=0
--------- Finished LB step 0 ---------
------------- Processor 0 Exiting: Called CmiAbort ------------
Reason: [0] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

[0] Stack Traceback:
  [0:0] lb_test 0x5d5f37
  [0:1] lb_test 0x5e99fa
  [0:2] lb_test 0x4cfcac TreeLB::objMovedIn(bool)
------------- Processor 2 Exiting: Called CmiAbort ------------
Reason: [2] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

------------- Processor 3 Exiting: Called CmiAbort ------------
Reason: [3] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

[2] Stack Traceback:
[3] Stack Traceback:
  [2:0] lb_test 0x5d5f37
  [0:3] lb_test 0x55e5dd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [0:4] lb_test 0x56d224 CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [0:5] lb_test 0x56d6e2 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [0:6] lb_test 0x52d7f4 CkDeliverMessageFree
  [0:7] lb_test 0x530638 _processHandler(void*, CkCoreState*)
  [0:8] lb_test 0x5e613b CsdScheduleForever
  [0:9] lb_test 0x5e691d CsdScheduler
  [0:10] lb_test 0x5df8c9
  [0:11] lb_test 0x5dfe64 ConverseInit
  [0:12] lb_test 0x5ba017 charm_main
  [3:0] lb_test 0x5d5f37
  [3:1] lb_test 0x5e99fa
  [3:2] lb_test 0x4cfcac TreeLB::objMovedIn(bool)
  [3:3] lb_test 0x55e5dd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [3:4] lb_test 0x56d224 CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [3:5] lb_test 0x56d6e2 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [3:6] lb_test 0x52d7f4 CkDeliverMessageFree
  [3:7] lb_test 0x530638 _processHandler(void*, CkCoreState*)
  [3:8] lb_test 0x5e613b CsdScheduleForever
  [3:9] lb_test 0x5e691d CsdScheduler
  [0:13] libc.so.6 0x7ffff6b51830 __libc_start_main
  [0:14] lb_test 0x4b7409 _start
Fatal error on PE 0> [0] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/circleci/project/src/ck-ldb/TreeLB.C line 465.

real	0m0.888s
user	0m0.005s
sys	0m0.000s
Makefile:34: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/circleci/project/netlrts-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'

UTH times out in benchmarks/converse/cthtest with +p2.

buildold make test failed with the same hang in benchmarks/ampi/alltoall exhibited by other CIs.

netlrts-linux, netlrts-darwin, and mpi-linux-smp are all hanging in the pingpong-cachemiss test of benchmarks/converse/commbench. #2076

The Isomalloc benchmark will require #2188 to work correctly so I've opted to disable the test for now.

Nitin mentioned in #2962 (comment) that CMA has a 2 GB limit per read() call, so #2188 may be overkill to solve this problem if we have to subdivide transfers into blocks anyway.

Besides the above SMP issue with lb_test, the only remaining failure is a timeout in mpi-linux-smp because pingpong_multipairs took 36 minutes to complete:
../../../bin/testrun  ./pingpong_multipairs +p2  +setcpuaffinity

Running on 2 processors:  ./pingpong_multipairs +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./pingpong_multipairs +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 1 worker threads (PEs) + 1 comm threads per process, 2 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: fe0fa19
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.080 seconds.
Multiple pair send/recv
 bytes 		 latency(us)	 bandwidth(MBytes/sec)
8		  9464.50   0.00
16		  18884.01   0.00
32		  28413.01   0.00
64		  38033.01   0.00
128		  47511.01   0.00
256		  56820.51   0.00
512		  66266.51   0.01
1024		  75828.51   0.01
2048		  85187.51   0.02
4096		  94754.01   0.04
8192		  112328.51   0.07
16384		  128587.51   0.13
32768		  138194.51   0.24
65536		  155832.51   0.42
131072		  173531.01   0.76
262144		  192566.01   1.36
524288		  212303.01   2.47
1048576		  232655.51   4.51
2097152		  253255.51   8.28
4194304		  273867.01   15.32
[Partition 0][Node 0] End of program

real	36m33.207s
user	61m1.942s
sys	11m54.468s
make[2]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/benchmarks/converse/pingpong'

@rbuch The same lb_test issue happened with MPI Linux SMP +p4 ++ppn 2
make -C lb_test testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
../../../../bin/testrun  +p4 ./lb_test $(( 25 * 4)) 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1  +setcpuaffinity ++ppn 2

Running on 2 processors:  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
traceprojections was off at initial time.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.072 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
Running lb_test on 4 processors with 100 elements
Print every 10 steps
Sync every 40 steps
First node busywaits 10 usec; last node busywaits 1000 usec

Selecting Topology Ring
Generating topology 0 for 100 elements
[0] Total work/step = 0.020908 sec
calibrated iterations 12163733
TIME PER STEP	10	0.580004	0.076805
TIME PER STEP	20	0.666661	0.086656
TIME PER STEP	30	0.768079	0.101418
--------- Started LB step 0 ---------
[0] num_procs=4 num_objs=100
[0] obj loads: min=0.000226 mean=0.014686 max=0.181409
[0] proc loads: min=0.262121 mean=0.372752 max=0.484037
[0] strategy Greedy time=0.000079 secs, maxLoad after strategy=0.372875, num_migrations=69 migrations_sum_hops=0
[2] Stack Traceback:
------------- Processor 2 Exiting: Called CmiAbort ------------
Reason: [2] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/travis/build/UIUC-PPL/charm/src/ck-ldb/TreeLB.C line 465.

--------- Finished LB step 0 ---------
[0] lb time = 0.008794
  [2:0] lb_test 0x5c861a CmiAbortHelper(char const*, char const*, char const*, int, int)
  [2:1] lb_test 0x5c873b
  [2:2] lb_test 0x5d285c
  [2:3] lb_test 0x4d368a TreeLB::objMovedIn(bool)
  [2:4] lb_test 0x553afd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [2:5] lb_test 0x563bea CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [2:6] lb_test 0x563e08 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [2:7] lb_test 0x5280c2 CkDeliverMessageFree
  [2:8] lb_test 0x52a9ce _processHandler(void*, CkCoreState*)
  [2:9] lb_test 0x5cefc8 CsdScheduleForever
  [2:10] lb_test 0x5cf2bd CsdScheduler
  [2:11] lb_test 0x5cb3c2
  [2:12] lb_test 0x5cbd7c
  [2:13] libpthread.so.0 0x7ffff7bc16ba
  [2:14] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m0.891s
user	0m0.012s
sys	0m0.004s
Makefile:34: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
Makefile:26: recipe for target 'testp-lb_test' failed
make[2]: *** [testp-lb_test] Error 2

There is also a segfault in examples/charm++/load_balancing/hello:
make -C hello testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2' P='4'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
../../../../bin/testrun  +p4 ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity ++ppn 2

Running on 2 processors:  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.047 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy GreedyRefine
Hello, I'm chare 0 on PE 0
Hello, I'm chare 1 on PE 0
Hello, I'm chare 2 on PE 1
Hello, I'm chare 3 on PE 1
--------- Started LB step 0 ---------
Hello, I'm chare 4 on PE 2
Hello, I'm chare 6 on PE 3
Hello, I'm chare 7 on PE 3
Hello, I'm chare 5 on PE 2
[0] num_procs=4 num_objs=8
[0] obj loads: min=0.000008 mean=0.500797 max=1.004737
[0] proc loads: min=0.000088 mean=1.001966 max=2.006403
[0] GreedyRefine: greedy maxload is 1.004796, tolerance set to 1.000000
[0] strategy GreedyRefine time=0.000011 secs, maxLoad after strategy=1.004796, num_migrations=4 migrations_sum_hops=0
------------- Processor 2 Exiting: Caught Signal ------------
Reason: Segmentation fault
[2] Stack Traceback:
  [2:0] hello 0x5c32f8
  [2:1] libpthread.so.0 0x7ffff7bcb390
  [2:2] hello 0x4cfd19 PELevel::processDecision(TreeLBMessage*, int&, int&)
  [2:3] hello 0x4cb73d TreeLB::receiveDecision(TreeLBMessage*, int)
  [2:4] hello 0x4cb94b TreeLB::sendDecisionDown(CkMessage*)
  [2:5] hello 0x51c042 CkDeliverMessageFree
  [2:6] hello 0x51e94e _processHandler(void*, CkCoreState*)
  [2:7] hello 0x5c762e CsdScheduleForever
  [2:8] hello 0x5c789d CsdScheduler
  [2:9] hello 0x5c39a2
  [2:10] hello 0x5c435c
  [2:11] libpthread.so.0 0x7ffff7bc16ba
  [2:12] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m2.179s
user	0m0.015s
sys	0m0.001s
Makefile:21: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
Makefile:25: recipe for target 'testp-hello' failed
make[2]: *** [testp-hello] Error 2

@rbuch The same lb_test issue happened with MPI Linux SMP +p4 ++ppn 2
make -C lb_test testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
../../../../bin/testrun  +p4 ./lb_test $(( 25 * 4)) 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1  +setcpuaffinity ++ppn 2

Running on 2 processors:  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./lb_test 100 100 10 40 10 1000 ring +balancer GreedyLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
traceprojections was off at initial time.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.072 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
Running lb_test on 4 processors with 100 elements
Print every 10 steps
Sync every 40 steps
First node busywaits 10 usec; last node busywaits 1000 usec

Selecting Topology Ring
Generating topology 0 for 100 elements
[0] Total work/step = 0.020908 sec
calibrated iterations 12163733
TIME PER STEP	10	0.580004	0.076805
TIME PER STEP	20	0.666661	0.086656
TIME PER STEP	30	0.768079	0.101418
--------- Started LB step 0 ---------
[0] num_procs=4 num_objs=100
[0] obj loads: min=0.000226 mean=0.014686 max=0.181409
[0] proc loads: min=0.262121 mean=0.372752 max=0.484037
[0] strategy Greedy time=0.000079 secs, maxLoad after strategy=0.372875, num_migrations=69 migrations_sum_hops=0
[2] Stack Traceback:
------------- Processor 2 Exiting: Called CmiAbort ------------
Reason: [2] Assertion "numLevels > 0 && awaitingLB[level]" failed in file /home/travis/build/UIUC-PPL/charm/src/ck-ldb/TreeLB.C line 465.

--------- Finished LB step 0 ---------
[0] lb time = 0.008794
  [2:0] lb_test 0x5c861a CmiAbortHelper(char const*, char const*, char const*, int, int)
  [2:1] lb_test 0x5c873b
  [2:2] lb_test 0x5d285c
  [2:3] lb_test 0x4d368a TreeLB::objMovedIn(bool)
  [2:4] lb_test 0x553afd CkLocRec::CkLocRec(CkLocMgr*, bool, bool, CkArrayIndex const&, unsigned long)
  [2:5] lb_test 0x563bea CkLocMgr::createLocal(CkArrayIndex const&, bool, bool, bool)
  [2:6] lb_test 0x563e08 CkLocMgr::immigrate(CkArrayElementMigrateMessage*)
  [2:7] lb_test 0x5280c2 CkDeliverMessageFree
  [2:8] lb_test 0x52a9ce _processHandler(void*, CkCoreState*)
  [2:9] lb_test 0x5cefc8 CsdScheduleForever
  [2:10] lb_test 0x5cf2bd CsdScheduler
  [2:11] lb_test 0x5cb3c2
  [2:12] lb_test 0x5cbd7c
  [2:13] libpthread.so.0 0x7ffff7bc16ba
  [2:14] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m0.891s
user	0m0.012s
sys	0m0.004s
Makefile:34: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/tests/charm++/load_balancing/lb_test'
Makefile:26: recipe for target 'testp-lb_test' failed
make[2]: *** [testp-lb_test] Error 2

There is also a segfault in examples/charm++/load_balancing/hello:
make -C hello testp OPTS='-g -Werror=vla -optimize -production' TESTOPTS='+setcpuaffinity ++ppn 2' P='4'
make[3]: Entering directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
../../../../bin/testrun  +p4 ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity ++ppn 2

Running on 2 processors:  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./hello +balancer GreedyRefineLB +LBDebug 1 +setcpuaffinity +ppn 2
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: 3024712
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Verbose level 1, load balancing period: -1 seconds
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 1 cores x 2 PUs = 2-way SMP)
Charm++> cpu topology info is gathered in 0.047 seconds.
Charm++> Warning: Running with more SMP threads (6) than physical cores (2).
         Use +CmiSleepOnIdle (default behavior) or +CmiSpinOnIdle to silence this message.
WARNING: Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy GreedyRefine
Hello, I'm chare 0 on PE 0
Hello, I'm chare 1 on PE 0
Hello, I'm chare 2 on PE 1
Hello, I'm chare 3 on PE 1
--------- Started LB step 0 ---------
Hello, I'm chare 4 on PE 2
Hello, I'm chare 6 on PE 3
Hello, I'm chare 7 on PE 3
Hello, I'm chare 5 on PE 2
[0] num_procs=4 num_objs=8
[0] obj loads: min=0.000008 mean=0.500797 max=1.004737
[0] proc loads: min=0.000088 mean=1.001966 max=2.006403
[0] GreedyRefine: greedy maxload is 1.004796, tolerance set to 1.000000
[0] strategy GreedyRefine time=0.000011 secs, maxLoad after strategy=1.004796, num_migrations=4 migrations_sum_hops=0
------------- Processor 2 Exiting: Caught Signal ------------
Reason: Segmentation fault
[2] Stack Traceback:
  [2:0] hello 0x5c32f8
  [2:1] libpthread.so.0 0x7ffff7bcb390
  [2:2] hello 0x4cfd19 PELevel::processDecision(TreeLBMessage*, int&, int&)
  [2:3] hello 0x4cb73d TreeLB::receiveDecision(TreeLBMessage*, int)
  [2:4] hello 0x4cb94b TreeLB::sendDecisionDown(CkMessage*)
  [2:5] hello 0x51c042 CkDeliverMessageFree
  [2:6] hello 0x51e94e _processHandler(void*, CkCoreState*)
  [2:7] hello 0x5c762e CsdScheduleForever
  [2:8] hello 0x5c789d CsdScheduler
  [2:9] hello 0x5c39a2
  [2:10] hello 0x5c435c
  [2:11] libpthread.so.0 0x7ffff7bc16ba
  [2:12] libc.so.6 0x7ffff67ad41d clone
application called MPI_Abort(comm=0x84000002, 1) - process 1

real	0m2.179s
user	0m0.015s
sys	0m0.001s
Makefile:21: recipe for target 'testp' failed
make[3]: *** [testp] Error 1
make[3]: Leaving directory '/home/travis/build/UIUC-PPL/charm/mpi-linux-x86_64-smp/examples/charm++/load_balancing/hello'
Makefile:25: recipe for target 'testp-hello' failed
make[2]: *** [testp-hello] Error 2

Yeah, I've been debugging it since yesterday, I'll make a separate issue. It seems like sometimes PEs get an LB decision that says that no objects are incoming even though there are incoming objects, but I've only seen this in cases with p > 1 and ppn > 1. I'll make a separate issue for this, thanks for pointing it out.

Our MPI Linux SMP build in particular appears to be buggy and I'm leaning toward disabling the +p4 ++ppn 2 part of its CI for this PR to unblock it. NetLRTS SMP does not have the same issues and can keep the PPN CI.

Something about the MPI machine layer with SMP is causing hangs in examples/charm++/zerocopy/entry_method_api/{reg,prereg,unreg}/stencil3d and tests/ampi/intercomm_coll.

I once got this failure in examples/charm++/zerocopy/entry_method_post_api/unreg/simpleZeroCopy, but could not reproduce it:

../../../../../../bin/testrun  +p4 ./simpleZeroCopy $(( 4 * 10 )) +balancer GreedyLB ++ppn 2 +setcpuaffinity

Running on 2 processors:  ./simpleZeroCopy 40 +balancer GreedyLB +ppn 2 +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./simpleZeroCopy 40 +balancer GreedyLB +ppn 2 +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.024 seconds.
[0] TreeLB in LEGACY MODE support
[0] TreeLB: Using PE_Root tree with strategy Greedy
send: completed
[NemeanLion:20932] *** An error occurred in MPI_Irecv
[NemeanLion:20932] *** reported by process [2524905473,1]
[NemeanLion:20932] *** on communicator MPI COMMUNICATOR 3 DUP FROM 0
[NemeanLion:20932] *** MPI_ERR_BUFFER: invalid buffer pointer
[NemeanLion:20932] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[NemeanLion:20932] ***    and potentially your MPI job)
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
------------- Processor 4 Exiting: Caught Signal ------------
Reason: Terminated
[4] Stack Traceback:
  [4:0] simpleZeroCopy 0x5555557425fb
  [4:1] libpthread.so.0 0x7ffff7bc88a0
  [4:2] mca_btl_sm.so 0x7fffe5b03d22 mca_btl_sm_component_progress
  [4:3] libopen-pal.so.20 0x7ffff625fabc opal_progress
  [4:4] mca_pml_ob1.so 0x7fffe56da903 mca_pml_ob1_iprobe
  [4:5] libmpi.so.20 0x7ffff716e31a PMPI_Iprobe
  [4:6] simpleZeroCopy 0x555555740d7b
  [4:7] simpleZeroCopy 0x555555741519 LrtsAdvanceCommunication(int)
  [4:8] simpleZeroCopy 0x5555557417ce CommunicationServerThread(int)
  [4:9] simpleZeroCopy 0x555555741b8a
  [4:10] simpleZeroCopy 0x555555742194 ConverseInit
  [4:11] simpleZeroCopy 0x555555722efc charm_main
  [4:12] libc.so.6 0x7ffff6794b97 __libc_start_main
  [4:13] simpleZeroCopy 0x55555562800a _start

real  0m0.311s
user  0m0.365s
sys 0m0.082s

Also this failure in megatest:

make[3]: Entering directory '/home/evan/Charmworks/charm/mpi-linux-x86_64-smp/tests/charm++/megatest'
../../../bin/testrun  ./pgm +p4  ++ppn 2 +setcpuaffinity

Running on 2 processors:  ./pgm +ppn 2 +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./pgm +ppn 2 +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.020 seconds.
Megatest is running on 2 nodes 4 processors.
test 0: initiated [groupring (milind)]
test 0: completed (0.12 sec)
test 1: initiated [nodering (milind)]
test 1: completed (0.00 sec)
test 2: initiated [varsizetest (mjlang)]
test 2: completed (0.00 sec)
test 3: initiated [varsizetest2 (phil)]
test 3: completed (0.00 sec)
test 4: initiated [varraystest (milind)]
test 4: completed (0.00 sec)
test 5: initiated [groupcast (mjlang)]
test 5: completed (0.00 sec)
test 6: initiated [groupmulti (gengbin)]
test 6: completed (0.00 sec)
test 7: initiated [groupsectiontest (ebohm)]
test 7: completed (0.00 sec)
test 8: initiated [multisectiontest (ebohm)]
test 8: completed (0.03 sec)
test 9: initiated [nodecast (milind)]
test 9: completed (0.00 sec)
test 10: initiated [synctest (mjlang)]
test 10: completed (0.01 sec)
test 11: initiated [fib (jackie)]
test 11: completed (0.01 sec)
test 12: initiated [arrayring (fang)]
test 12: completed (0.00 sec)
test 13: initiated [packtest (fang)]
test 13: completed (0.00 sec)
test 14: initiated [queens (jackie)]
test 14: completed (0.01 sec)
test 15: initiated [migration (jackie)]
test 15: completed (0.00 sec)
test 16: initiated [marshall (olawlor)]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
test 16: completed (0.32 sec)
test 17: initiated [priomsg (fang)]
test 17: completed (0.00 sec)
test 18: initiated [priotest (mlind)]
test 18: completed (0.03 sec)
test 19: initiated [rotest (milind)]
test 19: completed (0.01 sec)
test 20: initiated [statistics (olawlor)]
test 20: completed (0.00 sec)
test 21: initiated [templates (milind)]
test 21: completed (0.00 sec)
test 22: initiated [inherit (olawlor)]
test 22: completed (0.00 sec)
test 23: initiated [reduction (olawlor)]
test 23: completed (0.00 sec)
test 24: initiated [bitvector (jbooth)]
test 24: completed (0.00 sec)
test 25: initiated [immediatering (gengbin)]
test 25: completed (0.00 sec)
test 26: initiated [callback (olawlor)]
test 26: completed (0.00 sec)
test 27: initiated [inlineem (phil)]
test 27: completed (0.00 sec)
test 28: initiated [completion_test (phil)]
Starting test
Created detector, starting first detection
Started first test
Finished second test
Started third test
test 28: completed (0.00 sec)
test 29: initiated [groupdependence (nbhat4)]
test 29: completed (0.00 sec)
test 30: initiated [multi groupring (milind)]
test 30: completed (0.00 sec)
test 31: initiated [multi nodering (milind)]
test 31: completed (0.00 sec)
test 32: initiated [multi varsizetest (mjlang)]
test 32: completed (0.00 sec)
test 33: initiated [multi varsizetest2 (phil)]
test 33: completed (0.00 sec)
test 34: initiated [multi varraystest (milind)]
test 34: completed (0.00 sec)
test 35: initiated [multi groupcast (mjlang)]
test 35: completed (0.00 sec)
test 36: initiated [multi groupmulti (gengbin)]
test 36: completed (0.00 sec)
test 37: initiated [multi groupsectiontest (ebohm)]
test 37: completed (0.00 sec)
test 38: initiated [multi multisectiontest (ebohm)]
test 38: completed (0.11 sec)
test 39: initiated [multi nodecast (milind)]
test 39: completed (0.05 sec)
test 40: initiated [multi synctest (mjlang)]
[4] Stack Traceback:
  [4:0] pgm 0x555555812acb
  [4:1] libpthread.so.0 0x7ffff7bc88a0
  [4:2] mca_pml_ob1.so 0x7fffe56ebab4 mca_pml_ob1_recv_req_start
  [4:3] mca_pml_ob1.so 0x7fffe56da8e5 mca_pml_ob1_iprobe
  [4:4] libmpi.so.20 0x7ffff716e31a PMPI_Iprobe
  [4:5] pgm 0x55555581124b
  [4:6] pgm 0x5555558119e9 LrtsAdvanceCommunication(int)
  [4:7] pgm 0x555555811c9e CommunicationServerThread(int)
  [4:8] pgm 0x55555581205a
  [4:9] pgm 0x555555812664 ConverseInit
  [4:10] pgm 0x5555557a2bfc charm_main
  [4:11] libc.so.6 0x7ffff6794b97 __libc_start_main
  [4:12] pgm 0x55555568bb6a _start
[5] Stack Traceback:
  [5:0] pgm 0x555555812acb
  [5:1] libpthread.so.0 0x7ffff7bc88a0
  [5:2] libopen-pal.so.20 0x7ffff62bb093
  [5:3] libopen-pal.so.20 0x7ffff625f9a9 opal_progress
  [5:4] mca_pml_ob1.so 0x7fffe56da903 mca_pml_ob1_iprobe
  [5:5] libmpi.so.20 0x7ffff716e31a PMPI_Iprobe
  [5:6] pgm 0x55555581124b
  [5:7] pgm 0x5555558119e9 LrtsAdvanceCommunication(int)
  [5:8] pgm 0x555555811c9e CommunicationServerThread(int)
  [5:9] pgm 0x55555581205a
  [5:10] pgm 0x555555812664 ConverseInit
  [5:11] pgm 0x5555557a2bfc charm_main
  [5:12] libc.so.6 0x7ffff6794b97 __libc_start_main
  [5:13] pgm 0x55555568bb6a _start
------------- Processor 4 Exiting: Caught Signal ------------
Reason: Terminated
------------- Processor 5 Exiting: Caught Signal ------------
Reason: Terminated
[NemeanLion:22516] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[NemeanLion:22516] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

real	0m1.630s
user	0m3.254s
sys	0m0.046s

As an additional note on performance, pingpong_multipairs has an aberration in speed between iterations of size <= 8192 and >= 16384.

make[3]: Entering directory '/home/evan/Charmworks/charm/mpi-linux-x86_64-smp/benchmarks/converse/pingpong'
../../../bin/testrun  ./pingpong_multipairs +p4 ++ppn 2 +setcpuaffinity

Running on 2 processors:  ./pingpong_multipairs +ppn 2 +setcpuaffinity
charmrun>  /usr/bin/setarch x86_64 -R  mpirun -np 2  ./pingpong_multipairs +ppn 2 +setcpuaffinity
Charm++> Running on MPI version: 3.1
Charm++> level of thread support used: -1 (desired: 0)
Charm++> Running in SMP mode: 2 processes, 2 worker threads (PEs) + 1 comm threads per process, 4 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Converse/Charm++ Commit ID: v6.11.0-devel-384-g86fa91abd
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> cpu affinity enabled.
Charm++> Running on 1 hosts (1 sockets x 4 cores x 2 PUs = 8-way SMP)
Charm++> cpu topology info is gathered in 0.028 seconds.
Multiple pair send/recv
 bytes     latency(us)   bandwidth(MBytes/sec)
8     2.65   3.01
16      6.00   2.67
32      11.59   2.76
64      18.26   3.50
128     23.41   5.47
256     28.73   8.91
512     32.51   15.75
1024      37.44   27.35
2048      42.51   48.18
4096      60.71   67.47
8192      80.76   101.44
16384     2512.18   6.52
32768     6642.26   4.93
65536     6679.92   9.81
131072      6749.01   19.42
262144      6810.27   38.49
524288      7113.52   73.70
1048576     7505.03   139.72
2097152     8142.08   257.57
4194304     9566.39   438.44
[Partition 0][Node 0] End of program

real  2m16.499s
user  11m10.740s
sys 0m15.628s

Ready for review.

Should we create an issue for the tests that fail with oversubscription?

They don't fail, they just take a long time (20-35 minutes).
I will create follow-up PRs to track some of the tests that are entirely disabled.

intercomm_coll timed out. I can reproduce the same thing on multicore.

Needs one more review

authors: evan-charmworks matthiasdiener minitu nitbhat rbuch


text:
feat(FN-995): notify bank confirmation email

Introduction
We need to send confirmation to the banks when they have submitted a utilisation report.
Resolution

Updated endpoint in portal-api, which is called when user clicks Confirm and Send button on the utilisation report upload pages
Used sendEmail function to use Notify to send email
Added new email template to constants file

To Do

New bank data structure in MongoDB:

_id: ObjectId('string')
id: string
name: string
mga: Array (1)
	0: string
emails: Array (2)
	0: string
	1: string
companiesHouseNo: string
partyUrn: string
hasGefAccessOnly: boolean
paymentOfficerTeam:  Object
	teamName: string
        email: string

Pull email address on confirmation page from bank details too (FN-969 backend)

Codecov Report

:exclamation: No coverage uploaded for pull request base (main-stbr@410b4da). Click here to learn what that means.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             main-stbr    #2187   +/-   ##
============================================
  Coverage             ?   17.12%
============================================
  Files                ?      490
  Lines                ?    11279
  Branches             ?     2190
============================================
  Hits                 ?     1931
  Misses               ?     7768
  Partials             ?     1580

Flag
Coverage Δ

ui-test
64.24% <0.00%> (?)

unit-test
4.84% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

source: gharchive/pull-request

authors: codecov-commenter francescastocco

license: MIT

license_type: permissive

license_source: github-api


text:
Revert "Upgrade to node10-alpine"

Reverts UKHomeOffice/html-pdf-converter#22

so broken!

repo: UKHomeOffice/html-pdf-converter


text:
Increase robustness to installation on new computers

Others have reported unity is unable to find some files when installing this project. This issue can be resolved by testing install and fixing any issues on other computer

The apparent issue seems to have been with missing SteamVR assets. I've updated the install instruction to be more specific as to how to install the SteamVR plugin.
Please reopen or create a new issue if the installation leaves errors in the Unity file.
Note, unless you are in the Arm Lab at the University of Michigan, you will (almost certainly) not have the necessary robot interface to move the robot (Victor).

source: gharchive/issue


text:
is it necessary to perform GCTA-COJO before using PolarMorphism software？

Hello Joanna,
Thanks for your software and article.
My team encountered some problems reproducing the results in your article (PMID: 35758773).
In https://github.com/UMCUGenetics/PolarMorphism/blob/master/template_for_github.Rmd, (line 28: print(paste0(wdir, trait, ".cojo.gz"))), I noticed that the input file The suffix is ".cojo.gz".
Does the example in the article pre-process the GWAS summary statistics raw data with GCTA-COJO?
If so, is it necessary to perform GCTA-COJO before using PolarMorphism software?
Thank you
Mo

Hi Mo,
Sorry for getting back to you so late! PolarMorphism was one of my PhD projects, and I have since started a postdoc in a different field so I can't spend a lot of time on it anymore.
To answer your question: no, that is not necessary! The reason the suffix of the input data I used is ".cojo.gz", is because these files were preprocessed to be run for conditional analysis and it was very convenient for me to use the same preprocessed files. They have been preprocessed as described in the paper, and COJO had not been run yet.

Thank you for your answer.
I found that the reproduction result is different due to the n_max setting 10000.The problem was solved after canceling it, and it's too demanding for RAM.
Thank you for assisting me with this issue!

authors: Joannavonberg Moxxaaaa

repo: UMCUGenetics/PolarMorphism


text:
Make management of yarn/webpack dev server with gradle less awful.

Right now we're using some sketchy bash scripts to manage the webpack/yarn server processes through gradle. We're not killing the processes nicely, and none of this works on windows. This should be addressed.
Also:

It wouldn't hurt to add something like a "client status" task for the client subproject in gradle.
Management of the log files for these things needs to be improved. Students can't be expected to know how to tail log files...

This was actually fixed in 331b9ede8fdb1c652464b09bda5f0b638b10232b
The commit messages there references the wrong issue.


text:
Fix ingest specs

Finally all the specs pass again!

Coverage increased (+2.7%) to 45.425% when pulling 2cc2526c8b21b3575a43086070b92be80e0f6be8 on fix-ingest-specs into c8efa87f0e742216efb9cf98a33a104b20bbdbab on master.

authors: BRKalow coveralls

repo: UMNLibraries/ikidowinan

license: mit

license_source: bigquery


text:
Notify ssh restart handler upon config changes

Closes #11.

Patch looks good and is actually quite important to ensure the config is active.

authors: lkiesow tibroc


text:
Create Daily Backstop Emissions View

Context
Create Daily Backstop Emissions View
This can not be completed until import/export is done for Daily Backstop Emissions

Note: @JanellC follow up with @billmiller13 on getting additional infromation on this ticket.

Daily Backstop view has been added and can be tested in the official view in ECMPS dev using
Barry 1,2,CS0AAN
2023 Q1-Q2
2022 Q1-Q4
Export these periods and import them into the workspace to then test the workspace view as well as the export/import process.

Private Zenhub Image

Need 5582 to resolve the issue

repo: US-EPA-CAMD/easey-ui


text:
RETEST:  Beta environment: Submissions are not processing

While testing #5892, I did submissions for ORIS 404. The submissions are stuck in the "Submitted, Host Update Pending" status. This is in the Beta environment.
We need to figure out the underlying cause of this and fix it or put in logic to have the job restart on its own.

In #6108, this was remedied by restarting the Submission Job Queue job.

Will monitor submission data during beta testing to determine if this is still an issue.

FYI @esaber76, @mark-hayward-erg, @maheese.
Summary: After successfully evaluating a QA Test Data file, the QA Test Data file Submission Status changed to "Updated on Host." The QA record was visible in the submission module, however, I was not able to select the QA test data record to Sign & Submit.
Steps to recreate the issue:

Login to the test environment as a data submitter and select the following config: ORIS 404, Location 1.
Navigate to the MP module, export & import the MP data.
Navigate to the QA > Test Data module, click the import button, select Import from historical data, then select 2022 Q4, click the preview button.
Select the Test with the test number: 20224GFFM:112T1
In the QA Cert module UI, click the Test Type Group dropdown and select "Primary Element Inspection."
Click the Edit button for the record with the test #: 20224GFFM:112T1 to modify the Test Summary Data
Modify the Test Summary Data:

Modify the Test number to 20234GFFM:112T1
Modify the End Date to 03/02/2023
Append the existing Test comment with the word "test."

Click the Save & Close button.
Navigate to the Emissions module, import 2023Q1 emission data from historical.
Before evaluating the QA and Emissions data, an admin must go into the System Admin module to allow the resubmission of test number 20234GFFM:112T1 and reopen the 2023Q1 Emissions submission window for the ORIS 404 Location1.
After the Admin tasks have been completed, the data submitter user should navigate to the Evaluation module, select the ORIS 404, Location 1, and 2023Q1 as the filter options, and click "Apply Filters."
After applying the filter, select the MP, QA Test Data, and Emissions records to evaluate.
The evaluation status for each of the MP, QA, and Emissions records will result in the following (see the screenshot below):

Different Evaluation Status

MP: Premission to Resubmit Granted
QA: Updated to Host
Emissions: Resubmission Required

Closing this ticket based on the understanding that beta testing will generate specific bugs to act on. New ticket #6360 created to capture bugs

authors: alangmaid esaber76 ibarra-michelle mark-hayward-erg maxdiebold-erg


text:
Add XML-based Sparkler Configuration

Add Sparkler Configuration through sparkler-default.xml and sparkler-site.xml

Completed as per commit 6f2410b1b9cfc1e7ff5f2418f4eb3171915a6b26

@karanjeets Reopening the issue because the Configuration has to be java serializable. Hadoop Configuration is not java serializable so we have trouble using it directly in spark.
We may need to have our own implementation. Check out https://github.com/USCDataScience/sparkler/blob/365fc934ad0aad7f2dab1a2334f05bbdf0fd368d/sparkler-app/src/main/scala/edu/usc/irds/sparkler/model/SparklerJob.scala#L47 for some FIXME:
Lets discuss more of this.

authors: karanjeets thammegowda

repo: USCDataScience/sparkler


text:
Log summary_multiple_inst_ts.csv updated to current date

Log file summary_multiple_inst_ts.csv will be updated even if the data are the same for the time period of interest. The end_date column is the day of build for sites that are active, and the count_nu and count_ts will also update to the date of the build. For the sites that are active, could we report the end_date as the specified dummy_date and the number of samples up to the dummy date as count_nu?

I've edited this target in PR #47 so that if a site is active, the returned end_date is equal to the dummy_date. Regarding the other columns mentioned above,

the count_ts column is used to tally the number of unique ts_id's that get returned and should not change with build date unless a new ts_id gets added for the site. I've left this column as is.
I've omitted count_nu from this table. We're not necessarily using this log file to tally the observation-days available and so I leave this count to other log files within our pipeline, such as 3_visualize/sitelist_summary.csv.

I've also opted to move the code that was within the target itself into a separate function - with the changes described above, the target code was > 10 lines within 1_fetch.R, so probably too long for the phase file. I haven't changed the target name but I'm noting for the future that p1_nwis_sites_inst_multipleTS_csv is just too long 😄

repo: USGS-R/drb-inland-salinity-ml


text:
Flow cartogram build for Mar 2022

Let me know if you'd rather do April. I sent you the SVG on Teams.

Woww a very wet month

Switched to April! The file for March on S3 was deleted and April was added in its place. Running tar_make() with the changes in this PR will produce an SVG that looks like the image shown below in 40 seconds.

woah. going to merge

authors: ceenell lindsayplatt

repo: USGS-VIZLAB/viz-scratch


text:
Added events page search bar and title

Resolves #61

Thanks for making that change! It looks like this branch includes some older UI - do you know if this was branched from develop?

Thanks for making that change! It looks like this branch includes some older UI - do you know if this was branched from develop?

@Jake3231 @RubenOlano Yeah I totally forgot about that. It should be fixed now.

I take back what I said earlier. There is something very wrong with my branch so I'm going to restart.

repo: UTDNebula/jupiter


text:
🛑 Wiki is down

In cd324de, Wiki (https://online.huskysatellitelab.com/wiki) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Wiki is back up in f89fae9.

repo: UWCubeSat/selfhosted-upptime


text:
SEEM in remote sensing images segmantation task.

Thank you very much for your work! But I still find that there are still several problems. When segmenting remote sensing images, the results of the model used in your demo presentation perform very well. However, when I use several of the pre-trained models you posted (SEEM/X-Decoder), I find that the segmentation results are nowhere near as good, if not extremely bad, as your demo program. Do you have any idea what is going on here? Or can you open source the model (using Davit-d3 as backbone) that was used as demo demo?I choose the [panoramic] task when I use the app.py.

Also. I noticed that except for your demo model in huggingface (Davit-d3 as backbone), the other models don't have clear data boundaries (e.g., the "splash" problem mentioned in other issues), is this related to the fact that the other models are not trained on the SBD dataset?  (Noting that only Davit-d3 has SBD metrics in your checkpoint)

When I use the demo demo on huggingface for image segmentation, I get great results：

But when I use the published pre-trained model for segmentation, I get poor results：

Do you know what the problem is? What can I do to reproduce the results of the demo?

I see, thanks so much for your feedback, I could not release model checkpoint other than focalt, and focall, I sincerely apologize for the sub-optimal results.

Hi @MaureenZOU
May I ask why you can't release the model weights used in the online demo?

may I ask if you obtained this result through training on your own annotated remote sensing dataset?

No. I chose other pre-training models to get my work done. :<

may I ask if you obtained this result through training on your own annotated remote sensing dataset?
No. I chose other pre-training models to get my work done. :<

may I ask if you obtained this result through training on your own annotated remote sensing dataset?

None. I chose other pre-training models to get my work done :<.

authors: Aki-Tomoya EricZavier MaureenZOU antopost


text:
First Start: Größe des Soundboard Fensters im Dock

Das Dock rechts unten soll so in der Breite eingestellt werden, dass bei aktiviertem Soundboard immer das Ordner-Icon zu sehen ist. Das ist aber abhängig von der Gesamtgröße des Fensters.
Erweiterter Schwierigkeitsgrad: schauen, wie es sich auf Mehrmonitor-Setups verhält und inwiefern Retina-Auflösungen zu Problemen führen..

Duplicate mit #3

authors: mespotine rstockm

repo: Ultraschall/ultraschall-portable


text:
when is new update gonna be?

When is the new update gonna be?

In a few days when everything is done.

repo: UnamSanctam/UnamDownloader


text:
Anonymous variable "_" should not be assigned

On branch master (commit 5bec8870efb4d900e6526852ce032947c3a287ca), the anonymous variable is treated as any other normal variable, and in turn, is also assigned. For instance, let's consider the following fragment and let's analyze it with the Interval and FieldSensitivePointBasedHeap abstract domains.
package main

func f() (int,int) {
		return 1,2
}

func main() {
    x, _ := f()
}

The analysis result for the mainfunction is

where also the variable _ is assigned. This should not happen. Moreover, @olivieriluca I would ask you if you can check (and write some tests about that) if the anonymous variable wrongly follows the same scoping logic of the other variables you have implemented. In particular, I think the anonymous variable should be never scoped, that is:
func main() {
    x, _ := f()
    _, y := f()
}

should be a legal function.

The correct name variable "_" is blank identifier (https://golang.org/doc/effective_go#blank). Yes,  it should be never scoped

Great, thanks!

authors: VincenzoArceri olivieriluca

repo: UniVE-SSV/go-lisa


text:
Error for NC_HDF5_FILE_INFO

Hello,
I'm getting an error that previously had not happened:
libtool: compile:  /opt/openmpi/bin/mpicc -fPIC -DHAVE_CONFIG_H -I. -I.. -I../include -L/o                                                                                pt/openmpi/include -I/opt/hdf5/include -I/opt/openmpi/include -I/opt/hdf5/include -c hdf5f                                                                                ile.c -MD  -fPIC -DPIC -o .libs/libnchdf5_la-hdf5file.o
"hdf5file.c", line 227: error: struct "NC_HDF5_FILE_INFO" has no field "http"
      ncurifree(hdf5_info->http.uri);
                           ^

"hdf5file.c", line 228: error: struct "NC_HDF5_FILE_INFO" has no field "http"
      NC_authfree(hdf5_info->http.auth);
                             ^

2 errors detected in the compilation of "hdf5file.c".
make[2]: *** [Makefile:578: libnchdf5_la-hdf5file.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
"nc4hdf.c", line 2291: warning: integer conversion resulted in a change of sign
      size_t maxobjs = -1;
                       ^

libtool: compile:  /opt/openmpi/bin/mpicc -fPIC -DHAVE_CONFIG_H -I. -I.. -I../include -L/o                                                                                pt/openmpi/include -I/opt/hdf5/include -I/opt/openmpi/include -I/opt/hdf5/include -c hdf5a                                                                                ttr.c -MD -o libnchdf5_la-hdf5attr.o >/dev/null 2>&1
libtool: compile:  /opt/openmpi/bin/mpicc -fPIC -DHAVE_CONFIG_H -I. -I.. -I../include -L/o                                                                                pt/openmpi/include -I/opt/hdf5/include -I/opt/openmpi/include -I/opt/hdf5/include -c nc4in                                                                                fo.c -MD -o libnchdf5_la-nc4info.o >/dev/null 2>&1
libtool: compile:  /opt/openmpi/bin/mpicc -fPIC -DHAVE_CONFIG_H -I. -I.. -I../include -L/o                                                                                pt/openmpi/include -I/opt/hdf5/include -I/opt/openmpi/include -I/opt/hdf5/include -c nc4hd                                                                                f.c -MD -o libnchdf5_la-nc4hdf.o >/dev/null 2>&1
make[2]: Leaving directory '/home/odyhpc/netcdf-c/libhdf5'
make[1]: *** [Makefile:722: all-recursive] Error 1
make[1]: Leaving directory '/home/odyhpc/netcdf-c'
make: *** [Makefile:567: all] Error 2

I noticed #1028 so one wouldn't probably expect this to be an issue. The system is running CentOS8 and netcdf is cloned from the Github page. Thanks.

Same as #1888

Ouch! Didn't notice it.

I am surprised we did not see this during our travis testing. Anyway, those errors
should be conditionally included.

repo: Unidata/netcdf-c


text:
Moves common-proxies docs to its repo

This depends on UnifiedPush/common-proxies#14 to be merged.

@p1gp1g this can now be merged

repo: UnifiedPush/documentation


text:
UI Changes

Would be nice to experiment with the folllowing things:

Hashtags/searches
Permalinking
Rich previews for permalinks (Dynamic preview can be done in SSR)
SSR
Markdown preview for editor
Color in the UI

want to convert these to a single individual issues?
right now we have the:

Search
rich text editor (markdown preview)

no premallinking - this is good to have for sharing
i dont understand whats SSR
maybe we can put the "color in UI" on hold.

close this ticket, because its duplicated in Unirep Social, some of items are already implemented.

authors: ChialiTsai vimwitch

repo: Unirep/Unirep


text:
Using exclude with no include fails

With serverless-python-requirements@2.0.0.
If no include: directive is found, deploy triggers a failure.
https://github.com/UnitedIncome/serverless-python-requirements/blob/7044eeb46c5c0f6c8e74c46d68f36dba5dc6dfff/index.js#L83-L84
Example serverless.yml
package:
  exclude:
    - .git/**

Output
Running SLS_DEBUG=* serverless deploy:
Serverless: Installing required Python packages...
Serverless: Linking required Python packages...

  Type Error ---------------------------------------------

     Cannot read property 'push' of undefined

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

TypeError: Cannot read property 'push' of undefined
    at fse.readdirSync.map.file (/Users/miketheman/project/node_modules/serverless-python-requirements/index.js:83:50)
    at Array.map (native)
    at ServerlessPythonRequirements.linkRequirements (/Users/miketheman/project/node_modules/serverless-python-requirements/index.js:82:40)
    at runCallback (timers.js:666:20)
    at tryOnImmediate (timers.js:639:5)
    at processImmediate [as _immediateCallback] (timers.js:611:5)
From previous event:
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:156:22)
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:95:31)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:23:50)
    at process._tickCallback (internal/process/next_tick.js:109:7)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues

     Please report this error. We think it might be a bug.

  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       7.7.4
     Serverless Version: 1.9.0

Workaround
Add an include: with any file you expect to always be part of your deployment package:
package:
  include:
    - requirements.txt
  exclude:
    - .git/**

May be related to some of the code being discussed in #15 - should probably be split from the docker user PR if relevant.

Hmm. That's curious. Could you provide an example or provide more specific direction on how to duplicate this issue. I can't seem to duplicate the issue. Here's the service I used (simply the default aws-python template with requests & this plugin added. https://gist.github.com/dschep/1f2225b7c075e646357adc7ed3593b85

Tour example needs an active package: exclude statement

Perfect. Thank you. Fixed and released as v2.0.1. Good catch! :+1:

Thanks!

I'm still having this issue in v3.0.5.
I had to add a package: include statement

@cdimitroulas are you using the zip option? looking at the code that code path would have this bug I think.

I have a fix coming up, working on the test case right now.

PR submitted.

authors: cdimitroulas dschep miketheman


text:
Client cannot send named message to server through SendNamedMessage overload

Describe the bug
When using the overload on the class CustomMessagingMessenger.cs:
public static void SendNamedMessage(
      string name,
      List<ulong> clientIds,
      Stream stream,
      string channel = null,
      SecuritySendFlags security = SecuritySendFlags.None)

It will throw an error stating: "Can not send named messages to multiple users as a client" when called from a client. This happens regardless of there only being the server id in the list.
To Reproduce
Steps to reproduce the behavior:

Create a custom named message
Send it from a client to a server, but using the above mentioned overload

Expected behavior
The expected behaviour is that the method will send the method to a server, if the list only contains one id, which matches that of the server.

As I can see from the implemented SendNamedMessage function and the corresponding Send function, it isn't a bug. This behavior seems to be intended.
Nonetheless it would be handy to be able to send CustomNamedMessages from a client to the server?
I guess this is more like a feature request.

@blacktoby At this point in time you can send custom named messages from the client to the server, only the overload with multiple client ids will not even if there is only one id which corresponds to the server. I'm actually using the other (single clientId) overload to send messages from clients to the server right now. The message only gets suppressed when the client itself is a host.

Yes, I agree. Missed this point here.

This can be accomplished with the following overload:
public static void SendNamedMessage(string name, ulong clientId, Stream stream, string channel = null, SecuritySendFlags security = SecuritySendFlags.None)

The clientId should be NetworkingManager.Singleton.ServerClientId

authors: NAMiedema TwoTenPvP blacktoby


text:
test: Rpc queue unit tests for sending, receive, send parameters, and invoking at the appropriate update stage

The following pull request is for the migration of the RPC Queue tests from the netcode-scaledemo over to the MLAPI repository.
This Rpc Queue tests the following (with no network connection):

Sending and Receiving pipeline to validate that both sending and receiving pipelines are functioning properly.
Usage of the ServerRpcParams.Send.UpdateStage and ClientRpcParams.Send.UpdateStage functionality.
Rpcs receive will be invoked at the appropriate NetworkUpdateStage.

One testing cycle will send a ServerRpc and ClientRpc per NetworkUpdateStage.
This test performs two testing cycles.
This test will fail if:
It cannot instantiate the NetworkingManager.
If all tests are not completed within 30 seconds.
If any rpc update stage does not match with the originating value, server received value, and client received value.

@NoelStephensUnity — all the things I annotated are minor :) you wanted me to nit-pick, so I did :P

authors: MFatihMAR NoelStephensUnity


text:
Nodeserver RTA vs Elk Keypad Burglar Alarm

When an alarm is activated and is disarmed from Elk Nodeserver, or an user code is entered from Elk keypad, alarm is silenced and Elk Nodeserver returns to "Ready to Arm". However, Elk keypad is reading "Burglar Alarm" with the zone causing the alarm. As previously discussed, Elk requires the entry of user code twice to return to RTA. Nodeserver, requires two "Disarmed" entries to accept an arm entry.
Previous issue was for an update to documentation. This issue is for Nodeserver displaying RTA before it is ready. See photo & screen shot.

Nodeserver RTA in Alarm.docx

The NS doesn't have control over this, Elk is sending back Ready To Arm so that's what is displayed.  Does the ISY module behave differently?  If so then I can look deeper, but not sure what I can do?

ok

Just not sure what I can do, I will have to put my system in testing mode this weekend if I have time and see what comes thru in the API, but something must be there if ISY module is showing it...

Jim, I wouldn't worry too much about this. I just thought it would be nice to have something to grab your attention. The Arm Up State, Alarm State, & Armed Status match between the Nodeserver and the Elk Module. The difference being, the admin console shows a red flashing light next to Burglar Alarm that is readily visible.
Short of being able to change font color or flash work Burglar Alarm in the Nodeserver, you probably can't do anything and my guess is Polyglot doesn't support any of these options.

I still need to test this to see what the ELK returns and elkm1_lib has available.

Jim, I have a developers version of RP2 and a few other testing apps, I might be able to provide some helpful guidance. I don't know how often you check GitHub si I'll message you from the forum with a link..

I get an email on all replies here.
Do you have a proposal to resolve this issue?

I just updated IoX and PG3x, let me see if the upgrade was successful and we'll take it from there Jim.

Jim, would you like me to find out if Kelly Carter at Elk's Engineering Department to see if has a more recent version of elkm1_lib?

Thanks, but the elkm1_lib is not developed by Elk, unless this guy works for them https://github.com/gwww/elkm1

authors: Securitybill djc0739 jimboca


text:
Fix invalid xacro properties

Addresses #165

Codecov Report

Merging #166 (605bb36) into main (1b121b7) will decrease coverage by 0.12%.
The diff coverage is 0.00%.

@@           Coverage Diff            @@
##            main    #166      +/-   ##
========================================
- Coverage   3.59%   3.46%   -0.13%
========================================
  Files         13      13
  Lines        947     982      +35
  Branches     152     152
========================================
  Hits          34      34
- Misses       843     878      +35
  Partials      70      70

Flag
Coverage Δ

unittests
3.46% <0.00%> (-0.13%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ur_controllers/force_torque_sensor_broadcaster.hpp
0.00% <ø> (ø)

...controllers/scaled_joint_trajectory_controller.hpp
0.00% <0.00%> (ø)

...ontrollers/src/force_torque_sensor_broadcaster.cpp
0.00% <0.00%> (ø)

ur_controllers/src/gpio_controller.cpp
0.00% <0.00%> (ø)

...rollers/src/scaled_joint_trajectory_controller.cpp
0.00% <0.00%> (ø)

...ontrollers/src/speed_scaling_state_broadcaster.cpp
0.00% <0.00%> (ø)

...ver/include/ur_robot_driver/hardware_interface.hpp
0.00% <ø> (ø)

ur_robot_driver/src/hardware_interface.cpp
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ca5971...605bb36. Read the comment docs.


text:
Stats changes

Stats changes.
WARNING

Breaking changes (world area is applied everytime now, unlike before - only on missing areas)
Migrations add a new column.

Changes:

When no areas are found, apply unmatched area
Add world area everytime
Limit stats batches (those errored a loot)
Add amount to quest stats
Tiny optimizations/changes
Add mega energy amount to stats

Solves https://github.com/UnownHash/Golbat/issues/228 https://github.com/UnownHash/Golbat/issues/226

looks good to me assuming tested

I will know better tomorrow night/morning after quests.
Had no errors today, but N did something nasty, and I can't really call those verified results.

so this PR will introduce:

world which contains overall stats
unmatched which contains stats outside of fences
? this means we can't simple sum the countanymore, as it's world = areas + unmatched ? :)

so this PR will introduce:

world which contains overall stats
unmatched which contains stats outside of fences

? this means we can't simple sum the countanymore, as it's world = areas + unmatched ? :)

Under this PR, the total seen is in 'world' rather than sum of all areas.
So sum of all areas where <> 'world' should be equal to everything else.
Given you could get the total before with a sum, I guess the question is whether this is a beneficial change (I think renaming the old world to unmatched makes things clearer - the question is whether having world as well is a net benefit I think?)

A strong argument for having 'world' in this form is that there can easily be double counting with overlapping areas.  This means that the world entry is everything by itself.  This swings it for me.
A sum is not going to be accurate unless you take care to have non-overlapping areas.

Hmm, yeah, that's a correct point. I guess I will get rid of it then, as there won't be an easy way to exclude overlapping areas.

Hmm, yeah, that's a correct point. I guess I will get rid of world then, as there won't be an easy way to exclude overlapping areas.

No, I am saying we should have world because otherwise you can't have a correct count

Wait, right. We get all matched areas under one counts entry. So world is doing own job as it's applied for every bag of areas.
Correct. And is the only way to reliably get the total for the whole of golbat

Either way, stats looks much better than before. Got a bit surprised by amount, but we had event change today, hence numbers are doubled.
Sum of my country fences vs world is showing error at less than 0.0002%. Which is correct, as some of my city fences are a bit greater than country. City counts look good as well.
If there are no other questions, ideas - I'm ready to merge.

authors: Fabio1988 jfberry lenisko

repo: UnownHash/Golbat

license: Unlicense


text:
FLAPI-155: Support referencing the current block.

Make it possible to reference the current block, using a special constant like this. Ensure that the name is blocked from normal use!
This supports a few additional use cases:

access the descriptor block, without having to guess the block name
access an anonymous block recursively
decouples normal recursive calls from the block name

Imported from JIRA
Originally reported by: UnquietCode

Found it: 1cb45dbfbac7

repo: UnquietCode/Flapi


text:
Improve comment consistency

Comments are not overly consistent in style throughout the project.

Thanks for this feedback Mark, I'll take a look as I run through it.

You're welcome! Hopefully it is adopted.

(Referring to pull request I made #3 which does address this issue)

authors: BenTristem Markyroson


text:
BadImageFormatException Error When Using BlueprintEvent

I am encountering an issue when using BlueprintEvent in my project with version 5.3.2. Every time I try to use a BlueprintEvent, I encounter a BadImageFormatException error.
Here is an example of the code where I am using BlueprintEvent:
[UFunction(FunctionFlags.BlueprintEvent)]
public extern void Crouch();

[UFunction(FunctionFlags.BlueprintEvent)]
public extern void AddMappingContext(InputMappingContext mappingContext, ModifyContextOptions options = default);

I have ensured that all my dependencies are up to date and that I have not made significant changes to my code recently.

I updated the code to version 5.4, I still have the same error.

If you use extern, you need to implement it in BP.

authors: OlssonDev PourrezJ

repo: UnrealSharp/UnrealSharp


text:
unstructured-api-tools: Update API generation code to support uploading files as bytes

Summary
This pull request Introduced support for the text_files parameter as documented in Pipelines-and-APIs.
In a similar manner, add support for a binary files FASTApi API parameter. In the simplest form, this is implied by the presence of a file parameter in pipeline_api, e.g.:
def pipeline_api(file, response_type="application/json", ...)

The file argument in this function may be assumed to be the a file-like object. When called from from a FastAPI route, the value will be a UploadFile.file.
If the file argument is present in pipeline_api, two additional args may also be present, though optional: file_content_type and filename . This allows a mechanism for the FASTApi route to essentially pass UploadFile.filename and UploadFilename.content_type to pipeline_api  (without pipeline_api having knowledge of the UploadFile class).
Definition of Done

pipeline_api must have a file or text parameter (or both), otherwise an error is thrown by unstructured_api_tools convert-pipeline-notebooks. If file exists as a parameter in pipeline_api, the FastAPI route should accept files as parameter.

The new pipeline_api parameters are supported: file, file_content_type, and filename.

If a single file is specified in the request, the response is simply returned as the result from pipeline_api.

However, if multiple files are specified in the request, a multipart/mixed response should be constructed where each part is the result of a pipeline_api call.

When multiple files are provided in a request, if the Accept header is specified it must be Accept: multipart/mixed .

Verify that the generated pipeline-sec-filings API continues to work as expected with plain text files.

The version in unstructured-api-tools is incremented.

A test notebook with a simple pipeline_api definition is provided showing the ability to handle a POSTed pdf which is saved to disk. Any json response is acceptable. It is tested by saving the notebook in the /pipeline-notebooks directory of a pipeline-sec-filings checkout, running make generate-api , and then PYTHONPATH=. uvicorn prepline_sec_fiings.api.<API-NAME>:app --reload.

Hi, a question about this:
Must the m_params always come before other optional params (like response_type or file_content_type, etc.) or they can be in any order in the pipeline_api function definition?
CC: @cragwolfe

@asymness , any order should be fine. If it is substantially easier to enforce ordering, that could be OK too as long as there are clear error messages when trying to generate an API on how to fix if they are out of order. Generally though, I think it is fair to assume/enforce that the first parameter is file or text.

authors: MthwRobinson asymness cragwolfe

repo: Unstructured-IO/community


text:
Python L1 code follow up for MATLAB L1 code v1.0

Bulk update for aligning Python L1 code with MATLAB L1 code.

I appreciate your help, @mike-ivs. Your insight is the key to finding the root of issues.


text:
Test GPS examples on new board

Adding the board into the Arduino IDE and testing some of the examples. I uploaded some of the examples to the 2022-2023 repo.
Once everything was installed properly, this process went pretty smoothly. The GPS took a while to initialize, but that is to be expected. In the code, I'd like to change some things. I will create issues that we can work on later.

Create UofA image on startup
Delete HDOP
Add number of sattelites
Add units to altitude
Convert speed to MPH
Test with GPS antenna used last time

Actually decided against UofA image on startup. It would take up too much resources. Created progress bar instead for initialization.


text:
Issues when impoting multiple json files with same name/keys

Potentially related to: https://github.com/Updater/node-sass-json-importer/issues/21
I've noticed some issues with the wrong file being read/imported if the project has multiple files with the same name, like this:
[module1/
----[config.json
----[module1.scss
[module1/
----[config.json
----[module2.scss
I've also noticed issues when using multiple files with different names when the contents begin with the same first key, like this:
[module1/
----[module1.json
----[module1.scss
[module1/
----[module2.json
----[module2.scss
Where both module1.json and module2.json start like this:
{
    "default": {
        ...
    }
}

Sometimes the wrong json file will be imported, which obviously breaks the compilation.
My ideal setup would be for all my json files to be named config.json and begin with the default key, but as it stands I have to have unique names for both the file name and opening key.

Sorry we never followed up on this @esr360! If its still something you'd like to try to fix, please re-open with a reproducible scenario.

repo: Updater/node-sass-json-importer