text:
Run on H100 cluster "slurm+enroot+pyxis"

I tried to run ghcr.io/dlwh/levanter-cuda-cache:latest from Oct 28, 2023 on cluster with H100s thru "slurm+enroot+pyxis". I came up with the following batch file:
#!/bin/bash

#SBATCH -A <account>
#SBATCH -p <partition>
#SBATCH -J levanter.test
#SBATCH -N 1
#SBATCH -t 00:05:00

CONT="ghcr.io#dlwh/levanter-cuda-cache:latest"

srun --ntasks-per-node=1 --container-image="$CONT" --mpi=pmi2 --export NCCL_DEBUG=TRACE --export WANDB_MODE=offline python -m levanter.main.train_lm --config /opt/levanter/config/gpt2_small_fast_wiki.yaml --trainer.per_device_parallelism 16 --trainer.train_batch_size 3

The issue I saw:
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/levanter/src/levanter/main/train_lm.py", line 186, in <module>
    levanter.config.main(main)()
  File "/opt/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/opt/levanter/src/levanter/main/train_lm.py", line 77, in main
    config.trainer.initialize(config)
  File "/opt/levanter/src/levanter/trainer.py", line 533, in initialize
    self.distributed.initialize()
  File "/opt/levanter/src/levanter/distributed.py", line 248, in initialize
    device_ids = LevanterSlurmCluster.get_local_device_ids_for_process()
  File "/opt/levanter/src/levanter/distributed.py", line 108, in get_local_device_ids_for_process
    subprocess.check_output(["scontrol", "show", "hostnames", node_list], input=b"")
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'scontrol'

Looks like the train script thinks that it is run on the node and slurm API is available, but not inside the container.

After updating, I run the same batch script with latest laventer docker from Nov 03, 2023. The previous problem was fixed, but now I see another one:
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/levanter/src/levanter/main/train_lm.py", line 186, in <module>
    levanter.config.main(main)()
  File "/opt/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/opt/levanter/src/levanter/main/train_lm.py", line 77, in main
    config.trainer.initialize(config)
  File "/opt/levanter/src/levanter/trainer.py", line 533, in initialize
    self.distributed.initialize()
  File "/opt/levanter/src/levanter/distributed.py", line 276, in initialize
    device_ids = LevanterSlurmCluster.get_local_device_ids_for_process()
  File "/opt/levanter/src/levanter/distributed.py", line 111, in get_local_device_ids_for_process
    local_node_index = node_list.index(local_node)
ValueError: '<cluster-node>' is not in list

 is internal name of cluster node

i'm having a lot of trouble getting anything multinode to work (jax or not) on our slurm cluster. would you do me a favor and send me a hello world jax thing that works for you?

i got the clusters to detect, so i can mess with this a bit. I'd still like pointers on how you all run docker-on-slurm since i don't have a ton of experience with that.

try one more time? i got rid of our custom SlurmCluster env detector for this branch to see if it that fixes for you, so now it's vanilla jax.distributed...

JAX core container (ghcr.io/nvidia/jax:<version>) is a self contained. To test it, you can simple run test-jax.sh script:
test-jax.sh -b gpu

Thanks, I was asking more for the slurm incantation, but I can see you all
have a fancy plug-in thing that we don’t have, so it wouldn’t translate
anyway.
On Mon, Nov 6, 2023 at 5:12 PM Vladislav @.***> wrote:

JAX core container (ghcr.io/nvidia/jax:) is a self contained. To
test it, you can simple run test-jax.sh script:
test-jax.sh -b gpu
—
Reply to this email directly, view it on GitHub
https://github.com/stanford-crfm/levanter/issues/363#issuecomment-1797113943,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAACLIJHVCMHPV7NRAGQLN3YDGDIHAVCNFSM6AAAAAA677EIGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXGEYTGOJUGM
.
You are receiving this because you commented.Message ID:
@.***>

I was able to run it on a single node with 8xH100. Results looks reasonable:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                  learning_rate 0.0001
wandb:                    num_devices 8
wandb:                      num_hosts 1
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.09157
wandb: throughput/examples_per_second 349.45657
wandb:           throughput/hook_time 0.00217
wandb:        throughput/loading_time 5e-05
wandb:   throughput/tokens_per_second 357843.52947
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan

But with multinode run (2 nodes atually), I see this error:
2023-11-06 18:50:43,981 INFO worker.py:1664 -- Connected to Ray cluster. View the dashboard at ^[[1m^[[32m127.0.0.1:8265 ^[[39m^[[22m
2023-11-06 18:50:44,934 INFO node.py:1000 -- Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. Have you started Ray instsance using `ray start` or `ray.init`?
wandb: Tracking run with wandb version 0.15.12
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
2023-11-06T18:50:46 - 0 - levanter.data.shard_cache - shard_cache.py:970 - INFO :: Loading cache from cache/validation
2023-11-06T18:50:47 - 0 - levanter.data.shard_cache - shard_cache.py:970 - INFO :: Loading cache from cache/train
2023-11-06T18:50:47 - 0 - levanter.checkpoint - checkpoint.py:368 - WARNING :: No checkpoints found in checkpoints/ocykpdsx
2023-11-06T18:50:48 - 0 - __main__ - train_lm.py:134 - INFO :: No checkpoint found. Starting from scratch.
2023-11-06 18:50:54,945 INFO node.py:1000 -- Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. Have you started Ray instsance using `ray start` or `ray.init`?
2023-11-06 18:51:04,957 INFO node.py:1000 -- Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. Have you started Ray instsance using `ray start` or `ray.init`?
2023-11-06 18:51:14,968 INFO node.py:1000 -- Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. Have you started Ray instsance using `ray start` or `ray.init`?
2023-11-06 18:51:24,980 INFO node.py:1000 -- Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. Have you started Ray instsance using `ray start` or `ray.init`?
2023-11-06 18:51:34,991 INFO node.py:1000 -- Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. Have you started Ray instsance using `ray start` or `ray.init`?
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/levanter/src/levanter/main/train_lm.py", line 186, in <module>
    levanter.config.main(main)()
  File "/opt/levanter/src/levanter/config.py", line 84, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/opt/levanter/src/levanter/main/train_lm.py", line 77, in main
    config.trainer.initialize(config)
  File "/opt/levanter/src/levanter/trainer.py", line 535, in initialize
    self.ray.initialize()
  File "/opt/levanter/src/levanter/distributed.py", line 288, in initialize
    auto_ray_cluster(address=self.address, start_workers=self.start_workers)
  File "/opt/levanter/src/levanter/distributed.py", line 233, in auto_ray_cluster
    ray.init(address=address, namespace=namespace, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 1629, in init
    _global_node = ray._private.node.Node(
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/node.py", line 199, in __init__
    node_ip_address = self._wait_and_get_for_node_address()
  File "/usr/local/lib/python3.10/dist-packages/ray/_private/node.py", line 1007, in _wait_and_get_for_node_address
    raise ValueError(
ValueError: Can't find a `node_ip_address.json` file from /tmp/ray/session_2023-11-06_18-50-42_357600_1906786. for 60 seconds. A ray instance hasn't started. Did you do `ray start` or `ray.init` on this host?
2023-11-06 18:51:44,514 VINFO scripts.py:1085 -- Killed `^[[1m/usr/local/lib/python3.10/dist-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/tmp/ray/session_2023-11-06_18-50-42_312930_2583834/sockets/raylet --store_socket_

In multinode it was looking for a file node_ip_address.json. I assume, that it contains some information about nodes involved in the compute.
Btw, what is Ray?

hrm, ok, that's something.
Ray is a distributed computing framework (https://www.ray.io/). We use it for preprocessing, but we shouldn't need it for this test. (I don't know what's going on but we can cross that bridge we come to it.) Can you add --trainer.ray.auto_start_cluster false and see if that fixes?

Interesting, I added the flag you mentioned above (to turn ray off). It helped with Ray error.
The output is a very curios reading:
train:   0%|          | 1/200 [01:24<4:40:36, 84.60s/it, loss=11]
train:   0%|          | 1/200 [01:24<4:2023-11-06T20:43:38 - 0 - levanter.callbacks - callbacks.py:57 - INFO :: validation loss: nan
train: 100%|██████████| 200/200 [01:45<02023-11-06T20:43:58 - 0 - levanter.callbacks - callbacks.py:57 - INFO :: validation loss: nan
wandb: Waiting for W&B process to finish... (success).s, loss=nan]
train: 100%|██████████| 200/200 [01:45<02023-11-06T20:43:58 - 0 - levanter.callbacks - callbacks.py:57 - INFO :: validation loss: nan
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb:                    global_step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:                  learning_rate ▄███████▇▇▇▇▇▆▆▆▆▅▅▅▅▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁▁
wandb:            throughput/duration █▄▂▂▂▂▂▂▃▃▂▂▂▂▃▃▂▃▂▃▃▂▃▂▃▂▂▂▂▃▂▂▁▁▂▂▃▁▁▂
wandb: throughput/examples_per_second ▁▅▇▇▇▆▇▇▆▆▇▇▇▇▆▆▇▆▇▆▆▇▆▇▅▇▇▇▇▆▇▆▇█▇▇▆█▇▇
wandb:           throughput/hook_time █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        throughput/loading_time ▂▂▂▁▁▂▂▁█▂▂▂█▁▁▇▂▇▂▃▂▂▂▂▂▂▂▂▂▁▂█▂▂▃▂▁▃▂▂
wandb:   throughput/tokens_per_second ▁▅▇▇▇▆▇▇▆▆▇▇▇▇▆▆▇▆▇▆▆▇▆▇▅▇▇▇▇▆▇▆▇█▇▇▆█▇▇
wandb:        throughput/total_tokens ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                  learning_rate 0.0001
wandb:                    num_devices 8
wandb:                      num_hosts 1
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.09402
wandb: throughput/examples_per_second 340.36651
wandb:           throughput/hook_time 0.00177
wandb:        throughput/loading_time 5e-05
wandb:   throughput/tokens_per_second 348535.30354
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /opt/levanter/wandb/offline-run-20231106_204209-35lc1hod
wandb: Find logs at: ./wandb/offline-run-20231106_204209-35lc1hod/logs
wandb: WARNING Step only supports monotonically increasing values, use define_metric to set a custom x axis. For details see: https://wandb.me/define-metric
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'train/loss': nan, 'global_step': 199, '_timestamp': 1699332237.854266}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'learning_rate': 0.00010005665535572916, '_timestamp': 1699332237.8543367}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'eval/loss': nan, '_timestamp': 1699332238.0720947}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/total_tokens': 6520832, '_timestamp': 1699332238.0735772}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/examples_per_second': 340.3665073656403, 'throughput/tokens_per_second': 348535.30354241567, 'throughput/duration': 0.09401630098000169, '_timestamp': 1699332238.0736635}).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb:                    global_step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:                  learning_rate ▄███████▇▇▇▇▇▆▆▆▆▅▅▅▅▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁▁
wandb:            throughput/duration █▂▃▃▁▂▂▂▂▂▂▃▃▂▃▃▃▄▃▂▂▃▃▄▃▂▂▂▁▃▂▂▂▃▂▂▃▂▂▂
wandb: throughput/examples_per_second ▁▆▆▆█▇▇▇▇▇▇▆▆▇▆▅▆▅▆▇▇▆▆▅▆▇▆▇▇▆▇▇▇▆▆▇▆▇▇▇
wandb:           throughput/hook_time █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        throughput/loading_time ▂▃▂▂▁▂▂▁▂▂▃▂▃▃▃▃▂▃▃▃▂▂▃▃▃▂▂▃▂█▂▂▂▂▁▂▃▂▃▂
wandb:   throughput/tokens_per_second ▁▆▆▆█▇▇▇▇▇▇▆▆▇▆▅▆▅▆▇▇▆▆▅▆▇▆▇▇▆▇▇▇▆▆▇▆▇▇▇
wandb:        throughput/total_tokens ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                  learning_rate 0.0001
wandb:                    num_devices 8
wandb:                      num_hosts 1
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.09375
wandb: throughput/examples_per_second 341.3267
wandb:           throughput/hook_time 0.00193
wandb:        throughput/loading_time 5e-05
wandb:   throughput/tokens_per_second 349518.53681
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /opt/levanter/wandb/offline-run-20231106_204209-dms9ouev
wandb: Find logs at: ./wandb/offline-run-20231106_204209-dms9ouev/logs
wandb: WARNING Step only supports monotonically increasing values, use define_metric to set a custom x axis. For details see: https://wandb.me/define-metric
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'train/loss': nan, 'global_step': 199, '_timestamp': 1699332238.3478193}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'learning_rate': 0.00010005665535572916, '_timestamp': 1699332238.347889}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'eval/loss': nan, '_timestamp': 1699332238.5657177}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/total_tokens': 6520832, '_timestamp': 1699332238.5671923}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/examples_per_second': 341.32669610108235, 'throughput/tokens_per_second': 349518.5368075083, 'throughput/duration': 0.09375182300573215, '_ti

Is this output expected?
I see 2 similar outputs like from single-node run. I expected to see some combined output: num_deviced = 16 or similar.

Yeah I'd say they're not finding each other, which makes me think the SLURM
env variables aren't being set? Does the slurm auto-cluster stuff work
generally for you (in e.g. paxml)?
On Mon, Nov 6, 2023 at 8:53 PM Vladislav @.***> wrote:

Interesting, I added the flag you mentioned above (to turn ray off). It
helped with Ray error.
The output is a very curios reading:
train:   0%|          | 1/200 [01:24<4:40:36, 84.60s/it, loss=11]
train:   0%|          | 1/200 [01:24<4:2023-11-06T20:43:38 - 0 - levanter.callbacks - callbacks.py:57 - INFO :: validation loss: nan
train: 100%|██████████| 200/200 [01:45<02023-11-06T20:43:58 - 0 - levanter.callbacks - callbacks.py:57 - INFO :: validation loss: nan
wandb: Waiting for W&B process to finish... (success).s, loss=nan]
train: 100%|██████████| 200/200 [01:45<02023-11-06T20:43:58 - 0 - levanter.callbacks - callbacks.py:57 - INFO :: validation loss: nan
wandb: Waiting for W&B process to finish... (success).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb:                    global_step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:                  learning_rate ▄███████▇▇▇▇▇▆▆▆▆▅▅▅▅▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁▁
wandb:            throughput/duration █▄▂▂▂▂▂▂▃▃▂▂▂▂▃▃▂▃▂▃▃▂▃▂▃▂▂▂▂▃▂▂▁▁▂▂▃▁▁▂
wandb: throughput/examples_per_second ▁▅▇▇▇▆▇▇▆▆▇▇▇▇▆▆▇▆▇▆▆▇▆▇▅▇▇▇▇▆▇▆▇█▇▇▆█▇▇
wandb:           throughput/hook_time █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        throughput/loading_time ▂▂▂▁▁▂▂▁█▂▂▂█▁▁▇▂▇▂▃▂▂▂▂▂▂▂▂▂▁▂█▂▂▃▂▁▃▂▂
wandb:   throughput/tokens_per_second ▁▅▇▇▇▆▇▇▆▆▇▇▇▇▆▆▇▆▇▆▆▇▆▇▅▇▇▇▇▆▇▆▇█▇▇▆█▇▇
wandb:        throughput/total_tokens ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                  learning_rate 0.0001
wandb:                    num_devices 8
wandb:                      num_hosts 1
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.09402
wandb: throughput/examples_per_second 340.36651
wandb:           throughput/hook_time 0.00177
wandb:        throughput/loading_time 5e-05
wandb:   throughput/tokens_per_second 348535.30354
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /opt/levanter/wandb/offline-run-20231106_204209-35lc1hod
wandb: Find logs at: ./wandb/offline-run-20231106_204209-35lc1hod/logs
wandb: WARNING Step only supports monotonically increasing values, use define_metric to set a custom x axis. For details see: https://wandb.me/define-metric
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'train/loss': nan, 'global_step': 199, '_timestamp': 1699332237.854266}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'learning_rate': 0.00010005665535572916, '_timestamp': 1699332237.8543367}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'eval/loss': nan, '_timestamp': 1699332238.0720947}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/total_tokens': 6520832, '_timestamp': 1699332238.0735772}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/examples_per_second': 340.3665073656403, 'throughput/tokens_per_second': 348535.30354241567, 'throughput/duration': 0.09401630098000169, '_timestamp': 1699332238.0736635}).
wandb: - 0.000 MB of 0.000 MB uploaded (0.000 MB deduped)
wandb: Run history:
wandb:                    global_step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:                  learning_rate ▄███████▇▇▇▇▇▆▆▆▆▅▅▅▅▄▄▄▃▃▃▃▂▂▂▂▂▁▁▁▁▁▁▁
wandb:            throughput/duration █▂▃▃▁▂▂▂▂▂▂▃▃▂▃▃▃▄▃▂▂▃▃▄▃▂▂▂▁▃▂▂▂▃▂▂▃▂▂▂
wandb: throughput/examples_per_second ▁▆▆▆█▇▇▇▇▇▇▆▆▇▆▅▆▅▆▇▇▆▆▅▆▇▆▇▇▆▇▇▇▆▆▇▆▇▇▇
wandb:           throughput/hook_time █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        throughput/loading_time ▂▃▂▂▁▂▂▁▂▂▃▂▃▃▃▃▂▃▃▃▂▂▃▃▃▂▂▃▂█▂▂▂▂▁▂▃▂▃▂
wandb:   throughput/tokens_per_second ▁▆▆▆█▇▇▇▇▇▇▆▆▇▆▅▆▅▆▇▇▆▆▅▆▇▆▇▇▆▇▇▇▆▆▇▆▇▇▇
wandb:        throughput/total_tokens ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                  learning_rate 0.0001
wandb:                    num_devices 8
wandb:                      num_hosts 1
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.09375
wandb: throughput/examples_per_second 341.3267
wandb:           throughput/hook_time 0.00193
wandb:        throughput/loading_time 5e-05
wandb:   throughput/tokens_per_second 349518.53681
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /opt/levanter/wandb/offline-run-20231106_204209-dms9ouev
wandb: Find logs at: ./wandb/offline-run-20231106_204209-dms9ouev/logs
wandb: WARNING Step only supports monotonically increasing values, use define_metric to set a custom x axis. For details see: https://wandb.me/define-metric
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'train/loss': nan, 'global_step': 199, '_timestamp': 1699332238.3478193}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'learning_rate': 0.00010005665535572916, '_timestamp': 1699332238.347889}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'eval/loss': nan, '_timestamp': 1699332238.5657177}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/total_tokens': 6520832, '_timestamp': 1699332238.5671923}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/examples_per_second': 341.32669610108235, 'throughput/tokens_per_second': 349518.5368075083, 'throughput/duration': 0.09375182300573215, '_ti

Is this output expected?
I see 2 similar outputs like from single-node run. I expected to
see some combined output: num_deviced = 16 or similar.

—
Reply to this email directly, view it on GitHub
https://github.com/stanford-crfm/levanter/issues/363#issuecomment-1797821548,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAACLIM7B2MZIW4JDKYVLODYDG5FDAVCNFSM6AAAAAA677EIGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXHAZDCNJUHA
.
You are receiving this because you commented.Message ID:
@.***>

This is a multi-node run of T5X on our CI cluster:
https://github.com/NVIDIA/JAX-Toolbox/blob/f6aff1635070d94afaa2afd775e4fba34185cdb1/.github/workflows/_test_t5x.yaml#L76C23-L76C23

ok thanks! so looks like slurm discovery works for you with t5x. I don't really understand what's different. It should just be using the (same?) jax.distributed.init call?
I'll try to see what's going on on our end... we don't have pyxis and multinode is just totally broken on our cluster, so it's a bit of a pain to debug.

I have 2 qq:

Is there a way to change the log level to get more messages in the stdout?
Here I do see that you call jax.distributed.init (https://github.com/stanford-crfm/levanter/blob/main/src/levanter/distributed.py#L253), but don't see related log message in output. For instance, in output stream I

I'm trying to understanding how Levanter handles multi-node but it looks like the new container is built off a branch other than main, as in main we are still using scontrol. Could you please point me to the hotfix branch?

Yu-Hang, I think the branch with fix is fix_slurm_build

Thanks!
https://github.com/stanford-crfm/levanter/blob/fix_slurm_build/src/levanter/distributed.py#L253 is the branch as @DwarKapex says. I can force logger initialization to come before distributed.init

rebuilding an image that sets INFO logging early

Agree, that will be a long process. Run the latest one. The output is the same

i just pushed another. I don't think it had been updated by the time you wrote? But I don't know if it's going to work better. still trying to get hello-world multinode to run

OK, let take a closer look:
Running on hosts: <node1> <node2>
pyxis: imported docker image: ghcr.io#dlwh/levanter-cuda-cache:latest
pyxis: imported docker image: ghcr.io#dlwh/levanter-cuda-cache:latest
INFO:jax._src.distributed:JAX distributed initialized with visible devices: 0
INFO:jax._src.distributed:Connecting to JAX distributed service on <node1>:62634
INFO:jax._src.distributed:JAX distributed initialized with visible devices: 0
INFO:jax._src.distributed:Starting JAX distributed service on <node1>:62634
INFO:jax._src.distributed:Connecting to JAX distributed service on <node1>:62634
INFO:levanter.distributed:Initialized jax.distributed with 8 devices, 1 hosts, coordinator_address=None, process_id=None
INFO:levanter.distributed:Initialized jax.distributed with 8 devices, 1 hosts, coordinator_address=None, process_id=None
INFO:levanter.trainer:Setting run id to 2zp1ys3l
INFO:levanter.logging:Setting wandb code_dir to /opt/levanter
INFO:levanter.trainer:Setting run id to 6chr5bjy
INFO:levanter.logging:Setting wandb code_dir to /opt/levanter

Hm, interesting: according to logs, the second node was not involved at all, both runs happened on the the

this makes me think that the SlurmCluster class isn't sniffing the env variables correctly? WDYT?

@DwarKapex wait, just double checking, are you still doing -N 1?

@dlwh, Nope, I'm doing -N2

could you do the dumb thing of running python -c "import jax; jax.distributed.initialize(); print(jax.devices())" just so we can narrow it down

This is what I found on a 2-node job on our CI cluster: (also note that we recommend 1 process per GPU, rather than 1 process per node)
$ salloc -N 2  --exclusive -t 04:00:00
$ srun --tasks-per-node=8  --gpus-per-node=8 --container-name=test --container-image=ghcr.io#dlwh/levanter-cuda-cache:latest bash -c 'echo $SLURM_STEP_NODELIST'
pyxis: imported docker image: ghcr.io#dlwh/levanter-cuda-cache:latest
pyxis: imported docker image: ghcr.io#dlwh/levanter-cuda-cache:latest
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
compute-permanent-node-[905,954]
$ srun --tasks-per-node=8  --gpus-per-node=8 --container-name=test --container-image=ghcr.io#dlwh/levanter-cuda-cache:latest python -c 'import jax.distributed; jax.distributed.initialize(); print(jax.devices());'
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]
[cuda(id=0), cuda(id=1), cuda(id=2), cuda(id=3), cuda(id=4), cuda(id=5), cuda(id=6), cuda(id=7), cuda(id=8), cuda(id=9), cuda(id=10), cuda(id=11), cuda(id=12), cuda(id=13), cuda(id=14), cuda(id=15)]

Apparently, things appear fine on our CI cluster, but something went wrong when Vlad ran it on Eos.

yep, i didn't realize that when i started, and i made a few assumptions that it was 1 process per node in a few places (mostly w/ ray though)

(it is my stance that the slurm sniffer should work even if people do it the other way, fwiw)

(it is my stance that the slurm sniffer should work even if people do it the other way, fwiw)

agree

Quick update/summary.
The command below:

$>  salloc -N 2 --exclusive -t 01:00:00
$>  srun --tasks-per-node=8  --gpus-per-node=8 --container-name=test --container-image=ghcr.io#dlwh/levanter-cuda-cache:latest python -c 'import jax.distributed; jax.distributed.initialize(); print(jax.devices());'

showed that jax sees all 16 GPUs (see message from Yu-Hang above).
Today I run the similar cmd, but with levanter training:
$>  salloc -N 2 --exclusive -t 01:00:00
$> srun --tasks-per-node=8  --container-name=test --container-image=ghcr.io#dlwh/levanter-cuda-cache:latest python -m levanter.main.train_lm --config /opt/levanter/config/gpt2_small_fast_wiki.yaml --trainer.per_device_parallelism 1 --trainer.train_batch_size 32 --trainer.ray.auto_start_cluster false

The output is
INFO:levanter.distributed:Initialized jax.distributed with 8 devices, 1 hosts, coordinator_address=None, process_id=None

So, looks like levanter sees only 1 host with 8 GPUs, but jax sees all 16 GPUs

We can reach the point, where I create a local repo that syncs with git automatically, build the docker, push to our local registry, run single-node test (to check it's not broken) and ping me to test it in multi-node env.
Will that work?

wait, is it significant that you left out --gpus-per-node=8 in the second run?

ugh, i think i figured it out. jax.devices() is getting called at the top level in a random script i left hanging out in the src directory, and it's getting invoked before jax.distributed.initialize is called, which ruins it.

can you try ghcr.io/dlwh/levanter-cuda-cache:2023-11-14.2 . I think it might be ok, but slurm is being weird on my end.

Looks like the last container works:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                    num_devices 16
wandb:                      num_hosts 16
wandb:            optim/learning_rate 0.0001
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.0608
wandb: throughput/examples_per_second 526.30307
wandb:           throughput/hook_time 0.00239
wandb:        throughput/loading_time 2e-05
wandb:   throughput/tokens_per_second 538934.33868
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /opt/levanter/wandb/offline-run-20231115_130958-iupzsad2
wandb: Find logs at: ./wandb/offline-run-20231115_130958-iupzsad2/logs
wandb: WARNING Step only supports monotonically increasing values, use define_metric to set a custom x axis. For details see: https://wandb.me/define-metric
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'train/loss': nan, 'global_step': 199, '_timestamp': 1700082628.9168425}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'optim/learning_rate': 0.00010005665535572916, '_timestamp': 1700082628.9169188}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'eval/loss': nan, '_timestamp': 1700082629.0368288}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/total_tokens': 6520832, '_timestamp': 1700082629.03706}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/examples_per_second': 526.303065112598, 'throughput/tokens_per_second': 538934.3386753004, 'throughput/duration': 0.0608014699537307, '_timestamp': 1700082629.0371096}).

Nan train loss is worrying though
On Wed, Nov 15, 2023 at 1:11 PM Vladislav @.***> wrote:

Looks like the last container works:
wandb: Run summary:
wandb:                        backend gpu
wandb:                      eval/loss nan
wandb:                    global_step 199
wandb:                    num_devices 16
wandb:                      num_hosts 16
wandb:            optim/learning_rate 0.0001
wandb:                parameter_count 124439808
wandb:            throughput/duration 0.0608
wandb: throughput/examples_per_second 526.30307
wandb:           throughput/hook_time 0.00239
wandb:        throughput/loading_time 2e-05
wandb:   throughput/tokens_per_second 538934.33868
wandb:        throughput/total_tokens 6520832
wandb:                     train/loss nan
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /opt/levanter/wandb/offline-run-20231115_130958-iupzsad2
wandb: Find logs at: ./wandb/offline-run-20231115_130958-iupzsad2/logs
wandb: WARNING Step only supports monotonically increasing values, use define_metric to set a custom x axis. For details see: https://wandb.me/define-metric
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'train/loss': nan, 'global_step': 199, '_timestamp': 1700082628.9168425}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'optim/learning_rate': 0.00010005665535572916, '_timestamp': 1700082628.9169188}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'eval/loss': nan, '_timestamp': 1700082629.0368288}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/total_tokens': 6520832, '_timestamp': 1700082629.03706}).
wandb: WARNING (User provided step: 199 is less than current step: 200. Dropping entry: {'throughput/examples_per_second': 526.303065112598, 'throughput/tokens_per_second': 538934.3386753004, 'throughput/duration': 0.0608014699537307, '_timestamp': 1700082629.0371096}).
—
Reply to this email directly, view it on GitHub
https://github.com/stanford-crfm/levanter/issues/363#issuecomment-1813267472,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAACLIKJG42KKCNO5REJA33YEUVZ7AVCNFSM6AAAAAA677EIGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGI3DONBXGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

ok i think we're declaring victory here?!?!?

authors: DwarKapex dlwh yhtang

repo: stanford-crfm/levanter


text:
torch.cat consume large amount of memory

Dear author,
I saw that in `/home/dayu/convir/ColBERTv1/colbert/indexing/faiss.py", line 35 , this step
sample = torch.cat(sample).float().numpy()

will consume large amount of memory, which my machine cannot handle. Is there any alternative solution for this?
Best,
Dayu

Lower precision such as float16 might help

source: gharchive/issue

repo: stanford-futuredata/ColBERT

license: MIT

license_type: permissive

license_source: github-api


text:
Auto trigger does not work

When we moved from snowboy to mycroft-precise we lost the voice activity detection, and with that the ability to autotrigger when the user replies to a question.

Fixed by #62

repo: stanford-oval/almond-gnome


text:
BootstrapFewShot failing due to lm.copy for AzureOpenAI

Summary
BootstrapFewShot failing due to lm.copy for AzureOpenAI, likely due to positional argument issue from recent changes.
Environment

code: 98304a2eb9d1dddaaa846e30258cd5d8fc6b5d8d
model azure openai

Error message

during BootstrapFewShot run, code errored out due to missing arguments, api_version

>[dspy/dsp/modules/lm.py](https://file+.vscode-resource.vscode-cdn.net/Users/insop/Projects/Github/NLP/dspy/dsp/modules/lm.py)(106)copy()
    102         model = kwargs.pop('model')
    105
--> 106         return self.__class__(model, **kwargs)

Analysis
AzureOpenAI expects positional arguments, so it is failing.
I will add my fixes and see if that will impact other models.
class AzureOpenAI(LM):
...

    def __init__(
        self,
        api_base: str,
        api_version: str,
        model: str = "gpt-3.5-turbo-instruct",
        api_key: Optional[str] = None,
        model_type: Literal["chat", "text"] = "chat",
        **kwargs,
    ):

Resolved? Closing but lmk @insop , thanks a lot btw

Resolved? Closing but lmk @insop , thanks a lot btw

Thank you @okhat.

Update.
Related to the issue (https://github.com/stanfordnlp/dspy/issues/543) that caused by the PR (https://github.com/stanfordnlp/dspy/pull/522) from this issue:
I tried to reproduce the issue I ran into earlier, https://github.com/stanfordnlp/dspy/issues/521, but I am not able to reproduce it.
The strange thing is that the code gets to this part as part of BootstrapFewShot, but it doesn't seem like copy() is called. It seems as if something to do with cache, but I am not able to find it.
https://github.com/stanfordnlp/dspy/blob/669ecd7e04431a0e890ca61c60afafcae1544517/dsp/modules/lm.py#L104

authors: insop okhat

repo: stanfordnlp/dspy


text:
Usage for different languages

I am attempting to use this package for Chinese language segmenting and tagging. As #11 is 6 months old with no responses, I decided to open another issue.
I have downloaded stanford-chinese-corenlp-2018-02-27-model.jar, and I have the segmenter working for English. I am not sure how to specify a model .jar to use, and I cannot find a tutorial anywhere online. Some help and/or more in-depth walkthrough of how to use the segmenter would be much appreciated.

I also struggle with this issue. My solution was to run my own Stanford-Server.
It's really simple. Just follow the instructions given in this issue from a sister project: https://github.com/nltk/nltk/wiki/Stanford-CoreNLP-API-in-NLTK
There is even a detailled description for setting up a Chinese server model (but you have to first get some of the downloads from the top of the page):
https://github.com/nltk/nltk/wiki/Stanford-CoreNLP-API-in-NLTK#chinese
When the server is running, you can apply (in my case for german) (the endpoint is of course your server address with the correct port):
text= "Donald Trump besuchte Angela Merkel in Berlin."

with corenlp.CoreNLPClient(start_server=False, endpoint='http://localhost:9002') as client:
    ann = client.annotate(text)

sentence = ann.sentence[0]
print(sentence)

Oh! And don't bother because of the really, really, really slow start-up phase of the server and the first query. In this moment everything has to be loaded. The second query then is fast as hell. :)

authors: GrazingScientist tigerninjaman

repo: stanfordnlp/python-stanford-corenlp


text:
BEPINEX issue

After extracting bepinex core files game doesnt launch.
Maybe try melon loader?

the game will not start because of the file winhttp.dll

and it works with all the main staff

and it works with all the main staff

what should i do cuz without winhttp.dll it doesnt work

doorstop is patched, we are working on a solution


text:
fix(Icons): fixing errors with brand icons

Updating stop-color and stop-opacity to stopColor and stopOpacity.

Warnings

:warning:
There are no updates provided to CHANGELOG. Ensure there are no publicly visible changes introduced by this PR.

  Generated by :no_entry_sign: dangerJS

source: gharchive/pull-request

authors: DustyTheBot codepretty

repo: stardust-ui/react


text:
Fix first quiz example

In the first example, there is no answer2 signal, resulting in an error on the console.

Merged, thanks!

authors: bencroker datashaman

repo: starfederation/datastar


text:
Stargate docker image: Environment variable to enable materialized views

I need an environment variable that I can use to enable materialized views for testing; ENABLE_MATERIALIZED_VIEWS.

I believe this can be enabled by passing a cassandra.yaml using the system property stargate.unsafe.cassandra_config_path:
JAVA_OPTS="-Dstargate.unsafe.cassandra_config_path=/path/to/cassandra.yml" starctl ...

cassandra.yaml:
# ...
materialized_views_enabled: true

Many thanks! I'll try it later today

yes, that field alone will work (I justed tested it with C* 4.0). btw, I messed up the field name before (I edited my previous comment too), this is the correct field:
enable_materialized_views: true

I justed tested it with C* 4.0

@mpenick: Strange, this didn't work for me. I am using docker-py:
path_to_cassandra_yaml = Path(__file__).absolute().parent / "cassandra.yaml"
client.containers.run(
        "stargateio/stargate-4_0:v1.0.55",
        detach=True,
        name="stargate",
        ports={8081: 8081, 8082: 8082, 9042: 9042},
        environment={
            "CLUSTER_NAME": "stargate",
            "CLUSTER_VERSION": "4.0",
            "CQLENG_ALLOW_SCHEMA_MANAGEMENT": "1",
            "DEVELOPER_MODE": "1",
            "JAVA_OPTS": f"-Dstargate.unsafe.cassandra_config_path={path_to_cassandra_yaml}",
        },
    )

Error:
cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Materialized views are disabled. Enable in cassandra.yaml to use."

cassandra.yaml is on the same directory as the script above:
# ...
enable_materialized_views: true

Docker version:
docker run -e CLUSTER_NAME=stargate \
  -e CLUSTER_VERSION=4.0 -e DEVELOPER_MODE=1 \
  -e JAVA_OPTS=-Dstargate.unsafe.cassandra_config_path=/cassandra.yaml \
  -v $(pwd)/cassandra.yaml:/cassandra.yaml \
  -p 9042:9042 stargateio/stargate-4_0:v1.0.55

authors: caniko mpenick

repo: stargate/docker-images


text:
Feature Required: Add Line numbers from block of code

like this from obsidian-advanced-codeblock
using nums to add line numbers

+1

repo: stargrey/obsidian-better-codeblock


text:
Problem when install bucc to openstack : Openssl error

Hello,
I try to install BUCC on a iaas Openstack but i've error with openssl lib :
stdout: 'bundler: failed to load command: /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/bin/openstack_cpi (/home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/bin/openstack_cpi)
', stderr: 'LoadError: cannot load such file -- openssl

but
:
dpkg -l | grep ssl
ii  libflac8:amd64                   1.3.1-4                                    amd64        Free Lossless Audio Codec - runtime C library
ii  libgnutls-openssl27:amd64        3.4.10-4ubuntu1.4                          amd64        GNU TLS library - OpenSSL wrapper
ii  libssl-dev:amd64                 1.0.2g-1ubuntu4.10                         amd64        Secure Sockets Layer toolkit - development files
ii  libssl-doc                       1.0.2g-1ubuntu4.10                         all          Secure Sockets Layer toolkit - development documentation
ii  libssl1.0.0:amd64                1.0.2g-1ubuntu4.10                         amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl                          1.0.2g-1ubuntu4.10                         amd64        Secure Sockets Layer toolkit - cryptographic utility

and
ruby and openssl is install and works :
ruby -ropenssl -e "puts :OK"
OK

Can you help me ?
 bin/bucc  up --cpi openstack --keystone-v2
Using flags: --cpi openstack --keystone-v2
Flag files: '/home/ubuntu/BUCC/bucc/state/{cpi,lite,flags}'

Deployment manifest: '/home/ubuntu/BUCC/bucc/src/bosh-deployment/bosh.yml'
Deployment state: '/home/ubuntu/BUCC/bucc/state/state.json'

Started validating
  Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh'... Finished (00:00:00)
  Downloading release 'bosh-openstack-cpi'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
  Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
  Validating release 'os-conf'... Finished (00:00:00)
  Downloading release 'garden-runc'... Skipped [Found in local cache] (00:00:00)
  Validating release 'garden-runc'... Finished (00:00:00)
  Downloading release 'uaa'... Skipped [Found in local cache] (00:00:00)
  Validating release 'uaa'... Finished (00:00:01)
  Downloading release 'concourse'... Skipped [Found in local cache] (00:00:00)
  Validating release 'concourse'... Finished (00:00:03)
  Downloading release 'credhub'... Skipped [Found in local cache] (00:00:00)
  Validating release 'credhub'... Finished (00:00:00)
  Downloading release 'credhub-importer'... Skipped [Found in local cache] (00:00:00)
  Validating release 'credhub-importer'... Finished (00:00:00)
  Downloading release 'postgres'... Skipped [Found in local cache] (00:00:00)
  Validating release 'postgres'... Finished (00:00:00)
  Downloading release 'bucc-bbr'... Skipped [Found in local cache] (00:00:00)
  Validating release 'bucc-bbr'... Finished (00:00:01)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Skipped [Found in local cache] (00:00:00)
  Validating stemcell... Finished (00:00:02)
Finished validating (00:00:13)

Started installing CPI
  Compiling package 'ruby_openstack_cpi/ddb0f4a3013923fb1b074454d4314264c47d33c5'... Finished (00:00:00)
  Compiling package 'bosh_openstack_cpi/2ae4a479317fcc7300eda9125fa4572ec9a20e82'... Finished (00:00:00)
  Installing packages... Finished (00:00:00)
  Rendering job templates... Finished (00:00:00)
  Installing job 'openstack_cpi'... Finished (00:00:00)
Finished installing CPI (00:00:00)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-openstack-kvm-ubuntu-trusty-go_agent/3468.21'... Failed (00:00:00)
Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

creating stemcell (bosh-openstack-kvm-ubuntu-trusty-go_agent 3468.21):
  Executing external CPI command: '/home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/jobs/openstack_cpi/bin/cpi':
    Running command: '/home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/jobs/openstack_cpi/bin/cpi', stdout: 'bundler: failed to load command: /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/bin/openstack_cpi (/home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/bin/openstack_cpi)
', stderr: 'LoadError: cannot load such file -- openssl
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/vendor/bundle/ruby/2.4.0/gems/excon-0.60.0/lib/excon.rb:7:in `require'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/vendor/bundle/ruby/2.4.0/gems/excon-0.60.0/lib/excon.rb:7:in `<top (required)>'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/vendor/bundle/ruby/2.4.0/gems/fog-core-1.45.0/lib/fog/core.rb:5:in `require'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/vendor/bundle/ruby/2.4.0/gems/fog-core-1.45.0/lib/fog/core.rb:5:in `<top (required)>'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/vendor/bundle/ruby/2.4.0/gems/fog-openstack-0.1.24/lib/fog/openstack.rb:2:in `require'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/vendor/bundle/ruby/2.4.0/gems/fog-openstack-0.1.24/lib/fog/openstack.rb:2:in `<top (required)>'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/lib/cloud/openstack.rb:5:in `require'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/lib/cloud/openstack.rb:5:in `<top (required)>'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/bin/openstack_cpi:7:in `require'
  /home/ubuntu/.bosh/installations/82bac5cf-b7a5-443d-71d9-20d99f15b4b5/packages/bosh_openstack_cpi/bin/openstack_cpi:7:in `<top (required)>'
':
      exit status 1

Exit code 1

Sorry to close ticket. If you have further issues, can I ask you create them upstream at https://github.com/cloudfoundry/bosh-cli/ or https://github.com/cloudfoundry-incubator/bosh-openstack-cpi-release?

i've already all the dependencies :
sudo apt-get install -y build-essential zlibc zlib1g-dev ruby ruby-dev openssl libxslt-dev libxml2-dev libssl-dev libreadline6 libreadline6-dev libyaml-dev libsqlite3-dev sqlite3
sudo: unable to resolve host bucc
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'libxslt1-dev' instead of 'libxslt-dev'
build-essential is already the newest version (12.1ubuntu2).
libreadline6 is already the newest version (6.3-8ubuntu2).
libreadline6-dev is already the newest version (6.3-8ubuntu2).
libsqlite3-dev is already the newest version (3.11.0-1ubuntu1).
libyaml-dev is already the newest version (0.1.6-3).
ruby is already the newest version (1:2.3.0+1).
ruby-dev is already the newest version (1:2.3.0+1).
sqlite3 is already the newest version (3.11.0-1ubuntu1).
zlibc is already the newest version (0.9k-4.3).
libssl-dev is already the newest version (1.0.2g-1ubuntu4.10).
libxml2-dev is already the newest version (2.9.3+dfsg1-1ubuntu0.5).
libxslt1-dev is already the newest version (1.1.28-2.1ubuntu0.1).
openssl is already the newest version (1.0.2g-1ubuntu4.10).
zlib1g-dev is already the newest version (1:1.2.8.dfsg-2ubuntu4.1).
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

ok i will check wit bosh-cli  and bosh-openstack-cpi what's wrong ;)
thanks for your reply

authors: drnic jyriok

repo: starkandwayne/bucc


text:
add discussions-to link to SNIP-2

https://github.com/starknet-io/SNIPs/discussions/34

wouldn't it be better to number SNIPs that are inspired by ERC's to have same number or prefix the number by something, for example SNIP-E20?

hmm, what was the reason behind going with independent identification? I don't think we loose anything by adding a namespace for SNIPs directly inspired from EIPs. And we gain good DX

repo: starknet-io/SNIPs


text:
Refactor inlining logic.

This change is

crates/cairo-lang-lowering/src/test_data/destruct line 87 at r1 (raw file):

blk0 (root):
Statements:
  (v6: core::RangeCheck, v7: core::SegmentArena, v8: core::gas::GasBuiltin, v1: core::integer::u128) <- test::get_total_signed_weight(v3, v4, v5, v0)

This function is no-longer inline as we take destructors into account when inlining.
Code quote:
test::get_total_signed_weigh

crates/cairo-lang-lowering/src/inline/mod.rs line 84 at r3 (raw file):

) -> Maybe<InlineInfo> {
    let lowered = db.function_with_body_lowering(function_id)?;
    if db.in_cycle(function_id)? {

I could replace this with
in_cycle_postpanic or something like that but since we don't report error here it is rather pointless.
Code quote:
 if db.in_cycle(function_id)? {

crates/cairo-lang-lowering/src/test_data/destruct line 87 at r1 (raw file):
Previously, orizi wrote…
was it supposed to be inlined?

The current heuristic is to inline only a single function call, I think not counting destructors in the heuristic is unnatural.

crates/cairo-lang-sierra-generator/src/statement_location_test_data/simple line 42 at r3 (raw file):
Previously, orizi wrote…
i don't understand the statement.

the change is because bar was inline once,
I'll explicitly prevent that case.

crates/cairo-lang-runner/src/profiling_test_data/major_test_cases line 682 at r8 (raw file):
Previously, orizi wrote…
why not inlined?

I guess it has more then 10 statement with panic handling

repo: starkware-libs/cairo


text:
feat(consensus): add re proposal flow to sequencer context

This change is

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#1195  👈
#1194
#1193
#1192
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @matan-starkware and the rest of your teammates on  Graphite

Merge activity

Oct 15, 7:50 AM EDT: A user started a stack merge that includes this pull request via Graphite.

repo: starkware-libs/sequencer


text:
Fix test cases

Fixes #109
This PR builds on top of #108, so the diff in GitHub will not look right until that PR is merged. For now you can review this PR by looking at the one relevant commit: https://github.com/starlinglab/integrity-backend/commit/31a7b708ad93e5f5790d249f0556ae34ecba20d5
This PR cannot be merged until #108 is merged.

They don't show up anymore because of a GitHub UI change, but I've made comments for reviewers to look at on this commit: https://github.com/starlinglab/integrity-backend/commit/31a7b708ad93e5f5790d249f0556ae34ecba20d5. Scroll through the diff and you'll see them.

Changes to claim.py and test_claim.py were reverted, because better fixes to those files are in #112 and I don't want to cause unnecessary merge conflicts. This means some test cases don't pass, but it's only the ones related to those files, everything else passes.
Once both this PR and #112 are merged, all tests cases will pass.

repo: starlinglab/integrity-backend


text:
Preprocessor failes to validate wacz Files

Error occurs
 AttributeError: module 'validate' has no attribute 'Wacz'
https://github.com/starlinglab/integrity-preprocessor/blob/dev/lib/common/__init__.py#L102
Note: When commented process continues
Oct  3 18:54:59 org2 python3[936029]: 2022-10-03 18:54:59 INFO Recorder Metadata Change Detected
Oct  3 18:54:59 org2 python3[936029]: Traceback (most recent call last):
Oct  3 18:54:59 org2 python3[936029]:   File "/root/integrity-preprocessor/browsertrix/main.py", line 434, in
Oct  3 18:54:59 org2 python3[936029]:     meta_extra = common.parse_wacz_data_extra(wacz_path)
Oct  3 18:54:59 org2 python3[936029]:   File "/root/integrity-preprocessor/browsertrix/../lib/common/init.py", line 102, in parse_wacz_data_extra
Oct  3 18:54:59 org2 python3[936029]:     if not validate.Wacz(wacz_path).validate():
Oct  3 18:54:59 org2 python3[936029]: AttributeError: module 'validate' has no attribute 'Wacz'
Oct  3 18:54:59 org2 systemd[1]: integrity-preprocessor-browsertrix.service: Main process exited, code=exited, status=1/FAILURE
Oct  3 18:54:59 org2 systemd[1]: integrity-preprocessor-browsertrix.service: Failed with result 'exit-code'

This has been fixed for now, a broader fix is in #48

authors: YurkoWasHere makeworld-the-better-one

repo: starlinglab/integrity-preprocessor


text:
I have all green but will only let 1 RDP session

I can't seam to post my snip

I also met the same problem today! waiting for answers. Great thanks for the kind people.

I also met the same problem today! waiting for answers. Great thanks for the kind people in advance.

You may try to install 1.6.2 version of rdpwrap to see whether it's fixed or not. After that, reboot your computer.

I am having the same issue with a windows 10 workstation. I did install the 1.6.2 version, and am still only able to login as the same user one time, if i try a second time, it pulls the same session.

Seems that HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server:fSingleSessionPerUser=0 no longer works.

Same issue on 10.0.19041.789
Helped switching to GUI authentication, network level authentication seems to be broken

Same issue 19041.789
All RDPconf settings are green and the max limit of sessions has been set, but still showing connections is limited and will disconnect other users.
Searching answers for hours but still no progress.

Is that solved?
I have same issue.

Applicable to https://github.com/stascorp/rdpwrap/issues/1883

authors: Begin2aim Chendhome TWiley1 archiezhang sdunnaway sebaxakerhtc vivanusic xikavi xmostera

repo: stascorp/rdpwrap

license_source: bigquery


text:
WRAPPER STATE : 3rd Party

Please Help
I am unable to use Remote desktop. I'm using quick heal antivirus

I will be highly obliged.

Closed as duplicate of https://github.com/stascorp/rdpwrap/issues/1595

I have copied termserv file but it's still the same

Closed as duplicate of #1595

I have updated termser file still not able to use. It's the same issue. Please help

Closed as duplicate of #1595

I have updated termser file still not able to use. It's the same issue. Please help

Go to official Telegram channel for getting help

](url)
Can you please help on this case?


text:
ForceMove safety

This PR contains a draft blog post on the safety considerations taken in the ForceMove protocol.

Hi - I wonder if it might be worth adding in a little bit about who can issue a forceMove() transaction?  I'm a bit confused about who Eve is?  I managed to make sense of things when I decided that Eve and Bob were the same person, but that led to confusion in Version 4 when it appears that Bob is calling forceMove() when it's his/her turn to act next.
I went back to the code and this does appear to be allowed.
Could you possibly clarify this a little?

also - I'm 90% sure you want griefing rather than grieving throughout!

Hi - I wonder if it might be worth adding in a little bit about who can issue a forceMove() transaction? I'm a bit confused about who Eve is? I managed to make sense of things when I decided that Eve and Bob were the same person, but that led to confusion in Version 4 when it appears that Bob is calling forceMove() when it's his/her turn to act next.
I went back to the code and this does appear to be allowed.
Could you possibly clarify this a little?

Yes, Eve should be Bob. It's a mistake from an unfinished copy/paste.

@tomclose maybe close this in favour of the new blogging repo base.

This draft is no longer relevant, given @tomclose's draft post.

authors: andrewgordstewart snario zakalwe

repo: statechannels/monorepo


text:
This test relibly rails CI

Able to reproduce the issue in Codespaces. Need a hand to figure out how to fix it.

Code References

        test/specs/test.e2e.ts
    (Line: 79 - 91)

      it.skip('should be able to switch news channels', async () => {
        const firstArticle = await newsWidget.getArticle(0)
        const firstArticleText = await firstArticle.getText()
        expect(typeof firstArticleText).toBe('string')

        await newsWidget.switchChannel('Marquee')

        const newFirstArticle = await newsWidget.getArticle(0)
        const newFirstArticleText = await newFirstArticle.getText()

        expect(typeof newFirstArticleText).toBe('string')
        expect(firstArticleText).not.toBe(newFirstArticleText)
      })

@christian-bromann maybe we can pair on this one

repo: stateful/vscode-marquee


text:
Entry exist event state transition #12

here is finally my pull request base on the issue #12
I made a change for the constructor of the FSM, in that case we are sure to call the same constructor from other constructor, I need to trigger the first onEntry for the initial state
I test the build with netBeans 8.2, maybe I need to write some test for that ?
let me know what do I need to change

@developer-x any review done ?
or nobody work on it anymore ?

I'm still working on it when I can.  I've been heads down on my new venture.  Hope to get to this soon.

@developer-x, any update with such issue, hope it could be merged and it's better that statefulj could take scxml standard into action as statefulj lacks the support for standard.

Still haven't had a chance to look into it.  Hadn't heard of the scxml
effort - would be interested in looking into it's compliance.  I hope to
get to this soon.

Anderw

On Sun, Jul 2, 2017 at 6:18 AM, bwzhang2012 notifications@github.com
wrote:

@developer-x https://github.com/developer-x, any update with such
issue, hope it could be merged and it's better that statefulj could take
scxml standard into action as statefulj lacks the support for standard.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/statefulj/statefulj/pull/30#issuecomment-312491410,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADAYk82amK-GX05GLZJukb__qMcfaS3Eks5sJ5imgaJpZM4NhbTu
.

@developer-x, thanks for view and here came some suggestion from scxml:

scxml standard: https://www.w3.org/TR/scxml/
scxml apache commons: https://github.com/apache/commons-scxml
scxml java: https://github.com/carlos-verdes/scxml-java
as scxml standard seems more open with a lot of languages support and also scxml gui tool, I think long term that statefulj could take it into consideration. by the way, hope the onEntry and onExit could be merged once formed.

@bwzhang2011 you should do a fork of this project I think, looks like nobody care about this issue. you can grab my pull request if you need

I'm going to close it since nobody got back at me...

repo: statefulj/statefulj


text:
try more klee options

-pre-RA-sched
-randomize-fork
-search
-simplify-sym-indices
-use-construct-hash
-use-fast-cex-solver
-use-merge (probably nees instrumentation)
-cex-cache-exp
-cex-cache-superset
-cex-cache-try-all

check what is  -const-array-opt

repo: staticafi/symbiotic

license: mit


text:
add ARM support

It would be great if an ARM23/64 image would be available.

I decided to buildon ARM64 myself using the dockerfile provided

repo: staticfloat/docker-nginx-certbot


text:
Optimizing Replace method in MemoryExtensions

I was taking a look at some of the perf and poked around here. With 4,000 files the little things are starting to add up. Did two main things

Convert to a span outside of the loop. This is actually a pretty cheap call, but with this many NormalizedPath being created it gets called a ton
Pull the span outside of the loop to compare once.

Took the number of times the call to the Span getter and the indexer were being called from 200 million times down to 5 million. This helps out a bit with the constructor knocking the total amount of time spent building those up on my machine from 18s down to 5.
Before

After

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Went to bed and thought to myself - "wait a second, those were all getters...I bet I had it on Debug mode like a moron."
Did a second test and sure enough, compiler takes care of those things for me. Disregard!


text:
Update faq.md

more info about event dimensions in FAQ

🚀 Deployed on https://66ccbc90123b174a55aa6412--cozy-fox-0defec.netlify.app

authors: jasonwzm makris-statsig

repo: statsig-io/docs


text:
Utilizing code from main app within plugin

Hi there,
I'm looking at this project to possibly use for integrating with an existing Flask application. It seems pretty robust, but I do have a question tho.
It's going to be a bit difficult to explain as it's mostly theoretical, just thinking about ways I can utilize this. Lets say that the main program has a service that plugins should be able to access. For example, a background task runner. The main Flask application can, on startup, spin up a background task service (a singleton), and accept request to run tasks. A plugin could then send data to this background task runner.
This could be fairly easy with local plugins, since you can import directly from the parent application and module. What I'm interested in learning more about is how one might do this for a third party plugin that is on pypi. How would that plugin, which is more or less isolated from the main application, access things  from the main application? Is having a third party plugin not feasible in this situation, and a local plugin should be pursued?
Sorry if this doesn't make any sense. If not, let me know, maybe I can stand up a simple example :)
Thanks!

Hi there.
I probably understand what you mean, but it may not be very accurate (from China, English is not good).
The core of my analysis of your question is: how do third-party plugins import local modules?
If this is the problem, maybe I have tried, like this plugin: flask-pluginkit-isso,
the code is:
from __future__ import absolute_import
try:
    from config import PLUGINS
except ImportError:
    PLUGINS = {}

When I use it, there is a config.py locally in the project.

Hi, if the problem is solved, please tell me to close this issue.

authors: blitzmann staugur

repo: staugur/Flask-PluginKit


text:
Header shortest_paths.hpp adds dependency on fmt/format.h

This library should not depend on fmt/format.h, especially that on MSVC there is no reason to include it when we have <format>.

I chose to use a recent version of fmt to provide the latest capabilities in the library, as they continue to improve it, without requiring us to bump graph-v2 to use C++23. print() and println() are obvious things, but there are other features I can't recall off the top of my head.
Is this causing you any problems?

In my environment, I do not have the {fmt} library. That this should perevent me from doing the Shortest Paths algorithm is strange.
I tried to see if this is still the case, but I can see that the file shortest_paths.hpp is gone. I tried graph/algorithm/common_shortest_paths.hpp, but only through including it, I get error C2760 at line:
https://github.com/stdgraph/graph-v2/blob/master/include/graph/algorithm/common_shortest_paths.hpp#L18
This is about the undefined identifier is_arithmetic_v, and a bunch of other identifiers in this file (like edge_reference_t). It looks like they require the namespace prefix.

CMake should be downloading the {fmt} library if you "Delete Cache and Reconfigure".
There's now dijkstra_shortest_paths.hpp and bellman_ford_shortest_paths. This is similar to how Boost Graph does these algorithms.
In the move from std::graph to the graph namespace, I added the using statements in graph_using.hpp to make things look cleaner. I don't know if this is the best idea, but I find it improves readability for myself. I've added an #include for that in common_shortest_paths.hpp so people won't get that error. It will be in the next push, hopefully this week.


text:
Add @defgroup to define doxygen groups in graph.hpp

Also added @ingroup to some relevant concept, template alisas and function header docs to get an idea of how this works. This is experimental right now.
Changed doxygen comment style in a couple of places.

Error occurred  in Mac build. Unsure what that's about. I've only made changes to comments, so I'm going to merge.


text:
[RFC]: improve README examples of stats/base/dists/truncated-normal namespace

Description
This RFC proposes to improve the README examples of the stats/base/dists/truncated-normal namespace package. Currently, namespace package READMEs only include minimal examples which are not particularly informative and do not showcase namespace functionality.
Historically, namespace examples were not given much effort due to the rapidly changing nature of the project; however, now that development and organization has settled, it would be good to revisit these packages and add proper examples to better communicate to users how to use namespace contents.
For those wanting to contribute to this issue, you can use the following steps:

Browse through the contents of the stats/base/dists/truncated-normal namespace and identify interesting functions and packages to showcase.
Update the example code block in the "Examples" section of the README for the stats/base/dists/truncated-normal namespace and open a PR.
When updating the README, examples should follow existing project conventions (e.g., ES5).
Examples may require functionality from other packages to help make examples more interesting.

Related Issues
None.
Questions
No.
Other
No.
Checklist

[X] I have read and understood the Code of Conduct.
[X] Searched for existing issues and pull requests.
[X] The issue name begins with RFC:.

@Planeshifter @kgryte  till my PR is under review could it be possible for me to work on this issue?

@vr-varad This namespace is still very bare, containing only one function. Maybe choose a different one to improve the example code block for? Thanks!

authors: Planeshifter vr-varad

repo: stdlib-js/stdlib


text:
Go Kafka Consumers On Mesos with Marathon

Added Vagrantfile for running mesos and marathon on top of it with docker containers support for Go Kafka Client Consumers

old code, we are redoing this


text:
C Libraries (c_framework) causes memory problem

Most of C libraries causes memory problem, e.g. c_resolve.

Now we uses something special to solve it. see latest release


text:
Check DozerDB

https://dozerdb.org/
If cypherGUI works with it add it to readme.

checked. it works. added to readme.

b8fca2165ddb525b7dbc85b030c21430ea829522

repo: stefanak-michal/cyphergui


text:
A second 'fog of war' layer

Many modern strategy games have a second shroud layer that is revealed only when within living unit's sight radius, anything else is remained covered. Terrain is still visible, along with structures previously scouted, but this information is not updated until a unit's sight passes through again.
This makes it more important to keep units on the map, as you will have no knowledge of enemy troop movements, even through previous explored areas, unless your units are nearby to see them.

I believe this would add a very interesting element to strategy, as many contemporary strategy games have proven already.

Yes, this will be a very important feature.
In fact, I don't think keeping units on the map will do (because of the size of the map and because of the threat of worms). But, this is where the Spacing Guild can come in.
Just like in the books you should be able to bribe the Spacing Guild for revealing the map (clear fog of war).
But I also think this could have some variations, like:

when YOU clear the fog of war, you get what you see
when someone else clears the fog of war (ie the spacing guild) you will see what they want you to see. Enabling you to bribe the Spacing Guild to reveal map data, but also counter-bribe (if that is a word) to 'not show my buildings, only the map'. Which would make interesting strategic consequences.

This issue depends on #2 and #37 and #21

That does sound very interesting! It would also have consequences for an eventual multiplayer mode where your standing (measured in 'contributions'?) with the Spacing Guild influences how much, if any, of your units and structures are revealed to other players when they attempt to buy some vision in an area.
There is also a difference between buying an update for an area, so you can see if there are structures there or what the map looks like and buying sight range for a period of time, so you can see if units are moving through an area for the duration of that.
Maybe a choice needs to be made on which are available, or perhaps both are but for different prices?

This issue is ready to go. All required elements are built.

authors: RiderRockon stefanhendriks


text:
New complemetary tool

My name is Luis, I'm a big-data machine-learning developer, I'm a fan of your work, and I usually check your updates.
I was afraid that my savings would be eaten by inflation. I have created a powerful tool that based on past technical patterns  (volatility, moving averages, statistics, trends, candlesticks, support and resistance, stock index indicators).
All the ones you know (RSI, MACD, STOCH, Bolinger Bands, SMA, DEMARK, Japanese candlesticks, ichimoku, fibonacci, williansR, balance of power, murrey math, etc) and more than 200 others.
The tool creates prediction models of correct trading points (buy signal and sell signal, every stock is good traded in time and direction).
For this I have used big data tools like pandas python, stock market libraries like: tablib, TAcharts ,pandas_ta... For data collection and calculation.
And powerful machine-learning libraries such as: Sklearn.RandomForest , Sklearn.GradientBoosting, XGBoost, Google TensorFlow and Google TensorFlow LSTM.
With the models trained with the selection of the best technical indicators, the tool is able to predict trading points (where to buy, where to sell) and send real-time alerts to Telegram or Mail. The points are calculated based on the learning of the correct trading points of the last 2 years (including the change to bear market after the rate hike).
I think it could be useful to you, to improve, I would like to share it with you, and if you are interested in improving and collaborating I am also willing, and if not file it in the box.

If tou want, Please read the readme , and in case of any problem you can contact me ,
If you are convinced try to install it with the documentation.
https://github.com/Leci37/LecTrade/tree/develop I appreciate the feedback

Hi there – this repository is an accompaniment for my book, and this is well beyond its scope as the purpose is to instruct readers how to build a package. While I am unable collaborate on this, I quickly looked at your repository, and noticed that you don't have a license. This is crucial to have (especially if you want people to collaborate and use your library), so it's a good place for you to start.

repo: stefmolin/stock-analysis


text:
Better config with nconf

Would be nice to use e.g. nconf so the config.js looks like this:
'use strict';

var nconf = require('nconf');

nconf.env('__');

var config = {
  appLocales: ['en'],
  defaultLocale: 'en',
  googleAnalyticsId: 'UA-XXXXXXX-X',
  webUrl: 'localhost.com:8000',
  isProduction: process.env.NODE_ENV === 'production',
  piping: {
    ignore: /(\/\.|~$|\.(css|less|sass|scss|styl))/,
    hook: true
  },
  port: process.env.PORT || 8000,
  version: require('../../package').version,
  webpackStylesExtensions: ['css', 'less', 'sass', 'scss', 'styl'],
  mongo: {
    host: '127.0.0.1',
    port: 27017,
    database: 'sassie'
  }
};

nconf.defaults(config);

module.exports = nconf.get();

With the above config, we can now set env variables e.g in Heroku dashboard like this mongo__host to override development credentials.
WDYT?

I think it's great :+1:

Worth implementing or let's put it into wiki pages?

I suppose it's worth implementing. Can you create PR please?

Yeah, will send all the PR's tomorrow including other issues as well!

authors: grabbou steida

repo: steida/este


text:
Add into wiki how pingdom can ensure heroku hosted for free app staying alive

https://www.pingdom.com and https://www.youtube.com/watch?v=I_izvAbhExY

https://www.youtube.com/watch?v=I_izvAbhExY

or use http://kaffeine.herokuapp.com

Awesome, thank you.

@mikker Maybe you can help me again :-) I added https://github.com/steida/este/blob/master/package.json#L8 just for Heroku, but it slowdowns npm install. Can Heroku build automatically without writing own builder? Thank you.

If it can I am not aware. But you could probably specify multiple buildpacks: the node.js buildpack in combination with another that just gulp compiles.
https://github.com/ddollar/heroku-buildpack-multi

The benefit of using two buildpacks and not just copying and adding to the default node one would be that you would still get the updates to the node one. And I like updates.

This is probably defunct now with Heroku's new Free plan (6 hours of sleep per day required).

authors: mikker steida


text:
Support the v5 Tournament API

https://developer.riotgames.com/apis#tournament-v5
Riot games has released their v5 tournament endpoint. Would like to have that supported.

Fixed in f5b2064add4227879cccf40669dd7a964e03f79b


text:
Better communicate testnet reset

We need to better communicate to devs working with testnet that the ledger will be reset every 3 months. It may also be important to change the network passphrase and avoid any kind of accidental mixing between nodes & clients during testnet reset.
Ideas:

update Horizon errors on testnet to state the date of the last reset and the date of the next one, and link out to our testnet best practices
change the network passphrase, and update SDKs to auto-detect the current passphrase
modify Stellar lab to state reset information when in testnet mode

Any other ideas?

That seems fine.
For the date in the passphrase, does it need to be fully specified or just have the month/Quarter (and have a link to get the actual date)?
Something like Stellar Test Network Q3 2019; See http://www.stellar.org/testnet for more information
Do we need to use templates somehow for all the sample configurations that we have for Horizon/Core - also, some of those, I think, are packaged/tagged with releases

I like updating the passphrase with date + time, it will be a good motivator for automating the reset, which ultimately is the goal here.
With regards to configs, the stellar-core-postgres package has a sample Testnet configuration which would need to be updated every quarter. We could easily modify the NETWORK_PASSPHRASE during package build and then simply schedule a build of this package every 3 months.
Caveat: Package end-users utilising a modified config will still need to update the passphrase manually.

Yeah, the advantage of including the date + time is that https://horizon-testnet.stellar.org/ becomes the one source of truth for when the reset is happening.

Below are the 3 configs with a Testnet passphrase that end up in the .deb packages:

https://github.com/stellar/packages/blob/master/stellar-core-postgres/stellar-core_postgres.cfg
https://github.com/stellar/stellar-core/blob/master/docs/stellar-core_testnet.cfg
https://github.com/stellar/stellar-core/blob/master/docs/stellar-core_standalone.cfg

The standalone config should probably be updated to use "Standalone Network ; February 2017" in the same way that the docker quickstart standalone config does here.
For public docker images, we need to update:

https://github.com/stellar/docker-stellar-core-horizon/blob/master/testnet/core/etc/stellar-core.cfg
https://github.com/stellar/docker-stellar-core-horizon/blob/a2f1f5ff4717520e6a2c28aa468d1d8758fd7d8b/stat#L77
https://github.com/stellar/docker-stellar-core/blob/1ea7fa8c7b99732574ab5a20166f8eee1171ff5b/examples/compat_complete.env#L22
https://github.com/stellar/docker-stellar-core/blob/1ea7fa8c7b99732574ab5a20166f8eee1171ff5b/examples/compat_minimal.env#L20
https://github.com/stellar/docker-stellar-core/blob/1ea7fa8c7b99732574ab5a20166f8eee1171ff5b/examples/local.env#L34
https://github.com/stellar/docker-stellar-core/blob/1ea7fa8c7b99732574ab5a20166f8eee1171ff5b/examples/single.env#L36

There is also a bunch of documentation that needs updating as part of the process, either permanently or on each reset.
https://github.com/stellar/docs/search?q=Test+SDF+Network+%3B+September+2015&unscoped_q=Test+SDF+Network+%3B+September+2015
We wouldn't have to update all the configs/docs if we set the passphrase to NETWORK_PASSPHRASE="Test SDF Network ; Testnet is reset every 3 months - backup your data", or something along those lines.
But then new quickstarts might still get bad messages from non reset peers.
@MonsieurNicolas is there anything else we can do to ignore messages from stale peers ?

Maybe a stupid idea... but can’t we change our various packages and scripts to just curl a know url (like our packages repo) for the latest version of the testnet config?

As for ignoring other peers without changing the passphrase I don’t think we can do a lot more than what we’re doing

Resolving this as testnet resets seem to be going smoothly lately, thanks to Tom's work and better announcements.

@tomquisel do you think this issue can be closed?

authors: MonsieurNicolas bartekn brahman81 tomquisel

repo: stellar/go


text:
Add args to AP helm chart

PR Checklist
PR Structure

[x] This PR has reasonably narrow scope (if not, break it down into smaller PRs).
[x] This PR avoids mixing refactoring changes with feature changes (split into two PRs
otherwise).
[x] This PR's title starts with name of package that is most changed in the PR, ex.
paymentservice.stellar, or all or doc if the changes are broad or impact many
packages.

Thoroughness

[] This PR adds tests for the most critical parts of the new functionality or fixes.

What
Add args to AP helm chart
Why
This lets the user specify which services to run, so they can choose to run the sep server alone or with the payment observer.

Something went wrong with PR preview build please check

Reference Server Preview is available here:https://anchor-ref-pr584.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr584.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr584.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr584.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr584.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr584.previews.kube001.services.stellar-ops.com/

@erika-sdf: I've just realized there are some conflicts to solve before I can actually review it, since Jamie merged the config refactor PR.

@erika-sdf: I've just realized there are some conflicts to solve before I can actually review it, since Jamie merged the config refactor PR.

No worries, I'll resolve the conflicts

Reference Server Preview is available here:https://anchor-ref-pr584.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr584.previews.kube001.services.stellar-ops.com/

Something went wrong with PR preview build please check

Reference Server Preview is available here:https://anchor-ref-pr584.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr584.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr584.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr584.previews.kube001.services.stellar-ops.com/

Now passing!

repo: stellar/java-stellar-anchor-sdk


text:
Add u32::MAX overflow check in MeteredVector module

Currently, we have a check for u32::MAX overflow in vec_append host function. However, its better to add an u32::MAX overflow check in MeteredVector module.

Resolved by https://github.com/stellar/rs-soroban-env/pull/1001

repo: stellar/rs-soroban-env


text:
Add comments to each CostType; add very crude, untested model parameters

What
Resolves https://github.com/stellar/rs-soroban-env/issues/296 by putting in some cost parameters and budget.
Why
[TODO: Why this change is being made. Include any context required to understand the why.]
Known limitations
[TODO or N/A]

Given that you're away this week @jayz22 I just made the review edits I had in mind here myself and pushed them. Will merge now, thanks!


text:
Register External Contract for Unit Testing

What version are you using?
Commit b4cc8fc

What did you do?
Import an external contract and register it using the following function:
soroban_sdk::env::Env
pub fn register_contract<T>(&self, contract_id: &BytesN<32>, contract: T)

in an "integration" test format (within the tests folder). The function will not compile with the following error:
the trait `ContractFunctionSet` is not implemented for `ExampleContract`

The test works as expected in the "unit" test format.
Minimum example repository here: https://github.com/mootz12/soroban-external-contract-test

What did you expect to see?
Imported external contracts can be registered against the env without having to rely on test structs (as seen with consumption of the token contract https://github.com/stellar/soroban-token-contract/blob/main/src/testutils.rs)
What did you see instead?
Testing can still be completed using the "unit" test approach or with a test struct.

This is because the test utilities are only compiled in for test builds where the test cfg is enabled, or when testutils feature is enabled. Unit tests are built with the test cfg enabled. Integration tests in Rust do not. Integration tests build the crate without the test cfg.
I would recommend building integration tests that use the test utilities with a #![cfg(feature = "testutils")] at the top of the integration test and then enabling the testutils feature when running tests.

@leighmcculloch when I was trying to reproduce the issue reported by @mootz12 I did exactly what you suggested, and I started getting some other errors about missing conversions (coming from contractimpl)
error[E0277]: the trait bound `ScVec: TryFrom<(&soroban_sdk::BytesN<32_usize>, &str, &u64)>` is not satisfied
  --> src/lib.rs:15:1
   |
15 | #[contractimpl]
   | ^^^^^^^^^^^^^^^ the trait `TryFrom<(&soroban_sdk::BytesN<32_usize>, &str, &u64)>` is not implemented for `ScVec`
   |
   = help: the following other types implement trait `TryFrom<T>`:
             <ScVec as TryFrom<&std::vec::Vec<ScVal>>>
             <ScVec as TryFrom<(T0, T1)>>
             <ScVec as TryFrom<(T0, T1, T2)>>
             <ScVec as TryFrom<(T0, T1, T2, T3)>>
             <ScVec as TryFrom<(T0, T1, T2, T3, T4)>>
             <ScVec as TryFrom<(T0, T1, T2, T3, T4, T5)>>
             <ScVec as TryFrom<(T0, T1, T2, T3, T4, T5, T6)>>
             <ScVec as TryFrom<(T0, T1, T2, T3, T4, T5, T6, T7)>>
           and 6 others
   = note: required because of the requirements on the impl of `TryInto<ScVec>` for `(&soroban_sdk::BytesN<32_usize>, &str, &u64)`
   = note: this error originates in the attribute macro `contractimpl` (in Nightly builds, run with -Z macro-backtrace for more info)

That error is originating from derive_client and derive_fn. This error doesn't occur when setting up the code in "unit test" style, so there is probably some other feature issue going on.

@jonjove I also have the same issue, although my note claims it originates in the macro ::soroban_sdk::contractclient
Did a bit more poking around, and noticed that the example contract listed does not build with cargo build --features testutils (due to the error Jon documented). It appears to be caused by having a function argument of u64, since I can resolve the error with the following code update:
    // update trait and tests to match...

    fn do_thing(x: u32) -> u64 {
        u64::from(x + 1)
    }

At this point, unit testing and integration testing both work. (although, testing in Integration mode might warrant some additional documentation around the use of testutils features)

@mootz12 that was exactly the piece of information I needed to find what conversion was missing, thank you! See https://github.com/stellar/rs-stellar-xdr/pull/149. Can you try picking that change up and see if it fixes your issue?

Yup, confirmed that both of these work as expected after getting your change:
cargo test --features testutils

and
cargo build --features testutils

I added docs about integration tests on two pages in https://github.com/stellar/soroban-docs/pull/88 and https://github.com/stellar/soroban-docs/pull/91. See those issues for screenshots.

authors: jonjove leighmcculloch mootz12

repo: stellar/rs-soroban-sdk


text:
Update GitHub Actions packages

v3 are using deprecated Node v16, updated to v4.

Preview is available here:http://stellar-demo-wallet-client-pr351.previews.kube001.services.stellar-ops.com/

authors: quietbits stellar-jenkins

repo: stellar/stellar-demo-wallet


text:
Make it possible to analyze data related to validator activity

What problem does your feature solve?
One of the foundations of SCP is that entities can adjust who they trust as validators on the network.
In a system build on trust, there needs to be mechanisms to verify/audit that entities are behaving the way they claim to be.
What would you like to see?
The data set relevant to this issue is:

which validator "won" nomination (validators sign the transaction set they introduce)
the mapping at the time concensus took place between a validator key and the entity it represents (stellar.toml)
the exact structure of the generalized transaction set that the winning validator introduced

Some of the reports that become possible:

audit that there is no bias for or against specific entities, and if some bias exists to understand it better so that validators can take corrective actions

some entities may be biases against because of performance issues for example
some entities may be biases towards because of some bug in the stack somewhere

make more transparent situations where validators are traffic shaping in some way

penalizing single transactions (create a "surge pricing group of one transaction") - as identified in https://github.com/stellar/stellar-protocol/blob/master/core/cap-0042.md#security-concerns
giving priority to certain tokens/trades (MEV exploitation like behavior)

What alternatives are there?
None, other than implement some other data lake that would contain this data.

@anupsdf FYI

the exact structure of the generalized transaction set that the winning validator introduced
Also, how about adding some metrics around the total transactions, min fee, max fee among each lane (tx set) from that winning validators's mempool? This will help analyze the separate lanes better and have insight into any biases or bugs when different tx sets are proposed.

@anupsdf the mempool structure is out of scope of this. I'd like to keep this to what can be verified based on what goes through consensus.
That being said, we should be able to have reports per tier 1 organization of the how fees are distributed and if lanes are being used.

the mapping at the time concensus took place between a validator key and the entity it represents (stellar.toml)
@MonsieurNicolas or @anupsdf is this information written to the ledger close meta during time of consensus? Or would this require us to pull the stellar.toml at runtime to pull the information?

(cc @chowbao)

the mapping at the time concensus took place between a validator key and the entity it represents (stellar.toml)
There doesn't seem to be any straightforward and efficient way today to construct this mapping.

Stellar tomls live on websites, and describe validators. These websites aren't particularly discoverable, and in particular are not discoverable from on-chain data.
SCP messages provide validator public keys but there's no guarantee that a given website publishing a toml claiming to control that key actually does so.

It seems to me that a pre-requisite for this work would be hardening SEP 1. If standard on-chain data entries published the hash of a toml and also the toml location off-chain, and if the toml published the validator information and the account, then

This is self-consistent and doesn't need to be checked on every scp message - it is enough to check for data entry change deltas
A scanning of on-chain data would allow a complete list of tomls, and therefore a complete list of validator entities to be easily constructed.
As an additional benefit, all auxiliary data in the toml would be verifiable, and changes to the toml would require changes to the corresponding data entry, thus forming a traceable and transparent record. Contrast this with today's situation, where a toml file can change dynamically at any time, with no visible revision history.

The data set relevant to this issue is:

the exact structure of the generalized transaction set that the winning validator introduced

@MonsieurNicolas when you say exact structure of the generalized transaction set, do you want an aggregated view of the transaction set at the ledger level? For example: classic phase: 2 component(s): [{discounted txs:24, ops:152, base_fee:100}, {discounted txs:429, ops:500, base_fee:51234}], soroban phase: 1 component(s): [{discounted txs:1, ops:1, base_fee:100}]
We save the fee data at the transactional level, so we have the ability to derive the base fee + count of transactions for each base fee. But for ease of use, we can aggregate that data at the ledger level and append to the table, as well.

Ah cool, beyond your initial request to add node_id to history_ledgers, we already store all data necessary to ask these types of questions in Hubble. From a raw data perspective, this is request is complete.
It may be interesting in the future to build some aggregated tables that makes your example analysis easier to conduct, but we should discuss which questions are important/relevant before building those models out.

authors: MonsieurNicolas anupsdf ire-and-curses sydneynotthecity

repo: stellar/stellar-etl


text:
add stellargraph docker image

To help us move forward, this PR is split from #368 so that it's only building the stellargraph/stellargraph docker image, until we can confirm hosting datasets (since stellargraph-evaluation is now using the stellargraph/stellargraph image)

Coverage remained the same at 84.253% when pulling c52e986165fb9c5fc71d12587400b824da937aea on feature/docker-stellargraph-only into da453463aaf6d455f166fbb10d94596b8c71684b on develop.

authors: TheDen coveralls

repo: stellargraph/stellargraph


text:
chore: add a docker-compose.yml to make running opencast easier

Add a docker-compose.yml that runs the indexer and opencast
You should be able to run opencast by providing env values and running docker-compose up
Might require publishing an image of the lazy indexer to the docker registry

Putting 100 USDC reward on this one!
Add your wallet address with the PR and on merge, message me at https://warpcast.com/dalechyn.eth.

https://github.com/stephancill/opencast/pull/41

fixed by #41

authors: dalechyn gabrieltemtsen stephancill

repo: stephancill/opencast


text:
preprocess hansard w2v data

acre dale to acredale
cableicum to cablicum
demense to demesn -- purposefully removed the "e" to match with more terms -- will need to update this in 1 grams
demesnial to demesne
fire-bot to firebot
frank ferm to frank-ferm
land occupier to land-occupier
plowbote to ploughbote
rack rent to rack-rent
run rig to runrig
run-ridge to runrig
socman to sokeman
sub fief to sub-fief
zamandar to zemindar
thane land to thaneland
thane-land to thaneland
comonage to commonage

lessee should be counted w/ lemma lease
Spelling precedence:
Words that can be either two words or hyphenated words are transformed to hyphenated words. Words that can be single space are transformed to single space.

I need to add word boundary regexs so that I don't replace the substring.

I created the spreadsheet: https://docs.google.com/spreadsheets/d/1vKuIAltMjwB0nq_wb815ssTUk_EJKr_Viozz-faoosw/edit#gid=0
But this doesn't fix the lemmatization issue -- that will be separate.

repo: stephbuon/democracy-lab


text:
Hosting

Fly.io has replicated hosting for Postgres and SQLite
it’s free up to a certain level .

https://fly.io/blog/free-postgres/

https://fly.io/docs/litefs/

https://github.com/athena-logistics/athena-backend/blob/main/.github/workflows/part_deploy.yml shows how to build the Container to Github Container Registry and then Deploy to Fly.io.

I am not sure what this has to do with Bob.

It’s to easily run and scale out and host bob .

repo: stephenafamo/bob


text:
Add and and or functions

When you're programmatically building a query to execute, you have times where you build a variable number of clauses by which you filter a table. There isn't a great way to turn an Array<Expression<Bool>> into an Expression<Bool>, short of making a large (a AND (b AND (c AND (d AND e)))) expression.
and and or fix this by allowing you to turn an Array<Expression<Bool>> directly into a lower-complexity clause: (a AND b AND c AND d AND e)

Cool!

Would this replace the .filter?

A year and a half later, I find myself needing this again.
Is there anyone around who's interested in merging this?

authors: KyleLeneau davedelong stephanheilner


text:
Get imu data from svo file

Hello，
When I use zed sdk 2.8.4, can I get imu data from svo file through python api?
The python api description is as follows:
In SVO reading mode, the TIME_REFERENCE_CURRENT is currently not available (yielding :data:~pyzed.sl.ERROR_CODE.ERROR_CODE_INVALID_FUNCTION_PARAMETERS. Only the quaternion data at TIME_REFERENCE_IMAGE is available. Other values will be set to 0.
Although this seems impossible, but I want to ask who has a solution？

Hi,
Unfortunately, the SVO only contains the quaternion (image sync) and not the raw IMU data currently.
There is no easy solution. You could record the raw IMU data in a CSV file by yourself alongside the SVO recording, and read it back with manual synchronization.

authors: GeniusLight adujardin

repo: stereolabs/zed-python-api