text:
Using Polyglot in Journal Entries Impossible

I am unable to use Polyglot in Journal Entries since updating to V10.
In the default sheet configuration, Polyglot no longer appears in the drop down menu at all:

When I change the sheet to TinyMCE, Polyglot appears, but is greyed out:

Game System and version

Polyglot Version: current
Are you using Babele? no
Foundry Version: 10 (current)
System: The Dark Eye (system and modules all up to date)

I can't reproduce this on a brand new world. Can you send me your world's settings.db? It should be on your world's folder/data folder.

I use The Forge, here's the settings.db from the world export (zipped to be able to attach it here).
settings.zip

Hm, it still works for me. Are there any errors on the Console (F12) when opening the editor?

I deactivated and reactivated Polyglot, these are the errors that I get in the Console:

Activate the DSA5 Core Rules module. Any other errors?

Hm, that's strange, the DSA5 Core Rules module was active when I took the screenshot yesterday.
I just updated Polyglot, and the problem's still there, with just one change in the first row:
LanguageProvider.js:1218 Uncaught (in promise) TypeError: Cannot read
It does not seem to be the core module itself, but the core special abilities?
		if (game.modules.has("dsa5-core") && game.modules.get("dsa5-core")?.active) {
			const dsa5Pack = game.packs.get("dsa5-core.corespecialabilites");
			const dsa5ItemList = await dsa5Pack.getIndex();

I also tried switching the language of my world from German to English to see if there might be an issue there (as "special abilities" are "sonderfertigkeiten" in German and the system has been known to get confused with such terminology differences between languages), but the issue with Polyglot is still the same in English as well.

Hey, I installed the latest update just now (where you changed the line), but the error still presents the same:

If you enter game.packs.get("dsa5-core.coreabilities") in the console, does it return anything other than undefined?
The id for the compendium with languages was changed from V9 to V10. I don't own the Core Rules, so I can't test stuff for DSA5.

Just spoke with Huepfklotz, there was a typo on that compendium's id. Be sure to update it and the error should be gone.

authors: KnoepfchenFulla mclemente


text:
Middleware broken on embedded docs

Trying to update a document with an embedded doc that has a middleware produces a MongoError: have conflicting mods in update
To reproduce:
var mongoose = require('mongoose'),
    Schema = mongoose.Schema;

mongoose.connect('mongodb://localhost/test');

var UserSchema = new Schema({
  username: String
});

// works if this middleware is removed
UserSchema.pre('save', function(next) {
  this.username += 'bar';
  next();
});

var GroupSchema = new Schema({
  users: [UserSchema]
});

var Group = mongoose.model('Group', GroupSchema);

var group = new Group();
group.users.push({username: 'foo'});
group.save(function(error) {
  if (error) throw error;
  Group.findById(group._id, function(error, result) {
    if (error) throw error;
    result.users.push({username: 'candy'});
    result.save(function(error) {
      if (error) throw error; // MongoError: have conflicting mods in update
      console.log(arguments);
    });
  });
});

thanks for the test case. this is the same root issue as #438. closing this one.

source: gharchive/issue

authors: aheckmann jnordberg

repo: LearnBoost/mongoose

license: MIT

license_type: permissive

license_source: github-api


text:
Add Fuzzer

This fuzzer allows going through all signTx commands.
It needs to change how the UI works a little bit, and does so by replacing UI_STEP macros.
If the behaviour of UI components change later, this fuzzer will need to be adapted.

@jibeee  for review ?

This PR should be squashed and merged once approved.

source: gharchive/pull-request

repo: LedgerHQ/app-cardano


text:
feat: add bob

Description
Added BOB mainnet chain id, see Chainlist.
Changes include

[ ] Bugfix (non-breaking change that solves an issue)
[ ] New feature (non-breaking change that adds functionality)
[ ] Breaking change (change that is not backwards-compatible and/or changes current functionality)
[ ] Tests
[ ] Documentation
[ ] Other (for changes that might not fit in any category)

Hey @gregdhill , thanks for your PR ! It seems that we are missing your 64px icon in the glyphs folder though. You should find more informations here on how to craft one.
Note : The naming format for the glyphs icon will soon be updated on the developer portal, but it should now be chain_yourchainid_64px.gif

Hi @gregdhill , we want to merge the new networks PR today, so can you please have a look ?

@vforgeoux-ledger added the glyph icon now using the new format. Let me know if that will suffice.

It doesn't respect exactly the guidelines. I've crafted this one for you :

Does it suit you ?

@vforgeoux-ledger assuming the logo should be square? I don't think that is mentioned explicitly in the guidelines. Uploaded a new version now, please check

I did the commit with one I crafted. Let me know if you really want the full black to replace it @gregdhill

Ah no should be all black @vforgeoux-ledger but can you let us know what guideline we are still missing?

I just pushed on my branch the new one you sent. Your change respects the guidelines now (apart from the file naming)

Included in https://github.com/LedgerHQ/app-ethereum/pull/598

authors: gregdhill vforgeoux-ledger

repo: LedgerHQ/app-ethereum


text:
🛑 CompanyGuide3 is down

In b74dc63, CompanyGuide3 (https://comp4.fnguide.com/server_variables.asp) was down:

HTTP code: 0
Response time: 0 ms

Resolved: CompanyGuide3 is back up in 3afe4f6.


text:
🛑 CompanyGuide1 is down

In d32e502, CompanyGuide1 (https://comp2.fnguide.com/server_variables.asp) was down:

HTTP code: 0
Response time: 0 ms

Resolved: CompanyGuide1 is back up in 2a33a23.


text:
🛑 CompanyGlobal2 is down

In 3d1ee7a, CompanyGlobal2 (https://compglobal2.wisereport.co.kr/Home/ServerCheck) was down:

HTTP code: 0
Response time: 0 ms

Resolved: CompanyGlobal2 is back up in 5ddf4fd after 8 minutes.


text:
No window

Describe the bug
Dunno why, but suddenly I can't see any chats, no windows pop up either.

There must be an addon conflict or other error causing this. I'm sorry, I am unable to reproduce this.

authors: AHMilling Pazza


text:
added support for LT100 Smart Lighting Towers

CLP Boards 0.2.0 is required to use LT100, so update it using the Arduino Board Manager. Open "Tools">"Board:">"Boards Manager...">search for "clp" and install the update.

4 leds towers direct output
LT100_4towers.zip


text:
[FEAT] Spiel fortsetzen

Letztes gespeichertes Spiel fortsetzen, inklusive Timer

Thilo: Baut SaveFile
Gerrit: Baut Menü

Macht Fortschritte

repo: LeichtMatrosee/Sudoku


text:
Java heap space error with big for loop.

Error:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space 	at lemon.jpizza.Generators.Interpreter.visit_ForNode(Interpreter.java:292) 	at lemon.jpizza.Generators.Interpreter.visit(Interpreter.java:138) 	at lemon.jpizza.Generators.Interpreter.visit_ListNode(Interpreter.java:224) 	at lemon.jpizza.Generators.Interpreter.visit(Interpreter.java:149) 	at lemon.jpizza.Shell.run(Shell.java:295) 	at lemon.jpizza.Shell.run(Shell.java:286) 	at lemon.jpizza.Shell.main(Shell.java:164)
Code:
for (i -> 0:1000000000) => println(i);
Explanation:
When trying to create a for loop with a massive amount of iterations, the interpreter dies before running any part of the loop. This may be dependent on the amount of ram the system has, so you might need to tweak the number to reproduce. This worked on a 2020 macbook air with 8gb of ram.

Error is caught and resolved in JPizza RTError. Cannot really be fixed, as is due to running out of memory.

authors: Lemon-Chad OmegaMetor

repo: Lemon-Chad/jpizza


text:
🛑 flibustang.online is down

In fa3c34d, flibustang.online (https://flibustang.online) was down:

HTTP code: 530
Response time: 9201 ms

Resolved: flibustang.online is back up in 8be8948.


text:
TO DO (initial set up)

Quick list of changes needed in order to adapt this for processing traffic footage:

[x] Allow multiple detection classes
[ ] Improve display functions to allow window closing
[x] Export by condition and not as default.

Original code already export a txt with all detections, ids and frames.

Original code already export a txt with all detections, ids and frames.

Couldn't allow for window closing when processing the video for now, but lower priority on this project.

Couldn't allow for window closing when processing the video for now, but lower priority on this project.


text:
unable to install/run stormloader.application

i am getting an error when i try to run stormloader.application. i have attached a copy of the log. it appears some files are missing?
S0ISK17J.log

Can you attach the log from stormloader, the log you have attached is not the one generated by stormloader

Hello I just got this and tried it today I am having the same issues and I am not folder savvy still trying to self learn develop so I will send to files hope they are the logs you are looking for
log.txt
error details straight from Cannot Start Application.txt

Hello I just got this and tried it today I am having the same issues and I am not folder savvy still trying to self learn develop so I will send to files hope they are the logs you are looking for log.txt error details straight from Cannot Start Application.txt

is seems to not be able to load a file before the application actually starts - the error isnt logged by stormloader so its a windows error. have you extracted stormloader before running it_

Yeah I have done the guide here and what discord said to the point I dont think its an issue on my end as its not just me other communities have complained this mod is not working for them either so I know for sure not just my laptop having the issue

Yeah I have done the guide here and what discord said to the point I dont think its an issue on my end as its not just me other communities have complained this mod is not working for them either so I know for sure not just my laptop having the issue

This appears to be an issue with net framework loading data files, the error occurs BEFORE stormloader starts loading any of its internal stuff - the error is specifically a windows error: System.Deployment.Application.DeploymentDownloadException as part of the JIT compiler for .Net, can you try running 'sfc /scannow' in an administrator command prompt

Can I check too that you are running the prebuilt application? your error relates to the clickonce application deployment, stormloader DOESNT use this, so unless you are building yourself and deploying through clickonce, i cant see how this error can occur

I dunno I give up I just did manual just going to suck when issues come up this loader would prevent and I dont get that protection I have never had any issue with anything else but nothing else also had a click once application and not so I dont really know it says its missing specific files every time I have had this error

Hey there. Having an issue preventing me from loading Stormloader, it keeps saying "not enough quota is available to process this command." I have attached my log
log.txt

Hey there. Having an issue preventing me from loading Stormloader, it keeps saying "not enough quota is available to process this command." I have attached my log log.txt

This is typically associated with having your pagefile switched off. Check if you have the pagefile enabled under performance settings in windows.

this is what I see, not sure if this is what you are talking about or not though

 this is what I see, not sure if this is what you are talking about or not though

Try putting it on another drive, running it as admin or rebooting - this is specifically a WINDOWS issue and not anything to do with the code for stormloader.

thanks for the advice, I'll get back to you depending on how it goes!

Restarting seemed to do the job, thank you

authors: BoppinAfro Lewinator56 Pomfe SolomonCM


text:
Some numbers are being provided as terms, which is messing up scenario editor display

I'm getting answers back like
"args": [
                    {
                      "args": [],
                      "functor": 736436181.2434658
                    }
                  ],
                  "functor": "duration"

which would mean duration(736436181.2434658())
I have no idea why s(CASP) would be returning that sort of term. I guess the solution is to check to see if it is happening and convert it.

so far this has only shown up in durations,, but it's fixed, there in v1.6.4-alpha

repo: Lexpedite/blawx


text:
Fix numba-related deprecation warnings

Contributor Checklist:

[ ] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases

Reviewer Checklist:

[ ] /azp run libertem.libertem-live-data passed

/azp run libertem.libertem-live-data

repo: LiberTEM/LiberTEM-live


text:
Function bms_soc_update no longer called post-zephyr bms driver update

After the recent Zephyr driver update (commit https://github.com/LibreSolar/bms-firmware/commit/f0ff664c2687e68f9526ed6f4d2f659b1bb0fa19), the bms_soc_update function seems to have stopped being invoked. This has led to the State of Charge (SoC) no longer being calculated. Prior to this update, the function was regularly called at this location: https://github.com/LibreSolar/bms-firmware/blob/39bbaeaf668e310dca520255687771dbc1d52ff8/app/src/bq769x2/bms_bq769x2.c#L154-L162
Could you advise on the best location for invoking bms_soc_update in the current code framework to maintain precise SoC measurements?

I recommend placing the call to bms_soc_update(&bms) before line 66, where bms_state_machine(&bms) is called.
https://github.com/LibreSolar/bms-firmware/blob/f0ff664c2687e68f9526ed6f4d2f659b1bb0fa19/app/src/main.cpp#L61-L66

Thanks for raising the issue. I'm planning to do some further BMS testing tomorrow anyway, so I'll have al look into that.

I agree with your proposed fix and have created a PR (see link above).

authors: martinjaeger pasrom

repo: LibreSolar/bms-firmware


text:
Callback system v0

Before submitting

[ ] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
[ ] Did you read the contributor guideline, Pull Request section?
[ ] Did you make sure to update the docs?
[ ] Did you write any new necessary tests?

⚠ How does this PR impact the user? ⚠
Allow users to trigger arbitrary functions via a callback system. This enables to inject useful functionalities like logging inference time, metrics collection, etc.

What does this PR do?
Implements a callback system with certain hooks that get triggered on events.
import litserve as ls

class MetricLogger(ls.Callback):
    def on_litapi_predict_start(self, lit_api: "LitAPI"):
        t0 = time.perf_counter()
        self._start_time = t0

    def on_litapi_predict_end(self, lit_api: "LitAPI"):
        t1 = time.perf_counter()
        elapsed = t1 - self._start_time
        # print or write to a DB like Grafana
        print(f"Prediction took {elapsed:.2f} seconds")

lit_api = ls.test_examples.SimpleLitAPI()
server = ls.LitServer(lit_api, callbacks=[MetricLogger()])
server.run()

A caveat of this implementation would be that even though the callbacks are classes, it might not be straightforward to save state between to hooks like on_litapi_predict_end and on_server_setup_start because they run in different process and once the callback object is pickled then we don't sync the state.
I think this is okay, considering that when in same context (on_litapi_predict_start and on_litapi_predict_end) on_litapi_*_* we will have the state in same process.

As a result of this PR, we will be able to refactor the monitoring metrics PR as a callback.
from prometheus_client import Summary
from prometheus_client import multiprocess, CollectorRegistry, make_asgi_app

def make_metrics_app():
    generate_metrics_dir()
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return make_asgi_app(registry=registry)

class PrometheusCallback(ls.Callback):
    def __init__(self):
        self.summary = Summary("PREDICT_METRIC", "A metric to summarize the predict execution times")

	def on_server_setup_start(self, litserver):
        metrics_app = make_metrics_app()
		litserver.app.mount("/metrics", metrics_app)

    def on_litapi_predict_start(self, lit_api: "LitAPI"):
        t0 = time.perf_counter()
        self._start_time = t0

    def on_litapi_predict_end(self, lit_api: "LitAPI"):
        t1 = time.perf_counter()
        elapsed = t1 - self._start_time
        self.summary.observe(elapsed)

TODO

[ ] Write test for all the loops

PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

@aniketmaurya
for the hook names for the litapi, remove the litapi words:
GOOD
on_setup_start
BAD
on_litapi_setup_start
For the litserver, what do these 2 do?

on_server_setup_start
on_server_setup_end

authors: aniketmaurya williamFalcon

repo: Lightning-AI/LitServe


text:
Improve code coverage for flash.core.data.utilities.paths

What does this PR do?

Improves code coverage for flash.core.data.utilities.paths
Fixes #1391
Before submitting

[x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
[x] Did you make sure to update the documentation with your changes?
[x] Did you write any new necessary tests? [not needed for typos/docs]
[x] Did you verify new and existing tests pass locally with your changes?
[x] If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

[x] Is this pull request ready for review? (if not, please submit in draft mode)

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Hey, @JustinGoheen - Any updates on this?

@krshrimali I am working several other PRs for Lightning, and have this slated for next week
https://github.com/users/JustinGoheen/projects/16/views/3

@krshrimali can you please let me know if this submission looks good, or if anything needs to be be changed

Hi, @JustinGoheen - I apologize for the delay. The Flash CI is down with around 14 reasonable failures, and I've prioritized it for this week - trying to get it done as soon as possible. Once done, it will allow me to come back to the amazing work you and other contributors have done in these PRs - and help merge them once they look good.
Unfortunately, I believe I'll be able to get back to this either by the end of this week or by the start of the next week. ❤ I hope that's fine with you. Thank you again for your work!

sounds good to me. thank you for the quick update.

Hey, unfortunately the fork is deleted, so we'll be unable to fix the failures and merge it. Thank you for your interest though, we appreciate that. :)

authors: JustinGoheen krshrimali

repo: Lightning-AI/lightning-flash


text:
DDP (multi GPU) Iterable Dataset is not working as expected ?

Bug description
Hi,
I am currently testing with IterableDataset and DDP.
Total Examples - 10000
Batch_size - 32
NUM_GPUS - 2 .
While using IterableDataset , ideally with 2 GPUS, we are supposed to run 157 steps (10000 / 32 batch / 2 gpus)  in one epoch. But, instead of that, it is running for  314 steps (10000 / 32 batch) .
This issue is only with IterableDataset. When I am using normal Dataset (map dataset) from torch things are good and fine. Is there any reason for this particular behaviour ?
How to reproduce the bug
import os
os.environ["TOKENIZERS_PARALLELISM"] = "false"
import lightning as L
import torch
import time
from datasets import list_datasets, load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from torch.utils.data import DataLoader, Dataset

BATCH_SIZE = 32
NUM_WORKERS = 1

# Load Dataset in Memory

imdb_data = load_dataset("imdb")
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def tokenize_text(batch):
    return tokenizer(batch["text"], truncation=True, padding=True)

imdb_dataset = imdb_data
imdb_tokenized = imdb_dataset.map(tokenize_text, batched=True, batch_size=None)
imdb_tokenized.set_format("torch", columns=["input_ids", "attention_mask", "label"])

def custom_iterator():
    counter = 0
    for item in imdb_tokenized['train']:

        inputs = {'input_ids': item['input_ids'], 'attention_mask': item['attention_mask']}
        labels = {'labels': item['label']}
        counter += 1
        yield inputs, labels

class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self):
        super().__init__()

    def __iter__(self):
        yield from custom_iterator()

train_dataset = MyIterableDataset()

train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=BATCH_SIZE,
    num_workers=NUM_WORKERS,
    persistent_workers=False
)

# Load Model
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2)

# Ligntning Module
class LightningModel(L.LightningModule):
    def __init__(self, model, learning_rate=5e-5):
        super().__init__()

        self.learning_rate = learning_rate
        self.model = model

    def forward(self, input_ids, attention_mask, labels):
        return self.model(input_ids, attention_mask=attention_mask, labels=labels)

    def training_step(self, batch, batch_idx):

        inputs, labels = batch
        outputs = self(inputs["input_ids"], attention_mask=inputs["attention_mask"],
                    labels=labels["labels"])
        self.log("train_loss", outputs["loss"])

        # print(" Tensor sum ", torch.sum(inputs['input_ids']))
        # print("-------------------")
        # print(3*"\n")

        self.log("tensor_sum", torch.sum(inputs['input_ids']))

        return outputs["loss"]  # this is passed to the optimizer for training

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
        return optimizer

lightning_model = LightningModel(model)

from pytorch_lightning.loggers import CSVLogger, WandbLogger

name = "train_ddp_map-iterable"
logger = CSVLogger(save_dir="logs/", name=name)
wandb_logger = WandbLogger(project="DDP_exps", name=name)

def train_model():

    max_epochs = 2

    if os.path.exists('checkpoints'):
        import shutil
        shutil.rmtree('checkpoints')

    trainer = L.Trainer(
        max_epochs=max_epochs,
        callbacks=None,
        accelerator="gpu",
        devices=[0, 1],
        logger=[logger, wandb_logger],
        strategy='ddp',
        enable_progress_bar=True, # Disable progress bar
        log_every_n_steps=1,
    )

    trainer.fit(model=lightning_model,
                train_dataloaders=train_loader)

if __name__=='__main__':

    start_time = time.time()
    train_model()
    end_time = time.time()
    print()
    print("Time taken to train model is {} seconds".format(end_time-start_time))

Error messages and logs

# Error messages and logs here please

Environment

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 1.10):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):

More info
No response

Yes this is expected. Lightning can't know how to shard the data/iterator you provide. You need to make sure your iterator returns half of the data on GPU 0 and the other half on GPU 1. You can do this for example by changing your for loop to something like this (typos expected):
for item in imdb_tokenized['train'][rank::num_gpus]:
    ...

This shards your data. The rank can be accessed for example through trainer.global_rank.
Another way would be to use the DistribuedSampler inside your iterable dataset.

Makes sense.
One possible way is, set steps per epoch to half of iterable ( i know how
many records are there ).
In keras steps per epoch concept is there. Here, is there any way to
mention, every epoch say run 1000 steps.
On Mon, 21 Nov 2022 at 12:18 AM, Adrian Wälchli @.***>
wrote:

Yes this is expected. Lightning can't know how to shard the data/iterator
you provide. You need to make sure your iterator returns half of the data
on GPU 0 and the other half on GPU 1. You can do this for example by
changing your for loop to something like this (typos expected):
for item in imdb_tokenized['train'][rank::num_gpus]:
...
This shards your data. The rank can be accessed for example through
trainer.global_rank.
Another way would be to use the DistribuedSampler
https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler
inside your iterable dataset.
—
Reply to this email directly, view it on GitHub
https://github.com/Lightning-AI/lightning/issues/15734#issuecomment-1321212040,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACRE6KCZQMYKE7HWQOP5Y5LWJJXAZANCNFSM6AAAAAASFEIFJM
.
You are receiving this because you authored the thread.Message ID:
@.***>

Makes sense.
One more doubt. In my DataLoader  if my num_workers=2, in each GPU, the whole training loop runs 2 times.
Say for eg:
Assume, my IterableDataset has 10000 records and my batch size = 32 . So total steps = 10000//32 = 314 .
If my num_workers=2, it is looping 628 times in each GPU . Is this expected ?
Because, num_workers=2 is supposed to make DataLoader pipeline faster right.
Is there any concept of steps_per_epoch in lightning. Say, epochs=10, steps_per_epoch=1000, I want each epoch to run 1000 loops max.
In keras steps per epoch concept is there. Here, is there any way to
mention, every epoch say run 1000 steps.

If my num_workers=2, it is looping 628 times in each GPU . Is this expected ?

No. num_workers has nothing to do with the sampling of the data.

Because, num_workers=2 is supposed to make DataLoader pipeline faster right.

Read more about workers here: https://pytorch.org/docs/stable/data.html#multi-process-data-loading

Is there any concept of steps_per_epoch in lightning. Say, epochs=10, steps_per_epoch=1000, I want each epoch to run 1000 loops max.

Trainer(limit_train_batches=1000, max_epochs=10)

If my num_workers=2, it is looping 628 times in each GPU . Is this expected ?

No. num_workers has nothing to do with the sampling of the data.

Thanks. But, if you could execute the above code by NUM_WORKERS=2, you can notice that, in each GPUs ( 2 GPUS I have ) , it rans for 626 steps. Expected was 313 steps = 10000//32.

If NUM_WORKERS=1, its working as expected .

Sharing results.

Any update

Yes this is expected. Lightning can't know how to shard the data/iterator you provide. You need to make sure your iterator returns half of the data on GPU 0 and the other half on GPU 1. You can do this for example by changing your for loop to something like this (typos expected):
for item in imdb_tokenized['train'][rank::num_gpus]:
    ...

This shards your data. The rank can be accessed for example through trainer.global_rank. If you do this, you need to make sure the iterator returns the same amount of data on each rank (e.g., drop the remainder)
Another way would be to use the DistribuedSampler inside your iterable dataset.

@awaelchli, after trying to use PyTorch's DistributedSampler with an IterableDataset in my application, I observed that DistributedSampler raised an error saying it requires each input dataset to have a len() property. Does this match your understanding, given the context of this discussion? If so, how might we use the DistributedSampler to circumvent the original issue in this forum?

@awaelchli, I second this issue. I am also having difficulties figuring out the simplest way to enable multi-GPU and multi-dataloader worker support for IterableDatasets when using PyTorch Lightning. All the examples I have worked through so far do not seem to work when considering both of the following cases: (1) num_workers>0 and len(trainer.devices)>0.

Yes, these observations are all expected. This is not special behavior in Lightning, it's just how the IterableDataset and DataLoader are working in PyTorch. In short: When using an iterable dataset, you need to take care of the sampler inside your dataset yourself, and shard/partition the data yourself across workers and devices.

I really don’t know why Pytorch is so preferred, despite such a complicated
and clumsy distribution strategies .
In Tensorflow its a cakewalk.
On Thu, 29 Dec 2022 at 8:12 PM, Adrian Wälchli @.***>
wrote:

Yes, these observations are all expected. This is not special behavior in
Lightning, it's just how the IterableDataset and DataLoader are working in
PyTorch. In short: When using an iterable dataset, you need to take care of
the sampler inside your dataset yourself, and shard/partition the data
yourself across workers and devices.
—
Reply to this email directly, view it on GitHub
https://github.com/Lightning-AI/lightning/issues/15734#issuecomment-1367377541,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACRE6KHRFHTBEQSAXZGPQ3DWPWPLVANCNFSM6AAAAAASFEIFJM
.
You are receiving this because you authored the thread.Message ID:
@.***>

@s4sarath let's stay on topic.
Here is a notebook that explains the difference between the map dataset and iterable dataset with several examples, using dataloader workers and also shows how it behaves across multiple processes. At the bottom, I also show the example with Lightning. I hope this helps your understanding.

Thanks man. Will have a look
On Tue, 3 Jan 2023 at 2:49 PM, Adrian Wälchli @.***>
wrote:

@s4sarath https://github.com/s4sarath let's stay on topic.
Here is a notebook
https://colab.research.google.com/drive/1OFLZnX9y5QUFNONuvFsxOizq4M-tFvk-?usp=sharing
that explains the difference between the map dataset and iterable dataset
with several examples, using dataloader workers and also shows how it
behaves across multiple processes. At the bottom, I also show the example
with Lightning. I hope this helps your understanding.
—
Reply to this email directly, view it on GitHub
https://github.com/Lightning-AI/lightning/issues/15734#issuecomment-1369541608,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACRE6KAHQZKPOVGB2YFRAE3WQPVI5ANCNFSM6AAAAAASFEIFJM
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thanks @awaelchli for the detailed notebook, its really helpful! One question I have is, does this only work with PyTorch's inbuilt parallel strategies, or will it work for other strategies like DeepSpeed? Do we need to call a pytorch lightning get_rank/worker_info function which abstracts away the underlying strategy, or does calling the torch function always guarantee we get the correct information regardless of strategy?

@gorold It should work with deepspeed yes, but probably not with the TPU strategy.
I haven't mentioned it in the notebook, but PyTorch is developing torchdata which will address these issues completely, as it is heavily focusing on performant iterable-style data loading together with DataLoader2. It would eliminate most of the boilerplate code I show in that notebook.

Thanks a lot!

Thanks so much for sharing! @awaelchli https://github.com/Lightning-AI/lightning/issues/15734#issuecomment-1369541608
Is there an implementation that would work with an iterable dataset of unknown length?

With a pure PyTorch Iterable dataset, I don't know how to do that cleanly.
But I think if you define the dataset as a data pipe in torchdata, you should be able to add a sharding filter that can handle that.

@s4sarath let's stay on topic.
@s4sarath @amorehead Here is a notebook that explains the difference between the map dataset and iterable dataset with several examples, using dataloader workers and also shows how it behaves across multiple processes. At the bottom, I also show the example with Lightning. I hope this helps your understanding.

This notebook seems to work in it's most basic form for me but for some reason when I implement this strategy with a list of tensors, the distributed sampler doesn't return the tensors in their original form. Instead, I get tensors with totally shuffled numbers from my dataloader. Any idea why this would be the case?

@keenjo By default, the distributed sampler shuffles the data. It has an argument DistributedSampler(shuffle=True|False).

@awaelchli No, but in my iterable dataset every datapoint is essentially a list of integers and so when calling the dataloader I would expect it to create a batch which would be a list of lists in the end. However, instead the distributed sampler is sampling specific items from my lists, which totally messes up the data. Here is a toy example of what I mean below:
dataset = [[1,2,3,4], [5,6,7,8], [9,10,11,12], [13,14,15,16]]

I would expect a batch of 2, for instance, to look like this:
batch = [[1,2,3,4], [9,10,11,12]]

Instead it looks like this:
batch = [[1,11,9,3], [4,12,2,10]]

@awaelchli Are you suggesting that if you use torchdata's iterable dataset multi-GPU should be handled cleanly "out of the box"? That would be great because I'm struggling to get it to work still with the PyTorch IterableDataset.

I don't understand how this code from that Colab notebook actually works:
class DataParallelIterableDataset(IterableDataset):

    def __len__(self):
        # Caveat: When using DistributedSampler, we need to know the number of samples in our dataset!
        # Hence, we need to implement `__len__`.
        return NUM_SAMPLES

    def __iter__(self):
        worker_info = get_worker_info()
        num_workers = worker_info.num_workers if worker_info is not None else 1
        worker_id = worker_info.id if worker_info is not None else 0

        world_size = get_world_size()
        process_rank = get_rank()

        sampler = DistributedSampler(self, num_replicas=(num_workers * world_size), rank=(process_rank * num_workers + worker_id), shuffle=False)

        for i in iter(sampler):
            yield i

Where is the data actually coming from in this example?

When I add the two lines to get world size and process rank to my __iter__ code, it freezes my script. :(

I don't understand how this code from that Colab notebook actually works:

For distributed training, each process will call the dataset independently:
iterator = iter(dataset)
process 0: next(iterator)
process 1: next(iterator)
...
If we don't put any distribute sampling inside of our dataset, each process would get the same samples:
process 0: next(iterator), next(iterator), next(iterator) -> [0, 1, 2]
process 1: next(iterator), next(iterator), next(iterator) -> [0, 1, 2]
This would render data-parallel completely useless. Instead, if we add the distributed sampler, then we make each process will return different data:
process 0: next(iterator), next(iterator), next(iterator) -> [0, 2, 4]
process 1: next(iterator), next(iterator), next(iterator) -> [1, 3, 5]
You can study the output of the notebook cell to see the same thing.

What I'm failing to understand is how in practice to pass the rank and world_size to the dataset when that is being created by my DataModule, before the Trainer is created. It seems that for this to work the Trainer is supposed to pass the rank somehow to the dataset. I can't figure out from your example notebook how to do this. When I try to access the rank and/or world_size in my Dataset before the Trainer is created, it either freezes during runtime or says I need to use init_process_group. It would be great to see a full example in the Lightning docs how to use multi-GPU with IterableDataset to make it more clear.

@EvanZ I was also confused about this at first, but then figured it out. The Trainer does not need any information about the data to be instantiated. So I would recommend instantiating the Trainer first, then you can pass the trainer.world_size and trainer.global_rank to your data module without any issues. Hope this helps!

One question that I guess seems obvious to you guys but not to me, do I have to explicitly call init_process_group? If so, where should that be done in a typical Lightning script? As far as I can tell you have to pass a rank, but how do you pass a rank if it hasn't been initiated? This is super confusing to me. I was assuming Lightning takes care of these details.

@EvanZ I was also confused about this at first, but then figured it out. The Trainer does not need any information about the data to be instantiated. So I would recommend instantiating the Trainer first, then you can pass the trainer.world_size and trainer.global_rank to your data module without any issues. Hope this helps!

Hmm indeed that is helpful (in theory haha). Currently my training script looks like:
        dm = MyIterableDataModule(
            train=args.train,
            valid=args.valid,
            test=args.test,
            vocab_path=args.vocab_path,
            max_length=args.max_length,
            num_workers=args.num_workers,
            batch_size=args.batch_size,
            label_name=args.label,
            exposure_name=args.exposure
        )
        dm.setup(stage='fit')

        model = MyLightningModel(
            path=args.pretrained_model_path,
            use_exposure_as_weights=args.weights,
            num_tokens=len(vocab),
            dim_model=args.embed_dims,
            dim_h=args.hidden_dims,
            num_layers=args.num_layers,
            num_heads=args.num_heads,
            lr=args.lr
        )

        trainer = pl.Trainer(
            fast_dev_run=args.fast,
            devices=args.gpus,
            accelerator=args.acc,
            callbacks=[early_stopping_cb,
                       checkpoint_cb,
                       TQDMProgressBar(refresh_rate=1)],
            precision=args.precision,
            strategy='ddp',
            reload_dataloaders_every_n_epochs=1,
            limit_train_batches=train_batches,
            limit_val_batches=valid_batches,
            limit_test_batches=test_batches,
            auto_lr_find=True,
            accumulate_grad_batches=args.acc_grads
        )

You're saying if I just flip the Trainer before the DataModule then I will be able to access the rank and the world size inside the Dataset?

Yes, that's exactly it!

Ok...maybe that's the missing detail I needed. I'll work on it some more!

I should also mention that I used a different strategy to solve this problem in the end using itertools.islice to avoid repeating data and my Iterable Dataset ended up looking like this:
class CustomDataset(IterableDataset):
    def __init__(self, tokenizer, filepath, rank, world_size, stage='train'):
        super().__init__()
        self.tokenizer = tokenizer
        self.filepath = filepath
        self.stage = stage
        self.rank = rank
        self.world_size = world_size

    def __iter__(self):
        assert self.stage in ['train', 'val', 'test']
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is not None:
            num_workers = worker_info.num_workers
            worker_id = torch.utils.data.get_worker_info().id
            world_size = self.world_size
            rank = self.rank
        if self.stage == 'train':
            train_iter_source = open(f'{self.filepath}/train.source')
            train_iter_target = open(f'{self.filepath}/train.target')
            train_set = zip(train_iter_source, train_iter_target)
            mapped_itr = map(self.no_newlines, train_set)
            tok_itr = map(self.tokenize_inputs, mapped_itr)
        elif self.stage == 'val':
            val_iter_source = open(f'{self.filepath}/val.source')
            val_iter_target = open(f'{self.filepath}/val.target')
            val_set = zip(val_iter_source, val_iter_target)
            mapped_itr = map(self.no_newlines, val_set)
            tok_itr = map(self.tokenize_inputs, mapped_itr)
        elif self.stage == 'test':
            test_iter_source = open(f'{self.filepath}/test_both.source')
            test_iter_target = open(f'{self.filepath}/test_both.target')
            test_set = zip(test_iter_source, test_iter_target)
            mapped_itr = map(self.no_newlines, test_set)
            tok_itr = map(self.tokenize_inputs, mapped_itr)

        if worker_info is not None:
            if rank == 0:
                tok_itr = itertools.islice(tok_itr, worker_id, None, (num_workers * world_size))
            else:
                tok_itr = itertools.islice(tok_itr, worker_id + (num_workers * rank), None, (num_workers * world_size))

        return tok_itr

    def no_newlines(self, lines):
        '''
        Function to take new lines out of inputs
        '''
        lines = list(lines)

        for idx, line in enumerate(lines):
            lines[idx] = line.strip('\n')

        return lines

    def tokenize_inputs(self, lines):
        '''
        Function to tokenize a batch of lines that are read
        '''
        lines_tok = self.tokenizer.batch_encode_plus(lines,
                                                    return_special_tokens_mask=False,
                                                    add_special_tokens=False)['input_ids']
        return lines_tok

Hmm that's interesting and a different organization than I use. I define {train/val/test}_dataloader inside my DataModule. I do currently use islice as well though like this:
    def __iter__(self) -> Iterator[dict]:
        worker_total_num = torch.utils.data.get_worker_info().num_workers
        worker_id = torch.utils.data.get_worker_info().id
        for file in self.files:
            if self.compressed:
                fopen = BZ2File(filename=file, mode='r')
            else:
                fopen = open(file=file, mode='r')

            with fopen as f:
                for row in islice(f,worker_id,None,worker_total_num):
                    data = json.loads(row)
                    tokens = data['words']
                    random.shuffle(tokens)
                    tokens = tokens[:self.max_length]
                    indices, mask = self.tokens_to_indices(tokens)

                    item = {
                        'src': indices,
                        'mask': mask,
                        'label': data['label']
                    }

                    yield item

@keenjo Where do you pass in the rank and world size from? I assume you have a custom DataModule class somewhere?

If I try to use get_rank in the DataModule class, I get the following error:
Default process group has not been initialized, please make sure to call init_process_group
Even though I have instantiated the Trainer before the DataModule...Do I need to call init_process_group explicitly as well?

Basically anywhere in my script I try to call get_rank or get_world_size I get errors telling me to use init_process_group, but how can you initialize the process group if you don't have the rank? It seems like a chicken and egg scenario. What am I missing here?

I have a main training script which is where I get the initial rank and world size from and pass them to my Data Module:
    # Instantiate the trainer
    trainer = Trainer(accelerator='gpu',
                      devices=n_gpu,
                      precision=16,
                      val_check_interval=args.val_check_interval,
                      strategy='deepspeed_stage_2',
                      # logger=logger,
                      max_epochs=args.max_train_epochs,
                      callbacks=[checkpoint_callback,
                                 #grad_accumulation,
                                 lr_monitor,
                                 early_stop_callback,
                                 cb.TQDMProgressBar(),
                                 pred_writer])

    rank = trainer.global_rank
    world_size = trainer.world_size

    # Instantiate the data collator and data module
    collate_fn = DataCollatorForSeq2SeqWithMaskingAndPadding(tokenizer=tok, max_length=args.max_length, padding=True)
    dataset = LitRDF2TextDataModule(tokenizer=tok, train_batch_size=args.train_batch_size,
                                    eval_batch_size=args.eval_batch_size, collate_fn=collate_fn,
                                    data_path=args.data_path, rank=rank, world_size=world_size, buffer_size=args.buffer_size)

Then within my Data Module I pass the rank and world size the the Iterable Dataset that I posted earlier:
# Pytorch Lightning Data Module
class LitRDF2TextDataModule(pl.LightningDataModule):
    """
    Class to prepare data and split it into Dataloaders
    - Look into Pytorch Lightning 1.9.4 Documentation for more information:
        - https://lightning.ai/docs/stable/
    """

    def __init__(self, tokenizer, train_batch_size, eval_batch_size, data_path, collate_fn,
                 rank, world_size, buffer_size):
        super().__init__()
        self.tokenizer = tokenizer  # tokenizer to use on the dataset
        self.data_path = data_path  # path to dataset
        self.train_batch_size = train_batch_size  # train batch size
        self.eval_batch_size = eval_batch_size  # eval batch size
        self.collate_fn = collate_fn
        self.rank = rank
        self.world_size = world_size
        self.buffer_size = buffer_size

    def setup(self, stage):
        '''
        Method to prepare data to be passed to dataloaders
            - Specifically this creates a pytorch Dataset object for each split of the data
              and prepares the data for masking
        '''
        print(f'Preparing {stage} data...')
        if stage == 'fit' or stage == 'validation':
            self.dataset_train = CustomDataset(tokenizer=self.tokenizer,
                                               filepath=self.data_path,
                                               rank=self.rank,
                                               world_size=self.world_size,
                                               stage='train')
...
...

I'm confused. I thought the rank refers to each GPU. How does that work if you are calling it in the main script? How does it ever change?

When I try to call trainer.global_rank and trainer.world_size in my main script all I get is the following error:
ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set
There has to be something that I'm missing here.

I've ensured that my model is using all of the GPUs by printing out the ranks during various processes. Plus, before I solved my problem I had the issue of my data being duplicated across all of the GPUs. I'm not exactly sure what could be causing your error, but just in case there is incompatibility between the lightning versions we are working with, I'm using pytorch-lightning version 1.9.4

I am using 1.9.4 as well.
It boggles my mind. The rank is 0 in the main script. You are passing that to your dataset. How can it ever change? I just don't get the logic.

When I pass trainer.global_rank and trainer.world_size to my DataModule, I just get 0 and 4.

I don't understand how this code from that Colab notebook actually works:

For distributed training, each process will call the dataset independently:
iterator = iter(dataset) process 0: next(iterator) process 1: next(iterator) ...
If we don't put any distribute sampling inside of our dataset, each process would get the same samples: process 0: next(iterator), next(iterator), next(iterator) -> [0, 1, 2] process 1: next(iterator), next(iterator), next(iterator) -> [0, 1, 2]
This would render data-parallel completely useless. Instead, if we add the distributed sampler, then we make each process will return different data:
process 0: next(iterator), next(iterator), next(iterator) -> [0, 2, 4] process 1: next(iterator), next(iterator), next(iterator) -> [1, 3, 5]
You can study the output of the notebook cell to see the same thing.

When I try adding any code that accesses the rank or world size in my IterableDataset, I just keep getting an error saying I need to use init_process_group:
 Exception has occurred: RuntimeError Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last):   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop     data = fetcher.fetch(index)   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch     data.append(next(self.dataset_iter))   File "/home/ubuntu/deep-behavior-embedding/src/data/dataset.py", line 340, in __iter__     world_size = get_world_size()   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1067, in get_world_size     return _get_group_size(group)   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 453, in _get_group_size     default_pg = _get_default_group()   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 585, in _get_default_group     "Default process group has not been initialized, " RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.   File "/home/ubuntu/deep-behavior-embedding/finetune_frequency_iterable.py", line 159, in <module>     batch = next(tdl) RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last):   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop     data = fetcher.fetch(index)   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch     data.append(next(self.dataset_iter))   File "/home/ubuntu/deep-behavior-embedding/src/data/dataset.py", line 340, in __iter__     world_size = get_world_size()   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1067, in get_world_size     return _get_group_size(group)   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 453, in _get_group_size     default_pg = _get_default_group()   File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 585, in _get_default_group     "Default process group has not been initialized, " RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
But when I try to call init_process_group I then get an error saying I need to pass it a rank. But I can't pass it a rank because I can't get the rank! There has to be some logic that someone explain in a straightforward way to get this to work? I love Lightning, but this is getting really frustrating.

I have taken the example from the notebook and tried to create a minimal simple script to run the Lightning example on a 4-GPU machine:
from torch.utils.data import DistributedSampler, IterableDataset, get_worker_info, DataLoader
from torch.distributed import init_process_group, get_rank, get_world_size
import torch.nn as nn
import pytorch_lightning as pl
from torch.optim import Adam

NUM_PROCESSES = 4
NUM_SAMPLES = 16

def print_on_rank(string):
    rank = get_rank()
    print(f"[rank={rank}]{string}")

class DataParallelIterableDataset(IterableDataset):

    def __len__(self):
        # Caveat: When using DistributedSampler, we need to know the number of samples in our dataset!
        # Hence, we need to implement `__len__`.
        return NUM_SAMPLES

    def __iter__(self):
        worker_info = get_worker_info()
        num_workers = worker_info.num_workers if worker_info is not None else 1
        worker_id = worker_info.id if worker_info is not None else 0

        world_size = get_world_size()
        process_rank = get_rank()

        sampler = DistributedSampler(self, num_replicas=(num_workers * world_size), rank=(process_rank * num_workers + worker_id), shuffle=False)

        for i in iter(sampler):
            yield i

class Model(pl.LightningModule):

    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 2)

    def training_step(self, batch, batch_idx):
        print_on_rank(batch)
        return None

    def configure_optimizers(self):
        # required by Trainer, but not relevant for this test
        return Adam(self.parameters())

if __name__=="__main__":
    model = Model()

    dataset = DataParallelIterableDataset()
    dataloader_iterable_dataset = DataLoader(dataset, batch_size=4, num_workers=2, shuffle=False)

    EXTRA_TRAINER_OPTIONS=dict(
        max_epochs=1,
        enable_checkpointing=False,
        enable_model_summary=False,
        enable_progress_bar=False,
        logger=False,
    )
    trainer = pl.Trainer(
        accelerator="gpu",
        devices=NUM_PROCESSES,
        **EXTRA_TRAINER_OPTIONS,
    )
    trainer.fit(model, train_dataloaders=dataloader_iterable_dataset)

This throws the same type of error I am getting in my own script, so maybe we're getting somewhere now:
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/4
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4
Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/4
Initializing distributed: GLOBAL_RANK: 3, MEMBER: 4/4
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 4 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 3 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:209: UserWarning: num_workers>0, persistent_workers=False, and strategy=ddp_spawn may result in data loading bottlenecks. Consider setting persistent_workers=True (this is a limitation of Python .spawn() and PyTorch)
  "num_workers>0, persistent_workers=False, and strategy=ddp_spawn"
/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/lightning_fabric/utilities/data.py:64: UserWarning: Your `IterableDataset` has `__len__` defined. In combination with multi-process data loading (when num_workers > 1), `__len__` could be inaccurate if each worker is not configured independently to avoid having duplicate data.
  "Your `IterableDataset` has `__len__` defined."
Traceback (most recent call last):
  File "/home/ubuntu/deep-behavior-embedding/example_distributed_iterable.py", line 68, in <module>
    trainer.fit(model, train_dataloaders=dataloader_iterable_dataset)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 609, in fit
    self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 117, in launch
    start_method=self._start_method,
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 139, in _wrapping_function
    results = function(*args, **kwargs)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1214, in _run_train
    self.fit_loop.run()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 187, in advance
    batch = next(data_fetcher)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in __next__
    return self.fetching_function()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/utilities/fetching.py", line 265, in fetching_function
    self._fetch_next_batch(self.dataloader_iter)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/utilities/fetching.py", line 280, in _fetch_next_batch
    batch = next(iterator)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/supporters.py", line 569, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/pytorch_lightning/trainer/supporters.py", line 581, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/lightning_utilities/core/apply_func.py", line 51, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
    return self._process_data(data)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
    data.reraise()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch
    data.append(next(self.dataset_iter))
  File "/home/ubuntu/deep-behavior-embedding/example_distributed_iterable.py", line 26, in __iter__
    world_size = get_world_size()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1067, in get_world_size
    return _get_group_size(group)
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 453, in _get_group_size
    default_pg = _get_default_group()
  File "/home/ubuntu/deep-behavior-embedding/.virtualenv/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 585, in _get_default_group
    "Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

In the notebook example, as far as I can tell, init_process_group was not being called for the Lightning example either. What is going on here?

I finally got everything to "work" in the sense that I can see batches of data being sent to each GPU and the results appear to be similar to what I was getting before using just 1 GPU...only problem is I am not actually seeing any speedup. It seems all this effort was for not. :/

class DataParallelIterableDataset(IterableDataset):

    def __len__(self):
        # Caveat: When using DistributedSampler, we need to know the number of samples in our dataset!
        # Hence, we need to implement `__len__`.
        return NUM_SAMPLES

    def __iter__(self):
        worker_info = get_worker_info()
        num_workers = worker_info.num_workers if worker_info is not None else 1
        worker_id = worker_info.id if worker_info is not None else 0

        world_size = get_world_size()
        process_rank = get_rank()

        sampler = DistributedSampler(self, num_replicas=(num_workers * world_size), rank=(process_rank * num_workers + worker_id), shuffle=False)

        for i in iter(sampler):
            yield i

I want to ask if I want to read a file line by line using this class, and  I initialize one instance using method
class MyIterableDataset(IterableDataset):
    """
    https://colab.research.google.com/drive/1OFLZnX9y5QUFNONuvFsxOizq4M-tFvk-?usp=sharing
    https://github.com/Lightning-AI/lightning/issues/15734
    """

    def __init__(self, path, args, encoder_tokenizer, ent_tokenizer, rel_tokenizer, ins_num,  mode):

        super(MyIterableDataset, self).__init__()
        self.path = path
        self.args = args
        self.mode = mode
        self.encoder_tokenizer = encoder_tokenizer
        self.ent_tokenizer = ent_tokenizer
        self.rel_tokenizer = rel_tokenizer
        self.ins_num = ins_num

Where can I put the data path self.path = path into the tier() func ?

I’m also having difficulty understanding how the rank can change when it’s initially set to 0 in the main script and then passed to the dataset.

It boggles my mind. The rank is 0 in the main script. You are passing that to your dataset. How can it ever change? I just don't get the logic.

@EvanZ did you ever find an answer to your question? I’m facing a similar issue and would really appreciate some insight.
In the script @awaelchli provided, this seems to work in the non-Lightning version with:

mp.start_processes(fixed_data_parallel_with_iterable_dataset, nprocs=NUM_PROCESSES, start_method="fork")
init_process(rank).

However, I’m struggling to understand how to achieve the same behavior in the Lightning version, where mp.start_processes and init_process don’t seem to be utilized.

I'm also quite confused by this. I hoped PyTorch Lightning would provide more clear guidance + examples (though the notebook is an excellent start!).
Would using worker_init_fn() help better organize the code for a DataParallelIterableDataset()?
See how it is used in the PyTorch documentation: https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

A couple quick questions about the DataParallelIterableDataset() function provided in the above notebook:
class DataParallelIterableDataset(IterableDataset):

    def __len__(self):
        # Caveat: When using DistributedSampler, we need to know the number of samples in our dataset!
        # Hence, we need to implement `__len__`.
        return NUM_SAMPLES

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        num_workers = worker_info.num_workers if worker_info is not None else 1
        worker_id = worker_info.id if worker_info is not None else 0

        world_size = torch.distributed.get_world_size()
        process_rank = torch.distributed.get_rank()

        sampler = torch.utils.data.DistributedSampler(
            self,
            num_replicas=num_workers * world_size,
            rank=process_rank * num_workers + worker_id,
            shuffle=False
        )

        for i in iter(sampler):
            yield I

Why is there no __init__() class?
When we are dealing with real data, where + how do we pass this in? Does it need to get passed to the DistributedSampler() somehow via self?

authors: EvanZ YiandLi amorehead awaelchli denisbeslic gil2rok gorold keenjo s4sarath senzeyu

repo: Lightning-AI/lightning


text:
README stats are way off

What is the memory consumption of the default arguments in finetuning/adapter_v2.py?
This guide would lead one to believe it's 16GB-ish. I have a 2x48GB GPU and there are still OOM issues.
Either the guide is way off or the scripts have been updated?

Answering my own question with this link.

repo: Lightning-AI/lit-gpt


text:
🛑 Siyntra Yggdrasil Multiplayer Session Service is down

In e491e62, Siyntra Yggdrasil Multiplayer Session Service (https://auth.siyntrastudios.com/session) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Siyntra Yggdrasil Multiplayer Session Service is back up in 2e4a67c after 37 minutes.

repo: Limbo-Studios/siyntra-status-page


text:
🛑 SkyBlock is down

In 21c5554, SkyBlock ($SERVER_4) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SkyBlock is back up in 9fbc922.

repo: LimeNetwork/limestatus


text:
Test Debug - forkmalloc.c

per #150.
Forkmalloc fails, before printing pointer address.
Local/expected run:
@(1:60) $ ./forkmalloc
in child
pointer address is 0x564f78a6f6b0
done

Lind run:
@(1:64) $ lind /lind/forkmalloc.nexe

executing: [sel_ldr -a -- "/lib/glibc/runnable-ld.so" --library-path "/lib/glibc" /lind/forkmalloc.nexe]
[4940,2013030400:06:38:41.089810] BYPASSING ALL ACL CHECKS
Error in syscall: access - ENOENT: path does not refer to an existing file
in child

** Signal 11 from untrusted code: pc=297500034882

Full verbose output (-vvvv) can be found at:
forkmalloc-verbose.txt

I went back and built old lind, apparently this didn't work there either and was somehow overlooked.
What's weird is if we do program start -> malloc -> fork -> malloc in child != segfault
but in this case if we program start -> fork -> malloc in child = segfault
It seems to break in the malloc implementation from @youngfalcon 's digging, so this may be an arena thing.

@youngfalcon @kapkic
I'm relieving you guys from duty on this one, it seems a bit more complex.

Some scenario notes here:
Case 1: Malloc -> Fork -> Malloc-Child = No Segfault
Case 2: No Malloc -> Fork -> Malloc in both Parent + Child = Segfault
Case 3: No Malloc -> Fork -> Malloc in Parent = No Segfault
Case 4: No Malloc -> Fork -> Malloc in Child = Segfault
Case 5: Malloc -> Fork -> Malloc-Parent = No Segfault
This is reminiscent of the long malloc hunt I did a long time ago. That was due to problems with the TLS index variable. I have a strong gut feeling that is probably is whats happening here

As per this gdb output:
@ [#7:NaClSyscallCSegHook()] n
168       NaClSwitchToApp(natp);
@ [#7:NaClSyscallCSegHook()] s
NaClSwitchToApp (natp=0x7fffec003a10) at src/trusted/service_runtime/arch/x86_64/nacl_switch_to_app_64.c:99
99        NaClSwitch(&natp->user);
@ [#7:NaClSwitchToApp()] s
NaClSwitchSSE () at src/trusted/service_runtime/arch/x86_64/nacl_switch_64.S:128
128             switcher NaClSwitchSSE, 0
@ [#7:NaClSwitchSSE()] s
Warning:
Cannot insert breakpoint 0.
Cannot access memory at address 0x0

0x0000435a0003f6e0 in ?? ()
@ [#7:None()] n
Cannot find bounds of current function
@ [#7:None()] c
Continuing.

Thread 7 "sel_ldr" received signal SIGSEGV, Segmentation fault.
0x0000435a00034882 in ?? ()
@ [#7:None()] bt
#0  0x0000435a00034882 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x1008000000000000 in ?? ()
#3  0x00000000100709d0 in ?? ()
#4  0x0000435a011977a0 in ?? ()
#5  0x00000000110e07c0 in ?? ()
#6  0x0000000000000fa8 in ?? ()
#7  0x0000000000000000 in ?? ()

It can be seen that in NaClSwitch(&natp->user);, NaClSyscallCSegHook() faces a segmentation fault in NaClSwitchToApp(natp) after NaClSysBrk returns.

After linking GLibC to  GDB, we get the following stack at the point of failure:
Thread 7 "sel_ldr" received signal SIGSEGV, Segmentation fault.
0x00004ab500034882 in ?? ()
@ [#7:None()] bt
#0  0x00004ab500034882 in ?? ()
#1  0x0000000000000001 in ?? ()
#2  0x1008000000000000 in ?? ()
#3  0x00000000100709d0 in ?? ()
#4  0x00004ab5011977a0 in *__GI___sbrk (increment=0) at sbrk.c:50
#5  0x00004ab501108b60 in *__GI___default_morecore (increment=268700136) at morecore.c:49
#6  0x00004ab501103d40 in sYSMALLOc (av=<optimized out>, nb=<optimized out>) at ../malloc/malloc.c:3256
#7  _int_malloc (av=0x110e07c0, bytes=<optimized out>) at ../malloc/malloc.c:4580
#8  0x00004ab501106b00 in *__GI___libc_malloc (bytes=4000) at ../malloc/malloc.c:3551
#9  0x0000000000000fa9 in ?? ()
#10 0x0000000011000508 in ?? ()
#11 0x00000000100409d8 in ?? ()
#12 0x0000000011000508 in ?? ()
#13 0x0000000000000000 in ?? ()

Fixed with recent PR for break_addr

authors: kapkic rennergade youngfalcon


text:
🛑 MortyProxy is down

In 67b64d3, MortyProxy (https://proxy.linerly.tk) was down:

HTTP code: 0
Response time: 0 ms

Resolved: MortyProxy is back up in b12df7c.

repo: Linerly/status


text:
🛑 LinerlyBot (Discord) is down

In a4f79d2, LinerlyBot (Discord) (https://linerlybot-discord.linerly.tk) was down:

HTTP code: 0
Response time: 0 ms

Resolved: LinerlyBot (Discord) is back up in c00ac1b.


text:
🛑 KTS is down

In 59633ee, KTS (https://kts.flitswallet.app/api) was down:

HTTP code: 0
Response time: 0 ms

Resolved: KTS is back up in a1f311d.


text:
Area covered not calculated in main project list

The area covered appears XXm

I added the area parameter to be an input for the user as I could not find the correct equation to display an aprox. value of area. Added in commit 2b2c0d8e29cae4dbea4c930cd3c2be70f29e1f5b

authors: KarinePistili dronecoria


text:
Update master version in antora.yml

What should be the new content?
Update the versions antora.yml of the SDK 3.x and 4.x docs. New master version is SDK 4.x docs.

Already merged

repo: LiskHQ/lisk-docs


text:
请问如何才能不以类名为表名

因为项目中要求数据库类名统一加xx后缀便于识别，但是存在数据库中是不需要加后缀的，请问大神这种情况该如何自定义表名字

不好意思，litepal暂时还不支持这个功能，这个目前确实是无法实现的。

不好意思，litepal暂时还不支持这个功能，这个目前确实是无法实现的。

好的收到，谢谢郭神，希望litepal越来越强大

repo: LitePalFramework/LitePal

license_source: bigquery


text:
Send logs to stdout by default

This is what all the other kernels seem to be doing. Could have options that redirect it to a file, but sending it by default also helps in log capture when running on the cloud (see also http://12factor.net/) which is useful for installs like paws.wmflabs.org (where I'm hoping to get this installed!)

Hi @yuvipanda , can you confirm that all logs are sent to stdout? Is it possible that some logs are being sent to stderr?

Good point - I think they're actually going to stderr by default, not
stdout. Not going to a file though!
On Tue, Apr 5, 2016 at 2:36 PM, Andreu Correa Casablanca <
notifications@github.com> wrote:

Hi @yuvipanda https://github.com/yuvipanda , can you confirm that all
logs from other kernels are sent to stdout? Is it possible that some logs
are being sent to stderr?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Litipk/Jupyter-PHP/issues/10#issuecomment-205994260

--
Yuvi Panda T
http://yuvi.in/blog

Done.

Thanks!
On Wed, Apr 6, 2016 at 2:43 PM, Andreu Correa Casablanca <
notifications@github.com> wrote:

Done.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Litipk/Jupyter-PHP/issues/10#issuecomment-206584050

--
Yuvi Panda T
http://yuvi.in/blog

authors: castarco yuvipanda

repo: Litipk/Jupyter-PHP

license: mit


text:
🛑 invest is down

In 8e77255, invest (https://blog-pos-17/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: invest is back up in 8025437.

repo: LloydGain/UppTIME


text:
🛑 invest is down

In 869eaca, invest (https://blog-pos-17/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: invest is back up in abb3f78.


text:
游戏内汉化切换

在提问之前...

[X] 我填写了简短且清晰明确的标题，以便开发者在翻阅 issue 列表时能快速确定大致问题。而不是“一个建议”、“卡住了”等
[X] 我基本确定这是一个新功能/建议，而不是遇到了 bug（不确定的话请附上日志）

说说你遇到的问题？
No response
有什么好的想法？
RT
建议增加在游戏内切换英文，韩文，日文，中文功能。
其他内容
No response

如何简单的卸载模组:
重命名version.dll，例如_version.dll
如何简单的恢复:
重命名回来

下个版本将会增加


text:
Ultrawide monitor menu anchoring is messed up (makes menus unusable)

Hi!
I'm on an Ultrawide monitor (32:9 aspect ratio) and after I clicked the top right button on the menus, it seems to have sent the menus to the shadow realm. I am now unable to use the menus (I'll try restarting in a moment though).
If I had to guess, it seems the mod is assuming the aspect ratio is 16:9 and so puts the menus somewhere out of my reach above the screen.
Here's a screenshot. Note how the menus are supposed to be "activated" but only the interactable object menu is visible. I can't seem to be able to move that menu at all to make better use of it, so it's stuck in this position.

Edit: Restarting the game made the menus spawn in their original positions, so that's good I guess.

Same issue here

authors: CookieSalad Puchor

repo: Lodington/Aerolt


text:
🛑 SentinelTrails is down

In 3f67c91, SentinelTrails (https://app.logsentinel.com) was down:

HTTP code: 502
Response time: 332 ms

Resolved: SentinelTrails is back up in ace4e1d.

repo: LogSentinel/status


text:
🛑 LogSentinel SIEM API is down

In 9c3d792, LogSentinel SIEM API (https://api.logsentinel.com) was down:

HTTP code: 502
Response time: 637 ms

Resolved: LogSentinel SIEM API is back up in 6df0e77.


text:
🛑 LogSentinel SIEM dashboard is down

In 47434b1, LogSentinel SIEM dashboard (https://siem.logsentinel.com) was down:

HTTP code: 502
Response time: 332 ms

Resolved: LogSentinel SIEM dashboard is back up in 080b640.


text:
Rain refill loading chunks

Rain refill appears to be loading chunks, making it cause more lags than necessary
https://gist.github.com/someaddons/f5329ff9c87fa8071c8df31d84bdaa9d

Very interesting! Thank you for reporting!

ye happened to see it while debugging some other chunkloading :D
I guess its loading some chunks because the chunkmanager list has like these unfinished-half loaded chunks aswell

Hey this is a nice feature tbh, maybe there is some way to keep it but optimise it? Some kind of Server-side cache that, I stead of loading the chunks to fill it, just keeps a record/count of unloaded chunks that "have had rain" - then when the chunk is loaded by a player, only then block updates occur?
Or some kind of background worker thread that just makes the rain filling a non-blocking operation? Use the spare ticks left over in a tick and spread it out. It's not super important than rain refilling bodies is 100% realtime is it :)
Still, if water body updates to into a non-blocking task queue as described, you'll need to handle overflows. In that case, pause the whole server and do a full flush of the queue IMO.
...too complicated? I will try and learn your code base and implement it :D

forgot to merge the fix in? :D

I wanted to put this into a new release together with 1.17 support and loom/ForgeGradle updates but sadly ForgeGradle 5 is not working properly currently. I will probably do a release only containing this fix in a few days.

thanks^^

authors: LolHens cosmicdan someaddons

repo: LolHens/mc-fluid-physics


text:
windows 64位版本的几个疑似bug（稳定性问题）

请问能否尽快帮我解决，已经焦头烂额，谢谢。项目用到一半，又无法改变平台。我的qq:8221776
1)Post json时json大小超过一定尺寸后ngx.req.read_body()和local data = ngx.req.get_body_data()就返回nil，超过100k就一定会有这个问题。已经在配置文件设置client_max_body_size 10m;。日志中出现一行warning
2016/03/24 00:14:04 [warn] 5032#5716: *6 a client request body is buffered to a temporary file c:\App\GisService/temp/client_body_temp/0000000007, client: 127.0.0.1, server: , request: "POST /se/v1/test/12345678 HTTP/1.1", host: "127.0.0.1:8080"
2)lua在分析较大json时崩溃（大概100k左右，2000行），报错
nginx: lua atpanic: Lua VM crashed, reason: unknown reason
2016/03/24 08:58:12 [notice] 4756#8144: exiting
3)前期还发现日志输出太多时，开启代码缓存会直接不稳定，也报
nginx: lua atpanic: Lua VM crashed, reason: unknown reason
4)如果http配置段没有默认的defaulttype,在代码中设置content-type也会崩溃

如果需要示例的json，我可以提供

Win32下的openresty没有这些问题，但我们的C++模块只能用64位的。

Win32下的openresty没有这些问题，但我们的C++模块只能用64位的。

Win32下的openresty没有这些问题，但我们的C++模块只能用64位的。

Win32下的openresty没有这些问题，但我们的C++模块只能用64位的。

Win32下的openresty没有这些问题，但我们的C++模块只能用64位的。

麻烦提供一下你们的 请求发送格式。。用 curl 的命令行表示。

更新了新的 cjson.dll 已经解决。

repo: LomoX-Offical/nginx-openresty-windows


text:
🛑 Ronz AMOGUS Bot is down

In 5f9f407, Ronz AMOGUS Bot ($RONZAMOGUS_BOT_URL) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Ronz AMOGUS Bot is back up in 18d01ef.

repo: LordRonz/status


text:
🛑 Link Shortener is down

In fb6b268, Link Shortener (https://lr-link.vercel.app) was down:

HTTP code: 504
Response time: 10828 ms

Resolved: Link Shortener is back up in 8291be4.


text:
Ptt system

Hello, could you change the capacity system a potential?
It's not nice when you try to clear ftr 11 and score 9'700, even though it was 9'650, and you get ptt taken away because - yes
Please make it so that if the record was at least 00'005'000+ better (in the global top), the ptt is not taken away

You want to make PTT not decrease when breaking historic records, right?
At least currently, I don't have any ideas about adding this feature, because it is not in the offical server. Surely we haven't known all about the PTT system. If you think there are the new mechanisms, show me your evidence.
However, this feature is not complex. You can change the code by yourself. In line 481 & 485 of core/score.py, you can see a flag first_protect_flag and a compare between new and old scores, so just making flag true in that case can protect your PTT.

I can't provide evidence that such a system exists in Arcaea but I can't recall a single time when I had a score of 9'100 and improved it to 9'440~ on a song (ftr 9) without my ptt dropping (I was playing on a friend's account at the time with ~12.20 ptt

It is unpleasant...

Changing score.py seemed to help
Thanks

repo: Lost-MSth/Arcaea-server


text:
Cummulative update

Further localization

Also resolves #22

repo: LostPetInitiative/WebApp


text:
Running code ERROR

I think line 875 at function DataLoader, the collate_fn=pad should be collate_fn=NerDataset.pad ...

I have the same issue

me too

replace by "collate_fn=NerDataset.pad,"
i know ,should be changed to  collate_fn=NerDataset.pad,because the author use " collate_fn=NerDataset.pad" when load other data,i guess " collate_fn=pad" may be a mistake.

repo: Louis-udm/NER-BERT-CRF


text:
Some Latex Forms Have Broken / Misaligned Arms While Using Carry On.

not to mention the armor Layer is not aligned with the arms either,
I wonder if it is possible to instead create a animator for carry on to apply to the arms, as well as a held block offset for forms like the centaur?

You're right about the animator idea, I designed to be as close to the original mod as possible (as badly implemented as the original is). I'll start work on that.

authors: LtxProgrammer bijimio

repo: LtxProgrammer/Changed-Minecraft-Mod


text:
Implement Table Presets

Hi. Really nice piece of code!
I want to suggest the adiction of one "Preset" for Tables-like loading, with custom options to choose the line and columns number.
Thanks.

I'm also looking for a skeleton for tables. Did you end up creating one yourself @brunonetadmin ?

authors: SiamKreative brunonetadmin


text:
Basic Auth

I don't know if there is any support for Basic Auth, couldn't find it in the docs. If not, it would be nice to have. Currently I just set the header Authorization: Basic ... , but I think it would be nice to be able to set the username and password instead.

I like this idea. I think the best design would be adding a new field to the Recipe type called authorization, which has multiple variants. This would override the Authorization header. Some examples:
get_user:
  name: Get User
  method: GET
  url: "{{host}}/anything/{{user_guid}}"
  authorization: !basic
    username: "{{username}}"
    password: "{{password}}"

get_user:
  name: Get User
  method: GET
  url: "{{host}}/anything/{{user_guid}}"
  authorization: !bearer
    token: "{{token}}"

authors: LucasPickering snaggen

repo: LucasPickering/slumber


text:
Cannot import to mysql database

Description
Hi,
I am trying to import my export to a mysql database (with multiple servers) every time I do I get this "error"
[LP] (Import) -> Starting import process.
[LP] (Import) -> Reading data....
[LP] (Import) -> Waiting for initial update task to complete....
It stays like this for days and won't ever complete I have tried putting the export on a local server doing /luckperms import --upload and doing the 10 digit code on my actual server but the same thing happens I don't know what's wrong if someone could help me out and point me in the right direction that would be great, and if you need me to send another let me know, thank you
Reproduction Steps
export your groups + users
switch storage type to mysql
attempt to import and its stays stuck on this message:
[LP] (Import) -> Starting import process.
[LP] (Import) -> Reading data....
[LP] (Import) -> Waiting for initial update task to complete....
Expected Behaviour
It's supposed to import.
Server Details
git-Paper-160 (MC 1.19.2) - 1.19.2-R0.1-SNAPSHOT
LuckPerms Version
v5.4.4.6
Logs and Configs
https://pastes.dev/oDt9XNi7gD
Extra Details
No response

From just looking at it it seems it could be caused by like half of your skript plugins failing to load

From just looking at it it seems it could be caused by like half of your skript plugins failing to load

Why would skript be the issue?

Please generate a stack dump when the process gets stuck.
https://gist.github.com/lucko/81033b72927eee0246dc8e8c284664c4

authors: My1Legocraft NullTxt lucko

repo: LuckPerms/LuckPerms


text:
wont work on neoforge

Description
wont work on neoforge
Reproduction Steps
install on neoforge server
Expected Behaviour
the server launches
Server Details
neoforge 1.20.1-47.1.96
LuckPerms Version
LuckPerms-Forge-5.4.117
Logs and Configs
[18:57:52] [main/ERROR] [minecraft/Main]: Failed to start the minecraft server
net.minecraftforge.fml.LoadingFailedException: Loading errors encountered: [
LuckPerms (luckperms) encountered an error during the common_setup event phase
§7java.lang.NoClassDefFoundError: net/minecraft/server/network/ConfigurationTask
]
at net.minecraftforge.fml.ModLoader.waitForTransition(ModLoader.java:246) ~[core-47.1.47.jar%23140!/:47.1] {}
at net.minecraftforge.fml.ModLoader.lambda$dispatchAndHandleError$20(ModLoader.java:210) ~[core-47.1.47.jar%23140!/:47.1] {}
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?] {re:mixin}
at net.minecraftforge.fml.ModLoader.dispatchAndHandleError(ModLoader.java:210) ~[core-47.1.47.jar%23140!/:47.1] {}
at net.minecraftforge.fml.ModLoader.lambda$loadMods$15(ModLoader.java:190) ~[core-47.1.47.jar%23140!/:47.1] {}
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?] {re:mixin}
at net.minecraftforge.fml.ModLoader.loadMods(ModLoader.java:190) ~[core-47.1.47.jar%23140!/:47.1] {}
at net.minecraftforge.server.loading.ServerModLoader.load(ServerModLoader.java:31) ~[forge-1.20.1-47.1.96-universal.jar%23144!/:?] {re:classloading}
at net.minecraft.server.Main.main(Main.java:125) ~[server-1.20.1-20230612.114412-srg.jar%23139!/:?] {re:classloading}
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] {}
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] {}
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] {}
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] {}
at net.minecraftforge.fml.loading.targets.CommonLaunchHandler.runTarget(CommonLaunchHandler.java:126) ~[loader-47.1.47.jar%2351!/:47.1] {}
at net.minecraftforge.fml.loading.targets.CommonLaunchHandler.serverService(CommonLaunchHandler.java:118) ~[loader-47.1.47.jar%2351!/:47.1] {}
at net.minecraftforge.fml.loading.targets.CommonServerLaunchHandler.runService(CommonServerLaunchHandler.java:26) ~[loader-47.1.47.jar%2351!/:47.1] {}
at net.minecraftforge.fml.loading.targets.CommonLaunchHandler.lambda$launchService$4(CommonLaunchHandler.java:108) ~[loader-47.1.47.jar%2351!/:47.1] {}
at cpw.mods.modlauncher.LaunchServiceHandlerDecorator.launch(LaunchServiceHandlerDecorator.java:30) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.modlauncher.LaunchServiceHandler.launch(LaunchServiceHandler.java:53) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.modlauncher.LaunchServiceHandler.launch(LaunchServiceHandler.java:71) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.modlauncher.Launcher.run(Launcher.java:108) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.modlauncher.Launcher.main(Launcher.java:78) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.modlauncher.BootstrapLaunchConsumer.accept(BootstrapLaunchConsumer.java:26) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.modlauncher.BootstrapLaunchConsumer.accept(BootstrapLaunchConsumer.java:23) ~[modlauncher-10.0.9.jar%2358!/:?] {}
at cpw.mods.bootstraplauncher.BootstrapLauncher.main(BootstrapLauncher.java:141) ~[bootstraplauncher-1.1.2.jar:?] {}
Extra Details
No response

That LuckPerms version is for MC 1.20.3-1.20.4. For 1.20.1 you have to use v5.4.102, you can get it on modrinth or curseforge.

authors: TheKidThatCodes emilyy-dev


text:
[Enhancement 💡] KubeJs support ?

Could you add a kubeJs support please ? Or provide the way to make recipes in the json format ?

Hey 👋 @Elnhomantia
KubeJS compat is planned, I dont know for now how I will implement it. But for now, you can just see how to create custom recipes in KubeJS on their wiki and just change the recipe type or others fields like my JSON recipes ones like Here for casting brass ingot recipe.

Yep, try doing something like this in your script:
    let alloying = (input, time, fluidType, fluidAmount, heat) => {
        event.custom({
            "type": "createmetallurgy:alloying",
            "ingredients": input,
            "processingTime": time,
            "results": [
              {
                "fluid": fluidType,
                "amount": fluidAmount
              }
            ],
            "heatRequirement": heat
          })
    }

and use it later on
       alloying(
      [
        {"fluid": "createmetallurgy:molten_copper", "amount": 10},
        {"fluid": "createmetallurgy:molten_zinc", "amount": 10}
      ],
      80, "createmetallurgy:molten_brass", 20, "heated"
    );

I don't think it will be that easy.  I didn't really think about how to do it afterwards so idk

Actually I forgot I could actually go and see on the repo xD. Thx a lot for the help.
This actually work and let me add anything as input/ouput :
/**
   * Add melting recipe with create metallurgy
   * @param {{item: 'tag'}|{fluid: 'tag', amount: 'number'}[]} input
   * @param {number} time
   * @param {{item: 'tag'}|{fluid: 'tag', amount: 'number'}[]} output
   * @param {'no'|'heated'|'superheated'} heat
   */
  let melting = (input, time, output, heat) => {
    event.custom({
      type: "createmetallurgy:melting",
      ingredients: input,
      processingTime: time,
      results: output,
      heatRequirement: heat,
    });
  };
  /**
   * Add alloying recipe with create metallurgy
   * @param {{item: 'tag'}|{fluid: 'tag', amount: 'number'}[]} input
   * @param {number} time
   * @param {{item: 'tag'}|{fluid: 'tag', amount: 'number'}[]} output
   * @param {'no'|'heated'|'superheated'} heat
   */
  let alloying = (input, time, output, heat) => {
    event.custom({
      type: "createmetallurgy:alloying",
      ingredients: input,
      processingTime: time,
      results: output,
      heatRequirement: heat,
    });
  };

it should be and be used in the ServerEvents.recipes((event) => {} of kubeJs

Could you add a kubeJs support please ? Or provide the way to make recipes in the json format ?

I wrote a schema that can be applied to basin and table.
Although it is not perfect, the two most basic ones are not a big problem.

so good!

authors: ElMurlocElegante Elnhomantia Lucreeper74 Qi-Month


text:
Clarify or fix default IP usage

Almost all of the examples in the readme as well as the API Reference documentation show code examples like
const plexAPI = new PlexAPI({
  accessToken: "<YOUR_API_KEY_HERE>",
  xPlexClientIdentifier: "gcgzw5rz2xovp84b4vha3a40",
});

which implies that it is possible to use this library without specifying a server url. This is somewhat reinforced by the server index being optional.
However, it looks like the actual client library defaults to using a private IP address, 10.10.10.47, when no server url  is provided. This leads to the same error shown in #20 if using the client code as shown in all examples because that IP address obviously isn't used by everyone.
Please let me know if this address is supposed to be special to Plex somehow and I just don't understand.
Otherwise, it seems like the documentation should specify that serverURL always needs to be specified and, ideally, that it is NOT optional when initializing a new client.

It looks like all of the endpoints that require an actual PMS instance displays a prefix protocol://ip:port on the endpoint info in the documentation

and those that are hardcoded for use with plex.tv do not have this

If it's possible to modify the generated SDK code to the right hand side so that serverURL is present when the prefix is also present that would go a long way to clarifying which endpoints require a PMS instance.

It would also be helpful to include this clarification in at or in the SDK Example Usage section in the readme. The example usage code will fail with connection/timeout error since no serverURL is defined. (Also, I think the example is outdated since it does not show xPlexClientIdentifier ?)

repo: LukeHagar/plexjs


text:
Centered floating windows?

Are you planning on including centered floating windows in the future (e.g. with the center patch)?
I have been trying to apply the patch myself but it always fails, even when I manually edit the lines, so I am probably too inexperienced in C to find the reason for it failing...

the centered patch is essential to @LukeSmithxyz scratchpad's recent update, as the current scratchpad automatically centers all scratchpads in the middle of the screen, while there is an additional patch for the current scratchpad that disables that feature, and then with the centered patch you could assign the main scratchpad to be centered, while you could have the calculator or other scratchpads appear as floating windows on different parts of your screen

@electronblue Maybe patch it yourself?
https://dwm.suckless.org/patches/alwayscenter/
I use this and it works perfectly.

@electronblue  if u need a patch on that , mine is identical to luke main repo but i patch center floating windows...
and other patches like system try, pertag patch, and cool start patch..
https://github.com/goldcoders/dwm
if u wanna use it you need to transfer .xprofile startup script to your config.h in dwm
@electronblue  you dont need to be good at C, im not a C dev, all you need is learn to read the code
the lines where it should be added, your having hard time maybe because you are not used to patching
or git... but patching literrally has nothing to do with C.

authors: electronblue goldcoders kronikpillow paniash

repo: LukeSmithxyz/dwm


text:
Add feature to lock tasks

Just an idea I had while using phived:
sometimes you can modify the order of your tasks or even modify the internal text by accident, so, maybe it would be cool to add a feature to lock the task, only allowing the user to click Done or unlock it, like Notion does with pages. But it's just an idea.

gostei dessa ideia. vou implementar.

i no longer like this idea but i learned something! will not implement sorry
https://stackoverflow.com/questions/19293321/opposite-of-object-freeze-or-object-seal-in-javascript

authors: LukeberryPi datsfilipe

repo: LukeberryPi/phived


text:
StocksParsingException throws

I cloned project, started tests, but they all was failed with StocksParsingException exception.

Ok. i get it solved with setting
Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture im my program


text:
External dependencies

Hi, I'm currently working on an ebuild (i.e. packaging) of LuxCore for Gentoo Linux and got some questions on the build.

There are a few dependencies located in the deps dir, which are needed by the app. Yet, some of those deps are well known packages, probably provided by any modern linux distribution, i.e. eigen and openvdb. I suspect, most users who are installing a piece of software like LuxCore will already have those installed as a dependency of other, similar apps, like blender. Can the system packages be used to build LuxCore or is this currently not possible?

Do you have plans to provide an install target for cmake?

About #1, we have static build procedure (described here: https://github.com/LuxCoreRender/LinuxCompile), it has been developed to be able to distribute Linux binaries and avoid all the problems related version incompatibilities/availability across different Linux distributions. It is required because we need specific package versions.
The libraries included in apps are usually both small and (some time) a modified version of the original package. They also simplify the compilation across Linux/Windows/MacOS. So the short answer to #1 is nope.
About #2, the building procedure at https://github.com/LuxCoreRender/LinuxCompile delivers a .tar.bz2 for the default standalone installation. It is just a directory including everything, it doesn't need any particular installation procedure.
The Blender plugin is available as a .zip like other plugins (https://wiki.luxcorerender.org/BlendLuxCore_Installation) , it is packaged using the above binaries  and https://github.com/LuxCoreRender/BlendLuxCore (it includes a script to pack the .zip: https://github.com/LuxCoreRender/BlendLuxCore/blob/master/doc/release.md).

Thanks for your answer. Not unpacking external packages and using the packaged ones will be fine to get started.
I already looked at the LinuxCompile repo, but this is not of much use for Gentoo. Gentoo is a source based distribution, so we don't deliver packaged binaries, but instead we deliver scripts (called ebuild) to our users, which they pass to the package manager to compile a piece of software for their purpose, i.e. their machine configuration, optional packages, compiler used, compile and links flags and so on. So we already provide a complete, sandboxed build environment, which has to be used when building packages. Also, using the LinuxCompile task would mean, to install packages like jpeg, tiff, libpng or Qt inside an /opt environment (i.e. a self contained software) which is only usable by the piece of software that built it. But usually those packages are already installed on the system in question and used by lots of other software, so there's no need to compile them again, only for one software and would waste a lot of compilation and HD resources.
I had no problem compiling LuxCore, once all dependencies were installed, I was only missing an easy way to install it. I will need to setup a manual install then, using the install script from LinuxCompile as a reference, or patch the CMakeLists.txt with install targets.
Thanks again for your answer!

repo: LuxCoreRender/LuxCore


text:
🛑 EnderChest（末影箱-云盘） is down

In a75cc24, EnderChest（末影箱-云盘） (https://enderchest.anavi.cn) was down:

HTTP code: 0
Response time: 0 ms

Resolved: EnderChest（末影箱-云盘） is back up in f621b63 after 23 minutes.

repo: LxnChan/status


text:
🛑 Official（泠泫凝的异次元空间-主页） is down

In ccfc515, Official（泠泫凝的异次元空间-主页） (https://lxnchan.cn) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Official（泠泫凝的异次元空间-主页） is back up in 027eec7 after 13 minutes.


text:
🛑 Official（泠泫凝的异次元空间-主页） is down

In 6b46be2, Official（泠泫凝的异次元空间-主页） (https://lxnchan.cn) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Official（泠泫凝的异次元空间-主页） is back up in 095cea7 after 53 minutes.


text:
🛑 Official（泠泫凝的异次元空间-主页） is down

In 3cd483c, Official（泠泫凝的异次元空间-主页） (https://lxnchan.cn) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Official（泠泫凝的异次元空间-主页） is back up in febbe05 after 12 minutes.


text:
🛑 EnderChest（末影箱-云盘） is down

In 1c5c432, EnderChest（末影箱-云盘） (https://enderchest.anavi.cn) was down:

HTTP code: 0
Response time: 0 ms

Resolved: EnderChest（末影箱-云盘） is back up in 839b961 after 1 hour, 10 minutes.


text:
🛑 Authentication Server is down

In 3155f55, Authentication Server (https://id.lyrid.io/version) was down:

HTTP code: 500
Response time: 119 ms

Resolved: Authentication Server is back up in 64db921.

repo: LyridInc/statuspage


text:
first upload

CNN(really really basic)

Test commit wirtting


text:
Update to newer atoum versions

atoum 2.8 allows us to remove the bootstrap when it's only used to
require the autoloader (see https://github.com/atoum/atoum/pull/605)
atoum 3.0 allows us to run tests on latest PHP versions.

@jubianchi thanks, but the build is failing:

Error: Configuration file '.atoum.travis.php' does not exist

Everything should be fine now.