text:
issue #1998 - raise friendly error when checkpoint too close from tip

What was wrong?
Issue #1998
Trinity crashes with a HeaderNotFound exception when launched with --sync-from-checkpoint eth://block/byhash/ with the checkpoint too close from the chain's tip.
The issue concerns the beam sync when the checkpoint is not strictly more than MAX_BACKFILL_BLOCK_BODIES_AT_ONCE (100 000) blocks away from the tip.
The reason is because the beam sync backfills block by chunks of  MAX_BACKFILL_BLOCK_BODIES_AT_ONCE and looks first into the DB for their headers.
So, if the specified checkpoint is not before that, some headers will be missing which will trigger the crash.
How was it fixed?
As soon as we get the block number of the chain's tip, we check the checkpoint is strictly before the tip - MAX_BACKFILL_BLOCK_BODIES_AT_ONCE.
If not, we end the application with a clear error message.
To-Do

[ ] Add a specific unit test.

[ ] Add entry to the release notes

Cute Animal Picture

Hello @cburgdorf
The way I fixed this issue makes test_sync.py to fail.
Before I start digging into this test, could you please share your thoughts on my overall approach, when you have a few minutes ?

Also, I may need to add a check on self._enable_backfill along with:
if self._is_checkpoint_too_close_from_tip(final_headers[0].block_number):

or move the whole block inside:
    if self._enable_backfill:
        # There's no chance to introduce new gaps after this point. Therefore we can run this
        # until it has filled all gaps and let it finish.
        self.manager.run_child_service(self._header_backfill)

        # In contrast, block gap fill needs to run indefinitely because of beam sync pivoting.
        self.manager.run_daemon_child_service(self._block_backfill)

        # Now we can check the lag (presumably ~0) and start backfill
        self.manager.run_daemon_task(self._monitor_historical_backfill)

@cburgdorf : Thank you very much for your feedbacks ! :-) I'll look into them asap.

@cburgdorf :
Here is my understanding of the issue. Please correct me where I'm wrong ! :)

The HeaderNotFound is raised by BodyChainGapSyncer._setup_for_next_gap() when it tries to get the header of the block number #(Tip - MAX_BACKFILL_BLOCK_BODIES_AT_ONCE -1)

FromCheckpointLaunchStrategy.fulfill_prerequisites() does not raise it and does not know yet the tip number, which seems to be saved in DB only after the following lines from BeamSyncer.run():
 async with background_asyncio_service(self._header_persister) as manager:
     await manager.wait_finished()

Still according to my understanding, --sync-mode header would not backfill block bodies nor have a mechanism which looks for a header in DB before the checkpoint. That would explain why I don't reproduce the issue in this mode.

@Elnaril I think there's some misunderstanding about the involvement of block bodies here. Let's see if I can clear it up :)
SPOILER: There's a plot twist at 7. but I didn't want to throw away everything I had written before :joy:

Block bodies aren't a concern here at all. All we need to care about is that the block number (which can be read from the block header, no block body needed) that was chosen isn't too close to the tip of the chain. This check is important for every sync strategy that the --sync-from argument can be used with which includes --sync-mode header.

Because of the former, the constant that we need to care about is MAX_SKELETON_REORG_DEPTH not MAX_BACKFILL_BLOCK_BODIES_AT_ONCE

When syncing kicks off, it launches according to a given SyncLaunchStrategyAPI which has a method fulfill_prerequisites that can perform any kind of work that needs to be done before syncing can start. It's like a pre-hook.

https://github.com/ethereum/trinity/blob/aafc20c63716f018b0a628e3b632471b77a7c1dc/trinity/sync/common/strategies.py#L48-L56

There are cases where the client might fail to fulfill the prerequisites in which case an exception is caught and the client is stopped with an error message.

Header sync:
https://github.com/ethereum/trinity/blob/aafc20c63716f018b0a628e3b632471b77a7c1dc/trinity/sync/header/chain.py#L67-L75
Beam sync:
https://github.com/ethereum/trinity/blob/aafc20c63716f018b0a628e3b632471b77a7c1dc/trinity/sync/beam/chain.py#L210-L218
There's some duplication :point_up: with catching these failures for the different sync modes but I think that shouldn't be a big concern here.
5.There are multiple implementations of SyncLaunchStrategyAPI but the one we care about here is FromCheckpointLaunchStrategy which launches sync from a given checkpoint.

Currently, the only case where FromCheckpointLaunchStrategy would raise an exception is when we fail to get the chosen header from any peer within a finite set of attempts

https://github.com/ethereum/trinity/blob/aafc20c63716f018b0a628e3b632471b77a7c1dc/trinity/sync/common/strategies.py#L242-L244

Now that I wrote all this, I realize that there is already a check that checks if we are too close to the tip but I think that check might be based on a different (It bases of FULL_BLOCKS_NEEDED_TO_START_BEAM)

https://github.com/ethereum/trinity/blob/aafc20c63716f018b0a628e3b632471b77a7c1dc/trinity/sync/common/strategies.py#L196-L208
Looking back at the original ticket #1998 I noticed that the crash originates from a different place than what I had in mind when I wrote all of the above.
File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/sync/beam/chain.py", line 531, in run
    await self._setup_for_next_gap()
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/sync/beam/chain.py", line 452, in _setup_for_next_gap
    _starting_tip = await self._db.coro_get_canonical_block_header_by_number(start_num)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/trinity/_utils/async_dispatch.py", line 22, in wrapper
    return await loop.run_in_executor(
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.8/site-packages/eth/db/header.py", line 121, in get_canonical_block_header_by_number
    return self._get_canonical_block_header_by_number(self.db, block_number)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.8/site-packages/eth/db/header.py", line 129, in _get_canonical_block_header_by_number
    canonical_block_hash = cls._get_canonical_block_hash(db, block_number)
  File "/home/cburgdorf/Documents/hacking/ef/trinity/venv/lib/python3.8/site-packages/eth/db/header.py", line 114, in _get_canonical_block_hash
    raise HeaderNotFound(
eth.exceptions.HeaderNotFound: No canonical header for block number #3196818

It originates from _setup_for_next_gap here
https://github.com/ethereum/trinity/blob/aafc20c63716f018b0a628e3b632471b77a7c1dc/trinity/sync/beam/chain.py#L453
So maybe the real fix isn't demanding a bigger distance to the tip but figuring out why the gap syncer (the background sync that fills the gaps that occur when using checkpointing) gets confused about the gaps and crashes.
If I look at the original issue it suggests that one can reproduce this issue by using a checkpoint to close to the tip and fix the issue by choosing a checkpoint more far away from the tip but having spend the time to write this up I'm now not convinced if that is the actual proper fix or if there's a different problem to be solved. All I can say is that it probably needs more investigation...

@cburgdorf : Thank you very much for your feedback, I really appreciate it ! :-)

Hello @cburgdorf
I believe _get_next_gap() should return actual gaps when there is a checkpoint, meaning gaps which are only after this checkpoint.
In order to accomplish that, maybe we could have a specialized gap syncer:
class FromCheckpointBodyChainGapSyncer(BodyChainGapSyncer):
This gap syncer would know about the checkpoint and so would be able to filter the gaps.
I committed an illustration of this idea, so we can have a starting point of discussion in case this is of any interest.

@Elnaril First of all, thanks for taking the time to investigate that further. My head is currently elsewhere which is making it a bit harder for me to keep up with this. I haven't yet fully understood what the problem is so I have a few questions:

Is this problem only caused by the backfilling of blocks or also by backfilling of headers (which are two distinct operations)?

Can you describe step for step the scenario that leads to the crash?

Even better, can you come up with a test to reproduce the issue? We have some tests that cover the gap filler so if we can come up with a test that clearly demonstrates the problem that would be great!. Here's a good starting point.

https://github.com/ethereum/trinity/blob/07e259b9fcaa89dc008836fbd9bb180b56c715b8/tests/core/p2p-proto/test_sync.py#L832
Sorry, if that isn't much of support right now but the issue isn't at the top of my todo list right now so I'm trying to limit my review time while at the same time trying to provide some pointers to help you move it forward :sweat_smile:

@cburgdorf Thank you very much for your time ! :-) I will definitively try to make good use of your pointers ! :smiley:

Hello @cburgdorf
1/ As we saw previously, the error is raised at the beginning of the blocks backfilling, when the blocks syncer (BodyChainGapSyncer) tries to get the header of the block number #(Tip - MAX_BACKFILL_BLOCK_BODIES_AT_ONCE -1) from the DB (in the method _setup_for_next_gap()). This header is not in DB, hence the HeaderNotFound exception.
There is no error with the  backfilling of headers. The reason is that the headers syncer starts to backfill all headers from block 1 by intervals of 100000):
INFO  2020-11-08 15:44:22,011  HeaderChainGapSyncer  Initializing gap-fill header-sync; filling gap: (1, 100001)
and the headers backfilling process checks (in fulfill_prerequisites()) if the header of block 0 is in DB, which is the case for some reason:
self._get_canonical_block_header_by_number(self.db, 0) finds a header, but self._get_canonical_block_header_by_number(self.db, 1) and self._get_canonical_block_header_by_number(self.db, #Tip -100000 + 1) don't.
So no error for backfilling of headers, but I wonder if backfilling of headers from block 1 is expected for a FromCheckpointLaunchStrategy ?
2/ To reproduce the crash, we need to launch the beam syncer with --sync-from-checkpoint and, as checkpoint, a block whose number is not strictly more than MAX_BACKFILL_BLOCK_BODIES_AT_ONCE (100 000) blocks away from the tip.
So, for Goerli with a tip block number =  3718888

checkpoint 3630000 => crash with HeaderNotFound
checkpoint 3600000 => no crash

3/ I've added 2 tests in /trinity/tests/core/p2p-proto/test_sync.py to demonstrate the issue:

test_block_gapfill_from_checkpoint_syncer_1() which simulates the backfilling of bodies with BodyChainGapSyncer and fails (eth.exceptions.HeaderNotFound: No canonical header for block number #898 in the logs of py37-eth1-core).
test_block_gapfill_from_checkpoint_syncer_2() which simulates the backfilling of bodies with FromCheckpointBodyChainGapSyncer (the class I added previously to illustrate a fix) and succeeds.

Both tests create a new chain with headers 970 to 1000, and force the syncer._max_backfill_block_bodies_at_once to 100 (I added this field (which exists for SequentialHeaderChainGapSyncer(self._max_backfill_header_at_once)) to allow the test), then start the body backfilling. The tests are meant to succeed if the blocks 970 to 1000 are imported (but not block < 970).
I struggled a bit to make a test chain with only the headers and had to add a block otherwise no block gaps were found ...
4/ About the fix: In my understanding, the body syncer should backfill only blocks after the checkpoint, so it should not get gaps before that checkpoint. FromCheckpointBodyChainGapSyncer is made aware of the checkpoint number and therefore can filter those wrong gaps.

Hi @Elnaril thanks for your work and detailed writeup. If I understand the issue correctly, it could be that the bug might have been introduced in https://github.com/ethereum/trinity/commit/ae95f78d8fdfb01158df1711c9e3081e5cd5c559 (cc @carver) which tries to improve beam sync by prioritizing backfilling of more recent blocks over older ones. I'm speculating but it would probably be worth checking a previous commit to see if the bug existed before that change.

In my understanding, the body syncer should backfill only blocks after the checkpoint

I think the general assumption about these backfill tasks is that you should be able to start Trinity with any form of chain fragmentation (e.g. maybe you started it with several different checkpoints in the past and hence has lots of gaps) and if you just keep it running long enough it will eventually fill all gaps so that you arrive at a chain that is whole from genesis all the way to the tip.
Given this assumption, I don't think we can filter out gaps entirely.
To me, it seems that there is something wrong with how we recognize chain gaps.
Let's be clear about the two different forms of gaps that we track. There are gaps in the chain of headers that we can read via the get_header_chain_gaps API.
https://github.com/ethereum/trinity/blob/4bcd5dc3dd148039cf0de5a19b2472240d0e4693/trinity/sync/beam/chain.py#L480
If we don't have the header, we are also missing the block but that is not what we recognize as a gap in the chain of blocks. We do only count gaps in the chain of blocks for blocks where we do have the header. This is what we can read with the get_chain_gaps API.
https://github.com/ethereum/trinity/blob/4bcd5dc3dd148039cf0de5a19b2472240d0e4693/trinity/sync/beam/chain.py#L476
If there are no gaps in the chain of blocks, we check if there is room to catch up from our highest written block to our highest consecutive written header.
https://github.com/ethereum/trinity/blob/4bcd5dc3dd148039cf0de5a19b2472240d0e4693/trinity/sync/beam/chain.py#L477-L493
So, when we hit this code, our assumption is that we can always lookup the immediate header before the gap start.
https://github.com/ethereum/trinity/blob/4bcd5dc3dd148039cf0de5a19b2472240d0e4693/trinity/sync/beam/chain.py#L448-L452
But for some reason, that assumption doesn't hold and I haven't fully understood why. It seems to me as if gap_start might already be a checkpoint header and we are trying to reach one header backwards from here which then fails. But I don't see how we would end up with this scenario.
Anyway, I think that your current solution is trying to work around the issue in a way that will simply leave gaps unfilled which I don't think is acceptable (How are we supposed to fill them eventually?).
I could be wrong (and so I'll summon @carver for another pair of :eyes: because he's so much smarter than I :sweat_smile: ) but I want to believe the solution is hidden in def _get_next_gap and that this function may currently not always return what we think it does.

Hi @cburgdorf and thank you very much for your time and feedbacks ! :-)

I think the general assumption about these backfill tasks is that you should be able to start Trinity with any form of chain fragmentation (e.g. maybe you started it with several different checkpoints in the past and hence has lots of gaps) and if you just keep it running long enough it will eventually fill all gaps so that you arrive at a chain that is whole from genesis all the way to the tip.

Thank you for this clarification, it makes totally sense and indeed my solution was not heading this way at all !!
My understanding of the purposes of launching from a checkpoint was partially wrong. :sweat_smile:
I'll have a look at https://github.com/ethereum/trinity/commit/ae95f78d8fdfb01158df1711c9e3081e5cd5c559 and how it worked before this commit and based on all your very helpful clarifications, I'll try to figure out a better way to solve the issue.

Hello @cburgdorf ,
Here is where my investigation stands at the moment. Please let me know if you disagree with anything.
1/ From what I can see after a few tests, the commit https://github.com/ethereum/trinity/commit/ae95f78d8fdfb01158df1711c9e3081e5cd5c559 , per se, did not cause the issue : before this commit, there would be a crash because of the missing header 1 (header 0 is in DB).
2/ I had a look into test_eth1_chaindb.py/test_block_gap_tracking() and it seems to me there is a unit test corresponding to our situation:
https://github.com/ethereum/py-evm/blob/5b177029d4d08454fa097bcd451c89b1d4a92e89/tests/database/test_eth1_chaindb.py#L173-L184
In this test, header 4 and 5 are missing and as soon as block 7 is added, block 4, 5 and 6 are found missing by get_chain_gaps()
That seems to be very similar to our case where block 0 and some blocks near the tip are present but not all headers, and get_chain_gaps() finds the all blocks in between missing.

So, if my understanding is correct, would that mean _get_next_gap() should be adapted and use also get_header_chain_gaps() to compute the right block gaps ?

So, if my understanding is correct, would that mean _get_next_gap() should be adapted and use also get_header_chain_gaps() to compute the right block gaps ?

It does use get_header_chain_gaps() already.
https://github.com/ethereum/trinity/blob/f33bf1260be68539d5d706c8c0e62b1f79fe6b4a/trinity/sync/beam/chain.py#L488-L490
If there are no actual gaps in the chain of blocks (which I assume to be the case, especially if this error happens with a fresh database), then it calls get_header_chain_gaps() to find out if there is room to catch up to the highest consecutive written header. Where highest consecutive written header means the header with the highest number that we already have which is perfectly connected without any gaps all the way down to genesis.
So, lets assume we start Trinity and the DB currently looks like this:
Headers: 0 - 10000 (no gaps)
Blocks: None
We start Trinity with Checkpoint header 20000.
The check should say:

We don't have any block gaps
Oh but we have a perfect header chain up to header 10000
Let's fetch blocks 0 to 10000 then.

This leads me to the question: Can we confirm that this error does really happen with a fresh database (you can delete the db or use --trinity-tmp-root-dir to start it from a fresh temporary directory)
Because with a fresh database I assume it should play out like this:
Headers: 0 (Genesis Header is hard wired)
Blocks: None
We start Trinity with Checkpoint header 20000.
The check should say:

We don't have any block gaps
Our highest header is the genesis header
We can only fetch the genesis block

If this error happens with a fresh database and tries to fetch blocks far in the future, then I assume there must be some error in _get_next_gap even if it isn't obvious to me from starring at the code.
If this error doesn't happen with a fresh database, then it is likely more subtle.

Hello @cburgdorf and thank you again for your time, I'm sure we're getting close to figure out the root cause of this issue ! :-)
Firstly, I do all my debug sessions on Goerli after deleting the folders I specified in --trinity-root-dir. So, yes the issue does happen with a fresh database.

It does use get_header_chain_gaps() already.
https://github.com/ethereum/trinity/blob/f33bf1260be68539d5d706c8c0e62b1f79fe6b4a/trinity/sync/beam/chain.py#L488-L490
If there are no actual gaps in the chain of blocks (which I assume to be the case, especially if this error happens with a fresh database), then it calls get_header_chain_gaps() to find out if there is room to catch up to the highest consecutive written header. Where highest consecutive written header means the header with the highest number that we already have which is perfectly connected without any gaps all the way down to genesis.

Yes, I totally agree with the above. The thing is that the crash happens before we get a chance to go into that block.
Here is what exactly happens when I launch it on Goerli with a fresh DB and a checkpoint at block number 3690000 (for the record, according to goerli.etherscan.io, the last block was ~3781442 at the moment I launched Trinity and ~3781501 when BodyChainGapSyncer started to run):

I added lines 479 and 480 to help understanding the situation.
So when BodyChainGapSyncer runs, the very first time we enter _get_next_gap(self) we have the following (breakpoint on line 481):

As you can see:

self._db.get_chain_gaps() gets a blocks gap between blocks number 1 and 3781494
self._db.get_header_chain_gaps() gets a headers gap between 1 and 3689999
len(gaps) == 1 so we don't reach line 484. Instead the method returns gaps[-1]
Genesis block does exist (and not only its header)

Then we come back to _setup_for_next_gap(self):

where we look for header number 3681493 in DB which fails since it's in the headers gap (1, 3689999) and Trinity crash with the exception :

So, to summarize the issue:
1/ we start with a fresh DB, no blocks, no headers
2/ we launch Trinity from checkpoint 3690000
3/ at the moment the backfilling of blocks starts, we have in DB:

the headers after (and including) 3690000
the genesis block (so header + body)
the block 3781495 (and after):

4/ then, self._db.get_chain_gaps() report a blocks gap between 1 and 3781494
5/ we start to backfill from 3781494 - 100000 = 3681494
6/ for that we look for the previous header (3681494) in DB which does not exist
7/ crash
To conclude, it seems to me that the following assumption is violated, since the genesis block exists and not only its header:

Because with a fresh database I assume it should play out like this:
Headers: 0 (Genesis Header is hard wired)
Blocks: None

If we had only the genesis header, and not its body, no block gap would be retrieved before the checkpoint.
(sorry, for this long post ...)

Great analysis @Elnaril !,

4/ then, self._db.get_chain_gaps() report a blocks gap between 1 and 3781494

Ok, this is interesting because I would have assumed that for this to be a valid block gap it means we need to have block 3781494. But even if we ignore that, the test that you linked clearly shows that we can have block gaps for ranges that both blocks and headers are missing.
And clearly _get_next_gap does not account for that. It just blindly returns the block gap without verifying that it is actionable (meaning, we have the headers for that gap).
https://github.com/ethereum/trinity/blob/f33bf1260be68539d5d706c8c0e62b1f79fe6b4a/trinity/sync/beam/chain.py#L506-L507
So, I'm ignoring the questions why we see this specific block gap being reported because the tests actually demonstrate that we need to account for cases like that. So, it seems to me that what we need is an extra check to find out if the gap is actionable.
I think what we need goes roughly like this:
    def _get_next_gap(self) -> BlockRange:
        gaps, future_tip_block = self._db.get_chain_gaps()
        try:
            actionable_gap = get_topmost_actionable_gap(gaps)
        except NoActionableGap:
            # We do not have gaps in the chain of blocks but we may still have a gap from the last
            # block up until the highest consecutive written header.
            header_gaps, future_tip_header = self._db.get_header_chain_gaps()
            if len(header_gaps) > 0:
                # The header chain has gaps, find out the lowest missing header
                lowest_missing_header, _ = header_gaps[0]
            else:
                # It doesn't have gaps, so the future_tip_header is the lowest missing header
                lowest_missing_header = future_tip_header

            highest_consecutive_header = lowest_missing_header - 1
            if highest_consecutive_header >= future_tip_block:
                # The header before the lowest missing header is the highest consecutive header
                # that exists in the db and it is higher than the future tip block. That's a gap
                # we can try to close.
                return future_tip_block, BlockNumber(highest_consecutive_header)
            else:
                raise ValidationError("No gaps in the chain of blocks")
        else:
            return actionable_gap

Where get_topmost_actionable_gap would loop through the gaps and finds the most recent gap which is actionable, meaning where we do have the previous header available.

If we had only the genesis header, and not its body, no block gap would be retrieved before the checkpoint.

I'm not really sure if that is the case but it seems to me the tests are prove enough that gaps in the chain of blocks can be reported for ranges where both blocks and headers are missing. Hence my proposal above to address that.

(sorry, for this long post ...)

That was a really helpful analysis. Thanks for investing so much time into getting to the bottom of this case!

Thank you @cburgdorf for your always very helpful guidances ! :-)
I created the exception class NoActionableGap in trinity/sync/beam/chain.py , but maybe it would be better to define it elsewhere ? Maybe in trinity/exceptions.py ? Also, should it inherit from BaseTrinityError ?
Moreover, I created the get_topmost_actionable_gap() method.
Because of line 479 in the following code:
https://github.com/ethereum/trinity/blob/f33bf1260be68539d5d706c8c0e62b1f79fe6b4a/trinity/sync/beam/chain.py#L475-L482
we would need to have the headers for all the blocks in the block gap in order to be actionable, and not just the the header of the block preceding the gap. So this method uses the header and block gaps to return the most recent gap of blocks of max size = MAX_BACKFILL_BLOCK_BODIES_AT_ONCE for which the headers exist in DB, and also the preceding header.
This method leaves room for optimization (we could identify a smaller actionable gap i/o raising immediately a NoActionableGap), but I believe it is a good balance between complexity/maintenance and efficiency for a backfilling process, at least for now and in particular regarding the following.
So, speaking of optimization, I noticed the backfilling of the headers starts from block 1:

In our case, where we start from a checkpoint and especially with a fresh DB, that will make the block backfilling wait for all the headers to be imported up to the checkpoint, which seems to defeat the point of "preferring recency" (cf ae95f78d8fdfb01158df1711c9e3081e5cd5c559).
Back to my commits: I also added a pytest method test_block_gapfill_from_checkpoint_syncer() in tests/core/p2p-proto/test_sync.py to illustrate the new method and test the bug is fixed.
Looking at the CI checks, it looks like I will need to rebase ...

That was a really helpful analysis. Thanks for investing so much time into getting to the bottom of this case!

I am really glad if my PRs are helpful ! :-) I do hope they are worth the time you guys spend to review them !

but maybe it would be better to define it elsewhere ?

I think it's ok to live there locally

Also, should it inherit from BaseTrinityError ?

Yes, it should.

we would need to have the headers for all the blocks in the block gap in order to be actionable, and not just the the header of the block preceding the gap

I'm thinking that the body gap syncer will include the headers which means we only need to check that the previous header exists because everything that follows will be fetched in tandem header/block. I could be wrong but at a first glance this is how I think it works.
I'm not sure if get_topmost_actionable_gap needs to be concerned with MAX_BACKFILL_BLOCK_BODIES_AT_ONCE. I would think that we would just check if it is actionable (for which I continue to think we could just check for the parent header but I might be wrong. Did you try it out?) but don't actually care for the size of the gap because restricting the size of the gap is already done by the calling code in _setup_for_next_gap
https://github.com/ethereum/trinity/blob/f33bf1260be68539d5d706c8c0e62b1f79fe6b4a/trinity/sync/beam/chain.py#L458-L461

In our case, where we start from a checkpoint and especially with a fresh DB, that will make the block backfilling wait for all the headers to be imported up to the checkpoint, which seems to defeat the point of "preferring recency"

I think if you start with a fresh db and a checkpoint then it will play out somehow like this:

Start with Checkpoint 1000
The checkpoint is already a safety distance away from the tip so we first start syncing headers only
At some point we are close enough to the tip (say we are at 3000 now).
We start executing blocks so everything after 3000 we do have headers + blocks (and even executed them, yay beam sync!)
But now we can catch up blocks 1000 to 3000 because we only have the headers for those yet
Last but not least, we start filling the gap from 1 to 1000.

Looking at the CI checks, it looks like I will need to rebase

Yep, seems so.

I am really glad if my PRs are helpful ! :-) I do hope they are worth the time you guys spend to review them !

No, you do provide a useful work and I'm happy you have taken on this challenge which I initially thought to be much simpler but when I filed the issue I didn't have all the context that you provided with your ongoing work.

Hello @cburgdorf , and thanks for your feedback. I may miss something, but I'll try to clarify my points.

for which I continue to think we could just check for the parent header but I might be wrong. Did you try it out?

Actually I faced it during one of my previous investigations. The reason I figured out at this moment is that DatabaseBlockRangeHeaderSyncer(self._db, (fill_start, gap_end,)), is initialized by getting the header number gap_end from the DB (line 564 below) and then will yield all headers in range(fill_start, gap_end +1) from DB as well (lines 575 to 581 below).
=> I made the assumption at this moment that all the headers of the gap had to be in DB before the gap could be filled with blocks, but didn't push the investigation further.
https://github.com/ethereum/trinity/blob/f33bf1260be68539d5d706c8c0e62b1f79fe6b4a/trinity/sync/common/headers.py#L553-L581
So, to demonstrate it, I wrote an ad hoc test:
PART I
I built a chain with:

Header gaps: (1, 897), (899, 969)  => header 898 is in DB
Body gap: (1, 999)
and I force get_topmost_actionable_gap() to return (899, 950) which is a gap without header in DB. Only header 898 is in DB.

/!\ Now that I have done this part, I wonder if that is a realistic production case as we may have always some headers preceding the end of the body gap ? Not sure though, but you may want to skip this part and go to PART II directly ...
Here is the test code:

So, at breakpoint line 783, we have the following:

And then, once BodyChainGapSyncer runs, in _setup_for_next_gap() we have the following at breakpoint line 470:

DatabaseBlockRangeHeaderSyncer is then initialized and crashes at line 564 with a HeaderNotFound exception:

PART II
=> Until now, this test demonstrates we need at least the header of the last block of the actionable gap, in addition of the preceding header, but as I said at the beginning, I'm not sure yet if that is a realistic production case.
So, I changed get_topmost_actionable_gap() to return now (899, 999), for which we don't have the headers 899 to 969 in DB but we still have the precedent header (898).
At breakpoint line 783 we have:

This time, all good at DatabaseBlockRangeHeaderSyncer initialization:

But now we enter immediately new_sync_headers():

And a HeaderNotFound exception is raised:

PART III
I did a last test. I launched trinity in debug mode on Goerli with a checkpoint on a fresh DB, but with the following modifications:

I deactivated the header backfilling in order to avoid any unwanted interferences
I deactivated the shortening of the block gap in _setup_for_next_gap(): the aim is to have a block gap with the previous header (here, the genesis) being in DB.

we initialize DatabaseBlockRangeHeaderSyncer with fill_start = 1 and gap_end = 3816476:

No issue with the initialization and we get immediately into new_sync_headers()

A HeaderNotFound exception is raised at the first loop with block_number == 1 and Trinity crashes:

To conclude, I'm still convinced that either all headers of a block gap have to be in DB in order for the gap to be actionable, or some other substantial changes would be needed.
Regarding the relevance of MAX_BACKFILL_BLOCK_BODIES_AT_ONCE in the computing of actionable gaps, I let that for later when we have clarified the above.

you do provide a useful work and I'm happy you have taken on this challenge

:blush:

Ah yes, you are right, the DatabaseBlockRangeHeaderSyncer reads the headers out of the DB. I should know better since I wrote that code :see_no_evil:
But I guess since your solution checks the existence of the headers based on the header chain gaps information (as opposed to trying to make DB reads) checking each header for existence should be ok :ok_hand:

Regarding the relevance of MAX_BACKFILL_BLOCK_BODIES_AT_ONCE in the computing of actionable gaps, I let that for later when we have clarified the above.

Ok, let's move on to that one. It's actually not a big deal. I guess I don't see why it would be needed inside get_topmost_actionable_gap but maybe it is there to minimize the performance impact of that method (like, why keep checking if the gap is already bigger than what we would end up consuming anyway).
Also, you are much closer to the code right now than I am so I'm :+1: with this solution if it works. So, if you can clean this up and get CI to pass, I'm fine to go with this :)

Cool !! :-)
I actually added the use of MAX_BACKFILL_BLOCK_BODIES_AT_ONCE inside get_topmost_actionable_gap() to avoid discarding actionable gaps unnecessarily.
To fix ideas, let's consider the following example:

header gap: (1, 890)
body gap: (1, 999)
MAX_BACKFILL_BLOCK_BODIES_AT_ONCE = 100

Case with using MAX_BACKFILL_BLOCK_BODIES_AT_ONCE:
1/ get_topmost_actionable_gap() returns the actionable gap (899, 999)
2/ _setup_for_next_gap() keeps (899, 999) and the body backfiller can start immediately in parallel with the header backfiller
Case without using MAX_BACKFILL_BLOCK_BODIES_AT_ONCE:
1/ get_topmost_actionable_gap() returns no actionable gap, because the only body gap (1, 999) misses headers 1 to 890 in DB.
2/ The body backfiller will have to wait for the header backfiller to retrieve the missing headers.
Conclusion: In the case "without", the actionable gap (899, 999) has not been considered even though (1, 999) would have been truncated anyway by _setup_for_next_gap() precisely to (899, 999), resulting in an unnecessary wait for the body backfiller.
Please, Let me know if you think that would be better otherwise and I'll update accordingly.
Regarding the CI, despite having rebased, some py37 tests are still failing for reasons which seems, to me, unrelated to my changes. I'll try to dig further, though.
It seems we see the light at the end of the tunnel !! :smile:
Thanks again for all your help ! :-)

Hello @cburgdorf ,
It seems we have a conflict version between importlib-metadata and argcomplete because of the following:
On one hand:
argcomplete v<1.12.2 accepted only version < 3 for importlib-metada:
see argcomplete change here, line 8 of setup.py
On the other hand:
importlib-metada 3 was released on 22nd Nov
https://pypi.org/project/importlib-metadata/#history
Now all py37 checks are good. I guess this issue impacted only py37 checks because importlib-metada has been backported from Python 3.8.
That should also help the PR https://github.com/ethereum/trinity/pull/2109 though I'm not sure if that is the best way to solve it ?
And I guess this version fix should be in a separate PR ? Let me know if I should rollback setup.py.
I have now an error on py38-wheel-cli which I had not before. And py37-wheel-cli is all good. So I wonder if we should just retrigger the py38-wheel-cli test ? Is it possible ? Otherwise I'll try to dig into that tomorrow.

And I guess this version fix should be in a separate PR

Yes, if you could spin out the setup.py fix into its own PR that would be better.

I have now an error on py38-wheel-cli which I had not before

Yes, I think that error is unrelated. It's still interesting. It looks like a failure in our event bus endpoints cross connection ceremony. I've recently read about that error in the chat but had not seen it before myself.
   DEBUG  11-26 21:59:37  async_process_runner.py  b'\x1b[1m\x1b[33m WARNING  2020-11-26 21:59:37,211       EventBusService  Failed to connect network-db to one of discovery: [Errno 11] Resource temporarily unavailable\x1b[0m\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/cytoolz/compatibility.py:2: DeprecationWarning: The toolz.compatibility module is no longer needed in Python 3 and has been deprecated. Please import these utilities directly from the standard library. This module will be removed in a future release.\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  warnings.warn("The toolz.compatibility module is no longer "\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_lru.py:96: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  def _wait_closed(wrapped, *, return_exceptions, loop):\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_lru.py:191: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  def wrapped(*fn_args, **fn_kwargs):\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/aiohttp/helpers.py:105: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  def noop(*args, **kwargs):  # type: ignore\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'<Manager[AsyncioEventBusService] flags=SRcfe>: task _auto_connect_new_announced_endpoints[daemon=True] exited with error: [Errno 11] Resource temporarily unavailable\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'Traceback (most recent call last):\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_service/base.py", line 324, in _run_and_manage_task\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await task.run()\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_service/asyncio.py", line 35, in run\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await self._async_fn(*self._async_fn_args)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/trinity/extensibility/event_bus.py", line 103, in _auto_connect_new_announced_endpoints\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await endpoint.connect_to_endpoints(*endpoints_to_connect_to)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/lahja/asyncio/endpoint.py", line 529, in connect_to_endpoints\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await asyncio.gather(\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/lahja/asyncio/endpoint.py", line 537, in _await_connect_to_endpoint\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    conn = await AsyncioConnection.connect_to(endpoint.path)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/lahja/asyncio/endpoint.py", line 88, in connect_to\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    reader, writer = await asyncio.open_unix_connection(str(path))\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/usr/local/lib/python3.8/asyncio/streams.py", line 111, in open_unix_connection\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    transport, _ = await loop.create_unix_connection(\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "uvloop/loop.pyx", line 2223, in create_unix_connection\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "uvloop/loop.pyx", line 2218, in uvloop.loop.Loop.create_unix_connection\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'BlockingIOError: [Errno 11] Resource temporarily unavailable\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'<bound method AsyncioIsolatedComponent._do_run of <trinity.components.builtin.network_db.component.NetworkDBComponent object at 0x7fc913ddd040>> raised an unexpected exception\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'Traceback (most recent call last):\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/asyncio_run_in_process/_child.py", line 205, in run_process\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    runner(async_fn, args, to_parent)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/asyncio_run_in_process/_child.py", line 168, in _run_on_asyncio\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    result: Any = loop.run_until_complete(_do_async_fn(async_fn, args, to_parent, loop))\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/asyncio_run_in_process/_child.py", line 160, in _do_async_fn\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    return await async_fn_task\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/asyncio_run_in_process/_child.py", line 85, in _handle_coro\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    return await coro_task\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/trinity/extensibility/asyncio.py", line 97, in _do_run\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    self.logger.debug("%s: do_run() finished", self)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_generator/_util.py", line 42, in __aexit__\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await self._agen.asend(None)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_service/asyncio.py", line 458, in background_asyncio_service\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    raise MultiError(\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_service/base.py", line 324, in _run_and_manage_task\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await task.run()\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/async_service/asyncio.py", line 35, in run\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await self._async_fn(*self._async_fn_args)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/trinity/extensibility/event_bus.py", line 103, in _auto_connect_new_announced_endpoints\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await endpoint.connect_to_endpoints(*endpoints_to_connect_to)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/lahja/asyncio/endpoint.py", line 529, in connect_to_endpoints\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    await asyncio.gather(\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/lahja/asyncio/endpoint.py", line 537, in _await_connect_to_endpoint\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    conn = await AsyncioConnection.connect_to(endpoint.path)\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/home/circleci/repo/.tox/py38-wheel-cli/lib/python3.8/site-packages/lahja/asyncio/endpoint.py", line 88, in connect_to\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    reader, writer = await asyncio.open_unix_connection(str(path))\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "/usr/local/lib/python3.8/asyncio/streams.py", line 111, in open_unix_connection\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'    transport, _ = await loop.create_unix_connection(\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "uvloop/loop.pyx", line 2223, in create_unix_connection\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'  File "uvloop/loop.pyx", line 2218, in uvloop.loop.Loop.create_unix_connection\n'
   DEBUG  11-26 21:59:37  async_process_runner.py  b'BlockingIOError: [Errno 11] Resource temporarily unavailable\n'

I've kicked the build.

Thanks for the relaunch of the test, @cburgdorf , it's all green now !! :-)

Yes, if you could spin out the setup.py fix into its own PR that would be better.

Perfect, I'll do it. Should I also create an issue and reference it in the PR ?

Perfect, I'll do it. Should I also create an issue and reference it in the PR ?

Just the PR should be enough

@cburgdorf , I have rebased following the merge of the argcomplete version fix into master.
Now, it is the turn of py37-rpc-state-istanbul to fail ! :smile:

@cburgdorf , I have rebased following the merge of the argcomplete version fix into master.
Now, it is the turn of py37-rpc-state-istanbul to fail ! smile

That's a know issue: #2102

Ah ok, thank you for the info ! :+1:
So I guess it's worth trying to re-launch this test ? Can someone do it for me, please ? :pray:

I re-ran, but it can fail several times in a row, so better to just ignore it :-(

Indeed, it failed again, so yes it may be better to ignore it then.
Thanks again ! :-)

Merged! Thanks again @Elnaril for your endurance and time that you invested here!

And thank you @cburgdorf for your time and guidances ! :-)

authors: Elnaril cburgdorf gsalgado

repo: ethereum/trinity


text:
ununtu 18.04 intall error

Skipping 'fsevents' build as platform linux is not supported

fsevents@1.2.13 install /home/ipfsmain/yc/project/bee-status/node_modules/watchpack-chokidar2/node_modules/fsevents
node install.js

Skipping 'fsevents' build as platform linux is not supported
added 1929 packages in 11.001s

Thanks @yangchuang123456 !
It seems that this is just a warning and otherwise the install is working fine. Can you please confirm that this is the case?

Closing as there is not activity. If this is still a problem please create new issue.

source: gharchive/issue

authors: agazso vojtechsimetka yangchuang123456

repo: ethersphere/bee-dashboard

license_type: permissive

license_source: github-api


text:
Changes to make submapping possible

Changes are mainly to functions for saving and loading from protofiles. The reason these changes are necessary is that when saving a submap collection we want to be able to write collections of blocks interleaved with submap headers, not just a single collection of blocks.

@alexmillane rm this check https://github.com/ethz-asl/voxblox/blob/ee98e9cd465d26071712095ad3de3b147652332a/voxblox/src/integrator/tsdf_integrator.cc#L54

@mfehr Yeah, saw that... eventually.

source: gharchive/pull-request

authors: alexmillane mfehr

repo: ethz-asl/voxblox


text:
Does it work with Swift 3?

Hi. I've converted CryptoJS.swift to Swift 3 using Xcode 8 converter and the converted file seems to be ok but I can't make it work.
When I try
    // Load the TripleDES module
     let TripleDES = CryptoJS.TripleDES()

    // Basic TripleDES encryption
    let encrypted = TripleDES.encrypt("secretMessage", secretKey: "password123")
    print("SECRETMESSAGE: " + encrypted)

    // Basic TripleDES decryption
    let decrypted = TripleDES.decrypt(encrypted, secretKey: "password123")
    print("RESULT: " + decrypted)

I get:
Loaded tripledes.js
SECRETMESSAGE: Optional(undefined)
RESULT: Optional(undefined)
What am I missing?

Thanks for telling me.
I converted the project to swift 3, the problem is fixed.

license: MIT


text:
Update datasets to work with integer timestamp

🚀 Feature Request
Update datasets to work with integer timestamp
Proposal
TSDataset:

def __init__

Make it possible to work with integer timestamp
Make it possible to set freq=None
Check that index covers all the integer values from min to max

We expect that all the values in range(min, max+1) are present
Missing values are filled with NaNs like we do during asfreq

def make_future

Make it possible to work with integer timestamp
Expand index into the future by taking next integer values

def train_test_split

Parameters train_start, train_end, test_start, test_end should work with integer value

def to_dataset

Make it handle integer timestamp

def tsdataset_idx_slice

Clarify documentation that start_idx and end_idx are working like iloc even if timestamp is integer

def plot

Parameters start, end should work with integer value

def index

Fix the signature

def desribe

Clarify in the docs that timestamp could be integer

def info

Clarify in the docs that timestamp could be integer

utils:

def set_columns_wide

Parameters timestamps_left, timestamps_right should work with integer value

Test cases

Add tests for the fixed methods

Additional context
No response

Closed by #146.

repo: etna-team/etna


text:
Add contract possibility when employee is created

When a new employee is created there should be possible for the user to add contract for him

Not needed


text:
support postrges

Would be nice to support postgres as an alternative to mysql.

This should not be too difficult:

In phplib/Persistence.php set the PDO url to a postgres one
Use pgloader to migrate the schema (or modify schemas/* to be postgres compatible or create a postgres version of these)

authors: blysik dveeden

repo: etsy/morgue

license: mit

license_source: bigquery


text:
Phar packaging and deploying with travis-ci

This assumes #48 is accepted (made ontop of this)

Package all *.php files to PHAR

Add options for travis to automatically assemble PHAR on builds

Add options for travis to put built PHAR to release tags on GitHub

See my test release for phar package auto-deployed

Deployment occurs only with tags https://travis-ci.org/scaytrase/phan/builds/96430094

Phar package allows easy distributing of your project. Phar default stup points to phan.php file (made from you phan shell script) so all functioning is equals to calling ./phan
To make it work API_KEY (github token with public_repo access) env var should be setup for job at travis

Rebased onto accepted #48

repo: etsy/phan


text:
create accurate timer for metric flush

clamp timestamp to a precise interval #459

Thank you for this patch. However there is still no explanation of why this is useful and why it can't be implemented in backends if they need it. And the issue you are referencing is closed. I'm happy to discuss this further on issue #459 if there is a need I'm overlooking here.

You are using setInterval() to flush the metrics periodically. But the problem with JavaScript timers is that they’re not very accurate. Thus you have a minimal delay which is escalating with every flush. When the delay overstep a whole second then no data will be available for the affected second which leads to undesired gaps in the graph.
I observed that behaviour on my application every 10 minutes. I am wondering why nobody else is struggling. Maybe the delay is increasing further by the flushed data size or by a poor server.
This patch fixes the appearance of that undesired gaps by using an accurate JavaScript timer and by clamping timestamp to a precise interval which leads to nice and continuous graphs.
(reference: http://www.sitepoint.com/creating-accurate-timers-in-javascript/)

+1 for @thomasmuehlschlegel reason.  we have the same issue.  #513

Merged via command line

authors: fessyfoo mrtazz pathzzrd thomasmuehlschlegel

repo: etsy/statsd


text:
🛑 sip1.demo.prod (IPv4, HTTPS) is down

In c33baaf, sip1.demo.prod (IPv4, HTTPS) (https://sip1.demo.prod.ettelabs.com/metrics) was down:

HTTP code: 0
Response time: 0 ms

Resolved: sip1.demo.prod (IPv4, HTTPS) is back up in f828256 after 1 hour, 28 minutes.

repo: ettelabsinc/ette-uptimejs


text:
🛑 cr1.staging (IPv4, 5060/TCP) is down

In 361be36, cr1.staging (IPv4, 5060/TCP) (cr1.staging.ettelabs.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: cr1.staging (IPv4, 5060/TCP) is back up in c58854b after 5 minutes.


text:
Implement placeholder testing to be run in CI

Contributes to the the planned issue of #8
Delivers just placeholder because I'm still not familiar with the codebase.
And was the smallest step.

Create placeholder test for:

DgcLibAutoConfigurationTest
SignedCertificateMessageBuilderTest
CertficateUtilsTest

Add test runner on CI for pull request.

@M3kH currently I'm also working on the Unit tests. I would like to merge your PR into my local dev branch. Then your contribution is not lost.
Is it okay for you? Then I would close this PR.

Is it okay for you? Then I would close this PR.

Yes sure go ahead! :-)

repo: eu-digital-green-certificates/dgc-lib


text:
Connect the Batching process in the call back UC

integrate batch process into call back use case

send notification to all registered callback URI with the newly created batch tag
implement retry mechanism (extensive logging)

This is already done in our callback branch and will be merged when callback feature is finished.


text:
Calyptos: Stack trace when setting objectstorage.providerclient to walrus

[10.105.10.51] out: Ran source /root/eucarc && http_proxy="" //usr/sbin/euca-modify-property -p objectstorage.providerclient=walrus returned 1
[10.105.10.51] out: [2015-09-09T02:51:44-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Logs:
https://eucalyptus.atlassian.net/secure/attachment/25708/calyptos-1441792304.tgz

Think this could be related to the walrus server not being registered:
[root@odc-f-09 ~]# empyrean-describe-services --region @localhost | grep 'walrus\|storage'
SERVICE  objectstorage         API_10.105.10.67  API_10.105.10.67.objectstorage   broken
SERVICE  storage               az-01             az-01-sc-1                       broken
SERVICE  walrusbackend         walrus            walrus-1                         broken

[root@odc-f-09 ~]# nc -z 10.105.10.67 8779
Connection to 10.105.10.67 8779 port [tcp/*] succeeded!

Retried with identical symptoms:
https://eucalyptus.atlassian.net/secure/attachment/25709/calyptos-1441800742.tgz

This isnt related to the Walrus specifically. All java components are down right now.
Is anything different about the networking in this setup? It looks like the Java components arent able to talk to the CLC because the bind addr is set on the 10.104.x.y network rather than 10.105.x.y:
[root@odc-f-09 ~]# cat /etc/eucalyptus/eucalyptus.conf | grep bind
CLOUD_OPTS=" --bind-addr=10.104.10.51"
[root@odc-f-09 ~]#

I see the issue now. I had thought this PR was already merged:
https://github.com/eucalyptus/eucalyptus-cookbook/pull/201
This does the auto detection for the bind-interface. Ill get that merged ASAP.

@dmccue Its merged now. Sorry for the oversight.

Fixed, Thanks! - https://eucalyptus.atlassian.net/secure/attachment/25611/calyptos-1441138161.tgz
[root@odc-f-09 ~]# grep bind /etc/eucalyptus/eucalyptus.conf
CLOUD_OPTS=" --bind-addr=10.105.10.51"

authors: dmccue viglesiasce

repo: eucalyptus/calyptos


text:
Remove compile flags for better compatibility.

The flags are not necessary, as the Python CFFI build includes optimisation flags automatically (although x86-64, not native), and GCC uses a more recent C11 standard by default. Not specifying flags avoids compiler warnings on Windows where the Visual Studio compiler doesn't support GCC flags.
If there are any performance drawbacks (e.g. not as good vectorization of ChaCha20 by the compiler), we can add platform detection and different flags for Windows.

Codecov Report

Merging #3 (ea4dffa) into master (b1ddc33) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master        #3   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           13        13
  Lines          403       403
=========================================
  Hits           403       403

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b1ddc33...ea4dffa. Read the comment docs.

lgtm, cheers.

repo: eugene-eeo/monocypher-py


text:
Groovy module improvements

Upgraded Groovy to version 3.0.8
Align tests with the Spock testing framework
Code reformating
Minor fixes to tests

Thanks @Alex-Golub
We'll update the articles to match your changes as well.

repo: eugenp/tutorials


text:
Not Impled: 로그인하면 초기에 유저 정보를 못 들고옴

[ ] HomeView에서 onAppear할 때 비동기로 getUser정보를 FireAuth에서 받아온 id로 받아온다 -> API구현 해야함

로그인할때마다 국가 적어줘야하는 것도 이문제 때문인지 모르겠으나, 해당 이슈 있음.

repo: eunoiahyunseo/foreign-student-support


text:
anzahlDerGewerbeeinheiten bzw. anzahlDerGewerbeeinheitenImGebaeude wird bei Eigentumswohnungen nicht übertragen

siehe Titel

In der Web-Ansicht kann die Anzahl der Gewerbeeinheiten nur bei der Immobilien-Art Eigentumswohnung spezifiziert werden.
Allerdings behauptet die API-Doku für den Download, das eine solche Spezifikation nur für die Immobilien-Art vom Typ Haus ermittelt werden kann. So what?

repo: europace/baufismart-vorgaenge-api


text:
OpenStreetMap attributions doesn't respect rules

https://www.openstreetmap.org/copyright
OSM foundation asked OSM attributions doesn't need an interaction on map to be visible.
https://github.com/osm-fr/attributions/issues/97

This mechanism with an info button that shows the attribution dialog has been in use for a long time in the Mapbox SDK for Android (see also their docs), and is still the same in MapLibre, the fork that we are now using. I am quite surprised that this would not be in agreement with OSM's rules, as these SDKs are in use by thousands of apps, both FOSS and commercial. Has this policy changed recently?
If so, I think the more efficient solution would be to report this to MapLibre and Mapbox instead of making every app developer figure out their own solution to fix the attribution...

I'm not inside the OSM Foundation, but from my knowledge, it's not easy to respect OSM attributions by the enterprises.
It seems i button is allowed but I think not all contributors agree with this.

repo: ev-map/EVMap


text:
Not compatible with android gradle plugin v1.1.0-rc1

No such property: defaultConfigData for class: com.android.build.gradle.LibraryPlugin
	at me.tatarka.holdr.gradle.HoldrPlugin.getManifestPackage(HoldrPlugin.groovy:74)
	at me.tatarka.holdr.gradle.HoldrExtension.getHoldrPackage(HoldrExtension.groovy:25)
	at me.tatarka.holdr.gradle.HoldrExtension_Decorated.getHoldrPackage(Unknown Source)
	at org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.getProperty(BeanDynamicObject.java:153)
	at org.gradle.api.internal.BeanDynamicObject.getProperty(BeanDynamicObject.java:107)
	at org.gradle.api.internal.CompositeDynamicObject.getProperty(CompositeDynamicObject.java:78)
	at me.tatarka.holdr.gradle.HoldrExtension_Decorated.getProperty(Unknown Source)
	at me.tatarka.holdr.gradle.HoldrPlugin$_applyHoldrPlugin_closure4_closure5.doCall(HoldrPlugin.groovy:55)
	at org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:63)
	at org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:130)
	at org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:110)
	at org.gradle.api.internal.AbstractTask.configure(AbstractTask.java:482)
	at org.gradle.api.internal.project.AbstractProject.task(AbstractProject.java:879)
	at me.tatarka.holdr.gradle.HoldrPlugin$_applyHoldrPlugin_closure4.doCall(HoldrPlugin.groovy:54)
	at org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:63)
	at org.gradle.listener.ActionBroadcast.execute(ActionBroadcast.java:39)
	at org.gradle.api.internal.DefaultDomainObjectCollection.doAdd(DefaultDomainObjectCollection.java:165)
	at org.gradle.api.internal.DefaultDomainObjectCollection.add(DefaultDomainObjectCollection.java:160)
	at com.android.build.gradle.LibraryExtension.addVariant(LibraryExtension.groovy:68)
	at com.android.build.gradle.internal.VariantManager.createApiObjects(VariantManager.java:625)
	at com.android.build.gradle.internal.VariantManager.createAndroidTasks(VariantManager.java:246)
	at com.android.build.gradle.BasePlugin.createAndroidTasks(BasePlugin.groovy:431)
	at com.android.build.gradle.BasePlugin$_createTasks_closure9.doCall(BasePlugin.groovy:373)
	at org.gradle.listener.ClosureBackedMethodInvocationDispatch.dispatch(ClosureBackedMethodInvocationDispatch.java:40)
	at org.gradle.listener.ClosureBackedMethodInvocationDispatch.dispatch(ClosureBackedMethodInvocationDispatch.java:25)
	at org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java:83)
	at org.gradle.listener.BroadcastDispatch.dispatch(BroadcastDispatch.java:31)
	at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy13.afterEvaluate(Unknown Source)
	at org.gradle.configuration.project.LifecycleProjectEvaluator.notifyAfterEvaluate(LifecycleProjectEvaluator.java:79)

Fixed in 1.4.1

repo: evant/holdr


text:
Victron Batterylife SoC Limit wird beim Schnellladen verändert und nicht wieder zurückgestellt

Describe the bug
Hallo,
ich habe das evcc seit 2 Tagen installiert und teste die Ladung. Leider habe ich nur eine sehr kleine PV Einspeisung und das Wetter ist schlecht, daher habe ich mich auf das Schnell laden konzentriert.
Als Batterie Speicher Speicher für das Haus habe ich einen Victron MultiPlus-II GX mit einer 4,8 KwH Batterie. Die soll auch das Haus versorgen und ist viel zu klein um ein Auto zu laden. Da ist evcc genau richtig, es kann beim Laden des Autos das Entladen der Batterie unterbinden. Das funktioniert soweit auch.
Doch leider wird beim Starten des Schnell Ladens der Batterielife SoC Wert verändert. Es war auf 50% und evcc hat diesen dann auf 76% eingestellt. Beim Beenden des Ladevorgangs ist auch der verstellte Wert nicht wieder auf den ursprünglichen Wert zurück geändert worden.
Ich nehme an, dass das Limit beim Ladestart auf den aktuellen Doc Wert der Batterie gestellt wird (dieser war etwas über 70%).
Zudem wird beim Laden auch der der "ESS Batterie Life State" auch auf "Discharge disabled" gestellt und nicht wieder freigegeben.

Die Batterie Einstellungen:

Was kann ich einstellen, damit der Wert wieder auf den ursprünglichen Wert (vor dem Ladevorgang) gestellt und das Entladen der Batterie wieder freigegeben wird?
Über eine Hilfe würde ich mich sehr freuen!
Gruß
detlefh68
Steps to reproduce

...
Configuration details
network:
  schema: http
  host: evcc.local # .local suffix announces the hostname on MDNS
  port: 7070

interval: 30s # control cycle interval

# trial token, valid until 2024-03-23
sponsortoken:

# unique installation id
plant:

# database configuration for persisting charge sessions and settings
# database:
#   type: sqlite
#   dsn: <path-to-db-file>

# sponsor token enables optional features (request at https://sponsor.evcc.io)
# sponsortoken:

# telemetry enables aggregated statistics
#
# Telemetry allows collecting usage data (grid and green energy, charge power).
# Data is aggregated, no individual charging sessions are tracked. The collected,
# anonymous data can be retrieved using https://api.evcc.io.
#
# See https://github.com/evcc-io/evcc/pull/4343 or details.
#
# For time being, this is only available to sponsors, hence data is associated with
# the sponsor token's identity.
#
# telemetry: true

log: info
levels:
  site: debug
  lp-1: debug
  lp-2: debug
  cache: error
  db: error

# modbus proxy for allowing external programs to reuse the evcc modbus connection
# each entry will start a proxy instance at the given port speaking Modbus TCP and
# relaying to the given modbus downstream device (either TCP or RTU, RS485 or TCP)
modbusproxy:
  #  - port: 5200
  #    uri: solar-edge:502
  #    # rtu: true
  #    # readonly: true # use `deny` to raise modbus errors

# meter definitions
# name can be freely chosen and is used as reference when assigning meters to site and loadpoints
# for documentation see https://docs.evcc.io/docs/devices/meters
meters:
  - name: grid1
    type: template
    template: shelly-3em
    usage: grid
    host: 192.x.x.x
  - name: pv1
    type: template
    template: hoymiles-opendtu
    usage: pv
    host: 192.x.x.x
  - name: battery1
    type: template
    template: victron-energy
    usage: battery
    host: 192.x.x.x
    port: 502
    capacity: 4.8
    minsoc: -1
#    minsoc: 20
#    maxsoc: 95

# charger definitions
# name can be freely chosen and is used as reference when assigning charger to vehicle
# for documentation see https://docs.evcc.io/docs/devices/chargers
chargers:
  - name: wallbox
    type: template
    template: smaevcharger
    host: 192.x.x.x
    user:
    password:
  - name: ebike
    type: template
    template: tasmota
    host: 192.x.x.x # IP-Adresse oder Hostname
    user: # Standard-User ist admin (optional)
    password: # Passwort des Benutzerkontos (bei führenden Nullen bitte in einfache Hochkommata setzen) (optional)
    channel: 1 # Schaltkanal (1-8)
    standbypower: 15 # Leistung oberhalb des angegebenen Wertes wird als Ladeleistung gewertet (optional)
    integrateddevice: true
    icon: bike

# vehicle definitions
# name can be freely chosen and is used as reference when assigning vehicle to loadpoint
# for documentation see https://docs.evcc.io/docs/devices/vehicles
vehicles:
  - type: template
    template: vw
    title: ID.3
    icon: car
    user:
    password:
    vin:
    capacity: 58
    phases: 3
    cache: 15m
    mode: off
    minCurrent: 6
    maxCurrent: 16
    timeout: 10s
    name: id3

# site describes the EVU connection, PV and home battery
site:
  title: Mein Zuhause
  meters:
    grid: grid1
    pv:
      - pv1
    battery:
      - battery1
  residualPower: 100 # 100W Netzbezug im PV Modus erlaubt

# loadpoint describes the charger, charge meter and connected vehicle
loadpoints:
  - title: Garage
    charger: wallbox
    vehicle: id3
    mode: off
    enable:
      # einschalten, wenn 1 Minute lang mindestens 2000 W Überschuss vorhanden ist
      delay: 1m
      threshold: -800
    disable:
      # ausschalten, wenn 30 Minuten lang mehr als 2000 W aus dem Netz bezogen werden
      delay: 30m
      threshold: 2000
  - title: E-Bikes
    charger: ebike
    mode: off
#    mincurrent: 0.5 # 9A*220V=1980W minimum charge current (default 6A)
#    maxcurrent: 4 # 10A*220V=2200W maximum charge current (default 16A)
#    phases: 1

# tariffs are the fixed or variable tariffs
tariffs:
  currency: EUR
  grid:
    type: fixed
    price: 0.3784
  feedin:
    type: fixed
    price: 0.0
  co2:
    type: grünstromindex
    zip: xxxxx

# mqtt message broker
mqtt:
  broker: mqtt.home.lan:1883
  topic: evcc # root topic for publishing, set empty to disable
  clientid: evcc
  user:
  password:

# influx database
influx:
  # url: http://localhost:8086
  # database: evcc
  # user:
  # password:

# eebus credentials
eebus:
  # uri: # :4712
  # interfaces: # limit eebus to specific network interfaces
  # - en0
  # certificate: # local signed certificate, required, can be generated via `evcc eebus-cert`
  #   public: # public key
  #   private: # private key

# push messages
messaging:
  events:
    start:
      title: Ladevorgang gestartet
      msg: Ladevorgang wurde mit Modus "${mode}" gestartet
    stop:
      title: Ladevorgang beendet
      msg: ${chargedEnergy:%.1fk}kWh in ${chargeDuration} geladen
    connect:
      title: Fahrzeug verbunden
      msg: Ladepunkt ${loadpoint}
    disconnect:
      title: Fahrzeug getrennt
      msg: Ladepunkt ${loadpoint} nach ${connectedDuration} getrennt
  services:

Log details
...ist beim HA Addon nur eingeschränkt möglich

What type of operating system are you running?
HomeAssistant Add-on
Version
v0.124.10

Ein zurücksetzen geht nicht, aber du kannst den Standardwert konfigurieren. Siehe
evcc configure

/cc @VolkerK62 sollen wir die Batteriesteuerung per minsoc nochmal dokumentieren?

Bei meiner ersten Testinstalltion unter Proxmox habe ich folgendes erhalten....

name: battery1
type: template
template: victron-energy
usage: battery
host: 192.x.x.x
port: 502
capacity: 4.8
minsoc: -1

minsoc: 20
maxsoc: 95
minsoc und maxsoc waren nach "configure" eingetragen => damit ist das gleiche passiert. Bat Life SoC Limit verstellt und bat Life State auf disabled gestellt. Daher habe ich für den 2.Test minsoc auf -1 gestellt => ändert nichts :-(
Wenn diasabled wird, kann doch auch wieder enabled werden, oder muss ich das nach jedem Schnell Laden wieder manuell  enablen?

Wenn diasabled wird, kann doch auch wieder enabled werden, oder muss ich das nach jedem Schnell Laden wieder manuell enablen?

Kann er aktuell nicht. Du kannst entweder  die aktive Batteriesteuerung deaktivieren oder den minsoc auf die gewünschten 50% stellen.

ok ich stellen für den nächsten Test minsoc im evcc.yaml bei der Batterie ein.
Dann wird nach dem Laden das SoC Limit auf diesen Wert eingestellt?
Wie ich die aktive Batterie Steuerung dektiviere ist mir im Moment nicht ganz klar. Ich weis nicht wo und wie ich das aktiviere.
Ist halt blöd, dass es von Hand verstellt werden muss.

Muss es nicht. Du darfst sie nur nicht einschalten.

@andig

/cc @VolkerK62 sollen wir die Batteriesteuerung per minsoc nochmal dokumentieren?

Wegen Nichtbetroffenheit verfolge ich das Thema nur am Rande.
Ist minsoc ein Victron spezifisches Thema? Dann könnte man das im Template für die Doku "sichtbar" machen?

Das betrifft tatsächlich alle Batterien, bei denen hold/charge über den limitsoc anstatt über einen Mode gesteuert wird:

@andig
du meinst hier deaktivieren?

Genau. Der Haken kommt nicht von alleine ;)

Danke 🙂
Wenn ich es richtig verstanden habe...Die Sperre zum Entladen wird über (beim Victron) Batterylife SoCLimit realisiert. D.h. es wird ein Wert größer aktueller SoC gesetzt und der von mir eingestellte minsoc (z.B. 20%) in der evcc.yaml beim Beenden gesetzt.
Da der Victron ja immer nach Ladezyklus diesen Wert täglich verändert, wir durch das Auto Laden immer ein fix Wert gesetzt. Dann müsste ich ja abhängig von der Wetterlage und Jahreszeit diesen Wert in der evcc.yaml anpassen?
Wäre es super, wenn evcc den aktuelle limit Wert beim Beginn des Ladens liest und anschließend wieder setzt....
Habe ich den Mechanismus der Ladesperre in etwa verstanden?😬

Die bessere Variante wäre auch hier ein Weg ohne die Manipulation von SoC-Schwellen.

Ich habe jetzt minsoc auf 20 gestellt. Beim Starten des Ladenvorgangs wir minsoc auf den aktuellen soc Wert der Batterie geändert (heute auf 62%) und das soc limit auf Aktuelle soc+5%, also 67%.
Nach dem Laden wird auf den von mir auf minsoc gestellten Wert aus der evcc.yaml gestellt. doch leider bleibt des aktive soc limit auf 67%.
Das hat zur Folge, das die Hausbatterie immer nur auf den Wert entladen wird (soc+5%), der vor dem Autoladen war.
=> minsoc funktioniert, aber das Änerden des aktiven SoC Limit im Victron ist schlecht.
ist es nicht möglich das SoC Limit der Batterie von evcc nicht anzutasten?

Nach dem Laden wird auf den von mir auf minsoc gestellten Wert aus der evcc.yaml gestellt. doch leider bleibt des aktive soc limit auf 67%.

@philipptrenz hast Du eine Ahnung was hier los ist? minsoc vs. "aktives soc Limit"?

Der Victron hat zwei SoC Grenzen,  minsoc und Aktives SoC Limit....

...minsoc ist der Wert, der beim Laden auf den den aktuellen SoC Wert gestellt wird, damit das Entladen gesperrt ist.
Dieser wird nach dem Laden (Auto) wieder auf den im der evcc.yaml definierten Wert gestellt. Das funktioniert auch gut.
Der zweite Wert "Aktives SoC Limit" ist der Wert der beim Batterielife Betrieb vom Victron verstellt wird. Wird die Batterie am Tag auf annähernd 100% geladen, singt der Wert um 5%. Wird die Batterie bei schlechtem Wetter nur wenig geladen, also unter 100%, steigt der Wert täglich wieder um 5%. Somit soll vermieden werden, dass sich die Batterie über eine längere Zeit im sehr hohen oder tiefen Ladezustand befindet.
Durch das Anpassen von minsoc beim Start des Ladevorgangs (Auto) wird das "aktive SoC Limit" mit verstellt (vermutlich darf im Victron minsoc nicht größer als das SoC Limit sein).
Das hast in einer Zeit (Sommer), in der das Wetter gut ist und die Batterie ordentlich geladen wird, ist diese Limit sehr niedrig. Letzen Sommer war em immer fast auf 5%, Da heist die Batterie wurde nachts Immer entladen und den Grundbadarf des Hauses bis zum Morgen gedeckt.
Wenn ich dann am Nachmittag das Auto Lade und der die Batterie einen hohen Ladezustand hat, wird mein SoC Limit auch mit angehoben (macht öffentlich der Virctron selbstständig) und die Batterie nur bis zu dem Limit entladen.
Man müsste dann immer nach einem Ladevorgang des Autos überprüfen wie hoch das Limit war und manuell wieder anpassen.
Daher kam meine Frage, ob evcc nicht auch diesen Wert wieder auf den alten Wert zurückstellt, wenn ein Ladezyklus des Autos beendet ist.
Ich hoffe ich habe meine Gedanken einigermaßen verständlich erklärt.
Viele Grüße
detlefh68

Daher kam meine Frage, ob evcc nicht auch diesen Wert wieder auf den alten Wert zurückstellt, wenn ein Ladezyklus des Autos beendet ist.

Das geht in den Templates. Wir könnten es mit auf minsoc stellen?

Im sonnigen Sommer wäre das kein Problem, bei eine schlechten Wetterlage eher schlecht, da das Limit ja dann auch auf einen Kleinen Wert gestellt wird und die Batterie  wenig geladen stehen bleibt, weil die Sonne fehlt.
Super wäre beim Start lesen und dann wieder zurück stellen...fall so etwas beim "victron-energy" Temlate möglich ist.
Das Stellen auf minsoc würde das manuell nachstellen bei schlechter "Sonne" nach sich ziehen. Dieser Automatisch Batterie Modus beim Victron ist schon komfortabel :-)

Super wäre beim Start lesen und dann wieder zurück stellen...fall so etwas beim "victron-energy" Temlate möglich ist.

Oben fehlte das "nicht". Nein und nein.

Das ist schade :-(
Wenn das Wiederherstellen des Limits nicht möglich ist, wäre ein stellen auf minsoc besser.
Wäre das ein Sache für die Zukunft, feature request?
Dann muss ich versuchen dan Ladevorgang des Autos vom evcc im Homeassistant zu erfassen und dort in einer Automatisierung den Limit Wert Wiederherzustellen. Wird das Starten eines Ladevorgangs irgendwie per MQTT mitgeteilt?
Ich habe evcc ja erste ein paar Tage installiert und teste es und es gefällt mir soweit sehr gut.
So wie es im Moment die Hausbatterie beeinflußt in es nicht schlecht und mit Vorsicht und Korrigieren zu nutzen.
Ich hoffe, dass die Interaktion mit dem Victron System irgendwann besser läuft....

Das Verhalten der Victron Batterie scheint es nicht nur bei mir zu geben.
In diesen Beiträgen scheint es sich um das gleiche Verhalten zu handeln
#11286
#11252
Wenn Batterielife am Vitron an ist, müsste es die Veränderungen am minsoc ignorieren.
Es ist nicht so glücklich das Sperren der Batterieentladung über minsoc durchzuführen.

Du kannst gerne einen alternativen Vorschlag machen. Das ist der der umgesetzt ist...

Ich habe die Batterie jetzt erst mal so eingestellt

Und probiere erst nochmal weiter wie es sind bei den nächsten Ladevorgängen des Autos verhält.
Das Limit habe ich versucht über eine Homeassistant Automation beim Start zu lesen und wieder zurückzuschreiben.
Eine Frage habe ich noch...
Wenn ich minsoc in der Konfiguration auf -1 stelle, wird dann nichts an den Victron übertragen?
Trotzdem erst mal Danke für deine Hilfe 😊

repo: evcc-io/evcc


text:
PV Meter login timeout

SMA TRIPOWER inverter is connected via Ethernet / PowerLan to Router. A login timeout is reported as:
updating pv meter 0: update timeout
login failed: no packet received in timeout
When SMA TRIPOWER inverter is directly connected via Ethernet to Router there is no timeout reported.
Expected behavior
No login timeout when using PowerLAN / powerline communication.
EVCC details:
Show output of evcc -v:
evcc version 0.84

Show output of evcc dump -c configfile:
<- paste here

Show evcc configuration file evcc.yaml:
type: template
  template: sma-inverter
  usage: pv
  host: 192.168.178.26  # IP-Adresse oder Hostname
  password: xxxxxxx  # Wird nur benötigt, wenn das Standardpasswort geändert wurde
  name: pv2

Show evcc log output with --log debug:
[main  ] INFO 2022/02/28 10:46:55 evcc 0.84
[main  ] INFO 2022/02/28 10:46:55 using config file /home/pi/evcc.yaml
[main  ] INFO 2022/02/28 10:46:55 listening at 0.0.0.0:7070
[site  ] INFO 2022/02/28 10:46:58 site config:
[site  ] INFO 2022/02/28 10:46:58   meters:      grid ✓ pv ✓ battery ✗
[site  ] INFO 2022/02/28 10:46:58     grid:      power ✓ energy ✓ currents ✓
[site  ] INFO 2022/02/28 10:46:58     pv 0:      power ✓ energy ✓ currents ✓
[lp-1  ] INFO 2022/02/28 10:46:58 loadpoint 1:
[lp-1  ] INFO 2022/02/28 10:46:58   mode:        pv
[lp-1  ] INFO 2022/02/28 10:46:58   charger:     power ✓ energy ✓ currents ✗ phases ✗
[lp-1  ] INFO 2022/02/28 10:46:58   meters:      charge ✓
[lp-1  ] INFO 2022/02/28 10:46:58     charge:    power ✓ energy ✓ currents ✗
[lp-1  ] INFO 2022/02/28 10:46:58   vehicles:    ✗
[main  ] FATAL 2022/02/28 10:46:59 listen tcp 0.0.0.0:7070: bind: address already in use

[main  ] FATAL 2022/02/28 10:46:59 listen tcp 0.0.0.0:7070: bind: address already in use

Da laufen einfach zwei evcc Prozesse gleichzeitig und kommen sich in die Quere.

Ich habe keine zwei Prozesse gestartet.
Dies bekomme ich angezeigt:
pi@raspberrypi:~ $ sudo systemctl status evcc
● evcc.service - evcc
     Loaded: loaded (/lib/systemd/system/evcc.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-02-28 10:57:02 CET; 4min 25s ago
   Main PID: 764 (evcc)
      Tasks: 8 (limit: 415)
        CPU: 8.814s
     CGroup: /system.slice/evcc.service
             └─764 /usr/local/bin/evcc

Feb 28 10:57:31 raspberrypi evcc[764]: [site  ] INFO 2022/02/28 10:57:31   meters:      grid ✓ pv ✓ battery ✗
Feb 28 10:57:31 raspberrypi evcc[764]: [site  ] INFO 2022/02/28 10:57:31     grid:      power ✓ energy ✓ currents ✓
Feb 28 10:57:31 raspberrypi evcc[764]: [site  ] INFO 2022/02/28 10:57:31     pv 0:      power ✓ energy ✓ currents ✓
Feb 28 10:57:31 raspberrypi evcc[764]: [lp-1  ] INFO 2022/02/28 10:57:31 loadpoint 1:
Feb 28 10:57:31 raspberrypi evcc[764]: [lp-1  ] INFO 2022/02/28 10:57:31   mode:        pv
Feb 28 10:57:31 raspberrypi evcc[764]: [lp-1  ] INFO 2022/02/28 10:57:31   charger:     power ✓ energy ✓ currents ✗ phases ✗
Feb 28 10:57:31 raspberrypi evcc[764]: [lp-1  ] INFO 2022/02/28 10:57:31   meters:      charge ✓
Feb 28 10:57:31 raspberrypi evcc[764]: [lp-1  ] INFO 2022/02/28 10:57:31     charge:    power ✓ energy ✓ currents ✗
Feb 28 10:57:31 raspberrypi evcc[764]: [lp-1  ] INFO 2022/02/28 10:57:31   vehicles:    ✗
Feb 28 10:59:46 raspberrypi evcc[764]: [sma   ] ERROR 2022/02/28 10:59:46 login failed: no packet received in timeout

sudo systemctl stop evcc.service
evcc --log debug
sollte Klarheit schaffen.
Wenn diese Meldungen nur gelegentlich auftreten dann ist das Problem eher in deiner Netzwerkinstallation zu suchen.
Ein grundsätzliches Problem in dieser Richtung seitens evcc kann ich jedenfalls ziemlich sicher ausschließen. Einer meiner SMA WR ist auch per PowerLAN angebunden und funktioniert problemlos. Und wenn es bei dir auch ohne PowerLAN funktioniert grenzt dies den Fehler ebenso darauf ein. PowerLAN ist aber eine transparente LAN-Verbindung die aber erhöhtem Paketverlust unterliegt. Da SMA Speedwire über UDP läuft werden verlorere oder defekte Pakete ggf. nicht immer erkannt oder führen zu merkwürdigen Effekten.

Ich kann nur von Powerlan in Verbindung mit Speedwire abraten. Habe das selbst alles durch. Wirklich stabil ist nur ein Kabel. Wenn zu umständlich, lieber den Raspberry mit evcc in Richtung Wechselrichter bewegen und dort alles an einen Switch anschließen.

Wie gesagt: Hier wirklich gar keine Probleme mit Speedwire und PowerLAN. Funktioniert absolut störungsfrei.
SMA-WR kann man alternativ aber auch problemlos per SunSpec/Modbus abfragen (ID 126).

authors: Waldmensch1 premultiply vfricke