text:
added --s3-endpoint-url as default

This fixes the cloudformation package command

LGTM - thanks @pinzon 👍

This change seems to have broken s3api. Example:
It looks like --s3-endpoint-url is not being recognized by the cli.

I executed it without having a localstack running, that's why connection refused was thrown by v.0.9.0

Also seeing this using SNS...
awslocal sns create-topic --name topic-name
Note: AWS CLI version 2, the latest major version of the AWS CLI, is now stable and recommended for general use. For more information, see the AWS CLI version 2 installation instructions at: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help

Unknown options: --s3-endpoint-url=http://localhost:4566

Thanks for the pointer @heliomartins @rodnaph . This is now fixed in 847547fa860f2806351569c197b026afd345ae51 - a new version 0.11 has just been released. Please let us know if there are any other issues. Thanks!

Confirmed this fixes it for my use-case - thanks for the quick reply @whummer 👍

Hello @whummer - It also fixes for me, thanks for doing it so quickly. 👍

@whummer I am using 0.14 and I still get Unknown options: --s3-endpoint-url=http://localhost:4566,--s3-endpoint-url=http:localhost:4566 when trying to run awslocal cloudformation package ...

authors: heliomartins paul-uz pinzon rodnaph whummer

repo: localstack/awscli-local


text:
Localtunnel.me is down

Resulting with: 502 Bad Gateway

+1

+1

localtunnel.me is up right now for me.

source: gharchive/issue

repo: localtunnel/localtunnel

license: MIT

license_type: permissive

license_source: github-api


text:
Not connecting

the client console doesnt show anything

the server is running normally, no idea whats happening

I am using AWS EC2 and I set allow all traffic, it should work
screenshot of the settings

Hey can you tell me what this issue is and did you faced it too
localtunnel-server$ bin/server --port 1234
/usr/bin/env: ‘node -r esm’: No such file or directory
/usr/bin/env: use -[v]S to pass options in shebang lines

repo: localtunnel/server

license: mit

license_source: bigquery


text:
Python build zip for Maven

sbt pyrasterframes/package prints something like this:
[info] Maven Python .zip artifact written to '/Users/jbrown/src/raster-frames/pyrasterframes/target/scala-2.11/pyrasterframes-0.8.0-SNAPSHOT-py2.py3-none-any.whl'

But the statement does not make sense. Its saying the maven artifact is the whl? And I don't find a pyrasterframes zip anywhere in pyrasterframes/target after the command completes.

try sbt pyrasterframes/publishLocal and check out the ~/.ivy2/local to see what would be published...

Ok I am able to confirm that the wheel works just fine as far as i can tell

publish local : the source zip gets published to my local ivy caches as group/artifact/version/zips/name, in particular

[info] Maven Python .zip artifact written to '/Users/jbrown/src/raster-frames/pyrasterframes/target/scala-2.11/pyrasterframes-0.8.0-SNAPSHOT-py2.py3-none-any.whl'
[info] :: delivering :: org.locationtech.rasterframes#pyrasterframes_2.11;0.8.0-SNAPSHOT :: 0.8.0-SNAPSHOT :: integration :: Thu Aug 08 13:35:01 EDT 2019
[info] 	delivering ivy file to /Users/jbrown/src/raster-frames/pyrasterframes/target/scala-2.11/ivy-0.8.0-SNAPSHOT.xml
[info] 	published pyrasterframes_2.11 to /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/docs/pyrasterframes_2.11-javadoc.jar
[info] 	published pyrasterframes_2.11 to /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/jars/pyrasterframes_2.11.jar
[info] 	published pyrasterframes_2.11 to /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/zips/pyrasterframes_2.11-python.zip
[info] 	published pyrasterframes_2.11 to /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/poms/pyrasterframes_2.11.pom
[info] 	published pyrasterframes_2.11 to /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/srcs/pyrasterframes_2.11-sources.jar
[info] 	published ivy to /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/ivys/ivy.xml

It has python sources in it. (plus assembly jar)

zipinfo -l /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/zips/pyrasterframes_2.11-python.zip
Archive:  /Users/jbrown/.ivy2/local/org.locationtech.rasterframes/pyrasterframes_2.11/0.8.0-SNAPSHOT/zips/pyrasterframes_2.11-python.zip
Zip file size: 62669882 bytes, number of entries: 15
-rw-r--r--  2.0 unx        0 b-        2 defN 19-Jul-15 19:42 geomesa_pyspark/__init__.py
-rw-r--r--  2.0 unx     2520 b-      864 defN 19-Jul-15 19:42 geomesa_pyspark/types.py
-rw-r--r--  2.0 unx     8182 b-     2444 defN 19-Aug-08 17:18 pyrasterframes/__init__.py
-rw-r--r--  2.0 unx    26660 b-     4393 defN 19-Aug-07 15:37 pyrasterframes/rasterfunctions.py
-rw-r--r--  2.0 unx     2666 b-     1087 defN 19-Jul-15 19:42 pyrasterframes/rf_context.py
-rw-r--r--  2.0 unx     4100 b-     1735 defN 19-Jul-31 14:31 pyrasterframes/rf_ipython.py
-rw-r--r--  2.0 unx    15526 b-     3764 defN 19-Aug-01 13:08 pyrasterframes/rf_types.py
-rw-r--r--  2.0 unx     3188 b-     1351 defN 19-Aug-07 13:38 pyrasterframes/utils.py
-rw-r--r--  2.0 unx      801 b-      492 defN 19-Aug-08 17:33 pyrasterframes/version.py
-rw-r--r--  2.0 unx 71011581 b- 62648970 defN 19-Aug-08 17:34 pyrasterframes/jars/pyrasterframes-assembly-0.8.0-SNAPSHOT.jar
-rw-r--r--  2.0 unx      663 b-      410 defN 19-Aug-08 17:34 pyrasterframes-0.8.0.dev88.dist-info/LICENSE.txt
-rw-r--r--  2.0 unx     2750 b-     1267 defN 19-Aug-08 17:34 pyrasterframes-0.8.0.dev88.dist-info/METADATA
-rw-r--r--  2.0 unx      110 b-       95 defN 19-Aug-08 17:34 pyrasterframes-0.8.0.dev88.dist-info/WHEEL
-rw-r--r--  2.0 unx       31 b-       32 defN 19-Aug-08 17:34 pyrasterframes-0.8.0.dev88.dist-info/top_level.txt
?rw-rw-r--  2.0 unx     1329 b-      754 defN 19-Aug-08 17:35 pyrasterframes-0.8.0.dev88.dist-info/RECORD

It's ok to pass that zip into pyspark --py-files: I did the publishLocal and pass the resulting zips/pyrasterframes_2.11-python.zip in; I can import and and see my nonsenical local changes in the interpreter there. (see getting started page for more)

Will submit tiny PR to clarify the output statement there...

repo: locationtech/rasterframes


text:
Scaffold generates broken code in some failure cases

Description
cargo loco generate scaffold logs file_name:string hash:string created_at:ts
This will fail to generate the model and _entities files because created_at is already included (perhaps this could be documented), but the command will still generate controller and view which will be broken. There is some warnings displayed but they're not super intuitive without having already seen this bug.
To Reproduce
cargo loco generate scaffold logs file_name:string hash:string created_at:ts
Expected Behavior
The command either succeeds, or displays an error and doesn't make changes.
If a default column is specified by the user it could replace the column or it could just error (the latter might be more straightforward). I think either behavior would be fine.
Environment:
MBP
Additional Context
I haven't had time to dig into the code generation too much but afaict scaffold only generates whole new files with the exception of mod.rs files so maybe it could keep the original mod.rs contents in memory and there could be an unwind action on failures that removes created files and reverts the mod.rss

Indeed a special case! thanks for catching this!
I can think of two solutions:

check a field existence before adding (using the various injection tools we have)
keep a list of created_at, etc. that we don't allow from the CLI when it's generating models because we already attach these by default (I'm in favor of this one, all generators give you created/updated for free)

would you like to contribute a solution?

I’d be happy to but it might be a minute. I might be able to get a PR in
next week or this weekend if I get lucky.
On Wed, Dec 13, 2023 at 13:36 Dotan J. Nahum @.***>
wrote:

Indeed a special case! thanks for catching this!
I can think of two solutions:

check a field existence before adding (using the various injection
tools we have)
keep a list of created_at, etc. that we don't allow from the CLI
when it's generating models because we already attach these by default (I'm
in favor of this one, all generators give you created/updated for free)

would you like to contribute a solution?
—
Reply to this email directly, view it on GitHub
https://github.com/loco-rs/loco/issues/143#issuecomment-1854509350, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2S7RS256DEURJWEMLP72DYJHYTBAVCNFSM6AAAAABATQS43CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJUGUYDSMZVGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

@BWStearns be aware that i already fix it

Cool!

@kaplanelad fixed it

@BWStearns, no worries! Just wanted to mention that this pull request hasn't been merged yet. I'm waiting for the CI to pass.

authors: BWStearns jondot kaplanelad

repo: loco-rs/loco


text:
🛑 PotusTV is down

In df086c2, PotusTV (https://tv.potus.ar) was down:

HTTP code: 0
Response time: 0 ms

Resolved: PotusTV is back up in d61eaf0.


text:
Merry Christmas!

It's been a really tough two weeks.
Most of the issues are created between 1:00 and 5:00 a.m. KST.
However I couldn't sleep because it was too critical vulnerability.
I really appriciate to all contributors. Especially:

Christian Kemper (Afrouper), who sent many Pull Requests.
James Stewart (jgstew), who lead BigFix community.
Christian W. (doctore74), who made Checkmk automation and lead community.
And many engineers from Germany.

Since traffic is gradually decreasing, I think now is good time to say thank you.

It was amazing experience.
We all defended the Internet together.
Best Regards
From CEO of Logpresso

A heartfelt thank you for your dedication to detecting, reporting on and mitigating these vulnerabilities!
Like many people here, I was tasked with finding a tool that could scan the file system to provide an understanding of our level of risk. I have watched with awe the rapid pace of development and continual improvements during LONG hours of the day as well as a weekend of refactoring and further improvements and I wanted to take a moment to express my gratitude all of the contributors. Thank you!
Merry Christmas and best wishes to you all.


text:
Update sbt-microsites to 1.3.4

Updates com.47deg:sbt-microsites from 1.2.1 to 1.3.4.
GitHub Release Notes - Changelog - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.47deg", artifactId = "sbt-microsites" } ]

labels: sbt-plugin-update, semver-minor

Codecov Report

Merging #29 (086dab2) into master (4d7a553) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #29   +/-   ##
=======================================
  Coverage   97.43%   97.43%
=======================================
  Files          23       23
  Lines         895      895
  Branches        6        6
=======================================
  Hits          872      872
  Misses         23       23

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d7a553...086dab2. Read the comment docs.

Superseded by #77.

source: gharchive/pull-request

authors: codecov-commenter scala-steward

repo: loicknuchel/SafeQL


text:
esphomebtproxy

Hi,
Does this add on work with the esphomebtproxy?
https://esphome.github.io/bluetooth-proxies/

Probably not. This is not a native HA integration, even as a custom component, but a stand-alone script that communicates with HA vi aMQTT. As such the device that runs the script needs bluetooth communication and the scale needs to be within reach


text:
Change to suggested schema?

For graphite table, switching to the following moves us from 5 bytes/row to 0.6 bytes/row for a large dataset:
    `Path` LowCardinality(String),
    `Value` Float64,
    `Time` UInt32 Codec(DoubleDelta, LZ4),
    `Date` Date Codec(DoubleDelta, LZ4),
    `Timestamp` UInt32 Codec(DoubleDelta, LZ4)

toYYYYMMDD() might be a better partitioning key for large volumes of data
https://github.com/ClickHouse/ClickHouse/issues/12144#issuecomment-653951462 also suggests removing the Date col - perhaps this could be an option (although when compressed as above only takes a very small amount of space)

I also got similar results when experimenting with compression and low cardinality, but I do not think it's universal enough and need to be recommended as generic solution.
Same for  partitioning key, it's really depends on use case.
For Data column it's also very confusing, theoretically you need do exact opposite thing - get rid of Timestamp with zero-timestamp=true and use Date/Time for queries.

I'd have thought other than partitioning the suggested changes above should benefit everyone as Path will be repeated a lot and Time/Date/Timestamp are all usually increasing and with lots of duplicates

I mean, I think it's possible to remove Date, it's used by graphite-clickhouse. You can zero Timestamp and have mostly same effect with is documented and recommended way.

See this extract from one of my servers:
SELECT
    column,
    any(type),
    formatReadableSize(sum(column_data_compressed_bytes)) AS compressed,
    formatReadableSize(sum(column_data_uncompressed_bytes)) AS uncompressed,
    sum(rows)
FROM system.parts_columns
WHERE (table = 'graphite_data') AND active
GROUP BY column
ORDER BY column ASC

┌─column────┬─any(type)─┬─compressed─┬─uncompressed─┬──sum(rows)─┐
│ Date      │ Date      │ 69.84 MiB  │ 12.24 GiB    │ 6569285540 │
│ Path      │ String    │ 4.60 GiB   │ 1.04 TiB     │ 6569285540 │
│ Time      │ UInt32    │ 17.00 GiB  │ 24.47 GiB    │ 6569285540 │
│ Timestamp │ UInt32    │ 19.38 GiB  │ 24.47 GiB    │ 6569285540 │
│ Value     │ Float64   │ 6.40 GiB   │ 48.94 GiB    │ 6569285540 │
└───────────┴───────────┴────────────┴──────────────┴────────────┘

The Date column has not much impact. The Path might be useful to have as LowCardinality, although it only has big impact in uncompressed format.
I just found out the setting for zero-timestamp, I will change it so that Timestamp will only contain zero value. And will check and compare again later. There is not much documentation on this setting, so I'm wondering why it is enabled by default and why it is needed?

Be aware, that LowCardinality impacts the size of the marks. Since it doubles it for Path, I've had a lot of OOMs during experiments with it. In ClickHouse telegram chat people have suggested me to use ZSTD, and it works quite well too. Here's my own experiments' final state:
ATTACH TABLE data_lr
(
    `Path` String CODEC(ZSTD(3)),
    `Value` Float64 CODEC(Gorilla, LZ4),
    `Time` UInt32 CODEC(DoubleDelta, LZ4),
    `Date` Date CODEC(DoubleDelta, LZ4),
    `Timestamp` UInt32 CODEC(DoubleDelta, LZ4)
)
ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/graphite.data_lr/{shard}', '{replica}', 'graphite_rollup')
PARTITION BY toYYYYMMDD(Date)
ORDER BY (Path, Time)
SETTINGS index_granularity = 256```

4 months later and I have seen that compress ratio for column Timestamp has gone from 4-5 up to 220+
So again, why is zero-timestamp not set by default? Why is it useful to have an actual timestamp in it? I don't see the use-case here.
@Felixoid I didn't do anything special to the datamodel, I only added a TTL based on the Date column.

4 months later and I have seen that compress ratio for column Timestamp has gone from 4-5 up to 220+
So again, why is zero-timestamp not set by default? Why is it useful to have an actual timestamp in it? I don't see the use-case here.
@Felixoid I didn't do anything special to the datamodel, I only added a TTL based on the Date column.

The Timestamp column is used internally by GraphiteMergeTree as Version column during the rollup process. I'm afraid, I already don't remember details of how it's used, but the doc tells:

Version of the metric. Data type: any numeric. ClickHouse saves the rows with the highest version or the last written if versions are the same. Other rows are deleted during the merge of data parts.

Here's a comment from the code https://github.com/ClickHouse/ClickHouse/fc42851/master/src/Processors/Merges/Algorithms/GraphiteRollupSortedAlgorithm.h#L78-L90
    /* | path | time | rounded_time | version | value | unmodified |
     * -----------------------------------------------------------------------------------
     * | A    | 11   | 10           | 1       | 1     | a          |                     |
     * | A    | 11   | 10           | 3       | 2     | b          |> subgroup(A, 11)    |
     * | A    | 11   | 10           | 2       | 3     | c          |                     |> group(A, 10)
     * ----------------------------------------------------------------------------------|>
     * | A    | 12   | 10           | 0       | 4     | d          |                     |> Outputs (A, 10, avg(2, 5), a)
     * | A    | 12   | 10           | 1       | 5     | e          |> subgroup(A, 12)    |
     * -----------------------------------------------------------------------------------
     * | A    | 21   | 20           | 1       | 6     | f          |
     * | B    | 11   | 10           | 1       | 7     | g          |
     * ...
     */

I don't think setting zero-timestamp=true by default is a good idea. The much more valuable change is using proper codecs, like Timestamp UInt32 CODEC(DoubleDelta, LZ4). As you see, it helps significantly to store the data more effectively.
SELECT
    column,
    any(type),
    formatReadableSize(sum(column_data_compressed_bytes)) AS compressed,
    formatReadableSize(sum(column_data_uncompressed_bytes)) AS uncompressed,
    sum(column_data_uncompressed_bytes) / sum(column_data_compressed_bytes) AS ratio,
    sum(rows)
FROM system.parts_columns
WHERE ((table = 'data_lr') AND (database = 'graphite')) AND active
GROUP BY column
ORDER BY column ASC

┌─column────┬─any(type)─┬─compressed─┬─uncompressed─┬──────────────ratio─┬───sum(rows)─┐
│ Date      │ Date      │ 59.17 MiB  │ 45.58 GiB    │   788.813125428121 │ 24468984334 │
│ Path      │ String    │ 9.42 GiB   │ 1.41 TiB     │ 153.15145708165463 │ 24468984334 │
│ Time      │ UInt32    │ 961.97 MiB │ 91.15 GiB    │   97.0315116787169 │ 24468984334 │
│ Timestamp │ UInt32    │ 4.33 GiB   │ 91.15 GiB    │  21.03714874982845 │ 24468984334 │
│ Value     │ Float64   │ 55.59 GiB  │ 182.31 GiB   │  3.279419424878342 │ 24468984334 │
└───────────┴───────────┴────────────┴──────────────┴────────────────────┴─────────────┘

The Timestamp column is used internally by GraphiteMergeTree as Version column during the rollup process. I'm afraid, I already don't remember details of how it's used, but the doc tells:

Version of the metric. Data type: any numeric. ClickHouse saves the rows with the highest version or the last written if versions are the same. Other rows are deleted during the merge of data parts.

Here's a comment from the code https://github.com/ClickHouse/ClickHouse/fc42851/master/src/Processors/Merges/Algorithms/GraphiteRollupSortedAlgorithm.h#L78-L90
    /* | path | time | rounded_time | version | value | unmodified |
     * -----------------------------------------------------------------------------------
     * | A    | 11   | 10           | 1       | 1     | a          |                     |
     * | A    | 11   | 10           | 3       | 2     | b          |> subgroup(A, 11)    |
     * | A    | 11   | 10           | 2       | 3     | c          |                     |> group(A, 10)
     * ----------------------------------------------------------------------------------|>
     * | A    | 12   | 10           | 0       | 4     | d          |                     |> Outputs (A, 10, avg(2, 5), a)
     * | A    | 12   | 10           | 1       | 5     | e          |> subgroup(A, 12)    |
     * -----------------------------------------------------------------------------------
     * | A    | 21   | 20           | 1       | 6     | f          |
     * | B    | 11   | 10           | 1       | 7     | g          |
     * ...
     */

I don't think setting zero-timestamp=true by default is a good idea. The much more valuable change is using proper codecs, like Timestamp UInt32 CODEC(DoubleDelta, LZ4). As you see, it helps significantly to store the data more effectively.
SELECT
    column,
    any(type),
    formatReadableSize(sum(column_data_compressed_bytes)) AS compressed,
    formatReadableSize(sum(column_data_uncompressed_bytes)) AS uncompressed,
    sum(column_data_uncompressed_bytes) / sum(column_data_compressed_bytes) AS ratio,
    sum(rows)
FROM system.parts_columns
WHERE ((table = 'data_lr') AND (database = 'graphite')) AND active
GROUP BY column
ORDER BY column ASC

┌─column────┬─any(type)─┬─compressed─┬─uncompressed─┬──────────────ratio─┬───sum(rows)─┐
│ Date      │ Date      │ 59.17 MiB  │ 45.58 GiB    │   788.813125428121 │ 24468984334 │
│ Path      │ String    │ 9.42 GiB   │ 1.41 TiB     │ 153.15145708165463 │ 24468984334 │
│ Time      │ UInt32    │ 961.97 MiB │ 91.15 GiB    │   97.0315116787169 │ 24468984334 │
│ Timestamp │ UInt32    │ 4.33 GiB   │ 91.15 GiB    │  21.03714874982845 │ 24468984334 │
│ Value     │ Float64   │ 55.59 GiB  │ 182.31 GiB   │  3.279419424878342 │ 24468984334 │
└───────────┴───────────┴────────────┴──────────────┴────────────────────┴─────────────┘

And one additional note, when I've tested the LowCardinality(String) as a Path column, it did consume the double amount of memory for the marks. And my serves were killed by OOM quite often, especially during the wide Time+Path requests. Usage of String CODEC(ZSTD(3)) is more reliable. At least, it was the state of Sep 2019. Not sure how it's now.

And one additional note, when I've tested the LowCardinality(String) as a Path column, it did consume the double amount of memory for the marks. And my serves were killed by OOM quite often, especially during the wide Time+Path requests. Usage of String CODEC(ZSTD(3)) is more reliable. At least, it was the state of Sep 2019. Not sure how it's now.

So, I still don't see use case to have anything other than 0 in Timestamp col. Why would there be a case that there are multiple rows on the same Path and Time and different value? It seems to me those would be very specific use-case that should not be the reason for a default setting that uses more storage than needed?

So, I still don't see use case to have anything other than 0 in Timestamp col. Why would there be a case that there are multiple rows on the same Path and Time and different value? It seems to me those would be very specific use-case that should not be the reason for a default setting that uses more storage than needed?

Because neither I nor you know the internals of GraphiteMergeTree engine, I'd say. And I feel uncomfortable making a decision for everybody to change the default behavior.
Besides that, with the proper codec, it doesn't hurt so much. Plus, it just came to my mind, it would make sense to add a custom TTL to the Timestamp column. Something pretty big to be sure, that rollup is done at least once. Then it will be pretty insignificant.

Because neither I nor you know the internals of GraphiteMergeTree engine, I'd say. And I feel uncomfortable making a decision for everybody to change the default behavior.
Besides that, with the proper codec, it doesn't hurt so much. Plus, it just came to my mind, it would make sense to add a custom TTL to the Timestamp column. Something pretty big to be sure, that rollup is done at least once. Then it will be pretty insignificant.

But instead of changing the table schema, is there an issue in using toYYYYMMDD() for partitioning? Optimizing > 1.5TB partitions down to ~100G just needs an useless amount of empty space. (Does anybody know why clickhouse requires the 100% overhead?)

It's something I can answer

But instead of changing the table schema, is there an issue in using toYYYYMMDD() for partitioning?

It depends more on the amount of data. Partitions bring overhead. And if you have broad historical inserts (like over a year), then the memory consumption for such batches would be huge. On the other hand, the overhead is minimized, so one should find a balance for himself. I personally have used partitions for 3 days, but a half-year ago migrated to toYYYYMMDD() and don't see issues. So it sounds like a solution I can suggest.

Does anybody know why clickhouse requires the 100% overhead?

It's inherited from the MergeTree engine. The 100% space allocation required to be safe during merging all parts together. The server can't know in advance how much space may be really needed and how aggressive aggregation will be, so can't predict the necessary allocation more precisely than just taking an existing size.

It's something I can answer

But instead of changing the table schema, is there an issue in using toYYYYMMDD() for partitioning?

It depends more on the amount of data. Partitions bring overhead. And if you have broad historical inserts (like over a year), then the memory consumption for such batches would be huge.

Basically I'm inserting ~400k rows/s with data from the last 10..300 seconds. Old data is rarely inserted.

On the other hand, the optimizing overhead is minimized, so one should find a balance for himself. I personally have used partitions for 3 days, but a half-year ago migrated to toYYYYMMDD() and don't see issues. So it sounds like a solution I can suggest.

Thanks, appreciated.

authors: Felixoid Hipska bzed deniszh mzealey

repo: lomik/carbon-clickhouse


text:
fix: cursor jumps out of inputs in modal

fixes https://github.com/lonelyplanet/dotcom-profile/issues/407

@thomasthesecond thanks! should be resolved now

repo: lonelyplanet/backpack-ui


text:
feat: add tsconfig file config option

This allows the user to add a custom tsconfig file path.
Useful when nest makes tsconfig.app.json files.

cc @clement-estone

@Godefroy are you able to help review this? Would love to get this option in, it would help with my usage of this lib a lot! :)

Sorry I was away for a few weeks.
This LGTM, merging.
Thanks for your contribution ;)

Awesome. Thanks :)
What do you think about this one too: https://github.com/lonestone/nest-sdk-generator/pull/6/files ?

authors: clement-estone rek

repo: lonestone/nest-sdk-generator


text:
cache invalid default value

@cache() has expire: int = None which is invalid for memcached. Thus either this parameter should be mandatory or a sensible default should be introduced.

Fixed in dev branch


text:
🛑 Less Plastic Shop is down

In ea939ee, Less Plastic Shop (https://shop.lessplastic.jau.co.jp) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Less Plastic Shop is back up in cecfb3b.

repo: longforme/longforme-uptime-monitor


text:
Fix backupstore list issues

Previously when loading the backup names error we would return an error for the whole list call, instead, we should return volume info with an error set.
This way an issue on one volume cannot block the whole backup list from showing up.

@joshimoo We will need an engine test case and e2e test case for this. Can you file an issue? We might want to merge it for v1.0.1 instead.

authors: joshimoo yasker

repo: longhorn/backupstore


text:
Fix test_recurring_jobs_allow_detached_volume & add more recurring job test cases

https://github.com/longhorn/longhorn/issues/3517
https://github.com/longhorn/longhorn/issues/3470

Tried running these changes on my local env and it seems something didn't add up. just leave logs here.
Both test_recurring_job_delete_should_remove_volume_label and test_recurring_job_volume_label_when_job_and_group_use_same_name are not able to finish without error.

Test result logs:

root@longhorn-test:/integration/tests# ./run.sh test_recurring_job.py
+ flake8
+ pytest -v test_recurring_job.py
======================================================================================= test session starts =======================================================================================
platform linux -- Python 3.9.10, pytest-5.3.1, py-1.11.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /integration, inifile: pytest.ini
plugins: repeat-0.9.1
collected 28 items

test_recurring_job.py::test_recurring_job[s3] PASSED                                                                                [  3%]
test_recurring_job.py::test_recurring_job[nfs] PASSED                                                                                [  7%]
test_recurring_job.py::test_recurring_job_in_volume_creation PASSED                                                                                [ 10%]
test_recurring_job.py::test_recurring_job_duplicated PASSED                                                                                [ 14%]
test_recurring_job.py::test_recurring_job_in_storageclass[s3] PASSED                                                                                [ 17%]
test_recurring_job.py::test_recurring_job_in_storageclass[nfs] PASSED                                                                                [ 21%]
test_recurring_job.py::test_recurring_job_labels[s3] PASSED                                                                                [ 25%]
test_recurring_job.py::test_recurring_job_labels[nfs] PASSED                                                                                [ 28%]
test_recurring_job.py::test_recurring_job_kubernetes_status[s3] PASSED                                                                                [ 32%]
test_recurring_job.py::test_recurring_job_kubernetes_status[nfs] PASSED                                                                                [ 35%]
test_recurring_job.py::test_recurring_jobs_maximum_retain PASSED                                                                                [ 39%]
test_recurring_job.py::test_recurring_job_detached_volume PASSED                                                                                [ 42%]
test_recurring_job.py::test_recurring_jobs_allow_detached_volume[s3] PASSED                                                                                [ 46%]
test_recurring_job.py::test_recurring_jobs_allow_detached_volume[nfs] PASSED                                                                                [ 50%]
test_recurring_job.py::test_recurring_jobs_when_volume_detached_unexpectedly[s3] PASSED                                                                                [ 53%]
test_recurring_job.py::test_recurring_jobs_when_volume_detached_unexpectedly[nfs] PASSED                                                                                [ 57%]
test_recurring_job.py::test_recurring_jobs_on_nodes_with_taints SKIPPED                                                                                [ 60%]
test_recurring_job.py::test_recurring_job_groups[s3] PASSED                                                                                [ 64%]
test_recurring_job.py::test_recurring_job_groups[nfs] PASSED                                                                                [ 67%]
test_recurring_job.py::test_recurring_job_default PASSED                                                                                [ 71%]
test_recurring_job.py::test_recurring_job_delete PASSED                                                                                [ 75%]
test_recurring_job.py::test_recurring_job_delete_should_remove_volume_label FAILED                                                                                [ 78%]
test_recurring_job.py::test_recurring_job_volume_label_when_job_and_group_use_same_name FAILED                                                                                [ 82%]
test_recurring_job.py::test_recurring_job_multiple_volumes[s3] PASSED                                                                                [ 85%]
test_recurring_job.py::test_recurring_job_multiple_volumes[nfs] PASSED                                                                                [ 89%]
test_recurring_job.py::test_recurring_job_snapshot PASSED                                                                                [ 92%]
test_recurring_job.py::test_recurring_job_backup[s3] PASSED                                                                                [ 96%]
test_recurring_job.py::test_recurring_job_backup[nfs] PASSED                                                                                [100%]

============================================================================================ FAILURES =============================================================================================
______________________________________________________________________ test_recurring_job_delete_should_remove_volume_label _______________________________________________________________________

client = <longhorn.Client object at 0x7f832e75cee0>, batch_v1_beta_api = <kubernetes.client.api.batch_v1beta1_api.BatchV1beta1Api object at 0x7f832e255040>
volume_name = 'longhorn-testvol-0jmf3u'

    @pytest.mark.recurring_job  # NOQA
    def test_recurring_job_delete_should_remove_volume_label(client, batch_v1_beta_api, volume_name):  # NOQA
        """
        Scenario: test delete recurring job should remove volume labels

        Given 1 volume created.
        And  create `snapshot1` recurring job.
             create `backup1`   recurring job.
        And  `snapshot1` job exist in volume recurring job label.
             `backup1`   job exist in volume recurring job label.

        When delete `snapshot1` recurring job.
        Then `snapshot1` job not exist in volume recurring job label.
             `backup1`   job     exist in volume recurring job label.

        When delete `backup1` recurring job.
        Then `snapshot1` job not exist in volume recurring job label.
             `backup1`   job not exist in volume recurring job label.

        Given create `snapshot1` recurring job with `group-1` in groups.
        And   create `snapshot2` recurring job with `group-1` in groups.
        And   create `backup1`   recurring job with `group-1` in groups.
        And   remove volume recurring job label `default` job-group.
        And  `group-1` job-group     exist in volume recurring job label.
             `default` job-group not exist in volume recurring job label.

        When delete `snapshot1` recurring job.
        Then `group-1` job-group exist in volume recurring job label.

        When delete `snapshot2` recurring job.
        Then `group-1` job-group exist in volume recurring job label.

        When delete `back1` recurring job.
        Then `group-1` job-group not exist in volume recurring job label.
        """
        client.create_volume(name=volume_name, size=SIZE)
        volume = wait_for_volume_detached(client, volume_name)

        snap1 = SNAPSHOT + "1"
        snap2 = SNAPSHOT + "2"
        back1 = BACKUP + "1"
        group1 = "group-1"
        snap_job = {
            TASK: SNAPSHOT,
            GROUPS: [group1],
            CRON: SCHEDULE_1MIN,
            RETAIN: 1,
            CONCURRENCY: 2,
            LABELS: {},
        }
        back_job = {
            TASK: BACKUP,
            GROUPS: [group1],
            CRON: SCHEDULE_1MIN,
            RETAIN: 1,
            CONCURRENCY: 2,
            LABELS: {},
        }
        recurring_jobs = {
            snap1: snap_job,
            back1: back_job,
        }
        create_recurring_jobs(client, recurring_jobs)
        check_recurring_jobs(client, recurring_jobs)

        # Delete recurring job should remove volume label
        volume.recurringJobAdd(name=snap1, isGroup=False)
        volume.recurringJobAdd(name=back1, isGroup=False)
        wait_for_volume_recurring_job_update(volume,
                                             jobs=[snap1, back1], groups=[DEFAULT])

        snap1_recurring_job = client.by_id_recurring_job(snap1)
        back1_recurring_job = client.by_id_recurring_job(back1)
        client.delete(snap1_recurring_job)
>       wait_for_volume_recurring_job_update(volume,
                                             jobs=[back1], groups=[DEFAULT])

test_recurring_job.py:1302:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

volume = {'accessMode': 'rwo', 'backingImage': '', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'sta...reEndpoint': '', 'shareState': '', 'size': '16777216', 'staleReplicaTimeout': 0, 'standby': False, 'state': 'detached'}
jobs = ['backup1'], groups = ['default']

    def wait_for_volume_recurring_job_update(volume, jobs=[], groups=[]):
        ok = False
        for _ in range(RETRY_COUNTS):
            volumeJobs, volumeGroups = get_volume_recurring_jobs_and_groups(volume)
            try:
                assert len(volumeGroups) == len(groups)
                for group in groups:
                    assert group in volumeGroups

                assert len(volumeJobs) == len(jobs)
                for job in jobs:
                    assert job in volumeJobs

                ok = True
                break
            except AssertionError:
                time.sleep(RETRY_INTERVAL)
>       assert ok
E       AssertionError

common.py:4419: AssertionError
________________________________________________________________ test_recurring_job_volume_label_when_job_and_group_use_same_name _________________________________________________________________

client = <longhorn.Client object at 0x7f832e3558b0>, volume_name = 'longhorn-testvol-boeopp'

    @pytest.mark.recurring_job  # NOQA
    def test_recurring_job_volume_label_when_job_and_group_use_same_name(client, volume_name):  # NOQA
        """
        Scenario: test volume label recurring job when recurring job and
                  job-group uses the same name.

        Given volume-1 created.
              volume-2 created.
              volume-3 created.
        And  create `snapshot1` recurring job with `snapshot-1` in groups.
             create `snapshot2` recurring job with `snapshot-1` in groups.
        And  `snapshot1` job-group exist in volume-1 recurring job label.
             `snapshot1` job       exist in volume-2 recurring job label.
             `snapshot2` job       exist in volume-3 recurring job label.

        When delete `snapshot1` recurring job.
        Then `snapshot1` job-group     exist in volume-1 recurring job label.
             `snapshot1` job       not exist in volume-2 recurring job label.
             `snapshot2` job           exist in volume-3 recurring job label.

        When delete `snapshot2` recurring job.
        Then `snapshot1` job-group not exist in volume-1 recurring job label.
             `snapshot2` job       not exist in volume-3 recurring job label.
        """
        volume1_name = volume_name + "-1"
        volume2_name = volume_name + "-2"
        volume3_name = volume_name + "-3"
        client.create_volume(name=volume1_name, size=SIZE)
        client.create_volume(name=volume2_name, size=SIZE)
        client.create_volume(name=volume3_name, size=SIZE)
        volume1 = wait_for_volume_detached(client, volume1_name)
        volume2 = wait_for_volume_detached(client, volume2_name)
        volume3 = wait_for_volume_detached(client, volume3_name)

        snap1 = SNAPSHOT + "1"
        snap2 = SNAPSHOT + "2"
        snapshot = {
            TASK: SNAPSHOT,
            GROUPS: [snap1],
            CRON: SCHEDULE_1MIN,
            RETAIN: 1,
            CONCURRENCY: 2,
            LABELS: {},
        }
        recurring_jobs = {
            snap1: snapshot,
            snap2: snapshot,
        }
        create_recurring_jobs(client, recurring_jobs)
        check_recurring_jobs(client, recurring_jobs)

        # Delete recurring job should correctly remove volume label
        volume1.recurringJobAdd(name=snap1, isGroup=True)
        volume2.recurringJobAdd(name=snap1, isGroup=False)
        volume3.recurringJobAdd(name=snap2, isGroup=False)
        wait_for_volume_recurring_job_update(volume1,
                                             jobs=[], groups=[snap1, DEFAULT])
        wait_for_volume_recurring_job_update(volume2,
                                             jobs=[snap1], groups=[DEFAULT])
        wait_for_volume_recurring_job_update(volume3,
                                             jobs=[snap2], groups=[DEFAULT])

        snap1_recurring_job = client.by_id_recurring_job(snap1)
        snap2_recurring_job = client.by_id_recurring_job(snap2)
        client.delete(snap1_recurring_job)
>       wait_for_volume_recurring_job_update(volume2,
                                             jobs=[], groups=[DEFAULT])

test_recurring_job.py:1409:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

volume = {'accessMode': 'rwo', 'backingImage': '', 'backupStatus': [], 'cloneStatus': {'snapshot': '', 'sourceVolume': '', 'sta...reEndpoint': '', 'shareState': '', 'size': '16777216', 'staleReplicaTimeout': 0, 'standby': False, 'state': 'detached'}
jobs = [], groups = ['default']

    def wait_for_volume_recurring_job_update(volume, jobs=[], groups=[]):
        ok = False
        for _ in range(RETRY_COUNTS):
            volumeJobs, volumeGroups = get_volume_recurring_jobs_and_groups(volume)
            try:
                assert len(volumeGroups) == len(groups)
                for group in groups:
                    assert group in volumeGroups

                assert len(volumeJobs) == len(jobs)
                for job in jobs:
                    assert job in volumeJobs

                ok = True
                break
            except AssertionError:
                time.sleep(RETRY_INTERVAL)
>       assert ok
E       AssertionError

common.py:4419: AssertionError
====================================================================== 2 failed, 25 passed, 1 skipped in 8331.90s (2:18:51) =======================================================================

Tried running these changes on my local env and it seems something didn't add up. just leave logs here.
Both test_recurring_job_delete_should_remove_volume_label and test_recurring_job_volume_label_when_job_and_group_use_same_name are not able to finish without error.

The tests are dependent on this PR.

Is there a test case for the following corner case? Add/Remove a recurring job for a volume of which the job group already contains the recurring job.

It's covered in test_recurring_job_volume_label_when_job_and_group_use_same_name.

repo: longhorn/longhorn-tests


text:
task: Update CSI image tags

Now it's all Longhorn custom versions.
https://github.com/longhorn/longhorn-manager/pull/715

Done

repo: longhorn/longhorn


text:
[TASK] Check and update the networking doc & example YAMLs

What's the task? Please describe
Some entries in the networking doc are out-of-date. We need to update the doc for reference.
Describe the items of the task (DoD, definition of done) you'd like

[x] Check and update the entries in the doc.
[x] Check and update the NetworkPolicy examples.
[x] May need to add a pre-release test plan of NetworkPolicy No extra test or no change needed for existing tests. Re-running existing backing image related tests with the network policy is enough

Additional context
https://github.com/longhorn/longhorn/discussions/3031

Verified passed by following steps:

Create rke/rke2 cluster, which uses canal as CNI
Install Longhorn
Apply network policies in https://github.com/longhorn/longhorn/pull/5785
Run test cases in test_backing_image.py, there's no abnormal failed test case

authors: shuo-wu yangchiu


text:
[BUG][1.4.2] RWO volumes don't seem to mount with statefulsets

Describe the bug (🐛 if you encounter this issue)
RWO volumes are often failing to mount, despite longhorn ui and manager saying it's attached.
To Reproduce
Steps to reproduce the behavior:

Deploy https://github.com/samip5/k8s-cluster/blob/main/cluster/apps/search/elastic/resources/es.yaml
See that one of the elasticsearch replicas will not come up in about 50% of times
See that longhorn-manager on the node in question seems to think it's attached
Describe the pod and see error

Expected behavior
I would have expected it to not have any issues mounting it.
Log or Support bundle
Pastebin is at https://termbin.com/r4hc (not sure how long it lasts there)
Environment

Longhorn version: 1.4.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s v1.27.1

Number of management node in the cluster: 1
Number of worker node in the cluster: 5

Node config

OS type and version: Ubuntu Server 22.04  & Debian 11
CPU per node: Varies, some are Raspberry Pi 4's and some are thin pc's
Memory per node: Varies per node, 8-32GB
Disk type(e.g. SSD/NVMe): SSD
Network bandwidth between the nodes: 1G link

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 12

Additional context
Events:
 Type     Reason       Age                  From               Message
 ----     ------       ----                 ----               -------
 Normal   Scheduled    16m                  default-scheduler  Successfully assigned search/main-es-default-0 to m1
 Warning  FailedMount  3m53s (x3 over 11m)  kubelet            MountVolume.MountDevice failed for volume "pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
 Warning  FailedMount  47s (x7 over 14m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[elasticsearch-data], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition

Hello @samip5. Is it possible to provide a full support bundle for this issue? The longhorn-manager log you provided gives some context, but we are missing information from the replica controllers and instance managers on other nodes that may be helpful.
We are seeing a lot of logs like this:
time="2023-05-20T02:22:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"storage\", Name:\"m1\", UID:\"2ac23d23-170a-44fb-b2ce-1bcbf058bdc4\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"173046002\", FieldPath:\"\"}): type: 'Warning' reason: 'UnknownNodeConditionTrue' Unknown condition true of kubernetes node m1: condition type is ReadonlyFilesystem, reason is FilesystemIsReadOnly, message is EXT4-fs (sdg): Remounting filesystem read-only"
time="2023-05-20T02:22:44Z" level=info msg="Event(v1.ObjectReference{Kind:\"Node\", Namespace:\"storage\", Name:\"w-amd-3\", UID:\"0aa398c5-1b13-44e7-9dd8-44f65dd572fa\", APIVersion:\"longhorn.io/v1beta2\", ResourceVersion:\"173046007\", FieldPath:\"\"}): type: 'Warning' reason: 'UnknownNodeConditionTrue' Unknown condition true of kubernetes node w-amd-3: condition type is ReadonlyFilesystem, reason is FilesystemIsReadOnly, message is EXT4-fs (sdc): Remounting filesystem read-only"

Longhorn-manager is generating these events because it is seeing a node condition it doesn't recognize and broadcasting it as an event. We don't have any logic to handle this particular condition (as I think it is not one used by base-Kubernetes), but it may be providing us with the clue we need.
I think it is possible that sdg on m1 and sdc on w-amd-3 are corrupted. If we are trying to use these disks to run replicas (I'm not sure if we are with the logs available), it's not surprising that we are running into errors. If it is practical, you might try unmounting these disks and running fsck on them to confirm.

Is it possible to provide a full support bundle for this issue?

The support bundle generation seems to be failing, and I don't know how to troubleshoot that portion.

I think it is possible that sdg on m1 and sdc on w-amd-3 are corrupted.

Both of these seem to be virtual, aka created by longhorn (the sdg doesn't exist at the moment on m1)

cc @c3y1huang can we check the log of the terminated support bundle agent pod?

@samip5, could we have the log of the failing bundle agent pod? If there are issues with the support bundle functionality, we definitely want to know what they are!
Also, since we currently can't get a full support bundle, could you provide the output of:

kubectl get node -o yaml (I would like to see the node conditions Longhorn thinks it knows about)
kubectl logs -n longhorn-system <longhorn-csi-plugin-pod-name> for all plugin pods after a failure

@samip5, could we have the log of the failing bundle agent pod? If there are issues with the support bundle functionality, we definitely want to know what they are!

Here's the failed pod's log, and it's restart counter is at 88: https://termbin.com/wao5 (it doesn't appear to get all of the files added to the bundle and I cannot see why it gets to exit 2)

Also, since we currently can't get a full support bundle, could you provide the output of:

kubectl get node -o yaml (I would like to see the node conditions Longhorn thinks it knows about)

Pastebin of that is at: https://termbin.com/7tet5

kubectl logs -n longhorn-system <longhorn-csi-plugin-pod-name> for all plugin pods after a failure

Pastebin that contains all of the logs from that: https://termbin.com/u6h2w

Using --since=48h of the same logs: https://termbin.com/scy6y

Oh, I was just able to see the issue with crashing pod.
k logs -n storage supportbundle-agent-support-bundle-2023-05-23t06-16-18z-65jxq -f
+ exec tini -- /usr/bin/support-bundle-collector.sh
+ HOST_PATH=/host
+ OUTPUT_DIR=/tmp/support-bundle
+ '[' -z http://10.244.4.60:8080 ']'
+ '[' '!' -e /tmp/support-bundle ']'
+ mkdir -p /tmp/support-bundle
+ NODE_NAME=w1
+ BUNDLE_DIR=/tmp/support-bundle/w1
+ mkdir -p /tmp/support-bundle/w1
+ '[' -n longhorn ']'
+ OS_COLLECTOR=collector-longhorn
+ echo OS_COLLECTOR=collector-longhorn
OS_COLLECTOR=collector-longhorn
++ which collector-longhorn
+ '[' -x /usr/bin/collector-longhorn ']'
+ collector-longhorn /host /tmp/support-bundle/w1
+ cd /tmp/support-bundle
++ basename /tmp/support-bundle/w1
+ zip -r node_bundle.zip w1
  adding: w1/ (stored 0%)
  adding: w1/logs/ (stored 0%)
  adding: w1/logs/dmesg.log (deflated 83%)
  adding: w1/logs/kubelet.log (deflated 86%)
  adding: w1/logs/syslog (deflated 85%)
  adding: w1/logs/syslog.2.gz (deflated 13%)
  adding: w1/logs/syslog.1 (deflated 85%)
  adding: w1/hostinfos/ (stored 0%)
  adding: w1/hostinfos/hostinfo (deflated 38%)
  adding: w1/hostinfos/kernel_config (deflated 76%)
+ rm -rf bundle
+ set -o errexit
+ curl -v -i -H 'Content-Type: application/zip' --data-binary @node_bundle.zip http://10.244.4.60:8080/nodes/w1
curl: option --data-binary: out of memory
curl: try 'curl --help' or 'curl --manual' for more information

Oh, I was just able to see the issue with crashing pod. @ejweber

Ah, I see. I opened https://github.com/rancher/support-bundle-kit/issues/72 with a suggestion that may help mitigate this issue, though it it likely quite rare that we don't have enough memory to complete this cURL call.
There should also be a support-bundle-manager pod that spins up when you create a support bundle. Could you also provide the logs of that one? It seems to me that node collection should be "best-effort" (i.e. one failed node collection should not prevent us from getting the rest of the bundle).

It looks like the ReadonlyFilesystem condition Longhorn is picking up is coming from the node-problem-detector you have deployed in your cluster. I am not aware of that installing by default when deploying K3s, etc. Is it something you deployed manually?
I don't think it is causing any problems. (If anything, it may be giving us an extra clue.) I'm just curious to know a bit more about it, since I haven't seen Longhorn logging anything from it before.

From the CSI plugin logs you provided, and assuming pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 is still the one we are interested in:
We appear to be receiving multiple NodeStageVolume commands from kubelet and not responding to them. At 2023-05-23T04:39:31Z we receive the first request.
time="2023-05-23T04:39:31Z" level=info msg="NodeStageVolume: req: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"xfs\"}},\"access_mode\":{\"mode\":1}},\"volume_context\":{\"dataLocality\":\"disabled\",\"fromBackup\":\"\",\"fsType\":\"xfs\",\"numberOfReplicas\":\"2\",\"staleReplicaTimeout\":\"30\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1684546196523-8081-driver.longhorn.io\"},\"volume_id\":\"pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88\"}"
time="2023-05-23T04:39:31Z" level=info msg="volume pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 using user and longhorn provided xfs fs creation params: -ssize=4096 -bsize=4096"
time="2023-05-23T04:39:31Z" level=debug msg="volume pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 device /dev/longhorn/pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 contains filesystem of format "
time="2023-05-23T04:39:31Z" level=debug msg="trying to ensure mount point /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount"
time="2023-05-23T04:39:31Z" level=debug msg="mount point /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount try reading dir to make sure it's healthy"
I0523 04:39:31.754828 2682285 mount_linux.go:449] Disk "/dev/longhorn/pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88" appears to be unformatted, attempting to format as type: "xfs" with options: [-f /dev/longhorn/pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88]
I0523 04:39:34.257665 2682285 mount_linux.go:459] Disk successfully formatted (mkfs): xfs - /dev/longhorn/pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount

Normally we would log a couple of additional things after this and respond, but we don't.
We continue to receive NodeStageVolume requests that make it to the same basic point and never respond.
time="2023-05-23T04:41:31Z" level=info msg="NodeStageVolume: req: {\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"xfs\"}},\"access_mode\":{\"mode\":1}},\"volume_context\":{\"dataLocality\":\"disabled\",\"fromBackup\":\"\",\"fsType\":\"xfs\",\"numberOfReplicas\":\"2\",\"staleReplicaTimeout\":\"30\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1684546196523-8081-driver.longhorn.io\"},\"volume_id\":\"pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88\"}"
time="2023-05-23T04:41:31Z" level=info msg="volume pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 using user and longhorn provided xfs fs creation params: -ssize=4096 -bsize=4096"
time="2023-05-23T04:41:31Z" level=debug msg="volume pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 device /dev/longhorn/pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 contains filesystem of format xfs"
time="2023-05-23T04:41:31Z" level=debug msg="trying to ensure mount point /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount"
time="2023-05-23T04:41:31Z" level=debug msg="mount point /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/0d468721ad6d8561b3b2d03463fb6e12119f9befb16aee78af9e2a7791a721b3/globalmount try reading dir to make sure it's healthy"
time="2023-05-23T04:41:31Z" level=debug msg="found existing healthy mount point for volume pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 skipping mount"

Given what we are and aren't logging, it seems like we are getting stuck here: https://github.com/longhorn/longhorn-manager/blob/6856fd35ee1739f517b78be9e1c6df38171be458/csi/node_server.go#L444-L464. I will attempt to understand how that's possible.

Can this volume be mounted to other nodes?
Can this volume be manually attached to w2 and mounted using mount /dev/longhorn/pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 <some_path>?

This may be similar to https://github.com/longhorn/longhorn/issues/5296. Can you also check:

If the memory of the csi-plugin pod on w2 is skyrocketing?
And if there are many processes running in it by execing into it and running ps aux?

Is it something you deployed manually?

Yes, can be seen here: https://github.com/samip5/k8s-cluster/blob/main/cluster/apps/monitoring/node-problem-detector/helm-release.yaml

There should also be a support-bundle-manager pod that spins up when you create a support bundle. Could you also provide the logs of that one? It seems to me that node collection should be "best-effort" (i.e. one failed node collection should not prevent us from getting the rest of the bundle).

Pastebin of the support bundle manager: https://bpa.st/MIBMS

From the CSI plugin logs you provided, and assuming pvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 is still the one we are interested in.

It seems to currently be pvc-aee58912-e374-41eb-a289-624c798df8b7 for the elk one (age of which is 44h).

Can this volume be manually attached to w2 and mounted using mount /dev/longhorn/pvc-aee58912-e374-41eb-a289-624c798df8b7 <some_path>

Yes, it can.

If the memory of the csi-plugin pod on w2 is skyrocketing?

It doesn't appear to be, but then again I don't have good monitoring for memory usage.

And if there are many processes running in it by execing into it and running ps aux?

Pastebin of ps aux on w2: https://bpa.st/72WPQ

Ah, I see. I opened rancher/support-bundle-kit#72 with a suggestion that may help mitigate this issue, though it is likely quite rare that we don't have enough memory to complete this cURL call.

Oh, hey @ejweber you might want to see this, I checked the file sizes in /var/log on the w1 node and yeah, I can see why it's having issues. This is probably not really thought about if the file sizes are this large.
https://termbin.com/xa6d

There should also be a support-bundle-manager pod that spins up when you create a support bundle. Could you also provide the logs of that one? It seems to me that node collection should be "best-effort" (i.e. one failed node collection should not prevent us from getting the rest of the bundle).

Pastebin of the support bundle manager: https://bpa.st/MIBMS

Thanks @samip5. I submitted https://github.com/rancher/support-bundle-kit/issues/73 to address this.

The pvc with idpvc-6cd2b452-534d-4f7a-9020-7f59a7eaea88 doesn't exist anymore, as I had to re-create the pvc for it to attach it. Currently the elk pvc's are mounted and working but others are failing to mount.
Can I do something manually to the longhorn support bundle thing for it to actually work?
Where should I be executing the ps aux command? In the csi pod, or on the node itself?

Can I do something manually to the longhorn support bundle thing for it to actually work?

Nothing comes to mind, unfortunately. However, I think we've narrowed down to the particular logs we're interested in:

csi-plugin logs (which you have already provided)
ps aux output including the full executed command (this can either be from the csi-plugin or the node, as the information should be the same in this case)
dmesg or journal logs from the node (since it looks like some system call might be repeatedly failing)

ps aux output including the full executed command (this can either be from the csi-plugin or the node, as the information should be the same in this case)
dmesg or journal logs from the node (since it looks like some system call might be repeatedly failing)

m1's ps -aux: https://termbin.com/dfchr & journalctl since --today:  https://termbin.com/q4en

Running into this same issue, but I was able to generate the support bundle. Issue comes up most often with helm upgrades to update to new versions. This is also happening with multiple other apps, but just looking at heimdall to help diagnose in this case.
Statefulset in question here is heimdall with pod heimdall-0. PVC is pvc-ade5e7dc-5914-4726-b236-f1c271cbd0c1.
Support Bundle - https://drive.google.com/file/d/17mMOlSJkrZ3QBgp_l-GIlupIK-_I0gpn/view?usp=sharing

I also think that longhorn might be leaking fsnotifies as my m1 has run out of file descriptors.
inotifyd: no kernel support: No file descriptors available

Any update on this? Also a little bit more info, had redeployed my cluster with Longhorn 1.4.1 to test and this issue still exists, so looks like it may be something caused by the updates to K3S and Longhorn isn't communicating properly. I'm currently on v1.27.2+k3s1.

Oh, and I'm also on that same version of k3s.

It seems that this issue is definitely related to the k3s version, as after downgrading from v1.27.2+k3s1 to v1.26.3+k3s1 it appears to work much much better.

Hi Longhorn maintainers, any ideas about what to do about this? How can I help debug it? It's still happening on v1.26.3+k3s but less frequently.
Warning  FailedMount  4m13s (x738 over 2d1h)  kubelet  MountVolume.MountDevice failed for volume "pvc-584d06c6-9003-4e61-83bb-82952a562298" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

This is still happening on 1.5.0.

Sorry @samip5, this one fell off my radar. Can you provide the links in https://github.com/longhorn/longhorn/issues/5985#issuecomment-1574241106 again? I think they expired.

Sorry, but I don't have them anymore nor longhorn in that cluster due to stability issues and leaking of file handlers.


text:
[BUG] backingimage download server error

Describe the bug (🐛 if you encounter this issue)
When creating a new backing image for a volume via the web UI, the UI shows a spinner for a few seconds and then updates the list with the new backing image. There is nothing indicating the status of the backing image and it would be reasonable to assume the backing image is ready. However this may not necessarily be the case, and immediately clicking on the download button results in a server error (status code 500, see screenshot).
To Reproduce
Create a backing image by export from a longhorn volume of a few GB in size using raw format and immediately initiate download action.
Expected behavior

The backing image list should show a status of the respective backing images, indicating whether or not it's ready for download
The download action should be inactive while the backing image is not ready for download yet
It should be impossible to cause a server error while using the UI

Additional notes on UX:

When (successfully) initiating the download of a backing image, the file name of the downloaded file is the uuid of the backing image. However it would probably be more user friendly to set this to the name of the backing image to make the identification later on easier.
The list of backing images shows the creation method. In case of export from a volume, it would be user friendly to also show the name of the volume here. Right now this information is only available in the detailed view
The search/filter bar should also be able to search/filter by uuid and other properties, not just the name of the backing image
The detailed view shows the status of an individual backing image, but this has no coloration
The cleanup dialogue is confusing and uses incorrect English, it's unclear why it exists since the only operation (delete) is also directly supported
The backing image files seem to be uncompressed. This can lead to potentially long download/upload times. It would be nice to use compression at least while downloading.
Bulk downloads of multiple images are not supported in the UI, this can be tedious when dealing with many images/volumes.
The backing images view should have an action to directly create a volume from a backing image

Support bundle for troubleshooting

Environment

Longhorn version: 1.6-dev (docker.io/longhornio/longhorn-manager@sha256:16a45ba592814751061922ec3b45e5e6a2789aa73e1ededb00d96f5b6b8c4e8e)
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: Vanilla K8s (kubeadm):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.17", GitCommit:"22a9682c8fe855c321be75c5faacde343f909b04", GitTreeState:"clean", BuildDate:"2023-08-23T23:37:25Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}

Number of management node in the cluster: 1
Number of worker node in the cluster: 3
Node config

OS type and version: SLE 15 SP4
Kernel version: Linux kratos 5.14.21-150400.24.63-default #1 SMP PREEMPT_DYNAMIC Tue May 2 15:49:04 UTC 2023 (fd0cc4f) x86_64 x86_64 x86_64 GNU/Linux
CPU per node: 16
Memory per node: 64GiB
Disk type(e.g. SSD/NVMe/HDD): NVMe
Network bandwidth between the nodes: 1000Mb/s (1000baseT/Full)

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 1
Impacted Longhorn resources: backingimages

Additional context

cc @ChanYiLin

Hi @m-ildefons thanks for your feedbacks
Those are really helpful for use to improve the user experience!
Some ideas I think we can definitely create issues to improve them

BackingImage download server error:

You are right, we can disable download button when the BackingImage is not ready. And add status to the BackingImage list
Internal Server Error means Longhorn doesn't expect the operation. By applying above ideas I think we could prevent triggering Internal Server Error in this case.

BackingImage file name: Yes we should use BackingImage instead of UUID
BackingImage exported from Volume should show the volume name: agree
Filter: agree
Should apply compression method when download BackingImage: agree
Bulk download: agree
Create volumes from a BackingImage page: agree

For the above ideas, I will create UI/Backend tickets for them.
There are also some topics I would like to further discuss with you and get more information.

The detailed view shows the status of an individual backing image, but this has no coloration
The cleanup dialogue is confusing and uses incorrect English, it's unclear why it exists since the only operation (delete) is also directly supported

BackingImage is not just stored in one place in Longhorn, in facts, it will be copied and stored on different disks of the nodes if the replicas on that nodes/disks need the BackingImage.
That's why we need to list the status of the BackingImage on the different disk.
Cleanup operation then allows users to delete the BackingImage on specific disks to save some space if they know there won't be any replicas on that disk requiring BackingImage. Users can't delete the last BackingImage in this operation or there won't have BackingImage file in the system.
Delete operations means Longhorn will delete the BackingImage on all disks and also delete the BackingImage CR.

I have created a ticket to improve the user experience.
We can focus on disabling download when BackingImage is not ready in this issue.
I think from the UI side, it can check the diskFileStatusMap in the json.
If there is no BackingImage is ready, we can disable the download.

The detailed view shows the status of an individual backing image, but this has no coloration

What I mean by that is that the detailed view may show ready or in progress, but it's all in the same color:

I think we should use colors for the status text just like in other places of the UI. E.g. the nodes display their status with green when it's good or grey when they are unschedulable for some reason:

BackingImage is not just stored in one place in Longhorn, in facts, it will be copied and stored on different disks of the nodes if the replicas on that nodes/disks need the BackingImage.
That's why we need to list the status of the BackingImage on the different disk.
Cleanup operation then allows users to delete the BackingImage on specific disks to save some space if they know there won't be any replicas on that disk requiring BackingImage. Users can't delete the last BackingImage in this operation or there won't have BackingImage file in the system.
Delete operations means Longhorn will delete the BackingImage on all disks and also delete the BackingImage CR.

Thanks for explaining this. I still find it a bit convoluted from a UX perspective, but I understand why it is this way.
IMO, the user experience could be improved by merging the functionality of the cleanup dialogue in the detailed view and getting rid of the cleanup dialogue entirely. Also showing which node a copy of the backing image is stored on in the detailed view would be more helpful than showing the uuid of the disk.

I think from the UI side, it can check the diskFileStatusMap in the json.
If there is no BackingImage is ready, we can disable the download.

This sounds like a good approach to hide the error.

Hi @m-ildefons thanks for the details,
I have updated the feature requests in issue #7293

I have created a ticket to improve the user experience. We can focus on disabling download when BackingImage is not ready in this issue.
I think from the UI side, it can check the diskFileStatusMap in the json. If there is no BackingImage is ready, we can disable the download.

Hi @innobead
This issue can also be fixed by from UI side

Note
For this issue we can improve from the UI side
We can check the diskFileStatusMap in the json returned from the BackingImage API.
If there is no BackingImage is ready on any disk, we should disable the download button.

Note
For this issue we can improve from the UI side We can check the diskFileStatusMap in the json returned from the BackingImage API. If there is no BackingImage is ready on any disk, we should disable the download button.
cc @scures

@roger-ryao This is the info for testing.

Verified on master-head 20231228

longhorn master-head https://github.com/longhorn/longhorn/commit/07bdb419a968633059c24db35ec66d7d68d73f4c
longhorn-ui master-head https://github.com/longhorn/longhorn-ui/commit/d916e44fe449e00a68cc0fb829c728a1ec5da97a

The test steps
https://github.com/longhorn/longhorn/issues/7288#issuecomment-1855262472
Result Passed
When the state in the diskFileStatusMap dictionary is not ready, the Download icon is grayed out. After the state becomes ready, we can download the backing image from the UI.

Hi @scures :
Please help backport https://github.com/longhorn/longhorn-ui/pull/670 to v1.5.x and v1.4.x. After that, I can assist in verifying issues https://github.com/longhorn/longhorn/issues/7381 and https://github.com/longhorn/longhorn/issues/7382.
Thank you!

authors: ChanYiLin innobead m-ildefons roger-ryao


text:
[BUG] Delete error backup could cause v2 volume stuck in detaching/faulted state

Describe the bug
Delete replicas of a v2 volume during backup creation could cause the backup becomes Error state:
  error: 'proxyServer=10.42.2.36:8501 destination=10.42.1.31:20013: failed to get
    backup-ae5264e951594c20 backup status: rpc error: code = Internal desc = failed
    to get backup status: rpc error: code = NotFound desc = replica address 10.42.3.34:20007
    is not found in engine test-4-e-0 for getting backup backup-ae5264e951594c20 status'

And remove the error backup could cause the volume stuck in detaching/faulted state:

To Reproduce

Create v2 volume environment
Create a v2 volume test-1 with 3 replicas from UI and also create PV/PVC for it from UI
Create a pod for it:

cat << EOF > pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: sleep
      image: busybox
      imagePullPolicy: IfNotPresent
      args: ["/bin/sh", "-c", "while true;do date;sleep 5; done"]
      volumeMounts:
        - name: pod-data
          mountPath: /data
  volumes:
    - name: pod-data
      persistentVolumeClaim:
        claimName: test-1
EOF
kubectl apply -f pod.yaml

Write some data to the volume:

dd if=/dev/urandom of=/data/test-1 bs=3M count=1024

Create backup from UI, and during the backup in progress, delete some replicas to make the backup be in Error state:

$ kubectl get backups -n longhorn-system backup-ae5264e951594c20 -oyaml
apiVersion: longhorn.io/v1beta2
kind: Backup
metadata:
  creationTimestamp: "2024-01-08T03:32:35Z"
  finalizers:
  - longhorn.io
  generation: 1
  labels:
    backup-volume: test-4
  name: backup-ae5264e951594c20
  namespace: longhorn-system
  resourceVersion: "26161"
  uid: f81b9dd2-7d32-46ec-b982-a3905599ecb7
spec:
  labels:
    KubernetesStatus: '{"pvName":"test-4","pvStatus":"Bound","namespace":"default","pvcName":"test-4","lastPVCRefAt":"","workloadsStatus":[{"podName":"test-pod-4","podStatus":"Running","workloadName":"","workloadType":""}],"lastPodRefAt":""}'
    longhorn.io/volume-access-mode: rwo
  snapshotName: fdc06fc9-89b7-4bb5-a958-032f51d75a2c
  syncRequestedAt: null
status:
  backupCreatedAt: ""
  compressionMethod: ""
  error: 'proxyServer=10.42.2.36:8501 destination=10.42.1.31:20013: failed to get
    backup-ae5264e951594c20 backup status: rpc error: code = Internal desc = failed
    to get backup status: rpc error: code = NotFound desc = replica address 10.42.3.34:20007
    is not found in engine test-4-e-0 for getting backup backup-ae5264e951594c20 status'
  labels: null
  lastSyncedAt: "2024-01-08T03:32:54Z"
  messages: null
  ownerID: ip-10-0-1-238
  progress: 10
  replicaAddress: ""
  size: ""
  snapshotCreatedAt: "2024-01-08T03:32:35Z"
  snapshotName: fdc06fc9-89b7-4bb5-a958-032f51d75a2c
  state: Error
  url: ""
  volumeBackingImageName: ""
  volumeCreated: ""
  volumeName: ""
  volumeSize: "21474836480"

Detach the volume to trigger offline rebuilding:

kubectl delete -f pod.yaml

After offline rebuilding completed, re-attach the volume:

kubectl apply -f pod.yaml

The volume is attached and healthy without problems, and once delete the error backup from UI, the volume gets stuck in detaching and faulted:

Please see volume test-4 related logs in the support bundle for more details.
Expected behavior

Support bundle for troubleshooting
supportbundle_43c90f10-cff2-486a-bf7b-dc91761bf1ea_2024-01-08T04-09-06Z.zip
Environment

Longhorn version: master-head (longhorn-manager a601b9b)
Impacted volume (PV):
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: v1.27.1+k3s1

Number of control plane nodes in the cluster: 1
Number of worker nodes in the cluster: 3

Node config

OS type and version: ubuntu 22.04
Kernel version:
CPU per node:
Memory per node:
Disk type (e.g. SSD/NVMe/HDD):
Network bandwidth between the nodes (Gbps):

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
Number of Longhorn volumes in the cluster:

Additional context
<!-Please add any other context about the problem here.-->

The error is triggered by the actual size mismatching error of replica lvols. Checking the replica verification logics.
[longhorn-manager-dqzsx] time="2024-01-08T07:27:01Z" level=warning msg="Instance test-1-r-3fe7d78f is state error, error message: found mismatching lvol actual size 2750414848 with recorded prev lvol actual size 2097152 when validating lvol test-1-r-3fe7d78f-snap-rebuild-6d8f7e1c" func="controller.(*InstanceHandler).syncStatusWithInstanceManager" file="instance_handler.go:206

This is a transition error occuring in codes.  When deleting a snapshot lvol, the merge of lvols results in a change of actual size.
I'm thinking if we really need the actual size validation.
Can we remove the check and revisit it later if it is needed? WDYT? @shuo-wu @innobead

This is a transition error occuring in codes. When deleting a snapshot lvol, the merge of lvols results in a change of actual size.
I'm thinking if we really need the actual size validation. Can we remove the check and revisit it later if it is needed? WDYT? @shuo-wu @innobead

Sounds good to me.

Verified on master-head 20240109

longhorn master-head https://github.com/longhorn/longhorn/commit/96a0995708b4479bd29a55ee74fd36090dad9962
longhorn-spdk-engine main https://github.com/longhorn/longhorn-spdk-engine/commit/71e569aca8efcf8eb6e67dcdfb3ee0fc3afe1058

The test steps

https://github.com/longhorn/longhorn/issues/7575#issue-2069616052

Result passed

authors: derekbit innobead roger-ryao yangchiu


text:
[IMPROVEMENT] V2 volume snapshot supports labels

Is your improvement request related to a feature? Please describe (👍 if you like this request)
V2 volume snapshot should support labels
Describe the solution you'd like
The labels can be stored in the xattr of lovl.
Describe alternatives you've considered

Additional context

cc @DamiaSan @innobead

@DamiaSan Would you like to try this fix?
https://github.com/longhorn/longhorn-instance-manager/blob/master/pkg/proxy/snapshot.go#L68-L71?

Which milestone should we develop this feature for?

Which milestone should we develop this feature for?

It is an easy fix, so let's try to fix it in milestone v1.8.0.
Feel free to let me know if you need help. OK?

Don´t know how much it is easy, because as you can see here, actually we don't permit general labels but only predefined ones like UserCreated and Timestamp.

Is this for storing metadata for each snapshot? I thought we already had that. What use cases do we need this?

Is this for storing metadata for each snapshot? I thought we already had that. What use cases do we need this?

@innobead
v1 supports labels for snapshots, but v2 doesn't.
Can we make it best effort (P/1) in v1.8.0?

Is this for storing metadata for each snapshot? I thought we already had that. What use cases do we need this?

@innobead v1 supports labels for snapshots, but v2 doesn't. Can we do it best effort (P/1) in v1.8.0?

Sounds good.

authors: DamiaSan derekbit innobead


text:
Type SimpleSchema changed optional behavior

This changed somewhere between version 3.2.0 and the current version.
import SimpleSchema from 'simpl-schema'

export const Poi = new SimpleSchema({
  name: String,
  gPlaceId: { type: String, optional: true }
})

const PostSchema = new SimpleSchema({
  userId: String,
  location: { type: Poi, optional: true },
  // ....
})

const myInsertPostSchemaValidationContext = PostSchema.newContext()

console.log(myInsertPostSchemaValidationContext.validate({ userId: '1234' }))

When validating a PostSchema object without a location, I get the error "name" is required (the name of the Poi). It looks like although Poi is optional, all non-optional keys of Poi are required which of course doesn't make sense.

I think this is a dup of #475 (I am in the same boat looking for a solution)

Fixed in 3.4.6

authors: aldeed herchu paulincai

repo: longshotlabs/simpl-schema


text:
Spurious unknown Dialect Error.

Doesn't run, stack trace below.  Error message
Unknown Dialect undefined dialect
        query: q is table('malloytest.aircraft')->{
          group_by: state
        }

        explore: r is from(->q){
          query: foo is {
            group_by: state
          }
        }

        query: r->foo

stack trace, error text?

      at Object.getDialect (../malloy/src/dialect/dialect_map.ts:23:11)
      at new QueryStruct (../malloy/src/model/malloy_query.ts:2608:20)
      at Function.nextStructDef (../malloy/src/model/malloy_query.ts:1125:16)
      at Object.outputSpace (../malloy/src/lang/ast/ast-main.ts:1172:29)
      at PipelineDesc.appendOps (../malloy/src/lang/ast/ast-main.ts:1372:21)
      at PipelineDesc.getPipelineForExplore (../malloy/src/lang/ast/ast-main.ts:

Fixed

authors: lloydtabb mtoy-googly-moogly

repo: looker-open-source/malloy


text:
fixes #19

pinning down diplomat version since they are breaking contract
while major version is still 0

Please correct this commit message by describing what was changed before merging.

authors: blaines tommysullivan

repo: lookout/uat


text:
Updated Cryptoodles project description

Cryptoodles Loot Bag
Etherscan link: https://etherscan.io/address/0x2a16107ea0d28ae822168c8bb3ef5460b1924a78#code
Website: https://cryptoodles.com
Twitter: https://twitter.com/cryptoodles
Discord: https://discord.com/invite/g9SnvzdFs2
OpenSea: https://opensea.io/collection/toodle-loot-bag

Looks good!

authors: cryptoodles ravindranrahul

repo: lootproject/website


text:
improve rc and translate functions

[ ] Let both functions use more flexible input (e.g. case-insensitive)
[ ] Better handle ambiguous nts

Let's be real--you should just use the Biopython versions

repo: louiejtaylor/pyviko


text:
No alarm in ping monitoring

📑 I have found these related issues/pull requests
na
🛡 Security Policy

[X] I agree to have read this project Security Policy

Description
I use to ping of  monitoring url  , It will not alarm when a fault occurs, but will remind you when the fault is restored.
👟 Reproduction steps
webhook can not remind
👀 Expected behavior
There should be reminders when failures occur and when recovery fails
😓 Actual Behavior
can not remind when failures ,  Only recovered alarms
🐻 Uptime-Kuma Version
1.23.15
💻 Operating System and Arch
ubuntu 22.04
🌐 Browser
google
🖥 Deployment Environment

Runtime:
Database:
Filesystem used to store the database on:
number of monitors:
Docker

📝 Relevant log output
na

What is your notification template?

ping template

webhook

I meant: What did you put into this field

PST url  is  zoom webhook addresses;    request boday is  custom request  ; I see  message on dasboard , but webhook can't push this is message, only is normal webhook push message

erro message can't push  zoom webhook , Only restore information push on zoom

What did you put into this field

Uploading 1111.png…

Your image did not upload properly, could you try again?

There is a bug in how newlines are handled upstream:
JSON does not allow real line-breaks.
You need to replace all the line breaks with \n or split at the newlines separator and handle this in the target system.
Doing the replacement option in the notification provider would likely be a good idea (PR welcome).

"{{ msg | replace: "\n", "" }}"
THX !  I have fixed this issues

repo: louislam/uptime-kuma


text:
Multi-fire pop-over

Greetings,
I recently published a post where when one in-text citation is hovered over, the others also automatically show up... I'm pulling my hair out (what little is left ;-) ) trying to debug this.
Here is the post in question: https://hughandbecky.us/Hugh-CV/post/olac-subject-terms/
Here is another post rendered at the same time where the issue does not happen: https://hughandbecky.us/Hugh-CV/post/linguistic-literature-growth-rate/
I can make the JSON and the markdown available on request. But I'm wondering if anyone has seen this sort of error before and knows what it might be... Also not the change in font size in the inline citation... that is just happening. So something is being triggered and I'm not sure what it is (I looked at the JSON, ran it through a validator and it looks fine). Any pointers are appreciated.

The part that I think is challenging is that it is only happening on that specific page.

Hi, apologies for the delay, I haven’t had much time lately. Can you post a sample of the markdown file that produces this part? It seems like the subsequent citations (Svenomnius, Iwe, Anderson) are nested inside the first one (Daily).

authors: HughP loup-brun

repo: loup-brun/hugo-cite

license: WTFPL


text:
core_busy_o not protected by lockstep

Observed Behavior
The core_busy_o output signal of the non-lockstep core is used to gate the core clock, which is used by the regular Ibex core and its lockstep counterpart.
https://github.com/lowRISC/ibex/blob/3943a4eca3917d0a00406abb36f4951304c0b394/rtl/ibex_top.sv#L211-L219
https://github.com/lowRISC/ibex/blob/3943a4eca3917d0a00406abb36f4951304c0b394/rtl/ibex_top.sv#L286-L287
https://github.com/lowRISC/ibex/blob/3943a4eca3917d0a00406abb36f4951304c0b394/rtl/ibex_top.sv#L927-L928
This means that the lockstep core gets gated in the same clock cycle as the regular core, even though the lockstep core is a few cycles delayed. A low-glitch in the core_busy_o output of the non-lockstep core thus does not lead to a comparison mismatch and a major alert because the lockstep core is already clock-gated when it would produce the value of core_busy_o that is used in the comparison.
https://github.com/lowRISC/ibex/blob/3943a4eca3917d0a00406abb36f4951304c0b394/rtl/ibex_lockstep.sv#L453-L454

Expected Behavior
A low-glitch/mismatch in the core_busy_o output signal of the non-lockstep core should lead to a major alert by the lockstep core.
Steps to reproduce the issue
Run the chip_sw_ibex_lockstep_glitch test in OpenTitan (currently in https://github.com/lowRISC/opentitan/pull/15221) and override port_idx to the index of core_busy_o in output_ports.

Thanks @andreaskurth for creating this issue. I think this is a good catch!
I've now had a look at the RTL. If the clock is turned off, also the delay registers have no clock anymore. This means the outputs of the last couple of cycles of the core will be compared with the outputs of the lockstep/shadow core after resuming. Functionality-wise this seem okay to me.
From a security perspective, this could however be problematic. As we just discussed in person, by glitching core_busy_o of the main core to low, Ibex could be made to sleep instead of handling e.g. an alert or similar. Eventually, the watchdog timer will intervene and reset the whole chip but Andreas and I think it would be good to just protect the core_busy_o signal using a multi-bit encoding (similar to fetch_enable_i). Does this sound reasonable @GregAC ?
An alternative would be to only disable the clock once also the lockstep/shadow core outputs core_busy_o low. But this would probably require invasive changes to the core controller currently expects the clock to be switched off one clock cycle after setting busy low. I think the pipeline can not be made to not fetch instructions when the clock is on. Maybe, one could use the fetch_enable_i mechanism for the purpose but I am not sure we want to do that.

Eventually, the watchdog timer will intervene and reset the whole chip but Andreas and I think it would be good to just protect the core_busy_o signal using a multi-bit encoding (similar to fetch_enable_i). Does this sound reasonable @GregAC ?

I think so.
As you point out actually delaying the clock enable for the lockstep has it's own issues so hardening the signal instead makes sense. I think there's also limited scope for such a glitch attack on core_busy_o to achieve much as whilst you may be able to skip an alert you'll need to let the CPU start up again at some point at which point there will be an output mismatch (as the lockstep core won't produce a matching core_busy_o drop). So this helps justify a different approach here for the core_busy_o output.

@msfschaffner @tjaychen @cdgori any thoughts?

I think multi-bit hardening of core_busy_o is reasonable here, because as you say, when the CPU ultimately starts up again there will be an output mismatch.
Can anyone brainstorm a way for an adversary to suppress that mismatch?  I'm drawing a blank.  That would be the only reason to think about the more invasive change to combine the two core_busy_o signals to form the clock enable.

Thanks for the feedback @cdgori , I couldn't come up with a scenario for someone to suppress the mismatch.
I will have a look at doing the RTL modification for this.

The MUBI solution SGTM too!

authors: GregAC andreaskurth cdgori msfschaffner vogelpi

repo: lowRISC/ibex


text:
[bazel] Document preferred method to reconstruct failing tests

Failing build commands will echo their invocation and may be debugged using --debug_sandbox and --verbose_failures and then repeating and editing the command and environment, which isn't the most user friendly thing, but beats the alternative. Tests don't appear to have this feature and are wrapped by scripts making it hard to invoke the test environment without prior knowledge. Invocations of test tools like the verilator test bench could echo the invocation to test output before performing it and make it clear how to repeat the test while also passing args to verilator. Proper behavior would be broadly applicable but verilator gives us a space to explore best practices in the open.

@cfrantz has pointed out:
the log contains filenames and relative paths in an array
and suggests:
We change the statement to join them and use their realpaths which should work but wont be relative to the $repo_top

repo: lowRISC/opentitan


text:
[rv_dm,rtl] Fix acknowledgment of reset in dmstatus.allhavereset/dmstatus.anyhavereset

https://github.com/pulp-platform/riscv-dbg/issues/118:

Currently the debug module seems to assume any assertion of ndmreset_o will instantly cause a reset. The dmstatus.allhavereset/dmstatus.anyhavereset bits are set purely on ndmreset_o being asserted rather than any acknowledgement a reset has occurred.

This is incorrect behavior and should be fixed, but it is currently not urgent for us, because the rstmgr resets the system reset domain unconditionally whenever ndmreset_req is asserted (a few cycles after asserting this signal).

Triaged for rv_dm. Assigning to M2.5 with https://github.com/lowRISC/opentitan/labels/Priority%3AP3 because there are only a few clock cycles (between the assertion of ndmreset_o and the instant rstmgr resets the system) during which those bits are not correct -- so I think this can be easily worked around in software without noticeable impact, whereas the RTL changes involved in proper reset acknowledgment are out of scope for this release IMHO.

Depending on how we intend to implement the delayed debug unlock feature, this may be something that we have to fix as well.
CC @rswarbrick

authors: andreaskurth msfschaffner


text:
[dv] error when running on xcelium

Hmac when running on Xcelium has the following error:

TEST: hmac_sanity, SEED: 108809696
LOG: $scratch_path/0.hmac_sanity/out/run.log
UVM_ERROR @   1654118 ps: (hmac_scoreboard.sv:221) uvm_test_top.env.scoreboard [uvm_test_top.env.scoreboard] Check failed csr.get_mirrored_value() == item.d_data (1 [0x1] vs 3 [0x3]) intr_state

top_level test compile error:
BUILD: default
LOG: $scratch_path/default/build.log

xmelab: *E,ASNUSE (../src/lowrisc_dv_chip_sim_0.1/autogen/tb__xbar_connect.sv,59|42): Illegal use of a bit-select, part-select, member-select or mda element [9.3.1(IEEE)].
    force tb.dut.top_earlgrey.clkmgr_clocks.clk_main_hmac = clk_main;
                                          |
xmelab: *E,ASNUSE (../src/lowrisc_dv_chip_sim_0.1/autogen/tb__xbar_connect.sv,60|42): Illegal use of a bit-select, part-select, member-select or mda element [9.3.1(IEEE)].
    force tb.dut.top_earlgrey.clkmgr_clocks.clk_main_infra = clk_main;

Thanks for reporting this. I'm fixing it.

Can we file a bug with Cadence? This seems like a valid thing to do.

Can we file a bug with Cadence? This seems like a valid thing to do.

Seems like that violates the LRM
https://stackoverflow.com/questions/34299114/forcing-a-member-of-a-struct

Can we file a bug with Cadence? This seems like a valid thing to do.

Seems like that violates the LRM
https://stackoverflow.com/questions/34299114/forcing-a-member-of-a-struct

Interesting. Thanks.

close this issue as both PRs are merged.

authors: cindychip sriyerg weicaiyang


text:
Suggestion for prim_util.svh

Would it make sense to combine prim_util.svh with prim_assert.sv into one file?
My suggestion would be:

Rename prim_assert.sv to prim_util.sv.
Move contents from prim_util.svh into prim_util.sv.

That way, all helper functions and macros are in a single file and don't require any include files.
Note that prim_assert.sv is not an include file and therefore doesn't require an include line in each file.

I don't think I understand the last comment. prim_assert.sv defines various macros. These definitely need to be included into files that use them. Maybe its filename should actually be prim_assert.svh: I'm not sure whether there's a good reason that it isn't.
Re combining the two: I don't really have strong feelings but I can't think of any benefit of doing so.

Thanks @NilsGraf for opening this issue, I wanted to get back to you on your email suggesting exactly that.
A couple points from my side:

prim_assert.sv contains include file content, so it is an include file, and we need to keep it that way.
It's named .sv instead of .svh for historic reasons, it would make sense to change that.
prim_assert.sv currently also defines a package; this part is slightly ugly and shouldn't be in an include file. It currently works because of the include guards. I wouldn't mind cleaning that up.
Finally, on the question of one vs two include files: No strong feelings either way, especially in this case where both files are kind of a "catch-all". In general, including more than one file doesn't seem odd to me.

Ah, I see. Originally, no RTL file had an include prim_assert.sv line. prim_assert.sv was just at the beginning of each filelist, which I think worked well. I didn't realize that nowadays all RTL files have an 'include prim_assert.sv` line. I personally like the old way better because it removes the need for the include line in each file.

The "old way" was working only by chance, unfortunately. For a the full story see https://github.com/lowRISC/ibex/commit/23d4243acd4973c27e92ef82b86c9e721026449f.

Thank you for the link, I wasn't aware of this. The "old mechanism" has been used successfully for many chips, but I guess they were using commercial EDA tools.
Would it make sense to modify Fusesoc/edalize to support this? Adding @olofk
I believe the "old way" had the following advantages:

Less noisy source code as it doesn't require an include line in each file.
Doesn't need ifdef guards to avoid defining things multiple times.
Perhaps compiles faster because each include line results in an additional file access (so file prim_assert.sv is read 100 times if you have 100 RTL files that include it).

@NilsGraf It's not so easy. Edalize support 20+ different tools and they all behave slighly different. Many tools, as you say, allow you to put include files first in a list (this is normally referred to as multi file compilation units) but not all. Some Edalize backends also have options to use multi or single file compilation units. Vivado for example requires you to declare these files specifically as global include files as can be seen with SweRV where we need to source this file for a similar situation.
Implicit include files are also hard to handle in some cases. Imagine having a uart.vh and a spi.vh that both sets a DATA_WIDTH define to different values. Which will be used where if we just implcitly include the both files. This might be ok for a single project where you're in control of all sources, but it will quickly break down when you add IP from different vendors or groups so in the light of that I'd say that explicitly including works better.
Ideally, FuseSoC/Edalize should handle implcit include files just as well as the explicit ones but it inevitably becomes a trickier problem when porting between tools.

Thanks for your response. Actually, prim_assert.sv is not setting DATA_WIDTH parameters. It's defining a few global macros such as ASSERT_*. But I could see an issue if another IP defines the same macro names.
I'm curious to know which EDA tools don't support implicit include files. Thanks!

I'm curious to know which EDA tools don't support implicit include files. Thanks!

As mentioned above, Vivado needs these to be explicitly marked as global include files through its tcl interface. QuestaSim (and I think RivieraPro) needs to be setup for mfcu mode. For Quartus I think it's supported in the Pro version, but not in the Standard and Lite versions. Synplify Pro has a switch for Multi file compilation units. Not sure if that applies here though

Thinking about this some more: If another IP defined the same global macro ASSERT then this would be always an issue, even with explicit includes. The lint tool (and simulator) will flag macro redefinition.
Note that prim_assert.sv and prim_util.svh macros are global macros, they are included before the module in each file (see here) (to be local macros they would need to be included inside the module itself, which would also require undef at the end of each file).

As mentioned above, Vivado needs these to be explicitly marked as global include files through its tcl interface. QuestaSim (and I think RivieraPro) needs to be setup for mfcu mode. For Quartus I think it's supported in the Pro version, but not in the Standard and Lite versions. Synplify Pro has a switch for Multi file compilation units. Not sure if that applies here though

Could Fusesoc/edalize take care of this (e.g. set up QuestaSim for mfcu mode)? Thanks!

Yes. mfcu mode for questasim will be added as an option

prim_util.svh is now prim_util_pkg.sv for different reasons.

authors: NilsGraf imphil olofk rswarbrick