text:
[Spark] Skip collecting commit stats to prevent computing Snapshot State

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description
Before this PR, Delta computes a SnapshotState during every commit.  Computing a SnapshotState is fairly slow and expensive, because it involves reading the entirety of a checkpoint, sidecars, and log segment.
For many types of commit, it should be unnecessary to compute the SnapshotState.
After this PR, a transaction can avoid computing the SnapshotState of a newly created snapshot. Skipping the computation is enabled via a spark configuration option spark.databricks.delta.commitStats.collect=false
This change can have a big performance impact when writing into a Delta Table.  Especially when the table comprises a large number of underlying data files.
How was this patch tested?

Locally built delta-spark
Ran a small spark job to insert rows into a delta table
Inspected log4j output to see if snapshot state was computed
Repeated again, this time setting spark.databricks.delta.commitStats.collect=false

Simple demo job that triggers computing SnapshotState, before this PR:
val spark = SparkSession
  .builder
  .appName("myapp")
  .master("local[*]")
  .config("spark.sql.warehouse.dir", "./warehouse")
  .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
  .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
  .getOrCreate

spark.sql("""CREATE TABLE test_delta(id string) USING DELTA """)

spark.sql("""
  INSERT INTO test_delta (id) VALUES (42)
  """)

spark.close()

Does this PR introduce any user-facing changes?
Yes, after this PR the user can set spark config option spark.databricks.delta.commitStats.collect=false to avoid computing SnapshotState after a commit.

Can you provide more details what is the impact/perfomance degradation when reading from the table?
What would happen in batch read and stream read?

Hi @talgo, you asked about reading from the table.  This change has no impact whatsoever on reading, because it does not touch that part of the code.  This is entirely about writing to the table.
I can provide some more details about the impact on writing to the table.  Here is some log output from a spark job which wrote 1278 new rows into a Delta Table.
2024-02-29T08:30:58.129Z INFO org.apache.spark.sql.delta.OptimisticTransaction - [tableId=d19f5423,txnId=a3ba89ad] Attempting to commit version 251406 with 12 actions with Serializable isolation level
2024-02-29T08:31:13.146Z INFO org.apache.spark.sql.delta.DeltaLog - Creating a new snapshot v251407 for commit version 251406
2024-02-29T08:31:13.147Z INFO org.apache.spark.sql.delta.DeltaLog - Loading version 251407 starting from checkpoint version 251400.
2024-02-29T08:31:13.166Z INFO org.apache.spark.sql.delta.DeltaLogFileIndex - Created DeltaLogFileIndex(Parquet, numFilesInSegment: 1, totalFileSize: 457025090)
2024-02-29T08:31:13.166Z INFO org.apache.spark.sql.delta.DeltaLogFileIndex - Created DeltaLogFileIndex(JSON, numFilesInSegment: 7, totalFileSize: 77086)
2024-02-29T08:31:14.178Z INFO org.apache.spark.sql.delta.Snapshot - [tableId=d19f5423-16f3-4107-9454-f4c4816f9aa3] Created snapshot Snapshot(path=gs://<REDACTED>/prod/events/_delta_log, version=251407, metadata=REDACTED
2024-02-29T08:31:14.182Z INFO org.apache.spark.sql.delta.DeltaLog - Updated snapshot to Snapshot(path=gs://<REDACTED>/prod/events/_delta_log, version=251407, metadata=REDACTED
2024-02-29T08:31:14.182Z INFO org.apache.spark.sql.delta.Snapshot - [tableId=d19f5423-16f3-4107-9454-f4c4816f9aa3] DELTA: Compute snapshot for version: 251407
2024-02-29T08:32:19.519Z INFO org.apache.spark.sql.delta.Snapshot - [tableId=d19f5423-16f3-4107-9454-f4c4816f9aa3] DELTA: Done
2024-02-29T08:32:19.519Z INFO org.apache.spark.sql.delta.OptimisticTransaction - [tableId=d19f5423,txnId=a3ba89ad] Committed delta #251406 to gs://<REDACTED>/prod/events/_delta_log

From the timestamps on the left, you can see there is a 65 second gap between the lines DELTA: Compute snapshot for version: 251407 and DELTA: Done.  This is where the Delta library is computing the snapshot state.
After this PR, we completely eliminate that 65 second gap.  The overall time to write rows is made faster by 65 seconds.
This change will have most impact on streaming jobs that write into Delta, because those types of jobs need to do frequent commits.

@vkorukanti if you have a chance to review this. Looks a great improvement 137s vs 63s, reduction of 54%: https://github.com/delta-io/delta/pull/2718#issuecomment-1978290956
And it only impact Stats Logs.

Can you provide more details what is the impact/perfomance degradation when reading from the table?
What would happen in batch read and stream read?

you asked about reading from the table. This change has no impact whatsoever on reading, because it does not touch that part of the code. This is entirely about writing to the table.

@istreeter If I understand correctly, this change would shift the cost of state reconstruction from the writer who produces the post-commit snapshot, to a subsequent reader who consumes that snapshot on the same cluster. If no such reader exists (e.g. the table is write-mostly or the reader is on a different cluster), then the change is a net win. That's what the microbenchmark above was measuring: just writes, no read afterward.
For an iterated workload that does write -> read -> ... -> write -> read, I would expect no change in the overall latency of a write -> read pair. Does that sound correct?

Hi @scovich yes that sounds correct, as you described it.  If every write is always followed by a read on the same cluster, then there is no net change in the number of state reconstructions.  This PR would just shift the state reconstruction to a different place.
However, if you consider any other workload pattern, (in which not every write is followed by a read on the same cluster) then there is benefit from not reconstructing the state as part of the write.  For example, this workload pattern would also benefit: write -> write -> read -> write -> write -> read -> write -> write -> read.
At my company, we use a dedicated cluster just for writing.  So the pattern is simply write -> write -> write -> write -> write etc.  There should be no reason to ever do state reconstruction on that usage pattern.

At my company, we use a dedicated cluster just for writing. So the pattern is simply write -> write -> write -> write -> write etc. There should be no reason to ever do state reconstruction on that usage pattern.

Is the dedicated cluster writing repeatedly? Or is it ephemeral, with each cluster instance performing one write and then shutting down?
Asking because even "blind" writes actually do involve reads. At a minimum, they must access the snapshot's protocol and metadata. If the workload uses app ids to track stream progress, then that triggers a full state reconstruction as well. If it's a merge or update instead of a blind insert, then there's also a full-blown query in the write path.
So for write -> write on the same cluster, we have:

initial snapshot
P&M (and possibly state reconstruction)
commit
post-commit snapshot
P&M (and possibly state reconstruction)
commit
post-commit snapshot
etc

In such a case, this PR wouldn't change overall timing.
Note: I 100% agree the optimization is helpful when it applies, and that we should probably incorporate it into Delta. I just worry it might not apply as often as we wish it did, in practice. Is the microbenchmark fully representative of the actual workload you hope to optimize?

If we have multiple writers, computing the Snapshot on read also improves perf by skipping loading a snapshot (after my write) that needs to be refreshed anyway?

Hi @ryan-johnson-databricks the cluster we use for writing is long-lived and not ephemeral.  Once the cluster is up, and once the spark context is created, then we re-use that spark context and write a batch of rows in Delta every ~1 minute.  Each write is an append; never a merge, never an update.

Is the microbenchmark fully representative of the actual workload you hope to optimize?

The logs I shared in this comment are real logs from a production workload.  The logs were taken after the cluster had already been up and running for several hours.  Imagine a repeating loop with those log lines repeated again and again.
I am certain that eliminating the state reconstruction would make this workload faster.  However, I believe this PR in its current state only goes half way to solving that problem!  After this PR, Delta will avoid computing state for the post-commit snapshot.  But currently, Delta also computes state for the pre-commit snapshot on this line:
val readRowIdHighWatermark =
  RowId.extractHighWatermark(snapshot).getOrElse(RowId.MISSING_HIGH_WATER_MARK)

So far I have not worked out how to avoid  computing state on that line, although I believe it should be possible.  For our workload, we are not interested in row tracking or watermarks.  (And even if we were -- I still reckon it could be achieved without computing snapshot state).
One final thought... it would be awesome if we could have a unit test that proves that we don't compute snapshot state during writes.  This was beyond my abilities to add such a unit test!

the cluster we use for writing is long-lived and not ephemeral. Once the cluster is up, and once the spark context is created, then we re-use that spark context and write a batch of rows in Delta every ~1 minute. Each write is an append; never a merge, never an update.

Is the microbenchmark fully representative of the actual workload you hope to optimize?

The logs I shared in https://github.com/delta-io/delta/pull/2718#issuecomment-1978290956 are real logs from a production workload. The logs were taken after the cluster had already been up and running for several hours. Imagine a repeating loop with those log lines repeated again and again.

My questions actually came from looking at those logs -- they ONLY show the commit path itself, and creation of the post-commit snapshot (251407). They do not include creation of the snapshot the transaction read from (251406). If the workload runs in a loop, then one minute later we'd get a second transaction that starts from 251407 (and triggers P&M at a minimum), and which tries to create 251408. Thus, the first transaction avoids the cost but the second transaction ends up paying it one minute later.
Can you check what happens across 2-3 inserts instead of just the commit step of one insert in isolation? I expect that the 65 seconds which disappeared from that log snippet really just moved somewhere else.

it would be awesome if we could have a unit test that proves that we don't compute snapshot state during writes

Should be easy enough to check Snapshot.stateReconstructionTriggered after the operation you care about, at least for the really expensive computed state. It wouldn't cover protocol, metadata, or checkpoint size tho, because those don't rely on state reconstruction in the first place.
NOTE: In general, we can't avoid computing at least protocol and metadata, because they're required to interpret the snapshot correctly (table schema, table features, etc). But they're also much cheaper to compute than the full state reconstruction.

Delta also computes state for the pre-commit snapshot on this line:
val readRowIdHighWatermark =
  RowId.extractHighWatermark(snapshot).getOrElse(RowId.MISSING_HIGH_WATER_MARK)

So far I have not worked out how to avoid computing state on that line, although I believe it should be possible.

That actually looks like a simple (performance) bug: AFAIK, the rowid high watermark only matters if the table supports rowids, but the extractHighWatermark method body lacks any isSupported(snapshot.protocol) check that could early-return None?

Can you check what happens across 2-3 inserts instead of just the commit step of one insert in isolation? I expect that the 65 seconds which disappeared from that log snippet really just moved somewhere else

Sure, I ran a simple script like this:
(1 to 100).foreach { i =>
  spark.sql(s"INSERT INTO delta.`file:///tmp/test_delta` (v) values ($i)")
}

I attach here two versions of the stdout logs:

logs-before-pr.txt is when I ran the script using a checkout of delta-io immediately before my PR.
logs-after-pr.txt is running the same script including my PR and also changing this line to simply val readRowIdHighWatermark = RowId.MISSING_HIGH_WATER_MARK i.e. the other problem I mentioned which I've not fixed yet.

From the logs, you will see that before this PR, we get log lines like DELTA: Compute snapshot for version: x for every single commit.  Whereas after this PR, we only get that log line once every 10 commits, corresponding to the checkpoint interval.  For all other non-checkpointing commits, it does not compute the snapshot state.

That actually looks like a simple (performance) bug: AFAIK, the rowid high watermark only matters if the table supports rowids, but the extractHighWatermark method body lacks any isSupported(snapshot.protocol) check that could early-return None?

The delta spec here makes a distinction between when row tracking is supported vs when row tracking is enabled.  I think ideally we want to avoid computing snapshot state whenever row tracking is not enabled.  Whereas I think your suggested change would only avoid the computation if row tracking is not supported.

The delta spec here makes a distinction between when row tracking is supported vs when row tracking is enabled. I think ideally we want to avoid computing snapshot state whenever row tracking is not enabled. Whereas I think your suggested change would only avoid the computation if row tracking is not supported.

The spec does require keeping track of the high-water mark and setting base row ids / default row commit whenever the feature is supported, not only enabled.
The readRowIdHighWatermark value isn't currently used when the feature isn't supported: https://github.com/delta-io/delta/blob/aa0af003068a9491b7b6830712082af3f6fc87de/spark/src/main/scala/org/apache/spark/sql/delta/ConflictChecker.scala#L521
so as Ryan suggested, the simplest would be to skip the expensive snapshot recomputation if the feature isn't supported

running the same script including my PR and also changing this line to simply val readRowIdHighWatermark = RowId.MISSING_HIGH_WATER_MARK i.e. the other problem I mentioned which I've not fixed yet.
From the logs, you will see that before this PR, we get log lines like DELTA: Compute snapshot for version: x for every single commit. Whereas after this PR, we only get that log line once every 10 commits, corresponding to the checkpoint interval. For all other non-checkpointing commits, it does not compute the snapshot state.

Ah, that makes sense now. I didn't know you had made additional changes to avoid triggering snapshot state on the read path, in addition to the write path changes in this PR.

That actually looks like a simple (performance) bug: AFAIK, the rowid high watermark only matters if the table supports rowids, but the extractHighWatermark method body lacks any isSupported(snapshot.protocol) check that could early-return None?

The delta spec here makes a distinction between when row tracking is supported vs when row tracking is enabled. I think ideally we want to avoid computing snapshot state whenever row tracking is not enabled. Whereas I think your suggested change would only avoid the computation if row tracking is not supported.

I had to word the my suggested change the way I did, in order to comply with the spec:

When the feature rowTracking exists in the table protocol's writerFeatures, then we say that Row Tracking is supported. In this situation, writers must assign Row IDs and Commit Versions

Writers can't assign row ids without knowing the current high watermark, so if the feature is "supported" in the table protocol then we have to pull the high watermark in order to commit.
In practice, I don't think the distinction between "enabled" and "supported" should matter, because "supported" is anyway a transient state, used e.g. while a table is trying to enable rowids for the first time:

A backfill operation may be required to commit add and remove actions with the baseRowId field set for all data files before the table property delta.enableRowTracking can be set to true.

OK I read bit more of the protocol, and now I understand the point you're making: the writer MUST calculate the high watermark if rowTracking exists in the table protocol's writerFeatures.
That is OK for me, because the tables I write to do not have rowTracking in the table's features.  So I will take your suggestion and I will amend extractHighWatermark to return None if row tracking is not supported.
(I do wonder if it is possible to implement a cheap way of checking the high watermark.  Similar to how there is a cheap way to check protocol and metadata.  Then tables with row tracking enabled could also get the benefit of the change I'm making.  But I won't pursue that idea in this current PR).

I have a question for the people following this thread....
I do believe this PR will make a big performance difference for many uses of Delta.  But currently, users would only get that speed-up if they change commitStats.collect to false.  Do you think I should abandon this configuration option?  Maybe it is better to say that all writers should avoid computing snapshot state during transactions.  Users should get the benefit of this change without needing to know about the config option.
I only added the configuration option because I didn't want to upset someone who is depending on that logging.

I do believe this PR will make a big performance difference for many uses of Delta. But currently, users would only get that speed-up if they change commitStats.collect to false. Do you think I should abandon this configuration option? Maybe it is better to say that all writers should avoid computing snapshot state during transactions. Users should get the benefit of this change without needing to know about the config option.
I only added the configuration option because I didn't want to upset someone who is depending on that logging.

That's a great question and I don't have a good answer. More configs are always annoying, but people also complain (loudly) if their workload breaks.

I think numFilesTotal and sizeInBytesTotal are important stats and we should keep the config.
It could be useful to track the growth of a table

@felipepessoto I do see your point that it's useful to track the growth of a table over time.  But it seems that information just isn't naturally available inside OptimisticTransaction, and so I don't think it's the natural place to put that logging.
How would you feel if numFilesTotal and sizeInBytesTotal were tracked every time the snapshot state is computed?  I'm thinking I could add a line somewhere around here like:
val stats = SnapshotStateStats(
  numFilesTotal = _computedState.numOfFiles,
  sizeInBytesTotal = _computedState.sizeInBytes
)
recordDeltaEvent(deltaLog, "delta.snapshot.stats", data = stats)

Admittedly you would not get the stats for every transaction.  But you would get it every time Delta does a post-commit checkpoint.  And you would get it every time Spark reads a version of the table.

I also don’t have a good answer, depends on how people use it, that’s why I think we need the config flag.
@sezruby any thoughts about it?

Some users might use it to monitor their workload. There's no good way to monitor Delta table workload, so it would be better to keep the config. It's existing code, no regression risk on keeping it. Yet, there's a way to recalculate the metrics when it's needed, we could remove it I think.
We can't rely on tracking everytime, as concurrent updates could be done in different cluster/app.

If we keep the config option, and if it is set to true by default (i.e. the backwards-compatible behaviour) then we all know realistically most users will never change that config to something different.
I just think that is a real lost opportunity!  I'll stress it again: I strongly believe this PR can make Delta workloads much faster for many users under many circumstances.  I suspect, without proof, that Delta users will appreciate that performance gain more so than they would appreciate the logging.
I think the options available to us are:

Keep the config option, and set the default option to commitStats.collect = true.  Users do not get the performance benefit of this PR by default, but it means logging is the same as it was before.
Keep the config option, and set the default option to commitStats.collect = false.  Users do get the performance benefit by default.  And if someone cares about monitoring those stats then they can enable the config option again.
Simply do not have a config option.  All users get the performance gain.  To partially compensate for removing those stats from the logs, we can instead add some extra logging in the place I mentioned before

I really think 3 is the best option.  But you folks here have better knowledge than me of what Delta users might want.

I tend to agree that we should enable the config by default if we use a config. I wish there were a way to know whether people actually rely on the logging in practice to make a decision between config vs. no config. Safe choice might be to have the config (enabled by default) and remove the config in a future release if nobody squawks?

Thank you everyone for pointing me in the right direction :)
I have made the following changes to the PR:

The default value of commitStats.collect is now false
I added some extra logging whenever the snapshot state is computed anyway.  To partially compensate for the loss of logging in the optimistic transaction.

@istreeter, now that we are only conditionally collecting numFile and size, I wonder if would be possible to cheaply and reliably calculate it?
example:
size = previousSize + bytesNew
numFiles = previousNumFiles + addfiles - removedFiles

UPDATE: bytesNew is only added files, doesn't take removed files in consideration

@istreeter, now that we are only conditionally collecting numFile and size, I wonder if would be possible to cheaply and reliably calculate it?

@felipepessoto I think the only way to get previousSize and previousNumFiles is to compute the Snapshot State of the pre-commit snapshot.  The great thing about this PR in its current state is we don't need to compute the Snapshot State of either the pre- or post- commit snapshot states.
If you think it's possible to cheaply do size = previousSize + bytesNew then I'm willing to take a go, but you'll have to point me in the right direction how to do it.

@istreeter Thanks for working on this change. Please resolve the conflicts on the PR so that it could be merged?

Seems like some unit tests are failing with this change and needs to be fixed.

Hi @prakharjain09 the unit tests now all pass 🎉
I pushed one extra commit which amends a couple of pre-existing unit tests.  I think the fix is reasonable -- it just sets things up so the starting point is from the same place that is was before this PR.
I pushed it as an extra commit so it is obvious what I have changed.

authors: felipepessoto istreeter johanl-db prakharjain09 ryan-johnson-databricks scovich sezruby talgo10


text:
Example doesnt work

If I try to run the example, I get this:
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0xc0610e]

goroutine 1 [running]:
github.com/deluan/lookup.ensureGrayScale(0x0, 0x0, 0x0, 0x0)
        C:/lookup/helpers.go:14 +0x6e
github.com/deluan/lookup.(*OCR).Recognize(0xc00011df48, 0x0, 0x0, 0x0, 0xc000048320, 0xc00005a000, 0xc000043f48)
        C:/lookup/ocr.go:85 +0x45
main.main()
        C:/lookup/test/examples_ocr.go:30 +0xd2

https://github.com/deluan/lookup/blob/a12095427a5620a9f542a10f49c99eb9d1286597/helpers.go#L14
https://github.com/deluan/lookup/blob/a12095427a5620a9f542a10f49c99eb9d1286597/ocr.go#L85
https://github.com/deluan/lookup/blob/a12095427a5620a9f542a10f49c99eb9d1286597/examples_ocr_test.go#L30

Looks like the problem is here:
https://github.com/deluan/lookup/blob/a12095427a5620a9f542a10f49c99eb9d1286597/examples_ocr_test.go#L3-L8
need to add:
_ "image/png"

I think it is fair to add the import you suggested, as a "complete" example, but the examples in the repository clearly work as they are part of the documentation and they are run and show the results when generating the docs: https://pkg.go.dev/github.com/deluan/lookup#example-package-Lookup
Most probably because they are part of the lookup package, and some other file in the folder is import the image/png package. They fail only if ran individually.

source: gharchive/issue

repo: deluan/lookup

license: MIT

license_type: permissive

license_source: github-api


text:
proposal: Edit in Editor example

I had to make a Starlight component override for the <EditLink/> component to change its behavior in development mode so that the content file is opened in the user's editor. I don't think it's worth publishing a new package just for this, so I thought it could maybe be a small and interesting example.
Here is a demo in StackBlitz (this works in StackBlitz too): https://stackblitz.com/edit/github-mydczx?file=src%2Fcomponents%2FEditInEditor.astro
Any thoughts on adding a similar example? I don't mind at all if it's rejected, e.g. not useful enough, I just thought I would ask before submitting a PR or forgetting about it.

Definitely! The bar for admission into the examples is low, and this example is way over it 🚀

authors: HiDeoo delucis

repo: delucis/starlight-examples


text:
powershell documentation

Status
Ready
Related Issues
fixes: link to the issue
Description
PowerShell doc
Screenshots
Paste here any images that will help the reviewer

Actually started to write doc and now realized you made PR for it, I might create another PR later

source: gharchive/pull-request

authors: glicht yardensade

repo: demisto/content-docs


text:
Added new docker image for vcrpy

Status
Ready
Related Issues
related: https://github.com/demisto/content/pull/10830
Description
Added new docker image for vcrpy liberty, for partner contribution.

@ChanochShayner Just making sure you haven't missed the fact that the build is failing

@ChanochShayner Just making sure you haven't missed the fact that the build is failing

Docker Image Ready - Dev
Docker automatic build at CircleCI has deployed your docker image: devdemisto/vcrpy:1.0.0.15354
It is available now on docker hub at: https://hub.docker.com/r/devdemisto/vcrpy/tags
Get started by pulling the image:
docker pull devdemisto/vcrpy:1.0.0.15354

Docker Metadata

Image Size: failed querying size

Image ID: sha256:96a9fb5915a032e74cab8b653d77a61a8d14c32cf02a4f746426d91e6d3834af

Created: 2021-01-11T14:39:39.824728506Z

Arch: linux/amd64

Command: ["python3"]

Environment:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_VERSION=3.9.1
PYTHON_PIP_VERSION=20.3.3
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/5f38681f7f5872e4032860b54e9cc11cf0374932/get-pip.py
PYTHON_GET_PIP_SHA256=6a0b13826862f33c13b614a921d36253bfa1ae779c5fbf569876f3585057e9d2
DOCKER_IMAGE=devdemisto/vcrpy:1.0.0.15354

Labels:

org.opencontainers.image.authors:Demisto <containers@demisto.com>
org.opencontainers.image.revision:0b93577268f264db881517615edf5452183af3ca
org.opencontainers.image.version:1.0.0.15354

Docker Image Ready - Dev
Docker automatic build at CircleCI has deployed your docker image: devdemisto/vcrpy:1.0.0.15354
It is available now on docker hub at: https://hub.docker.com/r/devdemisto/vcrpy/tags
Get started by pulling the image:
docker pull devdemisto/vcrpy:1.0.0.15354

Docker Metadata

Image Size: failed querying size

Image ID: sha256:96a9fb5915a032e74cab8b653d77a61a8d14c32cf02a4f746426d91e6d3834af

Created: 2021-01-11T14:39:39.824728506Z

Arch: linux/amd64

Command: ["python3"]

Environment:

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_VERSION=3.9.1
PYTHON_PIP_VERSION=20.3.3
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/5f38681f7f5872e4032860b54e9cc11cf0374932/get-pip.py
PYTHON_GET_PIP_SHA256=6a0b13826862f33c13b614a921d36253bfa1ae779c5fbf569876f3585057e9d2
DOCKER_IMAGE=devdemisto/vcrpy:1.0.0.15354

Labels:

org.opencontainers.image.authors:Demisto <containers@demisto.com>
org.opencontainers.image.revision:0b93577268f264db881517615edf5452183af3ca
org.opencontainers.image.version:1.0.0.15354

Something is wrong with .dependabot/config.xml . Not clear why there are so many changes there.
@glicht It seems that the necessary changes have been added (as in other PRs)
One package_manager: python and one - package_manager: docker

Something is wrong with .dependabot/config.xml . Not clear why there are so many changes there.
@glicht It seems that the necessary changes have been added (as in other PRs)
One package_manager: python and one - package_manager: docker

Something is wrong with .dependabot/config.xml . Not clear why there are so many changes there.
@glicht It seems that the necessary changes have been added (as in other PRs)
One package_manager: python and one - package_manager: docker

Ok. I guess there was something wrong with the displayed diff. It looks good now.

Something is wrong with .dependabot/config.xml . Not clear why there are so many changes there.
@glicht It seems that the necessary changes have been added (as in other PRs)
One package_manager: python and one - package_manager: docker

Ok. I guess there was something wrong with the displayed diff. It looks good now.

I close the PR because docker image is not generated for Partner in this case.

I close the PR because docker image is not generated for Partner in this case.

authors: ChanochShayner dc-builder glicht guykeller

repo: demisto/dockerfiles


text:
🛑 SearxNG III is down

In be8b3b0, SearxNG III (https://sx3.eka.monster) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SearxNG III is back up in a2e7159 after 11 minutes.

repo: demonsteru/up


text:
'Extension host terminated unexpectedly' - Argdown

Please select the area your Bug applys to. (Multiple selections are Possible)

[ ] Onboard. Initial vault setup and import/export pods
[ ] Create. Note creation, lookup, snippets and templates
[ ] Retrieve. Backlinks, references, graph view
[ ] Structure. Refactoring, multi-vault and schemas
[ ] Publish. Sharing your repo with the world
[x] Misc (Choose this if your not sure)

Describe the bug
Getting 'Extension host terminated unexpectedly' error upon starting Dendron in VSCode no matter how many times I reloaded/restarted and rebooted my computer.
To Reproduce
Steps to reproduce the behavior:

Have the Argdown extension already installed.
Open Dendron in VSCode and either open the Calendar or the Dendron Preview V2.

Expected behavior
'Extension host terminated unexpectedly' error appears.
Desktop (please complete the following information)

OS: MS Windows 10
Browser: Chrome

Additional context
After uninstalling Argdown extension, the problem was resolved.

Thanks for the issue. We'll be investigating this separately when we have some more bandwidth 😅

I'm also seeing this. It appears to run out of memory:

Note that it's happening outside of Dendron workspaces (CC #1540).

This might have gotten worse recently, but disabling dendron.watchForNativeWorkspace seems to fix it for me.

Dendron 0.74 disables dendron.watchForNativeWorkspace by default now, and this crash should not occur anymore. If it continues to happen, please let us know. Otherwise this issue will close automatically in a few days.

Shouldn't this be kept open for tracking, since people might still enable that preference?

authors: SeriousBug kevinslin kzilla-git lnicola

repo: dendronhq/dendron


text:
Dendron fails to start

Describe the bug
Dendron fails to start. Upon launching VSCode, dendron tries to start (indicated by a popup in the lower right of the VSCode window) and just spins until it produces the ETIMEDOUT error listed in the log file.
To Reproduce
Steps to reproduce the behavior:

Launch VSCode.
Wait
See error

Expected behavior
Dendron should start automatically and (relatively) silently.
Desktop (please complete the following information):
VSCode version information:
Version: 1.52.1
Commit: ea3859d4ba2f3e577a159bc91e3074c5d85c0523
Date: 2020-12-16T16:30:02.420Z
Electron: 9.3.5
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Darwin x64 19.6.0

Additional context
Updated VSCode and on application restart this behavior began.
Dendron Log file
{"level":30,"time":1609795980196,"pid":47886,"hostname":"user-YMD6V-MBP","ctx":"FileStorageV2","wsRoot":"/Users/user/Documents/dendron","vaultsv3":[{"fsPath":"vault"}],"level":"info"}
{"level":50,"time":1609796057309,"pid":47886,"hostname":"user-YMD6V-MBP","name":"dendron","ctx":"ReloadIndex.execute","error":{"name":"RequestError","message":"Error: connect ETIMEDOUT 10.200.21.163:52747","cause":{"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"10.200.21.163","port":52747},"error":{"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"10.200.21.163","port":52747},"options":{"url":"http://localhost:52747/api/workspace/initialize","qs":{},"body":{"uri":"/Users/user/Documents/dendron","config":{"vaults":["/Users/user/Documents/dendron/vault"]}},"json":true,"method":"POST","simple":true,"resolveWithFullResponse":false,"transform2xxOnly":false}},"msg":"unable to initialize engine"}
{"level":50,"time":1609796239065,"pid":47886,"hostname":"user-YMD6V-MBP","name":"dendron","ctx":"ReloadIndex.execute","error":{"name":"RequestError","message":"Error: connect ETIMEDOUT 10.200.61.114:52747","cause":{"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"10.200.61.114","port":52747},"error":{"errno":"ETIMEDOUT","code":"ETIMEDOUT","syscall":"connect","address":"10.200.61.114","port":52747},"options":{"url":"http://localhost:52747/api/workspace/initialize","qs":{},"body":{"uri":"/Users/user/Documents/dendron","config":{"vaults":["/Users/user/Documents/dendron/vault"]}},"json":true,"method":"POST","simple":true,"resolveWithFullResponse":false,"transform2xxOnly":false}},"msg":"unable to initialize engine"}
{"level":50,"time":1609796258394,"pid":47886,"hostname":"user-YMD6V-MBP","name":"dendron","ctx":"LookupProvider:onAccept","err":{"type":"r","message":"","stack":"Error: \n\tat /Users/user/.vscode/extensions/dendron.dendron-0.23.0/dist/extension.js:905:160862","status":"unknown","msg":"","friendly":"something went wrong. please submit a bug report to https://github.com/dendronhq/dendron/issues/new?assignees=&labels=&template=bug_report.md&title= with the output of `Dendron: Open Log`","payload":"{\"msg\":\"Cannot read property 'forEach' of undefined\",\"stack\":\"TypeError: Cannot read property 'forEach' of undefined\\n\\tat d.refreshNotesV2 (/Users/user/.vscode/extensions/dendron.dendron-0.23.0/dist/extension.js:102:72964)\\n\\tat d.writeNote (/Users/user/.vscode/extensions/dendron.dendron-0.23.0/dist/extension.js:102:74225)\\n\\tat async t.LookupProviderV2._onAcceptNewNote (/Users/user/.vscode/extensions/dendron.dendron-0.23.0/dist/extension.js:905:156358)\\n\\tat async t.LookupProviderV2.onDidAcceptForSingle (/Users/user/.vscode/extensions/dendron.dendron-0.23.0/dist/extension.js:905:158000)\"}"}}

Note that the final line is repeated ~10 more times with only a different timestamp.

Nevermind, an OS update and a few restarts seemed to have fixed this issue and I can no longer reproduce it.

Nevermind, an OS update and a few restarts seemed to have fixed this issue and I can no longer reproduce it.


text:
chore: refactoring show preview classes

chore: refactoring show preview classes
This change refactors how the plugin-side classes around Preview work. The primary motivation is to get rid of circular dependencies in this area.

There's now a unified way for plugin code to perform Preview operations - by getting an instance of PreviewProxy. This interface now completely hides the vscode.webviewPanel, so no code calling preview has to worry about preview lifecycle events anymore. This interface also now exposes methods isOpen and isVisible.  If more properties are needed later, we can add them. Using this interface should help us avoid introducing circular dependencies in this area going forward.
All event registration around preview (like changing editors, updating text document) are encapsulated within PreviewPanel. Construction of PreviewPanel is abstracted through PreviewPanelFactory, which returns a PreviewProxy (not a PreviewPanel). This means we can pull out all lifecycle events outside of the window/workspace watchers (and hence remove more circular dependencies). The one exception is with the automaticallyShowPreview feature, which still needs to listen to onEditorChanged outside of PreviewPanel, as it needs to potentially activate the panel if it's not currently opened.
For simplification, I've pulled out the methods surrounding preview-link-click-handling logic into a separate interface and class in PreviewLinkHandler.ts. This also helps with testing.
Most logic in ShowPreviewCommand have been pulled out and moved to PreviewPanel and PreviewLinkHandler

Manual Tests:
There's a high chance of regression with this change, so it'd be good to have some eyes on it via Nightly. I've done some manual testing, and so far the following scenarios look good:

Regular Preview Operations
Enabled automaticallyShowPreview - feature worked; no undesired refocus effect when having the preview in the same editor group as a note.
Opened a markdown file not part of Dendron workspace via context menu; preview opened fine.

Existing Preview Issues
I did not actively try to fix existing preview bugs with this change (which seem to be happening on the webview side), those are out of scope for this particular change and include:

preview no longer updating after a while
preview going blank after a while
metadata info (such as backlinks and tags) disappearing once you start typing.

Pull Request Checklist
If this is your first time submitting a pull request to Dendron, copy and paste the full Dendron Review Checklist into this request and check off each item as necessary.
This template contains the short checklist which is used by the Dendron core team.
Testing

[ ] Write Tests
[ ] Confirm existing tests pass
[ ] Confirm manual testing
[ ] If your tests changes an existing snapshot, make sure that snapshots are updated
[ ] If you are adding a new language feature (graphically visible in VS Code/preview/publishing), make sure that it is included in the test workspace

Docs

[ ] if your change reflects documentation changes, also submit a PR to dendron-site and mention the doc PR link in your current PR

Analytics

[ ] if you are adding analytics related changes, make sure the Telemetry docs are updated

Other than the issue with preview always launching at startup regardless of the setting, looks good to me.

nice job. one request - can you update docs in [[Views|dendron://dendron.docs/pkg.plugin-core.dev.cook#views]] so folks know how to access the preview going forward?
also, for major refactoring operations, can you also add it to dev.process.close-loop about announcing this in the #dev channel when merged? similar to #2188 (review)

Updated docs at the following places:

[[Views|dendron://dendron.docs/dev.cook#views]]
[[Close Loop|dendron://dendron.docs/dev.process.close-loop]]

authors: SeriousBug jonathanyeung


text:
Correct a latex equation in the tutorial

Pull Request Checklist
If you are a community contributor, copy and paste the PR template from Dendron Community PR Checklist and add it to the body of the pull request.
If you are a team member, copy and paste the template from Dendron Extended PR Checklist.
To copy the template, click on the Raw button on the gist to copy the plaintext version of the template to this PR.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

authors: CLAassistant woodwm


text:
Add denncomp file

I have created an issue
created a new branch
created a text file
now I am going to created a pull request

Just a test because of a problem I have encountered

repo: denncomp/github-slideshow


text:
adding Hotkeys

is it possible to add hotkeys for classifications projects for faster labeling

100 this is required for any large scale annotation task :-)

repo: dennisbakhuis/pigeonXT


text:
Analyze all tracks

This feature was removed a couple of versions ago. However, I think that it would be very useful to get a quick overview of the training effort.
Also, it would be nice to be able to group by year, month or week.
Do you think this would be a valuable feature?

For sure - I would love to see this implementation properly!
I removed the available implementation as it was just summing up all recorded tracks: there was no differentiation by activity type, filtering or anything else.

Awesome. I'd love to help, but unfortunately I am not much of a java developer myself. So if there is anything I could do besides coding, e.g. testing or reviewing, I'd be happy to do this.

Coding is actually not that complicated in this case.
It is basically about accessing the database (via SQL) and hand the data to the UI (which is actually the complicated part).
What would be great if you could define the goals that you want to achieve (why do want statistics?) as well as a draft of a possible UI design.
Then somebody can just take and implement it.
One technical requirement: try to avoid introducing additional libraries for visualization.

An API for external plugins is now available: https://github.com/OpenTracksApp/OpenTracks/commit/a3eb1070686fe3a377ab50c4a9bb1c91bd62c279

authors: dennisguse mwip

repo: dennisguse/OpenTracks


text:
HTM doesn't work

I've created example project with deno run -A -r https://fresh.deno.dev my-project and tried to replace JSX with HTM. It renders fine but there is no interactivity. As I can see no JS related to Counter is sending to browser (e.g. no related <source> tag). I suppose it's a wrong behaviour.
Here are my changes:
// islands/Counter.ts (not tsx!)

import { html } from "htm/preact";
import { useState } from "preact/hooks";
import { IS_BROWSER } from "$fresh/runtime.ts";
import { tw } from "@twind";

interface CounterProps {
  start: number;
}

export default function Counter(props: CounterProps) {
  const [count, setCount] = useState(props.start);
  const btn = tw`px-2 py-1 border(gray-100 1) hover:bg-gray-200`;
  return html`
    <div class=${tw`flex gap-2 w-full`}>
      <p class=${tw`flex-grow-1 font-bold text-xl`}>${count}

      <button
        class=${btn}
        onClick=${() => setCount(count - 1)}
        disabled=${!IS_BROWSER}
      >
        -1
      </button>
      <button
        class=${btn}
        onClick=${() => setCount(count + 1)}
        disabled=${!IS_BROWSER}
      >
        +1
      </button>

  `;
}

// routes/index.ts (not tsx!)

import { html } from "htm/preact";
import { tw } from "@twind";
import Counter from "../islands/Counter.ts";

export default function Home() {
  return html`
    <div class=${tw`p-4 mx-auto max-w-screen-md`}>

      <p class=${tw`my-6`}>
        Welcome to \`fresh\`. Try update this message in the ./routes/index.tsx
        file, and refresh.

      <${Counter} start=${3} />

  `;
}

Thanks for the clue! Looks like HTM really has problems with hooks.
So, if get it right, this issue can be closed - Fresh doesn't relate to this.

Actually, I think that issue gave me an idea of something you can try. Can you try import htm like this: https://esm.sh/htm@3.1.1/preact?deps=preact@10.8.1?

Yes, it's working now!
// import_map.json
...
    "htm": "https://esm.sh/htm@3.1.1",
    "htm/preact": "https://esm.sh/htm@3.1.1/preact?deps=preact@10.8.1",
...

Then I'll close issue. Solution is using same version of preact.

authors: lucacasonato zlumyo

repo: denoland/fresh


text:
todo: add pagination to author pages

E.g. https://hunt.deno.land/user/iuioiua

Completed in #322.

repo: denoland/saaskit


text:
Github Token Authentication is coming this year

Hi Denis,
I've started getting emails from Github like this:
You recently used a password to access the repository at blahblah/obsidian with git using git/2.20.1.
Basic authentication using a password to Git is deprecated and will soon no longer work. Visit https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information around suggested workarounds and removal dates.
Thanks,
The GitHub Team
It looks like Github is going to be doing away with username/password authentication in July/August of this year, and if nothing is done about it in this plug-in, it will stop working for everyone then.
I'm pretty sure you are aware of this already, but I didn't see it listed in the open issues so I thought I would post this so we can track your  progress on it.  LOVE the plugin, by the way!

Hello, I think this is not due to the way the plugin work but how you configured git on your side.
This is the way all git server expect authentication currently: https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh

You have to change your global git credentials. Not related to this plugin.

repo: denolehov/obsidian-git


text:
Fix README (i.e. import)

Hi!
I invite you to update your README as the import section looks to be incorrect and when I tried to correct the URL even that is dead..?
Hope this helps
Sincerely

Thank you, done!

authors: denyncrawford ldexterldesign

repo: denyncrawford/deno-livereload


text:
Prompt disappears for one directory

Starship prompt disappears when cd'ing into one particular directory as shown in the screen shot. Using iTerm2 3.3.9; other versions shown in the screenshot.
Note that both iTerm shell integration and starship ask that configuration be the last line of your startup. In my case, I chose iTerm shell integration to be last, because the other way broke the status bar feature of iTerm.

<!--
Thanks for filing an issue 😄
<p>BEFORE SUBMITTING NEW ISSUE, PLEASE, READ THE TROUBLESHOOTING PAGE!
https://github.com/denysdovhan/spaceship-prompt/blob/master/docs/Troubleshooting.md
Check the other issue templates if you are trying to submit a bug report, feature request, or question.
Search open/closed issues before submitting since someone might have asked the same thing before!
-->

Please remove this issue to https://github.com/starship/starship/issues

Moving to starship-issues

authors: Runrioter edikus

repo: denysdovhan/spaceship-prompt


text:
Handle proxy errors

Add error event listener and don't crash

And send fail response event to UI

repo: deomitrus/utini

license: mit

license_source: bigquery


text:
Simplify PySpark testing with DataFrame equality functions

URL: https://www.databricks.com/blog/simplify-pyspark-testing-dataframe-equality-functions
Review (1-2 sentences): Finally we have PySpark functions for testing! Starting from Spark 3.5. No more additional libraries for testing. Or, maybe..
Created by: DE or DIE Bot :robot: on behalf of https://t.me/if_no_then_yes.

Published in Digest #36.

repo: deordie/deordie-digest


text:
[AP] Datadog Synthetic Monitor

Description
In order to test changes to reverse proxy, we will need a datadog synthetic monitors setup to test if the homepage is displayed.
There will also need to be another monitor that simulates a 404 to ensure the error page from content build is displayed.
Acceptance Criteria

[ ] Create a datadog synthetic monitor (HTTP) that asserts that va.gov is accessible
[ ] Create a datadog synthetic monitor (HTTP) that asserts the 404 behavior on va.gov

@mreed101 please review this ticket, has it fully met DoD requirements?

The monitor in question can be found at https://vagov.ddog-gov.com/monitors/277210. @mreed101 can you please review this and give your blessing to close the issue? Thank you so much!

LGTM. Closing this ticket

authors: LisaLindsey jcbolling mreed101


text:
Add support for Swift Package Manager

Initial work to add Swift support to Dependabot.
This is currently a work-in-progress, but I'd welcome any feedback on what we have so far.

Hey @mattt thx for creating this draft. Did you got any feedback? Is there anything I could help you with? (testing?) Would really love to have this feature available.

Thanks a lot for this initial work. Is there anything we can do to help get this moving? We'd also love to have SPM support in Dependabot.

Many thanks!! I'm also available to help @mattt.

Hi, I was just wondering if this feature is still in the works? I think it would make many iOS developers on GitHub very happy. Did I miss any developments on this?

Has this been abandoned @mattt ? Thanks

If / when we do pick this up, there's also prior art over in https://github.com/dependabot/dependabot-core/pull/5562/ that may be useful. That PR also has some NotImplemented bits so neither one is completely ready, but that one does look to have some additional test cases and some features that haven't been completed yet in this PR etc. So worth a skim to see if any commits should be cherry-picked into this.

authors: FranciscoPereira apps4everyone elu-dawn hernangonzalez jeffwidman lucasilverentand mattt

repo: dependabot/dependabot-core


text:
Add unit test for docker DependencyFileNotFound

Add a unit test to guard against regressions resulting in bugs like https://github.com/dependabot/dependabot-core/pull/6142.
I didn't include this with the original PR because I was uncertain if it'd be affected by the removal of the kubernetes_updates feature flag in https://github.com/dependabot/dependabot-core/pull/6144.

This passes just fine locally in the dev container, not sure why https://github.com/dependabot/dependabot-core/actions/runs/3474850880/jobs/5808469712 failed with Errno::ENOENT:...

Oh silly me, I forgot to git add the new file...


text:
deploy:release improvement

In deploy:release there is logic called "Fix collisions". This fix actually breaks the expected result which is release_name to be as specified. If there is conflict it is more proper to report it before the actual release. I propose to add additional task deploy:release:verify before deploy:release. If there is problem the deployment should stop. Only release_name to be responsible for the generated release name.

No more "Fix collisions" logic.

authors: antonmedv kanevbg

repo: deployphp/deployer


text:
Tidy artisan commands

This PR aims to keep the artisan tasks from the Laravel recipe up-to-date with the framework. It also cleans any redundant or obsolete tasks. Here's the complete changelog.

Removed task deploy:public_disk. This task was redundant since this can be achieved with the artisan:storage:link task since 5.3. 5.2 support was dropped in June 2016 so it is safe to stop supporting it ourselves.
Removed tasks that published assets. Namely: horizon:assets, horizon:publish, telescope:publish and nova:publish. These commands are made to generate some code in our local environment that should then be committed and pushed. These should never be needed directly in production.
Removed the min and max constraints on the optimize and optimize:clear tasks. Whilst its implementation changed slightly at some point, the command was actually always available. I believe this was a mistake on my side.
Added the following artisan tasks to stay consistent with what we provide. For example, if we provide config:cache and config:clear, we should also provide route:cache and route:clear.

route:clear
route:list
horizon:clear
horizon:list
event:list

Added the following artisan tasks that could be of great benefit for handling queue jobs in production.

horizon
horizon:continue
horizon:pause
horizon:purge
horizon:status
queue:failed
queue:flushed

Changed the order in which tasks are defined. First, by grouping them into categories, then alphabetically.

This PR breaks backward compatibility but I thought since we're still in Beta, it was the perfect opportunity for a cleanup. 😊

[ ] Bug fix #…?
[ ] New feature?
[x] BC breaks? Yes, some tasks have been deleted but they were not meant to be used directly in production.
[ ] Deprecations?
[ ] Tests added?
[ ] Changelog updated?

Wow! Awesome work! And totally perfect timing to introduce BC. Will release a new beta right now.

Thanks! :) Hang on, I forgot to update the changelog, I'm creating another quick PR.

authors: antonmedv lorisleiva


text:
🛑 Derailed Home is down

In 9516c11, Derailed Home (https://derailed.one) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Derailed Home is back up in 009f144 after 325 days, 9 hours, 21 minutes.

repo: derailed-org/derailed-oss-status


text:
🛑 Derailed API is down

In 570318e, Derailed API (https://derailed.one/api) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Derailed API is back up in 874080f after 326 days, 5 hours.


text:
🛑 Derailed Home is down

In f9ce5c9, Derailed Home (https://derailed.one) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Derailed Home is back up in f832b7c.


text:
Mike: trying to merge resonance decay with spin polz

It couldn't auto merge

looks ok

authors: derekeverett mjmcnelis


text:
Improve the documentation, pull in README

This makes the docs a bit nicer, pulling in all the great information in the README

Thanks!!

Pushed the updated docs, looking much better now :+1:


text:
How to pass function to options?

If I try to pass a function I get an error, that the function is not serializable.
Could you please add an example on how to pass functions? Maybe as string?

Thanks for opening an issue @stefaneidelloth,
If You want to pass a none serializable object the only way you can do that right now is by converting it the a JSON string yourself and passing the JSON string. You can do that by either replacing all none serializable objects with something else like a string representation for that object, or you can use json.dumps with the default  argument like so: json.dumps([print], default=lambda o: repr(o)).
I will update jsoneditor to be able to do this automatically.

If I pass the function as string, I get the error

=> Is there a way to actually execute a passed callback without changing the JavaScript code?
=> I am wondering how you got the callback working that is directly passed as argument to jsoneditor.editjson and if that strategy would also work for the callbacks that are passed as option.
Below Is my current code. I would like to trigger the threading event for the case where I close the editor.
    def edit(self, json_data):

        result = None

        def data_changed(threading_event, adapted_json_data):
            nonlocal result
            result = adapted_json_data
            threading_event.set()

        event = threading.Event()
        callback_with_event = functools.partial(data_changed, event)
        change_callback = json.dumps([print], default=repr)
        options = {
            "onChangeJSON": change_callback  # "function(json){alert(json)}"
        }
        jsoneditor.editjson(json_data, callback_with_event, options)
        event.wait()

        if result is not None:
            return result
        else:
            return json_data

Sorry, I thought you tried to pass a function as part of the JSON data (like a dict with function in it)
I looked over your code, and from what I understand you want to launch a window with some JSON data, halt the execution until the user edited the data and pressed the button, and then you want to take the data and return it from your function.
To do the you don't need threading event's because the editjson function halts the execution by default when you pass a callback function, and when you are ready to continue you can press the x button (which is right next to the check button) and that will continue the execution of the program which in your case will return the edited version of the JSON.
Edited you code to do just that:
import jsoneditor

def edit(json_data):

    result = None
    def data_changed(adapted_json_data):
        nonlocal result
        result = adapted_json_data

    jsoneditor.editjson(json_data, data_changed)

    return result

print(edit([]))

Hope this helps.

authors: dermasmid stefaneidelloth

repo: dermasmid/py-jsoneditor


text:
Camera unavailable + code showing

frigate 0.13.2-6476F8A
frigate Lovelace card v5.2.0
Since v13 and v5.2.0, I am having Camera unavailable + code (camera config) showing instead of "no frames received, etc.." (Frigate error) as in previous versions when the camera is not being picked up by Frigate (off or unavailable on network).
Now this might be more of a feature request, however it would be awesome if you can have options how to deal with cameras that are unavailable, being:

static picture
last image + with custom text in the middle
leave cameras that are unavailable out of the line up

Basically, I tend to switch off certain indoor cameras based upon the status of home (armed, disarmed).
Anyways thanks for the awesome work on this card!!!

I'm having the same issue. Presumably because I use wi-fi cameras, so they are not as stable as wired ones.

I'm having an issue with this to. I have a Frigate card with multiple cameras on it and if one camera has connection issues the entire card and all its cameras stop working. Feels like this shouldn't be the case.

same issue here. upgarded from working 0.13 to 0.14 beta, cards stopped working

This is already fixed to some degree on the dev branch, so feel free to test that out (no support offered, though): https://card.camera/#/advanced-installation?id=unreleased-versions .
I like the idea of having a configurable option of how this is treated as described in the original post on this issue, although some of those are easier than others (e.g. having a local copy of the last usable frame might be great, but would involve saving every frame which would have a performance impact -- but others are more doable).

The new beta build (https://github.com/dermotduffy/frigate-hass-card/releases/tag/v6.0.0-beta.2) will not block the whole card if a single camera goes unavailable. Please let me know if that is sufficient, or if you think the most advanced ideas described at the start are still worth it.

authors: DReffects GiowGiow Lockie85 dermotduffy nibagit

repo: dermotduffy/frigate-hass-card


text:
Let's define image icon

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
I setup th card and would like to use th status image as a privacy function. Bit the icon for the static icon is a picture and I would like to change this icon .any idea or do I have to add an icon myself with a button that calls the function to show the static image ?
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

There's no way to do that with the built-in default menu items. Your best bet (as you guessed) would be to do something like this:
elements:
  - type: custom:frigate-card-menu-icon
    icon: mdi:butterfly
    tap_action:
      action: custom:frigate-card-action
      frigate_card_action: image

There are two downsides to this:

With the in-built menu icons, they are styled/emphasized  when that view is active (e.g. click on 'Clips' and notice how the color of the icon will change to show it is selected). That would not happen with the above idea.
Support for custom:frigate-card-action is not in any released build, so you'd need to wait for the next one.

Then I wait  for next build or does Dev version already work with that action?

You'd need to wait for the next beta build.

Going to close this for now, but let me know if it doesn't work for you once released!

authors: dermotduffy thundergreen


text:
🛑 heise.de is down

In f4e2c42, heise.de (https://heise.de) was down:

HTTP code: 0
Response time: 0 ms

Resolved: heise.de is back up in 3b2b019.

repo: desisn/upptime


text:
🛑 tips4.app is down

In d8c6080, tips4.app (https://tips4.app) was down:

HTTP code: 0
Response time: 0 ms

Resolved: tips4.app is back up in dda4f28.


text:
PIL.UnidentifiedImageError

Hi, I am trying to run the command:
python thermal_analysis.py <flir_image_path>
but its throwing this error:
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x00000232575A1B88>
I did run requirements.txt and do the following:
Download exiftool ,
Extract 'exiftool(-k).exe' to the folder where I'm running the code from,
Rename it to 'exiftool.exe',
what do I do? Any kind of suggestions will be helpful. Thank you.

I assume you're running it in Windows. If possible, can you share your flir file?
Haven't seen this error before.

Thank you for your reply.Yes,I'm running it in Windows.This is my  flir file.

Can you upload the original file? I wanted to test it with the program. This seems to be a normal jpg image, not a FLIR thermal image.

Sorry, there was a little error in the picture I sent before. Now I sent the picture in PNG format which I collected from the laboratory through the infrared thermal imager.

What camera did you use to take this image?
While it is a 'thermal image', it doesn't contain any of the required metadata to use with this software, eg. raw sensor values.

        OK, I see. There is a problem with my infrared thermal image, which leads to a program error. Well, if you can, could you please send me an infrared thermal image of a person's face that can be used for my follow-up study? Thank you very much for your time to help me solve the problem.

                            15227887681

                                ***@***.***

    Signature is customized by Netease Mail Master

        On 04/03/2021 19:49, rakshith95 wrote:

What camera did you use to take this image?
While it is a 'thermal image', it doesn't contain any of the required metadata to use with this software, eg. raw sensor values.
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
@.": "http://schema.org",
@.": "EmailMessage",
"potentialAction": {
@.": "ViewAction",
"target": "https://github.com/detecttechnologies/Thermal_Image_Analysis/issues/6#issuecomment-812854455",
"url": "https://github.com/detecttechnologies/Thermal_Image_Analysis/issues/6#issuecomment-812854455",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
@.": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

I don't have any unfortunately. You could try the FLIR forums or reddit or something. Closing this issue


text:
OptionalWhenBraces false negative for nested when wrapped in braces

Using 1.18.1
The following code does not fail for OptionalWhenBraces:
fun main() {
  when(Random.nextBoolean()) {
    true -> {
      when(Random.nextBoolean()) {
        true -> {
          println("true")
        }
        false -> {
          println("false")
        }
      }

      println("end")
    }
  }
}

edited to simplify repro

@BraisGabin looks like there's either been a regression or another case that wasn't handled:
import android.icu.text.PluralRules
import kotlinx.serialization.json.JsonObject
import java.util.Locale

fun JsonObject.plural(
  plural: Number?,
  language: String = "en"
) = when(plural) {
  null -> get("other")

  else -> {
    when {
      "other" in this -> get(
        PluralRules.forLocale(
          Locale.forLanguageTag(language)
        ).toString()
      )

      // other must be present even if a present rule would match
      else -> null
    }
  }
}

Can you open it as a new issue? You could link there this issue. It doesn't seem like a regression because the fix had a test to avoid that regression. But it's a clear false negative. And thanks for the report, the false negatives are difficult to spot :)

authors: BraisGabin eygraber

repo: detekt/detekt


text:
OutdatedDocumentation should allow us to use @param even when constructor properties have val

Expected Behavior of the rule
Following code:
/**
 * @param firstParam Documentation for the first param
 * @param secondParam Documentation for the first param
 */
class MyClass(val firstParam: String, val secondParam: String)

fails the OutdatedDocumentation rule. Rule expects that all val constructor parameters are marked as @property instead.
But there is really no benefit to using @property that I can see.
Context
Main drawback of using @property is that there is no documentation when calling constructor of the class:

As opposed to @param:

However, even when using @param, IntellIJ still populates the property javadoc, so using @param actually gives us the best of both worlds:

Semi-related to the https://github.com/detekt/detekt/issues/4365

This looks more like a bug in KDoc, given that marking even one property as @property will cause the document generator to not display the Params: list. I've created an issue in YouTrack for Kotlin so that the language's authors can give feedback on this.

Regardless, I still don't see why @param is considered invalid here? Both @param and @property would be equally valid here (since this constructor represents both).

It's invalid because firstParam and secondParam are properties, not parameters, given they possess the val keyword. If this is an issue with KDoc - and I believe it is, given that marking one property as @property and another as @param produces the same lack of rendering in the KDoc popup - I would wait to see what Jetbrains says about whether they agree that this is an issue before creating a potential workaround, as once that workaround is in, it'll have to be maintained.

Well, at the same time, they are parameters, because they are part of the constructor.

Per KDoc's documentation:
@property name
Documents the property of a class which has the specified name. This tag can be used for documenting properties declared in the primary constructor, where putting a doc comment directly before the property definition would be awkward.

Yes, it's physically possible to mark a property in the constructor with @param, but that applies to lots of code that is flagged by other rules in Detekt as well, and I'm not seeing how this is different, especially given Jetbrains hasn't weighed in on the potential KDoc rendering issue yet.

I agree with severn here. It seems like a bug in the Jetbrains side. @property It feels more natural to describe the property instead of the param and the documentation say that it is there for that reason. Let's wait to get an answer in the other issue.

I agree with both of you but I just want to mention that answer from Jetbrains might take a while. In the meantime, the issue persist. If they do not answer in a meaningful time frame, I would opt for adding allowParamOnConstructorProperties option to the OutdatedDocumentation that allows this (again, I'm prepared to make PR for this).

Just got a response from the JetBrains team: this was already identified as an issue in the IntelliJ plugin... three years ago. It doesn't look like there's any movement on fixing it anytime soon, so maybe we should allow the option like @matejdro suggests.

I would opt for adding allowParamOnConstructorProperties option to the OutdatedDocumentation that allows this (again, I'm prepared to make PR for this).

Given that a fix from JB side would take some time, I believe that allowParamOnConstructorProperties sounds like a good compromise. If you end up making a PR @matejdro, please add a note with a reference to the YouTrack issue.

authors: BraisGabin cortinico matejdro severn-everett


text:
UnnecessaryParentheses: fix false positive for delegated expressions

Fixes #3851

This rule delegates all the heavy lifting on KtPsiUtil.areParenthesesUseless(expression). Shouldn't we open an issue to JetBrains to fix it?
I'm not sure if this is an issue in areParenthesesUseless or even the compiler. I don't see why those parentheses are needed.

I'm not sure if this is an issue in areParenthesesUseless or even the compiler. I don't see why those parentheses are needed.

True. I've created an issue here https://youtrack.jetbrains.com/issue/KT-47159
Will try to ask on Slack if someone has insights on this.

authors: BraisGabin cortinico


text:
IgnoredReturnValue: add option returnValueTypes to enable rule for particular types

@BraisGabin said:
We could extend IgnoredReturnValue and make it receive a list of types from the configuration. The rule will flag any call to a function that returns one of those types if the returned is ignored.

This PR adds config option returnValueTypes (is this option name OK?) to make it possible to enable the rule for functions returning specified types.
Also, I've added kotlinx.coroutines.flow.Flow to the default config.
Resolves #4640

By looking at this rule API, I think we should add also the restrictToReturnValueTypes boolean config.
  IgnoredReturnValue
    active: true
    restrictToAnnotatedMethods: true
    returnValueAnnotations:
      - '*.CheckResult'
      - '*.CheckReturnValue'
    ignoreReturnValueAnnotations:
      - '*.CanIgnoreReturnValue'
+   restrictToReturnValueTypes: true
+   returnValueTypes:
+     - 'kotlinx.coroutines.flow.Flow'

The rule should then check if restrictToReturnValueTypes is true and the returned type is one of the listed -> Then report.
It makes for a more explicit configuration and it aligns to restrictToAnnotatedMethods/returnValueAnnotations

By looking at this rule API, I think we should add also the restrictToReturnValueTypes boolean config.
The rule should then check if restrictToReturnValueTypes is true and the returned type is one of the listed -> Then report.
It makes for a more explicit configuration and it aligns to restrictToAnnotatedMethods/returnValueAnnotations

Thank you for API review!
Yes, i thought about it, but prefix restrictTo implies "exclusiveness" isn't it? In my mind restrictToAnnotatedMethods means "this rule will check only methods annotated with the specified annotations", and restrictToReturnValueTypes means "this rule will check only menthods returning the specified types", so it may be confusing if both of these flags are true.
But maybe it is only my feeling because of language barrier.

Yes, i thought about it, but prefix restrictTo implies "exclusiveness" isn't it? In my mind restrictToAnnotatedMethods means "this rule will check only methods annotated with the specified annotations", and restrictToReturnValueTypes means "this rule will check only menthods returning the specified types", so it may be confusing if both of these flags are true.

Yes that's correct. If enabled, the rule will flag only function which are returning the values types you listed. If this is not the case, then I misunderstood what was the semantic of your change.

Yes that's correct. If enabled, the rule will flag only function which are returning the values types you listed. If this is not the case, then I misunderstood what was the semantic of your change.

I mean sentence "rule will check only annotated methods" conflicts with "rule will check only methods returning the specified types" because both contain word "only". So maybe there may be better naming? Something that means "always check methods returning specified types even if checks enabled only for annotated methods"?

So maybe there may be better naming? Something that means "always check methods returning specified types even if checks enabled only for annotated methods"?

I would suggest to don't implement this logic. The two restrict should apply on top of each other and further restrict the set of return values we inspect. What you're suggesting instead seems harder to explain and understand for users IMHO

🤔 I think that we should reframe the configuration of this rule. To me this issue should flag a function if it returns a Flow (or any other class listed on the configuration) or if it is annotated with some of the listed annotations. Then, we should have another flag that would ignore all those configurations and will flag all the cases where the return value is not Unit nor Nothing. Do that have sense?
To do that I would deprecate the current boolean flag and rename it to some naming more clear.

Sorry for ping, but should it be merged?

Thank you for the PR :)

authors: BraisGabin cortinico osipxd


text:
.NET: Support debian-12 (bookworm)

Depends on https://hub.docker.com/_/microsoft-dotnet-sdk/

[ ] Open a PR
[ ] Create a release
[ ] Update templates if necessary
[ ] Update https://github.com/devcontainers/images/issues/90

Closing as completed!
📣  Released a new image version for dotnet with bookworm support. We would recommend pinning to newer major version 1 for continuing receiving security updates.

repo: devcontainers/images


text:
Add dos2unix to the dockerfile to deal with CRLF and LF issues.

Thanks for the great work!
I fixed issue #4.
I added dos2unix to the dockerfile to deal with CRLF and LF issues to provide cross-platform (Windows and Linux) support.

I think issue #4 could be closed.

repo: devecchijr/apiPub


text:
Where is the source code?

Where is the source code?

bg remover is not open source yet!

repo: developersharif/bgremover-app


text:
Service Worker or No Service Worker by default

Let's discuss:

Do we need SW on watch by default?

Do we need SW on build by default?

Things worth mentioning:
1- We have over 20 issues both open and closed on here (at least) with Service Worker problem and or how to disable it.
2- We don't have any tests for SW builds which can lead to a slight risk especially if the cli user was not aware or know enough about SW.
3- I'm a big SW & PWA fan and I run SydPWA Meetup & use it in all my websites.
My openion:

I want SW on watch, but only when I need to, I build a apps that fetch data from Cache first and need SW to cache responses, but that's only 5% of my Preact CLI apps and experiments and I'd like to be in no SW mode first until I decide my app needs it and I opt in with a flag in npm watch script.

in prod, I want to opt in also to get it, forcing SW on people is not a way to make the web faster and teach them about PWAs.

More soon when I'm not on mobile.. I want to hear everyone's opinion first.

Watch -- no
Build -- yes

An easy middle ground, for me, is preact watch --sw which build and attaches a service worker  for those who want it and/or are aware of it.
It should always be on & attached for production builds imo. This is a main pillar of PWA so must be there. The --no-sw will remain for those who don't want it for a particular reason.
Thanks for opening the debate! Let's see what the others / anyone has to say! 🎉
Looking for community feedback.

If we going to allow --no-sw in prod we should have something instead to unregister old one & clear the cache.. this is another real users issue.
Tests with puppeteer is recommended too.

Yes, this is mentioned in #420

Here are my 2 cents on this.
In dev mode (preact watch), I wouldn't cache the assets since wherever some changes made in the JS files, they will not reflect in the UI until you hard refresh because the SW strategy is cache first in CLI.
As @lukeed suggested, we can provide a flag --sw flag to preact watch command if the user wants the SW to be loaded in the development mode.
IMO, it would be better to change the strategy of the SW to network first while running preact watch for dev mode (this way we can avoid adding an extra flag --sw).

We can do the following while in dev mode:

Check for SW: if exists remove & unregister it)
Then leave the rest of the process as it is.

We always have to ship a SW file.
If preact watch --sw: file content are our SW stuff.
If preact watch: sw file content is SW unregisterar stuff.
Same for prod in my opinion.

Given that is always in prod. I wonder if disabling SW in dev might lead to devs freaking out about why their prod apps look stale?

I am supportive of no-sw behind a flag in watch mode. Cuz while it makes it comparable to the way of building your web app pre-PWA era, you start missing out on testing runtime strategies for your APIs.
Building offline UX for my web apps is something that my dev environment should support first class.
Its your choice if u don't want SW and hence we should provide a flag, but an opt-in flag might just paint the picture of offline UX as an after thought.
Already a bunch of best in the class players just support loading shell in offline mode and fail miserably in showing actual data.
Also

If we going to allow --no-sw in prod we should have something instead to unregister old one & clear the cache.. this is another real users issue.

This shouldn't be the concern of this flag at all. Managing your service workers should be a separate thing, may be a plugin but not this flag.

Closing this as Luke submitted a fix for this.

authors: elisechant lukeed prateekbh reznord zouhir

repo: developit/preact-cli


text:
Does it support vsis3?

I'm getting this error while passing a s3://bucket-name/AB000a12_001_MUL_L1T_COG3.tif in the api url:
{"errorMessage": "'/vsis3/bucket-name/AB000a12_001_MUL_L1T_COG3.tif' not recognized as a supported file format."}
thanks.

Hi @adrianmelero it should work. just make sure to add bucket-name in https://github.com/developmentseed/cogeo-tiler/blob/master/serverless.yml#L21-L27

Hi @vincentsarago yes, It is added, but still not working when passing url s3 object to the API

It was a problem of permissions, I have added Access Key and Secret Key to Dockerfile and now its working, thanks!!!

authors: adrianmelero vincentsarago

repo: developmentseed/cogeo-tiler


text:
Feature validates arbitrary dictionaries

Hi @vincentsarago, I stumbled across some unexpected behaviour in these pydantic models and thought I'd ask for guidance here!

Feature seems to validate arbitrary dictionaries:

>>> geojson_pydantic.Feature.parse_obj({"hi this is": "not a feature"})
Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None)

This seems kind of counter-productive, is this intentional? At least judging from the geojson json-schema, id should not be optional!

Feature validates the list ["vh", "vv"], but not the list ["not", "feature"]. This is causing problems in my application, because the list of bands ["vv", "vh"] is incorrectly parsed into a GeoJson, even though it's just a list.

>>> geojson_pydantic.Feature.parse_obj(["vh", "vv"])
Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None)

>>> geojson_pydantic.Feature.parse_obj(["not", "feature"])

ValueError Traceback (most recent call last)
File .venv/lib/python3.9/site-packages/pydantic/main.py:522, in pydantic.main.BaseModel.parse_obj()
ValueError: dictionary update sequence element #0 has length 3; 2 is required
The above exception was the direct cause of the following exception:
ValidationError Traceback (most recent call last) Cell In[4], line 1 ----> 1 geojson_pydantic.Feature.parse_obj(["not", "feature"])
File .venv/lib/python3.9/site-packages/pydantic/main.py:525, in pydantic.main.BaseModel.parse_obj()
ValidationError: 1 validation error for Feature root Feature expected dict not list (type=type_error)

This seems super strange too, do you have any ideas on why that specific list would validate and not the other? I have not been able to find another example.

I've run all these in a fresh virtual environment (Python 3.9.5) with only geojson_pydantic installed. Versions are the following:
print(geojson_pydantic.__version__)
print(pydantic.__version__)
>>> 0.5.0
>>> 1.10.2

Thanks!

That is indeed really weird and I don't think is what we expect!

Feature seems to validate arbitrary dictionaries:

I believe this is because the pydantic default is to ignore additional fields.  Pydantic checks if the hi is this field is on the model, it isn't, so the value is ignored.

Feature validates the list ["vh", "vv"], but not the list ["not", "feature"]. This is causing problems in my application, because the list of bands ["vv", "vh"] is incorrectly parsed into a GeoJson, even though it's just a list.

The BaseModel.parse_obj class method will coerce the object to a dictionary:
>>> dict(["vh", "vv"])
{'v': 'v'}

>>> dict(["not", "feature"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: dictionary update sequence element #0 has length 3; 2 is required

The first case works for the same reason, the v field is not on the model so pydantic ignores it.  The error  in the second case is because parse_obj is breaking when coercing the object to a dictionary, it doesn't even hit validation yet because the Feature class is never instantiated.
You can change the global behavior of pydantic to tell it to forbid extra fields (https://docs.pydantic.dev/usage/model_config/#change-behaviour-globally), but I think it would be a good idea to enforce this in geojson-pydantic.  I'm not really sure what to do about the second case.

but I think it would be a good idea to enforce this in geojson-pydantic. I'm not really sure what to do about the second case.

@geospatial-jeff not allowing extra could be a big breaking change and looking at https://datatracker.ietf.org/doc/html/rfc7946 I don't see why extra wouldn't be allowed.
I'm not sure if it's because with the latest changes but, to me, I believe that even if null you would have to set the geometr y in the feature
Feature.parse_obj({"type": "Feature", "geometry": None})
>> Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None)

# IMO The following shouldn't work but they do
Feature.parse_obj({"type": "Feature"})
>> Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None)

Feature.parse_obj({})
>> Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None)

ping @eseglem (ref: https://github.com/developmentseed/geojson-pydantic/pull/87/files#r939570703)

This does feel like it is more one of the flaws of current pydantic than the library itself. Unfortunately pydantic does some weird stuff like this. Pydantic v2 should hopefully fix a lot of it.
Are you able to provide a little more context on how that list ends up getting passed in? What does the data look like where a List or a Feature could be at the same level? Pydantic is order dependent, and will pick the first valid option unless you tell it to check every type first.
from typing import Union, List
from pydantic import BaseModel
from geojson_pydantic import Feature

class Foo(BaseModel):
    bar: Union[List[str], Feature]

Foo.parse_obj({"bar": ["vv", "vh"]})
# Foo(bar=['vv', 'vh'])
Foo.parse_obj({"bar": {}})
# Foo(bar=Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None))

class Foo(BaseModel):
    bar: Union[Feature, List[str]]

Foo.parse_obj({"bar": ["vv", "vh"]})
# Foo(bar=Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None))
Foo.parse_obj({"bar": {}})
# Foo(bar=Feature(type='Feature', geometry=None, properties=None, id=None, bbox=None))

I also believe extra should be allowed, so I don't think that would be the right route.
One thing that could be done in this library is moving those to geometry: Union[Geometry, None] = Field(...) and require the key actually be specified. That would eliminate the ability to do Feature() which seems like it should be allowed. It may be less bad though?

I kinda agree that we should do geometry: Union[Geometry, None] = Field(...)

That sounds good to me

Are there any other fields (across all the models) where it would make
sense to do the same? I think there are probably many instances where an
input that technically isn't valid GeoJSON would be accepted by
parse_obj. As it stands you can do Feature.parse_obj({"geometry": None}) which is less bad, but still technically not valid because the
"type" is missing. Would need to switch to something like type: Literal["Feature"] with no default to be truly strict about the input
parsing.
Personally, I like having defaults since I mostly use the library to build
and manipulate my own data. I may do feature = Feature() and then build
on it from there. But I also think the fact that it will accept a list
without error is wrong. I probably need to add a test to some of my other
code to see if this would cause issues for API input.  And I don't think
there is a great way to balance the two.
The dict coercion happens
https://github.com/pydantic/pydantic/blob/1.10.X-fixes/pydantic/main.py#L521
and I think it mostly makes sense from their side. But apparently dict()
on a list with only two character strings is a very interesting edge case.
I am trying to think about if there is some other way to not break there.
Obviously could override parse_obj but that would probably introduce more
issues. Trying to figure out if there is a way to enforce something along
the lines of "if (input is truthy) and (no overlap in keys): ValueError" So
if you pass in something that is dict like, or coercible to a dict, but
none of the keys are right, break. Seems like that should be an option
somewhere, but haven't found anything.
Would a strict_parse_obj function make any sense? Probably not the best
idea. Seems a bit rough too. Lots of other discussion here:
https://github.com/pydantic/pydantic/issues/1098 but I haven't been able to
read all of it yet.
On Thu, Jan 26, 2023 at 10:17 AM Jeff Albrecht @.***>
wrote:

That sounds good to me
—
Reply to this email directly, view it on GitHub
https://github.com/developmentseed/geojson-pydantic/issues/92#issuecomment-1405165428,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABNFN3DRGNJSBJGNAO2JK43WUKIRDANCNFSM6AAAAAAUGGULVA
.
You are receiving this because you were mentioned.Message ID:
@.***>

I've started a PR over #94
I understand this would have a lot of implication (requiring Type, Geometry and Properties) but I'm happy to discuss it more 🙏

authors: LukeWeidenwalker eseglem geospatial-jeff vincentsarago

repo: developmentseed/geojson-pydantic


text:
Fail more descriptively when time is not found

A request like
https://dev-titiler-cmr.delta-backend.com/tiles/WebMercatorQuad/0/0/0@1x?datetime=2024-10-20T22%3A00%3A00.000Z&resampling=bilinear&variable=precipitation&colormap_name=gnbu&rescale=0%2C46&concept_id=C2723754864-GES_DISC&backend=xarray
where the dataset only has time steps until say 2022-01-01
currently fails with a 500 error and
{"detail":"list index out of range"}

Error code and message could be more descriptive when the requested datetime is not found.

Thank you for opening this @j08lue - I'm going to move it to titiler-cmr

repo: developmentseed/titiler-xarray


text:
ThreadPool for StoreInput Validations

As part of #108 .
In this PR, I'm creating a ThreadPool that uses the task manager to N number of tasks for store_inputs validation.
Potential Changes:

Maybe we move the thread pool into the task manager and have certain use cases define their function signature?

Closing this, as it was just for POC


text:
Implement the OCI registry update functionality

/kind task
Which area/kind this issue is related to?
/area registry
Issue Description
The OCI registry should be up-to-date with the main branch of the provided env var registry repo URL. As a result, a task should be able to update the OCI registry artifacts whenever a trigger is sent to the webhook of the index/server.
The task should receive information about the:

commit_hash of the latest commit that we are updating the OCI registry.
the list of affected stacks that the task should look to update.

The task should update the content of the artifacts for the affected stacks and regenerate the index.json (with the new commit_hash inside).
Finally, a new endpoint should be created inside the index/server. This endpoint should only interact with github repo and trigger every new update of the oci-registry artifacts. Note that this endpoint should have an authentication method in order to be sure that the only requester will be the github repo.
Acceptance Criteria

[ ] Add functionality to be able to update specific artifacts inside the oci-registry,
[ ] An endpoint exists so we can receive webhooks from github regarding any updates of the stacks.
[ ] The endpoint is reachable only if the related env var (see https://github.com/devfile/api/issues/1507) is set.

Note the webhook on the index/server should be able to support two types of triggers from github:

Github sends a simple trigger to the webhook after a github event. The index/server checks the updated code for any affected stacks.
Github sends a trigger with information about affected stack (need to decide the format) so index/server updates directly the given updated stacks.

repo: devfile/api


text:
feat: Don't set CPU limits when possible

What does this PR do?
https://github.com/eclipse/che/issues/22198
What issues does this PR fix or reference?
feat: Don't set CPU limits when possible
Is it tested? How?

PR Checklist

[x] E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)

[x] v8-devworkspace-operator-e2e: DevWorkspace e2e test
[x] v8-che-happy-path: Happy path for verification integration with Che

@amisevsk
I am not sure about command to fix resources. Could you help me?

To regenerate all resources:
make generate manifests generate_default_deployment generate_olm_bundle_yaml

the steps are

Generate deepcopy/crd resources
Generate deployment manifests (update RBAC, etc. -- that's the change that's missing from this PR)
Generate OpenShift/Kubernetes depoyments (run kustomize)
Generate OLM bundle

Feel free to add a make generate-all rule to make this simpler in the future -- I just haven't gotten around to it since the combination above is in my shell history :)

I think there are too many issues with this approach to enable this functionality. A better approach, though less convenient, may be to allow configuration of the default container resources, to allow for specifying empty CPU limits, similar to what we have for the project clone init container, or to not set a default CPU limit (by default) and require configuration on clusters with limit ranges.

Closing PR as out of date.

authors: amisevsk tolusha

repo: devfile/devworkspace-operator


text:
Can't secure the docker-elk with TLS

Problem description
I want to enable the docker-elk with TLS encryption. I have followed the steps about generating the certificates (as decribed in  tls/README.md) and i have modified the corresponding .yml files according to the .yml files in tls folder in the repository. What am i missing? Any help would be appreciated.
$ docker-compose logs

kibana_1         | {"type":"log","@timestamp":"2021-06-22T12:43:18+00:00","tags":
                   ["info","http","server","Kibana"],"pid":953,"message":"http server running at http://0.0.0.0:5601"}

@georgios-kalomitsinis the tls branch enables secured communications between Elastic components, but not to Kibana.
For instance:

communications from Kibana to Elasticsearch are secured ✔
communications from your web browser to Kibana are not secured ❌

The reason for not enabling TLS communications to Kibana by default is because your web browser would reject the unknown certificate. For this setup to work, we would have to instruct users to either

add the generated CA certificate to their web browser so its "trusted"
resort to a third-party service such as Letsencrypt to generate server certificates from a CA that is already trusted by the browser

Both require some extra manual steps, which would void the works out-of-the-box approach we've taken so far.
Let me know if you have suggestions to solve this in a reasonable way, or if your need guidance on how to configure Kibana to enforce TLS connections.

Closing due to inactivity. Feel free to reopen in case this issue needs more clarification.

For posterity, there is now a new section inside tls/README.md which explains how to enable TLS in the Kibana server (between web browsers and Kibana), as well as a sample certificate and key which Kibana is pre-configured to use when TLS is enabled.

authors: antoineco georgios-kalomitsinis

repo: deviantony/docker-elk


text:
Question about the gpl license of WUT

Hi
from what i'm aware of, it seems like WUT, contrary to ctrulib for 3ds got licensed under the terms of GPL instead of zlib.
this for me raises some questions.
For instance, lots of headers from WUT solely define macros and structs to interact with the native wii u system library which would not result in any object files from WUT ending up in the final binary as long as the binary doesn't get linked to the WUT lib.
This makes me wonder if end binaries made this way would fall or not under GPL, as whb seems to be where most of the original implementations reside.
It is important for me to find ways to interact with the Wii U's system libraries without being restrained by GPL as what i'm working on is a game engine meant to be compatible with most consoles supporting homebrews as well as the more standard PC/mobile platforms which is why gpl wouldn't be an appropriate licence for my project
Do you have any informations on the matter ? because if that's not possible the wii u would end up being a console without any support from my engine

Changed to zlib in f6b123da1817a54ab7a12ad63e08d2fb1c96646d.

authors: SeleDreams fincs

repo: devkitPro/wut

license: Zlib


text:
fix: renew email regex, because the old regex is allow the special ch…

…aracter on the name field
Fixes Or Enhances
Make sure that you've checked the boxes below before you submit PR:

[ ] Tests exist or have been written that cover this particular change.

@go-playground/validator-maintainers

Gar ini kan kita fork dari golang validator, ikutin mereka punya standard sih. Btw regex yang baru ini sudah confirm OK ga terutama di produk yang lain misalnya prime.

hmm klo gitu aku bikin validasi baru aja khusus hi, nanti aku tambahin di netmonk libarary aja

repo: devnetmonk/validator


text:
Add an experience system for quizzes + track user's long-term word success rate

Expand the 'Account' button to a new view with information about how many quizzes have been taken and the user's overall word success rate.

Decided a user engagement feature like the one proposed in this issue did not best serve the intended purpose of the app.


text:
Throw exception when needed

The ts-merger lacked a proper exception handling. This PR fixes that problem.
@devonfw/cobigen

Seems that production line is broken. I have built this on a Linux machine and all tests are passing, therefore I will merge it.

repo: devonfw/tools-cobigen


text:
Fix invalid cross-device link on linux devices

There is a problem using the Flexible Config on Linux (on Mac works ok).
We can see that this lints (it's a valid file):

docker run -it -p 8080:8080 -v $PWD:/etc/krakend/ devopsfaith/krakend check -d -c "/etc/krakend/config/krakend.json"
But if we add the flexible config env vars (with empty folders) this crashes:

docker run -it -p 8080:8080 --env FC_ENABLE=1 --env FC_SETTINGS="/etc/krakend/config/settings" --env FC_PARTIALS="/etc/krakend/config/partials" --env FC_OUT="/etc/krakend/configuration.json" -v $PWD:/etc/krakend/ devopsfaith/krakend check -d -c "/etc/krakend/config/krakend.json"
This is due to a problem with the os.Rename (same as https://github.com/jenkins-x/jx/issues/449).

great catch!!!

authors: kpacha rgomezcasas

repo: devopsfaith/krakend-flexibleconfig


text:
The deprecation of github.com/auth0-community/go-auth0 breaks build

As per https://community.auth0.com/t/community-repo-deprecation-auth0-go/64665 , it was said that this line https://github.com/devopsfaith/krakend-jose/blob/90b8fe32a9fd493ded8e74d1fbcaddd0e8674ffb/go.mod#L6 has been deprecated since long ago by the new https://github.com/go-auth0/auth0 , last week IIRC the deprecated repo was still available so no error happened, but as of today the deprecated repository (github.com/auth0-community/go-auth0) seems to have return 404 (looks like the repo has been deleted in the end).
Do you accept a PR for this matter?

edit: the latest repository was here https://github.com/auth0/go-auth0

hi @gunturaf . we were aware of the deprecation notice but we did not keep track of the date (the process of releasing v2 was very demanding). I think the easiest way to fix this could be to publish a fork of the original repo at our github organization until we can invest the resources required for dropping the dependency and moving to the new lib.
once the fork is published, a single "replace" line on the go.mod of the krakend-jose and krakend-ce projects should be enough to keep everything up and running

Do you accept a PR for this matter?

of course. all contributions are welcome!

Mirror created at https://github.com/devopsfaith/go-auth0
Replace added to this project and to the CE itself
cheers!

authors: gunturaf kpacha

repo: devopsfaith/krakend-jose


text:
Proposal: View all supported plugins with the list command

Description
View all supported plugins by list command, and support group display

Great idea! @summingyu
It's a good idea to get the names of all dtm supported plugins directly from the command line. Because it is really cumbersome to query which plugins are available through the documentation.
Since there are not many plugins at present, I recommend outputting all the plugins' names directly without grouping.
When there are more plugins are supported in the future, we could implement group output at that time, even output all plugins' names of a group by specifying the group name like --group=ci or --kind=ci.
Go back to the implementation, I briefly thought about it, maybe we can design the command like:

dtm list-plugins

Maybe we can use the dtm list-plugins --kind=ci at future to support group output.
Of course there are many other options, I also list them for consideration

dtm list plugins
dtm plugins list

/cc @devstream-io/pmc

Great idea @summingyu
list-plugins, list-plugins --kind=ci look quite natural to my eyes.
Let's go!

/assign @summingyu

authors: IronCore864 daniel-hutao summingyu

repo: devstream-io/devstream


text:
⚠ Mobitopup has degraded performance

In 8190df2, Mobitopup (https://napcard.voz.tech) experienced degraded performance:

HTTP code: 200
Response time: 6410 ms

Resolved: Mobitopup performance has improved in 2c67ac2 after 416 days, 8 hours, 38 minutes.