text:
[SPARK-31801][API][SHUFFLE] Register map output metadata

This is a copy of #28618 but merged with the current master resolving all the merge conflicts.
All the credit goes to @mccheah I just would like to help out here and avoid his progress to be lost.
What changes were proposed in this pull request?
Adds a ShuffleOutputTracker API that can be used for managing shuffle metadata on the driver. Accepts map output metadata returned by the map output writers.
Requires #28616.
Why are the changes needed?
Part of the design as discussed in this document, and part of the wider effort of SPARK-25299.
Does this PR introduce any user-facing change?
Enables additional APIs for the shuffle storage plugin tree. Usage will become more apparent when the read side of the shuffle plugin tree is introduced.
How was this patch tested?
We've added a mock implementation of the shuffle plugin tree here, to prove that a Spark job using a different implementation of the plugin can use all of the plugin points for an alternative shuffle data storage solution. But we don't include it here, in order to minimize the diff and the code to review in this specific patch. See #28902.

Test build #132771 has started for PR 30763 at commit 5cb3e9d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37373/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37373/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37373/

Test build #132771 has finished for PR 30763 at commit 5cb3e9d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132771/

Test build #132783 has started for PR 30763 at commit 55c03a9.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37385/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37385/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37385/

Test build #132783 has finished for PR 30763 at commit 55c03a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132783/

cc @holdenk, @viirya, @mridulm, @shadowinlife, @tgravescs, @tianczha

cc @jiangxb1987 and @Ngone51 too FYI

Test build #133278 has started for PR 30763 at commit b468869.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37876/

Test build #133278 has finished for PR 30763 at commit b468869.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133278/

Test build #133942 has started for PR 30763 at commit a09ced3.

Test build #133942 has started for PR 30763 at commit a09ced3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38530/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38530/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38530/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38530/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38530/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38530/

Test build #133942 has finished for PR 30763 at commit a09ced3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133942 has finished for PR 30763 at commit a09ced3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133942/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133942/

Test build #134522 has started for PR 30763 at commit 88d370e.

Test build #134522 has started for PR 30763 at commit 88d370e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39107/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39107/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134522/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134522/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39107/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39107/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39107/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39107/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39110/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39110/

Test build #134525 has started for PR 30763 at commit 6f571a2.

Test build #134525 has started for PR 30763 at commit 6f571a2.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39110/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39110/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39110/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39110/

Test build #134525 has finished for PR 30763 at commit 6f571a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134525 has finished for PR 30763 at commit 6f571a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134525/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134525/

Test build #134568 has started for PR 30763 at commit b350258.

Test build #134568 has started for PR 30763 at commit b350258.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39154/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39154/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39154/

Test build #134568 has finished for PR 30763 at commit b350258.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134568 has finished for PR 30763 at commit b350258.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134568/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134568/

Test build #135856 has started for PR 30763 at commit abcf8f3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40438/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40438/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40438/

Test build #135856 has finished for PR 30763 at commit abcf8f3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135856/

Failure is totally unrealted:

org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceWithAdminSuite.subscribing topic by pattern from latest offsets (failOnDataLoss: false)

Let me move the mima excludes from 3.1.x to 3.2.x.

@Ngone51
Regarding cutting this to smaller pieces I can identify two potential sub-PRs:

introduction of MapTaskResult
introduction of ShuffleOutputTracker

I can do this cut if you think it is really needed and if you agree with the content of the sub-PRs.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40443/

Test build #135864 has started for PR 30763 at commit 8e54b41.

Test build #135864 has finished for PR 30763 at commit 8e54b41.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135864/

. Then, BlockManagerId would be a native implementation for Spark and users could implement Location to support custom storage.

To test the idea I try to come up with heavy situation this does not mean I am against the idea.
So if I understand correctly BlockManagerId would extend the Location class, right?
And here MapStatus#location would be a generic Location?
In this case we should check the references of this MapStatus#location and based on that decide where we are safe to cast Location to BlockManagerId or where else we would pass the location further as a  Location (or at least what else the generic location should contain to have the existing things working...).
As the current reader uses MapOutputTracker#getMapSizesByExecutorId you would like to keep that method and runtime throw an exception when it's called and location is not BlockManagerId? This is a central method to get blocksByAddress for fetching in the Spark shuffle.
For example as I see MapOutputTracker is tailored to satisfy the current shuffle solution. This should be checked for the idea.
On the other hand write side might be easier as there MapStatus is filled with the id of the current block manager. So a new writer implementation just uses its location.
But for the read side my worry is having runtime checks/assert/guards to enforce when allowed to use what.

I still think the location abstraction is good idea. I just have my doubts about the amount of the efforts we need to do:

Also, this way wouldn't change the existing framework of shuffle read/write. It'd allow us to be able to reuse the features and code paths without extra effort.

So if I understand correctly BlockManagerId would extend the Location class, right?
And here MapStatus#location would be a generic Location?

Yes

As the current reader uses MapOutputTracker#getMapSizesByExecutorId you would like to keep that method and runtime throw an exception when it's called and location is not BlockManagerId?

We don't. Actually, MapOutputTracker should be refactored to work with the Location instead of the specific BlockManagerId if Location introduced. Accordingly, blocksByAddress would be refactored to store the unique "address" generated by Localtion. That also means we'd always keep the generic Location inside ShuffleBlockFetchIterator instead of a specific Location, so we don't need casting.
I think it also answers this question:

In this case we should check the references of this MapStatus#location and based on that decide where we are safe to cast Location to BlockManagerId or where else we would pass the location further as a Location (or at least what else the generic location should contain to have the existing things working...).

Acutally, I only find one reference that need cast:
https://github.com/apache/spark/blob/f340857757034ac955862b34e60322ca5ee81758/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1661
And yes the custom reader should care more about casting. They should definitely cast the generic Location to their implemented one if they want to get the specific information. But the casting should always succeed because Spark would only use one type storage at a time.

I just have my doubts about the amount of the efforts we need to do

As far as I see,

in @hiboyang 's PR, he added getAllMapOutputStatusMetadata in MapOutputTracker. IIUC, this must need corresponding change to handle the metadata at the reader side, which would be a new code path.

in this PR, I see we added ShuffleOutputTracker, which is very similar to MapOutputTracker. And, MapOutputTracker has add a new interface - updateMapOutput to support node decommission recently. But ShuffleOutputTracker doesn't have it. Do we want to support decommission for custom storages too or only specific to the BlockManager? In the way of ShuffleOutputTracker, I think we must need extra effort if we want to support it in custom storages. However, if we have the generic Location,  we can reuse MapOutputTracker directly.

BTW, I'm thinking we still need to think carefully about the Location solution. I worry we'd overengineering if the use cases don't require such flexibility.  Because I can imagine how Location would widely refactor the code base, especially for those central parts. So I hope we'd discuss more use cases if you think the idea is generally good.

@Ngone51 @attilapiros do we want to proceed with the location idea?

I'm waiting for @attilapiros 's feedback.

We are all agree more abstraction here is really a good idea and reading https://github.com/apache/spark/pull/30763#issuecomment-792865534 gives me the impression we both worry about the impact of the change but as I see you have solutions for all the concerns: https://github.com/apache/spark/pull/30763#issuecomment-792861035.
@Ngone51 I am fine if you proceed and when it is ready we can see the real price of this change.

We are all agree more abstraction here is really a good idea and reading #30763 (comment) gives me the impression we both worry about the impact of the change but as I see you have solutions for all the concerns: #30763 (comment).
@Ngone51 I am fine if you proceed and when it is ready we can see the real price of this change.

+1

Sure, I'll give a try these days.

authors: AmplabJenkins HyukjinKwon Ngone51 SparkQA attilapiros hiboyang


text:
[SPARK-4665] [SPARK-4666] Improve YarnAllocator's parsing of "memory overhead" param

let it be specified as a fraction of the executor memory
add/generalize some utilities for parsing "memory strings" that are found around the Spark codebase.

fixes SPARK-4665 and SPARK-4666.

FWIW I think the logic behind the existing parameters is: 7% should be enough for anyone. And if it isn't, they know it, and have a clear idea of just how much they want the overhead to be. Or: if 7% is consistently not enough, it should be higher, like 10%. I'd leave the existing param unless it's obviously problematic.
But that's separate from the question here, of whether lots of these strings could be specified in mega or gigabytes. Is that still something that can move forward? the refactoring of all that seems helpful in any event.

Not a huge deal, but I think the converse is kind of true.  In my experience, the right percentage is totally unclear to everyone in all situations, so pretty much always needs to be set experimentally.  If I settle on a percentage that works for my job (or all my jobs), it would be nice for me not to need to reset this param every time I tweak spark.executor.memory.

OK maybe there's enough momentum for ... two flags? one of which overrides the other? But maybe we should also bump up the default a bit too.

i think memoryOverheadFraction is good to me.

I'm still not on board with having 2 different configs.  You then have to explain to the user what they are,  which one takes presendence, etc. I can see cases the % is useful but then again you can argue the other way too.  If my job needs 4G overhead, I may want to increase the executor memory without changing the overhead amount.  In that case then I have to go mess with % to get it back what I want.  Either way I think the user has to experiment to get the best number.  I haven't seen any reason to bump up the default at this point  but if others have more real world data lets consider it.
How many people are complaining or having real issues with this?
The main reason I'm against adding this is that I consider it an api change and we now have to support 2 configs until we can get rid of the deprecated one.  Its just more dev overhead, testing, and potential user confusion.   If its enough of a benefit to the user to change it then I would be ok with it but would rather deprecate the existing one in favor of the %.  To me the raw number is more flexible then the % but I see the % as being easier in situations.

before i think that both of two configs can be existed.  from @tgravescs i think overhead is more necessary than OverheadFraction , because at some time it has very large memory,we just use overhead to increase executor memory. now memoryOverheadFraction  is very large and inappropriate.

Thanks for resurrecting this, @srowen. I think there are 3 potentially separable changes here, in rough order of increasing controversial-ness:

Make spark.yarn.executor.memoryOverhead take a "memory string" (e.g. 1g or 100m)

defaulting to m as the unit for backwards-compatibility.

General refactoring of "memory string" handling in conf parameters in some other parts of the codebase.
Addition of spark.yarn.executor.memoryOverheadFraction

If people prefer, I can make a separate PR with 1∪2 or individual PRs for each of 1 and 2.
Re: 3, I am sympathetic to @tgravescs's arguments, but worry that we are conflating [wishing Spark's configuration surface area was smaller] with [forcing users to build custom Sparks to get at real, existing niches of said configuration surface area (in this case to change a hard-coded .07 that reasonable people can disagree on the ideal value of)]. Additionally, the arguments about possible confusion around interplay between memoryOverhead and memoryOverheadFraction are applicable to the status quo, imho.
However, I don't mind much one way or another at this point, so I'm fine dropping 3 if that's what consensus here prefers.

(until I hear otherwise I am pulling/rebasing 1∪2, roughly the SPARK-4666 bits, into their own PR)

Without clear consensus at this point, and given 3 is really a question of how you like to override a default in the same way you override others, I'd do 1+2 for the moment.

OK, this is ready to go again. Changes:

removed SPARK-4665 from the title and closed that JIRA
in the process of rebasing this change I found myself wanting "MB" suffixes on various Int variables that represent numbers of megabytes; I've included several commits here that perform such renames, but they're separate and easy to remove from this PR if that's controversial
I made Utils.getMaxResultSize(SparkConf) a method of SparkConf instead.
I cleaned up the semantics around which Utils and SparkConf methods assume Ints to represent numbers of megabytes, vs. ones that are generic across memory-size orders-of-magnitude.

Let me know what you think.

I personally definitely like the clarity this is adding in the code. Looks like it needs another rebase and test to see where we stand with Jenkins and Mima.

OK I think it is back to you @srowen

(thanks for the review!)

@srowen this one ready to go in?

@hammer @ryan-williams I'd prefer to hear from @JoshRosen or @pwendell before merging. The two key questions are

Gut feeling about the potential for merge conflict -- OK?
An opinion on https://github.com/apache/spark/pull/3525/files#r24632812 and maybe the for comprehension question though that's really not that important either way

I think that this PR's motivation may have been addressed by #5574.  @ilganeli @srowen @ryan-williams , can you confirm whether there's still an issue here and resolve this PR / JIRAs as appropriate?

I think this would have to be reworked in light of SPARK-5932. I don't think that resolved this particular suggestion. Proceed that way or close this PR?

@srowen @JoshRosen I think this should be refactored to use the updates from #5574 but I don't think #5574 resolves this on its own because of the need to handle the min/max allocation - my 2c.

@ryan-williams would you mind opening a new PR to reflect the changes in #5574?

I've lost state on this, closing, thanks all

source: gharchive/pull-request

authors: JoshRosen andrewor14 hammer ilganeli lianhuiwang ryan-williams srowen sryza tgravescs

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-40160][PYTHON][DOCS] Make pyspark.broadcast examples self-contained

What changes were proposed in this pull request?

This PR proposes to improve the examples in pyspark.broadcast by making each example self-contained with a brief explanation and a bit more realistic example.
Why are the changes needed?

To make the documentation more readable and able to copy and paste directly in PySpark shell.
Does this PR introduce any user-facing change?

Yes, it changes the documentation.
How was this patch tested?

Manually ran each doctest.

Can one of the admins verify this patch?

Merged to master.

authors: AmplabJenkins HyukjinKwon dcoliversun


text:
[SPARK-42770][CONNECT] Add truncatedTo(ChronoUnit.MICROS) to make SQLImplicitsTestSuite  in Java 17 daily test GA task pass

What changes were proposed in this pull request?
Run LocalDateTime.now() and Instant.now() with Java 8 & 11 always get microseconds on both Linux and MacOS, but there are some differences when using Java 17, it will get accurate nanoseconds on Linux, but still get the microseconds on MacOS.
On Linux(CentOs)
jshell> java.time.LocalDateTime.now()
$1 ==> 2023-03-13T18:09:12.498162194

jshell> java.time.Instant.now()
$2 ==> 2023-03-13T10:09:16.013993186Z

On MacOS
jshell> java.time.LocalDateTime.now()
$1 ==> 2023-03-13T17:13:47.485897

jshell> java.time.Instant.now()
$2 ==> 2023-03-13T09:15:12.031850Z

At present, Spark always converts them to microseconds, this will cause test implicit encoder resolution in SQLImplicitsTestSuite test fail when using Java 17 on Linux, so this pr add truncatedTo(ChronoUnit.MICROS) to ensure the accuracy of test input data is alway microseconds.
Why are the changes needed?
Make Java 17 daily test GA task run successfully. The Java 17 daily test GA task failed as follows:
[info] - test implicit encoder resolution *** FAILED *** (1 second, 329 milliseconds)
4429[info]   2023-03-02T23:00:20.404434 did not equal 2023-03-02T23:00:20.404434875 (SQLImplicitsTestSuite.scala:63)
4430[info]   org.scalatest.exceptions.TestFailedException:
4431[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
4432[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
4433[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
4434[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
4435[info]   at org.apache.spark.sql.SQLImplicitsTestSuite.testImplicit$1(SQLImplicitsTestSuite.scala:63)
4436[info]   at org.apache.spark.sql.SQLImplicitsTestSuite.$anonfun$new$2(SQLImplicitsTestSuite.scala:133)

Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual checked with Java 17

https://github.com/apache/spark/actions/runs/4318647315/jobs/7537203682
[info] - test implicit encoder resolution *** FAILED *** (1 second, 329 milliseconds)
4429[info]   2023-03-02T23:00:20.404434 did not equal 2023-03-02T23:00:20.404434875 (SQLImplicitsTestSuite.scala:63)
4430[info]   org.scalatest.exceptions.TestFailedException:
4431[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
4432[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
4433[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
4434[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
4435[info]   at org.apache.spark.sql.SQLImplicitsTestSuite.testImplicit$1(SQLImplicitsTestSuite.scala:63)
4436[info]   at org.apache.spark.sql.SQLImplicitsTestSuite.$anonfun$new$2(SQLImplicitsTestSuite.scala:133)
4437[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
4438[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
4439[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
4440[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
4441[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
4442[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
4443[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
4444[info]   at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
4445[info]   at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
4446[info]   at org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1564)
4447[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
4448[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
4449[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
4450[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
4451[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
4452[info]   at org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1564)
4453[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
4454[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
4455[info]   at scala.collection.immutable.List.foreach(List.scala:431)
4456[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
4457[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
4458[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
4459[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
4460[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
4461[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
4462[info]   at org.scalatest.Suite.run(Suite.scala:1114)
4463[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
4464[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
4465[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
4466[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
4467[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
4468[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
4469[info]   at org.apache.spark.sql.SQLImplicitsTestSuite.org$scalatest$BeforeAndAfterAll$$super$run(SQLImplicitsTestSuite.scala:34)
4470[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
4471[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
4472[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
4473[info]   at org.apache.spark.sql.SQLImplicitsTestSuite.run(SQLImplicitsTestSuite.scala:34)
4474[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
4475[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
4476[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
4477[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
4478[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
4479[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
4480[info]   at java.base/java.lang.Thread.run(Thread.java:833)

cc @HyukjinKwon
also cc @bjornjorgensen who reported this issue in dev mail list

@LuciferYang Thank you, sir.
LGTM

friendly ping @dongjoon-hyun

https://github.com/LuciferYang/spark/actions/runs/4412451542/jobs/7732979313

GA passed, status not update

Thanks @dongjoon-hyun @HyukjinKwon @bjornjorgensen

https://github.com/apache/spark/actions/runs/4420600519

The latest Java 17 daily test passed. Thanks all ~

authors: LuciferYang bjornjorgensen


text:
[MINOR][DOCS][PYTHON] Update some urls about deprecated repository pyspark.pandas

What changes were proposed in this pull request?
Fix some deprecated urls
Why are the changes needed?
Repository about https://github.com/pyspark.pandas was deprecated, using https://github.com/databricks/koalas replace.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Matched urls and existing tests

@zhengruifeng Hi, Would you mind have a review?

Those are actually not real JIRAs or TODOs. These are the pointers of the original fix or ticket (that contains examples or code change). So I guess it's fine as is.

I think it is fine if we don't have avaliable ticket link.
It seems that those links point to issues before moving Kolas to Apache Spark.

Thanks for your review. I think some examples that cannot be linked to the correct places, which can make confused, and the original points provides some clear references.

Merged to master.

Got it. Thanks. +1, LGTM too.

authors: HyukjinKwon chong0929 dongjoon-hyun zhengruifeng


text:
[SPARK-43273][SQL] Support lz4raw compression codec for Parquet

What changes were proposed in this pull request?
Parquet 1.13.0 supports LZ4_RAW codec. Please see https://issues.apache.org/jira/browse/PARQUET-2196.
This PR adds lz4raw to the supported list of spark.sql.parquet.compression.codec.
Why are the changes needed?
Support writing Parquet files with lz4raw compression codec.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test and manual testing:
spark.sql("set spark.sql.parquet.compression.codec=lz4raw")
spark.range(10).write.parquet("/tmp/spark/lz4raw")
spark.read.parquet("/tmp/spark/lz4raw").show(false)

yumwang@LM-SHC-16508156 lz4raw % ll /tmp/spark/lz4raw
total 16
-rw-r--r--@ 1 yumwang  wheel    0 Jun  8 12:10 _SUCCESS
-rw-r--r--@ 1 yumwang  wheel  487 Jun  8 12:10 part-00000-c6786f4d-b5a6-406d-96a1-37bf0ceeeac7-c000.lz4raw.parquet
-rw-r--r--@ 1 yumwang  wheel  489 Jun  8 12:10 part-00001-c6786f4d-b5a6-406d-96a1-37bf0ceeeac7-c000.lz4raw.parquet

The first commit passed all tests already (except pyspark-pandas-slow-connect). And, I verified the second commit manually. Merged to master.
[info] ParquetCodecSuite:
01:42:55.154 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] - write and read - file source parquet - codec: none (10 seconds, 419 milliseconds)
[info] - write and read - file source parquet - codec: uncompressed (1 second, 513 milliseconds)
[info] - write and read - file source parquet - codec: snappy (1 second, 457 milliseconds)
[info] - write and read - file source parquet - codec: gzip (1 second, 201 milliseconds)
[info] - write and read - file source parquet - codec: zstd (1 second, 435 milliseconds)
[info] - write and read - file source parquet - codec: lz4 (1 second, 163 milliseconds)
[info] - write and read - file source parquet - codec: lz4raw (1 second, 282 milliseconds)
01:43:14.972 WARN org.apache.spark.sql.execution.datasources.ParquetCodecSuite:

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.datasources.ParquetCodecSuite, threads: ForkJoinPool.commonPool-worker-6 (daemon=true), QueryStageCreator-0 (daemon=true), ForkJoinPool.commonPool-worker-4 (daemon=true), QueryStageCreator-5 (daemon=true), QueryStageCreator-4 (daemon=true), ForkJoinPool.commonPool-worker-7 (daemon=true), QueryStageCreator-1 (daemon=true), rpc-boss-3-1 (daemon=true), ForkJoinPool.commonPool-worker-5 (daemon=true), QueryStageCreator-6 (daemon=true), shuffle-boss-6-1 (daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true), ForkJoinPool.commonPool-worker-3 (daemon=true), QueryStageCreator-2 (daemon=true), QueryStageCreator-3 (daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true) =====
[info] Run completed in 22 seconds, 883 milliseconds.
[info] Total number of tests run: 7
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 487 s (08:07), completed Jun 8, 2023 1:43:15 AM

authors: dongjoon-hyun wangyum


text:
[SPARK-45070][SQL][DOCS] Describe the binary and datetime formats of to_char/to_varchar

What changes were proposed in this pull request?
In the PR, I propose to document the recent changes related to the format of the to_char/to_varchar functions:

binary formats added by https://github.com/apache/spark/pull/42632
datetime formats introduced by https://github.com/apache/spark/pull/42534

Why are the changes needed?
To inform users about recent changes.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
By CI.
Was this patch authored or co-authored using generative AI tooling?
No.

Merging to master. Thank you, @dongjoon-hyun and @cloud-fan for review.


text:
[SPARK-4979][MLLIB] Streaming logisitic regression

This adds support for streaming logistic regression with stochastic gradient descent, in the same manner as the existing implementation of streaming linear regression. It is a relatively simple addition because most of the work is already done by the abstract class StreamingLinearAlgorithm and existing algorithms and models from MLlib.
The PR includes

Streaming Logistic Regression algorithm
Unit tests for accuracy, streaming convergence, and streaming prediction
An example use

cc @mengxr @tdas

Test build #26503 has started for   PR 4306 at commit d88f717.

This patch merges cleanly.

Test build #26503 has finished for   PR 4306 at commit d88f717.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class StreamingLogisticRegressionWithSGD (

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26503/
Test FAILed.

Test build #26504 has started for   PR 4306 at commit 85320b1.

This patch merges cleanly.

Test build #26504 has finished for   PR 4306 at commit 85320b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class StreamingLogisticRegressionWithSGD (

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26504/
Test PASSed.

LGTM except minor inline comments.

Thanks for the review! I think I dealt with everything, and in a couple places I tweaked the corresponding point in StreamingLinearRegression for parity.

Test build #26550 has started for   PR 4306 at commit 275f8bd.

This patch merges cleanly.

Test build #26550 timed out     for PR 4306 at commit 275f8bd     after a configured wait of 120m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26550/
Test FAILed.

test this please

Test build #26567 has started for   PR 4306 at commit 275f8bd.

This patch merges cleanly.

Test build #26567 has finished for   PR 4306 at commit 275f8bd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26567/
Test FAILed.

@mengxr the test failure is real, I think there's a conflict due to this recent change to master (https://github.com/apache/spark/commit/b1aa8fe988301b924048039529234278aeb0298a), just merged into my local branch so I can sort it out...

Ok, the problem is that we've been setting initialWeights to null by default and then checking that weights are defined before starting training (because we don't know the number of features ahead of time). But that now yields a null pointer exception because the default constructor for LogisticRegressionModel checks the size of the weights.

Should we just set initialWeights=Vectors.dense(0.0) by default?

Could we try to set model type to Option[Model] and check whether it is set in the code?

Nice idea! Just took a stab, see what you think. Was reproducing the test failure locally, and it is now fixed with this change.

Test build #26599 has started for   PR 4306 at commit 5c2c70b.

This patch merges cleanly.

Test build #26599 has finished for   PR 4306 at commit 5c2c70b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26599/
Test PASSed.

Merged into master. Thanks!!

authors: AmplabJenkins SparkQA freeman-lab mengxr


text:
[SPARK-45755][SQL] Improve Dataset.isEmpty() by applying global limit 1

What changes were proposed in this pull request?
This PR makes Dataset.isEmpty() to execute global limit 1 first. LimitPushDown may push down global limit 1 to lower nodes to improve query performance.
Note that we use global limit 1 here, because the local limit cannot be pushed down the group only case: https://github.com/apache/spark/blob/89ca8b6065e9f690a492c778262080741d50d94d/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L766-L770
Why are the changes needed?
Improve query performance.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual testing:
spark.range(300000000).selectExpr("id", "array(id, id % 10, id % 100) as eo").write.saveAsTable("t1")
spark.range(100000000).selectExpr("id", "array(id, id % 10, id % 1000) as eo").write.saveAsTable("t2")
println(spark.sql("SELECT * FROM t1 LATERAL VIEW explode_outer(eo) AS e UNION SELECT * FROM t2 LATERAL VIEW explode_outer(eo) AS e").isEmpty)

Before this PR
After this PR

Was this patch authored or co-authored using generative AI tooling?
No.

I revised the PR title a little, @wangyum . You can change it back if you want.

Thank you @dongjoon-hyun The new PR title looks better than the previous one.

Merged to master. Thank you @wangyum
Thank you @dongjoon-hyun @HyukjinKwon @yaooqinn too.

authors: beliefer dongjoon-hyun wangyum


text:
[SPARK-46801][PYTHON][TESTS] Do not treat exit code 5 as a test failure in Python testing script

What changes were proposed in this pull request?
This PR proposes to avoid treating the exit code 5 as a test failure in Python testing script.
Why are the changes needed?
...
========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.12']
Will test the following Python modules: ['pyspark-core', 'pyspark-streaming', 'pyspark-errors']
python3.12 python_implementation is CPython
python3.12 version is: Python 3.12.1
Starting test(python3.12): pyspark.streaming.tests.test_context (temp output: /__w/spark/spark/python/target/8674ed86-36bd-47d1-863b-abb0405557f6/python3.12__pyspark.streaming.tests.test_context__umu69c3v.log)
Finished test(python3.12): pyspark.streaming.tests.test_context (12s)
Starting test(python3.12): pyspark.streaming.tests.test_dstream (temp output: /__w/spark/spark/python/target/847eb56b-3c5f-49ab-8a83-3326bb96bc5d/python3.12__pyspark.streaming.tests.test_dstream__rorhk0lc.log)
Finished test(python3.12): pyspark.streaming.tests.test_dstream (102s)
Starting test(python3.12): pyspark.streaming.tests.test_kinesis (temp output: /__w/spark/spark/python/target/78f23c83-c24d-4fa1-abbd-edb90f48dff1/python3.12__pyspark.streaming.tests.test_kinesis__q5l1pv0h.log)
test_kinesis_stream (pyspark.streaming.tests.test_kinesis.KinesisStreamTests.test_kinesis_stream) ... skipped "Skipping all Kinesis Python tests as environmental variable 'ENABLE_KINESIS_TESTS' was not set."
test_kinesis_stream_api (pyspark.streaming.tests.test_kinesis.KinesisStreamTests.test_kinesis_stream_api) ... skipped "Skipping all Kinesis Python tests as environmental variable 'ENABLE_KINESIS_TESTS' was not set."

----------------------------------------------------------------------
Ran 0 tests in 0.000s

NO TESTS RAN (skipped=2)

Had test failures in pyspark.streaming.tests.test_kinesis with python3.12; see logs.
Error:  running /__w/spark/spark/python/run-tests --modules=pyspark-core,pyspark-streaming,pyspark-errors --parallelism=1 --python-executables=python3.12 ; received return code 255
Error: Process completed with exit code 19.

Scheduled job fails because of exit 5, see https://github.com/pytest-dev/pytest/issues/2393. This isn't a test failure.
Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
Manually tested.
Was this patch authored or co-authored using generative AI tooling?
No,

Merged to master/3.5/3.4.

authors: HyukjinKwon dongjoon-hyun


text:
[SPARK-4012] exit JVM process when the exception is thrown from an infinite loop

https://issues.apache.org/jira/browse/SPARK-4012
This patch is a resubmission for https://github.com/apache/spark/pull/2864
What I am proposing in this patch is that when the exception is thrown from an infinite loop, we should stop the process, instead of let JVM throws exception forever
So, in the infinite loops where we originally wrapped with a  logUncaughtExceptions, I changed to tryOrExit, so that the JVM process is stopped
Early stopped JVM process is helpful for HA scheme design, for example,
The user has a script checking the existence of the pid of the Spark Streaming driver for monitoring the availability; with the code before this patch, the JVM process is still available but not functional when the exceptions are thrown
@andrewor14, @srowen , mind taking further consideration about the change?

Test build #28525 has started for   PR 5004 at commit 6322959.

This patch merges cleanly.

Test build #28525 has finished for   PR 5004 at commit 6322959.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28525/
Test PASSed.

Do you think calling Thread.setDefaultUncaughtExceptionHandler(SparkUncaughtExceptionHandler) in the driver side is acceptable? Or at least, I think we should make sure every thread created by Spark should set SparkUncaughtExceptionHandler.

Hi, @zsxwing , Thread.setDefaultUncaughtExceptionHandler(SparkUncaughtExceptionHandler) is fine, though it's equivalent to the current patch in terms of functionality
but I'm not sure if we need to set exceptionHandler of all threads as SparkUncaughtExceptionHandler, because that means, once there is an uncaught exception, we stop the program...

once there is an uncaught exception, we stop the program...

Yes. So I suggests only threads created by Spark should set SparkUncaughtExceptionHandler. I think if a Spark internal thread throws an uncaught exception, it often means some Spark internal module has crashed. Just my 2 cents about improving the robustness.
Of course, look good to me about your changes.

@zsxwing , sounds reasonable
let's wait for more eyes on this

I tend to favor this change for ContextCleaner and AsynchronousListenerBus, because an exception there kills a thread that is clearly intended to run forever. I'm not as sure about FsHistoryProvider, since there it is not as clear (yet) that the Runnable should cause the worker to die if it hits an exception. why this one?
I'd like it if, say, @aarondav might comment, since he added one of the lines being changed.

@srowen thanks for the comments,
the reason to change FsHistoryProvider is that the runner generated by this function  is essentially executed by the threads with fixed rate
https://github.com/CodingCat/spark/blob/SPARK-4012-1/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L138

Is it really OK to System.exit() the driver JVM? This may be user code that has an embedded SparkContext. The SparkUncaughtExceptionHandler is suitable for Executors, where we have full control over the JVM, and AppClient for the same reason, but I'm not sure TaskSchedulerImpl should be using it, or ContextCleaner, for instance.

Test build #28613 has started for   PR 5004 at commit 6087864.

This patch merges cleanly.

@aarondav , thanks for the insightful suggestion
I just updated the patch
the change becomes a bit bigger, as I need to create a new method in Utils as tryOrStopSparkContext, which is a curried method in the current patch
also, I need to pass a SparkContext reference to AsynchronousListenerBus to be able to pass sparkContext as the parameter, so I changed the start() to start(sparkContext: SparkContext) in which I set the value of the newly added sparkContext variable member of  AsynchronousListenerBus

Test build #28613 has finished for   PR 5004 at commit 6087864.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28613/
Test PASSed.

@aarondav thanks for the comments, I just updated the patch

Test build #28619 has started for   PR 5004 at commit 3c72cd8.

This patch merges cleanly.

Test build #28619 has finished for   PR 5004 at commit 3c72cd8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28619/
Test PASSed.

Alright, this looks good to me. @srowen and @zsxwing, any comments?

I don't have a strong opinion on this one. I suppose there's a question of what may happen if you stop the SparkContext but then immediately kill the calling thread with an uncaught Throwable. Does that undermine the action of stop() in some cases?

@srowen I checked the code, one suspicious part might be the asynchronous shutdown of ActorSystem, but that should be OK in Spark case...
@aarondav ?

Since tryOrStopSparkContext is in Utils, I think it's better to check if it's reasonable in the general case. I'm concerning that swallowing fatal errors (OOM) or InterruptException may cause some issue.

Test build #28825 has started for   PR 5004 at commit 589276a.

This patch merges cleanly.

Test build #28825 has finished for   PR 5004 at commit 589276a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28825/
Test PASSed.

@zsxwing @srowen @aarondav how about the current version?

LGTM

LGTM

thanks, guys~

Cool, merging this into master. Thanks!

authors: AmplabJenkins CodingCat SparkQA aarondav srowen zsxwing


text:
[SPARK-8149][SQL] Break ExpressionEvaluationSuite down to multiple files

Also moved a few files in expressions package around to match test suites.

Ah damn github sync is behind so more diffs are showing up.

Merged build triggered.

Merged build started.

Test build #34399 has started for   PR 6693 at commit c0eb74b.

Test build #34399 has finished for   PR 6693 at commit c0eb74b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34400 has started for   PR 6693 at commit 857599f.

Test build #34400 has finished for   PR 6693 at commit 857599f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

authors: AmplabJenkins SparkQA rxin


text:
[SPARK-8693][Project Infra]: profiles and goals are not printed in a nice way

Hotfix to correct formatting errors of print statements within the dev and jenkins builds. Error looks like:
-Phadoop-1[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  -Dhadoop.version=1.0.4[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  -Pkinesis-asl[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  -Phive-thriftserver[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  -Phive[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  package[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  assembly/assembly[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments:  streaming-kafka-assembly/assembly

Merged build triggered.

Merged build started.

Test build #35986 has started for   PR 7085 at commit c5575f1.

Not a python person here, but I suspect that is indeed a good fix.

LGTM.  I guess the problem in the old code was a precedence issue that was causing the implicitly-concatenated string to be joined to each profile instead of concatenating the joined profiles to the start of the log message:
>>> print("hello" " world")
hello world
>>> print("hello " ",".join(["a", "b", "c"]))
ahello ,bhello ,c

Test build #35986 has finished for   PR 7085 at commit c5575f1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

The script's lint checks looked fine and I manually verified that the message was logged properly, so I'm going to merge this into master.  Thanks!

authors: AmplabJenkins JoshRosen SparkQA brennonyork srowen


text:
[SPARK-8914][SQL] Remove RDDApi

As @rxin suggested in #7298 , we should consider to remove RDDApi.

Merged build triggered.

Merged build started.

Test build #36851 has started for   PR 7302 at commit cb7ebb9.

LGTM.

Test build #36851 has finished for   PR 7302 at commit cb7ebb9.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #36853 has started for   PR 7302 at commit e495d35.

Test build #36853 has finished for   PR 7302 at commit e495d35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Thanks - merged.

authors: AmplabJenkins SparkQA rxin sarutak


text:
[SPARK-10705][SQL] Avoid using external rows in DataFrame.toJSON

JIRA: https://issues.apache.org/jira/browse/SPARK-10705
As described in the JIRA ticket, DataFrame.toJSON uses DataFrame.mapPartitions, which converts internal rows to external rows. We should use queryExecution.toRdd.mapPartitions that directly uses internal rows for better performance.

Merged build triggered.

Merged build started.

Test build #42827 has started for   PR 8865 at commit a0c6e60.

Test build #42827 has finished for   PR 8865 at commit a0c6e60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42827/
Test PASSed.

LGTM. @yhuai Could you please have a look at this one?

LGTM. Merging to master.

authors: AmplabJenkins SparkQA liancheng viirya yhuai


text:
Uncaught Error: Target container is not a DOM element.

When running npm run dev-server this route:
http://localhost:9000/chart/add Leads to a React JS error: Uncaught Error: Target container is not a DOM element.
Which indicates that the Dom isnt available yet for the React Scripts to kick off.
This only happens when kicking off webpack-dev-server
It works just fine when running the app without webpack-dev-server.... so it must be a config issue, but not sure what it could be.
Any insight would be great on this.
Node: 12.14.1
typescript 3.8.3
Superset:  0.37.0 as of (2020/08/01 06:59 +00:00)

I too am getting this error. It does not happen all the time. I encounter this only when try to add a chart (on the page /chart/add). Adding a database, table or dashboard works fine.

Closing this as it should not be an issue in more current/supported versions. If anyone is seeing this in 1.5.x or 2.0.x, please say the word, and we'll re-open it!

source: gharchive/issue

authors: badman-rodriguez rusackas thisishantzz

repo: apache/superset


text:
Control panel not showing controls on initial load for custom plugins

Workaround seems to be to change either the datasource/dataset or the viz type. However, on initial load, the controls are missing. The reason it says "Empty Query?" is because it tries to query the backend because the validator constraints for all the controls are not in play as the controls are not rendered.
How to reproduce the bug

Point to a plugin in Settings -> Plugins
Initiate creating a new chart via Charts -> + Chart
See error

Expected results
Expect control panel to load normally.
Actual results
Control panel missing controls. They reappear only after changing datasource or viz type (does not matter from which to which).
Screenshots
Above.
Environment
(please complete the following information):

browser type and version: Chromium 95
superset version: 1.3.2
python version: 3.7.3
any feature flags active:

FEATURE_FLAGS ={
    "DYNAMIC_PLUGINS": True,
    "ROW_LEVEL_SECURITY": True,
    "VERSIONED_EXPORT": True,
    "THUMBNAILS": True,
    "THUMBNAILS_SQLA_LISTENERS": True,
    "LISTVIEWS_DEFAULT_CARD_VIEW": False,
    "ALERT_REPORTS": True,
}

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Hi Guys,
I've reproduced the issue on 2.0.1,
It seems that the plugin is called before it finished initialisation.
Looking into the code of superset-frontend/./src/explore/components/ExploreViewContainer/index.jsx,
it looks as we try to triggerQuery on a component that did not complete loading,
The following fix works, but needs to be checked for side effects.

*** 387,393 ****
control =>
control.validationErrors && control.validationErrors.length > 0,
);
!     if (!hasError) {
props.actions.triggerQuery(true, props.chart.id);
}
}, []);
--- 390,396 ----
control =>
control.validationErrors && control.validationErrors.length > 0,
);
!     if (!hasError && !isDynamicPluginLoading ) { // If we have a Dynamic plug in, make sure it finished loading
props.actions.triggerQuery(true, props.chart.id);
}
}, []);

Hello @krsnik93 thanks for reporting this. As this issue has been stale for a while I'll be closing it. Feel free to re-open it if you are still experiencing this problem in recent versions of Superset.

authors: geido krsnik93 viewers


text:
Ability to support indian numbering system

Screenshot
[drag & drop image(s) here!]
Description
Indian numbering system has numbers such as lakh(100,000), crore (10,000,000) etc
This issue is to check if there is already a support for formatting numbers in this way in superset. if not, what would it take to do so? is it a concern of superset or of the d3-format library?
PS: I'd like to contribute the code to implement this feature, if it doesnt exist yet :)
[describe the issue here!]
Design input
[describe any input/collaboration you'd like from designers, and
tag accordingly. For design review, add the
label design:review. If this includes a design proposal,
include the label design:suggest]

I guess the new option D3_FORMAT in superset_config.py on the master branch can set the number format to Indian locale,
how ever, even if it's implemented, d3-format looks like have some limitations:
Can not format two columns with different currency prefixes.
Some regions do not use large number suffixes such as 'G' or 'M' , but insted use large number suffixes in their local language.

Closing this as stale, and because it's more feature request than bug... but that said, the situation HAS improved since this thread was touched. You can now add currency prefix/suffix in many spots, and you can also implement these numbering systems in config. If this needs further improvement, we're open to PRs and proposals on Github Discussions

Hi @Raymanrrr @rusackas  , did this feature request to include the Indian Number formats got processed? I am facing the issue  where I want to show a number in Indian format eg 3.2 crore OR 4lakhs , but superset or specifically the d3.js dosent support this. So any suggestions?

authors: Raymanrrr himanshudube97 rusackas sahoss


text:
Tags on Saved Queries don't work after upgrading to Superset 3.0.0

Adding tags to saved queries does not seem to be possible from the frontend UI. When creating Charts or Dashboards you have the option of adding Tags!
This is very useful for organising elements into logic groups.
Under the Saved Queries tab you can clearly see a column & search bar dedicated to Tags on Saved Queries, however there is no way to add these queries from the Frontend.

In a discussion on Slack a suggestion was to use Bulk Select which would then expose the Add Tags option, however no such option becomes available.

When adding the tag manually directly in the database it does render to the frontend

Expected results
Something similar to Charts but the ability at add Tags to Saved Queries.

Environment
(please complete the following information):

browser type and version: Chrome Version 116.0.5845.187 (Official Build) (arm64)
superset version: Version: 3.0.0rc2
python version: python 3.9
FEATURE_FLAGS = {
'ALERT_REPORTS': True,
'ENABLE_TEMPLATE_PROCESSING': True,
'ALLOW_FULL_CSV_EXPORT': True,
'DASHBOARD_RBAC': True,
'TAGGING_SYSTEM': True,
'HORIZONTAL_FILTER_BAR': True,
}

Any suggestions?

I think this feature was more fully fleshed out with the 3.1 release, where 3.0 was still fairly "beta" in nature. @yousoph do you know the current status? If it's all good in 3.1, I think we can close the 3.0 complaint as resolved since we won't be backporting features (just fixes).

Hello! We are running image tag 4.0.2-dev and still can not see this in the UI

authors: duvenagep gparrello rusackas


text:
feat: Disables HTML rendering in Toast by default

SUMMARY
This PR disables HTML rendering in toast messages by default. I couldn't find any instances of toast messages with HTML content but it is a valid use case, so I created an allowHtml property for this scenario.
TESTING INSTRUCTIONS
1 - Make sure toast messages don't render HTML by default
2 - Make sure toast messages render HTML if allowHtml is enabled
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Reviewers, please let me know if we actually have an instance where we use HTML in toast messages already. I couldn’t find one.


text:
fix(drillby): Enable DrillBy in charts w/o filters (dimensions)

SUMMARY
This fix resolves #27940 by enabling "Drll by" for Timeseries and Mixed Timeseries charts that do not have dimensions configured.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before:

After

TESTING INSTRUCTIONS
Open any dashboard that includes either Timeseries or Mixed Timeseries charts without dimensions and right-click on any data point in those charts. The "Drill by" should be available and open the modal for the selected dimension.
ADDITIONAL INFORMATION

[X] Has associated issue:
[ ] Required feature flags:
[X] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Thanks @sowo this looks promising! Curious what you see left to do here, but we look forward to reviewing it when it's ready. So far, the code looks good to me!

Hi,
Thanks for the response.
I found an issue when clicking on the breadcrumbs. Some index misalignment
after filtering out the empty initial filter. Should be fixed soon.
Best,
Wolfgang
On Mon, Apr 8, 2024 at 6:37 PM Evan Rusackas @.***>
wrote:

Thanks @sowo https://github.com/sowo this looks promising! Curious what
you see left to do here, but we look forward to reviewing it when it's
ready. So far, the code looks good to me!
—
Reply to this email directly, view it on GitHub
https://github.com/apache/superset/pull/27941#issuecomment-2043201328,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAVDI5JQDA7KHY2TYGGOGYDY4LBWLAVCNFSM6AAAAABF4YSE7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGIYDCMZSHA
.
You are receiving this because you were mentioned.Message ID:
@.***>

/testenv up

Maybe if @sfirke's hypothesis is right, we'll get lucky and this will help with https://github.com/apache/superset/issues/25334 as well... but maybe not. Just something to test when the ephemeral env spins up.

Seems the breadcrumbs and drillByConfigs need to be in sync to make breadcrumb navigation work. So rather than filtering out the first breadcrumb item when it is empty (for charts without dimensions), I decided to just hide the first breadcrumb. This is somewhat ugly as it leaves the separator (/) visible.

Let me know if anyone has a better idea on how to non-intrusively fix this.

/testenv up

Hmm... closing/reopening to see if that kicks CI into gear.

@kgabryje initially, I used the X axis value plus the selected dimension for the drill by chart. But this created a time series chart with just one bar broken down by the selected dimension. For this to really be useful, the chart type would need to be changed - e.g. pie chart. Is there any plan to make the chart type selectable as one drills down?
Thanks

Maybe if @sfirke's hypothesis is right, we'll get lucky and this will help with #25334 as well... but maybe not. Just something to test when the ephemeral env spins up.

Alas this did not, just confirmed while testing 4.0.1rc1 which contains this PR.  This fix works through and is great!

authors: kgabryje rusackas sfirke sowo


text:
THRIFT-4766: Fix JDK11 build

Pull Request Guidance
Otherwise it will fail with:
make[4]: Entering directory '/thrift/tutorial/cpp'
make[4]: Nothing to be done for 'install-exec-am'.
make[4]: Nothing to be done for 'install-data-am'.
make[4]: Leaving directory '/thrift/tutorial/cpp'
make[3]: Leaving directory '/thrift/tutorial/cpp'
make[2]: Leaving directory '/thrift/tutorial/cpp'
Making install in java
make[2]: Entering directory '/thrift/tutorial/java'
/usr/bin/ant  compile
Buildfile: /thrift/tutorial/java/build.xml

init:
    [mkdir] Created dir: /thrift/tutorial/java/build
    [mkdir] Created dir: /thrift/tutorial/java/build/log

generate:

compile:
    [javac] Compiling 7 source files to /thrift/tutorial/java/build
    [javac] /thrift/tutorial/java/gen-java/shared/SharedService.java:10: error: package javax.annotation does not exist
    [javac] @javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2019-01-30")
    [javac]                  ^
    [javac] /thrift/tutorial/java/gen-java/shared/SharedStruct.java:10: error: package javax.annotation does not exist
    [javac] @javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2019-01-30")
    [javac]                  ^
    [javac] /thrift/tutorial/java/gen-java/tutorial/Calculator.java:10: error: package javax.annotation does not exist
    [javac] @javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2019-01-30")
    [javac]                  ^
    [javac] /thrift/tutorial/java/gen-java/tutorial/Work.java:19: error: package javax.annotation does not exist
    [javac] @javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2019-01-30")
    [javac]                  ^
    [javac] /thrift/tutorial/java/gen-java/tutorial/InvalidOperation.java:13: error: package javax.annotation does not exist
    [javac] @javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2019-01-30")
    [javac]                  ^
    [javac] /thrift/tutorial/java/gen-java/tutorial/Operation.java:14: error: package javax.annotation does not exist
    [javac] @javax.annotation.Generated(value = "Autogenerated by Thrift Compiler (0.13.0)", date = "2019-01-30")
    [javac]                  ^
    [javac] 6 errors

BUILD FAILED
/thrift/tutorial/java/build.xml:55: Compile failed; see the compiler error output for details.

Review the following items to ensure a smooth pull request experience.

[ ] Did you make a breaking change?  If so:

[ ] Add (or reference) an Apache Jira THRIFT ticket.
[ ] Add a Breaking-Change label to the Jira ticket.
[ ] Add a note to the lib/<language>/README.md file.
[ ] Add a line to the CHANGES.md file.

[x] Is this change significant enough to be in release notes?

This will add JDK9+ compatibility to Thirft

[x] If there is an Apache Jira ticket:

[x] Is the Apache Jira THRIFT ticket identifier in the PR title?
THRIFT-4766: an example pull request title

[x] Is the Apache Jira THRIFT ticket identifier and affected languages in the commit message?
THRIFT-4766: [summary of fix, one line if possible]
Client: [language(s) affected, comma separated, use lib/ directory names please]

[x] Did you squash your changes to a single commit?
  Committers can squash when they merge, but sometimes we forget, and it makes the history
  pretty dirty.  Please squash your pull requests to a single commit if you can.

For more information about committing, see CONTRIBUTING.md

I prefer using the headless as well. I've added libasound2 package to the Dockerfile.

Still some issues with Chrome:
Running "qunit:ThriftDeepConstructor" (qunit) task
>> There was an error with headless chrome
Fatal error: Failed to launch chrome!
/thrift/src/lib/js/node_modules/puppeteer/.local-chromium/linux-609904/chrome-linux/chrome: error while loading shared libraries: libatk-bridge-2.0.so.0: cannot open shared object file: No such file or directory

@jeking3 can you restart the failed builds? The error looks unrelated.

This has broken the CI builds.  It looks like the java runs slower than expected... I think we need to adjust the timeouts on the cross test builds.

I just pushed a fix by increasing the ruby client timeout.

repo: apache/thrift


text:
Build fix  fix D build for LDC: The LLVM-based D Compiler.

https://github.com/ldc-developers/ldc
LDC:  The LLVM-based D Compiler.
actually is a popular D compiler than DMD
It has build failure:
/home/./.dub/packages/apache-thrift-0.14.1/apache-thrift/lib/d/src/thrift/transport/websocket.d(335,39): Error: function ldc.intrinsics.llvm_bswap!uint.llvm_bswap at /home/./project/ldc2-1.26.0-linux-x86_64/bin/../import/ldc/intrinsics.di(405,7) conflicts with function std.bitmanip.nativeToBigEndian!ushort.nativeToBigEndian at /home/./project/ldc2-1.26.0-linux-x86_64/bin/../import/std/bitmanip.d(2998,6)
This can be easily fixed by using fully qualified names.
The new change has been tested in my local D build.

[ ] Did you create an Apache Jira ticket?  (not required for trivial changes)
[ ] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
[X] Did you squash your changes to a single commit?  (not required, but preferred)
[X] Did you do your best to avoid breaking changes?  If one was needed, did you label the Jira ticket with "Breaking-Change"?

https://travis-ci.org/github/apache/thrift/jobs/773531587
====================
All 100 tests passed
make[4]: Leaving directory '/thrift/src/lib/d'
make[3]: Leaving directory '/thrift/src/lib/d'
Making check in test
make[3]: Entering directory '/thrift/src/lib/d/test'

Downgrading openssl version? Because someone says something about some compiler? Really?

Downgrading openssl version? Because someone says something about some compiler? Really?

openssl is not downgraded!
If you look closely, the openssl version in the dub.json is D's (not C's) openssl version, see here:
https://code.dlang.org/packages/openssl
Versions:
2.0.3+1.1.0h | 2021-May-13
2.0.2+1.1.0h | 2021-Mar-09
2.0.1+1.1.0h | 2019-May-12
2.0.0+1.1.0h | 2018-Sep-17
1.1.6+1.0.1g | 2017-Nov-05

(the D openssl's version format is: +
For C's openssl both use_openssl_1_1 () and use_openssl_1_0 are supported.
Regard the popularity LDC compiler, please see the D's forum:
https://forum.dlang.org/

@Jens-G Before this PR, the dub.json only supported any version of the D openssl library greater than 1.1.6, which corresponds to the C openssl version 1.0.1. One thing this commit does is add support for the user to choose either the 1.0.1 C openssl library or the 1.1.0 one. I would guess that one reason for this is to allow for backwards compatibility for people who are using the library currently.

Downgrading openssl version? Because someone says something about some compiler? Really?

Regard the popularity LDC compiler

I think there is a misunderstanding here due to language barriers and an insufficient structure of this PR and description of the changes done here.

authors: CyberShadow Jens-G jmh530 mingwugmail


text:
BZ 63627: Implement more fine-grained handling in RealmBase#authenticate(GSSContext, boolean)

Tests are pending, but I don't expect any related failures here.

If I don't see ayn objection, I'll merge this by tomorrow.

Thanks, tests are already running for backports...

repo: apache/tomcat


text:
Enables switching SSL certificates on QUIC with QUICHE

This enables using certificates loaded by QUICMultiCertConfigLoader (not only the default cert but also certs for specific servernames and IP addresses).
This also makes other SNI stuff work on QUIC.

This depends on #9322, and most code change will be gone after merging it.
The most important change on this PR is use of quiche_conn_new_with_tls. This requires the both ATS and Quiche use the same version of BoringSSL.

This requires BoringSSL if we use Quiche implementation, and I think the autest failure is because of the build environment, which use OpenSSL(quictls).
@bneradt Can we use BoringSSL on the image for Quiche build?

[approve ci autest]

[approve ci autest]

This breaks the original QUIC implementation, and autests for HTTP/3 would be ran only if ATS is built with Quiche.

repo: apache/trafficserver


text:
[TEST] Improve CheckGrad Numerical Gradient checking

Followup of https://github.com/apache/tvm/issues/8579
Right now our numerical r numerical gradient checker trying to do numerical gradient for all directions which scales linearly to the number of entries.
We can instead sample a few random directions to check so that the time complexity won't grow linearly to the number of entries. see page 8 of this slide

cc @altanh

repo: apache/tvm


text:
Tutorial for running TVM on Arm(R) Cortex(R)-M55 CPU and Ethos(TM)-U55 NPU

This PR introduces a tutorial for running TVM on Arm(R) Cortex(R)-M55 CPU and Ethos(TM)-U55 NPU, using bare metal.
In order to run the application without having access to a Cortex(R)-M55 and Ethos(TM)-U55 development board, the sample application will be run on a Fixed Virtual Platform (FVP). The FVP based on Arm(R) Corstone(TM)-300 software, models a hardware system containing a Cortex(R)-M55 and Ethos(TM)-U55.
This tutorial compiles a MobileNet v1 model and instructs TVM to offload operators to the Ethos(TM)-U55 where possible.
@manupa-arm @Mousius @leandron @areusch

@grant-arm thanks for sending this! will defer to others for review

@leandron @mbaret Would you mind having a look at this, since you have some experience developing the tvmc tutorial.

authors: areusch grant-arm


text:
[WIP][Runtime] Pipeline Executor Add Set and Get Input/Output interfaces.

RFC PR: apache/tvm-rfcs#0014
GitHub Issue: apache/tvm#8596

Add "param" connection into pipeline config to support such case
like set params into module 1 by a param name "param0"

Add using input name to locate backend runtime index and input index
implemention.

Add interface like run/stop/set_input/get_output etc.

Add a implemention of serialized pipeline backend runtime execution
for the purpose to test all the said interface.

Thanks for contributing to TVM!   Please refer to guideline https://tvm.apache.org/docs/contribute/ for useful information and tips. After the pull request is submitted, please request code reviews from Reviewers by @ them in the pull request thread.

@comaniac @masahi please take a look.

thanks @comaniac , all review comments addressed, please take a look.

@comaniac @masahi

I can take a look this week end / next week.

thanks @masahi.

Thanks @masahi for the review, besides of fixing grammar issue, do you think that splitting this PR into couple small PR can make it be more review friendly? I will go to split this PR if you think that can help for the review.

Thanks @masahi for the review, besides of fixing grammar issue, do you think that splitting this PR into couple small PR can make it be more review friendly? I will go to split this PR if you think that can help for the review.

Yes! That sounds great, thanks.

thanks @masahi @comaniac for the review, and sorry for inconvenience caused by the big PR code size,  I start to split this PR into 4 small PR to make it more review friendly, following is the plan of patch splitting, associated tracking issue also get updated.

Pipeline executor set_input implementation and input configuration loading.
Pipeline executor set_params implementation and params configuration loading.
Pipeline executor runtime interface declare and implementation.
Pipeline executor pipe line sequence execution.

authors: huajsj masahi


text:
Conversion issue with nested bold and italic

There is at least one example of a failed conversion of nested bold and italic:
https://www.staged.apache.org/dev/pmc.html#private-or-public
has the text:
Most projects do their work on their **dev@**project.apache.org mailing list.

The word project is in italic, which is correct, but the asterisks are wrong: the entire list address should be bold, see:
https://www.apache.org/dev/pmc.html#private-or-public
There may be other such examples.

I will note that both the CMS and Pelican versions are wrong for different interpretation and parsing of three *** in a row.
The fix is to mix * and _ giving what was clearly the original intent.

Reference here: https://sindresorhus.com/github-markdown-css/#emphasis-1

repo: apache/www-site


text:
[ZEPPELIN-4870] Improve parsing of the paragraph properties

What is this PR for?
We can provide properties that are local to the paragraph, that could be used to pass an additional information for interpreter that could affect its behavior.  Unfortunately existing parsing functionality relies on the fact that key/value pairs need to be separated by , character, and doesn't handle values with special characters (,, =, ...) inside, like this:
%cassandra(locale=ruRU, timeFormat="E, d MMM yy", floatPrecision = 5, outputFormat=cql)

This PR changes the parsing logic to perform character-by-character parsing, and handling of the quoted keys & values, escaping of the special characters, etc.
What type of PR is it?
Improvement
What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-4870

How should this be tested?

https://travis-ci.org/github/alexott/zeppelin/builds/697522260
additional unit tests were added

Over LGTM, one last question about this PR, what happens in frontend if user specify invalid properties ?

Overall LGTM, one last question about this PR, what happens in frontend if user specify invalid properties ?

Interpreter should throw an exception... The parsing itself just do the general checks that syntax is correct etc., the rest is responsibility of the interpreter - we'll generalize common properties, like, time format, etc. over the time - I'll file a JIRA for it.

Would the frontend display the proper error message ?

Should be, at least I saw them in current implementation of Cassandra interpreter, when I specified incorrect time format, etc.

Thanks for the confirmation @alexott , +1 from me

authors: alexott zjffdu

repo: apache/zeppelin

license_source: bigquery


text:
Chocolatey package

I was wondering if you guys can make a package for installation via Chocolatey on windows.
https://chocolatey.org/packages

Worth noting it does already have a Scoop package though https://github.com/lukesampson/scoop/blob/master/bucket/apex.json

repo: apex/apex

license: MIT


text:
Menu isn't closed if you click outside a chart component.

Description
The menu doesn't close if you click beyond the chart component.
Steps to Reproduce

Got to  https://apexcharts.com/angular-chart-demos/line-charts/basic/ and click on menu.
Click outside a chart component.
Menu still open.

Expected Behavior
The menu should be closed if you click beyond a chart component.
Actual Behavior
The menu is still open if you click beyond a chart component.
If you click on the chart component you will close menu.
Screenshots

Reproduction Link
https://apexcharts.com/angular-chart-demos/line-charts/basic/

Not a bug.
The sample you linked is rendered as an iframe on the page. So, it is undetected if a click event happens outside the iframe. Try clicking inside anywhere around the chart and it should close.

authors: damianmalinowski junedchhipa


text:
Using One Yaxis for multiple series

Discussed in https://github.com/apexcharts/apexcharts.js/discussions/4338

Originally posted by BlerinaP March 20, 2024
Im using react-apexchart on my project and i have four series, one of them has opposite y-axis the others needs to associate to the first serie, like this
yaxis: [
            {
                seriesName: 'Positive',
                show: true
            },
            {
                seriesName: 'Positive',
                show: false
            },
            {
                seriesName: 'Positive',
                show: false
            },
            {
                seriesName: 'Average',
                opposite: true
            }
        ],

        Everything was working fine until i had to update react-apexchart to the latest version, then i got this error

        Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'result') in apexcharts.common.js

        Any idea why this happend, and how to solve this prob.

Confirming this as a bug in 3.48.0. I believe I'll post a pull request soon, depending on more testing.


text:
feat(node): expose endpointParameters to all pre/post-processing steps

Closes #1673.
I implemented this by passing endpointParameters to the VM context in unsafeEvaluate and unsafeEvaluateAsync for both pre- and post-processing. This allows the endpointParameters to be immutable with respect to the parameters of the aggregatedApiCall (more relevant to pre-processing).
Note to achieve the above immutability and to avoid breaking existing post-processing code, I did not modify the input (the raw API call response data) of the reduce of postProcessingSpecifications.

The description looks good, I'll review as soon as possible

Thanks, looks good. I can test both #1738 and #1673 once merged. It would be nice to release 0.12 before the end of July.

Thanks and releasing v0.12 by end of July should be no problem- the only issue left for the v0.12 milestone is https://github.com/api3dao/airnode/issues/1692 for which I am finishing up a PR (https://github.com/api3dao/airnode/pull/1836).

authors: bbenligiray dcroote


text:
Change airnode-validator ChainOptions fulfillmentGasLimit to be optional

Documentation for https://github.com/api3dao/airnode/issues/1437

Closing because the corresponding Airnode issue is closed (see that issue for the reason).

authors: amarthadan dcroote


text:
Enable change of min/maxConcurrency on fly

Sometimes the website may get overloaded by a high number of requests. Some users reported that the unoptimized website started returning 500 error codes even with low concurrency.
If this problem is identified I want to be able to lower the concurrency limit of AutoscaledPool (PuppeteerCrawler / BasicCrawler) on the fly and also to pause a crawler for certain time.

Done.

@mtrunkat could you please give details on how to use it?
Thank you

It's not released yet. We will release it next week after we merge this PR https://github.com/apifytech/apify-js/pull/219/files
There will be new method autoScaledPool.setMax(Min)Concurrency()

authors: gsouf mtrunkat

repo: apifytech/apify-js


text:
fix istio specs, readme

Change-Id: I36c4aee896e8a5c223855eae6b288636a4d98fe5

done

repo: apigee/istio-mixer-adapter


text:
Cannot deploy openbank to latest NPM and Gulp

#Issue/Request:
Cannot deploy openbank to Apigee Edge with the following npm and gulp versions. Below are the errors from gulp openbankdeploy:
[12:02:09] deploying proxy products
[12:02:10] error deploying proxy locations
[12:02:10] Error: Error uploading target: 400
[12:02:10] error deploying proxy products
[12:02:10] Error: Error uploading target: 400
[12:02:14] error deploying proxy oauth
[12:02:14] Error: Error uploading target: 400
[12:02:19] deployed proxy payments

$ npm -v
5.6.0
$ gulp -v
[23:35:17] CLI version 2.0.1
[23:35:17] Local version 3.9.1

I had the same issue, and determined that it was caused by the config.yml file not being updated correctly. See #91

Closing as the project has been refactored and this bug issue no longer applies - please see https://github.com/apigee/openbank#previous-versions

authors: jorisvddonk laughingbiscuit phriscage

repo: apigee/openbank


text:
🛑 Gitea is down

In c945226, Gitea (https://gitea.openstorage.xyz) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Gitea is back up in 2ede425.

repo: apinter/OSC-mon


text:
Add ssh protocol support.

Currently, I have so it creates a new ssh connection on every request. Do you think the ssh connection should persist? So you think it should be an option?
Example:
var Docker = require("dockerode");
var fs = require('fs');
var key = fs.readFileSync('/path/to/ssh-key');

var docker = new Docker({
    protocol: 'ssh',
    host: 'localhost',
    ssh: {
        privateKey: key,
        username: 'root'
    }
});

docker.listImages((e,d) => {
    console.log('img', e, d);
});

docker.info((e,d) => {
    console.log('con', e, d);
});

fixes #99

I think it's better for it to be an option.
If someone wants persistent connections it will be able to specify it.
Awesome and much needed PR :)

repo: apocas/docker-modem


text:
Cannot read properties of undefined (reading 'method')

I'm getting this error when trying to make an http call using localstack.
Debugging the issue I found that instead of trying to read from event.requestContext.http.method it should actually be event.requestContext.httpMethod.

Nevermind, I think localstack uses v1 APIGateway events


text:
WatchKeys相关疑问

Client端启动的时候，会像NotificationControllerV2发起长轮询，如图所示，总是会watch自己应用所设置的集群和default集群，也就是default配置变动的时候也会通知对应的Client端，但是Client端发起请求拉取配置的时候却不会拉取default的配置，这么做的原因是什么呢？

client 实际生效的配置会按照优先级查找多个集群，在订阅的时候不感知具体哪个集群有配置，所以都会订阅

repo: apolloconfig/apollo


text:
Android Kotlin codegen does not compile after 2.0.0 upgrade

Kotlin codegen generates  code which causes "unnecessary safe call on a non-null receiver of type " for queries after 2.0.0 upgrade
Sample codegen where it throws error during build

data class _Backrefs(
val __typename: String = "CentreBackrefs",
val stores__via__centre: List<Stores__via__centre>?
) {
fun marshaller(): ResponseFieldMarshaller = ResponseFieldMarshaller.invoke { writer ->
writer.writeString(RESPONSE_FIELDS[0], this@_Backrefs.__typename)
writer.writeList(RESPONSE_FIELDS[1], this@_Backrefs.stores__via__centre) { value,
listItemWriter ->
value?.forEach { value ->
listItemWriter.writeObject(value?.marshaller())}
}
}

stores__via__centre list item is expecting it to be nullable but its not and is causing build to fail. Have tried invalidating cache, rebuild, clean and try to re-generate without much sucess
Screenshot of issue

Issue is similar in 12 other queries across the android app, and it was working fine with 1.4.5
Version info:
apolloGradlePlugin = "2.0.0"
androidGradlePlugin = "3.6.3"

Hi, this is definitely something we should fix but the screenshot and the report looks like a warning rather than an error.
If the compilation is failing, there must be another error. Can you have another look?

I'm guessing @mmahesh2019 has  kotlinOptions.allWarningsAsErrors = true. We are seeing the same compilation failures.

Thanks for clarifying. I'm going to set kotlinOptions.allWarningsAsErrors = true in our tests and will prevent this from happening in the future.

Just to clarify. No behavior has changed here. Since we converted API to Kotlin, Kotlin Compiler has more knowledge about nullability. That's why we have extra warnings now. They are not because of any change in code generation.

authors: BenSchwab mmahesh2019 tasomaniac

repo: apollographql/apollo-android


text:
Modular types and incremental imports

Follow up on: https://twitter.com/mdebbar/status/869249679862546432
I'll try to illustrate the example from Twitter with more details.
Let's say we have a simple schema for only posts and comments. The QueryType has one field getPost. And the PostType has a field comments.
When using graphql-js, the top-level schema file doesn't have to import everything. It only imports the types necessary for the top-level query fields, and everything else will be included through incremental imports:
// schema.js
import { GraphQLSchema, GraphQLObjectType } from 'graphql';
import PostType from './PostType.js';

const QueryType = new GraphQLObjectType({
  name: 'Query',
  fields: {
    getPost: {
      type: PostType,
      args: { id: ... },
      resolve: (root, { id }) => getPostFromDB(id),
    },
  },
});

export default new GraphQLSchema({
  query: QueryType,
});

// PostType.js
import { GraphQLObjectType, GraphQLList, GraphQLInt } from 'graphql';
import CommentType from './CommentType'

export default new GraphQLObjectType({
  name: 'Post',
  fields: {
    id: { type: GraphQLInt },
    comments: {
      type: new GraphQLList(CommentType),
      resolve: (post) => post.getCommentsFromDB(),
    },
  },
});

I like the approach of graphql-tools which looks cleaner and less verbose, but I couldn't find a way to achieve a structure similar to the above.
cc @stubailo

Hmm, right now you would have to do:
// PostType.js
import Comment from './CommentType';
import { merge } from 'lodash';

export default {
  typeDefs: [`
    type Post {
      id: Int
      comments: [Comment]
    }
  `, ...Comment.typeDefs],

  resolvers: merge({
    Post: {
      comments: (post) => post.getCommentsFromDB()
    }
  }, Comment.resolvers)
};

import { makeExecutableSchema } from 'graphql-tools';
import { merge, uniq } from 'lodash';
import PostType from './PostType.js';

const typeDefs = [`
  type Query {
    getPost(id: Int!): Post
  }
`, ...PostType.typeDefs];

const resolvers = merge({
  Query: {
    getPost: (root, { id }) => getPostFromDB(id),
  }
}, PostType.resolvers);

export default makeExecutableSchema({
  typeDefs: uniq(typeDefs),  // Need the uniq to avoid having the same string twice
  resolvers
});

I can see where you're coming from that it could be easier. I think something which makes it more natural to pass around a schema/resolver pair and merge them together would be a good fit inside this package, and would probably be just a few lines of code!
Would love to talk about a design for that.

One point here is that I think I just have a bit of a different mental model - in my mind I feel like all of the types to be kind of part of a whole that are all merged together, rather than small modules with interdependencies.
But we should support both ways of doing stuff, it's just a matter of having a nice unit to pass around/import.

Right. I agree with your point about the different mental models.
I think of types as an interconnected graph. And to define a schema, all we need to do is point to the Query node of that graph, and let everything else be figured out automatically:
                            Like *-------------|
                             *                 |
                             |                 |
                             *                 |
Schema * ---- * Query *---* Post *---* Comment |
                  *          *           *     |
                  |          |           |     |
                  |          *           |     |
                  |-------* User *-------|     |
                              *                |
                              |----------------|

So yeah, each type in my mind is a standalone node in the graph.

Sweet diagram! But yes I'm definitely into having a nicer API for that model.

@stubailo I'm trying to come up with a pattern for exporting types as a unit. One thing that I couldn't figure out is how to do the equivalent ofisTypeOf for unions and interfaces.

Any progress on this? Seems like still a great idea!

I think we should address this as part of a way to make things more modular/incremental, and I'm planning to use #185 as the main issue for that topic. Closing this but that doesn't mean it's not important!

authors: mdebbar stubailo

repo: apollographql/graphql-tools


text:
[qontract-reconcile-base] terraform (1.5.7) -> terraform (1.6.6)

upgrade from terraform 1.5.7 to terraform 1.6.6
let's choose between this and #110

going with this in light of https://www.hashicorp.com/blog/hashicorp-joins-ibm

repo: app-sre/container-images


text:
ace: Fix manifest arch error

The manifest contains values for the ACI arch and OS, not the go
language values.  To avoid confusion, change the symbols 'GOOS" and
'GOARCH' to 'ACI_OS' and 'ACI_ARCH', and provide conversion as
needed.
Fixes errors like these when run an ARM64 machines:
bad arch "arm64" for linux (must be one of: [amd64 i386 aarch64 aarch64_be armv6l armv7l armv7b ppc64 ppc64le s390x])

/cc @squeed

LGTM too.

authors: glevand lucab squeed

repo: appc/spec


text:
audacity 2.3.3

When running
C:\WINDOWS\system32>appget install audacity
I get:
Getting package audacity:latest
Beginning installation of 'audacity'

Downloading installer from https://fossies.org/windows/misc/audacity-win-2.3.3.exe
Request to https://fossies.org/windows/misc/audacity-win-2.3.3.exe failed. 410:Gone

I am able to ping fossies.org
C:\Users\username>ping fossies.org

Pinging fossies.org [144.76.163.196] with 32 bytes of data:
Reply from 144.76.163.196: bytes=32 time=156ms TTL=52
Reply from 144.76.163.196: bytes=32 time=168ms TTL=52
Reply from 144.76.163.196: bytes=32 time=160ms TTL=52
Reply from 144.76.163.196: bytes=32 time=157ms TTL=52

Ping statistics for 144.76.163.196:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 156ms, Maximum = 168ms, Average = 160ms

fossies doesn't store older versions of Audacity, only the current version (now 2.4.1) in that folder.
This issue is short-term fixed by: https://github.com/appget/appget.packages/pull/5504
Longer-term, every update of Audacity will break Appget until it's updated.
https://www.fosshub.com/Audacity-old.html has older versions and seems to also grab the current version ( https://www.fosshub.com/Audacity-old.html?dwl=audacity-win-2.4.1.exe ), so that should be a stable link across version updates.
Note: The Fosshub site is what's linked to on the main Audacity website: https://www.audacityteam.org/download/

authors: jcandresen skyeskie


text:
[ca]: Updates and fixes to Catalan translation

Updates and fixes for a very old translation

To: @michaelrsweet
Catalan CUPS translation ready!

Please submit this to the OpenPrinting CUPS project.

authors: bellaperez michaelrsweet

repo: apple/cups


text:
Can't specify --image xx (image2image)

Does the current ml-stable-diffusion support image2image? I'm having the following error when trying to use it. Any help/comment is appreciated.

ml-stable-diffusion version: the latest
Prompt entered:
swift run StableDiffusionSample --resource-path /Users/myUserName/ml-stable-diffusion/checkPoints/coreml-Deliberate/split-einsum/deliberate_v2_split-einsum --step-count 50 --compute-units cpuAndNeuralEngine --disable-safety --output-path ~/Downloads "A test description" --image-count 5 --image /Users/myUserName/Downloads/test\.png
Output with error:

Build complete! (0.08s)
Loading resources and creating pipeline
(Note: This can take a while the first time using these resources)
Sampling ...
StableDiffusion/Encoder.swift:96: Fatal error: Unexpectedly found nil while unwrapping an Optional value
[1]    20995 trace trap  swift run StableDiffusionSample --resource-path  --step-count 50        5```

Hey @ToddCool, could you please confirm that you have VAEEncoder.mlmodelc in your --resource-path? If not, you will need to run --convert-vae-encoder to generate it during the PyTorch to Core ML conversion phase.

authors: ToddCool atiorh

repo: apple/ml-stable-diffusion


text:
Package.swift: fix deprecated dependency initializers

Package dependency initializer overload with a name: argument was deprecate in recent versions of SwiftPM. Let's replace it with a non-deprecated version to fix the dependency warning.

@swift-ci test

@swift-ci test

@swift-ci test

@swift-ci test windows

@swift-ci test

@swift-ci test windows

@swift-ci test macos

repo: apple/sourcekit-lsp


text:
build: ensure that we build CoreFoundation for re-export

We wish to re-export CoreFoundation from Foundation.  In order to
accomplish this, the objects need to be built as if they are being built
for a DLL as they will be subsumed into the Foundation DLL.  We do not
necessarily wish to do this if we are building Foundation static
however.  Doing this ensures that we can re-export the symbols from
CoreFoundation on Windows.

@swift-ci please test

The problem with that is that because it is an external project, we cant do a TARGET_OBJECTS to get the objects and statically link the library.  We want to have the ability to build the CoreFoundation project as shared or static, and in the case of Foundation, statically with shared semantics.  I guess I can add another flag.

We haven't been sure we wanted to make CF "public" API on other platforms via Swift. It's not a great Swift API. I'd rather people use it as a C library.

@parkera - am I misunderstanding something, or are you recommending that we dynamically link to CoreFoundation then?  If so, that would solve the problem.

I'm just curious what the desired effect is here. If, from Swift, you import Foundation, do you want API like CFStringCreate to be available?

@parkera - oh, the desired effect here is that users (i.e. XCTest) can use the CoreFoundation APIs just like they currently can on Darwin and Linux (as @millenomi has stated is the desired behaviour)

Another flag seems like the best option; overriding CMAKE_C_FLAGS will just leave us with the same Linux build failures as last time 😕

I talked with @parkera and @compnerd.
We want to limit how much people use CF; we want them to use the Foundation API instead, and we already prevent CF symbols from being imported by import Foundation on Linux.
I have proposed to @compnerd that, since Windows is a new platform here, if it requires going out of our way to achieve export, that we just don't on that platform, and we correct the XCTest issues by providing SPI for their use only. If we determine downstream support requires CF for interesting packages, we can always revisit. The idea here is to keep the status quo on Linux, so we don't break existing downstream code.

Addedum: @compnerd: can we limit XCTest SPI presence to Windows for now?

Closing this off, I'll add the SPI in a different PR; this was motivated for XCTest, so, no longer need this.

authors: compnerd gwynne millenomi parkera

repo: apple/swift-corelibs-foundation