text:
[SPARK-48398] Add Helm Chart

What changes were proposed in this pull request?
This PR proposes Spark Kubernetes Operator Helm chart, which describes deployment manifest for the operator.
Why are the changes needed?
Helm chart helps users to install, upgrade and uninstall Spark Operator in Kubernetes cluster. It provides a cloud native way of managing Operator in place. This chart helps us to complete the end-to-end flow for the operator app.
Does this PR introduce any user-facing change?
No (not yet released)
How was this patch tested?

CI added for chart linting
Chart test for operator deployment RBAC validation
We are also working on SPARK-49214 to include the test automation in workflow. Till that is resolved, this patch can be validated manually by running the following steps:

Start minikube
Start miniKube and make it access locally-built image
minikube start
eval $(minikube docker-env)

Build Spark Operator Locally
# Build a local container image which can be used for minikube.etc.
docker build --build-arg APP_VERSION=0.1.0 -t spark-kubernetes-operator:0.1.0 -f build-tools/docker/Dockerfile  .

# Generate CRD yaml and make it available for chart deployment
./gradlew spark-operator-api:relocateGeneratedCRD

Install the Spark Operator
helm install spark-kubernetes-operator --create-namespace -f build-tools/helm/spark-kubernetes-operator/values.yaml build-tools/helm/spark-kubernetes-operator/

Verify the Installation
helm test spark-kubernetes-operator

Here's a sample spark app yaml
apiVersion: org.apache.spark/v1alpha1
kind: SparkApplication
metadata:
  name: spark-pi
spec:
  mainClass: "org.apache.spark.examples.SparkPi"
  jars: "local:///opt/spark/examples/jars/spark-examples_2.13-4.0.0-preview1.jar"
  sparkConf:
    spark.executor.instances: "5"
    spark.kubernetes.container.image: "spark:4.0.0-preview1"
    spark.kubernetes.authenticate.driver.serviceAccountName: "spark"
  applicationTolerations:
    resourceRetainPolicy: OnFailure
  runtimeVersions:
    scalaVersion: "2.13"
    sparkVersion: "4.0.0-preview1"

Create a file named spark-pi.yaml and run
kubectl create -f spark-pi.yaml

The operator should be able to operate given app from start to complete.
You can delete job with the following.
$ kubectl delete -f spark-pi.yaml
sparkapplication.org.apache.spark "spark-pi" deleted

Uninstallation
To remove the installed resources from your cluster, reset environment to the defaults and
shutdown the cluster:
helm uninstall spark-kubernetes-operator
eval $(minikube docker-env --unset)
minikube stop

Was this patch authored or co-authored using generative AI tooling?

No

I revised this PR, @jiangzho and @viirya .

Use Apache Spark 4.0.0-preview1 as the example.
Use helm install instead of helm upgrade to avoid the initial failure.
Add Spark Job deletion example, kubectl delete -f spark-pi.yaml
Add --strict on helm linting.
Renaming during copy CRDs, rename '(.+).yml', '$1.yaml'.
Minor code clean up to focus on the main feature.

diff --git a/.github/.licenserc.yaml b/.github/.licenserc.yaml
index 6cafe63..5507e4d 100644
--- a/.github/.licenserc.yaml
+++ b/.github/.licenserc.yaml
@@ -19,6 +19,5 @@ header:
     - 'build/**'
     - '**/*.json'
     - '**/.helmignore'
-    - '**/*.yml'

   comment: on-failure
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 2ca7bd2..fd99dfc 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -46,7 +46,7 @@ jobs:
           ./gradlew build
       - name: Validate helm chart linting
         run: |
-          helm lint build-tools/helm/spark-kubernetes-operator
+          helm lint --strict build-tools/helm/spark-kubernetes-operator
   build-image:
     name: "Build Operator Image CI"
     runs-on: ubuntu-latest
diff --git a/.gitignore b/.gitignore
index d5afef7..b174a96 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,6 +6,7 @@
 .vscode
 /lib/
 target/
+build-tools/helm/spark-kubernetes-operator/crds

 # Gradle Files #
 ################
@@ -16,7 +17,3 @@ build
 dependencies.lock
 **/dependencies.lock
 gradle/wrapper/gradle-wrapper.jar
-
-# Generated Files #
-###################
-build-tools/helm/spark-kubernetes-operator/crds
diff --git a/build-tools/helm/spark-kubernetes-operator/.helmignore b/build-tools/helm/spark-kubernetes-operator/.helmignore
index 0e8a0eb..5920392 100644
--- a/build-tools/helm/spark-kubernetes-operator/.helmignore
+++ b/build-tools/helm/spark-kubernetes-operator/.helmignore
@@ -1,15 +1,4 @@
-# Patterns to ignore when building packages.
-# This supports shell glob matching, relative path matching, and
-# negation (prefixed with !). Only one pattern per line.
 .DS_Store
-# Common VCS dirs
-.git/
-.gitignore
-.bzr/
-.bzrignore
-.hg/
-.hgignore
-.svn/
 # Common backup files
 *.swp
 *.bak
diff --git a/build-tools/helm/spark-kubernetes-operator/Chart.yaml b/build-tools/helm/spark-kubernetes-operator/Chart.yaml
index 46e612a..deed565 100644
--- a/build-tools/helm/spark-kubernetes-operator/Chart.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/Chart.yaml
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,9 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################
-
----
 apiVersion: v2
 name: spark-kubernetes-operator
 description: A Helm chart for the Apache Spark Kubernetes Operator
diff --git a/build-tools/helm/spark-kubernetes-operator/conf/log4j2.properties b/build-tools/helm/spark-kubernetes-operator/conf/log4j2.properties
deleted file mode 100644
index 3a0256b..0000000
--- a/build-tools/helm/spark-kubernetes-operator/conf/log4j2.properties
+++ /dev/null
@@ -1,51 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-status=info
-strict=true
-dest=out
-name=PropertiesConfig
-property.filename=/opt/spark-operator/logs/spark-operator
-filter.threshold.type=ThresholdFilter
-filter.threshold.level=debug
-# console
-appender.console.type=Console
-appender.console.name=STDOUT
-appender.console.layout.type=PatternLayout
-appender.console.layout.pattern=%d %p %X %C{1.} [%t] %m%n
-appender.console.filter.threshold.type=ThresholdFilter
-appender.console.filter.threshold.level=info
-# rolling JSON
-appender.rolling.type=RollingFile
-appender.rolling.name=RollingFile
-appender.rolling.append=true
-appender.rolling.fileName=${filename}.log
-appender.rolling.filePattern=${filename}-%i.log.gz
-appender.rolling.layout.type=JsonTemplateLayout
-appender.rolling.layout.eventTemplateUri=classpath:EcsLayout.json
-appender.rolling.policies.type=Policies
-appender.rolling.policies.size.type=SizeBasedTriggeringPolicy
-appender.rolling.policies.size.size=100MB
-appender.rolling.strategy.type=DefaultRolloverStrategy
-appender.rolling.strategy.max=20
-appender.rolling.immediateFlush=true
-# chatty loggers
-rootLogger.level=all
-logger.netty.name=io.netty
-logger.netty.level=warn
-log4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
-rootLogger.appenderRef.stdout.ref=STDOUT
-rootLogger.appenderRef.rolling.ref=RollingFile
diff --git a/build-tools/helm/spark-kubernetes-operator/conf/spark-operator.properties b/build-tools/helm/spark-kubernetes-operator/conf/spark-operator.properties
deleted file mode 100644
index e644970..0000000
--- a/build-tools/helm/spark-kubernetes-operator/conf/spark-operator.properties
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Property Overrides. e.g.
-# spark.kubernetes.operator.reconciler.intervalSeconds=60
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl b/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl
index 526b778..e8ee901 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl
+++ b/build-tools/helm/spark-kubernetes-operator/templates/_helpers.tpl
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################

 {{/*
 Expand the name of the chart.
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/sparkapps-resource.yaml b/build-tools/helm/spark-kubernetes-operator/templates/app-rbac.yaml
similarity index 96%
rename from build-tools/helm/spark-kubernetes-operator/templates/sparkapps-resource.yaml
rename to build-tools/helm/spark-kubernetes-operator/templates/app-rbac.yaml
index 613e489..f76d56b 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/sparkapps-resource.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/app-rbac.yaml
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################

 {{/*
 RBAC rules used to create the app (cluster)role based on the scope
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
similarity index 95%
rename from build-tools/helm/spark-kubernetes-operator/templates/rbac.yaml
rename to build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
index e03809e..f8364e6 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################

 {{/*
 RBAC rules used to create the operator (cluster)role
@@ -61,7 +59,7 @@ Labels and annotations to be applied on rbacResources
 {{- end }}

 ---
-#Service account and rolebindings for operator
+# Service account and rolebindings for operator
 apiVersion: v1
 kind: ServiceAccount
 metadata:
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
index 555b239..5edd228 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################

 apiVersion: apps/v1
 kind: Deployment
@@ -25,7 +23,6 @@ metadata:
     {{- include "spark-operator.commonLabels" . | nindent 4 }}
 spec:
   replicas: {{ .Values.operatorDeployment.replicas }}
-  revisionHistoryLimit: 2
   strategy:
     {{- toYaml .Values.operatorDeployment.strategy | nindent 4 }}
   selector:
diff --git a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
index 03fc4b1..33d8fa3 100644
--- a/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/templates/tests/test-rbac.yaml
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,8 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################
-
 apiVersion: v1
 kind: Pod
 metadata:
diff --git a/build-tools/helm/spark-kubernetes-operator/values.yaml b/build-tools/helm/spark-kubernetes-operator/values.yaml
index 16ecc5e..87fda1e 100644
--- a/build-tools/helm/spark-kubernetes-operator/values.yaml
+++ b/build-tools/helm/spark-kubernetes-operator/values.yaml
@@ -1,4 +1,3 @@
-################################################################################
 # Licensed to the Apache Software Foundation (ASF) under one or more
 # contributor license agreements.  See the NOTICE file distributed with
 # this work for additional information regarding copyright ownership.
@@ -13,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-################################################################################

 image:
   repository: spark-kubernetes-operator
@@ -43,7 +41,7 @@ operatorDeployment:
     # https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/
     topologySpreadConstraints: [ ]
     operatorContainer:
-      jvmArgs: "-XX:+UseG1GC -Xms3G -Xmx3G -Dfile.encoding=UTF8"
+      jvmArgs: "-Dfile.encoding=UTF8"
       env:
       envFrom:
       volumeMounts: { }
@@ -76,15 +74,7 @@ operatorDeployment:
         seccompProfile:
           type: RuntimeDefault
     additionalContainers: { }
-    # additionalContainers:
-    #  - name: ""
-    #    image: ""
     volumes: { }
-    # volumes:
-    #   - name: spark-artifacts
-    #     hostPath:
-    #       path: /tmp/spark/artifacts
-    #       type: DirectoryOrCreate
     securityContext: { }
     dnsPolicy:
     dnsConfig:
@@ -191,5 +181,5 @@ operatorConfiguration:
     annotations:
       "helm.sh/resource-policy": keep
     data:
-    # Spark Operator Config Runtime Properties Overrides. e.g.
+      # Spark Operator Config Runtime Properties Overrides. e.g.
       spark.kubernetes.operator.reconciler.intervalSeconds: 60
diff --git a/dev/.rat-excludes b/dev/.rat-excludes
index a556cb7..4234634 100644
--- a/dev/.rat-excludes
+++ b/dev/.rat-excludes
@@ -16,4 +16,3 @@ build
 .*gradle
 .*json
 .helmignore
-.*yml
diff --git a/spark-operator-api/build.gradle b/spark-operator-api/build.gradle
index 13b80e4..8b1dfc3 100644
--- a/spark-operator-api/build.gradle
+++ b/spark-operator-api/build.gradle
@@ -38,4 +38,5 @@ tasks.register('relocateGeneratedCRD', Copy) {
   dependsOn finalizeGeneratedCRD
   from "build/classes/java/main/META-INF/fabric8/sparkapplications.org.apache.spark-v1.yml"
   into "../build-tools/helm/spark-kubernetes-operator/crds"
+  rename '(.+).yml', '$1.yaml'
 }

authors: dongjoon-hyun jiangzho

repo: apache/spark-kubernetes-operator


text:
[SPARK-12024][SQL] More efficient multi-column counting.

In https://github.com/apache/spark/pull/9409 we enabled multi-column counting. The approach taken in that PR introduces a bit of overhead by first creating a row only to check if all of the columns are non-null.
This PR fixes that technical debt. Count now takes multiple columns as its input. In order to make this work I have also added support for multiple columns in the single distinct code path.
cc @yhuai

Test build #46814 has started for PR 10015 at commit 3fe9cf3.

Test build #46814 has finished for PR 10015 at commit 3fe9cf3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * case class Count(children: Seq[Expression]) extends DeclarativeAggregate \n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46814/
Test PASSed.

LGTM. Merging to master and 1.6.

source: gharchive/pull-request

authors: AmplabJenkins SparkQA hvanhovell yhuai

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-15350][mllib]add unit test function for LogisticRegressionWithLBFGS in JavaLogisticRegressionSuite

What changes were proposed in this pull request?
add unit test function for LogisticRegressionWithLBFGS in JavaLogisticRegressionSuite,
and update the test function name in the class to distinct LR with SGD and LBFGS.
How was this patch tested?
The testsuite I modified.

Test build #58643 has started for PR 13136 at commit a8593ab.

Test build #58643 has finished for PR 13136 at commit a8593ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58643/
Test PASSed.

authors: AmplabJenkins SparkQA WeichenXu123


text:
FIX: AggregatedDialect does not quote identifier correctly

What changes were proposed in this pull request?
If we write an custom MySQL Dialect and register it, it will be fused as an AggregatedDialect. But AggregatedDialect does not override quoteIdentifier method, so it will use JDBCDialect's quoteIdentifier method, quote identifier with ", which is not correct.
How was this patch tested?
I have manually test this patch. It works as expected.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

authors: AmplabJenkins HelloCodeMing


text:
[WIP][TEST] Experiment with running Scoverage in tests

WIP to test Scoverage integration in AMPLab Jenkins.

Test build #65996 has started for PR 15269 at commit 9f77155.

Test build #65996 has finished for PR 15269 at commit 9f77155.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65996/
Test FAILed.

Test build #65998 has started for PR 15269 at commit 9b8e284.

Test build #65998 has finished for PR 15269 at commit 9b8e284.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65998/
Test FAILed.

Test build #66049 has started for PR 15269 at commit e22b406.

Test build #66049 has finished for PR 15269 at commit e22b406.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66049/
Test FAILed.

Merged build finished. Test FAILed.

I know this is only WIP but I just want to say +1 to the idea assuming we can iron out the kinks :) (It's been on my back burner for far too long).

authors: AmplabJenkins JoshRosen SparkQA holdenk


text:
[SPARK-24695] [SQL]: To add support to return Calendar interval from udf.

What changes were proposed in this pull request?
This change adds capability to return Calender interval from udf.
Earlier, the  udf  of Type (String => CalendarInterval) was throwing Exception stating:
Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
java.lang.UnsupportedOperationException: Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
at org.apache.spark.sql.catalyst.ScalaReflection391anonfun.apply(ScalaReflection.scala:781)
How was this patch tested?
Added test case in ScalaReflectionSuite.scala and ExpressionEncoderSuite.scala
Also, tested by creating an udf that returns Calendar interval.
jira entry for detail: https://issues.apache.org/jira/browse/SPARK-24695

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Since CalendarInterval is an internal class, I think users are not intended to use the class directly...

btw, can you update the title like [SPARK-24695][SQL]...

org.apache.spark.unsafe.types.CalenderInterval is already public, am i missing something.
Also, what if i want to do some computation on any data type and return Calender Interval.. How should i solve this problem in current scenario.

Actually, the unsafe package does not include user-facing classes.

Yea, so it's kind of half public now but not completely exposed. It should be exposed before we extend its support.

What if i make changes to expose it?

Can one of the admins verify this patch?

I would better open a discussion thread in the mailing list before making a PR to expose it. Shell we leave this close this meanwhile? I think it's better to leave active and actionable PRs only.

Can one of the admins verify this patch?

I think we should close this for now then.

For your information, @cloud-fan and I are discussing how we expose calendar interval data types. Will post the proposal later.

I think this one is reasonable. We already exposed it as an argument to a UDF, so it is pretty poor UX to not allow a return type. As for making CalendarInterval public, what is there to do besides moving it?
Reopening this one.

@priyankagargnitk can you update this one.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@priyankagargnitk ping, are you still there?

I am working on this, I'll update the PR in a day or two.

A new PR has been raised.

https://github.com/apache/spark/pull/25022

authors: AmplabJenkins HyukjinKwon gatorsmile hvanhovell maropu priyankagargnitk


text:
[SPARK-25662][TEST] Refactor DataSourceReadBenchmark to use main method

What changes were proposed in this pull request?

Refactor DataSourceReadBenchmark

How was this patch tested?
Manually tested and regenerated results.
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.DataSourceReadBenchmark"

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #97227 has started for PR 22664 at commit cf61f1c.

Test build #97227 has finished for PR 22664 at commit cf61f1c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97227/
Test FAILed.

Hi, @peter-toth .
Could you review and merge https://github.com/peter-toth/spark/pull/1 which contains the result on EC2 r3.xlarge?

Test build #97237 has started for PR 22664 at commit 7cef8db.

Could you add [SQL] before [TEST], too?

Test build #97237 has finished for PR 22664 at commit 7cef8db.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97237/
Test FAILed.

Retest this please.

Test build #97241 has started for PR 22664 at commit 7cef8db.

Hi, @dbtsai .
Since this is DataSourceReadBenchmark, could you review and merge this?

Test build #97241 has finished for PR 22664 at commit 7cef8db.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97241/
Test FAILed.

Retest this please.

Test build #97272 has started for PR 22664 at commit 7cef8db.

Thanks @dongjoon-hyun for ping me. LGTM too. We're working on some parquet reader improvement, and this will be useful.

Test build #97272 has finished for PR 22664 at commit 7cef8db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97272/
Test PASSed.

Thank you, @dbtsai and @peter-toth .

@peter-toth I assigned to you. Thanks for contribution.

authors: AmplabJenkins SparkQA dbtsai dongjoon-hyun peter-toth


text:
[SPARK-28912][STREAMING] Fixed MatchError in getCheckpointFiles()

What changes were proposed in this pull request?

This change fixes issue SPARK-28912.
Why are the changes needed?

If checkpoint directory is set to name which matches regex pattern used for checkpoint files then logs are flooded with MatchError exceptions and old checkpoint files are not removed.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Manually.

Start Hadoop in a pseudo-distributed mode.

In another terminal run command  nc -lk 9999

In the Spark shell execute the following statements:
val ssc = new StreamingContext(sc, Seconds(30))
ssc.checkpoint("hdfs://localhost:9000/checkpoint-01")
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.print()
ssc.start()
ssc.awaitTermination()

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Welcome to Apache Spark community. Thank you for your first contribution, @avkgh .

ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15042/
Test PASSed.

Test build #110022 has started for PR 25654 at commit b990196.

Test build #110022 has finished for PR 25654 at commit b990196.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110022/
Test FAILed.

Test build #110043 has started for PR 25654 at commit 812d867.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15062/
Test PASSed.

Test build #110043 has finished for PR 25654 at commit 812d867.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110043/
Test PASSed.

Nice catch, +1 on unit test since old checkpoint files are not removed can be asserted.

Gentle ping, @avkgh .

I will add a unit test when I have the time. I plan to do it within next 2 days.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15235/
Test PASSed.

Test build #110239 has started for PR 25654 at commit 1835fe4.

Test build #110239 has finished for PR 25654 at commit 1835fe4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110239/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15239/
Test PASSed.

Test build #110242 has started for PR 25654 at commit 4c3300d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15247/
Test PASSed.

Test build #110250 has started for PR 25654 at commit 6c50007.

Test build #110250 has finished for PR 25654 at commit 6c50007.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110250/
Test FAILed.

Test build #110242 has finished for PR 25654 at commit 4c3300d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110242/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15249/
Test PASSed.

Test build #110252 has started for PR 25654 at commit eebc393.

Test build #110252 has finished for PR 25654 at commit eebc393.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110252/
Test PASSed.

OK, I will replace the unit test.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15264/
Test PASSed.

Test build #110268 has started for PR 25654 at commit 90f1e8d.

Test build #110268 has finished for PR 25654 at commit 90f1e8d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110268/
Test PASSed.

You are added to the Apache Spark contributor group, @avkgh .

Thank you, @dongjoon-hyun, @HyukjinKwon , @srowen , @gaborgsomogyi !

[error] /home/jenkins/workspace/spark-branch-2.4-compile-maven-hadoop-2.6/streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala:850: not found: value withTempDir
[error]     withTempDir { tempDir =>
[error]     ^
[error] one error found
[error] Compile failed at Sep 6, 2019 6:08:02 PM [8.547s]

We need to revert this from 2.4 branch.

Reverted.

@avkgh Could you submit a PR against 2.4 branch?

I submitted a PR against branch-2.4.

Oops. My bad. Thank you for recovering branch-2.4, @gatorsmile !

authors: AmplabJenkins SparkQA avkgh dongjoon-hyun gaborgsomogyi gatorsmile


text:
[SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched

What changes were proposed in this pull request?
Use TaskSetManager. abort  to abort a barrier stage instead of throwing exception within resourceOffers.
Why are the changes needed?
Any non fatal exception thrown within Spark RPC framework can be swallowed:
https://github.com/apache/spark/blob/100fc58da54e026cda87832a10e2d06eaeccdf87/core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala#L202-L211
The method TaskSchedulerImpl.resourceOffers is also within the scope of Spark RPC framework. Thus, throw exception inside resourceOffers won't fail the application.
As a result, if a barrier stage fail the require check at require(addressesWithDescs.size == taskSet.numTasks, ...), the barrier stage will fail the check again and again util all tasks from TaskSetManager dequeued.   But since the barrier stage isn't really executed, the application will hang.
The issue can be reproduced by the following test:
initLocalClusterSparkContext(2)
val rdd0 = sc.parallelize(Seq(0, 1, 2, 3), 2)
val dep = new OneToOneDependency[Int](rdd0)
val rdd = new MyRDD(sc, 2, List(dep), Seq(Seq("executor_h_0"),Seq("executor_h_0")))
rdd.barrier().mapPartitions { iter =>
  BarrierTaskContext.get().barrier()
  iter
}.collect()

Does this PR introduce any user-facing change?
Yes, application hang previously but fail-fast after this fix.
How was this patch tested?
Added a regression test.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26533/
Test PASSed.

cc @cloud-fan @tgravescs @mridulm

Test build #121862 has started for PR 28356 at commit 0f9a186.

Test build #121862 has finished for PR 28356 at commit 0f9a186.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121862/
Test FAILed.

retest this please

Test build #121875 has started for PR 28356 at commit 0f9a186.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26546/
Test PASSed.

Test build #121875 has finished for PR 28356 at commit 0f9a186.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121875/
Test FAILed.

retest this please

Test build #121882 has started for PR 28356 at commit 0f9a186.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26554/
Test PASSed.

Test build #121882 has finished for PR 28356 at commit 0f9a186.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121882/
Test PASSed.

thanks, merging to 3.0!

authors: AmplabJenkins Ngone51 SparkQA cloud-fan


text:
[SPARK-32645][INFRA] Upload unit-tests.log as an artifact

What changes were proposed in this pull request?
This PR proposes to upload target/unit-tests.log into the artifact so it will be able to download here:

Why are the changes needed?
Jenkins has this feature. It should be best to have the same dev functionalities with it.
Also, note that this was pointed out https://github.com/apache/spark/pull/29225#discussion_r471485011.
Does this PR introduce any user-facing change?
No, dev-only
How was this patch tested?
GitHub Actions build in the current PR will test it out.

Test build #127521 has started for PR 29454 at commit 0e18705.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127521/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32144/
Test PASSed.

cc @Ngone51 and @tgravescs FYI. This is being tested at https://github.com/apache/spark/actions/runs/213000777

Test build #127523 has started for PR 29454 at commit bd7ff11.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32146/
Test PASSed.

Should be ready for a look or possibly to be merged.

Test build #127523 has finished for PR 29454 at commit bd7ff11.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127523/
Test PASSed.

cc @dongjoon-hyun, @gengliangwang, @viirya FYI

Test build #127565 has started for PR 29454 at commit d787cd1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32188/
Test PASSed.

I've downloaded the log from: https://github.com/apache/spark/actions/runs/213000777 and it looks good!
LGTM

Test build #127565 has finished for PR 29454 at commit d787cd1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127565/
Test FAILed.

GitHub keeps 90 days (https://docs.github.com/en/actions/configuring-and-managing-workflows/managing-a-workflow-run#viewing-logs-to-diagnose-failures) but I think there's another internal limit .. given my random googling.

Nice feature!

Thanks guys. Merged to master.

authors: AmplabJenkins HyukjinKwon Ngone51 SparkQA maropu


text:
[SPARK-35338][PYTHON] Separate arithmetic operations into data type based structures

What changes were proposed in this pull request?

The PR is proposed for pandas APIs on Spark, in order to separate arithmetic operations shown as below into data-type-based structures.
__add__, __sub__, __mul__, __truediv__, __floordiv__, __pow__, __mod__, __radd__, __rsub__, __rmul__, __rtruediv__, __rfloordiv__, __rpow__,__rmod__
DataTypeOps and subclasses are introduced.
The existing behaviors of each arithmetic operation should be preserved.
Why are the changes needed?

Currently, the same arithmetic operation of all data types is defined in one function, so it’s difficult to extend the behavior change based on the data types.
Introducing DataTypeOps would be the foundation for pandas APIs on Spark: Separate basic operations into data type based structures..
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Tests are introduced under pyspark.pandas.tests.data_type_ops. One test file per DataTypeOps class.

Test build #138639 has started for PR 32469 at commit 578ac9f.

Test build #138639 has finished for PR 32469 at commit 578ac9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43159/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138639/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43159/

Test build #138648 has started for PR 32469 at commit 86e2809.

Test build #138648 has finished for PR 32469 at commit 86e2809.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138648/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43168/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43168/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43168/

Test build #138656 has started for PR 32469 at commit 114908a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43176/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43176/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43176/

Test build #138656 has finished for PR 32469 at commit 114908a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138656/

Test build #138688 has started for PR 32469 at commit 4a0f272.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43209/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43209/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43209/

Test build #138688 has finished for PR 32469 at commit 4a0f272.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138688/

Test build #138694 has started for PR 32469 at commit 9cfb617.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43215/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43215/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43215/

Test build #138694 has finished for PR 32469 at commit 9cfb617.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138694/

Thanks! merging to master.

Seems like this breaks the Python linter.
mypy checks failed:
python/pyspark/pandas/data_type_ops/base.py:101: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/base.py:104: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/base.py:107: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/base.py:110: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/base.py:113: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/base.py:116: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/base.py:119: error: Invalid signature "def (pyspark.pandas.data_type_ops.base.DataTypeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/string_ops.py:79: error: Invalid signature "def (pyspark.pandas.data_type_ops.string_ops.StringOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/string_ops.py:85: error: Invalid signature "def (self: pyspark.pandas.data_type_ops.string_ops.StringOps, left: Any, right: Any) -> Any"
python/pyspark/pandas/data_type_ops/string_ops.py:88: error: Invalid signature "def (pyspark.pandas.data_type_ops.string_ops.StringOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/string_ops.py:94: error: Invalid signature "def (self: pyspark.pandas.data_type_ops.string_ops.StringOps, left: Any, right: Any) -> Any"
python/pyspark/pandas/data_type_ops/string_ops.py:97: error: Invalid signature "def (self: pyspark.pandas.data_type_ops.string_ops.StringOps, left: Any, right: Any) -> Any"
python/pyspark/pandas/data_type_ops/string_ops.py:100: error: Invalid signature "def (self: pyspark.pandas.data_type_ops.string_ops.StringOps, left: Any, right: Any) -> Any"
python/pyspark/pandas/data_type_ops/string_ops.py:103: error: Invalid signature "def (self: pyspark.pandas.data_type_ops.string_ops.StringOps, left: Any, right: Any) -> Any"
python/pyspark/pandas/data_type_ops/num_ops.py:123: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.NumericOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:131: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.NumericOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:138: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.NumericOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:145: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.NumericOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:156: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.NumericOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:247: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.IntegralOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:260: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.IntegralOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:354: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.FractionalOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/num_ops.py:367: error: Invalid signature "def (pyspark.pandas.data_type_ops.num_ops.FractionalOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/datetime_ops.py:60: error: Invalid signature "def (pyspark.pandas.data_type_ops.datetime_ops.DatetimeOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
python/pyspark/pandas/data_type_ops/date_ops.py:59: error: Invalid signature "def (pyspark.pandas.data_type_ops.date_ops.DateOps, Any, Any) -> Union[pyspark.pandas.series.Series[Any], pyspark.pandas.indexes.base.Index]"
Found 25 errors in 5 files (checked 304 source files)

I'm going to revert this. @xinrong-databricks Could you take a look?

authors: AmplabJenkins SparkQA ueshin xinrong-databricks


text:
[SPARK-37411][PYTHON][ML] Inline hints for pyspark.ml.regression

What changes were proposed in this pull request?

This PR migrates type pyspark.ml.regression annotations from stub file to inline type hints.
Why are the changes needed?

Part of ongoing migration of type hints.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Existing tests.

cc @itholic @ueshin  @xinrong-databricks, in case you have time to take a look :)

Looks pretty good

Merged into master.
Thanks!


text:
WIP: [SPARK-27998] [SQL] Add support for double-quoted named expressions

What changes were proposed in this pull request?

Spark SQL currently only supports backticks for identifiers. ANSI SQL allows double-quoted strings for identifiers.
This PR adds support for double-quoted identifiers for named expressions (column aliases) without changing the use of double-quoted strings as SQL strings in other contexts. Backticks are also still supported.
Specific changes:

Split STRING lexer rule into SINGLE_QUOTED_STRING and DOUBLE_QUOTED_STRING
Update most parser rules to allow SINGLE_QUOTED_STRING or DOUBLE_QUOTED_STRING instead of STRING
Update parser rule for namedExpression to allow quotedIdentifier (backticks) or DOUBLE_QUOTED_STRING

Why are the changes needed?

To allow ANSI SQL double-quoted identifiers to be used for column aliases.
Does this PR introduce any user-facing change?

Yes, it allows double quotes to be used as well as backticks for named expressions.
How was this patch tested?

New UT added.

Can one of the admins verify this patch?

I think this this is worrysome for two reasons

It adds confusion between what is a string and what is an identifier.
It singles out a specific kind of an identifier (column) and a specific narrow kind of context (column aliasing in the select list).

I totally  for a switch that sets whether double quotes are used for identifiers or strings.
but it should be:

Pick one
Holistic

Can one of the admins verify this patch?

I think this this is worrysome for two reasons
1. It adds confusion between what is a string and what is an identifier.

2. It singles out a specific kind of an identifier (column) and a specific narrow kind of context (column aliasing in the select list).

I'm totally for a switch that sets whether double quotes are used for identifiers or strings. but it should be:
1. Pick one

2. Holistic

My original PR was to change the meaning of double quotes to be identifiers instead of strings but the feedback was that it was too late to make a change this large. However, maybe we can introduce a configuration option to determine how this is handled, as suggested here.

please retest

I think this this is worrysome for two reasons
1. It adds confusion between what is a string and what is an identifier.

2. It singles out a specific kind of an identifier (column) and a specific narrow kind of context (column aliasing in the select list).

I'm totally for a switch that sets whether double quotes are used for identifiers or strings. but it should be:
1. Pick one

2. Holistic

My original PR was to change the meaning of double quotes to be identifiers instead of strings but the feedback was that it was too late to make a change this large. However, maybe we can introduce a configuration option to determine how this is handled, as suggested here.

I think a config would be appropriate.

authors: AmplabJenkins andygrove runitao srielau


text:
Revert "[SPARK-40049][SQL][TESTS] Add ReplaceNullWithFalseInPredicateWithAQEEndToEndSuite"

What changes were proposed in this pull request?
This PR reverts commit 1a1e4938f97ecedbb99a440df27d365439a9a621.
Why are the changes needed?
Based on the follow up discussion, concluded  ReplaceNullWithFalseInPredicateWithAQEEndToEndSuite is not necessary
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests

cc @cloud-fan

Merged to master.

Thank you for the quick merge

LGTM, thanks!

authors: HyukjinKwon cloud-fan kazuyukitanimura


text:
[SPARK-41087][BUILD] Remove duplicate -Xmx4g from dev/make-distribution.sh and make build/mvn use the same JAVA_OPTS

What changes were proposed in this pull request?
This pr aims to make build/mvn use the same JAVA_OPTS as dev/make-distribution.sh to speed up mvn build.
In addition, this pr deletes the duplicate -Xmx4g option  from make-distribution.sh
Why are the changes needed?
To speed up mvn build.
Run the following commands:
build/mvn clean -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -DskipTests
build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -DskipTests

Before

x86 Linux: [INFO] Total time:  01:22 h ([INFO] Spark Project Examples ............................. SUCCESS [41:30 min])
M1 MacOs: [INFO] Total time:  01:26 h ([INFO] Spark Project Examples ............................. SUCCESS [41:59 min])

After

x86 Linux: [INFO] Total time:  29:11 min（[INFO] Spark Project Examples ............................. SUCCESS [01:22 min])
M1 MacOs: [INFO] Total time:  25:14 min ([INFO] Spark Project Examples ............................. SUCCESS [01:08 min])

From above result, new JAVA_OPTS reduce mvn build time by 65%+
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions

cc @dongjoon-hyun is this one ok?

Wait me check maven test all modules

Could you revise the PR title according to the new change?

done

Tests are failing because of this.  https://github.com/apache/spark/commit/c2a8e48e70abfb6bd101c99c5a0f6017151fc85e

rebased, thanks @bjornjorgensen

maven test all passed

BTW, @LuciferYang . Did you run the full Maven testing with this configuration? dev/make-distribution.sh is simply for building while build/mvn should support testing. Please run a full testing to verify your PR doesn't break anything in Maven testing.

maven test all passed

Did you run the full Maven testing with this configuration?

Yes. 4 days ago, I checked the full UTs with this configuration and all test passed.
For -XX:ReservedCodeCacheSize=128m, there will be still CodeCache is full. Compiler has been disabled. related warnings during the full test, but there is no test failed. I am investigating a suitable value to unify this option, and want to change other places, such as scala-maven-plugin, maven-surefire-plugin, and scalatest-maven-plugin.
@dongjoon-hyun Are you sure -XX:ReservedCodeCacheSize=1g is a more suitable option?  If confirmed, I can use -XX:ReservedCodeCacheSize=1g for compilation and full UTs check again, and then change the option in other places to -XX:ReservedCodeCacheSize=1g if it is better

If you verified it, it will be okay. So, could you describe your 4-days-ago test environment a little more to us, @LuciferYang ?

The test is divided into two parts:

Bare metal server, Intel(R) Xeon(R) Gold 6271C CPU @ 2.60GHz with Java 8u352, test command as follows:

build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -fn

All test passed, but in this part , I skip the connector/connect and connector/protobuf modules because they can't compiled on the centos7u4 OS now（I'm trying to solve this issue）
2.Mac OS/Apple Silicon, M1 with Java 17.0.5, test command as follows:
build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -fn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest

Thanks @dongjoon-hyun, as mentioned earlier, I will further find for a more suitable ReservedCodeCacheSize  to make the build test process more stable and fast


text:
[WIP] [SPARK-41659] [CONNECT] Enable doctests in pyspark.sql.connect.readwriter

What changes were proposed in this pull request?
This PR proposes to enable doctests in pyspark.sql.connect.readwriter that is virtually the same as pyspark.sql.readwriter.
Why are the changes needed?
To make sure on the PySpark compatibility and test coverage.
Does this PR introduce any user-facing change?
No, doctest's only.
How was this patch tested?
New Doctests Added

@zhengruifeng i think we should cover dictionary cases too (like we did in https://github.com/apache/spark/pull/39313):
Failed example:
    with tempfile.TemporaryDirectory() as d:
        # Write a DataFrame into a JSON file
        spark.createDataFrame(
            [{"age": 100, "name": "Hyukjin Kwon"}]
        ).write.mode("overwrite").format("json").save(d)

        # Read the JSON file as a DataFrame.
        spark.read.format('json').load(d).show()
Expected:
    +---+------------+
    |age|        name|
    +---+------------+
    |100|Hyukjin Kwon|
    +---+------------+
Got:
    +---+------------+
    | _1|          _2|
    +---+------------+
    |100|Hyukjin Kwon|
    +---+------------+

A lot of test fails because of that

Filed a JIRA: SPARK-41746

After https://github.com/apache/spark/pull/39344 is merged, most of them should pass.

Merged to master.

authors: HyukjinKwon techaddict


text:
[SPARK-43324][SQL] Handle UPDATE commands for delta-based sources

What changes were proposed in this pull request?

This PR adds RewriteUpdateTable, similar to RewriteDeleteFromTable, to handle UPDATE commands for delta-based sources. Support for group-based sources will come in a follow-up PR.
Why are the changes needed?

These changes are needed per SPIP SPARK-35801.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

This PR comes with tests. There are more tests in AlignUpdateAssignmentsSuite, which was merged earlier.

cc @dongjoon-hyun @viirya @huaxingao @cloud-fan @sunchao

Thank you, @dongjoon-hyun @cloud-fan!


text:
[SPARK-43487][SQL] Fix wrong error message used for ambiguousRelationAliasNameInNestedCTEError

What changes were proposed in this pull request?
The pr aims to fix  wrong error message used for ambiguousRelationAliasNameInNestedCTEError.
Why are the changes needed?
Fix bug.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.

@panbingkun Isn't this a duplicate of https://github.com/apache/spark/pull/41155?

@panbingkun Isn't this a duplicate of #41155?

It seems like it's duplicated. Should I turn it off?

It seems like it's duplicated. Should I turn it off?

Please, keep it alive so far.

@panbingkun Could you close this PR, please.

@panbingkun Could you close this PR, please.

OK

authors: MaxGekk panbingkun


text:
[WIP] [SPARK-47893][BUILD] Upgrade ASM to 9.7

What changes were proposed in this pull request?
This PR aims to upgrade ASM to 9.7.
Why are the changes needed?
xbean-asm9-shaded 4.25 upgrade to use ASM 9.7 and ASM 9.7 is for Java 23:
https://asm.ow2.io/versions.html
https://github.com/apache/geronimo-xbean/pull/40
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.

@panbingkun the Apache XBean 4.25 has been released and is available on central.

@panbingkun the Apache XBean 4.25 has been released and is available on central.

@fpapon
Okay, let's trigger this job again.
Thank you very much! ❤

authors: fpapon panbingkun


text:
[SPARK-4226] [SQL] Add Exists support for where clause

Rewrite the [NOT] EXISTS clause as left semi join. Still we need to add the [NOT] IN support.

Test build #28058 has started for   PR 4812 at commit 3d6934c.

This patch merges cleanly.

Test build #28058 has finished for   PR 4812 at commit 3d6934c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class LeftSemiType extends JoinType
case class Exists(left: LogicalPlan, right: LogicalPlan, exist: Boolean) extends BinaryNode

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28058/
Test PASSed.

@chenghao-intel Thank you for your implementation, following are my observations
Implementation seems simple but it comes with lot of limitations. The query like below
select C from R1 where exists (Select B from R2 where R1.X = R2.Y)

would be converted as below in case of your implementation I guess
select C from R1 left semi join R2 on  R1.X = R2.Y

But it syntactically not correct. it supposed to be converted as follow.
select C
from R1 left semi join
(select B, R2.Y as sq1_col0 from R2) sq1
on R1.X = sq1.sq1_col0

Both exists and in implementations should be similar. Just add exists support in parser would be enough and remaining implementation is almost similar. Not only the above case there are lot of other scenarios need to be taken care in subquery expressions. Please refer https://issues.apache.org/jira/secure/attachment/12614003/SubQuerySpec.pdf.
I am waiting to get my code merged so that I am planning to add all the remaining features on top of it.

So syntactically, what's the difference between
select C from R1 left semi join R2 on  R1.X = R2.Y

and
select C
from R1 left semi join
(select B, R2.Y as sq1_col0 from R2) sq1
on R1.X = sq1.sq1_col0

We never select any values from R2 right?

Sorry, I meant semantically.

@marmbrus, any comment on this?

@chenghao-intel Sorry for late reply. I think semantically it looks fine.

LGTM

Hi @chenghao-intel can you rebase this PR?

I think we can use PhysicalOperation to make this rule more general. Namely extract all filters first, deal with all Exists, and then apply other filters back.

Thank you @liancheng @scwf for the review. I'd like to support the subquery combines filter in where clause in another PR. Probably we can do that after the feature of IN subquery being added.

Test build #29829 has started for   PR 4812 at commit 81877a5.

Test build #29829 has finished for   PR 4812 at commit 81877a5.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

abstract class LeftSemiType extends JoinType

case class Exists(left: LogicalPlan, right: LogicalPlan, exist: Boolean) extends BinaryNode

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29829/
Test FAILed.

Test build #29830 has started for   PR 4812 at commit b36b906.

Test build #29830 has finished for   PR 4812 at commit b36b906.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

abstract class LeftSemiType extends JoinType

case class Exists(left: LogicalPlan, right: LogicalPlan, exist: Boolean) extends BinaryNode

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29830/
Test PASSed.

Test build #30113 has started for   PR 4812 at commit de529db.

Test build #30113 has finished for   PR 4812 at commit de529db.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

abstract class LeftSemiType extends JoinType

case class Exists(

case class InSubquery(left: SubqueryConjunction, right: LogicalPlan, positive: Boolean)

case class SubqueryConjunction(child: LogicalPlan,

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30113/
Test FAILed.

Test build #30114 has started for   PR 4812 at commit beb4e21.

Test build #30114 has finished for   PR 4812 at commit beb4e21.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

abstract class LeftSemiType extends JoinType

case class Exists(

case class InSubquery(left: SubqueryConjunction, right: LogicalPlan, positive: Boolean)

case class SubqueryConjunction(child: LogicalPlan,

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30114/
Test PASSed.

Hi @chenghao-intel, thanks for working on this.  It seems like this branch has gone stale and there are some questions about the implementation.  Can we close this issue for now and discuss design on the JIRA?

authors: AmplabJenkins SparkQA adrian-wang chenghao-intel liancheng marmbrus ravipesala scwf


text:
[SPARK-50489][SQL][PYTHON] Fix self-join after applyInArrow

What changes were proposed in this pull request?
Fix self-join after applyInArrow, the same issue of applyInPandas was fixed in https://github.com/apache/spark/pull/31429
Why are the changes needed?
bug fix
before:
In [1]: import pyarrow as pa

In [2]: df = spark.createDataFrame([(1, 1)], ("k", "v"))

In [3]: def arrow_func(key, table):
   ...:     return pa.Table.from_pydict({"x": [2], "y": [2]})
   ...:

In [4]: df2 = df.groupby("k").applyInArrow(arrow_func, schema="x long, y long")

In [5]: df2.show()
24/12/04 17:47:43 WARN CheckAllocator: More than one DefaultAllocationManager on classpath. Choosing first found
+---+---+
|  x|  y|
+---+---+
|  2|  2|
+---+---+

In [6]: df2.join(df2)
...
Failure when resolving conflicting references in Join:
'Join Inner
:- FlatMapGroupsInArrow [k#0L], arrow_func(k#0L, v#1L)#2, [x#3L, y#4L]
:  +- Project [k#0L, k#0L, v#1L]
:     +- LogicalRDD [k#0L, v#1L], false
+- FlatMapGroupsInArrow [k#12L], arrow_func(k#12L, v#13L)#2, [x#3L, y#4L]
   +- Project [k#12L, k#12L, v#13L]
      +- LogicalRDD [k#12L, v#13L], false

Conflicting attributes: "x", "y". SQLSTATE: XX000
	at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
	at org.apache.spark.SparkException$.internalError(SparkException.scala:79)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$2(CheckAnalysis.scala:798)

after:
In [6]: df2.join(df2)
Out[6]: DataFrame[x: bigint, y: bigint, x: bigint, y: bigint]

In [7]: df2.join(df2).show()
+---+---+---+---+
|  x|  y|  x|  y|
+---+---+---+---+
|  2|  2|  2|  2|
+---+---+---+---+

Does this PR introduce any user-facing change?
bug fix
How was this patch tested?
added tests
Was this patch authored or co-authored using generative AI tooling?
no

Merged to master.

Hi @zhengruifeng does this pull request fix a bug introduced in https://github.com/apache/spark/pull/41347 ? We maintain a fork of spark, and we're wondering if we need to cherry-pick this bug fix now. We don't have https://github.com/apache/spark/pull/41347 in our fork. (If we don't need to cherry-pick this bug fix, we'll get all these commits when we upgrade.) Thank you!

authors: HyukjinKwon Zand100 zhengruifeng


text:
[SPARK-50530] Fix bad implicit string type context calculation

What changes were proposed in this pull request?
Catch more cases where string type context should have implicit strength (lateral column aliases, outer references etc)
Why are the changes needed?
Before we would only assign implicit strength when we encountered AttributeReference and Alias. However, this should be the case for all NamedExpressions.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added new unit tests.
Was this patch authored or co-authored using generative AI tooling?
No.

@MaxGekk please take a look when you have the time


text:
[SPARK-6604][PySpark]Specify ip of python server scoket

In driver now will start a server socket and use a wildcard ip, use 127.0.0.0 is more reasonable, as we only use it by local Python process.
/cc @davies

Can one of the admins verify this patch?

ok to test

Jenkins, test this please.

LGTM

Test build #631 has started for   PR 5256 at commit c88bee9.

Test build #631 has finished for   PR 5256 at commit c88bee9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

There was some previous discussion of localhost vs 127.0.0.1 at #3425 and I think the conclusion there was against this change.  @davies, do your comments there about 127.0.0.1 sometimes being not portable (e.g. IPv6-only machines) still apply here?

@JoshRosen Changing the bind address to localhost is reasonable, but we have not figured out a good way to specify local interface, localhost or 127.0.0.1 both have PRO and CON, either one is better than another, so I against to change localhost to 127.0.0.1 without good reason.
I'd vote +1 for * -> 127.0.0.1 change, -0 for localhost -> 127.0.0.1.

(Related discussion was https://github.com/apache/spark/pull/3425)

@Sephiroth-Lin can you rebase this? and does it make sense to refer to "localhost" instead? Consider also that we have Utils.localHostName() which is an attempt to standardize what is meant by "the address of the local machine"

@srowen OK, thanks.
Jenkins, test this please.

LGTM pending tests.

Test build #685 has started for   PR 5256 at commit 7b3c633.

Test build #685 has finished for   PR 5256 at commit 7b3c633.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds no public classes.

This patch adds the following new dependencies:

commons-math3-3.4.1.jar
snappy-java-1.1.1.7.jar

This patch removes the following dependencies:

commons-math3-3.1.1.jar
snappy-java-1.1.1.6.jar

Jenkins, retest this please.

Test build #688 has started for   PR 5256 at commit 7b3c633.

Test build #688 has finished for   PR 5256 at commit 7b3c633.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test failure looks unrelated as it's not in Python, but let me try one more time.

Jenkins, retest this please

Test build #30474 has started for   PR 5256 at commit 7b3c633.

Test build #30474 has finished for   PR 5256 at commit 7b3c633.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30474/
Test PASSed.

authors: AmplabJenkins JoshRosen Sephiroth-Lin SparkQA WangTaoTheTonic davies srowen


text:
[SPARK-6855] [SPARKR] Set R includes to get the right collate order.

This prevents tools like devtools::document creating invalid collate orders

cc @cafreeman @davies

Test build #30058 has started for   PR 5462 at commit f3db562.

You may want to pull in the schema changes from earlier today since SQLtypes.R was replaced with schema.R

Test build #30058 has finished for   PR 5462 at commit f3db562.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30058/
Test PASSed.

@cafreeman We will pull that change in #5436 and @davies will take care of merging it correctly.

LGTM.
@shivaram Could you merge this today?  then I can rebase #5436

Test build #684 has started for   PR 5462 at commit f3db562.

Test build #684 has finished for   PR 5462 at commit f3db562.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Thanks @davies - I don't think the error has anything to do with this PR, but let me retest it anyways to be safe
Jenkins, retest this please

Test build #686 has started for   PR 5462 at commit f3db562.

Test build #686 has finished for   PR 5462 at commit f3db562.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

authors: AmplabJenkins SparkQA cafreeman davies shivaram


text:
OAuthError: invalid_client :  The client MUST NOT use more than one authentication method in each request

We configured the Superset_config.py as suggested on the documentation page and a custom sso as suggested in documentation . It is getting authenticated using OIDC and returning with a valid code. However it is not proceding after that. Error is
authlib.integrations.base_client.errors.OAuthError: invalid_client: The client MUST NOT use more than one authentication method in each request.
superset_app            | 2021-12-23 14:19:50,024:ERROR:superset.views.base:invalid_client: The client MUST NOT use more than one authentication method in each request.
superset_app            | Traceback (most recent call last):
Changes in Superset_cofig.py file*****
from custom_sso_security_manager import CustomSsoSecurityManager
from flask_appbuilder.security.manager import AUTH_OAUTH,AUTH_OID
CUSTOM_SECURITY_MANAGER = CustomSsoSecurityManager
AUTH_TYPE = AUTH_OAUTH
OAUTH_PROVIDERS = [
{ 'name':'PING',
'token_key':'access_token', # Name of the token in the response of access_token_url
'icon':'fa-address-card', # Icon for the provider
'remote_app': {
'client_id':'client_id', # Client Id (Identify Superset application)
'client_secret':'client_secret', # Secret for this Client Id (Identify Superset application)
'client_kwargs':{
'scope': 'openid profile address email phone', # Scope for the Authorization
},
'access_token_method':'POST', # HTTP Method to call access_token_url
'access_token_params':{ # Additional parameters for calls to access_token_url
'client_id':'client_id',
'grant_type' : 'authorization_code',
'client_secret' :'client_secret',
'redirect_uri' : 'http://url-blaa-bla:8088/'
},
'access_token_headers':{ # Additional headers for calls to access_token_url
'Authorization': 'Basic Base64EncodedClientIdAndSecret',
'Content-Type': 'application/x-www-form-urlencoded'
},
'api_base_url':'api_base_url',
'access_token_url':'access_token_url',
'authorize_url':'authorize_url'
}
}
]
AUTH_USER_REGISTRATION = True
AUTH_USER_REGISTRATION_ROLE = "Public"
#***********
Actual results
Authentication happening with Get call and returing with a valid code and state on the browser.
what actually happens.
http://url-bla-bla:8088/
Eneter MS ID and password
Auth success
in browser it comes with a valid code:
https://url-bla-bla:8088/oauth-authorized/PING?code=aaaaaaaaaaaaaaaaa&state=bbbbbbbbbbb.cccccccccc.ddddddd-ffffff-gggggggg-Y
Error on browser is "invalid redirect uri
LOGS:
superset_app | 2021-09-20 14:45:19,097:DEBUG:authlib.integrations.base_client.base_app:Saving authorize data: {'redirect_uri': 'http://localhost:8088/oauth-authorized/PING', 'nonce': 'S6JvBApadi4z3wOIyMWE', 'url': 'https://url/as/authorization.oauth2?response_type=code&client_id=client_id&redirect_uri=http%3A%2F%2Flocalhost%3A8088%2Foauth-authorized%2FPING&scope=openid+profile+address+email+phone&state=aaaaaaa.bbbbbbb.cccccc-ddddd-eeeee-Y&nonce=S6JvBApadi4z3wOIyKVU', 'state': 'eyL0eEAiOiMNS1QiLCJhbGciOiJIUzI1NiJ9.eeeeeee.bbbbbb-vvvvv-vvvvvv-Y'}
superset_app | 172.18.0.1 - - [20/Sep/2021:14:45:19 +0000] "GET /login/PING?next= HTTP/1.1" 302 951 "http://localhost:8088/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36"
superset_app | 127.0.0.1 - - [20/Sep/2021:14:45:22 +0000] "GET /health HTTP/1.1" 200 2 "-" "curl/7.64.0"
How to reproduce the bug
Git clone: https://github.com/apache/superset.git
configure superset_config.py with Oauth changes
add a custum sso file as defined in documention
build docker image : docker build -t superset-dev:latest .
docker-compose -f docker-compose-non-dev.yml up
http://localhost:8088/
enter MS id and password
Auth success
in browser it comes with a valid code:
https://localhost:8088/oauth-authorized/PING?code=aaaaaaaaaaaaaaaaa&state=bbbbbbbbbbb.cccccccccc.ddddddd-ffffff-gggggggg-Y
Error on browser is "This site can’t be reached"
Environment
Local : https://localhost:8088/
(please complete the following information):
File "/usr/local/lib/python3.8/site-packages/authlib/integrations/requests_client/oauth2_session.py", line 117, in handle_error
superset_app            |     raise OAuthError(error_type, error_description)
superset_app            | authlib.integrations.base_client.errors.OAuthError: invalid_client: The client MUST NOT use more than one authentication method in each request.
superset_app            | 2021-12-23 14:19:50,024:ERROR:superset.views.base:invalid_client: The client MUST NOT use more than one authentication method in each request.
superset_app            | Traceback (most recent call last):
superset_app            |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
superset_app            |     rv = self.dispatch_request()
superset_app            |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
superset_app            |     return self.view_functionsrule.endpoint
superset_app            |   File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/views.py", line 659, in oauth_authorized
superset_app            |     resp = self.appbuilder.sm.oauth_remotes[provider].authorize_access_token()
superset_app            |   File "/usr/local/lib/python3.8/site-packages/authlib/integrations/flask_client/remote_app.py", line 76, in authorize_access_token
superset_app            |     token = self.fetch_access_token(**params)
superset_app            |   File "/usr/local/lib/python3.8/site-packages/authlib/integrations/base_client/remote_app.py", line 112, in fetch_access_token
superset_app            |     token = client.fetch_token(token_endpoint, **kwargs)
superset_app            |   File "/usr/local/lib/python3.8/site-packages/authlib/oauth2/client.py", line 203, in fetch_token
superset_app            |     return self._fetch_token(
superset_app            |   File "/usr/local/lib/python3.8/site-packages/authlib/oauth2/client.py", line 226, in _fetch_token
superset_app            |     return self.parse_response_token(resp.json())
superset_app            |   File "/usr/local/lib/python3.8/site-packages/authlib/oauth2/client.py", line 380, in parse_response_token
superset_app            |     self.handle_error(error, description)
superset_app            |   File "/usr/local/lib/python3.8/site-packages/authlib/integrations/requests_client/oauth2_session.py", line 117, in handle_error
superset_app            |     raise OAuthError(error_type, error_description)
superset_app            | authlib.integrations.base_client.errors.OAuthError: invalid_client: The client MUST NOT use more than one authentication method in each request.
superset_app            | 10.175.238.7 - - [23/Dec/2021:14:19:50 +0000] "GET /oauth-authorized/PING?code=GYAb-14DB_65NwWzDSQiw4AUzeGpvc04Oasasasasas&state=eyJ0eXAiOiJKV1QiLCJhbGciqwqwwweeUzI1NiJ9.eyJuZXh0IjpbIiJdfQ.fdfto7RZFzoca-sdsdsdsd-jsGjhSPMH_ltWoj85-Y HTTP/1.1" 500 0 "https://hats-superset.mandeep.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"

@eschutho @dpgaspar @junlincc  Any help here please? I am trying since 2 months on Oauth and stuck. I am not getting any help anywhere and even from superset team. I know its opensource but some help would really do it. If this is not resolved management has asked to sidelined the superset explore something else :(  because i have already wasted 2 months on just SSO and posting questions here since 2 months. :( :(

@amitmiran137 @dpgaspar @eschutho Can i expect any help here?

I'm sorry I'm not familiar with that side.
I believe this is mostly relies on FAB realm

Hi @mandeeplohan I'm not familiar with this part of the app either, but I'd suggest taking a look at the source of the error code from the oauth documentation: https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-31#section-2.3
My first guess would be that you have the default AUTH_USER_REGISTRATION_ROLE set to "Public" when that role shouldn't require authentication. Have you tried setting that config to a different role like Admin?

@amitmiran137 @eschutho  Thanks for your time and response. This would help me for sure.  I have one question based on your response.

How this functionality lies outside of Superset?  Superset is providing the way to configure Oauth in the Superset_config.py file.  https://superset.apache.org/docs/installation/configuring-superset

I think somewhere Superset is using some library which user dont have access to modify code.
I am good with superset and many charts has been created easily. However, i am stuck on last part I.e.  to enable OAuth or LDAP.  Management does not want separate  creds which we can create easily in superset.
I dont know what should i do.  I was following this link : https://superset.apache.org/docs/installation/configuring-superset

@mandeeplohan Since Amit and I aren't very familiar with this part of the code, I would suggest going to the community slack and asking a question there, and someone that has configured their application with OIDC can help. Since this is more of a question than a bug, I'm going to close this issue.

source: gharchive/issue

authors: amitmiran137 eschutho mandeeplohan

repo: apache/superset


text:
How do you integrate superset with Datadog?

I think it must be something like this in superset_config.py but I cant get it working:
from superset.stats_logger import BaseStatsLogger
from datadog import statsd
from datadog import initialize

class DatadogdStatsLogger(BaseStatsLogger):
    def __init__(self, prefix='superset'):
        options = {
            'api_key':'<DATADOG_API_KEY>'
        }
        initialize(**options)

    def incr(self, key):
        statsd.increment(key)

    def decr(self, key):
        statsd.decr(key)

    def gauge(self, key):
        # pylint: disable=no-value-for-parameter
        statsd.gauge(key)

STATS_LOGGER = DatadogdStatsLogger(prefix='superset')

First confirm that the methods you've defined are being called, if that's the case then it's not a Superset issue but likely a configuration issue or something you can test and make work in isolation of Superset.

Have you found any solution @frankfarrell
I am fighting with it on how to send the logs & metrics to Datadog.

authors: MaheshB0ngani frankfarrell mistercrunch


text:
feat(native-filters): improve inverse selection indicators

SUMMARY
Currently setting a select filter to "inverse selection" doesn't visually indicate that the emitted WHERE clause is in fact a "NOT IN" as opposed to the normal "IN". This replaces the normal CheckOutlined icon with StopOutlined when inverse selection is applied. In addition, a new reserved parameter label is introduced on filterState, which makes it possible to customize the indicator label.
We might also want to consider adding support for being able to pass ReactNode labels to be able to further customize the indicator label. If this seems like a good approach, I'll add the label property to the DataMask type in @superset-ui/core.
BEFORE
Currently the dropdown icon for both regular and inverse selection are a checkmark:

The filter indicator also only shows the values as such, without indicating that the values are excluded:

AFTER
Now the dropdown icon signals that the selected values are excluded:

In addition, the filter indicator text now signals that the selected values are in fact excluded, not included:

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

this is much clearer than before already! can we implement the same icon in Explore NOT IN? thank you Ville!

can we implement the same icon in Explore NOT IN?

@junlincc do you mean here instead of "NOT IN"?

@junlincc do you mean here instead of "NOT IN"?

yes, but i mean the check mark showing in the "NOT IN" select could be confusing. this one is good to go! thank you so much! 🟢

@junlincc do you mean here instead of "NOT IN"?

yes, but i mean the check mark showing in the "NOT IN" select could be confusing. this one is good to go! thank you so much! 🟢

Ah yes of course, now I understand 😄 I will follow up later with this addition!

authors: junlincc villebro


text:
fix(explore): fix chart embed code modal glitch

SUMMARY

This PR fixes chart embed code modal glitch. I believe it's enough to vertical dragging when user wants to see the embed code. Maybe it's not necessary to drag the box beyond the modal part. So here I make resize to vertical.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

before
https://user-images.githubusercontent.com/25153919/104322949-d607a000-54e5-11eb-9fb1-c1b27bea1459.mov
after
https://user-images.githubusercontent.com/11830681/146972446-44dd1091-089c-452f-977d-fdb799e47453.mov
TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[x] Has associated issue: https://github.com/apache/superset/issues/12445
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

cc @rusackas

/testenv up

authors: pkdotson stephenLYZ


text:
issue with Test.ContinueOnFail = True

The test is skipped even we have ContinueOnFail set to True.
 # need Curl
 Test.SkipUnless(
     Condition.HasProgram("curl", "Curl need to be installed on system for this test to work"),
     Condition.HasCurlFeature('http2')
 )
 Test.ContinueOnFail = True

If this is still an issue please reopen.

authors: bryancall zizhong

repo: apache/trafficserver


text:
Rename vol to stripe (defined in the CacheVC.h)

Part of #10628.
Main changes are renaming members of CacheVC.
Stripe *vol;
int scanVol(int event, Event *e);
char *scan_vol_map;

to
Stripe *stripe;
int scanStripe(int event, Event *e);
char *scan_stripe_map;

Draft until https://github.com/apache/trafficserver/pull/10710 is merged.

Resolved conflicts. Please take another look.


text:
Cleanup: Move dir_valid macros to Stripe

Move below macros into Stripe member function.

#define dir_valid(_d, _e)
#define dir_agg_valid(_d, _e)
#define dir_agg_buf_valid(_d, _e)

Remove below macro because it's dead code.

#define dir_write_valid(_d, _e)

Rebased on the latest mater due to conflicted with another cleanup (https://github.com/apache/trafficserver/commit/d0b6a4b49ba8f2738fe91606c9cea55b06220cec). PTAL.

[approve ci autest]


text:
Update epel from 7-5 to 7-9.

Current instructions aren't immediately usable, fetch leads to a 404 error. The rpm in the repo is 7-9.

Thanks, but maybe this one is more better?
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Could you squash the commits?

Anything else?

No, that's it, thanks.
[approve ci]

authors: lbordowitz maskit


text:
Convert an ink_release_assert into logic to reset the rbio

Replaces an assert with logic to reset the memory based rbio to finish draining the buffer.

Fails clang-format unfortunately :).

Making a new PR against master

authors: shinrich zwoop


text:
Revert "Two more places to check whether attempting half_closed connecton logic is feasible"

This reverts commit 653927f9aab9a3cbe8f09521dd8c8154ccbc1614.
Erroneous fix caused KEEPALIVE to fail on HTTP/1.1 over HTTPS
Found by @zwoop, @mlibbey, @masaori335  and @maskit to have bad interactions with PR #4213.  In a separate PR I will add tests to catch regressions in client-side keep alive.

Cherry-picked to 8.0.x


text:
[bugfix][frontend][keras] Fix go_backwards attribute of LSTM in keras frontend

This PR fixed the bug proposed in #15176 and added bug-triggering test cases.
If go_backwards=True, the LSTM layer processes the input sequence backward and returns the reversed sequence.
Missing the support for go_backwards will lead to wrong inference results in TVM.
cc @Hzfengsy @echuraev

@Hzfengsy, Thanks for your review!

repo: apache/tvm


text:
[ZEPPELIN-5379] LIST_CONFIGURATIONS command cannot return the default…

What is this PR for?
when download distirbution package ,  run zeppelin server ,do not config zeppelin-site.xml ,i found LIST_CONFIGURATIONS command cannot return the default configuration in the ZeppelinConfiguration class code,so the web  lack some configuration
What type of PR is it?
[Bug Fix]
Todos

[ ] - Task

What is the Jira issue?

[ZEPPELIN-5379] https://issues.apache.org/jira/browse/ZEPPELIN-5379

How should this be tested?

Ci pass and manually tested

Screenshots (if appropriate)

Questions:

Does the licenses files need update?
Is there breaking changes for older versions?
Does this needs documentation?

LGTM, will merge if no more comment

authors: mnmhouse zjffdu

repo: apache/zeppelin

license_source: bigquery


text:
P110 warnings - power consumption related

Describe The Bug:
Homebridge logs get filled with this message
[homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info.
To Reproduce:
Always reproducible with a P110 configured
Expected behavior:
No warning messages and 'Total Consumption' gets passed in
Logs:
[24/03/2022, 22:30:57] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info.
[24/03/2022, 23:30:50] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info.
[24/03/2022, 23:30:50] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info.

Debug logs:
[26/03/2022, 09:19:56] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info.
[26/03/2022, 09:19:56] [homebridge-tapo] Error:
    at Characteristic.characteristicWarning (/usr/lib/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:2142:105)
    at Characteristic.validateUserInput (/usr/lib/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:2030:14)
    at /usr/lib/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:1617:24
    at /usr/lib/node_modules/homebridge/node_modules/hap-nodejs/src/lib/util/once.ts:10:18
    at P110Accessory.getTotalConsumption (/usr/lib/node_modules/homebridge-tapo/src/platformP110Accessory.ts:197:5)
    at Characteristic.emit (node:events:390:28)
    at /usr/lib/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:1592:14
    at new Promise (<anonymous>)
    at Characteristic.<anonymous> (/usr/lib/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:1590:12)
    at step (/usr/lib/node_modules/homebridge/node_modules/tslib/tslib.js:143:27)

Plugin Config:
            "name": "Tapo Smart Platform",
            "username": "",
            "password": "",
            "devices": [
                {
                    "name": "Dishwasher",
                    "host": "192.168.1.54",
                    "type": "PowerPlug",
                    "updateInterval": 30,
                    "timeout": 2
                },
                {
                    "name": "Washing Machine",
                    "host": "192.168.1.115",
                    "type": "PowerPlug",
                    "updateInterval": 30,
                    "timeout": 2
                }
            ],
            "platform": "TapoP100"
        }

Screenshots:
N/A
Environment:

Plugin Version:  v1.4.16
Homebridge Version:  1.4.0
Node.js Version:  v16.13.2
NPM Version:  8.1.2
Operating System:  Raspbian (5.10.103-v7+)

I too get the same issue with my two P110  power plugs.
This is my log:
[3/27/2022, 9:41:34 AM] [homebridge-tapo] This plugin generated a warning from the characteristic 'Current Consumption': Characteristic not in required or optional characteristic section for service Outlet. Adding anyway.. See https://git.io/JtMGR for more info. [3/27/2022, 9:41:34 AM] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': Characteristic not in required or optional characteristic section for service Outlet. Adding anyway.. See https://git.io/JtMGR for more info. [3/27/2022, 9:41:34 AM] [homebridge-tapo] This plugin generated a warning from the characteristic 'Current Consumption': Characteristic not in required or optional characteristic section for service Outlet. Adding anyway.. See https://git.io/JtMGR for more info. [3/27/2022, 9:41:34 AM] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': Characteristic not in required or optional characteristic section for service Outlet. Adding anyway.. See https://git.io/JtMGR for more info. [3/27/2022, 9:46:16 AM] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info. [3/27/2022, 9:46:16 AM] [homebridge-tapo] This plugin generated a warning from the characteristic 'Total Consumption': characteristic value expected valid finite number and received "undefined" (undefined). See https://git.io/JtMGR for more info.
The last two lines get repeated in the log.


text:
Which parameter in configs corresponds to the number of components R

In the paper, Eqns 1-5, 7-10 contain the parameter R (R_\sigma, R_c) which indicates the number of components (for both CP and VM decompositions). Which variable in the configs actually refers to this parameter R?

same question. have you solved it?

n_lamb_sigma and n_lamb_sh denote the number of components for sigma and shader respectively. For VM, they use different components for the 3 decompositions - hence R is a tuple. For CP, it's a scalar.
@apchenstu please correct me if I've understood it wrong.

repo: apchenstu/TensoRF

license: MIT


text:
IE maximum URL length

CAS Version: 5.0.3.1
Container: Tomcat 8.5 on Linux
When clicking the link on the login message view in IE it gets truncated after 2083 characters, which is the limit IE has with URL's.  To fix this issue with I updated the casLoginMessageView.html file to post a form instead.  Below is a link to the update.
casLoginMessageView.html.zip

Thanks. I applied the fix to master. You're more than welcome to submit a pull request for the 5.0.x branch as well.

authors: mmoayyed todd-pratt

repo: apereo/cas


text:
Move Async methods off of CasWebApplication

This moves the async application ready event method off of the CasWebApplication class. I think it may be confusing spring native but whether it is or not, I think it is better off in a bean.

@apereocas-bot runci