text:
[SPARK-29607][SQL] Move static methods from CalendarInterval to IntervalUtils

What changes were proposed in this pull request?
In the PR, I propose to move all static methods from the CalendarInterval class to the IntervalUtils object. All those methods are rewritten from Java to Scala.
Why are the changes needed?

For consistency with other helper methods. Such methods were placed to the helper object IntervalUtils, see https://github.com/apache/spark/pull/26190
Taking into account that CalendarInterval will be fully exposed to users in the future (see https://github.com/apache/spark/pull/25022), it would be nice to clean it up by moving service methods to an internal object.

Does this PR introduce any user-facing change?
No
How was this patch tested?

By moved tests from CalendarIntervalSuite to IntervalUtilsSuite
By existing test suites

@cloud-fan @HyukjinKwon @dongjoon-hyun Please, take a look at the PR.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17642/
Test PASSed.

Test build #112688 has started for PR 26261 at commit 21c30c6.

Test build #112688 has finished for PR 26261 at commit 21c30c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112688/
Test FAILed.

jenkins, retest this, please

Test build #112692 has started for PR 26261 at commit 21c30c6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17646/
Test PASSed.

Test build #112692 has finished for PR 26261 at commit 21c30c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112692/
Test PASSed.

@MaxGekk . I left a few minor comments. In general, the logic is correctly migrated from Java to Scala.
And, I agree with @MaxGekk 's opinion. How do you think about this, @cloud-fan ?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17658/
Test PASSed.

Test build #112713 has started for PR 26261 at commit a417cbe.

Test build #112713 has finished for PR 26261 at commit a417cbe.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112713/
Test FAILed.

jenkins, retest this, please

Test build #112717 has started for PR 26261 at commit a417cbe.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17662/
Test PASSed.

Test build #112717 has finished for PR 26261 at commit a417cbe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112717/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17665/
Test PASSed.

Test build #112720 has started for PR 26261 at commit e443921.

Test build #112720 has finished for PR 26261 at commit e443921.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112720/
Test FAILed.

Test build #112721 has started for PR 26261 at commit a6fbc4d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17666/
Test PASSed.

Test build #112721 has finished for PR 26261 at commit a6fbc4d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112721/
Test FAILed.

The following failure happens in master branch, too. It seems that irrelevant to this PR.
org.scalatest.exceptions.TestFailedException: 1572217200000000 was not 1572213600000000 plus or minus 30000000

jenkins, retest this, please

Test build #112727 has started for PR 26261 at commit a6fbc4d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17672/
Test PASSed.

Test build #112727 has finished for PR 26261 at commit a6fbc4d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112727/
Test FAILed.

jenkins, retest this, please

Test build #112737 has started for PR 26261 at commit a6fbc4d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17682/
Test PASSed.

Test build #112737 has finished for PR 26261 at commit a6fbc4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112737/
Test PASSed.

Yea this is the right direction. If we expose CalendarInterval, it should be a pure data class.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17725/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112780/
Test FAILed.

Retest this please.

Test build #112791 has started for PR 26261 at commit 38f7c78.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17736/
Test PASSed.

Test build #112791 has finished for PR 26261 at commit 38f7c78.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112791/
Test FAILed.

Test build #112792 has started for PR 26261 at commit 38f7c78.

Test build #112792 has finished for PR 26261 at commit 38f7c78.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112792/
Test FAILed.

retest this please

Test build #112817 has started for PR 26261 at commit 38f7c78.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17754/
Test PASSed.

Test build #112817 has finished for PR 26261 at commit 38f7c78.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112817/
Test FAILed.

retest this please

Test build #112826 has started for PR 26261 at commit 38f7c78.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17761/
Test PASSed.

Test build #112826 has finished for PR 26261 at commit 38f7c78.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112826/
Test FAILed.

retest this please

retest this please

Test build #112832 has started for PR 26261 at commit 38f7c78.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17767/
Test PASSed.

Test build #112832 has finished for PR 26261 at commit 38f7c78.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112832/
Test PASSed.

thanks, merging to master!

Thank you all!

authors: AmplabJenkins HyukjinKwon MaxGekk SparkQA cloud-fan dongjoon-hyun


text:
[SPARK-30373][SQL] Avoid unnecessary sort for ParquetUtils.splitFiles

What changes were proposed in this pull request?

In ParquetUtils.splitFiles() I changed the logic to:

if spark.sql.parquet.mergeSchema=false, then don't sort any files.
if spark.sql.parquet.mergeSchema=true,
2.1) if spark.sql.parquet.respectSummaryFiles=false, then sort all files
2.1) if spark.sql.parquet.respectSummaryFiles=true, then only sort metadata files

Why are the changes needed?

According to SPARK-11500, files' order only matters when schema merged is need. So, we cloud avoid unnecessary sort when we don't merge schema, especially when there're plenty of files.
Another minor improvement is grouping metadata files and data files firstly to avoid traversing all file for three times while constructing FileTypes.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Partially covered by SPARK-11500. And I also want to test metadata file part, but I don't find any metadata files while using write parquet files...

Test build #115889 has started for PR 27031 at commit 085d9d4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20679/
Test PASSed.

cc @cloud-fan @HyukjinKwon

Test build #115889 has finished for PR 27031 at commit 085d9d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115889/
Test PASSed.

@Ngone51 for non-trivial changes like this, we should make sure there are significant benefits. I agree with @HyukjinKwon and don't think sorting the files can be a bottleneck.

+1 with @HyukjinKwon .
I am closing this one.
@Ngone51 You can reopen this one if you have more thoughts.

source: gharchive/pull-request

authors: AmplabJenkins Ngone51 SparkQA cloud-fan gengliangwang

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-31400][ML]The catalogString doesn't distinguish Vectors in ml and mllib

What changes were proposed in this pull request?
add package check in org.apache.spark.ml.util.SchemaUtils#checkColumnType
and add unit test in org.scalatest.FunSuiteLike#test
Why are the changes needed?
the catalogString doesn't distinguish Vectors in ml and mllib when mllib vector misused in ml
https://issues.apache.org/jira/browse/SPARK-31400
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit test is added including both negative and positive cases.
test("Valid class of dataType") {     val schema = StructType(Array[StructField] {       StructField("features", new org.apache.spark.mllib.linalg.VectorUDT)     })     val e = intercept[IllegalArgumentException] {       SchemaUtils.checkColumnType(schema, "features", new VectorUDT)     }     assert(e.getMessage.contains("must belong to ml package"), "package is not valid to ml")     val normalSchema = StructType(Array[StructField] {       StructField("features", new VectorUDT)     })     SchemaUtils.checkColumnType(normalSchema, "features", new VectorUDT)   }

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Jenkins test this please

Test build #121768 has started for PR 28291 at commit 21415dc.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26445/
Test PASSed.

Test build #121768 has finished for PR 28291 at commit 21415dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121768/
Test PASSed.

Oh @TJX2014 I just reailzed you opened this against branch 2.4. It needs to be open vs master. The change is fine but it needs a PR. Feel free to open a new one vs master with the same change and ping me.

authors: AmplabJenkins SparkQA TJX2014 srowen


text:
[BugFix]fix the bug in issue SPARK-34759.

What changes were proposed in this pull request?
In JavaSparkSQLExample, exception is thrown  in function runDatasetCreationExample, when execute ‘spark.read().json(path).as(personEncoder)’.
The exception is  'Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot up cast age from bigint to int.'
Change age's type to long.

Why are the changes needed?
run JavaSparkSQLExample successfully.

Does this PR introduce any user-facing change?
NO

How was this patch tested?
test in JavaSparkSQLExample.

Can one of the admins verify this patch?

authors: AmplabJenkins zengruios


text:
[SPARK-36401][PYTHON] Implement Series.cov

What changes were proposed in this pull request?

Implement Series.cov
Why are the changes needed?

That is supported in pandas. We should support that as well.
Does this PR introduce any user-facing change?

Yes. Series.cov can be used.
>>> from pyspark.pandas.config import set_option, reset_option
>>> set_option("compute.ops_on_diff_frames", True)
>>> s1 = ps.Series([0.90010907, 0.13484424, 0.62036035])
>>> s2 = ps.Series([0.12528585, 0.26962463, 0.51111198])
>>> s1.cov(s2)
-0.016857626527158744
>>> reset_option("compute.ops_on_diff_frames")

How was this patch tested?

Unit tests

Can one of the admins verify this patch?

cc @xinrong-databricks @itholic FYI

Thanks @itholic , I resolved your reviews except one. Could you please review it again?

Would you add entry in python/docs/source/reference/pyspark.pandas/series.rst?

Could you please re-review, is it ready to merge ?  @srowen @itholic @xinrong-databricks

ok to test

Test build #142929 has started for PR 33752 at commit e0a8fe6.

Test build #142929 has finished for PR 33752 at commit e0a8fe6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142929/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47431/

Test build #142946 has started for PR 33752 at commit 74f354a.

Test build #142946 has finished for PR 33752 at commit 74f354a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142946/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47447/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47447/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47447/

Thanks! merging to master.

authors: AmplabJenkins HyukjinKwon SparkQA dgd-contributor ueshin xinrong-databricks


text:
[SPARK-37993][SHUFFLE] Avoid multiple calls to configuration parameter values.

What changes were proposed in this pull request?
Avoid multiple calls to configuration parameter values.
Why are the changes needed?
Improve performance.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existed UT.

Can one of the admins verify this patch?

Friendly ping @srowen

Simple enough, but can this make a difference to perf?

If you have a case that this increases perf a tiny bit, are there other similar changes that would benefit?

@srowen Thanks. It may not benefit performance enough, but it can make the code cleaner.

I don't think this is worth it, just too small

authors: AmplabJenkins srowen weixiuli


text:
[SPARK-40004][CORE] Remove redundant LevelDB.get in RemoteBlockPushResolver

What changes were proposed in this pull request?
This pr just remove redundant LevelDB.get(key) in RemoteBlockPushResolver.
Why are the changes needed?
This is no need to check db.get(key) != null before db.delete(key). LevelDB will handle this scene.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GitHub Actions

This minor change should not be coupled with the refactoring  work of SPARK-38909, I think this change may worth doing even if SPARK-38909 is not accepted in the end, .

Merged to master.
Thanks for the PR @LuciferYang !
Thanks for the review @srowen :-)

thanks @srowen @mridulm  ~

authors: LuciferYang mridulm


text:
[SPARK-40169][SQL] Don't pushdown Parquet filters with no reference to data schema

What changes were proposed in this pull request?

Currently in Parquet V1 read path, Spark will pushdown data filters even if they have no reference in the Parquet read schema. This can cause correctness issues as described in SPARK-39833.
The root cause, it seems, is because in the V1 path, we first use AttributeReference equality to filter out data columns without partition columns, and then use AttributeSet equality to filter out filters with only references to data columns.
There's inconsistency in the two steps, when case sensitive check is false.
Take the following scenario as example:

data column: [COL, a]
partition column: [col]
filter: col > 10

With AttributeReference equality, COL is not considered equal to col (because their names are different), and thus the filtered out data column set is still [COL, a]. However, when calculating filters with only reference to data columns, COL is considered equal to col. Consequently, the filter col > 10, when checking with [COL, a], is considered to have reference to data columns, and thus will be pushed down to Parquet as data filter.
On the Parquet side, since col doesn't exist in the file schema (it only has COL), when column index enabled, it will incorrectly return wrong number of rows. See PARQUET-2170 for more detail.
In general, where data columns overlap with partition columns and case sensitivity is false, partition filters will not be filter out before we calculate filters with only reference to data columns, which is incorrect.
Why are the changes needed?

This fixes the correctness bug described in SPARK-39833.
Does this PR introduce any user-facing change?

No
How was this patch tested?

There are existing test cases for this issue from SPARK-39833. This also modified them to test the scenarios when case sensitivity is on or off.

@viirya please see test cases added in SPARK-39833.
cc @sadikovi @cloud-fan @HyukjinKwon too.

This seems like a corner case when data columns and partition columns overlap (assuming you didn't set the case sensitivity flag to true).
When data columns and partition columns overlap, Spark reads the actual values from partition columns and ignore the overlapping data columns. See HadoopFsRelation.schema. That said, in your example, the filter col > 10 should be a partition filter, not data filter.

Thanks! merged to master/branch-3.3 (test failure unrelated).

Thank you, @sunchao and all!

authors: cloud-fan dongjoon-hyun sunchao


text:
[SPARK-41474][PROTOBUF][BUILD] Exclude proto files from spark-protobuf jar

What changes were proposed in this pull request?
This PR aims to exclude .proto files from spark-protobuf JAR file.
Why are the changes needed?
The proto files are not used.
BEFORE
$ jar tvf target/spark-protobuf_2.12-3.4.0-SNAPSHOT.jar | grep '\.proto$'
  5909 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/any.proto
  7734 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/api.proto
  8754 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/compiler/plugin.proto
 38497 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/descriptor.proto
  4895 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/duration.proto
  2363 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/empty.proto
  8185 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/field_mask.proto
  2341 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/source_context.proto
  3779 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/struct.proto
  6459 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/timestamp.proto
  6126 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/type.proto
  4042 Wed Oct 26 17:25:38 PDT 2022 google/protobuf/wrappers.proto

AFTER
$ jar tvf target/spark-protobuf_2.12-3.4.0-SNAPSHOT.jar | grep '\.proto$'

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manually check the jar file contents.

Could you review this, @SandishKumarHN and @gengliangwang ?

@dongjoon-hyun LGTM, thanks for the PR.

Thank you, @SandishKumarHN .

Thank you, @gengliangwang !

All tests passed. Merged to master for Apache Spark 3.4.0.

authors: SandishKumarHN dongjoon-hyun


text:
[SPARK-42410][CONNECT][TESTS][FOLLOWUP] Fix PlanGenerationTestSuite together

What changes were proposed in this pull request?
This is a follow-up of #39982 to fix PlanGenerationTestSuite together.
Why are the changes needed?
SPARK-42377 added two test suites originally which fails at Scala 2.13, but SPARK-42410 missed PlanGenerationTestSuite  while fixing ProtoToParsedPlanTestSuite.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.
$ dev/change-scala-version.sh 2.13
$ build/sbt "connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite" -Pscala-2.13
[info] PlanGenerationTestSuite:
...
[info] - function udf 2.13 (514 milliseconds)
...
$ SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite" -Pscala-2.13
...
[info] PlanGenerationTestSuite:
...
[info] - function udf 2.13 (574 milliseconds)

I missed SPARK-42377 added two test suites originally. Could you review this too, @HyukjinKwon ?

Could you review this PR, @viirya ? While fixing https://github.com/apache/spark/pull/39933#pullrequestreview-1294785750, I only fixed one of two test suites only mistakenly.

Module test passed. I'll merge this to recover master/3.4 branch.

Thanks for your fix @dongjoon-hyun

lgtm2

Looks good to me.

authors: HyukjinKwon LuciferYang dongjoon-hyun viirya


text:
[SPARK-39281][SQL] Speed up Timestamp type inference with legacy format  in JSON/CSV data source

What changes were proposed in this pull request?
Follow up https://github.com/apache/spark/pull/36562 , performance improvement when Timestamp type inference with legacy format.
In the current implementation of CSV/JSON data source, the schema inference with legacy format relies on methods that will throw exceptions if the fields can't convert as some data types .
Throwing and catching exceptions can be slow. We can improve it by creating methods that return optional results instead.
The optimization of DefaultTimestampFormatter has been implemented in https://github.com/apache/spark/pull/36562 , this PR adds the optimization of legacy format. The basic logic is to prevent the formatter from throwing exceptions, and then use catch to determine whether the parsing is successful.

Why are the changes needed?
Performance improvement when Timestamp type inference with legacy format.

Does this PR introduce any user-facing change?
No

How was this patch tested?
Add new test

Do the benchmarks CSVBenchmark and JsonBenchmark show any improvements? Could you regenerate the results JsonBenchmark.*.txt and CSVBenchmark.*.txt, please.

I'm doing benchmarks, but I found a problem, like @gengliangwang said in https://github.com/apache/spark/pull/36562#issuecomment-1127600354 .The benchmarks no case for the string inputs are not valid timestamps. The speed up only work when string input are not valid timestamps. I'm worry about the benchmarks can't prove anything. Can I create a PR for add benchmarks for type inference when string input are not valid timestamps

@Hisoka-X Please, resolve the conflicts and rebase on the recent master.

@Hisoka-X Please, resolve the conflicts and rebase on the recent master.

Ok, I will add benchmarks for this too. Please wait. Thanks!

+1, LGTM. Merging to master.
Thank you, @Hisoka-X.

authors: Hisoka-X MaxGekk


text:
[SPARK-43714][SQL][TESTS] When formatting error-classes.json file with SparkThrowableSuite , the last line of the file should be empty line

What changes were proposed in this pull request?
The pr aims to generate a blank line when formatting error-classes.json file using SparkThrowableSuite.
Why are the changes needed?

When I format error-classes.json file using SparkThrowableSuite, I found the last blank line of the file will be erased, which does not comply with code specifications.
Promote developer experience.

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual testing.

cc @MaxGekk

which does not comply with code specifications

Which specifications do you mean BTW. Please, point them out in the PR description.

1.I think it's a universal underlying rule, eg:
python: https://www.flake8rules.com/rules/W391.html
2.Why?
https://stackoverflow.com/questions/2287967/why-is-it-recommended-to-have-empty-line-in-the-end-of-a-source-file
3.All Spark source codes follow this rule
4.BTW, Perhaps we need to clarify this rule in https://github.com/databricks/scala-style-guide or other docs? @MaxGekk

+1, LGTM. Merging to master.
Thank you, @panbingkun.

authors: MaxGekk panbingkun


text:
STORM-3076:Fix the bug of examples in external can not be compiled

In the  external package,the modules include storm-hbase,storm-hdfs and storm-kafka , can not be compiled successfully, due to the lack of specific implements of ILocalCluster in the storm-core-0.9.5

what is wrong

repo: apache/storm


text:
SUBMARINE-412. Basic GitHub aciton support for submarine project

What is this PR for?
Add build/rat/linter GitHub actions for a fast check to help developers to fix trivial things
What type of PR is it?
Improvement
Todos

[ ] - rm same logic in travis untils this is stable

What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-412
How should this be tested?
GitHub action triggered
Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

oops.. seems only Only repository collaborators can create workflows.

@yaooqinn Thanks for working on it.
We need to create a JIRA ticket. more info could refer to this guide
submarine jira: https://issues.apache.org/jira/projects/SUBMARINE/issues/

thanks @pingsutw

@yaooqinn Thank you for contribution this feature.
Will merge if no more comments.

authors: liuxunorg pingsutw yaooqinn

repo: apache/submarine


text:
Custom Logging Class not working

Hi !
I am currently trying to use a custom EVENT_LOGGER in superset, i added the code snippets from the official doc but it isn't working:
I think the EVENT_LOGGER has changed and isn't the default DBEventLogger() anymore because we have some new logs on stdout, and there are no more "action logs" in the superset UI.
However, my JSONStdOutEventLogger() isn't used either, i added some prints in the log method to see if it was even called, and it doesn't seem so.
Here are the code snippets i added:
event_logger.py: |-
    from superset.utils.log import AbstractEventLogger
    import json
    class JSONStdOutEventLogger(AbstractEventLogger):
        def log(self, user_id, action, *args, **kwargs):
          records = kwargs.get('records', list())
          dashboard_id = kwargs.get('dashboard_id')
          slice_id = kwargs.get('slice_id')
          duration_ms = kwargs.get('duration_ms')
          referrer = kwargs.get('referrer')
          print("bip boup 1")
          for record in records:
              log = dict(
                  action=action,
                  json=record,
                  dashboard_id=dashboard_id,
                  slice_id=slice_id,
                  duration_ms=duration_ms,
                  referrer=referrer,
                  user_id=user_id
              )
              print(json.dumps(log))
              print("bip boup 2")

configFile: |-
...
  from event_logger import JSONStdOutEventLogger
  EVENT_LOGGER = JSONStdOutEventLogger()

Expected results
I expect the custom log method to be used (ie to see "bip boup" and "bip boup 2" on stdout)
Actual results
Some debug logs and logs on the requests done are on stdout (cf screenshot)
Screenshots

Environment

superset version: 1.0.1
python version: 3.8.8
node.js version: 12

I hope i'm not missing something, thank in advance for your help !

Hello! Have you solved this issue somehow or found some workaround for that?

Hello, yes we managed to fix it. It turned out that logs were not written at the right place by the logging class. We added this to the code :
with open("/var/lib/superset/action.log","w") as f:
                f.write("\n [LOG]: " + json.dumps(log) + "\n")

The full code snippet is now:
event_logger.py: |-
    from superset.utils.log import AbstractEventLogger
    import json
    class JSONStdOutEventLogger(AbstractEventLogger):
      def log(self, user_id, action, *args, **kwargs):
          records = kwargs.get('records', list())
          dashboard_id = kwargs.get('dashboard_id')
          slice_id = kwargs.get('slice_id')
          duration_ms = kwargs.get('duration_ms')
          referrer = kwargs.get('referrer')
          for record in records:
              log = dict(
                  action=action,
                  json=record,
                  dashboard_id=dashboard_id,
                  slice_id=slice_id,
                  duration_ms=duration_ms,
                  referrer=referrer,
                  user_id=user_id
              )
              with open("/var/lib/superset/action.log","w") as f:
                f.write("\n [LOG]: " + json.dumps(log) + "\n")

Thanks a lot for your answer.
I think it's worth considering editing documentation here: https://github.com/apache/superset/blob/master/docs/docs/installation/event-logging.mdx and then it would be possible to close this issue.
May I do it and link this issue to PR or do you want to open your own PR?

Sure you can do it, hope it can help people.

I've managed to do both JSONStdOutEventLogger and DBEventLogger at the same time. Now events are saved to logs in JSON format and they are still visible in Superset's dashboard in activity tab.
event_logger.py: |
from superset.utils.log import DBEventLogger
import json
class JSONStdOutEventLogger(DBEventLogger):
    def log(self, user_id, action, *args, **kwargs):
        records = kwargs.get('records', list())
        dashboard_id = kwargs.get('dashboard_id')
        slice_id = kwargs.get('slice_id')
        duration_ms = kwargs.get('duration_ms')
        referrer = kwargs.get('referrer')
        super().log(user_id, action, *args, **kwargs)

        for record in records:
            if 'rison' in record:
                del record["rison"]
            log = dict(
                action=action,
                json=record,
                dashboard_id=dashboard_id,
                slice_id=slice_id,
                duration_ms=duration_ms,
                referrer=referrer,
                user_id=user_id
            )
            print(json.dumps(log))

logger_settings: |
  from event_logger import JSONStdOutEventLogger
  EVENT_LOGGER = JSONStdOutEventLogger()

Hi! My last PR is merged, finally I've just updated docs. I think this issue could be closed now?
@bouleq could you close it? Or is it anything more still needed for this issue?

If anyone is coming here in 2024. This is what worked for us with the Superset Helm Chart 0.12.9 version: (We are running Superset in Kubernetes)
# values.yaml file:

configOverrides:
  # We use a custom JSON logger here to output everything to stdout -> FluentBit
  # Also, we prevent superset from using the default logger and flooding the DB with Logs
  # More info: https://superset.apache.org/docs/configuration/event-logging/
  logger_settings: |
    # log everything as JSON
    from superset.utils.log import AbstractEventLogger
    import json
    class JSONStdOutEventLogger(AbstractEventLogger):
        def log(self, user_id, action, *args, **kwargs):
          records = kwargs.get('records', list())
          dashboard_id = kwargs.get('dashboard_id')
          slice_id = kwargs.get('slice_id')
          duration_ms = kwargs.get('duration_ms')
          referrer = kwargs.get('referrer')
          for record in records:
              log = dict(
                  action=action,
                  json=record,
                  dashboard_id=dashboard_id,
                  slice_id=slice_id,
                  duration_ms=duration_ms,
                  referrer=referrer,
                  message=f"Action: {action} | DashID: {dashboard_id} | Time: {duration_ms}",
                  user_id=user_id
              )
              print(json.dumps(log))

    # use JSON logger and disable logging to the DB Table
    EVENT_LOGGER = JSONStdOutEventLogger()

This code creates a new logger that will log events as JSON in stdout and then prevent the DB Table from overflowing from time to time. The logs can then be flushed out from stdout to a search tool using a logger daemon.

source: gharchive/issue

authors: bouleq gugacavalieri wiktor2200

repo: apache/superset


text:
Intermittent report sent failures - Error: 'NoneType' object has no attribute 'apply_limit_to_sql'

Observing intermittent report sent failures with status - Error: 'NoneType' object has no attribute 'apply_limit_to_sql'
This issue is occurring with all kinds of reports - PNG, CSV, Slack Or email.
Noticed intermittent failures with all kinds of reports.
Environment

superset version: 1.0.0

Checklist
Stacktrace
[2021-12-16 11:05:01,978: INFO/ForkPoolWorker-7] Report sent to email
Report state: 'NoneType' object has no attribute 'apply_limit_to_sql'
[2021-12-16 11:05:02,100: INFO/ForkPoolWorker-7] Report state: 'NoneType' object has no attribute 'apply_limit_to_sql'
Scheduling alert test_201221 eta: 2021-12-16 11:10:00
[2021-12-16 11:10:00,749: INFO/ForkPoolWorker-7] Scheduling alert test_201221 eta: 2021-12-16 11:10:00
Init selenium driver
[2021-12-16 11:10:01,684: INFO/ForkPoolWorker-7] Init selenium driver
Taking a PNG screenshot or url http://superset:8088/superset/slice/125/?standalone=true
[2021-12-16 11:10:08,757: INFO/ForkPoolWorker-7] Taking a PNG screenshot or url http://superset:8088/superset/slice/125/?standalone=true
Report sent to slack
[2021-12-16 11:10:10,331: INFO/ForkPoolWorker-7] Report sent to slack
Scheduling alert test_201221 eta: 2021-12-16 11:15:00
[2021-12-16 11:15:00,686: INFO/ForkPoolWorker-7] Scheduling alert test_201221 eta: 2021-12-16 11:15:00
Check if the alert is in grace period
[2021-12-16 11:15:01,504: INFO/ForkPoolWorker-7] Check if the alert is in grace period
Report sent to email
[2021-12-16 11:15:02,517: INFO/ForkPoolWorker-7] Report sent to email
Report state: 'NoneType' object has no attribute 'apply_limit_to_sql'
[2021-12-16 11:15:02,902: INFO/ForkPoolWorker-7] Report state: 'NoneType' object has no attribute 'apply_limit_to_sql'
Scheduling alert test_201221 eta: 2021-12-16 11:20:00
[2021-12-16 11:20:00,395: INFO/ForkPoolWorker-7] Scheduling alert test_201221 eta: 2021-12-16 11:20:00
Init selenium driver
[20

Closing as stale since
a) nobody has been able to validate/reproduce this in over a year
b) the superset version mentioned is quite old at this point, so I'm optimistic it's no longer an issue.

authors: ashishtadose rusackas


text:
fix(header navlinks): link navlinks to path prefix

SUMMARY

The navlinks were previously highlighted using click events, which meant that if a user was pushed to another portion of the site, the navlinks wouldn't highlight the correct tab that the user was located under. The proposed fix introduces logic to set the highlighted navlink based upon the url prefix rather than click events alone.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before:
https://www.loom.com/share/67f59bc0320043dcbcdc6e2715b7da44?sid=c1821e43-cfc4-43f1-a8d6-cb87674da138
After:
https://www.loom.com/share/37b1679dce32469986ae743e318c132c?sid=c97fe956-581a-4adc-a551-204794c1f586
TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@fisjac can you remove the package-lock.json file?

authors: eschutho fisjac


text:
[SYSTEMML-1799] Remove ppred from test scripts

Replace ppred functions with relational operators in test scripts since
ppred() is deprecated.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1797/

Thanks for reviewing @dusenberrymw. I renamed the test classes in the first commit and then updated the majority of the content in the second commit to preserve the history for these files.

Thanks @dusenberrymw. I'll merge after successful completion of tests.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1804/

authors: akchinSTC deroneriksson

repo: apache/systemml


text:
THRIFT-4825: Aligned the TTlsSocketServerTransport with the TSocketServerTransport

Updated the existing constructor to to use default values for buffering and replaced the simplified constructor with one that allows passing in a TcpListener instance.

[X] Did you create an Apache Jira ticket?  (not required for trivial changes)
[X] Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?  (not required for trivial changes)
[X] Did you squash your changes to a single commit?
[X] Did you do your best to avoid breaking changes?  If one was needed, did you label the Jira ticket with "Breaking-Change"

As pointed out by jeking here, should I have simply created a TcpListener instance in the port only constructor? You wouldn't be able to rethrow an exception as a TTransportException, but is that a requirement?

In these cases I typically refactor the common initialization code to a single method and call that from both constructors.  It avoids future situations where someone modifies one constructor and not the other.  This typically happens when someone adds a variable and then forgets to initialize it in one of the constructors.  Some languages help protect against this, and it's been a while since I did any meaningful C#, so maybe that's not necessary and the compiler catches it?

Unfortunately .NET does not handle this for us. I could, however, pass null to the TcpListener parameter in the chain, and then use the existing code to set the _server value. It would fix the problem.
        public TTlsServerSocketTransport(
            int port,
            X509Certificate2 certificate,
            Buffering buffering = Buffering.None,
            RemoteCertificateValidationCallback clientCertValidator = null,
            LocalCertificateSelectionCallback localCertificateSelectionCallback = null,
            SslProtocols sslProtocols = SslProtocols.Tls12)
            : this(null, certificate, buffering, clientCertValidator, localCertificateSelectionCallback, sslProtocols)
        {
            try
            {
                // Create server socket
                _server = new TcpListener(IPAddress.Any, port);
                _server.Server.NoDelay = true;
            }
            catch (Exception)
            {
                _server = null;
                throw new TTransportException($"Could not create ServerSocket on port {port}.");
            }
        }

I'd think that you would want to raise an exception if someone passed in a null TcpListener though.  I see what you mean though.  Anyway, it was just a thought.  It doesn't have to be done.  This is still an improvement as-is, I was just thinking about how to consolidate the initialization.

@jeking3 the Initializer() way would be great, but we will have to sacrifice readonly then, since readonly vars can conly be set in a CTOR in C#. We could make it an R/O property though.

authors: Jens-G animusblue jeking3

repo: apache/thrift


text:
Crash while using next hop strategy

Strategies yaml config got loaded but seeing crash after sending curl request. I tested with 9.0.x and 9.1.x branches and crashed on both. I tried multiple "hash_key:path" and "hash_key:hostname" in yaml.
Please find the configuration and logs below.
cat remap.config
map /nexthop-test/ http://httpbin.org/ @strategy=nexthop-test-strategy

cat strategies.yaml
hosts:
  - &nexthop-test-host
    host: test-host.xyz.com
    protocol:
      - scheme: http
        port: 80
      - scheme: https
        port: 443

groups:
  - &nexthop-test-group
    - <<: *nexthop-test-host
      weight: 1.5

strategies:
  - strategy: 'nexthop-test-strategy'
    policy: consistent_hash
    hash_key: path
    go_direct: false
    groups:
      - *nexthop-test-group
    scheme: http
    failover:
      ring_mode: alternate_ring
      response_codes:
        - 404
      markdown_codes:
        - 503
      health_check:
        - passive

Curl
curl -vvv -o /dev/null http://localhost/nexthop-test/get
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to localhost port 80 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 80 (#0)
> GET /nexthop-test/get HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:00:34 --:--:--     0* Empty reply from server
  0     0    0     0    0     0      0      0 --:--:--  0:00:34 --:--:--     0
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

Stack trace
traffic_server: received signal 11 (Segmentation fault)
traffic_server - STACK TRACE:
/opt/ats/bin/traffic_server(crash_logger_invoke(int, siginfo_t*, void*)+0x94)[0x55edd0cc3524]
/lib64/libpthread.so.0(+0xf630)[0x7f8c4352f630]
/opt/ats/lib/trafficserver/libtscore.so.9(ATSConsistentHash::lookup(char const*, std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__tree_node<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, void*>*, long> >*, bool*, ATSHash64*)+0xf3)[0x7f8c44a177d3]
/opt/ats/bin/traffic_server(NextHopConsistentHash::findNextHop(tsapi_httptxn*, void*, long)+0x87f)[0x55edd0dc7e3f]
/opt/ats/bin/traffic_server(+0x254617)[0x55edd0d93617]
/opt/ats/bin/traffic_server(+0x2387be)[0x55edd0d777be]
/opt/ats/bin/traffic_server(HttpTransact::handle_response_from_parent(HttpTransact::State*)+0xbaa)[0x55edd0d85f0a]
/opt/ats/bin/traffic_server(HttpTransact::HandleResponse(HttpTransact::State*)+0x4a5)[0x55edd0d7adc5]
/opt/ats/bin/traffic_server(HttpSM::handle_server_setup_error(int, void*)+0x3a1)[0x55edd0d52491]
/opt/ats/bin/traffic_server(HttpSM::state_read_server_response_header(int, void*)+0x540)[0x55edd0d51e90]
/opt/ats/bin/traffic_server(HttpSM::main_handler(int, void*)+0x1ac)[0x55edd0d4099c]
/opt/ats/bin/traffic_server(+0x409883)[0x55edd0f48883]
/opt/ats/bin/traffic_server(UnixNetVConnection::net_read_io(NetHandler*, EThread*)+0xa94)[0x55edd0f47574]
/opt/ats/bin/traffic_server(NetHandler::process_ready_list()+0xa3)[0x55edd0f38793]
/opt/ats/bin/traffic_server(NetHandler::waitForActivity(long)+0x54e)[0x55edd0f38e6e]
/opt/ats/bin/traffic_server(non-virtual thunk to NetHandler::waitForActivity(long)+0xd)[0x55edd0f38f3d]
/opt/ats/bin/traffic_server(EThread::execute_regular()+0x55f)[0x55edd0f68dff]
/opt/ats/bin/traffic_server(EThread::execute()+0x179)[0x55edd0f69079]
/opt/ats/bin/traffic_server(+0x428556)[0x55edd0f67556]
/lib64/libpthread.so.0(+0x7ea5)[0x7f8c43527ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f8c42d288dd]
Segmentation fault

Complete Logs
GDB

@meeramn I've tried your exact configuration and curl command on 9.1.x.  I've tried the config and am unable to duplicate the crash.  I've tried different failure scenarios with the parent, unreachable, trafficserver shutdown, etc and am not able to duplicate the crash.  Everything is working as expected.

@meeramn I can re-create the crash when I'm running trafficserver on macOS.  I cannot duplicate it on a linux build with devtoolset-9

We build with llvm 12, so maybe a difference in how gcc/llvm handles things.
Full stack of crashed thread:
#0  0x00007febf7b221d9 in waitpid () from /lib64/libpthread.so.0
#1  0x00005639c1c44504 in crash_logger_invoke(int, siginfo_t*, void*) (signo=11, info=0x7febeff05b70, ctx=0x7febeff05a40)
    at traffic_server/Crash.cc:168
#2  <signal handler called>
#3  __tree_min<std::__tree_node_base<void *> *> (__x=0x600) at /opt/bazinga/include/c++/v1/__tree:147
#4  __tree_next_iter<std::__tree_end_node<std::__tree_node_base<void *> *> *, std::__tree_node_base<void *> *> (
    __x=0x7febf5789fb0) at /opt/bazinga/include/c++/v1/__tree:183
#5  operator++ (this=<optimized out>) at /opt/bazinga/include/c++/v1/__tree:842
#6  operator++ (this=<optimized out>) at /opt/bazinga/include/c++/v1/map:804
#7  operator++ (this=<optimized out>) at /opt/bazinga/include/c++/v1/map:809
#8  ATSConsistentHash::lookup(char const*, std::__1::__map_iterator<std::__1::__tree_iterator<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, std::__1::__tree_node<std::__1::__value_type<unsigned long, ATSConsistentHashNode*>, void*>*, long> >*, bool*, ATSHash64*) (this=0x7febf5789f98, url=0x0, i=0x7feb4fd33f68, w=<optimized out>, h=<optimized out>)
    at ConsistentHash.cc:109
#9  0x00005639c1d48e3f in chash_lookup (ring=..., hash_key=298391872744178962, iter=0x7feb4fd33f68,
    wrapped=0x7febeff068bf, hash=0x7febeff068c0, hash_init=<optimized out>, mapWrapped=<optimized out>,
    sm_id=<optimized out>) at NextHopConsistentHash.cc:48
#10 NextHopConsistentHash::findNextHop(tsapi_httptxn*, void*, long) (this=0x7febf59ff018, txnp=0x7feb4fd33500,
    ih=<optimized out>, now=0) at NextHopConsistentHash.cc:337
#11 0x00005639c1d14617 in nextParent(HttpTransact::State*) (s=0x7feb4fd33600) at HttpTransact.cc:249
#12 0x00005639c1cf87be in find_server_and_update_current_info(HttpTransact::State*) (s=0x7feb4fd33600)
    at HttpTransact.cc:584
#13 0x00005639c1d06f0a in HttpTransact::handle_response_from_parent(HttpTransact::State*) (s=0x7feb4fd33600)
    at HttpTransact.cc:3665
#14 0x00005639c1cfbdc5 in HttpTransact::HandleResponse(HttpTransact::State*) (s=0x7feb4fd33600) at HttpTransact.cc:3462
#15 0x00005639c1cd3491 in call_transact_and_set_next_state (this=0x7feb4fd33500, f=<optimized out>) at HttpSM.cc:7391
#16 HttpSM::handle_server_setup_error(int, void*) (this=0x7feb4fd33500, event=<optimized out>, data=<optimized out>)
    at HttpSM.cc:5790
#17 0x00005639c1cd2e90 in HttpSM::state_read_server_response_header(int, void*) (this=0x7feb4fd33500, event=104,
    data=0x7febe9742e60) at HttpSM.cc:2063
#18 0x00005639c1cc199c in HttpSM::main_handler(int, void*) (this=0x7feb4fd33500, event=104, data=0x7febe9742e60)
    at HttpSM.cc:2715
#19 0x00005639c1ec9883 in handleEvent (this=<optimized out>, event=104, data=0x7feb4fd33f68)
    at /usr/src/debug/bazinga-trafficserver-9.1.1/iocore/eventsystem/I_Continuation.h:219
#20 read_signal_and_update(int, UnixNetVConnection*) (event=104, vc=0x7febe9742c80) at UnixNetVConnection.cc:83
#21 0x00005639c1ec8574 in read_signal_done (event=104, nh=0x7febf5211d50, vc=0x7febe9742c80) at UnixNetVConnection.cc:150
#22 read_from_net (nh=<optimized out>, vc=<optimized out>, thread=<optimized out>) at UnixNetVConnection.cc:292
#23 UnixNetVConnection::net_read_io(NetHandler*, EThread*) (this=<optimized out>, nh=0x7febf5211d50,
    lthread=<optimized out>) at UnixNetVConnection.cc:921
#24 0x00005639c1eb9793 in NetHandler::process_ready_list() (this=0x7febf5211d50) at UnixNet.cc:415
#25 0x00005639c1eb9e6e in NetHandler::waitForActivity(long) (this=<optimized out>, timeout=<optimized out>)
    at UnixNet.cc:552
#26 0x00005639c1eb9f3d in non-virtual thunk to NetHandler::waitForActivity(long) ()
#27 0x00005639c1ee9dff in EThread::execute_regular() (this=0x7febf520dbc0) at UnixEThread.cc:303
#28 0x00005639c1eea079 in EThread::execute() (this=0x7febf520dbc0) at UnixEThread.cc:364
#29 0x00005639c1ee8556 in spawn_thread_internal(void*) (a=0x7febf62519e0) at Thread.cc:92
#30 0x00007febf7b1aea5 in start_thread () from /lib64/libpthread.so.0
#31 0x00007febf731b8dd in clone () from /lib64/libc.so.6

authors: jrushford meeramn randall

repo: apache/trafficserver


text:
Update shields on README

Remove distributions that are no longer built: Debian 10, Fedora 37 and 38.
Add new distributions: Debian 12, Fedora 39 and 40.

Cherry-picked to v10.0.x

authors: cmcfarlen moonchen


text:
coverity 1199998 : return code not handled

Address issue with TSHttpHdrUrlGet()

clang format failed! https://ci.trafficserver.apache.org/job/clang-format-github/408/

RAT check successful! https://ci.trafficserver.apache.org/job/RAT-github/419/

AU check failed! https://ci.trafficserver.apache.org/job/autest-github/404/

FreeBSD11 build failed! https://ci.trafficserver.apache.org/job/freebsd-github/2102/

Linux build failed! https://ci.trafficserver.apache.org/job/linux-github/1996/

Intel CC build failed! https://ci.trafficserver.apache.org/job/icc-github/531/

clang-analyzer build successful! https://ci.trafficserver.apache.org/job/clang-analyzer-github/664/

RAT check successful! https://ci.trafficserver.apache.org/job/RAT-github/458/

should be fixed

clang format failed! https://ci.trafficserver.apache.org/job/clang-format-github/448/

Intel CC build successful! https://ci.trafficserver.apache.org/job/icc-github/571/

[approve ci]

AU check failed! https://ci.trafficserver.apache.org/job/autest-github/445/

RAT check successful! https://ci.trafficserver.apache.org/job/RAT-github/462/

Linux build successful! https://ci.trafficserver.apache.org/job/linux-github/2037/

clang format failed! https://ci.trafficserver.apache.org/job/clang-format-github/452/

Intel CC build successful! https://ci.trafficserver.apache.org/job/icc-github/575/

AU check successful! https://ci.trafficserver.apache.org/job/autest-github/449/

FreeBSD11 build failed! https://ci.trafficserver.apache.org/job/freebsd-github/2141/

clang-analyzer build successful! https://ci.trafficserver.apache.org/job/clang-analyzer-github/702/

Linux build successful! https://ci.trafficserver.apache.org/job/linux-github/2041/

FreeBSD11 build failed! https://ci.trafficserver.apache.org/job/freebsd-github/2145/

clang-analyzer build successful! https://ci.trafficserver.apache.org/job/clang-analyzer-github/706/

authors: atsci dragon512 zwoop


text:
[Bug] [Unity] Error with 'munmap_chunk(): invalid pointer' when importing 'torch' after importing 'tvm'

I encountered an error when importing the 'torch' library in Python after importing 'tvm' previously. The error message displayed was "Error in `python': munmap_chunk(): invalid pointer: 0x00007f0c685b53f0". However, importing only 'tvm' without importing 'torch' beforehand or importing 'tvm' after importing 'torch' does not trigger the error.
Actual behavior
*** Error in `python': munmap_chunk(): invalid pointer: 0x00007f0c685b53f0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f0d11a467f5]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x1a8)[0x7f0d11a536e8]
/usr/lib/x86_64-linux-gnu/libLLVM-8.so.1(+0x161dc4c)[0x7f0cfb469c4c]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a008)[0x7f0d11a09008]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a055)[0x7f0d11a09055]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf7)[0x7f0d119ef847]
python(+0x1c24db)[0x55a5a61814db]
======= Memory map: ========
55a5a5fbf000-55a5a601a000 r--p 00000000 fd:00 177515246                  /workplace/software/miniconda3/envs/torch_gpu/bin/python3.7
55a5a601a000-55a5a61af000 r-xp 0005b000 fd:00 177515246                  /workplace/software/miniconda3/envs/torch_gpu/bin/python3.7
55a5a61af000-55a5a624d000 r--p 001f0000 fd:00 177515246                  /workplace/software/miniconda3/envs/torch_gpu/bin/python3.7
55a5a624e000-55a5a6251000 r--p 0028e000 fd:00 177515246                  /workplace/software/miniconda3/envs/torch_gpu/bin/python3.7
55a5a6251000-55a5a62ba000 rw-p 00291000 fd:00 177515246                  /workplace/software/miniconda3/envs/torch_gpu/bin/python3.7
...
Aborted (core dumped)

Environment

TVM version: v0.14.dev0-595-gfa0d35b
Python version: Python 3.7.10
torch version: 1.7.0

Steps to reproduce
import tvm
import torch

Triage

needs-triage

cc @quic-sanirudh

Hi @Thrsu perhaps you can try the suggestion here: https://github.com/apache/tvm/issues/9362#issuecomment-955263494

Hi @Thrsu perhaps you can try the suggestion here: #9362 (comment)

Thank you for your prompt response!

repo: apache/tvm


text:
[TE] Support Scan of non-highest axis

For algorithms like cumsum, most of the cases requires scanning over non-highest axis. The current workaround for this issue is to transposing the target axis to the highest and then do the scanning. It would be great to add direct support for scanning over other axis though.
Background https://github.com/apache/incubator-tvm/pull/5289

We have cumsum op now.

authors: masahi tqchen


text:
[Vulkan] Support uniform buffer object for passing many scalar arguments (Take 2)

A follow up to https://github.com/apache/tvm/pull/7717 per discussion https://github.com/apache/tvm/pull/7717#issuecomment-817316477
All issues raised have been addressed. Now one UBO is allocated per VulkanThreadEntry, in a similar way as staging buffer.
We always allocate UBO in a host visible memory. @tqchen asked if we can assume that we can always find such memory. Actually we are already making such assumption when allocating staging buffer here https://github.com/apache/tvm/blob/be87d56de011b4f18a3984ddff1da21c0675dac0/src/runtime/vulkan/vulkan.cc#L658
I did find a slide https://gpuopen.com/wp-content/uploads/2016/05/Most-common-mistakes-in-Vulkan-apps.pdf where it says "At least one memory type is host-visible & host-coherent" (page 19) but so far I'm not finding the official statement in the spec.
Since the way we allocate and use UBO is similar to staging buffer, I refactored vk runtime code to cleanly support both cases.
please review @tqchen @tmoreau89 @ajtulloch

cc @echuraev @csullivan @ajtulloch @antinucleon would be great if you can help review this PR

@tqchen I believe I've addressed all of your comments

@tqchen good to go?

Thanks @masahi for the great work to bring this new feature the codebase, more importantly make the codebase more readable! This PR is now merged


text:
Address review comments on Arm(R) Ethos(TM)-U PR 3/6

This patch addresses the review comments left on https://github.com/apache/tvm/pull/8806. Small changes are made to the API of ScheduleBuilder so that it can be reused in the NPU compiler, allowing us to remove a largely duplicated piece of code.

cc @mbs-octoml @jroesch PTAL

Apologies for the delay fixing CI here, having some environment problems locally. Will hopefully be fixed by next week.

authors: areusch mbaret


text:
Small typos in Spark introduction

What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
What type of PR is it?
Bug Fix
Improvement
Feature
Documentation
Hot Fix
Refactoring
Please leave your type of PR only
Todos

[ ] - Task

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
Put link here, and add [ZEPPELIN-Jira number] in PR title, eg. [ZEPPELIN-533]

How should this be tested?

Strongly recommended: add automated unit tests for any new or changed behavior
Outline any manual steps to test the PR here.

Screenshots (if appropriate)
Questions:

Does the licenses files need to update?
Is there breaking changes for older versions?
Does this needs documentation?

LGTM, @antonvasilev52 Could you update the PR description?

@zjffdu, updated. I'm sorry that I didn't do it right away.

No problem @antonvasilev52 Thanks for your contribution, LGTM

repo: apache/zeppelin

license_source: bigquery


text:
Paragraph 73 missing data

Paragraph 73, IAB Findings Not Supported by Preponderance of the Evidence, only have IMR 1 through 6 displayed.

Data was missing in db, skipped in scraping process

repo: apdforward/smartCASA

license: MIT


text:
allow jwt signing algorithm per service

Allow to specify JWT signing algorithm per service (like idTokenSigningAlg in OIDC client)
Actually, RS512 and HS512 are CAS defaults, but IETF spec recommends or requires others, like RS256 and HS256.
These algorithms are often required by third-party RPs.
The PR is another attempt after PR 5193 but without breaking changes for existing setup.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

repo: apereo/cas


text:
Generate new functions in existing projects

Given an existing project, it'd be nice to be able to generate new functions following those that are produced by apex init.
Perhaps a new parameter to init would work for this, such as apex init <function-name>? Although that may be confused with naming a new project, so maybe behind a flag instead?

Creating function dir is pretty straightforward, because it's just a dir inside ./functions. Not sure why that would be helpful?

touch ./functions/yofile.js

Closing for now, we can probably revisit if things get more complicated but hopefully we can keep it simple enough that a quick mkdir functions/foo && touch functions/foo/index.js is good enough.
cheers

I had in mind something that also copies in the boilerplate index.js, but erring on the side of simplicity is never a bad thing ;)

For that sort of thing I usually just have snippets in my editor since that's pretty easy for file-level boiler

authors: morgondag mthenw tj tlvince

repo: apex/apex


text:
Issue with Radial chart rounded lineCap.

Codepen
https://codepen.io/xanrick/pen/gOOBvMv
Explanation
When I enter a series value I expect the person who is seeing to estimate the value. However, when I enter a series value of above 95 its very hard to distinguish it almost looks like 100% or almost 100%.
There is no error message, it's just a visual bug.
Ofcourse I can stop using
{
lineCap: 'round'
}, however it looks really good with the round.
Issue 2:
In bar chart, I am using this option       {endingShape: 'rounded'}, the rounded shape at the end is not similar to the rounded shape on the radial bar, it's less tangential to the bar. It looks good, but not consistent with the round shape on the radial bar, so it kinda looks odd!

The endingShape: 'rounded' in a bar chart is drawn manually while radialbar round shape is achieved by setting stroke-linecap: round of SVG. So, there will be some differences.
Sorry, but as this is standard SVG behavior, I have no fix for it.

authors: junedchhipa saicharanallam


text:
[Docs Error] Default values for padding are incorrect

Hi, I noticed small error in docs.
Here: https://apexcharts.com/docs/options/grid/
In example we have:
padding: {
   top: 0,
   right: 0 // missing comma
   bottom: 0 // missing comma
   left: 0 // missing comma
}

I think in source code, commas are also missing because those 0 values do not apply.
Adding commas in options.grid solves problem and changes padding value to 0.
Great library btw. Keep it up 👍

Thanks for pointing it out, fixed it!

authors: junedchhipa venits


text:
Error reporting when accessing non-grid services

When accessing non-service grid services in the sidecar proxy container of the apisix application, a 503 gateway is prompted, such as Baidu

kubectl exec -it ratings-v1-679fc7b4f-9zglr bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to ratings.
Use 'kubectl describe pod/ratings-v1-679fc7b4f-9zglr -n default' to see all of the containers in this pod.
node@ratings-v1-679fc7b4f-9zglr:/opt/microservices$
node@ratings-v1-679fc7b4f-9zglr:/opt/microservices$ curl www.baidu.com

502 Bad Gateway

502 Bad Gateway
openresty

node@ratings-v1-679fc7b4f-9zglr:/opt/microservices$ curl https://www.baidu.com
curl: (77) error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs


text:
Nested arrays aren't expanded

Given following blueprint:
# Resource [/r]
## Retrieve [GET]

+ Response 200 (application/json)
    + attributes
        + a
        + b (array[X])

## Data Structures

### X
- c

The  Retrieve  response JSON is never expanded.

This is an issue with boutique. Need more digging to check if there are other similar test cases.

Related issue – http://stackoverflow.com/questions/29810814/trying-to-describe-the-request-and-response-using-data-structures-in-api-bluepri

This has been solved by #32

authors: pksunkara zdne

license: mit


text:
feat: Enable alphabetize for imports

Based off https://github.com/apify/apify-core/blob/develop/src/app/.eslintrc.js#L40-L60
The rule is auto-fixable by eslint, so we won't have to do too much manual work (hopefully :D)
Slack discussion for reference https://apifier.slack.com/archives/C01VBUV81UZ/p1688991614682009

I wish there was also a rule to enforce the node: prefix, but it looks like we're not getting it any time soon: https://github.com/import-js/eslint-plugin-import/issues/2717

Plus one more thing, I'd really also add this rule from core repo:
    "newline/import-module": ["warn", {
        "minItems": 2,
        "tab": 2,
    }],

as by having imports spread per multiple lines, we beware of merge conflicts in many cases. But we can leave this one for another discussion.

Could we also put there the part about the apify-packages grouping? It's only used in private repositories, so this won't affect any open source repo. It makes it cleaner.
    "import/order": ["error", {
        "groups": ["builtin", "external", ["parent", "sibling"], "index", "object"],
        "alphabetize": { "order": "asc", "caseInsensitive": true },
        "newlines-between": "always",
        "pathGroups": [
            // There is probably some bug in meteor that doesn't respect topological order of packages
            // and doesn't refresh intl strings properly. Importing intl before other packages fixes it
            // but we might run into similar issues in the future.
            {
                "pattern": "@apify-packages/intl**",
                "group": "external",
                "position": "after"
            },
            {
                "pattern": "@apify-packages/**",
                "group": "external",
                "position": "after"
            }
        ],
        "pathGroupsExcludedImportTypes": ["builtin"]
    }],

or as the comment says it's probably bug in the meteor we don't really need this elsewhere?

I think we don't need it elsewhere. I wanted to keep it as simple as possible. If it causes issues, we can always add the grouping...

Plus one more thing, I'd really also add this rule from core repo:
    "newline/import-module": ["warn", {
        "minItems": 2,
        "tab": 2,
    }],

as by having imports spread per multiple lines, we beware of merge conflicts in many cases. But we can leave this one for another discussion.

I'd be for adding this rule too.
What do @fnesveda @B4nan @gippy think?

I don't like that very much, 2 import statements seem to be too few for such a rule (at least for our usage in crawlee and similar), and I generally dislike warnings in linter, as they can be ignored and pollute the logs for no reason (aka it can make logs worse to navigate, and won't force people to actually follow the rule).
Maybe something like 5-6 would make more sense, if you want to enforce line breaks on (pretty much) all the import statements (except the single ones), it sounds better to put this into your project instead.
We already enforce the line length which will force us to wrap too long import statements anyway, to me that's good enough, personally, I don't like to line break the imports so aggressively.

I don't mind adding it, but does it help with merge conflicts really? With 5 lines of git diff context by default, you'd have to have like 10 lines between the changed imports so that you don't get a merge conflict anyway, that wouldn't happen in most cases.

Ok guys, thanks for the discussion. I'm going to merge this PR as is. We can discuss the other rules later, since we don't perfectly agree on it.

when u say "eslint" 3 times, Pablo will appear 🦄

authors: B4nan Strajk fnesveda jirimoravcik mtrunkat

repo: apify/apify-eslint-config


text:
Swagger global security definition not working!

Using the global security definition does not work.
...
securityDefinitions:
   APIKeyHeader:
     type: apiKey
     in: header
     name: X-jwttoken
security:
        - APIKeyHeader: []
...
/someendpoint:
    x-swagger-router-controller: someendpoint
    get:
      description: a description
      operationId: someendpointMethod
      produces:
        - application/json
      parameters:
        - name: X-jwttoken
          in: header
          description: jwt wso2 token
          required: false
          type: string
      responses:
        '200':
          description: Success

The error I get is:
_Cannot read property 'type' of undefined","stacktrace":"TypeError: Cannot read property 'type' of undefined
at getScopeOrAPIKey (D:\devcat\dev_workspace\workspace\backend-marketplace-productactivation\node_modules\swagger-node-runner\fittings\swagger_security.js:80:25)
at andCheck (D:\devcat\dev_workspace\workspace\backend-marketplace-productactivation\node_modules\swagger-node-runner\fittings\swagger_security.js:51:43)
If I put the security definition within each of my paths (instead of global) it works.
Also, with swagger-node-mw 0.1.0 the global definition works.

This will be resolved via #145.

#145 was merged, and will come in the next release.
@randomcatgamer can you pull master, link to the local copy and verify that the fix works? If you can't do this, please verify as soon as there is a release so that we can close the issue.

This is a duplicate of #120.

authors: noahdietz randomcatgamer whitlockjc


text:
update apollo-server implementation with latest schema changes

Update to expected schema to enable verifying @requires directive functionality.
Changes:

add missing User entity and type resolvers
add averageProductsCreatedPerYear: Int @requires(fields: "yearsOfEmployment") on User type
add new yearsOfEmployment: Int! @external field on User type`

Related Issues:

https://github.com/apollographql/apollo-federation-subgraph-compatibility/issues/128

LibraryFederation 1 SupportFederation 2 Support

Apollo Server_service🟩@key (single)🟩@key (multi)🟩@key (composite)🟩@requires🟩@provides🟩@ftv1🟩@link🟩@shareable🟩@tag🟩@override🟩@inaccessible🟩


text:
Add ESM build for tree-shaking

TODO:

[x] If this PR is a new feature, reference an issue where a consensus about the design was reached (not necessary for small changes)
[x] Make sure all of the significant new logic is covered by tests
[x] Rebase your changes on master so that they can be merged easily
[x] Make sure all tests and linter rules pass
[x] Update CHANGELOG.md with your change. Include a description of your change, link to PR (always) and issue (if applicable). Add your CHANGELOG entry under vNEXT. Do not create a new version number for your change yourself.

Prior discussion of this in #343 and #344.

[x] Produce build with ESM modules in it into es folder
[x] Fix require in tests by using esModuleInterop into tsconfig.json
[x] Add module field so that tooling can find the ESM build

Once merged the ESM build will just get done alongside the ES5 build. I did npm pack to make sure the es folder will get included.
I linked this version into an AWS Lambda function where I'm only importing makeExecutableSchema and nothing else from graphql-tools. I saw a 100KiB reduction in a production build size.

@sheepsteak: Thank you for submitting a pull request! Before we can merge it, you'll need to sign the Apollo Contributor License Agreement here: https://contribute.apollographql.com/

Closed in favor of #1320

authors: apollo-cla sheepsteak yaacovCR

repo: apollographql/graphql-tools


text:
elasticsearch 5

closes #618

elasticsearch 5

test with:

make clean install
make TAG=local build-image
amp pf stop ; amp pf pull && amp pf start --local && make test

tested on Docker for Mac (1.12.5) ✅

failing on Linux (Docker 1.12.5): elasticsearch fails to start

passing on Linux Docker 1.12.5 now.

thanks @freignat91 , LGTM

repo: appcelerator/amp


text:
fix(generate): fixed typo in alloy generate components

https://jira.appcelerator.org/browse/EDITOR-42

Warnings

:warning:

src/commands/alloyGenerate.ts#L45 - src/commands/alloyGenerate.ts line 45 – Forbidden non-null assertion. (@typescript-eslint/no-non-null-assertion)

Messages

:book:

:floppy_disk: Here's the built vsix for this change.

:book:

:white_check_mark: All tests are passing
Nice one! All 32 tests are passing.

  Generated by :no_entry_sign: dangerJS against 6c50e589e643bf5c94768c984ea1c0be5bed48d9


text:
Add Concurrency

Hi!
Here the Concurrency framework is being added to the project. Now, in case of having multiple string catalogs, they will be checked in parallel.
The changes have been tested using the compete concurrency check provided with the Swift Tools version 5.10 (.enableExperimentalFeature("StrictConcurrency")), to avoid including data-race safety issues and to assure the code is Swift 6 compatible.
Cheers!!

LGTM - just one question!

Nice catch!! Fixed!!

I also fixed the message shown with every missing translation (MissingTranslation structure's message property)

repo: appdecostudio/SwiftPolyglot


text:
Endpoint structure

It might be a good thing to separate what should be in the path and what should be in the GET parameters.
For instance, when requesting to the API, it might be more readable to include page number as a GET parameter instead of as part of the path:
http://<api>/top/?page=5 instead of http://<api>/top/5.
This would make future functionality integration a bit easier and paths much simpler.
Thoughts?
(Sort filters are implemented using GET in #5 as example.)

Sure. I am new to this and still learning. It will be helpful if you open a PR


text:
Add ability to slide carousel from parent component

Hi everyone!
We wanted to be able to slide the carousel to n-th page with a press of a button. Maybe it might be useful for others as well.
You can pass the page you want the carousel to slide to as animateTo. Then, every time you want the carousel to slide send a new non-zero number as refreshAnimation (the easiest way to do this is to increment it in parent).

I realised it is possible to do this by calling _animateTo from parent. Making that function public would be a better solution than mine.


text:
Added IME commands

@jlipps @moizjv @Jonahss please review.

commented; also agree with @imurchie's comments. then LGTM!

merged, grats on your first PR scott!

authors: jlipps scottdixon

repo: appium/appium-android-driver


text:
🛑 Reddit is down

In 3afcec5, Reddit (https://www.reddit.com) was down:

HTTP code: 503
Response time: 53 ms

Resolved: Reddit is back up in a788f6b.

repo: applabomatic/siteuptimely


text:
TRANSACTION_ID_NOT_FOUND with valid Transaction ID

I'm getting error TRANSACTION_ID_NOT_FOUND
I use transactionId from data from Apple. Example
{"receipt": {"productId": "threeDaysForGirl", "transactionDate": 1698142245000, "transactionId": "2000000442477236", "transactionReceipt":"***"}

I this case use transactionId "2000000442477236" in sendConsumptionData() or getTransactionInfo() methods invoke and getting 404 error.
APIException{httpStatusCode=404, apiError=TRANSACTION_ID_NOT_FOUND} com.apple.itunes.storekit.client.APIException
	at com.apple.itunes.storekit.client.AppStoreServerAPIClient.makeHttpCall(AppStoreServerAPIClient.java:114)
	at com.apple.itunes.storekit.client.AppStoreServerAPIClient.getTransactionInfo(AppStoreServerAPIClient.java:290)
	at ru.ruamo.web.impl.AppleServiceImpl.sendConsumptionRequest(AppleServiceImpl.java:65)
	at ru.ruamo.web.controllers.TestController.apl(TestController.java:397)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:365)
	at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:151)
	at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:129)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
	at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.session.ForceEagerSessionCreationFilter.doFilterInternal(ForceEagerSessionCreationFilter.java:45)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:186)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:91)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1740)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:833)

Sorry to bother you. Bundle ID was wrong in my request.

repo: apple/app-store-server-library-java


text:
cupsCheckDestSupported() does not work as expected when value = NULL

CUPS version: latest snapshot of repo
The CupsPM  states that

If the value argument is NULL, the cupsCheckDestSupported function returns whether the option is supported by the destination. Otherwise, the function returns whether the specified value of the option is supported.

However, this is not working as expected for me. For instance,
cupsCheckDestSupported(http, dest, dinfo, CUPS_COPIES , NULL) returns 0;
But
cupsCheckDestSupported(http, dest, dinfo, CUPS_COPIES , "1" ) returns 1.
Also,  cupsFindDestSupported(http, dest, dinfo, "job-creation-attributes")  lists "copies" as one of the values.
Shouldn't cupsCheckDestSupported(http, dest, dinfo, CUPS_COPIES , NULL)  have returned 1 according to the CUPS PM ?
Am I missing something in the CUPS Manual?

Will investigate; the API should definitely work as documented.

[master 49b6c6af7] The cupsCheckDestSupported function did not support NULL values (Issue #5031)

I pulled the latest changes of the repo; but the problem seems to persist. :disappointed_relieved:

Hmm, that should definitely not be the case. I've pushed a change to the "testdest" unit test program to verify this - after making CUPS the "testdest" program will be found in the "cups" directory, which can then be run with:
cups/testdest PRINTER-NAME supported copies

this works as expected with one printer I tested with:
cups/testdest Brother_MFC_J6920DW supported copies
copies (supported)
  1-9999

authors: NilanjanaLodh michaelrsweet

repo: apple/cups


text:
How  connects to cup server via ipps/https from windows

what cupsd.conf configuration need to modify?

Share the printer over the https connection to the cup server


text:
Nix support

Is there gonna be nix configuration support??

Duplicate of #113

Is it really a duplicate? What was the authors intention?

@adrian-gierakowski nope not a duplicate.
My intention was to use apple pkl to add support for nix configuration files such as configuration.nix and flakes to easily assist anyone looking to setup NixOS as a server or personal worksation. #113 speaks of installation of pkl on NixOS.

Re-opened!

Hi @stackoverflow , this is absolutely not a duplicate.
I've been thinking about & need this use-case as well @alexthotse . I think this is where pkl could truly shine for large scale adoption, in addition to all other strong use-cases/migrations like k8s workflows. I would have liked to move my ~/.config/home-manager/home.nix to ~/.config/home-manager/home.pkl as well.
pkl-cli eval command needs to support nix language:
$ jpkl eval -f nix home.pkl

So far based on my investigation, it could provide that as long as eval-ed pkl objects don't have "entries" that are public(maybe "hidden" entries could pass the output render checks). Objects with pkl entries also fail to be converted to json or yaml. A json/yaml can always be converted to pkl and nix however pkl environment will miss the possible pkl type declaration unless certain nix renderer is used.
Another option is to build a typed home-manager in pkl and interop will be challenging but earlier we start is better I think. Regardless a nix format is absolutely needed for $ pkl eval -f nix.
I'm still learning/investigating Pkl, I already see Pkl as an improvement to nix, the language overall(not the nix, the store).

@izelnakri You clearly have a bigger picture as to how Nix and Pkl can coexist with one another. This is what I had envisioned in a nutshell. Thank you for wording it out.
@bioball Appreciate the re-open as this will help bring a new approach to future configuration of systems as a whole.
So the question would be how can we join our efforts to make this a possibility?

Can a pkl file export a function?

Can a pkl file export a function?

very good question @adrian-gierakowski  & yes! :) However as far as I know the current language design discourages "one-function-per-file" concept unless its being generated.
Pkl has a special output object that can have the value defined. Dependening on the non-pkl output mode, the relevant parser/generator will always read this namespace. Example:
function.pkl:
output {
  value: (n) -> n * 2 // our custom function
}

main.pkl:
my_function = import("function.pkl") // if you want this key ignored by output make it "hidden my_function = import()"

result = my_function.output.value.apply(22)

jpkl eval -f json main.pkl
# json output will be { result: 44 }

Functions are internal to Pkl; not exportable.
I don't have any experience working with nix. But: you can write an in-language renderer to turn Pkl into nix.
For example, here is a renderer that writes Pkl -> TOML: https://github.com/apple/pkl-pantry/blob/main/packages/pkl.toml/toml.pkl

A nix output format is certainly a reasonable way in, although that could just be a renderer in Pkl (see the TOML renderer). That said, the nix format has a bunch of programming constructs that you'll not get out of Pkl. Doesn't nix tooling also read static formats anywhere? Alternatively, the real support would be got by getting nix to use Pkl as a library, so that it can 'natively' consume .pkl files.

Nix can import json (and toml) natively so if you generate, say k8s config in pkl you could import it into nix and manipulate further using nix language
but things like configuration.nix or home manager config are usually functions. Not sure how one could use pkl for that and what’s the benefit over just using nix

but things like configuration.nix or home manager config are usually functions. Not sure how one could use pkl for that and what’s the benefit over just using nix

@adrian-gierakowski  I think the benefit would be that nix, home-configuration.nix & configuration.nix could use Pkl documentation generator for documentation in the future and perhaps all the typed data structures would be easily debuggable, interactively peeked with $ jpkl eval. Both functions return an expected object and debugging/peeking that returned object before manipulation make things very productive including the benefit of being able to debug the imported flakes interactively.
Missing or impartial documentation for data structures has been my biggest DX issue with learning and doing anything advanced with nix/home-manager. I can see how Pkl could ease the learning curve of nix and development of advanced configurations and modules.

authors: adrian-gierakowski alexthotse bioball holzensp izelnakri stackoverflow

repo: apple/pkl


text:
[logging] Don't pass arbitrary string to NSLog

NSLog expects a format string, so don't pass it arbitrary log strings,
which may contain format specifiers.

@swift-ci please test

repo: apple/sourcekit-lsp


text:
Initialize pthread mutexes in NSRecursiveLock

I was going to suggest a test for NSRecursiveLock fix in #2269, as promised. But test turned out extremely useful from first run. Found that pthread-based implementation is also missing some initialization.
I suppose it will be ok to add this new fix and test in one PR, right?

Yeah, having that in a single change is fine.

@swift-ci please test

Can you make the timeout looser? Maybe 30 seconds. Sometimes these machines get hit really hard.

Can you make the timeout looser? Maybe 30 seconds. Sometimes these machines get hit really hard.
Sure. I think I need to add better timeout logic then, because current code waits 2 sec unconditionally. I'll change it to wait until all threads finish but not greater than timeout.

@swift-ci please test

@swift-ci please test

@swift-ci please test

@swift-ci please test

I see, Linux build fails. Will look into that, guess I missed something in ifdefs.

@swift-ci please test

@swift-ci please test

authors: compnerd gottesmm lxbndr

repo: apple/swift-corelibs-foundation


text:
[DNM] Test Creating strings without CF

This tests directly using Swift instead of CF for a range of strings created with different encodings although it falls back to CF for encodings it doesn't understand.

@swift-ci test linux

@compnerd Could you test this on Windows when you get the chance, I want to see if it works as a fix for the string corruption issue - I reverted your fix in the PR to test against.

I'm lazy, I'll let the CI test it: https://dev.azure.com/compnerd/swift-build/_build/results?buildId=26417&view=results

https://dev.azure.com/compnerd/swift-build/_build/results?buildId=26417&view=logs&j=ec47c80d-c7cf-5f2b-7138-13cd268bea3d&t=5cfc8e20-74e6-5b9b-ff2a-9c787c40fff4
Seems like the failures persist with that

@swift-ci test linux

@swift-ci test linux

@swift-ci test linux

The Swift project moved the default branch to main and deleted master branch, so GitHub automatically closed the PR. Please re-create pull request with main branch.
More detail about the branch update - https://forums.swift.org/t/updating-branch-names/40412

authors: compnerd shahmishal spevans


text:
Removed Unneccessary !'s from public variables in NSDateFormatter

Each public variable was cast as a forced unwrap.  I do not believe this is necessary as each variable has a getter & setter.
I do have a question with
internal var _dateFormat: String?
    public var dateFormat: String {
        get {
            if let format = _dateFormat {
                return format
            } else {
                if let obj = __cfObject {
                    return CFDateFormatterGetFormat(obj)._swiftObject
                } else {
                    return ""
                }
            }
        }
        set {
            _reset()
            _dateFormat = newValue
        }
    }

I changed the return to String, from the original String?.  If the desired return is String?, I believe the get should be
get {
            if let format = _dateFormat {
                return format
            } else {
                if let obj = __cfObject {
                    return CFDateFormatterGetFormat(obj)._swiftObject
                } else {
                    return nil
                }
            }
        }

The reason that these are ! is because they are null_resettable in Objective-C. That is, it's valid to set them to nil but you will never get nil back. We should preserve that contract in Swift as well.

Here is the API in Objective-C:
@property (null_resettable, copy) NSString *dateFormat;
@property NSDateFormatterStyle dateStyle;
@property NSDateFormatterStyle timeStyle;
@property (null_resettable, copy) NSLocale *locale;
@property BOOL generatesCalendarDates;
@property NSDateFormatterBehavior formatterBehavior;
@property (null_resettable, copy) NSTimeZone *timeZone;
@property (null_resettable, copy) NSCalendar *calendar;
@property (getter=isLenient) BOOL lenient;
@property (nullable, copy) NSDate *twoDigitStartDate;
@property (nullable, copy) NSDate *defaultDate;
@property (null_resettable, copy) NSArray<NSString *> *eraSymbols;
@property (null_resettable, copy) NSArray<NSString *> *monthSymbols;
@property (null_resettable, copy) NSArray<NSString *> *shortMonthSymbols;
@property (null_resettable, copy) NSArray<NSString *> *weekdaySymbols;
@property (null_resettable, copy) NSArray<NSString *> *shortWeekdaySymbols;
@property (null_resettable, copy) NSString *AMSymbol;
@property (null_resettable, copy) NSString *PMSymbol;
@property (null_resettable, copy) NSArray<NSString *> *longEraSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *veryShortMonthSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *standaloneMonthSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *shortStandaloneMonthSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *veryShortStandaloneMonthSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *veryShortWeekdaySymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *standaloneWeekdaySymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *shortStandaloneWeekdaySymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *veryShortStandaloneWeekdaySymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *quarterSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *shortQuarterSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *standaloneQuarterSymbols NS_AVAILABLE(10_5, 2_0);
@property (null_resettable, copy) NSArray<NSString *> *shortStandaloneQuarterSymbols NS_AVAILABLE(10_5, 2_0);
@property (nullable, copy) NSDate *gregorianStartDate NS_AVAILABLE(10_5, 2_0);
@property BOOL doesRelativeDateFormatting NS_AVAILABLE(10_6, 4_0);

authors: natemann parkera