text:
[SPARK-36876][SQL][FOLLOWUP] Support Dynamic Partition pruning for HiveTableScanExec

What changes were proposed in this pull request?
remove redundant branches
Why are the changes needed?
clean code
Does this PR introduce any user-facing change?
NO
How was this patch tested?
Exist UT

Can one of the admins verify this patch?

Can you explain why is this redundant in the Pr description?

@whcdjj how about "remove unnecessary matching cases for hive table tests in DynamicPartitionPruningHiveScanSuite" ? plz also update your description in pr.

ok to test

Can you explain why is this redundant in the Pr description?

Remove redundant branches. In this suite, table format is 'hive table', so these branches won't enter.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49686/

jenkins, test this please

Can one of the admins verify this patch?

Test build #145219 has started for PR 34587 at commit 66e2c6f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49689/

Test build #145219 has finished for PR 34587 at commit 66e2c6f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145219/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49689/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49689/

Test build #145224 has started for PR 34587 at commit b770df4.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49694/

Can one of the admins verify this patch?

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49694/

Test build #145224 has finished for PR 34587 at commit b770df4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49694/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145224/

Merged to master.

@whcdjj It's a little strange we still have FileSourceScanExec in a hive suite, then we can't make sure which code path the suite goes through. Could you investigate more for the failed case, or should we move it to DynamicPartitionPruningDataSourceSuiteBase?

@whcdjj It's a little strange we still have FileSourceScanExec case in a hive suite, then we can't make sure which code path the suite goes through. Could you investigate more for the failed case, or should we move it to DynamicPartitionPruningDataSourceSuiteBase?

ok, i will try it.

authors: AmplabJenkins HyukjinKwon SparkQA whcdjj wzhfy


text:
[SPARK-38230][SQL] InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions in most cases

What changes were proposed in this pull request?
Add a spark conf in order to just fetch partitions' name instead of fetching partitions' details. This can reduce requests on hive metastore.
Why are the changes needed?

method listPartitions is order to get locations of partitions and compute custom partition locations(variable customPartitionLocations), but in most cases we do not have custom partition locations.
method listPartitionNames just fetchs partitions' name, it can reduce requests on hive metastore db.

Does this PR introduce any user-facing change?
Yes, we should config "spark.sql.hasCustomPartitionLocations = false"
How was this patch tested?

recompile InsertIntoHadoopFsRelationCommand.scala
update spark-sql_2.12-3.0.2.jar
run insert into cases

Can one of the admins verify this patch?

Could you fix the GA?
Besides, any prof showing that this is taking too much time?

@stczwd
spark-sql run insert into table xxx partition xxx (x1, x2) select 1,2:
before optimization:

after optimization:

Hm, I got your mind. After looking at the relevant logic, customPartitionLocations will only be used while overwriting hive static partition.
Thus, changing listPartitions to listPartitionNames by default, and calling listPartitions before deleteMatchingPartitions is a better idea. It will not add much time for static partition overwriting as listPartitions for staticPartition won't take any time.
cc @cloud-fan

@stczwd
If so, there are also performance problems while overwriting hive static partitions.

source: gharchive/pull-request

authors: AmplabJenkins coalchan stczwd

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-40034][SQL] PathOutputCommitters to support dynamic partitions

What changes were proposed in this pull request?

Uses the StreamCapabilities probe in MAPREDUCE-7403 to identify when a
PathOutputCommitter is compatible with dynamic partition overwrite.
This patch has unit tests but not integration tests; really needs
to test the SQL commands through the manifest committer into gcs/abfs,
or at least local fs. That would be possible once hadoop 3.3.5 is out...
Uses the StreamCapabilities probe in MAPREDUCE-7403 to identify when a
PathOutputCommitter is compatible with dynamic partition overwrite.
Why are the changes needed?
Hadoop 3.3.5 adds a new committer in mapreduce-core which works fast and correctly on azure and gcs. (it would also work on hdfs, but its optimised for the cloud stores).
The stores and the committer do meet the requirements of Spark SQL Dynamic Partition Overwrite, so it is OK to for spark to work through it.
Spark does not know this; MAPREDUCE-7403 adds a way for any PathOutputCommitter to declare that they are compatible; the IntermediateManifestCommitter will do so.
(https://github.com/apache/hadoop/pull/4728)
Does this PR introduce any user-facing change?
No.
There is documentation on the feature in the hadoop manifest committer docs.
How was this patch tested?

Unit tests in hadoop-cloud which work with hadoop versions with/without the matching change.
New integration tests in https://github.com/hortonworks-spark/cloud-integration which require spark to be built against hadoop with the manifest committer declaring compatibility

Those new integration tests include

spark sql test derived from spark's own CloudRelationBasicSuite.scala#L212
Dataset tests extended to verify support for/rejection of dynamic partition overwrite AbstractCommitDataframeSuite.scala#L151

Tested against azure cardiff with the manifest committer; s3 london (s3a committers reject dynamic partition overwrites)

this should interest @sunchao and @dongjoon-hyun -know that this doesn't add support to the s3a committers; s3 itself doesn't do the right thing (rename()). Does for abfs and gcs through the manifest committer.

the hadoop side of this change is now merged in.
@attilapiros do you have any time to review this again?

added a new test case and updated the docs. I've not yet rebased it/merged it with your committer work, but the docs shouldn't clash. once this PR is in I will set my local build up to run your tests against s3 london

BTW, when is the ETA for Apache Hadoop 3.3.5, @steveloughran ?

@dongjoon-hyun I'm off on vacation next week; we will fork off the branch the week after.
things i'd like in if anyone has the time

upgraded shaded parquet
the shaded avro PR
get that arm64 docker image working for a release there

Merged to master for Apache Spark 3.4.0.

Thank you, @steveloughran and @attilapiros .

@attilapiros FYI, there is a new PR for this area.

https://github.com/apache/spark/pull/40221

This is reverted from branch-3.4 only while being kept in master branch for Apache Spark 3.5.

https://github.com/apache/spark/commit/57aa3d1ca6e17e4c6b934d74176ea22ca56d60f7

authors: dongjoon-hyun steveloughran


text:
[SPARK-42808][CORE] Avoid getting availableProcessors every time in MapOutputTrackerMaster#getStatistics

What changes were proposed in this pull request?
The return value of Runtime.getRuntime.availableProcessors is generally a fixed value. It is not necessary to obtain it every time getStatistics is called to avoid a native method call.
Why are the changes needed?
Does this PR introduce any user-facing change?
No
How was this patch tested?
exist UT

Merged to master


text:
[SPARK-45056][PYTHON][SS][CONNECT] Termination tests for streamingQueryListener and foreachBatch

What changes were proposed in this pull request?

Add termination tests for StreamingQueryListener and foreachBatch. The behavior is mimicked by creating the same query on server side that would have been created if running the same python query on client side. Refactored the code a bit for testing purpose.
Why are the changes needed?

Necessary tests
Does this PR introduce any user-facing change?

No
How was this patch tested?

Test only addition
Was this patch authored or co-authored using generative AI tooling?

No

CC @bogao007 @rangadi @ueshin Can you guys take a look? Thanks in advance!

Reminder to myself: https://github.com/apache/spark/pull/42687 and this PR will have conflicts in streamingForeachBatchRunnerCleanerCache

Now it fails with No module named 'grpc_status', let me search through how to resolve this

Merged to master.

let me make a quick followup or revert.

@WweiL mind creating a followup to fix them?

authors: HyukjinKwon WweiL


text:
[SPARK-48286] Fix analysis of column with exists default expression - Add user facing error

What changes were proposed in this pull request?
FIRST CHANGE
Pass correct parameter list to org.apache.spark.sql.catalyst.util.ResolveDefaultColumns#analyze when it is invoked from org.apache.spark.sql.connector.catalog.CatalogV2Util#structFieldToV2Column.
org.apache.spark.sql.catalyst.util.ResolveDefaultColumns#analyze method accepts 3 parameter

Field to analyze
Statement type - String
Metadata key - CURRENT_DEFAULT or EXISTS_DEFAULT

Method org.apache.spark.sql.connector.catalog.CatalogV2Util#structFieldToV2Column
pass fieldToAnalyze and EXISTS_DEFAULT as second parameter, so it is not metadata key, instead of that, it is statement type, so different expression is analyzed.
Pull requests where original change was introduced
https://github.com/apache/spark/pull/40049 - Initial commit
https://github.com/apache/spark/pull/44876 - Refactor that did not touch the issue
https://github.com/apache/spark/pull/44935 - Another refactor that did not touch the issue
SECOND CHANGE
Add user facing exception when default value is not foldable or resolved. Otherwise, user would see message "You hit a bug in Spark ...".
Why are the changes needed?
It is needed to pass correct value to Column object
Does this PR introduce any user-facing change?
Yes, this is a bug fix, existence default value has now proper expression, but before this change, existence default value was actually current default value of column.
How was this patch tested?
Unit test
Was this patch authored or co-authored using generative AI tooling?
No

@urosstan-db  can you re-trigger the test?

@cloud-fan Sure, sorry, I did not see this failed

can we add a test? Besically any default column value that is unfoldable, like current_date(), can trigger this bug

It is really strange, but I seen this issue on describe table also.

can we add a test? Besically any default column value that is unfoldable, like current_date(), can trigger this bug

Also, alter table add column default should trigger this bug?

can we add a test? Basically any default column value that is unfoldable, like current_date(), can trigger this bug

Also, alter table add column default should trigger this bug?

For example, ALTER TABLE t ADD COLUMN c DEFAULT CURRENT_DATE()

thanks, merging to master!

@cloud-fan @urosstan-db It looks like this broke the build for Spark 3.5 on scala 2.13...
[ERROR] [Error] ...sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala:299: value defaultValueNotConstantError is not a member of object org.apache.spark.sql.errors.QueryCompilationErrors
I don't see that definition defined on the 3.5 branch

https://issues.apache.org/jira/browse/SPARK-48608

@urosstan-db let's add defaultValueNotConstantError to branch 3.5

@cloud-fan Here it is https://github.com/apache/spark/pull/46978

authors: cloud-fan dtenedor tgravescs urosstan-db


text:
[SPARK-4454] Properly synchronize accesses to DAGScheduler cacheLocs map

This patch addresses a race condition in DAGScheduler by properly synchronizing accesses to its cacheLocs map.
This map is accessed by the getCacheLocs and clearCacheLocs() methods, which can be called by separate threads, since DAGScheduler's getPreferredLocs() method is called by SparkContext and indirectly calls getCacheLocs().  If this map is cleared by the DAGScheduler event processing thread while a user thread is submitting a job and computing preferred locations, then this can cause the user thread to throw "NoSuchElementException: key not found" errors.
Most accesses to DAGScheduler's internal state do not need synchronization because that state is only accessed from the event processing loop's thread.  An alternative approach to fixing this bug would be to refactor this code so that SparkContext sends the DAGScheduler a message in order to get the list of preferred locations.  However, this would involve more extensive changes to this code and would be significantly harder to backport to maintenance branches since some of the related code has undergone significant refactoring (e.g. the introduction of EventLoop).  Since cacheLocs is the only state that's accessed in this way, adding simple synchronization seems like a better short-term fix.
See #3345 for additional context.

Test build #27651 has started for   PR 4660 at commit 12d64ba.

This patch merges cleanly.

/cc @pwendell @markhamstra

Test build #27651 has finished for   PR 4660 at commit 12d64ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27651/
Test PASSed.

This LGTM Josh - thanks for giving this such a thorough look.

I pushed a a51fc7ef9adb6a41c4857918217f800858fced2c hotfix to roll back a bit of code cleanup that I did which added a getOrElseUpdate call, since this might have a performance impact as this code is called O(num tasks) times.

authors: AmplabJenkins JoshRosen SparkQA pwendell


text:
[SPARK-50085][PYTHON] Make lit(ndarray) with np.int8 respect the numpy datatype

What changes were proposed in this pull request?
Make lit(ndarray) with np.int8 respect the numpy datatype
Why are the changes needed?
there was a py4j limitation so NumpyArrayConverter for Py4J converts Array[Byte] to Array[Short], we should convert it back to Array[Byte] to respect the numpy datatype
Does this PR introduce any user-facing change?
before:
In [8]: spark.range(1).select(sf.lit(np.int8(123)), sf.lit(np.array([1,2,3], np.int8)))
Out[8]: DataFrame[123: tinyint, ARRAY(1S, 2S, 3S): array<smallint>]

after:
In [3]: spark.range(1).select(sf.lit(np.int8(123)), sf.lit(np.array([1,2,3], np.int8)))
Out[3]: DataFrame[123: tinyint, ARRAY(1S, 2S, 3S): array<tinyint>]

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Merged to master.

authors: HyukjinKwon zhengruifeng


text:
[SPARK-6620] Speed up toDF() and rdd() functions by constructing converters in ScalaReflection

cc @marmbrus

Test build #29441 has started for   PR 5279 at commit ab7585b.

Test build #29441 has finished for   PR 5279 at commit ab7585b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29441/
Test PASSed.

I know you are probably still working on this - any benchmark numbers?

cc @davies since you guys are both changing this part of the code lately.

Went though very quickly for the first time, left some styling comments.

Test build #29499 has started for   PR 5279 at commit 6a35425.

Test build #29499 has finished for   PR 5279 at commit 6a35425.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29499/
Test PASSed.

Here are the benchmark numbers: http://pastie.org/private/6vg7kg2yfwop2ov5zu2eq
TLDR: On real cluster the new implementation made rdd() 1.9 times faster and toDF() 2.2 times faster.

Test build #29514 has started for   PR 5279 at commit 99d333a.

Test build #29514 has finished for   PR 5279 at commit 99d333a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29514/
Test PASSed.

ping

@vlyubin who are you pinging? is this still "WIP"?

@rxin Sorry, I just removed the WIP tag. There isn't anything more to add, as it turned out that there are no places here where we could use SpecificMutableRow to speed things up.
So I'm pinging whoever wants to add more comments on the code ...

Jenkins, retest this please.

Test build #29688 has started for   PR 5279 at commit 99d333a.

Test build #29688 has finished for   PR 5279 at commit 99d333a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29688/
Test FAILed.

Test build #29749 has started for   PR 5279 at commit 3425446.

Test build #29749 has finished for   PR 5279 at commit 3425446.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29749/
Test FAILed.

Test build #29751 has started for   PR 5279 at commit 8cad6e2.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29750/
Test FAILed.

Jenkins, retest this please.

Test build #29752 has started for   PR 5279 at commit 8cad6e2.

Test build #29751 has finished for   PR 5279 at commit 8cad6e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29751/
Test PASSed.

Test build #29752 has finished for   PR 5279 at commit 8cad6e2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29752/
Test PASSed.

@vlyubin This optimization seems good to me, we also did something similar in Python API.
Another king of optimization we could add is that we only call converter for some data types, bypass others. For example, If you do not have UDF/DataType in your schema, you do not even need a converter. This also apply to some nested type (StructType, MapType, ArrayType).
For UDT, we may need another converter for the result from udf.serialize().

This looks great.  Thanks for working on it!  Two high level pieces of feedback:

Can you move all this conversion logic to a separate object?  Its my fault this is all in one place, and I'd like to fix that.
Did you try removing the other => other cases?  I'd rather actually have errors now if we are making mistakes on types.

Otherwise this LGTM.

@marmbrus Not having case other => other in createScalaConverter makes java beans fail. As of now I also need it to handle nulls.
As for createCatalystConverter, you're right - those shouldn't be necessary in most cases. I'll replace them with asInstanceOf.

What is the java beans error?  Maybe we can fix that.

@marmbrus In
val elementConverter = createScalaConverter(arrayType.elementType)
(item: Any) => item match {
  case s: Seq[_] => s.map(elementConverter)
  case other => other
}```
item is a java array. So we can't do anything with it apart from leaving it the same (that's the intended behavior, right?).

That brings up a good point - should ArrayTypes be converted into properly primitive Java Arrays if they began as Seqs?

Test build #29821 has started for   PR 5279 at commit 881dc60.

Test build #29821 has finished for   PR 5279 at commit 881dc60.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29821/
Test FAILed.

Test build #29822 has started for   PR 5279 at commit 939c225.

Test build #29822 has finished for   PR 5279 at commit 939c225.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29822/
Test FAILed.

Jenkins retest this please

Test build #29823 has started for   PR 5279 at commit afa3aa5.

Test build #29823 has finished for   PR 5279 at commit afa3aa5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29823/
Test PASSed.

Test build #29832 has started for   PR 5279 at commit 74301fa.

Test build #29832 has finished for   PR 5279 at commit 74301fa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29832/
Test FAILed.

Test build #29896 has started for   PR 5279 at commit dec6802.

Test build #29897 has started for   PR 5279 at commit c327bc9.

Test build #29896 has finished for   PR 5279 at commit dec6802.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29896/
Test PASSed.

Test build #29897 has finished for   PR 5279 at commit c327bc9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29897/
Test PASSed.

@vlyubin Would you mind to add [SQL] to the PR title? Thanks!

LGTM, @marmbrus would you mind doing a final pass?

Test build #29988 has started for   PR 5279 at commit 11a20ec.

Test build #29988 has finished for   PR 5279 at commit 11a20ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29988/
Test PASSed.

Test build #30051 has started for   PR 5279 at commit e75a387.

Test build #30051 has finished for   PR 5279 at commit e75a387.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30051/
Test PASSed.

authors: AmplabJenkins SparkQA aarondav davies liancheng marmbrus rxin vlyubin


text:
[SPARK-6917] [SQL] DecimalType is not read back when non-native type exists

cc @yhuai

Merged build triggered.

Merged build started.

Test build #33917 has started for   PR 6558 at commit 3b4a94f.

Regression test?

Yeah, let's add a test.

regression test added.

Merged build triggered.

Merged build started.

Test build #33927 has started for   PR 6558 at commit b43845c.

Test build #33917 has finished for   PR 6558 at commit 3b4a94f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test build #33927 has finished for   PR 6558 at commit b43845c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@yhuai fixed.

Merged build triggered.

Merged build started.

Test build #33937 has started for   PR 6558 at commit 48cc57c.

Test build #33937 has finished for   PR 6558 at commit 48cc57c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33938 has started for   PR 6558 at commit c877ca8.

Test build #33938 has finished for   PR 6558 at commit c877ca8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

authors: AmplabJenkins JoshRosen SparkQA davies yhuai


text:
[SPARK-8640] [SQL] Enable Processing of Multiple Window Frames in a Single Window Operator

This PR enables the processing of multiple window frames in a single window operator. This should improve the performance of processing multiple window expressions wich share partition by/order by clauses, because it will be more efficient with respect to memory use and group processing.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

cc @yhuai

ok to test

Merged build triggered.

Merged build started.

Test build #38795 has started for   PR 7515 at commit 300d9f6.

Test build #38795 has finished for   PR 7515 at commit 300d9f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #39223 has started for   PR 7515 at commit f0e1c21.

Test build #39223 has finished for   PR 7515 at commit f0e1c21.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

LGTM. Merging!

authors: AmplabJenkins SparkQA hvanhovell yhuai


text:
[chart] Pie chart left side elements -> label disappears

Screenshot
Open chart Are you an ethnic minority in your city?
https://user-images.githubusercontent.com/25153919/104327526-2b927b80-54eb-11eb-89d0-1b3e90b12d86.mov
Description
When you put mouse on left elements of chart then label dissapears. it should remain.
Design input

resolved by bumping Echarts to 5.0.2

source: gharchive/issue

authors: adam-stasiak junlincc

repo: apache/superset


text:
Problem upgrading Superset from 0.38.0 (python 3.6) to 1.0.0 (python 3.8)

Hi,
I'm facing an issue upgrading Superset from 0.38.0 to 1.0.0. Currently I have a 0.38.0 Superset version installed in a virtual env, on a Linux Ubuntu 18.04 server, with python 3.6. To upgrade to the new major 1.0.0 release, I have to upgrade to python 3.7 to make it works. For this reason, I upgraded the development server to an Ubuntu 20.04 with a python 3.8 distribution. But I can't find how to upgrade the Superset installation, how to upgrade the virtual environment with the Superset install...
Do I have to reinstall Superset from scratch ?
Any help will be appreciated ;-)
Best regards

@JeanBaptisteINITIATIVES you don't have to install superset from scratch (it would be nice from a immutable infrastructure perspective). You need to update/reinstall superset's virtual environment, install superset, upgrade the db, superset init. The normal install steps
Note: Superset 1.0.0 has a dependency problem on python 3.8 and PyJWT, so use python 3.7 or pin PyJWT at 1.7.1

Hi @dpgaspar,
Thanks for the reply, I'll do all these steps and give some feedback if I encounter some problem during this process.
Best regards

For anyone concerned by kind of upgrading (Python + superset), following the install steps worked fine :

reinstall the superset's virtual environment
install superset (pip install apache-superset --upgrade)
upgrade the db, changing if needed the sql alchemy url in the config.py file (superset db upgrade)
initiate superset (superset init)

authors: JeanBaptisteINITIATIVES dpgaspar


text:
Error Upgrading Bar Charts with SQL Functions in Series to Superset 1.0.0

I am migrating from superset version 0.34.0rc2 to 1.0.0. In the old version we use the "Bar Chart" visualization with a sql function in the "Series" box successfully. In the new version, this chart is broken. For example, we have a chart that renders successfully in the old version has a "Series" column of datepart(year,shipment_month). This chart fails in the new version. The data source is in redshift.
Expected results
I expected the bar chart to render correctly.
Actual results
I saw an error message stating Columns missing in datasource: ['datepart(year,shipment_month)']
Screenshots
Configuration from the old version:

Successfully rendered chart in the old version:

Configuration in the new version:

Error message in the new version:

How to reproduce the bug

View a Redshift dataset with a datetime column in Explore mode
Choose "Bar Chart" visualization
Add "datepart(year, )" as a "Series" entry
Click the "Run" button
See the error

Environment
(please complete the following information):

superset version: superset version

Loaded your LOCAL configuration at [/home/superset/superset_config.py]
logging was configured successfully
INFO:superset.utils.logging_configurator:logging was configured successfully
/usr/local/lib/python3.7/site-packages/flask_caching/__init__.py:192: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
  "Flask-Caching: CACHE_TYPE is set to null, "
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Superset 1.0.0
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

python version: python --version

Python 3.7.9

node.js version: node -v

v10.23.2

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
Request log:

[2021-02-04 20:29:35 +0000] [63] [DEBUG] POST /superset/explore_json/
DEBUG:parsedatetime:eval - with context - False, False
Cache key: f540b9108c6c05f255250f9c259ec9a9
INFO:superset.viz:Cache key: f540b9108c6c05f255250f9c259ec9a9
127.0.0.1 - - [04/Feb/2021:20:29:35 +0000] "POST /superset/explore_json/ HTTP/1.1" 400 981 "http://localhost:8088/superset/explore/?form_data=%7B%22datasource%22%3A%221__table%22%2C%22viz_type%22%3A%22dist_bar%22%2C%22url_params%22%3A%7B%7D%2C%22time_range_endpoints%22%3A%5B%22inclusive%22%2C%22exclusive%22%5D%2C%22granularity_sqla%22%3A%22created_on%22%2C%22time_range%22%3A%22Last+week%22%2C%22metrics%22%3A%5B%22count%22%5D%2C%22adhoc_filters%22%3A%5B%5D%2C%22groupby%22%3A%5B%22datepart%28year%2C+created_on%29%22%5D%2C%22columns%22%3A%5B%5D%2C%22row_limit%22%3A10000%2C%22color_scheme%22%3A%22supersetColors%22%2C%22label_colors%22%3A%7B%7D%2C%22show_legend%22%3Atrue%2C%22y_axis_format%22%3A%22SMART_NUMBER%22%2C%22bottom_margin%22%3A%22auto%22%2C%22x_ticks_layout%22%3A%22auto%22%7D" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36"

[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
We are upgrading to superset 1.0.0 by cloning our taking a snapshot of the metadata db and spinning up a new superset instance on top of it. The failing chart is a copy of a chart that was working in the old version.

@akud
what is the data type of that column? can you also provide the sql query that generated in the old version? thanks

Hi @junlincc the column is a DATE column. Here is an obfuscated version of the query produced by the old version of superset:
SELECT version AS version,
                  (datepart(year, shipment_month)) AS "datepart(year,shipment_month)",
                  product_group AS product_group,
                  sum(net_shipped_amount_usd) AS "sum(net_shipped_amount_usd)"
FROM
  (... Data Source custom SQL ...)
GROUP BY version, (datepart(year, shipment_month)), product_group
ORDER BY "sum(net_shipped_amount_usd)" DESC
LIMIT 1000;

For anyone stumbling across this, I was able to solve the problem in the UI by editing the datasource and adding a "Derived Column" with the sql function, then using that in the chart Series.

Since this issue is in support of upgrading to a version that the project no longer supports, and the thread hasn't seen any attention in quite some time, I'll close this. If you have a similar issue in current versions of Superset, I highly encourage you to open a new issue linking back to this one, with updated context/screenshots. Thank you!

authors: akud junlincc rusackas


text:
Applicable driver not found; attempting to install with Selenium Manager (Beta)

I got this INFO in celery log:
[WARNING/ForkPoolWorker-3] Applicable driver not found; attempting to install with Selenium Manager (Beta)
Originally posted by @Habeeb556 in https://github.com/apache/superset/issues/29401#issuecomment-2228313179

@dosu I am not using docker.

@dosu I am not using a docker

@dosu find the below now:
[WARNING/ForkPoolWorker-1] Failed at generating thumbnail Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line
Traceback (most recent call last):
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/screenshots.py", line 170, in compute_and_cache
   payload = self.get_screenshot(user=user, window_size=window_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/screenshots.py", line 93, in get_screenshot
   self.screenshot = driver.get_screenshot(self.url, self.element, user)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/webdriver.py", line 354, in get_screenshot
   driver = self.auth(user)
            ^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/webdriver.py", line 276, in auth
   driver = self.create()
            ^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/superset/utils/webdriver.py", line 273, in create
   return driver_class(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/firefox/webdriver.py", line 201, in __init__
   super().__init__(command_executor=executor, options=options, keep_alive=True)
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 286, in __init__
   self.start_session(capabilities, browser_profile)
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 378, in start_session
   response = self.execute(Command.NEW_SESSION, parameters)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 440, in execute
   self.error_handler.check_response(response)
 File "/swloc/.virtualenvs/supersetvenv4/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
   raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line

To conclude this thread error with Selenium:
All I need to do is install geckodriver manually since the $PATH automatically finds /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin.
Then, install Firefox using:
sudo yum install firefox

This should resolve the following issue:
selenium.common.exceptions.SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line

as suggested here.
However, I've encountered another issue now:
selenium.common.exceptions.WebDriverException: Message: Reached error page: about:neterror?e=connectionFailure&u=http%3A//0.0.0.0%3A8080/login/&c=UTF-8&d=Firefox%20can%E2%80%99t%20establish%20a%20connection%20to%20the%20server%20at%200.0.0.0%3A8080.
Stacktrace:
RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:187:5
UnknownError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:736:5
checkReadyState@chrome://remote/content/marionette/navigate.sys.mjs:61:24
onNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39
emit@resource://gre/modules/EventEmitter.sys.mjs:154:20
receiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:35:25

@dosu

Resolved with #9135
change WEBDRIVER_BASEURL = "http://0.0.0.0:8080/" to WEBDRIVER_BASEURL = "http://0.0.0.0:8088/" and add this to superset_config.py
Then, faced another error:
[2024-07-30 16:27:50,021: ERROR/ForkPoolWorker-4] Selenium timed out waiting for chart containers to draw at url http://0.0.0.0:8088/superset/dashboard/24/?standalone=3
And I think it is related to #29730 and resolved with this https://github.com/apache/superset/issues/29401#issuecomment-2228250274
SCREENSHOT_LOCATE_WAIT = 200  # Increase the wait time for locating elements
SCREENSHOT_LOAD_WAIT = 1200   # Increase the wait time for loading the page


text:
perf: Implement model specific lookups by id to improve performance

SUMMARY
Currently the access checks are happening 2ce in the explore flow, once during the object lookup and later doing an explicit check.  Existing implementation was spending  > 500 ms in our deployment on doing object by id lookup and that contributed to ~1 - 1.5 second slowdown of the explore endpoint.
The proposed solution skips the filter on the chart / datasource lookup and let's explore to handle the access checks and return 403 if it was denied. Prior behavior was to return 404.
Out of scope
https://github.com/apache/superset/blob/f0ca158989644b793719884b52d04f93c05de1ba/superset/views/utils.py#L104 function takes > 1 second in our deployment, it is part of the explore flow while bootsrapping user data https://github.com/apache/superset/blob/f0ca158989644b793719884b52d04f93c05de1ba/superset/views/utils.py#L72  I will have separate PR to optimize the permissions lookup.
Before

After

TESTING INSTRUCTIONS
[x] Deployed and tested on dropbox staging
ADDITIONAL INFORMATION
More granular breakdown
@classmethod
def find_by_id(
    cls, model_id: Union[str, int], session: Session = None
) -> Optional[Model]:
    """
    Find a model by id, if defined applies `base_filter`
    """
    with newrelic.agent.FunctionTrace("superset.dao.base.find_by_id.session", group="DAO"):
        session = session or db.session
    with newrelic.agent.FunctionTrace("superset.dao.base.find_by_id.query_contruction", group="DAO"):
        query = session.query(cls.model_cls)
        if cls.base_filter:
            data_model = SQLAInterface(cls.model_cls, session)
            query = cls.base_filter(  # pylint: disable=not-callable
                cls.id_column_name, data_model
            ).apply(query, None)
        id_filter = {cls.id_column_name: model_id}
    try:
        with newrelic.agent.FunctionTrace("superset.dao.base.find_by_id.query_execution", group="DAO"):
            return query.filter_by(**id_filter).one_or_none()
    except StatementError:
        # can happen if int is passed instead of a string or similar
        return None

I wonder if the slowness in your check permission query is because postgres did not automatically create indexes on the sqlatable_user table (ref).
superset_test=> \d sqlatable_user;
                             Table "public.sqlatable_user"
  Column  |  Type   | Collation | Nullable |                  Default
----------+---------+-----------+----------+--------------------------------------------
 id       | integer |           | not null | nextval('sqlatable_user_id_seq'::regclass)
 user_id  | integer |           |          |
 table_id | integer |           |          |
Indexes:
    "sqlatable_user_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "sqlatable_user_table_id_fkey" FOREIGN KEY (table_id) REFERENCES tables(id)
    "sqlatable_user_user_id_fkey" FOREIGN KEY (user_id) REFERENCES ab_user(id)

I wonder if the slowness in your check permission query is because postgres did not automatically create indexes on the sqlatable_user table (ref).
superset_test=> \d sqlatable_user;
                             Table "public.sqlatable_user"
  Column  |  Type   | Collation | Nullable |                  Default
----------+---------+-----------+----------+--------------------------------------------
 id       | integer |           | not null | nextval('sqlatable_user_id_seq'::regclass)
 user_id  | integer |           |          |
 table_id | integer |           |          |
Indexes:
    "sqlatable_user_pkey" PRIMARY KEY, btree (id)
Foreign-key constraints:
    "sqlatable_user_table_id_fkey" FOREIGN KEY (table_id) REFERENCES tables(id)
    "sqlatable_user_user_id_fkey" FOREIGN KEY (user_id) REFERENCES ab_user(id)

sqlatable_user

Yeah - we have both indexes:

new relic states that a lot of time is spent in python code, not sql execution. I have 2 more places I want to optimized for better perf: https://apache-superset.slack.com/archives/G013HAE6Y0K/p1659722615103529?thread_ts=1659395755.971919&cid=G013HAE6Y0K and will have PRs coming up soon

authors: bkyryliuk ktmud


text:
Ant fixes.

Fix mkdir calls to use the mkdir Ant task. Otherwise, this does not build on Windows.

+1

Tracked in THRIFT-3962

authors: Jens-G bufferoverflow garydgregory

repo: apache/thrift


text:
cdn-in-a-box add tenant_id with user

adduser.pl was missing tenant id which is now required for users

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2008/
Test PASSed.

authors: asfgit dangogh

repo: apache/trafficcontrol


text:
TS-5040: Forward CONNECT without parent proxying.

Unless parent proxy is enabled, CONNECT methods always result in
setting up a direct tunnel to the origin server. Parent proxy,
however, has the option of forwarding the CONNECT to the next hop
and allowing it to set up the tunnel.  This change adds the
proxy.config.http.forward_connect_method configuration parameter
that can be enabled to forward CONNECT methods even if we are not
parent proxying.

Sending early for review. Will add another commit with documentation.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1182/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1075/ for details.

No docs for the new setting?

@zwoop Added docs.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1085/ for details.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1192/ for details.

authors: atsci jpeach zwoop

repo: apache/trafficserver


text:
Avoid varargs warnings when compiling LuaJIT

This has to go on 7.1.x as well, since it fails the CI currently.

clang format successful! https://ci.trafficserver.apache.org/job/clang-format-github/133/

RAT check successful! https://ci.trafficserver.apache.org/job/RAT-github/147/

FreeBSD11 build successful! https://ci.trafficserver.apache.org/job/freebsd-github/1829/

AU check failed! https://ci.trafficserver.apache.org/job/autest-github/130/

Intel CC build successful! https://ci.trafficserver.apache.org/job/icc-github/259/

Linux build successful! https://ci.trafficserver.apache.org/job/linux-github/1720/

clang-analyzer build successful! https://ci.trafficserver.apache.org/job/clang-analyzer-github/391/

authors: atsci zwoop


text:
Null Transform Plugin Autest finished

Autest files for null transform plugin

[approve ci]

[approve ci autest]

You'll want to make your first PR against master instead of 7.1.x.

authors: dyrock persiaAziz shinrich


text:
Option to enable stand-alone Leak Sanitizer

If leak detection is needed without the Address Sanitizer slowdown.

Cherry-picked to 7.1.x

authors: gtenev zwoop


text:
Disable ja3 plugin when building with boringssl

SSL_client_hello_get0_legacy_version is not available under boringssl

Cherry-picked to v9.1.x branch.

authors: randall zwoop


text:
tidy: Remove RecModeT from the code.

Once TM was removed the RecModeT became obsolete. This PR removes all references to this type. This incldues the removal of TSRecordModeType.

[approve ci autest]


text:
Try to do less work in hot function HttpHookState::getNext

This function was showing up as a hotspot but this rewritten one does not.

[approve ci autest]

Cherry-picked to v9.2.x

authors: cmcfarlen zwoop


text:
[CMSIS-NN][ARM] ARM_MATH_ARGUMENT_ERROR due to not setting the context buffer in arm_avgpool_s8

Using TVM main to build keyword spotting model with CMSIS NN (same version used on ci_qemu) would lead to error ARM_MATH_ARGUMENT_ERROR. Here are some details:
calling arm_avgpool_s8 function would result in calling this line and here since the context buffer is null and arm_avgpool_s8_get_buffer_size returns non-zero, it results in ARM_MATH_ARGUMENT_ERROR error.
cc @Mousius @ashutosh-arm @areusch

@grant-arm

Here is a gist to reproduce this error:
https://gist.github.com/mehrdadh/2f3aa844e2df21e22e5ccf9708a56e0c#file-tvm_kws_cmsis-py
Note:

You don't see the error unless you use west debug to step by step go through the code. In the generated code ARM_MATH_ARGUMENT_ERROR would result in returning -1 and code execution just fails.
If you run this with older CMSIS version like (5.8.0 which we used before on ci_qemu) it will show wrong output. The reason is that previous CMSIS didn't check for if context->buff is not NULL and it would just overwrite some memory.

cc @ashutosh-arm

Thanks @mehrdadh for putting together the script to reproduce this issue. I will get to this early next week.

@mehrdadh  I would guess, that this is related to #11702 which was merged one month ago. You you please try again on the latest TVM commit?

@PhilippvK thanks for reporting this! I checked today and the issue seems to be resolved!
I add some missing CMSIS NN source files to cmake: https://github.com/apache/tvm/pull/12642

authors: PhilippvK ashutosh-arm mehrdadh

repo: apache/tvm


text:
[VOTE] Release Apache TVM v0.11.0.rc0

Dear TVM community,
This is a call for vote to release Apache TVM version v0.11.0.  https://github.com/apache/tvm/issues/13899 for any edits you would like to see included in the release notes.
Link to release notes:
https://github.com/apache/tvm/issues/13899
Link to release candidate:
https://dist.apache.org/repos/dist/dev/tvm/tvm-v0.11.0-rc0/
Link to github pre-release page:
https://github.com/apache/tvm/releases/tag/v0.11.0.rc0
Everyone is welcomed to vote. Please vote by replying to this thread explicitly.
+1 = approve
+0 = no opinion
-1 = disapprove (provide reason)
NOTE: this thread is being mirrored in dev@
Vote will close Mar. 1st at 23:59PM GMT.

I checked
+1 , I checked
Signatures and hashes good
LICENSE, NOTICE
Signatures and hashes
No unexpected binary files
Code compiles and import in python

+1

+1

+1

+1

+1

+1

Thanks for all who voted, I'll summarise the results and publish a [RESULT] ticket.

authors: Hzfengsy areusch echuraev junrushao leandron lhutton1 tqchen vinx13


text:
Fix threshold calculation logic in PyTorch frontend

This pr is targeted to fix the bug in the issue-14805
cc @echuraev @Hzfengsy

@tvm-bot rerun

@tvm-bot rerun


text:
[CI] Pin setuptools to v58.4.0 in CI to circumvent breaking change in v58.5

Pin setuptools version to v58.4.0 to circumvent a breaking change in v58.5, see https://github.com/apache/tvm/issues/9445
cc @leandron

Thanks for fixing this @jtuyls, I think this is a sensible change but I suggest removing
https://github.com/apache/tvm/blob/f656a2274aaac64489da1ece3ed110e214e549e7/docker/install/ubuntu_install_vela.sh#L23
so we aren't surprised by the version of setuptools we get in the eventual image, I presume this is a high enough version of setuptools given it was working until a few days ago 😸

Done. And yeah, this is the last version that I could find that doesn't have the issue.


text:
[ZEPPELIN-4305] LocalStorageConfig.atomicWriteToFile throws exception

What is this PR for?
The hotfix that was made several weeks ago changed behavior of LocalStorageConfig.atomicWriteToFile, that started to atomically move files to destination. But this works without errors only when temporary directory and destination are on the same disk. When they are on different disks, atomic move isn't possible, so the exception is thrown.
This PR fixes this by performing non-atomic move to temp file on destination file system, and then atomically rename it to destination file.
What type of PR is it?
Bug Fix
What is the Jira issue?
ZEPPELIN-4305
Questions:

Does the licenses files need update?
No
Is there breaking changes for older versions?
No
Does this needs documentation?
No

Could you please make a simple test for it?

Have added the unit test...

@jongyoul @felixcheung does the current version look ok? If yes, could it be merged?

Can we merge this PR? it's really blocking me from working on new code :-(

LGTM, will merge if no more comment

@alexott CI of master branch is failed due to this PR, could you help take a look at it ? Thanks
https://api.travis-ci.org/v3/job/580601145/log.txt
08:42:53,199 ERROR org.apache.zeppelin.server.ZeppelinServer:269 - Error while stopping servlet container
MultiException stack 1 of 8
java.nio.file.NoSuchFileException: /home/travis/build/apache/zeppelin/conf_ClusterEventTest/interpreter.json2384379691433869008.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    at java.nio.file.Files.newByteChannel(Files.java:361)
    at java.nio.file.Files.createFile(Files.java:632)
    at java.nio.file.TempFileHelper.create(TempFileHelper.java:138)
    at java.nio.file.TempFileHelper.createTempFile(TempFileHelper.java:161)
    at java.nio.file.Files.createTempFile(Files.java:852)
    at org.apache.zeppelin.storage.LocalConfigStorage.atomicWriteToFile(LocalConfigStorage.java:118)
    at org.apache.zeppelin.storage.LocalConfigStorage.save(LocalConfigStorage.java:60)
    at org.apache.zeppelin.interpreter.InterpreterSettingManager.saveToFile(InterpreterSettingManager.java:311)
    at org.apache.zeppelin.interpreter.InterpreterSettingManager.init(InterpreterSettingManager.java:318)
    at org.apache.zeppelin.interpreter.InterpreterSettingManager.<init>(InterpreterSettingManager.java:189)
    at org.apache.zeppelin.interpreter.InterpreterSettingManager.<init>(InterpreterSettingManager.java:145)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1375)
    at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:272)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:366)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:116)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:698)
    at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:78)
    at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:212)
    at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:229)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:358)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:685)
    at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:78)
    at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:212)
    at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:229)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:358)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:83)
    at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:71)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
    at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
    at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:122)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:777)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:740)
    at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:710)
    at org.apache.zeppelin.server.ZeppelinServer.lambda$main$3(ZeppelinServer.java:266)
    at java.lang.Thread.run(Thread.java:748)

@zjffdu I've opened #3439 that should fix this problem - I believe that it's caused by fact that destination directory doesn't exist yet. I've added corresponding test case as well

authors: alexott jongyoul zjffdu

repo: apache/zeppelin

license_source: bigquery


text:
[ZEPPELIN-5497] Use AutoCloseable interface to close all IO

What is this PR for?
Should close clientPool explicitly, otherwise it won't be garbage collected.
Take over of #4208
What type of PR is it?

Bug Fix

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-5497

How should this be tested?

via CI

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

I will merge this into the master on Wednesday if no other comments are received.

LGTM

authors: Reamer zjffdu


text:
filtering subdirectory names.

What is this PR for?
It needs to filtering subdirectory names in navbar.
What type of PR is it?
Bug Fix
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-886
How should this be tested?
try filter the note name on navbar.
Screenshots (if appropriate)

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

Looks good to me +1

@astroshim Tested, works well, except for those 2 comments LGTM.
The code handling those notebook lists and their filtering became quite messy and error prone.
I will take some time to refactor it later.

@corneadoug Thank you for your review.
I'll refactor the code as you mentioned.

@corneadoug I refactored the codes that you pointed out. please review.

Tested, Merging if no more discussions

authors: astroshim cloverhearts corneadoug


text:
Providers not showing on login screen. CAS 5.0.0.RC3

I have added the cas-server-support-pac4j-webflow dependency and configured the cas.authn.pac4j.cas.loginUrl to point to the delegate CAS server's /login url and set the protocol to CAS30.
When redirected to the login screen, the CAS provider is not shown.
In Chrome I disable the 'hidden-xs' style and the provider appears.
Clicking the CAS provider gives the following exception.
org.springframework.webflow.engine.NoMatchingTransitionException: No transition was matched on the event(s) signaled by the [1] action(s) that executed in this action state 'clientAction' of flow 'login'; transitions must be defined to handle action result outcomes -- possible flow configuration error? Note: the eventIds signaled were: 'array['stopWebflow']', while the supported set of transitional criteria for this action state is 'array[success, error, stop]' 	at org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:130) 	at org.springframework.webflow.engine.State.enter(State.java:194) 	at org.springframework.webflow.engine.Flow.start(Flow.java:527) 	at org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:368) 	at org.springframework.webflow.engine.impl.FlowExecutionImpl.start(FlowExecutionImpl.java:223) 	at org.springframework.webflow.executor.FlowExecutorImpl.launchExecution(FlowExecutorImpl.java:140) 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 	at java.lang.reflect.Method.invoke(Method.java:497) 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) 	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) 	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) 	at com.sun.proxy.$Proxy166.launchExecution(Unknown Source) 	at org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:263) 	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963) 	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) 	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) 	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) 	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apereo.cas.web.support.CurrentCredentialsAndAuthenticationClearingFilter.doFilter(CurrentCredentialsAndAuthenticationClearingFilter.java:28) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apereo.cas.security.RequestParameterPolicyEnforcementFilter.doFilter(RequestParameterPolicyEnforcementFilter.java:261) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apereo.cas.security.ResponseHeadersEnforcementFilter.doFilter(ResponseHeadersEnforcementFilter.java:238) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apereo.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ClientInfoThreadLocalFilter.java:62) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apereo.cas.logging.web.ThreadContextMDCServletFilter.doFilter(ThreadContextMDCServletFilter.java:90) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) 	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) 	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) 	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) 	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) 	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) 	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) 	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) 	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) 	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) 	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620) 	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677) 	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) 	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) 	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) 	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) 	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) 	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 	at java.lang.Thread.run(Thread.java:745)
I assume that it is hidden because it is not yet complete?!

No. It's complete as far as it should be. Switch to RC4-SNAPSHOT and try again.

Ok, under RC4-SNAPSHOT, things are a little different.
I still don't see the CAS delegate services 'Or login with:' listed. The parent div has a class of hidden-xs.
This time, when unhiding it and clicking it, it takes me to the delegate CAS 4.x server correctly. I can login successfully (Both TGT and ST are created).
On redirection back to my CAS server (5.0.0.RC4-SNAPSHOT) I get an Unauthorized Access panel with empty values across the row.
Debugging tells me that there was 'No principal was found in the response from the CAS server'.
N.B. I have changed my protocol to CAS20 and CAS30 (just in case!)

One note, I have tgc.secure=false in both CAS servers.

You will have to share logs at DEBUG before any diagnosis can be made, and your settings.

The first thing that I noticed moving from RC3 to RC4-SNAPSHOT was the service name/description had changed from my service to 'OAuth Callback url'/'OAuth Wrapper Callback Url'...
I have attached the logfiles from the two CAS servers.
CAS4 (4.2.1) is standard base with
-support-rest
-support-generic
CAS5 (5.0.0.RC4-SNAPSHOT) is standard base with
-support-jdbc
-support-json-service-registry
-support-pac4j-webflow
-support-oauth-webflow
Both servers have tgc.secure=false
If this is not enough or you require different logging, just tell me what to turn on.
Thanks for your help so far!

yes. We do need your logs at DEBUG when CAS starts up so we can see what's going on, and also your entire settings in application/cas.properties.
Note that there is no such thing as tgc.secure in 5. And you're welcome to submit your overlay to the issues project.

The login screen acts correctly, showing the CAS provider when accessed directly from the browser but not when redirected to from OAuth.

but not when redirected to from OAuth.

Explain this please. What do you mean?

If I hit http://myhost:8080/cas/login directly rather than an OAuth secured app redirecting to it, the CAS provider is shown as well as the 'Links to CAS Resources' panel.
The usual events are that I hit http://myhost:8181/myapp and that redirects me to the CAS login page without the 'Links to CAS Resources' panel or the 'Or login with:' panel.
I am using the spring annotation @EnableOAuth2Sso on myapp.
I will put a cut down example together tomorrow to showcase it...
Cheers

Ah I see. OK. That sounds like an issue; logs should help but I can probably attempt to diagnose this further based on your report here.

I have created a cut down application showing App -> [oauth2] -> CAS5 -> [cas20] -> CAS3.
This is not my original setup as I was using a CAS4 instead of the CAS3 server. However, the third party are still on CAS 3.5.2, so I need to get that working first!
The CAS delegate is being rendered correctly in this setup but on return from the CAS3 login we get the CAS5 'Log In Successful' page instead of the OAuth2 'Authorise the application' page.
GitHub example

Thanks. This behavior isnt supported today. You should probably rename the issue title to make the request more explicit.

authors: CobraFlow mmoayyed

repo: apereo/cas


text:
Only need one MongoClient for this MongoAuthenticationHandler bean.  …

…Everytime getAuthentication is called resource are created which are never being returned.   MongoClient has connection pool and threads which monitor that pool.  Probably only need one MongoClient for the application.  At min we just need one for this bean.  @PostConstruct will create the connection after bean is initialized and the @PreDestroy will close it returning resources.  Old methods was causing thousands of threads on our server until we restarted CAS.
I have tested something similar but not exactly this change.  I've built my own MongoAuthenicatorHandler(s) as we have both PIN and password codes.  It's the same code just in my own classes and packages.  Which I tested with 5.0.5 of CAS.  Mostly I wanted all to be aware of this issue and fix it as suggested in this pull request or another way.

Thank you. Could you please provide the same branch for 5.1 as well?

authors: mmoayyed philliprower


text:
CentOS 7 中连接出现 {"error": "connect error: timeout: no recent network activity"} 问题

已确认服务器444端口已放行并且可以通过外网进行访问

hysteria 正确运行了并且监听了该端口。

配置文件如下

v2rayn已加载最新的 hysteria2 核心

连接出现 {"error": "connect error: timeout: no recent network activity"} 问题

执行 /usr/local/bin/hysteria server --config /etc/hysteria/config.yaml

补充：
新发现！如果关闭了 hysteria 服务后执行 /usr/local/bin/hysteria server --config /etc/hysteria/config.yaml 显示正常？

但是 v2rayn 依旧无法连接

为什么感觉是v2rayN的问题，和hysteria server应该没有关系吧

这个我也不确定，有空我试试clash

遇到了同样问题 应该和v2rayN没关系 单独启动hy client也是这个错误

遇到了同样问题 应该和v2rayN没关系 单独启动hy client也是这个错误

据说centos7比较容易触发该问题，不知道是不是真的

客户端和服务端版本是不是不一致

客户端和服务端版本是不是不一致

已经手动替换客户端了啊

破案了，centos7的更新linux内核，这个搞了好久死活不行，更新完内核就正常了，因为我的另一个centos7 内核时5点几的，搭建是正常的

破案了，centos7的更新linux内核，这个搞了好久死活不行，更新完内核就正常了，因为我的另一个centos7 内核是5点几的，搭建是正常的，可以尝试这个更新Linux内核文章

所以真的是系统问题啊，hhh

一样的问题，udp端口都通，centos内核也升级了
服务器端：
[root@kabirgoyal ~]# netstat -ntplu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
... ...
udp6       0      0 :::30040                :::*                                1208/./hysteria-lin
服务器是centos7，版本如下
[root@kabirgoyal yc]# uname -a
Linux kabirgoyal.com 6.7.6-1.el7.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 17:06:52 EST 2024 x86_64 x86_64 x86_64 GNU/Linux
[root@kabirgoyal yc]# uname -r
6.7.6-1.el7.elrepo.x86_64
[root@kabirgoyal yc]# cat server.yaml
listen: :30040
tls:
cert: "./cert.pem"
key: "./key.pem"
bandwidth:
up: 100 mbps
down: 20 mbps
auth:
type: password
password: chika
resolver:
type: https
https:
addr: 1.1.1.1:443
timeout: 10s
[root@kabirgoyal yc]# cat startserver.log
2024-02-28T11:39:42Z    INFO    server mode
2024-02-28T11:39:42Z    INFO    server up and running   {"listen": ":30040"}
服务器的ip地址是209.141.55.218，我hysteria程序启用的端口是30040，并且防火墙的状态如下
[root@kabirgoyal ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2024-02-28 11:40:52 UTC; 12min ago
Docs: man:firewalld(1)
Process: 538 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 538 (code=exited, status=0/SUCCESS)
Feb 28 11:38:42 localhost.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Feb 28 11:38:43 localhost.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.
Feb 28 11:38:43 localhost.localdomain firewalld[538]: WARNING: AllowZoneDrifting is enabled. This is considered an inse... now.
Feb 28 11:38:43 localhost.localdomain firewalld[538]: ERROR: Failed to read file "/proc/sys/net/netfilter/nf_conntrack_...lper'
Feb 28 11:38:43 localhost.localdomain firewalld[538]: WARNING: Failed to get and parse nf_conntrack_helper setting
Feb 28 11:40:50 kabirgoyal.com systemd[1]: Stopping firewalld - dynamic firewall daemon...
Feb 28 11:40:52 kabirgoyal.com systemd[1]: Stopped firewalld - dynamic firewall daemon.
Hint: Some lines were ellipsized, use -l to show in full.
端口也都通的
客户端发送：
echo "Test UDP message" | nc -u 209.141.55.218 30041
服务器：
[root@kabirgoyal yc]# nc -u -l 30040
Ncat: bind to :::30040: Address already in use. QUITTING.
[root@kabirgoyal yc]# nc -u -l 30041
Test UDP message

客户端和服务端版本是不是不一致
要在运营商的控制端放行UDP协议的端口，而不是TCP端口

https://blog.yywq.me/article/39f87fd6-1c3e-4826-90d9-5d031393ca8d
这个打不开
blog.yywq.me currently does not have any sponsors for you.

破案了，centos7的更新linux内核，这个搞了好久死活不行，更新完内核就正常了，因为我的另一个centos7 内核是5点几的，搭建是正常的，可以尝试这个更新Linux内核文章

所以真的是系统问题啊，hhh

https://blog.yywq.me/article/39f87fd6-1c3e-4826-90d9-5d031393ca8d
这个打不开
blog.yywq.me currently does not have any sponsors for you.

authors: 2602875449 Airskotex DreamFirstSnow Phascolarctos chengxuylongweiguo hanlray yaochong-06

repo: apernet/hysteria

license: MIT


text:
Inventory UI

We need UI for inventory so player's can select weapons. A pre-existing example of this exists in the pre-rewrite branch, but we are not keeping the old styling, so feel free to experiment. https://github.com/apetavern/sbox-grubs/tree/pre-rewrite/code/UI/Gamemode

https://github.com/apetavern/sbox-grubs/pull/153

authors: matekdev trundler-dev

repo: apetavern/sbox-grubs


text:
nuget package

Unable to install package via nuget or power shell.is there any other way to add GitHub projects in project file ?

Hi,
Check the Include prerelease checkbox

it's came in results with pre-release checkbox tick.

authors: joadan kiranmaya

repo: apexcharts/Blazor-ApexCharts


text:
toggleDataPointSelection causes error in pie chart

Bug report
toggleDataPointSelection causes error in a pie chart in latest 3.18.0 version
while in 3.17 version it works just fine.
Codepen
https://codesandbox.io/s/vue-basic-example-5t1jf
Explanation
I'm using a donut chart and I call toggleDataPointSelection(0) method a long time after the chart is loaded (after user clicks a button).
I get the following error and a donut piece is disappearing instead of selecting.
<path> attribute d: Expected number, "M NaN NaN A NaN Na…".

Thanks for reporting, the issue will be address soon.

What is the cause of this error. I am also getting the same issue.

Any update on this? I'm facing this issue too

Hí anh em

is there any fix for this i am having big problems

I have solved all those errors by destroying the chart when component unmounts.
I did it on sveltekit, and there the chart needs to be renderen inside the onMount method and binding the html container element with bind:this to avoid another annoying error (window is not defined), and to be able to identify the chart inside the onDestroy method a callback function needs to be created where the chart is rendered (onMount).
So the workaround gets like this:
` let graphContainer: HTMLElement;
let destroyer: any;
onMount(async () => {
const ApexCharts = (await import('apexcharts')).default
const container = graphContainer;
const chart = new ApexCharts( container, setOptions(pairData))
chart.render();
destroyer = () => chart.destroy()
})
onDestroy(async () => {
destroyer()
})`

any fix

authors: AlexParton DasunMuthuruwan Mominjon Rao-Mudasir junedchhipa khoanld-2442 poddarkhushbu07 vitalcrazz


text:
Cannot install apexcharts using npm-asset

Description
I would like to add apexcharts as a dependency in my Composer.json file.
"npm-asset/apexcharts": "^3.49"
Steps to Reproduce

create composer file ( link to my composer.json )
add npm-asset/apexcharts as dependency
run composer update

Expected Behavior
Actual Behavior
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - npm-asset/svg.resize.js 1.4.3 requires npm-asset/svg.select.js >=2.1.2,<3.0.0 -> satisfiable by npm-asset/svg.select.js[2.1.2].
    - You can only install one version of a package, so only one of these can be installed: npm-asset/svg.select.js[2.1.2, 3.0.1].
    - npm-asset/apexcharts[3.49.0, ..., 3.49.2] require npm-asset/svg.resize.js >=1.4.3,<2.0.0 -> satisfiable by npm-asset/svg.resize.js[1.4.3].
    - npm-asset/apexcharts[3.49.0, ..., 3.49.2] require npm-asset/svg.select.js >=3.0.1,<4.0.0 -> satisfiable by npm-asset/svg.select.js[3.0.1].
    - Root composer.json requires npm-asset/apexcharts ^3.49 -> satisfiable by npm-asset/apexcharts[3.49.0, 3.49.1, 3.49.2].

Screenshots

Reproduction Link

the problem has existed since version 3.16.1
https://github.com/apexcharts/apexcharts.js/compare/v3.16.0...v3.16.1


text:
The node should not fork if there are no actionable requests

It should return early

@andreogle Do we close this issue?

Yeah we can close this

authors: andreogle bbenligiray


text:
feat(zookeeper): support zookeeper service discovery

support zookeeper service discovery

Codecov Report

Merging #30 (431cf16) into main (dfc787e) will decrease coverage by 6.14%.
The diff coverage is 6.00%.

@@            Coverage Diff             @@
##             main      #30      +/-   ##
==========================================
- Coverage   51.88%   45.74%   -6.15%
==========================================
  Files          22       24       +2
  Lines        1403     1598     +195
==========================================
+ Hits          728      731       +3
- Misses        615      806     +191
- Partials       60       61       +1

Impacted Files
Coverage Δ

internal/discoverer/zookeeper.go
1.15% <1.15%> (ø)

internal/conf/zookeeper.go
10.52% <10.52%> (ø)

internal/core/components/rewriter.go
56.89% <100.00%> (-5.61%)
:arrow_down:

internal/discoverer/discoverer.go
88.46% <100.00%> (ø)

internal/discoverer/nacos.go
75.57% <100.00%> (+0.09%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfc787e...431cf16. Read the comment docs.

@soulbird  I can't find you in the reviewers list, please help to review too.

hi, @spacewander @soulbird the above problem has been fixed, please review.

authors: codecov-commenter shuaijinchao


text:
How does one know if markdown is in Blueprint API format?

How does one know if markdown is in Blueprint API format? For example, say I have an incoming markdown file. If the file is in Bluepring API format I should render it as and API, otherwise I render it as a standard Markdown HTML. But how do I know if the markdown is in Blueprint API format?

@streamich We try to parse the incoming markdown file and generate the API Blueprint AST assuming that the given file is an API Blueprint file. You can probably look at the generated AST and guess if a file is an API Blueprint or not. But it is not always deterministic since the normal markdown file can use API Blueprint keywords. Other than that, there are no ways to identify a API Blueprint file.
Please remember that, every API Blueprint file is a markdown file where as not every markdown file is an API Blueprint file.

@streamich We try to parse the incoming markdown file and generate the API Blueprint AST assuming that the given file is an API Blueprint file. You can probably look at the generated AST and guess if a file is an API Blueprint or not. But it is not always deterministic since the normal markdown file can use API Blueprint keywords. Other than that, there are no ways to identify a API Blueprint file.
Please remember that, every API Blueprint file is a markdown file where as not every markdown file is an API Blueprint file.

But how do I know if the markdown is in Blueprint API format?

You can't really.  Similarly as you can't distinguish plain text from markdown. The only way is to get the metadata (e.g. Content-Type) with the file.
In the case of Content-Type or Accept HTTP headers the value for API Blueprint Markdown should be text/vnd.apiblueprint+markdown.

Note the FORMAT: 1A is Apiary-specific metadata and it is not required in API Blueprints outside of Apiary. In a future it will be removed from Apiary as well, so I wouldn't really rely on it.

May API blueprints should have their file extensions? Like minified JS files have .min.js, API blueprints could be something like .api.md.

We are using .apib at Apiary. But I like the .api.md as well – this allows you to benefit from GitHub rendering of markdown files. So while we are using apib I would say sticking with api.md is a good alternative.

To wrap up this issue – the correct approach is to look at the file media type – it should be text/vnd.apiblueprint+markdown for every API Blueprint version to-date.
Please reopen if a further clarification is needed.

authors: pksunkara streamich zdne

repo: apiaryio/api-blueprint

license: mit


text:
[Meta] OKHttp Client implementation

This is a parent ticket for the implementation of the OkHttpClient implementation. The action items are as follows;-

[x] #2
[x] #5
[x] Vet Naming and related inner documentation of classes
[x] Update ReadMe when Classes become stable
[x] #3
[x] Add CD workflow for package publishing

Closing this tracker issue as all sub-tasks are done.

repo: apimatic/okhttp-client-adapter


text:
🛑 OSC is down

In 6ccba6d, OSC (https://openstorage.xyz) was down:

HTTP code: 0
Response time: 0 ms

Resolved: OSC is back up in 4ddd5d3.

repo: apinter/OSC-mon


text:
Error trying to get request size using django middleware

Hi guys, english isn't my first language, so please excuse any mistakes.
I'm trying to use the last version of django middleware (0.6.1) i have the next error:
invalid literal for int() with base 10: ''
Request Method:	GET
Request URL:	http://localhost:8001/api/job/
Django Version:	5.0.3
Exception Type:	ValueError
Exception Value:
invalid literal for int() with base 10: ''
Exception Location:	/usr/local/lib/python3.10/site-packages/apitally/client/base.py, line 135, in add_request
Raised during:	job.views.JobList
Python Executable:	/usr/local/bin/python
Python Version:	3.10.13
Python Path:
['/app',
'/usr/local/lib/python310.zip',
'/usr/local/lib/python3.10',
'/usr/local/lib/python3.10/lib-dynload',
'/usr/local/lib/python3.10/site-packages']
It looks like that when you are trying to get the size of a request ( request.headers.get("Content-Length")) and the result is '' (an empty string) it impossible to make the convertion to an integer object in line 135 of /apitally/client/base.py file and there is not validation for an empty string in line 134 of /apitally/client/base.py file.
Right now, the only option i have is using the 0.3.4 of the library.

Thanks for reporting this! I will push a fix shortly.
Do you use the django.middleware.common.CommonMiddleware in your application? If not, you may want to add it after the ApitallyMiddleware as that is where the Content-Length header is being set in Django.

This is now fixed in the latest version v0.6.2.

Hi, yes, i was using django.middleware.common.CommonMiddleware but i changed the library version to the last one (v0.6.2) and it looks like everything is working fine, thank you for the fix.

authors: itssimon kevyder

repo: apitally/python-client


text:
The last column of the list of packages on a group's homepage is wrongly named "Tags"

Should be "Is User Command"

Fixed in 0.123.0

repo: aplteam/Tatin


text:
custom_infowindow_class in master

Hi i'm using master branch and want to customize infobox. Is any methods to use smth like custom_infowindow_class in old versions?
Thanks!

here is my solution https://github.com/apneadiving/Google-Maps-for-Rails/wiki/Change-handler-behavior

repo: apneadiving/Google-Maps-for-Rails


text:
Dockerod create instead of run

Hi everyone,
So I have this very simple code:
 docker.run('mouli', ['python', 'mouli.py'], function (err, data, container) {       console.log('done');   });
but when I do a docker ps, it seems that the container was created and not runned:
1a0297000245        mouli                         "python mouli.py"        7 minutes ago       Created                                            adoring_fermat , which is weird because when I do a docker run mouli python mouli.py, everything happens well...
My conf:

Node v6.6.0
Docker version 1.11.2, build b9f10c9

Thanks for your support

Are there any logs for your container?

Nope, but the bug fix itself, I'm not able to reproduce it ...

Is err null ?

Going to close this one then :)

authors: JarrodAMcEvers Wats0ns apocas kilianc

repo: apocas/dockerode


text:
Update service does not provide version query parameter

This is the response from the docker host
Error: (HTTP code 400) unexpected - invalid service version '': strconv.ParseUint: parsing "": invalid syntax

I'm working on a PR

My bad, it can actually be added to the opts Object


text:
chore(deps): update secops orb to v2.0.4

This PR contains the following updates:

Package
Type
Update
Change

secops
orb
patch
2.0.3 -> 2.0.4

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - "after 8am and before 4pm on tuesday" in timezone Etc/UTC.
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

This PR has been generated by Renovate Bot.

Renovate Ignore Notification
Because you closed this PR without merging, Renovate will ignore this update (2.0.4). You will get a PR once a newer version is released. To ignore this dependency forever, add it to the ignoreDeps array of your Renovate config.
If you accidentally closed this PR, or if you changed your mind: rename this PR to get a fresh replacement PR.

repo: apollographql/apollo-ios-pagination


text:
Publish extension to Open VSX

Thanks for this theme! love it!
Would it be possible to publish this extension to the Open VSX registry as well? I use VSCodium, so can't install it directly.
In the meantime, I've created: https://github.com/open-vsx/publish-extensions/pull/247/files
Thanks!

Hey @yorch, thanks! Glad you like it. 😄
I will look into publishing there. Looks pretty straightforward, it may just take me a bit of time to get to running through it. Stay tuned!

Hey @yorch, thanks! Glad you like it. 😄
I will look into publishing there. Looks pretty straightforward, it may just take me a bit of time to get to running through it. Stay tuned!

FYI, putting this on the backburner because in order to publish to the VSX registry I'll have to sign the VSX publisher agreement on behalf of Apollo, which will require legal review.

FYI, putting this on the backburner because in order to publish to the VSX registry I'll have to sign the VSX publisher agreement on behalf of Apollo, which will require legal review.

@jglovier thanks for the heads up! would be great once you guys can publish directly. Luckily, the theme is already available in OpenVSX with their automatic process (the PR I mentioned earlier).

@jglovier thanks for the heads up! would be great once you guys can publish directly. Luckily, the theme is already available in OpenVSX with their automatic process (the PR I mentioned earlier).

Thanks for adding it manually in the meantime. :bow: 💛

Thanks for adding it manually in the meantime. :bow: 💛

authors: jglovier yorch

repo: apollographql/apollo-midnight


text:
Use default resolver from GraphQL.js package

GraphQL.js now exports the default resolver: https://github.com/graphql/graphql-js/pull/527
We should replace our internal version: https://github.com/apollostack/graphql-tools/blob/40aa4be6d1b2b62e52c1c41d41830810aa02f895/src/schemaGenerator.ts#L488

I can replace it. Happy veterans day

BTW here's the real commit with the export, which is slightly different from the PR https://github.com/graphql/graphql-js/commit/c6a6388339b8c5af6b75ca14b6861497018bd19f

Is this essentially the same as #180?

That's right

Anyone who gets to this issue from my tweet refer to #180 instead!

authors: Anichale helfer stubailo

repo: apollostack/graphql-tools