text:
[SPARK-32564][SQL][TEST][3.0] Inject data statistics to simulate plan generation on actual TPCDS data

What changes were proposed in this pull request?

TPCDSQuerySuite currently computes plans with empty TPCDS tables, then checks if plans can be generated correctly. But, the generated plans can be different from actual ones because the input tables are empty (e.g., the plans always use broadcast-hash joins, but actual ones use sort-merge joins for larger tables). To mitigate the issue, this PR defines data statistics constants extracted from generated TPCDS data in TPCDSTableStats, then injects the statistics via spark.sessionState.catalog.alterTableStats when defining TPCDS tables in TPCDSQuerySuite.
Please see a link below about how to extract the table statistics:

https://gist.github.com/maropu/f553d32c323ee803d39e2f7fa0b5a8c3

For example, the generated plans of TPCDS q2 are different with/without this fix:
==== w/ this fix: q2 ====
== Physical Plan ==
* Sort (43)
+- Exchange (42)
   +- * Project (41)
      +- * SortMergeJoin Inner (40)
         :- * Sort (28)
         :  +- Exchange (27)
         :     +- * Project (26)
         :        +- * BroadcastHashJoin Inner BuildRight (25)
         :           :- * HashAggregate (19)
         :           :  +- Exchange (18)
         :           :     +- * HashAggregate (17)
         :           :        +- * Project (16)
         :           :           +- * BroadcastHashJoin Inner BuildRight (15)
         :           :              :- Union (9)
         :           :              :  :- * Project (4)
         :           :              :  :  +- * Filter (3)
         :           :              :  :     +- * ColumnarToRow (2)
         :           :              :  :        +- Scan parquet default.web_sales (1)
         :           :              :  +- * Project (8)
         :           :              :     +- * Filter (7)
         :           :              :        +- * ColumnarToRow (6)
         :           :              :           +- Scan parquet default.catalog_sales (5)
         :           :              +- BroadcastExchange (14)
         :           :                 +- * Project (13)
         :           :                    +- * Filter (12)
         :           :                       +- * ColumnarToRow (11)
         :           :                          +- Scan parquet default.date_dim (10)
         :           +- BroadcastExchange (24)
         :              +- * Project (23)
         :                 +- * Filter (22)
         :                    +- * ColumnarToRow (21)
         :                       +- Scan parquet default.date_dim (20)
         +- * Sort (39)
            +- Exchange (38)
               +- * Project (37)
                  +- * BroadcastHashJoin Inner BuildRight (36)
                     :- * HashAggregate (30)
                     :  +- ReusedExchange (29)
                     +- BroadcastExchange (35)
                        +- * Project (34)
                           +- * Filter (33)
                              +- * ColumnarToRow (32)
                                 +- Scan parquet default.date_dim (31)

==== w/o this fix: q2 ====
== Physical Plan ==
* Sort (40)
+- Exchange (39)
   +- * Project (38)
      +- * BroadcastHashJoin Inner BuildRight (37)
         :- * Project (26)
         :  +- * BroadcastHashJoin Inner BuildRight (25)
         :     :- * HashAggregate (19)
         :     :  +- Exchange (18)
         :     :     +- * HashAggregate (17)
         :     :        +- * Project (16)
         :     :           +- * BroadcastHashJoin Inner BuildRight (15)
         :     :              :- Union (9)
         :     :              :  :- * Project (4)
         :     :              :  :  +- * Filter (3)
         :     :              :  :     +- * ColumnarToRow (2)
         :     :              :  :        +- Scan parquet default.web_sales (1)
         :     :              :  +- * Project (8)
         :     :              :     +- * Filter (7)
         :     :              :        +- * ColumnarToRow (6)
         :     :              :           +- Scan parquet default.catalog_sales (5)
         :     :              +- BroadcastExchange (14)
         :     :                 +- * Project (13)
         :     :                    +- * Filter (12)
         :     :                       +- * ColumnarToRow (11)
         :     :                          +- Scan parquet default.date_dim (10)
         :     +- BroadcastExchange (24)
         :        +- * Project (23)
         :           +- * Filter (22)
         :              +- * ColumnarToRow (21)
         :                 +- Scan parquet default.date_dim (20)
         +- BroadcastExchange (36)
            +- * Project (35)
               +- * BroadcastHashJoin Inner BuildRight (34)
                  :- * HashAggregate (28)
                  :  +- ReusedExchange (27)
                  +- BroadcastExchange (33)
                     +- * Project (32)
                        +- * Filter (31)
                           +- * ColumnarToRow (30)
                              +- Scan parquet default.date_dim (29)

This comes from the @cloud-fan comment: https://github.com/apache/spark/pull/29270#issuecomment-666098964
This is the backport of #29384.
Why are the changes needed?

For better test coverage.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Existing tests.

Test build #127221 has started for PR 29390 at commit 750a632.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31842/
Test PASSed.

Test build #127221 has finished for PR 29390 at commit 750a632.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127221/
Test PASSed.

Thanks, @maropu . Merged to branch-3.0.

Thanks a lot, @dongjoon-hyun !

authors: AmplabJenkins SparkQA dongjoon-hyun maropu


text:
[SPARK-39156][SQL] Clean up the usage of ParquetLogRedirector in ParquetFileFormat.

What changes were proposed in this pull request?
SPARK-17993 introduce ParquetLogRedirector for Parquet version < 1.9, PARQUET-305 change to use slf4j instead of jul in Parquet 1.9,  Spark uses Parquet 1.12.2 now  and no longer relies on Parquet version 1.6 now , the ParquetLogRedirector is no longer needed, so this pr clean up the usage of ParquetLogRedirector in ParquetFileFormat.
Why are the changes needed?
Clean up the usage of ParquetLogRedirector in ParquetFileFormat.
Does this PR introduce any user-facing change?
No
How was this patch tested?

Pass GA
Manual test:

Build Spark client manually before and after this pr
Change parquet log4j level to debug:

logger.parquet1.name = org.apache.parquet
logger.parquet1.level = debug
logger.parquet2.name = parquet
logger.parquet2.level = debug

Try to read Parquet file write with 1.6 , for example sql/core/src/test/resources/test-data/dec-in-i32.parquet.

java -jar parquet-tools-1.10.1.jar meta /${basedir}/dec-in-i32.parquet
file:        file:/${basedir}/dec-in-i32.parquet
creator:     parquet-mr version 1.6.0
extra:       org.apache.spark.sql.parquet.row.metadata = {"type":"struct","fields":[{"name":"i32_dec","type":"decimal(5,2)","nullable":true,"metadata":{}}]}

file schema: spark_schema
--------------------------------------------------------------------------------
i32_dec:     OPTIONAL INT32 O:DECIMAL R:0 D:1

row group 1: RC:16 TS:102 OFFSET:4
--------------------------------------------------------------------------------
i32_dec:      INT32 GZIP DO:0 FPO:4 SZ:131/102/0.78 VC:16 ENC:RLE,PLAIN_DICTIONARY,BIT_PACKED ST:[no stats for this column]

spark.read.parquet("file://${basedir}/ptable/dec-in-i32.parquet").show()

The log contents before and after this pr are consistent, and there is no error log mentioned in SPARK-17993

Looks OK. Could you cross link the fix (JIRA) from Parquet side?

Should be PARQUET-305 in Parquet 1.9

hmm... @sunchao any other need changes?

Thanks! Merged to master.

thanks @huaxingao @sunchao

source: gharchive/pull-request

authors: LuciferYang huaxingao

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-36681][CORE][TESTS][FOLLOW-UP] Handle LinkageError when Snappy native library is not available in low Hadoop versions

What changes were proposed in this pull request?
This is a follow-up to https://github.com/apache/spark/pull/36136 to fix LinkageError handling in FileSuite to avoid test suite abort when Snappy native library is not available in low Hadoop versions:
23:16:22 FileSuite:
23:16:22 org.apache.spark.FileSuite *** ABORTED ***
23:16:22   java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: org.apache.spark.FileSuite
23:16:22   at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)
23:16:22   at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
23:16:22   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
23:16:22   at scala.collection.Iterator.foreach(Iterator.scala:941)
23:16:22   at scala.collection.Iterator.foreach$(Iterator.scala:941)
23:16:22   at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
23:16:22   at scala.collection.IterableLike.foreach(IterableLike.scala:74)
23:16:22   at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
23:16:22   at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
23:16:22   at scala.collection.TraversableLike.map(TraversableLike.scala:238)
23:16:22   ...
23:16:22   Cause: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
23:16:22   at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
23:16:22   at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
23:16:22   at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:136)
23:16:22   at org.apache.spark.FileSuite.$anonfun$new$12(FileSuite.scala:145)
23:16:22   at scala.util.Try$.apply(Try.scala:213)
23:16:22   at org.apache.spark.FileSuite.<init>(FileSuite.scala:141)
23:16:22   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
23:16:22   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
23:16:22   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
23:16:22   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

Scala's Try can handle only NonFatal throwables.
Why are the changes needed?
To make the tests robust.
Does this PR introduce any user-facing change?
Nope, this is test-only.
How was this patch tested?
Manual test.

cc @HyukjinKwon, @viirya, @dongjoon-hyun

Thanks all for the review.


text:
[SPARK-39748][SQL][FOLLOWUP] Add missing origin logical plan on DataFrame.checkpoint on building LogicalRDD

What changes were proposed in this pull request?
This PR adds missing origin logical plan on building LogicalRDD in DataFrame.checkpoint, via review comment https://github.com/apache/spark/pull/37161#discussion_r919204026.
Why are the changes needed?
This is missing spot on previous PR and @viirya helped to find out.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
N/A

cc. @viirya

lgtm

Thanks! Merging to master.

authors: HeartSaVioR viirya


text:
[SPARK-41790][SQL] Set TRANSFORM reader and writer's format correctly

What changes were proposed in this pull request?
We'll get wrong data when transform only specify reader or writer 's row format delimited, the reason is using the wrong format to feed/fetch data to/from running script now. we should set the format correctly.
Currently in Spark:
spark-sql> CREATE TABLE t1 (a string, b string);

spark-sql> INSERT OVERWRITE t1 VALUES("1", "2"), ("3", "4");

spark-sql> SELECT TRANSFORM(a, b)
         >   ROW FORMAT DELIMITED
         >   FIELDS TERMINATED BY ','
         >   USING 'cat'
         >   AS (c)
         > FROM t1;
c

spark-sql> SELECT TRANSFORM(a, b)
         >   USING 'cat'
         >   AS (c)
         >   ROW FORMAT DELIMITED
         >   FIELDS TERMINATED BY ','
         > FROM t1;
c
1    23    4

The same sql in hive:
hive> SELECT TRANSFORM(a, b)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY ','
    >   USING 'cat'
    >   AS (c)
    > FROM t1;
c
1,2
3,4

hive> SELECT TRANSFORM(a, b)
    >   USING 'cat'
    >   AS (c)
    >   ROW FORMAT DELIMITED
    >   FIELDS TERMINATED BY ','
    > FROM t1;
c
1    2
3    4

Why are the changes needed?
Fix transform writer format and reader format.
Does this PR introduce any user-facing change?
When we set transform's row format delimited in the sql, we may get the wrong data.
How was this patch tested?
New tests.

Can one of the admins verify this patch?

cc @AngersZhuuuu

Merged to master.

authors: AmplabJenkins HyukjinKwon mattshma


text:
[SPARK-41591][PYTHON][FOLLOW-UP] Remove gRPC version check for Distributor

What changes were proposed in this pull request?
Removing redundant check for whether GPUs exist on the driver node.
Why are the changes needed?
For slightly cleaner code. Could close PR if we don't need to merge it in.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
As long as normal tests work, we don't expect any other failures.

Yeah, let's close. I don't think this is an issue.

Thank you, @HyukjinKwon and @rithwik-db .

authors: HyukjinKwon dongjoon-hyun rithwik-db


text:
[SPARK-42779][SQL][FOLLOWUP] Allow V2 writes to indicate advisory shuffle partition size

What changes were proposed in this pull request?

This PR addresses non-blocking comments for PR #40421.
Why are the changes needed?

These changes are needed to make sure the new logic only applies in expected cases.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Existing tests.

cc @cloud-fan @dongjoon-hyun

thanks, merging to master!

Thank you, @cloud-fan !

Thanks, @dongjoon-hyun @cloud-fan!

authors: aokolnychyi cloud-fan dongjoon-hyun


text:
[SPARK-43228][SQL] Join keys also match PartitioningCollection in CoalesceBucketsInJoin

What changes were proposed in this pull request?
This PR updates CoalesceBucketsInJoin.satisfiesOutputPartitioning to support matching PartitioningCollection. A common case is that we add an alias on the join key. For example:
SELECT *
FROM   (SELECT /*+ BROADCAST(t3) */ t1.i AS t1i, t1.j AS t1j, t3.*
        FROM   t1 JOIN t3 ON t1.i = t3.i AND t1.j = t3.j) t
       JOIN t2 ON t.t1i = t2.i AND t.t1j = t2.j

The left side outputPartitioning is:
(hashpartitioning(t1i#41, t1j#42, 8) or hashpartitioning(i#46, t1j#42, 8) or hashpartitioning(t1i#41, j#47, 8) or hashpartitioning(i#46, j#47, 8))

Why are the changes needed?
Enhance CoalesceBucketsInJoin to support more cases.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.

cc @cloud-fan


text:
[SPARK-42945][CONNECT][FOLLOWUP] Disable JVM stack trace by default

What changes were proposed in this pull request?
This is a follow-up of #40575.
Disables JVM stack trace by default.
% ./bin/pyspark --remote local
...
>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> spark.sql('select 1/0').show()
...
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.ArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 8) ==
select 1/0
       ^^^

>>>
>>> spark.conf.set("spark.sql.pyspark.jvmStacktrace.enabled", True)
>>> spark.sql('select 1/0').show()
...
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.ArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 8) ==
select 1/0
       ^^^

JVM stacktrace:
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 8) ==
select 1/0
       ^^^

	at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:226)
	at org.apache.spark.sql.catalyst.expressions.DivModLike.eval(arithmetic.scala:674)
...

Why are the changes needed?
Currently JVM stack trace is enabled by default.
% ./bin/pyspark --remote local
...
>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> spark.sql('select 1/0').show()
...
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.ArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 8) ==
select 1/0
       ^^^

JVM stacktrace:
org.apache.spark.SparkArithmeticException: [DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 8) ==
select 1/0
       ^^^

	at org.apache.spark.sql.errors.QueryExecutionErrors$.divideByZeroError(QueryExecutionErrors.scala:226)
	at org.apache.spark.sql.catalyst.expressions.DivModLike.eval(arithmetic.scala:674)
...

Does this PR introduce any user-facing change?
Users won't see the JVM stack trace by default.
How was this patch tested?
Existing tests.

Merged to master. Thank you, @ueshin and @allisonwang-db .

authors: dongjoon-hyun ueshin


text:
[SPARK-48394][3.5][CORE] Cleanup mapIdToMapIndex on mapoutput unregister

This PR backports https://github.com/apache/spark/pull/46706 to branch 3.5.

What changes were proposed in this pull request?

This PR cleans up mapIdToMapIndex when the corresponding mapstatus is unregistered in three places:

removeMapOutput
removeOutputsByFilter
addMapOutput (old mapstatus overwritten)

Why are the changes needed?

There is only one valid mapstatus for the same mapIndex at the same time in Spark. mapIdToMapIndex should also follows the same rule to avoid chaos.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Unit tests.
Was this patch authored or co-authored using generative AI tooling?

No.

https://github.com/apache/spark/pull/46749 should fix the issue in the build in this case.
For now, you could rebase/force push and that should fix up the build

It seems I mistakenly pushed the branch to apache repo rather than my own repo. Will remove that branch after the PR merged.

@Ngone51 the build won't trigger if the branch is in apache repo. Let's just open a new PR with your forked repository.

@HyukjinKwon Oh, I see. Thanks for the reminder.

FYI created a new PR (https://github.com/apache/spark/pull/46768) to replace this one.


text:
[MINOR][SQL][TESTS] Changes the test:runMain in the code comments to Test/runMain

What changes were proposed in this pull request?
This PR only changes the test:runMain description related to run command in the code comments to Test/runMain.
Why are the changes needed?
When we use the execution command in the code comments, we will see the following compilation warning:
build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.TopKBenchmark"

[warn] sbt 0.13 shell syntax is deprecated; use slash syntax instead: sql / Test / runMain

The relevant comments should be updated to eliminate the compilation warnings when run the command.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manually run the test using the updated command and check that the corresponding compilation warning is no longer present.
Was this patch authored or co-authored using generative AI tooling?
No

Please add [TEST] to the PR title

Please add [TEST] to the PR title

done

Merged into master. Thanks @yaooqinn

authors: LuciferYang yaooqinn


text:
[SPARK-50649] Fix inconsistencies with casting between different collations

What changes were proposed in this pull request?
Fixing the inconsistent behavior of casts between different collations. Currently, we are allowed to do casts between them in dataframe API but not in the SQL API. I propose allowing casts in SQL as well (we are already allowing them for complex types anyways).
Also, this means changing the behavior or CAST(x AS STRING) which was previously not altering the collation of x, and will now change it to the default collation.
Why are the changes needed?
To make collation casts between the dataframe and SQL api consistent.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added new unit tests for the dataframe API which we didn't have before and also updated the existing tests for the SQL API to match the new behavior.
Was this patch authored or co-authored using generative AI tooling?
No.

@cloud-fan please take a look when you can

The Spark Connect test failure is unrelated and flaky, I'm merging it to master, thanks!

authors: cloud-fan stefankandic


text:
[SPARK-7069][SQL] Rename NativeType -> AtomicType.

Also renamed JvmType to InternalType.

Test build #30817 has started for   PR 5651 at commit cbd4028.

LGTM pending Jenkins.

Test build #30817 has finished for   PR 5651 at commit cbd4028.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

protected[sql] abstract class AtomicType extends DataType

abstract class NumericType extends AtomicType

 class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]

  class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])

 class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]

  class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])

 class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]

  class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30817/
Test PASSed.

authors: AmplabJenkins SparkQA liancheng rxin


text:
[SPARK-10482] [ML] Add Python interface for ml.CountVectorizer

jira: https://issues.apache.org/jira/browse/SPARK-10482
Add Python interface for feature transformer: ml.CountVectorizer

Merged build triggered.

Merged build started.

Test build #42112 has started for   PR 8650 at commit 0f1fa34.

Test build #42112 has finished for   PR 8650 at commit 0f1fa34.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42112/
Test FAILed.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #42122 has started for   PR 8650 at commit d22ba5a.

Test build #42122 has finished for   PR 8650 at commit d22ba5a.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42122/
Test FAILed.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #42125 has started for   PR 8650 at commit dd0e933.

Test build #42125 has finished for   PR 8650 at commit dd0e933.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class CountVectorizer(JavaEstimator, HasInputCol, HasOutputCol):
class CountVectorizerModel(JavaModel):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42125/
Test PASSed.

LGTM except some minor issues

This seems to do the same work as the outstanding PR https://github.com/apache/spark/pull/8561

@holdenk Yes, I just noticed it. Could you merge some changes in this PR into yours? I think the doctest from @hhbyyh is better and the default values are specified correctly in this PR. I will make a pass after.
@hhbyyh Since this duplicates #8561, do you mind closing this PR? You can check opening PRs at https://spark-prs.appspot.com/#mllib.

Ok, I'll merge in the doc tests.

@mengxr Sorry for the extra effort during review.

authors: AmplabJenkins SparkQA hhbyyh holdenk mengxr


text:
[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites

This patch fixes an issue where the spark.sql.TungstenAggregate.testFallbackStartsAt SQLConf setting was not properly reset / cleared at the end of TungstenAggregationQueryWithControlledFallbackSuite. This ended up causing test failures in HiveCompatibilitySuite in Maven builds by causing spilling to occur way too frequently.
This configuration leak was inadvertently introduced during test cleanup in #9618.

Test build #46402 has started for PR 9857 at commit ffe29f7.

The failing HiveCompatibilitySuite test, mapjoin_mapjoin, has passed in the Maven pull request builder, and the modified TungstenAggregationQueryWithControlledFallbackSuite also passed tests, so I'm going to merge this now so that the overnight Maven builds have the opportunity to exhibit new test failures now that this one has been fixed.

Test build #46402 has finished for PR 9857 at commit ffe29f7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46402/
Test FAILed.

authors: AmplabJenkins JoshRosen SparkQA


text:
[STORM-848] Shade external dependencies

Shading following dependencies:
org.apache.httpcomponents:http*
org.apache.zookeeper:zookeeper
org.apache.curator:*
com.twitter:carbonite
com.twitter:chill-java
org.objenesis:objenesis
org.tukaani:xz
org.yaml:snakeyaml
org.jgrapht:jgrapht-core
commons-httpclient:commons-httpclient
org.apache.commons:commons-compress
org.apache.commons:commons-exec
commons-io:commons-io
commons-codec:commons-codec
commons-fileupload:commons-fileupload
commons-lang:commons-lang
com.googlecode.json-simple:json-simple
org.clojure:math.numeric-tower
org.clojure:tools.cli
org.clojure:tools.macro
joda-time:joda-time

Can we also shade jetty? Also why there are new dependency on some connectors.

@Parth-Brahmbhatt Primarily, anything that's getting directly called from withing clojure is hard to shade. Hence (disruptor, jetty..) etc are not shaded. As such newer version of jetty are self-shaded ( the packages are different.). But it would be a much bigger change to accomplish.
As the storm-core is shading these dependencies, they need to be called out explicitly by connectors now if they implicitly depended on storm-core so far.

I agree with @Parth-Brahmbhatt. I think for components in "/external" we should use the use the shaded classes in storm-core instead of manually adding the dependency. Doing so is just a matter of changing import statements.

@kishorvpatil what issues did you run into when you shaded Jetty?

@ptgoetz my problem with that is the distribution mechanism is different for external from what it is for storm proper.  Say I create a topology jar with storm-hdfs in it.  At some point in the future storm upgrades the version of snake.yaml and it is a non-backwards compatible change.  Now my topology will not run on a newer version of storm.

@revans2 Good point.

@ptgoetz  I get following exceptions when I try to shade Jetty
Compiling storm.starter.clj.word-count to /home/kpatil/commun/incubator-storm/examples/storm-starter/target/classes
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes, compiling:(word_count.clj:16:1)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3558)
	at clojure.lang.Compiler.compile1(Compiler.java:7226)
	at clojure.lang.Compiler.compile1(Compiler.java:7216)
	at clojure.lang.Compiler.compile(Compiler.java:7292)
	at clojure.lang.RT.compile(RT.java:398)
	at clojure.lang.RT.load(RT.java:438)
	at clojure.lang.RT.load(RT.java:411)
	at clojure.core$load$fn__5066.invoke(core.clj:5641)
	at clojure.core$load.doInvoke(core.clj:5640)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invoke(core.clj:5446)
	at clojure.core$compile$fn__5071.invoke(core.clj:5652)
	at clojure.core$compile.invoke(core.clj:5651)
	at clojure.lang.Var.invoke(Var.java:379)
	at clojure.lang.Compile.main(Compile.java:81)
Caused by: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
	at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
	at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
	at java.util.jar.JarVerifier.processEntry(JarVerifier.java:317)
	at java.util.jar.JarVerifier.update(JarVerifier.java:228)
	at java.util.jar.JarFile.initializeVerifier(JarFile.java:348)
	at java.util.jar.JarFile.getInputStream(JarFile.java:415)
	at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
	at sun.misc.Resource.cachedInputStream(Resource.java:77)
	at sun.misc.Resource.getByteBuffer(Resource.java:160)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:190)
	at storm.starter.clj.word_count$loading__4958__auto__.invoke(word_count.clj:16)
	at clojure.lang.AFn.applyToHelper(AFn.java:152)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)

can you try adding following to the shade-plugin part. See http://maven.apache.org/plugins/maven-shade-plugin/examples/includes-excludes.html for a full example.

*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA

@kishorvpatil I can't reproduce it. What does your shade config for jetty look like?
Here are the sections I added to test:
The include:
<include>org.eclipse.jetty:*</include>

The relocation:
                     <pattern>org.eclipse.jetty</pattern>
                            <shadedPattern>org.apache.storm.jetty</shadedPattern>
                        </relocation>

I left out the filter @Parth-Brahmbhatt mentioned, expecting to get the error you did, but everything succeeded.

I tried What @Parth-Brahmbhatt  suggested to successfully build it. Testing it now.

Thank you @ptgoetz. The changes worked. And Jetty is shaded now.

Thanks @kishorvpatil!
+1

+1.

+1

With this patch, there are a lot of ClassNotFoundException when nimbus or ui load Plugins such as AutoHdfs, what can we do to fix this.

@tedxia I responded on your pull request #600

@revans2 thanks for your replay.

authors: Parth-Brahmbhatt kishorvpatil ptgoetz revans2 tedxia

repo: apache/storm


text:
[STORM-949] On the topology summary UI page, added Elapsed time since error column.

Current on the topology summary UI page and tells you the last error and highlights it red if it happened within the last 30 min.  However, I think its useful if there was a column that told you the time that has elapsed since the most recent error occurred.  I found this useful for monitoring the well being of my storm cluster.

Not sure why travis is failing again.  I just built and ran all tests on my local machine and everything was fine

Hi, there's a storm-hive compilation issue.
Your patch modifies html pages, so you don't need to worry about build. :)
I'll take a look when I have some time. Thanks!

@jerrypeng
I'd like to see applied screenshot first before taking a detail look. Could you post it to comment?

fixed the formating issues

I'm more in favor of putting a time/date here than an elapsed time.
The actual description for STORM-949 is "On the topology summary UI page, last shown error should have the time and date".

Yes but the exact time and date can be found if you drill down to the component.  Having the elapsed time since time I feel like will tell administrators in a finer grain when the error happened

@HeartSaVioR can you take a look at my pull request again

@jerrypeng
I'm with @knusbaum. PR is a bit different from JIRA title, and I'm also more in favor of putting time/date.
How about gathering consensus about this feature from dev mailing list and reflect feedback?

Just modified the UI to have error time shown as a time and date.  If you hover over that time a tooltip will pop up displaying the elapsed time.

Modified version of UI is what I and @knusbaum , @d2r suggested, and it seems that there're no other opinions now.
Build failure is not related to.
So I'm +1.

authors: HeartSaVioR jerrypeng knusbaum


text:
[Explore]'Is temporal' checkbox should take effect

Current behavior:
Only columns that have 'Is Temporal' box checked in the Edit dataset  shows in Time Column select dropdown which is the right behavior ✅
What need to be done:

ICON CHANGE - When user uncheck  'Is Temporal'  on a column that was detected as Time, column icon should change to "?" or "#", vise versa to use the 🕑 icon
if the column was previous selected as time column, time range where clause should be removed automatically from the query
related project
[explore]Can't remove unnecessary Datetime column from time filter

https://user-images.githubusercontent.com/67837651/119166999-2a7ed700-ba14-11eb-9e26-a75d668ef8ec.mov
other related project:
[Explore]Search data panel column by key words
Drag and Drop
Not able to sort column in Edit dataset modal is super annoying.. 🤣

@geido

another related issue
when clicking "SYNC COLUMNS FROM SOURCE", Is temporal is not detected

source: gharchive/issue

repo: apache/superset


text:
Custom Color Theming Not working for embedded chart

While adding custom color theme for dashboard using below (where testpart23456789,Info,due,beforeTime,High are label names), the colors are getting applied only when charts are seen in the dashboard or during dashboard embed.
Issue we are facing is that when chart is explored or embedded, these label colors are getting overridden by default theming.
Also, while exploring chart, these label colors appear when "Superset Colors" theme is chosen but after saving, again the label colors get overridden by default theming.
"label_colors": {
"testpart23456789": "#FFFF00",
"Info": "#FFFF00",
"due": "#8B0000",
"beforeTime": "#008000",
"High": "#8B0000"
}
We got reference for above from FAQ- https://superset.apache.org/docs/frequently-asked-questions ( Is there a way to force the use specific colors?)
Expected results
While chart embed or explore, the label colors should appear as provided i.e should follow below:
"label_colors": {
"testpart23456789": "#FFFF00",
"Info": "#FFFF00",
"due": "#8B0000",
"beforeTime": "#008000",
"High": "#8B0000"
}
Actual results
While adding custom color theme for dashboard using below (where testpart23456789,Info,due,beforeTime,High are label names), the colors are getting applied only when charts are seen in the dashboard or during dashboard embed.
Issue we are facing is that when chart is explored or embedded, these label colors are getting overridden by default theming.
Also, while exploring chart, these label colors appear when "Superset Colors" theme is chosen but after saving, again the label colors get overridden by default theming.
Screenshots

How to reproduce the bug

Edit dashboard to add above provided config in Advanced Dashboard properties and save.
Open dashboard and explore any chart under it.
You will see that the colors provided are not being followed for labels.

Environment
(please complete the following information):

superset version: 1.1.0
python version: 3.8.5
node.js version: v14.17.0

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

This should be solved in 1.4


text:
Unable to display data table in sqlite?

I have successfully installed superset in docker mode, and the example data is also loaded successfully. When I add local sqlite for testing, it can't display the specific table name in sql lab.

superset version: 0.28.0
python version: 2.7.5
node.js version: 10.16.3
npm version: 6.9.0

Checklist

[ X] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ X] I have reproduced the issue with at least the latest released version of superset.
[ X] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
no any table display in main schema, when i exec select sql: sqlite error: no such table: xxx

add datasource:
Test Connection: OK
expose in SQL Lab: selected
No data table displayed in bottom !

View In SQL Lab Editor:
schema: main(only)
see table schema(0 in main)

View database by sqlite3 in Centos7:  display table

This issue refers to an old version of Superset (also deprecated version of Python); please try the most recent official release (0.34) and reopen if the issue persists.

I installed the superset as per the tutorial via Docker Compose on Windows 11 and added a SQLite database using the PREVENT_UNSAFE_DB_CONNECTIONS = False directive
The connection test is successful.
The database is fully functional and accessible.
The superset version is 0.0.0dev (as stated in the About section of the Settings menu).
But the same thing is happening to me:
View In SQL Lab Editor:

schema: main (only)
see table schema (0 in main)

I installed the superset as per the tutorial via Docker Compose on Windows 11 and added a SQLite database using the PREVENT_UNSAFE_DB_CONNECTIONS = False directive
The connection test is successful.
The database is fully functional and accessible.
The superset version is 0.0.0dev (as stated in the About section of the Settings menu).
But the same thing is happening to me:
View In SQL Lab Editor:

schema: main (only)
see table schema (0 in main)

Can we repoen this?
I cloned the repo just now and got it running  with docker compose and followed this comment https://github.com/apache/superset/issues/9748#issuecomment-1124323169 to get the sqlite db added but I have the same experience, adding the sqlite db file via the path copied into superset_home works with the driver connection string properly formatted, but adding a dataset from the sqlite db results in a single schema main and no tables found.

exact same experience just now as capttrousers.  Brand new install, managed to get sqlite db to connect, but main only and no data :(

I just installed superset using manual steps and have it running. I added a sqlite3 db file under database connections , where it showed the connection to be ok with unsafe flag set to FALSE. I also only see main as schema under the newly attached database and don't see the table which exists when I query the db file via sqlite3 commain line.

Infra:
SQLite 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da
zlib version 1.2.13
gcc-11.2.0
Loaded your LOCAL configuration at [/home/vibhu/src/talkAItive/superset/superset_config.py]
Python 3.10.13
Flask 2.2.5
Werkzeug 2.3.8
Help please.

the same problem persists even as of May 2024. A solution please?

authors: LoveMyBaby capttrousers everton3x hiwaveSupport mechgt sammigachuhi villebro


text:
How to change Big Number CSS to be like that?

How to configure to make big number like that?
Example:

@syazwan0913 you'll probably have to add your own component for this. Superset abstracts the common components to their plugins and exposes them as individual packages. https://github.com/apache-superset/superset-ui-plugins/blob/master/packages/.
If you take a look at the big number chart imported via the MainPresets.js file. You will find it does use a mainColor variable (ie the colorPicker) in the linear gradient.
The source code is pretty straightforward if you want to read it here: https://github.com/apache-superset/superset-ui-plugins/blob/484d63993b81d593183f1f1a2b8f9d91aeef310f/packages/superset-ui-legacy-preset-chart-big-number/src/BigNumber/BigNumber.jsx#L207
As you can see it's tightly coupled with the AreaSeries chart from @data-ui/xy-chart. You might want a new component or a new render section here that just takes a bg image. Probably making an issue request / PR and seeing if you can add it to the ui-plugins would be best.
I'd like that feature too :)

@timurista i see. thanks for your guide.

@syazwan0913 You could, in theory, do this with CSS. When you "Edit Dashboard" there's an option to add CSS. If the number and sequence of these Big Number components on your dashboard is static, you can use CSS nth-of-type trickery. I made a quick example to illustrate the case:

Syles applied here are as follows.

Set the background color of each instance.
.superset-legacy-chart-big-number:nth-of-type(1){  background: orange; }

Put the subheader where you want it (you can set the width so it wraps, etc.)
.superset-legacy-chart-big-number .subheader-line {  text-align: left;  position: absolute;  bottom: 10px;  left: 10px; }

Add a CSS pseudo element, and give it the icon you want as a background.
.superset-legacy-chart-big-number:nth-of-type(1)::after {  content: '';  display: block;  height: 60px;  width: 60px;  background: url(https://image.flaticon.com/icons/png/512/121/121901.png);  background-size: contain;  position: absolute;  bottom: 20px;  right: 20px; }

That's super hacky, but might get you the result you want without having to make new components.

@rusackas That is great. I will try it out. Thanks

@rusackas What version of superset do you have installed?, I have tried this hack on v.0.28.1 and did not work.
Thanks

@rusackas What version of superset do you have installed?, I have tried this hack on v.0.28.1 and did not work.
Thanks

I'm not sure what I was running at the time, but I usually run the latest code on master. Not sure where you added the CSS, but just for clarity, do the following (which I did on the example Baby Names dashboard):

click "Edit Dashboard"
click the dropdown arrow at the far right next to "Switch to view mode", and select "Edit CSS" from the dropdown menu.
Paste in this block of CSS:

.superset-legacy-chart-big-number:nth-of-type(1){
    background: orange;
}
.superset-legacy-chart-big-number .subheader-line {
    text-align: left;
    position: absolute;
    bottom: 10px;
    left: 10px;
}
.superset-legacy-chart-big-number:nth-of-type(1)::after {
    content: '';
    display: block;
    height: 60px;
    width: 60px;
    background: url(https://image.flaticon.com/icons/png/512/121/121901.png);
    background-size: contain;
    position: absolute;
    bottom: 20px;
    right: 20px;
}

You should see the result instantly, but you can close the modal/overlay, and click "Switch to view mode" to finish editing.

The result should look like so:

@muneneg  Another hacky way I found out is using the chart-id
#chart-id-225{background: green;}
     #chart-id-226{background: orange;}
      #chart-id-227{background: red;}

@B-Cheye How did you manage to make the entire box coloured. With the above CSS you have shared only the chart area will get coloured but not the dashboard-component.  Using your code above results in the image shown.

@stevensuting If you want the whole dashboard-component color to change then you will need to target the whole dashboard div

@B-Cheye How do you do that when this is how the CSS is ordered?

Could you share your CSS snippet?

@stevensuting it looks like @B-Cheye 's solution references the id attribute on the same line I'd annotated with "But it's here..." in the screenshot. So I don't think it is coloring the entire chart wrapper on the dash, but just the chart area itself. You could get hacky with nth-child/nth-of-type CSS selectors on the dashboard to color the whole wrapper, if your layout is fairly stable.

So I am asking in 2024 with v 3.1.1. how do we do this with CSS?  I particularly want my value to be centered

authors: B-Cheye Nikomahal muneneg rusackas stevensuting syazshafei syazwan0913 timurista


text:
feat: more SIP-40 errors

SUMMARY

Make more SQL Lab error message SIP-40 compliant.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Hi @betodealmeida this PR cause Sql Lab can not handle some error message correctly:

I have reverted this from airbnb's release branch, but you probably have to fix it in open source master branch.

authors: betodealmeida graceguo-supercat


text:
fix(deck.gl): update view state on property changes (#17720)

SUMMARY
Updates viewport state of deck.gl on filter updates
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before (screeshot 1 & 2):

After:
https://user-images.githubusercontent.com/2187389/146766468-a031dad9-b4d1-48ff-b307-dbfa2dd79417.mov
TESTING INSTRUCTIONS

Create a Dashboard containing a deck.gl polygon chart (auto zoom = True) with a column included in filter
Open the dahsboard => the map is zoomed according to show features (screenshot 1)
Change native filter value
The map is now zoomed to the new bounding box (see video)

ADDITIONAL INFORMATION

[x] Has associated issue: fixes #17720
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@hbruch can you rebase this? I believe there may have been some CI issues when you pushed these last changes

@hbruch sorry about this, but there's yet some more flakiness in our CI pipeline: #17918 . Bear with us while we get it sorted..

@villebro Is there anything I could do to have this merged? Thx

authors: hbruch villebro


text:
Code to add support for TOMEE-542

Signed-off-by: Doychin Bondzhev doychin@dsoft-bg.com

@doychin Thanks for the PR
It looks good to me.
Even if it's a small change, I think it should be visible on release notes.
Do you mind creating a JIRA ticket and updating the ticket title with it?

This looks great, and thank you for the PR! I'm just going to run a build with this on a Windows box and get it merged in for you.

authors: doychin jeanouii jgallimore

repo: apache/tomee


text:
Can mark an empty ip address as service when perform /POST servers API

I'm submitting a ...

[x] bug report
[ ] new feature / enhancement request
[ ] improvement request (usability, performance, tech debt, etc.)
[ ] other

Traffic Control components affected ...

[ ] CDN in a Box
[ ] Documentation
[ ] Grove
[ ] Traffic Control Client
[ ] Traffic Monitor
[x] Traffic Ops
[ ] Traffic Ops ORT
[ ] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[ ] unknown

Current behavior:

User can create a servers with empty IP address and can mark it as service.
For example using this payload:
{
"cachegroupId":   3,
"cachegroup":     "infrastructure",
"cdnId":          2,
"cdnName":       "testCDN",
"domainName":     "test.net",
"hostName":       "testingServer",
"interfaceMtu":   1500,
"interfaceName":  "eth0",
"ipAddress":      "",
"ipGateway":      "0.0.0.2",
"ipNetmask":      "255.255.255.0",
"ip6Address":     "0:0:0:0:0:0:0:1",
"ip6Gateway":     "::1",
"physLocationId": 1,
"physLocation":   "Augusta-ME",
"profileId":      34,
"profile":         "ATS_Edge_MKGA",
"statusId":       2,
"typeId":         11,
"updPending":     false,
"ipIsService":    true,
"ip6IsService":   false

}

The server will return 200 OK
Expected / new behavior:

Should return 400 with message like "an empty IP or IPv6 address cannot be marked as a service address"
Minimal reproduction of the problem with instructions:

Anything else:

There's also another problem with that. You said it returns 200 OK, but ignoring the fact that you made an empty IP a service address, it also, therefore, doesn't have any service addresses. It should have at least rejected it on that basis.

@ocket8888 Yeah, you right about that. I just check and server still create without ip and ip6

@ocket8888 Also even if user input the ipv6 and ip address in. The TP still clean out ip of the servers.

repo: apache/trafficcontrol


text:
BACKPORT: fixes #3223. updated type of SteeringTargetNullable.Value to JSONIntStr

(cherry picked from commit 3e11da43d31ce8e4469e033d633a47976a3f783b)
Which issue is fixed by this PR? If not related to an existing issue, what does this PR do?
Fixes #3223
Can't update or create steering target in 3.0.0 or 3.0.1
Which TC components are affected by this PR?

[ ] Documentation
[ ] Grove
[ ] Traffic Analytics
[ ] Traffic Monitor
[x] Traffic Ops
[ ] Traffic Ops ORT
[x] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[ ] Other _________

What is the best way to verify this PR? Please include manual steps or automated tests.
(If no tests are part of this PR, please provide explanation as to why no tests are included.)
Check all that apply

[ ] This PR includes tests
[ ] This PR includes documentation updates
[ ] This PR includes an update to CHANGELOG.md
[ ] This PR includes all required license headers
[ ] This PR includes a database migration (ensure that migration sequence is correct)
[ ] This PR fixes a serious security flaw. Read more: www.apache.org/security

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3402/
Test PASSed.

authors: asfgit smalenfant


text:
Rewrite current stats from Perl to Golang

What does this PR (Pull Request) do?

[x] This PR fixes #3853

This PR rewrites current_stats from Perl to Golang and adds API documentation for it.
Currently no tests as we do not have Traffic Stats available for TO Api tests.
Which Traffic Control components are affected by this PR?

Documentation
Traffic Golang Control Client
Traffic Ops

What is the best way to verify this PR?

Build traffic ops with PR code and then hit /current_stats, it should return the same data as perl implementation
If this is a bug fix, what versions of Traffic Control are affected?

The following criteria are ALL met by this PR

[x] This PR includes tests OR I have explained why tests are unnecessary
[x] This PR includes documentation OR I have explained why documentation is unnecessary
[x] This PR includes an update to CHANGELOG.md OR such an update is not necessary
[x] This PR includes any and all required license headers
[x] This PR ensures that database migration sequence is correct OR this PR does not include a database migration
[x] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

Additional Information

Still need to look into why Perl implementation did https://github.com/apache/trafficcontrol/blob/master/traffic_ops/app/lib/Utils/Helper.pm#L39 on each cdn name for the query and if that escaping is handled for us in the Golang client

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4780/

So then is a work in progress or ready for review?

@ocket8888 this is ready for review. The perl code did replace on ,',",\n in the CDN names prior to going to influxdb but all are blocked from even being apart of a CDN name in the first place -> https://github.com/apache/trafficcontrol/blob/master/traffic_ops/traffic_ops_golang/cdn/cdns.go#L112

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4808/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4847/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4848/

authors: asf-ci mhoppa ocket8888


text:
Video streaming issue on  traffic server

Hi ,
We are using traffic server as video streaming CDN. We have 2 server each 1,4TB SSD and 10G network interface. During peak hour video stream hard to watch and buffering in client side. Server's load average and Memory are higher than normal state. But streaming bandwidth is around 3Gbps on all 3 server total. We wondering that we have potential only 3Gbps bandwidth on streaming but it has high load on traffic servers. Does those server can handle this load?
load average 106.94, 93.30, 88.32 as 32 core CPU
memory usage total 62GB used 62GB
established tcp connection is around 1200 on each server
traffic server's error log is increasing as following
20201115.16h42m02s CONNECT:[0] could not connect [CONNECTION_CLOSED] to 127.0.0.1 for 'http://localhost/vod/encrypt/prod/8a01918b72167aad01722a8007db243e/8a01918b72167aad01722a8007db243e_1500_2/media-88320000.mp4'
20201115.16h42m02s CONNECT:[0] could not connect [CONNECTION_CLOSED] to 127.0.0.1 for 'http://localhost/vod/encrypt/prod/8a01918b752cbbb7017577a9a1fc05ba/8a01918b752cbbb7017577a9a1fc05ba_1500/media-46260040.mp4'
20201115.16h42m02s CONNECT:[0] could not connect [CONNECTION_CLOSED] to 127.0.0.1 for 'http://localhost/vod/encrypt/prod/8a01918b752cbbb7017549e7cc523b37/8a01918b752cbbb7017549e7cc523b37_1000_2/media-220416000.mp4'
20201115.16h42m02s CONNECT:[0] could not connect [CONNECTION_CLOSED] to 127.0.0.1 for 'http://localhost/vod/encrypt/prod/8a01918b69817ebf016a951962ab7ccc/8a01918b69817ebf016a951962ab7ccc_1500/media-4513608.mp4'
Best Regards,

how about disk util,and how much MB every disk,if each sever around 1200 on each server,it is abort 1.2G,the disk may be over used

loadavg is a very broad metric, which includes CPU, memory wait, disk wait, disk usage, and potentially other things.
Can you look at specific metrics on your system, and see what specific things have high load? Is it just CPU? Memory? Disk io_wait? All of the above?
ATS will use as much memory as you tell it to. You can allocate ramdisks and give them to ATS as block devices. Each disk given to ATS also has a memory cache in front of it, the size of which is configurable.
See:
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/records.config.en.html#ram-cache
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/storage.config.en.html
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/volume.config.en.html
ATS does have some known memory leaks, but they're generally pretty small. It shouldn't use much more memory than what you allocated for storage and ram_cache, and the memory shouldn't grow much over time.
Many people run ATS in production with bandwidth much higher than 3Gbps. My company has caches doing in excess of 20Gbps. If you're having trouble achieving those speeds, another possibility is Linux Kernel Parameters. It's common to have to do a lot of tuning of Linux Kernel Parameters to achieve high performance. Though I wouldn't expect a great deal of tuning to be necessary under 10Gbps.
I assume this is somewhat recent hardware, with decent CPUs? We do have some Prod servers that struggle to exceed 10Gbps, from underpowered CPUs with few PCI lanes. Platforms with too few PCI lanes can also cause network bottlenecks like that.
If your high load average is being caused by disk io_wait, are you certain your SSDs are fast? Some SSD brands have poor performance. It may be worth testing their sequential and random speeds, just to be sure that isn't the problem.

traffic server's error log is increasing as following
20201115.16h42m02s CONNECT:[0] could not connect [CONNECTION_CLOSED] to 127.0.0.1 for 'http://localhost/vod/encrypt/prod/8a01918b72167aad01722a8007db243e/8a01918b72167aad01722a8007db243e_1500_2/media-88320000.mp4'

I'm not sure I understand. Your initial question is about bandwidth bottlenecks and high loadavg, but this looks like an error? This looks like an origin (on localhost?) is misconfigured, or unable to handle the requests or load?
Are you saying you see a lot of these errors as you approach 3Gbps? That sounds like the Origin server isn't able to handle the load, that the problem is with the Origin, not ATS. Can you verify your Origin itself is capable of the request load?
Are these requests mostly Cache Hits or Misses? For a CDN, ATS should be caching I assume. Is the full traffic going to the Origin? Could that be causing the problem? Maybe the Origin can't handle the full 3Gbps, because everything is a Cache Miss, and you need to set Cache-Control to make ATS cache the content so the Origin can handle it.
Certain SSL Certificates can also cause high CPU usage, especially RSA. Are you using HTTPS?
In short, there are a huge number of factors that can cause bottlenecks like you're seeing. You'll have to narrow it down further, and inspect your hardware usage to figure out what the bottleneck is, and how to fix it. But ATS can definitely do +20Gbps, potentially even 100Gbps, and many large corporations are doing so in Production.

Please try to reproduce this in ATS 9.1, and reopen this issue if you still see it.

repo: apache/trafficserver


text:
swoc: install swoc_ip_util.h

Adding swoc_ip_util.h to the list of header sources so that it will be installed in the expected swoc include location.

[approve ci autest]

Cherry-picked to v10.0.x

authors: bneradt cmcfarlen


text:
Remove multiprocessing.Queue.qsize() from traffic_replay

Because this raise NotImplementedError on Unix platforms like Mac OS X.
Details in below.
http://python.readthedocs.io/en/stable/library/multiprocessing.html#multiprocessing.Queue.qsize

[ci approve autest]

[approve ci autest]


text:
TS-4396: fix number_of_redirections off-by-one

please test

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/406/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/300/ for details.

FreeBSD build failed! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/412/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/306/ for details.

[approve ci]

Closing since there is a new PR for this #2092

authors: PSUdaemon atsci bryancall mingzym


text:
[ZEPPELIN-1076] Set hbase.client.retries.number for JDBC

What is this PR for?
If a user has "org.apache.phoenix:phoenix-core:4.x.x" jar added as a dependency in JDBC interpreter, and for some reason phoenix was not accessible or not properly configured; then the phoenix tries to for 35 times (which is default for hbase.client.retries.number) and each retires is 8 second apart, before it finally fails.
What type of PR is it?
[Bug Fix]
Todos

[x] - Set phoenix.hbase.client.retries.number for JDBC

What is the Jira issue?

ZEPPELIN-1076

How should this be tested?
In JDBC interpreter add org.apache.phoenix:phoenix-core:4.4.0-HBase-1.0 as dependency, but don't configure phoenix setting.
Then try to run any sql query with any of the configured JDBC driver (like show tables)

Without this it will take slightly more than about 5 mins
With this it should fetch result sooner (in less than a minute)

Screenshots (if appropriate)
Questions:

Does the licenses files need update? n/a
Is there breaking changes for older versions? n/a
Does this needs documentation? n/a

LGTM. That property hbase.client.retries.number will be passed to phoenix jdbc, right?

Yes, phoenix.hbase.client.retries.number, and have tested with and without this string.

Merging this if no more discussion.

authors: jongyoul prabhjyotsingh

repo: apache/zeppelin

license_source: bigquery


text:
[ZEPPELIN-3773] - add check permission on write.

What is this PR for?
Sometimes when drawing the result of a paragraph, a call is made commit paragraph. And if the user does not have permission on write paragraph a window appears with a warning.
This PR fix it.
What type of PR is it?
Bug Fix
What is the Jira issue?
ZEPPELIN-3773
Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

@Savalek
I see an infinite number of GET request like http://localhost:8080/api/helium/suggest/2DMKVSPYC/20180807-154514_2063688624 with response "401 Unauthorized" is case if user don't have permission for write. Could you fix?

@mebelousov, Perhaps you have incorrectly configured shiro.ini.

@felixcheung, there is already a check on the server. Due to some errors, a commit of the paragraph went to the server and an error occurred.

@Savalek
You're right. I fix my shiro.ini.
But I still see a huge amount of background queries (up to 100 per second) connected with Helium.
I can reproduce the issue in development mode.
Tested on Ubuntu 16.04, Chromium (Version 69.0.3497.92) and Firefox Quantum 62.0
Check please.

This seems to be the bug that we are also facing currently. I have a notebook with some users that only have read permissions. These users receive an error message that the do not have the update permission on the notebook and therefore cannot open it.
Any estimate when this will be merged and released?

Need someone else to check this PR. Because I could not reproduce this error.

authors: Savalek deradam mebelousov


text:
🛑 Element is down

In 264ed85, Element (https://element.interhop.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Element is back up in b8d5422 after 17 minutes.

license: MIT


text:
[Features] support create Kibana for Elasticsearch cluster

Kibana is an essential analysis and visualization platform for Elasticsearch.

Hi @shuoshadow ,
It is a good suggestion. We plan to support by the end of Oct.

authors: shanshanying shuoshadow

repo: apecloud/kubeblocks-addons


text:
Support one way hash for OAuth2 client secrets

Add one way hash function support for OAuth2 clientSecret storage mechanism as recommended by OWASP

Thank you for the pull request. It does not look like this would be something we'd want to accept just yet. If need does come up and there is time, we can review this again.

repo: apereo/cas


text:
Error ill-formed ticket found in the URL when ticket is encrypted

If the cas server is encrypting the ticket value (cas.ticket.security.cipherEnabled=true) the php client fails with the error:  Error ill-formed ticket found in the URL
You have to set this property to false for the client to work.

Related to #180

I don't believe it is related to #180 because php CAS doesn't need to decrypt the ticket instead the criteria for what is a valid ticket needs to accept that the encrypted ticket is valid and just pass it back.

Can you please supply a debug log?

3424 .START (2017-08-22 12:11:13) phpCAS-1.3.5 ****************** [CAS.php:468]
3424 .=> phpCAS::client('3.0', 'castst.conncoll.edu', 443, 'cas') [index.php:12]
3424 .|    => CAS_Client::__construct('3.0', false, 'castst.conncoll.edu', 443, 'cas', true) [CAS.php:360]
3424 .|    |    Starting a new session cq25i8qv8bbq8uqoiud8hh2g75 [Client.php:932]
3424 .|    |    Session is not authenticated [Client.php:938]
3424 .|    |    => phpCAS::error('ill-formed ticket found in the URL (ticket=eyJhbGciOiJIUzUxMiJ9.WlhsS05tRllRV2xQYVVwRlVsVlphVXhEU21oaVIyTnBUMmxLYTJGWVNXbE1RMHBzWW0xTmFVOXBTa0pOVkVrMFVUQktSRXhWYUZSTmFsVXlTVzR3TGk1NFYxaDNTbWhOY25KMFoxQm9aM1ExVkc4eGFGVjNMakJhZHpRdE0zcFRVV2hFUlhsaGNYVTVibWw1VlUxT05EazFURkpRWDNkRWVHcERXR0l3YUVGcldXSkdPWGxMTmpWRFIyazBiV2M1VUdocldWbHVVa0l1VVUxWmFtZFVUWGRLWm1VM2NVNVhabVk1VjBsclp3PT0.QCt2Ma0yxcfigVaNE5DYlwog1Vz8bIRB_EzoJjs85wWnXKCEwaxlvQKoIMU7C4HdxFbJya-Pj6URByRfpMwbsg\')') [Client.php:1028] 3424 .|    |    |    ill-formed ticket found in the URL (ticket=eyJhbGciOiJIUzUxMiJ9.WlhsS05tRllRV2xQYVVwRlVsVlphVXhEU21oaVIyTnBUMmxLYTJGWVNXbE1RMHBzWW0xTmFVOXBTa0pOVkVrMFVUQktSRXhWYUZSTmFsVXlTVzR3TGk1NFYxaDNTbWhOY25KMFoxQm9aM1ExVkc4eGFGVjNMakJhZHpRdE0zcFRVV2hFUlhsaGNYVTVibWw1VlUxT05EazFURkpRWDNkRWVHcERXR0l3YUVGcldXSkdPWGxMTmpWRFIyazBiV2M1VUdocldWbHVVa0l1VVUxWmFtZFVUWGRLWm1VM2NVNVhabVk1VjBsclp3PT0.QCt2Ma0yxcfigVaNE5DYlwog1Vz8bIRB_EzoJjs85wWnXKCEwaxlvQKoIMU7C4HdxFbJya-Pj6URByRfpMwbsg') in /cwd/cwassets/httpd/alias/tp/cas/cas5-php-test/index.phpon line 12 [CAS.php:566]
3424 .|    |    <= ''

Thanks, looks like we simply need to adjust the regexp security filter for the ticket so that it allows all formats including the new encryption.

Has there been any movement on this? I'm running against master, with all of the fixes for ticket/session length (#248, #257, and #224), but enabling cas.ticket.security.cipherEnabled still throws an error.

authors: MrDys atilling jfritschi

repo: apereo/phpCAS


text:
Add UP_STAGE environment variable

Technically I think it's more correct to set env vars for the environment, not based on the env such as NODE_ENV. However we need it for logging anyway.
This may have to be a hack, setting the env on first request, since stageVariables is the only way to get at the stage (AFAIK).

It would be handy for UP_STAGE to be made available to the hooks too, allowing particularly the build script to adjust it's behaviour accordingly.
I'm not sure if this issue refers to the application environment (in AWS) only, or making UP_STAGE available everywhere including local environment when building.

Hmm yeah it's a little tricky, since the "ideal" way of deploying would be to stage the application, then promote the staged version to production. UP_STAGE would will be "staging" at that point

Promoting the stage build to production doesn't work for statically built files like next.js does.

@cgarvis yea not if you differentiate config in those two stages. It depends how you structure things I suppose, since you can pass UP_STAGE to your client JS for example and choose config that way.
It would be a more ideal way to deploy since you know exactly what you're getting.

@tj I use webpack's DefinePlugin to replace process.env in my builds.  That way I don't leak environment vars by accident.  Also makes it more difficult for someone on the team to hit the wrong endpoints.

authors: cgarvis jamesramsay tj

repo: apex/up


text:
GLIBC_2.32 not found / CGO_ENABLED=0

Prerequisites

[X] I am running the latest version. (up upgrade)
[X] I searched to see if the issue already exists.
[X] I inspected the verbose debug output with the -v, --verbose flag.
[X] Are you an Up Pro subscriber?

Description
On Arch glibc is not compatible with AWS's. So the workaround is to use CGO_ENABLED=0
However CGO_ENABLED=0 often means poor support with sqlite drivers or example. CGO_ENABLED=0 versions of sqlite drivers exist but they are significantly slower for example.
Perhaps it's having some sort of cross compile option as suggest here https://twitter.com/benbjohnson/status/1370955506471165956 ?
Steps to Reproduce
Try deploy https://github.com/kaihendry/dfts

Hmm I'm not sure I'd want a dependency on Docker, it could get really complicated quick. I think that sort of thing is probably best left to CI like GH Actions where you're building in the target environment already, but something to think about for sure

I'm told this what I need to do: https://gist.github.com/egonelbre/01bbf7ca97d6b5588438da36a2578e7b
I do really want a smooth way to deploy locally without CI.

I have no previous experience with apex/up, but as far as I'm able to deduce it seems that hooks.build is a way to override with arbitrary build command? So it should be possible to pretty much use the same invocation -- maybe modify it to output "server" as the binary. Although, I'm not sure in which computer/server context will run.
The only concern is invoking subcommands such as $(pwd) and $(go env GOPATH) within the command. Maybe it can invoke a separate script? Based on https://github.com/apex/up/blob/9770c5062e3a39563d183f84ce51cec52bf683ec/up.go#L67, I'm guessing it will work?

@egonelbre yep it runs as a shell command, so you should be fine to use stuff like that
@kaihendry yeah I definitely hear you, but the number of variables explodes once you start introducing platform specifics. Using Docker manually locally and putting Up inside would more or less have the same UX as Up trying to use Docker, so I'm not sure there would be much of a benefit, since Up would have to support all the various OSes and shared libraries etc

authors: egonelbre kaihendry tj


text:
Always have the fill below the curve in area charts

I'm creating trend line widgets, showing the angle of the current trend line (another request btw) for a given time period. Some chart values are however negative and that makes the chart "flip" the area or even worse, cross the line. Like in this codepen:
https://codepen.io/jkohlin/pen/BaNgbOy
What I want is an option for fill that might look like this:
fill: {
  colors: '#0000ff',
  opacity: 0.9,
  type: 'gradient',
  fillTo: -20,
}

Where fillTo is a numeric value that, if set, forces the fill to flow all the way down to that value on the y-axis.
Then you could get this:

instead of this:

Describe alternatives you've considered
At the moment I have to recalculate the y values and add a positive value big enough to force the curve above zero. This works on sparklines, where the y-axis is hidden, but not otherwise of course.
This would be an awesome feature!

I would, too, like to request this. As of now it looks rather strange for negative values, if the area has a gradient:
. It makes it quite difficult to see the lowest values.

Sorry for the late response.
I have added a new option that will extend the area beneath zero-line and fill it till the end.
plotOptions: {
  area: {
    fillTo: 'end'
  }
}

This will be available from v3.19.3

authors: jkohlin junedchhipa oherik


text:
x-axis labels are not appears

Codepen
https://codepen.io/anon/pen/YMjPEa
added to code-pen
Explanation

What is the behaviour you expect?
the x-axis labels are too long
What is happening instead?
so it is not appearing correctly.
What error message are you getting?
half of it replaced with ...  and I need it all to be visible

thanks

Increase the yaxis labels maxWidth by
yaxis: {
   labels: {
     maxWidth: 200
   }
}

authors: bashairm junedchhipa


text:
Airnode functions refactor

This updates wallet functions, loadConfig, logger, getGasPrice, callApi to be imported from airnode-node v0.5
Note: to get this to build locally, you will need to copy the built airnode-node and airnode-utilities /dist folders from Airnode PR 944 into your Airkeeper node_modules/@api3
This probably shouldn't be merged until we can update to v0.5 and then we can temporarily update the package.json to use airnode-node for PR944 as a dependency with:
'git+https://gitpkg.now.sh/api3dao/airnode/packages/airnode-node?ec0340aabc3be174e62f315b9cc0bbe62b43d0da' using this which allows us to use a monorepo subdirectory as a package

@acenolaza Do the example configs work for you in the main branch? I had to change the airnodeAddress in airkeeper.json (these changes) to make them work.

I'm actually using 0xA30CA71Ba54E83127214D3271aEA8F5D6bD4Dace because in secrets.env I have the test mnemonic we are using everywhere else

authors: acenolaza vponline


text:
BEC258: Terraform ECS

I'm creating this as a WIP because I wanted to get feedback as soon as possible and also because the Airseeker docket image is not yet in docker hub.
Some notes:

Created an ecs cluster, task definition and service to tie them together.
Since I'm using FARGATE I had to add a vpc resource but since we are not exposing a frontend I didn't think it was necessary to deal with internet gateways, load balancers and security groups. The simplest way I found to add a vpc is to just define a default vpc resource with default subnets.
Since I wanted to add logs to the task (awslogs set in the Task Definition) I had to also attach the AmazonECSTaskExecutionRolePolicy.
I've hardcoded the airnode-client docker image url just for testing until airseeker image is ready. Once that happens I guess I'll replace it with a variable.
I wasn't sure if we needed to deploy to different stages so I decided to keep it simple and have all terraform files in the same directory. Also I have not created any modules until I see the need for them.

I kept it as a Draft PR because I wasn't sure if airseeker-dev was the right image to use. Also, running docker pull api3/airseeker-dev:latest doesn't seem to work for me. I need to use a specific task in order to be able to pull the image from docker hub. Same thing happens when I use latest to pull airkeeper image. Maybe @aquarat has any idea what might be happening thinking I did found something about having to set some config to true in docker hub in order to be able to pull using latest.

I think the problem is that there is no @api3/airseeker-dev tagged as latest :smile: Set the default to @api3/airseeker:0.1.0 (even though it doesn't exist yet). You can pass a dev one (with commit hash) as an variable to test that it's actually working.

@amarthadan I feel like this is much better now and it's ready for final review.

@amarthadan I feel like this is much better now and it's ready for final review.

Yes, I'll review it later today :slightly_smiling_face:

I've rebased this branch on top of main and I started getting an issue with the use of console.log in scripts/terraform-fmt.ts which I fixed by just putting the whole scripts folder in the eslintignore file because those scripts are only used for development.

authors: acenolaza amarthadan


text:
Incorrect highlighting when MSON attribute description contains underscore

See apiaryio/api-blueprint#202. Sample at line 126.

Same problem here with wrong rendering
+ Request Client credentials (application/json)

    + Attributes

        + grant_type: `client_credentials` (string, required)
        + client_id: `88888888-4444-4444-4444-cccccccccccc` (string, required)
        + client_secret: `clientsecret` (string, required)

    + Body

            {
                "grant_type": "client_credentials",
                "client_id": "88888888-4444-4444-4444-cccccccccccc",
                "client_secret": "clientsecret"
            }

@danielgtaylor Now that I look at this with fresh eyes, this might not be an issue at all, since underscores in MSON are not allowed and they need to be escaped. If that rule is followed, the highlighter would not make any mistake.

MSON are not allowed and they need to be escaped

This is true for property names and values. Not for description.

Same hightlighting issue happens if you have a JSON payload that contains a key that starts with an underscore.
For instance:
## Videos Collection [/videos]

Provides access to all videos.

+ Model (application/json)

    JSON representation of videos

    + Body

            {
                "_pagination":
                {
                    "next": "/videos?offset=7",
                    "total_count": 100
                }
            }

Highlighting is incorrect starting from the underscore.

any update?

authors: danielgtaylor edwardaa freezy-sk jeanregisser pksunkara zdne

repo: apiaryio/api-blueprint-sublime-plugin

license: mit


text:
Add Wercker and more

Summer cleanup 😎

Preparing for CircleCI 2.0 support in Dredd's dredd init
Adding Wercker
Using simpler naming for several files
Being more explicit in the README about where all the files are, to which service they belong, etc.
Adding Jenkins to the README even though it's not live-tested
Swagger is dead, long live the OpenAPI 2 (preparing for OpenAPI 3 support in Dredd and elsewhere)

@nadade @kylef Call for help 🙋♂
I'm trying to make Wercker to npm install --no-optional, but the no optional flag seems to be ignored - it still installs protagonist:
...
export WERCKER_NPM_INSTALL_OPTIONS="--no-optional"
source "/pipeline/npm-install-83ebc85d-4d44-47f3-a21f-75237025acae/run.sh" < /dev/null
Using wercker cache
Creating $WERCKER_CACHE_DIR/wercker/npm
Configuring npm to use wercker cache
Starting npm install, try: 1
npm WARN deprecated json-schema-faker@0.5.0-rc13: Broken not support
npm WARN deprecated nomnom@1.5.2: Package no longer supported. Contact support@npmjs.com for more info.

> protagonist@1.6.8 install /pipeline/source/node_modules/protagonist
> node-gyp rebuild
...

Even if I tried to create my own step, it was still ignored and Wercker just installed the project as without --no-optional, compiling protagonist. Any ideas?

@honzajavorek I think this may be NPM bug: https://npm.community/t/npm-install-no-optional-not-actually-filtering-optionals-in-cli-6-0-1-or-6-1-0/257 / https://github.com/npm/npm/issues/17633#issuecomment-403938408
Perhaps you can work around the issue with having a package-lock.json generated without protagonist. Probably something you should do regardless due to transient dependency licensing approval.

Oh, package-lock.json is a different can of worms I'll need to resolve later. Thanks for looking into it, I think I'm fine with Wercker installing protagonist now.

authors: honzajavorek kylef

repo: apiaryio/dredd-example


text:
httpx.InvalidURL: Invalid non-printable ASCII character in URL

Hey, I try to scrap music but it seems that the crawler with the await context.enqueue_links(strategy="all") add invalid url, I run my code but I have the error:
[crawlee.autoscaling.autoscaled_pool] INFO  Waiting for remaining tasks to finish
Traceback (most recent call last):
  File "/home/jourdelune/dev/Crawler/src/main.py", line 21, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/jourdelune/dev/Crawler/src/main.py", line 14, in main
    await crawler.run(
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/crawlee/basic_crawler/basic_crawler.py", line 359, in run
    await run_task
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/crawlee/basic_crawler/basic_crawler.py", line 398, in _run_crawler
    await self._pool.run()
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/crawlee/autoscaling/autoscaled_pool.py", line 185, in run
    await run.result
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/crawlee/autoscaling/autoscaled_pool.py", line 336, in _worker_task
    await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/crawlee/basic_crawler/basic_crawler.py", line 734, in __run_task_function
    await self._commit_request_handler_result(crawling_context, result)
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/crawlee/basic_crawler/basic_crawler.py", line 653, in _commit_request_handler_result
    destination = httpx.URL(request_model.url)
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/httpx/_urls.py", line 115, in __init__
    self._uri_reference = urlparse(url, **kwargs)
  File "/home/jourdelune/dev/Crawler/env/lib/python3.10/site-packages/httpx/_urlparse.py", line 163, in urlparse
    raise InvalidURL("Invalid non-printable ASCII character in URL")
httpx.InvalidURL: Invalid non-printable ASCII character in URL

the invalid url is: https://www.linkedin.com/company/nic-br/
Code:
import re
import urllib.parse

from crawlee.basic_crawler import Router
from crawlee.beautifulsoup_crawler import BeautifulSoupCrawlingContext
from crawlee.playwright_crawler import PlaywrightCrawlingContext

router = Router[PlaywrightCrawlingContext]()

regex = r"https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()!@:%_\+.~#?&\/\/=]*)\.(mp3|wav|ogg)"

@router.default_handler
async def default_handler(context: BeautifulSoupCrawlingContext) -> None:
    url = context.request.url
    html_page = str(context.soup).replace("\/", "/")

    matches = re.finditer(regex, html_page)

    audio_links = [html_page[match.start() : match.end()] for match in matches]

    for link in audio_links:
        link = urllib.parse.urljoin(url, link)

        data = {
            "url": link,
            "label": "audio",
        }

        await context.push_data(data)

    await context.enqueue_links(strategy="all")

hey, thanks you for the answer, the code that imports the router:
"""
main script for the crawler
"""

import asyncio

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler

from routes import router
from utils import process

async def main() -> None:
    """
    Function to launch the crawler
    """

    crawler = BeautifulSoupCrawler(
        request_handler=router,
    )

    await crawler.run(
        ["https://www.cgi.br/publicacao/revista-br-ano-07-2016-edicao-09/"]
    )
    await crawler.export_data("results.json")
    process("results.json")

if __name__ == "__main__":
    asyncio.run(main())

I want to crawl the full web to create a dataset of song url (to create an AI music generation model), that why I use strategy="all",
if you run the code, you should get the error.
The url where it get the invalid url is: https://www.cgi.br/publicacao/revista-br-ano-07-2016-edicao-09/
the invalid url is: https://www.linkedin.com/company/nic-br/

Huh, this is getting interesting. I added this to the request handler:
context.log.info(f'links found: {"\n".join([repr(link.attrs.get('href')) for link in context.soup.select('a')])}', )

...and it showed me that the linkedin link in fact contains a line break:

However unusual this is, I'll add a .strip() to the enqueue_links implementation.

authors: Jourdelune janbuchar

repo: apify/crawlee-python


text:
Styleguides: improvements in styleguide/conformance protos

With the aim of displaying the lint results in the UI, the conformance report protos need the following updates:

The spec field should include the revision ID of the spec.
RuleReport should include the metadata of the rule defined in the styleguide. The metadata fields which should be included in the rule are as follows:

display_name
description
doc_uri

cc @michaelyara

repo: apigee/registry


text:
Feature/api metadata i18n. Closes #779

Added i18n strings to view metadata page. This does not include AutoForm label translations, as that currently seems non-trivial.

@elnzv will you please review thie PR?

It may not be necessary to add the translation strings to the fi.i18n.json. Did you read somewhere that this is a requirement?

@brylie, no, that is not a requirement, but just more comfortable to translate, no need to search for new lines in en.json.
Anyway, I added those myself, resolved conflicts.
Merging.

authors: brylie elnzv

repo: apinf/api-umbrella-dashboard


text:
Requires Tatin to intialise before opening project

Don't know whether this is documented and I missed it, but
      ]CIDER.OpenProject \g\ParquetDotNet
* Command Execution Failed: VALUE ERROR: Undefined name: Tatin

goes away if I first do
      ]TATIN.Version
┌─────┬───────────┬──────────┐
│Tatin│0.90.0+1485│2023-02-27│
└─────┴───────────┴──────────┘

I would expect not to have to call a different UCMD before being able to use Cider

Tatin it's will be part of a 19.0 standard installation. In older versions it's up to the user to make sure that it's loaded into ⎕SE.
However, the error message should be more explicit about what's required.

Changed my mind. Solved in version 0.23.4

authors: aplteam rikedyp

repo: aplteam/Cider


text:
proxy support

A production server I use has outgoing connections firewalled, and we have to use a proxy.  This plugin needs to be able to connect via the proxy for geocoding requests, similar to https://rails.lighthouseapp.com/projects/8994/tickets/2133-activeresource-http-proxy-support

#126 has code attached

repo: apneadiving/Google-Maps-for-Rails


text:
Using exec.start()

Hi again,
So I read the examples you referred me to in issue #309. I've even "sort of" run your example. Specifically, rather than create an instance of an Ubuntu container, I did a
docker.getContainer(ID)
The container in question is running MongoDB and I'd like to be able to present to its shell (bash) several commands and get their responses. But I am uncertain about how to accomplish this. As matters stand, based on your example, I can see the usual console messages from MongoDB as its shell starts. These were raised in response to the original Cmd, 'mongo', presented by
container.exec()
But now I need to present a few more commands to the MongoDB shell & parse the responses. These commands don't reside in a file. They are strings defined in the program itself.
Would you be so kind as to provide an example of how to do this with dockerode's exec facility?
Thanks.
Cordially,
Paul

Hmmm...maybe hold off for a bit on any kind of answer. I may have gotten it working. The "trick" appears to be the use of 'hijack: true', 'stdin: true' in the exec.start options as well as 'AttachStdin: true' in the container.exec options.

👍

authors: apocas moonlitSpider

repo: apocas/dockerode


text:
Fix inconsistent behaviour of Container.inspect

container.inspect() is supposed to return a Promise not a stringified object

Yup legacy pre-promise code left behind.
Could you please do the same for the other object's inspects? (image, network, etc)
So everything is consistent.

@apocas done!

Published v2.4.0


text:
TypeScript optional inputs

Similar to #155, but for TypeScript, fixes #81

@tgriesser @lewisf: I'm sorry I'm only getting to this now, but I wanted to publish a new release. It turns out some of the TypeScript snapshot tests are failing however, and it seems the results do not match the expected behavior in this PR. Could you have a look at this?

@martijnwalraven will take a look now

@martijnwalraven this actually looks like a small oversight. Some changes weren't pulled in. I opened up #162 to get the tests passing

authors: lewisf martijnwalraven tgriesser

repo: apollographql/apollo-codegen


text:
OkHttpExecutionContext replacement

Question.
OkHttpExecutionContext was available in version 2 after migration to 3.3.0 this context key was removed and I can't find another way how to get okhttp response from ApolloResponse

Hi 👋 Thanks for reaching out
In 3.x you can use response.executionContext[HttpInfo]. That will give you the status code and headers from your HTTP call. If you need access to the body, I'd recommend using an HttpInterceptor so that you can read the body as it is streamed. If you need something else, let us know and we can investigate together how to get it.

thanks @martinbonnin for quick reply! The reason why I need okhttp  Response is because we have backend driven logic when automatically refresh page.
This is an example how we did it previously.
apolloClient.query(...).rx().singleOrError().map {
            val okHttpResponse  = it.executionContext[OkHttpExecutionContext]?.response
            val maxAgeInSeconds = okHttpResponse?.cacheControl?.maxAgeSeconds ?: 0
        }

I see, thanks for providing the details!
Looking at the OkHttp code, looks like it's getting this value from a header so you could do something like:
    val maxAgeInSeconds = response.executionContext[HttpInfo]
        ?.headers
        ?.firstOrNull {
          it.name.lowercase() == "max-age"
        }
        ?.value
        ?.toIntOrNull()

Could that work?

That's so neat! thank you for help and have a nice weekend 😊

Thank you, you too :) !

authors: denys-meloshyn martinbonnin

repo: apollographql/apollo-kotlin


text:
Firing queries from Android Studio plugin with @nonull directive

Question
Hey, I am using Apollo plugin with Android Studio and I used to test my graphql files directly from IDE. Noticed that adding @nonull does not allow plugin to work and returns error message "message": "directive 'nonnull' is not defined in the schema"
The question is if that can be somehow workaround?

Can you try enabling the "Apollo Kotlin" framework in the GraphQL plugin?

Hey @martinbonnin I had this enabled

IDE support is still being worked on (see https://github.com/JetBrains/js-graphql-intellij-plugin/issues/697) so it'll come but the red underlines for @semanticNonNull are still expected at this stage.
I would still expect @nonnull to be detected though(see https://github.com/JetBrains/js-graphql-intellij-plugin/blob/21d5800921d07176992e3d006728a89a1c1eb242/resources/definitions/ApolloKotlin.graphql#L23)

@nonnull works just fine when editing graphql file, there is support for that. It just fails when executing graphql queries from IDE

Thanks. They work just fine from the app. Been using it for some time now on prod. It's just very convenient sometimes to verify something right from the IDE.

I created https://github.com/apollographql/apollo-kotlin/issues/5507 as a follow up before I forget.

authors: damianpetla martinbonnin


text:
Crash when use policy

Version
3.8.2
Summary

override suspend fun  getMerchantsByKeyWord(keyword:String): Flow<Response> {
return apolloClient.query(MerchantsByKeyWordQuery(keyword)).fetchPolicy(FetchPolicy.CacheAndNetwork)
.toFlow()
.map { response ->
try {
//response.data.merchants
Response.Success(response.data?.toRestaurantsSearch() as T)
} catch (e: Exception) {
Response.Failure( e.message ?: "unknown error")
}
}
}
Steps to reproduce the behavior
No response
Logs
(Your logs here)

Hi! It looks like you're having a network issue. This can happen if your device has a connectivity issue, or e.g. if the backend is not responding.

I enabled working offline from cache policy

CacheAndNetwork will both go to the cache and the network, which will throw if the network fails.

I disable wifi to test app  working offline

In that case what you're seeing is definitely expected. You should probably add a .catch {} call to the Flow chain, to handle the exception.

authors: BoD enggazzar


text:
Not possible to control the types of items in a list

Given a field that returns an array that composed of Union or Interface types, it is not possible to control the type of each item in the list. For example, given the following schema:
      union AdvertOrPost = Advert | Post

      enum SupportedTypes {
        Advert
        Post
      }

      type Advert {
        id: ID!
        productName: String!
      }

      type Post {
        id: ID!
        text: String!
      }

      type RootQuery {
        getAll(type: SupportedTypes): [AdvertOrPost]
      }

      schema {
        query: RootQuery
      }

It should be possible to apply the following mocks:
    const mockMap = {
      RootQuery: () => ({
        getAll: (o: any, a: { [key: string]: any }) => new MockList(2, () => ({typename: a['type']})),
      }),
      Advert: () => ({
        productName: 'supercoolproduct',
      }),
      Post: () => ({
        text: 'superlongpost',
      }),
    };

However, using the above code would result in objects that do not have the type-specific mocks applied, for example a result to querying getAll(type: Advert) would look like this:
{
  "getAll": [
     {"__typename": "Advert", "productName": "Hello World"},
      {"__typename": "Advert", "productName": "Hello World"},
  ]
}

@ajs139 Sorry for the late reply. I think in order for the server to know which mock to apply, your schema has to define the __resolveType or __ofType functions. I might also be wrong however, in which case a failing test case would be much appreciated.

Sure, please see https://github.com/apollographql/graphql-tools/pull/282/files#diff-b9774ff344f81ba2175bf0fb4973ac7a for a test (this PR also contains a fix, but I closed it because it didn't deal with a nested array return type (e.g. someField: [[Foo]]) and haven't had the chance to revisit.

I think this should work in 1.1.0.

repo: apollographql/graphql-tools


text:
New: Added Unit Tests for Graph Generators

**Merge PR https://github.com/aporeto-inc/trireme-statistics/pull/31 before merging this PR **
--> Added unittests for server functions
--> Added mock for influxdb

Thanks for addressing those changes.

I will let you merge this PR (whenever you feel it is in a ready state)

authors: bvandewalle sibicramesh

repo: aporeto-inc/trireme-statistics


text:
Add a new field subnets for machine pool

They can span across more than one subnet for multi-az cluster. This way it's backwards compatible until we are ready to migrate to this new field.

So this will not be useful as we remove the default machine pool which creates nodes in all subnets then create one machine pool designated for each AZ/subnet, and there is no way to provide more than one subnet ids when creating a machine pool through ROSA CLI and OCM API.

repo: app-sre/qontract-schemas


text:
React-Native: Add Documentation

[x] We need to create a version switcher for React Native. Using a different repo

[x] Getting Started Section

[x] Install and Quickstart
[x] Import Data
[x] ReactiveBase
[x] Components Overview

[x] Component References
One doc per component

[x] Advanced

[x] Add a snack snippet for the ReactiveComponent doc.
[x] Performance Tips @metagrover
[x] Creating Themes, Styles Unsupported in current version

[ ] Link innerStyle to all the components
[ ] Link innerProps to all the components
[ ] Update theme prop for ReactiveBase.

[ ] Add a doc on how to publish the app - With expo and as an ejected app.

repo: appbaseio/reactive-manual


text:
New Zoom/Pinch implementation

After working with the zoom and pinch functionalities I made in my last PR we (myself and a colleague) came across some inconsistencies.
We figured that iOS and Android use very different approaches to these gestures, so we worked on a new implementation for these that works much more consistent.

do you want to take out the generic approach all together?

Well, the problem is that the Appium server needs different parameters for the different platforms. So the only alternative to this approach would be to define an interface with all of those parameters. But this way one implementation would just ignore a half of these parameters and the other implementation would ignore the other half. And I think this doesn't really add anything useful to the code.

Was there something wrong with the multi touch way of doing things instead
of running the script directly?
On Fri, Jan 15, 2016 at 1:30 PM, trbnb notifications@github.com wrote:

Well, the problem is that the Appium server needs different parameters for
the different platforms. So the only alternative to this approach would be
to define an interface with all of those parameters. But this way one
implementation would just ignore a half of these parameters and the
other implementation would ignore the other half. And I think this
doesn't really add anything useful to the code.
—
Reply to this email directly or view it on GitHub
https://github.com/appium/appium-dotnet-driver/pull/132#issuecomment-172097985
.

Yes, after observing it a bit we found it to be quite unreliable. Especially on iOS we sometimes struggled to make it work.

I'll review this. I've some remark...

Sorry for the late responce, I was busy
I have a a question to @Jonahss. Let him to take a look at your implementation. I have some question to the original code... If it is correct then it makes sense to port your code to Java client.

I'm not really sure how because the gestures don't change the size the view or something like this, just how the content is presented and I don't think that's something we can retrieve with Appium.

Any news on this?

Waiting for @Jonahss (I've told about this PR) because I think that this change could be ported to Java client too.
As I know current gestures were ported from Java client. I dont know why original code was designed the way as it is now. But it is most likely I'm Ok with your changes.

Ok, good to know.
And yes, these gestures were initially ported from the Java client, I made the pull request for that a few weeks/months ago.

@trbnb
i'm ok with your changes.
There are tons of work with appium node 1.5.x. I'm working on java client now. You fix is supposed to be included in 1.6.0.1. There will be many fixes related to node 1.5.x.

Alright, thanks for letting me know.

Just for your information. We (trbnb and myself) ported the new pinch and zoom behavior from the python appium driver. More information can be found here:
https://github.com/appium/python-client/blob/master/appium/webdriver/webdriver.py
Be careful. They are calling pinch and zoom in the code a little bit different. It is internally called pinchClose and pinchOpen. But it does the same.
It is a good idea to port our PR to the java bindings too. Pinch and Zoom is broken there too. We initialy ported the pinch and zoom command from the java bindings to the dotnet bindings without testing if it worked in the Java bindings. Seems that it did not work under iOS at all. So thats the reason we did it again. This time we ported it from the python bindings. And now it should work on Android and iOS.

Guys,
I'm going to play around with proposed changes. I think I'll propose a PR with your authorship if everything is ok.
However, I'm not sure that it is compatible with node server 1.5.x

Sorry.
I need to close it without merging. This functionality is not supported since Appium 1.5.0
https://github.com/appium/appium/blob/master/CHANGELOG.md#general-3
Server is throwing this exception now.

Method has not yet been implemented (WARNING: The server did not provide any stacktrace information)

See also:
https://github.com/appium/appium/issues/2234

Can we add docs for the new way of performing pinch and zoom?

@Jonahss It has to work the same way.
I and @ShrivanTarget are searching for something instead.

repo: appium/appium-dotnet-driver