text:
[SPARK-36271][SQL] Unify V1 insert check field name before prepare writter

What changes were proposed in this pull request?
Unify DataSource V1 insert schema check field name before prepare writer.
Why are the changes needed?
Unify code
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added UT

Test build #141800 has started for PR 33566 at commit 33e9031.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46313/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46313/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46313/

Test build #141800 has finished for PR 33566 at commit 33e9031.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141800/

Test build #141824 has started for PR 33566 at commit 378752d.

Test build #141824 has finished for PR 33566 at commit 378752d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141824/

Test build #141825 has started for PR 33566 at commit f94585d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46338/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46337/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46337/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46338/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46338/

Test build #141825 has finished for PR 33566 at commit f94585d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141825/

ping @wangyum @maropu @maropu

Any suggestion?

Test build #142099 has started for PR 33566 at commit e4c2b80.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46611/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46611/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46611/

Test build #142106 has started for PR 33566 at commit d9c82ad.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46618/

Test build #142108 has started for PR 33566 at commit 7cc7340.

Test build #142109 has started for PR 33566 at commit da2a416.

Test build #142108 has finished for PR 33566 at commit 7cc7340.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AvroV1Suite extends AvroSuite

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142108/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46618/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46621/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46618/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46620/

Test build #142099 has finished for PR 33566 at commit e4c2b80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142099/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46620/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46621/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46620/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46621/

Test build #142106 has finished for PR 33566 at commit d9c82ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142106/

Test build #142109 has finished for PR 33566 at commit da2a416.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142109/

ping @cloud-fan

GA failure is unrelated, thanks, merging to master/3.2!

authors: AmplabJenkins AngersZhuuuu SparkQA cloud-fan


text:
[SPARK-36897][PYTHON] Use NamedTuple with variable type hints instead of namedtuple

What changes were proposed in this pull request?
Use NamedTuple with variable type hints instead of namedtuple.
Why are the changes needed?
Per discussion under https://github.com/apache/spark/pull/34133#discussion_r718833451, we wanted to replace collections.namedtuple() by typing.NamedTuple.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing tests.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48517/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48517/

Merged to master.

source: gharchive/pull-request

authors: HyukjinKwon SparkQA ueshin

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-41891][CONNECT][TESTS] Enable test_add_months_function, test_array_repeat, test_dayofweek, test_first_last_ignorenulls, test_inline, test_window_time, test_reciprocal_trig_functions

What changes were proposed in this pull request?
Enabling tests in connect/test_parity_functions.py
Why are the changes needed?
Improved coverage
Does this PR introduce any user-facing change?
No
How was this patch tested?
New Tests

Merged to master.

authors: HyukjinKwon techaddict


text:
[SPARK-42191][SQL] Support udf 'luhn_check'

What changes were proposed in this pull request?
This PR  adds a built-in function to check if a given number string is a valid Luhn number. It shall return true if the number string is a valid Luhn number, and false otherwise.

Why are the changes needed?
This checksum function is widely applied to credit card numbers and government identification numbers to distinguish valid numbers from mistyped, incorrect numbers
Ref : Trino
Postgresql

Does this PR introduce any user-facing change?
Yes, new udf luhn_check

How was this patch tested?
Added test cases

Hi @dtenedor, @dtenedor @srielau, @HyukjinKwon @cloud-fan , @gengliangwang
Could please review this PR?

The general algorithm and test coverage look correct

I'm not super against this, but this seems too niche to put into the Spark API

Hi @srowen I get where you're coming from. For background, we have a Jira [1] to add a suite of data masking functions into Spark. This is one of a family of such proposed functions. The idea is to support ETL with redaction of original datasets to make it safer for sharing.
[1] https://issues.apache.org/jira/browse/SPARK-40686

Right, I don't think most of those belong in the Spark API. This is what UDF and utility libs are for

@srowen yeah, we should apply discretion for those functions on that list.  There's something to be said for having a built-in library of functions that are useful enough for general-purpose consumption, but not everything belongs in there.
Maybe for this particular case we can let it through as it is in fact present in Postgres and Trino (as linked in the PR description) but we could do a review of the others and make go/no-go decisions on each one together. WDYT?

thanks, merging to master!

authors: cloud-fan dtenedor srowen vinodkc


text:
[WIP] Similarity search fix

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

What is this?

@srowen fixed description, didn't expect you to be so quick

+1, LGTM. Merging to master.
Thank you, @vitaliili-db.

authors: MaxGekk srowen vitaliili-db


text:
[SPARK-45143][PYTHON][CONNECT] Make PySpark compatible with PyArrow 13.0.0

What changes were proposed in this pull request?
1, in PyArrow 13.0.0, the behavior of Table#to_pandas and ChunkedArray#to_pandas changed, set the coerce_temporal_nanoseconds=True
2, there is another undocumented breaking change in data type conversion TimestampType#to_pandas_dtype:
12.0.1:
In [1]: import pyarrow as pa

In [2]: pa.timestamp("us", tz=None).to_pandas_dtype()
Out[2]: dtype('<M8[ns]')

In [3]: pa.timestamp("ns", tz=None).to_pandas_dtype()
Out[3]: dtype('<M8[ns]')

In [4]: pa.timestamp("us", tz="UTC").to_pandas_dtype()
Out[4]: datetime64[ns, UTC]

In [5]: pa.timestamp("ns", tz="UTC").to_pandas_dtype()
Out[5]: datetime64[ns, UTC]

13.0.0:
In [1]: import pyarrow as pa

In [2]: pa.timestamp("us", tz=None).to_pandas_dtype()
Out[2]: dtype('<M8[us]')

In [3]: pa.timestamp("ns", tz=None).to_pandas_dtype()
Out[3]: dtype('<M8[ns]')

In [4]: pa.timestamp("us", tz="UTC").to_pandas_dtype()
Out[4]: datetime64[us, UTC]

In [5]: pa.timestamp("ns", tz="UTC").to_pandas_dtype()
Out[5]: datetime64[ns, UTC]

Why are the changes needed?
Make PySpark compatible with PyArrow 13.0.0
Does this PR introduce any user-facing change?
NO
How was this patch tested?
CI
Was this patch authored or co-authored using generative AI tooling?
NO

@dongjoon-hyun I don't see failure in docker build
#21 [15/19] RUN python3.9 -m pip install numpy pyarrow 'pandas<=2.0.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*'
#21 43.17 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#21 DONE 46.2s

we don't pin the mlflow version, instead we only set a lowerbound mlflow>=2.3.1, it looks like mlflow 2.7.0 can work with PyArrow 13.0.0:
pyarrow [required: >=4.0.0,<14, installed: 13.0.0]

Oh, it's great to have mlflow 2.7.0 on time! Thank you.

Merged to master for Apache Spark 4.0.0. Thank you, @zhengruifeng and @HyukjinKwon .

thank you @dongjoon-hyun and @HyukjinKwon

authors: dongjoon-hyun zhengruifeng


text:
[SPARK-46369][CORE] Remove kill link from RELAUNCHING drivers in MasterPage

What changes were proposed in this pull request?
This PR aims to remove kill hyperlink from RELAUNCHING drivers in MasterPage.
Why are the changes needed?
Since Apache Spark 1.4.0 (SPARK-5495), RELAUNCHING drivers have kill hyperlinks in the Completed Drivers table.

However, this is a bug because the driver was already terminated by definition. Newly relaunched driver has an independent ID and there is no relationship with the previously terminated ID.
https://github.com/apache/spark/blob/7db85642600b1e3b39ca11e41d4e3e0bf1c8962b/core/src/main/scala/org/apache/spark/deploy/master/DriverState.scala#L27
If we clicked Master always complains like the following.
23/12/11 21:25:50 INFO Master: Asked to kill driver 202312112113-00000
23/12/11 21:25:50 WARN Master: Driver 202312112113-00000 has already finished or does not exist

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual review.
Was this patch authored or co-authored using generative AI tooling?
No.

Could you review this one-line removal of MasterPage UI, @HyukjinKwon ?

Could you review this PR, @MaxGekk ?


text:
[SPARK-47512][SS] Tag operation type used with RocksDB state store instance lock acquisition/release

What changes were proposed in this pull request?
Tag operation type used with RocksDB state store instance lock acquisition/release
Why are the changes needed?
Currently its impossible to tell which operation/thread tried to acquire/release the RocksDB instance lock which is crucial in debugging issues around RocksDB instance lock acquisition
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added unit tests
[info] RocksDBSuite:
12:34:41.365 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] - disallow concurrent updates to the same RocksDB instance - with colFamiliesEnabled=true (with changelog checkpointing) (1 second, 634 milliseconds)
[info] - disallow concurrent updates to the same RocksDB instance - with colFamiliesEnabled=true (without changelog checkpointing) (654 milliseconds)
[info] - disallow concurrent updates to the same RocksDB instance - with colFamiliesEnabled=false (with changelog checkpointing) (609 milliseconds)
[info] - disallow concurrent updates to the same RocksDB instance - with colFamiliesEnabled=false (without changelog checkpointing) (584 milliseconds)
12:34:45.319 WARN org.apache.spark.sql.execution.streaming.state.RocksDBSuite:

Verified updated logs
sql/core/target/unit-tests.log:1958:13:38:41.698 concurrent-test-thread-3 INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(158)] for opType=load_store
sql/core/target/unit-tests.log:2235:13:38:41.886 concurrent-test-thread-3 INFO RocksDB [Thread-17]: RocksDB instance was released by [ThreadId: Some(158)] for opType=load_store
sql/core/target/unit-tests.log:2236:13:38:41.886 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(17)] for opType=close_store
sql/core/target/unit-tests.log:2239:13:38:41.889 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was released by [ThreadId: Some(17)] for opType=close_store
sql/core/target/unit-tests.log:2249:13:38:41.896 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(17)] for opType=load_store
sql/core/target/unit-tests.log:2518:13:38:41.950 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(17)] for opType=load_store
sql/core/target/unit-tests.log:2562:13:38:42.139 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was released by [ThreadId: Some(17)] for opType=load_store
sql/core/target/unit-tests.log:2563:13:38:42.139 concurrent-test-thread-2 INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(181)] for opType=load_store
sql/core/target/unit-tests.log:2607:13:38:42.293 concurrent-test-thread-2 INFO RocksDB [Thread-17]: RocksDB instance was released by [ThreadId: Some(181)] for opType=load_store
sql/core/target/unit-tests.log:2608:13:38:42.293 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(17)] for opType=load_store
sql/core/target/unit-tests.log:2611:13:38:42.319 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(17)] for opType=rollback_store
sql/core/target/unit-tests.log:2612:13:38:42.319 pool-1-thread-1-ScalaTest-running-RocksDBSuite INFO RocksDB [Thread-17]: RocksDB instance was released by [ThreadId: Some(17)] for opType=rollback_store
sql/core/target/unit-tests.log:2614:13:38:42.319 concurrent-test-thread-3 INFO RocksDB [Thread-17]: RocksDB instance was acquired by [ThreadId: Some(193)] for opType=load_store
sql/core/target/unit-tests.log:2942:13:38:42.525 concurrent-test-thread-3 INFO RocksDB [Thread-17]: RocksDB instance was released by [ThreadId: Some(193)] for opType=load_store

Was this patch authored or co-authored using generative AI tooling?
No

@sahnib @HeartSaVioR - PTAL, thx !

Thanks! Merging to master.

authors: HeartSaVioR anishshri-db


text:
[SPARK-48209][CORE] Common (java side): Migrate error/warn/info with variables to structured logging framework

What changes were proposed in this pull request?
The pr aims to
1.migrate error/warn/info in module common with variables to structured logging framework for java side.
2.convert all dependencies on org.slf4j.Logger & org.slf4j.LoggerFactory to org.apache.spark.internal.Logger & org.apache.spark.internal.LoggerFactory, in order to completely prohibit importing org.slf4j.Logger & org.slf4j.LoggerFactory in java code later.
Why are the changes needed?
To enhance Apache Spark's logging system by implementing structured logging.
Does this PR introduce any user-facing change?
No.
How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?
No.

cc @gengliangwang

LGTM overall. Thanks for the work!

LGTM overall. Thanks for the work!

Updated, thanks! ❤

Thanks, merging to master

authors: gengliangwang panbingkun


text:
[SPARK-9490] [DOCS] [MLLIB] MLlib evaluation metrics guide example python code uses deprecated print statement

Use print(x) not print x for Python 3 in eval examples
CC @sethah @mengxr -- just wanted to close this out before 1.5

Merged build triggered.

Merged build started.

Test build #39191 has started for   PR 7822 at commit bd7f7fb.

Test build #39191 has finished for   PR 7822 at commit bd7f7fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class MulticlassClassificationEvaluator (override val uid: String)
class NaiveBayes(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol):
class NaiveBayesModel(JavaModel):
class MulticlassClassificationEvaluator(JavaEvaluator, HasLabelCol, HasPredictionCol):

Merged build finished. Test PASSed.

Actually, I see a number of instances of the same print x in the other docs' Python examples. I'll push an additional commit for those. If I'm wrong, I can back it out easily.

Merged build triggered.

Merged build started.

Test build #39256 has started for   PR 7822 at commit 01abeba.

Test build #39256 has finished for   PR 7822 at commit 01abeba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

LGTM. Merged into master. Thanks!

authors: AmplabJenkins SparkQA mengxr srowen


text:
[SPARK-10068][MLlib] Adds links to MLlib types, algos, utilities listing

@mengxr @jkbradley

Merged build triggered.

Merged build started.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41061/
Test FAILed.

Merged build finished. Test FAILed.

Merged into master and branch-1.5. Thanks! (This is a doc-only change. I tested it on my local and skipped Jenkins.)

authors: AmplabJenkins feynmanliang mengxr


text:
[STORM-4003] add missing jakarta.xml.bind dependency to storm-kafka-monitor

What is the purpose of the change
When running Storm UI with Java 17 runtime, the lag of Kafka consumers is not visible in the topology view for the topologies which are consuming Kafka topics. Based on exceptions found in ui.log, this is because storm-kakfa-monitor is missing JAXB API.
How was the change tested
Using Storm 2.6.0 snapshot dated 2023/11/14, doing a copy of  jakarta.xml.bind-api-2.3.2.jar from lib-worker/ to lib-tools/storm-kafka-monitor/ then restarting Storm UI process solves the problem.

i see that there's an integration error , but I don't understand how it could relate to my PR. Any idea ?

Most likely flaky test (as it passes on 17,21). Retriggered.

okay tests are passing now :)

repo: apache/storm


text:
STORM-1073: Refactor AbstractHdfsBolt

STORM-969 changed the execution of HdfsBolt to be more robust to errors.  This change moves that logic up to AbstractHdfsBolt and adds two new abstract methods to AbstractHdfsBolt: writeTuple and syncTuples.  In other words AbstractHdfsBolt contains all of the error handling and retry logic for writing to HDFS, and its implementing classes will specify how to write specific content to HDFS.

@arunmahadevan please review this

+1 Overall it looks good once the minor comments are addressed.

Thank you for catching those, @arunmahadevan.  PR updated per your feedback.

@arunmahadevan @harshach Do either of you have additional feedback on this PR?  Thank you!

looks good to me

@dossett the code looks fine.  I have two minor nits, but they both existed in the previous code prior to the re-factor so I am +1 either way.
The build failure looks like it was a network hiccup with the maven repository.

@revans2 Thank you for feedback. I was the author of the previous code, so I have address those issues in this PR.  Thanks!

The code looks fine I am +1 on this, but I would like to give @harshach and @arunmahadevan some time if they want to comment. Any feedback?

Thanks for catching those @arunmahadevan. Fixed those and squashed everything into one commit.

+1

authors: arunmahadevan dossett harshach revans2


text:
Proposed TextProvider feature addition (and cleanup)

TextProvider feature addition and cleanup:

Introduce (optional) control flag STRUTS_I18N_SEARCH_DEFAULTBUNDLES_FIRST to request TextProviders read from default resource bundles first, instead of their standard lookup ordering.  Defaults to false.
Minor refactor of AbstractLocalizedTextProvider hierarchy to introduce getDefaultMessageWithAlternateKey() method that repackages existing logic used by the SrutsLocalizedTextProvider and GlobalLocalizedTextProvider.
Update GlobalLocalizedTextProvider method comments to properly reflect the actual logic of some of its methods.
New unit tests to confirm standard and default bundle first lookup ordering.
Cleanup of some unused imports, a few typos, and code formatting items.
Cleanup of default.properties comments to make them more consistent.

Coverage increased (+0.04%) to 49.863% when pulling f62384a471bdf93ce53d7bf9f6e86b4287f4e23a on JCgH4164838Gh792C124B5:localS2_26_TextProvider_improvement_1 into dcc53aebf983f6c0a48932262facb81f5d82815c on apache:master.

Coverage increased (+0.04%) to 49.863% when pulling f62384a471bdf93ce53d7bf9f6e86b4287f4e23a on JCgH4164838Gh792C124B5:localS2_26_TextProvider_improvement_1 into dcc53aebf983f6c0a48932262facb81f5d82815c on apache:master.

Hello Apache Struts Team.
This is an initial attempt to add a feature to the I18N TextProvider hierarchy to read resource entries from the default bundles first, and then continue with "normal" lookup if nothing is found.  It arose from a discussion on the developer list raised by G. Huber concerning the number of lookup calls when most resource entries are defined in the main application bundle.
This implementation allows a developer to "opt-in" to an alternative lookup strategy where the default bundles take precedence over entries anywhere else (if an entry does not exist in the default bundles, the class hierarchy bundles are then searched normally).  It should be slightly more efficient for applications that only use a "global" properties file (default resource bundle), while still providing lookups along the class hierarchy for any entries that are not defined in the default (global) resource bundles.  The default value of the flag is false, which preserves the normal (original) behaviour for resource lookup (requires explicitly setting the configuration to use the alternative lookup flow).
There is no JIRA yet for this item.  If after reviewing this PR there is support for the change, a JIRA can be opened to cover it for tracking.

Hello Apache Struts Team.
This is an initial attempt to add a feature to the I18N TextProvider hierarchy to read resource entries from the default bundles first, and then continue with "normal" lookup if nothing is found.  It arose from a discussion on the developer list raised by G. Huber concerning the number of lookup calls when most resource entries are defined in the main application bundle.
This implementation allows a developer to "opt-in" to an alternative lookup strategy where the default bundles take precedence over entries anywhere else (if an entry does not exist in the default bundles, the class hierarchy bundles are then searched normally).  It should be slightly more efficient for applications that only use a "global" properties file (default resource bundle), while still providing lookups along the class hierarchy for any entries that are not defined in the default (global) resource bundles.  The default value of the flag is false, which preserves the normal (original) behaviour for resource lookup (requires explicitly setting the configuration to use the alternative lookup flow).
There is no JIRA yet for this item.  If after reviewing this PR there is support for the change, a JIRA can be opened to cover it for tracking.

A new JIRA WW-5112 has been opened to track the proposed change in this PR.
If there are any additional changes to consider, please let me know.

A new JIRA WW-5112 has been opened to track the proposed change in this PR.
If there are any additional changes to consider, please let me know.

repo: apache/struts


text:
[viz][treemap v2] click on corner the chart will move around and the section color is not correct with the represented number scale

2 issues seeing in the Treemap V2
1, the Treemap chart will move if clicking on the corner section of the chart
2, the bigger the number, the lighter the color in the treemap section, should i expect the other way?
Expected results
1, click on the corner of the chart will not move it
2, color is darker with larger number?
Actual results
1, chart is shifting position when click on the corner section
2, color is lighter with larger number
Screenshots
https://user-images.githubusercontent.com/81597121/120585338-663b7880-c3e6-11eb-8a51-fb76a0c0126c.mov
How to reproduce the bug

Open treemap v2 chart
Click on the corner or edge of the chart
observe
See error

Environment
(please complete the following information):

superset version: master

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ ] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.

fixed https://github.com/apache-superset/superset-ui/pull/1151

Closing due to provided fix. Please re-open if the situation persists.

source: gharchive/issue

authors: jinghua-qa rusackas stephenLYZ

repo: apache/superset


text:
[feat]Develop a feature to export/import Row Level Security filters from one instance to another

Problem: We don’t want to manually create Row Level Security filters in each instance.
We are using flask fab export-roles and flask fab import-roles to export/import Roles with permissions. However, we are still facing the problem of exporting/importing Row Level Security filters from one instance to another in Superset.

Closing this as inactive, and because it's not a bug but a feature request. I'd recommend opening an Ideas thread in Discussions if you still want this, and we can ping a few people there to get the discussion rolling.

authors: lulugll rusackas


text:
Import dashboard/assets with overwrite flag does not replace dashboards

When importing a dashboard via /api/v1/dashboard/import/ with the ovewrite flag set to true, the charts within the dashboard are merged with existing charts. I would expect if passing the overwrite flag that whatever you upload would replace the existing dashboard. This is useful for a version control where you want the .yml in the repository to exactly reflect what is in the superset database. This is also the same for the /api/v1/assets/import/ API although this doesn't take an overwrite parameter.
How to reproduce the bug

Export dashboard with chartA and chartB
Unzip exported file
Edit yml to remove /charts/chartB.yml and remove references to it in /dashboards/dashboard.yml
Zip up files and upload with overwrite flag
Dashboard is updated but still contains both charts

Expected results
Expect that dashboards should be replaced when imported and the overwrite flag is passed.
Actual results
Imported dashboards seem to be merged with what is already in the database.
Screenshots
N/A
Environment
(please complete the following information):

browser type and version: firefox 106.0.5 but this is also the behaviour in the api
superset version: 2.0.1rc2
python version: 3.9.7
node.js version: 16.14.2
any feature flags active:

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[ ] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Slack discussion: https://apache-superset.slack.com/archives/C015WAZL0KH/p1667918679617939

I'm running into this as well (superset 2.0.0) by using the import / export feature from the UI.
The example given by @cleslie is about removing a chart, but the same things happens for updates :

Export Dashboard through the UI with chartA, datasetA
Modify dashboard, chartA, datasetA
Import back dashboard file obtained in step 1, OVERWRITE is specified in the UI
Dashboard configuration is rolled back, chartA and datasetA are left in their state after (2)

Manually extracting datasets / charts from the archive, and importing them through the dedicated dataset / chart import tools applies updates as expected
I would add it may be interesting to give the user a chance to select what actually gets updated. For my specific use case I would like charts / datasets to be updated as dependencies of the dashboard, but I need databases to remain untouched (dashboards are moved between staging / production environements which use different source databases)

any solution for this issue.

Still have this issue
Will the above PR be reviewed anytime soon?

https://github.com/apache/superset/pull/23489 aims to fix this, but needs a rebase, a PR description, and some reviews. If anyone wants to review that code, test the solution, or open a rebased PR based on it, please go for it! Thanks in advance :D
CC @betodealmeida for awareness of import/export issues, as usual :D

Anyone still facing this? Tempted to close it as stale, but it's probably still an issue. Anyone who wants to open a PR here (including @Naveench131) would be appreciated :)

Yeah this is still an issue. The workaround that we've been doing on our end is to export as dashboards, change the type in the metadata file from Dashboard to assets, and then just use the assets API to import, though that obviously isn't ideal. I had a PR over in https://github.com/apache/superset/pull/25102 but it's a year old and I'm not certain that fix still works. I'll see if I get some time this or next week to fix it up.

authors: ashayargal cleslie ey3ball lindenh musicmuthu rusackas


text:
Time range filter on Clickhouse datasource with DateTime(timezone) column

When I'm trying to use time range filter on chart built on top of datasource from ClickHouse that has DateTime('UTC') as a time column it fails.
How to reproduce the bug

Create clickhouse table with DateTime('UTC') column and make a dataset in Superset from it. Create any chart and put it on the dashboard
Create time range filter and apply it
See error

Expected results
I've expected the values to be correctly filtered
Actual results
After applying the filter all the charts that have this column as time column disappear and the error is shown
Screenshots

Environment
(please complete the following information):

browser type and version: Chrome Version 112.0.5615.137 (Official Build) (arm64)
superset version: 2.0.1
python version: 3.8.12
node.js version: not relevant
any feature flags active: no

Checklist
Completed
Additional context
superset logs for the error
2023-04-28 08:57:53,906:ERROR:clickhouse_connect.driver.httpclient:Code: 53. DB::Exception: Cannot convert string 2023-04-21 00:00:00.000000 to type DateTime('UTC'): while executing 'FUNCTION greaterOrEquals(fact_new_at : 1, '2023-04-21 00:00:00.000000' : 2) -> greaterOrEquals(fact_new_at, '2023-04-21 00:00:00.000000') UInt8 : 4'. (TYPE_MISMATCH) (version 22.3.19.6 (official build))
Query SELECT toStartOfDay(toDateTime(fact_new_at)) AS __timestamp,
count(order_id) AS COUNT(order_id)
FROM marts.fact_orders
WHERE fact_new_at >= '2023-04-21 00:00:00.000000'
AND fact_new_at < '2023-04-28 00:00:00.000000'
GROUP BY toStartOfDay(toDateTime(fact_new_at))
ORDER BY COUNT(order_id) DESC
LIMIT 10000 on schema marts failed
Traceback (most recent call last):
File "/app/superset/connectors/sqla/models.py", line 1920, in query
df = self.database.get_df(sql, self.schema, mutator=assign_column_label)
File "/app/superset/models/core.py", line 458, in get_df
self.db_engine_spec.execute(cursor, sqls[-1])
File "/app/superset/db_engine_specs/base.py", line 1261, in execute
raise cls.get_dbapi_mapped_exception(ex)
File "/app/superset/db_engine_specs/base.py", line 1259, in execute
cursor.execute(query)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/dbapi/cursor.py", line 50, in execute
query_result = self.client.query(operation, parameters)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/client.py", line 178, in query
return self._query_with_context(query_context)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/httpclient.py", line 207, in _query_with_context
response = self._raw_request(body,
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/httpclient.py", line 387, in _raw_request
self._error_handler(response)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/httpclient.py", line 323, in _error_handler
raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for http://chi-simple-nonreplicated-0-0.stable.svc.cluster.local:8123/ returned response code 400)
Code: 53. DB::Exception: Cannot convert string 2023-04-21 00:00:00.000000 to type DateTime('UTC'): while executing 'FUNCTION greaterOrEquals(fact_new_at : 1, '2023-04-21 00:00:00.000000' : 2) -> greaterOrEquals(fact_new_at, '2023-04-21 00:0
2023-04-28 08:57:53,907:WARNING:superset.connectors.sqla.models:Query SELECT toStartOfDay(toDateTime(fact_new_at)) AS __timestamp,
count(order_id) AS COUNT(order_id)
FROM marts.fact_orders
WHERE fact_new_at >= '2023-04-21 00:00:00.000000'
AND fact_new_at < '2023-04-28 00:00:00.000000'
GROUP BY toStartOfDay(toDateTime(fact_new_at))
ORDER BY COUNT(order_id) DESC
LIMIT 10000 on schema marts failed
Traceback (most recent call last):
File "/app/superset/connectors/sqla/models.py", line 1920, in query
df = self.database.get_df(sql, self.schema, mutator=assign_column_label)
File "/app/superset/models/core.py", line 458, in get_df
self.db_engine_spec.execute(cursor, sqls[-1])
File "/app/superset/db_engine_specs/base.py", line 1261, in execute
raise cls.get_dbapi_mapped_exception(ex)
File "/app/superset/db_engine_specs/base.py", line 1259, in execute
cursor.execute(query)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/dbapi/cursor.py", line 50, in execute
query_result = self.client.query(operation, parameters)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/client.py", line 178, in query
return self._query_with_context(query_context)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/httpclient.py", line 207, in _query_with_context
response = self._raw_request(body,
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/httpclient.py", line 387, in _raw_request
self._error_handler(response)
File "/usr/local/lib/python3.8/site-packages/clickhouse_connect/driver/httpclient.py", line 323, in _error_handler
raise OperationalError(err_str) if retried else DatabaseError(err_str) from None
clickhouse_connect.driver.exceptions.DatabaseError: :HTTPDriver for http://chi-simple-nonreplicated-0-0.stable.svc.cluster.local:8123/ returned response code 400)
Code: 53. DB::Exception: Cannot convert string 2023-04-21 00:00:00.000000 to type DateTime('UTC'): while executing 'FUNCTION greaterOrEquals(fact_new_at : 1, '2023-04-21 00:00:00.000000' : 2) -> greaterOrEquals(fact_new_at, '2023-04-21 00:0

See also my discussion of this behavior in clickhouse-connect repo https://github.com/ClickHouse/clickhouse-connect/issues/177

Hi @maratbakiev2! Can you test the issue in 2.1.0 version of superset?

@EugeneTorap I'll try to do it in the nearest future and come back here

Hi @villebro @sebastianliebscher! Do you know who to fix it?

@EugeneTorap sorry for the delay. I've tested it in 2.1.0. No errors :)
So it could be closed. At least from my point of view.

Let's close this as @maratbakiev2 can't reproduce on 2.1.0 (thanks for reporting back!)  If anyone else disagrees / can reproduce it on a current version, please say so.

authors: EugeneTorap maratbakiev2 sfirke


text:
refactor: Move fields from Extra section

SUMMARY
Move the existing fields from the "Extra" section to the following sections. This also includes moving them to be fields instead of one json string.

SQL Lab

allows_virtual_table_explore

Performance

Metadata Cache Timeout with Schema and Table options (metadata_cache_timeout)

Security

Schema select for uploading data (schemas_allowed_for_csv_upload)
Impersonate logged in user
Allow data upload

Other

engine_params and metadata params (in a json field)
version

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

BEFORE
Extra

SQL Lab

Performance

Security

TEST PLAN

From the "Databases" page, open the database modal by clicking either the "+ DATABASE" button or the "Edit" action button on any listed database.
Observe the described changes on the following tabs: Other (formerly Extra), SQL Lab, Performance, Security

ADDITIONAL INFORMATION

[x] Has associated issue: Clubhouse Ticket
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Continued at #14293


text:
chore: run babel_update.sh to update po files

Summary
Periodically we need to run this script to rebuild the list of translatable strings from static analysis of the code base.
Also adding --no-location to pybabel extract fixing merge conflicts issues we've been having around these .po files.
Also fixing the docs builds, they've been broken.
About the fact that merge conflicts are really tricky around translation work in general, I was thinking it may be doable to write a simple utility that would strip all comments from po files, as they are just comments referencing where the msgid exists, and operate at that level if/when needed. Like you hit a merge conflict, strip both files from comment, diff them, and commit without comments. I think the next pybabel script run would bring back the comments.
~~Maybe we even want to pass a flag to pybabel to compeltely avoid adding source references as comments. We'd loose the ability to easily find where the string lives in the codebase, but for that you can just git grep. ~~

Oh wow look at that ->


text:
fix(Jinja): Extra cache keys for calculated columns and metrics using Jinja

SUMMARY
This is a follow up to https://github.com/apache/superset/pull/30715. It adds logic to check Jinja in calculated columns and also metrics (both ad-hoc and from the dataset level).
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
No UI changes.
TESTING INSTRUCTIONS
Added testing to this flow.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@villebro thanks for initial pass 🙏 I addressed your feedback + removed a redundant import. Let me know if you have any questions!


text:
[DOC] Documentation for builtin toOnehot function

This pull request adds the documentation  for builtin toOnehot function.

Thank you, @stanp7 - I will review today.

Thank you, @stanp7 - LGTM. 👍
🎉 🚀

repo: apache/systemml


text:
THRIFT-4237 Fix data races in Go TServerSocket

Incorrect lock semantics resulted in a race condition in the TServerSocket
implementation. This commit adds a unit test that detects this, and fixes up
the locking semantics.
https://issues.apache.org/jira/browse/THRIFT-4237

@Jens-G it appears I've introduced some issue that causes the tests to hang. Is there a good way to run just the Go tests in verbose mode? I've connected directly to a shell in the docker build container but haven't yet found a way.

Does running make check from /thrift/src/test/go not do what you want?

$ pwd
/Users/zwass/dev/go/src/git.apache.org/thrift.git/lib/go
$ make check
 cd ../.. && /bin/bash /thrift/src/missing automake-1.14 --foreign lib/go/Makefile
/bin/bash: /thrift/src/missing: No such file or directory
make: *** [Makefile.in] Error 127

authors: econner zwass

repo: apache/thrift


text:
TOMEE-2896 Translate to Portuguese: examples/tomee-jersey-eclipselink

Translate document Readme example

thanks @sercheo87 for translate and @junioorcelistre  for review.

repo: apache/tomee


text:
Add Traffic Monitor Uptime API and UI

Add a Traffic Monitor uptime time and/or start timestamp API, and add to the HTML UI.
This will make it easier to debug, to know if someone else has restarted the monitor, or if the SystemD service restarted it. This will also make it easier for operators to know at a glance if there are recent or ongoing problems requiring restarts.
This should be easy to add, it's a trivial endpoint and a few lines of js/html.

Incidentally, I'm creating this from a need in practice, not theory. I recently fixed a nil panic issue, and it would have been helpful to see for certain that monitors hadn't restarted. This isn't the first time, either.

repo: apache/trafficcontrol


text:
Add TCP accept metric which tracks the total number of TCP connections accepted.

proxy.process.net.tcp.accept
This is useful to track the rate of inbound connections accepted. This can be compared to transactions, open connections, or other OS network metrics to better identify performance issues.

Linux build successful! See https://ci.trafficserver.apache.org/job/linux-github/1457/ for details.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/freebsd-github/1563/ for details.

clang-analyzer build successful! See https://ci.trafficserver.apache.org/job/clang-analyzer-github/128/ for details.

Why are there tsconfig changes with this?

Bad merge, I'll fix the TSConfig stuff.

Intel CC build successful! See https://ci.trafficserver.apache.org/job/icc-github/6/ for details.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/freebsd-github/1573/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/linux-github/1467/ for details.

clang-analyzer build successful! See https://ci.trafficserver.apache.org/job/clang-analyzer-github/138/ for details.

Hmmm, how is this different from proxy.process.http.total_incoming_connections ?

The http stat measures only HTTP 1.X connections and only those that are successfully created. If, for example, there is a connection that fails the TLS handshake, this stat will record it but http.total_incoming_connections will not. Tunnels will also be tracked by tcp_accept and not by the http stat.
It also looks like HTTP 2 connections are not tracked by that stat either, which is also distinct. It might be reasonable to either make the existing stat track both or track them independently and sum.

It seems like we should have some sort of layered approach, where we keep track of tcp (l4) http (L5?) and TLS (L7). We have metrics for some of this it sounds, but IIRC they are a bit patchy.
So I'm in favor of a raw "tcp accepted", and IMO the http incoming_connections should probably be 1/2 agnostic (if we want an http_stream_accepted-- we can do that too).

Yes, if there are subtle differences like this, we should break them out accordingly, and then make an aggregated "sum" metrics in the Lua metrics.config.
It almost sounds like a bug / misfeature that the H2 connection is not included in the proxy.process.http.total_incoming_connections. And it's very inconsistent with other metrics; Whereas these two connection metrics works as you describe (independent), the request metrics are not. E.g.
Sending 4 HTTP/2 request, I see this:
./bin/traffic_ctl metric get proxy.process.http2.total_client_streams proxy.process.https.incoming_requests proxy.process.http.incoming_requests
proxy.process.http2.total_client_streams 4
proxy.process.https.incoming_requests 4
proxy.process.http.incoming_requests 4

Note how each H2 stream is counted as an HTTPS request, as well as an HTTP request (I'm guessing because of how H2 uses HTTP internally). This seems rather inconsistent to me, but does make some sense I guess

http2 connections are tracked by proxy.process.http2.current_client_sessions and proxy.process.http2.total_client_connections.
All successfully negotiated SSL/TCP connections would be the sum of proxy.process.http2.total_client_connections and proxy.process.http.total_client_connections

We should have a metric that tracks the total number of accepts at the TCP layer.  I recommend that the name be inline with the current metrics proxy.process.net.total_accepts

@bryancall - do you mean you want proxy.process.net.tcp.total_accepts instead of proxy.process.net.tcp.accepts?

AU check failed! https://ci.trafficserver.apache.org/job/autest-github/279/

clang format successful! https://ci.trafficserver.apache.org/job/clang-format-github/283/

RAT check successful! https://ci.trafficserver.apache.org/job/RAT-github/297/

Linux build successful! https://ci.trafficserver.apache.org/job/linux-github/1870/

Intel CC build successful! https://ci.trafficserver.apache.org/job/icc-github/409/

FreeBSD11 build successful! https://ci.trafficserver.apache.org/job/freebsd-github/1978/

clang-analyzer build failed! https://ci.trafficserver.apache.org/job/clang-analyzer-github/542/

authors: SolidWallOfCode atsci bryancall jacksontj shinrich zwoop

repo: apache/trafficserver


text:
[Bug] Unable to install PAPI dependency 'linux-tools-generic'

Expected behavior
Successfully install linux-tools-generic in Docker containers based on Ubuntu Jammy 22.04.02, when running the script ubuntu_install_papi.sh.
Actual behavior
Apt install fails with the message:
The following packages have unmet dependencies:
 linux-tools-generic : Depends: linux-tools-5.15.0-76-generic but it is not installable
E: Unable to correct problems, you have held broken packages.

The command to install linux-tools-generic is ambiguous, so as to pick up the latest available version. At the moment however, it is fetching version 5.15.0.76-generic which is not yet present in the Ubuntu apt repositories. Currently, the latest available version in 22.04.02 is 5.15.0.75-generic.
Environment
Ubuntu Jammy 22.04.02 (amd64).
Steps to reproduce
Can be reproduced by building a new ci_cpu Docker container, or in a vanilla Ubuntu Jammy 22.04.02 VM.
Triage

needs-triage
dev:ci

Issue has now been resolved in Ubuntu package repository.

repo: apache/tvm


text:
Fix out of bound enum conversion

This PR fixes Wenum-constexpr-conversion which is introduced in recent version of clang. Explciitly use integer instead of enum to store the device that may contain invalid options.

cc @Wheest background https://discuss.tvm.apache.org/t/tvm-web-build-error-and-unrecognised-target/14324

cc @Wheest background https://discuss.tvm.apache.org/t/tvm-web-build-error-and-unrecognised-target/14324

Cheers, pulling this PR allows me to run make in web/ without the error being triggered.

authors: Wheest tqchen


text:
[Unity] Added known tir.Expr to relax.PrimValue

Prior to this commit, a relax.PrimValue could have a datatype, but couldn't have a corresponding tir.PrimExpr.  As a result, it could not be used to specify tensor shapes.  This makes some expressions require fallback to R.Tensor(ndim=ndim), even though the shape could still be inferred.
# Prior to this commit, attempts to use `R.Prim` when
# a `tir.PrimExpr` is required would result in an error.
@R.function
def func(
    A: R.Tensor(16, 16),
    first_n_rows: R.Prim("int64"),
) -> R.Tensor([first_n_rows, 16]):
    #          ^^^^^^^^^^^^
    #          R.Tensor requires a PrimExpr, not relax.Expr
    #
    #                               Operations may require PrimExpr
    #                                                  vvvvvvvvvvvv
    out = R.op.strided_slice(axis=[0], begin=[0], end=[first_n_rows])
    return out

This commit adds a Optional<PrimExpr> value field to the PrimStructInfo.  This field acts similarly to the PrimExpr fields already used in ShapeStructInfo, and may contain symbolic variables.
@R.function
def func(
    A: R.Tensor(16, 16),

    # TIR definitions in signature allow in-line definitions,
    # similar to R.Tensor and R.Shape.  R.Prim takes `dtype` or
    # `value` kwarg to distinguish between in-line symbolic variable
    # and string representation of dtype.
    first_n_rows: R.prim(value="first_n_rows_tir"),
) -> R.Tensor(["first_n_rows_tir", 16]):

    # Body contains a TIR variable definition, which may be used
    # in function calls, inferred shape annotations.
    first_n_rows_tir = T.int64()
    out = R.op.strided_slice(axis=[0], begin=[0], end=[first_n_rows])
    return out

@Hzfengsy Thank you, and I do think the value should be a tir.PrimExpr and not just a tir.Var for a few reasons.

API consistency when applying BindSymbolicVars.  When a function is updated using BindSymbolicVars, the allowed values for each parameter are restricted, but the calling convention is otherwise kept identical.  For example, binding {n: 16} would reduce the set of allowed values for R.Tensor([n, n]) (the set of all square tensors) to R.Tensor([16, 16]) (the set of all tensors of shape [16,16]), and remains part of the function signature.  Similarly, a PrimValue parameter would be reduced from R.Prim(dtype='int64', value=n) (the set of all 64-bit integers) to R.Prim(dtype='int64', value=16) (the set containing only the integer 16), and remains part of the function signature.
If R.Prim were instead restricted to a tir.Var, it wouldn't be possible to write R.Prim(dtype='int64', value=16), and the parameter would need to be removed entirely.  As a result, the calling scope would need to be updated (e.g. from my_func(tensor, 16) to my_func(tensor)).

API consistency with R.Shape.  Similar to a R.Prim, a R.Shape can contain only static parameters.  However, for similar reasons, parameters of type R.Shape([n, n]) are specialized to R.Shape([16,16]) and retained in the function signature, even though they only have a single legal value of ShapeTuple([16,16]).

Usefulness as constituents in other StructInfo objects.  Currently, many types of StructInfo may internally contain a PrimExpr (e.g. TensorStructInfo, ShapeStructInfo), and so visitors that inspect the struct info for PrimExpr instances must handle each case independently, performing the handoff from StructInfoFunctor to tir::ExprFunctor.  If these were instead expressed in terms of PrimStructInfo, this would only need to occur in a single location.  Because TensorStructInfo and ShapeStructInfo may contain either static values or expressions in terms of other symbolic variables, this potential usage would require PrimStructInfo to also contain these values.


text:
[CUBLAS][FP8] Enable fusing astype operation for matmul multiply pattern

This PR adds fusing of the astype operation to matmul for cublas. This change is needed to improve the performance for fp8.

do we need to update cublas codegen or runtime to support the cast?


text:
[CI] add the --net=host cmd line arg to the docker/bash.it script

introduce the --net=host command line argument to the docker/bash.sh. This is already available
for https://github.com/apache/tvm/blob/main/docker/build.sh#L71
Without this change it is not possible to have an arbitrary command executed through docker?bash.sh
and have interactivity enabled since the following section https://github.com/apache/tvm/blob/main/docker/bash.sh#L52
imposes running bash and overwrites the value of COMMAND.
With this change, one would be able to have bridged host networking in addition to executing an
arbitrary command.
on another note, i think it would be quite helpful to check if e.g CI_DOCKER_EXTRA_PARAMS is defined
as an environment variable, then docker/bash.sh would append variables, so the calling sequence would be:
    export CI_DOCKER_EXTRA_PARAMS="--net=host -v foo:bar .... some_other_docker_flags"
    ./docker/bash.sh -i tvm.ci_cpu SOME_COMMANDS_HERE

Thank you @mherkazandjian !

@tqchen i can spend some time implementing the option that i described:
on another note, i think it would be quite helpful to check if e.g CI_DOCKER_EXTRA_PARAMS is defined
as an environment variable, then docker/bash.sh would append variables, so the calling sequence would be:

do you think this would be merged or not worth implementing?

authors: mherkazandjian tqchen


text:
[ROCM] Fix undefined symbols by adding library

PR #8178 added a check for undefined symbols when the target was linked.
This broke the ROCm build because some symbols were resolved by libhsaruntime64.so which is present at execution time, but not present at link time.  This PR fixes the unresolved symbol issue by linking with libhsaruntime64.so if present.

Thanks @mvermeulen I've also hit this problem.
cc @Lunderberg

@mvermeulen Thank you for the fix!  I had initially thought that the CI would catch a failing compile, but it looks like the CI's OSX compilation only has LLVM/Metal runtimes enabled, and not ROCm.

authors: Lunderberg masahi mvermeulen


text:
[YUNIKORN-522] smoke test for dynamic queue clean up

What is this PR for?
Add a TestDynamicQueueCleanUp function to smoke_test.go.
When app is Completed, partition manager should be able to clean up the dynamically created queue.
What type of PR is it?

[ ] - Bug Fix
[x] - Improvement
[ ] - Feature
[ ] - Documentation
[ ] - Hot Fix
[ ] - Refactoring

Todos

[ ] - Task

What is the Jira issue?
https://issues.apache.org/jira/projects/YUNIKORN/issues/YUNIKORN-522
How should this be tested?
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

@craigcondit Thank you for your review.
Add a SetCompletingTimeout function in pkg/scheduler/objects/application.go, change 30 seconds to 100 milliseconds.

repo: apache/yunikorn-core


text:
[ZEPPELIN-1461] Update Flink with latest version 1.1.2

What is this PR for?
Flink has had two releases since 1.0.3, we are now on 1.1.2
This includes new functionality for streaming support in repl environment.
What type of PR is it?
Improvement
Todos

[x] - Update pom.xml
[x] - Update single (batch) environment to batch and streaming environments

What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1416?filter=-1
How should this be tested?
Tests for previous versions are the same as new version.
Screenshots (if appropriate)
Questions:

Does the licenses files need update?
No
Is there breaking changes for older versions?
Yes* older code written in the Flink interpreter will now have to use benv in place of env
Does this needs documentation?
No

Changes look good to me. +1 for merging after fixing the failing test case.

Flink Tests all passing now.
Profile3 Cassandra Test Timeout Unrelated
Profile8
- should provide onclick method *** FAILED ***
Display system 'onclick' method timeout. Known flaky test (I'll find it and link soon)
LGTM?

@Leemoonsoo all systems go on re-triggered build.

Thanks @rawkintrevo, @tillrohrmann. It looks good to me!

Merge to master and branch-0.6, if there're no more dicussions

authors: Leemoonsoo rawkintrevo tillrohrmann

repo: apache/zeppelin

license_source: bigquery


text:
ZEPPELIN-2161 Nested Group Support in LdapRealm for AD

What is this PR for?
A common use case in LDAP/AD setup is the hierarchical structuring of
groups - a.k.a. adding groups to other groups. Such nesting groups can
help reduce the number of roles that need to be managed.
Current zeppelin realm implementations doesn't have support for looking
up memberships throughout nested group structures.
E.g. consider the following nested group scenario:
acme_employees
 \__department_a
     \__sub_department_x

User 'bob' is in Group 'sub_department_x'.
Notebook 'note1' has a Reader Role assignment for 'department_a' or
'acme_employees'.
Then access must be granted for 'bob' on 'note1'.
In AD enviroments this scenarios can be efficiently implemented using
the so called LDAP_MATCHING_RULE_IN_CHAIN operator
'1.2.840.113556.1.4.1941'.
This PR introduces a property 'groupSearchEnableMatchingRuleInChain' to
org.apache.zeppelin.realm.LdapRealm which defaults to false. If enabled,
all roles of potential nested group hierarchies will be resolved using
the LDAP_MATCHING_RULE_IN_CHAIN operator.
What type of PR is it?
Improvement
Todos

What is the Jira issue?
[ZEPPELIN-2161]
How should this be tested?
Set groupSearchEnableMatchingRuleInChain = true for the ldap realm.
Screenshots (if appropriate)
Questions:

Does the licenses files need update? n
Is there breaking changes for older versions? n
Does this needs documentation? y

any idea why the build has failed?


text:
Extract unused contracts to an external repo

In reference to https://api3workspace.slack.com/archives/C02AYRX8D89/p1680698758234279
There are a lot of contracts in @api3/airnode-protocol that are not used in practice. Some of these are less viable alternatives to RequesterAuthorizerWithErc721. Once we have full RequesterAuthorizerWithErc721 support, we should remove these alternative user flows in favor of simplicity.

For issue tracking purposes, the RequesterAuthorizerWithErc721 support issue is here: https://github.com/api3dao/airnode/issues/1675

authors: bbenligiray dcroote

license: MIT


text:
_type endpoints reservedParameter is required

Currently OIS validation allows an empty reservedParameters endpoints array. I confirmed, based on a reported issue on discord, that gateway requests to an Airnode deployed with such an empty array causes a difficult to debug error: {"error":"Error: Cannot read property 'length' of undefined"}.
See this and subsequent comments for a discussion on needing _type and not merely checking for a non-empty array.

@Siegrift - shall we release v1.1.2?

Yeah, we should.

Just FYI v1.1.2 has just been published: https://www.npmjs.com/package/@api3/ois
and https://github.com/api3dao/ois/commit/52fdfa7e44597979208e8ed877bf83b7363dee07
:)

Hi @Siegrift, as per this thread, is this being retired for the time being?

Hi, I've tested this by installing the package via npm and testing according to the usage information in the OIS repo. The package can be installed successfully, shows the correct version, and checks provided JSON against the OIS specification.

Thanks @martinkolenic!

authors: Siegrift aquarat dcroote martinkolenic


text:
docs: fix outdated docs re: create new actor button

add new screenshot with correct UI
change prose to reflect the current state of UI

The original create-new-actor.png is linked from other pages, or could it be removed now? I don't see it removed it the PR.

authors: TC-MO honzajavorek

repo: apify/apify-docs


text:
feat: add SDK and Crawlee version to the ApifyClient's user agent

Closes #331

@mnmkng This adds both SDK and Crawlee versions to the UA, since SDK uses code from crawlee, it's technically always there (and we validate that the version it uses is the same), do you think we should differentiate calls just from SDK and those from crawlee? I wouldn't care about that personally, especially since users can use the static Actor... methods in crawlee projects too, so what would be trying to differentiate in the first place? The only place crawlee uses the client internally is to communicate with the storages, that is handled via the Actor.init call, and that comes from the user (and will be there even without crawlee being in use explicitly), so I am not sure if/how could this be done.

I'll merge this in, we can always improve later.

repo: apify/apify-sdk-js


text:
swagger-ui middleware redirects without trailing slash

Whenever you access /docs, it redirects (or attempts to) to /docs/ and the initially displayed page never fully renders.  I know that the node-static middleware we're using has a configuration to disable the redirect but we also could benefit from figuring out why this is happening.  This originally was found while working on #339.

+1

I cannot reproduce this with Connect or Express but with Restify, I see the issue.  I think the redirect that the swagger-ui middleware is performing is not handled by Restify since Restify's res.redirect is different than Connect/Express.  That being said, I cannot fix this in swagger-tools.  But I was able to get this working using a simple Restify redirect (modified from #339):
'use strict';

var restify = require('restify')
var server = restify.createServer();
var swaggerTools = require('swagger-tools');

var serverPort = 3000;

// swaggerRouter configuration
var options = {
  controllers: './controllers',
  useStubs: process.env.NODE_ENV === 'development' ? true : false // Conditionally turn on stubs (mock mode)
};

// The Swagger document (require it, build it programmatically, fetch it from a URL, ...)
var swaggerDoc = require('./api/swagger.json');

// Initialize the Swagger middleware
swaggerTools.initializeMiddleware(swaggerDoc, function (middleware) {
  var middlewares = [
    middleware.swaggerMetadata(),
    middleware.swaggerValidator(),
    middleware.swaggerRouter(options),
    middleware.swaggerUi()
  ];

  // Required since swagger-ui uses CORS to fetch the Swagger document
  server.use(restify.CORS());
  // Use Restify versions of body/query parsing middlewares
  server.use(restify.queryParser());
  server.use(restify.jsonp());
  server.use(restify.bodyParser());

  // Fix issue with swagger-ui redirect not working with Restify (Not sure who is to blame here)
  server.get('/docs', function (req, res, next) {
    res.redirect('/docs/', next);
  });

  // Supported Swagger HTTP verbs (options and patch are not Restify server APIs)
  // https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#path-item-object
  ['get', 'put', 'post', 'delete', 'head'].forEach(function (verb) {
    if (verb === 'delete') {
      verb = 'del';
    }

    // Register the middlewares with Restify
    server[verb](/.*/, middleware.swaggerMetadata(), middleware.swaggerValidator(), middleware.swaggerRouter(options), middleware.swaggerUi());
  });

  // Start the server
  server.listen(serverPort, function () {
    console.log('Your server is listening on port %d (http://localhost:%d)', serverPort, serverPort);
  });
});

I think at the end of the day, the middleware in swagger-tools is written to be Connect compatible, which Restify is not.  So while I'm sure we can make swagger-tools work just fine with Restify, there may be some hoops you have to jump through to do so.  (The code above was able to make all functionality of the swagger-tools 2.0 example work using Restify.)

      server.get(`${BaseUrl}/docs`, (request, response, next) => {
        response.redirect(`${BaseUrl}/docs/`, next);
      });

      [`get`, `put`, `post`, `del`, `head`].forEach((verb) => {
        server[verb](/.*/, middleware.swaggerUi({
          swaggerUi: `${BaseUrl}/docs`,
          apiDocs: `${BaseUrl}/api-docs`
        }));
      });

Not working. getting Error 310 (net::ERR_TOO_MANY_REDIRECTS) - because /docs redirects to /docs/, than /docs/ redirects to /docs because of swaggerUi: `${BaseUrl}/docs

I'm sorry but this is an issue with Restify and the serve-static middleware not working with this.  swagger-tools cannot work around this.  I'm not sure if the options you're passing are causing the issue but since you're passing in the default values, you might as well just pass no options.  Does that help?  Also, it seems you're only using swagger-tools for its swagger-ui middleware.  If that's the case, save yourself the trouble and just use Restify's built-in static file serving and serve swagger-ui yourself.  I only suggest this because I'm sure using Restify to do this would be much simpler than trying to work around it.

authors: lironess whitlockjc

license: mit


text:
🛑 Oceania is down

In a5f198a, Oceania (https://epembelajaran.umt.edu.my/oceania/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Oceania is back up in 5a133dd.

repo: apikmeister/umt-web-uptime


text:
Install and configure Prometheus for internal service monitoring

Install Prometheus monitoring service to a VM.
Configure the service to monitor apinf.io, api umbrella and other internal services as agreed with @ccsr. Make sure that email alerts are created when services are down.

@bajiat, @ccsr Am I right understood that we need these metrics:

Shutdown with email alert.
CPU, disk I/O, network statistics.

For second requirement I plan use https://github.com/prometheus/node_exporter

@bajiat @brylie I configured Prometheus + Graphana for collect server metrics, now Prometheus monitor sever where it installed. The next step is install node_exporters on other internal servers.

We have 2 options for install node_exporter:

From source code.
With Docker image based on BusyBox.

Installation with Docker looks simpler but require Docker on each monitoring server.
What way will be preferable for that task?

Is there an option for passive monitoring? E.g. ping check?

I will try configure only alert about downtime (not tried). By default Prometheus want url where it can fetch metrics.

See related user story:

As an Apinf user I want to see the percentage of failures/uptime (timeout, bad gateway, internal server error) for a given API so that I can determine the reliability of a given API backend

OK, here is the related user story:

As an Apinf administrator I want to be notified if any of the Apinf platform services are down

@brylie Hm, We splitted original story https://github.com/apinf/docs/pull/7#issuecomment-223569828
By this story we need observer internal services not API backends.

OK, cool. Do you have the access privileges you need on our Apinf services?

Ping @ccsr

@brylie I have access to servers.
Before I will install node_exporter on these servers let's double check:

What metrics we need?

a) Shutdown with email alert.
b) CPU, disk I/O, network statistics.
I think both?
Is it OK install Docker for run "node_exporter" or will be better installation from source code?

Hm, it might depend on the server setup. E.g. apinf.io has an Apinf Docker deployment. Would the two services conflict in any way? Also, there is a Meteor package for Prometheus stats, that we might be able to include at some point.

@brylie Will not have conflict, "node_export" can be runed on any port, usually on :9100.
Good point about "Meteor package for Prometheus".

OK, cool. @ccsr is double checking our server list, and will make sure you have the credentials you need.

@bajiat I tested and merged @shaliko 's work https://github.com/apinf/monitoring/pull/1
I suppose this could be closed or is there still something to do?

@jykae I not started monitoring agent on 2 servers, because had issue with previous docker installation. Waiter @brylie for consultation (scare change old running soft), today will manage last 2 servers and then we can close that task.

@shaliko should we close this task?

@shaliko, should we close this task?

@nnn Yes

authors: bajiat brylie jykae nnn shaliko

repo: apinf/api-umbrella-dashboard


text:
in Consumes and Produces section, for more than 2 MIME types, if intermediate fields are not filled up, preview shows only 1st value

Bug has been fixed with the new UI.
Closing the issue.

repo: apinf/open-api-designer


text:
[Bug]: App interrupts streaming and freezes when trying to stop stream

Version
v1.2.3
Which operating systems have you used?

[X] Android
[ ] iOS

Environment that reproduces the issue
Samsung Galaxy S9 - Android 10
Is it reproducible in the example application?
Yes
RTMP Server
rtmp://***.in.streamster.io/in (key: ***)
Reproduction steps

Start streaming with any parameters
After some time (usually around 10-15 seconds) streaming will stop - server will not receive data
Preview will still work fine
Try to stop streaming, app will freeze but preview will be visible

Expected result
App does not freeze
Actual result
App freezes, no logs on react-native side, no errors in logcat, only anr crash report is saved
https://gist.github.com/Pitros/e3d1239dd1875538d57d3b7725bbf281
Additional context
Streaming is also stuttering on this device, other phone (Galaxy S20) does not stutter and does not have this freeze issue.
Relevant logs output
No response

Hi,
There seems to have a lot a lock thread in the ANR. It is hard to see what is wrong as there is a mix with your application threads.
Could you try to reproduce directly with our internal Android library: https://github.com/ThibaultBee/StreamPack/suites/10371571998/artifacts/511567553?

About the stuttering, could you test locally with ffmpeg:
ffplay -listen 1 -i rtmp://0.0.0.0:1935/s/streamKey

And on your phone, set the RTMP url to your computer IP.

@ThibaultBee so i checked stuttering locally and it works fine, totally smooth, but still with rmtp server from restream.io it's having difficulties. Every few seconds there's pause for second or multiple seconds. Also there's warning:
Check your encoder settings
Keyframe interval: 0.589
Recommended: 2

Tried to reproduce those stutterings on emulator, seems like Pixel 5 API 30 has some issues, not sure if that will apply to you as well.
Here's crash log from your app related to app not responding:
https://gist.github.com/Pitros/48a0a8bc4a1575509c5e59dddb69084c

Not sure it is going to fix your issue, but to respect the key frame interval required, you can set:
video.gopDuration = 2

See README.md:
video: {
    ...
    // duration between 2 key frames in seconds
    gopDuration: number;
  };

I am also seeing a similar issue with the app and using the RTMP server from https://github.com/gwuhaolin/livego.
The server throws an error while using the library
WARN[2023-02-23T11:11:42+05:30] rtmp.go:419 github.com/gwuhaolin/livego/protocol/rtmp.(*VirReader).Read.func1() rtmp read packet panic: runtime error: slice bounds out of range [:12910591] with capacity 512000

but other libraries like https://github.com/NodeMedia/react-native-nodemediaclient working fine with the server.

Any update ?
Having the same issue with a Samsung Galaxy S9 on Android 10 too

I saw that it depends on whether we use the front or back camera too.
When i use the phone's back camera with a bitrate of 2000000 and a gopDuration equal to 1 I can normaly stream but if I try with the front camera I can't view my stream on my NodeMediaServer.

Ok, I have the feeling there are several issues described here.
Let's focus on the freeze on stopStream. Please open other issues for any other odd behavior.
Is the freeze on stopStream random? Does it only appear on your S9/Android 10?
As this library depends on another live streaming library, could you test directly on it? https://github.com/apivideo/api.video-android-live-stream. I can build it for you if it is easier.

Could you test the original issue with 2.0.0?

@ThibaultBee thanks for keeping the lib updated!
It seems like it's stable now, tried all sort of actions with updated lib and didn't crash. Also now bitrate can be changed dynamically.

Ok. Thanks. I am closing this.

authors: Pitros ThibaultBee gokugen tinkalgogoi


text:
AppIntro runs everytime I open the app

AppIntro Version: 4.2.2

Device/Android Version: Works great on every device. I use Yu Yureka as physical and Nexus 5X as emulated device for testing.

Issue details: I've implemented how to run the AppIntro slides only for the first time from the AppIntro Wiki. I also added the Manifest declaration  but still, everytime I open the app from my physical or emulated device, it runs each time! How do I resolve this?

Your Code: My MainActivity.java (actually it's EarthquakeActivity.java)-
//Start the intro slide with an intent
        Intent intent = new Intent(this, IntroActivity.class);
        startActivity(intent);

        /* Start the intro only once */
        //  Declare a new thread to do a preference check
        Thread t = new Thread(new Runnable() {
            @Override
            public void run() {
                //  Initialize SharedPreferences
                SharedPreferences getPrefs = PreferenceManager.getDefaultSharedPreferences(getBaseContext());

                //  Create a new boolean and preference and set it to true
                boolean isFirstStart = getPrefs.getBoolean("firstStart", true);

                //  If the activity has never started before...
                if (isFirstStart) {

                    //  Launch app intro
                    final Intent i = new Intent(EarthquakeActivity.this, IntroActivity.class);

                    runOnUiThread(new Runnable() {
                        @Override public void run() {
                            startActivity(i);
                        }
                    });

                    //  Make a new preferences editor
                    SharedPreferences.Editor e = getPrefs.edit();

                    //  Edit preference to make it false because we don't want this to run again
                    e.putBoolean("firstStart", false);

                    //  Apply changes
                    e.apply();
                }
            }
        });

        // Start the thread
        t.start();

My Manifest file-
<activity android:name="com.example.android.quakereport.IntroActivity"
            android:label="@string/app_intro" />

I've added a new String resource in Strings.xml-
<string name="app_intro">Intro</string>

I'm sorry, but this is not an AppIntro issue -- yes, looks like the wiki needs to be expanded further for those that are making use of preferences for the first time, but we can't take the time to guide everyone who uses the library on how to open an activity only when needed.
Stackoverflow has plenty of questions regarding this subject and it is the resource I recommend all developer to query (not post).

repo: apl-devs/AppIntro