text:
[ISSUE #8584] fix missing brokerName in sendMessageBack request

Which Issue(s) This PR Fixes

Fixes #8584
Brief Description

How Did You Test This Change?

Codecov Report
Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 47.19%. Comparing base (a1ea1eb) to head (60d7fc3).

Files
Patch %
Lines

...ocketmq/client/consumer/DefaultMQPullConsumer.java
0.00%
1 Missing :warning:

...ocketmq/client/consumer/DefaultMQPushConsumer.java
0.00%
1 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8585      +/-   ##
=============================================
- Coverage      47.27%   47.19%   -0.08%
+ Complexity     11467    11450      -17
=============================================
  Files           1277     1277
  Lines          89342    89342
  Branches       11480    11480
=============================================
- Hits           42234    42168      -66
- Misses         41918    41970      +52
- Partials        5190     5204      +14

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
[feature][datasource] Main add doris

Purpose of this pull request

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs

Please fix conflicts

Hi All, Is this this still going on ?

source: gharchive/pull-request

repo: apache/seatunnel-web

license_type: permissive

license_source: github-api


text:
[Bug] [website] Checkpoint-Storage description is incorrect.

Purpose of this pull request
Resolves https://github.com/apache/seatunnel/issues/5848 →
https://seatunnel.incubator.apache.org/docs/2.3.3/seatunnel-engine/checkpoint-storage
Among the HDFS section, the kerberosKeytab configuration item is incorrect.
Among them, the seatunnel.hadoop.dfs.client.failover.proxy.provider.usdp-bing configuration item has an extra double quotation mark at the end.

Does this PR introduce any user-facing change?
no
How was this patch tested?
before:

after:

Check list
~* [ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide~

[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
~* [ ] If you are contributing the connector code, please check that the following files are updated:~
~1. Update change log that in connector document. For more details you can refer to connector-v2~
~2. Update plugin-mapping.properties and add new connector information in it~
~3. Update the pom file of seatunnel-dist~
[x] Update the release-note.

Great, thanks ;)

repo: apache/seatunnel


text:
[Improve][Zeta] Add log config for job mdc

Purpose of this pull request
[Zeta] Add log config for job mdc
Does this PR introduce any user-facing change?
No
How was this patch tested?

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist
Add ci label in label-scope-conf
Add e2e testcase in seatunnel-e2e
Update connector plugin_config

[ ] Update the release-note.

link
https://github.com/apache/seatunnel/pull/7491


text:
StackOverflowError

Question
StackOverflowError reported when multiple selectors were configured

The ports of the two selectors are different

Can you show me how to reproduce the beatings?

There is no problem with local startup, and it will appear when deployed to a Linux server

Redeploy will be normal

source: gharchive/issue

repo: apache/shenyu


text:
feature: add support for rocketMQ

summary
RockeMQ is a native go client maintained by the apache rocketMQ team.
test report
I intercepted the SendSync, SendAsync, and SendOneWay methods in the producer, as shown in the figure below.

Could you confirm, is the further process driven by the message going to be in the same context?

Ref https://github.com/apache/skywalking/pull/11877

Could you confirm, is the further process driven by the message going to be in the same context?

I think so, so I added headKey in properties, I'm not sure if I'm articulating the point you want to ask about
type Message struct {
  properties map[string]string
}

I mean, the consumer codes, it would do business operations, such as accessing SQL database, cache, which are supported to be traced. Are they im the same trace?
The reason I am asking for is, message consuming sometimes has a kind of async mechanisms, which may change the tracing context incorrectly. It is better to recheck with demo codes.

I double checked, they are not in the same trace. The chain between producer and consumer is independent, as shown in the diagram, when I make additional http requests within the consumer code. I think these other business operations in consumer will not affect the tracing context, maybe I have overlooked something, please kindly give me your advice

The use case should be

A produces a message
B consumes the message
B uses this message to continue the processes.

These three should be in one trace. If 1 and 2 are, 3 is not, it means you miss one local span to wrap the 2 and 3 in a context. Java agent plugin should have covered this case.

The further process driven by the message will be in the same context, the context can be propagated and intercepted as it is passed down the execution chain

Good to me. Let's wait for @mrproliu review.

Sorry for the recurring problem on CI/Tests. My golangci-lint attempt on the local mac seems to be inconsistent with that on github. Is there any additional configuration required besides make lint?

@mrproliu Any idea?

Sorry for the recurring problem on CI / Tests (pull_request) . My golangci-lint attempt on the local mac seems to be inconsistent with that on github. Is there any additional configuration required besides make lint?

For now, you can delete the local lint command, then, it could be download the specific lint command by make lint automatically.

For now, you can delete the local lint command, then, it could be download the specific lint command by make lint automatically.

Thanks, it works, from yesterday's test, the order of consuming messages is not in the order of message sending, this will affect the stability of plugins test, this time I adjusted the relevant files again

authors: CodePrometheus mrproliu wu-sheng

repo: apache/skywalking-go


text:
SLING-2781

Now valuemaps hold the same state by sharing a common cache. Unit test have been added to test the common state of ValueMap and JcrModifialbeValueMap, other unit tests are passing ofc

With the upcoming migration to git this repository will be deleted. If the pull request contains data that should be re-submitted I suggest that you save the changes locally and resubmit them after the migration, which will likely happen on Monday the 9th of October.

authors: mmanski rombert

repo: apache/sling

license_source: bigquery


text:
SOLR-15367 Convert "rid" functionality into a default Tracer

https://issues.apache.org/jira/browse/SOLR-15367

Description
Trace Id propagation only - based on OTEL mechanism. this will only pickup the id from the incoming request (or generate a new one) and make sure to attach it to all internal communication.
It can be a much better alternative to the existing rid mechanism, without necessarily replacing it.
The way to enable it is to enable OTEL and set a system property (but this could be merged into the OTE config so it's a flag on the main OTEL).
The header is configurable, and could be extended into a list of headers. I would be ok with seeding this from a system property to make config simpler.
Solution
Please provide a short description of the approach taken to implement your solution.
Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[x] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

@dsmiley would love your thoughts on this one.

It's kind of a shame to require the whole open tracing SDK for this functionality but so be it.

you raise a very interesting point, and I think this could work without the sdk. it only really needs a barebones otel impl to make this work, and everything should already be in solr-core (otel api and otel context). if it does turn out it's light on dependencies this could actually move to solr core and no longer depend on the otel module to function. let me check this.

@dsmiley thank you for the review so far. I managed to successfully move the functionality to solr-core without extra dependencies (just api and context). I think this allows for much more freedom with enabling by default and eventually (maybe) replacing the rid.
I updated the test and something weird failed. it seems the UpdateRequest does not actually propagate headers to internal communication? I'm not even sure if this is by design or not. I added a todo to verify.
My plan (given this moved to core) would be to turn this into a 'on by default' behavior (add a flag to allow disabling). Optionally I would source rid from this value, so that there is no confusion between multiple similar sets of values.
Please take another look and let me know what you think so far.

I'm so glad you managed to implement this in core!  Changes looks good.
FYI, one gap in propagation is that our Apache HttpClient isn't instrumented with tracing, only the Jetty HttpClient.  I actually have a fix for this locally.  Of course, ideally we would only use Jetty HttpClient.

FYI, one gap in propagation is that our Apache HttpClient isn't instrumented with tracing, only the Jetty HttpClient. I actually have a fix for this locally.

could you be a bit more specific? I am not sure where the gap is, the context injection (addition of http headers) should ideally happen on both client, but maybe you have seen a specific flow that is broken.
I am down a rabbit hole now looking for new UpdateRequest() calls that wipe out the headers so propagation would be broken.

My local change is to InstrumentedHttpRequestExecutor to add a client span and to inject trace headers.  This class is used in UpdateShardHandler to create the "defaultClient" (Apache HttpClient).  Without this, any communication using Apache HttpClient won't propagate tracing.  If you were to make changes about passing certain headers in a generic way, that would be a work-around but the ideal fix is one I made and could share in InstrumentedHttpRequestExecutor.

I think I understand what you are saying. but I think it was misleading to say propagation doesn't work (I think you partially corrected later by saying "won't propagate tracing"). a more correct thing to say is 'tracing in general does not work with the apache client' and so this propagation proposal is broken due to the same underlying cause. the fix would be for the otel tracing in general and it would also enable the propagation. along the same lines, the existing 9.x tracing impl is broken due to the same reason I believe, the apache client is not aware of the trace context. please correct if I am misunderstanding your point.
I see why you would choose InstrumentedHttpRequestExecutor. it offers a cross cutting access point into the execution, but I don't think it's a very good idea to mix functionality like this (metrics collection with tracing headers). I am looking at alternatives, maybe we could setup a HttpRequestInterceptor which would allow for a cleaner separation of concerns.

I don't think it's a very good idea to mix functionality like this (metrics collection with tracing headers)

I disagree; both are "instrumentation" and the class name was already chosen in this generic way instead of being called MetricsBlahBlahBlah.  Nonetheless the concerns could be split up to work in parallel but it's extra code for unclear gain.
BTW again in my fork locally, I did similarly for the Jetty side in InstrumentedHttpListenerFactory moving the previous logic to this central location, and I think in so doing picked up more use-cases.

I disagree; both are "instrumentation" and the class name was already chosen in this generic way instead of being called MetricsBlahBlahBlah. Nonetheless the concerns could be split up to work in parallel but it's extra code for unclear gain.

sure, I see the point and I don't disagree with it.

BTW again in my fork locally, I did similarly for the Jetty side in InstrumentedHttpListenerFactory moving the previous logic to this central location, and I think in so doing picked up more use-cases.

is this code you mention open source? if so could you link to it? it would make the discussion much easier.

I took a stab at having this on by default,  just to see what the tests say. and it looks ok so far. I am also open to not doing this right away, to allow for some more testing time.

@dsmiley did a recap above and pushed a few more commits. I would like to leave it on by default to catch regressions early, I will disable it if it turns out to be too noisy.
I think this is good to go, please let me know if you see anything blocking.

authors: dsmiley stillalex

repo: apache/solr


text:
SPARK-20220 Documentation Add thrift scheduling pool config to scheduling docs

What changes were proposed in this pull request?
The thrift scheduling pool configuration was removed from a previous release. Adding this back to the job scheduling configuration docs.
How was this patch tested?
Manually tested the changes locally.
Please review http://spark.apache.org/contributing.html before opening a pull request.

Can one of the admins verify this patch?

ok to test

Test build #78371 has started for PR 17536 at commit 7953ec2.

Test build #78371 has finished for PR 17536 at commit 7953ec2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78371/
Test PASSed.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Ping @mrchristine

I submitted https://github.com/apache/spark/pull/21778 to take this over. Will close this manually since the author looks inactive.

authors: AmplabJenkins HyukjinKwon SparkQA gatorsmile mrchristine srowen

repo: apache/spark


text:
[Spark-21842][Mesos] Support Kerberos ticket renewal and creation in Mesos

What changes were proposed in this pull request?
tl;dr: Add a class, MesosCredentialRenewer that updates delegation tokens on a schedule on the behalf of Spark Drivers. Broadcast renewed credentials to the executors.
The problem
We recently added Kerberos support to Mesos-based Spark jobs as well as Secrets support to the Mesos Dispatcher (SPARK-16742, SPARK-20812, respectively). However the delegation tokens (hereafter referred to as Credentials)  have a defined expiration (1 day by default, I believe). This poses a problem for long running Spark jobs (e.g. Spark Streaming applications). YARN has a solution for this where a thread is scheduled to renew the Credentials when they the time reaches 75% of their way to expiration. It then writes the Credentials to HDFS for the executors to find (uses a monotonically increasing suffix).
This solution
We maintain the same method for obtaining Credentials using common code in CoarseGrainedSchedulerBackend, but then schedule their renewal with the MesosCredentialRenewer. Renewed Credentials are broadcasted to the executors. This maintains all transfer of Credentials within Spark (as opposed to Spark-to-HDFS). It also does not require any writing of Credentials to disk. It also does not require any GC of old files.
How was this patch tested?
Manually against a Kerberized HDFS cluster.
Thank you for the reviews.

Can one of the admins verify this patch?

When executor is started it asks from CoarseSchedulerBackend the spark config which contains the hadoop credentials https://github.com/apache/spark/blob/7c92351f43ac4b1710e3c80c78f7978dad491ed2/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L235.
As we have discussed this is not safe. Using rpc an arbitrary executor could register to the scheduler and get tokens. Does this code handle this, do we authenticate executors? How about encryption at least at the rpc level? The latter is not supported on mesos (spark.io.encryption.enabled).

Hey @kalvinnchau good catch on the first renewal time. I believe I addressed it. Have a look. Thanks again.

@ArtRand thanks! I've been testing a local version of doing that, I'll pull that change in and test it as well.

@ArtRand
I'm running into an issue where it seems like the delegation tokens are being sent to the executors (as in I see the logs stating that the tokens are being broadcast), but the old delegation tokens are still in use.
When the job first starts up it created token 777222, then ~18 hours in the refresh occurs:
[INFO ] 2017-09-26 17:59:20.639 [Credential Refresh Thread-0] DFSClient - Created HDFS_DELEGATION_TOKEN token 772826 for <principal> on ha-hdfs:<namenode>
[INFO ] 2017-09-26 17:59:20.747 [Credential Refresh Thread-0] MesosCredentialRenewer - Sending new tokens to all executors
Then at ~24 hour mark I get the exception:
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 772222 for <principal>) is expired
Are you aware of anything that needs to be done to tell the thread to use the newer tokens? Or do the older tokens need to be removed from the UGI?

Hey @kalvinnchau thanks for having the patience to try this. This is a curious error though.
If you look at the addAll method called by UserGroupInformation.addCredentials() it should overwrite the current credentials.
I tried to reproduce your error, but being less patient, I changed my HDFS setup to request the tokens be updated every minute instead of everyday by adding the following to hdfs-site.xml:
    <property>
        <name>dfs.namenode.delegation.token.max-lifetime</name>
        <value>60000</value>
    </property>

I added some logging to the executor backend to check if they were indeed being updated.
case UpdateDelegationTokens(tokens) =>
      logInfo("Got request to update tokens")
      val oldCreds = UserGroupInformation.getCurrentUser.getCredentials
      for (t <- oldCreds.getAllTokens.asScala) {
        logInfo(s"Old Creds ${DelegationTokenIdentifier.stringifyToken(t)}")
      }
      val creds = SparkHadoopUtil.get.deserialize(tokens)
      for (t <- creds.getAllTokens.asScala) {
        val s = DelegationTokenIdentifier.stringifyToken(t)
        logInfo(s"Got new tokens $s")
      }
      SparkHadoopUtil.get.addDelegationTokens(tokens, env.conf)
      val newCreds = UserGroupInformation.getCurrentUser.getCredentials
      for (t <- newCreds.getAllTokens.asScala) {
        logInfo(s"New creds ${DelegationTokenIdentifier.stringifyToken(t)}")
      }

and indeed when I check the logs, indeed the token number has been updated.
17/09/28 03:32:58 INFO CoarseGrainedExecutorBackend: Got request to update tokens
17/09/28 03:32:58 INFO CoarseGrainedExecutorBackend: Old Creds HDFS_DELEGATION_TOKEN token 29 for hdfs on ha-hdfs:hdfs
17/09/28 03:32:59 INFO CoarseGrainedExecutorBackend: Got new tokens HDFS_DELEGATION_TOKEN token 31 for hdfs on ha-hdfs:hdfs
17/09/28 03:32:59 INFO CoarseGrainedExecutorBackend: New creds HDFS_DELEGATION_TOKEN token 31 for hdfs on ha-hdfs:hdfs

then some time later (in fact there was another update in the middle):
17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: Got request to update tokens
17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: Old Creds HDFS_DELEGATION_TOKEN token 34 for hdfs on ha-hdfs:hdfs
17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: Got new tokens HDFS_DELEGATION_TOKEN token 35 for hdfs on ha-hdfs:hdfs
17/09/28 03:35:14 INFO CoarseGrainedExecutorBackend: New creds HDFS_DELEGATION_TOKEN token 35 for hdfs on ha-hdfs:hdfs

I will run a 24h experiment to verify, but hopefully there is a way to validate that the update is working without waiting that long just to debug!
@vanzin Could you eyeball this, am I missing something obvious?

What you wrote sounds correct. However, I've seen errors like the ones above in the past, but haven't been able to fully debug them due to lack of logs. Part of it is because the user running the app didn't provide me the full logs; but also Spark currently doesn't have the logs you added in your debug code, not even at debug level, so we have only partial information about exactly what's going on from existing logs. (And these error tend to happen only after x days, so it's kind of a pain to reproduce.)
That all being said, though, as I mentioned, what you wrote sounds correct AFAIK.

Another thing to keep in mind is that different Hadoop versions have different bugs in this area, so if you use a version of the Hadoop client library that suffers from some issue, or are talking to an HDFS version that has some bug, that can cause problems.
Without researching more I only remember an issue that affects HA mode (HDFS-9276), but there might be others.

Hello @vanzin
Thanks for taking a look at this. Good to know that there can be downstream errors depending on the situation.
Would very much appreciate a proper review on this work when you have some time, very keen on getting this into the next release.

@ArtRand curious, what version of hadoop are you build spark against and what version is the cluster that you're running?

@kalvinnchau I'm running Hadoop 2.6 on a DC/OS cluster with Mesos 1.4.0

ok to test

add to whitelist

Test build #82602 has started for PR 19272 at commit f5925fd.

Test build #82602 has finished for PR 19272 at commit f5925fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82602/
Test PASSed.

@vanzin Thanks for the review. I'll address the comments ASAP.

Test build #82777 has started for PR 19272 at commit 3f22efe.

Test build #82777 has finished for PR 19272 at commit 3f22efe.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82777/
Test PASSed.

Build finished. Test PASSed.

Test build #82870 has started for PR 19272 at commit e522150.

Test build #82870 has finished for PR 19272 at commit e522150.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82870/
Test PASSed.

Test build #82919 has started for PR 19272 at commit 837157d.

Test build #82919 has finished for PR 19272 at commit 837157d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82919/
Test PASSed.

Test build #82941 has started for PR 19272 at commit c95f80b.

Test build #82941 has finished for PR 19272 at commit c95f80b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82941/
Test FAILed.

Retest this please.

Test build #82955 has started for PR 19272 at commit c95f80b.

Test build #82955 has finished for PR 19272 at commit c95f80b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82955/
Test PASSed.

Test build #83235 has started for PR 19272 at commit 864ab7e.

Test build #83235 has finished for PR 19272 at commit 864ab7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83235/
Test PASSed.

Merged build finished. Test PASSed.

Test build #83262 has started for PR 19272 at commit f93c551.

Test build #83262 has finished for PR 19272 at commit f93c551.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83262/
Test PASSed.

@vanzin Thanks for the continued review. I addressed the comments you left. PTAL.
It seems simplest and more robust to keep the serialized Credentials within CoarseGrainedSchedulerBackend. The change is that now the renewal time is also kept along side. I tried to move as much duplicate code to HadoopDelegationTokenManager which remains a "toolbox" as before and doesn't retain much state. The abstract class HadoopCredentialRenewer is now part of the scheduler backend also. I refrained from changing too much of the YARN specific code.

Test build #83468 has started for PR 19272 at commit b2fbcf2.

Test build #83468 has finished for PR 19272 at commit b2fbcf2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83468/
Test PASSed.

Test build #83493 has started for PR 19272 at commit 4558cea.

Test build #83493 has finished for PR 19272 at commit 4558cea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83493/
Test PASSed.

Hello @vanzin thanks for continuing to help with this. Please take another look at this refactor.
In this change, there is one place to interact with hadoopDelegationTokens from CoarseGrainedSchedulerBackedn: initializeHadoopDelegationTokens. This method contains the logic for initializing the tokens and setting a token renewer. It's also now resource-manager specific. This seems cleaner than having a HadoopDelegationTokenManager in CoarseGrainedSchedulerBackend because any "token management" will always want to wrapHadoopDelegationTokenManager so you can keep all the necessary information in one place. Of course, happy to discuss further.

Test build #83607 has started for PR 19272 at commit 5f254e5.

Test build #83607 has finished for PR 19272 at commit 5f254e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83607/
Test PASSed.

Merged build finished. Test PASSed.

Test build #83660 has started for PR 19272 at commit 8df7e37.

Test build #83661 has started for PR 19272 at commit 45b46ed.

Test build #83660 has finished for PR 19272 at commit 8df7e37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83660/
Test PASSed.

Test build #83661 has finished for PR 19272 at commit 45b46ed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83661/
Test PASSed.

Hello @vanzin thanks for the continued help with this, anything else needed?

We want to use this feature and will be great if this can be merged. Any idea which spark release will have it.

Hello @maverick2202, hopefully 2.3 (and maybe back ported?) but that's up to the Committers.

Test build #83863 has started for PR 19272 at commit 18d77ff.

@vanzin PTAL.
I removed the awkward mode parameter from the token manager. Now we only start the renewer thread when using a keytab/principal. The condition is logged appropriately.

Test build #83863 has finished for PR 19272 at commit 18d77ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83863/
Test PASSed.

Test build #83912 has started for PR 19272 at commit 049e4b5.

Test build #83912 has finished for PR 19272 at commit 049e4b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83912/
Test PASSed.

Merging to master.

@vanzin Thanks for the reviews and mentorship!

authors: AmplabJenkins ArtRand SparkQA kalvinnchau maverick2202 skonto vanzin


text:
[WIP][SPARK-24375][Prototype] Support barrier scheduling

What changes were proposed in this pull request?
This PR is to add new RDDBarrier and BarrierTaskContext to support barrier scheduling in Spark. It also modifies how the job scheduling works to accommodate the new feature.
Note: this is a prototype to facilitate the discussion. It's not meant for the final design or anything. It just shows one way that might works.
How was this patch tested?
Simple unit test and integration test.

Test build #91471 has started for PR 21494 at commit 84cdc68.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3807/
Test PASSed.

Test build #91471 has finished for PR 21494 at commit 84cdc68.

This patch fails from timeout after a configured wait of `300m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91471/
Test FAILed.

Merged build finished. Test FAILed.

Hi, @jiangxb1987 , can you explain more for what is barrier scheduling in spark and  elaborate an example which would only works with  barrier scheduling( but could not work under current spark schedule mechanism) for better understanding ?

@Ngone51 You can refer to the SPIP that xiangrui proposed in SPARK-24374 for a basic background and major goal of barrier scheduling, and you can also refer to SPARK-24375 for a design sketch. If you have further comments please feel free to talk on the JIRA (recommended because that works better for something we may want to revisit later) or here :)

Close this in favor of #21758 and #21898 , thanks for your comments! I hope they're addressed in the new code.


text:
[SPARK-24521][SQL][TEST] Fix ineffective test in CachedTableSuite

What changes were proposed in this pull request?
test("withColumn doesn't invalidate cached dataframe") in CachedTableSuite doesn't not work because:
The UDF is executed and test count incremented when "df.cache()" is called and the subsequent "df.collect()" has no effect on the test result.
This PR fixed this test and add another test for caching UDF.
How was this patch tested?
Add new tests.

Test build #91672 has started for PR 21531 at commit 658539a.

cc @maryannxue

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3908/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/18/
Test PASSed.

Test build #91672 has finished for PR 21531 at commit 658539a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91672/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/48/
Test PASSed.

Test build #91713 has started for PR 21531 at commit 658539a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3938/
Test PASSed.

Test build #91713 has finished for PR 21531 at commit 658539a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91713/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3943/
Test PASSed.

Test build #91719 has started for PR 21531 at commit 658539a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/55/
Test PASSed.

Test build #91719 has finished for PR 21531 at commit 658539a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91719/
Test PASSed.

Thanks for the review! I addressed the comments

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4000/
Test PASSed.

Merged build finished. Test PASSed.

Merged build finished. Test PASSed.

Test build #91800 has started for PR 21531 at commit c9db68d.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/110/
Test PASSed.

Test build #91800 has finished for PR 21531 at commit c9db68d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91800/
Test FAILed.

Jenkins, retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4013/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/124/
Test PASSed.

Test build #91819 has started for PR 21531 at commit c9db68d.

Test build #91819 has finished for PR 21531 at commit c9db68d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91819/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #91848 has started for PR 21531 at commit c9db68d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/144/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4033/
Test PASSed.

Test build #91848 has finished for PR 21531 at commit c9db68d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91848/
Test PASSed.

LGTM
Thanks! Merged to master/2.3

authors: AmplabJenkins SparkQA felixcheung gatorsmile icexelloss


text:
[SPARK-25832][SQL] remove newly added map related functions from FunctionRegistry

What changes were proposed in this pull request?
According to the discussion in RC4 voting thread and https://issues.apache.org/jira/browse/SPARK-25829, Spark current has a very weird behavior when we have duplicated keys in map. The newly added map related functions in 2.4 make it worse.
Before we entire fix the map behavior, we should not expose these functions to users
How was this patch tested?
N/A

cc @dongjoon-hyun @gatorsmile @rxin

Test build #98000 has started for PR 22821 at commit 7e919e3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4462/
Test PASSed.

Test build #98000 has finished for PR 22821 at commit 7e919e3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4467/
Test PASSed.

Test build #98005 has started for PR 22821 at commit 7e919e3.

Test build #98005 has finished for PR 22821 at commit 7e919e3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98005/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4476/
Test PASSed.

Test build #98017 has started for PR 22821 at commit 726fc30.

@cloud-fan . That's sounds like a Tech. Preview for the advance users, doesn't it?
It looks like an excuse to ignore the whole context of the discussion and to try to ship in any way. I cann't agree with you with this approach. IMO, we need to fix this or we need to not ship any code like this to any users.

@dongjoon-hyun if there are advanced users who know all the background, and still want to use these functions, why shall we stop them? If end users can't hit the bug with public APIs, I think we are fine.

IIUC, the reason why we are not fixing this is that we want to keep the release cadence. If then, please simply remove this. I already gave you the code.

https://github.com/cloud-fan/spark/pull/11

I'm just confused here. Shall we finish the discussion on the email thread? @cloud-fan and @gatorsmile . If the decision is officially made like that (providing tech. preview to advance users) in the email thread, I'm okay with this.

We seem to be splitting hairs here. Why are we providing tech preview to
advanced users? Are you saying they construct expressions directly using
internal APIs? I doubt that’s tech preview.
Users can construct a lot of invalid plans that lead to weird semantics or
behaviors if they try, and this doesn’t really make it worse.
In either case it’s not that difficult to remove them and add them back so
I could see it going  either way.
On Thu, Oct 25, 2018 at 7:48 AM Dongjoon Hyun notifications@github.com
wrote:

I'm just confused here. Shall we finish the discussion on the email
thread? @cloud-fan https://github.com/cloud-fan and @gatorsmile
https://github.com/gatorsmile . If the decision is officially made like
that (providing tech. preview to advance users) in the email thread, I'm
okay with this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/22821#issuecomment-433080200, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AATvPODR0wtirRLTLDDSb7agOF-U8fqLks5uoc8ogaJpZM4X5fHp
.

Thank you, @rxin . In that case, +1 for complete removal.
It's easier for us to add the expressions back instead of updating exising expressions.

@cloud-fan . I'll update my PR to you once more.

I will submit a PR to revert all these changes. Thanks!

Please ping me on the R removal.

Test build #98017 has finished for PR 22821 at commit 726fc30.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98017/
Test FAILed.

authors: AmplabJenkins SparkQA cloud-fan dilipbiswal dongjoon-hyun felixcheung gatorsmile rxin


text:
[SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* in tuning guide be consistent with that of SQLConf.

What changes were proposed in this pull request?
This pr is a follow up of https://github.com/apache/spark/pull/26200.
In this PR, I modify the description of spark.sql.files.* in sql-performance-tuning.md to keep consistent with that in SQLConf.
Why are the changes needed?
To keep consistent with the description in SQLConf.
Does this PR introduce any user-facing change?
No, It just
How was this patch tested?
Existed UT.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #118282 has started for PR 27545 at commit 25b3feb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23041/
Test PASSed.

Test build #118282 has finished for PR 27545 at commit 25b3feb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118282/
Test PASSed.

authors: AmplabJenkins HyukjinKwon SparkQA turboFei


text:
[SPARK-25271][SQL][2.4] Hive ctas commands should use data source if it is convertible

What changes were proposed in this pull request?

In Spark 2.3.0 and previous versions, Hive CTAS command will convert to use data source to write data into the table when the table is convertible. This behavior is controlled by the configs like HiveUtils.CONVERT_METASTORE_ORC and HiveUtils.CONVERT_METASTORE_PARQUET.
In 2.3.1, we drop this optimization by mistake in the PR SPARK-22977. Since that Hive CTAS command only uses Hive Serde to write data.
This patch adds this optimization back to Hive CTAS command. This patch adds OptimizedCreateHiveTableAsSelectCommand which uses data source to write data.
This is to backport #22514 to branch-2.4.
Why are the changes needed?

This bug was originally reported in 2.3.1, but only fixed in 3.0. We should have it in branch-2.4 because the branch is LTS.
Does this PR introduce any user-facing change?

Yes. Users can use the config to use built-in data source writer instead of Hive serde in CTAS.
How was this patch tested?

Unit tests.

cc @dongjoon-hyun

Test build #129698 has started for PR 30017 at commit e3ffaaf.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34304/

Thank you, @viirya .

cc @anuragmantri

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34304/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34304/
Test PASSed.

cc @cloud-fan

Test build #129698 has finished for PR 30017 at commit e3ffaaf.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait CreateHiveTableAsSelectBase extends DataWritingCommand
case class CreateHiveTableAsSelectCommand(
case class OptimizedCreateHiveTableAsSelectCommand(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129698/
Test PASSed.

Thanks!

authors: AmplabJenkins SparkQA dongjoon-hyun viirya


text:
[SPARK-33721][SQL] Support to use Hive build-in functions by configuration

What changes were proposed in this pull request?
Hive and Spark SQL engines have many differences in built-in functions.The differences between several functions are shown below：

build-in functions
SQL
result of Hive SQL
result of Spark SQL

unix_timestamp
select unix_timestamp(concat('2020-06-01',  ' 24:00:00'));
1591027200
NULL

to_date
select to_date('0000-00-00');
0002-11-30
NULL

datediff
select datediff(CURRENT_DATE, '0000-00-00');
737986
NULL

collect_set
selectc1,c2,concat_ws('##', collect_set(c3)) c3_set frombigdata_offline.test_collect_set group by c1, c2; bigdata_offline.test_collect_set contains data:(1, 1, '1')，(1, 1, '2'),(1, 1, '3'),(1, 1, '4'),(1, 1, '5')
c1  c2  c3_set1   1   2##3##4##5##1
c1  c2      c3_set1   1   3##1##2##5##4

Why are the changes needed?

There is no conclusion on which engine is more accurate. Users prefer to be able to make choices according to their real production environment.
I think we should do some improvement for this.
Does this PR introduce any user-facing change?

No
How was this patch tested?

manual

Can one of the admins verify this patch?

There are too many differences, for example:
hive> select timestamp '2020-06-01 24:00:00';
OK
2020-06-02 00:00:00
Time taken: 0.034 seconds, Fetched: 1 row(s)

spark-sql> select to_timestamp('2020-06-01 24:00:00');
NULL

Could we add these differences to http://spark.apache.org/docs/latest/sql-migration-guide.html#compatibility-with-apache-hive?

Which version of hive is based on in this migration guide?
I found some inconsistency not listed in the doc，but based on hive 1.x，so I guess this doc is based on hive 2.x？
Since Spark 2.x is prebuilt with hive 1.x， meaning hive 1.x is still widely used，so I think we may note these inconsistency  in the doc？

I believe we just comprehensively document. If there are some specific behaviours for a specific Hive version, we should better document it as well.

Yeah, I don't support this change either, I guess @wangyum too. Maybe we should just document the diff as guided by @wangyum.

Yea, a better approach would be to switch the catalog to a Hive-compatible implememtation, but that requires we add FunctionCatalog API first.

According to the above advice, I'll close this (AS-IS) PR.
@southernriver . You can reopen this if you change the approach or open a new PR.
Thank you for making a PR and sorry for the decision on this PR at this time.

authors: AmplabJenkins HyukjinKwon cloud-fan dongjoon-hyun southernriver sqlwindspeaker


text:
[WIP][SPARK-32833][SQL] JDBC V2 Datasource aggregate push down

What changes were proposed in this pull request?
This PR refactor https://github.com/apache/spark/pull/29695,  it has some limitations in terms of implementation and abstraction:

Don't support count
Don't support AVG in case of multiple partitions, and actually the current implementation is wrong even in single partition.
Don't know how to extend for supporting more aggregation case, for example, sum(if()).
Don’t know how to extend for supporting join push down

see the discussion here, The SPIP PDF could be viewed here.
Why are the changes needed?
for better performance
Does this PR introduce any user-facing change?
No
How was this patch tested?
using tests from https://github.com/apache/spark/pull/29695

Can one of the admins verify this patch?

cc @huaxingao

retest please

retest this please

retest this please

ok to test.

add to whitelist.

Test build #136977 has started for PR 32061 at commit 75f8fdc.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41554/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41554/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41554/

ok to test.

Test build #136991 has started for PR 32061 at commit dcc16a1.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41568/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41568/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41568/

Test build #136977 has finished for PR 32061 at commit 75f8fdc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class JDBCTableCatalog extends TableCatalog with SupportsNamespaces with SupportsSQL with Logging
abstract sealed class PushQuery extends Logging
class OldPush (
case class PushScanQuery(
case class PushAggregateQuery(
case class SingleSQLStatement(
case class SingleCatalystStatement(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136977/

Test build #136991 has finished for PR 32061 at commit dcc16a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136991/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43714/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43714/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43714/

authors: AmplabJenkins SparkQA baibaichen wangyum


text:
[SPARK-35902][UI] the auto-generated rdd's name in the storage tab should be truncated if it is too long.

What changes were proposed in this pull request?
the auto-generated rdd's name in the storage tab should be truncated  as a single line if it is too long.
Why are the changes needed?
to make the ui shows more friendly.
Does this PR introduce any user-facing change?
No
How was this patch tested?
just a simple modifition in css, manual test works well like below:

Can one of the admins verify this patch?

cc @gengliangwang and @sarutak FYI

Hi @kyoty, could you show a screen shot in the description from which we can see the effect of this change?

ok to test.

Test build #137455 has started for PR 32191 at commit 7312c99.

Test build #137455 has finished for PR 32191 at commit 7312c99.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137455/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42030/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42030/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42030/

@sarutak  the screenshots have been attached, please help to reveive this, thanks!

Test build #137481 has started for PR 32191 at commit 6e16f16.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42056/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42056/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42056/

I left one minor comment. Otherwise, it LGTM and I confirmed this change works on my laptop.

Test build #137481 has finished for PR 32191 at commit 6e16f16.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137481/

Test build #137504 has started for PR 32191 at commit b2f2ff3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42078/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42078/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42078/

Test build #137504 has finished for PR 32191 at commit b2f2ff3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137504/

@kyoty
I checked whether the GitHub Actions jobs finished successfully but I see the following message.
Actions jobs for this repository have been disabled by GitHub staff. If you are the repository owner, you can contact support via https://github.com/contact for more information.

Do you have any idea of this?
Anyway, I don't think this change affects the GA failure because it's just a style change and Jenkins passes so I'll merge this.

@HyukjinKwon do you have any idea about sarutak mentioned above ? I have never seen this warning before..

Can one of the admins verify this patch?

Test build #137577 has started for PR 32191 at commit b2f2ff3.

@kyoty I think it's better to ask GitHub stuff as the error message suggests.

Test build #137577 has finished for PR 32191 at commit b2f2ff3.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137577/

@sarutak  Thanks for your help and review!

authors: AmplabJenkins HyukjinKwon SparkQA kyoty sarutak


text:
[SPARK-35356][SQL] Fix issue of the createTable when externalCatalog is InMemoryCatalog

What changes were proposed in this pull request?
In the case of externalCatalog is InMemoryCatalog, Using a different SparkSession will report the following error when create the same table , as follow:
sparkSession.sql("create table if not exists t1 (id Int) using orc")
sparkSession.sql("insert into t1 values(1)")

sparkSession2.sql("create table if not exists t1 (id Int) using orc") // will report error as follow

Can not create the managed table('`default`.`t1`'). The associated location('file:/***/t1') already exists.
org.apache.spark.sql.AnalysisException: Can not create the managed table('`default`.`t1`'). The associated location('file:/***/t1') already exists.
	at org.apache.spark.sql.errors.QueryCompilationErrors$.cannotOperateManagedTableWithExistingLocationError(QueryCompilationErrors.scala:571)

Why are the changes needed?
Fix bug.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Add new test

Can one of the admins verify this patch?

Can you fix the test failures? https://github.com/yikf/spark/actions/runs/828366343

authors: AmplabJenkins HyukjinKwon yikf


text:
[SPARK-38235][SQL][TESTS] Add test util for testing grouped aggregate pandas UDF

What changes were proposed in this pull request?
This PR proposes to add test util TestGroupedAggPandasUDF to help testing grouped aggregate pandas UDF more comfortable.
Why are the changes needed?
To improve testability by integrating various UDF tests with existing test utilities.
Does this PR introduce any user-facing change?
No, it's test only.
How was this patch tested?
Manually tested, and this will be tested in the SPARK-38107 when testing grouped aggregate pandas UDF related errors.

Merged to master.

authors: HyukjinKwon itholic


text:
[YARN][SPARK-4929] Bug fix: fix the yarn-client code to support HA

Nowadays, yarn-client will exit directly when the HA change happens no matter how many times the am should retry.
The reason may be that the default final status only considerred the sys.exit, and the yarn-client HA cann't benefit from this.
So we should distinct the default final status between client and cluster, because the SUCCEEDED status may cause the HA failed in client mode and UNDEFINED may cause the error reporter in cluster when using sys.exit.

Test build #25021 has started for   PR 3771 at commit c02bfcc.

This patch merges cleanly.

@tgravescs your comment is much more clear than what I said, I have use it instead of mine.Thx.
when Yarn HA event happens, the previous ApplicationMaster will throw a
java.io.IOException: Failed on local exception: java.io.EOFException

So the yarn cluster the catch the exception and change the final status.
But the yarn client will directly go into the ShutDownHook and cause the problem.
I think it haven't go into the  DisassociatedEvent yet, because the driver is still alive.

Test build #25021 has finished for   PR 3771 at commit c02bfcc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25021/
Test PASSed.

Test build #25154 has started for   PR 3771 at commit c02bfcc.

This patch merges cleanly.

Test build #25154 has finished for   PR 3771 at commit c02bfcc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25154/
Test PASSed.

thanks @SaintBacchus  the changes look good.

authors: AmplabJenkins SaintBacchus SparkQA tgravescs


text:
[SPARK-41178][SQL] Fix parser rule precedence between JOIN and comma

What changes were proposed in this pull request?

This PR fixes a long-standing parser bug in Spark: JOIN should take precedence over comma when combining relations. For example, FROM t1, t2 JOIN t3 should result to t1 X (t2 X t3). However, it's (t1 X t2) X t3 today.
You can easily verify this behavior in other databases by running the query SELECT * FROM t1, t2 JOIN t3 ON t1.c and t3.c. It should fail as t1.c is not available when joining t2 and t3. I tested MySQL, Oracle and PostgreSQL, all of them fail, but Spark can run this query due to the wrong join order.
However, this bug has a large impact: it changes the join order which can lead to worse performance or unexpected analysis errors. To be safe, this PR hides the fix under a new ANSI sub-config.
Why are the changes needed?

bug fix
Does this PR introduce any user-facing change?

No, as the fix is turned off by default.
How was this patch tested?

new tests

cc @viirya @dongjoon-hyun @gengliangwang @MaxGekk

thanks for review, merging to master!


text:
[SPARK-42052][SQL] Codegen Support for HiveSimpleUDF

What changes were proposed in this pull request?

As a subtask of SPARK-42050, this PR adds Codegen Support for HiveSimpleUDF
Extract aHiveUDFEvaluatorBase class for the common behaviors of HiveSimpleUDFEvaluator & HiveGenericUDFEvaluator.

Why are the changes needed?

Improve codegen coverage and performance.
Following https://github.com/apache/spark/pull/39949. Make the code more concise.

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Add new UT.
Pass GA.

cc @cloud-fan

@cloud-fan Can we merge it to master? After it I will try to refactor HiveGenericUDTF & HiveUDAFFunction. Thanks!

thanks, merging to master!

authors: cloud-fan panbingkun


text:
[SPARK-44385][SQL] Use PartitionEvaluator API in MergingSessionsExec & UpdatingSessionsExec

What changes were proposed in this pull request?
SQL operator MergingSessionsExec, UpdatingSessionsExec updated to use the PartitionEvaluator API to do execution.

Why are the changes needed?
To avoid the use of lambda during distributed execution.
Ref: SPARK-43061 for more details.

Does this PR introduce any user-facing change?
No

How was this patch tested?
exist test

ping @cloud-fan @viirya


text:
[SPARK-47980][SQL][TESTS] Reactivate test 'Empty float/double array columns raise EOFException'

What changes were proposed in this pull request?

SPARK-29462 have been resolved, so let's enable this test
Why are the changes needed?

test cov
Does this PR introduce any user-facing change?
no
How was this patch tested?

this test
Was this patch authored or co-authored using generative AI tooling?
no

Merged to master. Thank you @HyukjinKwon


text:
[SPARK-48717][FOLLOWUP][PYTHON][SS] Catch Cancelled Job Group wrapped by Py4JJavaError in StreamExecution

What changes were proposed in this pull request?

The previous commit https://github.com/apache/spark/commit/1581264a5d77e1731a987020a448ddc3f3ea8f27 doesn't capture the situation when a job group is cancelled. This patches that situation.
Why are the changes needed?

Bug fix, without this change, calling query.stop() would sometimes (when there is a python foreachBatch function, and this error is thrown) cause query appears as failed.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Added unit test.
Was this patch authored or co-authored using generative AI tooling?

No

cc @HyukjinKwon @allisonwang-db PTAL! TY!

Merged to master.

authors: HyukjinKwon WweiL


text:
[SPARK-9756][ML] Make constructors in ML decision trees private

These should be made private until there is a public constructor for providing rootNode: Node to use these constructors.
@jkbradley

Merged build triggered.

Merged build started.

Test build #40208 has started for   PR 8046 at commit 2cbdf08.

Test build #40208 has finished for   PR 8046 at commit 2cbdf08.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

LGTM.  Merging with master and branch-1.5

authors: AmplabJenkins SparkQA feynmanliang jkbradley


text:
[WIP] [SPARK-9772] [PySpark] [ML] Add Python API for ml.feature.VectorSlicer

Add Python API for ml.feature.VectorSlicer.
Todo:
Add user guide and examples.

Merged build triggered.

Merged build started.

Test build #40446 has started for   PR 8102 at commit 54bffab.

Test build #40446 timed out     for PR 8102 at commit 54bffab     after a configured wait of 175m.

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Merged build finished. Test PASSed.

test this please

Merged build triggered.

Merged build started.

Test build #40826 has started for   PR 8102 at commit 54bffab.

Test build #40826 has finished for   PR 8102 at commit 54bffab.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Merged build finished. Test PASSed.

reviewing now

One more comment: need to add VectorSlicer to list __all__ at top of file

Build triggered.

Build started.

Test build #41230 has started for   PR 8102 at commit 5d2f75b.

Merged build triggered.

Merged build started.

Test build #41232 has started for   PR 8102 at commit 879474d.

Test build #41232 has finished for   PR 8102 at commit 879474d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41232/
Test PASSed.

Test build #41230 timed out     for PR 8102 at commit 5d2f75b     after a configured wait of 175m.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41230/
Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Test build #41298 has started for   PR 8102 at commit 879474d.

Test build #41298 has finished for   PR 8102 at commit 879474d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41298/
Test PASSed.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #42195 has started for   PR 8102 at commit 67bf664.

Test build #42195 has finished for   PR 8102 at commit 67bf664.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class VectorSlicer(JavaTransformer, HasInputCol, HasOutputCol):

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42195/
Test PASSed.

Merged build finished. Test PASSed.

LGTM.  Merging with master.  Thank you!

authors: AmplabJenkins SparkQA jkbradley yanboliang


text:
A better way to get default value

A better way to get default value.

+1

authors: HeartSaVioR lujinhong

repo: apache/storm


text:
fix: Support dynamic assets in Maven plugin to generate docs

Purpose

Modify the streampipes-maven-plugin so that also dynamic assets are resolved when generating the Markdown documentation. Dynamic assets are used by the PLC4X adapters to auto-generate the available configuration options.
Remarks

PR introduces (a) breaking change(s): no
PR introduces (a) deprecation(s): no

Discussion: It seems we're currently using the name Asset for two different concepts, which might lead to confusion. To make the distinction clearer, I suggest renaming one of them to something more descriptive—perhaps PipelineElementAsset.
What are your thoughts on this proposal? Do you have any alternative name suggestions?

Discussion: It seems we're currently using the name Asset for two different concepts, which might lead to confusion. To make the distinction clearer, I suggest renaming one of them to something more descriptive—perhaps PipelineElementAsset.
What are your thoughts on this proposal? Do you have any alternative name suggestions?

Yes, good idea, we can harmonize this to ExtensionAsset which we already use in the model definition.

authors: dominikriemer tenthe

repo: apache/streampipes


text:
[Explore] Data pane in explore not resizing on widow resize

Screenshot / Description
In Explore, the Data pane is not updating its height when the window is vertically resized. This has two easily reproduced cases:

Start with a TALL window and open the data pane. Shorten the window. You'll see that the entire chart+table area scrolls due to the lack of table resize. Collapse and re-open the Data area, and you'll see the height is adjusted (no more scrollbar).

https://user-images.githubusercontent.com/812905/103616825-fdf17500-4ee1-11eb-9970-919755364505.mp4

Start with a SHORT window an open the data pane. Lengthen the window. You'll see that the table is unnecessarily cropped off. Collapse and re-open the Data area, and you'll see the height is adjusted (no more cropping).

https://user-images.githubusercontent.com/812905/103616970-38f3a880-4ee2-11eb-86ba-40314038af30.mp4

@maloun96 this stems from the other PR we just merged.

@maloun96 this stems from the other PR we just merged.

confirmed that now datatable is resizing proof!! Thank you @kgabryje and @maloun96 !

confirmed that now datatable is resizing proof!! Thank you @kgabryje and @maloun96 !

authors: junlincc rusackas

repo: apache/superset


text:
Helm

The helm installation stops at superset-init-db.
I tested it in k8s cluster and microk8s cluster, same results.
Screenshots

i checked the redis logs and found this:

Why have you closed the bug?

authors: mvoitko pavimimos


text:
fix: Calendar color change

SUMMARY
This changes the Calendar button on the chart page to be consistent with the one on the dashboard page.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Chart:

Dashboard:

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

🏷 2021.31

authors: AAfghahi rosemarie-chiu


text:
fix(altered-modal): displayed the metric value in altered modal correctly

SUMMARY
Altered Modal Doesn't Show Metrics Properly
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before:

After:

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Hello @prosdev0107 thanks for the PR. There are some linting issues to fix. Also, I don't think you need to tick the "Changes UI" checkbox in the PR description as that is intended for changes that affect the UI, in this case, this is just a fix of standard behavior.

Hello @prosdev0107 thanks for the PR. There are some linting issues to fix. Also, I don't think you need to tick the "Changes UI" checkbox in the PR description as that is intended for changes that affect the UI, in this case, this is just a fix of standard behavior.

Agreed on the checkbox, though I appreciate over-checking rather than under-checking :)
To run the linter locally, just run npm run lint-fix from within the superset-frontend folder you can also install the pre-commit hooks if you'd like (I would need to dig up the command if so, since I don't remember it offhand)

Approved and waiting for CI. As with all fix PRs, it'd be great if there were a test. Let me know if that looks like a reasonably easy addition here.

authors: geido prosdev0107 rusackas


text:
Use s instead of hyphens

A small enhancement to tenant drop-down selection lists in Traffic Portal would be to use the HTML5 <optgroup> tag to group tenants by their parents. I'd suggest making "root" its own parent.
Example.:
<label for="tenant">Select a tenant</label>
<select id="tenant">
    <optgroup label="root">
        <option>root</option>
        <option>tenantWithNoChildren</option>
        <option>tenentWithTwoChildren</option>
    </optgroup>
    <optgroup label="tenantWithTwoChildren">
        <option>Child1</option>
        <option>Child2</option>
    </optgroup>
</select>

Rendered (in Firefox 63):

I agree we need to improve the "select a tenant" dropdown list, but I don't think this flattened layout would be best. The user would still have to bounce around the list to follow the hierarchy. If we could visualize the full tree (unflattened) in a form that's not a list as it is today but more like an actual tree, the user could more easily follow the hierarchy down to the desired tenant.
Something like a modal that pops up and renders a tree like this where you just have to click to select the tenant.
           __________________[root]____
           |                          |
[tenantWithTwoChildren]      [tenantWithNoChildren]
      |            |
[child1]         [child2]

That would be ideal but probably too much work unless there's a decent js library that exists for that already.

Could be implemented as an Angular Material "Tree"

repo: apache/trafficcontrol


text:
TP Migrate Jobs test from E2E to integrate test

What does this PR (Pull Request) do?

[x] This PR  is not related to any Issue

Which Traffic Control components are affected by this PR?

CDN in a Box
Traffic Portal
CI tests

What is the best way to verify this PR?

Make sure all tests are migrate in integration/Data/jobs.ts directory.
https://github.com/apache/trafficcontrol/blob/master/traffic_portal/test/e2e_deprecated/jobs/jobs-spec.js
Go to traffic_portal/test/integration and run
npm test -- --params.baseUrl https://localhost --capabilities.shardTestFiles true and make sure test all pass.
Make sure CI tests pass also.
If this is a bug fix, what versions of Traffic Control are affected?

The following criteria are ALL met by this PR

[ ] This PR includes tests OR I have explained why tests are unnecessary
[ ] This PR includes documentation OR I have explained why documentation is unnecessary
[ ] This PR includes an update to CHANGELOG.md OR such an update is not necessary
[ ] This PR includes any and all required license headers
[ ] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

Additional Information

Tests pass on local.
Code LGTM after proposed changes.


text:
wasm plugin: provide defaults for struct SanitizationConfig

This should resolve gcc release build warnings for fedora and rocky9.

Cherry-picked to v10.0.x

authors: cmcfarlen traeak

repo: apache/trafficserver


text:
TS-4732: Changing the do_io_read API so it can be called with NULL and

0 byte values.  Allowing do_io_read and do_io_write to not warn on closed
connection when we are only trying to disable the reads and writes for it.
Removed some macros that were used in a few places (not all the time) and
a couple that were not used at all.
(cherry picked from commit 1975d679663453348057bb7155bdd3c17ac73880)

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1138/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1031/ for details.

authors: atsci bryancall


text:
H2 test with traffic -replay

This patch :

Uses traffic replay as h2 client for sending multiple requests
removes unnecessary imports
fixes config to make it usable with autest

[approve ci autest]

[approve ci autest]

[approve ci autest]


text:
TS-4227: Remove auto_ptr in SPDY plugin

This only applies to 5.3.x and is because F22 has a version of GCC that has deprecated auto_ptr.

+1

authors: PSUdaemon zwoop


text:
[TOPHUB] Remove tophub context from VM and Graph Executor

Remove tophub context from VM and Graph Executor because 1. it is no longer up to date and 2. it causes differences in lowering between vm.VMCompiler.optimize and vm.VMCompiler.lower.
Will this cause any impacts? I'm not sure how much tophub is used.
@masahi @comaniac

Although I'm not 100% for sure, I'm afraid of performance impact. The tricky points are:

Although TOPHUB does out of date, many tuned schedules are still applicable. For example, the tile sizes tuned for Conv2Ds in ResNet on NVIDIA T4, V100, etc should still applicable, as AutoTVM and TVM CUDA codegne are rarely updated in the past year.

We do know that some ops (especially Conv2Ds on x86 and CUDA) do demonstrate obvious performance improvements with the TOPHUB schedules. General speaking, as long as we still support AutoTVM, we should not abandon TOPHUB.

@comaniac My goal with this PR was the fix the issue that some codepaths use the tophub context and some do not. This leads the really confusing results where VMCompiler.optimize and VMCompiler.lower result in different lowered functions (usually different layouts due to alter op layout). I see three ways to make everything consistent:

Require users to manually use tophub (this PR). It seems like this approach may have a negative performance impact if the user is not using tuned schedules.

Use tophub as the default context when no other context has been specified. This will make VMCompiler.optimize and VMCompiler.lower consistent in their results. On the other hand, some alteroplayout code may no longer fire correctly because it depends on a fallback context (unclear if it was even correct in the first place).

Add with tophub to VMCompiler.optimize. This doesn't solve the underlying problem. I expect us see more inconsistent results in other parts of the codebase where tophub is not used.

or 2. appear to be the best solutions. Do you prefer 2. over 1.? If so, I will submit a PR with that change.

Thanks for the explanation. I personally prefer 2 for sure as it minimizes the changes from user's perspective.
cc @masahi to see if there's any obvious concern for the alter op layout.

some alteroplayout code may no longer fire correctly because it depends on a fallback context (unclear if it was even correct in the first place)

@tkonolige Can you point us to an example of such code? Sounds weird...

@masahi If we set the default context to the tophub context, then this code will never fire: https://github.com/apache/tvm/blob/main/python/tvm/topi/x86/conv2d_alter_op.py#L98-L109. There a couple more similar blocks in this function that also will not fire.

Is "not firing this code path" a bad thing? To me it looks like it only prevents fallback schedules to be applied, and alter layout itself still kicks in?

I'm not sure if the fallback schedules are necessary or not. We can try dropping them and see what happens?

From my recent work on vnni dense, I remember that if I don't have is_fallback code path, some random (?) value was chosen to instantiate my schedule. So things should run I believe without is_fallback.
I think is_fallback is mostly for the case where autotvm is not used and developers want to apply more sensible default values than the ones chosen arbitrary by autotvm. I think the question is: are tophub schedules supposed to be better than those hand picked params in fallback schedules? If the answer is not a definite yes, I'm inclined to remove tophub altogether and possibly port their params to whatever _get_default_config is doing.
So how about this: We go ahead with this PR and remove tophub. If later someone complains about perf regression, which means they rely on tophub, we port tophub params to _get_default_config.

@masahi It looks like using the fallback schedules caused CI to fail for this pr. So unfortunately I guess that could count as a perf regression.

Please be aware that default schedule is not always valid on GPU, because a hand pick tile size may not fit to a certain GPU architecture.
IMHO, based on the above discussion this change might be worth to have a short RFC to inform the community, as TopHub has been used for a long while and we have no clue about the impact. If we are going to apply the change in this PR, at least a deprecate warning is necessary.

@masahi It looks like using the fallback schedules caused CI to fail for this pr. So unfortunately I guess that could count as a perf regression.

Ok, so the CI is really pulling the tophub GPU schedule for mobilenet v2...

Please be aware that default schedule is not always valid on GPU, because a hand pick tile size may not fit to a certain GPU architecture.

This is also true of the tophub schedules as they are not specific to a GPU model, just target kind.

IMHO, based on the above discussion this change might be worth to have a short RFC to inform the community, as TopHub has been used for a long while and we have no clue about the impact. If we are going to apply the change in this PR, at least a deprecate warning is necessary.

Do RFCs have the broadest reach? Or would it make more sense to post on the forum? I'm not sure who the users of tophub are.
How long should we provide a deprecation warning for? I'd prefer to have this fixed as soon as possible instead of waiting. We change a lot of things in tvm that are more impactful than this without warning.

Do RFCs have the broadest reach? Or would it make more sense to post on the forum? I'm not sure who the users of tophub are.

Posting to the forum is also fine. My intention is just making an announcement, because I believe now only 3 of us know that we are going to do this.

How long should we provide a deprecation warning for? I'd prefer to have this fixed as soon as possible instead of waiting. We change a lot of things in tvm that are more impactful than this without warning.

Usually we provide a warning for a release cycle. For example, now is 0.9.dev, so the warning should be at the 0.9 release and remove at 0.10 (or 1.0 not sure what's going to happen). Of course this could be flexible as long as most community members have an agreement. This is also a reason of posting to the forum or filing an RFC.

blocked on https://github.com/apache/tvm-rfcs/pull/59

authors: areusch comaniac masahi tkonolige

repo: apache/tvm


text:
[ONNX] Add imports for Gelu, BiasGelu

As title. Adds imports for Gelu, BiasGelu from the com.microsoft onnx op domain.

cc @MargaretQian @sfvaroglu

Thanks @anwang2009 @sfvaroglu

Thanks @anwang2009!


text:
[Unity][TensorIR][Schedule] Fix dtype bugs in Rfactor rule

Currently test test_reduction_rfactor_zero_dim in file tests/python/unittest/test_tir_schedule_rfactor.py fails on unity branch.
The reason is there exists mixture of int32/int64 dtype in this testcase as well as in TIR RFactor schedule rule.
This PR fixes these problems by unifying dtype to int64 as current unity branch sets  int64 as default dtype.
Besides, this PR adds a tvm::tir::DefaultIndexType() interface to determine the correct dtype.
test_reduction_rfactor_zero_dim is a regression test so we does not need more unit tests.

For TIR primitive fixes we can go to the main branch.

For TIR primitive fixes we can go to the main branch.

The problem only occurs in unity branch, but not in main branch. The reason for that should be we require int64 as default dtype (and due to some limitations, in only a portion of places) only in unity, and this resulted in specific bugs in Unity.

@Ubospica would be great to followup on this. please send a PR to main

The essence of the problem that this PR wants to solve is the inconsistency between the main and unity branches regarding dtype (int32 vs int64). However, current fix in the test cannot be unified in the main branch and unity branch. Since it is not a blocker, this PR will be closed temporarily and reopened after conprehensive consideration.

authors: MasterJH5574 Ubospica tqchen


text:
Set tvm.micro.project_api as a Python Module

Add missing tvm.micro.project_api module file:

The missing init.py makes it impossible to import tvm.micro.project_api, which is indirectly done in tvmc
It is more evident when using tlcpack as the module is not included in the package at all

Reopening as I accidentally closed #8944.
cc @tqchen @areusch @gromero @manupa-arm @Mousius for reviews

Can someone merge this, as CI seems happy? Thanks

Thanks @leandron @areusch . This is merged now.

authors: leandron manupa-arm


text:
[WICKET-7067] WebSocketResponse does not allow setting a header thus,…

WebSocketResponse does not allow setting a header thus,  disableCaching fails with an UnsupportedOperationException

@martin-g Many thanks for review! Merging and cherrypicking to master

repo: apache/wicket


text:
[YUNIKORN-2119] check wildcard limit in different queue level

What is this PR for?
If a non-wildcard limit is not defined in parent queue, but it's defined in child queue, we need to check whether it's large than wildcard limit in parent queue.
What type of PR is it?

[X] - Improvement

What is the Jira issue?
https://issues.apache.org/jira/browse/YUNIKORN-2119
How should this be tested?
Covered by unit test.

@FrankYang0529 Any updates?

repo: apache/yunikorn-core


text:
[YUNIKORN-2855] Rethink the call for update metrics with nil resource

What is this PR for?
We will not update the nil resource to resource metrics, we need to fix it.
What type of PR is it?

[ ] - Bug Fix
[x] - Improvement
[ ] - Feature
[ ] - Documentation
[ ] - Hot Fix
[ ] - Refactoring

Todos

[ ] - Task

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/YUNIKORN-2855
Put link here, and add [YUNIKORN-Jira number] in PR title, eg. [YUNIKORN-2] Gang scheduling interface parameters

How should this be tested?
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

Thanks @craigcondit for review, i addressed the code to a more reasonable way and setting zero resource for nil update case.

@craigcondit , we support setting maxResource/ guaranteed resource to nil, but the metrics update is wrong because we don't update it from previous value update to zero/nil.

Then it seems the proper solution is to retrieve the previous value of the metric and update the values there (an "unprune" so to speak). Missing values n the new map should be set to zero. Setting arbitrary resources is just wrong. Doing it this way also has the advantage of ensuring that any resource types that were emitted by the metric previously are still emitted, but with zero instead of missing.

Thanks, @craigcondit i like the idea unprune, i will address this way.

I see... @craigcondit let me change my code.

Addressed in latest PR:

Getting the existed metrics.
Setting those field to zero.

authors: craigcondit zhuqi-lucas


text:
[HOTFIX] fix: DON'T run npm clean in maven clean phase (CI related, Build failure in web)

What is this PR for?
maven clean executes npm run clean after #1824. but running npm commands requires installing, it's impossible to clean before installing it.
This PR resolve CI break (see broken RAT build profiles)

https://travis-ci.org/apache/zeppelin/builds/188895921
https://travis-ci.org/apache/zeppelin/builds/188782988

What type of PR is it?
[Bug Fix | Hot Fix]
Todos
Nothing
What is the Jira issue?
ZEPPELIN-1850
How should this be tested?
The first build profile (RAT) should pass CI.
Screenshots (if appropriate)
[INFO] --- frontend-maven-plugin:1.3:npm (npm clean) @ zeppelin-web ---
...
[INFO] Running 'npm run clean' in /Users/lambda/github/apache-zeppelin/zeppelin-bug/zeppelin-web
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.701 s
[INFO] Finished at: 2017-01-05T03:55:32+09:00
[INFO] Final Memory: 21M/982M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm clean) on project zeppelin-web: Failed to run task: 'npm run clean' failed. java.io.IOException: Cannot run program"/Users/lambda/github/apache-zeppelin/zeppelin-bug/zeppelin-web/node/node" (in directory "/Users/lambda/github/apache-zeppelin/zeppelin-bug/zeppelin-web"): error=2, No such file or directory -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm clean) on project zeppelin-web: Failed to run task
...

[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Questions:

Does the licenses files need update? - NO
Is there breaking changes for older versions? - NO
Does this needs documentation? - NO

LGTM. It fixes CI build failure. Merge to master as a hotfix

repo: apache/zeppelin


text:
ZEPPELIN-614 Part 1. Merge PostgresqlInterpreter's sql completer into JDBC Interpreter

What is this PR for?
Supporting Sql completer into JDBC Interpreter which is merged by the function of Postgresql Interpreter. And this is the first step to merge Postgresql into generic JDBC Interpreter
What type of PR is it?
[Improvement]
Todos

[x] - Adopted the function of auto completion

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-614

How should this be tested?
Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

@tzolov Could you please test this PR with Postrgres, HAWQ and so on? I've heard that you are one of the heavy users.

Great work @jongyoul !
Looks good to me except minor things listed above

@bzz I updated license files. could you please check it again?
https://github.com/apache/incubator-zeppelin/pull/974/commits/99ce452b68c757c1959b9d4b7983f7d81000256e

npm build error

@jongyoul thank you for prompt update!
Looks great to me.

authors: bzz jongyoul


text:
[BUG] ElasticSearch remove the cluster.initial_master_nodes from node's configuration after the cluster has formed

Describe the bug
The Elasticseach reports the following error:
{"@timestamp":"2024-11-27T04:04:32.523Z", "log.level": "WARN", "message":"this node is locked into cluster UUID [CGVDtE9sQWSNrJSmdlm2hQ] but [cluster.initial_master_nodes] is set to [elastic2-mdit-0, elastic2-mdit-1, elastic2-mdit-2]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.8/important-settings.html#initial_master_nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"ES_ECS","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"elastic2-mdit-0","elasticsearch.cluster.name":"elastic2"}

elasticsearcy.yml
cluster:
  name: elastic2
  routing:
    allocation:
      awareness:
        attributes: k8s_node_name
  initial_master_nodes:
  - elastic2-mdit-0
  - elastic2-mdit-1
  - elastic2-mdit-2

discovery:
# the default of discovery.type is multi-node, but can't set it to multi-node explicitly in 7.x version
  seed_hosts:
  - elastic2-mdit-0.elastic2-mdit-headless.default.svc.cluster.local
  - elastic2-mdit-1.elastic2-mdit-headless.default.svc.cluster.local
  - elastic2-mdit-2.elastic2-mdit-headless.default.svc.cluster.local

http:
  cors:
    enabled: true
    allow-origin: "*"
    allow-headers: Authorization,X-Requested-With,Content-Type,Content-Length
  publish_host: ${POD_FQDN}

network:
  host: "0"
  publish_host: ${POD_IP}

node:
  attr:
    k8s_node_name: ${NODE_NAME}
  name: ${POD_NAME}
  store:
    allow_mmap: false
# https://www.elastic.co/guide/en/elasticsearch/reference/7.7/modules-node.html
  roles:
  - master
  - data
  - ingest
  - transform

path:
  data: /usr/share/elasticsearch/data
  logs: /usr/share/elasticsearch/logs

xpack:
  security:
    enabled: "false"
  ml:
    enabled: "false"

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Index can be used to indicate whether the cluster is formed.

repo: apecloud/kubeblocks-addons


text:
Fix Nexmo support 4.0.x

Fix Nexmo SMS support
Nexmo client needs private key if applicationId is set. DisplayBeforePasswordlessAuthenticationAction uses email configuration for sms notifications.
[x] Remove applicationId default value from NexmoProperties
[x] Add privateKeyContentsPath to NexmoProperties
[x] Use SMS configuration for sms notifications (DisplayBeforePasswordlessAuthenticationAction)
[x] Use default applicationId "CAS" if a private key path is set and applicationId not

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

So, I've removed applicationId, it's only used for JWT token generation for phone call with private key. Added signatureSecret for signed sms and updated documentation.

Thank you very much for the contribution. Keep 'em coming!

authors: CLAassistant ftreguer mmoayyed

repo: apereo/cas


text:
Patch 1

I noticed the keywords are inconsistently using bold and non bold letters. I see no reason why to highlight some keywords and some not.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.5 out of 6 committers have signed the CLA.:white_check_mark: mmoayyed:white_check_mark: ilgrosso:white_check_mark: leleuj:white_check_mark: fcolin-odigo:white_check_mark: antoine777:x: MIPMHannesYou have signed the CLA already but the status is still pending? Let us recheck it.

authors: CLAassistant MIPMHannes


text:
Use FHS compliant paths

According to the Filesystem Hierarchy Standard
man pages should be placed under /usr/share/man.

There is a potential conflict when $(PREFIX) and $(prefix) contain different values, but this only is a problem if a packager does stupid things.

I have manually merged this on master. Thanks for you contribution @rscholer :+1:

authors: aperezdc rscholer

repo: aperezdc/dwt

license: mit


text:
Update go runtime to use node 4.3

I already tested it on AWS and it works great.
Let me know if there's something else I need to do to get this approved.

Not sure about that but based on https://aws.amazon.com/blogs/compute/node-js-4-3-2-runtime-now-available-on-lambda/ (shim)[https://github.com/apex/apex/blob/master/shim/index.js] needs to be updated also.

@mthenw I updated to use the new callback argument however it was optional and actually I had to set callbackWaitsForEmptyEventLoop to false otherwise it'd timeout.
it says it's optional here: http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html#nodejs-prog-model-handler-callback
let me know if I should revert this change.

Oh sorry, I didn't read to the end. If it's working without that you can revert last commit.

I tested both(with and without the new callback) and I didn't notice any difference

Let's leave it as it is. Thanks.

I have some stuff I can use to test and see if there's a perf benefit to upgrading or not at some point

I think 8a270e84 does everything necessary. I manually update my apex go deployment to use Nodejs 4.3 via the AWS Console, and everything works as expected.
b9be3f765965 reverted this PR #365 ; I think if 8a270e84 is reapplied it should work fine without the changes in e59845fe2a0d1c2584df96292dda72cb2079f79

I'll have to give it a try with something non-production haha, I don't see a compelling reason to change it really since the shim doesn't do much

authors: dcu mthenw simshanith tj

repo: apex/apex

license: MIT