text:
NIFI-8616 - Migrate PutTCP, PutUDP and PutSplunk to use Netty

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[x] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Just doing some further testing on this but wanted to get a PR up for feedback.

Thanks for the adjustments and updates @thenatog! Verified PutUDP and PutTCP with various settings. +1 Merging.

authors: exceptionfactory thenatog


text:
{bp-15097} litex_serial: use small lock in arch/risc-v/src/litex/litex_serial.c

Summary
reason:
We hope to remove all instances of spin_lock_irqsave(NULL).
Impact
RELEASE
Testing
CI

[Experimental Bot, please feedback here]
No, this PR does not fully meet the NuttX requirements. While it provides a reason for the change, it lacks crucial details.  Here's a breakdown of what's missing:

Summary:

Missing: "What functional part of the code is being changed?"  (e.g., "The scheduler," "Interrupt handling subsystem," specific drivers, etc.)
Missing: "How does the change exactly work?" (e.g., "All instances of spin_lock_irqsave(NULL) are replaced with calls to up_irq_save()," or a more detailed explanation if the solution is more complex.)
Missing: Related issue references (if any).

Impact:  "RELEASE" is insufficient. While it implies a potentially significant impact, it doesn't explain what the impact is.  Each "Impact" sub-item needs a proper "YES" or "NO" and a description if "YES." For example:

Impact on user:  Likely YES.  If this changes interrupt behavior, user code that relies on specific interrupt timings might break. This needs to be explained.
Impact on build: Possibly YES. If changes are made to core kernel files, rebuilds might be required.
Impact on hardware: Potentially YES, depending on which parts of the code are changed. Specific architectures, boards, or drivers affected should be listed.
And so on for all other impact categories.

Testing: "CI" is insufficient. While Continuous Integration is important, the PR needs to show specific testing results.

Missing: Build Host details (OS, CPU, compiler).
Missing: Target details (architecture, board, configuration).
Missing: Actual "before" and "after" testing logs demonstrating the change in behavior and that the issue is fixed.  Just stating "CI" doesn't provide this evidence.

In short, the PR needs to be much more descriptive and provide concrete details for each required section.  The current description offers very little information for reviewers to assess the change properly.

source: gharchive/pull-request

authors: jerpelea nuttxpr

repo: apache/nuttx

license_type: permissive

license_source: github-api


text:
Mysql driver is basically updated

The tests are not green
I need some basic review on shall I proceed fixing the tests
One of the errors is
java.sql.SQLSyntaxErrorException: Syntax error for TIMESTAMP escape sequence '2009-08-25 00:00:00.'.
Caused by: java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
	at java.sql.Timestamp.valueOf(Timestamp.java:257)
	at com.mysql.cj.jdbc.EscapeProcessor.processTimestampToken(EscapeProcessor.java:356)

Only 2 errors remain:
[ERROR] org.apache.openjpa.persistence.query.TestTimeoutException.testQueryTimeOutExceptionWhileQueryingWithLocksOnAlreadyLockedEntities  Time elapsed: 1.135 s  <<< FAILURE!
junit.framework.AssertionFailedError: org.apache.openjpa.persistence.PersistenceException was raised but expected javax.persistence.QueryTimeoutException

[ERROR] org.apache.openjpa.persistence.query.TestTimeoutException.testQueryTimeOutExceptionWhileFindWithLocksOnAlreadyLockedEntities  Time elapsed: 1.149 s  <<< FAILURE!
junit.framework.AssertionFailedError: org.apache.openjpa.persistence.PersistenceException was raised but expected javax.persistence.LockTimeoutException

I have no idea why is that :(((
Any help would be appreciated :)

repo: apache/openjpa


text:
Update the license to Apache License

Since the license of the donated code does not conform to the Apache license, we need to update them.

closed by #6

closed by #6

source: gharchive/issue

repo: apache/openwhisk-vscode-extension


text:
Suppor merge file in C++

I notice there is merge file function in Java. Is there plan to support it in C++?

It seems that you are looking at C++ tools and Java tools. Is there a reason which you cannot use Java tools, @jnwan ?

Since there is no update, let me close this question. We can still continue to have Q&A.

authors: dongjoon-hyun jnwan

repo: apache/orc

license_source: bigquery


text:
HDDS-5401. Add more metrics to ReplicationManager to help monitor replication progress.

What changes were proposed in this pull request?
Add more metrics to ReplicationManager to help monitor replication progress.
Note that metrics are placed in a separated new class.
More detailed description could be found in the JIRA.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5401
How was this patch tested?
extended ut.
manual test.

@ChenSammi please help review this, thanks~

Hi @bshashikant , could you please help review this as Sammi is on vacation this week, thanks~

Thanks @guihecheng  for working on this.  I just left a few comments.
Hi @sodonnel, would you help to have a look at this patch?  There are two changes to the ReplicationManager metrics. One is the metrics name changing from INFLIGHT_REPLICATION to inflightReplication, another is the metrics source name changing from ReplicationManager to ReplicationManagerMetrics.   Though I think the new names are more consistent with current other module metrics style,  we would like to have your thoughts here.

@ChenSammi thanks for mentioning me. I think this change looks generally good aside from the points you already raised. I checked with a few others on the team and changing the metrics names should be ok. I don't think anything is integrated with Ozone that depends on these names so far.

@ChenSammi thanks for mentioning me. I think this change looks generally good aside from the points you already raised. I checked with a few others on the team and changing the metrics names should be ok. I don't think anything is integrated with Ozone that depends on these names so far.

Oh, this is a good news, thanks again for confirmation.

@sodonnel @ChenSammi updated, thanks~
Also invite @JacksonYao287 for review since some stuff of 'move' is changed.

thanks @guihecheng for this work, please fix the CI first

Ah, let's retrigger the CI since these metrics should not fail the mr jobs.

ping @JacksonYao287 @ChenSammi @sodonnel

The last patch LGTM, +1.

Thanks @guihecheng  for the contribution and @sodonnel @JacksonYao287 for th code review.

authors: ChenSammi guihecheng sodonnel

repo: apache/ozone


text:
HDDS-5556. GrpcReplication Client might fail in SCM HA Cluster

What changes were proposed in this pull request?
Add all CA Certs to trust manager while creating GRPC Replication Server between datanodes.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5556
How was this patch tested?
Tested with docker-compose by bringing up the following containers initially in ozonesecure-ha environment:
docker-compose up -d kdc kms scm1.org scm2.org scm3.org recon om1 om2 om3 datanode1
// To change the SCM leader node
docker-compose stop scm1.org

// Start datanode2 to get its cert signed by new SCM leader (scm2.org)
docker-compose up -d datanode2

// Change leader again
docker-compose start scm1.org
docker-compose stop scm2.org

// Start dn3
docker-compose up -d datanode3

// put a file
docker-compose exec om bash
kinit -kt /etc/security/keytabs/om.keytab om/om
ozone sh vol create vol1
ozone sh bucket create vol1/bucket1
ozone sh key put /vol1/bucket1/key1 <path_to_file>

// stop dn3 and remove
docker-compose stop datanode3
docker-compose rm datanode3

// start dn3 as a fresh container
docker-compose up -d datanode3

After this, verified that SCM scheduled replication of the container to datanode3 and it succeeded without any errors.

@bharatviswa504 Thanks for the review. PTAL.

Thank You @vivekratnavel for the contribution.


text:
HDDS-5956. Speed up TestOzoneRpcClientAbstract#testZReadKeyWithUnhealthyContainerReplica

Please see HDDS-5956 description for details.

Thanks @sodonnel for the +1. After retriggering a couple more times I finally got a green run. Will merge this in a minute.


text:
HDDS-8665. Implemented CopyObject for SnapshotInfo and moved out snapshot chain update form OMSnapshotPurgeResponse

What changes were proposed in this pull request?
Issue is that at final stage of snapshot purge we update the snapshot chain and next global & path level snapshots in DB. When we update next global and path level snapshots in DB, it doesn’t not update the cache. SnapshotInfo read happens from the cache. When causes that test failures mentioned in HDDS-8665.
This change contains two main change.

Snapshot Info to implement CopyObject.
Move out SnapshotChain update form OMSnapshotPurgeResponse to OMSnapshotPurgeRequest.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8665
How was this patch tested?
Existing unit tests.

Thanks @hemantk-12 for the fix. Thanks @szetszwo for raising the issue. Thanks @aswinshakil for reviewing this.

@hemantk-12 , @smengcl , thanks a lot for fixing this!

authors: hemantk-12 smengcl szetszwo


text:
PHOENIX-6210 Change credentials for Yetus result postback to github

try 9fa62308-3a11-4abe-a22a-b6bb63d07362

Glad to know this change is coming :)
+1 (non-binding) if corresponding bot account is able to comment.

try 9fa62308-3a11-4abe-a22a-b6bb63d07362

@stoty  you tried this in first commit and then changed it to 683f5dcf-5552-4b28-9fb1-6a6b77cf53dd in 2nd commit?

Yes, I tested 4 different credentials, but none of them worked :(

Sad, message is still same :(
Could not update commit status, please check if your scan credentials belong to a member of the organization or a collaborator of the repository and repo:status scope is selected

GitHub has been notified of this commit’s build result

authors: stoty virajjasani

repo: apache/phoenix


text:
Add segment metadata deletion code to segment deletion path

Whats in the PR:
Added segment metadata deletion from remote store, in the segment deletion path.
Why its needed:
Through PR, we got the ability to upload segment metadata to remote store, which removes the need to download the entire segment to parse metadata, while creating segment metadata in Zk (metadata push in controller). This change here is to ensure that we always clean up segment metadata (catch all deletion), when the associated segment gets deleted from remote store. Segment and segment metadata are stored in the same remote directory.

@snleee - Could you take a look? Thanks !

Codecov Report
Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (6cc1915) 61.65% compared to head (a98dc08) 0.00%.
Report is 10 commits behind head on master.

Files
Patch %
Lines

.../controller/helix/core/SegmentDeletionManager.java
0.00%
12 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12350       +/-   ##
=============================================
- Coverage     61.65%    0.00%   -61.66%
=============================================
  Files          2421     2348       -73
  Lines        131872   128763     -3109
  Branches      20346    19922      -424
=============================================
- Hits          81308        0    -81308
- Misses        44600   128763    +84163
+ Partials       5964        0     -5964

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.54%)
:arrow_down:

skip-bytebuffers-false
0.00% <0.00%> (-61.65%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <0.00%> (-61.66%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter swaminathanmanish

repo: apache/pinot


text:
Ensure FieldConfig.getEncodingType() is never null

This small PR ensures that FieldConfig.getEncodingType() is never null.
That method should in fact never return null in the past given it wasn't annotated with @Nullable, but there was no check to ensure that. This was in fact pretty common when the FieldConfig was deserialized from a JSON document like in TableConfigs.
With this PR, whenever these objects are created their value will be DICTIONARY by default. As seen in DictionaryIndexType.createDeserializer(), this was the logical default value before this PR, but there are some checks that may fail before reaching that code. One of the examples is the recently added check in TableConfigUtils.validateFieldConfigList().
I've been looking for usages of FieldConfig.getEncodingType() that assume the value may be null, but didn't find them.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (38d86b0) 61.74% compared to head (fcc4217) 61.75%.

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #12430   +/-   ##
=========================================
  Coverage     61.74%   61.75%
  Complexity      207      207
=========================================
  Files          2436     2436
  Lines        133108   133108
  Branches      20617    20618    +1
=========================================
+ Hits          82191    82195    +4
+ Misses        44876    44865   -11
- Partials       6041     6048    +7

Flag
Coverage Δ

custom-integration1
<0.01% <0.00%> (ø)

integration
<0.01% <0.00%> (ø)

integration1
<0.01% <0.00%> (ø)

integration2
0.00% <0.00%> (ø)

java-11
61.74% <100.00%> (+0.04%)
:arrow_up:

java-21
27.73% <0.00%> (-33.90%)
:arrow_down:

skip-bytebuffers-false
61.75% <100.00%> (+0.01%)
:arrow_up:

skip-bytebuffers-true
0.00% <0.00%> (-61.61%)
:arrow_down:

temurin
61.75% <100.00%> (+<0.01%)
:arrow_up:

unittests
61.74% <100.00%> (+<0.01%)
:arrow_up:

unittests1
46.85% <100.00%> (-0.04%)
:arrow_down:

unittests2
27.75% <0.00%> (+0.02%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter gortiz


text:
vectorize predicate evaluation

This vectorizes SV predicate evaluation at the same batch size as is optimal for batch iterators.
Note that a little bit of copy and paste is necessary in the predicate evaluators to ensure that applySV can be inlined into the batch loop, which increases the line count here more than might be expected.
Partially addresses #8634

Note that a little bit of copy and paste is necessary in the predicate evaluators to ensure that applySV can be inlined into the batch loop, which increases the line count here more than might be expected.

Just to be sure: The reason why applySV(int, int[], int[]) may not be inlined if it is defined in the superclass is that it calls applySV(int), which is overloaded by each predicate, right?

Note that a little bit of copy and paste is necessary in the predicate evaluators to ensure that applySV can be inlined into the batch loop, which increases the line count here more than might be expected.

Just to be sure: The reason why applySV(int, int[], int[]) may not be inlined if it is defined in the superclass is that it calls applySV(int), which is overloaded by each predicate, right?

Yes, that’s the idea. It won’t show up as a problem until there are three or more predicates per data type though, so it would be easy to craft a benchmark to convince oneself erroneously that this isn’t necessary.

Regarding performance, there's likely to be about 25% improvement here, but it's not that widely applicable because the number of cases where applyAnd is called is quite small:
Benchmark               (_numRows)                                                                        (_query)  (_scenario)  Mode  Cnt   Score   Error  Units
BenchmarkQueries.query     1500000  SELECT RAW_INT_COL FROM MyTable WHERE INT_COL = 1 AND RAW_INT_COL IN (0, 1, 2)   EXP(0.001)  avgt    5   1.866 ± 1.525  ms/op
BenchmarkQueries.query     1500000  SELECT RAW_INT_COL FROM MyTable WHERE INT_COL = 1 AND RAW_INT_COL IN (0, 1, 2)     EXP(0.5)  avgt    5  16.552 ± 4.065  ms/op
BenchmarkQueries.query     1500000  SELECT RAW_INT_COL FROM MyTable WHERE INT_COL = 1 AND RAW_INT_COL IN (0, 1, 2)   EXP(0.999)  avgt    5  14.501 ± 3.665  ms/op

Benchmark               (_numRows)                                                                        (_query)  (_scenario)  Mode  Cnt   Score   Error  Units
BenchmarkQueries.query     1500000  SELECT RAW_INT_COL FROM MyTable WHERE INT_COL = 1 AND RAW_INT_COL IN (0, 1, 2)   EXP(0.001)  avgt    5   1.648 ± 0.434  ms/op
BenchmarkQueries.query     1500000  SELECT RAW_INT_COL FROM MyTable WHERE INT_COL = 1 AND RAW_INT_COL IN (0, 1, 2)     EXP(0.5)  avgt    5  12.627 ± 0.410  ms/op
BenchmarkQueries.query     1500000  SELECT RAW_INT_COL FROM MyTable WHERE INT_COL = 1 AND RAW_INT_COL IN (0, 1, 2)   EXP(0.999)  avgt    5  12.075 ± 0.872  ms/op

Codecov Report

Merging #8759 (0fb2eda) into master (8219766) will decrease coverage by 55.50%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #8759       +/-   ##
=============================================
- Coverage     69.67%   14.16%   -55.51%
+ Complexity     4619      168     -4451
=============================================
  Files          1732     1687       -45
  Lines         90722    88970     -1752
  Branches      13505    13354      -151
=============================================
- Hits          63210    12605    -50605
- Misses        23117    75428    +52311
+ Partials       4395      937     -3458

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
?

unittests2
14.16% <0.00%> (-0.03%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...e/operator/dociditerators/SVScanDocIdIterator.java
0.00% <0.00%> (-86.89%)
:arrow_down:

...ter/predicate/EqualsPredicateEvaluatorFactory.java
0.00% <0.00%> (-83.34%)
:arrow_down:

...icate/FSTBasedRegexpPredicateEvaluatorFactory.java
0.00% <0.00%> (-90.00%)
:arrow_down:

.../filter/predicate/InPredicateEvaluatorFactory.java
0.00% <0.00%> (-79.42%)
:arrow_down:

.../predicate/NotEqualsPredicateEvaluatorFactory.java
0.00% <0.00%> (-78.38%)
:arrow_down:

...lter/predicate/NotInPredicateEvaluatorFactory.java
0.00% <0.00%> (-80.19%)
:arrow_down:

.../operator/filter/predicate/PredicateEvaluator.java
0.00% <0.00%> (ø)

...lter/predicate/RangePredicateEvaluatorFactory.java
0.00% <0.00%> (-85.42%)
:arrow_down:

...predicate/RegexpLikePredicateEvaluatorFactory.java
0.00% <0.00%> (-27.28%)
:arrow_down:

...src/main/java/org/apache/pinot/sql/FilterKind.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1361 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8219766...0fb2eda. Read the comment docs.

Once both this https://github.com/apache/pinot/pull/8764/ are done, I will try to run the same queries in production that I earlier used to gather the profile attached in https://github.com/apache/pinot/issues/8634

authors: codecov-commenter gortiz richardstartin siddharthteotia


text:
[Doc] [Do-not-Merge] Add maxSubscriptionsPerTopic to broker and standalone configurations

Modifications
To align doc with code https://github.com/apache/pulsar/pull/8289, the following changes have been made:

Add maxSubscriptionsPerTopic to both broker and standalone configurations
Copy/paste maxProducersPerTopic, maxConsumersPerTopic and maxConsumersPerSubscription from standalone configurations to broker configurations because these three parameters are in the broker.conf file.

Please kindly note that the same changes will be applied to more historical versions as long as it gets approval.

Documentation

[x] doc

Hi @hangc0276 @Anonymitaet can you please review this doc PR? Thanks.
Please kindly note that the same changes will be applied to more historical versions as long as it gets approval.

Looks good to me.

repo: apache/pulsar


text:
Set function channel to idle to prevent DNS resolution of deleted pod

Motivation
When running the Kubernetes runtime and deleting a function, it is possible to observe the following error log:
15:36:17.025 [worker-scheduler-0] INFO  org.apache.pulsar.functions.utils.Actions - Sucessfully completed action [ Deleting statefulset for function ds/default/gluon-revolut-ord-rsp-assembler ]
15:36:17.405 [grpc-default-executor-307] WARN  io.grpc.internal.ManagedChannelImpl - [Channel<99>: (pf-ds-default-gluon-revolut-ord-rsp-assembler-0.pf-ds-default-gluon-revolut-ord-rsp-assembler.pulsar.svc.cluster.local:9093)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host pf-ds-default-gluon-revolut-ord-rsp-assembler-0.pf-ds-default-gluon-revolut-ord-rsp-assembler.pulsar.svc.cluster.local, cause=java.lang.RuntimeException: java.net.UnknownHostException: pf-ds-default-gluon-revolut-ord-rsp-assembler-0.pf-ds-default-gluon-revolut-ord-rsp-assembler.pulsar.svc.cluster.local: Name or service not known
	at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
	at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
	at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.UnknownHostException: pf-ds-default-gluon-revolut-ord-rsp-assembler-0.pf-ds-default-gluon-revolut-ord-rsp-assembler.pulsar.svc.cluster.local: Name or service not known
	at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1519)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
	at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
	... 5 more
}

This error happens because the ManagedChannel is created to target the StatefulSet pods, and the function worker deletes the pods before shutting down the ManagedChannel. Note that there is nothing about pod deletion that triggers the GRPC client to connect to the functions. The error happens due to the way that GRPC handles DNS and its frequent DNS resolution.
There are two solutions. First, we could shutdown the managed channel first, or we could set the channel to idle and prevent any new DNS resolution (as long as there are new connections). Given that the StatefulSet or the Service could fail to get deleted, it seems simpler to just set the channel to idle and then delete it after successfully deleting the function.
Although, it's worth noting enterIdle is labeled as experimental. It looks like it is implemented in the ManagedChannelImpl to do exactly what we want: stop DNS resolution for as long as the channel stays in "idle" state. Here is the associated Javadoc
  /**
   * Invoking this method moves the channel into the IDLE state and triggers tear-down of the
   * channel's name resolver and load balancer, while still allowing on-going RPCs on the channel to
   * continue. New RPCs on the channel will trigger creation of a new connection.
   *
   *
This is primarily intended for Android users when a device is transitioning from a cellular
   * to a wifi connection. The OS will issue a notification that a new network (wifi) has been made
   * the default, but for approximately 30 seconds the device will maintain both the cellular
   * and wifi connections. Apps may invoke this method to ensure that new RPCs are created using the
   * new default wifi network, rather than the soon-to-be-disconnected cellular network.
   *
   *
No-op if not supported by implementation.
   *
   * @since 1.11.0
   */
  @ExperimentalApi("https://github.com/grpc/grpc-java/issues/4056")
  public void enterIdle() {}

Modifications

Set all channels to idle before deleting the function pods in the K8s function runtime
Fix some copy/pasted log lines that were a bit confusing
Fix typos of sucess.

Verifying this change
This is a trivial change that will not affect the logic of function deletion. It just ensures graceful shutdown and avoids benign errors that might otherwise confuse users.
Does this pull request potentially affect one of the following parts:
This is a backwards compatible change.
Documentation

[x] no-need-doc

This is an internal cleanup.

@freeznet @nlu90 Please help review this PR.

authors: codelipenghui michaeljmarshall


text:
Skip crowdin download

Master issue: #14824
Motivation
The crowdin translations do not get updated often enough to warrant 3 downloads a day, especially given that the download and upload are currently timing out and preventing the english version of the website to get published.
Modifications

Make the crowdin-download step only run once a day.
Add license to the yarn package.json.

Verifying this change
This is a trivial change.

FYI @urfreespace

@Anonymitaet @urfreespace - I think we need to look at upgrading the crowdin cli version, so I created this issue: https://github.com/apache/pulsar/issues/14837. Also, is there a way committers can get access to the crowdin account?

authors: Anonymitaet michaeljmarshall


text:
[broker]tidy update subscriptions dispatcher rate-limiter

Motivation

When updating topic-level and namespace-level  subscriptions dispatcher rate-limiter,  the code is a little different  but we can unify them as one method, such as updateSubscriptionsDispatcherRateLimiter(),  which will make the  code more clear

Modifications
Create  updateSubscriptionsDispatcherRateLimiter() to update subscriptions dispatcher rate-limiter on both topic-level and namespace-level
Verifying this change

[x] Make sure that the change passes the CI checks.

Documentation

[x] no-need-doc

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@AnonHxy Please help create a PR for branch-2.10, I'm not able to cherry-pick it due to lots of conflicts with branch-2.10

@AnonHxy Please help create a PR for branch-2.10, I'm not able to cherry-pick it due to lots of conflicts with branch-2.10

The PR has been created, PTAL @codelipenghui

authors: AnonHxy codelipenghui


text:
[feat][txn] Implement the AbortedTxnProcessor for TransactionBuffer

Master Issue: https://github.com/apache/pulsar/issues/16913
Motivation
Implement an abortedTxnProcessor to handle the storage of the aborted transaction ID.
Modifications
The structure overview:

The main idea is to move the logic of the operation of checking and persistent aborted transaction IDs(take snapshots)  and the operation of updating maxReadPosition into the AbortedTxnProcessor.
And the AbortedTxnProcessor can be implemented in different designs.
Add persistentWorker to handle snapshot persistenting ：

The first four items below are the corresponding four tasks in the figure. The fifth item is not strictly a task, but a part of the first two tasks.

takeSnapshotSegmentAsync -> writeSnapshotSegmentAsync

These two method is used to persist the snapshot segment.

deleteSnapshotSegment

This method is used to delete the snapshot segment.

updateIndexMetadataForTheLastSnapshot

Using to update index metadata (the latest snapshot).

clearSnapshotSegmentAndIndexes

Delete all segments and then delete the index of this topic.

updateSnapshotIndex

Called by the deleteSnapshotSegment and writeSnapshotSegmentAsync. Do update the index after writing snapshot segment.
Called by recovery as a compensation mechanism for updating the index.

Documentation

[x] doc-not-needed
(Please explain why)

Matching PR in the forked repository
PR in forked repository: https://github.com/liangyepianzhou/pulsar/pull/7

Codecov Report

Merging #17847 (7b1b817) into master (0866c3a) will increase coverage by 31.81%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master   #17847       +/-   ##
=============================================
+ Coverage     38.97%   70.79%   +31.81%
+ Complexity     8311      438     -7873
=============================================
  Files           683       26      -657
  Lines         67325     2246    -65079
  Branches       7217      245     -6972
=============================================
- Hits          26239     1590    -24649
+ Misses        38079      483    -37596
+ Partials       3007      173     -2834

Flag
Coverage Δ

unittests
70.79% <ø> (+31.81%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...apache/pulsar/proxy/server/LookupProxyHandler.java
57.75% <0.00%> (-0.44%)
:arrow_down:

.../pulsar/broker/service/AbstractBaseDispatcher.java

...sar/broker/service/persistent/PersistentTopic.java

...ransaction/buffer/impl/InMemTransactionBuffer.java

...ransaction/buffer/impl/TopicTransactionBuffer.java

...nsaction/buffer/impl/TransactionBufferDisable.java

.../metadata/v2/TransactionBufferSnapshotIndexes.java

...ad/jcloud/impl/OffloadIndexBlockV2BuilderImpl.java

...apache/bookkeeper/mledger/ManagedLedgerConfig.java

...apache/pulsar/client/impl/ProducerBuilderImpl.java

... and 648 more

authors: codecov-commenter liangyepianzhou


text:
[fix] Combination of autocreate + forced delete of partitioned topic with active consumer leaves topic metadata inconsistent.

Local fork PR: https://github.com/dlg99/pulsar/pull/2
Motivation
Forced delete of partitioned topic with active consumer on the namespace where the topic autocreate is enabled leaves the namespace in the state where one cannot create partitioned topic with the same name (because it exists already) and cannot delete it (because it does not exist at the same time)
https://github.com/apache/pulsar/pull/17308 -> https://github.com/apache/pulsar/pull/17566 -> this rework of 17308
Modifications
Prevents autocreation of the topic partitions during deletion. Correctly detects partitioned topic as non-existing if namespace level metadata is deleted.
Verifying this change
This change added tests
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes
nothing AFAICT

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed

Bug fix

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

@poorbarcode @eolivelli Please take a look

Codecov Report

Merging #18193 (78b899e) into master (6c65ca0) will increase coverage by 14.94%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #18193       +/-   ##
=============================================
+ Coverage     34.91%   49.86%   +14.94%
- Complexity     5707     8279     +2572
=============================================
  Files           607      586       -21
  Lines         53396    51366     -2030
  Branches       5712     5494      -218
=============================================
+ Hits          18644    25613     +6969
+ Misses        32119    22879     -9240
- Partials       2633     2874      +241

Flag
Coverage Δ

unittests
49.86% <0.00%> (+14.94%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...in/java/org/apache/pulsar/PulsarBrokerStarter.java
0.00% <ø> (ø)

.../org/apache/pulsar/PulsarClusterMetadataSetup.java
0.00% <ø> (ø)

...g/apache/pulsar/PulsarClusterMetadataTeardown.java
0.00% <ø> (ø)

...org/apache/pulsar/PulsarInitialNamespaceSetup.java
0.00% <ø> (ø)

.../main/java/org/apache/pulsar/PulsarStandalone.java
0.00% <0.00%> (ø)

...ava/org/apache/pulsar/PulsarStandaloneBuilder.java
0.00% <ø> (ø)

...ava/org/apache/pulsar/PulsarStandaloneStarter.java
0.00% <ø> (ø)

...ersistentStickyKeyDispatcherMultipleConsumers.java
44.71% <0.00%> (-11.06%)
:arrow_down:

.../pulsar/broker/service/SharedConsumerAssignor.java
61.11% <0.00%> (-7.41%)
:arrow_down:

...er/systopic/NamespaceEventsSystemTopicFactory.java
73.33% <0.00%> (-5.24%)
:arrow_down:

... and 222 more

@poorbarcode Please help review this PR. Thanks.

Two-phase delete, that's a good way to do it.

Maybe after the https://github.com/apache/pulsar/pull/16590 done we can explore this option when it becomes available / stable.
Current approach should be good enough (and cherry-pickable into 2.10, if needed).

@dlg99 - should this have gone to older release branches?

authors: codecov-commenter codelipenghui dlg99 michaeljmarshall


text:
[fix][broker] Fix typos: lister -> listener

Motivation
Fixed some files where "listener" was mistyped as "lister".
Verifying this change

[x] Make sure that the change passes the CI checks.

Documentation

[x] doc-not-needed

@heesung-sn PTAL. I'm not sure if it's intentionally "Lister".

Codecov Report

Merging #21068 (275c8b3) into master (30073db) will increase coverage by 36.40%.
Report is 3 commits behind head on master.
The diff coverage is 68.84%.

@@              Coverage Diff              @@
##             master   #21068       +/-   ##
=============================================
+ Coverage     36.73%   73.14%   +36.40%
- Complexity    12197    32377    +20180
=============================================
  Files          1696     1887      +191
  Lines        130039   139836     +9797
  Branches      14187    15380     +1193
=============================================
+ Hits          47775   102280    +54505
+ Misses        75926    29463    -46463
- Partials       6338     8093     +1755

Flag
Coverage Δ

inttests
24.05% <6.52%> (-0.08%)
:arrow_down:

systests
25.00% <3.62%> (-0.01%)
:arrow_down:

unittests
72.43% <68.84%> (+40.38%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...ache/pulsar/broker/namespace/NamespaceService.java
70.94% <0.00%> (+28.17%)
:arrow_up:

...va/org/apache/pulsar/client/impl/ConsumerBase.java
73.63% <ø> (+22.67%)
:arrow_up:

...he/pulsar/metadata/impl/AbstractMetadataStore.java
84.90% <ø> (+31.69%)
:arrow_up:

...a/org/apache/pulsar/websocket/ConsumerHandler.java
62.50% <ø> (+37.09%)
:arrow_up:

...he/pulsar/client/api/DummyCryptoKeyReaderImpl.java
33.33% <33.33%> (ø)

...r/websocket/service/WSSDummyMessageCryptoImpl.java
66.66% <66.66%> (ø)

...a/org/apache/pulsar/websocket/ProducerHandler.java
65.51% <67.30%> (+34.48%)
:arrow_up:

...sar/broker/authentication/SaslRoleTokenSigner.java
92.30% <100.00%> (+92.30%)
:arrow_up:

...che/pulsar/client/impl/crypto/MessageCryptoBc.java
68.12% <100.00%> (+2.12%)
:arrow_up:

...va/org/apache/pulsar/client/impl/ConsumerImpl.java
76.87% <100.00%> (+21.91%)
:arrow_up:

... and 5 more

... and 1436 files with indirect coverage changes

Merging...

authors: codecov-commenter massakam tisonkun


text:
[improve][pip] PIP-305: Customize DNS servers to use for Pulsar Client

Motivation
A PIP for making Pulsar Client used DNS servers configurable.

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[x] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/apache/pulsar/pull/21227

PIP vote passed. Merging...

authors: diegosalvi tisonkun


text:
Fixed static linking on C++ lib on MacOS

Motivation
Static linking of C++ client lib on MacOS (which is used in building the Python wheel file) got broken in #4071. Making sure we link statically with all the libs minus libz and libcurl.

it's broken in jenkins
 CMakeFiles/Makefile2:351: recipe for target 'examples/CMakeFiles/SampleConsumerListener.dir/all' failed
make[1]: *** [examples/CMakeFiles/SampleConsumerListener.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 81%] Linking CXX shared library _pulsar.so
[ 81%] Linking CXX executable perfConsumer
[ 81%] Linking CXX executable perfProducer
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::internal::WireFormatLite::WriteInt64(int, long, google::protobuf::io::CodedOutputStream*)'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `vtable for google::protobuf::MessageLite'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::MessageLite::ParseFromArray(void const*, int)'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::io::CodedOutputStream::~CodedOutputStream()'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::internal::WireFormatLite::WriteDouble(int, double, google::protobuf::io::CodedOutputStream*)'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::internal::VerifyVersion(int, int, char const*)'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::internal::WireFormatLite::WriteBool(int, bool, google::protobuf::io::CodedOutputStream*)'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to `google::protobuf::internal::WireFormatLite::WriteUInt32(int, unsigned int, google::protobuf::io::CodedOutputStream*)'
../lib/libpulsar.so.2.5.0-SNAPSHOT: undefined reference to

@merlimat  l will change the Milestone to 2.4.3. So we can cut 2.4.2 and if needed
2.4.3 in a few weeks.

we need to exclude the python lib from the static build /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python
	@rpath/_pulsar.so (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python (compatibility version 3.7.0, current version 3.7.0)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)

run java8 tests
run cpp tests

run java8 tests
run cpp tests

run java8 tests
run integration tests

retest this please

run cpp tests
run java8 tests

authors: aahmed-se jiazhai merlimat tuteng wolfstudy


text:
[Reopen][Issue 5597] Retry when getPartitionedTopicMetadata failed

Motivation
Fixes #5597
Add backoff retries when getting partitioned metadata from brokers.
The change in #5734 (copy from #5603) used the wrong time unit when inited Backoff which failed to trigger the retry logic as expected.
Modifications
Correct the time unit and add some useful log.

@murong00  Seems there was a test case testGetPartitionedTopicDataTimeout() , maybe we should improve it to make it really protecting the code.

@jiazhai ok, will improve it later.

@jiazhai fix the unit test, please help to review again.

Thanks @murong00 for the help, over all looks good to me. left some minor comments.

run java8 tests
run cpp tests

run java8 tests
run cpp tests

need fix NPE in ConnectionTimeoutTest:
2019-12-20\T\05:33:18.457 [ERROR] testLowTimeout(org.apache.pulsar.client.impl.ConnectionTimeoutTest)  Time elapsed: 31.139 s  <<< FAILURE!
java.lang.NullPointerException
	at org.apache.pulsar.client.impl.ConnectionTimeoutTest.testLowTimeout(ConnectionTimeoutTest.java:53)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

run java8 tests
run cpp tests

run cpp tests
run java8 tests

run cpp tests
run java8 tests

@sijie The retry logic may skip the limit of lookup throttle which can be set on client or server side (e.g. maxConcurrentLookupRequest,maxLookupRequests), for example unit test testLookupThrottlingForClientByClient() may fail since all consumer is connected.

Meanwhile, fixed some unit test by setting operationTimeout to 1s (default 30s) in order to reduce unit testing time.

retest this please

run java8 tests

run java8 tests

run integration tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests

authors: jiazhai murong00 sijie tuteng


text:
Fix filesystem offload oom

Motivation
to fix filesystem offload oom #6692
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes
Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)
Documentation
Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@congbobo184 thanks for your pull request. I have applied the modification. However, the OOM still occurs.

@congbobo184 thanks for your pull request. I have applied the modification. However, the OOM still occurs.

I think when one entry be written and then release it, if

@congbobo184 thanks for your pull request. I have applied the modification. However, the OOM still occurs.

I think you offload to hdfs is to slowly, and then you produce message is too fast. and then many ledger will offload int the same time. so we maybe should control the read-ahead number or the whole fileSystem offload task number

quote from @congbobo184  "so we maybe should control the read-ahead number" this is what my PR want to do

Add label release/2.5.1 and remove label release/2.5.2

authors: congbobo184 pheecian sijie tuteng


text:
Attach names for all producers/readers in worker service

(If this PR fixes a github issue, please add Fixes #<xyz>.)
Fixes #
(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)
Master Issue: #
Motivation
This pr attaches names for all the producers/readers that worker service utilizes. This is very useful for debugging.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@srkukarni can you also add for the producer in the FunctionMetadataManager

@jerrypeng Its already added.
https://github.com/apache/pulsar/pull/7165/files#diff-e583d3ebd03ff417566e9ecadda7c3a7R436
Or is there other place that you have in mind.

@srkukarni that is the place.  I must have missed it

authors: jerrypeng srkukarni


text:
Does the dashboard need to provide a visual way to configure acl?

FEATURE REQUEST
Hello,
In the year before, I developed a feature that can support config acl int the console for my team, such as following:

Just now，I migrate these code to the dashboard in my local.
I would like to ask this feature is helpful to the community? If it is helpful , can I commit a pr, otherwise I need not going to do that.

Yes, it is helpful.
You can submit a pull request, and the community will review it.
Thanks.

repo: apache/rocketmq-dashboard


text:
Add prehandler and fix some bugs

Add prehandler for preprocessing on the upstream processing link
fix some bugs

Codecov Report

Merging #105 (e902f0b) into main (7a9adcc) will decrease coverage by 0.45%.
The diff coverage is 8.33%.

@@            Coverage Diff             @@
##             main     #105      +/-   ##
==========================================
- Coverage   48.92%   48.47%   -0.46%
==========================================
  Files          98       98
  Lines        4194     4227      +33
  Branches      655      659       +4
==========================================
- Hits         2052     2049       -3
- Misses       1790     1824      +34
- Partials      352      354       +2

Impacted Files
Coverage Δ

...s/protocol/mqtt/handler/MqttDisconnectHandler.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...mqtt/cs/protocol/mqtt/handler/MqttPingHandler.java
88.88% <0.00%> (-11.12%)
:arrow_down:

...tt/cs/protocol/mqtt/handler/MqttPubAckHandler.java
77.77% <0.00%> (-9.73%)
:arrow_down:

...t/cs/protocol/mqtt/handler/MqttPubCompHandler.java
87.50% <0.00%> (-12.50%)
:arrow_down:

...tt/cs/protocol/mqtt/handler/MqttPubRecHandler.java
90.00% <0.00%> (-10.00%)
:arrow_down:

...tt/cs/protocol/mqtt/handler/MqttPubRelHandler.java
88.88% <0.00%> (-11.12%)
:arrow_down:

...t/cs/protocol/mqtt/handler/MqttPublishHandler.java
73.80% <0.00%> (-20.13%)
:arrow_down:

...cs/protocol/mqtt/handler/MqttSubscribeHandler.java
83.67% <0.00%> (-1.75%)
:arrow_down:

.../protocol/mqtt/handler/MqttUnSubscribeHandler.java
78.57% <0.00%> (-2.92%)
:arrow_down:

...rocketmq/mqtt/cs/session/loop/SessionLoopImpl.java
44.68% <ø> (-0.18%)
:arrow_down:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a9adcc...e902f0b. Read the comment docs.

authors: codecov-commenter pingww

repo: apache/rocketmq-mqtt


text:
[ISSUE #3154]delete the unuse code

#3154
Make sure set the target branch to develop
What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.03%) to 54.012% when pulling d6e33764ad65a68ea7bf48f8e4bf714a3ac6a6e9 on yglovewms:master into e4daba5ff5e4b51a81ad690b678370adb3d00632 on apache:develop.

Coverage decreased (-0.01%) to 54.035% when pulling d6e33764ad65a68ea7bf48f8e4bf714a3ac6a6e9 on yglovewms:master into e4daba5ff5e4b51a81ad690b678370adb3d00632 on apache:develop.

authors: coveralls yglovewms

repo: apache/rocketmq


text:
[ISSUE #3924] It may be better to change the method StoreStatsService#initPutMessageDistributeTime() to private.

Make sure set the target branch to develop
What is the purpose of the change
For #3924
Brief changelog

Change the method StoreStatsService#initPutMessageDistributeTime() to private.
Raname the method StoreStatsService#initPutMessageDistributeTime() to  StoreStatsService#resetPutMessageDistributeTime()

Codecov Report

Merging #3925 (a908eea) into develop (5ae4a10) will decrease coverage by 0.10%.
The diff coverage is 60.00%.

@@              Coverage Diff              @@
##             develop    #3925      +/-   ##
=============================================
- Coverage      47.51%   47.41%   -0.11%
+ Complexity      4935     4926       -9
=============================================
  Files            633      633
  Lines          42572    42572
  Branches        5590     5590
=============================================
- Hits           20229    20186      -43
- Misses         19833    19870      +37
- Partials        2510     2516       +6

Impacted Files
Coverage Δ

...a/org/apache/rocketmq/store/StoreStatsService.java
36.97% <60.00%> (+0.56%)
:arrow_up:

.../broker/subscription/SubscriptionGroupManager.java
64.70% <0.00%> (-16.48%)
:arrow_down:

...in/java/org/apache/rocketmq/test/util/MQAdmin.java
38.88% <0.00%> (-5.56%)
:arrow_down:

...org/apache/rocketmq/store/ha/WaitNotifyObject.java
66.07% <0.00%> (-5.36%)
:arrow_down:

...ache/rocketmq/common/stats/MomentStatsItemSet.java
39.13% <0.00%> (-4.35%)
:arrow_down:

...lient/impl/consumer/DefaultMQPushConsumerImpl.java
40.17% <0.00%> (-2.29%)
:arrow_down:

...e/rocketmq/remoting/netty/NettyRemotingServer.java
57.34% <0.00%> (-1.90%)
:arrow_down:

...etmq/client/latency/LatencyFaultToleranceImpl.java
50.00% <0.00%> (-1.29%)
:arrow_down:

...e/rocketmq/remoting/netty/NettyRemotingClient.java
45.48% <0.00%> (-1.13%)
:arrow_down:

...nt/impl/consumer/ConsumeMessageOrderlyService.java
49.64% <0.00%> (-0.71%)
:arrow_down:

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ae4a10...a908eea. Read the comment docs.

Coverage increased (+0.03%) to 51.593% when pulling a908eea11d9b90a351a8349c1b6228d45acbd721 on XiaoyiPeng:issue#3924 into 5ae4a106bdd83848ae12e870e8f0f587bd107500 on apache:develop.

Coverage increased (+0.03%) to 51.593% when pulling a908eea11d9b90a351a8349c1b6228d45acbd721 on XiaoyiPeng:issue#3924 into 5ae4a106bdd83848ae12e870e8f0f587bd107500 on apache:develop.

@duhenglucky ping~  ^_^

Hi, @RongtongJin
Can this PR be merged?

authors: XiaoyiPeng codecov-commenter coveralls


text:
[ISSUE #8647] Fix the issue where lmq cannot update consumer offset

Which Issue(s) This PR Fixes

Fixes #8647
Brief Description

If it is a group of lmq, return true directly during the check.
How Did You Test This Change?

Codecov Report
Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 47.17%. Comparing base (632943e) to head (38b7351).
Report is 2 commits behind head on develop.

Files with missing lines
Patch %
Lines

...oker/subscription/LmqSubscriptionGroupManager.java
0.00%
3 Missing :warning:

...bscription/RocksDBLmqSubscriptionGroupManager.java
0.00%
3 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8648      +/-   ##
=============================================
- Coverage      47.19%   47.17%   -0.02%
- Complexity     11458    11469      +11
=============================================
  Files           1277     1277
  Lines          89425    89436      +11
  Branches       11502    11502
=============================================
- Hits           42201    42191      -10
- Misses         42014    42032      +18
- Partials        5210     5213       +3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: RongtongJin codecov-commenter


text:
RemoteObject needs to have exactly the same properties in server and client objects

Using AMF/RemoteObject you can bring Typed objects from a backend (Java, PHP, ...) that has more properties in the server class that the ones we have in AS3 counterpart class. Said in other words, AS3 class can have all properties in the backend, or only a subset. This is very common and in Flex the deserialization works flawlessly. In Royale the first case (same properties in both side) works, but the second one fails.

@carlosrovira what will happen if you make AS3 class dynamic ?

Hi Piotr, I did not try "dynamic" not Transient. Just the same objects we have in our application that is working in Flex. in AMFBinaryData.as there's a else that goes that route. I solved the current case, we should try as well marking as dynamic. But since the fix was a try-catch needed, I think it should work.

repo: apache/royale-asjs


text:
Can't install helm chart 'skywalking'

Please answer these questions before submitting your issue.

Why do you submit this issue?
[x] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?

Which company or project?
Trying to test Istio's "Book app" with "skywalking"

What happen?
Trying to install skywalking

If possible, provide a way for reproducing the error. e.g. demo application, component version.
helm install --name my-release skywalking
Error: failed to download "skywalking" (hint: running helm repo update may help)
helm search skywalking
No results found

Requirement or improvement

Please describe about your requirements or improvement suggestions.

I think you need helm file in our repo, https://github.com/apache/skywalking-kubernetes/tree/master/helm-chart.
We haven't pushed them into official helm repo, and the recent severl release hasn't provided helm chart by @hanahmily

@innerpeacez Please push new helm for skywalking 6.4.0, thanks.
@DarthRevanXX The new helm3 chart for 6.3.0 is there already.

thanks guys

Trying with helm 2 but getting Error: failed to download "skywalking" (hint: running helm repo update may help). Tried to run helm repo update but no help

helm search skywalking
No results found

Again, there is no helm chart in helm official repo.

You need to use local helm chart.

Ok, sorry

when i use the repo:https://github.com/apache/skywalking-kubernetes/blob/master/README.md
i meet a problem when execute command:"helm dep up skywalking",
the response is "No requirements found in /root/skywalking-kubernetes/chart/skywalking/charts."
how to fix it?

You should pick up "helm 3" instead of "helm 2"
Alen King notifications@github.com 于2019年12月30日周一 下午3:00写道：

when i use the repo:
https://github.com/apache/skywalking-kubernetes/blob/master/README.md
i meet a problem when execute command:"helm dep up skywalking",
the response is "No requirements found in
/root/skywalking-kubernetes/chart/skywalking/charts."
how to fix it?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/apache/skywalking/issues/3559?email_source=notifications&email_token=AAO6UWIIW2GCUC6SG5J2GADQ3GL77A5CNFSM4I5MPV32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZWL7A#issuecomment-569599484,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAO6UWKRZJKYYP35KHKQBSTQ3GL77ANCNFSM4I5MPV3Q
.

--
Hongtao Gao
Apache SkyWalking && Apache ShardingSphere
Twitter, @hanahmily

authors: AlinePyther DarthRevanXX hanahmily wu-sheng

repo: apache/skywalking


text:
Support define multiple OAL configuration

Please answer these questions before submitting pull request

Why submit this pull request?

[ ] Bug fix

[ ] New feature provided

[ ] Improve performance

Related issues

Bug fix

Bug description.

How to fix?

New feature or improvement

Describe the details and related test reports.

Codecov Report

Merging #4748 into master will decrease coverage by 6.56%.
The diff coverage is 82.14%.

@@             Coverage Diff              @@
##             master    #4748      +/-   ##
============================================
- Coverage     50.82%   44.25%   -6.57%
+ Complexity     2651     2445     -206
============================================
  Files          1287      731     -556
  Lines         28109    17979   -10130
  Branches       3060     1728    -1332
============================================
- Hits          14285     7957    -6328
+ Misses        13171     9348    -3823
- Partials        653      674      +21

Impacted Files
Coverage Δ
Complexity Δ

...he/skywalking/oal/rt/output/DispatcherContext.java
100.00% <ø> (ø)
1.00 <0.00> (ø)

...pache/skywalking/oal/rt/parser/AnalysisResult.java
67.85% <ø> (-3.58%)
15.00 <0.00> (ø)

...arch/JaegerStorageModuleElasticsearchProvider.java
37.50% <ø> (+7.50%)
3.00 <0.00> (ø)

...ver/tool/profile/core/mock/MockSourceReceiver.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...sticsearch/StorageModuleElasticsearchProvider.java
64.47% <50.00%> (-1.28%)
9.00 <1.00> (ø)

...icsearch7/StorageModuleElasticsearch7Provider.java
64.61% <50.00%> (-1.52%)
8.00 <1.00> (ø)

...arch/ZipkinStorageModuleElasticsearchProvider.java
37.50% <50.00%> (+7.50%)
3.00 <1.00> (ø)

.../apache/skywalking/oal/rt/parser/ScriptParser.java
68.75% <66.66%> (-24.11%)
2.00 <0.00> (ø)

...skywalking/oap/server/core/CoreModuleProvider.java
80.76% <66.66%> (-0.58%)
8.00 <1.00> (ø)

...oap/server/core/oal/rt/OALEngineLoaderService.java
76.47% <76.47%> (ø)
3.00 <3.00> (?)

... and 899 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8872909...37ce145. Read the comment docs.

I can't update your branch, seems you close it. You have to update the branch by yourself.

@arugal can you elaborate more on how to define more than one OAL scripts, or give us a typical case when a user want to add a customized OAL script? I didn't see any related configurations

@arugal can you elaborate more on how to define more than one OAL scripts, or give us a typical case when a user want to add a customized OAL script? I didn't see any related configurations or docs

This is not part of the user action, just like official_analysis.oal and browser_analysis.oal. It can also be used for meter-system.

can you elaborate more on how to define more than one OAL scripts, or give us a typical case when a user want to add a customized OAL script? I didn't see any related configurations or docs

One is he mentioned above, the other is, I am expecting to separate the OAL into every receiver. Such as if CLR receiver is not active, then CLR metrics process/index are not there automatically. The same story is in the mesh solution, Zipkin receiver, and Jaeger too. Make sense?

Take a look at the storage provider, such as StorageModuleElasticsearchProvider#start.  Storage installation happens there
StorageEsInstaller installer = new StorageEsInstaller(getManager(), config);
installer.install(elasticSearchClient);

The OAL engine is required to process before that, but currently, there is no guarantee, right?

The OAL engine is required to process before that

Yes.

but currently, there is no guarantee, right?

I've moved Installer#install to notifyAfterCompleted , AnnotationScan#scan as well.

Take a look at my PR, don't change in that way. Moving it there could make no index in ttl timer and cache timer, Because the same startup sequence issue too.
You could keep the codes in previous way, I have made the whole thing works.

Please fix the conflict including follow the latest installation logic.

authors: arugal codecov-io kezhenxu94 wu-sheng


text:
[SPARK-13067][SQL] workaround for a weird scala reflection problem

A simple workaround to avoid getting parameter types when convert a logical plan to json。

CC @davies @yhuai @marmbrus
@davies can you try it out locally?

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50310/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #50316 has started for PR 10970 at commit 7e66869.

Test build #50316 has finished for PR 10970 at commit 7e66869.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50316/
Test FAILed.

retest this please

Test build #50320 has started for PR 10970 at commit 7e66869.

Test build #50320 has finished for PR 10970 at commit 7e66869.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50320/
Test FAILed.

Test build #2472 has started for PR 10970 at commit 7e66869.

Test build #2472 has finished for PR 10970 at commit 7e66869.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

"。" replace with "." ?

Tested it locally, test passed. I'm merging this into master, thanks!

@cloud-fan is this fixing a bug? If yes, please add a test somewhere.

Existing tests were failing locally, but not on Jenkins.  We are not sure why.

OK got it. @cloud-fan  Please add that to the pull request description next time.

Sorry I thought the JIRA description tells it, will add it to PR description next time.

Yea the pull request description gets merged as commit description, and is the only thing that sticks around forever.

How can we workaround this problem when using 1.6.1?

This solution does not seem to work for datasets on tables using complex types. e.g. Array[Int] still throws the exception java.lang.AssertionError: assertion failed: Unsound substitution from List(type T, type U) to List()

@atronchi can you create a JIRA and put the code that can reproduce the bug? thanks!

The solution mentioned in [SPARK-17424] by @rdblue fixes this issue.

authors: AmplabJenkins SparkQA atronchi cloud-fan davies marmbrus rxin

repo: apache/spark


text:
[SPARK-16287][SQL] Implement str_to_map SQL function

What changes were proposed in this pull request?
This PR adds str_to_map SQL function in order to remove Hive fallback.
How was this patch tested?
Pass the Jenkins tests with newly added.

Test build #61525 has started for PR 13990 at commit 1f888ab.

Test build #61525 has finished for PR 13990 at commit 1f888ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61525/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61685 has started for PR 13990 at commit fa294bc.

cc: @cloud-fan @rxin

Test build #61685 has finished for PR 13990 at commit fa294bc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61685/
Test PASSed.

Test build #61690 has started for PR 13990 at commit 6b2390d.

Test build #61690 has finished for PR 13990 at commit 6b2390d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StringToMap(child: Expression, pairDelim: Expression, keyValueDelim: Expression)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61690/
Test PASSed.

Test build #61767 has started for PR 13990 at commit ca74f9a.

Test build #61767 has finished for PR 13990 at commit ca74f9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class StringToMap(text: Expression, pairDelim: Expression, keyValueDelim: Expression)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61767/
Test PASSed.

Test build #61806 has started for PR 13990 at commit f7c03c5.

cc @dongjoon-hyun can you help review this

Test build #61815 has started for PR 13990 at commit d1573b6.

Test build #61806 has finished for PR 13990 at commit f7c03c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61806/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61819 has started for PR 13990 at commit 94c18ff.

Sure, @rxin .

Oh, @techaddict . Could you check Hive code again? The default parameter is different from the documentation. We had better follow the code instead of document.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java

I've done my first pass. Feel free to fix the PR.

Test build #61815 has finished for PR 13990 at commit d1573b6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61815/
Test PASSed.

Test build #61819 has finished for PR 13990 at commit 94c18ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61819/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61844 has started for PR 13990 at commit 2725445.

Test build #61844 has finished for PR 13990 at commit 2725445.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61844/
Test PASSed.

Merged build finished. Test PASSed.

Thank you for fast updating, @techaddict !

cc @dongjoon-hyun or @petermaxlee want to take over the pull request and bring it to completion?

@rxin no need, I will update this today.

Test build #62090 has started for PR 13990 at commit cc96fea.

Test build #62090 has finished for PR 13990 at commit cc96fea.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62090/
Test FAILed.

Test build #62091 has started for PR 13990 at commit 0b4419f.

Test build #62091 has finished for PR 13990 at commit 0b4419f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62091/
Test PASSed.

Test build #62166 has started for PR 13990 at commit 1592ffc.

Test build #62166 has finished for PR 13990 at commit 1592ffc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test build #62174 has started for PR 13990 at commit 34db7bb.

Test build #62174 has finished for PR 13990 at commit 34db7bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62174/
Test PASSed.

Test build #62208 has started for PR 13990 at commit 34db9cf.

@cloud-fan all comments addressed.

Test build #62209 has started for PR 13990 at commit 4dff9b4.

Test build #62209 has finished for PR 13990 at commit 4dff9b4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class JavaStructuredNetworkWordCountWindowed
case class CallMethodViaReflection(children: Seq[Expression])
      TypeCheckFailure(s\"class $className not found\")
case class OptimizeMetadataOnlyQuery(
class TextSocketSource(host: String, port: Int, includeTimestamp: Boolean, sqlContext: SQLContext)
abstract class Filter
case class EqualTo(attribute: String, value: Any) extends Filter
case class EqualNullSafe(attribute: String, value: Any) extends Filter
case class GreaterThan(attribute: String, value: Any) extends Filter
case class GreaterThanOrEqual(attribute: String, value: Any) extends Filter
case class LessThan(attribute: String, value: Any) extends Filter
case class LessThanOrEqual(attribute: String, value: Any) extends Filter
case class IsNull(attribute: String) extends Filter
case class IsNotNull(attribute: String) extends Filter
case class And(left: Filter, right: Filter) extends Filter
case class Or(left: Filter, right: Filter) extends Filter
case class Not(child: Filter) extends Filter
case class StringStartsWith(attribute: String, value: String) extends Filter
case class StringEndsWith(attribute: String, value: String) extends Filter
case class StringContains(attribute: String, value: String) extends Filter

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62209/
Test FAILed.

Test build #62210 has started for PR 13990 at commit 0da97da.

Test build #62210 has finished for PR 13990 at commit 0da97da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62210/
Test PASSed.

Test build #62208 has finished for PR 13990 at commit 34db9cf.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62208/
Test PASSed.

Test build #62250 has started for PR 13990 at commit cbc8798.

Test build #62256 has started for PR 13990 at commit e701716.

Test build #62250 has finished for PR 13990 at commit cbc8798.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62250/
Test PASSed.

Test build #62257 has started for PR 13990 at commit 8172bd5.

Test build #62256 has finished for PR 13990 at commit e701716.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62256/
Test PASSed.

Test build #62257 has finished for PR 13990 at commit 8172bd5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62257/
Test PASSed.

@cloud-fan anything else, it good to merge ?

Sorry I was OOO last few days, LGTM except some minor comments, thanks for working on it!

@cloud-fan np, 👍

Test build #62471 has started for PR 13990 at commit 1e35779.

Test build #62471 has finished for PR 13990 at commit 1e35779.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62471/
Test PASSed.

Merged build finished. Test PASSed.

@cloud-fan Comment addressed, test passed 👍

Test build #62681 has started for PR 13990 at commit 8aabd37.

Test build #62681 has finished for PR 13990 at commit 8aabd37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62681/
Test PASSed.

thanks, merging to master and 2.0!

https://github.com/apache/spark/pull/14315 fixed the odd compile error for this.
Is this really something we should be merging in branch 2.0 now? this looks like part of a new feature, and not even obviously something for 2.0.1.

@srowen please see https://issues.apache.org/jira/browse/SPARK-16275, there is an explaination why we wanna merge them into 2.0

authors: AmplabJenkins SparkQA cloud-fan dongjoon-hyun rxin srowen techaddict


text:
[SQL]RowBasedKeyValueBatch reuse valueRow too

What changes were proposed in this pull request?
reuse the cached valueRow in RowBasedKeyValueBatch
How was this patch tested?
existing ut

cc @ooq   @sameeragarwal @davies is it right and necessary?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Thanks for the patch @yaooqinn . As per comments for getValueRow, because getValueRow(id) is always called after getKeyRow(id) with the same id, we use `getValueFromKey(id) to retrieve value row and use the id as a flag for cached/uncached. The patch is unnecessary IMO. Also, with the patch code, it seems value row is not pointing to the correct position?

@ooq Thanks very much for your review.   As we always getKeyRow(id) before we getValueRow(id) with the same id, we don't retreive the keyRow from the batch but use the cached one. IMO, we can mark the valueRow as cached too because no action seems to overwritten it, then we'd rather use the cached valueRow than retrieve a new one .

@yaooqinn Do you have any benchmarks on the performance difference? I think pointTo() is pretty cheap. And does the patch pass the tests? I think valueRow is not updated correctly in your patch.

authors: AmplabJenkins ooq yaooqinn


text:
[SPARK-18943][SQL] Avoid per-record type dispatch in CSV when reading

What changes were proposed in this pull request?
CSVRelation.csvParser does type dispatch for each value in each row. We can prevent this because the schema is already kept in CSVRelation.
So, this PR proposes that converters are created first according to the schema, and then apply them to each.
I just ran some small benchmarks as below after resembling the logics in https://github.com/apache/spark/blob/7c33b0fd050f3d2b08c1cfd7efbff8166832c1af/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala#L170-L178 to test the updated logics.
test("Benchmark for CSV converter") {
  var numMalformedRecords = 0
  val N = 500 << 12
  val schema = StructType(
    StructField("a", StringType) ::
    StructField("b", StringType) ::
    StructField("c", StringType) ::
    StructField("d", StringType) :: Nil)

  val row = Array("1.0", "test", "2015-08-20 14:57:00", "FALSE")
  val data = spark.sparkContext.parallelize(List.fill(N)(row))
  val parser = CSVRelation.csvParser(schema, schema.fieldNames, CSVOptions())

  val benchmark = new Benchmark("CSV converter", N)
  benchmark.addCase("cast CSV string tokens", 10) { _ =>
    data.flatMap { recordTokens =>
      parser(recordTokens, numMalformedRecords)
    }.collect()
  }
  benchmark.run()
}

Before
CSV converter:                           Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
cast CSV string tokens                        1061 / 1130          1.9         517.9       1.0X

After
CSV converter:                           Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
cast CSV string tokens                         940 / 1011          2.2         459.2       1.0X

How was this patch tested?
Tests in CSVTypeCastSuite and CSVRelation

Test build #70403 has started for PR 16351 at commit e72d1bc.

Test build #70404 has started for PR 16351 at commit 22c9a8a.

Test build #70403 has finished for PR 16351 at commit e72d1bc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70403/
Test PASSed.

cc @cloud-fan, could I please ask to take a look? I remember a similar PR was reviewed by you before.

Test build #70404 has finished for PR 16351 at commit 22c9a8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70404/
Test PASSed.

Merged build finished. Test PASSed.

mostly LGTM, do you have some performance numbers about this optimization?

Thank you @cloud-fan, will add some in the PR description soon after cleaning up.

Test build #70461 has started for PR 16351 at commit 3289726.

Test build #70462 has started for PR 16351 at commit c75bd05.

Test build #70463 has started for PR 16351 at commit 192bc6e.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70462/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70461/
Test FAILed.

Merged build finished. Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70463/
Test FAILed.

retest this please

Test build #70519 has started for PR 16351 at commit 192bc6e.

Test build #70519 has finished for PR 16351 at commit 192bc6e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70519/
Test PASSed.

Test build #70566 has started for PR 16351 at commit a036b7f.

Test build #70566 has finished for PR 16351 at commit a036b7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70566/
Test PASSed.

thanks, merging to master!

@cloud-fan Thank you!

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan


text:
[SPARK-33621][SQL] Add a way to inject data source rewrite rules

What changes were proposed in this pull request?

This PR adds a way to inject data source rewrite rules.
Why are the changes needed?

Right now SparkSessionExtensions allow us to inject optimization rules but they are added to operator optimization batch. There are cases when users need to run rules after the operator optimization batch (e.g. cases when a rule relies on the fact that expressions have been optimized). Currently, this is not possible.
Does this PR introduce any user-facing change?

Yes.
How was this patch tested?

This PR comes with a new test.

cc @holdenk @dbtsai @dongjoon-hyun @rdblue @cloud-fan @sunchao @viirya @HyukjinKwon

Test build #132072 has started for PR 30577 at commit a76f4c2.

We should probably wait for clarity on the discussion here before moving on with this one.

cc @gatorsmile

This looks good to me, but I agree that we should change the name if anyone comes up with a better one.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36671/

Test build #132072 has finished for PR 30577 at commit a76f4c2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132072/

I'll wait to update this until we resolve the name issue.

@aokolnychyi . Could you address the existing comments first without waiting for the others? They are orthogonal. You don't need to wait. For example,

This PR breaks Scala 2.13 build.
Test case prefix.

Will do today, @dongjoon-hyun!

Gentle ping, @aokolnychyi .

Will do today, @dongjoon-hyun!

Sorry for the delay, @dongjoon-hyun! I've updated the PR now.

Gentle ping @gatorsmile on the name suggestion discussed here.

Test build #132361 has started for PR 30577 at commit 6b38928.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36961/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36961/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36961/

Test build #132361 has finished for PR 30577 at commit 6b38928.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132361/

Test failures are in HiveThriftHttpServerSuite.

authors: AmplabJenkins SparkQA aokolnychyi dongjoon-hyun rdblue