text:
Fix non-latest partition last flush time cannot recover

Description
Before this fixing, if there is not only one time partition in system, the last flush time of non-latest partition cannot recover because of a redundant partitionFiles.getKey() == latestPartitionId judgement condition.
How to Test
Execute the following sql.
insert into root.sg.d1(time,s1) values(604799999, 1);
flush;
insert into root.sg.d1(time,s1) values(604800000, 1);
flush

Then restart IoTDB and execute
insert into root.sg.d1(time,s1) values(1, 1);

You will see the timestamp 1 was inserted into sequence space.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ea37a1f) 49.25% compared to head (4d1764d) 49.26%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #11999      +/-   ##
============================================
+ Coverage     49.25%   49.26%   +0.01%
- Complexity    26105    26111       +6
============================================
  Files          2933     2933
  Lines        183723   183704      -19
  Branches      22080    22074       -6
============================================
+ Hits          90490    90508      +18
+ Misses        93233    93196      -37

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: HTHou codecov-commenter

repo: apache/iotdb


text:
Jsieve 103

This PR :

Adds activation time for active sieve scipt (necessary for SIEVE vacation extension)
Refactor so that ResourceLocator is a class parameter and reinstanciated on each call
Add tests for vacation.

See https://github.com/apache/james-jsieve/pull/9

No joda-time ?

rework - SieveRepository API should rely on yoda-time ...
Seems like I forgot to push this one...

source: gharchive/pull-request

authors: chibenwa mbaechler

repo: apache/james-project

license_type: permissive

license_source: github-api


text:
MINOR: Change KStreamKstreamOuterJoinTest to use distinct left and right types

This test uses the same value types on the left and right, and so wouldn't be sensitive to a mixup between left and right values. So I changed one of the stream types to Long, and updated the assertions to match.
Because the implementation of the KStreamKstreamJoin is not type-safe, it was unclear just from the code that a mixup was not possible.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@gharris1727 @cadonna -- do we still want to merge this PR?

Yes, why not? It is a good change to the test, no?
I just did not have time to review it since my last review.

Was just wondering if the other PR contained tests that cover this already.
@gharris1727 could you rebase so a new PR build is triggered to make sure everything still works? Happy to reveiw/merge afterwards.

@mjsax It was already up-to-date without merge conflicts, but I synced it up again.

Thanks @gharris1727 -- merged to trunk.

authors: cadonna gharris1727 mjsax

repo: apache/kafka


text:
DONOTMERGE: Instrumentation to debug PR build failures

Trying to get to the bottom of all the jdk11 build failures.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Closing since Ismael figured out the issue.


text:
KAFKA-9441: Add internal TransactionManager

Upfront refactoring for KIP-447.
Introduces TransactionManager that allows to share a producer over multiple tasks and track the TX status.
Call for review @guozhangwang @abbccdda

Are the test failures in org.apache.kafka.streams.TopologyTestDriverTest relevant?

Updates this PR -- also fixed the TDD issue.

Rebase to resolve merge conflict. Call for review @guozhangwang

Thanks for the review @abbccdda -- updated this PR.

Java 8: org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenDeleted
Java 11: org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.shouldForwardAllDbOptionsCalls
(No need to retest atm. Waiting for reviews.)

retest this please

I couldn't find one of my important previous comments anywhere, so I will just rephrase it here:
I could see that we still need to keep eosEnabled flag inside RecordCollector, and we are overusing TransactionManager for a lot of non-transaction related work such as send, flush, etc.
I'm thinking whether it makes sense to define a ProgressCommitManager instead to take in both Producer and Consumer, and the only time it shows up in RecordCollector is to do the commit. This way we could internalize all the EOS related logic inside ProgressCommitManager and record collector shall still own normal producer operations. And collector should no longer be aware of EOS logic.
WDYT?

Update this PR. Let me know if I addresses all comments sufficiently. If yes, I hope we can merge if Jenkins is green.

LGTM.

Rebase this PR to resolve merge conflicts.

authors: abbccdda guozhangwang mjsax


text:
[KARAF-7423] Add logback configuration option

I've added an xml based configuration service implementation for logback configurations and corresponding unittests and documentation.

Are we sure we want logback by default ? I would rather prefer to keep log4j2 style by default, and document how to have logback instead. Or eventually create another framework/static feature (with logback classifier for instance) to easily use logback.
Thoughts ?

The default option is still log4j2. The logback service just also exists now and can be included if needed.

I understand the confusion. I made a mistake in including pax-logging-logback in the wrong place in assembly I'll remove it and update my pull request

retest this please

If you don't mind, I will squash the 5 commits in a single one.

If you don't mind, I will squash the 5 commits in a single one.

That would be fine.

authors: i-thiemann jbonofre

repo: apache/karaf


text:
[KARAF-5649] Add a feature for the manual

Adding a documentation feature in order to published the generated manual in html with the http service.

The documentation feature doesn't validate, it's because the manual module is built only with the release profile.

retest this please

retest this please

As the manual profile is defined after the assemby, the error is "expected".
I propose the following:

Remove the manual profile in the main pom.xml
Include manual module in the main pom.xml (reactor) before the assembly

It means the manual will be systematically built, which is not bad IMHO. Especially, the PDF generation is isolated in a dedicated profile.

I removed the manual profile from pom parent and include the manual module in the reactor.
Ready for the review ;)

authors: fpapon jbonofre


text:
KYLIN-5291 introducing redis as a distributed cache for kylin

Proposed changes

Build the AbstractRedisClient class for introduction by other redis clients
Introduce jedis client
Create a MultiLevelCacheManager. Data is cached locally and remotely to prevent cache avalanches, and can be downgraded to local when distributed cache is unavailable

Branch to commit

[ ] Branch kylin3 for v2.x to v3.x
[x] Branch kylin4 for v4.x
[ ] Branch kylin5 for v5.x

Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have created an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[x] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin.apache.org or dev@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Hi, here I share a completed version on kylin5 branch for reference : https://github.com/apache/kylin/blob/kylin5/src/common-service/src/main/java/org/apache/kylin/rest/cache/RedisCache.java .

Besides, where can I find the config entry of enable/disable redis cache?

Besides, where can I find the config entry of enable/disable redis cache?

Originally, the bean was configured in server/src/main/resources/applicationContext.xml to enable/disable redis, but this method is not very good, I will modify it to enable/disable redis cache through the parameters of the configuration file

@SeventeenToOne , I am glad to see you have fix these issues, but do you know why UT is not passed in Travis CI?

authors: SeventeenToOne hit-lacus

repo: apache/kylin

license_source: bigquery


text:
Change default EC spec to secp521r1 for internal session variable signature

Why are the changes needed?

change the default EC spec from secp256k1 to secp521r1, as secp256k1 not supported on Java 17

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #4850 (73e0dad) into master (b8ee006) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #4850      +/-   ##
============================================
- Coverage     58.09%   58.05%   -0.04%
  Complexity       13       13
============================================
  Files           584      584
  Lines         32593    32593
  Branches       4321     4321
============================================
- Hits          18934    18922      -12
- Misses        11839    11849      +10
- Partials       1820     1822       +2

Impacted Files
Coverage Δ

.../main/scala/org/apache/kyuubi/util/SignUtils.scala
87.87% <100.00%> (ø)

... and 8 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks, merged to master.

Docs about “Recommended Curves” in the SunEC Provider

for Oracle Java 8: https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunEC
for Oracle Java 11: https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-091BF58C-82AB-4C9C-850F-1660824D5254
for Oracle Java 17: https://docs.oracle.com/en/java/javase/17/security/oracle-providers.html#GUID-091BF58C-82AB-4C9C-850F-1660824D5254

Curiously, both secp256k1 and secp521r1 are on the list.
Not sure whether the same for the temurin, and also wondering why secp256k1 failed on temurin java 17.
cc @pan3793

secp256k1 failed on Oracle JDK 17.0.4 on my laptop
(kyuubi) ➜  apache-kyuubi git:(jdk-17) java -version
java version "17.0.4" 2022-07-19 LTS
Java(TM) SE Runtime Environment (build 17.0.4+11-LTS-179)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.4+11-LTS-179, mixed mode, sharing)

OK, and could you check secp384r1 and secp521r1 on Oracle java 17 as well?

secp384r1 and secp521r1 are fine

repo: apache/kyuubi


text:
SoftLayer/IBM provider is broken

Summary
Can't obtain a driver for SOFTLAYER provider
Detailed Information
I'm using Python 3.9/3.10 and apache-libcloud==3.8.0
When trying to get driver for the SOFTLAYER provider the following error obtained:
from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver
cls = get_driver(Provider.SOFTLAYER)
  File "/usr/local/lib/python3.10/enum.py", line 437, in __getattr__
    raise AttributeError(name) from None
AttributeError: SOFTLAYER

I tried to debug it and noticed that the cls object doesn't include the SOFTLAYER member. It includes the IBM instead. But when I try to use the Provider.IBM, the following exception is displayed:
Exception: Provider no longer supported: http://www.ibm.com/midmarket/us/en/article_cloud6_1310.html, please visit: IBM SmartCloud Enterprise has been deprecated in favour of IBM SoftLayer Public Cloud, please use the SOFTLAYER provider.

I've noticed that the code related to SoftLayer was just removed between tags 3.7.0 and 3.8.0.
For me, it is OK to use 3.7.0, but please mention such breaking changes in the release notes.

source: gharchive/issue

repo: apache/libcloud


text:
Unset the http_proxy environment variable before running tests.

Unset the http_proxy environment variable before running tests.
Description
Two connection tests (test_constructor and test_connection_to_unusual_port)
assume that there is no external real proxy in the test environment. Turns
out LibcloudConnection() will honor a http_proxy shell variable if it's set,
so if the test suite is run in such an environment, these tests will fail:
self = <libcloud.test.test_connection.BaseConnectionClassTestCase testMethod=test_connection_to_unusual_port>

    def test_connection_to_unusual_port(self):
        conn = LibcloudConnection(host='localhost', port=8080)
>       self.assertEqual(conn.proxy_scheme, None)
E       AssertionError: 'http' != None

libcloud/test/test_connection.py:107: AssertionError

self = <libcloud.test.test_connection.BaseConnectionClassTestCase testMethod=test_constructor>

    def test_constructor(self):
        conn = LibcloudConnection(host='localhost', port=80)
>       self.assertEqual(conn.proxy_scheme, None)
E       AssertionError: 'http' != None

libcloud/test/test_connection.py:80: AssertionError

I also added a test for this behavior (the honoring of http_proxy) to the existing test_constructor test.
Status

done, ready for review

Checklist (tick everything that applies)

[ ] Code linting (required, can be done after the PR checks)
[ ] Documentation
[x] Tests
[ ] ICLA (required for bigger changes)

Turns out I was able to do it now. Thanks @panlinux! Merged in trunk.

authors: panlinux pquentin


text:
FreeMarker integration is broken on Windows due to path issues

As this failed CI run indicates:
java.lang.RuntimeException: failed exporting release from directory `D:\a\logging-log4j2\logging-log4j2\src\changelog\2.0-alpha1`
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.main (ChangelogExporter.java:71)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:279)
    at java.lang.Thread.run (Thread.java:750)
Caused by: java.lang.RuntimeException: failed rendering template `D:\a\logging-log4j2\logging-log4j2\src\changelog\2.0-alpha1\.changelog.adoc.ftl` to file `D:\a\logging-log4j2\logging-log4j2\target\generated-sources\site\asciidoc\changelog\2.0-alpha1.adoc`
    at org.apache.logging.log4j.changelog.exporter.FreeMarkerUtils.render (FreeMarkerUtils.java:80)
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.exportRelease (ChangelogExporter.java:170)
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.exportRelease (ChangelogExporter.java:139)
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.main (ChangelogExporter.java:63)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:279)
    at java.lang.Thread.run (Thread.java:750)
Caused by: freemarker.template.TemplateNotFoundException: Template not found for name "D:\\a\\logging-log4j2\\logging-log4j2\\src\\changelog\\2.0-alpha1\\.changelog.adoc.ftl".
The name was interpreted by this TemplateLoader: FileTemplateLoader(baseDir="D:\", canonicalBasePath="D:\").
Warning: The name contains backslash ("\") instead of slash ("/"); template names should use slash only.
    at freemarker.template.Configuration.getTemplate (Configuration.java:[28](https://github.com/apache/logging-log4j2/actions/runs/3943007132/jobs/6747337526#step:9:29)83)
    at freemarker.template.Configuration.getTemplate (Configuration.java:2685)
    at org.apache.logging.log4j.changelog.exporter.FreeMarkerUtils.render (FreeMarkerUtils.java:63)
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.exportRelease (ChangelogExporter.java:170)
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.exportRelease (ChangelogExporter.java:139)
    at org.apache.logging.log4j.changelog.exporter.ChangelogExporter.main (ChangelogExporter.java:63)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:279)
    at java.lang.Thread.run (Thread.java:750)

Fixed in fd8462341106f6f522c7508e183a1c14b638fe7d.


text:
Make DocValuesIterator class public

This change makes DocValuesIterator public so we use the advanceExact() method on a "generic" doc values iterator.

This has come up not just once but twice.  Even then I suspect maybe a third might be somewhere ;-)
https://issues.apache.org/jira/browse/LUCENE-9203
Discuss in JIRA.

and similar to those previous 2 times, where it turned out to be a bad idea, i think it might be here too. why is this method needed? if we expose it, it will make it harder to switch docvalues to use block-level postings-like compression (see recent dev discussions)

Sorry, I didn't search for the existing issues before opening this PR. My need is a stronger upper type bound than DocIdSetIterator for docValues iterators. I am okay to close this PR if it prevents future optimizations.

authors: dnhatn dsmiley rmuir

repo: apache/lucene


text:
LUCENE-9907: Make MonotonicBlockPackedReader/writer depend on DirectReader/writer

MonotonicBlockPackedReader and MonotonicBlockPackedWriter  are used in the FixedGapTermsIndexReader and FixedGapTermsIndexWriter respectively. Those readers/ writers from internally from PackedInts#getReader. This PR removes that dependency by using internally DirectReaders and DirectWriters instead.

Thanks @jpountz, I did not realise about that constraint.
I took a different approach: Similarly to Packed64, I read the data into a backing byte array and implement a get method that works directly on that array. MonotonicBlockPackedReader seems to be always used on heap (direct = false), therefore I remove the flag.


text:
Updates to k64f aka kinetis trng driver

trng: k64f: rename to trng_kinetis - This makes it less specific, since the driver should also work on other Kinetis families.

trng: kinetis: update driver for newer families - Older Kinetis families used the RNGA HW module for random number generation, newer models have a slightly different HW module called TRNG; this commit adds TRNG API usage compatibility, and adds syscfg to allow BSPs to choose from one of the HW modules.

hw: bsp: frdm-k82f: enable TRNG usage

trng: kinetis: remove delay in poller - This delay was causing a race condition on trng_test during console_printf, and can be safely removed.

trng: kinetis: seed RNGA with SIM->UIDL - This follows the recommendation from NXP SDK.

Fix for CI was submitted here: https://github.com/apache/mynewt-mcumgr/pull/113

repo: apache/mynewt-core


text:
Ability for ble_gatts_notify_custom() to wait for new packet buffers

I suggest a new feature where a caller can wait for new packet buffers and specify a timeout limiting how long to wait for a packet buffer or queue entry to become available. I have an idea about how to implement this, but wanted to propose it here first.
The problem I face is caused by two threads calling ble_gatts_notify_custom() on FreeRTOS. One low-priority thread generates packets at up to 250Hz(many will fail to enqueue to the stack, and this is ok). The high-priority thread is generating packets at 1Hz, but when it calls ble_gatts_notify_custom(), BLE_HS_ENOMEM almost always occurs because the other thread has kept the queue full, so the 1Hz packets are usually dropped too.
If I'm understanding the problem right, adding the ability to wait for new packet buffers or queue slots would allow my 1Hz high-priority thread to block a short while until a connection event frees some room, and my 250Hz low-priority thread would continue to behave as it does now, usually keeping the queue full, but never blocking if the queue is already full.
Another use-case would be a serial port to BLE bridge, where the caller of ble_gatts_notify_custom() may want to block to allow the serial port hardware flow control to throttle the other end of the serial link according to BLE buffer availability.

I am also using NimBLE for BLE development but on RIOT OS, and yes, I agree with this post, since I am facing the same issue now.

repo: apache/mynewt-nimble


text:
publish live website

changing the branch and asf file to publish live.
Should remove the all content of asf-site branch on netbeans-website

lib/rssfeed.js will do that looking for tag blogentry and procude the xml file.  Will merge wait until it's published, and then remove the antora branch

repo: apache/netbeans-antora


text:
NIFI-786 AWS credential refactoring and enhancements

Changes to AWS credential handling:

Refactoring the creation of AWS Credentials into a factory class for use by AWSCredentialsProviderControllerService (included) and future shared use by processors or other components.
Centralized the PropertyDescriptors used for AWS Credential configuration to standardize behavior and improve documentation.
Improved self-documentation by making Default Credentials an explicit and visible option, while preserving the behavior of using it implicity if no other credential type is configured.
In this commit, the explicit Use Default Credential option defaults to "false", to maintain backward compatibility upgrading to this implementation.
Credential enhancements
New credential option - Named Profile
New credential option - Anonymous
Added External ID as an optional parameter for Assume Role

I'm starting to review and test this.

This is really nice, it was great being able to just provide the AWS CLI profile and have setting automatically picked up.
Great improvement, thanks James!

@jskora, thanks for your help reviewing and merging.

authors: jskora jvwing

repo: apache/nifi


text:
NIFI-1915 Fixed '$' escape logic.

Fixed '$' escape logic (if not followed by number) to use Matcher.quoteReplacement(java.lang.String) to
suppress its special meaning so it could be treated as literal.

LGTM +1


text:
NIFI-11773 - Upgrade groovy to 3.0.18

Summary
NIFI-11773 - Upgrade groovy to 3.0.18
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[ ] Apache NiFi Jira issue created

Pull Request Tracking

[ ] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[ ] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

+1 LGTM, checked the release notes and noticed a potential breaking change GROOVY-8164, but it appears to be an improvement versus a backwards-incompatible change (adding support for Java static interface methods). Thanks for the upgrade! Merging to support/nifi-1.x and main


text:
NIFI-2609: Ensure that we handle URIs for Remote Process Groups that …

…do not have a path of /nifi or /nifi/

Reviewing....

Looks good, will merge to master.


text:
{Bp-13906} local_sock: fix accept use-after-free

Summary
we should get next waiter before acceptor released
Impact
RELEASE
Testing
CI

Failed in sim-02:
In file included from /github/workspace/sources/apps/system/uorb/sensor/gesture.h:28,
                 from /github/workspace/sources/apps/system/uorb/sensor/gesture.c:25:
Error: /github/workspace/sources/apps/system/uorb/sensor/gesture.c:40:33: error: invalid application of 'sizeof' to incomplete type 'struct sensor_wake_gesture'
   40 | ORB_DEFINE(sensor_wake_gesture, struct sensor_wake_gesture,
      |                                 ^~~~~~
/github/workspace/sources/apps/system/uorb/uORB/uORB.h:160:12: note: in definition of macro 'ORB_DEFINE'
  160 |     sizeof(structure), \
      |            ^~~~~~~~~

https://github.com/apache/nuttx/actions/runs/11254577963/job/31292376197?pr=13996

Failed in sim-02:
In file included from /github/workspace/sources/apps/system/uorb/sensor/gesture.h:28,
                 from /github/workspace/sources/apps/system/uorb/sensor/gesture.c:25:
Error: /github/workspace/sources/apps/system/uorb/sensor/gesture.c:40:33: error: invalid application of 'sizeof' to incomplete type 'struct sensor_wake_gesture'
   40 | ORB_DEFINE(sensor_wake_gesture, struct sensor_wake_gesture,
      |                                 ^~~~~~
/github/workspace/sources/apps/system/uorb/uORB/uORB.h:160:12: note: in definition of macro 'ORB_DEFINE'
  160 |     sizeof(structure), \
      |            ^~~~~~~~~

https://github.com/apache/nuttx/actions/runs/11254577963/job/31292376197?pr=13996

need update apps folder.

please merge the apps PRs

please merge the apps PRs

authors: jerpelea lupyuen xiaoxiang781216

repo: apache/nuttx


text:
drivers: usbhost_hub: usb communication memory should use DRVR_ALLOC

usbhost_hub: usb communication memory should use DRVR_ALLOC
Summary
DRVR_CTRLIN buffer must have been allocated using DRVR_ALLOC.  For example, some mcu memory should align cache line.
Impact
only usb host hub.
Testing

MacOS is breaking again


text:
HDDS-4171. DEFAULT_PIPELIME_LIMIT typo in MiniOzoneCluster.java

What changes were proposed in this pull request?
DEFAULT_PIPELIME_LIMIT was changed to DEFAULT_PIPELINE_LIMIT in MiniOzoneCluster.java
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4171?jql=project in (HDDS%2C DAFFODIL) AND status in (Open%2C "In Progress") AND text ~ "typos"
How was this patch tested?
N/A since a typo

Thank you for taking this up @shashank195! the changes look good to me, awaiting green CI.

Thanks @shashank195 for the patch, @JyotinderSingh for the review.

How was this patch tested?
N/A since a typo

@shashank195 Note that testing such a change can be as simple as compiling the project.
Also, if you enable workflows in your fork, you can link to a successful CI run.  Here is an example from my fork: https://github.com/adoroszlai/hadoop-ozone/actions/runs/1979061759

authors: JyotinderSingh adoroszlai shashank195

repo: apache/ozone


text:
HDDS-9676. SORT_DATANODE audit log should include request details

What changes were proposed in this pull request?
SCM's audit log entry for sort datanodes request does not have information about the request (client, node list).  This change adds them.
https://issues.apache.org/jira/browse/HDDS-9676
How was this patch tested?
Checked audit log entry, example:
2023-11-13 08:41:13,856 | INFO  | SCMAudit | user=hadoop | ip=10.5.0.70 | op=SORT_DATANODE {client=10.5.0.71, nodes=[6a9a42ad-9ecd-4e96-bf45-5255f6be66bf, a12806c3-6e7f-479a-be63-20cb3ecb0eb4, cfac92e5-cdce-471d-a995-e6f4dc22ea6f]} | ret=SUCCESS |

CI:
https://github.com/adoroszlai/ozone/actions/runs/6847145541

Thanks @sodonnel for the review.


text:
[Feature] CDC Ingestion should support -U and +U messages

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
I want that reading -U and +U messages from audit log table of paimon table with input changelog producer, but paimon CDC parser does not produce -U and +U messages.
Solution
I suggest paimon CDC parser produce -U and +U messages for the downstream tasks.
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Hi @JingsongLi,  Should CDC Ingestion support -U and +U messages ?

repo: apache/paimon


text:
PARQUET-671: performance improvements for rle/bit-packed decoding

Testing on my own data shows an order-of-magnitude improvement.
I separated the commits for clarity, each one gives an imcremental improvement.
The motivation for the last commit (allowing NULL for def_levels/rep_level) is a workaround for Spark  which doesn't seem to be able to generate columns without def_level, even when a column is specified as "not nullable".

This is great, thank you! I will review as soon as I can — optimizing the bit decoding was high on my list of performance improvements.
Can you open a JIRA on issues.apache.org for this and prefix your PR title with "PARQUET-XXX:" (where XXX is the JIRA number?)

Left some minor comments, and running clang-format would be good, but otherwise this is great. Any chance you would want to add a gbenchmark case? Doesn't need to be in this patch if you want to create a separate JIRA as a reminder to add a benchmark

Thanks for the review!  I pushed a commit to fix the style. I'm working on my dev environment so I can run clang-format and look into benchmarks.

I think I addressed every comment. I can squash commits once everyone is OK with the current state.

+1

No need to squash; we have a patch tool for that

+1, merging this despite the small nit

authors: edani majetideepak wesm

repo: apache/parquet-cpp


text:
parquet-tools doesn't filter _SUCCESS file

Currently parquet-tools command fails when input is a directory with _SUCCESS file from mapreduce. Filtering those out like ParquetFileReader does fixes the problem.
parquet-cat /tmp/parquet_write_test
Could not read footer: java.lang.RuntimeException: file:/tmp/parquet_write_test/_SUCCESS is not a Parquet file (too small)

$ tree /tmp/parquet_write_test
/tmp/parquet_write_test
├── part-m-00000.parquet
└── _SUCCESS

Reporter: Neville Li / @nevillelyh
Note: This issue was originally created as PARQUET-142. Please see the migration documentation for further details.

Ryan Blue / @rdblue:
Issue resolved by pull request 89
https://github.com/apache/incubator-parquet-mr/pull/89

repo: apache/parquet-java


text:
Document how a schema can have columns splitted over different files

In the design overview is stated that the format supports that columns of a given schema are splitted over several files .
To date ,other than the reference in the parquet homepage there are no other references to these feature , not even if it is actually implemented.
As it currently stands , I believe that for a given row group you cannot have some columns in one file and other column group in another file belonging to the same schema .
If these feature was actually implemented it would open the door to faster data updates which affect a single column in very wide tables, which is typical on big data use cases .

Reporter: Carlos Diogo
Note: This issue was originally created as PARQUET-1987. Please see the migration documentation for further details.

Micah Kornfield / @emkornfield:
This was discussed a little bit on https://github.com/apache/parquet-format/pull/164 there is a proposed implementation for C++ at https://github.com/apache/arrow/pull/8130 but there is at least in having a clear specification if people are willing to provide reference implementations.

Carlos Diogo:
Looking at the mentioned discussions it seem that the ability to columns groups in different files , the same way row groups are in different files could have a wide application advantage.
For me , the use case is pretty simple. If i have 1000 columns in a parquet schema and i split it in files of 100 columns and then i need to "update" a single one, i only need to re-write a file with 1/10 of the data. Furthermore this would not have any impact on columns compression of page stats.
This is a critical use case (slow changing dimensions with low cardinality) that both Delta, Iceberg or Hudi fail to address  as all of them require a full row to be rewritten.
Nevertheless , the question is only then if the statement on the Parquet home page is accurate as i assume that the functionality is far from being completed.

Micah Kornfield / @emkornfield:
CC @raduteo


text:
One GsubWorker per font

Load GsubWorker only once per font, by maintaining a Map of GsubWorkers.
Thanks,
Palash.

Commit details:
Commit 1830346 from Tilman Hausherr in branch 'pdfbox/trunk'
[ https://svn.apache.org/r1830346 ]

repo: apache/pdfbox


text:
PHOENIX-4817 Fixed Phoenix Tracing Web Application

Fixed check null, trace table name, webapp path, column names. Fixed traceserver.py

Fixed Phoenix Tracing Web Application, fixed traceserver.py

traceserver.py start        NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet

It's ok, just info message, try to open localhost:8864
@mygood

Tracing List is ok  but  Trace Count Chart Graph
Request URL: http://192.168.178.31:8864/trace/?action=getCount
Request Method: GET
Status Code: 500 Unknown tracename: description

see trace log

It's ok, clone my branch and compile only web application, then rename jar for your version

Thank you very much for the problem.
@MrSandmanRUS

Hi @MrSandmanRUS
I have a problem apache-phoenix-4.11.0-HBase-1.1-bin, hadoop-2.7.1
1.hbase-site.xml modification

phoenix.trace.frequency
always

2. Copy $PHOENIX_HOME/bin/hadoop-metrics2-hbase.properties to $HBASE_HOME/conf

The client writes data but SYSTEM.TRACING_STATS has no data to write.

You use version from apache phoenix 、hbase 、hadoop
@MrSandmanRUS

Yes
@mygood

I have a problem that the data does not into SYSTEM.TRACING_STATS
0: jdbc:phoenix:localhost:2181:/hbase> trace on;
+----------------------+
|       trace_id       |
+----------------------+
| 2701156331755118993  |
+----------------------+
1 row selected (0.001 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> upsert into T(a,b) values(2,2);
1 row affected (0.015 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> trace off;
+----------------------+
|       trace_id       |
+----------------------+
| 2701156331755118993  |
+----------------------+
1 row selected (0.001 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> trace off;
+-----------+
| trace_id  |
+-----------+
+-----------+
No rows selected (0 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> select * from SYSTEM.TRACING_STATS where trace_id=2701156331755118993;
+-----------+------------+----------+--------------+-------------+-----------+-----------+--------+--------+
| TRACE_ID  | PARENT_ID  | SPAN_ID  | DESCRIPTION  | START_TIME  | END_TIME  | HOSTNAME  | COUNT  | COUNT  |
+-----------+------------+----------+--------------+-------------+-----------+-----------+--------+--------+
+-----------+------------+----------+--------------+-------------+-----------+-----------+--------+--------+
No rows selected (0.109 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> select * from SYSTEM.TRACING_STATS;
+-----------+------------+----------+--------------+-------------+-----------+-----------+--------+--------+
| TRACE_ID  | PARENT_ID  | SPAN_ID  | DESCRIPTION  | START_TIME  | END_TIME  | HOSTNAME  | COUNT  | COUNT  |
+-----------+------------+----------+--------------+-------------+-----------+-----------+--------+--------+
| 2         | 2          | 2        | upsert       | 20180720    | 20180721  | host1     | 20     | 30     |
| 3         | 3          | 3        | upsert       | 20180720    | 20180721  | host1     | 10     | 30     |
+-----------+------------+----------+--------------+-------------+-----------+-----------+--------+--------+
2 rows selected (0.098 seconds)
@MrSandmanRUS

Add to hbase-site.xml:

phoenix.trace.enabled
true

trace on not data into trace table
I use apache-phoenix-4.13.1-HBase-1.1-bin and hbase-1.1.6 and hadoop-2.7.1
cp $PHOENIX_HOME/phoenix-4.13.1-HBase-1.1-server.jar $HBASE_HOME/lib
cp $PHOENIX_HOME/bin/hadoop-metrics2-hbase.properties $HBASE_HOME/conf
cp $PHOENIX_HOME/bin/hadoop-metrics2-phoenix.properties $HBASE_HOME/conf
add hbase-sit.xml

phoenix.trace.frequency
always

phoenix.trace.statsTableName
SYSTEM.TRACING_STATS

phoenix.trace.frequency
true

run sqlline.py
create SYSTEM.TRACING_STATS
CREATE TABLE SYSTEM.TRACING_STATS (
trace_id BIGINT NOT NULL,
parent_id BIGINT NOT NULL,
span_id BIGINT NOT NULL,
description VARCHAR,
start_time BIGINT,
end_time BIGINT,
hostname VARCHAR,
tags.count SMALLINT,
annotations.count SMALLINT,
CONSTRAINT pk PRIMARY KEY (trace_id, parent_id, span_id)
)
0: jdbc:phoenix:localhost:2181:/hbase> trace on;
trace_id
-7450651397611882467
1 row selected (0.001 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> create table t(a bigint not null PRIMARY key,b bigint);
No rows affected (2.523 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> !ta
------------------------------------------------------------------------------------------------------------+
TABLE_CAT   TABLE_SCHEM TABLE_NAME  TABLE_TYPE  REMARKS     TYPE_NAME   SELF_REFERENCING_COL_NAME     REF_GENERATIO
------------------------------------------------------------------------------------------------------------+
  SYSTEM      CATALOG     SYSTEM TABLE
  SYSTEM      FUNCTION    SYSTEM TABLE
  SYSTEM      SEQUENCE    SYSTEM TABLE
  SYSTEM      STATS SYSTEM TABLE
  SYSTEM      TRACING_STATS     SYSTEM TABLE
        T     TABLE

------------------------------------------------------------------------------------------------------------+
0: jdbc:phoenix:localhost:2181:/hbase> upsert into t(a,b) values(1,1);
1 row affected (0.046 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> select * from SYSTEM.TRACING_STATS where trace_id=-7450651397611882467;
TRACE_ID    PARENT_ID   SPAN_ID     DESCRIPTION START_TIME  END_TIME    HOSTNAME    COUNT COUNT

No rows selected (0.09 seconds)
0: jdbc:phoenix:localhost:2181:/hbase> select * from SYSTEM.TRACING_STATS;
TRACE_ID    PARENT_ID   SPAN_ID     DESCRIPTION START_TIME  END_TIME    HOSTNAME    COUNT COUNT

No rows selected (0.09 seconds)
@MrSandmanRUS @joshelser

authors: MrSandmanRUS mygood

repo: apache/phoenix


text:
Add table indexes API

This API is primarily for the controller UI -> tables ->  -> reload status -> STATUS ui element. Right now it uses the /segments/{tableName}/metadata?type={type}&columns=* API to display the column level index details. This has a few problems

This is an expensive API call especially for large tables. This leads to bad ux when using the reload status UI feature
The UI logic to build tableLevel column -> index mapping using this segmentName level metadata is simply wrong (uss the first segment's metadata to build the view).
Since Index-spi changes are merged, this API does not return metadata of dynamically added index types, but only some hardcoded ones.

This new API's reponse looks like this
{
    "totalSegments": 31,
    "columnToIndexesCount":
    {
        "FlightNum":
        {
            "dictionary": 31,
            "bloom": 0,
            "null": 0,
            "forward": 31,
            "fst": 0,
            "json": 0,
            "range": 0,
            "h3": 0,
            "text": 0,
            "inverted": 0,
            "some-dynamically-injected-index-type": 31,
        },
        "Origin":
        {
            "dictionary": 31,
            "bloom": 0,
            "null": 0,
            "forward": 31,
            "fst": 0,
            "json": 0,
            "range": 0,
            "h3": 0,
            "text": 0,
            "inverted": 0,
            "some-dynamically-injected-index-type": 0,
        }
        ...
}

I'll followup with @jayeshchoudhary to move the UI API call to this new one, and how best to display the UI element once this PR is merged.

API calls involved when viewing this modal. For a large enough table, this UI element takes ~40s to load

Codecov Report

Merging #11576 (a6b0bdb) into master (bc0d9fb) will decrease coverage by 48.58%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #11576       +/-   ##
=============================================
- Coverage     63.05%   14.47%   -48.58%
+ Complexity     1108      201      -907
=============================================
  Files          2325     2326        +1
  Lines        124797   124869       +72
  Branches      19056    19061        +5
=============================================
- Hits          78693    18079    -60614
- Misses        40508   105266    +64758
+ Partials       5596     1524     -4072

Flag
Coverage Δ

integration
?

integration1
?

integration2
?

java-11
14.46% <0.00%> (-35.58%)
:arrow_down:

java-17
14.47% <0.00%> (-48.45%)
:arrow_down:

java-20
14.47% <0.00%> (-48.43%)
:arrow_down:

temurin
14.47% <0.00%> (-48.58%)
:arrow_down:

unittests
14.47% <0.00%> (-48.58%)
:arrow_down:

unittests1
?

unittests2
14.47% <0.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...on/response/server/TableIndexMetadataResponse.java
0.00% <0.00%> (ø)

...oller/api/resources/PinotTableRestletResource.java
43.15% <0.00%> (-4.23%)
:arrow_down:

...che/pinot/server/api/resources/TablesResource.java
54.09% <0.00%> (-4.42%)
:arrow_down:

... and 1500 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

repo: apache/pinot


text:
Fix v1 query engine behavior for aggregations without group by where the limit is zero

Aggregation queries without GROUP BY clauses typically aren't used with a LIMIT clause since it semantically doesn't make sense.
However, it isn't considered invalid, and most databases (including Postgres and MySQL) return the same aggregation results (i.e., over the entire table) regardless of what the LIMIT is - except if the LIMIT is 0, in which case an empty result set is returned.
The v1 query engine in Pinot follows the former but not the latter - i.e., it returns the same aggregation results (i.e., over the entire table) regardless of what the LIMIT is (even if it is 0).
This patch standardizes the Pinot behavior in this case by introducing an EmptyAggregationOperator (somewhat similar to the EmptySelectionOperator):

> EXPLAIN PLAN FOR SELECT SUM(ArrDelay) FROM airlineStats LIMIT 0;

PLAN_START(numSegmentsForThisPlan:31)
BROKER_REDUCE(limit:0)
COMBINE_AGGREGATE
AGGREGATE_EMPTY

> SELECT SUM(ArrDelay) FROM airlineStats LIMIT 0;

{"resultTable":{"dataSchema":{"columnNames":["sum(ArrDelay)"],"columnDataTypes":["DOUBLE"]},"rows":[]},"numRowsResultSet":0,"partialResult":false,"exceptions":[],"numGroupsLimitReached":false,"timeUsedMs":3,"requestId":"11345490000000018","brokerId":"Broker_192.168.29.25_8000","numDocsScanned":0,"totalDocs":9746,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":31,"numSegmentsProcessed":31,"numSegmentsMatched":0,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"brokerReduceTimeMs":0,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"traceInfo":{}}

This patch fixes one of the cases described in https://github.com/apache/pinot/issues/13563.
Note that this isn't an issue in the v2 multi-stage query engine because the Calcite planner is smart enough to prune sections of query plans that are known to never produce any rows.

> SET useMultiStageEngine = true; EXPLAIN PLAN FOR SELECT SUM(ArrDelay) FROM airlineStats LIMIT 0;

Execution Plan
LogicalValues(tuples=[[]])

The existing test testGroupByAggregationWithLimitZero recently added in https://github.com/apache/pinot/pull/13555 is updated because the integration test actually doesn't do any comparison for aggregation group by queries without an order by clause (also in this case we just want to verify that 0 rows are returned and that the data schema is still returned)   - https://github.com/apache/pinot/blob/efa43007adc1dd7736580d882f33956e359b0678/pinot-integration-test-base/src/test/java/org/apache/pinot/integration/tests/ClusterIntegrationTestUtils.java#L812-L817

Codecov Report
Attention: Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (6184918).
Report is 731 commits behind head on master.

Files
Patch %
Lines

.../core/operator/query/EmptyAggregationOperator.java
0.00%
8 Missing :warning:

...erator/blocks/results/AggregationResultsBlock.java
0.00%
3 Missing :warning:

.../combine/merger/AggregationResultsBlockMerger.java
0.00%
2 Missing :warning:

...rg/apache/pinot/core/plan/AggregationPlanNode.java
0.00%
2 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (6184918). Click for more details.
HEAD has 48 uploads less than BASE

Flag
BASE (59551e4)
HEAD (6184918)

integration
7
2

integration2
3
2

temurin
12
2

java-21
7
2

skip-bytebuffers-true
3
1

skip-bytebuffers-false
7
1

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13564       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2484       +48
  Lines        133233   137259     +4026
  Branches      20636    21314      +678
=============================================
- Hits          82274        0    -82274
- Misses        44911   137259    +92348
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <0.00%> (-61.75%)
:arrow_down:

skip-bytebuffers-true
0.00% <0.00%> (-27.73%)
:arrow_down:

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@jackjlli  - Can you help review this please ?


text:
have query table list show search bar; fix InstancesTables filter

This is a ui bugfix

adds the searchbox to the query tables list. every since pagination was added here, it's become a lot more challenging to scroll through tables when you have > 10 tables
this fixes the InstancesTables page to only show the provided instances. right now it shows empty tables for all the other instances

Codecov Report

Merging #9742 (e44e411) into master (2f640ff) will decrease coverage by 44.74%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #9742       +/-   ##
=============================================
- Coverage     70.08%   25.33%   -44.75%
+ Complexity     4980       44     -4936
=============================================
  Files          1951     1939       -12
  Lines        104561   104208      -353
  Branches      15836    15798       -38
=============================================
- Hits          73279    26400    -46879
- Misses        26155    75092    +48937
+ Partials       5127     2716     -2411

Flag
Coverage Δ

integration1
25.33% <ø> (-0.11%)
:arrow_down:

integration2
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/NoOpRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/user/RoleType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/data/MetricFieldSpec.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/stream/StreamMessage.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/common/tier/TierFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/org/apache/pinot/spi/config/table/TableType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../org/apache/pinot/spi/data/DimensionFieldSpec.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1417 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@joshigaurava are you able to review this?


text:
Flaky-test: PulsarFunctionMetadataStoreTest.setup

example failure
Error:  Tests run: 21, Failures: 1, Errors: 0, Skipped: 11, Time elapsed: 142.439 s <<< FAILURE! - in org.apache.pulsar.functions.worker.PulsarFunctionMetadataStoreTest
  Error:  setup(org.apache.pulsar.functions.worker.PulsarFunctionMetadataStoreTest)  Time elapsed: 19.964 s  <<< FAILURE!
  org.awaitility.core.ConditionTimeoutException: Assertion condition defined as a org.apache.pulsar.functions.worker.FileServer null within 10 seconds.
  	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
  	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
  	at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)
  	at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
  	at org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:769)
  	at org.apache.pulsar.functions.worker.FileServer.waitUntilServerIsAvailable(FileServer.java:82)
  	at org.apache.pulsar.functions.worker.FileServer.start(FileServer.java:72)
  	at org.apache.pulsar.functions.worker.PulsarFunctionLocalRunTest.setup(PulsarFunctionLocalRunTest.java:305)
  	at jdk.internal.reflect.GeneratedMethodAccessor958.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
  	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61)
  	at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366)
  	at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320)
  	at org.testng.internal.TestInvoker.runConfigMethods(TestInvoker.java:701)
  	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:527)
  	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
  	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
  	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
  	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
  	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
  	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
  	at org.testng.TestRunner.privateRun(TestRunner.java:764)
  	at org.testng.TestRunner.run(TestRunner.java:585)
  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
  	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
  	at org.testng.TestNG.runSuites(TestNG.java:1069)
  	at org.testng.TestNG.run(TestNG.java:1037)
  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90)
  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
  	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
  Caused by: java.util.concurrent.TimeoutException
  	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
  	at org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:101)
  	at org.awaitility.core.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:81)
  	at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:103)
  	... 44 more

<!-- optionally provide the full stacktrace ->

Full exception stacktrace

full exception stacktrace here

+1 https://github.com/apache/pulsar/runs/7523859795?check_suite_focus=true

authors: codelipenghui shibd

repo: apache/pulsar


text:
[feature][doc] Add offload metrics for broker and some minor fixes for tiered storage

Doc updates for  #13833

Add new offload metrics for broker.
Preview:

In the Tiered Storage chapter, add a reference for offload metrics and fix minor issues for single-source purposes.
a) add a link to offload metrics.
Preview:

b) add a streamlined sentence and remove duplicate information in each offloader page.
Preview:

Fix style syntax issue for breaks in related doc pages.

Unify the expression for rates.

[x] doc

@tjiuming @hangc0276 @Anonymitaet can you please review this PR as soon as #13833 is merged? Thank you.

@momo-jun I've added DO NOT MERGE  to the PR title since we're waiting for the code PR to be merged first.

@momo-jun I've added DO NOT MERGE to the PR title since we're waiting for the code PR to be merged first.

Makes sense. Thank you for doing this for me :)

The https://github.com/apache/pulsar/pull/13833 has merged already.
Please update the doc.

NAME
TYPE
DESCRIPTION

brk_ledgeroffloader_offload_error
COUNTER
Number of failed operations to offload

brk_ledgeroffloader_offload_rate
GAUGE
Rate of offload(bytes/s)

brk_ledgeroffloader_read_offload_error
COUNTER
Number of failed operations to read offload files

brk_ledgeroffloader_read_offload_rate
GAUGE
Rate of read entries from offload files(bytes/s)

brk_ledgeroffloader_write_storage_error
COUNTER
Number of failed operations to write to storage

brk_ledgeroffloader_read_offload_index_latency
SUMMARY
Latency of read index from offload files

brk_ledgeroffloader_read_offload_data_latency
SUMMARY
Latency of read data from offload files

brk_ledgeroffloader_read_ledger_latency
SUMMARY
Latency of read entries from bookkeeper

brk_ledgeroffloader_delete_offload_ops
COUNTER
Number of both successful and failed operations to delete offload

@tjiuming @hangc0276 @Anonymitaet Can you pls review this doc PR? Thanks.

LGTM @momo-jun

LGTM @momo-jun

authors: Anonymitaet momo-jun tjiuming


text:
Fix PersistentAcknowledgmentsGroupingTracker set BitSet issue.

Motivation
When consumers set enableBatchIndexAcknowledgment=true, client will execute PersistentAcknowledgmentsGroupingTracker#doIndividualBatchAckAsync :
https://github.com/apache/pulsar/blob/8928c3496a61c588b50461d6adaab089dd421619/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java#L357-L372
There is an error in line 367, it should be
value.set(0, batchMessageId.getBatchSize());
But batchMessageId.getBatchSize() always return acker.getBatchSize():
https://github.com/apache/pulsar/blob/8928c3496a61c588b50461d6adaab089dd421619/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java#L137-L139
If line 362 is false, BatchMessageIdImpl only has acker with BatchMessageAckerDisabled which batch is always 0.
So I have added getOriginalBatchSize to return the user-specified batch size.
Then, when print logs in line 556, pendingIndividualBatchIndexAcks is always empty. Should replace with entriesToAck
Documentation

[x] no-need-doc

Does this fix apply for the maintenance branches (branch-2.8, branch-2.9)?

The patch applies cleanly to branch-2.8 and I don't see a reason why it shouldn't be cherry-picked to branch-2.8 since this is a bug fix.

So I have added getOriginalBatchSize to return the user-specified batch size.

@Technoboy- what does the "user-specified batch size" mean? I'd assume that each batch message has a batch size of what the actual batch size for that message is?

So I have added getOriginalBatchSize to return the user-specified batch size.

@Technoboy- what does the "user-specified batch size" mean? I'd assume that each batch message has a batch size of what the actual batch size for that message is?

Sorry for my bad english. I mean it's the batch size that from the constructor, which is the same meaning with you.

So I have added getOriginalBatchSize to return the user-specified batch size.

@Technoboy- what does the "user-specified batch size" mean? I'd assume that each batch message has a batch size of what the actual batch size for that message is?

@lhotari Actually the new getOriginalBatchSize() is confusing. The getBatchSize() should not retrieve the batch size from the BatchMessageAcker because the acker could be a BatchMessageAckerDisabled whose batch size field is 0. I removed this confusing method and fix the getBatchSize() method in https://github.com/apache/pulsar/pull/19031.

authors: BewareMyPower Technoboy- lhotari


text:
[improve][test] Reduce logging overhead in tests

Motivation

tests use a logging configuration where the location of the log statement is logged. this is very expensive and unnecessary.
this was noticed in test profiling (see #22250 for details of how profiling was performed)

Modifications

don't log the location where the log was added since this causes a high overhead

Additional context

The overhead is explain in https://logging.apache.org/log4j/2.x/manual/layouts.html#location-information

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Wow. I didn't know that!
Is this layout also used in the default log4j.xml we ship?

Is this layout also used in the default log4j.xml we ship?

@asafm no, it was only used in tests.

authors: asafm lhotari


text:
Fix documentation: how to start server for cpp tests

Modifications
Out-of-date instructions in pulsar-client-cpp/README.md about how to start test server

run java8 tests
run integration tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests

run cpp tests

run java8 tests

authors: Anonymitaet Jennifer88huang candlerb sijie wolfstudy


text:
[RIP-67] RocketMQ EventBridge Runtime Management

When subscription rule increase, a single runtime worker cannot support it, we need to add more worker to provide services at the same time.Here are several key issues:

How many workers are needed in the current system？
How to manage the life cycle of these workers？
How to assign tasks to workers in order to maximize performance?
How to achieve elastic scaling?
... ...

show the Chinese version detail please

问题背景
当订阅规则增加时，单个运行时worker无法支持，需要添加更多worker同时提供服务。这里有几个关键问题：

现有系统需要多少Worker？
如何管理这些worker的生命周期？
如何给Worker分配任务才能最大化性能？
如何实现弹性伸缩？
如何实现不同任务之间的隔离？
……

整体架构设计

引入 Runner Task
当在规则中创建一个Target订阅时，EventBridge会为每个任务创建一个Target Runner，用来描述当前这个Target订阅的运行态信息，详细信息可以参考表 event_target_runner:

但是目前的方式存在几个问题：

所有的Target Runner 默认都运行在一个Worker节点上；
无法实现不同Worker，运行不同的Target Runner；
无法支持同一个Target Runner，在多个Worker上同时运行；

为了解决这个问题，我们在DB中，以声明的方式，引入表：event_runner_task

其中cluster_id、worker_id 指名了当前runner_name运行的集群信息和Worker信息：
虚拟Cluster 管理
Cluster 是一个虚拟的概念，用来标注同一类Worker，这类Worker拥有相同的资源配置，镜像版本等。引入Cluster的目的主要包括：

资源隔离：不同的Cluster管理不同规模的Worker，同一个Target Runner只能分配到一个Cluster中的一个或多个Worker上；
方便Worker管理：Worker的生命周期管理，以Cluster为粒度，我们可以Cluster方便的升级该Cluster下所有的Worker；也可以通过Cluster来控制Worker的数量；

为此，我们需要在DB中，同样以声明的方式，引入一张表 event_cluster：

Worker管理
Worker和Runtime的区别是：Runtime代表软件的运行时，而Worker是Runtime运行时的资源节点。这个关系类似进程和系统之间的关系。理论上，一个Worker可以包含多个Runtime，但是为了简化模型，常常一个Worker只包含一个Runtime进程。
为了方便worker的管理，我们在DB中引入一张表event_worker：

如何创建Worker
方式1: 直接注册物理Node到event_worker表中

当有新的Node加进来时，只需要注册到event_worker表，配置IP地址；
前提条件：新增的Node和现有集群在同一个网络中；

方式2: 通过K8S创建资源节点

创建Depolyment, 默认replica为1：

Worker如何获取被分配的Runner Task
Runtime支持文件、API的方式加载Task配置，对应的我们有几种方式让Worker获取被分配的Runner Task：

方式1：Manager主动将Task 配置分发给Worker

比如通过K8S configMap，将Task配置以文件的形式挂载到Pod某个文件目录

方式2: 通过Manager与Worker的通信

如何升级Worker的镜像版本
方式1: 如果是Node直接注册的Worker，则由客户自己管理和维护Worker的镜像
方式2: 如果是通过K8S创建的Worker，则由系统负责worker的镜像升级

event_images

发布流程：

镜像发布时，在connect_images添加新发布的镜像版本，通过upper_limit限制本次发布更新的cluster数量；
conductor定时JOB，每隔1分钟，扫描connect_cluster表，判断当前cluster是否需要更新：
首先，判断当前cluster指定了expect_image_id，如果指定了，则优先更新到此镜像版本。
否则，镜像更新到最新版本，但是必须同时满足：

当前cluster对应的imageId不是最新的；
当前cluster的升级优先级高于其他cluster；
upper_limit为达到上限；

如果都满足，则更新当前cluster下的worker到最新镜像。

问题背景
当订阅规则增加时，单个运行时worker无法支持，需要添加更多worker同时提供服务。
下面重点回答弹性资源问题：

现有系统需要多少Worker？
如何实现弹性伸缩？
如何实现不同任务之间的隔离？
如何新增加Worker
如何实现高可用

技术架构

总体架构是master-worker，master包括资源消费者(包括调度器)，资源管理器；worker是任务管理器，管有资源，上报资源。
数据架构，资源是组件的核心数据，分两条线a线 现有资源，b线 待定资源
a)	4资源请求->5a 分配可用资源-> 6a 请求使用资源-> 7a 提供资源->8a 提交任务
b)	4资源请求->5b 分配待定资源-> 6b 请求新worker-> 7b 启动任务管理器->8b 注册/报告资源
a线是分配现有资源；b线请求新资源，新资源注册后成为现有资源，在a线分配
### 现有系统需要多少Worker？
作业管理器询问所有connector task数量，总和就是资源槽数据量，Worker数量=资源槽数据量/每个Worker资源槽数
### 如何实现弹性伸缩/如何新增加Worker
两个问题其实是一个问题，弹性资源通过增减Worker实现；参考flink的声明式资源管理，资源使用分为，资源申请，检查资源请求/检查资源声明，资源提供

资源申请
资源消费者负责，向资源管理器提出申请；
检查资源请求/检查资源声明
检查资源请求/检查资源声明是交汇点，检查资源请求，该分配的分配，该请求新的请求新的资源；检查资源声明
哪些资源可以释放，需要新资源请求新worker。
资源提供
资源管理器向k8s集群管理器申请新建Worker，Worker启动后，注册/报告自身的资源槽；

如何实现不同任务之间的隔离？
使用docker的容器
高可用&健壮性设计

master的高可用是k8s的复制机制提供，选主机制k8s提供，资源消费者和资源管理器参与选主，资源消费者/任务管理器接入资源管理器主节点监听，获取最新资源管理器地址；任务管理器接入资源消费者主节点监听，获取最新的资源消费者主节点
2 资源消费者/资源管理器/任务管理器直接都有心跳检测，超时重连，保证连接有效性

authors: 2011shenlin Jashinck leehom

repo: apache/rocketmq-eventbridge


text:
为什么cn帮助文档中的PNG图片是404了呢？？

https://raw.githubusercontent.com/apache/rocketmq/master/docs/cn/image/rocketmq_architecture_2.png
比如上面的URL网址，帮方是否可以处理一下，不然帮助文档中引用的PNG图片看不到啊，对学习和理解不利。

https://github.com/apache/rocketmq/tree/master/docs/cn/image
all png file status is 404 !~

自己下载源代码，里面images中的png文件，看到了，自行解决！

repo: apache/rocketmq


text:
The field clientIP in ClientConfig.class maybe static?

The field clientIP in ClientConfig.class maybe static?
The problem is that how much consumer service has, it is need to register same number of times.
And I alse use windows dev environment, this make me worried。NetworkInterface.getNetworkInterfaces method cost too much time, and it need cost coumser nums multiply method cost times, this make my application start too slow.
So, I think the filed The field clientIP in ClientConfig.class maybe static?Or it has other uses?But I only konw little of it.
Can someone tell me?
ClientConfig.class 中的 clientIP 字段或许应该是static的？
目前我使用windows作为本地的开发环境，NetworkInterface.getNetworkInterfaces方法每获取一次都需要花费不少的时间，而且每有一个consumer，就会在注册的时候调用一次这方法，这将花费consumer数量X单次时间 的巨长的时间，甚至在我启用懒加载的时候超出了服务启动完成的时间。
所以我想ClientConfig.class中的字段是否可能是static的？可能这个字段还有其他的用途？我对它了解太少了。
有人能告诉我吗？

public class ClientConfig {     public static final String SEND_MESSAGE_WITH_VIP_CHANNEL_PROPERTY = "com.rocketmq.sendMessageWithVIPChannel";     private String namesrvAddr = NameServerAddressUtils.getNameServerAddresses();     private String clientIP = RemotingUtil.getLocalAddress();     private String instanceName = System.getProperty("rocketmq.client.name", "DEFAULT");     private int clientCallbackExecutorThreads = Runtime.getRuntime().availableProcessors();     protected String namespace;     protected AccessChannel accessChannel = AccessChannel.LOCAL;

ClientIp is dynamically acquired at startup. Static variables are generally used to save constants determined during compilation. Static variables are not suitable.
Obtaining ip information locally does not go through the network, and generally does not take a long time. You can locate the specific time-consuming time.

Thanks for your help.
There is 100+ consumes in my one service.All register completed cost more then 40 sconds.If i disable Disable some network adapters like wlan, it will Halve the time spent，but i think it still too slow。
Also，clientIp in all consumes is same(I don't use clusters or anything like that)，so I think this field meybe static, only query once when class has been loaded.
Or meybe provide one way when in this environment that can set use NetworkInterface.getNetworkInterfaces method only once?
Thank you.

The clientIP will be obtained only when the clientInstance is started for the first time. Do the 100+ consumptions you mentioned use the same clientInstance?


text:
Correct constant name WROTE_POSITION_UPDATER in DefaultMappedFile

WROTE_POSITION_UPDATER should be changed to WRITE_POSITION_UPDATER ？

#5187

My Fault! Use the past tense here.


text:
[Bug] [CheckPoint] checkpoint ineffective

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

SeaTunnel Version
2.3.5
SeaTunnel Config
env {
  execution.parallelism = 2
  job.mode = "BATCH"
  checkpoint.interval = 2000
  read_limit.rows_per_second=1
  job.name = cell-Job
}

source {

    InfluxDB {
        result_table_name = "test"
        url = "http://localhost:8086"
        sql = "select number from test1"
        database = "mydb"
        username = "admin"
        password = "123456"
        upper_bound = 10
        lower_bound = 1
        partition_num = 10
        split_column = "number"
        query_timeout_sec = 60
        connect_timeout_ms = 150000
        chunk_size = 1
        chunk_read = true
        schema {
                fields {
                         number = INT
                }
        }
    }

}

sink {
  Console {
          source_table_name = "test"
        }

}

Running Command
./start-seatunnel-flink-15-connector-v2.sh --config ../jobs/job.txt

Error Exception
**BATCH    checkpoint ineffective**

Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

using streaming mode

using streaming mode
this source support batch mode only

repo: apache/seatunnel


text:
[Feature][Connector-v2] Support streaming read for paimon

Purpose of this pull request

Support streaming read for paimon. Link pr #6766
Does this PR introduce any user-facing change?

How was this patch tested?

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist
Add ci label in label-scope-conf
Add e2e testcase in seatunnel-e2e
Update connector plugin_config

[ ] Update the release-note.

cc @hawk9821 and @TyrantLucifer as well.

authors: Hisoka-X dailai


text:
Closed [type: feature] e2e-spring-cloud test e2e apollo

test e2e apollo.

Codecov Report

Merging #4827 (f333cba) into master (fa306f4) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4827      +/-   ##
============================================
+ Coverage     64.53%   64.55%   +0.02%
+ Complexity     8177     8173       -4
============================================
  Files          1147     1147
  Lines         33805    33805
  Branches       3016     3016
============================================
+ Hits          21816    21823       +7
+ Misses        10268    10259       -9
- Partials       1721     1723       +2

see 4 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: HaiqiQin codecov-commenter

repo: apache/shenyu


text:
Remove local span and exit span OP name register permanently

Today, there are endpoint register happens for entry, exit and local span, but we don’t need exit and local span OP names in 99% cases. We should remove it.
The only two scenarios is

Segment query including endpoint id for endpoint dashboard page, but that page only shows entry span as endpoint, so it is safe to keep the column empty
Endpoint dependency in ref header dependency analysis in the backend. Let’s ignore the scenario of end parent endpoint is local or entry. Treat it as none, and don’t generate it as endpoint relation in the OAP side.

@kezhenxu94 Would you want to do this? Or I should run this removal.

Local and exit register caused too much loads to the storage and waste a lot of endpoint ID for nothing. And entry span is the one could be used for metrics, endpoint dependency, and waste more time to finish register(because more ID required register).
So, from all perspectives, this mechanism should be removed. Actually, this should be removed earlier, but I forgot to propose this.

repo: apache/skywalking


text:
Kafka reporter block Main thread

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Bug

Which version of SkyWalking, OS and JRE?
skywalking-oap-server:8.1.0
with kafka reporter plugin kafka-reporter-plugin-8.1.0.jar

What happened?

dubbo service
kafka server is down
all dubbo-server-thread block, throws "dubbo thread pool is exhausted"
threads dump

"DubboServerHandler-xxxx-thread-197" Id=271 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68b95e85
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68b95e85
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:143)
        at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:218)
        at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:942)
        at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:865)
        at org.apache.skywalking.apm.dependencies.org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:752)
        at org.apache.skywalking.apm.agent.core.kafka.KafkaTraceSegmentServiceClient.afterFinished(KafkaTraceSegmentServiceClient.java:81)
        at org.apache.skywalking.apm.agent.core.context.TracingContext$ListenerManager.notifyFinish(TracingContext.java:489)
        at org.apache.skywalking.apm.agent.core.context.TracingContext.finish(TracingContext.java:453)
        at org.apache.skywalking.apm.agent.core.context.TracingContext.stopSpan(TracingContext.java:376)
        at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:180)
        at org.apache.skywalking.apm.agent.core.context.ContextManager.stopSpan(ContextManager.java:172)
        at org.apache.skywalking.apm.plugin.dubbo.DubboInterceptor.afterMethod(DubboInterceptor.java:101)
        at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:97)
        at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java)
        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69)

Requirement or improvement

KafkaTraceSegmentServiceClient  should be A IConsumer instead of A TracingContextListener

@dmsolr I think this is a high priority bug we need to locate.

Does Kafka client have the configuration to ignore data when the local queue is full?
Why we don't keep DataCarrier as the buffer mechanism?

authors: nileblack wu-sheng


text:
kafka-fetcher topic prefix is not generated

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-fetcher.md#kafka-fetcher

Bug

Which version of SkyWalking, OS, and JRE?
oap.images.tag: 8.5.0-es7
oap.images.storageType: elasticsearch7

helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}"
-f ./skywalking/values-my-es.yaml
--set oap.env.SW_CLUSTER=kubernetes
--set oap.env.SW_CLUSTER_K8S_NAMESPACE=sky
--set oap.env.SW_KAFKA_FETCHER=default
--set oap.env.SW_KAFKA_FETCHER_SERVERS=xxx:9092
--set oap.env.SW_KAFKA_FETCHER_PARTITIONS=4
--set oap.env.SW_KAFKA_FETCHER_IS_SHARDING=true
--set oap.env.SW_NAMESPACE=pre
--set oap.env.SW_CONFIGURATION=k8s-configmap

What happened?

After deployment according to the document description,
SW_NAMESPACE is set, topic is not prefixed,
but elasitcsearch index is prefixed by default
Example after topic generation:skywalking-segments
Example after index generation:pre_segment-20210524

Should have been fixed by https://github.com/apache/skywalking/pull/6952.

authors: queenns wu-sheng


text:
Allowing GitHub Issues and Projects

What changes were proposed in this pull request?
Based on the discussion and vote on the Spark Mailinglist, this PR enables the ability to use Github Issues and Projects for this repository.
https://lists.apache.org/thread/oyogozm8y642w0bc5h1cx2xst7fyz3ms

Merging to master.

repo: apache/spark-connect-go


text:
[SPARK-49380] Add cluster-with-template.yaml example

What changes were proposed in this pull request?
This PR aims to add cluster-with-template.yaml example.
Why are the changes needed?
Apache Spark K8s Operator starts to support template-based Spark Cluster creation via the following.

https://github.com/apache/spark-kubernetes-operator/pull/80

We had better provide a concrete example like the following use cases.

Custom annotations and labels.
Custom full spec like priorityClassName, securityContext, sidecar containers

Does this PR introduce any user-facing change?
No. This is an unreleased new feature.
How was this patch tested?
Manual review.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @jiangzho

Also, cc @viirya , too.

Thank you, @viirya . Merged to main.

repo: apache/spark-kubernetes-operator


text:
[SPARK-13905][SPARKR] Change implementation of as.data.frame() to avoid conflict with the ones in the R base package.

What changes were proposed in this pull request?
SparkR provides a method as.data.frame() to collect a SparkR DataFrame into a local data.frame. But it conflicts the methods with the same name in the R base package.
This PR improves the implementation of as.data.frame() in SparkR to avoid conflict with those in the R base package.
How was this patch tested?
dev/lint-r
SparkR unit tests

Test build #53190 has started for PR 11726 at commit 150c79a.

Test build #53190 has finished for PR 11726 at commit 150c79a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53190/
Test PASSed.

It seems that I made mistake. This patch is problematic. I will close this PR. Will submit a new PR if find the solution.

authors: AmplabJenkins SparkQA sun-rui

repo: apache/spark


text:
[TRIVIAL][DOCS][STREAMING][SQL] The return type mentioned in the Javadoc is incorrect for toJavaRDD, …

What changes were proposed in this pull request?
Change the return type mentioned in the JavaDoc for toJavaRDD / javaRDD to match the actual return type & be consistent with the scala rdd return type.
How was this patch tested?
Docs only change.

Test build #61412 has started for PR 13954 at commit f8533ca.

Test build #61412 has finished for PR 13954 at commit f8533ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61412/
Test PASSed.

merging this to master and 2.0

authors: AmplabJenkins SparkQA holdenk tdas


text:
[SPARK-17096][SQL][STREAMING] Improve exception string reported through the StreamingQueryListener

What changes were proposed in this pull request?
Currently, the stackTrace (as Array[StackTraceElements]) reported through StreamingQueryListener.onQueryTerminated is useless as it has the stack trace of where StreamingQueryException is defined, not the stack trace of underlying exception.  For example, if a streaming query fails because of a / by zero exception in a task, the QueryTerminated.stackTrace will have
org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:211)
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:124)

This is basically useless, as it is location where the StreamingQueryException was defined. What we want is
Here is the right way to reason about what should be posted as through StreamingQueryListener.onQueryTerminated

The actual exception could either be a SparkException, or an arbitrary exception.

SparkException reports the relevant executor stack trace of a failed task as a string in the the exception message. The Array[StackTraceElements] returned by SparkException.stackTrace() is mostly irrelevant.
For any arbitrary exception, the Array[StackTraceElements] returned by exception.stackTrace() may be relevant.

When there is an error in a streaming query, it's hard to reason whether the Array[StackTraceElements] is useful or not. In fact, it is not clear whether it is even useful to report the stack trace as this array of Java objects. It may be sufficient to report the strack trace as a string, along with the message. This is how Spark reported executor stra
Hence, this PR simplifies the API by removing the array stackTrace from QueryTerminated. Instead the exception returns a string containing the message and the stack trace of the actual underlying exception that failed the streaming query (i.e. not that of the StreamingQueryException). If anyone is interested in the actual stack trace as an array, can always access them through streamingQuery.exception which returns the exception object.

With this change, if a streaming query fails because of a / by zero exception in a task, the QueryTerminated.exception will be
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1, localhost): java.lang.ArithmeticException: / by zero
	at org.apache.spark.sql.streaming.StreamingQueryListenerSuite$$anonfun$5$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply$mcII$sp(StreamingQueryListenerSuite.scala:153)
	at org.apache.spark.sql.streaming.StreamingQueryListenerSuite$$anonfun$5$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply(StreamingQueryListenerSuite.scala:153)
	at org.apache.spark.sql.streaming.StreamingQueryListenerSuite$$anonfun$5$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply(StreamingQueryListenerSuite.scala:153)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:370)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:232)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:226)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
	at org.apache.spark.scheduler.Task.run(Task.scala:86)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1429)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1417)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1416)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1416)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
...

It contains the relevant executor stack trace. In a case non-SparkException, if the streaming source MemoryStream throws an exception, exception message will have the relevant stack trace.
java.lang.RuntimeException: this is the exception message
	at org.apache.spark.sql.execution.streaming.MemoryStream.getBatch(memory.scala:103)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$5.apply(StreamExecution.scala:316)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$5.apply(StreamExecution.scala:313)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at org.apache.spark.sql.execution.streaming.StreamProgress.foreach(StreamProgress.scala:25)
	at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
	at org.apache.spark.sql.execution.streaming.StreamProgress.flatMap(StreamProgress.scala:25)
	at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatch(StreamExecution.scala:313)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches$1.apply$mcZ$sp(StreamExecution.scala:197)
	at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:43)
	at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:187)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:124)

Note that this change in the public QueryTerminated class is okay as the APIs are still experimental.
How was this patch tested?
Unit tests that test whether the right information is present in the exception message reported through QueryTerminated object.

Test build #63885 has started for PR 14675 at commit 60eabcc.

@marmbrus @zsxwing Could you please take a look.

Test build #63885 has finished for PR 14675 at commit 60eabcc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63885/
Test PASSed.

LGTM

Merging this master and 2.0

authors: AmplabJenkins SparkQA tdas zsxwing


text:
[SPARK-17389][Follow-up][ML] Change KMeans k-means|| default init steps from 5 to 2.

What changes were proposed in this pull request?
#14956 reduced default k-means|| init steps to 2 from 5 only for spark.mllib package, we should also do same change for spark.ml and PySpark.
How was this patch tested?
Existing tests.

cc @srowen

Test build #65223 has started for PR 15050 at commit bd71eb6.

Oh, whoops, completely missed that there are also defaults specified in the wrapper classes. Thank you

Test build #65223 has finished for PR 15050 at commit bd71eb6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65223/
Test PASSed.

Merged build finished. Test PASSed.

Also merged to master

authors: AmplabJenkins SparkQA srowen yanboliang


text:
[SPARK-19306][Core] Fix inconsistent state in DiskBlockObject when expection occurred

What changes were proposed in this pull request?
In DiskBlockObjectWriter, when some errors happened during writing, it will call revertPartialWritesAndClose, if this method again failed due to some issues like out of disk, it will throw exception without resetting the state of this writer, also skipping the revert. So here propose to fix this issue to offer user a chance to recover from such issue.
How was this patch tested?
Existing test.

Test build #71712 has started for PR 16657 at commit f9819d5.

CC @mridulm please help to review, thanks a lot.

Test build #71712 has finished for PR 16657 at commit f9819d5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71712/
Test FAILed.

Jenkins, retest this please.

Test build #71716 has started for PR 16657 at commit f9819d5.

Test build #71716 has finished for PR 16657 at commit f9819d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71716/
Test PASSed.

Merged build finished. Test PASSed.

Test build #71800 has started for PR 16657 at commit b0fe795.

LGTM, will wait for @vanzin's comments too.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71800/
Test FAILed.

Merged build finished. Test FAILed.

Jenkins, retest this please.

Test build #71801 has started for PR 16657 at commit b0fe795.

Test build #71801 has finished for PR 16657 at commit b0fe795.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71801/
Test PASSed.

LGTM. Merging to master / 2.1.

authors: AmplabJenkins SparkQA jerryshao mridulm vanzin


text:
Power Iteration Clustering in SparkML throws exception, when the ID in IntType

While running the following code, PIC throws exception.
val data = spark.createDataFrame(Seq(
      (0, Array(1), Array(0.9)),
      (1, Array(2), Array(0.9)),
      (2, Array(3), Array(0.9)),
      (3, Array(4), Array(0.1)),
      (4, Array(5), Array(0.9))
    )).toDF("id", "neighbors", "similarities")

val result = new PowerIterationClustering()
      .setK(2)
      .setMaxIter(10)
      .setInitMode("random")
      .transform(data)
      .select("id", "prediction")

Result
org.apache.spark.sql.AnalysisException: cannot resolve 'prediction`' given input columns: [id, neighbors, similarities];;
'Project [id#215, 'prediction]
+- AnalysisBarrier
+- Project [id#215, neighbors#216, similarities#217]
+- Join Inner, (id#215 = id#234)
:- Project [_1#209 AS id#215, _2#210 AS neighbors#216, _3#211 AS similarities#217]
:  +- LocalRelation [_1#209, _2#210, _3#211]
+- Project [cast(id#230L as int) AS id#234]
+- LogicalRDD [id#230L, prediction#231], false
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:88)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:85)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)

`
What changes were proposed in this pull request?

PIC needs to return only "id" and "predictions". Currently it returns the entire data, including neighborhood array and similarity array.
MLLib PIC returns "id" as Long, and "predictions" as Int. So, In ML, we don't need to typecast to the user input ID type. We can directly display the output of MLLib PIC.

How was this patch tested?
Added a UT

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Thanks for the patch!  I just commented on https://issues.apache.org/jira/browse/SPARK-24213 though and would like to replace this with https://github.com/apache/spark/pull/21274
Could you please close this issue and help with reviewing the other PR?  Thanks!

Thank you @jkbradly. Actually one more issue is there. Currently we are skipping some of the nodes which are not there in the ID column, but there in the neighboring column. Spark MLLib is diplaying cluster indices corresponding to all the nodes.
So, Is it necessary for the join operation?Shall I open a new PR, adressing the issue? Kindly reply

@shahidki31 Seemingly what you said above is anothor issue ? You can create another jira for that. :)

@WeichenXu123 Thanks for the comment. I have created another Jira and I have raised a PR for that. That PR will fix this issue as well. Can you please review the PR?
Jira : https://issues.apache.org/jira/browse/SPARK-24217
PR: https://github.com/apache/spark/pull/21277