text:
KAFKA-17766: Fixing deadlock in TopicBasedRemoteLogMetadataManager

KAFKA-17766: Issue Details: Inside TopicBasedRemoteLogMetadataManager::close, one thread(t1) is calling join on initializationThread thread after taking writeLock on "lock" object => t1 will wait for initializationThread to complete. Internally initializationThread is also using writeLock on "lock" object. This can cause deadlock in below situation

initializationThread is started
close has been invoked as part of a separate thread. But this thread is not yet scheduled by OS.
At line 430, initializationThread is preempted and OS has started running close thread. close takes writeLock and invoked join on initializationThread.
Now OS schedules initializationThread again and at line 433 this thread also tries to take writeLock. But since writeLock is already held by close thread => both are waiting on each other to complete. initializationThread will wait on close to release the writeLock, while close thread will wait for completion of initializationThread

Fix Details: We can avoid taking lock inside close() method as there no operations with any side effects. closing instance variable is of type AtomicBoolean => no race condition when updating it to true. .
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@kamalcph Even though I am working on adding UT for initialization failure, but JFYI, the instance of TopicBasedRemoteLogMetadataManager we are getting from RemoteLogMetadataManagerTestUtils is already initialized one (i.e. initialized = true and initializationFailed = false). Should I still go ahead and add a UT ?

I found this document : https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage?uclick_id=044a9cd6-6406-43df-ab2e-a89d84e404c0

@kamalcph Please check my previous comment. Do you still want me to add the UT ?
My approach will be to : Avoid below check in RemoteLogMetadataManagerTestUtils and inducing initialization failure . Then inside TopicBasedRemoteLogMetadataManagerTest we can get that TopicBasedRemoteLogMetadataManager instance and check for initialization failure
assertDoesNotThrow(() -> TestUtils.waitForCondition(topicBasedRemoteLogMetadataManager::isInitialized, 60_000L,
"Time out reached before it is initialized successfully"));
Ideally, RemoteLogMetadataManagerTestUtils should have been written in such as way to allow consumers to test for pre initialization cases as well.

I've filed KAFKA-17830 to track adding the unit tests separately for init failure cases and unblock this patch as it fixes the deadlock in the CI run.

Ideally, RemoteLogMetadataManagerTestUtils should have been written in such as way to allow consumers to test for pre initialization cases as well.

Instead of refactoring the existing method in RemoteLogMetadataManagerTestUtils class, we can directly update it in the test: (eg)
@ClusterTest
public void testInitializationFailure() throws IOException, InterruptedException {
    try (TopicBasedRemoteLogMetadataManager rlmm = new TopicBasedRemoteLogMetadataManager()) {
        // configure rlmm such that it throws exception during initialization
        Map<String, String> configs = new HashMap<>();
        rlmm.configure(configs);
        TestUtils.waitForCondition(rlmm::isInitializationFailed, "Initialization should fail");
    }
}


text:
KAFKA-17641: Update Vote RPC with PreVote field

KIP-996 Part 1: https://issues.apache.org/jira/browse/KAFKA-17641
Introduces v2 of vote RPC and includes handling of PreVote requests. Includes KafkaRaftClientPreVoteTest.
Some behaviors to note:

Many references to candidate are changed to more generic replica. In the next PreVote PR we will be adding Prospective state and it will not be guaranteed that candidates are sending vote requests.
Replicas receiving PreVote requests (vote request with prevote=true) with an epoch equal to their own will not transition to Unattached state. They will only grant the vote if they do not know of the leader (in unattached state) and the request's last epoch and offset are up-to-date with theirs.
If a replica receives a PreVote request with an epoch greater than their current epoch, they will transition to Unattached state (setting their epoch to one from the prevote request) and then grant the vote if the above conditions are met.
Vote requests with prevote=false are handled the same way as before.
Not included in this PR are changes to Follower state's canGrantVote. ~This may need to change in later PreVote PR to address a ping-pong scenario brought up by Jack Vanlightly. (ex: 3 node quorum, leader node A disconnects from quorum, node B goes into prospective state first before node C, node B sends prevote request to node C still in follower state and receives back that node A is leader, node B transitions to follower while node C transitions to prospective after election timeout, repeating this cycle) One way to tackle this is by having follower state nodes grant prevote requests based off last fetch time, which would involve changing canGrantVote, but this makes more sense to tackle after the Prospective state exists.~ It is possible to address the ping-pong scenario by having Prospective state send fetch requests and not transition to Follower state unless they receive a BeginQuorum request or other request which shows a leader with a larger epoch. This is explained under https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=272927537#KIP996:PreVote-QuorumStatechanges

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

cc @jsancio for review!

This depends on https://github.com/apache/kafka/pull/17352

addressing these failures now
Found 6 test failures:
FAILED ❌ UnattachedStateWithVoteTest > "testCanGrantVoteWithoutDirectoryId(boolean).isLogUpToDate=false"
FAILED ❌ ResignedStateTest > "testGrantVote(boolean).isLogUpToDate=true"
FAILED ❌ RaftUtilTest > "testSingletonVoteResponseForAllVersion(short, String).version=2, expectedJson={"errorCode":0,"topics":[{"topicName":"topic","partitions":[{"partitionIndex":0,"errorCode":0,"leaderId":1,"leaderEpoch":1,"voteGranted":true,"preVote":true}]}],"nodeEndpoints":[{"nodeId":1,"host":"localhost","port":9990}]}"
FAILED ❌ CandidateStateTest > "testGrantVote(boolean, boolean).isLogUpToDate=true, withDirectoryId=true"
FAILED ❌ CandidateStateTest > "testGrantVote(boolean, boolean).isLogUpToDate=true, withDirectoryId=false"
FAILED ❌ AbstractCoordinatorTest > testWakeupAfterSyncGroupReceived()

source: gharchive/pull-request

repo: apache/kafka

license_type: permissive

license_source: github-api


text:
MINOR: Update TestLogCleaning tool to use Java consumer

Add a system test to verify the log compaction
Enable log compaction in System tests
Remove configs with values same as server defaults from "kafka.properties" template file.
Update the kafka.py logic to handle the duplicates between kafka.properties and server_prop_overides.

@ijuma  Pls take a look. I have  added a system test to verify compaction using this tool.

retest this please

@cmccabe  Thanks for the review.  Addressed the review comments.
Also I have updated the logic in kafka.py to cleanup the duplicates between kafka.properties and server_prop_overides. Updated the logic to load the kafka.properties first and then override any server_prop_overides configs.
Pls take a look.

@ijuma @cmccabe  rebased the PR.  Verified most of the system tests on my local machine.
Is it possible to start system test run for this branch?

Thanks, @omkreddy .  Posted some comments

@cmccabe Thanks for the detailed review. Updated the PR. Ran few systems tests to verify the changes. Please take a look.

retest this please

Thanks @omkreddy, it looks really good.  I ran the test locally using ducker-ak and it passed.
One last thing.  There still seems to be a bit of name inconsistency.  The Java class is called kafka.tools.TestLogCleaning, but the python class is named LogCompactionTool.  A log message refers to TestLogCleaning tool.
The Java class isn't really a "tool" since tools are something we expect users to run outside of a testing context.  It's also not really a "test" in the sense of being something you directly run.
Maybe it makes sense to name the Java class LogCompactionTester.  Then the Python service class would be LogCompactionTester.  The Python test that uses the service class could be LogCompactionTest.  What do you think?

@cmccabe makes sense. Updated the class names and ran the system test. Pls take a look.

LGTM.  Thanks, @omkreddy

@ijuma Can you please take a look?

@ijuma  Thanks for the review.  Updated the review . Pls take one more look.

retest this please

authors: cmccabe omkreddy


text:
MINOR: rename InternalProcessorContext.initialized

We named this method initialized because it marks the context as having been initialized.
The downside is that it sounds more like a getter upon casual reading, which is weird since it's void.
Note also that the inverse method is already called uninitialize.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hmm. The java 10 build failed for :core:compileTestScala, claiming there were 4 errors, but I can't find any in the output. I also can't repro the failure on my machine in java 10 or java 8.
Retest this, please.


text:
MINOR: Safe string conversion to avoid NPEs

Should be ported back to 2.0

@ewencp @rhauch @C0urante I think this fix will clear up some NPEs observed in some connectors.


text:
MINOR: Refactor unit tests around RocksDBConfigSetter

Extract the mock RocksDBConfigSetter into a separate class.
De-dup unit tests covering RocksDBConfigSetter.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

I created this PR mainly to illustrate that one can pass in shared object or configurable values to the setter as well.

test this.

Thanks @abbccdda for the reviews!


text:
Transferring resources out of the Incubator after graduation

The vote for Kvrocks to graduate as a Top Level Project has been passed[1] and also established in the board meeting(2020-06-21). So we would like to transfer resources out of the incubator according to Transferring Resources out of the Incubator and this issue is used to track the checklist:
Update the Incubator status records

[x] Update the podling status page. All sections should now be filled in including EXIT. (Owner: @git-hulk)

Source

[x] Git repositories will be renamed to drop the incubator- prefix, please ensure that developers change their remotes. After you’ve created the TLP request, request the rename.(Owner: @git-hulk) - https://issues.apache.org/jira/browse/INFRA-24718
[x] SVN repositories will be moved from the incubator to other locations, if you need the move done please raise an infra ticket after the TLP Parent Request.(Owner: @git-hulk) - https://issues.apache.org/jira/browse/INFRA-24718
[x] Post an announcement to the development list telling everyone that the repository is about to be moved

Websites

[x] If you have any fully qualified links to your podling.incubator.apache.org on your website change them

Distribution mirrors

[x] Dist area (dist.apache.org) release/${project} and dev/${project} folders can be created by PMC members.

Do not forget to copy KEYS file from release/incubator/${project} location.
You have two major options then:

you can keep incubating released artifacts at release/incubator/${project} and remove artifacts once your TLP project does their first release.
you can move the last incubating released artifacts to your release/${project}, taking care that incubating stay in path (you will need to change website and mail for the new location)

Final Revision of Podling Incubation Records

[x] Update the svn incubator/trunk/content/projects/${project}.xml file to show the project’s status. (Owner: @git-hulk)
[x] Update the podling summary metadata file, incubator/trunk/content/podlings.xml svn file. (Owner: @git-hulk)
[x] After your project has finished reporting to the Incubator, then remove the "reporting" element from that podlings.xml file.(Owner: @git-hulk)
[x] Ensure that other svn resources for your project have moved to your new home.
[x] Review this whole graduation guide.

[1] https://lists.apache.org/thread/hbl8fsy5pw0k2k6prx62r1v1r7n9psjr

@git-hulk could you link to the board resolution about the graduation so that it's transparent a resolution?

@git-hulk could you link to the board resolution about the graduation so that it's transparent a resolution?

@tisonkun I only received the new role change notification: https://lists.apache.org/thread/dyknwbfb2d4rtk47nfd1h6cs1dq8bpm6 for now, it contains the transfer resources guide.

Resources are transferred out of the incubator according to the guide. Will close this issue, can help to review and correct while you’re free. @PragmaTwice @tisonkun @torwig

source: gharchive/issue

authors: git-hulk tisonkun

repo: apache/kvrocks


text:
[KYLIN-5288] set sampling button 'off' by default

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Branch to commit

[ ] Branch kylin3 for v2.x to v3.x
[ ] Branch kylin4 for v4.x
[x] Branch kylin5 for v5.x

Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have created an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[x] Compiling and unit tests pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin.apache.org or dev@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

LGTM

authors: Mukvin Syleechan

repo: apache/kylin

license_source: bigquery


text:
KYLIN-5304 Query cache enable should check both 'KylinConfig' and 'BackdoorToggles'

currently search cache only check query cache enable with 'KylinConfig'
Branch to commit

[ ] Branch kylin3 for v2.x to v3.x
[ ] Branch kylin4 for v4.x
[x] Branch kylin5 for v5.x

Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have created an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID
[ ] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin.apache.org or dev@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@hit-lacus

LGTM thx

authors: hit-lacus zznlime


text:
Tilde ('~') character is incorrectly URL-encoded with Python 2

I'm referring to the older, unresolved issue here (it seems I cannot create a new issue nor comment on the older one over at issues.apache.org though I have a long-time user there...):
https://issues.apache.org/jira/browse/LIBCLOUD-979
I can confirm that the reported issue happens also for Amazon S3, not only for CEPH as originally reported. And the problem is indeed that libcloud URL-encodes the tilde '~' sign to '%7E', which it shouldn't according to RFC 3986.
The problem seems specific to Python 2 because in Python 3 urllib.quote() has been already updated to not URL-encode the tilde '~' sign anymore:
https://bugs.python.org/issue16285
Because of this, I think the correct fix (other than the patch proposed in LIBCLOUD-979) would be adding the tilde '~' character to the safe characters in the urlquote() utility function in utils/py3.py to ensure consistent behavior between Python 2 and Python 3.
--- libcloud/utils/py3.py       2020-04-30 12:58:55.804591828 +0000
+++ libcloud_fixed/utils/py3.py 2020-04-30 12:59:25.975368813 +0000
@@ -211,7 +211,7 @@

     tostring = ET.tostring

-    def urlquote(s, safe='/'):
+    def urlquote(s, safe='/~'):
         if isinstance(s, _real_unicode):
             # Pretend to be py3 by encoding the URI automatically.
             s = s.encode('utf8')

Thanks for reporting this and thanks for contribution.
The change looks reasonable for me, but I would feel safer if there were some tests.
In addition to that, I'm a bit worried that some providers are relying on clients incorrectly encoding ~...

I just tested urlqoute from Python 3.6 and it seems to handle it in the same manner as Python 2 - aka it encodes it as %7E.
Python 3.6.9 (default, Sep  9 2019, 13:49:23)
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from urllib.parse import quote as urlquote
>>> urlquote('~')
'%7E'

Having said that, to be on the safe side - I think we should only add that change to S3 driver to begin with.
Who knows how many (if any) provider APIs rely on ~ being url encoded as %7E so changing it could introduce regressions in many other drivers.

@mmilitzer #1457 should fix it.
Can you please confirm (I tested it myself, but more eyes and testing never hurt)?

@Kami: I can confirm that your patch fixes the issue for S3.
My concern would be just that there'll still be inconsistent handling of '~' character now depending on Python version used though I understand your concerns. I'd just not rely on potential issues with other backends and the changed '~' tilde handling in Python 3.7+ being quickly reported. After all, the very issue here has apparently been present for years and gone unnoticed. So ideally there should be a test for "tilde in the name" for all backends to determine what the proper handling is that the other provider APIs expect...

@mmilitzer Thanks for confirming.
For now, I'll merge #1457 and then in the future look into making it consistent across Python versions, but that will likely be included in major version bump (just in case).

authors: Kami mmilitzer

repo: apache/libcloud


text:
Fix impossible location condition

Changes Title: Fix impossible location condition
Description
When requesting a list of publishers from other parts of the code (such as when listing images), a condition can be encountered where the location has been passed in (and therefore is not None), but  self.default_location is also set. The location that is passed in should be used, but the condition doesn't allow it to be.
Status

done, ready for review

Checklist (tick everything that applies)

[ ] Code linting (required, can be done after the PR checks)
[ ] Documentation
[ ] Tests
[ ] ICLA (required for bigger changes)

makes sense. thanks @techhat and thanks for making your first contribution to the project!

👍  merging

authors: techhat tonybaloney


text:
Disable JMX bean creation by default

https://github.com/apache/logging-log4j2/issues/2462#issuecomment-2045821938
Checklist

Base your changes on 2.x branch if you are targeting Log4j 2; use main otherwise
./mvnw verify succeeds (if it fails due to code formatting issues reported by Spotless, simply run ./mvnw spotless:apply and retry)
Non-trivial changes contain an entry file in the src/changelog/.2.x.x directory
Tests for the changes are provided
Commits are signed (optional, but highly recommended)

There are currently no tests for the JMX support.
org.apache.logging.log4j.core.jmx.Server#isJmxDisabled is private and PropertiesUtil.getProperties() lazily caches its values in a static field, i.e. on the JVM level.
So there are several options:

keep the status quo, i.e. no tests
make org.apache.logging.log4j.core.jmx.Server#isJmxDisabled package-private and have test along the lines of:

@Tag("needsforkedvm")
class JmxEnablementTest {

  @ReadsSystemProperty
  @Test
  void isJmxDisabledDefault() {
   assertTrue(Server.isJmxDisabled());
  }

  @SetSystemProperty(key = "log4j2.disableJmx", value = "false")
  @Test
  void isJmxDisabledFalse() {
    assertFalse(Server.isJmxDisabled());
  }

  @SetSystemProperty(key = "log4j2.disableJmx", value = "true")
  @Test
  void isJmxDisabledTrue() {
    assertTrue(Server.isJmxDisabled());
  }
}

call org.apache.logging.log4j.core.jmx.Server#reregisterMBeansAfterReconfigure and and have test along the lines of:

@Tag("needsforkedvm")
class JmxEnablementTest {

  @ReadsSystemProperty
  @Test
  void reregisterMBeansAfterReconfigureJmxSupportDefault() {
      Server.reregisterMBeansAfterReconfigure();
      final MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
      assertThat(mbs.getDomains()).doesNotContain("org.apache.logging.log4j2");
  }

  @SetSystemProperty(key = "log4j2.disableJmx", value = "false")
  @Test
  void reregisterMBeansAfterReconfigureJmxSupportEnabled() {
      Server.reregisterMBeansAfterReconfigure();
      final MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
      assertThat(mbs.getDomains()).contains("org.apache.logging.log4j2");
  }

  @SetSystemProperty(key = "log4j2.disableJmx", value = "true")
  @Test
  void reregisterMBeansAfterReconfigureJmxSupportDisabled() {
      Server.reregisterMBeansAfterReconfigure();
      final MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
      assertThat(mbs.getDomains()).doesNotContain("org.apache.logging.log4j2");
  }

Both 2. and 3. need a forked VM setup:
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
      <properties>
        <includeTags>needsforkedvm</includeTags>
      </properties>
      <forkCount>1</forkCount>
      <reuseForks>false</reuseForks>
    </configuration>
  </plugin>

https://junit.org/junit5/docs/current/user-guide/#writing-tests-tagging-and-filtering
https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html#forked-test-execution

The double negative log4j2.disableJmx=false is unfortunate, but we would have to deprecate an existing system property and introduce a new one log4j2.enableJmx otherwise.

There are currently no tests for the JMX support.
...

keep the status quo, i.e. no tests

I'd be in favor of this approach. I doubt if this feature has any significant usage at all. No need to complicate the code base and increase the build time (which is already excessive!) with tests of a hardly used feature.

@sdavids,
Thank you for your contribution.

authors: ppkarwasz sdavids vy


text:
Parallel processing

Description
Description
Please note- This is not a new PR- Original PR (apache/lucene-solr#2345) was raised on old apache/lucene-solr github repository. This is just a copy in new repo.
Lucene Benchmark Scaling Problem with Reuters Corpus
While Indexing 1 million documents with reuters21578 (plain text Document derived from reuters21578 corpus), we observed that with higher number of Index threads, the Index throughput does not scale and degrades. Existing implementation with synchronization block allows only one thread to pick up a document/file from list, at any given time – this code is part of getNextDocData() in ReutersContentSource.java. With multiple index threads, this becomes a thread contention bottleneck and does not allow the system CPU resource to be used efficiently.
Solution
We developed a strategy to distribute total number of files across multiple number of Indexing threads, so that these threads work independently and parallelly.
Tests
We mainly modified existing getNextDocData(), which is not altering functionality, hence not added any new test cases.
Passed existing tests
Checklist
Please review the following and check all that apply:

[X] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[ ] I have created a Jira issue and added the issue ID to my pull request title.
[X] I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
[X] I have developed this patch against the main branch.
[X] I have run ./gradlew check.
[ ] I have added tests for my changes.

Thanks for the updates!
It looks like gradle check is upset -- if you run ./gradlew tidy it will re-format your changes and it should pass again!

@mikemccand
Mike, We've committed the code 4 days before and it still  in Checking phase. Ideally 'musedev' should not take longer time. Looks like, there is some problem is the checking process. It would be really helpful if you could advise us for the next steps.

@mikemccand
Mike, I was wondering, are there any suggestions to incorporate in this PR. Also, need your help to resolve this ** pending checks** issues, it has been more than 15 days and "musedev" still showing in pending state.

it has been more than 15 days and "musedev" still showing in pending state.

Egads!  I don't know why it's stuck in Pending.  We can skip it -- I'll confirm gradlew check is happy.

Thanks @balmukundblr -- I merged this with Lucene's main branch and also backported to 8.x (for eventual future 8.10.0 release).

@mikemccand
Thank you very much for your great suggestions which really helped us to improve the code at greater level.

authors: balmukundblr mikemccand

repo: apache/lucene


text:
LUCENE-10551: switch to PUAFIF

Just a fast follow from https://github.com/apache/lucene/pull/858, switching from if to PUAFIF.

I came here to find out what a PUAFIF is! TBH I'm not really sure why this is better than the ISE it replaces?

Oh sorry -- see the discussion on the previous PR.

I see, got it thanks

I actually see this exception, trying to figure out the situation
java.lang.IllegalStateException: 7 <> 9 execservicetest.oc1.iad.amaaaaaait4ivdqawqdvv26mn2naeqgm22k5hnei3von6shgquawho746smaexecservicetest.oc1.iad.amaaaaaam5mjnqia6tcffpj2iwrpehlm2usduwdqctplnussr3bjuivi774qexecservicetest.oc1.iad.amaaaaaasolc3naa7gyscffke4m4oeyieul7tfoftatmjvn72tnzsu4t3jiqexecservicetest.oc1.iad.amaaaaaasolc3naafn2zpux3fp32xmb4cfvd7gljqlgiy4rgrrustq6tndmqexecservicetest.oc1.iad.amaaaaaasolc3naaj3am4rxciadcpl5ki3522brua2lc6red5ofjxtgiefjaexecservicetest.oc1.iad.amaaaaaazxun5saaxupdsqip3olsif6xejfeeu5w7y2wousv6gcwa37jqcqqfnfunc.oc1.iad.aaaaaaaaiu5kyxn3rsuxrnr2ryywb3fsjjdcwbuncmxucyahh2xt6rt6yykafnfunc.oc1.iad.aaaaaaaaxzddddwf2i5mb6jwgfwfrz5ktz5ogjagav77vydzrasbxmm7f22afnfunc.oc1.phx.aaaaaaaabwdvk36adgujikfpjaetd3ho6stvkn7qjgsft7m25tp7sb2shs6qfnfunc.oc1.phx.aaaaaaaame6xt7522xeftk2gyxuzaoouekrv5o4k244zzvclgbu7zov7ewrqfnfunc.oc1.phx.aaaaaaaarmxlylskbhzu4atnhbvlngkqdtamovhlxvfgwug57opx6ck5xv3ainstance.oc1.iad.anuwcljrwmvdzuyc2wd26svc6iypp5uydwlxbislw3foazz4jkemhunelkhqinstance.oc1.iad.anuwcljrwmvdzuyc3azq6nqj5lddnhoxqbvbvujkkkotnjltmq3kror25qjainstance.oc1.iad.anuwcljrwmvdzuyc47ucrewbikbzoq533thznkxatppjw3fbtrpmcevmsstqinstance.oc1.iad.anuwcljrwmvdzuychw3bygq673uvbedgj74xvi5mveittolswygjy2dy6cjqinstance.oc1.iad.anuwcljryl446jac3jysbgv5vysahbqashx3mghnuruug2ibxcbhctt3iiaainstance.oc1.iad.anuwcljt2olcr6acxxksd2juun4uydm7jxcn3bsvl2aewd7ol7wvs77f27hqsaml2idp.oc1..aaaaaaaa44vdivaglnbd6efepi43vxawi5s55etehzpjrjlg2pfedzlkixoa/admibreitsaml2idp.oc1..aaaaaaaa4nub35egudgxeimz7ga3thf55fnmwbgwwnhzrtt552kmkfwbzzea/adrian.magno@cencosud.com.arsaml2idp.oc1..aaaaaaaa4ymeemwgshzvpnhtvril3rsddroagoji5f5lyvttoeffo44gw7va/nteske@federalsignal.comsaml2idp.oc1..aaaaaaaanvn3rcyq5eb2yqbbfhtnbc4tv6gorjbwsvrhmvnqpnyfw53ppozq/mack.yao@optum.comsaml2idp.oc1..aaaaaaaarmpb5k7zrpeveosltjlr6j5ahtlxczac7i33gbt5um7mahybxj6a/richardbqualls@gmail.comuser.oc1..aaaaaaaa4k365c3ow7eqboxcdez443unelg6txqlw54ruzvqfel7rj24smfquser.oc1..aaaaaaaa76uagtnylt5ovrzupfdpnxqlu3n3edbdik5jl7tljwwxskka25jauser.oc1..aaaaaaaa7jmjfsserglnfew6doizijmvipi25jds6wgri5qoiqjjarnw7nfquser.oc1..aaaaaaaae7z2zgzhljbngnempv2jbmf6i5n4mesgz3sg2fqrnglramghe2gauser.oc1..aaaaaaaafc2wu6twlgpkugddy57dnqyzmlr66y4sxk6kuqbo4xvhuoq5nmxquser.oc1..aaaaaaaaizixnmedb4kdwedsaxnd42z2ww3ybfc7hooz325fzccddbzrrkbauser.oc1..aaaaaaaal5fjqe4xnpqoxfl3gaq72ncdthvxwbyyuzb6jvl2njhgs2legkwquser.oc1..aaaaaaaanndpesy5y3an4j7qa6pi4jhl62xj2w6umhwfekaelubhmc7itvaauser.oc1..aaaaaaaaqpog2576mpcyqi273moz7vdhfuhodcfm4emqjg42sl4okdd4pbxquser.oc1..aaaaaaaaxgvqmllcuj2mlvt4gc62k5vls2p72i3jvlhvr3wzgq3qldl2yxcauser.oc1..aaaaaaaazkqhlufgk7qv366kxfa57or25d224wl4gzgsfhlu5x5xjre6ha6a
        at org.apache.lucene.util.compress.LowercaseAsciiCompression.compress(LowercaseAsciiCompression.java:115)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:834)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:628)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.pushTerm(BlockTreeTermsWriter.java:947)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:912)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:318)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170)
        at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120)
        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:267)
        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476)
        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:390)
        at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:436)
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1757)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1400)

authors: mikemccand msokolov taroplus


text:
Improve Code Quality

Enables C#10
Use null propagation
Removes various unused using statements

Ready for review.

repo: apache/lucenenet


text:
[MENFORCER-273] Adding RequireUpperBoundDeps.excludes option

MENFORCER-273
Needed to implement JENKINS-41631: there are certain known cases (that for various reasons cannot be fixed any time in the near future) where a dependency is older than a transitive version, yet this has been manually confirmed as acceptable, and we wish to enforce the ban for everything else. Analogous to exclude options on various other rules.
@reviewbybees for my colleages

This pull request originates from a CloudBees employee. At CloudBees, we require that all pull requests be reviewed by other CloudBees employees before we seek to have the change accepted. If you want to learn more about our process please see this explanation.

Done.

@reviewbybees done

@jglick can you close this PR as it was merged in https://svn.apache.org/viewvc?view=revision&revision=1798819

@stephenc nope, you forgot to add the IT files.

r1798831 @jglick

Thanks!

@jglick WDYT about adding special exclude options for test dependencies? I have a couple of Jenkins plugins, where the conflicts come from the test dependencies which do not go into the release version. Although it is not perfect due to the test false-positive risks, such conflicts do not impact the real distributable.

authors: jglick oleg-nenashev reviewbybees stephenc

repo: apache/maven-enforcer


text:
Fixing the compiler args (should be passed as list)

Minor issue, it seems like compiler args should be passed as list, not as string, per plugin documentation (https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#compilerArgs)
<compilerArgs>
  <arg>-Xmaxerrs=1000</arg>
  <arg>-Xlint</arg>
  <arg>-J-Duser.language=en_us</arg>
</compilerArgs>

Thanks!

think maven has a converter for that which makes it working, no?

It seems like it depends on Maven version, f.e. 3.5.x is fine whereas 3.2.x fails:
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile (default-compile) on project meecrowave-oauth2: Unable to parse configuration of mojo org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile for parameter compilerArgs: Cannot assign configuration entry 'compilerArgs' with value '-XDignore.symbol.file' of type java.lang.String to property of type java.util.List -> [Help 1]

@reta oki, makes sense. Applies, thanks a lot!

authors: reta rmannibucau

repo: apache/meecrowave


text:
DataContextFactory.elasticsearchDataContext returns UpdateableDataContext

Brings DataContextFactory up to latest ElasticSearchDataContext capabilities

Oh right, we should update that :)

authors: bobby kaspersorensen

repo: apache/metamodel


text:
NIFI-1710 Resolve path name to nifi.sh in start script

This adds realpath to nifi.sh to resolve location of nifi.sh when its symlinked (instead of installed using the install command). Recent versions of Linux and BSD make realpath available.

I cant find packages for centos 6 that will install this in the os repos, epel repos etc. Have not checked other OS's. I upgraded coreutils and still did not get this command.

@arpitgupta Ok, yeah, older versions of coreutils don't have it. I was doing this with readlink but BSD readlink doesn't have the same options as GNU. Looks like it might be necessary to push this down into detectOS(), if people want this sort of change at all.

what about moving to python2.4 for these scripts. That way we get a unified file for linux and windows.

Please do not attempt to use python as a substitute for OS shell scripts. It's a path to ruin, the mechanics of properly managing things on Bash-having systems and Windows are fundamentally different (until we can just switch to bash-only for windows 10).

@jfrazee
could you rebase this PR?

authors: arpitgupta busbey jfrazee trixpan

repo: apache/nifi


text:
NIFI-6025: Include Processor 'scheduled state' (i.e., Enabled or Disa…

…bled) in the VersionedProcessor when pushing to Flow Registry and take into account when updating flows on the NiFi side
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Will review...

@markap14 If I only changed the scheduled state to disabled, it does not show as a local change. I expected that disabling a processor in my flow would be recognized as a local change. It was not. Once I made another change, I was able to save a new revision. Upon import of that revision that processor was correctly disabled.

@markap14 it seems like the scheduled state only gets applied if some other change is also made to the component which triggers the update.
Example:

Created a PG with proc1 enabled, proc2 disabled
Start version control
Import another instance of it and verify the procs have correct state
Change original PG so that proc1 is disabled, and proc2 is enabled, also moved position of proc2 to make a local change
Commit version 2
Change version on second PG and it correctly updates proc2, but leaves proc1 enabled

When stepping through in the debugger it never enters the updateProcessor method for proc1. The only reason I can think is because it wasn't considered an affected component b/c nothing in the flow diff showed proc1.
This probably ties back into Matt's question because if we start adding schedule state into the flow diff then it will become a local change by default, and then we have to decide if we want to add the code to filter it out on NiFi side.

@bbende I think you and @mcgilman are saying the same thing. That was my intent, but I don't have a strong preference whether or not to consider that a local change, one way or the other. But given that you both were surprised by the behavior, I will update the PR so that enabling or disabling the component will be considered a local change. Thanks.

OK I pushed an update to the PR. However, since we are changing the comparison logic, that has to be done in the nifi-registry project, so this is now blocked by NIFIREG-294.

+1 Tested the latest commit and it resolved the issue I described above.
Will merge this right after next registry release.

authors: bbende markap14 mcgilman


text:
NIFI-12079 - Remove Variables / Variable Registry

Summary
NIFI-12079 - Remove Variables / Variable Registry
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[ ] Apache NiFi Jira issue created

Pull Request Tracking

[ ] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[ ] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

The user guide still needs to be updated with new screenshots. I'll add this tomorrow but opening the draft PR to get some builds running and if someone wants to have a look.

Thanks David, yeah, I noticed that one (had some troubles running system tests locally). I'm trying to figure out some UI issues with my changes but not my domain of expertise.

What a thorough update!   I can tell there was some complexity with what to name this thing.  EnvironmentRegistry.  EnvSystemRegistry.  EnvironmentSystemRegistry, etc..  I recommend we just consistently call this thing ENVIRONMENT as the scope like you did then just use the term Environment everywhere else.  Define this scope as providing values in the order of JVM System Properties first then Environment variables available to the process and just leave it referred to as 'Environment'.    What do you think?

What a thorough update! I can tell there was some complexity with what to name this thing. EnvironmentRegistry. EnvSystemRegistry. EnvironmentSystemRegistry, etc.. I recommend we just consistently call this thing ENVIRONMENT as the scope like you did then just use the term Environment everywhere else. Define this scope as providing values in the order of JVM System Properties first then Environment variables available to the process and just leave it referred to as 'Environment'. What do you think?

+1 for ENVIRONMENT as the name of the scope going forward.

@pvillard31 you mind rebasing given the changes?  What do you think about getting rid of the EnvSys naming and just referring to it uniformly as EnvironmentVariables instead?

For now I just rebased against the main branch (I hope I did it correctly). Going through the changes to be more consistent with the naming.

Full clean build with contrib check good.  And Github CI looking great.
Reviewed each changed file all looks good.  Some of the changes in files seemed massive due to perhaps whitespace but doesn't seem worth delaying this further.
Ran a series of tests using system properties.  Environment variables. Parameter contexts.  Looking each up either directly in the processor properties of UpdateAttribute or via parameter context lookup.  All worked as expected.
Great work Pierre.  This is a massive PR.  But dont want to delay it further as it will impact many things going into 2.x from here out.  This one is so expansive it likely starts to really make the break betweeen 2.x and 1.x.
Other notable changing the scope from VARIABLE to ENVIRONMENT will surely make 1.x processors not run on 2.x but I think we're likely well into needing people to rebuild for 2.x anyway a thing.
+1 merging to main

authors: exceptionfactory joewitt pvillard31


text:
boards/esp32s3: Link stack checking function and data to SRAM when enable flash or PSRAM driver

Summary
This PR intends to fix the PSRAM support on ESP32S3 when Stack Smash Protection is enabled (CONFIG_STACK_CANARIES=1).
During PSRAM initialization and flash operations, the Cache needs to be disabled, so all data and code for the aforementioned scope is required to be placed in Internal RAM (DRAM and IRAM regions).
Impact
Only for PSRAM-enabled configurations on ESP32S3.
Testing

Successful execution of ramtest in esp32s3-devkit:psram_octal with CONFIG_STACK_CANARIES=1

@cwespressif
Please resolve the conflicts.

@cwespressif Please resolve the conflicts.

Done

repo: apache/nuttx


text:
sched: move etc romfs mount from nsh to sched/init

Summary
Usually the startup script is placed under /etc. The contents of the etc directory are compiled and linked with Nuttx binary in the form of romfs. After startup, it will be  mounted by Nsh.
etc is generated by the different boards, that use genromfs and xxd tools to generate and compile it into the Nuttx, for example: boards/arm/at32/at32f437-mini/tool/mkromfs.sh
The more common method is etc image generated from the content in the corresponding board/arch/board/board/src/etc directory, and added by Makefile for example: boards/sim/sim/sim/src/etc.
But in kernel/protected mode, Nuttx kernel and apps are run in different privileged/non-privileged mode or the isolated binarys, so as that nsh should use syscall to access Nuttx kernel by exported API. In this scenario, nsh can not mount the etc image content, because that is generated in board and as a part of Nuttx kernel.
changes:

move etc romfs mount from nsh to Nuttx, but keep the script to parse and execute.
move and rename the related CONFIG, move customized nsh_romfsimg.h to etc_romfs.c in boards, and no need declaration for romfs_img/romfs_img_len.

This commit changes and updates all configurations in Nuttx arch/board as much as possible,
but if any missing, please refer to the following simple guide:

rename CONFIG_NSH_ROMFSETC to CONFIG_ETC_ROMFS, and delete CONFIG_NSH_ARCHROMFS in defconfig
rename the etc romfs mount configs, for example CONFIG_NSH_FATDEVNO to CONFIG_ETC_FATDEVNO
move customized nsh_romfsimg.h to etc_romfs.c in board/arch/board/board/src and no need declaration for romfs_img/romfs_img_len.
delete default nsh_romfsimg.h, if ROMFSETC is enabled, should generate and compile etc_romfs.c in board/arch/board/board/src.

Impact
all boards config, could break CI need correspond to https://github.com/apache/nuttx-apps/pull/2256
Testing
qemu/sim

Please refresh boards/sim/sim/sim/configs/unionfs/defconfig @fxysunshine.

Please refresh boards/sim/sim/sim/configs/unionfs/defconfig @fxysunshine.

done

@fxysunshine This looks to be a massive breaking change.  What is the advantage of it?  What are all the implications it will have on downstream NuttX users.  What are all the steps thy will need to do to migrate to it?

@fxysunshine This looks to be a massive breaking change. What is the advantage of it?

This change is to support the mount /etc in kernel/protected mode.

What are all the implications it will have on downstream NuttX users. What are all the steps thy will need to do to migrate to it?

@fxysunshine please add a migration guide in commit message and pr.

@fxysunshine Why was this approach taken and why were all the build changes needed? Are thy needed? May be you could explain the Problem you are solving, and why this is a good solution.

@fxysunshine Why was this approach taken and why were all the build changes needed? Are thy needed? May be you could explain the Problem you are solving, and why this is a good solution.

@davids5 Thanks for your suggestions. Please refer to the problem and solution description in the updated commit and pr.

@fxysunshine Why was this approach taken and why were all the build changes needed? Are thy needed? May be you could explain the Problem you are solving, and why this is a good solution.

@davids5 Thanks for your suggestions. Please refer to the problem and solution description in the updated commit and pr.

@fxysunshine Thank you for the excellent  problem and solution description. This is super helpful!

@xiaoxiang781216 Do you know why CI is failing?

@fxysunshine I'll merge it, but I think you need to update our Documentation/, because there a lot of reference to CONFIG_NSH_ROMFSETC etc...

@xiaoxiang781216 Do you know why CI is failing?
@xiaoxiang781216 @davids5
lack of the apps corresponding commit https://github.com/apache/nuttx-apps/pull/2256, could not pass case scanftest.

Temporary ignore to check scanftest result https://github.com/apache/nuttx/pull/11498/files#diff-8c28e250b38860383a1f2e7a7610574b2358da67d92b6d523474bf30cefcf013, can pass CI.
Can we merge the PR, and revert the temp change after https://github.com/apache/nuttx-apps/pull/2256 is merged?

@fxysunshine I'll merge it, but I think you need to update our Documentation/, because there a lot of reference to CONFIG_NSH_ROMFSETC etc...

Sure.

@fxysunshine I'll merge it, but I think you need to update our Documentation/, because there a lot of reference to CONFIG_NSH_ROMFSETC etc...

@xiaoxiang781216 Do you know why CI is failing?

@xiaoxiang781216 @davids5 lack of the apps corresponding commit apache/nuttx-apps#2256, could not pass case scanftest.
Temporary ignore to check scanftest result https://github.com/apache/nuttx/pull/11498/files#diff-8c28e250b38860383a1f2e7a7610574b2358da67d92b6d523474bf30cefcf013, can pass CI.
Can we merge the PR, and revert the temp change after apache/nuttx-apps#2256 is merged?

Done, please revert the temp change.

Hi all, @xiaoxiang781216 @acassis @jerpelea
I just came accross this PR while trying to update the nuttx. Sorry to rant afterwards, I never noticed this before. This change just breaks all my configurations, and I think is wrong in many aspects

This is conceptually wrong. NSH is an application, a shell (or a combined shell and busybox kind of app...). NSH should reside ON the file ROMFS filesystem. In the end, you don't really want to "mount the romdisk from the NSH" nor even use the NSH as an "sbin/init" app (even though this is what is currently being done)
If we want a "common" mounting mechanism for "root" fs, we should have a separate simplified "init" in the kernel and NOT use NSH as an init process. You may just refer to other OSs (like linux). They are not using a shell to mount the root file system. This just doesn't make any sense!
There was no issues previously. You can just call "romdisk_register" and mount the (sbin) from your board initialization code

For sure we could adapt to this change, but I feel that it is taking us, and the role of the NSH application into a wrong direction.
I strongly suggest re-considering this approach, for now I am just reverting this change in our own branches.


text:
ORC-1393: Returning the possibility to change stream length when resetting UncompressedStream

What changes were proposed in this pull request?
Adding a possibility to re-set the length of the UncompressedStream when calling reset() on it.
Why are the changes needed?
In short, in some cases, after resetting an UncompressedStream, its actual length is longer than its initial length.
Before 'ORC-516 - Update InStream for column compression', InStream.UncompressedStream class had 'length' field and the length was modifiable in reset() method. ORC-516 was the root cause of this regression reported in HIVE-27128.
The reset() method was used in SettableUncompressedStream class in setBuffers() method:
public void setBuffers(DiskRangeInfo diskRangeInfo) {
  reset(diskRangeInfo.getDiskRanges(), diskRangeInfo.getTotalLength());
  setOffset(diskRangeInfo.getDiskRanges());
}

After Orc version upgrade in Hive to 1.6.7., and since SettableUncompressedStream class was removed from Orc code base, Hive manages it own copy of SettableUncompressedStream which doesn't pass new length to UncompressedStream when calling reset (because UncompressedStream doesn't accept new length any more in the reset method):
public void setBuffers(DiskRangeInfo diskRangeList) {
  reset(diskRangeList.getDiskRanges());
  setOffset(diskRangeList.getDiskRanges());
}

When investigating the issue reported in HIVE-27128 and comparing the lengths of the InStream.UncompressedStream prior to the upgrade of ORC version in Hive to 1.6.7. (which included ORC-516) and after I noticed that the issue happens with ORC-516 changes because the length of the InStream.UncompressedStream is set once for all row groups, while without those changes the length is dynamic and sometimes is set to bigger value than the initial value.
How was this patch tested?

Passing CI pipeline tests
Running Hive Q-test https://github.com/difin/hive/commits/orc_read_err_qtest on the local dev env with a change in Hive's SettableUncompressedStream to pass  diskRangeList.getTotalLength() to the reset method of UncompressedStream.

cc @omalley , @williamhyun , @guiyanakuang

Although this impl should not be used outside, I believe we need this because Apache Hive is the mother of Apache ORC initially.

Could we add a simple unit test like this:
Invocation the new reset(DiskRangeList input, long length) and test the real avaliable bypes of Instream?

I've added unit tests as suggested.

Thank you for updating, @difin .

Thank you for your contribution. ORC-1393 is resolved and assigned to you.
Welcome to the Apache ORC community.

authors: deshanxiao difin dongjoon-hyun

repo: apache/orc


text:
HDDS-4595. RDBScanner should print table name if it is not present in the db definition.

What changes were proposed in this pull request?
Recently discovered that with branch HDDS-2939, the newly added directory table is missing from the DB definition array.
So the tool wasn't able to parse the table which really exists on the rocksdb.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4595
How was this patch tested?
Manual testing

+1

repo: apache/ozone


text:
HDDS-4587. Merge remote-tracking branch 'upstream/master' into HDDS-3698

What changes were proposed in this pull request?
Merge remote-tracking branch 'upstream/master' into HDDS-3698
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4587
How was this patch tested?
Basic build and UTs. I will address CI failures.

The integration and acceptance test failures are caused by upgrading the Ratis version, which introduced RATIS-1305. They are the same errors seen in CI runs for HDDS-4610, which also requires the Ratis version increase. They are not related to the merge since the upgrade branch needs the Ratis version increased anyways.

LGTM +1

thanks @errose28 for fixing the review and fixing failures.


text:
HDDS-5849. [Ozone-Streaming]Write exceptions occur after checksum is enabled

What changes were proposed in this pull request?
This bug should be caused by HDDS-5742. We need to pass in a ReadOnlyBuffer, otherwise there will be problems with the written data. An error is reported while reading.
In addition, we did not detect this problem because checksum was turned off by default in the previous UT.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5849
How was this patch tested?
Fixed UT

Thanks @captainzmc for the quick fix.

authors: bshashikant captainzmc


text:
C++: Clear out license issues

Before the initial release, check that LICENSE and NOTICE contain the relevant information.

Also, license should be used to record third-party licensed works that are
included in the source distribution. The bit packing code should be in
there, rather than in notice. Notice is for required third-party notices
and isn't the file where third-party licensing information should be
accumulated.

Reporter: Uwe Korn / @xhochy
Assignee: Wes McKinney / @wesm
Note: This issue was originally created as PARQUET-776. Please see the migration documentation for further details.

Uwe Korn / @xhochy:
Yes, @wesm sorted that out before the release.

Deepak Majeti / @majetideepak:
@xhochy Is this resolved?

repo: apache/parquet-java


text:
GH-3029: Fix EncryptionPropertiesHelper not to use java.nio.file.Path

Rationale for this change
EncryptionPropertiesHelper has been changed to use java.nio.file.Path but the file: URI will fail on Windows.
See #3029 for detail.
What changes are included in this PR?
Change EncryptionPropertiesHelper not to use java.nio.file.Path.
Are these changes tested?
Pass CI.
Are there any user-facing changes?
No
Closes #3029

@gszadovszky Could you help review this? Thanks!


text:
PHOENIX-7038 : Implement Phoenix Connection Profile Metrics

This PR will add Connection Profile Level Metrics. It will be helpful when single JVM will create multiple Connections with variations in configuration of the connection. This can help in identifying, debug and monitor individual client behaviour.

I am in general very wary of overriding HBase properties from a Phoenix application, as this can very easily lead to failures for properties that must be in sync across the cluster for HBase to work.
In my ideal world we'd have some kind of whitelist for properties that are known to be safe to override (or an inverse blacklist)
(Of course users could still wreak havoc by modifying the hbase-site.xml used by the client)
I would tie the id propagation part to https://issues.apache.org/jira/browse/PHOENIX-5974.
We could either correlate that single ID with the ConnectionInfo/profile (though it requires extra bookeepeing), or just add it alongside the query id.

Just opened a related [DISCUSS] thread on dev, @dbwong .

Thanks for the reviews and comments @stoty @ranganathg @dbwong @jpisaac
I will go through comments and DISCUSS Thread today.

My main concern is this is too sfdc focused. A conection profile doesn't exist within the concept space of phoenix. I'd rebrand this as ConnectionInfo metrics.

That's right. I have changed it to ConnecitonQueryService Metrics since this metrics are calculated at CQS Level.

My second concern is there apears to be a bunch of copy pasted or similar code... I opened a Jira for a better overall metrics infrastrucutre and while that is out of scope is it possiblegenericfy some of this.

That's right there are multiple class/code blocks which are copy pasted. I will add notes of these PHOENIX-6174 to simplify in future.
@dbwong

I am in general very wary of overriding HBase properties from a Phoenix application, as this can very easily lead to failures for properties that must be in sync across the cluster for HBase to work.
In my ideal world we'd have some kind of whitelist for properties that are known to be safe to override (or an inverse blacklist)
(Of course users could still wreak havoc by modifying the hbase-site.xml used by the client)
I would tie the id propagation part to https://issues.apache.org/jira/browse/PHOENIX-5974.
We could either correlate that single ID with the ConnectionInfo/profile (though it requires extra bookeepeing), or just add it alongside the query id.

QUERY_SERVICE_NAME config property is only for Phoenix Connection and for metrics related to CSQI. It won't be used anywhere in HBase layer. As far as I understand, PHOENIX-5974 is different from this JIRA and having a default name for CQSI Object.
@stoty

Test failures are not related.

In my ideal world we'd have some kind of whitelist for properties that are known to be safe to override (or an inverse blacklist)
(Of course users could still wreak havoc by modifying the hbase-site.xml used by the client)

@stoty You are right, in the SFDC world, we do have such a whitelist in the app. Something we can think of bringing down into the Phoenix layer.

Test failure is not related.

authors: jpisaac mihir6692 stoty

repo: apache/phoenix


text:
Support empty segments for realtime tables

Addresses #12703
This patch allows persistence of empty segments in case of no messages in realtime ingestion. Right now, it can happen that a segment is stuck in CONSUMING state for months / years if no offset advances for it.
We also add support for retention manager finding empty segments aggressively and purging them.

Codecov Report
Attention: Patch coverage is 0% with 26 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (46469b5).
Report is 599 commits behind head on master.

Files
Patch %
Lines

...retention/strategy/CompositeRetentionStrategy.java
0.00%
22 Missing :warning:

...ention/strategy/EmptySegmentRetentionStrategy.java
0.00%
3 Missing :warning:

...troller/helix/core/retention/RetentionManager.java
0.00%
1 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13362       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2474       +38
  Lines        133233   136277     +3044
  Branches      20636    21167      +531
=============================================
- Hits          82274        0    -82274
- Misses        44911   136277    +91366
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
?

skip-bytebuffers-true
0.00% <0.00%> (-27.73%)
:arrow_down:

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Sajjad and I will review this and get back to you. Please hold merging for a bit.

I believe we already allow empty segments during force commit, so that by itself should not be a problem. @sajjad-moradi ?

I believe we already allow empty segments during force commit, so that by itself should not be a problem.

+1!
And we are ensuring here that we don't remove the last committed empty segment for keeping the metadata info so should be pretty safe imo.

I'm concerned that deleting the empty ones might make debugging a bit harder. If something goes wrong with segments of a partition, it would be hard by looking at IdealState, ExternalView, or SEGMENTS in PropertyStore to distinguish if a segment was purged due no-data case or something has indeed gone wrong.

In that case, we can enable the empty segment retention manager behind a config: deleteEmptySegments by default we can keep it to false. Ideally keeping a lot of empty segments in a table doesn't make sense if the commit threshold time is few hours and retention is high. Wdyt?

What is the current mechanism of removing empty segments?

What is the current mechanism of removing empty segments?

I see time-based retention manager should take care of it because we get segment.end.time info in metadata for empty segment as well.
But one scenario, specifically for upserts is that if we are setting indefinite retention and relying on UpsertCompaction to take care of purging segments then empty segments won't be a part of it. This is because we don't add EmptyIndexSegment to _trackedSegments and ultimately skip in snapshot flow as well.

Good point on the infinite retention case. I'm okay always removing empty segment as long as it is not the last committed one (used to resume consumption or mark the end of a partition). I think it is okay because removed segments are either expired or empty. I don't see much value keeping all empty segments around even for debugging purpose. @sajjad-moradi wdyt?

We won't have a lot of segments if we don't commit empty segments. A good thing about the existing behavior is that when we see a consuming segment that's started a long time ago, we can immediately tell that there hasn't been any new events on the stream, without needing for further investigation!
To get around the issue that this PR is trying to fix, can't we just issue a force commit, and then deleted those segments?
I don't like the idea of committing the empty consuming segments and then removing them in another task, retention manager. Retention manager, as its name suggests, should only delete segments that are older than the retention.
If you guys still think it's better to go with the approach suggested in this PR, let's have a config for committing empty consuming, and keep the existing behavior behind the default value of that config.

@sajjad-moradi One problem with force commit is that it won't commit the empty segment for the same reason.
I'm okay adding a config to allow committing empty segment, then allow retention manager to remove it if it is not the last committed segment.

when we see a consuming segment that's started a long time ago, we can immediately tell that there hasn't been any new events on the stream, without needing for further investigation!

We can monitor various metrics to determine if this scenario occurs after allowing this empty segments behaviour.
I will move this empty segment behaviour behind a config but somehow I am not convinced to not make this default behaviour just because of better debuggability. It's just that overtime we will have to manage a lot of configs for getting the correct functionality out of Pinot.

There are at least a couple of more considerations I can think of.  @tibrewalpratik17 can you please follow up on these?

The server keeps a rolling statistics buffer on one of the segments of the table that helps in deciding how much memory to allocate for consuming segments. A commit of a zero-sized segment may end up skewing these statistics unfavorably. See ReatimeSegmentStatsHistory.
When one segment completes, the controller computes the number of rows to be consumed for the next segment, based on the size of the current segment. Will this computation also get skewed with zero-sized segments? See FlushThresholdUpdater and implementations.

Thanks @mcvsubbu for the pointers!
Briefly going through the code, I observe the following:

The server keeps a rolling statistics buffer on one of the segments of the table that helps in deciding how much memory to allocate for consuming segments. A commit of a zero-sized segment may end up skewing these statistics unfavorably. See ReatimeSegmentStatsHistory.

I noticed that we only update the stats when _numRowsIndexed is strictly greater than 0 (Ref). Therefore, the behavior remains the same: if we do not commit an empty segment, the stats history is not updated, and this applies to the current scenario as well.

When one segment completes, the controller computes the number of rows to be consumed for the next segment, based on the size of the current segment. Will this computation also get skewed with zero-sized segments? See FlushThresholdUpdater and implementations.

Reviewing the implementation of SegmentSizeBasedFlushThresholdUpdater, it appears that empty segments are already handled (Ref), as the new threshold is set to the same value as the previous committed segment threshold. In the case of committing consecutive empty segments, if a table's partition has never received a message, it defaults to the autotuneInitialRows configuration value  (Ref).

retention manager always keeps the last completed segment even if it is already expired. I think we need to fix this behavior because it is violating the retention agreement.

but for very high / indefinite retention we would still end up with the consuming segment open for quite long.

@tibrewalpratik17 Consuming segment will be opened for long time only if there is no event consumed. As long as we can clean up all the expired segments it should be fine

authors: Jackie-Jiang codecov-commenter mcvsubbu sajjad-moradi tibrewalpratik17

repo: apache/pinot


text:
Define configuration store settings on proxy configmap

Motivation
This is required to active websockets and while it can be provided via
proxy.configData this is more ergonomic for users of the chart.
Modifications
Added zookeeperServers/configurationStoreServers to proxy config map
Verifying this change

[ ] Make sure that the change passes the CI checks.

@codelipenghui was this closed accidentally?

@josephglanville that was my screw-up - apologies. @codelipenghui can you help? Thanks!

authors: frankjkelly josephglanville

repo: apache/pulsar-helm-chart


text:
[Bug] Can't consume messages by using a subset of sticky key hash range

Search before asking

[X] I searched in the issues and found nothing similar.

Version
2.10.0
Minimal reproduce step
Here are a reproduce code by adding test on KeySharedSubscriptionTest class.
@Test
public void testSpecifiedStickyKeyRanges() throws PulsarClientException, InterruptedException {
    String topic = TopicName.get("persistent", "public", "default",
            "testSpecifiedStickyKeyRanges" + UUID.randomUUID()).toString();
    String subscriptionName = "my-sub";

    String keyToRead = "someKey1";
    String keyToExclude = "someKey2";
    int numMessages = 100;
    byte[] keyBytes = keyToRead.getBytes(StandardCharsets.UTF_8);
    int hash = Murmur3_32Hash.getInstance().makeHash(keyBytes) % KeySharedPolicy.DEFAULT_HASH_RANGE_SIZE;

    // start consumer
    CountDownLatch count = new CountDownLatch(numMessages);
    Consumer<String> consumer = pulsarClient.newConsumer(Schema.STRING)
            .topic(topic)
            .subscriptionName(subscriptionName)
            .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
            .subscriptionType(SubscriptionType.Key_Shared)
            .keySharedPolicy(KeySharedPolicy.stickyHashRange().ranges(Range.of(hash, hash)))
            .messageListener((c, msg) -> {
                c.acknowledgeAsync(msg).whenComplete((m, e) -> {
                    if (e != null) {
                        log.error("error", e);
                    } else {
                        count.countDown();
                    }
                });
            })
            .subscribe();

    pulsar.getExecutor().submit(() -> {
        try {
            try (Producer<String> producer = pulsarClient.newProducer(Schema.STRING)
                    .topic(topic)
                    .enableBatching(false)
                    .create();) {
                for (int i = 0; i < numMessages; i++) {
                    String value = "test" + i;
                    producer.newMessage()
                            .key(keyToRead)
                            .value(value)
                            .send();
                    producer.newMessage()
                            .key(keyToExclude)
                            .value(value)
                            .send();
                    Thread.sleep(100);
                }
            }
        } catch (Throwable t) {
            log.error("error", t);
        }
    });

    assertThat(CompletableFuture.runAsync(() -> {
        try {
            count.await();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(e);
        }
    })).succeedsWithin(Duration.ofMinutes(1));

    consumer.close();
}

What did you expect to see?
Consumed the sent messages with given key.
What did you see instead?
No messages can be consumed.
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

@eolivelli I got the reproduce test code passed after comment the code you added in #14014.
        for (Entry entry : entries) {
            int stickyKeyHash = getStickyKeyHash(entry);
            Consumer c = selector.select(stickyKeyHash);
            if (c != null) {
                groupedEntries.computeIfAbsent(c, k -> new ArrayList<>()).add(entry);
-                consumerStickyKeyHashesMap.computeIfAbsent(c, k -> new HashSet<>()).add(stickyKeyHash);
            } else {
                addMessageToReplay(entry.getLedgerId(), entry.getEntryId(), stickyKeyHash);
                entry.release();
            }
        }

For sticky mode, the consumer will only receive messages of hashes in the specified ranges.
In the above case, the consumer only config one hash range. But the produced messages hash are in other ranges, so the consumer is blocked.

Yep, you are right. I have found another workaround for this issue.
https://github.com/streamnative/flink/blob/pull/FLINK-26182/flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/PulsarConsumerBuilder.java

authors: Technoboy- syhily

repo: apache/pulsar


text:
should support IPv4/IPv6 dualstack

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
the Pular and broker should support the IPv4/IPv6 dualstack
Solution
No response
Alternatives
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

@zhangdejin-tom could you please provide more background and context? Better if you can have a reproduced repo for what's proposed here. One line "request" just does not work.

Closed as stale.

authors: tisonkun zhangdejin-tom


text:
[Doc] color code

Search before asking

[X] I searched in the issues and found nothing similar.

What issue do you find in Pulsar docs?

The color codes don't corresponds to the diagram.
What is your suggestion?
Either re-do the diagram using the color codes in the texts or remove the color-coded texts.
Any reference?
https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#message-retention-and-expiry
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Hi @anguillanneuf! Thanks for reporting this issue.
Since you click the "I'm willing to submit a PR!" box, I'll double-check that if you'll draw the diagram. This is not quite easy for software developers, though.

I unchecked the box and won't be submitting a fix @tisonkun. Thanks for the quick response!

Hi @momo-jun I've checked the commit history and found that seems you're the author of this change? If so, can you give us more context?

@Anonymitaet I've no idea about this one and didn't notice this image change. My commit related to message retention was about the cookbook topic, not the concept one.
I'm not sure about the motivation to change the images (maybe for style alignment ?) in 2.11 docs, but it did break the consistency between the content and the image. Just FYI, I think the easiest way to resolve this inconsistency can be:

Revert the original images used in 2.10.x
Remove the hard-coded color code in the MD file (maybe better).

@momo-jun OK, thanks!
Updating the color text to black text is a better way since black text and the image in NEXT and 2.11.x matches our current design style (while the image in 2.10.x does not)

Image in NEXT and 2.11.x

Image in 2.10.x

authors: Anonymitaet anguillanneuf momo-jun tisonkun


text:
[Issue:4726] Fix the problem that the log folder in pulsa-function-go is ignored

Signed-off-by: xiaolong.ran ranxiaolong716@gmail.com
Fixes #4726
Motivation
We repackaged the logrus log library and named it log. But at .gitignore#L37, we ignored all log files. The log file in pulsar-function-go is not uploaded.
In pulsar-function-go, the log file is the key to implementing the logtopic function, we can't ignore it. So this pull request renames the log to logutil to avoid this problem.

Thanks @freeznet for pointing this out, PTAL.

run java8 tests

run integration tests

run integration tests

run integration tests

run java8 tests
run cpp tests
run integration tests

run java8 tests

authors: sijie wolfstudy


text:
Redeliver response implement

Motivation

Redeliver response implement
add the tracker for response timeout

implement
Verifying this change
Add the tests for it
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes
Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (yes)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

This pr will close, because TC will handle the abort logical, then return the response to client. So, we don't need to handle this in client for redeliver response!


text:
Refactor Pulsar CI (with retries)

Test refactored workflow, with retries.

I opened the PR here too early. I'm currently testing this by opening PRs to my own fork, lhotari/pulsar .


text:
client与broker网络不一致情况下,findConsumerIdList机制是否合理

rocketmq使用的客户端负载均衡，hash算法是在客户端实现。
该算法使用的是数据来源为 namesrv提供的broker集合，和随机到一台broker上获取findConsumerIdList的集合。
改机制可靠性成立的两个关键核心是，broker与namesrv之间的网络可靠性必须一致，和client与broker之间的网络可靠性必须一致。
一旦发生局部网络不一致则有概率产生，两个client计算出统一个MessageQueue，这样commit操作是自相冲突的。会产生重复消费。

@beijixiongzzj 在网络出现问题或者是系统抖动的情况下确实有可能导致这些情况，所以rocketmq客户端有定时获取元数据做rebalance的机制，而且出现上述问题后，最大的可能在于重复消息，而不会丢失消息，我们认为这个是可以接受的，感谢评论，如果有诸如此类的咨询问题的话麻烦在我们的邮件列表里面讨论：https://rocketmq.apache.org/about/contact/，issue用于feature request 以及bug report。

authors: beijixiongzzj duhenglucky

repo: apache/rocketmq


text:
Why do the PullMessageService and RebalanceService threads also start when the producer starts,in release4.7.1

Before Creating the Enhancement Request

[X] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary
PullMessageService and RebalanceService seem to work only for consumers。If it is wrong, please help to interpret the role of these two threads on the producer, thanks!!!
Motivation
PullMessageService and RebalanceService, which only block or idle threads when the producer start
Describe the Solution You'd Like
Both threads are prohibited from starting when the producer starts
Describe Alternatives You've Considered
Both threads are prohibited from starting when the producer starts
Additional Context
No response

Both the producer and consumer at the bottom layer have an MQClientInstance. Currently, this is not differentiated between production and consumption, so both will start.

Both the producer and consumer at the bottom layer have an MQClientInstance. Currently, this is not differentiated between production and consumption, so both will start.

So this two threads, on the producer side, consuming performance but do nothing???

RebalanceService will perform empty polling after startup, waste of resources.

authors: leizhiyuan yihonglei zangchenyu0918


text:
[RIP-15]Add Ipv6 support for RocketMQ

What is the purpose of the change
add the support of IPv6
Brief changelog

Modified the message system flag to indicate which IP protocol the client and broker itself came from.
Modified the method of generating and decoding msgID
Modified the class related to message storage and decoding

Verifying this change
https://docs.google.com/presentation/d/1d4GXH0pdwQiUgzlwqJqWLmR5Punz6GG0RvMP0IgHvbM/edit?usp=sharing
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

@supercym It seems that the ci check failed, could you solve this problem first?

@supercym

Did you follow the ·Verifying this change· checklist to push  your pr?
Could you only commit your modified parts ? I have seen many irrelvant commits in other commits.

authors: duhenglucky supercym vongosling


text:
[ISSUE #1648]Fix the send back message sent into transaction half…

fix(transaction) fix the send back message sent into transaction half topic referred in this issue
What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.2%) to 50.488% when pulling 118bf22a4c1fe8d17495cc71f64c3f1ed74412ce on fix_transaction_sendback into f8f6fbe4aa7f5dee937e688322628c366b12a552 on develop.

authors: coveralls duhenglucky


text:
[Feature][Transform] Move transform from seatunnel-starter.jar to transform directory

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
Currently, transform is packaged in the seatunnel-starter.jar package. If users need to extend transform, they need to restart the zeta engine. We hope that transform, like connector, exists in a separate folder and is dynamically loaded when a task is submitted. At the same time, users can also dynamically add transform without restarting the engine.
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

please assign it to me, thx.

authors: Hisoka-X liunaijie

repo: apache/seatunnel


text:
akka.actor.ActorNotFound: Actor not found for

Caused by: akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://alpha-akka/), Path(/user/9cb2c448-09e7-4896-9a72-629a6a91539c)]
at akka.actor.ActorSelection.$anonfun$resolveOne$1(ActorSelection.scala:70)
at scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:60)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:77)
at akka.dispatch.BatchingExecutor.execute(BatchingExecutor.scala:120)
at akka.dispatch.BatchingExecutor.execute$(BatchingExecutor.scala:114)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:76)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallback(Promise.scala:312)
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:303)
at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:68)
at org.apache.servicecomb.pack.alpha.fsm.sink.SagaActorEventSender.send(SagaActorEventSender.java:70)
... 2 more
2019-09-02 15:30:44.505 ERROR Fxdemon --- [torEventChannel] o.a.s.p.a.f.c.MemoryActorEventChannel    : akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://alpha-akka/), Path(/user/9cb2c448-09e7-4896-9a72-629a6a91539c)]
java.lang.RuntimeException: akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://alpha-akka/), Path(/user/9cb2c448-09e7-4896-9a72-629a6a91539c)]
at org.apache.servicecomb.pack.alpha.fsm.sink.SagaActorEventSender.send(SagaActorEventSender.java:79)
at org.apache.servicecomb.pack.alpha.fsm.channel.MemoryActorEventChannel$EventConsumer.run(MemoryActorEventChannel.java:60)
at java.lang.Thread.run(Thread.java:748)
Caused by: akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://alpha-akka/), Path(/user/9cb2c448-09e7-4896-9a72-629a6a91539c)]
at akka.actor.ActorSelection.$anonfun$resolveOne$1(ActorSelection.scala:70)
at scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:60)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:73)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:77)
at akka.dispatch.BatchingExecutor.execute(BatchingExecutor.scala:120)
at akka.dispatch.BatchingExecutor.execute$(BatchingExecutor.scala:114)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:76)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallback(Promise.scala:312)
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:303)
at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:68)
at org.apache.servicecomb.pack.alpha.fsm.sink.SagaActorEventSender.send(SagaActorEventSender.java:70)
... 2 more
2019-09-02 15:30:44.507 ERROR Fxdemon --- [torEventChannel] o.a.s.p.a.f.c.MemoryActorEventChannel

saga状态机模式 这个异常是什么原因？

saga状态机模式 这个异常是什么原因？

0.5.0版本中actor获取是同步阻塞的模式，一旦超时，就会throw一个ActorNotFound的runtimexception，需要调整actor获取的超时时间或者等下一个版本采用异步的方式
  private static final Timeout lookupTimeout = new Timeout(Duration.create(1, TimeUnit.SECONDS));

authors: Fxdemon liulzz

repo: apache/servicecomb-pack


text:
Menu selected style not updated

Click any menu first, and then click home. The previous menu style is still selected.

repo: apache/shenyu-dashboard


text:
[BUG] I use master branch and can't start shenyu-admin

Is there an existing issue for this?

[ ] I have searched the existing issues

Current Behavior
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-07-05 16:46:01.940 ERROR 16648 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'shiroFilterFactoryBean' defined in class path resource [org/apache/shenyu/admin/shiro/config/ShiroConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.shiro.spring.web.ShiroFilterFactoryBean]: Factory method 'shiroFilterFactoryBean' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:270) ~[spring-context-5.3.28.jar:5.3.28]
at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:764) ~[spring-context-5.3.28.jar:5.3.28]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:567) ~[spring-context-5.3.28.jar:5.3.28]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.13.jar:2.7.13]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731) [spring-boot-2.7.13.jar:2.7.13]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) [spring-boot-2.7.13.jar:2.7.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) [spring-boot-2.7.13.jar:2.7.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303) [spring-boot-2.7.13.jar:2.7.13]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292) [spring-boot-2.7.13.jar:2.7.13]
at org.apache.shenyu.admin.ShenyuAdminBootstrap.main(ShenyuAdminBootstrap.java:36) [classes/:na]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.shiro.spring.web.ShiroFilterFactoryBean]: Factory method 'shiroFilterFactoryBean' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.3.28.jar:5.3.28]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-5.3.28.jar:5.3.28]
... 19 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.shenyu.admin.shiro.config.ShiroConfiguration.shiroFilterFactoryBean(ShiroConfiguration.java:75) ~[classes/:na]
at org.apache.shenyu.admin.shiro.config.ShiroConfiguration$$EnhancerBySpringCGLIB$$9b40b978.CGLIB$shiroFilterFactoryBean$2() ~[classes/:na]
at org.apache.shenyu.admin.shiro.config.ShiroConfiguration$$EnhancerBySpringCGLIB$$9b40b978$$FastClassBySpringCGLIB$$4c017df.invoke() ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.3.28.jar:5.3.28]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.3.28.jar:5.3.28]
at org.apache.shenyu.admin.shiro.config.ShiroConfiguration$$EnhancerBySpringCGLIB$$9b40b978.shiroFilterFactoryBean() ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_65]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_65]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.28.jar:5.3.28]
... 20 common frames omitted
Expected Behavior
No response
Steps To Reproduce
No response
Environment
ShenYu version(s):

Debug logs
No response
Anything else?
No response

no problem

还是不行啊，shenyu-admin启动不起来

repo: apache/shenyu


text:
Bump up BanyanDB 0.3.0 client.

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[x] Update the CHANGES log.

Follow the latest BanyanDB release.

@lujiajing1126 @hanahmily 0.3.0 DB or client may have issues. I have tried to rerun, but still fail in e2e.

repo: apache/skywalking