text:
TransactionView: ReceiveAndBuffer

Problem

Need to ingest new tx type into banking stage

Summary of Changes

Implement a new TransactionViewReceiveAndBuffer to allow banking stage to ingest TransactionView-based transactions
Add argument to BankingStage::new to switch between transaction representations
Hook up CLI for validator and relevant benchmarks
Run all BankingStage and SchedulerController tests using both options

Fixes #

@tao-stones @jstarry if you are back from vacation(s) would really appreciate review on this.
Somewhat significant changes to banking and I've been holding up some other stuff until this gets merged to avoid having 100 conflicts


text:
[syscalls] Return SyscallError::InvalidAttribute on invalid curve or op id

Problem
Summary of Changes
Fixes #

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.8%. Comparing base (b6d2237) to head (a203935).

Additional details and impacted files
@@            Coverage Diff            @@
##           master     #412     +/-   ##
=========================================
- Coverage    81.9%    81.8%   -0.1%
=========================================
  Files         840      840
  Lines      228058   228058
=========================================
- Hits       186795   186769     -26
- Misses      41263    41289     +26

source: gharchive/pull-request

authors: codecov-commenter samkim-crypto

repo: anza-xyz/agave

license_type: permissive

license_source: github-api


text:
add some get helper functions

work in progress

probably can make these more useful by separating the part that does trial aligning from the part that gets data, i.e.
def get_trial_aligned_data(data, preproc_dir, preprocessed_filename, trial_start_codes, trial_end_codes,
                           trial_filter=lambda x:True):
    '''
    '''
    data = aopy.data.load_hdf_group(preproc_dir, preprocessed_filename, 'exp_data')
    metadata = aopy.data.load_hdf_group(preproc_dir, preprocessed_filename, 'exp_metadata')

    event_codes = data['events']['code']
    event_times = data['events']['timestamp']

    trial_segments, trial_times = aopy.preproc.get_trial_segments(event_codes, event_times,
                                                                  trial_start_codes, trial_end_codes)
    trajectories = np.array(aopy.preproc.get_data_segments(data, trial_times, samplerate), dtype='object')
    trial_segments = np.array(trial_segments, dtype='object')
    success_trials = [trial_filter(t) for t in trial_segments]

    return trajectories[success_trials], trial_segments[success_trials]

repo: aolabNeuro/analyze

license: MIT


text:
Adapt to the side keys of the Onyx reader

I don't know if this is actually possible.

I can configure the side buttons to scroll the Obsidian screen. I was wondering if it's possible for me to capture this signal, or event?

Implemented by https://github.com/aoout/onyx_side_button

source: gharchive/issue

repo: aoout/obsidian-epub-importer


text:
Faild To Lanch camt find the required mod:origins

it wont start i tryed like  50times # #

Based on the title, sounds like you need to install a newer version of Origins for one of your dependencies.
However, this is not the place for installation help, please join the Fabric or Origins Discord for that.


text:
GarbageCollectorIT.gcLotsOfCandidatesIT failure

The GarbageCollectorIT is currently failing, specifically gcLotsOfCandidatesIT is timing out.

@milleruntime ,  I am looking into this.  It fails one time out of 10 consecutive runs for me.  I am trying to see how I can check if the lock is held before trying to delete it in the killMacGc function in the GarbageCollectorIT.  It only times out trying to remove the lock for some reason.

@jzgithub1 Do you know where in the code it is getting stuck? Can you run jstack on the process to see what it's doing?

@ctubbsii  between 10-20% of the runs I do fail here:
020-03-02T17:01:02,678 [functional.ConfigurableMacBase] ERROR: Unable to delete ZooLock for mini accumulo-gc java.lang.IllegalStateException: No lock is held at /accumulo/ebab5cc3-cdad-43fc-980b-c79a4fbda200/gc/lock 	at org.apache.accumulo.fate.zookeeper.ZooLock.deleteLock(ZooLock.java:499) ~[classes/:?] 	at org.apache.accumulo.test.functional.GarbageCollectorIT.killMacGc(GarbageCollectorIT.java:106) [classes/:?] 	at org.apache.accumulo.test.functional.GarbageCollectorIT.gcLotsOfCandidatesIT(GarbageCollectorIT.java:145) [classes/:?]

@jzgithub1 thanks for looking at this.
I have seen that TestTimedOutException but I am not sure how useful that is.  The same goes with the similar client exception I have seen:
java.lang.IllegalStateException: No lock is held at /accumulo/1d1562ff-4b75-47e1-b7d0-edf08778bd96/gc/lock
	at org.apache.accumulo.fate.zookeeper.ZooLock.deleteLock(ZooLock.java:499) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
	at org.apache.accumulo.test.functional.GarbageCollectorIT.killMacGc(GarbageCollectorIT.java:106) [classes/:?]
	at org.apache.accumulo.test.functional.GarbageCollectorIT.gcLotsOfCandidatesIT(GarbageCollectorIT.java:145) [classes/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12]
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) [junit-4.12.jar:4.12]
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) [junit-4.12.jar:4.12]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

We need to figure out what is happening on the server side that is causing the timeout.  Try doing a jstack on the pid of one of the IT processes.

Could there be a race condition with the shutdown?  From one of the stack traces it seemed to hint that the gc was shutdown (which should remove the lock) and then the clean-up is trying to delete the now non-existant lock.
If that is the case, then if gc shutdown is called, the delete could accept that the lock has already been removed by catching / ignoring the node does not exist exception.
Or, is this test also trying to validate the gc behavior w.r.t. losing the lock?

Looking at the IT, it is waiting for a specific output, which could have changed:
      while (!output.contains("delete candidates has exceeded")) {

Looking at the IT, it is waiting for a specific output, which could have changed:
      while (!output.contains("delete candidates has exceeded")) {

The output I think it is waiting for is here:
https://github.com/apache/accumulo/blob/e4b69f976c7223ff7dc3fd930462c323da0ef7ae/server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java#L219-L222
Perhaps with recent improvements, it is not running out of memory so never meets this condition.

I never hit those lines when I set breakpoints in them.

Are you seeing this in the SimpleGarbageCollector error log?
cat test/target/mini-tests/org.apache.accumulo.test.functional.GarbageCollectorIT_gcLotsOfCandidatesIT/logs/SimpleGarbageCollector_*.err

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.apache.accumulo.gc.SimpleGarbageCollector"

Let me look.

@milleruntime ,  I saw:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "org.apache.accumulo.gc.SimpleGarbageCollector" Exception in thread "Thread-5" java.lang.OutOfMemoryError: Java heap space

OK it looks like the SimpleGarbageCollector is running out of memory before it can call almostOutOfMemory I posted above.

I wrapped the call gc.runServer() in the main function of SimpleGarbageCollector in a try catch like this and did not hit a breakpoint set in the log.error statement I wrote:
public static void main(String[] args) throws Exception {
    try (SimpleGarbageCollector gc = new SimpleGarbageCollector(new ServerOpts(), args)) {
      try {
        gc.runServer();
      } catch (OutOfMemoryError outOfMemoryError) {
        log.error("Out of memory error in SimpleGarbageCollector");
      }
    }

@milleruntime or @EdColeman ,  I tailed all of the logs while running the test and when it does fail I see a connection loss to ZooKeeper in the TabletServer log.
./test/target/mini-tests/org.apache.accumulo.test.functional.GarbageCollectorIT_gcLotsOfCandidatesIT/logs/TabletServer_812067704.out <== 2020-03-03T15:02:01,081 [zookeeper.ZooReader] WARN : Saw (possibly) transient exception communicating with ZooKeeper org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /accumulo/9182a659-cfaf-4a19-b4c7-fb875091fc7b/tservers/CoolJeff:35337

I think the changes I made in 87ad69931f79c0755ac967d74ad3c9ca3868fa40 changed the timing of the test since it used to run out of memory but it would happen at a different time.  I am going to play around with the IT to see if I can get the timing right.

If things are running out of physical memory I suppose that all sorts of errors could pop up.  You might want to check if things are swapping to disk while the test is running.  (linux free, top... command) If not, then it could be hitting the JVM limits and doing expensive stop-the-world gcs - might be info in the logs if gc details are enabled, if not you may be able to enable them.  On a physical system, just add the jvm options in accumulo-env.sh - but I'm not certain what mini uses, but if not the same, there might be an equivalent

I am pushing a fix... it had to do with the Java GC not having enough memory to start the process.  The test was manually setting the default memory to 16MB, which wasn't enough.  The Java GC for mini was changed in 33d8f2039541addc96849bce55e6fff3265957c8

@milleruntime ,  thanks  a lot.  Looking forward to testing it out.

The G1GC (the default GC in newer JVMs) has more overhead, and can very quickly run out of memory on a JVM configured with too little memory. We saw the same issue in Upgrader9to10 and fixed it in #1441. I checked other places in our ITs to see where we set the JVM memory size for processes launched by mini, and this was the only one as low as 16MB. 32MB seems to be a reasonable minimum for G1GC.
Thanks for looking into this @jzgithub1 and @milleruntime . Glad we got it figured it.

authors: EdColeman ctubbsii jzgithub1 milleruntime

repo: apache/accumulo


text:
ARTEMIS-2716 Implements pluggable Quorum Vote

https://issues.apache.org/jira/browse/ARTEMIS-2716

Sadly canAcquireLockOnMajorityRestart can still get some failures because of https://issues.apache.org/jira/browse/CURATOR-594, I'm awaiting the PR I've sent on https://github.com/apache/curator/pull/383 to be merged

I see that https://issues.apache.org/jira/browse/ARTEMIS-2297 can cause ReplicationManager::sendSynchronizationDone to behave differently with the pluggable quorum vote:
the original JIRA was meant to force an isolated master to suicide to save split brain to happen; while the new quorum this is not needed because network isolation is already handled by the distributed lock listener, that should force the broker to suicide.
I probably need to modify ReplicationManager to ignore the quorum members count or just save QuorumManager to be created.

Some notes on the existing test failures:

org.apache.activemq.artemis.tests.smoke.quorum.PluggableQuorumSinglePairTest#testBackupFailoverAndPrimaryFailback can get a failing test using forceKill=false (that's using a gentle SIGTERM to stop live primary): it's likely a bug/feature of Atomixwhile using Raft, because if a killed live was a Raft leader too, a soft kill would make it able to release its live lock and then crash, forcing a re-election and causing the failing over backup to "reconsider" the ownership of the live lock just obtained:

[main] 17:47:51,694 INFO  [org.apache.activemq.artemis.tests.smoke.quorum.PluggableQuorumSinglePairTest] killing primary
**********************************
Killing server java.lang.UNIXProcess@674658f7
**********************************
atomixReplicationBackup-out:2021-05-26 17:47:51,706 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to localhost/127.0.0.1:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
atomixReplicationBackup-out:2021-05-26 17:47:51,706 WARN  [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure to localhost/127.0.0.1:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
atomixReplicationBackup-out:2021-05-26 17:47:51,765 INFO  [org.apache.activemq.artemis.quorum.atomix.AtomixDistributedLock] Failed to acquire lock ac6f986d-be39-11eb-981a-8cc68169c75b
atomixReplicationBackup-out:2021-05-26 17:47:51,861 INFO  [org.apache.activemq.artemis.quorum.atomix.AtomixDistributedLock] Acquired lock ac6f986d-be39-11eb-981a-8cc68169c75b with version Version{version=10}
atomixReplicationBackup-out:2021-05-26 17:47:51,863 INFO  [org.apache.activemq.artemis.core.server] AMQ221037: ActiveMQServerImpl::serverUUID=ac6f986d-be39-11eb-981a-8cc68169c75b to become 'live'
atomixReplicationBackup-out:2021-05-26 17:47:51,872 WARN  [org.apache.activemq.artemis.core.client] AMQ212004: Failed to connect to server.
atomixReplicationBackup-out:2021-05-26 17:47:51,974 INFO  [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address exampleTopic supporting [MULTICAST]
atomixReplicationBackup-out:2021-05-26 17:47:51,974 INFO  [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address exampleQueue supporting [ANYCAST]
atomixReplicationBackup-out:2021-05-26 17:47:51,975 INFO  [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue exampleQueue on address exampleQueue
atomixReplicationBackup-out:2021-05-26 17:47:52,164 WARN  [org.apache.activemq.artemis.core.server.impl.ReplicationBackupActivation] org.apache.activemq.artemis.quorum.UnavailableStateException: io.atomix.primitive.PrimitiveException: java.net.ConnectException: Failed to connect to the cluster
atomixReplicationBackup-out:2021-05-26 17:47:52,164 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=This server cannot check its role as a live: activation is failed]
atomixReplicationBackup-out:	at org.apache.activemq.artemis.core.server.impl.ReplicationBackupActivation.startAsLive(ReplicationBackupActivation.java:194) [artemis-server-2.18.0-SNAPSHOT.jar:2.18.0-SNAPSHOT]
atomixReplicationBackup-out:	at org.apache.activemq.artemis.core.server.impl.ReplicationBackupActivation.run(ReplicationBackupActivation.java:157) [artemis-server-2.18.0-SNAPSHOT.jar:2.18.0-SNAPSHOT]
atomixReplicationBackup-out:	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:4240) [artemis-server-2.18.0-SNAPSHOT.jar:2.18.0-SNAPSHOT]
atomixReplicationBackup-out:
atomixReplicationBackup-out:2021-05-26 17:47:55,018 WARN  [io.atomix.protocols.raft.roles.FollowerRole] RaftServer{data-partition-1}{role=FOLLOWER} - io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:7777
atomixReplicationBackup-out:2021-05-26 17:47:55,019 INFO  [io.atomix.protocols.raft.impl.RaftContext] RaftServer{data-partition-1} - Transitioning to CANDIDATE
atomixReplicationBackup-out:2021-05-26 17:47:55,021 INFO  [io.atomix.protocols.raft.roles.CandidateRole] RaftServer{data-partition-1}{role=CANDIDATE} - Starting election
atomixReplicationBackup-out:2021-05-26 17:47:55,032 WARN  [io.atomix.protocols.raft.roles.CandidateRole] RaftServer{data-partition-1}{role=CANDIDATE} - io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:7777
atomixReplicationBackup-out:2021-05-26 17:47:55,034 INFO  [io.atomix.protocols.raft.impl.RaftContext] RaftServer{data-partition-1} - Transitioning to LEADER
atomixReplicationBackup-out:2021-05-26 17:47:55,041 INFO  [io.atomix.protocols.raft.impl.RaftContext] RaftServer{data-partition-1} - Found leader backup
[raft-server-data-partition-1] 17:47:55,052 INFO  [io.atomix.protocols.raft.impl.RaftContext] RaftServer{data-partition-1} - Found leader backup
[main] 17:47:55,283 INFO  [org.apache.activemq.artemis.tests.smoke.quorum.PluggableQuorumSinglePairTest] killed primary

It shows that
atomixReplicationBackup-out:2021-05-26 17:47:51,765 INFO  [org.apache.activemq.artemis.quorum.atomix.AtomixDistributedLock] Failed to acquire lock ac6f986d-be39-11eb-981a-8cc68169c75b
atomixReplicationBackup-out:2021-05-26 17:47:51,861 INFO  [org.apache.activemq.artemis.quorum.atomix.AtomixDistributedLock] Acquired lock ac6f986d-be39-11eb-981a-8cc68169c75b with version Version{version=10}

live lock is acquired by the failing over backup, but
atomixReplicationBackup-out:2021-05-26 17:47:52,164 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=This server cannot check its role as a live: activation is failed]

The lock is not considered available and the failover process is stopped.
The critical part seems:
atomixReplicationBackup-out:2021-05-26 17:47:55,034 INFO  [io.atomix.protocols.raft.impl.RaftContext] RaftServer{data-partition-1} - Transitioning to LEADER

It shows that backup has become the RAFT leader (given that live isn't around and that's ok, but: why backup hasn't suicide itself given that it's not able to start as a live? This maybe need some attention.

org.apache.activemq.artemis.tests.smoke.quorum.ZookeeperPluggableQuorumSinglePairTest is hitting some thread leaks on ListenerHandler threads
eg

Thread Thread[ListenerHandler-/127.0.0.1:38567,5,main] is still alive with the following stackTrace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
...

@franz1981 i think thats a blocker issue, you should be able to safely force kill, and also soft kill. I like the phrasing "bug/feature" LOL arent all bugs just features to be implemented :P

@michaelpearce-gain agree. Just would like to understand if I can just force witness node to always be a raft leader, making it similar to a zookeeper external node. But seems a bit too limiting, maybe.
The other important bit, maybe my fault is

why backup hasn't suicided itself given that it's not able to start as a live?

Tomorrow I am going to check this one instead.

Update on

why backup hasn't committed suicide given that it's not able to start as a live?

Assuming a correct Atomix behaviour It seems related to a bug on my side:
   private void startAsLive(final DistributedLock liveLock) throws Exception {
         // ...
         // IMPORTANT:
         // we're setting this activation JUST because it would allow the server to use its
         // getActivationChannelHandler to handle replication
         final ReplicationPrimaryActivation primaryActivation = new ReplicationPrimaryActivation(activeMQServer, distributedManager, policy.getLivePolicy());
         liveLock.addListener(primaryActivation);
         activeMQServer.setActivation(primaryActivation);
         activeMQServer.initialisePart2(false);
         final boolean stillLive;
         try {
            stillLive = liveLock.isHeldByCaller();
         } catch (UnavailableStateException e) {
            LOGGER.warn(e);
            throw new ActiveMQIllegalStateException("This server cannot check its role as a live: activation is failed");
         }
         if (!stillLive) {
            throw new ActiveMQIllegalStateException("This server is not live anymore: activation is failed");
         }
         // ...

If the quorum is lost before liveLock.addListener(primaryActivation), the current AtomixDistributedLock::onStateChanged:
   private void onStateChanged(PrimitiveState state) {
      LOGGER.info(state);
      switch (state) {
         case SUSPENDED:
         case EXPIRED:
         case CLOSED:
            for (LockListener listener : listeners) {
               listener.stateChanged(LockListener.EventType.UNAVAILABLE);
            }
            break;
      }
   }

It's going to find empty listeners and ReplicationPrimaryActivation::stateChanged won't be called to async stop the server.
The late check liveLock.isHeldByCaller should fail and throw an exception, but it won't cause the server to stop.
In short, the issue is that liveLock.isHeldByCaller must be able to stop the server and it should call ``AtomixDistributedLock::onStateChanged` in case of a lost lock, before throwing the exception.
This same timing issue could happen with Zookeeper too, so it worth to be fixed.
Re the Atomix issue instead, I'm going to investigate a bit more what's going on, because it doesn't seem that AtomixDistributedLock::onStateChanged has been called while  liveLock.isHeldByCaller has failed, that seems a bug.

I believe to have ported enough integration tests: I'm now going to add some more smoke tests, improve logs/configs and maybe prepare some doc

Re the thread leak on smoke tests on

org.apache.activemq.artemis.tests.smoke.quorum.ZookeeperPluggableQuorumSinglePairTest is hitting some thread leaks on ListenerHandler threads

I see that https://github.com/apache/zookeeper/blob/c74658d398cdc1d207aa296cb6e20de00faec03e/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/QuorumCnxManager.java#L915 seems a Thread created by Zookeeper: despite usiug a leak-free zk client version (see https://issues.apache.org/jira/browse/CURATOR-595) this seems a separate issue: going to take a better look why/what it is.

waiting https://issues.apache.org/jira/browse/ZOOKEEPER-4309 to be fixed and ported to any zk version

I decide 2 important things re this PR:

going to drop Atomix support -community is not reactive and the solution itself doesn't seem solid as expected -
there are some loop in the primary state diagram that can be dangerous:
on primary start: we loop forever searching some broker with the same Node ID to pair with (to failback) or we try acquire the live lock -> both should be attempted just once
during failback: if the failback process is going to fail, the primary is restarting as primary, getting into the loop mentioned above -> it should just stop

@jbertram @gtully Any opinons?

I think you are right.
I would go further, and drop the fail-back feature in favour of a shutdown with delayed (after lock release sleep(epoch)) jvm exit.
Have a broker role be determined by the shared lock, the first to acquire becomes primary, the others potential backups.
The existing roles flow from the possibility of unilateral decisions, two primary brokers could be two live brokers in error. However with a reliable distributed lock and no unilateral decisions (exit once state is unknown) multiple competing brokers are fine.
This is a simpler model, easier to reason about and totally deterministic. Availability is tied to the quorum. To have more HA in the event of a lock service outage (ie quorum nodes fail), introduce another cluster of nodes.

I disagree failback is very important, feature relied on by many, i dont think its something we can simply just drop.

sorry, I need to give some more context, dropping failback b/c it is no longer necessary is what I am thinking and I am considering/assuming some additional state from the lock.
Let me elaborate and see if this makes sense/holds water!
The bottom line here is: we must avoid unilateral decisions b/c our journal cannot reconcile concurrent writes on different nodes. ie: we cannot deal with split brain, so we must avoid it.
With a simple distributed lock, there is an implicit timeout or epoch. If we track the current epoch we can coordinate our local state transitions and avoid unilateral decisions.
in a simpler state model, a broker transitions from:
 BROKER - PRIMARY - REPLICATED
or:
 BROKER - BACKUP - IN_SYNC_REPLICA
On restart:
For the next 'in sequence' epoch the REPLICATED and IN_SYNC_REPLICA are identical, either can become PRIMARY and race. The winner becomes PRIMARY and the loser BACKUP.
In this case, delaying the exit of one can replicate or force 'failback'.
The more interesting case, if an epoch is skipped, our local state is stale, revert to BROKER and exit in the hope/knowledge/expectation that PRIMARY will restart. PRIMARY is no longer REPLICATED and the IN_SYNC_REPLICA is stale. PRIMARY must restart.
Here is the crux, we trade off availability for consistency. The only failover that can happen is from REPLICATED to an IN_SYNC_REPLICA

in short, with a reliable quorum (distributed lock with epoch), we use it:

to handle the transition that we can currently support, one failover from primary to backup.
to completely avoid split brain (that term is odd!) and trade availability.

If we can do this with 3nodes, replicating that setup (2 more nodes that share a witness node) can give us better HA

So to avoid split brain is simple with distributed lock and ZK, only one process can hold that lock, and to be active you must have the lock, application with the lock should constantly be checking it still has the lock and if not disabling itself. You should never become active unless you get the lock, there should be no epoch about it, if you do that, you're asking for disaster.
With ZK how many artemis nodes you have makes no longer any difference to the avoiding split, you should be able to run just 2 artemis nodes for HA  (for ZK yes it requires 3 minimum) but those scale differently you're not scaling those for data plane use case.

Maybe this is something better to be discussed with whiteboards and a community call?

You should never become active unless you get the lock, there should be no epoch about it, if you do that, you're asking for disaster.

what you want is to have the lock and the current data. The lock alone is not sufficient.
Consider this:
primary starts, backup starts and becomes an in sync replica. Both die. primary comes back up for a while, and dies. backup restarts, It gets the lock but has stale data. boom!
The distributed lock only guards access to the journal, but something else (the epoch counter) has to guard the singular valid transition from primary to in sync backup. It is only valid if the lock is next held by the backup.

So the instance shouldn't even be trying to even get the lock until it is back "back-in-sync", it should only be trying to get the lock in case it "could" take over.

I believe the epoch/version that should both mark the journal (or replicated journal) and a live role acquisition (hence should be something on ZK, indexed on the NodeID?) to be key to save any journal misalignment to happen, that's something even more important then classic split-brain: as @michaelpearce-gain already mentioned, the distributed lock should already prevent it to happen. Happy that both have raised a similar/same idea: this make me more convinced that something like that is the way to go.
Re this PR: I wanted to provide the above "enhancement" in a later phase really, as a separate improvement, while here I'm "just" trying to abstract away the quorum algorithm + providing a RI that allow a single pair to work fine + addressing some dark corners of the state diagram in the "classic" native quorum replication.
The loops I've mentioned above are my fault while trying to fill some of the gaps in the existing replication state transitions.
A brief summary here:

classic: master's start by searching for existing live node with the same node ID to pair with and, if none are around, just starting straight as live -> this one is an easy cause of split brain with network partition
now: primary's start involves a loop, first searching for existing live node with the same node ID to pair with and, if none is around, try just once to acquire the live lock before starting as live: if the live lock acquisition fail, retry to search existing live nodes etc etc

I think that this loop isn't needed, but could be performed just once ie search for other lives, if not found, try acquire live lock (for some time) and if failed, just exit.
It saves a completely isolated broker to keep on retrying to acquire the lock and/or search other lives forever. wdyt?
Another "hidden" loop:

classic: during a master fail-back failure (see SharedNothingBackupQuorum.BACKUP_ACTIVATION.FAILURE_REPLICATING on the code) the broker is restarted as pure backup (no able to failback anymore!) deleting it's data folder too, including the node ID.
That's dangerous because deleting the data folder will cause the master to loose it's identity (NodeID is going to change or restart) and if the node is manually restarted it will search for an non-existing node ID and able to start (SPLIT BRAIN served!).
new: during a primary fail-back failure the broker is restarted as primary with its node ID preserved (so, looping to search for existing lives to pair with or trying to acquire live lock if not found any, in the loop mentioned above).

The new approach is better because node ID is preserved and there's no risk a primary node restart with a backup acting as a live will ever cause a split brain to happen, but still, restarting as a primary while isolated can still cause the start process to last forever, while a failback should be a all-or-nothing mechanism IMO: the primary should try to failback to the existing live and just failing if not, without "restarting", wdyt?

If the primary was restarted whilst isolated, it shouldnt be able to get the lock from ZK....... im clearly missing something.

community call

I'd be interested in attending if a community call on this is held.  Thanks

@mattrpav Happy to hear that let me schedule something for the next week, everyone interested can join :)
@gtully @michaelpearce-gain @jbertram @clebertsuconic are invited too, just need to check my calendar and decide which tech use for the call ;)

@gtully @michaelpearce-gain @clebertsuconic @jbertram @mattrpav The proposal for the community call is for next Wed (09) on 15 UK time: will write here or on slack a link for the call.
Some of the subjects of the call are about:

key differences from "classic" replication process
improvements to handle on this PR or delay as separate enhancements
eg Ideas about journal epoch/versioning: worth doing it now or later? Discuss about issue and solutions
some key decision/discussions on existing behaviours

The last one is very important to move on with this PR, and the behaviour I was thinking are:

classic replication: single pair with pinger vs multiple pairs using quorum -> former deactivate/activate master based on network availability, latter just stop master on cluster connection drop (see https://github.com/apache/activemq-artemis/blob/0b95cb7754e1ae732f1ae9a3ddef657ea13ac90d/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/impl/SharedNothingLiveActivation.java#L263) -> which behaviour to apply on the pluggable quorum vote? impl both and make it configurable?
classic replication: failure during backup failover (including inability to become live due to network split/majority agreement) cause it to restart as a pure backup (it includes the failback case) -> what's I've impl now and what should be better to do instead?

If new things will popup on my head I'm going to drop them here, but first replies if the time/day of the call is fine ;)

@franz1981 thanks, yes this time slot works fine for me, i have reserved it out in my calendar, please let us know dial in link once youve sorted.

@michaelpearce-gain @gtully @jbertram @clebertsuconic @brusdev @mattrpav
10th Wed 15 UK time
The meeting link is: https://meet.google.com/eve-hnos-ikg
The calendar link: https://calendar.google.com/event?action=TEMPLATE&tmeid=Mzg5bGM3MXNmZXVpZ2Jta3VsNjllN3FuNHAgbmlncm8uZnJhQG0&tmsrc=nigro.fra%40gmail.com

hi @mattrpav can I share on youtube our community video? I'm going to link it in a dev Apache post too

hi @mattrpav can I share on youtube our community video? I'm going to link it in a dev Apache post too

Sure thing

To summarize the current status of this PR, re
Comparison with classic replication
Same as classic replication:

fail-fast (JVM-wise) live on quorum loss: retrying to acquire lock as soon as quorum is restored can cause journal misalignments, see https://issues.apache.org/jira/browse/ARTEMIS-3340 - network pinger act differently

Differently from classic replication:

failing-back primary preserve its node ID during journal rotation on activation start
failing-back primary sync with live brokers with its same node ID
fail-fast (JVM-wise) failing-back primary due to unavailable quorum/live still alive: classic replication cause it to restart as backup, loosing its node ID. On restart, it will cause it to activate as live (!) although with a different node ID.

it may make sense to categorise those difference as existing bugs and give them jiras.
on the third one, it looks like a very risky operation, without some coordination and ARTEMIS-3340. It is impossible for the primary to know if the backup has shutdown if it does not get a reply.

@gtully
Good idea about the bug JIRAs!

on the third one, it looks like a very risky operation

You mean the classic behaviour ie restarting as backup while forgetting node ID, or,
You mean the new behaviour ie fail fast while preserving node ID instead of retrying?

It is impossible for the primary to know if the backup has shutdown if it does not get a reply to the shutdown request.

These failures I have mentioned are just while the primary failing-back broker believe to have an up to date journal, is proceeding to fail-over, and failing, because of missing consensus or no majority of nodes around (both classic and new replication).
In the original version, it was "safely" restarting as backup, but forgetting it's node ID:

If a user restart it, it's going to become a live broker with a different node ID (clients can connect to it)

In the new version it's restarted by user, it's preserving its node ID:

If the backup it's still around as live with the same node ID, the primary will search for it and try to failback again
if the backup is not around, it will start as live but with a misaligned journal

My change on new aim to reduce the chance of a misbehaviour, but is not solving completely the misalignment issue, as said.

Still WIP, I'm trying to add a new mechanism to list for unavailable managers and see if It improves backup reliability @gtully

@gtully @michaelandrepearce
Next week I'm going 2 weeks on PTO, so I would like to move this one first.
Today I'm going to add some other bits of doc and smoke tests: other improvements on both doc, tests and changes in the state diagram will be pushed after this.
The failback case has been made as much as possible able to fail-fast, but I see that this should cover several other parts we didn't cover with classic replication too eg

while awaiting the live to appear: is it supposed to wait forever? IMO,nope
while trying to connect to its designated live: it's supposed to retry forever? IMO, nope

These 2 points requires to define the failback operation a bit differently from what was before, but it's something that could be addressed later, given that what I'm already delivering is on par (or better) then classic replication.

@gtully
I've added some docs and this is ready for review and eventually merged, assuming there's still a lot of work to do on this, as the linked JIRAs already explain.

Well why shouldn't it try for ever to connect. In both situations. All you are doing by it not is making it manual intervention which you operationally want to not do. And what is an operator to do. Restart it.... so your not getting any win.

All you are doing by it not is making it manual intervention which you operationally want to not do. And what is an operator to do. Restart it.... so your not getting any win.

Are you talking about primary start loop, now missing, or about the automatic activation/deactivation of primary because of unavailable quorum service?
The latter has been discussed during the community call and the reason why it hasn't been implemented was because of missing journal versioning: without journal versioning there is no point to quickly restart, because it just increase the chance of journal misalignments.
We can safely introduce it later, when the version allow to check if the broker is not just live, but with the most up to date journal.
Right now, it's not suggested to quickly restart a primary without checking who was the last broker alive: the same with classic replication.
I'm not against self-healing but I believe this should be introduced with the right feature to make it reliable, or ca became self-misaligning.

Im confused here because i thought we agreed on the call that versioned journal was needed and the assumption i took there was that would then need to be implemented before we move forwards

During the call we have agreed that I would have raised the JIRAs for the missing features to enhance it and making it more robust in front of journal misalignments scenario, if compared to che classic replication.
The already delivered values here are:

support for single pair too
many fixes on some dark corners of the original state diagram
easy migration path (configuration-wise)
pluggable quorum service

Users can already use what we have here getting same or better value then classic replication.
In the meantime we can:

collect feedbacks on it as it is
design journal versioning as an enhancement
any other improvements we need

Agree about not been in hurry, but given the magnitude of this feature I believe delivering it in chunks/iteratively is the way to go (assuming each chunk to have its value).
I was sure I have explained it during the call
@gtully any comment?

So i thought we had a data state issue and thus a solution was must before we went ahead. As such the need for the journal versioning to address that. Im very cautious about us delivering this to main and to users without a solution to that issue as its asking for production issues

I think what we have here is a very good line in the sand. the problem that a lock version solves for a computer can more easily be solved by a capable human. What we have here is a clear definition of the problem. This is a nice incremental improvement.

And here lies the biggest issue, the human. This is what and why this was something to be solved, because too often than not, the human cannot make the right decision, starts accidentally the wrong node, so forth and so on.
And thus why we had the call to find the solution. Now the agreed solution from call was the journal solution proposed by youself, but we do need something. If we dont think journal solution now work, then we should probably have a follow up call. but i suspect thats not the case here.

this PR is just a core part of the solution, https://issues.apache.org/jira/browse/ARTEMIS-3340 can layer over the versioning.

Indeed, but i think we need both parts before we goto main, id be concerned about this going into main before that as then we could accidentally release just one part and people getting themselves into an issue in prod, without both parts being done, and protecting people from themselves.

@michaelandrepearce I hear you. there is more work to do for sure, but it won't be in any way easier to shoot ones self in the foot with these changes.
It is all about collaboration, it is difficult to get folks to commit to code on a PR or even a feature branch.
The reality is that If it is not on main, it does not exist.
This feature is ripe for collaboration because it pulls in a distributed lock service, getting more users of that would be great to shape the API. The journal exclusive lock is just one use case (all be it an important one).

Right now in testing this, yes i have been building and running it, i have started the wrong node after taking it all down, and thus ended up running the older journal, right now with leader follower (mstr, slv) setups, because of the leader (mstr) concept you always know which node should be taken down last and backup first. And on restart if you do accidentally start slave you're safe, because it wont start till master is there.

If it does go to main, then it should have docs written with very explicitly this is experimental not for production and work in progress.

If it does go to main, then it should have docs written with very explicit statements (big red/orange banner like) that this is experimental not for production and work in progress.

+100 agree
It would ease the pain of using a separate branch chasing main and would ease collaboration IMO (it's just personal).
I would like to add a comment on this one in the doc + a couple of other minor changes ie I would prefix all ZK live locks with namespace/locks/ to prepare for the journal version change (that will likely use something like namespace/longs/ prefixes
@gtully
I believe we should doc somewhere the data misalignments issue of classic replication too (not just the JIRA with the fix using a journal version) and what are the expected mitigation actions to help: the pluggable version will going to see a fix for this, but for classic probably not, hence it worths to help users to be aware of it IMO.

Def want to be able to give artemis zk a base i the tree (configurable) after all many places will share a zk with other stuff or one zk for multiple brokers

Yep, that's already implemented using the "namespace" parameter of the manager, but I would like to already separate locks from "others" (journal timestamps or any other primitives we will need in the future)

@michaelandrepearce @gtully
I hove both added the ZK default  locks path (but I'm playing with ZK to see if it satisfies me) and the warning for users mentioning the upstream JIRA that explain the data misalignment issue.
I'm going to add a doc JIRA to produce some meaningful mitigation actions for replication in general, given that classic replication is affected by this, but is very unlikely that's gonna be fixed.

I have pulled all of the above into a new PR and added in the activation sequencing. I think we can move forward on that.
https://github.com/apache/activemq-artemis/pull/3646

Closing as being superseded by https://github.com/apache/activemq-artemis/pull/3646

authors: franz1981 gtully mattrpav michaelandrepearce michaelpearce-gain

repo: apache/activemq-artemis

license_source: bigquery


text:
[AMBARI-24843] Make Ambaripreupload.py more configurable - fallback to default driver

What changes were proposed in this pull request?
If server.jdbc.driver.path property in ambari.properties points to a non-existent file, the new behavior of Ambaripreupload.py script after #2556 was to exit with error.  This is not fully backward compatible, since before the change the script had ignored the property.
This change makes the script fallback to the previous default driver if the specified driver does not exist, but only if the default one does exist.
How was this patch tested?
Default behavior:
$ Ambaripreupload.py
2018-11-05 12:09:52,624 - Using SQL driver from /var/lib/ambari-server/resources/sqljdbc41.jar
...
2018-11-05 12:09:52,662 - Waiting to read appropriate value of fs.defaultFS from /etc/hadoop/conf/core-site.xml ...
2018-11-05 12:10:02,668 - Waiting to read appropriate value of fs.defaultFS from /etc/hadoop/conf/core-site.xml ...
...

Nonexistent JDBC driver, default driver exists:
$ touch /var/lib/ambari-server/resources/sqljdbc41.jar
$ Ambaripreupload.py -d /usr/share/java/no-such-file.jar
2018-11-05 12:01:53,151 - SQL driver file /usr/share/java/no-such-file.jar does not exist
2018-11-05 12:01:53,151 - Fallback to SQL driver /var/lib/ambari-server/resources/sqljdbc41.jar
2018-11-05 12:01:53,151 - Using SQL driver from /var/lib/ambari-server/resources/sqljdbc41.jar
...

Nonexistent JDBC driver, default driver does not exist:
$ rm -f /var/lib/ambari-server/resources/sqljdbc41.jar
$ Ambaripreupload.py -d /usr/share/java/no-such-file.jar
2018-11-05 12:11:51,417 - SQL driver file /usr/share/java/no-such-file.jar does not exist

JDBC driver taken from ambari.properties:
$ touch /usr/share/java/driver-from-properties.jar
$ echo 'server.jdbc.driver.path=/usr/share/java/driver-from-properties.jar' >> /etc/ambari-server/conf/ambari.properties
$ Ambaripreupload.py
2018-11-05 12:16:48,132 - Using SQL driver from /usr/share/java/driver-from-properties.jar
...

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4457/
Test PASSed.

@g-boros please review

authors: adoroszlai asfgit

repo: apache/ambari


text:
Ambari 9016 branch 2.7

What changes were proposed in this pull request?
Ambari almost always responds with "Content-Type:text/plain" even if the response content itself is a JSON. Also, if Ambari is requested with "Content-Type:application/json" if fails with an exception.
This changes proposes a filter that:

Change the Content-Type of a request to text/plain if the original Content-Type is application/json to prevent Ambari failing on the application/json content.

Change to response's Content-Type to application/json if the original request's Content-Type is application/json

How was this patch tested?
The test is still in progress.
So far the unit tests of ambari-server have been run along with manual testing.
(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)
Please review Ambari Contributing Guide before opening a pull request.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5299/
Test FAILed.
Test FAILured.

@dlysnichenko  @mpapirkovskyy  @aonishuk can any of you pls review?

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5304/
Test FAILed.
Test FAILured.

retest this please

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5306/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5320/
Test PASSed.

authors: asfgit g-boros payert


text:
[Feature]: Support multi-instance AMS REST service

Description
Currently, only the leader AMS would provide REST service. When an AMS leader transition happens, the standby AMS would take 30+ seconds to initialize optimizing, after which the REST service is available. That affects the availability of the REST service.
I proposed to make the REST service independent of the leadership. That's to say, the stand-by AMS instances need to initialize and run REST service on start. Their only difference to a leader instance is no optimizing service.
Use case/motivation
No response
Describe the solution
I drew a diagram of the main modules in AMS.

The green ones are the modules that should be run by both the leader and the followers when starting.
The yellow ones are the leader-only services that should be run when an AMS becomes the leader.
The remaining red one is a new module that syncs catalog changes from the leader through reloading from the database.

Subtasks
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Are you still working on this? I'm interesting in this too. @link3280

Are you still working on this? I'm interesting in this too. @link3280

I'm reconsidering the proposal since @majin1102 plans to support multi-instance AMS. We could align the efforts to avoid duplicate work. What do you think? @baiyangtx

I started a discussion and provided a design.  #2997
You are welcome to participate in the discussion

+1 We also need this part, design can be completed together will reduce duplication of work

The implementation of this issue can be divided into several steps:

Refactor and implement the statelessness of the front-end interfaces. (I have created an issue #3346 and broken it down into multiple subtasks that can be advanced in parallel.)
Implement the sharing of HTTP Session across AMS instances. (There is already a PR #3340 in progress for this project.)
Refactor the AMS startup process to launch the Rest HTTP service before the leader election. (This step depends on the completion of the first two steps.)

repo: apache/amoro


text:
feat: Add support for primary key for external tables.

Which issue does this PR close?

Closes #7154
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Hello @mustafasrepo , I don't know how to add Constraints in protobuf, Could you have some timw to help me ?

Hello @mustafasrepo , I don't know how to add Constraints in protobuf, Could you have some time to help me ? Thanks you!!!

Thanks for working on this issue. I have made a implementation that adds support for protobuf in here on top of your work. You can see the last commit. Basically, In short you have to update datafusion.proto file to add new protobuf types (with its own syntax). Once this file is updated, you have to run bash regen.sh script to automatically generate rust code. All the files under datafusion/proto/src/generated/ are automatically generated with this script. You shouldn't touch these files yourself. However, to add this support I had to make some structs under datafusion_common public. If you can come up with a solution where those structs doesn't have to be public. It would be very nice.

Thanks @mustafasrepo

repo: apache/arrow-datafusion


text:
[Parquet] Recover from panic in file reader

see https://github.com/apache/arrow/pull/43607
original author: @don4get

Fixes apache/arrow#43605.

FYI: We can use Co-authored-by trailer for this case. See also: https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors

authors: kou zeroshade

repo: apache/arrow-go


text:
Fix possibly unaligned writes in MutableBuffer

Which issue does this PR close?

Closes #1410.
Rationale for this change
Changes the push, push_unchecked and extend_from_iter functions so that data is transfered using copy_nonoverlapping which works for unaligned destinations. LLVM will most likely transform this again into a single unaligned write. This also makes these methods actually use the ToByteSlice trait. In extend_from_slice, which this PR did not change, the trait is more used like a marker trait because the whole slice is copied without going through the trait methods. This is probably fine since it's supposed to be an internal trait that is only implemented for ArrowNativeType.
All newly added tests were previously failing when running under miri.

In from_trusted_len_iter the MutableBuffer is created inside the method, so there should be no way for the pointer to become unaligned. Probably still a good idea to adjust it for consistency.

Merged, thanks!

authors: jhorstmann sunchao

repo: apache/arrow-rs


text:
feat: add token provider authorization to azure store

Which issue does this PR close?
Closes #2373
cc @tustvold @alamb
Rationale for this change
Allow a wider range of use by offering more ob the azure native auth methods. While this change might eventually becoem irrelevant, once we move away from SDKs, it is quite helpful in the meantime.
What changes are included in this PR?
Add an option to provide a TokenProvider credential via the MicrosoftAzureBuilder.
Are there any user-facing changes?
users now have more authorization options.

@tustvold - had to convert this back to draft, since the currently released version of the crates do not support identity based credentials just yet ... Maybe I'll wait until this gets released, and do the updates then, or just get it over with and migrate the implementation to a custom one in this crate ...

The latest version of the Azure crates finally got released. Since there were many api changes between the last attempt (#2193) I did not update that PR but included it all here. The relevant update - authorizing via identity - is also part of the latest azure release.
Working on this made me realize once more that we really want to use our own, more focussed implementation of API interactions. I will take a look at that asap. in the meantime i was hoping to get the token auth in the next release, so while a bit finicky in some places, using the SDKs was more realistic to get doe in time (hopefully :)).

There do appear to be failing lints and tests btw

There do appear to be failing lints and tests btw

All tests and lints should be passing now.

Very exciting, btw if you haven't seen #2352 that moves away from rusuto for S3, so the Azure SDKs are the only ones that still remain.

Very exciting indeed - one more reason to get this done and have everything nice and clean :)

authors: roeap tustvold


text:
pyarrow install error

command: pip install  pyarrow
error ：
cmake --build . --config release --
    Scanning dependencies of target lib_pyx
    [ 33%] Compiling Cython CXX source for lib...
    [ 33%] Built target lib_pyx
    Scanning dependencies of target lib
    [ 66%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
    make[2]: *** No rule to make target '/usr/lib/x86_64-linux-gnu/libarrow_python.so', needed by 'release/lib.cpython-37m-x86_64-linux-gnu.so'。 停止。
    CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/lib.dir/all' failed
    make[1]: *** [CMakeFiles/lib.dir/all] Error 2
    Makefile:83: recipe for target 'all' failed
    make: *** [all] Error 2
    error: command 'cmake' failed with exit status 2

How to solve this?

i use python3.6 ,then this  problem is solved

I think that your CMake is old.
Recent CMake ships FindPythonInterp.cmake that supports Python 3.7 or later.

FWIW, I have the latest CMake, but it doesn't work for me on Py 3.7 (and it does again work on Py 3.6 for me too):
    INFOCompiler id: GNU
    Selected compiler gcc 5.4.0
    -- Performing Test CXX_SUPPORTS_SSE3
    -- Performing Test CXX_SUPPORTS_SSE3 - Success
    -- Performing Test CXX_SUPPORTS_ALTIVEC
    -- Performing Test CXX_SUPPORTS_ALTIVEC - Failed
    Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
    -- Build Type: RELEASE
    -- Build output directory: /tmp/pip-install-pauej064/pyarrow/build/temp.linux-x86_64-3.7/release
    -- Found PythonInterp: /home/oberstet/cpy370_5/bin/python3 (found version "3.7")
    -- Searching for Python libs in /home/oberstet/cpy370_5/lib64;/home/oberstet/cpy370_5/lib;/home/oberstet/cpy370/lib/python3.7/config-3.7m-x86_64-linux-gnu
    -- Looking for python3.7m
    -- Found Python lib /home/oberstet/cpy370/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a
    -- Found PythonLibs: /home/oberstet/cpy370/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a
    -- Found NumPy: version "1.15.2" /home/oberstet/cpy370_5/lib/python3.7/site-packages/numpy/core/include
    -- Searching for Python libs in /home/oberstet/cpy370_5/lib64;/home/oberstet/cpy370_5/lib;/home/oberstet/cpy370/lib/python3.7/config-3.7m-x86_64-linux-gnu
    -- Looking for python3.7m
    -- Found Python lib /home/oberstet/cpy370/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a
    -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
    -- Checking for module 'arrow'
    --   Found arrow, version 0.10.0
    -- Arrow SO and ABI version: 10
    -- Arrow full SO version: 10.0.0
    -- Found the Arrow core library: /usr/lib/x86_64-linux-gnu/libarrow.so
    -- Found the Arrow Python library: ARROW_PYTHON_LIB_PATH-NOTFOUND
    Added shared library dependency arrow: /usr/lib/x86_64-linux-gnu/libarrow.so
    Added shared library dependency arrow_python: /usr/lib/x86_64-linux-gnu/libarrow_python.so
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /tmp/pip-install-pauej064/pyarrow/build/temp.linux-x86_64-3.7
    -- Finished cmake for pyarrow
    -- Running cmake --build for pyarrow
    cmake --build . --config release --
    Scanning dependencies of target lib_pyx
    [ 33%] Compiling Cython CXX source for lib...
    [ 33%] Built target lib_pyx
    Scanning dependencies of target lib
    [ 66%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
    make[2]: *** Keine Regel vorhanden, um das Ziel „/usr/lib/x86_64-linux-gnu/libarrow_python.so“,
      benötigt von „release/lib.cpython-37m-x86_64-linux-gnu.so“, zu erstellen.  Schluss.
    CMakeFiles/Makefile2:72: die Regel für Ziel „CMakeFiles/lib.dir/all“ scheiterte
    make[1]: *** [CMakeFiles/lib.dir/all] Fehler 2
    Makefile:83: die Regel für Ziel „all“ scheiterte
    make: *** [all] Fehler 2
    error: command 'cmake' failed with exit status 2

    ----------------------------------------
Command "/home/oberstet/cpy370_5/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-pauej064/pyarrow/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-vh43akes/install-record.txt --single-version-externally-managed --compile --install-headers /home/oberstet/cpy370_5/include/site/python3.7/pyarrow" failed with error code 1 in /tmp/pip-install-pauej064/pyarrow/
(cpy370_5) oberstet@thinkpad-t430s:~/scm/crossbario/zlmdb$ which cmake
/home/oberstet/cmake-3.12.2-Linux-x86_64/bin/cmake
(cpy370_5) oberstet@thinkpad-t430s:~/scm/crossbario/zlmdb$ cmake --version
cmake version 3.12.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
(cpy370_5) oberstet@thinkpad-t430s:~/scm/crossbario/zlmdb$

Well -- Found the Arrow Python library: ARROW_PYTHON_LIB_PATH-NOTFOUND looks wrong, I would start there to see what's not found

authors: fjptwenger kou oberstet wesm

repo: apache/arrow


text:
[Java] new FileSystem is called repeatedly and cause performance degrade

Describe the bug, including details regarding any error messages, version, and platform.
I am integrating Arrow to Spark using jni, it seems FileSystem is created repeatedly and causing performance degrade. like this:

where did the JNI callback to java method ?  maybe this Object should be provided in the interface.
Component(s)
Java

it seems the dataset creation api will always create the filesystem from scratch when calling into libHdfs hdfsBuilderConnect
I'd suggest we create a config to pass that fileSystem in, rather than create it.


text:
[C++][Parquet] release-candidate script should not push to master

This currently introduces the need for a force push if the vote does not pass.
Reporter: Uwe Korn / @xhochy
Assignee: Uwe Korn](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=uwe) / @xhochy
Related issues:

C++: Release parquet-cpp 1.2.0 (blocks)

Note: This issue was originally created as PARQUET-994. Please see the migration documentation for further details.

Uwe Korn / @xhochy:
PR: https://github.com/apache/parquet-cpp/pull/372

Wes McKinney / @wesm:
Issue resolved by pull request 372
https://github.com/apache/parquet-cpp/pull/372


text:
ARROW-15709: [C++] Compilation of ARROW_ENGINE fails if doing an "inline" build

Initial attempt at making something functional to solve this issue properly.
The use of add_arrow_lib in ThirdpartyToolchain.cmake is certainly odd, and I'm sure I'm not following best practices in that file in general. I could use some advice on what the proper way to do this would be. Some of the issues:

The CMake property specifying that a path refers to a generated file is scoped only to the current CMake file, so only moving the externalproject_add over to ThirdpartyToolchain.cmake resulted in the add_arrow_lib in src/arrow/engine failing due to missing source files. An object library didn't seem to resolve it either, and there are probably portability issues with that anyway, so that's why I ended up just using add_arrow_lib.
Unlike all the other third-party dependencies (AFAICT), Substrait can't currently be installed, so Substrait_SOURCE=SYSTEM makes no sense. In the end I just decided to override it, but that's probably not ideal.
Substrait doesn't have releases yet, so I had to resort to a git hash instead.

@kou @pitrou We could use input on this approach.  The Substrait integration is a little unique because the .proto files themselves are not owned by us.  We could fork a copy of the proto files but that would make keeping up with updated versions a bit tricky.
Protobuf is also a bit tricky because the version of the code generation tool (protoc) must match the version of the linked protobuf library exactly (this is different from flatbuffers which doesn't require a runtime dependency).
Previously we were downloading the proto files using git and running protoc within the engine directory.  This had some negative side effects (https://github.com/apache/arrow/pull/12454) mainly due to our use of git.  Even beyond git however it was rather strange that this was resulting in a call to externalproject_add outside of ThirdPartyToolchain.cmake so @jvanstraten and I agreed it might be clear to treat Substrait as a "build-from-source only" third party dependency.  This PR is @jvanstraten 's attempt to make that work but it is also odd as it requires a call to add_arrow_lib inside of ThirdPartyToolchain.cmake.
Any suggestions on a better approach or feedback on our current approach would be very much welcomed.

How about using just an OBJECT library https://cmake.org/cmake/help/latest/command/add_library.html#object-lib instead of arrow_add_lib()?
diff --git a/cpp/cmake_modules/BuildUtils.cmake b/cpp/cmake_modules/BuildUtils.cmake
index 391c43e0ac..3afecfc562 100644
--- a/cpp/cmake_modules/BuildUtils.cmake
+++ b/cpp/cmake_modules/BuildUtils.cmake
@@ -297,6 +297,14 @@ function(ADD_ARROW_LIB LIB_NAME)
       target_precompile_headers(${LIB_NAME}_objlib PRIVATE ${ARG_PRECOMPILED_HEADERS})
     endif()
     set(LIB_DEPS $<TARGET_OBJECTS:${LIB_NAME}_objlib>)
+    # We need to specify $<TARGET_OBJECTS:XXX> explicitly to SHARED
+    # library and STATIC library because $<TARGET_OBJECTS:XXX> in
+    # OBJECT isn't propagated to SHARED library and STATIC library.
+    foreach(ARG_SOURCE ${ARG_SOURCES})
+      if(ARG_SOURCE MATCHES "^\\$<TARGET_OBJECTS:")
+        list(APPEND LIB_DEPS ${ARG_SOURCE})
+      endif()
+    endforeach()
     set(LIB_INCLUDES)
     set(EXTRA_DEPS)

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 8b53272452..9143550df9 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -1687,22 +1687,13 @@ macro(build_substrait)

   add_custom_target(substrait_gen ALL DEPENDS ${SUBSTRAIT_PROTO_GEN_ALL})

-  add_arrow_lib(arrow_substrait
-                SOURCES
-                ${SUBSTRAIT_SOURCES}
-                DEPENDENCIES
-                substrait_gen
-                SHARED_LINK_LIBS
-                ${ARROW_PROTOBUF_LIBPROTOBUF}
-                STATIC_LINK_LIBS
-                ${ARROW_PROTOBUF_LIBPROTOBUF}
-                PRIVATE_INCLUDES
-                ${SUBSTRAIT_CPP_DIR})
-
-  set(SUBSTRAIT_DEPENDENCIES substrait_gen)
-  set(SUBSTRAIT_SHARED arrow_substrait_shared)
-  set(SUBSTRAIT_STATIC arrow_substrait_static)
-  set(SUBSTRAIT_INCLUDES ${SUBSTRAIT_CPP_DIR})
+  add_library(substrait OBJECT ${SUBSTRAIT_SOURCES})
+  set_target_properties(substrait
+    PROPERTIES
+    INCLUDE_DIRECTORIES ${SUBSTRAIT_CPP_DIR}
+    POSITION_INDEPENDENT_CODE ON)
+  add_dependencies(substrait substrait_gen)
+  get_target_property(SUBSTRAIT_INCLUDES substrait INCLUDE_DIRECTORIES)
 endmacro()

 if(ARROW_WITH_SUBSTRAIT)
diff --git a/cpp/src/arrow/engine/CMakeLists.txt b/cpp/src/arrow/engine/CMakeLists.txt
index 50a84464c0..ff88f47e4a 100644
--- a/cpp/src/arrow/engine/CMakeLists.txt
+++ b/cpp/src/arrow/engine/CMakeLists.txt
@@ -36,9 +36,10 @@ add_arrow_lib(arrow_engine
               OUTPUTS
               ARROW_ENGINE_LIBRARIES
               DEPENDENCIES
-              ${SUBSTRAIT_DEPENDENCIES}
+              substrait
               SOURCES
               ${ARROW_ENGINE_SRCS}
+              $<TARGET_OBJECTS:substrait>
               PRECOMPILED_HEADERS
               "$<$<COMPILE_LANGUAGE:CXX>:arrow/engine/pch.h>"
               SHARED_LINK_FLAGS
@@ -46,11 +47,9 @@ add_arrow_lib(arrow_engine
               SHARED_LINK_LIBS
               arrow_shared
               arrow_dataset_shared
-              ${SUBSTRAIT_SHARED}
               STATIC_LINK_LIBS
               arrow_static
               arrow_dataset_static
-              ${SUBSTRAIT_STATIC}
               PRIVATE_INCLUDES
               ${SUBSTRAIT_INCLUDES})

@kou Thanks for the input! It's certainly a lot nicer to do it that way, but unfortunately, we're not quite there yet... I got it working in most build environments using your diff with some minor changes (had to add protobuf to the include list for the object library, and add its runtime library to arrow_engine), but it fails here:
CMake Error at cmake_modules/BuildUtils.cmake:287 (add_library):
  OBJECT library "arrow_engine_objlib" contains:

    capabilities.pb.cc.o
    expression.pb.cc.o
    extensions.pb.cc.o
    function.pb.cc.o
    parameterized_types.pb.cc.o
    plan.pb.cc.o
    relations.pb.cc.o
    type.pb.cc.o
    type_expressions.pb.cc.o

  but may contain only sources that compile, header files, and other files
  that would not affect linking of a normal library.

I feel like the addition to BuildUtils.cmake is either causing this or supposed to resolve this (and failing to do so in this particular case), but other than that, my skill level for CMake is insufficient to see what exactly is going on here.

Failure of AMD64 Ubuntu 20.04 GLib & Ruby for
fffa65c seems to be unrelated (zlib download failure, I think). It builds correctly when I run it with archery locally.

@github-actions crossbow submit python-sdist

@kou @pitrou @westonpace I believe this is ready to be merged, or am I still missing something?

Perhaps @kou can also double-check the CMake changes.

authors: jvanstraten kou pitrou westonpace


text:
ARROW-17620: [R] as_arrow_array() ignores type for StructArrays

As described in https://issues.apache.org/jira/browse/ARROW-17620, as_arrow_array() ignores user-provided types passed by the type argument.
This PR explicitly adds the type name argument to the Map call. Currently, the argument gets passed to the ellipsis and gets ignored.
I also added a test. Let me know if there is a better place for it (or a better test).

With this fix, 2 tests were failing. Previously, the type argument was ignored, making the tests pass even though the tests were using the incorrect dataset. 07e75ebd917287e575f74a807e428f8a16e76821 fixes that.


text:
GH-36284: [Python][Parquet] Support write page index in Python API

Rationale for this change

Support write_page_index in Parquet Python API
What changes are included in this PR?

support write_page_index in properties
Are these changes tested?

Currently not
Are there any user-facing changes?
User can generate page index here.

Closes: #36284

@jorisvandenbossche Mind take a look? I'm not so familiar with Python part, so maybe make something wrong

@pitrou @westonpace Would you mind take a look? This patch support Python to write page_index.

Can this patch be merged? Or should I wait for other committers review?

@github-actions crossbow submit -g python

test-conda-python-3.10-spark-master
test-cuda-python
test-conda-python-3.8-spark-v3.1.2
test-conda-python-3.10-spark-master

These cases failed, how can I try to fix them?

@mapleFU Those are unrelated to this PR. Can you try to rebase?

test-conda-python-3.10-spark-master
test-conda-python-3.8-spark-v3.1.2
test-conda-python-3.9-spark-v3.2.0

Spark failures are known and have an issue opened.

test-conda-python-3.11-hypothesis

Hypothesis failure is a new one but I do not see how it could be related to this PR.

test-cuda-python

I have seen nightlies fail with this error today already, so this is not related to the PR either.

Hypothesis failure is a new one but I do not see how it could be related to this PR.

Hmm, that seems very similar to the one that I fixed last week (https://github.com/apache/arrow/issues/36349, but now with another unknown timezone). In any case, you can ignore it here.

@github-actions crossbow submit -g python

Still these failed, lol

@pitrou @jorisvandenbossche I've tried to fix the comment here. Would you mind take a look?

@github-actions crossbow submit -g python

authors: AlenkaF jorisvandenbossche mapleFU pitrou


text:
GH-40040: [C++][Gandiva] Make Gandiva's default cache size to be 500000 for object code cache

Rationale for this change
Gandiva's default cache is object code cache, however, the default cache size is still the old value for LLVM module based cache, which is too small.
More details about the GANDIVA_ENABLE_OBJECT_CODE_CACHE flag can be found in GH-40040
What changes are included in this PR?
Remove the unused GANDIVA_ENABLE_OBJECT_CODE_CACHE flag and make the default cache size to be 500000 for object code cache.
Are these changes tested?
No
Are there any user-facing changes?
Yes, default cache size will be changed from 500 to 500000, and it may help the default deployment's performance.

Closes: #40040

cc @js8544

Happy Chinese New Year!
@pitrou I think this PR is to fix the regression introduced in https://github.com/apache/arrow/pull/12742 rather than a feature. #12742 sets GANDIVA_ENABLE_OBJECT_CODE_CACHE default to true but forget to change the default cache size. @niyue Is my understanding correct?
However I do agree that we can further investigate the size limit of the cache. 500000 objects should require too much memory. Suppose one expression takes 1KB (my wild guess), that would be 500MB.

Yes, 5000 would be much better here.
Le 13 février 2024 11:45:25 GMT+01:00, Yue @.***> a écrit :

@niyue commented on this pull request.

@@ -23,11 +23,7 @@

namespace gandiva {
-#ifdef GANDIVA_ENABLE_OBJECT_CODE_CACHE
static const size_t DEFAULT_CACHE_SIZE = 500000;
Indeed, the unit in question pertains to the number of entries rather than bytes, but there's no misunderstanding of the cache size and the original value of 500 was also defined in terms of the number of entries. This PR aims to address an oversight by removing a previously missed flag.
This link [1] has more data about module cache vs. object code cache in Gandiva, and in the limited expressions tested, the memory is down to 0.8%~6% after using object code cache.
The current default value 500 is likely too small in production, and it will probably only use 8MB of memory using the stats in link [1], but you are right it seems 500000 is indeed too large (I have no idea how it was defined this way and I should do more calculation for it). According to the stats in link [1], when using module cache, it will probably take 750MB memory for 500 entries.
Other initiatives [2] have attempted to enhance the cache eviction algorithm, but such attempts were reversed due to other issues [3]. I've previously reviewed these efforts and believe I have distinct ideas for advancing Gandiva's cache. I plan to propose a PR if I fully understand the workflow. To avoid complicating this PR, my goal is solely to refine the default value. Would it be acceptable to adopt a more conservative default value, such as 5000/10000?
[1] https://github.com/apache/arrow/pull/11193#issue-1001547667
[2] https://github.com/apache/arrow/pull/10465
[3] https://github.com/apache/arrow/pull/11957
--
Reply to this email directly or view it on GitHub:
https://github.com/apache/arrow/pull/40041#discussion_r1487603910
You are receiving this because you were mentioned.
Message ID: @.***>

https://github.com/apache/arrow/pull/12742 sets GANDIVA_ENABLE_OBJECT_CODE_CACHE default to true but forget to change the default cache size. @niyue Is my understanding correct?
Happy Chinese New Year!
That's correct. I didn't explicitly mentioned this in this PR description, but it is described in https://github.com/apache/arrow/issues/40040 issue description.

@js8544 @pitrou
Thanks for the review, and I updated the default value and the PR title to be 5000 now.

Thank you @niyue !

authors: js8544 niyue pitrou


text:
ARROW-10631: [Rust] Fixed error in computing equality of fixed-sized binary.

This fixes a bug on the equality for fixed-sized binary arrays. For these types, the number of bytes of a slot equals the specified size. However, we were not taking that into account when offsetting in number of slots in the comparison.

LGTM. Logic now matches primitive_equal which also has to depend on the size of the type.

authors: jhorstmann jorgecarleitao


text:
fix csharp build error

add app.config to Avro.Pref project
remove '-framework' parameter from nunit-console test command.

Please review it.

@rdblue This looks good to me. I had to make these changes as well to get it to build properly.

Merged. Thanks @Naruto for the contribution and @jcustenborder for reviewing!

authors: Naruto jcustenborder rdblue

repo: apache/avro


text:
AVRO-2335: Remove Joda Time library

Make sure you have checked all steps below.
Removes the Joda time library and makes Java JSR310 dates the default.
Jira

[ ] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-XXX
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

+1

authors: Fokko nandorKollar


text:
[Bug]: In Tour of Beam the code snippets are not selectable.

What happened?
In Tour of Beam, the code snippets in the left hand side orange panes are not selectable.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[X] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

Also consider introducing a separate component or tag for TOB issues, and pre-select issues with this tag, when forwarding users to GitHub to file an issue. This would make it easy to see if similar issues have already been reported.

Also looks like that when selecting the text outside code snippets, there are some rendering artifacts, and text is selected only within a paragraph.

.take-issue

authors: nausharipov tvalentyn

repo: apache/beam


text:
[BEAM-2535] Support outputTimestamp and watermark holds in processing timers.

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

retest this please

Run Java PreCommit

retest this please

Run Java PreCommit

run dataflow validatesrunner

run flink validatesrunner

Pinging to trigger tests.

Run Python2_PVR_Flink PreCommit

authors: aaltay iemejia kennknowles rehmanmuradali reuvenlax


text:
[BEAM-10486] Update OffsetRestrictionTracker to honor failed claim attempt marking the claim as done when checkpointing.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @boyuanzz

Run Python PreCommit

Run Python2_PVR_Flink PreCommit

Run Python PreCommit


text:
Update Dataflow worker harness container image version.

Update Dataflow worker harness image to latest published version.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Dataflow ValidatesRunner

Run Dataflow ValidatesRunner

Run Dataflow ValidatesRunner Java 11

Run Dataflow ValidatesRunner Java 11

Run Java examples on Dataflow Java 11

Run Java examples on Dataflow Java 11

R: @kennknowles
R: @emilymye

R: @kennknowles
R: @emilymye

I've verified that the images exist in gcr global but didn't find any log messages to confirm that it is being used in the tests I ran.

I've verified that the images exist in gcr global but didn't find any log messages to confirm that it is being used in the tests I ran.


text:
[BEAM-14506] Adding testcases and examples for xlang Python RunInference

./gradlew :sdks:python:test-suites:direct:xlang:validatesCrossLanguageRunnerJavaUsingPython --tests RunInferenceTransformTest

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Java PostCommit

Run Java PreCommit

Run XVR_Direct PostCommit

Run XVR_Direct PostCommit

Run XVR_Flink PostCommit

Run XVR_Direct PostCommit

Run XVR_Direct PostCommit

Run XVR_Direct PostCommit

https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct_PR/126

Run SQL PreCommit

Run Java PreCommit

R: @robertwb

Run SQL PreCommit

Run Python PreCommit

Run Java PreCommit

Run SQL PreCommit

Run Java PreCommit

Run Portable_Python PreCommit

Run Python PreCommit

Run PythonDocs PreCommit

Run PythonDocs PreCommit

Run Portable_Python PreCommit

Run Python PreCommit

Run Portable_Python PreCommit

Run Portable_Python PreCommit

Run PythonDocs PreCommit

authors: ihji pabloem


text:
Ensure Java JMH benchmark tasks run sequentially

Ensure Java JMH benchmark tasks run sequentially to prevent failure when acquiring lock (addresses #22238)
Exception in thread "main" org.openjdk.jmh.runner.RunnerException: ERROR: Another JMH instance might be running. Unable to acquire the JMH lock (/tmp/jmh.lock), exiting. Use -Djmh.ignoreLock=true to forcefully continue.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @aromanenko-dev


text:
[Do not merge] Test vendor guava 32.1.2

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

https://github.com/google/guava/commit/b1350a6a99b9164a6cdc15f177cbda50fa1baf3d makes the value returned from ImmutableMap nullable, breaking checkerframeworks in various places

Run Java_IOs_Direct PreCommit

Run Java_PVR_Flink_Batch PreCommit

artifact released, closing this PR


text:
Fix flaky test StreamingDataflowWorkerTest

Use MockClock instead of real system clock to fix the test flakiness.

Thank you, @olalamichelle !

Interestingly, after merging this test worked well for a while became highly flaky since Sept 8th: #28383

authors: Abacn olalamichelle tvalentyn


text:
Cherry-pick of #3149 into release-2.0.0

Cherry-pick of #3149 readAvros should't have proto Message upper bound
R: @davorbonaci

This doesn't meet the bar at this stage, right? It seems to affect PubsubIO only, and only for certain types of messages.

Without this change, PubsubIO.readAvros() is unusable (it was untested too - my bad) because most classes used with AvroIO won't be protobufs. It might be ok - users have the workaround of manually doing what it does. But on the other hand this change is fully safe. Your call, I'm fine either way.

Understood. I say -1, unless there are other reasons why the RC must be rebuilt. At that point, it will change to +1.

Changes Unknown when pulling 1f8b36b3993a069f97c78757b025cdc8d983b1b6 on jkff:read-avros into ** on apache:release-2.0.0**.

authors: coveralls davorbonaci jkff


text:
[BEAM-6113] Small changes to avoid user errors around beam.Init()

Making some small changes that will help prevent users from forgetting
or not knowing about calling beam.Init(). Making sure all examples
include a call to beam.Init(), and that the direct runner (which
currently runs if beam.Init() is not called) contains a warning about
it.

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

R: @lostluck, @aaltay


text:
Document what parts of BuildStream / plugins depend on host packages

See original issue on GitLab
In GitLab by [Gitlab user @jjardon] on Apr 10, 2018, 15:22
AFAIK, we depend on

host's ssh to push to the bst cache
host's bubblewrap for sandboxing
host's ostree for caching artifacts

Also, this plugins:

patch depends on host's patch
tar depends on host's lzip
git depends on host's git
bzr depends on host's bzr

In GitLab by [Gitlab user @jjardon] on Apr 12, 2018, 18:15
changed the description

In GitLab by [Gitlab user @jjardon] on Apr 12, 2018, 18:20
changed the description

In GitLab by [Gitlab user @jjardon] on Apr 12, 2018, 18:21
changed the description

In GitLab by [Gitlab user @jjardon] on Apr 17, 2018, 11:11
changed the description

In GitLab by [Gitlab user @jjardon] on Apr 17, 2018, 11:13
changed the description

In GitLab by [Gitlab user @jjardon] on Apr 17, 2018, 14:52
mentioned in merge request !406

In GitLab by [Gitlab user @jjardon] on Jun 10, 2018, 21:23
mentioned in merge request !495

In GitLab by [Gitlab user @jjardon] on Jun 11, 2018, 23:54
mentioned in commit f70ca7a3e3f210267937b6604399001f01fb4784

In GitLab by [Gitlab user @jjardon] on Jun 11, 2018, 23:54
mentioned in commit b1b1f64a7dd843ef03c574bce98a60f794e8a654

In GitLab by [Gitlab user @jjardon] on Jun 11, 2018, 23:54
mentioned in commit 0bed777b3e1af9b69f7a955cf8a006e320d6d899

In GitLab by [Gitlab user @jjardon] on Jun 11, 2018, 23:54
mentioned in commit 1e17ab8fbf6ae1cd3d49413327f3cb60ecea3455

In GitLab by [Gitlab user @jjardon] on Jun 11, 2018, 23:54
mentioned in commit 0ff544fed28b2481ddb3038024080ff761864f57

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:44
mentioned in commit 0e4dad21e7d8709f7c19e0731ead3b7013f2cb6f

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:44
mentioned in commit 1aa6e08277fbe57389ad4a558c5cd8f0985d6856

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:44
mentioned in commit 9f2f4e2c909a184fda3fb2099c7032e8c480b9fd

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:44
mentioned in commit d0a333995df93130da96ee8ad12e5147e829eaf2

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:44
mentioned in commit bbe718748fe5cfe00c441abd1dec6af5c14a1d34

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:46
mentioned in commit bbb17b6f168f2b84da1220584747dba77ba002bd

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:46
mentioned in commit fea856dd8ad17baf478dd6818e922328d5b7ccf2

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:46
mentioned in commit 2c94faf827a94357b467334c6e007322fd74e45c

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:46
mentioned in commit 0396d804f8698179e07aac8c67e8b4d6fae4e4e8

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:46
mentioned in commit fc103f0a1ae78626a3a11ba038dd7b84ef1841a7

In GitLab by [Gitlab user @jjardon] on Jun 21, 2018, 19:47
changed the description

In GitLab by [Gitlab user @tristanvb] on Jun 22, 2018, 18:00
changed title from Document what parts of Build{-s-}tream / plugins depend on host packages to Document what parts of Build{+S+}tream / plugins depend on host packages

In GitLab by [Gitlab user @jjardon] on Jun 22, 2018, 18:09
mentioned in commit e90098e0e0decbd40f8e1980772d921e7843b8ab

In GitLab by [Gitlab user @jjardon] on Jun 22, 2018, 18:09
mentioned in commit b5d917940d79b859c7ebab0d25f512a805647b25

In GitLab by [Gitlab user @jjardon] on Jun 22, 2018, 18:09
mentioned in commit 3b1e869bd8c64750a2e725a523acd55dc8e94360

In GitLab by [Gitlab user @jjardon] on Jun 22, 2018, 18:09
mentioned in commit be92cc8d7ff2dafee1a15b07a43802be01d5ad5a

In GitLab by [Gitlab user @jjardon] on Jun 22, 2018, 18:09
mentioned in commit 7d97c6d3f3d59cf078df3bedd178e7977868d9d5

In GitLab by [Gitlab user @tristanvb] on Jun 23, 2018, 01:08
This is closed by !495 (landed as a part of !516).

In GitLab by [Gitlab user @tristanvb] on Jun 23, 2018, 01:08
closed

repo: apache/buildstream


text:
bzr source plugin can introduce non-determinism

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:53
Summary
When staging the bzr checkout at Source.stage() time, we are not making any effort to ensure that the staged files have deterministic file attributes.
As far as I can see, similar to issue #527 and more similar to #543, the checkout will result in file permissions which depend on the user's umask.

In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:53
marked this issue as related to #527

In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:53
marked this issue as related to #540

In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:53
marked this issue as related to #541

In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:53
marked this issue as related to #542

In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:53
marked this issue as related to #543

In GitLab by [Gitlab user @tristanvb] on Aug 2, 2018, 13:57
marked this issue as related to #545

In GitLab by [Gitlab user @toscalix] on Aug 6, 2018, 09:51
assigned to [Gitlab user @valentindavid]

In GitLab by [Gitlab user @valentindavid] on Aug 6, 2018, 10:42
marked this issue as related to #555

In GitLab by [Gitlab user @valentindavid] on Aug 6, 2018, 17:09
mentioned in commit 4d142206bdba11bcef52e54c8c2ea4e1ad525d03

In GitLab by [Gitlab user @valentindavid] on Aug 7, 2018, 17:33
mentioned in commit 4d49d8c0a2ad3e94f44080bfe55cc853dd22198e

In GitLab by [Gitlab user @valentindavid] on Aug 9, 2018, 15:18
mentioned in commit 298fa0483ec407e0cc54c216eefcbeaf328d7f0a

In GitLab by [Gitlab user @valentindavid] on Aug 10, 2018, 09:53
mentioned in commit c5362746c10efc734fe65d5978d8c1ad47b3f0d9

In GitLab by [Gitlab user @valentindavid] on Aug 10, 2018, 11:08
mentioned in commit 9daa8eace89f750a227e38b59137e6861c256dd1

In GitLab by [Gitlab user @valentindavid] on Aug 10, 2018, 14:50
mentioned in commit 8757117c2fe3abd89e9682909587657125e6a10d

In GitLab by [Gitlab user @valentindavid] on Aug 10, 2018, 15:22
mentioned in commit 1b9c9277e92f57d4ec2bf510eaf00d45bf33ee12

In GitLab by [Gitlab user @valentindavid] on Aug 12, 2018, 06:54
mentioned in commit 322cab58e255916da10e0a59ae91e92f50898504

In GitLab by [Gitlab user @valentindavid] on Aug 12, 2018, 06:57
mentioned in commit 465909ddc6d39822f7669d554f3033e497b6f762

In GitLab by [Gitlab user @tristanvb] on Aug 12, 2018, 07:59
closed via merge request !616

In GitLab by [Gitlab user @valentindavid] on Aug 12, 2018, 07:59
closed via commit 322cab58e255916da10e0a59ae91e92f50898504

In GitLab by [Gitlab user @tristanvb] on Aug 12, 2018, 07:59
mentioned in commit c1fdebb3137891305bea7db8b4f1ad9f0b1e2043


text:
flatpak-autotools example broken

See original issue on GitLab
In GitLab by [Gitlab user @finnball] on Oct 2, 2018, 14:42
Summary
Tried following this example:
https://buildstream.gitlab.io/buildstream/examples/flatpak-autotools.html
Steps to reproduce
Run the above example on debian
What is the current bug behavior?
buildgrid[[Gitlab user @client-vm]](https://gitlab.com/client-vm):~/buildstream/doc/examples/flatpak-autotools$ bst build hello.bst
[--:--:--][][] START   Build
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[--:--:--][][] BUG     'Repo' object has no attribute 'reload_config'

    Traceback (most recent call last):
      File "/home/buildgrid/.local/bin/bst", line 8, in <module>
        sys.exit(cli())
      File "/home/buildgrid/.local/lib/python3.5/site-packages/click/core.py", line 764, in __call__
        return self.main(*args, **kwargs)
      File "/home/buildgrid/buildstream/buildstream/_frontend/cli.py", line 162, in override_main
        standalone_mode=standalone_mode, **extra)
      File "/home/buildgrid/.local/lib/python3.5/site-packages/click/core.py", line 717, in main
        rv = self.invoke(ctx)
      File "/home/buildgrid/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/buildgrid/.local/lib/python3.5/site-packages/click/core.py", line 956, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/buildgrid/.local/lib/python3.5/site-packages/click/core.py", line 555, in invoke
        return callback(*args, **kwargs)
      File "/home/buildgrid/.local/lib/python3.5/site-packages/click/decorators.py", line 27, in new_func
        return f(get_current_context().obj, *args, **kwargs)
      File "/home/buildgrid/buildstream/buildstream/_frontend/cli.py", line 330, in build
        build_all=all_)
      File "/home/buildgrid/buildstream/buildstream/_stream.py", line 182, in build
        dynamic_plan=True)
      File "/home/buildgrid/buildstream/buildstream/_stream.py", line 931, in _load
        self._pipeline.resolve_elements(elements)
      File "/home/buildgrid/buildstream/buildstream/_pipeline.py", line 139, in resolve_elements
        element._update_state()
      File "/home/buildgrid/buildstream/buildstream/element.py", line 1066, in _update_state
        self.__update_source_state()
      File "/home/buildgrid/buildstream/buildstream/element.py", line 2008, in __update_source_state
        source._update_state()
      File "/home/buildgrid/buildstream/buildstream/source.py", line 650, in _update_state
        self.__consistency = self.get_consistency()
      File "/home/buildgrid/buildstream/buildstream/plugins/sources/ostree.py", line 169, in get_consistency
        self.ensure()
      File "/home/buildgrid/buildstream/buildstream/plugins/sources/ostree.py", line 181, in ensure
        self.repo = _ostree.ensure(self.mirror, True)
      File "/home/buildgrid/buildstream/buildstream/_ostree.py", line 66, in ensure
        repo.reload_config()
    AttributeError: 'Repo' object has no attribute 'reload_config'

What is the expected correct behavior?
It should work.

In GitLab by [Gitlab user @finnball] on Oct 2, 2018, 14:57
closed

In GitLab by [Gitlab user @traveltissues] on Nov 14, 2019, 12:55
https://gitlab.com/BuildStream/buildstream/pipelines/95903565 this element is continuing to have some issues
the problem seems to be in the sdk.bst since sdk.gnome.org is offline

In GitLab by [Gitlab user @traveltissues] on Nov 14, 2019, 12:55
reopened

In GitLab by [Gitlab user @abderrahimk] on Nov 19, 2019, 19:23
mentioned in merge request !1719

In GitLab by [Gitlab user @jjardon] on Nov 19, 2019, 22:00
closed via merge request !1719

In GitLab by [Gitlab user @jjardon] on Nov 19, 2019, 22:00
mentioned in commit 9bf0eae1cb887f1bb214b72e11f1212b6a8bc2c4