text:
Fate Admin print doesn't filter on status

The status filter on the new Admin fate command does not change the results:

04:26:21 {main} ~/workspace/uno/install/logs/accumulo$ accumulo admin fate -p -s IN_PROGRESS
2022-09-27T16:26:29,795 [conf.SiteConfiguration] INFO : Found Accumulo configuration on classpath at /home/mpmill4/workspace/uno/install/accumulo-2.1.0-SNAPSHOT/conf/accumulo.properties
2022-09-27T16:26:31,085 [metrics.MetricsUtil] INFO : Metric producer PropStoreMetrics initialize
TABLE_BULK_IMPORT2 txid: 6324179035588914  status: SUBMITTED          locked: []              locking: [R:3]           op: PrepBulkImport  created: 2022-09-27T20:25:40.716Z
TABLE_COMPACT   txid: 4d7bf30c0676eb5d  status: SUBMITTED          locked: [R:+default]    locking: [R:3]           op: CompactRange    created: 2022-09-27T20:26:09.732Z
TABLE_BULK_IMPORT2 txid: 0176dfaebcec9591  status: SUBMITTED          locked: []              locking: [R:3]           op: PrepBulkImport  created: 2022-09-27T20:26:10.648Z
TABLE_MERGE     txid: 1eeba54e6e769e7c  status: IN_PROGRESS        locked: [R:+default, W:3] locking: []              op: TableRangeOpWait created: 2022-09-27T20:25:39.58Z
TABLE_COMPACT   txid: 09760433eb43a5ce  status: IN_PROGRESS        locked: [R:+default, R:1] locking: []              op: CompactionDriver created: 2022-09-27T20:26:00.977Z
TABLE_MERGE     txid: 37d000bf5948ab30  status: SUBMITTED          locked: [R:+default]    locking: [W:1]           op: TableRangeOp    created: 2022-09-27T20:26:25.915Z

I think we should also make the options the same. For example, Admin uses fate -p -s IN_PROGRESS while the Shell command uses -t for filtering on status Type.

make the options the same. For example, Admin uses fate -p -s IN_PROGRESS while the Shell command uses -t for filtering on status Type.

which option do you think is better?

Yeah I like -s better, like Ed said. So we can just change that one. But this ticket was more for making sure the filter actually works.

It looks like the filter is not getting picked up properly and for some reason the value is null by the time the filtering actually occurs.

It looks like the filter is not getting picked up properly and for some reason the value is null by the time the filtering actually occurs.

Right. I have a simple fix ready to go but was waiting on #2976 since that is also breaking the command.

Do we really need this? I feel like a user can just use grep themselves, if each item is already printed on separate lines.

Do we really need this? I feel like a user can just use grep themselves, if each item is already printed on separate lines.

Not really. Now that its in Admin you can easily pipe it to grep. The only thing that is kinda nice is that the description also prints all the possible types:

          -s, --state
            [ -s ...] Print transactions in the state(s) {NEW,
            IN_PROGRESS, FAILED_IN_PROGRESS, FAILED, SUCCESSFUL}
            Default: []

authors: DomGarguilo ctubbsii milleruntime

repo: apache/accumulo


text:
ARTEMIS-2482 Large messages could leak native ByteBuffers

Perform chunked read/write of large message files to save
NIO from leaking native ByBuffers:
see https://bugs.openjdk.java.net/browse/JDK-8147468

@franz1981 do we have any other reading operation that would lead the same?
Can you also change our scripts to include the property to avoid caching if anything like that is ever used?

On paging we cache manually so is a different "issue" and probably for many user is fine.
Re the property I'm not sure: imo is a good idea but it will impact existing users where it is happening without some; they will start to perceive slower performance in a stealthy way..
Anyway I don't have a strong opinion on that so will do it if you prefer (just need to verify if is present in jdk 11 as well)

We shouldn't be using native cache anyways... just in case it happens in our codebase, i would rather have a limit.

@franz1981 can you add labels on github? use the DO-NOT-MERGE-YET?

@clebertsuconic seems that I cannot add the label...please check the code, because I've found another part on replication where it could happen...

@wy96f please take a look that you've recently have fun with ByteBufs :)

@wy96f

PooledByteBufAllocator::directBuffer uses java's ThreadLocal. Wouldn't this result in leak in the case if thread terminates due to idle timeout as in #2199 ?

Do you mean the thread local arena? Or the thread local NIO ByteBuffer?
Anyway, at the end of its usage we always release the ByteBuf and Netty will take care to release any referenced resources: it's a pool so it leaks "by definition" :+1:

@wy96f

PooledByteBufAllocator::directBuffer uses java's ThreadLocal. Wouldn't this result in leak in the case if thread terminates due to idle timeout as in #2199 ?

Do you mean the thread local arena? Or the thread local NIO ByteBuffer?
Anyway, at the end of its usage we always release the ByteBuf and Netty will take care to release any referenced resources: it's a pool so it leaks "by definition"

I mean tinySubPageDirectCaches/smallSubPageDirectCaches/normalDirectCaches in PoolThreadCache. PoolChunk would be added into these caches first if released(not yet into shared arena). Given PoolThreadCache is thread local, would theses caches leak if thread terminates?

@wy96f Yes and no :)
The would leak (at first look, but let me take a better look), but they would contain just wrappers to direct memory that cannot be released (is part of the pool) and no cleaner (so they won't collect anything, when contains ByteBuffers). The impact should be way less then IOUtil that pool native ByteBuffers holding exclusively native memory that won't be reused anymore...

@wy96f I've taken a second look to the way Netty handle thread local pools and I can confirm that -Dio.netty.allocator.useCacheForAllThreads=false should save non-netty threads from creating such thread local regions for tiny/small/medium sized buffers.
While related to

but they just contain wrappers to direct memory that cannot be released (is part of the pool and they have no cleaner). The impact should be way less then IOUtil that pool native ByteBuffers holding exclusively native memory that won't be reused anymore...

I have to add more detail and I was wrong: the tiny/small/medium thread local caches are actually leaking memory regions (with default io.netty.allocator.useCacheForAllThread) that need the holding thread to die in order to make the available to other threads, but their size is quite small by default and the size I've chosen as LARGE_MESSAGE_CHUNK_SIZE (ie 100 * 1024) is medium but doesn't fall in any of those caches, so no real leak will happen until we allocate something smaller (could happen but is rare).
IMO we should consider (for a separate issue) 2 separate improvements:

allow threads to stay alive forever, but just idle (and maybe using the FJ thread pool executor instead of the AMQ thread pool executor in that case): in that case we can consider such caches to be ok to be used by all threads (including non-netty ones), because the leak is just to cope with future load
for the AMQ thread pool executor as it is, make the presence of such caches configurable and off by default

wdyt?
Thanks for the comment, good catch!

@franz1981

I have to add more detail and I was wrong: the tiny/small/medium thread local caches are actually leaking memory regions (with default io.netty.allocator.useCacheForAllThread) that need the holding thread to die in order to make them available to other threads again

I assumed memory released into tiny/small/medium cache would be leaked if holding thread terminates until I looked at the code in PoolThreadCache:
/// TODO: In the future when we move to Java9+ we should use java.lang.ref.Cleaner.
    @Override
    protected void finalize() throws Throwable {
        try {
            super.finalize();
        } finally {
            free();
        }
    }

    /**
     *  Should be called if the Thread that uses this cache is about to exist to release resources out of the cache
     */
    void free() {
        // As free() may be called either by the finalizer or by FastThreadLocal.onRemoval(...) we need to ensure
        // we only call this one time.
        if (freed.compareAndSet(false, true)) {
            int numFreed = free(tinySubPageDirectCaches) +
                    free(smallSubPageDirectCaches) +
                    free(normalDirectCaches) +
                    free(tinySubPageHeapCaches) +
                    free(smallSubPageHeapCaches) +
                    free(normalHeapCaches);

            if (numFreed > 0 && logger.isDebugEnabled()) {
                logger.debug("Freed {} thread-local buffer(s) from thread: {}", numFreed,
                        Thread.currentThread().getName());
            }

            if (directArena != null) {
                directArena.numThreadCaches.getAndDecrement();
            }

            if (heapArena != null) {
                heapArena.numThreadCaches.getAndDecrement();
            }
        }
    }

So the cache will be freed and used by other threads when thread dies? If so, it seems no any leak problems with this pr, no need with improvements?

@wy96f

So the cache will be freed and used by other threads when thread dies?

Yes, it is, but my concern is that it needs a finalization run before getting there ie not that deterministic as I wish :)

As you said, size of tiny/small/medium caches is quite small. Even if not freed in time, the impact is very little compared to IOUtil that pool native ByteBuffers(default TEMP_BUF_POOL_SIZE=8) for each thread :)

@wy96f fair enough :)
Still waiting some results from the integration tests on the CI and will be ready to be merged, although I prefer to improve code quality of this PR, there are some changes due to JDBC that makes is uglier than I wished

@clebertsuconic @michaelandrepearce Thanks to the work of @wy96f on replication, I've taken a further look into https://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html#getChannel() and I think this issue can be solved much more easily by using RandomAccessFile when necessary (to write/read byte[] without leaks :))
I will provide soon a PR for it, but I can close this one, that is very complex and uneasy to be maintained

source: gharchive/pull-request

repo: apache/activemq-artemis

license_type: permissive

license_source: bigquery


text:
ARTEMIS-770 AMQP Message Transformer refactor

Refactor the AMQP Message transformers both for better performance and
also to fix a number of issues with the transformers creating inbound
and outbound messages with incorrectly mapped values or extra data
appended where it should not be.

There are a few regressions under org.apache.activemq.artemis.tests.integration.amqp.*
AmqpTransactionTest::testReceiversCommitAndRollbackWithMultipleSessionsInSingleTX
and
ProtonPubTest has a few failing.
You think you could look into those and amend the commit?
But overall it looks great!

Updated code fixes the test failures, at least the one's that aren't already failing on master.


text:
AMQ-8398 - Fix Stomp to OpenWire UTF-8 translation

This commit fixes conversion of messages that are sent or received using Stomp when the body contains characters that require 4 bytes for encoding using standard UTF-8.
ActiveMQ and OpenWire currently use a modified UTF-8 encoding that only uses 3 bytes so the conversion previously was breaking because the body was encoded using standard JDK UTF-8 encoding and set directly on an ActiveMQText message which was leading to decoding errors later when the ActiveMQMessage tried to decode using the modified encoder. The reverse was also was true and was breaking in some cases.
The fix now makes sure to correctly decode the Stomp message back to a String first and set that on the ActiveMQ message so it can be re-encoded correctly. The reverse is fixed as well so both conversion from Stomp -> OpenWire and OpenWire -> Stomp work. Tests have been added for Stomp -> OpenWire, OpenWire -> Stomp, and Stomp -> Stomp which is really Stomp -> OpenWire -> Stomp.

Something else is I didn't look to see if the conversion is correct for other protocols such as AMQP or MQTT. We should probably look at that too and apply the same or similar fix if needed. Also, we should check things like Websockets although websockets should just delegate to the same conversion logic I would think for STOMP/MQTT.

Change seems reasonable to me.  The idea that you need to change the openwire bits might be going to far given the expectation is that those bits are using modified UTF8 and has been for ages.  What really matters is the protocols on the periphery which are less likely to be expecting modified UTF8 vs the standard UTF8 encoding like AMQP.
For AMQP the proton-j bits will be doing that work to encode and decode and should only be populating the ActiveMQ message types with the decoded String content on ingress and then encoding the Strings from TextMessage and message properties as they move out of the broker (If the transformer is configured to 'jms').
For MQTT the protocol is a binary encoding and as far as I'm aware there is nothing internal that's creating TextMessage views of those MQTT messages so that should be safe (probably want to check for some weird option that does though).

repo: apache/activemq

license_source: github-api


text:
SQL Error [22023]: ERROR: label_id must be 1 .. 65535

Describe the bug
Upon trying to create new Nodes, getting the below error:
SQL Error [22023]: ERROR: label_id must be 1 .. 65535
The issue is very specific to Nodes whose Labels are already existing,  If I try to create Nodes with new Labels, no issues.
The volume of data is <100 (total number of Nodes across various Labels).
There is no steps I could think of to replicate this issue, it came randomly and exists.
How are you accessing AGE (Command line, driver, etc.)?

DBeaver, AGE Viewer

What is the necessary configuration info needed?

Postgres 16

What is the command that caused the error?
SELECT * FROM cypher('SchemaGraph', $$
CREATE (:Category {name: 'CategoryTest1'})
CREATE (:Category {name: 'CategoryTest2'})
$$) AS (n agtype);

SQL Error [22023]: ERROR: label_id must be 1 .. 65535

Expected behavior
The Nodes should have been created.
Environment (please complete the following information):

Release v1.5.0 for PG16

Additional context
Add any other context about the problem here.

The volume of data is <100 (total number of Nodes across various Labels).

Roughly, how many distinct labels?

The volume of data is <100 (total number of Nodes across various Labels).

Roughly, how many distinct labels?
There is roughly 26 distinct labels. The size of data is very small.
Edit: Additionally, how are you inserting all of these nodes
I am not doing any bulk insertions rather very simple node creations.
E.g:- The below query will create issue

SELECT * FROM cypher('SchemaGraph', $$
CREATE (:Category {name: 'CategoryTest1'})
$$) AS (n agtype);

The issue is very specific to Nodes whose Labels are already existing, If I try to create Nodes with new Labels, no issues.

@devcode100 This is happening when using the command line in PostgreSQL?

@jrgemignani , Both through psql cli and DBearver(SQL GUI Client), getting the issue.

@devcode100 I have not been able to reproduce this and I created 120,000 nodes as shown below, in both the latest and PG16 release 1.5.0. It could be that your install got messed up?
psql-16.1-5432-pgsql=# DO
$do$
BEGIN
    FOR i in 1..100000 LOOP
        PERFORM * FROM cypher('test', $$ CREATE (:Category {name: 'CategoryTest1'}) CREATE (:Category {name: 'CategoryTest2'}) CREATE (:Category2 {name: 'CategoryTest1'}) CREATE (:Category2  {name: 'CategoryTest2'}) $$) AS (n agtype);
end LOOP;
end
$do$;
DO
psql-16.1-5432-pgsql=# DO
$do$
BEGIN
    FOR i in 1..100000 LOOP
        PERFORM * FROM cypher('test', $$ CREATE (:Category {name: 'CategoryTest1'}) CREATE (:Category {name: 'CategoryTest2'}) CREATE (:Category2 {name: 'CategoryTest1'}) CREATE (:Category2  {name: 'CategoryTest2'}) $$) AS (n agtype);
end LOOP;
end
$do$;
DO
psql-16.1-5432-pgsql=# DO
$do$
BEGIN
    FOR i in 1..100000 LOOP
        PERFORM * FROM cypher('test', $$ CREATE (:Category {name: 'CategoryTest1'}) CREATE (:Category {name: 'CategoryTest2'}) CREATE (:Category2 {name: 'CategoryTest1'}) CREATE (:Category2  {name: 'CategoryTest2'}) $$) AS (n agtype);
end LOOP;
end
$do$;
DO
psql-16.1-5432-pgsql=# select * from cypher('test', $$ match (u) return count(u) $$) as (result agtype);
 result
---------
 1200000
(1 row)

psql-16.1-5432-pgsql=#

@devcode100 I should note that, for each of my tests, I used fresh installs of PostgreSQL version 16.1 and Apache AGE master and release 1.5.0. I'm not sure what specific version of 16 you are using, though.
Here I have created over 13M nodes without issue -
psql-16.1-5432-pgsql=# select * from cypher('test', $$ match (u) return count(u) $$) as (result agtype);
  result
----------
 13600000
(1 row)

psql-16.1-5432-pgsql=#

@jrgemignani , appreciate your effort to replicate.
The current postgres server setup  have is a single node one and I haven't encountered any issues with DB so far.
I would suggest to keep the issue open(at least till ~2 releases ) as its critical for us to monitor the AGE extension as we are planning to move to production with the same.

@devcode100 I have tried this against PostgreSQL version 16.2 and the master branch, without issue -
psql-16.2-5432-pgsql=# drop extension age; create extension age; load 'age'; set search_path TO ag_catalog;
DROP EXTENSION
CREATE EXTENSION
LOAD
SET
psql-16.2-5432-pgsql=# select * from create_graph('test');
NOTICE:  graph "test" has been created
 create_graph
--------------

(1 row)

psql-16.2-5432-pgsql=# DO
$do$
BEGIN
    FOR i in 1..10000000 LOOP
        PERFORM * FROM cypher('test', $$ CREATE (:Category {name: 'CategoryTest1'}) CREATE (:Category {name: 'CategoryTest2'}) CREATE (:Category2 {name: 'CategoryTest1'}) CREATE (:Category2  {name: 'CategoryTest2'}) $$) AS (result agtype);
end LOOP;
end
$do$;
DO
psql-16.2-5432-pgsql=# select * from cypher('test', $$ match (u) return count(u) $$) as (result agtype);
  result
----------
 40000000
(1 row)

psql-16.2-5432-pgsql=#

@devcode100 I have tried this against PostgreSQL version 16.2 and release 1.5.0, without issue -
psql-16.2-5432-pgsql=# drop extension age; create extension age; load 'age'; set search_path TO ag_catalog;
DROP EXTENSION
CREATE EXTENSION
LOAD
SET
psql-16.2-5432-pgsql=# select * from create_graph('test');
NOTICE:  graph "test" has been created
 create_graph
--------------

(1 row)

psql-16.2-5432-pgsql=# DO
$do$
BEGIN
    FOR i in 1..10000000 LOOP
        PERFORM * FROM cypher('test', $$ CREATE (:Category {name: 'CategoryTest1'}) CREATE (:Category {name: 'CategoryTest2'}) CREATE (:Category2 {name: 'CategoryTest1'}) CREATE (:Category2  {name: 'CategoryTest2'}) $$) AS (result agtype);
end LOOP;
end
$do$;
DO
psql-16.2-5432-pgsql=# select * from cypher('test', $$ match (u) return count(u) $$) as (result agtype);
  result
----------
 40000000
(1 row)

psql-16.2-5432-pgsql=#

@jrgemignani  - I have dropped the Graph where I got the issue and working on new Graph schema now. I have not encountered the issue so far.
But to be frank the issue seemed weird and I still feel there is a corner case which triggers the issue.
Even I couldn't replicate the issue( means in the new Graph).

I'm facing the same error while trying to load the nodes from a CSV

I'm using the docker image. The cities.csv is the same as in the repo https://github.com/apache/age/blob/master/regress/age_load/data/cities.csv
services:
  age:
    image: apache/age
    container_name: age
    environment:
      - POSTGRES_USER=postgresUser
      - POSTGRES_PASSWORD=postgresPW
      - POSTGRES_DB=postgresDB
    ports:
      - "5430:5432"
    dns:
      - 8.8.8.8
      - 8.8.4.4
      - ```

I'm facing the same error but while trying to use load_labels_from_file. I'm using the age docker image

services:
  age:
    image: apache/age
    container_name: age
    environment:
      - POSTGRES_USER=postgresUser
      - POSTGRES_PASSWORD=postgresPW
      - POSTGRES_DB=postgresDB
    ports:
      - "5430:5432"
    dns:
      - 8.8.8.8
      - 8.8.4.4

@dhilst Thank you for the information, I will see if I can repeat it.

@dhilst I was able to reproduce it, see below. I will try to find a resolution.
psql-16.2-5432-psql=# drop extension age; create extension age; load 'age'; set search_path TO ag_catalog;
DROP EXTENSION
CREATE EXTENSION
LOAD
SET
psql-16.2-5432-psql=# \dx age
             List of installed extensions
 Name | Version |   Schema   |      Description
------+---------+------------+------------------------
 age  | 1.5.0   | ag_catalog | AGE database extension
(1 row)

psql-16.2-5432-psql=# select create_graph('test_graph');
NOTICE:  graph "test_graph" has been created
 create_graph
--------------

(1 row)

psql-16.2-5432-psql=# select load_labels_from_file('test_graph', 'Cities', '/home/jgemignani/POSTGRESQL/cities.csv');
ERROR:  label_id must be 1 .. 65535
psql-16.2-5432-psql=#

@dhilst This issue is that you aren't creating a vlabel for the label you are importing from the CSV. Note how the regression tests do the load -
SELECT create_graph('agload_test_graph');

SELECT create_vlabel('agload_test_graph','Country');
SELECT load_labels_from_file('agload_test_graph', 'Country',
    'age_load/countries.csv');

SELECT create_vlabel('agload_test_graph','City');
SELECT load_labels_from_file('agload_test_graph', 'City',
    'age_load/cities.csv');

SELECT create_elabel('agload_test_graph','has_city');
SELECT load_edges_from_file('agload_test_graph', 'has_city',
     'age_load/edges.csv');

If applied to your issue -
psql-16.2-5432-psql=# drop extension age; create extension age; load 'age'; set search_path TO ag_catalog;
DROP EXTENSION
CREATE EXTENSION
LOAD
SET
psql-16.2-5432-psql=# select create_graph('test_graph');
NOTICE:  graph "test_graph" has been created
 create_graph
--------------

(1 row)

psql-16.2-5432-psql=# select load_labels_from_file('test_graph', 'Cities', '/home/jgemignani/POSTGRESQL/cities.csv');
ERROR:  label_id must be 1 .. 65535
psql-16.2-5432-psql=# SELECT create_vlabel('test_graph','Cities');
NOTICE:  VLabel "Cities" has been created
 create_vlabel
---------------

(1 row)

psql-16.2-5432-psql=# select load_labels_from_file('test_graph', 'Cities', '/home/jgemignani/POSTGRESQL/cities.csv');
 load_labels_from_file
-----------------------

(1 row)

psql-16.2-5432-psql=#

The issue is that the label you are importing the vertices or edges into, doesn't exist.

Hi @jrgemignani Thanks for the answer.
Yes it works for the vertices, and the documentation states the creation of the labels before loading the file, I'm sorry I missed that. Anyway, now I'm having the same problem with the edges, and creating the elabel before the load_edges_from_file is not helping

I don't know if I'm missing something. These lines are in the documentation and your comment https://github.com/apache/age/issues/1837#issuecomment-2207356787 I double-checked it, and it looks right to me, let me know if you want me to test something. I'm using the same docker image as before, I recreated the container to ensure that I have the extension that comes from the image and not the compiled from source from the previous test.

source: gharchive/issue

authors: devcode100 dhilst jrgemignani

repo: apache/age


text:
[AMBARI-24582] - Ambari Alert - HiveServer2 Process - False negative in Certain Scenarios

What changes were proposed in this pull request?
Cherry pick of https://github.com/apache/ambari/pull/2228

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3814/
Test FAILed.
Test FAILured.

authors: asfgit jonathan-hurley

repo: apache/ambari


text:
[AMBARI-23573] HDFS install fails at before-ANY hook due to no created instances (dsen)

What changes were proposed in this pull request?
Commented out hadoop related code from the hooks.
How was this patch tested?
Manually

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1838/
Test FAILed.
Test FAILured.


text:
[Subtask][Trino]: Support UnifiedCatalog in Trino Engine.

Search before asking

[X] I have searched in the issues and found no similar issues.

Description
No response
Parent issue
https://github.com/NetEase/arctic/issues/1061
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Closed as not planned.

authors: baiyangtx zhoujinsong

repo: apache/amoro


text:
help request: Java Plugin Runner thorws warning

Description
开发java版的外部程序遇到的问题：
程序：apisix是在win11上通过docker-compose安装的容器部署的
java外部插件是clone的demo项目进行改造的

根据文档描述的，我将外部插件编译好的jar可执行文件，上传到了apisix的容器内部

之后修改了配置文件并重启了apisix

然后在Dashboard配置路由信息

测试路由转发报错

错误日志是这些

我不知道是哪一步出了问题，麻烦可以帮忙处理一下吗
Environment

APISIX version (run apisix version):2.13-alpine
Operating system (run uname -a):
OpenResty / Nginx version (run openresty -V or nginx -V):
etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
APISIX Dashboard version, if relevant:
Plugin runner version, for issues related to plugin runners:
LuaRocks version, for installation issues (run luarocks --version):

2022/07/14 05:46:45 [warn] 58#58: *257 [lua] init.lua:931: respawning new runner..., context: ngx.timer
2022/07/14 05:46:45 [error] 58#2945: lua pipe child execvp() failed while executing java (2: No such file or directory)
2022/07/14 05:46:45 [warn] 58#58: *257 [lua] init.lua:919: runner exited with reason: exit, status: 1, context: ngx.timer
2022/07/14 05:46:45 [warn] 58#58: *257 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 58#58: *257 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 58#58: *257 [lua] init.lua:929: respawn runner 3 seconds later with cmd: ["java","-jar","-Xmx1g","-Xms1g","/usr/local/apisix/external_plugin/apisix-java-plugin-runner.jar"], context: ngx.timer
2022/07/14 05:46:45 [warn] 49#49: *1467811 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 49#49: *1467811 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 51#51: *1467812 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 51#51: *1467812 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 46#46: *1467813 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 54#54: *1467814 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 46#46: *1467813 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 54#54: *1467814 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 45#45: *1467815 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 45#45: *1467815 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 44#44: *1467816 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 53#53: *1467817 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 44#44: *1467816 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 52#52: *1467818 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 53#53: *1467817 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 52#52: *1467818 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 55#55: *1467823 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 50#50: *1467824 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 50#50: *1467824 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 55#55: *1467823 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 47#47: *1467825 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 47#47: *1467825 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer
2022/07/14 05:46:45 [warn] 48#48: *1467826 [lua] init.lua:361: flush_token(): flush conf token in shared dict, context: ngx.timer
2022/07/14 05:46:45 [warn] 48#48: *1467826 [lua] init.lua:818: flush conf token lrucache, context: ngx.timer

2022/07/14 05:46:45 [error] 58#2945: lua pipe child execvp() failed while executing java (2: No such file or directory)

APISIX cannot find the jar file path, please check the file path inside docker and whether APISIX has permission to read and write the jar file.

我执行了 chmod 777，restart container again，all the same this error

It looks like you wrote the wrong path.
where apisix-java-plugin-runner.jar? /usr/local/apisix-plugin-runner or /usr/local/apisix/external_plugin?

/usr/local/apisix-plugin-runner this path is configured by me today，/usr/local/apisix/external_plugin this path was configured by me yesterday. I repackaged the image and configured it again

I tried it. I put the plugin runner and apimix together to create a new apimix image. There is no problem deploying and using external plug-ins.
However, for apifix image, my external plug-ins may change frequently. What I hope is that apifix does not need to re mirror, but only needs to modify the executable package of the external plug-ins.
I thought of a way to mount the directory of external plug-ins to the outside of the container, and then update the executable package of external plug-ins in this mounted directory. I don't know whether this method is feasible?

After experiments, I came to a conclusion that I need to create an apifix image containing plugin runner first, then start the image container, and then modify and upload the executable package of plugin runner.
However, this situation is not explained in the document, or I did not find the corresponding document information. I think in the development process, we don't know that we need to develop external plug-ins independently from the beginning, but we will consider developing external plug-ins only when we find that some requirements are not met in the process of using apifix.
Therefore, we first deployed an image container that does not contain plugin runner. Later, when we need to develop external plug-ins, we only need to upload the developed plugin runner to the APIs IX container to use it directly.
I hope the external plug-in development process should be like this. Although repacking the image and starting the container is not a very complicated work, I think it can be saved

From the community feedback, it appears that the re-build image deployment method also exists. ref docs: https://github.com/apache/apisix-java-plugin-runner/blob/main/docs/zh/quick-start.md#部署

Yes, I mean, the problem I found out was that when I first started using the official Apisix image, if I wanted to use plugin-Runner, I had to rebuild an image that included plugin-Runner before I could use the external plug-in, instead of what the documentation said, Only deploy in the same container. This is not reflected in the documentation.

Then, I would like to say that, as mentioned in the documentation, to use plugin-runner, only plugin-runner and Apisix need to be in the same container, so that developers only need to focus on the development of plugin-runner. Plugin-runner's executable package is provided, and there is no need to pay attention to whether plugin-Runner and Apisix have rebuilt the new image

Plugin-runner's executable package is provided, and there is no need to pay attention to whether plugin-Runner and Apisix have rebuilt the new image

Would you like to send this topic to the mailing list on dev@apisix.apache.org for discussion about providing an APISIX image with the plugin runnner. (Since this issue is not unique to java runner, plugin runners such as go also do not have mirrors)

好的

@MorganArthur is there any solution from the apisix ?

apisix start命令 ，必须在 /usr/local/apisix路径下执行，这样才能生成pid.sock文件！！！！

authors: MorganArthur lefreet tomcmd tzssangglass

repo: apache/apisix-java-plugin-runner


text:
[Site]: improve static files performance

Current State
Resources Sample

https://apisix-website-static.apiseven.com/zh/assets/js/runtime~main.26c4290b.js
https://apisix-website-static.apiseven.com/assets/js/runtime~main.e301679c.js

TODO

return"assets/js/" → with CDN
return"assets/css/ → with CDN

Other

https://askubuntu.com/questions/513046/how-to-search-strings-inside-files-like-in-windows-without-search-monkey

Desired State
Description of the change you are proposing:
See Current State
Would you like to fix this issue?
No response

I think we need to consider code-splitting first 🤔

I'm also trying to remove gsap, but in order to keep the original animation as much as possible, I may need to introduce other dependencies, so the exact bundle size needs to be analyzed further.

Umm, I am pretty sure removing gsap isn't the only culprit that might speed up the website drastically, meaning the profit and loss is heavily unbalanced, less profit, more loss.
I would almost prefer to say that if converting the animation's timing and spacing to what was originally the case is problematic, we are hurting the website, and making it look amateurish (you know, font sizes, animations, hierarchy etc. being off) .
This is really important brand wise
I mean to say, if we really don't want to use gsap, than either use framer or something, as the animations I see of today are very off and uneven in easing and timing. So, forcing to recreate the same animations in css, if seems problematic then why force yourself, it's not leading to any major gain anyways (performance wise, as was the thought goal I suppose)
Plus we are losing some key interactions like the snap fix and show on scroll features in the middle section, which are like some things that stay in minds of users and make them return.
All I am trying to say is we are losing on brand front here if this doesn't get fixed 😬
Also, why are we changing things like this stepwise, and that too on the live website ? I mean such drastic changes so quickly might confuse the user on whether the website is even working properly. Maybe @SkyeYoung you should work out everything separately first than push after review, just a suggestion.
Sorry, for the long rant but this is something I really wanted to say.

@1502shivam-singh I don't have the ability to make everything better in one commit, but our SEO, performance needs are obviously more pressing, so I choose to tweak the site as soon as possible rather than wait until all the issues are resolved before submitting changes.
On matters such as removing GSAP and animations, you can see through my PR submission that my attention has clearly not been on those areas lately.
I've recently been trying to cut up the original site to achieve some capabilities that were previously unattainable, and you can see those changes at #1225 #1218.
Anyway, things will always get better, so let's stay patient together!

@1502shivam-singh I don't have the ability to make everything better in one commit,

What I meant was distributing similar type of changes in a single PR. If it seems unfeasible no worries, use your own work style.

On matters such as removing GSAP and animations, you can see through my PR submission that my attention has clearly not been on those areas lately.

Sure I get this, but let me rephrase what i said, my main concern is having difficulty in imitating exactly the animations with same spacing and timing. GSAP is just a medium through which I am trying to communicate that you can either use Framer or something more lightweight for your needs, if CSS animations seem painful for the task.

I've recently been trying to cut up the original site to achieve some capabilities that were previously unattainable, and you can see those changes at #1225 #1218.

Really commend your work on this, few weeks back I was also thinking of something like this. Great work !

Anyway, things will always get better, so let's stay patient together.

Sure, I am just trying to highlight the negative points here so you are aware of an overall design perspective, so i may seem impatient (Especially from the rant above 😅), but don't let it interfere with your work, okay.

authors: 1502shivam-singh SkyeYoung juzhiyuan

repo: apache/apisix-website


text:
Bump sqllogictest to v0.11.1

Which issue does this PR close?

Closes #.
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Thank you @xudong963

repo: apache/arrow-datafusion


text:
ArrowArray::try_from_raw Misleading Signature

Describe the bug

ArrowArray::try_from_raw accepts *const FFI_ArrowArray and *const FFI_ArrowSchema. As of https://github.com/apache/arrow-rs/pull/1449 no longer takes ownership of the pointers, and instead "moves" the contents into its own allocations. The method is unsafe, and so this technically isn't unsound, but it is fairly surprising to people unfamiliar with the interface who might assume they can pass in a const reference.
Related to this, the fact it doesn't take ownership makes it very easy to accidentally leak memory https://github.com/apache/arrow-rs/pull/1878 and https://github.com/apache/arrow-rs/issues/3683
To Reproduce

Expected behavior

Additional context

label_issue.py automatically added labels {'arrow'} from #3685

repo: apache/arrow-rs


text:
[Go] array.RecordFromJSON() method does not return for a significant amount of time for large JSON file

Describe the usage question you have. Please include as many useful details as  possible.
When I attempt to use the array.RecordFromJSON() method to read a relatively large JSON file (approximately 200MB in size), Ithe method does not return for a significant amount of time. It seems to be either blocked indefinitely or possibly stuck in an ongoing data processing loop. I am currently unable to determine whether this is a bug or how to successfully utilize the method to achieve my objective.
Here is my code:
func readJsonFile() {
schema := arrow.NewSchema([]arrow.Field{
{Name: "instruction", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "input", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "output", Type: arrow.BinaryTypes.String, Nullable: true},
}, nil)
jsonFile, err := os.Open("/home/data/mydata.json")
if err != nil {
fmt.Println("open json file:", err)
return
}
defer jsonFile.Close()
mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
rec, _, err := array.RecordFromJSON(mem, schema, jsonFile)
fmt.Printf("record number is： %d", rec.NumRows())
fmt.Println("field name 0 -", rec.Schema().Field(0).Name)
fmt.Println("field name 1 -", rec.Schema().Field(1).Name)
fmt.Println("field name 2 -", rec.Schema().Field(2).Name)
}
My json file format looks like this:
[{"instruction":"a", "input":"b", "output","o1"}, {"instruction":"c", "input":"d", "output","o2"}, ...]
Component(s)
Go

I'll take a look at this sometime this week.

Would you be able to provide the actual data file that causes the problem? I created a 200MB JSON file with random values using the format schema you provided and then ran the code you provided to read it. It took just around 7 seconds or so to run and I wasn't able to reproduce indefinite blocking that you reported.

ok， here is my data file
data.zip

Now i convert the json format and use WithMultipleDocs option, it works well.
func TestReadJson(t *testing.T) {
	file, err := os.Open("/home/data/180000.json")
	if err != nil {
		fmt.Println("open json:", err)
	}
	defer file.Close()
	content, err := io.ReadAll(file)
	if err != nil {
		fmt.Println("read file:", err)
	}

	var jsonData []map[string]interface{}
	err = json.Unmarshal(content, &jsonData)
	if err != nil {
		fmt.Println("Unmarshal failed:", err)
	}
	schema := arrow.NewSchema([]arrow.Field{
		{Name: "instruction", Type: arrow.BinaryTypes.String, Nullable: true},
		{Name: "input", Type: arrow.BinaryTypes.String, Nullable: true},
		{Name: "output", Type: arrow.BinaryTypes.String, Nullable: true},
	}, nil)
	mem := memory.NewCheckedAllocator(memory.DefaultAllocator)
	var recs []arrow.Record
	for _, obj := range jsonData {
		bytes, err := json.Marshal(obj)
		if err != nil {
			fmt.Println("Marshal failed:", err)
		}
		rec, _, err := array.RecordFromJSON(mem, schema, strings.NewReader(string(bytes)), array.WithMultipleDocs())
		recs = append(recs, rec)
	}
	fmt.Println("value -", recs[0].Column(0))
	fmt.Println("value -", recs[0].Column(1))
	fmt.Println("value -", recs[0].Column(2))
}

@AlwaysZhi was using the WithMultipleDocs option enough to solve the issue? or is there more of a question here?
Typically you need the WithMultipleDocs option if the format of the json file is:
{"a": 1, "b": 2......}
{"a": 2, "b": 3......}
{.......}

Rather than having them all wrapped inside of [ ] as a single JSON document

authors: AlwaysZhi zeroshade

repo: apache/arrow


text:
[C++] Acero kernel function returning int64 or float64 not working

Describe the bug, including details regarding any error messages, version, and platform.
I am trying to create a custom function that takes an uniary input float64 and returns a float64. But, the output is not accurate. Mostly it is all 0. Look into the sample code to reproduce the issue
Arrow version: 1800.0.0
Gcc: 13
// main.cpp
#include <iostream>
#include <arrow/compute/api.h>
#include <arrow/status.h>
#include <arrow/type.h>

#include <arrow/array.h>
#include <arrow/builder.h>
#include <arrow/compute/exec.h>
#include <arrow/compute/function.h>

using namespace arrow;
using namespace arrow::compute;

// Define the function that computes the square of input values.
Status SquareFunctionExec(KernelContext* ctx, const ExecSpan& batch, ExecResult* out) {

    // Ensure the input is numeric.
    if (!batch[0].is_array() || batch[0].type()->id() != Type::DOUBLE) {
        return Status::Invalid("Expected double type for input");
    }

    const auto& exec_batch = batch.ToExecBatch();

    const std::shared_ptr<arrow::DoubleArray>& arg0 = exec_batch.values[0].array_as<arrow::DoubleArray>();
    const int64_t length = arg0->length();
    arrow::DoubleBuilder builder;
    ARROW_RETURN_NOT_OK(builder.Resize(length));

    for (int64_t i = 0; i < length; ++i)
    {
        if (arg0->IsNull(i))
            builder.UnsafeAppendNull();
        else
        {
            const double d = arg0->Value(i);
            ARROW_RETURN_NOT_OK(builder.Append(d * d));
        }
    }
    ARROW_ASSIGN_OR_RAISE(const std::shared_ptr<arrow::Array> output_array, builder.Finish());
    out->value = output_array->data();
    std::cout << " Inside kernel operation (Expected)" << output_array->ToString() << std::endl;
    return arrow::Status::OK();
}

Status RegisterSquareFunction(FunctionRegistry* registry) {
    auto square_func = std::make_shared<ScalarFunction>("square", Arity::Unary(), arrow::compute::FunctionDoc::Empty());

    const auto signature = arrow::compute::KernelSignature::Make(
           {arrow::float64()}, arrow::float64());
    arrow::compute::ScalarKernel kernel(signature, &SquareFunctionExec);

    RETURN_NOT_OK(square_func->AddKernel(std::move(kernel)));
    return registry->AddFunction(std::move(square_func));
}

void assertOK(const arrow::Status& status)
{
    if (!status.ok())
        throw std::runtime_error(status.message());
}

int main() {
    // Initialize the compute function registry.
    const auto registry = arrow::compute::GetFunctionRegistry();

    // Register the custom square function.
    auto status = RegisterSquareFunction(registry);
    if (!status.ok()) {
        std::cerr << "Error registering custom function: " << status.ToString() << std::endl;
        return 1;
    }

    // Use the function (example).
    ExecContext exec_ctx;
    FunctionOptions* options = nullptr;

    arrow::DoubleBuilder builder;

    // Append scalar values
    assertOK(builder.Append(1.1));
    assertOK(builder.Append(2.2));
    assertOK(builder.Append(3.3));

    // Finalize the array (convert builder to Array)
    std::shared_ptr<arrow::DoubleArray> input_array;
    assertOK(builder.Finish(&input_array));

    const auto call_result = CallFunction("square", {input_array}, options, &exec_ctx);

    if (call_result.ok()) {
        const auto& output = call_result.ValueOrDie();
        std::cout << "Result (Unexpected): " << output.ToString() << std::endl;
    } else {
        std::cerr << "Function execution failed: " << call_result.status().ToString() << std::endl;
    }

    return 0;
}

Cmake
cmake_minimum_required(VERSION 3.28)
project(ArrowTest)

set(CMAKE_CXX_STANDARD 23)

set(CMAKE_CXX_FLAGS "-Wall")
set(CMAKE_CXX_STANDARD_REQUIRED True)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wreturn-type")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -static-libgcc -static-libstdc++")

find_package(Arrow REQUIRED)
find_package(Parquet REQUIRED)
find_package(ArrowDataset REQUIRED)

message("Arrow_FOUND: ${Arrow_FOUND}")
message("Arrow Version: ${ARROW_VERSION}")
message("Arrow SO Version: ${ARROW_FULL_SO_VERSION}")

set(ARROW_LIBRARY_TARGET arrow_shared)
set(PARQUET_LIBRARY_TARGET parquet_shared)
set(ARROW_DATASET_LIBRARY_TARGET arrow_dataset)
set(ARROW_ACERO arrow_acero)

file(MAKE_DIRECTORY "${ARROW_INCLUDE_DIR}")
#add_library(${ARROW_LIBRARY_TARGET} SHARED IMPORTED)
#add_library(${PARQUET_LIBRARY_TARGET} SHARED IMPORTED)

add_executable(ArrowTest main.cpp)
target_link_libraries(ArrowTest
        PRIVATE
        ${ARROW_ACERO}
        ${ARROW_LIBRARY_TARGET}
        ${ARROW_DATASET_LIBRARY_TARGET}
        ${PARQUET_LIBRARY_TARGET}
)

ExpectedOutput: [
1.2100000000000002,
4.840000000000001,
10.889999999999999
]
Actual Output: Array([
2.548682823166e-311,
0,
0
])
Component(s)
C++

Have you tried stepping in SquareFunctionExec to see what happens?

One important thing: the ScalarKernel defaults are as follows:
https://github.com/apache/arrow/blob/ded148c2ef5dd441a3b6ab9496d0ed0aeb940f71/cpp/src/arrow/compute/kernel.h#L566-L569
So out->value has already been allocated, its validity bitmap has been computed from the input args, and you should just write inside its value buffer for non-null inputs.

Inside kernel operation (Expected)

Yes, the function produces expected result. In the sample code you you can see i have printed the output from that function. Search for ... "Inside kernel operation (Expected)"

Setting the memory allocator to NO_PREALLOCATE resolved the issue.
kernel.mem_allocation = arrow::compute::MemAllocation::NO_PREALLOCATE;

Thank you @pitrou for the hint.

authors: abulbasar pitrou


text:
feat: Add binary support for CSV

Adding binary support for CSV

@yevgenypats Can you create a corresponding issue for this and then update the title to be GH-<issue#>: [Go] Add binary support for CSV thanks!

@zeroshade updated :)

authors: yevgenypats zeroshade


text:
MINOR: [Go] Remove deprecated equality checks

Rationale for this change
There are several deprecated comparison functions that have long been deprecated.
This PR unblocks the removal of those functions as nothing depends on them anymore.
Given the functions have been deprecated for a long time and might have been removed in v9, I removed them in this PR already, given this is for v13.

What changes are included in this PR?

Are these changes tested?
Unit tests are passing locally.

Are there any user-facing changes?
Removal of old deprecated functions for comparing arrays.

Could you open an issue for this pull request because this is not a MINOR change?
See also our MINOR definition: https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes

I agree with @kou's statement that this isn't a MINOR change and should have an issue opened for it. Also it looks like you missed one ArrayEqual in the cdata_test.go file.

authors: kou metalmatze zeroshade


text:
ARROW-7608: [C++][Dataset] Add the ability to list files in FileSystemSource

It also enables instrospection in R of dataset classes.

The OSX failure is a Flight unrelated issue.

I don't know how that information is exactly stored, but: would it also be possible to have a way to get the partition expression for each file?

Absolutely.

Perhaps (partition_expression, file) iterator would be better to expose under files()
for partition, path in source.files():
    pass

Either way we should have a jira about it.

There's a windows path formatting issue: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/30694260/job/8rs3y80x55fclkrk#L5502
I'll take a look

Merci Neal!

Is this ready to be merged?

it isn't due to the windows path issue, I'm not sure if I should dig into it or just remove the R test for now (it'll still work, just the path is funky).

I've pushed a commit to try to make the tests in R work on windows (basically, tell file.path() to use / as path separator so that the paths will compare as equal. If that doesn't work and we don't want to lose more time on it, we can add skip_on_os("windows") before the tests and move on.

The odd path segments are introduced by tempfile() (before the failing test runs) so I think the solution is to wrap it with normalizePath(tempfile(), winslash = "/"))

authors: bkietz fsaintjacques jorisvandenbossche kszucs nealrichardson


text:
ARROW-12215: [C++] Allow null values in fixed-size binary columns read from CSV

Give the CSV converter for reading fixed size binary columns the same behavior for detecting nulls as other datatypes.
Update tests to reflect the fact that fixed size binary types are now nullable in this context.

Of course! Thank you!


text:
AVRO-3195: Include rust in the docker build image

Make sure you have checked all steps below.
Jira

[X] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-3195
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason: The ubertool is part of the manual release process.

Commits

[X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

R: @martin-g Can you take a look?  The dockerfile is used to build release artifacts and as an example for the requirements to build all of the Avro SDKs

repo: apache/avro


text:
[Feature Request]: Override external SDK container URLs for Dataflow by default in Java SDK

What would you like to happen?
Override external SDK container URLs for Dataflow by default in Java SDK so users don't need to manually specify --sdkHarnessContainerImageOverrides for Java cross-language pipelines using Python transforms.
Issue Priority
Priority: 2
Issue Component
Component: cross-language

Why would the user have to manually (or automatically) specify this?

@chamikaramj what do you think of this?

I think this is done. Users do not have to specify "sdkHarnessContainerImageOverrides" when using Java multi-language on Dataflow with released Beam.

authors: chamikaramj ihji kennknowles robertwb

repo: apache/beam


text:
[Task]: Use Batched DoFn API in the DataFrame API

What needs to happen?
We should leverage Batched DoFns to do schema <-> DataFrame conversion in the DataFrame API
Issue Priority
Priority: 2
Issue Component
Component: dsl-dataframe

Part of #21650


text:
[BEAM-9813] Replace outputs with respect to the correct tag.

This is triggered in an interaction between https://github.com/apache/beam/commit/cebe6286e0 and https://github.com/apache/beam/pull/11335

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @chamikaramj

LGTM. Thanks.

Run Portable_Python PreCommit

Run Python PreCommit

authors: chamikaramj robertwb


text:
[WIP] BEAM-12561 method truncate on series and dataframe

Implementation of truncate for DeferredDataFrame and DeferredSeries

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

retest this please

@AlikRodriguez Are the unit test failures related to this change?

authors: AlikRodriguez kileys


text:
[BEAM-1742] Update Checkpoint and Finalize Documentation

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

This more precisely specifies how a runner is permitted to interact with
a CheckpointMark, and the requirements on implementors for what a source
must be capable of handling.
Existing CheckpointMarks should be audited to ensure that they meet
these new requirements.

R: @kennknowles @dhalperi

Coverage increased (+0.01%) to 70.129% when pulling bd7272422d5ff3eb9790a94d0f0af57f04c295fa on tgroh:finalize_spec into b672cde111b2f11d185479e1d208e63ef6ff322e on apache:master.

Coverage increased (+0.01%) to 70.129% when pulling bd7272422d5ff3eb9790a94d0f0af57f04c295fa on tgroh:finalize_spec into b672cde111b2f11d185479e1d208e63ef6ff322e on apache:master.

Coverage increased (+0.003%) to 70.12% when pulling bd7272422d5ff3eb9790a94d0f0af57f04c295fa on tgroh:finalize_spec into b672cde111b2f11d185479e1d208e63ef6ff322e on apache:master.

@tgroh, any progress here?

Coverage increased (+0.9%) to 70.662% when pulling bd7272422d5ff3eb9790a94d0f0af57f04c295fa on tgroh:finalize_spec into 810db7f990a9e39179f3613347ab5cde7e7c650c on apache:master.

Changes Unknown when pulling 70cc4039d88237dfd1e9cec4adce87e2ac194a74 on tgroh:finalize_spec into ** on apache:master**.

LGTM, might as well merge for the added detail.

Coverage decreased (-0.004%) to 70.472% when pulling 588965efd12f1625b8b830523c14f293e9f72936 on tgroh:finalize_spec into 727476590c3b4c805a23b06d985c81447d19895a on apache:master.

authors: coveralls davorbonaci kennknowles tgroh


text:
Use WindowedValue.withValue on hot paths #21250

This removed about half of the overhead for outputting a value in the common scenario where we are already using a valid timestamp (the input timestamp) and also that we can use the withValue hot path which is optimized for certain use cases (e.g. the globally windowed value case).
Before:
Benchmark                                Mode  Cnt     Score     Error  Units
ProcessBundleBenchmark.testLargeBundle  thrpt   15  3616.761 ± 157.844  ops/s

After:
Benchmark                                Mode  Cnt     Score     Error  Units
ProcessBundleBenchmark.testLargeBundle  thrpt   15  3666.889 ± 151.448  ops/s

This is for #21250.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @bhisevishal

retest this please

Run Java PreCommit

Precommits passed, github UI hasn't updated yet:
https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/22565/
https://ci-beam.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Java11_Commit/14601/


text:
Remove deprecated week and weekofyear for Pandas 2

Partially fixes #27221.

R: @tvalentyn


text:
BIGTOP-4025:Improve the relevant conf directory for flink rpm build scripts

Description of PR
Improve the relevant conf directory for Flink rpm build scripts:
1.remove redundant parameter passing
2.Improve the default conf directory for Hadoop and Flink
How was this patch tested?
Build Flink rpm package and install Flink service
For code changes:

[1 ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
[1] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

@sekikn Please help me review the code，thanks!

+1, thank you for your contribution @LiJie20190102.

repo: apache/bigtop


text:
Comment out catalog items coming from brooklyn-libraries in Karaf dist

brooklyn-libraries features are not installed at boot so can't reference them in the default catalog. The reason is that brooklyn-libraries is not yet built when the Karaf dist is assembled.

I've raised PRs to move the karaf distribution into brooklyn-dist, see
https://github.com/apache/brooklyn-server/pull/271
https://github.com/apache/brooklyn-library/pull/53
https://github.com/apache/brooklyn-dist/pull/41
As part of that I split out the catalog and feature files for the brooklyn library stuff, so if the above are accepted then this PR may no longer be needed.

Closing in favour of https://github.com/apache/brooklyn-server/pull/271

authors: geomacy neykov

repo: apache/brooklyn-server


text:
[WIP] [JCLOUDS 2.1.0] Upgrade to jclouds 2.1

These are the changes I needed to implement in order to run Brooklyn against current jclouds master.
Leaving them around to be reused for the jclouds 2.1.0 upgrade once released.

Thanks @neykov - can you include "[WIP]" or some such in the PR name, so we know not to merge this yet (given it points at 2.1.0-SNAPSHOT, and 2.1.0 isn't released yet).

Github closed this by mistake, confused by the already merged commit (and later reverted). Cherry-picked the same commit and reopening.

hi @neykov I think this can be retired now as superseded by https://github.com/apache/brooklyn-server/pull/950.  Andrea cross checked against this one before his was merged.

authors: aledsage geomacy neykov


text:
fix(helm): proper CRDs

Ref #4497
This PR should patch the problem for Helm chart with version 1.12.1.

Release Note
fix(helm): proper CRDs

No checks required. Merging.

repo: apache/camel-k


text:
Create a Camel Netty TCP Consumer/Producer extension

I will say from my point of view that may or may not reflect others, but the idea of a quarkus-based camel has a lot of interest when you are working lower-level TCP payloads to EIP-style solve into different solutions.
This ticket may also be used to identify challenges.

note - I'm barely ready to contribute, so if someone can fast track and build on top of it, please do!

The idea should be to simply create a camel-netty extension.

Contributions are welcome! @dhartford you or anybody else may want to try to implement this following https://camel.apache.org/camel-quarkus/latest/contributor-guide.html Feel free to ask if you have specific questions.

Reviewing, I was hoping for a sample init commit of just the netty-http to see what the scope was, but no luck. Reviewing these, let me know if I missed something from a scope point of view for 'writing a new quarkus-camel extension':

sample extension scope:
https://github.com/apache/camel-quarkus/tree/master/extensions/netty-http
https://github.com/apache/camel-quarkus/tree/master/integration-tests/netty-http/
And parent-pom and documentation additions:
https://github.com/apache/camel-quarkus/search?q=netty-http&unscoped_q=netty-http

Yes you need to add:

3 new modules under https://github.com/apache/camel-quarkus/tree/master/extensions : netty-parent, netty runtime and netty deployment
An integration test under https://github.com/apache/camel-quarkus/tree/master/integration-tests
Entries to poms/bom/pom.xml and to poms/deployment/pom.xml

Special in this case is that camel-netty-http already depends on camel-netty, so actually the existing camel-quarkus-netty-http should be refactored to depend on your new camel-quarkus-netty extension.

Pre-review request - I've started doing the integration tests directly with Socket code, which feels incorrect, but I didn't want to burden with unnecessary test dependencies/libraries.
Is there a recommendation/preference from the team of how to do TCP testing (whereas RestAssured is currently the preferred HTTP testing framework)?
https://github.com/dhartford/camel-quarkus/blob/camel-quarkus-netty/integration-tests/netty/src/test/java/org/apache/camel/quarkus/component/netty/it/NettyTest.java

I'd be fine with using a java.net.Socket directly, but I am not the highest instance here. WDYT, @lburgazzoli and others?

I'll keep moving forward with Socket, thanks for the double-check.  I simulated the Poem test from the original camel-netty tests and is working fine with normal java, but failing in native with this. I'm still tracking it down, just sharing status update or if this is an issue previously experienced in other camel projects:
org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: netty://tcp://localhost:8999?sync=true&textline=true due to: There are 2 parameters that couldn't be set on the endpoint. Check the uri if the parameters are spelt correctly and that they are properties of the endpoint. Unknown parameters=[{sync=true, textline=true}]

using sockets is fine but you can use netty too (i.e. to resuse some pre-defined handers/codecs.
about the exception in native mode is usually because of a reflective access to a class so you need to add it to the reflective class list

Alright, native tests pass, adding the the reflective classes with an additional dependency camel-endpointdsl was the solution.

This can be closed.

And once again, @dhartford, many thanks for the contribution!

authors: dhartford lburgazzoli oscerd ppalaga

repo: apache/camel-quarkus


text:
add HazelcastReplicatedmapConsumer test fixes apache#2095

CAMEL-16018 being fixed, I've added the integration test for the HazelcastReplicatedmapConsumer

I manually merged this to the camel-master branch.

I manually merged this to the camel-master branch.

thanks @jamesnetherton

thanks @jamesnetherton

authors: jamesnetherton zbendhiba


text:
Removed workaround for strimzi kafka container and J17

fixes https://github.com/apache/camel-quarkus/issues/6097

The failure is my bad, I'll take care of that.

I reworked the fix, verified on FIPS machine, all should be ok


text:
CAMEL-19439 - Import jackson-bom to build and test with a single deterministic set of jackson dependencies

https://issues.apache.org/jira/browse/CAMEL-19439
We currently have jackson2-version at 2.15.1 in camel-parent and it is used in several components. However Jackson artifacts are pulled by many other components as transitive dependencies. The lack of dependency management causes that many different versions of Jackson artifacts are downloaded and used throughout Camel. Also the effective version of Jackson in end user applications depends not only on components used but also on their ordering in the given project's pom.xml file.
Importing jackson-bom solves all the above problems.

@essobedo could you please give a hint what to do with the mvnd failure?

@ppalaga If you check the log file, the failure is due to tests failing in Camel Core:
[camel-core] [INFO] Results:
[camel-core] [INFO]
[camel-core] [ERROR] Failures:
[camel-core] [ERROR]   FileConsumerBeginAndCommitRenameStrategyTest.testRenameSuccess mock://report Body of message: 0. Expected: <Hello Paris> but was: <>
[camel-core] [ERROR]   FileConsumerCommitRenameAbsolutePathStrategyTest.testRenameSuccess mock://report Body of message: 0. Expected: <Hello Paris> but was: <>
[camel-core] [ERROR]   FileConsumerDeleteExchangePooledTest.testDelete mock://result Body of message: 0. Expected: <Hello World> but was: <>
[camel-core] [ERROR]   FileConsumerIdempotentKeyNameAndSizeTest.testIdempotent mock://result Body of message: 0. Expected: <Hello World> but was: <>
[camel-core] [ERROR]   DefaultSupervisingRouteControllerTest.testSupervising:71 expected: <Started> but was: <Stopped>
[camel-core] [INFO]
[camel-core] [ERROR] Tests run: 6122, Failures: 5, Errors: 0, Skipped: 36

authors: essobedo ppalaga

repo: apache/camel


text:
CAMEL-16639: Use symbolic links

With Antora 3 we can now use symbolic links within the git repository, and we no longer need to copy files over to the docs directory. This is a major refactor of how the Gulp is configured not only for creating symbolic links but also removing duplicate functionality and utilizing a configuration object to generate Gulp tasks from instead of manual task generation.
Having the configuration now effectively declarative (look for details in the gulpfile.js) it should be easier to tweak and maintain how the content is managed in the docs directory.
Other relevant changes also included as they were part of the refactoring:

images are now stored in ../images instead of ../assets/images
templates for nav.adoc files have a uniform naming
a summary document (dsl/src/main/docs/dsl.adoc) was added to help group DSL documents in the other category
adds symbolic links for JSON Camel metadata documents, currently disabled (see NOTE in gulpfile.js)

This work takes is largely inspired by the work by David Jencks on #6040
This PR also contains two other commits for fixing grouping attribute. And to keep the PR as small as possible doesn't include a regen commit which would apply the changes in the Gulp build.

I've converted this to draft, I need to investigate why the website build fails with the broken xref's. Could be on my end, but just to make sure this doesn't break the website build.

I'm having quite a bit of weird issues with this, when running locally, that is pointing antora-playbook.yml to my local git repository two issues occur:

_attachments/Hohpe_EIP_camel_20150622.zip is a directory not a ZIP file
source points to my GitHub fork not referenced in the paybook

[17:07:09.934] INFO (asciidoctor): possible invalid reference: MicrometerComponent-counter
    file: docs/components/modules/ROOT/pages/micrometer-component.adoc
    source: https://github.com/zregvart/camel.git (refname: camel-2.x, start path: docs/components)
EISDIR: illegal operation on a directory, open '.../camel-website/documentation/components/latest/eips/_attachments/Hohpe_EIP_camel_20150622.zip'

So I've pushed to a test branch, with regenerated documentation (i.e. I ran yarn gulp in docs), and pointed the playbook to that (instead of main), and that seems to work without issues (still need to run checks).
So unless I'm doing something wrong, and I can't find what, this seems to cause issues when working locally, which would point to a bug in Antora/isomorphic-git?

Sorry, I should have mentioned those problems.

I have to run rm core/camel-core-engine/src/main/docs/modules/eips/attachments/* to get a build against local  sources to succeed. The build works fine against remote repos.
There's a problem with logging that doesn't reference the local worktree source, instead it references the origin repo. I mentioned this one on Zulip but haven't seen a response. https://antora.zulipchat.com/#narrow/stream/282400-users/topic/.5BAntora.203.20a.2E9.5D.20Logger.20appears.20to.20specify.20the.20wrong.20source

Thanks for the hint @djencks, seems that inability to build from a local branch other than main is a preexisting issue. I've tested on a copy of main, i.e. without these changes and received the same error.
I think this is now ready to be merged. I'll leave it open for a bit if anyone else wishes to provide some feedback.

I'm amazed by this naming trick! I still don't understand why it works... What would you think of encapsulating it in a method, like

JavaScript has some interesting properties, I was also trying to create a function that would make this a bit readable, but did not succeed. This is how I tested:
const name='xyz'
const { [name]: f1 } = { [name]: () => void 0 }
const { [name]: f2 } = { [name]: function() {} }

const lambda = (n, f) => {
  const { [n]: ret } = { [n]: f }
  return ret
}

const func = function(n, f) {
  const { [n]: ret } = { [n]: f }
  return ret
}

console.log(f1)
console.log(f2)
console.log(lambda('xyz', () => void 0))
console.log(lambda('xyz', function() {}))
console.log(func('xyz', () => void 0))
console.log(func('xyz', function() {}))

When run it gives:
[Function: xyz]
[Function: xyz]
[Function (anonymous)]
[Function (anonymous)]
[Function (anonymous)]
[Function (anonymous)]

So that halted that effort...

You have to construct the lambda in the closure in order for the name to stick (at least that's my interpretation of the behavior).  So my first suggestion
function nameTask (name, task) {
  const { [name]: x } = { [name]: () => task() }
  return x
}

works because of the new lambda () => task() in the function.  After some more consideration, I like this better:
function nameTask2 (name, task, args) {
  const { [name]: x } = { [name]: () => task(...args) }
  return x
}

called like
    const c = nameTask2(`clean:asciidoc:${type}`, clean, [asciidoc.destination, asciidoc.keep])

or maybe
    allTasks.push(series(
      nameTask2(`clean:asciidoc:${type}`, clean, [asciidoc.destination, asciidoc.keep]),
      nameTask2(`symlink:asciidoc:${type}`, createSymlinks, [asciidoc.source, asciidoc.destination]),
      nameTask2(`nav:asciidoc:${type}`, createNav, [asciidoc.destination] )
    ))

I included the suggestion from @djencks and the changes resulting from running the build in docs. I'm going to merge this and keep an eye on the website build.

authors: djencks zregvart


text:
[TestCase][HOTFIX] Added drop database in beforeEach to avoid exception

Problem: Sometimes while running these test suites in local some test cases are failing with database already exists exception.
Solution: Add drop database command in beforeEach so that each test case can easily create new one.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Can one of the admins verify this patch?

add to whitelist

add to whitelist

add to whitelist

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2730/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2731/

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10992/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2961/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2967/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2737/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2738/

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10999/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2968/

LGTM

authors: CarbonDataQA YatinGudwani brijoobopanna kunal642 qiuchenjian zzcclp

repo: apache/carbondata


text:
[ISSUE-4305] Add @Override for override method

Why is this PR needed?
There are some code style， which should be optimized
What changes were proposed in this PR?
There are some code style， which should be optimized
Does this PR introduce any user interface change?

No

Is any new testcase added?

No, There are some code style， which should be optimized

Build Failed  with Spark 3.1, Please check CI http://159.138.8.58:12602/job/ApacheCarbon_PR_Builder_3.1/763/