text:
Add as_any() to the ObjectStore to make it able to identify which ObjectStore is using for the related trait object

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The context is we are working on introducing a data source cache layer for Ballista & DataFusion https://github.com/apache/arrow-ballista/issues/645. The cache layer will be able to multiple ObjectStore data. And we need to leverage the identification of the ObjectStore.
Describe the solution you'd like

Add as_any() to the ObjectStore to make it able to identify which ObjectStore is using for the related trait object.
Describe alternatives you've considered

Additional context

And we need to leverage the identification of the ObjectStore.

Could you expand a bit on what you mean by this, why does the cache need to know the backing implementation?

authors: tustvold yahoNanJing

repo: apache/arrow-rs


text:
CI Failure in json writer tests after upgrade from lexical-core 1.0.2 to lexical-core 1.0.3

Describe the bug
CI is failing on main, apparently related to the upgrade from lexical-core 1.0.2 to lexical-core 1.0.3
https://github.com/apache/arrow-rs/actions/runs/12216285082/job/34079214477
---- writer::tests::test_write_single_batch stdout ----
thread 'writer::tests::test_write_single_batch' panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lexical-write-integer-1.0.4/src/jeaiii.rs:304:29:
range end index 20 out of range for slice of length 19
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- writer::tests::test_write_multi_batches stdout ----
thread 'writer::tests::test_write_multi_batches' panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lexical-write-integer-1.0.4/src/jeaiii.rs:304:29:
range end index 20 out of range for slice of length 19

---- writer::tests::write_arrays stdout ----
thread 'writer::tests::write_arrays' panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lexical-write-integer-1.0.4/src/jeaiii.rs:304:29:
range end index 20 out of range for slice of length 19

---- writer::tests::write_basic_rows stdout ----
thread 'writer::tests::write_basic_rows' panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lexical-write-integer-1.0.4/src/jeaiii.rs:304:29:
range end index 20 out of range for slice of length 19

failures:
    writer::tests::test_write_multi_batches
    writer::tests::test_write_single_batch
    writer::tests::write_arrays
    writer::tests::write_basic_rows

To Reproduce
# update to get lexical-core 1.0.3
cargo update
cargo test -p arrow-json --lib

Expected behavior
Tests should pass
Additional context

@jayzhan211  observed this downstream in DataFusion: https://github.com/apache/datafusion/issues/13686
@jonahgao  fixed by pinning the lexical core version: https://github.com/apache/datafusion/pull/13689
Upstream ticket filed: https://github.com/Alexhuszagh/rust-lexical/issues/191

I am pretty sure this is a regression in upstream lexical-core
I filed an upstream ticket (with reproducer): https://github.com/Alexhuszagh/rust-lexical/issues/191

https://github.com/Alexhuszagh/rust-lexical/issues/191 has been fixed upstream ❤

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
ci: Fail Miri CI on first failure

Which issue does this PR close?

Closes #5159
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

With just adding the set -e the Miri CI check fails as expected: https://github.com/apache/arrow-rs/actions/runs/7321312775/job/19941544005?pr=5243

Thank you

source: gharchive/pull-request

authors: Jefffrey tustvold


text:
arrow flight sql jdbc drive with Lz4Compression

Describe the usage question you have. Please include as many useful details as  possible.
I'm new to java jdbc and flight. But i want to use flight sql jdbc drive to connect a db.
this is my code
package com.test;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Properties;
import org.apache.arrow.compression.CommonsCompressionFactory;
import org.apache.arrow.compression.Lz4CompressionCodec;

public class App {
    public static void main(String[] args) {
        try {
            // Load the JDBC driver
            Class.forName("org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver");

            String url = "jdbc:arrow-flight-sql://localhost:50050?useEncryption=0";

            Properties p = new Properties();
            p.setProperty("user", "admin");
            p.setProperty("password", "password");
            Connection connection = DriverManager.getConnection(url, p);

            // Create a statement
            Statement statement = connection.createStatement();

            // Execute a query
            String query = "SELECT 1"; // Replace "your_table" with your actual table name
            ResultSet resultSet = statement.executeQuery(query);

            // Process the result set
            while (resultSet.next()) {
                System.out.println(resultSet.getInt(1)); // Adjust the index or use column names depending on your
                                                            // schema
            }

            // Close the resources
            resultSet.close();
            statement.close();
            connection.close();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

and my pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.fusiongrid</groupId>
    <artifactId>arrow-flight-sql-example</artifactId>
    <version>1.0-SNAPSHOT</version>
    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.6.0</version>
                <configuration>
                    <mainClass>com.test.App</mainClass>
                    <executable>java</executable>
                    <arguments>
                        <argument>--add-opens=java.base/java.nio=ALL-UNNAMED</argument>
                        <argument>-Darrow.memory.debug.allocator=true</argument>
                    </arguments>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.apache.arrow</groupId>
            <artifactId>flight-sql-jdbc-driver</artifactId>
            <version>16.0.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.arrow</groupId>
            <artifactId>arrow-compression</artifactId>
            <version>16.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>2.0.13</version>
        </dependency>

        <dependency>
        <groupId>ch.qos.logback</groupId>
        <artifactId>logback-classic</artifactId>
        <version>1.5.5</version>
    </dependency>

    </dependencies>
</project>

when i run it, a error show like that
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
4月 30, 2024 2:38:01 下午 org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.memory.BaseAllocator <clinit>
信息: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
4月 30, 2024 2:38:01 下午 org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.memory.DefaultAllocationManagerOption getDefaultAllocationManagerFactory
信息: allocation manager type not specified, using netty as the default type
4月 30, 2024 2:38:01 下午 org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.memory.CheckAllocator reportResult
信息: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class
java.sql.SQLException: Error while executing SQL "SELECT 1": java.lang.IllegalArgumentException: Please add arrow-compression module to use CommonsCompressionFactory for LZ4_FRAME
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:56)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.createException(Helper.java:41)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:164)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:228)
	at com.test.App.main(App.java:29)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Please add arrow-compression module to use CommonsCompressionFactory for LZ4_FRAME
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
	at org.apache.arrow.driver.jdbc.utils.FlightEndpointDataQueue.next(FlightEndpointDataQueue.java:117)
	at org.apache.arrow.driver.jdbc.utils.FlightEndpointDataQueue.next(FlightEndpointDataQueue.java:155)
	at org.apache.arrow.driver.jdbc.ArrowFlightJdbcFlightStreamResultSet.getNextEndpointStream(ArrowFlightJdbcFlightStreamResultSet.java:260)
	at org.apache.arrow.driver.jdbc.ArrowFlightJdbcFlightStreamResultSet.loadNewFlightStream(ArrowFlightJdbcFlightStreamResultSet.java:131)
	at org.apache.arrow.driver.jdbc.ArrowFlightJdbcFlightStreamResultSet.populateData(ArrowFlightJdbcFlightStreamResultSet.java:146)
	at org.apache.arrow.driver.jdbc.ArrowFlightJdbcFlightStreamResultSet.execute(ArrowFlightJdbcFlightStreamResultSet.java:138)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:669)
	at org.apache.arrow.driver.jdbc.ArrowFlightMetaImpl.prepareAndExecute(ArrowFlightMetaImpl.java:205)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:677)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:157)
	... 2 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Please add arrow-compression module to use CommonsCompressionFactory for LZ4_FRAME
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.arrow.driver.jdbc.utils.FlightEndpointDataQueue.next(FlightEndpointDataQueue.java:106)
	... 11 more
Caused by: java.lang.IllegalArgumentException: Please add arrow-compression module to use CommonsCompressionFactory for LZ4_FRAME
	at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.vector.compression.NoCompressionCodec$Factory.createCodec(NoCompressionCodec.java:69)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:82)
	at org.apache.arrow.driver.jdbc.shaded.org.apache.arrow.flight.FlightStream.next(FlightStream.java:270)
	at org.apache.arrow.driver.jdbc.utils.FlightEndpointDataQueue.lambda$enqueue$2(FlightEndpointDataQueue.java:187)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at org.apache.arrow.driver.jdbc.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1570)

Component(s)
FlightRPC

Duplicated: #41457

authors: Curricane kou

repo: apache/arrow


text:
ARROW-13956: [C++] Add a RETURN_NOT_OK_ELSE_WITH_STATUS to support changing the Status

Signed-off-by: Junwang Zhao zhjwpku@gmail.com

Sorry for the delay. I'm closing as explained on the JIRA.

authors: pitrou zhjwpku


text:
ARROW-2971: [Python] Give some modules in arrow/python more descriptive names

Follow-up work to ARROW-2814. Renaming files only

Codecov Report

Merging #2446 into master will increase coverage by 1.28%.
The diff coverage is 89.11%.

@@            Coverage Diff             @@
##           master    #2446      +/-   ##
==========================================
+ Coverage   85.49%   86.78%   +1.28%
==========================================
  Files         301      241      -60
  Lines       46273    42748    -3525
==========================================
- Hits        39563    37097    -2466
+ Misses       6636     5651     -985
+ Partials       74        0      -74

Impacted Files
Coverage Δ

cpp/src/arrow/python/arrow_to_pandas.cc
88.72% <ø> (ø)
:arrow_up:

cpp/src/arrow/python/python-test.cc
100% <ø> (ø)
:arrow_up:

cpp/src/arrow/python/numpy_to_arrow.cc
94.11% <ø> (ø)
:arrow_up:

cpp/src/arrow/python/deserialize.cc
91.62% <ø> (ø)

cpp/src/arrow/python/decimal.h
100% <ø> (ø)
:arrow_up:

cpp/src/arrow/python/serialize.h
0% <0%> (ø)

cpp/src/arrow/python/python_to_arrow.h
100% <100%> (+100%)
:arrow_up:

cpp/src/arrow/python/serialize.cc
88.88% <88.88%> (ø)

cpp/src/arrow/python/python_to_arrow.cc
89.52% <89.51%> (+0.63%)
:arrow_up:

cpp/src/arrow/compute/kernels/cast.cc
90.82% <0%> (-0.72%)
:arrow_down:

... and 69 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16bbec8...d13016b. Read the comment docs.

+1

authors: codecov-io wesm


text:
GH-37587: [C++] Move integration machinery into its own directory and namespace

Rationale for this change
The JSON integration machinery currently lives in the arrow::testing namespace and the corresponding directory, but it can be compiled and used even when testing is disabled.
What changes are included in this PR?
Move the integration machinery source files into a new arrow/integration directory.
Move the integration APIs into a new arrow::internal::integration namespace.
Are these changes tested?
By existing tests and CI jobs.
Are there any user-facing changes?
No, as these are not supposed to be public-facing facilities.

Closes: #37587

@github-actions crossbow submit -g cpp

The underlying motivation for this is #37537, where I plan to expose some C integration entrypoints from libarrow.so and invoke them from the integration runner.

I'm seeing an unrelated error here:
https://github.com/ursacomputing/crossbow/actions/runs/6099155746/job/16551111386#step:6:1592
[ 92%] Building CXX object src/parquet/CMakeFiles/parquet-file-deserialize-test.dir/file_deserialize_test.cc.o
/arrow/cpp/src/parquet/file_deserialize_test.cc:18:10: fatal error: gmock/gmock.h: No such file or directory
   18 | #include <gmock/gmock.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.

@kou is it a known issue?

I'm seeing an unrelated error here: https://github.com/ursacomputing/crossbow/actions/runs/6099155746/job/16551111386#step:6:1592
[ 92%] Building CXX object src/parquet/CMakeFiles/parquet-file-deserialize-test.dir/file_deserialize_test.cc.o
/arrow/cpp/src/parquet/file_deserialize_test.cc:18:10: fatal error: gmock/gmock.h: No such file or directory
   18 | #include <gmock/gmock.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.

I've opened a new issue for this: #37601

authors: kou pitrou


text:
ARROW-5098: [Website] Update how to install .deb by APT

The APT repository for 0.13.0 is unofficial because we don't use the
voted APT metadata. The voted APT metadata was broken.

@kou I'm trying to update the site, but a lot of file have changed with updated script tag integrity attribute. I suppose that's normal, but I'm asking before committing :)

Could you show the diff?
Or should we go to dev@?

@kszucs it could be caused by a gem version change, if you could open a JIRA or start an email thread about it we can take a closer look

authors: kou kszucs wesm


text:
ARROW-1312: [C++] Make ARROW_JEMALLOC OFF by default until ARROW-1282 is resolved

See ARROW-1282, ARROW-1311 for related discussions.

+1. Appveyor build: https://ci.appveyor.com/project/wesm/arrow/build/1.0.845


text:
FakeJobsService tries to base64 decode a plain QueryString resulting in an error.

https://github.com/apache/beam/blob/fbeae980e93ea64fc8cc3ad074cbc8aebd157691/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/FakeJobService.java#L488
Not Sure if I am doing something wrong setting up the. FakeBigQueryServices
I have a pipeline reading from BigQuery Table select * from project:dataset.table, the contents is the published to a topic on a queue .
The Pipeline is configured with the FakeBigQueryServices and PubsubTestClient to shield of the real interactions with GCP.
After setting up the expected tables with te expected rows, the pipeline in BigquerIO wants to execute this part:
private JobStatus runQueryJob(JobConfigurationQuery query)
      throws IOException, InterruptedException  {
    List<TableRow> rows = FakeBigQueryServices.rowsFromEncodedQuery(query.getQuery());
    datasetService.createTable(new Table().setTableReference(query.getDestinationTable()));
    datasetService.insertAll(query.getDestinationTable(), rows, null);
    return new JobStatus().setState("DONE");
  }

The query from the job is in plain string:  select * from project:dataset.table, but somehow expects a base64 encoded version or encoded results?
resulting in:
com.google.common.io.BaseEncoding$DecodingException: Unrecognized character: 0x20"

What am I missing?

here is the code which illustrates the error:
`package com.kramphub.davinci.pipelines.bigquery.job;
import com.google.api.services.bigquery.model.*;
import com.google.common.collect.ImmutableList;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
import org.apache.beam.sdk.io.gcp.bigquery.BigQueryOptions;
import org.apache.beam.sdk.io.gcp.bigquery.TestBigQueryOptions;
import org.apache.beam.sdk.io.gcp.testing.FakeBigQueryServices;
import org.apache.beam.sdk.io.gcp.testing.FakeDatasetService;
import org.apache.beam.sdk.io.gcp.testing.FakeJobService;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.junit.Rule;
import org.junit.jupiter.api.Test;
import org.junit.rules.TemporaryFolder;
import org.junit.rules.Timeout;
import java.io.IOException;
import java.nio.file.Files;
import java.util.List;
public class TestPipeline {
@Rule
public transient Timeout globalTimeout = Timeout.seconds(60);

private FakeDatasetService fakeDatasetService = new FakeDatasetService();
private FakeJobService fakeJobService = new FakeJobService();
private FakeBigQueryServices fakeBigQueryServices =
        new FakeBigQueryServices()
                .withDatasetService(fakeDatasetService)
                .withJobService(fakeJobService);

@Test
public void testFakeDataService() throws IOException, InterruptedException {
    String testFolder = Files.createTempDirectory("tmpDirPrefix").toFile().getAbsolutePath();

    FakeDatasetService.setUp();

    List<TableRow> data =
            ImmutableList.of(
                    new TableRow().set("name", "a").set("number", 1L),
                    new TableRow().set("name", "b").set("number", 2L),
                    new TableRow().set("name", "c").set("number", 3L),
                    new TableRow().set("name", "d").set("number", 4L),
                    new TableRow().set("name", "e").set("number", 5L),
                    new TableRow().set("name", "f").set("number", 6L));

    String tableSpec = "project:data_set.table_name";
    TableReference table = BigQueryHelpers.parseTableSpec(tableSpec);

    fakeDatasetService.createDataset("project", "data_set", "europe-west1", "", null);
    fakeDatasetService.createTable(
            new Table()
                    .setTableReference(table)
                    .setSchema(
                            new TableSchema()
                                    .setFields(
                                            ImmutableList.of(
                                                    new TableFieldSchema().setName("name").setType("STRING"),
                                                    new TableFieldSchema().setName("number").setType("INTEGER")))));
    fakeDatasetService.insertAll(table, data, null);

    PipelineOptions options = PipelineOptionsFactory.create();
    options.setTempLocation(testFolder);
    BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
    bqOptions.setProject("project");

    Pipeline pipeline = Pipeline.create(options);
    pipeline.apply(BigQueryIO.readTableRows()
            .withTestServices(fakeBigQueryServices)
            .fromQuery(String.format("SELECT * FROM `%s`", tableSpec.replace(":", ".")))
            .withQueryLocation("europe-west1")
            .usingStandardSql()
            .withTemplateCompatibility()
            .withoutValidation()
    );

    // RUN
    pipeline.run().waitUntilFinish();

}

}
/*
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException: Query job beam_bq_job_QUERY_testpipeline0531064239441eccb1_7835b3c830b4431dbfd01ebde163708b_22936c33de6f45798b1b43d18d6c2e1a failed, status: {
"errorResult" : {
"message" : "Job GenericData{classInfo=[copy, dryRun, extract, jobTimeoutMs, jobType, labels, load, query], {query=GenericData{classInfo=[allowLargeResults, clustering, connectionProperties, continuous, createDisposition, createSession, defaultDataset, destinationEncryptionConfiguration, destinationTable, flattenResults, maximumBillingTier, maximumBytesBilled, parameterMode, preserveNulls, priority, query, queryParameters, rangePartitioning, schemaUpdateOptions, tableDefinitions, timePartitioning, useLegacySql, useQueryCache, userDefinedFunctionResources, writeDisposition], {allowLargeResults=true, createDisposition=CREATE_IF_NEEDED, destinationTable=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=temp_dataset_beam_bq_job_QUERY_testpipelinedrutjes0531064239441eccb1_7835b3c830b4431dbfd01ebde163708b, projectId=project, tableId=temp_table_beam_bq_job_QUERY_testpipelinedrutjes0531064239441eccb1_7835b3c830b4431dbfd01ebde163708b}}, flattenResults=true, priority=BATCH, query=SELECT * FROM project.data_set.table_name, useLegacySql=false, writeDisposition=WRITE_TRUNCATE}}}} failed: java.lang.IllegalArgumentException: com.google.common.io.BaseEncoding$DecodingException: Unrecognized character: 0x20"
},
"state" : "FAILED"
}
*/`

This happen to me too, have you found the solution?

authors: DennisRutjes hanalaydrus

repo: apache/beam


text:
TOIL: Update Unified worker image

R: @ananvay
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Python Dataflow V2 ValidatesRunner

Thanks a lot Ankur!
LGTM

Flink error seems to be unrelated.

Note: All the Dataflow Runner V2 tests are passing. yeeee

authors: ananvay angoenka


text:
BEAM-13714 Get rid of unnecessary logs for BigQuery streaming writes

As reported in https://issues.apache.org/jira/browse/BEAM-13714 BigQuery connector logs every batch in streaming mode.
For real-world pipeline it causes Cloud Logging throttling.
As long as BigQuery insertAll API metrics (request count and percentiles for request sizes) are available in StackDriver, logging seems to be redundant, so I decided to remove logging at all.
@nehsyc - I would like to ask you for the review as an author of relevant changes in BigQuery connector.
https://github.com/apache/beam/commit/c1e5b3547b0dc23b4ae4804b410be2d018781470

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @scwhittle
You agree that these logs are superfluous and probably good to remove them to reduce volume?

Yes it is definitely spammy, it could perhaps be kept as a debug log if possibly of interest in debugging how effective batching is. But perhaps some metrics or periodic logging would be better to surface that.

authors: kennknowles mkuthan scwhittle


text:
Apply spotless.

Follow up for https://github.com/apache/beam/pull/16489
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Merging based on > Spotless ("Run Spotless PreCommit") — Build finished.

Whoops, that was my bad. Thanks for running Spotless.

authors: tvalentyn youngoli


text:
[BEAM-1579] Use Batch Replacement API in the Dataflow Runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @jkff

Run Dataflow RunnableOnService

retest this please

Changes Unknown when pulling 29ffe5ee2a30df801730278971e3c74f180a34b2 on tgroh:dataflow_batch_replacement into ** on apache:master**.

This failure is an actual failure: https://github.com/apache/beam/pull/2387 should fix it, once it's complete

Run Dataflow ValidatesRunner

Changes Unknown when pulling 446d6c34e3a35c35c286cff3f9211c37a31bc3ea on tgroh:dataflow_batch_replacement into ** on apache:master**.

run Dataflow ValidatesRunner

Coverage decreased (-0.002%) to 70.132% when pulling 27f1548b57170c61d79c00a5ad7e033d63bd1cc2 on tgroh:dataflow_batch_replacement into 570d0e2a119716dfdfc8b902e07ee3963561a419 on apache:master.

Coverage decreased (-0.1%) to 70.024% when pulling 27f1548b57170c61d79c00a5ad7e033d63bd1cc2 on tgroh:dataflow_batch_replacement into de36e8398ecb37b9826175c4723f94d127e63d03 on apache:master.

authors: coveralls tgroh


text:
[BEAM-1672] Accumulable MetricsContainers.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @bjchambers for Java core sdk changes.
R: @aljoscha for Flink runner related changes.
R: @staslev for Spark runner related changes.

Working on getting this working for direct runner as well.

R: @bjchambers for Java SDK and Direct runner changes.
R: @aljoscha for Flink runner changes.
R: @staslev for Spark runner changes.

Run Spark ValidatesRunner

Run Flink ValidatesRunner

Run Dataflow ValidatesRunner

Coverage increased (+0.2%) to 69.986% when pulling b87a860bf229543e85e899ab7f67d59945624285 on aviemzur:accumulable-metricscontainer into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

Rebased on top of master
Addressed comments in the review.
PTAL: @bjchambers @aljoscha @staslev

Retest this please.

Run Spark ValidatesRunner

Run Flink ValidatesRunner

Run Dataflow ValidatesRunner

Coverage increased (+0.2%) to 69.993% when pulling aeb3000ecbd9f7d2c3d1d67375b0bba3306fd4c5 on aviemzur:accumulable-metricscontainer into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

Retest this please.

Retest this please.

Coverage increased (+0.2%) to 69.993% when pulling aeb3000ecbd9f7d2c3d1d67375b0bba3306fd4c5 on aviemzur:accumulable-metricscontainer into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

Run Dataflow ValidatesRunner

@staslev Addressed your comments in my latest commit.

I'd import the static methods and use asMetricResults, asAttemptedOnlyMetricResults etc. directly, rather than in a qualified manner like so MetricsContainerStepMap.asAttemptedOnlyMetricResults which long-ish.
Other than that the parts I have looked at LGTM.

I looked over the Flink parts again. This LGTM, now that Flink Metrics support is retained. 👍

Run Spark ValidatesRunner

Run Flink ValidatesRunner

Rebased on top of master.
Removed all code changes from direct runner.
Will merge when tests pass.

Coverage increased (+0.2%) to 70.646% when pulling f943e18571985760f720cf047612b0bd42013657 on aviemzur:accumulable-metricscontainer into a629f73ee4e64c470e0c78cc6f51b8625d781b41 on apache:master.

Two test failures of flaky tests, verified they passed locally.

authors: aljoscha aviemzur coveralls staslev


text:
Fix incorrect Work.java cast and logging

Work.java no longer implements the Runnable interface, ExecutableWork does this.
Fix the bad cast and correctly extract work token in BoundedQueueExecutor.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

presubmit failure is unrelated @scwhittle


text:
DSL_SQL: merge master into DSL_SQL feature branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x ] Make sure tests pass via mvn clean verify.
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

I would not do a merge, but a rebase instead. Do you mind if I try on my side ?

This is actually a rebase, not merge, just rebase seems not enough, some code change is also needed. Feel free to try on your side @jbonofre .

authors: jbonofre xumingming


text:
Revert #6752 #6798 #6807 #6837 to fix Dataflow test breakage

This change reverts the following changes since they break Dataflow tests:
72e0e6d710 Merge pull request #6752 [BEAM-5791] Dataflow harness data plane pushback.
2fbc4c3509 Merge pull request #6798: [BEAM-5833] Fix checkstyle breakage
7800c3078d Merge pull request #6807: [BEAM-5833] Fix java-harness build by adding flush() to BeamFnDataWriteRunnerTest
a9aaaace38 Merge pull request #6837 [BEAM-5853] Fix in RemoteGrpcPortWriteOperation

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Noting that these changes are primarily to the Dataflow worker so invoking internal Dataflow testing is a fair point. Even so, it would be better if you could provide the community insight into the problem, and this arrangement should be temporary.

LGTM, approval. I'll try to figure out what's happening.
Whenever the internal tests fail despite all external tests passing, we should take an action item to rectify this situation (if feasible). I suspect using the newly contributed dataflow worker code has insufficient coverage (at least that seems the cause for the last two fixups).

@boyuanzz Has been working on enabling more Dataflow tests and has some PRs that are outstanding.

authors: charlesccychen kennknowles lukecwik robertwb


text:
[BEAM-5462] get rid of .options deprecation warnings in tests

Background:

Split Pipeline and PipelineOptions so make Pipeline more like pure PTransform
Need to access PipelineOptions in PipelineRunner.run_pipeline(self, pipeline) but currently the only accessing point to PipelineOptions is pipeline.options() or pipeline._options
Best possible solution might be adding a new parameter options as PipelineRunner.run_pipeline(self, pipeline, options) however it may cause backward compatibility issues

Proposed solution:
Creating an instance variable options and a method set_options in PipelineRunner. Setting the same options to PipelineRunner in the constructor of Pipeline. Using PipelineRunner.options instead of pipeline.options in PipelineRunner.run_pipeline.
Two possible ways to run a pipeline are:

build Pipeline(runner=..., options=...) and call
pipeline.run()
call pipeline_runner.run(transform, options=...)

This change covers both of the cases and does not introduce
any public API signature change.
There will be two same copies of options in Pipeline and PipelineRunner. Pipeline.options is marked as deprecated so we can motivate users to use PipelineRunner.options instead.

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

run python postcommit

@udim Looks like it's not easy to fix this issue without public API signature changes and this is a proposal for it. Any idea / feedback is welcomed.

@robertwb Any comment?

run python postcommit

run python postcommit

I'm not sure we want to bind pipelines to a runner, e.g. one could imagine using the same runner to multiple pipelines possibly with different options. Instead, the options should be passed into run_pipeline.

Then it would be better to wait until 3.0 release and add an additional parameter to run_pipeline. How about just removing deprecated annotation for now? There's no way to avoid that deprecation warning with the current method structure.

new implementation #7132

Can you update the existing PR instead of opening a new one so that review can be continued at the same location ?
If above is not possible for some reason please close this PR instead of the new one.

@chamikaramj I'll close the new one and update this.

run python postcommit

Thanks Heejong.
Looks like Robert's above comment has been addressed as well. I'll go ahead and merge.

authors: chamikaramj ihji robertwb


text:
LogWatcher - don't change log level if not needed

prevents the logs from missing useful output if testing for a higher-level message,
useful when investigating tests, esp SoftwareProcessStopsDuringStartTest.testStopDuringProvisionWaitsForCompletion where the log messages we want aren't showing because we reset it to warn.
also use try-close syntax for LogWatcher.

thx @aledsage , done and merged

repo: apache/brooklyn-server


text:
Error "Fail to wait EPOLLOUT of fd=...: Connection timed out" when channel timeout is -1

Describe the bug (描述bug)
According to channel.h, setting timeout_ms to -1 will make the Channel block on requests.
However, this fails and shows the following error over and over again (even if no more requests are being made): W1106 15:03:26.337735 3784702 4294969859 /.../brpc/src/brpc/socket.cpp:1361] Fail to wait EPOLLOUT of fd=3: Connection timed out .
However, this does not happen if the IP provided for the channel to connect is "localhost" more precisely, "127.0.0.1".
To Reproduce (复现方法)
For example,

Compile the echo_c++ example provided by brpc (but comment out all logging instructions for easier reading).
Run it like so: ./echo_client --timeout_ms=-1 --server="128.0.0.1:50000".
(I am aware that this IP is not for "localhost" but, again, this fails for IPs that are not "localhost").

Expected behavior (期望行为)
The request should be blocked until an answer is received.
An example of correct behavior is following the example above but run it like so: ./echo_client --timeout_ms=-1 --server="127.0.0.1:50000".
Versions (各种版本)
OS: 5.4.0-187-generic
Compiler: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
brpc: 1.11.0
protobuf: 3.21.6.0
Additional context/screenshots (更多上下文/截图)
Resulting code portion after commenting out logging calls in the echo_c++ example:

You also need to set connect_timeout_ms to -1.

Strangely enough, I was doing that on the project I was actually working on and had the exact same problem. However, it is true that in the example above, setting options.connect_timeout_ms to -1 avoids this problem.
Thanks for the help; I'll see if this was, in fact, a problem on my side.

authors: chenBright vaavaav

repo: apache/brpc


text:
aborting bst push command causes stack trace

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Oct 27, 2017, 18:15
When running

bst push --deps all apps/meta-gnome-apps-tested.bst

And things are failing, perhaps connectivity reasons, I tried retry option once, it failed, and decided to hit ^C on the next prompt.
The result is an ugly unhandled exception:
00:00:00][b1dbab84][ push:apps/accerciser.bst           ] FAILURE Sending artifact
[00:00:08][b1dbab84][ push:apps/accerciser.bst           ] FAILURE Pushing Artifact
[00:00:08][b1dbab84][ push:apps/accerciser.bst           ] FAILURE gnome/apps-accerciser/b1dbab84-push.19019.log

    Failed to push artifact gnome/apps-accerciser/b1dbab84ef8722a28a2062314171155f16e8f6a954395e25dc52bcb6ee4bb003: Expected reply, got none

        PKG_CONFIG_PATH: /usr/local/lib/pkgconfig:/usr/lib/pkgconfig
        SHELL: /bin/sh
        SOURCE_DATE_EPOCH: '1320937200'
        TERM: dumb
        TZ: UTC
        USER: tomjon
        USERNAME: tomjon
        V: '1'
    [--:--:--] START   [apps/accerciser.bst]: Pushing Artifact
    [--:--:--] START   [apps/accerciser.bst]: Preparing compressed archive
    [00:00:07] SUCCESS [apps/accerciser.bst]: Preparing compressed archive
    [--:--:--] START   [apps/accerciser.bst]: Sending artifact
    pushreceive: INFO: Executing ssh -l artifacts -p 10007 gnome7.codethink.co.uk bst-artifact-receive --verbose artifacts
    pushreceive: INFO: Receiving repository information
    ssh: connect to host gnome7.codethink.co.uk port 10007: Connection refused
    [00:00:00] FAILURE [apps/accerciser.bst]: Sending artifact
    [00:00:08] FAILURE [apps/accerciser.bst]: Pushing Artifact
    [00:00:08] FAILURE [apps/accerciser.bst]: Push

        Failed to push artifact gnome/apps-accerciser/b1dbab84ef8722a28a2062314171155f16e8f6a954395e25dc52bcb6ee4bb003: Expected reply, got none

[--:--:--][d621e192][ main:apps/evolution.bst            ] STATUS  Push suspending
[--:--:--][7a00f6e6][ main:apps/ghex.bst                 ] STATUS  Push suspending
[--:--:--][6e4be59e][ main:apps/libgit2-glib.bst         ] STATUS  Push suspending

Push failure on element: apps/accerciser.bst

Choose one of the following options:
  continue  - Continue queueing jobs as much as possible
  quit      - Exit after all ongoing jobs complete
  terminate - Terminate any ongoing jobs and exit
  retry     - Retry this job
  log       - View the full log file

Pressing ^C will terminate jobs and exit

Choice: [continue]: ^C

Terminating all jobs

[--:--:--][d621e192][ main:apps/evolution.bst            ] STATUS  Push resuming
[--:--:--][7a00f6e6][ main:apps/ghex.bst                 ] STATUS  Push resuming
[--:--:--][6e4be59e][ main:apps/libgit2-glib.bst         ] STATUS  Push resuming
[--:--:--][d621e192][ main:apps/evolution.bst            ] STATUS  Push terminating
[--:--:--][7a00f6e6][ main:apps/ghex.bst                 ] STATUS  Push terminating
[--:--:--][6e4be59e][ main:apps/libgit2-glib.bst         ] STATUS  Push terminating
[--:--:--][d2fca14e][ main:apps/gitg.bst                 ] STATUS  Push terminating
Process Process-13:
Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/ostreecache.py", line 352, in push
    [ref, weak_ref], output_file)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/pushreceive.py", line 675, in push
    pusher.run()
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/pushreceive.py", line 459, in run
    args = self.reader.receive_info()
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/pushreceive.py", line 266, in receive_info
    cmdtype, args = self.receive([PushCommandType.info])
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/pushreceive.py", line 260, in receive
    raise PushException('Expected reply, got none')
buildstream._artifactcache.pushreceive.PushException: Expected reply, got none

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/job.py", line 265, in child_action
    result = self.action(element)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/pushqueue.py", line 39, in process
    return element._push()
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1173, in _push
    return self.__artifacts.push(self)
  File "/codethink/GNOME/buildstream/buildstream/_artifactcache/ostreecache.py", line 354, in push
    raise _ArtifactError("Failed to push artifact {}: {}".format(ref, e)) from e
buildstream.exceptions._ArtifactError: Failed to push artifact gnome/apps-evolution/d621e19297b2df27ed9607cb0a355a7dafcab6f92a5417bc6bbd78ad48666fab: Expected reply, got none

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1289, in _logging_enabled
    yield self.__log_path
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/job.py", line 281, in child_action
    self.child_shutdown(1)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/job.py", line 314, in child_shutdown
    sys.exit(exit_code)
SystemExit: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/codethink/GNOME/buildstream/buildstream/_scheduler/job.py", line 297, in child_action
    logfile=filename)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1291, in _logging_enabled
    self.__log_path = None
  File "/codethink/GNOME/buildstream/buildstream/_signals.py", line 37, in terminator_handler
    terminator()
  File "/codethink/GNOME/buildstream/buildstream/element.py", line 1282, in flush_log
    .format(action_name, self.name))
ValueError: I/O operation on closed file.
^C^C

[00:00:29][9ac8d8be][ main:apps/meta-gnome-apps-tested.bst] FAILURE Push failed
ERROR:

In GitLab by [Gitlab user @tlater] on Nov 13, 2017, 15:53
created branch 138-aborting-bst-push-command-causes-stack-trace

In GitLab by [Gitlab user @cs-shadow] on Jun 7, 2018, 21:42
changed the description

In GitLab by [Gitlab user @coldtom] on Jun 20, 2018, 18:05
assigned to [Gitlab user @coldtom]

In GitLab by [Gitlab user @coldtom] on Jun 21, 2018, 09:06
[Gitlab user @cs]-shadow What changes did you make to the description?

In GitLab by [Gitlab user @cs-shadow] on Jun 21, 2018, 11:22
[Gitlab user @coldtom] nothing major, I just fixed the formatting of ^C, which was previously rendering as a superscript (^C). Sorry if I caused more confusion than before :)

In GitLab by [Gitlab user @coldtom] on Jun 21, 2018, 11:29
Thanks, I was hoping it was just a cosmetic change :)

In GitLab by [Gitlab user @coldtom] on Jun 21, 2018, 17:53
created branch 138-aborting-bst-push-command-causes-stack-trace-2

In GitLab by [Gitlab user @coldtom] on Jun 21, 2018, 17:54
mentioned in merge request !513

In GitLab by [Gitlab user @coldtom] on Jun 21, 2018, 18:16
created branch 138-aborting-bst-push-command-causes-stack-trace-3

In GitLab by [Gitlab user @coldtom] on Jun 21, 2018, 18:16
mentioned in merge request !514

In GitLab by [Gitlab user @coldtom] on Jul 2, 2018, 17:27
I could not reproduce the behaviour despite my best efforts, and as a result it was recommended this issue be closed or set to needs info for the time being.

In GitLab by [Gitlab user @coldtom] on Jul 2, 2018, 17:27
unassigned [Gitlab user @coldtom]

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jul 2, 2018, 17:44
Thanks for the update. After reading the comments on your associated MR that says:
'In this case, I think we really need to reproduce if we're going to fix. Otherwise, I think we should put this to sleep, closed in a NEEDINFO state in case it happens again.'
Then I think the action here is to close the issue (and then the MR) and add the label 'NeedInfo'. [Gitlab user @tristanvb] what do you think?

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jul 2, 2018, 17:44
assigned to [Gitlab user @tristanvb]

In GitLab by [Gitlab user @juergbi] on Jul 10, 2018, 08:30
mentioned in commit a615e00185b097a3ff3ce15f58e79a803339d9f8

In GitLab by [Gitlab user @juergbi] on Jul 10, 2018, 08:51
mentioned in commit 76cbcd4ec52c26fc77134533df64c64221610e22

In GitLab by [Gitlab user @juergbi] on Jul 10, 2018, 17:06
mentioned in commit 8c7ab2f0a3582cde19d5be7f293b47bfa51efbc4

In GitLab by [Gitlab user @juergbi] on Jul 10, 2018, 18:12
mentioned in commit 1792150b63971e2c4335f785cb9c7da1243cfcc4

In GitLab by [Gitlab user @juergbi] on Jul 16, 2018, 15:33
mentioned in commit cb79b711f0c653c10d78e44f3154eee4f9d123b9

In GitLab by [Gitlab user @juergbi] on Jul 16, 2018, 17:24
mentioned in commit fdbc1e33f84fd45f6ac4527846fc3cc8a14d009b

In GitLab by [Gitlab user @juergbi] on Jul 17, 2018, 06:58
mentioned in commit 1f8b4aa290a908a697f008a29ea143a9320dd639

In GitLab by [Gitlab user @juergbi] on Jul 17, 2018, 08:59
closed via merge request !337

repo: apache/buildstream


text:
Fix #4193 to ban com.github.stephenc.jcip:jcip-annotations

I am working on reusing banned dependencies XML files distributed by Quarkus. Merging this would conflict, so let me adopt this change in my branch.

Replaced by https://github.com/apache/camel-quarkus/pull/4441

authors: ppalaga zhfeng

repo: apache/camel-quarkus


text:
CAMEL-10398 SJMS - Avoid NPE when neither of ConnectionResource nor C…

…onnectionFactory is configured

@igarashitm Thanks for the contribution!
The PR has been merged. Can you close this?

Thanks! closing

authors: igarashitm oscerd

repo: apache/camel


text:
groupId for camel-servlet-starter was incorrect

The camel-servlet-starter artifact is present in groupId org.apache.camel and in org.apache.camel.springboot. However in org.apache.camel.springboot only release 3.0.0 is present. In org.apache.camel all versions including 3.0.0 are present. For the documentation to provide a consistent dependency for all releases, groupId org.apache.camel seems a better fit. Another solution would be to mention two distinct dependencies in the documentation, depending on the version; maybe for the future versions you want to make org.apache.camel.springboot the groupId and keep org.apache.camel for the old ones.

The documentation is up to date with the current master so it must be as is. This PR is invalid. Thanks anyway.

This is mentioned in migration guide too

In top right corner on website you can switch to 2.x version. By default is loaded 3.0.0 (latest) for which is groupId org.apache.camel.springboot correct.

You're correct, I did not check the migration guide and also did not check the documentation for 2.x and 3.x and look at the difference there. My apologies for this pull request and thank you for explaining!

authors: MaartenSmeets bedlaj oscerd


text:
CAMEL-8185: Changed parameter of scr reference event methods

Modified the default bind/unbind scr reference method parameter from a ComponentResolver to a ServiceReference to avoid the possibility of an argument type mismatch exception. This can occur if multiple versions of Camel are installed (such as JBoss Fuse 6.1) and have registered their own versions of services within the OSGi ServiceRegistry.
Since these methods do not utilize the ComponentResolver themselves, moving to a ServiceReference does not change the functionality of the component and the end user can define bind/unbind methods of their own.

Thanks for this PR. It has been merged. Do you mind closing this.


text:
[CAMEL-13812] Split FAQ into separate module under user-manual

Created a separate module for FAQ under docs/user-manual/modules.
Added nav.doc for FAQ, and moved all related pages in docs/user-manual/modules/FAQ/pages.
Replaced links to and from the FAQ module in the entire camel repository. I'm not sure how to run link checker locally.

I hope these changes make more sense now. I have made them in accordance with @djencks's comments on the JIRA issue

I have often failed to run a complete link-check locally :-)
What I do is:

in my local camel-website checkout:

have an antora-playbook-local.yml where I've modified the source urls for the parts I'm working on to point to my local checkout(s).
I've been working on 3 branches of camel: latest, 2.x, and 3.0.x. For each issue I've created branches off each of the official branches.
In order to be able to see all 3 branches "at once" I have one 'real' or full camel checkout for my 'master-issue-number' branch and used 'git worktree' to check out my other two branches into separate source trees so I can edit them all at once.
Since Antora does not yet support multiple work trees, the urls for all these branches have to point to my "main" checkout.  I use HEAD for the main checkout of master, and the actual branch names for the other branches.  This means that although Antora will see even uncommitted changes in HEAD, I have to (locally) check in the changes to the other two branches.

I also update the package.json scripts to use my alternate playbook.

Then I can run yarn run build or other scripts, or mvn clean install.
Here's what the content section of my local playbook looks like, from the last issue I worked on:
content:
  sources:
    - url: ./../camel
      branches: HEAD
      start_path: docs/user-manual
# eip
    - url: ./../camel
      branches: HEAD
      start_path: core/camel-core-engine/src/main/docs
# languages
    - url: ./../camel
      branches: HEAD
      start_path: core/camel-core-languages/src/main/docs
    - url: ./../camel
      branches: HEAD
      start_path: core/camel-xml-jaxp/src/main/docs
    - url: ./../camel
      branches: camel-3.0.x-issue-14782-task-14784
      start_path: core/camel-base/src/main/docs
    - url: ./../camel
      branches: camel-3.0.x-issue-14782-task-14784
      start_path: core/camel-jaxp/src/main/docs
    - url: ./../camel
      branches: camel-2.x-issue-14782-task-14791
      start_path: camel-core/src/main/docs
# main components doc
    - url: ./../camel
      branches: HEAD, camel-2.x-issue-14782-task-14791, camel-3.0.x-issue-14782-task-14784
      start_path: docs/components

    - url: git@github.com:apache/camel-k.git
      branches: master
      start_path: docs
    - url: git@github.com:apache/camel-quarkus.git
      branches: master
      start_path: docs
    - url: git@github.com:apache/camel-kafka-connector.git
      branches: master
      start_path: docs
    - url: git@github.com:apache/camel-spring-boot.git
      branches: master
      start_path: docs

Hey, thanks! I have tried to check as many links as possible via the website. The links across different modules in user-manual and between different components (like user-manual and components) seem to work fine. I will run a more comprehensive check as you mentioned above, but I am only dealing with the master branch for now, I haven't considered other branches. Is that why the PR build is failing?

The xref changes look appropriate to me: I looked at a few but certainly not all.
The PR builds always failed for me too, and it looks to me as if the failure has nothing to do with your work.
I would REALLY prefer you rename the module to faq rather than FAQ.  ROOT is special, otherwise the usual convention is to use lowercase or camel-case names for modules.  Having FAQ rather than faq in a url looks very peculiar.

Yes, I noted that from your comment on the JIRA issue. I'll update the name in a bit!

@djencks I hope the changes are fine now. I initially thought that since FAQ is an acronym it should be in uppercase, but I understand your point. Thanks for making me aware!

@zregvart @djencks any verdict on this PR?

The changes FAQ >> faq all look good to me, although I didn't build the PR myself to check it.
Someone sometime should remove the docs/user-manual/fix-eip-xrefs.sh script I added accidentally :-)
I wonder if the three files linked to in the faq section of the nav that aren't in the faq module should be: this would be better judged by someone more familiar with camel than I am.
We're definitely going to need fixes for the 2.x and 3.0.x branches, and I'm fine with applying this now as long as the other branches are fixed soon.

Hey! I think I'll delete those extra files if they're not needed. I would be happy to make the changes for both the other branches once this is merged - I think we will also get to know if there were any issues with this PR once we merge?

Also, even I checked for those three files. Turn out, those three files are linked from other sections in the menu as well - languages under DSL, support under Community etc, which is probably why they are kept with others of their kind.

To be clear, there are several active branches, but these changes only need to be made on these two - 3.0.x and 2.x - branches?

To be clear, there are several active branches, but these changes only need to be made on these two - 3.0.x and 2.x - branches?

I think that we should make them on 2.x and 3.2.x that's going to get released really soon (but won't include this). Backporting to 3.2 should be fairly easy just cherry-picking changes from here (we'll squash them) so it's easier.
That way on the website we build 2.x, 3.2.x (latest release), and master (latest version).
I'm just testing this locally to see for any issues, if I don't find any (other than the expected breaking of 2.x 3.0.x) then we can proceed and merge this.

I'm seeing errors like this one:
[INFO] --- camel-package-maven-plugin:3.2.0-SNAPSHOT:xref-check (xref-check) @ docs ---
[ERROR] Unresolved xrefs:
[ERROR]   manual:ROOT:nav.adoc (/home/zregvart/tmp/camel/docs/user-manual/modules/faq/nav.adoc) at line 2: xref:can-i-get-commercial-support.adoc[Can I get commercial support?]

Reported by the camel-package-maven-plugin:xref-check plugin, if I run node_modules/@antora/cli/bin/antora --generator @antora/xref-validator antora-playbook-local-xref-check.yml no errors are reported. I'll create an issue to follow-up in the camel-package-maven-plugin.

xref:can-i-get-commercial-support.adoc link is present in user-manual/modules/faq/nav.adoc and linked file is present at /user-manual/modules/faq/pages/can-i-get-commercial-support.adoc. Both nav.adoc and can-i-get-commercial-support.adoc are in the same folder faq, so xref link should be fine right? The links worked fine when I checked on the website..

I'm trying to build the website with this playbook
content:
  sources:
    - url: git@github.com:aashnajena/camel.git
      branches: split-faq
      start_path: docs/user-manual
# eip
    - url: git@github.com:aashnajena/camel.git
      branches: split-faq
      start_path: core/camel-core-engine/src/main/docs
# core languages
    - url: git@github.com:aashnajena/camel.git
      branches: split-faq
      start_path: core/camel-core-languages/src/main/docs
    - url: git@github.com:aashnajena/camel.git
      branches: split-faq
      start_path: core/camel-xml-jaxp/src/main/docs
    - url: git@github.com:apache/camel.git
      branches: camel-3.0.x
      start_path: core/camel-base/src/main/docs
    - url: git@github.com:apache/camel.git
      branches: camel-3.0.x
      start_path: core/camel-jaxp/src/main/docs
    - url: git@github.com:aashnajena/camel.git
      branches: split-faq-2.x
      start_path: camel-core/src/main/docs
# main components/dataformats doc
    - url: git@github.com:aashnajena/camel.git
      branches: split-faq, split-faq-2.x, split-faq-3.2.x
      start_path: docs/components

But I'm getting some errors saying Duplicate nav: latest@components:ROOT:nav.adoc
What could be the possible reason for this error? I'm trying to mimic the original playbook with my branches.

I think we're very close to getting this merged and the other two pull requests for 3.2.x and 2.x branches. Here is the playbook I'm using to test with, the difference with the one @aashnajena posted is that I removed 3.0.x bits, as we won't be building those for the website.
antora-playbook.yml.txt

xref:can-i-get-commercial-support.adoc link is present in user-manual/modules/faq/nav.adoc and linked file is present at /user-manual/modules/faq/pages/can-i-get-commercial-support.adoc. Both nav.adoc and can-i-get-commercial-support.adoc are in the same folder faq, so xref link should be fine right? The links worked fine when I checked on the website..
Can you please help me understand what's the error here?

I'm not seeing this, so not sure, what is here should work.

xref:can-i-get-commercial-support.adoc link is present in user-manual/modules/faq/nav.adoc and linked file is present at /user-manual/modules/faq/pages/can-i-get-commercial-support.adoc. Both nav.adoc and can-i-get-commercial-support.adoc are in the same folder faq, so xref link should be fine right? The links worked fine when I checked on the website..
Can you please help me understand what's the error here?

I'm not seeing this, so not sure, what is here should work.

Yes I figured it out, everything seems to work fine for me now. Added all requested changes in this PR as well as for 3.2.x

@aashnajena we need to move this file:
docs/user-manual/modules/ROOT/examples/core/camel-management/src/test/java/org/apache/camel/management/ManagedCustomProcessorTest.java
to
docs/user-manual/modules/faq/examples/core/camel-management/src/test/java/org/apache/camel/management/ManagedCustomProcessorTest.java
We also need this change in the docs/gulpfile.js:
diff --git a/docs/gulpfile.js b/docs/gulpfile.js
index 95e79e44f1d..7aa5c3c4415 100644
--- a/docs/gulpfile.js
+++ b/docs/gulpfile.js
@@ -239,7 +239,7 @@ const extractExamples = function(file, enc, next) {
 }

 function deleteExamples(){
-    return del(['user-manual/modules/ROOT/examples/', 'components/modules/ROOT/examples/']);
+    return del(['user-manual/modules/ROOT/examples/', 'user-manual/modules/faq/examples/', 'components/modules/ROOT/examples/']);
 }

 function createUserManualExamples() {
@@ -248,6 +248,12 @@ function createUserManualExamples() {
         .pipe(dest('user-manual/modules/ROOT/examples/'));
 }

+function createFAQExamples() {
+    return src('user-manual/modules/faq/**/*.adoc')
+        .pipe(through2.obj(extractExamples))
+        .pipe(dest('user-manual/modules/faq/examples/'));
+}
+
 function createEIPExamples() {
     return src('../core/camel-core-engine/src/main/docs/modules/eips/**/*.adoc')
         .pipe(through2.obj(extractExamples))
@@ -271,7 +277,7 @@ const symlinks = parallel(
     series(deleteComponentImageSymlinks, createComponentImageSymlinks)
 );
 const nav = parallel(createComponentNav, createComponentDataFormatsNav, createComponentLanguagesNav, createEIPNav);
-const examples = series(deleteExamples, createUserManualExamples, createEIPExamples, createUserManualLanguageExamples, createComponentExamples);
+const examples = series(deleteExamples, createUserManualExamples, createFAQExamples, createEIPExamples, createUserManualLanguageExamples, createComponentExamples);

 exports.symlinks = symlinks;
 exports.nav = nav;

@zregvart @djencks can you have a look?

Looks good to me.  It would be good to squash into one commit.  Lets get this in soon before it needs to be rebased:-)
Nice work!

I'm doing one final round of testing with this and the two other PRs. I'll update shortly.

Should I squash the commits now?

Squashed commits into one. I'll do this for the other branches as well.

Thanks!

authors: aashnajena djencks oscerd zregvart


text:
CASSANDRA-18662: Fix cassandra-analytics-core-example

This commit fixes the SampleCassandraJob available under the cassandra-analytics-core-example subproject.

Closed via https://github.com/apache/cassandra-analytics/commit/69766bca399cc779e0f2f8e859e39f7e29a17b7a

repo: apache/cassandra-analytics


text:
Console proxy timeouts when there are multiple subnets

ISSUE TYPE

Bug Report

COMPONENT NAME

Console Proxy (NoVNC)

CLOUDSTACK VERSION

4.14 onwards

CONFIGURATION

Advanced networking
Multiple subnets for management cidr (172.30.34.* and 172.30.36.*)
OS / ENVIRONMENT

Ubuntu 16 hosts
SUMMARY

The console for a vm timeouts when there are multiple subnets configured.
If we try to access the console of the vm which is running on the host with primary subnet then everything works fine.
If we try to open the console of the vm which is running on the host with second subnet then the console hangs
Looks like console proxy creates a static route for the second subnet which breaks the connection
Moving the vm back to the hypervisor which has the primary subnet, fixes the issue.
This issue was not present in cloudstack 4.7 version but the issues exists in 4.14 onwards. It probably might exist in 4.11 also
STEPS TO REPRODUCE

1. Configure multiple subnets for the management cidr (172.30.34.* and 172.30.26.*)
2. In our case few hypervisors had 172.30.34* subnet configured and few others had 172.30.36.*
3. Here 172.30.34.* is the main subnet and 172.30.26.* is second subnet
4. Deploy a VM on host with 172.30.34* and 172.30.36.* subnets
5. open the console for the console proxy vm
6. Now try to open the console for the vm running on host with subnet 172.30.34.* This works fine
7. Now open the console for the vm running on the host with 172.30.36.* as the subnet
8. The console hangs

Below is the routing table

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         37.48.69.14     0.0.0.0         UG    0      0        0 eth2
10.0.0.0        172.30.34.254   255.0.0.0       UG    0      0        0 eth1
37.48.69.0      0.0.0.0         255.255.255.240 U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.16.0.0      172.30.34.254   255.240.0.0     UG    0      0        0 eth1
172.30.34.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
172.30.36.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
172.30.36.3     172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.25    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.37    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.42    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.56    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
192.168.0.0     172.30.34.254   255.255.0.0     UG    0      0        0 eth1

Below are the Ip address of the hosts where vm is running and these routes are created the moment we tried to access console of that vm.

So these routes should not be created. When we delete this route manually, it works fine.

If we do a refresh, these routes are added again and it breaks

172.30.36.3     172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.25    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.37    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.42    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1
172.30.36.56    172.30.34.254   255.255.255.255 UGH   0      0        0 eth1

EXPECTED RESULTS

The console for all vm's running on all hosts should work fine without any timeout

ACTUAL RESULTS

The console for the vm's running on the hosts with secondary subnet timeouts/hangs

@weizhouapache @rhtyd have you guys faces this issue?

@ravening is it related to your pr https://github.com/apache/cloudstack/pull/4678 ?

@ravening is it related to your pr #4678 ?

@weizhouapache no... because that pr adds the cidr to the route but not the specific ip address.
if you see the route -n output, we can see that only specific ip routes are added when we access the console

@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under CsRoute.py cs/CsStaticRoutes.py but didnt see that in /var/log/cloud.log

@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under CsRoute.py cs/CsStaticRoutes.py but didnt see that in /var/log/cloud.log

@ravening
it is added by java
    public void ensureRoute(String address) {
        if (_localgw != null) {
            if (s_logger.isDebugEnabled())
                s_logger.debug("Ensure route for " + address + " via " + _localgw);

            // this method won't be called in high frequency, serialize access
            // to script execution
            synchronized (this) {
                try {
                    addRouteToInternalIpOrCidr(_localgw, _eth1ip, _eth1mask, address);
                } catch (Throwable e) {
                    s_logger.warn("Unexpected exception while adding internal route to " + address, e);
                }
            }
        }
    }

@weizhouapache so the console works for initial 30 seconds and then freezes.
the static route is added by python scripts or the java code? I added some logs under CsRoute.py cs/CsStaticRoutes.py but didnt see that in /var/log/cloud.log

@ravening
I think you could consult networking engineers how to solve this issue in switch configurations

@weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also

Never seen/heard it @ravening

@weizhouapache we checked the code you mentioned yesterday but that was not modified from long time. so the same issue should have been in 4.7 also

@ravening
the issue might exist with legacy vnc console and new novnc console introduced in 4.14
ConsoleProxy.ensureRoute(uri.getHost());

You did not see the issue because you did not use legacy vnc console but use a different novnc implementation.
I suggest you to fix it by network configuration changes.

Hi @ravening is this issue still valid?

@nvazquez no.. I will close this

authors: nvazquez ravening rhtyd weizhouapache

repo: apache/cloudstack