text:
Not reading parquet statistics correctly

I have a dataset generated by spark which shows it has statistics for the string column when using the java parquet-mr code (shown by using parquet-tools meta) however reading from pyarrow shows that the statistics for that column are not set.  I should not the column only has a single value, though it still seems like a problem that pyarrow can't recognize it (it can recognize statistics set for the long and double types).
See metadata.zip for file example.
Pyarrow Code To Check Statistics:
from pyarrow import parquet as pq

meta = pq.read_metadata('/tmp/metadata.parquet')
# No Statistics For String Column, prints false and statistics object is None
print(meta.row_group(0).column(1).is_stats_set)

Example parquet-meta output:
file schema: spark_schema
--------------------------------------------------------------------------------
int:         REQUIRED INT64 R:0 D:0
string:      OPTIONAL BINARY O:UTF8 R:0 D:1
float:       REQUIRED DOUBLE R:0 D:0

row group 1: RC:8333 TS:76031 OFFSET:4
--------------------------------------------------------------------------------
int:          INT64 SNAPPY DO:0 FPO:4 SZ:7793/8181/1.05 VC:8333 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 100, num_nulls: 0]
string:       BINARY SNAPPY DO:0 FPO:7797 SZ:1146/1139/0.99 VC:8333 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE ST:[min: hello, max: hello, num_nulls: 4192]
float:        DOUBLE SNAPPY DO:0 FPO:8943 SZ:66720/66711/1.00 VC:8333 ENC:PLAIN,BIT_PACKED ST:[min: 0.0057611096964338415, max: 99.99811053829232, num_nulls: 0]

I realize the column only has a single value though it still seems like pyarrow should be able to read the statistics set. I made this here and not a JIRA since I wanted to be sure this is actually an issue and there wasnt a ticket already made there (I couldnt find one but I wanted to be sure). Either way I would like to understand why this is

I migrated this to JIRA: https://issues.apache.org/jira/browse/ARROW-2800
I just created a wiki page for Python Parquet support: https://cwiki.apache.org/confluence/display/ARROW/Python+Parquet+Format+Support. You can add issues to the JIRA filter by adding the "parquet" label to Arrow issues
Please feel free to add anything else to the wiki page you like (you'll have to create an account)

authors: rgruener wesm

repo: apache/arrow


text:
[Java] Incorrect calculation of data buffer length in BufferImportTypeVisitor for slice of string array

Describe the bug, including details regarding any error messages, version, and platform.
This bug is found during debugging the issue https://github.com/apache/datafusion-comet/issues/540.
We found that some string arrays' offsets are out of the range of their value buffer. I.e., a string array's value buffer is only 147456 bytes, but the offsets of the last string is (294894, 294912).
The string is output from DataFusion aggregation operator AggregateExec. When producing the output batch, the operator will possibly slice the output batch if it is larger than a configured size. The slice of a string array in arrow-rs, keeps original value buffer and moves the pointer of offset buffer so it is a zero-copy slice.
During importOffsets call in BufferImportTypeVisitor, the slice of offset can be imported correctly as it uses the moved pointer and calculates the offset buffer correctly.
But when it goes to import value buffer, it calculates the capacity of it by using the difference between imported last offset and first offset. Because the imported offsets are from the slice, the calculated capacity is only for the certain slice of the value buffer.
For example, the original string array's value buffer is 346536 bytes, last offset is 346536. We take a slice of 8192 strings from it. The slice array's last offset is 294912 but the value buffer is the same (346536 bytes).
When BufferImportTypeVisitor imports the slice, the imported offsets are [147456, ..., 294912]. It calculates the length of value buffer is 294912 - 147456 = 147456. But actually the length of value buffer is 346536.
Obviously the offsets are now out of range of the incorrect value buffer size 147456.
Actually seems either ArrayImporter or BufferImportTypeVisitor doesn't consider offset in ArrowArray.
Component(s)
Java

Ah, thanks for the explanation.
In general Arrow Java does not support offset in the same way that C++ and Rust do (this is purely a fiction of their implementations and not part of IPC, though now C Data forces Java to deal with it). But we can correct the problem here at least
CC @vibhatha

I will take a look.

To be clear, the source of the issue comes from https://github.com/apache/arrow-rs/issues/5896 where it exports moved pointer of offer buffer of a slice of string array. Instead, we should use offset field in ArrowArray for it. We are going to fix it in arrow-rs for that.
But actually seems either ArrayImporter or BufferImportTypeVisitor also doesn't consider offset in ArrowArray.

Ah, thanks. Yes, either way, we need to consider offset in Java and figure out how best to deal with it.

Ah, thanks. Yes, either way, we need to consider offset in Java and figure out how best to deal with it.

Yes. I'm going to fix arrow-rs to have correct offset field and we also need to handle offset in Java to make it work.

@lidavidm @bkietz I have been working on a solution for this issue in the Java side, but I later realized that we don't have any C data interface tests to evaluate something like slicing.

One easy method to support importing sliced arrays in Java would be to import all arrays as if they had offset 0, then use splitAndTransfer() to effect the slicing after import is complete. IIUC, since importing is zero copy and splitAndTransfer() is too (when possible) this should be about as efficient as handling the slice offset directly

If implementations can handle offset buffers that do not start at zero, then C data arrays with offsets should be possible to import following one of these two cases:

For fixed-width types, slice the value buffer starting at offset*width bytes.
For binary or list types, instead slice the offset buffer at offset bytes.

The latter means that offset buffers won't necessarily start at zero, which not all implementations support right now. This is something we fixed recently in Go, also related to datafusion's shared buffer usage: #41993.
If implementations can support this case, then implementing C data offsets should become simpler.

source: gharchive/issue

authors: bkietz joellubi lidavidm vibhatha viirya

license_type: permissive

license_source: github-api


text:
[Python] How to use fs.FileSystem.from_uri with Azurite

Describe the usage question you have. Please include as many useful details as  possible.
I want to use fs.FileSystem.from_uri with azurite.
Here a short sample of code:
import pyarrow.fs as fs
local_new, path_new = fs.FileSystem.from_uri("abfs://myaccount.blob.core.windows.net/mycontainer")
local_new.create_dir("test")

The error I received is:
az: error: unrecognized arguments: --scope https://storage.azure.com/.default
How can I set account_key to be able to use FileSystem.from_uri?
Component(s)
Python

The feature was removed by #44220.
Could you use an environment variable?
AZURE_PASSWORD may work: https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.environmentcredential?view=azure-python
BTW, the error message is strange. Our Azure file system implementation doesn't use the az command.

I had to install az because otherwise I received an error because az was not here.
I tried by setting AZURE_PASSWORD, AZURE_STORAGE_ACCOUNT or AZURE_STORAGE_KEY (https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-data-operations-cli#set-environment-variables-for-authorization-parameters) but it didn't work.
I tried also setting query parameter credential_kind to environment but I got
terminate called after throwing an instance of 'Azure::Core::Credentials::AuthenticationException'
  what():  EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Aborted

Do you have any example to test FileSystem.from_uri with Azurite?

I tested using fs.AzureFileSystem and it works. But I need to use from_uri so I don't knwo how to use it.

authors: fabricebaranski kou


text:
[C++] Implement ArraySpan support for Variadic Types

Describe the enhancement requested
While investigating https://github.com/apache/arrow/issues/44336 I believe I ran into the limitation that the compute modules pass along an ExecSpan object which contains a collection of ArraySpan objects. AFAICT, the ArraySpan is hard-coded to assume 3 fixed buffers, making it difficult to implement compute for variadic types:
https://github.com/apache/arrow/blob/8ac434fb97ab5fb0adc7591736826b2d1db76b9d/cpp/src/arrow/array/data.h#L441
I assume that the ArraySpan therefore needs to be updated to support more than 3 buffers, although if I am going down the wrong path with this happy to be corrected
Component(s)
C++

After some more research here I noticed the ArraySpan type actually should support binary view types, and does so by making all the variadic buffers available through the third buffer:
https://github.com/apache/arrow/blob/b655852b260d3b8c3fe457795df0f42a2ff9c98c/cpp/src/arrow/array/data.h#L636
Coming from nanoarrow I was a bit surprised by that, as nanoarrow dynamically resizes the buffers array to fit variadic buffers, but if that's not an issue maybe there isn't much to be done here


text:
MINOR: [Archery] Add collaborators to list of roles with access to trigger bot tasks

I've been added as triager but now if I try to trigger crossbow tasks they fail with permissions issues:
https://github.com/apache/arrow/pull/14715#issuecomment-1326243342
Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/3539688563

This should fix the issue

I didn't know about the "collaborator" role. This looks good to me. @kou What do you think?

@pitrou for reference: https://github.com/apache/arrow/pull/14719

+1

source: gharchive/pull-request

authors: kou pitrou raulcd


text:
GH-35059: [C++] Fix "hash_count" for run-end encoded inputs

Rationale for this change
Fixing a bug.
What changes are included in this PR?
Changes to the "hash_count" kernel implementation to handle REE and union arrays correctly.

[x] Generic (potentially slow) implementation
[x] REE-specialized implementation

Are these changes tested?
Yes, by modifying the existing unit tests.

Closes: #35059

@westonpace @jorisvandenbossche


text:
MINOR: [R] Replace use of show_query in test-duckdb.R with dbplyr::sql_build

Rationale for this change
I just ran the R package tests and saw a printed query mixed amongst testthat output:
[ FAIL 0 | WARN 0 | SKIP 1 | PASS 15 ]<SQL>
SELECT *
FROM arrow_010

I thought it would be good to silence this in some way.
What changes are included in this PR?
I silenced this by changing out the call to dplyr::show_query for dbplyr::sql_build which produces different output but (1) doesn't print as as side-effect, (2) is specifically made for testing, and (3) still produces output we can use in this test.
For reference, this is what show_query prints (assuming via cat):
> show_query(table_four)
<SQL>
SELECT *
FROM arrow_011

Whereas sql_build:
> dbplyr::sql_build(table_four)
<dbplyr_table_ident[1]>
[1] `arrow_011`

Are these changes tested?
Yes, but just manually on my system.
Are there any user-facing changes?
No.

Merged. Thanks for taking a look at this so fast @paleolimbot.


text:
ARROW-79: [Python] Add benchmarks

Run them using asv run --python=same or asv dev.

What is your opinion on putting the benchmarks in pyarrow/benchmarks versus at the top level $ROOT/python/benchmarks. This is what we do with pandas, but I don't have a strong opinion: https://github.com/pydata/pandas/tree/master/asv_bench

Actually I'm used to have benchmarks and unit tests at the top level. I only put them into pyarrow/benchmark as the unit tests where also located there. Should I move them?

Sure, that would be great. The main reason I've embedded the test suite in packages is to be able to have a PACKAGE.test() for users to verify their installation (for example numpy.test or pandas.test), but it's not a hard and fast rule. For benchmarks this is less important

+1, thank you

authors: wesm xhochy


text:
ARROW-10034: [Rust] Fix Rust build on master

Reverts ARROW-9977 and fixes one other conflict.
I was seeing compilation errors like this:
error[E0599]: no method named `value` found for reference `&array::array::StringArray` in the current scope
  --> arrow/src/compute/kernels/aggregate.rs:39:35
   |
39 |                 let item = $array.value(i);
   |                                   ^^^^^ method not found in `&array::array::StringArray`

@jorgecarleitao fyi

build error seems unrelated so I am going to go ahead and merge this

Thanks a lot for the quick response on solving this.
One was related to a new trait in Arrow, StringArrayOps, that has .value(i) -> &str, and is needed when using value in strings, in the commit for take.
The other one is related to the Ops in logical plans.
Essentially, this is all related to multiple merges, with backward incompatible changes on them.
@andygrove , is there anything we can do to help mitigating this during merge of multiple merges?
One idea is to have a WARNING in the PR text that the merge is backward incompatible, so that the committer knows of potential conflicts with other merges.
Another idea is to first perform the merges locally, one by one, and then push a branch with those changes to test them in integration, and then merge them at once.

I think I will start testing each PR locally now, after merging latest from
master.
There is quite a bit of process in place so it is important that PRs are
merged to master individually.
On Thu, Sep 17, 2020, 7:55 PM Jorge Leitao notifications@github.com wrote:

Thanks a lot for the quick response on solving this.
One was related to a new trait in Arrow, StringArrayOps, that has .value(i)
-> &str, and is needed when using value in strings, in the commit for take
.
The other one is related to the Ops in logical plans.
Essentially, this is all related to multiple merges, with backward
incompatible changes on them.
@andygrove https://github.com/andygrove , is there anything we can do
to help mitigating this during merge of multiple merges?
One idea is to have a WARNING in the PR text that the merge is backward
incompatible, so that the committer knows of potential conflicts with other
merges.
Another idea is to first perform the merges locally, one by one, and then
push a branch with those changes to test them in integration, and then
merge them at once.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow/pull/8213#issuecomment-694602195, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHEBREOAR5VEFXSRKRONFLSGK42NANCNFSM4RRFKIAA
.

authors: andygrove jorgecarleitao


text:
BigQuery Storage Write API fails with type mismatch error for DATE type

If DATE type is specified in schema, BigQueryIO with Storage Write API may fail with the following error:

Got error com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The proto field mismatched with BigQuery field at <table>, the proto field type string, BigQuery field type DATE Entity: <write_stream_name>
.
DATE type in BEAM for BigQuery is converted to string in proto request [1], while BigQuery Storage Write API requires an integer type for DATE type in BigQuery [2].

A workaround is to use INTEGER in BEAM and manually compute the number of days since epoch.
Actually, a feature request to BigQuery is filed at [3]. If it's implemented, we can use DATE type successfully.
[1] https://github.com/apache/beam/blob/v2.35.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L55-L73
[2] https://cloud.google.com/bigquery/docs/write-api#data_type_conversions
[3] https://issuetracker.google.com/issues/205174128
Imported from Jira BEAM-13753. Original Jira may contain additional context.
Reported by: baeminbo.

This was fixed by #17404

.close-issue

repo: apache/beam


text:
[BEAM-14049][Playground] Add new API method to get Precompiled Object

[BEAM-14049]
Add and implement new API method to get Precompiled Object by cloud path

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @KhaninArtur @ilya-kozyrev

retest this please

R: @pabloem

lgtm thanks. Can you please add more detail in the bug (BEAM-14049) as to why this is necessary? (I suppose the backend needs to get the examples for the catalog. this will be useful in the future for maintenance

authors: AydarZaynutdinov pabloem


text:
Remove stringx package

We've said this will is deprecated since February. This removes it and replaces one usage in the proper codebase and a couple in examples that had crept in.
Fixes #21293

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @jrmccluskey


text:
Validate that a URN is set on all non-composite transforms.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Java_GCP_IO_Direct PreCommit

Run Java PreCommit

The failure (e.g. NullPointerException in org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldRelaxation[3] ) look unrelated.

noting that this seems to break a bunch of Dataflow tests (they may or may not be mocking things out and that is why it happens)

When a user corrupts the graph by, say, capturing a PCollection via a closure then returning it as an output, this detects it as a non-composite. It would be nice to catch that error sooner with more context for the user.

And it instructs said users to register a URN (which does "fix" the issue... though with a corrupted graph I could see issues with the topological outward-in visitor having random problems)

authors: kennknowles robertwb


text:
[Release-2.49.0] Fix staged RC source name in GA (#27394)

cherry pick #27394 into release-2.49.0

Fix staged RC source name format in GitHub Action

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @damccorm


text:
Add Error Handlers to File IO and related IOs (TextIO, AvroIO)

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

the flink precommit is known to be flaky


text:
Revert "Revert #30533: Automatically execute unbounded pipelines in streaming mode."

Reverts apache/beam#30706
This change was initially correct

We're taking a different (runner driven) approach here


text:
Beam-2329 ABS function

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify.
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Hi @xumingming , can you please take a look? I have break it into jira tickets wise.

Changes Unknown when pulling 6e5ad90878ff69064ee8cfd1f624b1e927bdcb00 on app-tarush:beam-2329-abs-function into ** on apache:DSL_SQL**.

LGTM

@jbonofre can you take a peek at this one?

R: @jbonofre
Sure ! Thanks !

Merged on DSL_SQL. Can you please close this PR ? Thanks !

@jbonofre @xumingming Thanks!!

authors: app-tarush coveralls jbonofre xumingming


text:
[BEAM-4088] fix thread leak in tests on PR #4965

This PR is based on @rmannibucau's https://github.com/apache/beam/pull/4965 that fixes a thread leak in direct runner. It just fixes its tests

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand:

[ ] What the pull request does
[ ] Why it does it
[ ] How it does it
[ ] Why this approach

[X] Each commit in the pull request should have a meaningful subject line and body.
[X] Run ./gradlew build to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[X] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @rmannibucau

Run Java PreCommit


text:
[BEAM-6623] Complete DataflowValidatesRunner test coverage for Python 3.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Seed Job

Run Dataflow Python ValidatesRunner

Run Python Dataflow ValidatesRunner

Run Python Dataflow ValidatesRunner

Run Seed Job

Run Seed Job

Run Python Dataflow ValidatesRunner

Run Seed Job

Run Seed Job

Run Seed Job

Run Python Dataflow ValidatesRunner

@markflyhigh could you please help with the merge?
cc: @fredo838 , @Juta

Thanks, @markflyhigh !


text:
[BEAM-8036] disable failed Postcommit Test

Relevant java call stack:
java.lang.NoSuchMethodError: org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.parseTableSpec(Ljava/lang/String;)Lorg/apache/beam/repackaged/sql/com/google/api/services/bigquery/model/TableReference;
	at org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTable.getRowCountFromBQ(BigQueryTable.java:100)
	at org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BigQueryTable.getTableStatistics(BigQueryTable.java:63)
	at org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable.getStatistic(BeamCalciteTable.java:91)
	at org.apache.beam.sdk.extensions.sql.impl.BeamCalciteTable.getStatistic(BeamCalciteTable.java:46)
	at org.apache.beam.repackaged.sql.org.apache.calcite.prepare.RelOptTableImpl.getMonotonicity(RelOptTableImpl.java:341)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.TableNamespace.getMonotonicity(TableNamespace.java:79)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:223)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:995)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:955)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3109)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3091)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3363)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:995)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:955)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:216)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:930)
	at org.apache.beam.repackaged.sql.org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:637)
	at org.apache.beam.repackaged.sql.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:215)
	at org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144)
	at org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103)

It might because of this relocation, which is the change used for making zetasql jar work.
The problem is we haven't found a resolution for it by changing build.gradle. For example, we tried exclude relocation for com.google.api.services, which didn't help (any help on this idea will be appreciated).
Vendor Calcite could help because if we have vendor calcite, we can separate zetasql to another package with its own relocation. However by doing so we need wait for vendor calcite effort.
@apilloud suggests we revert Beam ZetaSQL PR. I would more leaning to ignore this IT test for now.
R: @kennknowles What do you think? Is there a better option?
cc: @Ardagan

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Update:
I occasionally find this relocation is not needed. I will send another PR to remove it.

https://github.com/apache/beam/pull/9409 is supposed to fix this.


text:
source-bundle: generated script fails when a build element has no source

See original issue on GitLab
In GitLab by [Gitlab user @hashpling] on Aug 24, 2017, 14:31
If a build element doesn't have any sources, then the tarball won't contain a source directory for that element but the generate build- script will attempt to copy the contents of the directory to /buildstream/build and will error out due to set -e.
The simplest fix would seem to be to check for the existence of the source directory before attempting the copy in the build-module.sh.in template, but the exact shape of that change depends on the choice of fix to issue #67.
(Perhaps I should be using kind script instead of kind manual for an element without sources, but I think that these elements are omitted from source-bundle generated tarballs?)

In GitLab by [Gitlab user @tlater] on Sep 25, 2017, 14:01
created branch 75-source-bundle-generated-script-fails-when-a-build-element-has-no-source

In GitLab by [Gitlab user @tlater] on Sep 25, 2017, 15:25
mentioned in merge request !97

In GitLab by [Gitlab user @tristanvb] on Sep 27, 2017, 14:46
closed via merge request !97

repo: apache/buildstream


text:
New components not appearing on karavan comopent pallette

I took camel and karavan both repository in my local and I kept my test component in the camel/components at same place where other components are and ran the build. i can see my component is listed in component.properties and its json is also included in component.json. Now i built the karavan app as per the documentation(https://github.com/apache/camel-karavan/blob/main/DEV.md

). But can't see the test component in the karavan component Pallette. Please help me what should be exact step to get my component listed in karavan component Pallette.
What more i have tried is , I generate the code then pasted the code in the installed folder of karavan vsCode extension but that also didn't work, it shows the component in component catalogue but not on the actual component Pallette which appears after we click on create integration. screenshot attached.

Current version of Karavan works with Camel 3.20+
Current version of Karavan works with Components from Camel project.

However Karavan supports custom Kamelets, so, as a workaround I would recommend to create a custom Kamelet.

Thanks.

I am using camel version 3.20.3.
Please correct my understanding. I took code for camel, i took code for karavan in my local. build camel code, generated new properties and json file for camel catalogue. now i built the karavan and those updated files got copied into karavan app and designer. So will it still take Apache camel catalogue or will it refer my files now. Which class fetches catalogue information? I want to test my component first locally with its look and feel in karavan integration before submitting to apache repository.

karavan-generator uses camel-catalog jar to generate component list.
you could manually add your component metadata json to components.json created by karavan-generator

authors: VandanaSinha mgubaidullin

repo: apache/camel-karavan


text:
Delete project from Git repo when user deletes the project

Added method deleteProject() and addDeletedFolderToIndex() to delete project from Repository.

Related to https://github.com/apache/camel-karavan/issues/551

authors: mgubaidullin mrinalsharma


text:
CAMEL-10190: Fallback lookup components, data-formats and languages using suffix in name to avoid clash

I changed the behavior of the default, blueprint and OSGi resolvers to introduce the fallback names.
I refactored all three resolvers to use a common Helper instead of repeating the code.
Unit tests added. Executed all unit tests on all changed modules.

Thanks for the PR it has been merged. Do you mind closing this?

authors: davsclaus nicolaferraro

repo: apache/camel


text:
feat(components): add camel-observability-services

With this PR we're adding a new component which serves as an adapter and helper to expose a set of opinionated observability services. This first MVP adds camel-micrometer-prometheus components and a default configuration for it. Once this is onboarded I'll proceed with creating the proper extension/starter for the other runtimes.
Iteratively we'll be adding into this adapter a few more components in order to expose other sensitive observability metrics (telemetry, health, ...). The main goal is to provide services required when running a microservice oriented architecture on cloud, but the component can be used outside cloud as well with the same meaning.
Closes CAMEL-21384
Description

Target

[x] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[x] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[x] I checked that each commit in the pull request has a meaningful subject line and body.

[x] I have run mvn clean install -DskipTests locally from root folder and I have committed all auto-generated changes.

@claudio4j @oscerd FYI. I am not sure if I've missed some configuration required to onboard the new component. Just let me know.


text:
Backport CAMEL-17949 and netty-http test fixes

Tests passed.

The HeaderFilterStrategy that some component uses is potentially a better and more general solution. But this strategy is also a little bit confusing to use for component developers.

Thanks. I created a ticket so we can take a look at that when time allow us: https://issues.apache.org/jira/browse/CAMEL-18158


text:
[Don't merge] Only for test apache jenkin CIs, to avoid printing too much error msg.

This PR is making test case failed, to check if print too much error msg after configuring jenkins CI

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/816/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/740/Failed Tests: 1carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 1org.apache.carbondata.spark.testsuite.aggquery.IntegerDataTypeTestCase.select empno from integertypetableAgg

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/88/Failed Tests: 1carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-spark-common-test: 1org.apache.carbondata.spark.testsuite.aggquery.IntegerDataTypeTestCase.select empno from integertypetableAgg

SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/138/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3406/

retest this please

retest this please

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/823/

SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/141/

retest this please

SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/158/

retest this please

retest this please

SDV Build Success with Spark 2.1, Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/244/

@chenliang613 close this if not needed anymore

authors: CarbonDataQA asfgit chenliang613 jackylk jbonofre ravipesala

repo: apache/carbondata


text:
[CARBONDATA-3176] Optimize quick-start-guide documentation

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[x] Any interfaces changed?

[x] Any backward compatibility impacted?

[x] Document update required?

[x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

LGTM

@lamber-ken Please finish the checklist in content of this PR.

@lamber-ken Can you optimize all documentation in this PR? not only for quick-start-guide, such as "docs/ddl-of-carbondata.md", it also has the similar place need to be optimized . There are many files have similar place, we shouldn't raise many PRs for these file.

@xubo245, thanks for review. I'll update as your suggestion

@xubo245
"and plan to support alluxio path too."
I think there is no need to add this currently. We should only describe the feature implemented.

@xuchuanyin I mean it's plan, just explain why do we change  => <carbon_store_path> , not require add it to document now.

Hi admins, can you please take a look? :)

add to whitelist

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1860/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1862/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2070/

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10117/

hi，admins，what do I need to do next for this pr? thanks

@xubo245, thanks for review. I had update docs as suggestion，please take a look.

LGTM

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1934/

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10188/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1964/

LGTM, Thank you for working on this.

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10216/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2237/

authors: CarbonDataQA lamber-ken sraghunandan xubo245 xuchuanyin


text:
[CARBONDATA-3897] Fixed external table with location creation and deleting issues

Why is this PR needed?

Creating external table without Fs schema fails on HDFS
Desc formatted on an external table throws StackOverflowException

What changes were proposed in this PR?

Added default FS Schema to the table path
Added a check for non-transactional table in refreshIndexInfo method.

Does this PR introduce any user interface change?

No

Is any new testcase added?

No

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3355/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1614/

retest this please

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3360/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1619/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1621/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3363/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1622/

LGTM.
Will merge once build is passed.

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1628/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3369/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1634/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3375/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1635/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3376/


text:
use.system.public.ips does nothing when domain doesn't have a range

ISSUE TYPE

Bug Report

COMPONENT NAME

IP assignment

CLOUDSTACK VERSION

4.17,
4.18,
main (*)

(*): Although I didn't test that on main branch, the code pard didn't change so it should be there as well.
SUMMARY

By default when user use all of their assigned public IPs, they can use system public IPs. Although by using the use.system.public.ips setting admins can prevent that action, the setting doesn't have any affect when user doesn't have any reserved range.
STEPS TO REPRODUCE

Login as a root admin
Create a domain and a domain admin account for the domain.
Set use.system.public.ips to false in the account, the domain or globally.
Assign an IP range to the domain or the account. [Range 1]
Login as a domain admin and create an isolated network in domain.
Go to the isolated network page and click on the Public IP addresses then click on Acquire new IP.
The list shows only the IPs in the assinged IP range [Range 1]
Logout and login as a root admin again
Remove the [Range1] IP range assignment.
Repeat step 5, 6, and 7. The output list now includes the system IPs.

EXPECTED RESULTS

User can NOT see the system's public IP addresses.

ACTUAL RESULTS

User can see the system's public IP addresses.

@DaanHoogland able to reproduce the issue

@soreana are you creating a PR for this?

Hey @DaanHoogland Yes, I'm going to create a PR for this issue. Any suggestion appreciated :)

@weizhouapache Honestly, Before double checking the global setting definition I wasn't agree with you. But now it makes sense not to touch the global settings behaviour.
Shall I add a new setting called restric.system.public.ips.access which restricts public IP access even if user doesn't have the dedicated range?
Definition of "use.system.public.ips"
If true, when account has dedicated public ip range(s), once the ips dedicated to the account have been consumed ips will be acquired from the system pool

@weizhouapache Honestly, Before double checking the global setting definition I wasn't agree with you. But now it makes sense not to touch the global settings behaviour.
Shall I add a new setting called restric.system.public.ips.access which restricts public IP access even if user doesn't have the dedicated range?
Definition of "use.system.public.ips"
If true, when account has dedicated public ip range(s), once the ips dedicated to the account have been consumed ips will be acquired from the system pool

@soreana good idea. go for it.
can we close this ticket as it is not a real issue ?

@weizhouapache Honestly, Before double checking the global setting definition I wasn't agree with you. But now it makes sense not to touch the global settings behaviour.
Shall I add a new setting called restric.system.public.ips.access which restricts public IP access even if user doesn't have the dedicated range?
Definition of "use.system.public.ips"
If true, when account has dedicated public ip range(s), once the ips dedicated to the account have been consumed ips will be acquired from the system pool

@soreana good idea. go for it.
can we close this ticket as it is not a real issue ?

I would create a PR for that.
Btw, I think it is better to keep it open. I would change it in a way that it reflects our discussion.

authors: DaanHoogland kiranchavala soreana weizhouapache

repo: apache/cloudstack