text:
fix: Visit subqueries in Expr::Alias

Which issue does this PR close?
Closes #4898
Rationale for this change
See issue description.
Are these changes tested?
Yes. A new test is included to test the changes.
Are there any user-facing changes?
The users who enable tracing of LogicalPlan might see additional subqueries in the logs.

BTW here is a follow on proposal to avoid bugs like this in the future: https://github.com/apache/arrow-datafusion/pull/4906

authors: alamb askoa

repo: apache/arrow-datafusion


text:
Rename weekday and weekday0 kernels to to num_days_from_monday and days_since_sunday

Which issue does this PR close?
Closes https://github.com/apache/arrow-rs/issues/2065
Rationale for this change
Depending on the usecase sometimes one wants the days of week to start from Sunday or Monday. Dates :shrug:
@ovr added the weekday0 kernel in https://github.com/apache/arrow-rs/pull/2052
But now the names are weekday and weekday0 which I think might be confusing to the average reader
What changes are included in this PR?
Rename the kernels to follow the  chrono naming scheme num_days_from_sunday and num_days_from_monday
https://docs.rs/chrono/0.4.19/chrono/enum.Weekday.html#method.num_days_from_sunday
https://docs.rs/chrono/0.4.19/chrono/enum.Weekday.html#method.num_days_from_monday
Are there any user-facing changes?
Yes, name of the compute::kernels::temporal::weekday (which is present in arrow 18) is renamed to compute::kernels::temporal::num_days_since_monday

cc @nl5887 who contributed the original weekday kernel

Codecov Report

Merging #2066 (43c5f97) into master (4444cb7) will increase coverage by 0.00%.
The diff coverage is 100.00%.

:exclamation: Current head 43c5f97 differs from pull request most recent head 01bb3fb. Consider uploading reports for the commit 01bb3fb to get more accurate results

@@           Coverage Diff           @@
##           master    #2066   +/-   ##
=======================================
  Coverage   83.57%   83.57%
=======================================
  Files         222      222
  Lines       58244    58244
=======================================
+ Hits        48675    48677    +2
+ Misses       9569     9567    -2

Impacted Files
Coverage Δ

arrow/src/compute/kernels/temporal.rs
94.98% <100.00%> (ø)

parquet_derive/src/parquet_field.rs
66.21% <0.00%> (+0.22%)
:arrow_up:

arrow/src/datatypes/datatype.rs
65.68% <0.00%> (+0.36%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4444cb7...01bb3fb. Read the comment docs.

Labelled as development process to hide from changelog (as not released)

Labelled as development process to hide from changelog (as not released)
I think since weekday is released this should be in the changelog
https://docs.rs/arrow/18.0.0/arrow/compute/kernels/temporal/fn.weekday.html
though weekday0 was not released (perhaps I can update the title)

source: gharchive/pull-request

authors: alamb codecov-commenter tustvold

repo: apache/arrow-rs

license_type: permissive

license_source: github-api


text:
Refactor Binary Builder and String Builder Constructors

Which issue does this PR close?
Closes #2054.
Rationale for this change
Removes capacity from new constructors
What changes are included in this PR?
Removes capacity from new constructors
Are there any user-facing changes?
Yes

I have replace all new calls with with_capacity(1024, capacity) . This way the logic wont change.


text:
[Java] Remove Netty dependency from arrow-vector

Describe the enhancement requested
#14913 removes a lot of uses of Netty from the core Arrow modules in favor of our own classes. If we could replace the remaining uses of Netty, we could drop the hard dependency on Netty entirely, so that it's only required if you use arrow-memory-netty or arrow-flight.
From https://github.com/apache/arrow/pull/14913#issuecomment-1348686193:

PlatformDependent was used in arrow-algorithm, arrow-jdbc, arrow-memory-netty, arrow-vector. I've removed the usages in arrow-algorithm, arrow-jdbc and arrow-vector, and removed the dependency of netty-common in arrow-algorithm and arrow-jdbc. It would take a bit more job to remove dependency of netty-common  in arrow-vector as it is using some Netty collection utils llike IntObjectMap. Maybe do that in follow up.

Component(s)
Java

Interested to work on this issue.

@sudip-unb if you're interested, please feel free to file a PR; you can comment 'take' here to have the bot assign this issue to you as well

take

We probably don't want to re-implement IntObjectHashMap. Can we simply take the Netty implementation and add it to the Arrow project? It Netty has an Apache 2.0 license.

Vendoring is fine, yes (just update LICENSE to reflect that). Or we could possibly pull in Eclipse or Apache Commons equivalents if they exist.

source: gharchive/issue

authors: jduo lidavidm sudip-unb

repo: apache/arrow


text:
BUG: libhdfs3 not getting picked up correctly when using conda

Describe the bug, including details regarding any error messages, version, and platform.
#14832 switched on the test suite within the conda-builds, which itself synced with https://github.com/conda-forge/arrow-cpp-feedstock/pull/875
In the process of doing so, I removed
https://github.com/apache/arrow/blob/2c768a1767c0d7b33644910e0e96a949bc6f6f42/dev/tasks/conda-recipes/arrow-cpp/meta.yaml#L351-L352
because the tests always showed: SKIPPED [24] test_hdfs.py:48: No libhdfs available on system
Even with a patch as follows:
diff --git a/cpp/src/arrow/io/hdfs_internal.cc b/cpp/src/arrow/io/hdfs_internal.cc
index 4592392b8..9f8f70389 100644
--- a/cpp/src/arrow/io/hdfs_internal.cc
+++ b/cpp/src/arrow/io/hdfs_internal.cc
@@ -144,9 +144,9 @@ Result<std::vector<PlatformFilename>> get_potential_libhdfs_paths() {
 #ifdef _WIN32
   file_name = "hdfs.dll";
 #elif __APPLE__
-  file_name = "libhdfs.dylib";
+  file_name = "libhdfs3.dylib";
 #else
-  file_name = "libhdfs.so";
+  file_name = "libhdfs3.so";
 #endif

   // Common paths
@@ -155,6 +155,8 @@ Result<std::vector<PlatformFilename>> get_potential_libhdfs_paths() {
   // Path from environment variable
   AppendEnvVarFilename("HADOOP_HOME", "lib/native", &search_paths);
   AppendEnvVarFilename("ARROW_LIBHDFS_DIR", &search_paths);
+  AppendEnvVarFilename("CONDA_PREFIX", "lib", &search_paths);
+  AppendEnvVarFilename("LIBRARY_LIB", &search_paths);

   // All paths with file name
   for (const auto& path : search_paths) {

this remained the case. I thought this might be necessary because in the conda-forge world, the name of the libhdfs binary contains the "3", but there seem to be other issues at play here as well.
To fix this issue, the line quoted above should be uncommented, and the conda tests should not show SKIPPED [24] test_hdfs.py:48: No libhdfs available on system anymore (at least on unix).
Component(s)
C++, Continuous Integration

It seems that libhdfs.so and libhdfs3.so are different project.
libhdfs.so is a Java (JNI) library but libhdfs3.so is not.
https://github.com/martindurant/libhdfs3-downstream/tree/master/libhdfs3
I think that we need to implement libhdfs3.so based HDFS backend instead of reusing the current libhdfs.so based implementation. (I think that this is a feature request not a bug.)

The Arrow HDFS implementation is based on the JNI libhdfs, so it is expected that it doesn't work with libhdfs3. And since libhdfs doesn't seem to be packages by conda-forge, I don't think there is a way to run the hdfs tests in the conda-forge build purely based on conda packages (our own tests install the JNI library manually on top of the conda env, see eg ci/docker/conda-python-hdfs.dockerfile)
We have had integration with libhdfs3 as well in the past (and you could switch between both drivers), but this was removed almost 3 years ago (https://github.com/apache/arrow/pull/6432), because the libhdfs3 project is unmaintained. Also the dask filesystem wrapper using libhdfs3 is archived (https://github.com/dask/hdfs3).
Unless the libhdfs3 project would be revived, I don't think we should currently consider adding support for it again.

Cool, thanks for the input. I wasn't aware of the split between libhdfs / libhdfs3; in this case I think it might be worthwhile medium term to package libhdfs in conda-forge.

but this was removed almost 3 years ago (https://github.com/apache/arrow/pull/6432)

Oh, the author of the pull request is me... I didn't remember it...
I close this because we don't support libhdfs3.

authors: h-vetinari jorisvandenbossche kou


text:
[Python] Unable to build pyarrow wheel for macOS 10.13 when installing from PyPI

Describe the bug, including details regarding any error messages, version, and platform.
I'm trying to install pyarrow on macOS 10.13 using pip.
As of pyarrow 10.0 there are no pre-built wheels for macOS 10.13, so pip tries to build a wheel from source.
I get the following error message:
  CMake Error at CMakeLists.txt:63 (find_package):
    By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
    asked CMake to find a package configuration file provided by "Arrow", but
    CMake did not find one.

    Could not find a package configuration file provided by "Arrow" with any of
    the following names:

      ArrowConfig.cmake
      arrow-config.cmake

    Add the installation prefix of "Arrow" to CMAKE_PREFIX_PATH or set
    "Arrow_DIR" to a directory containing one of the above files.  If "Arrow"
    provides a separate development package or SDK, be sure it has been
    installed.

  -- Configuring incomplete, errors occurred!
  See also "/private/var/folders/2h/gnd6b40x36n7xwscj8712hsc0000gn/T/pip-install-gqi21gpr/pyarrow_24109b85dd934301b68bc56161834085/build/cpp/CMakeFiles/CMakeOutput.log".
  error: command '/opt/local/bin/cmake' failed with exit code 1
  [end of output]

See full install log below:

❯ python3 -m venv venv
❯ . venv/bin/activate
❯ pip install pyarrow
Collecting pyarrow
  Using cached pyarrow-10.0.1.tar.gz (994 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy>=1.16.6
  Using cached numpy-1.24.1-cp310-cp310-macosx_10_9_x86_64.whl (19.8 MB)
Building wheels for collected packages: pyarrow
  Building wheel for pyarrow (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyarrow (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [318 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.13-x86_64-cpython-310
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/orc.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/conftest.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_generated_version.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/benchmark.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_compute_docstrings.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/ipc.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/util.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/flight.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/cffi.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/substrait.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/filesystem.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/__init__.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/plasma.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/types.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/dataset.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/cuda.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/feather.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/pandas_compat.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/fs.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/csv.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/jvm.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/hdfs.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/json.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/serialization.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/compute.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_tensor.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_ipc.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/conftest.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_convert_builtin.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_misc.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/arrow_16597.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_gandiva.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/strategies.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_adhoc_memory_leak.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/arrow_7980.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/util.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_orc.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_table.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_array.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_deprecations.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_serialization_deprecated.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/__init__.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_io.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_util.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_cuda_numba_interop.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_cpp_internals.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_cffi.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_schema.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_jvm.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_plasma_tf_op.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_fs.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_udf.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_filesystem.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/pandas_threaded_import.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/pandas_examples.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_cython.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_sparse_tensor.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_dataset.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_builder.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_cuda.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_extension_type.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_feather.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_pandas.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_memory.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_exec_plan.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_flight.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/read_record_batch.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_json.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_serialization.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_compute.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_hdfs.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/deserialize_buffer.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_strategies.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_csv.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_plasma.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_scalars.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_gdb.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_types.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/test_substrait.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/vendored
      copying pyarrow/vendored/version.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/vendored
      copying pyarrow/vendored/__init__.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/vendored
      copying pyarrow/vendored/docscrape.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/vendored
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/parquet
      copying pyarrow/parquet/encryption.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/parquet
      copying pyarrow/parquet/__init__.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/parquet
      copying pyarrow/parquet/core.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/parquet
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_basic.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/conftest.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/encryption.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_parquet_writer.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_metadata.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/__init__.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_datetime.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/common.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_dataset.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_data_types.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_pandas.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_parquet_file.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_compliant_nested_type.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      copying pyarrow/tests/parquet/test_encryption.py -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/parquet
      running egg_info
      writing pyarrow.egg-info/PKG-INFO
      writing dependency_links to pyarrow.egg-info/dependency_links.txt
      writing entry points to pyarrow.egg-info/entry_points.txt
      writing requirements to pyarrow.egg-info/requires.txt
      writing top-level names to pyarrow.egg-info/top_level.txt
      listing git files failed - pretending there aren't any
      reading manifest file 'pyarrow.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '../LICENSE.txt'
      warning: no files found matching '../NOTICE.txt'
      warning: no previously-included files matching '*.so' found anywhere in distribution
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '*~' found anywhere in distribution
      warning: no previously-included files matching '#*' found anywhere in distribution
      warning: no previously-included files matching '.git*' found anywhere in distribution
      warning: no previously-included files matching '.DS_Store' found anywhere in distribution
      no previously-included directories found matching '.asv'
      writing manifest file 'pyarrow.egg-info/SOURCES.txt'
      copying pyarrow/__init__.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_compute.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_compute.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_csv.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_csv.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_cuda.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_cuda.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_dataset.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_dataset.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_dataset_orc.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_dataset_parquet.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_exec_plan.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_feather.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_flight.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_fs.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_fs.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_gcsfs.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_hdfs.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_hdfsio.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_json.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_orc.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_orc.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_parquet.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_parquet.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_parquet_encryption.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_parquet_encryption.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_plasma.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_pyarrow_cpp_tests.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_pyarrow_cpp_tests.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_s3fs.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/_substrait.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/array.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/benchmark.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/builder.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/compat.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/config.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/error.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/gandiva.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/io.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/ipc.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/lib.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/lib.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/memory.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/pandas-shim.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/public-api.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/scalar.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/serialization.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/table.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/tensor.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      copying pyarrow/types.pxi -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/common.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_feather.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/__init__.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libgandiva.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_python.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_flight.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_dataset_parquet.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_dataset.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_substrait.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_cuda.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libarrow_fs.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/includes/libplasma.pxd -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/includes
      copying pyarrow/tests/bound_function_visit_strings.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      copying pyarrow/tests/pyarrow_cython_example.pyx -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tensorflow
      copying pyarrow/tensorflow/plasma_op.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tensorflow
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src
      copying pyarrow/src/ArrowPythonConfig.cmake.in -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src
      copying pyarrow/src/ArrowPythonFlightConfig.cmake.in -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src
      copying pyarrow/src/CMakeLists.txt -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src
      copying pyarrow/src/arrow-python-flight.pc.in -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src
      copying pyarrow/src/arrow-python.pc.in -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/feather
      copying pyarrow/tests/data/feather/v0.17.0.version.2-compression.lz4.feather -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/feather
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/README.md -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/TestOrcFile.emptyFile.jsn.gz -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/TestOrcFile.emptyFile.orc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/TestOrcFile.test1.jsn.gz -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/TestOrcFile.test1.orc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/TestOrcFile.testDate1900.jsn.gz -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/TestOrcFile.testDate1900.orc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/decimal.jsn.gz -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      copying pyarrow/tests/data/orc/decimal.orc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/orc
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/parquet
      copying pyarrow/tests/data/parquet/v0.7.1.all-named-index.parquet -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/parquet
      copying pyarrow/tests/data/parquet/v0.7.1.column-metadata-handling.parquet -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/parquet
      copying pyarrow/tests/data/parquet/v0.7.1.parquet -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/parquet
      copying pyarrow/tests/data/parquet/v0.7.1.some-named-index.parquet -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/tests/data/parquet
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow
      creating build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/CMakeLists.txt -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/api.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/arrow_to_pandas.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/arrow_to_pandas.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/arrow_to_python_internal.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/benchmark.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/benchmark.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/common.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/common.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/csv.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/csv.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/datetime.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/datetime.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/decimal.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/decimal.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/deserialize.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/deserialize.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/extension_type.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/extension_type.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/filesystem.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/filesystem.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/flight.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/flight.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/gdb.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/gdb.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/helpers.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/helpers.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/inference.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/inference.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/init.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/init.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/io.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/io.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/ipc.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/ipc.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/iterators.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/numpy_convert.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/numpy_convert.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/numpy_internal.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/numpy_interop.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/numpy_to_arrow.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/numpy_to_arrow.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/parquet_encryption.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/parquet_encryption.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/pch.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/platform.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/pyarrow.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/pyarrow.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/pyarrow_api.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/pyarrow_lib.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/python_test.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/python_test.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/python_to_arrow.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/python_to_arrow.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/serialize.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/serialize.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/type_traits.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/udf.cc -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/udf.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      copying pyarrow/src/arrow/python/visibility.h -> build/lib.macosx-10.13-x86_64-cpython-310/pyarrow/src/arrow/python
      running build_ext
      creating /private/var/folders/2h/gnd6b40x36n7xwscj8712hsc0000gn/T/pip-install-gqi21gpr/pyarrow_24109b85dd934301b68bc56161834085/build/cpp
      -- Running CMake for PyArrow C++
      cmake -DARROW_BUILD_DIR=build -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_INSTALL_PREFIX=/private/var/folders/2h/gnd6b40x36n7xwscj8712hsc0000gn/T/pip-install-gqi21gpr/pyarrow_24109b85dd934301b68bc56161834085/build/dist -DPYTHON_EXECUTABLE=/Users/dwales/Desktop/test/venv/bin/python3 -DPython3_EXECUTABLE=/Users/dwales/Desktop/test/venv/bin/python3 -DPYARROW_CXXFLAGS= -DPYARROW_WITH_DATASET=off -DPYARROW_WITH_PARQUET_ENCRYPTION=off -DPYARROW_WITH_HDFS=off -DPYARROW_WITH_FLIGHT=off /private/var/folders/2h/gnd6b40x36n7xwscj8712hsc0000gn/T/pip-install-gqi21gpr/pyarrow_24109b85dd934301b68bc56161834085/pyarrow/src
      -- The C compiler identification is AppleClang 10.0.0.10001145
      -- The CXX compiler identification is AppleClang 10.0.0.10001145
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      CMake Error at CMakeLists.txt:63 (find_package):
        By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
        asked CMake to find a package configuration file provided by "Arrow", but
        CMake did not find one.

        Could not find a package configuration file provided by "Arrow" with any of
        the following names:

          ArrowConfig.cmake
          arrow-config.cmake

        Add the installation prefix of "Arrow" to CMAKE_PREFIX_PATH or set
        "Arrow_DIR" to a directory containing one of the above files.  If "Arrow"
        provides a separate development package or SDK, be sure it has been
        installed.

      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/2h/gnd6b40x36n7xwscj8712hsc0000gn/T/pip-install-gqi21gpr/pyarrow_24109b85dd934301b68bc56161834085/build/cpp/CMakeFiles/CMakeOutput.log".
      error: command '/opt/local/bin/cmake' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects

Component(s)
Python

You're using Homebrew, right?
If so, could you install apache-arrow (brew install apache-arrow) before pip install pyarrow?

I'm using MacPorts. (Homebrew doesn't support 10.13 anymore.) But I'll try the equivalent and report back after work.

Ah, then you need to install Apache Arrow C++ manually.
See also:

https://arrow.apache.org/docs/dev/developers/python.html#building-on-linux-and-macos
#33814

I don't think 10.13 is still maintained by Apple. I'm still using it because I still have 32 bit apps, and 32 bit apps aren't supported by 10.15+.

BTW, is macOS 10.13 still maintained?

Officially, also 10.14 and 10.15 are no longer maintained (the release notes still shows them, but the last releases for 10.14 was more than a year ago, wiki says "Unsupported as of October 2021"), but I think we should still try to provide binaries for MacOS < 11 at the moment (as long at that is reasonably possible).
(note, I don't use Mac myself, I don't have a good idea about how common it is to have a Mac that can't be updated to at least 10.14. According to this site, 10.14 (Mojave) should run typically on machines from 2012)

In general, I want to drop support for platforms that aren't maintained by upstream including macOS 10.13. We already have many supported platforms. I think that we don't have enough resources to support unmaintained platforms.
Anyway, we can discuss this on dev@arrow.apache.org.

authors: daviewales jorisvandenbossche kou


text:
Can't build Arrow on M1 Mac

Describe the bug, including details regarding any error messages, version, and platform.
I'm getting the following error on an M1 Mac:
Undefined symbols for architecture arm64:
  "testing::Matcher<std::__1::basic_string_view<char, std::__1::char_traits<char> > const&>::Matcher(char const*)", referenced from:
      testing::Matcher<std::__1::basic_string_view<char, std::__1::char_traits<char> > const&> testing::internal::MatcherCastImpl<std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, char const*>::CastImpl<true>(char const* const&, std::__1::integral_constant<bool, true>, std::__1::integral_constant<bool, true>) in array_test.cc.o
      testing::Matcher<std::__1::basic_string_view<char, std::__1::char_traits<char> > const&> testing::internal::MatcherCastImpl<std::__1::basic_string_view<char, std::__1::char_traits<char> > const&, char const*>::CastImpl<true>(char const* const&, std::__1::integral_constant<bool, true>, std::__1::integral_constant<bool, true>) in array_binary_test.cc.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Component(s)
C++

This is a known issue with brew's googletest distribution (it's not built in way that is usable by C++17 projects).  You will need to build googletest from source.  The simplest way to do this is to specify -DGTest_SOURCE="BUNDLED" to cmake.  This will configure the Arrow build to download and build googletest.

authors: judahrand westonpace


text:
[C++][Parquet] Remove TODO file, update READMEs

These files are outdated
Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wesm) / @wesm
Note: This issue was originally created as PARQUET-680. Please see the migration documentation for further details.

Wes McKinney / @wesm:
Outdated now after monorepo merge


text:
ARROW-1869: [JAVA] Fix LowCostIdentityHashMap name

JIRA: https://issues.apache.org/jira/browse/ARROW-1869
This PR fixes spelling error in class name for LowCostIdentityHashMap.
Follow-up for https://github.com/apache/arrow/pull/1150.

@wesm @yufeldman could you review? Thanks.

LGTM, but I suggest you open a separate JIRA, even though it is very small one

@yufeldman Thanks for review. I created JIRA.

authors: sadikovi yufeldman


text:
ARROW-17868: [C++][Python] Restore the ARROW_PYTHON CMake option

Restore it but it's marked as a deprecated option. Because the Python component in Apache Arrow C++ was moved to PyArrow by ARROW-16340. It' removed in a feature release.
Users should use CMake presets instead of ARROW_PYTHON but CMake presets requires CMake 3.19 or later.

@github-actions crossbow submit -g nightly-tests -g nightly-release -g nightly-packaging

@github-actions crossbow submit -g nightly-tests -g nightly-release -g nightly-packaging

@github-actions crossbow submit -g nightly-tests -g nightly-release -g nightly-packaging

I think that this is ready to merge.
Build failures aren't related to this change.


text:
ARROW-17940: [Java][Gandiva] Implement Reserve for JavaBuffer

See previous discussions at https://github.com/apache/arrow/pull/14261

This PR is based on changes from https://github.com/apache/arrow/pull/14310 so will need to wait for that to merge.

@github-actions crossbow submit java-jars

@github-actions crossbow submit java-jars


text:
GH-35600: [Python] Allow setting path to timezone db through python API

Rationale for this change
Add a function to change the path where timezone db should be found as a small wrapper around the setting of a C++ option GlobalOptions.
What changes are included in this PR?
New function configure_tzdb.
Are these changes tested?
Are there any user-facing changes?
No.

Closes: #35600

Are these changes tested?

For testing, you can certainly add a test that it raises an error on non-Windows platforms (the C++ function should already do that if the path cannot be configured but is determined at compile time)
I would maybe also use this function ourselves in for example the Appveyor build, by putting the downloaded db somewhere else and pointing to that location?

Yes, I was thinking same but adding a test in python test suite. It would be similar to the one in C++ were the db is copied to another location and the initialisation of the db is tested in that new location.
But having it only on AppVeyor would be better.
I am also thinking, do I only need to expose the wrapper in the __init__ or do I need another helper function defined there?

AppVeyor build is successful though there is an error I have to correct:
(arrow) C:\projects\arrow\python>python -c "import pyarrow;import os;path = os.path.expandvars(r'C:\Users\appveyor\Downloads\test\tzdata');pa.set_timezone_db_path(path)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'pa' is not defined

I guess the AppVeyor build passes because the timezone database is moved only after building Arrow C++ (to not have C++ tests failing as they expect the timezone database to be in the standard location, we are configuring the path only for the Python part, if I am understanding this correctly) and the python tests are somehow not affected by it - or the affected ones are skipped.

I guess the AppVeyor build passes because the timezone database is moved only after building Arrow C++ ...  and the python tests are somehow not affected by it - or the affected ones are skipped.

Yeah, we indeed have a windows_has_tzdata utility function, and the tests are skipped if the tz database is not found in that location.

Yeah, we indeed have a windows_has_tzdata utility function, and the tests are skipped if the tz database is not found in that location.

Oh, of course! 🤦♀
I will need to change the utility function also. Even this will probably be easier if we use the env var for setting the path.

I am trying to see why the code is failing if the database is copied/installed into a new location.
For the purpose of testing on the CI, the path to the new location of the tzdata db is set with an env variable PYARROW_TZDATA_PATH here. I check for it in the python/pyarrow/tests/conftest.py and if the env var is set, I call set_timezone_db_path  on the new path (initializing the C++ GlobalOptions with a new path).
If I set the env var to the default path (%USERPROFILE%\Downloads\tzdata) the code works.
But if I install (or copy) the tzdata db into another location, I get OSError: Unable to get Timezone database version from C:\Users\appveyor\Downloads\test\tzdata\
3496` https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/47978107#L3495
Any ideas what am I missing @pitrou ?

❤

Ha, you're welcome :-)

Thanks all!

authors: AlenkaF jorisvandenbossche pitrou


text:
GH-39855: [Python] ListView support for pa.array()

WIP
Rationale for this change
Add pa.array() instantiation support for ListView and LargeListView formats.
What changes are included in this PR?

pa.array() supports creating ListView and LargeListView types
ListArray, LargeListArray now have their size initialized before adding elements during python-to-arrow conversion. This allows these types to be convertible to ListViewArray and LargeListViewArray types.

Are these changes tested?
WIP
Are there any user-facing changes?
Yes, new feature added.

Closes: #39855

>>> import pyarrow as pa
>>> pa.array([[1, 2], [3, 4], None], type=pa.list_view(pa.int8()))
<pyarrow.lib.ListViewArray object at 0x13df6e740>
[
  [
    1,
    2
  ],
  [
    3,
    4
  ],
  null
]


text:
GH-43349: [R] Fix altrep string columns from readr

Rationale for this change
To resolve the reverse dependency issue with parquetize
What changes are included in this PR?
One step towards resolving the issue
Are these changes tested?
yes
Are there any user-facing changes?
no

GitHub Issue: #43349

@ursabot please benchmark

@github-actions crossbow submit -g r


text:
GH-43902: [Java] Support for Long memory addresses

Rationale for this change
The usage of Integer instead of Long must be encouraged with the usage of memory sizing, indexing and addresses.
What changes are included in this PR?
This PR refactors the usage of Integer into Long along with utilities refactors.
Are these changes tested?
Existing test cases.
Are there any user-facing changes?
Yes, certain API calls may subject changes.

GitHub Issue: #43902

I think this is the right idea but we should be adding the long version as an overload in most cases (and if it's in return position we need to add a separate method), then we need to deprecate the int variants

Right. I will update. Thank you!

@lidavidm I updated. Sorry I have missed a few things earlier.

Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  The following options were not recognized by any processor: '[skipDefs, atfDoNotCache]'
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[995,19] [removal] reserve(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[894,19] [removal] add(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[64,46] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[77,16] Unsafe is internal proprietary API and may be removed in a future release

Can we ensure we fix the warnings? (Also can we enable warnings-as-errors?)

Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  The following options were not recognized by any processor: '[skipDefs, atfDoNotCache]'
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[27,15] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[995,19] [removal] reserve(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/BaseAllocator.java:[894,19] [removal] add(int) in AllocationReservation has been deprecated and marked for removal
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[36,23] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[64,46] Unsafe is internal proprietary API and may be removed in a future release
Warning:  /build/java/memory/memory-core/src/main/java/org/apache/arrow/memory/util/MemoryUtil.java:[77,16] Unsafe is internal proprietary API and may be removed in a future release

Can we ensure we fix the warnings? (Also can we enable warnings-as-errors?)

Right, we discussed this earlier as well. We can create a ticket to follow up with this. But in relation to this context, I think we need to sort of organize the warning -> error migration as Unsafe components probably need some waiting. But for the rest and for each module I can enable the warnings to errors iteratively.
I created a parent issue https://github.com/apache/arrow/issues/43941, and I will update each module upon feasibility and create new issues for cases like Unsafe to be fixed when we are supporting minimal JDKs in future.

At least here, let's fix any warnings about deprecation of things we deprecated in this PR

@lidavidm fixed the warnings and locally they don't appear, but let's check via the CI logs as well.

Sorry for not having commented earlier but I wonder how much the changes do increase the memory footprint overall. When many buffers are created, this has a non negligeable impact abd I wonder if we should not adapt the fields based on the actual need for more than 2gb (4gb if we consider it unsigned) allocations

Hmm, is there a use case where you'd expect many tens of thousands of buffers allocated but memory is constrained (measured in megabytes, I suppose)?

Hmm, is there a use case where you'd expect many tens of thousands of buffers allocated but memory is constrained (measured in megabytes, I suppose)?

The query engine I work with runs multiple fragments concurrently and we are regularly seing hundreds of thousands of buffers allocated and being live. We try to keep the heap size at a minimum size (4g/8g) and use as much direct memory as possible and so arrow bufs footprint (along with related objects) have a large impact for us

But even for 1 million buffers, 4 bytes of overhead per buffer is 4 megabytes (or we can say 8 bytes of overhead: 8 megabytes).
I suppose we could continue storing int but require all interfaces to use long...

authors: laurentgo lidavidm vibhatha


text:
ARROW-961: [Python] Rename InMemoryOutputStream to BufferOutputStream

Better API naming consistency with C++

Here's the appveyor build: https://ci.appveyor.com/project/wesm/arrow/build/1.0.441


text:
ARROW-7391: [C++][Dataset] Remove Expression subclasses from bindings

Serialization is implemented by converting Expressions to Arrays then writing a tiny IPC file. This is a ridiculous way to serialize Expressions but it should be acceptable since these classes are destined to be replaced by arrow::engine::Expr

Agree that exposing ds.field and ds.scalar should be sufficient on the python side.

@kszucs @jorisvandenbossche PTAL

@github-actions autotune everything

authors: bkietz kszucs nealrichardson


text:
[Task]: Embed public Beam IO Performance Metrics

What needs to happen?
The output of this task are embedded visualizations of Beam IO performance metrics.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[X] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue

repo: apache/beam


text:
[BEAM-8078] streaming_wordcount_debugging.py is missing a test

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @udim
Kindly review the request

retest this please

Waiting to check that streaming_wordcount_debugging_test runs in python precommit

retest this please

For some reason my comment launched tests on dbf0396 instead of ffefeba. Trying again

And now the previous comment launched tests on ffefeba

Run Portable_Python PreCommit

sigh, again ran the test on the previous commit

Run Portable_Python PreCommit

authors: Tesio udim


text:
[BEAM-9278] hbase-shaded-client as provided dependency

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @iemejia


text:
HIFIO Cassandra tests were failing if run twice in a row without a clean

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

The embedded cassandra tests work correctly the first time they are run after a mvn clean - on any test phase execution of the jdk1.8-tests directory after that (ie, any time you run this unit tests again), they will fail. Thus, this passes our "mvn clean verify" runs and is not breaking CI, but will likely break any devs running tests in this directory locally without clean-ing. (aka, me)
R @dhalperi
R @tgroh
...whoever gets to this first

The root cause here is that the embedded cassandra service is not entirely in-memory - it stores some files to disk (which get wiped out by mvn clean.)
So the unit tests should clean up the cassandra instance when it destroys the instance in the @afterclass (as we do in other embedded tests), but in this case they don't.
To verify that this works, I ran this command sequence:
mvn clean verify -pl sdks/java/io/hadoop/jdk1.8-tests/
mvn verify -pl sdks/java/io/hadoop/jdk1.8-tests/

Coverage decreased (-0.01%) to 70.193% when pulling e72b13577d9d836aa7f0bc494725b1607aa8eb83 on ssisk:fix-cassandra-uts into 7587d29f50654397f00a754596bfc5a225d4a64f on apache:master.

authors: coveralls ssisk


text:
Fix Dataflow v2 test failed tag container if project contains colon

Follow up of #31938 . Tested with command
./gradlew :runners:google-cloud-dataflow-java:validatesRunnerV2Test --tests *test*StringSetMetrics -PgcpProject=google.com:clouddfe -PgcpRegion=us-central1 -PgcpTempRoot=gs://.../tmp

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @damccorm


text:
[BEAM-1347] Provide an abstraction which creates an Iterator view over the Beam Fn State API

Combining this with the DataStreams.DataStreamDecoder converts the Beam Fn State API into a
an input stream backed by multiple logical chunks.
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @vikkyrk

Retest this please

LGTM once tests pass.

Coverage increased (+0.003%) to 69.971% when pulling 701f0a2b23be172c1b385910dd585fd833923a41 on lukecwik:state_api into 95cd37faff1c57dc4184ed001d8d168bc42d174e on apache:master.

retest this please

Coverage increased (+0.01%) to 69.977% when pulling 701f0a2b23be172c1b385910dd585fd833923a41 on lukecwik:state_api into 95cd37faff1c57dc4184ed001d8d168bc42d174e on apache:master.

authors: coveralls lukecwik vikkyrk


text:
DSL editor: allow ref to entity anywhere

This change means that the DSL editor will let you choose a reference to an entity in any config key (rather than only if the config key is of type Entity).
This makes it more consistent with the rest of the DSL editor: e.g. if generating attributeWhenReady, it doesn't type-check the attributes' types to filter the view to only those that match the config key's type. This is a good thing, because otherwise we'd have to repeat all the type coercion rules in the UI, for it to know what is legal!

I tested this by running npm run start in brooklyn-ui/ui-modules/blueprint-composer. I chose a couple of arbitrary entities, and successfully wired up a config key's value to reference the other entity:

retest this please
(as been aborted because build time > 40min. I tweaked the config to allow for more time)

retest this please
Failed with seemingly unrelated failure in a different module:
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project brooklyn-ui-location-manager: Failed to run task: 'npm install' failed. java.lang.InterruptedException -> [Help 1]

Ah, you beat me to it @tbouron (and I should have refreshed my browser page to see your comment!). The build worked after you requested the retest. My retest was unnecessary.

authors: aledsage tbouron

repo: apache/brooklyn-ui


text:
#358 Renamed postman demo to parceldemo

Renamed postman demo to parceldemo

@Divyansh1908 thanks.
Could you also rename the folder

Yeah renaming the folder would be good

authors: Divyansh1908 davsclaus mgubaidullin

repo: apache/camel-karavan


text:
Initial source code - Package names

The root java package name should be renamed to

org.apache.camel.quarkus

I wonder whether the java package renaming should be a part of the code migration from the Quarkus repo?
I could try to modify my import script that I used to produce the current state of #3 .  Not sure how well that can be done using git filter-branch --filter-tree and sed s/import io.quarkus/import org.apache.camel.quarkus/ , but it would result in a cleaner history for sure.
I am also not sure modifying the commits in this way would be acceptable from the legal point of view.
WDYT?

I think we cannot leave the package with the old name. We can import with that package name, merge and then change the package name completely with a new commit done in the ASF Camel repo.

authors: davsclaus oscerd ppalaga

repo: apache/camel-quarkus


text:
Deprecate kotlin extension

Rethinking my comments in #6438. I think we should deprecate (for 3.15.0) and remove camel-quarkus-kotlin (in 3.16.0) along with the kotlin-dsl extension.
I don't see any major benefits to keeping camel-quarkus-kotlin. All it does is provide a Kotlin function wrapper around RouteBuilder.configure.
If folks want to build with Kotlin then it's simple enough to add quarkus-kotlin and write a RouteBuilder impl.
class Routes : RouteBuilder() {
    override fun configure() {
        from("timer:tick?period=5s")
            .log("Hello World")
    }
}

Camel Quarkus 3.15 will be based on Camel 4.7?
Camel Quarkus 3.16 will be based on Camel 4.9?
Double-checking that we can remove the Kotlin DSl part with Camel 4.9.

CQ 3.15.x will be LTS and will stay aligned to Camel 4.8.x
CQ 3.16.0 will probably be based on whatever the latest Camel 4.8.x release is. It depends if Camel 4.9.0 is available (unlikely TBH). So I guess we delay the extension removal until we integrate Camel 4.9.0.

authors: apupier jamesnetherton


text:
CAMEL-11879: Upgraded lucene version to 7.0.0

https://issues.apache.org/jira/browse/CAMEL-11879
Upgraded Lucene to latest version 7.0.0.
Passed the iTests for Karaf and OSGi without issues, including the camel-lucene component tests.

I think camel-solr may also need to be updated, or does it pull in lucene 7 as well?
And do the spring-boot tests work? There is an itest for spring boot where you can run tests for lucene and solr as well.

Hi Claus,
Okay, i just realized i missed to update Solr as well. I am updating it.  Well, running the spring-boot Itests does pose the below,
Caused by: java.lang.ClassNotFoundException: ch.qos.logback.classic.spi.ThrowableProxy
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
so, i need to check if these are related with the upgrade.

The logging error is just some noise when the test shutdown - its harmless - some of the others do that too

Okay.. i missed to upgrade Solr. I will do that and still check the spring-boot iTests again!

Okay, so i have completed the upgrade for both components, camel-lucene and camel-solr. There were some api changes in Solr.
Tests pass, except the spring-boot iTests.

Failed spring-boot test for Lucene: https://gist.github.com/vrlgohel/ec9eff21efabe8a66aee10a12aef3770

Failed spring-boot test for Solr:
https://gist.github.com/vrlgohel/aa94a12716bec565afcd4b9bed88fcc5

Yeah, interestingly i saw this even after upgrading,
7.0.0
[5,7)
and the spring-boot iTests have the managed spring-boot-dependencies import,

org.springframework.boot
spring-boot-dependencies
${spring-boot-version}
pom
import

I am not sure if am understanding this right.  Does it mean we are stranded at the moment until we add support for Spring boot 2.x ?

I would make the commit though and then squash the changes, so we can maybe reopen this later ?

Yeah its a bit harder to override those spring boot dependencies. Also Lucene 7.0 is very new so not all users are using it.

Yeah sure lets look at this for 2.21 version of Camel with Spring Boot 2

Hi Claus,
Okay,  the upgrade code is what i think should work without any issues for both camel-lucene and camel-solr. Do you want me to commit the changes ? and then wait for the next 2.21 release ? Let me know so, i can do accordingly.

Ah yeah sure commit your current changes then we can pickup this PR later

Cool. Thank you Claus. Doing it in just some time..

Done..  I have committed the changes.

Will this PR be integrated in the next release ?

For the record the Sorl 7.0.0 version contains a zero-day exploit. As the hardest part of the job is done it may be good to upgrade to the Solr 7.1.0 fixed version.
What do you think ?

@kevinlacire, thanks for the suggestion. Unless, we do not have much api changes and solr version 7.1.0 which is just released, plays well with lucene, we can look to upgrading to the latest to cover the issue you highlighted. However, this will happen once we upgrade spring-boot.

Any update on this PR ?
Will it be integrated into the next release ?
Sorry for my impatience :)

When we have boot2ga work done and merged to master. We can revisit this and try to upgrade solr and lucene as SB2 ought to support newer versions now

Maybe, since we merged boot2ga branch, we can start to look at this.

By the way the OSGi bundle for Lucene version range is wrong, because we need to use (6,7] and use the right bundle version

Yeah we should try to get this updated now that master has been upgraded to SB2

Solr 7.3.0 was just released last week.
It contains a security fix :
CVE-2018-1308: XXE attack through Apache Solr's DIH's dataConfig request parameter
Severity: Major
Vendor:
The Apache Software Foundation
Versions Affected:
Solr 1.2 to 6.6.2
Solr 7.0.0 to 7.2.1
I think that would be a great idea to use this version.
Even if that does not really affect the Apache Camel's code.

We first need to understand what version is using SB 2.0.1 and as far as I can see, it is based on Solr 6.6.3

Perhaps could we relocate the Solr classes with the maven shade plugin ?
This way we would not have any conflict in the future.

@davsclaus Hi Claus, We have upgraded the solr version of our project to 7.2.1 and the camel-solr doesn't work anymore because of the deprecated methods. Can you please give me an update on this PR? i would really like this to be patched soon.

We can update both Lucene and Solr, but we need to be sure SB 2.0.3 will be released before our release of 2.22.0

@davsclaus Hello, i believe the release of 2.22.0 is on 13th june. Can you please let me know if we are including the SOLR upgrade with this release? As i mentioned before it's very crucial.

The release is by the end of June.

I merged this PR and then I updated Lucene and Solr to 7.2.1
https://github.com/apache/camel/commit/f76d8bcbff81c7180003a68f32fa3dd880ec8f9b
Now we just need SB 2.0.3

@oscerd Thank you. :)

authors: chirag0103 davsclaus kevinlacire ludovic-boutros oscerd vrlgohel

repo: apache/camel


text:
Added ability to create schemas only when using cloudstack-setup-data…

…bases
Description
This PR adds the ability to run the cloudstack-setup-databases script without having to know the root user credentials.
The cloud and cloud_usage databases and cloud user can be manually configured by a different admin prior to running the script.
It will allow the script to be called like this:
./cloudstack-setup-databases user:password@host:port -s
or
./cloudstack-setup-databases user:password@host:port --schema-only
Using this flag will instruct to skip the SQL files responsible for creating the databases, user and granting permissions to the user. It will still run all other SQL scripts that need to be run before starting the management server for the first time. Initial databases and user creation will be the responsibility of the administrator.
Doc PR here: https://github.com/apache/cloudstack-documentation/pull/229

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[x] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

This has been tested by creating the cloud and cloud_usage databases and cloud user prior to executing the script, and then executing the script like this:
./cloudstack-setup-databases user:password@host:port -s
or
./cloudstack-setup-databases user:password@host:port --schema-only
An extra option is listed from:
./cloudstack-setup-databases -h
  -s, --schema-only     Creates the db schema without having to pass root
                        credentials - Please note: The databases (cloud,
                        cloud_usage) and user (cloud) has to be configured
                        manuallyprior to running this script when using this
                        flag.

@blueorangutan package

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 487

@blueorangutan test

@Spaceman1984 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1215)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43476 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5187-t1215-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 87 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Failure
530.12
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Failure
502.25
test_vpc_redundant.py

@blueorangutan package

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 506

@blueorangutan package

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 509

@blueorangutan test

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1226)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40205 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5187-t1226-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 87 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_02_deploy_and_upgrade_kubernetes_cluster
Failure
552.13
test_kubernetes_clusters.py

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 680

@blueorangutan test

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1404)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 55784 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5187-t1404-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_primary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 84 look OK, 5 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_add_primary_storage_disabled_host
Error
0.55
test_primary_storage.py

test_01_primary_storage_nfs
Error
0.10
test_primary_storage.py

ContextSuite context=TestStorageTags>:setup
Error
0.18
test_primary_storage.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Failure
521.22
test_internal_lb.py

test_01_invalid_upgrade_kubernetes_cluster
Failure
3606.28
test_kubernetes_clusters.py

test_02_deploy_and_upgrade_kubernetes_cluster
Failure
3614.50
test_kubernetes_clusters.py

test_03_deploy_and_scale_kubernetes_cluster
Failure
0.05
test_kubernetes_clusters.py

test_04_basic_lifecycle_kubernetes_cluster
Failure
0.05
test_kubernetes_clusters.py

test_05_delete_kubernetes_cluster
Failure
0.04
test_kubernetes_clusters.py

test_07_deploy_kubernetes_ha_cluster
Failure
0.05
test_kubernetes_clusters.py

test_08_deploy_and_upgrade_kubernetes_ha_cluster
Failure
0.04
test_kubernetes_clusters.py

test_09_delete_kubernetes_ha_cluster
Failure
0.04
test_kubernetes_clusters.py

ContextSuite context=TestKubernetesCluster>:teardown
Error
89.41
test_kubernetes_clusters.py

test_02_list_snapshots_with_removed_data_store
Error
1.17
test_snapshots.py

test_01_secure_vm_migration
Error
158.48
test_vm_life_cycle.py

test_02_unsecure_vm_migration
Error
271.83
test_vm_life_cycle.py

test_03_secured_to_nonsecured_vm_migration
Error
144.80
test_vm_life_cycle.py

test_08_migrate_vm
Error
43.66
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Failure
1510.42
test_vm_life_cycle.py

Trillian test result (tid-1448)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46832 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5187-t1448-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 87 look OK, 2 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Failure
490.01
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Failure
528.37
test_vpc_redundant.py

test_disable_oobm_ha_state_ineligible
Error
1512.83
test_hostha_kvm.py

LGTM

authors: Spaceman1984 alexandremattioli blueorangutan borisstoyanov sureshanaparti

repo: apache/cloudstack


text:
UI: Fix Volume Snapshot from user VM as admin

Description
Fixes: #6795

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[x] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[x] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?
Take volume snapshots for user VMs from the instance view as admin, the volume is correctly displayed and snapshot operation succeeded

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

Tested and fixes the issue for me. Thank-you @nvazquez

authors: blueorangutan kohrar nvazquez