text:
configure: error: please install magic library

checking for libpcap... /home/elastic/moloch-master/thirdparty/libpcap-1.9.1
checking for nl_socket_alloc in -lnl-3... no
checking for nl_socket_alloc in -lnl... no
checking for nl_handle_alloc in -lnl... no
checking for yara... thirdparty/yara/yara-3.10.0
checking for maxminddb... thirdparty/libmaxminddb-1.3.2
checking for glib2... thirdparty/glib-2.56.2
checking for curl... thirdparty/curl-7.66.0
checking for lua... thirdparty/lua-5.3.5
checking for magic... checking for main in -lmagic... no
configure: error: please install magic library
make: *** No targets specified and no makefile found.  Stop.
MOLOCH: make failed

install moloch ，configure: error: please install magic library。

Need to install the libmagic/libfile library

repo: aol/moloch


text:
NLQ Analysis for QALD-9 test dataset is not working as expected

NLQ Analysis for QALD-9 test dataset is not working as expected (upper/lower case issue):
For example (qald-9-test-multilingual) the following question will not be identified as "Is" question:
{       "language" : "en",       "string" : "Is Pamela Anderson a vegan?",       "keywords" : "Pamela Anderson,  vegan"     }

Thanks for your suggestion. I just like to mention that this class is deprecated and not in use. We replaced it with the "OneNLQAnalysis.java" class in our work to determine the NLQ type. We are also preparing for the Web version for CBench and it will be available soon.
I will remove this class in a few days.
Thanks again
Abdelghny Orogat

source: gharchive/pull-request

authors: aorogat muncca

repo: aorogat/CBench

license_type: permissive

license_source: github-api


text:
Installation is failing w/ storybook dependency

Describe the bug
Pulled recent changes and tried to run builds on different envs and got this bug:
error Couldn't find package "storybook__addon-knobs@*" required by "@types/storybook__addon-knobs@^5.0.0" on the "npm" registry.

To Reproduce
Run yarn install
Expected behavior
Installation should go fine
Environment (please complete the following information):

superset-ui version: latest/any

This should be fixed now.

The issue seem to be still present.

source: gharchive/issue

authors: b2kdaman kristw oziniak

repo: apache-superset/superset-ui-plugins


text:
refactor: Relocating/renaming various chart controls

💔 Breaking Changes
🏆 Enhancements
Takes advantage of tweaks to tabOverride in control layouts from https://github.com/apache/incubator-superset/pull/11708
I'll post screenshots inline with code comments below, so it's not quite so overwhelming in the description.
📜 Documentation
🐛 Bug Fix
🏠 Internal

Opening this one for review, as it's been lingering for a while! Maybe we can merge these changes in, bump the versions, and start another PR with additional tweaks on other plugins.

@ktmud would you mind giving this round of tweaks a review? More to come on the heels of this one :)

LGTM!

authors: ktmud rusackas

repo: apache-superset/superset-ui


text:
ARTEMIS-1093 Full qualified queue name support

Broker should support full qualified queue names (FQQN)
as well as bare queue names. This means when clients access
to a queue they have two equivalent ways to do so. One way
is by queue names and the other is by FQQN (i.e. address::qname)
names. Currently only receiving is supported.

@clebertsuconic @mtaylor This is a resubmit of FQQN PR, with all tests fixed.

guys please hold on this commit, I found some garbage files need to remove.

OK, I removed the garbage files and reverted AmqpSendReceiveTest.

Please.  While the whole testsuite ?

There might be a problem with Jenkins machine, the failure relates to 'memory allocation'. Not related to my PR.
@clebertsuconic I ran the whole test suite on my local machine, there are few test failures but none of them related to my PR. They failed both with and without my PR. (I'll investigate those failures further when I got time).

@mtaylor I will let you merge this.. but this PR is +1 from me..
nice job @gaohoward

@gaohoward actually, the example managmeent is broken after this. Can you look please?

@clebertsuconic sure I'll take care of it. Thanks.

@clebertsuconic @mtaylor turns out I compared a String address with SimpleString, which always return false. Fixed that and added a simple test.
Jenkins for the moment seems down, I'll kick off a build when it's back.

Jenkins looking good. :)

authors: clebertsuconic gaohoward

repo: apache/activemq-artemis

license_source: bigquery


text:
Can't figure out how to make a create prepared statement with more than one key

This works :
	PREPARE cypher_stored_procedure(ag_catalog.agtype) AS
        SELECT * FROM ag_catalog.cypher('agg', $$
            CREATE (a:user {
				mail: $mail
            }) $$, $1) AS (a ag_catalog.agtype);

But this doesn't work :
	PREPARE cypher_stored_procedure(ag_catalog.agtype) AS
        SELECT * FROM ag_catalog.cypher('agg', $$
            CREATE (a:user {
				mail: $mail,
				address: $address
            }) $$, $1, $2) AS (a ag_catalog.agtype);

 ERROR: function ag_catalog.cypher(unknown, unknown, ag_catalog.agtype, unknown) does not exist (SQLSTATE 42883)
But this does not  work. What is the correct way to build this prepared statement procedure with more than one key ?

got it, you don't need to pass more than one placeholder

repo: apache/age


text:
[AMBARI-24020] Remove HDFS Disk Usage widget from host summary page

What changes were proposed in this pull request?
There are NameNode widgets on host summary page which are specific for the namespace containing this host. Since HDFS DIsk Usage metrics is an aggregate from all the namespaces, the corresponding widget on host page is confusing and should be removed from there.
How was this patch tested?
UI unit tests:
21798 passing (26s)
48 pending

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/2576/
Test PASSed.

authors: aBabiichuk asfgit

repo: apache/ambari


text:
AMBARI-25854: Adapt Ambari 2.7.5 from python2 to python3.9

What changes were proposed in this pull request?
This PR show the necessary adaption for Ambari 2.7.5 to adapt python3.9.
related JIRA Issue: https://issues.apache.org/jira/browse/AMBARI-25854
How was this patch tested?
unit tests,
manual tests

I am trying to apply this patch on branch-2.7, but https://github.com/wuzhuoming/ambari-python3/blame/AMBARI-25854-release-2.7.5/ambari-agent/src/main/python/ambari_agent/listeners/ConfigurationEventListener.py#L53 from line 53 to 56 these changes are not present in branch-2.7, seems you haven't applied the changes on top of branch-2.7.
Can you tell me the branch on top of which you have done your changes?

I am trying to apply this patch on branch-2.7, but https://github.com/wuzhuoming/ambari-python3/blame/AMBARI-25854-release-2.7.5/ambari-agent/src/main/python/ambari_agent/listeners/ConfigurationEventListener.py#L53 from line 53 to 56 these changes are not present in branch-2.7, seems you haven't applied the changes on top of branch-2.7. Can you tell me the branch on top of which you have done your changes?

Can you tell me the branch on top of which you have done your changes?

Hello, here ambari 2.7.5 source code is downloaded from the official website of ambari. Link: https://archive.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz

I am trying to apply this patch on branch-2.7, but https://github.com/wuzhuoming/ambari-python3/blame/AMBARI-25854-release-2.7.5/ambari-agent/src/main/python/ambari_agent/listeners/ConfigurationEventListener.py#L53 from line 53 to 56 these changes are not present in branch-2.7, seems you haven't applied the changes on top of branch-2.7. Can you tell me the branch on top of which you have done your changes?

Can you tell me the branch on top of which you have done your changes?

Hello, here ambari 2.7.5 source code is downloaded from the official website of ambari. Link: https://archive.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz

Please use the latest code from https://github.com/apache/ambari/tree/branch-2.7

I am trying to apply this patch on branch-2.7, but https://github.com/wuzhuoming/ambari-python3/blame/AMBARI-25854-release-2.7.5/ambari-agent/src/main/python/ambari_agent/listeners/ConfigurationEventListener.py#L53 from line 53 to 56 these changes are not present in branch-2.7, seems you haven't applied the changes on top of branch-2.7. Can you tell me the branch on top of which you have done your changes?

Can you tell me the branch on top of which you have done your changes?

Hello, here ambari 2.7.5 source code is downloaded from the official website of ambari. Link: https://archive.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz

Please use the latest code from https://github.com/apache/ambari/tree/branch-2.7

Working on it! We are comparing the code between branch-2.7 and release-2.7.5 and making some modification to solve the conflit. Thank you for your comment!

Do not delete .../.gitignore this file...

Hi I am getting below error, while creating ambari rpms
07:46:07,345 [ERROR] error /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass: Command failed.
07:46:07,346 [ERROR] Exit code: 1
07:46:07,346 [ERROR] Command: sh
07:46:07,346 [ERROR] Arguments: -c node scripts/build.js
07:46:07,346 [ERROR] Directory: /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass
07:46:07,346 [ERROR] Output:
07:46:07,346 [ERROR] Building: /workspace/ambari/contrib/views/files/src/main/resources/ui/node/node /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
07:46:07,347 [ERROR] gyp info it worked if it ends with ok
07:46:07,347 [ERROR] gyp verb cli [ '/workspace/ambari/contrib/views/files/src/main/resources/ui/node/node',
07:46:07,347 [ERROR] gyp verb cli   '/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js',
07:46:07,347 [ERROR] gyp verb cli   'rebuild',
07:46:07,347 [ERROR] gyp verb cli   '--verbose',
07:46:07,347 [ERROR] gyp verb cli   '--libsass_ext=',
07:46:07,347 [ERROR] gyp verb cli   '--libsass_cflags=',
07:46:07,347 [ERROR] gyp verb cli   '--libsass_ldflags=',
07:46:07,347 [ERROR] gyp verb cli   '--libsass_library=' ]
07:46:07,347 [ERROR] gyp info using node-gyp@3.6.0
07:46:07,347 [ERROR] gyp info using node@4.5.0 | linux | x64
07:46:07,347 [ERROR] gyp verb command rebuild []
07:46:07,347 [ERROR] gyp verb command clean []
07:46:07,347 [ERROR] gyp verb clean removing "build" directory
07:46:07,347 [ERROR] gyp verb command configure []
07:46:07,347 [ERROR] gyp verb check python checking for Python executable "python2" in the PATH
07:46:07,347 [ERROR] gyp verb which failed Error: not found: python2
07:46:07,347 [ERROR] gyp verb which failed     at getNotFoundError (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:13:12)
07:46:07,347 [ERROR] gyp verb which failed     at F (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:68:19)
07:46:07,347 [ERROR] gyp verb which failed     at E (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:80:29)
07:46:07,347 [ERROR] gyp verb which failed     at /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:89:16
07:46:07,347 [ERROR] gyp verb which failed     at /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/isexe/index.js:42:5
07:46:07,347 [ERROR] gyp verb which failed     at /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/isexe/mode.js:8:5
07:46:07,346 [INFO] info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
07:46:07,347 [ERROR] gyp verb which failed     at FSReqWrap.oncomplete (fs.js:82:15)
07:46:07,347 [ERROR] gyp verb which failed  python2 { [Error: not found: python2] code: 'ENOENT' }
07:46:07,347 [ERROR] gyp verb check python checking for Python executable "python" in the PATH
07:46:07,347 [ERROR] gyp verb which succeeded python /usr/bin/python
07:46:07,347 [ERROR] gyp verb check python version /usr/bin/python -c "import platform; print(platform.python_version());" returned: "3.9.6\n"
07:46:07,347 [ERROR] gyp ERR! configure error
07:46:07,347 [ERROR] gyp ERR! stack Error: Python executable "/usr/bin/python" is v3.9.6, which is not supported by gyp.
07:46:07,347 [ERROR] gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
07:46:07,347 [ERROR] gyp ERR! stack     at Object.failPythonVersion (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/lib/configure.js:491:19)
07:46:07,347 [ERROR] gyp ERR! stack     at Object. (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/lib/configure.js:473:14)
07:46:07,347 [ERROR] gyp ERR! stack     at ChildProcess.exithandler (child_process.js:204:7)
07:46:07,347 [ERROR] gyp ERR! stack     at emitTwo (events.js:87:13)
07:46:07,347 [ERROR] gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
07:46:07,347 [ERROR] gyp ERR! stack     at maybeClose (internal/child_process.js:829:16)
07:46:07,347 [ERROR] gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
07:46:07,347 [ERROR] gyp ERR! System Linux 3.10.0-1160.88.1.el7.x86_64
07:46:07,347 [ERROR] gyp ERR! command "/workspace/ambari/contrib/views/files/src/main/resources/ui/node/node" "/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
07:46:07,347 [ERROR] gyp ERR! cwd /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass
07:46:07,348 [ERROR] gyp ERR! node -v v4.5.0
07:46:07,348 [ERROR] gyp ERR! node-gyp -v v3.6.0
07:46:07,348 [ERROR] gyp ERR! not ok

Hi I am getting below error, while creating ambari rpms
07:46:07,345 [ERROR] error /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass: Command failed. 07:46:07,346 [ERROR] Exit code: 1 07:46:07,346 [ERROR] Command: sh 07:46:07,346 [ERROR] Arguments: -c node scripts/build.js 07:46:07,346 [ERROR] Directory: /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass 07:46:07,346 [ERROR] Output: 07:46:07,346 [ERROR] Building: /workspace/ambari/contrib/views/files/src/main/resources/ui/node/node /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library= 07:46:07,347 [ERROR] gyp info it worked if it ends with ok 07:46:07,347 [ERROR] gyp verb cli [ '/workspace/ambari/contrib/views/files/src/main/resources/ui/node/node', 07:46:07,347 [ERROR] gyp verb cli '/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js', 07:46:07,347 [ERROR] gyp verb cli 'rebuild', 07:46:07,347 [ERROR] gyp verb cli '--verbose', 07:46:07,347 [ERROR] gyp verb cli '--libsass_ext=', 07:46:07,347 [ERROR] gyp verb cli '--libsass_cflags=', 07:46:07,347 [ERROR] gyp verb cli '--libsass_ldflags=', 07:46:07,347 [ERROR] gyp verb cli '--libsass_library=' ] 07:46:07,347 [ERROR] gyp info using node-gyp@3.6.0 07:46:07,347 [ERROR] gyp info using node@4.5.0 | linux | x64 07:46:07,347 [ERROR] gyp verb command rebuild [] 07:46:07,347 [ERROR] gyp verb command clean [] 07:46:07,347 [ERROR] gyp verb clean removing "build" directory 07:46:07,347 [ERROR] gyp verb command configure [] 07:46:07,347 [ERROR] gyp verb check python checking for Python executable "python2" in the PATH 07:46:07,347 [ERROR] gyp verb which failed Error: not found: python2 07:46:07,347 [ERROR] gyp verb which failed at getNotFoundError (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:13:12) 07:46:07,347 [ERROR] gyp verb which failed at F (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:68:19) 07:46:07,347 [ERROR] gyp verb which failed at E (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:80:29) 07:46:07,347 [ERROR] gyp verb which failed at /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/which/which.js:89:16 07:46:07,347 [ERROR] gyp verb which failed at /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/isexe/index.js:42:5 07:46:07,347 [ERROR] gyp verb which failed at /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/isexe/mode.js:8:5 07:46:07,346 [INFO] info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command. 07:46:07,347 [ERROR] gyp verb which failed at FSReqWrap.oncomplete (fs.js:82:15) 07:46:07,347 [ERROR] gyp verb which failed python2 { [Error: not found: python2] code: 'ENOENT' } 07:46:07,347 [ERROR] gyp verb check python checking for Python executable "python" in the PATH 07:46:07,347 [ERROR] gyp verb which succeeded python /usr/bin/python 07:46:07,347 [ERROR] gyp verb check python version /usr/bin/python -c "import platform; print(platform.python_version());" returned: "3.9.6\n" 07:46:07,347 [ERROR] gyp ERR! configure error 07:46:07,347 [ERROR] gyp ERR! stack Error: Python executable "/usr/bin/python" is v3.9.6, which is not supported by gyp. 07:46:07,347 [ERROR] gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0. 07:46:07,347 [ERROR] gyp ERR! stack at Object.failPythonVersion (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/lib/configure.js:491:19) 07:46:07,347 [ERROR] gyp ERR! stack at Object. (/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/lib/configure.js:473:14) 07:46:07,347 [ERROR] gyp ERR! stack at ChildProcess.exithandler (child_process.js:204:7) 07:46:07,347 [ERROR] gyp ERR! stack at emitTwo (events.js:87:13) 07:46:07,347 [ERROR] gyp ERR! stack at ChildProcess.emit (events.js:172:7) 07:46:07,347 [ERROR] gyp ERR! stack at maybeClose (internal/child_process.js:829:16) 07:46:07,347 [ERROR] gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5) 07:46:07,347 [ERROR] gyp ERR! System Linux 3.10.0-1160.88.1.el7.x86_64 07:46:07,347 [ERROR] gyp ERR! command "/workspace/ambari/contrib/views/files/src/main/resources/ui/node/node" "/workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library=" 07:46:07,347 [ERROR] gyp ERR! cwd /workspace/ambari/contrib/views/files/src/main/resources/ui/node_modules/node-sass 07:46:07,348 [ERROR] gyp ERR! node -v v4.5.0 07:46:07,348 [ERROR] gyp ERR! node-gyp -v v3.6.0 07:46:07,348 [ERROR] gyp ERR! not ok

Sorry, the Ambari adapts to the ARM architecture and Python3 based on the openEuler system. Currently, the code of the x86 architecture branch is not verified or adapted. Our team is considering whether to proceed with this task.

Hi I am getting below error for both ambari-agent as well as ambari-server, while adding zookeeper service
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/hook.py", line 37, in
BeforeInstallHook().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 302, in execute
logging_level_str = logging._levelToName[self.logging_level]
KeyError: 'INFO'

What your system environment and architecture is? And how did you add the Zookeeper service?

@himanshumaurya09876

Hi,what your system environment and architecture is? And how did you add the Zookeeper service?

I am using RHEL 8 machines. I have installed ambari-agent, ambari-server and through Ambari UI install wizard I am adding Zookeeper service

@himanshumaurya09876

Hi,what your system environment and architecture is? And how did you add the Zookeeper service?

I am using RHEL 8 machines. I have installed ambari-agent, ambari-server and through Ambari UI install wizard I am adding Zookeeper service

You can create an issue and attach the error log or image. I'm not sure where you went wrong.

Hi I am getting below error while adding service trough Ambari UI Install wizard
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install     self._pkg_manager.install_package(package_name, self.__create_context())    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 211, in install_package     elif context.is_upgrade or context.use_repos or not self._check_existence(name):    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 278, in _check_existence     return self.yum_check_package_available(name)    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 287, in yum_check_package_available     import yum  # Python Yum API is much faster then other check methods. (even then "import rpm") ModuleNotFoundError: No module named 'yum'
While analyzing this I found that yum module is not available in python3, have you also observed same error?

Hi I am getting below error while adding service trough Ambari UI Install wizard
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install     self._pkg_manager.install_package(package_name, self.__create_context())    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 211, in install_package     elif context.is_upgrade or context.use_repos or not self._check_existence(name):    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 278, in _check_existence     return self.yum_check_package_available(name)    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 287, in yum_check_package_available     import yum  # Python Yum API is much faster then other check methods. (even then "import rpm") ModuleNotFoundError: No module named 'yum'
While analyzing this I found that yum module is not available in python3, have you also observed same error?

@himanshumaurya09876

Hi I am getting below error while adding service trough Ambari UI Install wizard
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install     self._pkg_manager.install_package(package_name, self.__create_context())    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 211, in install_package     elif context.is_upgrade or context.use_repos or not self._check_existence(name):    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 278, in _check_existence     return self.yum_check_package_available(name)    File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 287, in yum_check_package_available     import yum  # Python Yum API is much faster then other check methods. (even then "import rpm") ModuleNotFoundError: No module named 'yum'
While analyzing this I found that yum module is not available in python3, have you also observed same error?

@himanshumaurya09876
Hi
Python3 has deprecated the use of the yum module. I think we can abandon the call to yum_check_package_available and use the call to rpm_check_package_available.

Hi, I am getting below error while adding Ambari-Metrics to my cluster
Traceback (most recent call last):
File "//AMBARI_METRICS/package/scripts/metrics_collector.py", line 117, in
AmsCollector().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 361, in execute
method(env)
File "//AMBARI_METRICS/package/scripts/metrics_collector.py", line 38, in install
self.install_packages(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 863, in install_packages
retry_count=agent_stack_retry_count)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 164, in init
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 169, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 133, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
self._pkg_manager.install_package(package_name, self.__create_context())
File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
shell.repository_manager_executor(cmd, self.properties, context)
File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 744, in repository_manager_executor
call_result = subprocess_executor(cmd, timeout=-1, env=env)
File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 446, in subprocess_executor
lines = [line for line in output]
File "/usr/lib64/python3.6/contextlib.py", line 92, in exit
raise RuntimeError("generator didn't stop")
RuntimeError: generator didn't stop

File "/usr/lib64/python3.6/contextlib.py", line 92, in exit
raise RuntimeError("generator didn't stop")

This PR seemed to be trying to use Python 3.9, not 3.6. What happens when you upgrade that? @himanshumaurya09876

File "/usr/lib64/python3.6/contextlib.py", line 92, in exit
raise RuntimeError("generator didn't stop")

This PR seemed to be trying to use Python 3.9, not 3.6. What happens when you upgrade that? @himanshumaurya09876
Also, use code fences for formatting, please

Hi, I have tried both python 3.6 as well as 3.9. It's giving same error in both cases.

File "/usr/lib64/python3.6/contextlib.py", line 92, in exit
raise RuntimeError("generator didn't stop")

This PR seemed to be trying to use Python 3.9, not 3.6. What happens when you upgrade that? @himanshumaurya09876
Also, use code fences for formatting, please

Hi, I have tried both python 3.6 as well as 3.9. It's giving same error in both cases.

@himanshumaurya09876
Hi, thank you for your reply. You are advised to use Python 3.7 or later for verification.

Hi I am getting failures for below UT
`======================================================================
FAIL: test_count_hash_sum (TestResourceFilesKeeper.TestResourceFilesKeeper)
Traceback (most recent call last):
File "ambari-server/src/test/python/TestResourceFilesKeeper.py", line 349, in test_count_hash_sum
self.assertEqual(hash_sum, self.DUMMY_UNCHANGEABLE_PACKAGE_HASH)
AssertionError: 'a3a6ed30f109be83e02a4281d427ee237947248a' != 'c83bb4e4e34213300af27bf48dabf2b7568a174b'

a3a6ed30f109be83e02a4281d427ee237947248a

c83bb4e4e34213300af27bf48dabf2b7568a174b

======================================================================
FAIL: test_zip_directory (TestResourceFilesKeeper.TestResourceFilesKeeper)
Traceback (most recent call last):
File "ambari-common/src/test/python/mock/mock.py", line 1208, in patched
return func(*args, **keywargs)
File "ambari-server/src/test/python/TestResourceFilesKeeper.py", line 434, in test_zip_directory
self.assertEqual(hash_val, self.DUMMY_UNCHANGEABLE_PACKAGE_HASH)
AssertionError: 'a3a6ed30f109be83e02a4281d427ee237947248a' != 'c83bb4e4e34213300af27bf48dabf2b7568a174b'

a3a6ed30f109be83e02a4281d427ee237947248a

c83bb4e4e34213300af27bf48dabf2b7568a174b

======================================================================
FAIL: test_collect_zookeeper (TestPortAlert.TestPortAlert)
Traceback (most recent call last):
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1208, in patched
return func(*args, **keywargs)
File "/workspace/ambari/ambari-agent/src/test/python/ambari_agent/TestPortAlert.py", line 408, in test_collect_zookeeper
alert.collect()
File "/workspace/ambari/ambari-agent/src/main/python/ambari_agent/alerts/base_alert.py", line 189, in collect
self.collector.put(self.cluster_name, data)
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 962, in call
return _mock_self._mock_call(*args, **kwargs)
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1025, in _mock_call
ret_val = effect(*args, **kwargs)
File "/workspace/ambari/ambari-agent/src/test/python/ambari_agent/TestPortAlert.py", line 400, in collector_side_effect
self.assertEqual(data['state'], expected_state)
AssertionError: 'CRITICAL' != 'OK'

CRITICAL

OK

======================================================================
FAIL: test_collect_zookeeper_connectionTimeout (TestPortAlert.TestPortAlert)
Traceback (most recent call last):
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1208, in patched
return func(*args, **keywargs)
File "/workspace/ambari/ambari-agent/src/test/python/ambari_agent/TestPortAlert.py", line 525, in test_collect_zookeeper_connectionTimeout
alert.collect()
File "/workspace/ambari/ambari-agent/src/main/python/ambari_agent/alerts/base_alert.py", line 189, in collect
self.collector.put(self.cluster_name, data)
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 962, in call
return _mock_self._mock_call(*args, **kwargs)
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1025, in _mock_call
ret_val = effect(*args, **kwargs)
File "/workspace/ambari/ambari-agent/src/test/python/ambari_agent/TestPortAlert.py", line 518, in collector_side_effect
self.assertEqual(data['text'], expected_text)
AssertionError: "Connection failed: 'str' object has no attribute 'decode' to 192.168.0.1:2181" != 'Connection failed: Socket Timeout to 192.168.0.1:2181'

Connection failed: 'str' object has no attribute 'decode' to 192.168.0.1:2181

Connection failed: Socket Timeout to 192.168.0.1:2181

======================================================================
FAIL: test_collect_zookeeper_warning (TestPortAlert.TestPortAlert)
Traceback (most recent call last):
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1208, in patched
return func(*args, **keywargs)
File "/workspace/ambari/ambari-agent/src/test/python/ambari_agent/TestPortAlert.py", line 466, in test_collect_zookeeper_warning
alert.collect()
File "/workspace/ambari/ambari-agent/src/main/python/ambari_agent/alerts/base_alert.py", line 189, in collect
self.collector.put(self.cluster_name, data)
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 962, in call
return _mock_self._mock_call(*args, **kwargs)
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1025, in _mock_call
ret_val = effect(*args, **kwargs)
File "/workspace/ambari/ambari-agent/src/test/python/ambari_agent/TestPortAlert.py", line 458, in collector_side_effect
self.assertEqual(data['state'], expected_state)
AssertionError: 'CRITICAL' != 'WARNING'

CRITICAL

WARNING

======================================================================
FAIL: test_status_commands_clear_structured_out (TestScript.TestScript)
Traceback (most recent call last):
File "/workspace/ambari/ambari-common/src/test/python/mock/mock.py", line 1208, in patched
return func(*args, **keywargs)
File "/workspace/ambari/ambari-agent/src/test/python/resource_management/TestScript.py", line 85, in test_status_commands_clear_structured_out
self.assertFalse(open_mock.called)
AssertionError: True is not false

`

 ** wuzhuoming **

@wuzhuoming are you observing the same?

@himanshumaurya09876  @bhavikpatel9977  Thank for pointing out the issue, we're trying to reproduce this problem locally.
BTW, can someone tell us what os the CI/CD uses, we need to figure out which python version is supported.

BTW, can someone tell us what os the CI/CD uses, we need to figure out which python version is supported.

Good point. Are we also planning to have new branch for python-3 or we will merge it in the current 2.7.5 branch?

I think we better merge this feature into a new branch, trunk or 2.8 or others.
OK.

When are you guys planning to complete the validation?

When are you guys planning to complete the validation?
we may need one more week to fix some issues and we also welcome more contributors to join for fully validation on different os or scenary.

we may need one more week to fix some issues and we also welcome more contributors to join for fully validation on different os or scenary.

Sounds good!! Let's connect on slack channel, we will also contribute on this.

Due to above changes stderr:/var/lib/ambari-agent/data/errors.txt is filled with below logs WARNING:root:An exception occurred when calling the hasattr method...... WARNING:root:An exception occurred when calling the hasattr method...... WARNING:root:An exception occurred when calling the hasattr method......

@wenwj0 have you observed the same behaviour and fixed the same?

Due to above changes stderr:/var/lib/ambari-agent/data/errors.txt is filled with below logs WARNING:root:An exception occurred when calling the hasattr method...... WARNING:root:An exception occurred when calling the hasattr method...... WARNING:root:An exception occurred when calling the hasattr method......

We have fixed this issue now and you guys can try to test it. @bhavikpatel9977 @himanshumaurya09876
This problem occurs when we try hasattr(xxx) or getattr(xxx), if we want to set a parameter which is not in configuration dictionary, python3 will throw an exception while  python2 will not throw exception.

Hi everyone, we create a new PR (https://github.com/apache/ambari/pull/3751) to the trunk branch.
Please take a look :）


text:
Support showing Project's name on the navbar

When we visit FAQ from the FAQ dropdown:

It will show Docs:

Expected behavior
It will show the target repo's name:

Maybe related to https://github.com/apache/apisix-website/blob/master/website/src/theme/DocPage/index.js#L41 ?

@Baoyuantop Do you have time to take a look?

@Baoyuantop Do you have time to take a look?

Ok, let me take a look

Waiting for https://github.com/apache/apisix-ingress-controller/pull/605

authors: Baoyuantop juzhiyuan

repo: apache/apisix-website


text:
Update main DataFusion README

Which issue does this PR close?
re https://github.com/apache/arrow-datafusion/issues/3058 and https://github.com/apache/arrow-datafusion/issues/1814
Rationale for this change
I attended a conference last week CIDR and it was clear to me that DataFusion is more widely applicable than it is currently used, and part of that is lack of awareness in the broader community. Thus I wanted to try and communicate what DataFusion was / was good for more clearly.
Also as we have gathered more users we can now point to some more concrete examples
I also hope/plan to try and writeup a "datafusion architecture guide" soon as an additional way to encourage / grow our user base by making it more accessable for new people to see what we have (which is a lot!). Related to https://github.com/apache/arrow-datafusion/issues/980 from @xudong963
What changes are included in this PR?

Update the main readme with: more specific information about what datafusion is, and what it has been used for
Added some compare/contrast with pola.rs, DuckDB and Velox.

See the rendered page here: https://github.com/alamb/arrow-datafusion/tree/alamb/improve_user_docs#datafusion
Discussion
Dicsussion:
How much of this content should go in the user guide https://arrow.apache.org/datafusion?
I like the idea of all the non-release specific content being in the user guide . However, the website isn't updated all that often (as it takes a manual process as I recall)
The README is what shows up on the main crates.io page: https://crates.io/crates/datafusion as well as the landing page of github.

@xudong963 or @Dandandan  I wonder if you have any thoughts on this content?

This is a much better than what we currently have right now, thank you for writing this up.
I think having this stuff in the README still makes sense at this time since the website is not updated often (and we don't have a habit/rule for it AFAICT).
To fuel community growth, I think we should avoid all kinds of friction that inhibit advertising/publishing what we have (and what we are working on). Since the README file is easy to update, and it is a common gateway page for many new users, I think it is OK to keep it "rich" during this phase of the project lifecycle.

authors: alamb ozankabak

repo: apache/arrow-datafusion


text:
fix: ORDER BY window definition should work on null literal

Which issue does this PR close?

Closes #.
Rationale for this change

A small follow up to #8419. In coerce_window_frame, if first literal is Null, it returns an error "Cannot run range queries ...".
What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Thank you @alamb


text:
Add pub api for checking column index is sorted

I think parquet 23.0.0 will have properly exposed arrow definitions (added by @tustvold  in https://github.com/apache/arrow-rs/pull/2626)
Though now that I see this, I see the proposal is probably to move the functions columns_sorted and check_is_ordered into the parquet crate which makes sense to me (and will likely get reviewed by some others with more parquet knowledge)
@Ted-Jiang can you file a ticket / PR to do so?
Originally posted by @alamb in https://github.com/apache/arrow-datafusion/pull/3477#discussion_r971211638

label_issue.py automatically added labels {'parquet'} from #2849

authors: Ted-Jiang alamb

repo: apache/arrow-rs


text:
Drop Seek requirement for ParquetWriter

The Seek trait bound on arrow::file::writer::ParquetWriter prevents streaming parquet directly over a messaging bus. In the implementation, it seems to be used mainly for writing metadata about the size of the data body and the footer: https://github.com/apache/arrow-rs/blob/master/parquet/src/file/writer.rs#L196
It looks like the trait bound can be removed if the information on the number of bytes written is kept track of.

Fwiw I was able to achieve this in parquet2, but required re-writing the thirft library to offer us that number of written bytes. This is not doable without a backward incompatible change in thirft library.

Fwiw I was able to achieve this in parquet2, but required re-writing the thirft library to offer us that number of written bytes. This is not doable without a backward incompatible change in thirft library.

Oh, good to know. Do you think it might be possible to wrap the writer to just collect that information? We only need a cumulative count in that place, so a single struct that wraps a usize + the actual writer might be enough...

That is an excelent idea. Kudos for that!

Closed by https://github.com/apache/arrow-rs/pull/1719

authors: jorgecarleitao tustvold vlushn


text:
[R] Inconsistent application of type in Datasets via the schema

It looks like at least filter is not handling a column type specified by schema when specified in open_dataset. Reprex:

options("max.print" = 5)
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
## Set up the data
tf <- tempfile()
dir.create(tf)
write_dataset(quakes, tf)
## Works as expected
open_dataset(tf) %>%
 filter(stations == 41) %>%
 collect()
#> lat long depth mag stations
#> 1 -20.42 181.62 562 4.8 41
#> [ reached 'max' / getOption("max.print") -- omitted 11 rows ]
## errors as expected
open_dataset(tf) %>%
 filter(stations == "41") %>%
 collect()
#> Error: NotImplemented: Function equal has no kernel matching input types (array[int32], scalar[string])
## Ok let's change a column type
tf_reg <- open_dataset(tf)$schema
tf_reg$stations <- string()
## ok returns a character
open_dataset(tf, schema = tf_reg) %>%
 pull(stations) %>%
 typeof()
#> [1] "character"
## So if `stations` is character I think this should work?
open_dataset(tf, schema = tf_reg) %>%
 filter(stations == as.character("41")) %>%
 collect()
#> Error: Filter expression not supported for Arrow Datasets: stations == as.character("41")
#> Call collect() first to pull data into R.
## previous behaviour no longer works
open_dataset(tf, schema = tf_reg) %>%
 filter(stations == 41) %>%
 collect()
#> Error: NotImplemented: Function equal has no kernel matching input types (array[string], scalar[double])

Reporter: Sam Albers / @boshek
Note: This issue was originally created as ARROW-14324. Please see the migration documentation for further details.

Still reproducible on 12.0.1

authors: asfimport thisisnic

repo: apache/arrow


text:
[Website] Improve project description

Currently the Apache Arrow project descriptions that appear prominently at the top of the website and GitHub repo do not match and have not been updated in quite some time. Currently the description on the website is:

A cross-language development platform for in-memory analytics

and the description on GitHub is:

A multi-language toolbox for accelerated data interchange and in-memory processing

Given the immense growth in the adoption of Arrow that has occurred since we last updated these descriptions, and the current status of the Arrow format as a de facto standard with no directly comparable alternatives, I think it would be appropriate for us to be somewhat bolder in how we introduce the project. I also think that the description should include some mention of the fact that Arrow is a format in addition to a toolbox. And I think we should prefer simpler words ("fast" over "accelerated"; "toolbox" over "development platform).
Following this rationale, I propose that we change the description on both the website and GitHub to:

The universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

Thoughts?
Component(s)
Website

+1
(We may want to add "zero-copy" as a columnar format modifier.)

(We may want to add "zero-copy" as a columnar format modifier.)

I agree that it's important to highlight the fact that Arrow can enable zero-copy data interchange. But it might be difficult to incorporate "zero-copy" into this succinct description in a way that is accurate. Many successful applications of Arrow for data interchange are not truly "zero-copy"; instead they minimize the number of copies made while eliminating slow and computationally expensive data serialization/deserialization and transposition steps. But that's too many words to say in a succinct description. So I think we might be better off explaining this in other text below the description (which we already do to some extent, although maybe it could be improved).

It makes sense.

Issue resolved by pull request 44492
https://github.com/apache/arrow/pull/44492

authors: ianmcook kou


text:
ARROW-15067: [C++] Add tracing spans to the scanner

Continuing the work of @lidavidm in https://github.com/apache/arrow/pull/11964 in this PR. Rebased on master.

This can be closed in favor of the other PR, right?

authors: lidavidm mbrobbel


text:
ARROW-2537: [Ruby] Import

Imported from:

https://github.com/red-data-tools/red-arrow
https://github.com/red-data-tools/red-arrow-gpu

@kou, this is great, thank you! Since this is a large code import, we will want to go through the IP clearance process like with Go recently http://incubator.apache.org/ip-clearance/. This should not be too painful but will require a vote on the mailing list and then a lazy consensus clearance vote on general@incubator

@wesm OK. I'll read the document.

@wesm I read the document. I think that I don't have "incubator karma". Can you help me to process this?

IP Clearance processing must be executed either by an Officer or a Member of the ASF. If you are not an Officer or a Member, please contact your project chair who will find an appropriate volunteer. Incubator karma is also required. Please request karma from the incubator pmc if you do not have it.

I filled some fields:
Title:
Arrow Ruby Library Intellectual Property (IP) Clearance Status
Description:
The Arrow Ruby Library is a Ruby language bindings of the Apache Arrow columnar format
Project info:
he Apache Arrow PMC will be responsible for the code.
It will be integrated into the Apache Arrow project, into a new Ruby area of the main Arrow source tree and build system.
The following people will be managing this contribution:

TBD
...

Completed tasks are shown by the completion date (YYYY-MM-dd).
Identify the codebase:

Origin: https://github.com/red-data-tools/red-arrow as at commit ID abb0d3c0349c94189a700c25ab7ca97b17b09a6f and https://github.com/red-data-tools/red-arrow-gpu as at commit ID 29025cfdc92135fd81b42fbdc29497466a7454aa.
Submitted to Apache Arrow as GitHub pull request https://github.com/apache/arrow/pull/1990

Copyright:
Identify name recorded for software grant: Ruby Bindings of Apache Arrow
Verify distribution rights:
Corporations and individuals holding existing distribution rights:

Kouhei Sutou

DISCUSS: https://github.com/red-data-tools/red-arrow/commit/701386c16eec1aa72d3c349ae58a8ff94945862f is only a commit made by me. It just fix a URL. Is there a copyright of mikisou? If there is a copyright of him, I'll contact to him.
Generally, the result of checking off these items will be a Software Grant, CLA, and Corporate CLA for ASF licensed code, which must have no dependencies upon items whose licenses that are incompatible with the Apache License.
Organizational acceptance of responsibility for the project:
Related votes:

TBD

hi @kou -- yes, I will help you with the IP clearance. I'm an ASF Member now so I believe I have enough karma to make changes to the IP clearance files.
First, we need to have a PMC vote to accept the Ruby bindings. I'm going to start this now on the mailing list. We can get the IP clearance page set up while that is running, so we can have the Incubator clearance vote next week and then merge the PR.
Sorry for the delay; I have been traveling the last week and not available much on e-mail or GitHub

Thanks! I understand the schedule.

Sorry for the delay; I have been traveling the last week and not available much on e-mail or GitHub

No problem. Thanks for your help.

The IP clearance vote has passed. I'm going to rebase this and review, and merge on a green build if there are no problems

Codecov Report

Merging #1990 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1990   +/-   ##
=======================================
  Coverage   86.33%   86.33%
=======================================
  Files         242      242
  Lines       41103    41103
=======================================
  Hits        35487    35487
  Misses       5616     5616

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c5e95a...cb41c7a. Read the comment docs.

+1

authors: codecov-io kou wesm


text:
MINOR: [C++] Slight improvement for ArrayData device_type

Responding to feedback on #40807:
          This condition implies that, conversely, in non-debug mode we could immediately return when we encounter a buffer? Instead of continue looping on all buffers and children...

Originally posted in https://github.com/apache/arrow/pull/40807#discussion_r1611641179

FTR, I removed my handle from the PR description, as I don't want to receive notification spam when people merge commits in their Arrow fork :-)

authors: pitrou zeroshade


text:
ARROW-6284: [C++] Allow references in std::tuple when converting tuple to arrow array

Currently std::vector<std::tuple> can be converted to a table by treating each tuple as a row and by building arrays corresponding to each element of the tuple. This PR allows reference types (and const reference types) in elements of std::tuple which is to be converted. This is useful for converting user defined types. For example a vector of CustomType:
struct CustomType {
  int8_t i8;
  uint64_t u64;
  bool b;
  std::string s;

  auto tie() const {
    return std::tie(i8, u64, b, s);
  }
};

can be converted to a table like this:
using boost::adaptors::transform;

std::vector<CustomType> rows();
std::shared_ptr<Table> table;
auto rng_rows =
  transform(rows(), [](const CustomType& c) { return c.tie(); });
TableFromTupleRange(default_memory_pool(), rng_rows, names, &table);

This makes it easier to convert user defined types to arrow tables.

Codecov Report

Merging #5112 into master will increase coverage by 1.58%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #5112      +/-   ##
==========================================
+ Coverage   87.64%   89.22%   +1.58%
==========================================
  Files        1014      731     -283
  Lines      145922   104999   -40923
  Branches     1437        0    -1437
==========================================
- Hits       127887    93690   -34197
+ Misses      17673    11309    -6364
+ Partials      362        0     -362

Impacted Files
Coverage Δ

cpp/src/arrow/stl.h
100% <ø> (ø)
:arrow_up:

cpp/src/arrow/stl-test.cc
100% <100%> (ø)
:arrow_up:

cpp/src/arrow/util/thread-pool-test.cc
97.66% <0%> (-0.94%)
:arrow_down:

python/pyarrow/tests/test_parquet.py
96.38% <0%> (-0.07%)
:arrow_down:

js/src/util/fn.ts

go/arrow/memory/memory_avx2_amd64.go

rust/datafusion/src/execution/filter.rs

rust/arrow/src/csv/writer.rs

rust/datafusion/src/bin/main.rs

go/arrow/ipc/file_reader.go

... and 277 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ba0566...6376d10. Read the comment docs.

authors: codecov-io ozars


text:
ARROW-6866: [Java] Improve the performance of calculating hash code for struct vector

Improve the performance of hashCode(int) method for StructVector:

We can get the child vectors directly, so there is no need to get the name from the child vector and then use the name to get the vector.
The child vectors cannot be null, so there is no need to check it.

The performance improvement depends on the complexity of the hash algorithm. For computational intensive hash algorithms, the improvement can be small; while for simple hash algorithms, the improvement can be notable.

Improve the performance of hashCode(int) method for StructVector:

We can get the child vectors directly, so there is no need to get the name from the child vector and then use the name to get the vector.
The child vectors cannot be null, so there is no need to check it.

The performance improvement depends on the complexity of the hash algorithm. For computational intensive hash algorithms, the improvement can be small; while for simple hash algorithms, the improvement can be notable.

I don't see a code change for (1)

Improve the performance of hashCode(int) method for StructVector:

We can get the child vectors directly, so there is no need to get the name from the child vector and then use the name to get the vector.
The child vectors cannot be null, so there is no need to check it.

The performance improvement depends on the complexity of the hash algorithm. For computational intensive hash algorithms, the improvement can be small; while for simple hash algorithms, the improvement can be notable.

I don't see a code change for (1)

Thanks a lolt for your comments.
Please let me explain in more details. This is the old way of implementation:
for (String child : getChildFieldNames()) {
ValueVector v = getChild(child);
...
}
the getChildFieldNames method first gets the child vector by the getChildren method, and then retrieves the vector name from the child vectors. This is the first line of the code
In the second line, we get the vector by calling the getChild method with the vector name.
So essentially, we first get child name by child vectors, and then get the vector from the child name.
In the new code, we directly get child vectors by the getChildren method.

LGTM, otherwise you need rebase this PR.

Rebased. Thank you.

@siddharthteotia Would you please take a look?


text:
ARROW-8428: [C++] GCC 4.8 Implicit move-on-return failure in C++ tests

@kszucs ideally this would be caught by CI. It's not caught by manylinux1 since that job doesn't build the C++ tests

@github-actions crossbow submit gandiva-jar-trusty

Created a follow-up ARROW-8431
[C++][CI] Configure a build to build and execute the C++ tests with GCC 4.8

The gandiva build fails, but with an error related to deployment.

authors: bkietz kszucs


text:
white space

removed white spaces

Thanks for the changes.
I committed a full sweep "remove trailing spaces" yesterday of which this single file is also part.
This can pull request can be closed.

authors: dorisugita nielsbasjes

repo: apache/avro


text:
[BEAM-8928] Update clickhouse-jdbc to version 0.2.4

Update clickhouse-jdbc to version 0.2.4

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @iemejia

Looks like GH action failed on:
Execution failed for task ':buildSrc:pluginUnderTestMetadata'.
> Could not resolve all files for configuration ':buildSrc:runtimeClasspath'.
   > Could not resolve gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.22.0.
     Required by:
         project :buildSrc
      > Could not resolve gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.22.0.
         > Could not get resource 'https://jcenter.bintray.com/gradle/plugin/com/palantir/gradle/docker/gradle-docker/0.22.0/gradle-docker-0.22.0.pom'.
            > Could not GET 'https://jcenter.bintray.com/gradle/plugin/com/palantir/gradle/docker/gradle-docker/0.22.0/gradle-docker-0.22.0.pom'. Received status code 502 from server: Bad Gateway

retest this please

Ah.. I am unable to re-run GH action...

Run Java PreCommit

Run Java PreCommit

pinging @kanterov in case he might have any issue with this update if not answer before monday I will merge.

@iemejia @pawelpasterz thanks! should be fine as soon as tests pass :)

Thanks for confirming @kanterov and thanks for working in all these upgrades @pawelpasterz !

authors: iemejia kanterov pawelpasterz

repo: apache/beam


text:
[BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Straightforward, this adds Dataflow support to the script, as well as a corresponding Gradle command for Dataflow.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @lostluck

R: @lostluck


text:
[BEAM-1116] Support for new Timer API in Flink runner (streaming)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Coverage decreased (-0.003%) to 69.224% when pulling d8628be6a64307f613481ac680ff7e511e79f47c on JingsongLi:BEAM-1116-newTimerApiInFlink into 36ed3f6c54d35ac48241789fd77e2657d54c5831 on apache:master.

Run Flink RunableOnService

I stared a Flink Runner post-commit hook here: https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_RunnableOnService_Flink/1674/

Thank you, it passed the test.

Perfect! Let's wait for an answer by @kennknowles to your question on the Jira Issa regarding how to extract the window from a namespace. Then I would merge this. 👍

Sorry for the delay! I missed the real comment between the bot noise on the JIRA. I answer there, but the basic thing is that namespaces are generalized beyond what user timers need. You only need WindowNamespace.

OK, I got it. User timers all come from StateNamespaces.window, so the namespace must be WindowNamespace.

Coverage decreased (-0.01%) to 69.217% when pulling ddbbb11f68245869e305e4005ee4303797e482cf on JingsongLi:BEAM-1116-newTimerApiInFlink into 36ed3f6c54d35ac48241789fd77e2657d54c5831 on apache:master.

authors: JingsongLi aljoscha coveralls kennknowles


text:
Fix Illegal to use destination for jms producer

Fixes #25849
The issue is that the producer was created with a destination in #24973. This kind of producer cannot set destination again in send()
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @sriram23kmm
CC: @Amraneze
This indicates we need some integration test for JmsIO (entered #25887). Would help if you are interested in adding integration tests.

Hey @Abacn, We got the same issue in our pipeline too. We added some workaround. But, I will create a new PR to add more IT tests.

For the IT tests with other connectors, we don't have access to the library TIBCO. What should we do ?

What is 300 coming from? Is it the number of thread per worker, or total number of bundles? This sounds like connection leaking.

For now, we are testing with numberOfWorkerHarnessThreads to confirm the hypothesis that it's related to number of thread per worker. I will let you know about it.

Also, what is the timeline for the fix? If not before the cut of 2.47 (beginning of April we should revert #25887 at the moment

The problem is that we tested the same code (JmsIO) that we merged in our pipelines with the version 2.45 and we didn't have any issues (for the last 2 weeks). It's just when we upgraded the apache beam to 2.46, so I'm not sure if it's related to JmsIO.

@Amraneze is both pipeline (with/without excessive port occupying) running on same version of Dataflow runner (v1 or v2) ?
Anyway this is another issue than that reported in #25849. @sriram23kmm are you able to try the fix of this PR, that is compile your own jmsio artifact and replace that shipped with Beam v2.46.0 to validate it fixes your issue?

@Abacn It's the same pipeline one was running with the version 2.45 + the new code for JmsIO and one was with the version 2.46. Both of them have the same configuration (same number of workers, ....) and they use Dataflow Runner v1 (we had some drawbacks with v2)
Here is a screenshot of the port usage after running the pipeline (using only beam 2.46) with numberOfWorkerHarnessThreads = 32. It stabilizes at 2k open ports.

A screenshot of the port usage after running the pipeline (using beam 2.46 + the fix of this PR). It stabilizes at 2k open ports.

A screenshot of the port usage after running the pipeline (using beam 2.45 + the fix of this PR + new JmsIO code). It stabilizes at 2k open ports.

Note: The pipelines are running with the same configuration, 18 workers and n2-highmem-4 without autoscaling

I guess, we will need to setup the connection at the setup cycle instead of startBundle cycle

@Abacn I tried this PR fix + 2.46 beam it solves our problem.
Also, when it is going release in the next version(2.47.0)?

@sriram23kmm that's great. Let us get this in first then. The port usage is another issue than the original bug and could be fixed separately. The next version v2.47.0 which is expected be released in late April.

Thanks, @Abacn so then can you merge this PR and get this immediately since it was affecting our production

done. The nightly snapshot is in the directory https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-jms/2.47.0-SNAPSHOT/ and will be updated to include the fix in the next snapshot.

authors: Abacn Amraneze sriram23kmm


text:
Restore trailing slash accidentally removed in https://github.com/apache/beam/pull/25665

This unbreaks the container.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @abacn


text:
Test GBK immediately followed by stateful ParDo

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify.
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Coverage decreased (-0.003%) to 70.716% when pulling 4037ee08cd4a32a7be95c8e939a2fb05417bcf35 on kennknowles:GBK-then-state into e8c5574483edc28d8bea30e55aa2d54b1d566722 on apache:master.

Coverage decreased (-0.01%) to 70.709% when pulling acc35e25371ec08c38be4839b5f90d170af1258b on kennknowles:GBK-then-state into e8c5574483edc28d8bea30e55aa2d54b1d566722 on apache:master.

Coverage remained the same at 70.72% when pulling 5d9fe885b2581a98aeb1d470229b733eda52d1cd on kennknowles:GBK-then-state into e8c5574483edc28d8bea30e55aa2d54b1d566722 on apache:master.

authors: coveralls kennknowles


text:
Log when Handler Delegation Throws

In addition to returning an instruction response with the error field
set, log the error that caused it.
Some formatting also got caught here

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand:

[ ] What the pull request does
[ ] Why it does it
[ ] How it does it
[ ] Why this approach

[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Run ./gradlew build to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

run java precommit

authors: lukecwik tgroh


text:
[BEAM-4602][BEAM-4598] Datetime type comparison

Support datetime type in comparison expression
Test datetime type comparison at the DSL level
R: @kennknowles @XuMingmin
cc: @akedin @apilloud @amaliujia

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Spark

Go

---
---
---
---
---

Java

Python

---

---
---
---

run java precommit

It's ready to be merged.
On Mon, Jun 25, 2018, 11:53 XuMingmin notifications@github.com wrote:

@XuMingmin approved this pull request.
LGTM
@vectorijk https://github.com/vectorijk can you fix the conflict then I
can merge it
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/5749#pullrequestreview-131738856,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADQu6XkuOUmLSM3FEyuSGQhq-E9oVP8zks5uATGUgaJpZM4U1VeX
.

There are three commits but I think it should probably just be one.

rebased it.

authors: kennknowles vectorijk


text:
[BEAM-5759] Ensuring JmsIO checkpoint state is accessed and modified safely

As described in BEAM-5759, a ConcurrentModificationException can be thrown when JmsIO source checkpoint finalisation occurs simultaneously to further messages being added for finalisation. This is due to the state of JmsCheckpointMark not being thread safe.
This change adds a unit test which demonstrates the issue by

Adding several messages to a queue.
Reading some of these messages with an UnboundedJmsReader (which adds messages for finalization).
Calling finalizeCheckpoint on the reader's checkpoint mark in a separate thread.
Reading the remainder of the messages in parallel with the above finalization.

In order to reliably reproduce this issue, the test uses a JMS consumer which decorates received messages with a callback which sleeps for a short period, introducing some "lag" into the finalization process.
To fix the issue, JmsCheckpointMark has been updated with thread safety constructs. In order to avoid potentially long blocks for consumers during checkpoint finalization, a read/write lock is used and finalization takes a snapshot of state before processing in a lock-free manner, only locking to update state post-finalization.
@jbonofre
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

retest this please

retest this please

authors: andrew-flumaion jbonofre


text:
WIP: Brings back the helm code

Revert "Revert "This closes #830""
This reverts commit 4236ff272903a8ffe77a06d9b146aea40b3f057e.
Added a few changes, played around with the version to avoid conflicts and added OSGI setup.

Awesome stuff!
Hellish dependencies.  But if you got it working, let's merge once comments are addressed.
Mainly it's comments to help us in future though there is one bit about cross-platform builds that needs to be checked.
In future it might be nicer to jarjar the odd helm deps in a separate project rather than pull them all in here via OSGi.  But not worth the effort if you have it working.
One other question, do you have an Apache contributor license agreement @iuliana ?

Putting a break on this because of the hellish dependencies.

authors: ahgittin iuliana

repo: apache/brooklyn-server


text:
服务启动后没流量，但是bthread_count显示存在大量bthread

如题，这些bthread是brpc内部起的后台任务吗？哪些地方会起这么多bthread？谢谢

可以用 #903 排查一下。

authors: chenBright hsueh-yh

repo: apache/brpc


text:
Integration tests dont work by default, and write outside of repo

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Feb 23, 2018, 12:43
When running the following command, integration tests dont work:
./setup.py test --addopts '--integration tests/integration/shell.py'

This results in tests trying to write to the tmpfs in /tmp on my host, where ostree fails to set extended attributes.
Which leads to the second part of this, that the integration tests leave behind debris on the host by default; that should not happen.
I later discovered the env var INTEGRATION_CACHE, which fixes things when setting it to the tmp/ subdir of BuildStream repo.
Instead of defaulting to a place which likely wont work and leaving behind debris, it's better to default to somewhere local to the repository. If it's desirable to set INTEGRATION_CACHE to somewhere else, for performance reasons; this should be documented in the HACKING.rst file.

In GitLab by [Gitlab user @tristanvb] on Feb 24, 2018, 11:29

I later discovered the env var INTEGRATION_CACHE, which fixes things when setting it to the tmp/ subdir of BuildStream repo.

NOTE: The tmp/ subdir of the buildstream repo is recreated every time you run the test suite, so it's probably not a suitable location.
I also noticed that the INTEGRATION_CACHE is not an env var which represents where to put the caches used for integration tests.
Instead, INTEGRATION_CACHE is a directory under which the tests will create an integration-cache subdirectory... weird.

In GitLab by [Gitlab user @samthursfield] on Feb 26, 2018, 12:48
mentioned in commit 6e01005fc1fb65154479c5ad25da06c881dea529

In GitLab by [Gitlab user @samthursfield] on Feb 26, 2018, 12:48
mentioned in commit 2a294d0dfa57ab83b3f93f50b090a2b20fc29fe7

In GitLab by [Gitlab user @samthursfield] on Feb 26, 2018, 12:52
mentioned in merge request !294

In GitLab by [Gitlab user @samthursfield] on Feb 26, 2018, 12:57
mentioned in commit 28deda0a4a8d72de3eaa71bf325d3a1f5af5a4d5

In GitLab by [Gitlab user @samthursfield] on Feb 26, 2018, 12:59
mentioned in commit 1dfb1897a6dc9bc5f10ba900f565abca06c7a4e7

In GitLab by [Gitlab user @samthursfield] on Feb 28, 2018, 09:17
mentioned in commit 5ebb78240e674afb4ae86ecf41f9ca03192a6dba

In GitLab by [Gitlab user @samthursfield] on Feb 28, 2018, 09:17
mentioned in commit ad0369fd1b1255407418d5789e83503f52fcd49b

In GitLab by [Gitlab user @tristanvb] on Feb 28, 2018, 09:34
closed via merge request !294

repo: apache/buildstream