text:
BIGTOP-4126. Remove obsolete test resources for Hue.

https://issues.apache.org/jira/browse/BIGTOP-4126
Obsolete test resources for Hue causes completion error. This causes issue on mvn deploy in the release process.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/iwasakims/srcs/bigtop/bigtop-tests/test-artifacts/hue/src/main/groovy/org/apache/bigtop/itest/huesmoke/TestHueSmoke.groovy:[40,23] 1. ERROR in /home/iwasakims/srcs/bigtop/bigtop-tests/test-artifacts/hue/src/main/groovy/org/apache/bigtop/itest/huesmoke/TestHueSmoke.groovy (at line 40)
        Shell sh = new Shell();
                              ^
Groovy:expecting ']', found ';' @ line 40, column 25.

[ERROR] /home/iwasakims/srcs/bigtop/bigtop-tests/test-artifacts/hue/src/main/groovy/org/apache/bigtop/itest/huesmoke/TestHueSmoke.groovy:[48,1] 2. ERROR in /home/iwasakims/srcs/bigtop/bigtop-tests/test-artifacts/hue/src/main/groovy/org/apache/bigtop/itest/huesmoke/TestHueSmoke.groovy (at line 48)
        sh.exec("curl -m 60 --data '${creds}' ${loginURL}");
        ^
Groovy:unexpected token: sh @ line 48, column 5.

...

+1, I made sure that the mvn commands before deploying artifacts described the release process are successful with this PR. Thanks @iwasakims.

cherry-picked this to master branch.

authors: iwasakims sekikn

repo: apache/bigtop


text:
Move source cache to proto based service

See original issue on GitLab
In GitLab by [Gitlab user @raoul].hidalgocharman on May 29, 2019, 13:48
Background
The source cache should move towards using a protocol buffer based format similar to new artifact service architecture (overall plan described in #909). The reference service should be kept for now to allow it to be used with older buildstream clients.
Task description

[x] Design new source protos. This should be similar to artifact protos and may include metadata such as the sources provenance data.
[x] Implement new SourceCacheService that uses this.
[x] Use new SourceCacheService in SourceCache for pulling and pushing sources.

Acceptance Criteria
All old source cache tests pass using new source protos.

In GitLab by [Gitlab user @raoul].hidalgocharman on May 29, 2019, 13:48
mentioned in merge request !1362

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 19, 2019, 12:11
mentioned in merge request !1410

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 21, 2019, 14:13
assigned to [Gitlab user @raoul].hidalgocharman

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 09:30
mentioned in commit 8bbea0cc9d1c07dbcf7bba563082c31694d92220

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 09:30
mentioned in commit 35214d0d87c759788a1be22d5e9b77ddecf9806e

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 09:30
mentioned in commit 8e264f093816a63f77e52d58332e8b32d713eb92

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 09:30
mentioned in commit 55bf188cb681ca0683dda3c82202ee21369cb47d

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 12:16
marked the task Design new source protos. This should be similar to artifact protos and may include metadata such as the sources provenance data. as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 12:16
marked the task Implement new SourceCacheService that uses this. as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:08
mentioned in commit c9f8581531a0d583ef3cb21519aefb9e1ba66bd4

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:08
mentioned in commit ef712320ebd39f2259b90ce7cd3f7d9ffdb28a5c

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:08
mentioned in commit 544c02367784e2e401760bf171d8b61ae8d3959a

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:15
mentioned in commit 8c9df8706969b5346b404c1581299fbcf4e3676e

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:15
mentioned in commit 4533989e94d68b80ea1dfd7a59dd7177417f91e7

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:15
mentioned in commit 0d6a1e9fa891e27883689bd65a8f7b46e991162c

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:15
mentioned in commit 83352742821de343b499b597c751667fc4d6a763

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 14:15
mentioned in commit 12874bd9a67492a6d58d5aadd9ed8d5b737a1e0a

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 26, 2019, 15:50
mentioned in commit 4dc530b338a4e434bd315d5f2d4102b563f49c77

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:12
mentioned in commit 4032c81019e8dd69ca541e87f99f484ffde3db52

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:12
mentioned in commit 357ab273a01711a592b0ca8de99b501e2c1cb62e

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:12
mentioned in commit 78d567dddea3c28f4e737db00a092ae8a82c405e

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:12
mentioned in commit 1581ff9cd6f6d8a2ce32d61446de300d85bae4e8

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:12
mentioned in commit a8e04529d62f5c7c9894c45b83330823d4eb6bba

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:42
mentioned in commit 6c540804827d18deb562afb3da19f4039a25c81a

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:42
mentioned in commit 0397b66e9bda61035253fa718eff59538a7d211d

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:48
mentioned in merge request !1435

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:55
mentioned in commit c76881a12a4dc10194ea3999f6d7d76181c34244

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 27, 2019, 15:55
marked the task Use new SourceCacheService in SourceCache for pulling and pushing sources. as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:30
mentioned in commit 5ce4cc7499d8eab81f0fa5f9f3e93249d443605f

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:30
mentioned in commit 6b42c2da0cc248ef0d0819c892feba6d10ca19f1

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:31
mentioned in commit abea6490cb6036b1fa9c898879e5e20956007393

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:31
mentioned in commit caba7d3a59ed24f1edf86366b8a7cf3675b9c1fc

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:31
mentioned in commit b5e84029b07e95b281dfcc0da73f274e1b55b1c3

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:31
mentioned in commit 47ace4076c34f43b9c92ebc01ef3f354b501f5d2

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:45
mentioned in commit 4e2c3b28d89ded00f62e226ee5a3db8a96530c73

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:45
mentioned in commit f9ffbfe9af2bb8563af54b4b7ccc98f5292da030

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:45
mentioned in commit ac7a02fbef5fea5f8ace2357111e3e99794fb515

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:45
mentioned in commit b80260e694c9ac88bd45b79319e10e8c82f7c84f

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 28, 2019, 15:45
mentioned in commit 3f3a29fed5e3dac3634208bf7b724314cfb2896d

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 16:59
mentioned in commit 9497b02f9eae40e9dca97c1b325cc6ca6a52fe81

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 16:59
mentioned in commit 2a58f66d09600bdc79d6d25d771274bcc1544434

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 16:59
mentioned in commit 6b13a80e5f4e9792768b04e45266b74944daa643

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit eee1512447206b28d62b7bce38e86d96e6c19a3c

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit e41517f305c23d5efce24ae4a91326fc81d93a32

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 82de872b3ec4eba90e8c022476bf760f350b81b2

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 5fc9ba356b967100260bbfc2aedfa19a193fbf86

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 2259e0453aedfa35f1a7539c19a629302fa9fd82

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 3cd1e50a38595da169e10f9c7f6b5912cb745264

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 14be3c3070f06f3d8194daa8474c035b5776fbc3

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 3063d1df3dba3ceab7ffa9fc7227cf4102d13c3d

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit 97858e21acf8497543e94f5338b035b12ff1ca1c

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 1, 2019, 17:39
mentioned in commit ec012e5cd21ce3e0840ad54cd0becd2cb21eb889

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 2, 2019, 16:23
mentioned in commit 0d69662b5d4711098d0bbf6f15b5c0f7da8098a7

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 2, 2019, 16:23
mentioned in commit 5dc76fdb7833fa71d89c2f56a4be098cecf7495b

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 2, 2019, 16:23
mentioned in commit a4e8907c60bffcb51c4057327a9300938df87956

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 2, 2019, 16:23
mentioned in commit 5158c9fce3dae4f88f20a1e6bf48a0d426ac2036

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 2, 2019, 16:23
mentioned in commit c12587ce8e1ef32f1b66ff0399d5b76650f57c90

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 13:06
mentioned in commit ab06c00ded23866ec88ce9132fe3000c9bfe823b

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 13:06
mentioned in commit d06bacd228313b943224abb92435915b3a177d23

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 13:06
mentioned in commit 46418bf79a740fb6c906962e52c243243d2849bb

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 13:06
mentioned in commit 8d7afd7514f701870ed8333b822a2bb2682ef0de

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 13:06
mentioned in commit 1fb4716df7caae754501717ad519f5f1e5268a7c

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 13:47
mentioned in commit 2d48dc85f2814d6d2b04e48917c09a415cdbd540

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 5, 2019, 15:44
mentioned in commit b15b32376f6abe1735233bd5d85c42fd1ad5a703

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 11:40
mentioned in commit bb2cf18be0aef7d6e394a0c6ff6d83eac737c60b

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 11:40
mentioned in commit 6b6e04ddb1c03f683e3e5591f057207d1303e6b6

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 11:40
mentioned in commit d493682609f8f96ed127f4083bad42fa2fabb250

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 11:40
mentioned in commit c20eac1e7ac80e1dc36b23b04affacfbe2cca338

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 11:40
mentioned in commit d61e21448953942ce90d457dad7189c0dda61bc7

In GitLab by [Gitlab user @marge-bot123] on Jul 8, 2019, 12:17
closed via merge request !1435

In GitLab by [Gitlab user @marge-bot123] on Jul 8, 2019, 12:17
mentioned in commit cf0516ba92fb4a220ae0086c411314cec4974df5

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 16:41
mentioned in commit d1be05c771bdbe054f01eaaea977d4b20e401354

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 16:41
mentioned in commit dc689098d164510eb22820776f9c8cf1cf1fd642

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 16:41
mentioned in commit 38f7bffd87ffd901e316a7966fefc9d189658e19

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 16:41
mentioned in commit c02c0170058f36eff722bb341edbed3feae18145

In GitLab by [Gitlab user @raoul].hidalgocharman on Jul 8, 2019, 16:41
mentioned in commit 107bae99f159d22cbedd155c5db9255782fad3c0

source: gharchive/issue

repo: apache/buildstream

license_type: permissive

license_source: github-api


text:
Add API for composing lists in yaml nodes

See original issue on GitLab
In GitLab by [Gitlab user @tlater] on Jul 3, 2019, 11:14
Background
As part of #1061 we need to perform composition between lists like so:
# bar.bst

kind: autotools
dependencies:
  (>):
    - foo-lib.bst

# project.conf

elements:
  autotools:
    dependencies:
      - autotools.bst

These lists are expected to be composed into this:
  - autotools.bst
  - foo-lib.bst

This means we need to do something along the lines of:
default_deps = _yaml.node_get(project_conf, list, "dependencies")
deps = _yaml.node_get(element, list, "dependencies")
_yaml.composite(default_deps, deps)

But this will not work, because _yaml.composite() will only deal with _yaml.Nodes.
Task description
We should add some form of an API to allow doing this - I can see either of these things working:

_yaml.composite() learns to deal with plain lists - the problem here is that we'd struggle providing provenance data, and the behavior of composite(list, list) isn't obvious (although !1601 will probably make that "safe append", at least for dependencies).
_yaml.get_node() returns proper _yaml.Nodes when type=_yaml.Node for lists - this feels a bit more reasonable, but I'm likely overlooking something :)

Acceptance Criteria
We should be able to compose lists without creating naughty synthetic nodes.

In GitLab by [Gitlab user @tlater] on Jul 3, 2019, 11:15
changed the description

In GitLab by [Gitlab user @tlater] on Jul 3, 2019, 11:15
changed the description

In GitLab by [Gitlab user @tlater] on Jul 3, 2019, 11:16
changed the description

In GitLab by [Gitlab user @tlater] on Jul 3, 2019, 11:18
changed the description

In GitLab by [Gitlab user @tlater] on Jul 3, 2019, 12:02
changed the description


text:
Checkout needs sandbox even when run with --no--intergration

See original issue on GitLab
In GitLab by [Gitlab user @willsalmon] on Jul 7, 2020, 11:27
Summary
Checkout needs sandbox even when run with --no--intergration
This means that if a project was built in CI on a different arch or with RE and the RE bots are busy, artefacts can not be checked out even if they are just a tar/docker-image/single file.
We could do it with a approach like https://gitlab.com/BuildStream/buildstream/-/merge_requests/1983 or we tweak how the sandbox is invoked so that it dose not need a real sandbox.

In GitLab by [Gitlab user @cs-shadow] on Jul 14, 2020, 22:02
I'm not sure if I follow. This is already possible, and precisely the reason we have the dummy sandbox (SandboxDummy). This should be automatically get created when buildbox-run isn't available for whatever reason.
If this is not happening, this would be a bug in BuildStream. If so, please share more details about it, and how to reproduce it.

In GitLab by [Gitlab user @cs-shadow] on Jul 14, 2020, 22:02
I'm not sure if I follow. This is already possible, and precisely the reason we have the dummy sandbox (SandboxDummy). This should be automatically get created when buildbox-run isn't available for whatever reason.
If this is not happening, this would be a bug in BuildStream. If so, please share more details about it, and how to reproduce it.

In GitLab by [Gitlab user @willsalmon] on Jul 15, 2020, 10:51
For this case buildbox-run is available but dose not support the target arch.
The element was made by using remote exicution or CI with a different arch. by setting the sandbox arch you can get the right cache key and pull down the artefact. But when you try to check it out then the sandbox dose not create a dummy but complains that the arch is not supported for a full sandbox.
This makes seance when --no-intergration is not used but dose not make sense if --no-intergration is specified.

In GitLab by [Gitlab user @willsalmon] on Jul 15, 2020, 10:51
For this case buildbox-run is available but dose not support the target arch.
The element was made by using remote exicution or CI with a different arch. by setting the sandbox arch you can get the right cache key and pull down the artefact. But when you try to check it out then the sandbox dose not create a dummy but complains that the arch is not supported for a full sandbox.
This makes seance when --no-intergration is not used but dose not make sense if --no-intergration is specified.

In GitLab by [Gitlab user @cs-shadow] on Jul 15, 2020, 11:03
Thanks. I think the fix in that case should be to ensure that we do use the dummy sandbox in such code paths, rather than circumventing that in places other than the sandbox module. This keeps all related logic in one place and avoids unncessary forks in the code.

In GitLab by [Gitlab user @cs-shadow] on Jul 15, 2020, 11:03
Thanks. I think the fix in that case should be to ensure that we do use the dummy sandbox in such code paths, rather than circumventing that in places other than the sandbox module. This keeps all related logic in one place and avoids unncessary forks in the code.

In GitLab by [Gitlab user @cs-shadow] on Jul 15, 2020, 11:04
mentioned in merge request !1983

In GitLab by [Gitlab user @cs-shadow] on Jul 15, 2020, 11:04
mentioned in merge request !1983

In GitLab by [Gitlab user @willsalmon] on Jul 15, 2020, 13:49
The issue that i had was that AFAICT we pick which sandbox to use for the run really early, at the platform level so you lose the chance to fall back at the point were we actually invoke it. Im not sure if that's true but that's what it looked like when I looked. Duno if [Gitlab user @cs-shadow] or [Gitlab user @juergbi] can point me in the right direction for how to fix this sensibly.

In GitLab by [Gitlab user @willsalmon] on Jul 15, 2020, 13:49
The issue that i had was that AFAICT we pick which sandbox to use for the run really early, at the platform level so you lose the chance to fall back at the point were we actually invoke it. Im not sure if that's true but that's what it looked like when I looked. Duno if [Gitlab user @cs-shadow] or [Gitlab user @juergbi] can point me in the right direction for how to fix this sensibly.


text:
removed deprecated version of copy to clipboard method to clipboard.w…

…riteText

Thanks for the copyToClipboard refactoring.
Could you please remove changes in render() method from PR. They make code cleaner but they are not related to the PR (removed deprecated version of copy to clipboard method). It is better to make them in the separate one.

source: gharchive/pull-request

authors: httpsOmkar mgubaidullin

repo: apache/camel-karavan


text:
Camel Avro RPC component native support

This one is for camel-avro-rpc component. We already have the avro dataformat in native https://github.com/apache/camel-quarkus/issues/782

Please, assign to me.

authors: JiriOndrusek ppalaga

repo: apache/camel-quarkus


text:
(chores) camel-optaplanner: reduce the time spent trying to solve the problems

Signed-off-by: Otavio R. Piske angusyoung@gmail.com

Before: Total time:  14:04 min
After: Total time:  01:14 min

repo: apache/camel


text:
[CARBONDATA-1259] CompareTest improvement

changes:

check query result details, report error if result is not the same
add support for comparison with ORC file
add decimal data type

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/294/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2878/

retest this please

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/314/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2900/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2902/

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/316/

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/317/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2903/

LGTM

authors: CarbonDataQA chenliang613 jackylk

repo: apache/carbondata


text:
[CARBONDATA-1903] Fix code issues in carbondata

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[x] Any interfaces changed?
No

[x] Any backward compatibility impacted?
No

[x] Document update required?
No

[x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
No, only fixed code related issues.
- How it is tested? Please attach test report.
Tested in local machine
- Is it a performance related change? Please attach the performance test report.
No, only fixed code related issues.
- Any additional information to help reviewers in testing this change.
No

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
Not related

Modification

Remove unused code like FileUtil

Fix/Optimize code issues in carbondata

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2126/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/902/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2412/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2437/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2153/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/924/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2162/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/933/

retest this please

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2241/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1017/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2499/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2242/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1018/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2246/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1023/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2505/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2251/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1028/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2507/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2514/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2262/

retest this please

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1040/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2285/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1069/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2291/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1075/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1078/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2307/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1091/

retest this please

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1110/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2330/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2352/

retest this please

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1135/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2355/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1137/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1139/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2568/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1143/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2373/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2404/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1181/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2435/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1211/

authors: CarbonDataQA ravipesala xuchuanyin


text:
[CARBONDATA-2440] doc updated to set the property for SDK user

[x] Any interfaces changed? NO

[x] Any backward compatibility impacted? No

[x] Document update required? ==> Yes

[x] Testing done ==>  All UT and SDV succes report is enough.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.  NA

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4745/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5667/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4507/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4746/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5671/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4511/

LGTM

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4686/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5842/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4887/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5883/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4918/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4731/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4919/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5885/

@xubo245  review comments resolved .

please import CarbonProperties before using it.

@xubo245 import done

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5016/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4844/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6003/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6014/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4856/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6022/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5029/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4863/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5036/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4874/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6033/

retest this please

LGTM

authors: CarbonDataQA rahulforallp ravipesala sgururajshetty xubo245


text:
CLIMATE-379 - Allows dataset customisation

This patch allows the user to customise the name of the local dataset that is being uploaded in the web-app.

Can one of the admins verify this patch?

hey @lewismc @MJJoyce, please have a look at this patch.

I am +1, any comments @MJJoyce ?

:+1:

Please commit @Omkar20895

I haven't been granted the write access to the repository yet. right??

No you have, the canonical source is here
https://git-wip-us.apache.org/repos/asf/climate.git
The Github code is merely a mirror and is not the canonical source as it is
not hosted at the Apache Software Foundation.
The link above is hosted at the ASF and therefore canonical.
Thanks
Lewis
On Wed, Jan 27, 2016 at 9:38 AM, Omkar notifications@github.com wrote:

I haven't been granted the write access to the repository yet. right??
—
Reply to this email directly or view it on GitHub
https://github.com/apache/climate/pull/276#issuecomment-175762787.

--
Lewis

authors: MJJoyce OCWJenkins Omkar20895 lewismc

repo: apache/climate

license_source: bigquery


text:
packaging: enforce new min. CloudStack version 4.15 starting GA/1.0

There are many changes, including API changes in upstream master/4.15
which makes it challenging to maintain backward compability of Primate
with older versions of CloudStack. Therefore we need to ensure that the
rpm and deb Primate pkgs require CloudStack 4.15 as minimum version.
This would still leave some flexibility for advanced users of archive
builds (which adds risks that some features don't work with 4.14 or
older versions).

Following this we need to update https://github.com/apache/cloudstack-documentation/pull/150 as well wrt the min. version Primate will support and installation instructions. By default, we'll ship primate with every cloudstack repo so users won't need to setup the repo themselves (the other way is for cloudstack-management to install the repo config automatically).

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/841 (JID-3629)

authors: blueorangutan rhtyd

repo: apache/cloudstack-primate


text:
Misuses of cryptographic APIs

Hi
The following lines have cryptographic API misuses.  File name => utils/src/main/java/com/cloud/utils/ssh/SSHKeysHelper.java: Line number => 75: API name => MessageDigest: File name => utils/src/main/java/com/cloud/utils/nio/Link.java: Line number => 371: API name => KeyStore:Second parameter should never be of type java.lang.String. File name => utils/src/main/java/org/apache/cloudstack/utils/security/DigestHelper.java: Line number => 30: API name => MessageDigest:Unexpected call to method <java.security.MessageDigest: byte[] digest()> on object of type java.security.MessageDigest. Expect a call to one of the following methods <java.security.MessageDigest: void update(byte[])>,<java.security.MessageDigest: void update(byte[],int,int)>,<java.security.MessageDigest: byte[] digest(byte[])>,<java.security.MessageDigest: void update(java.nio.ByteBuffer)>,<java.security.MessageDigest: void update(byte)> File name => utils/src/main/java/org/apache/cloudstack/utils/security/DigestHelper.java: Line number => 37: API name => MessageDigest: File name => utils/src/main/java/org/apache/cloudstack/utils/security/DigestHelper.java: Line number => 52: API name => MessageDigest:Unexpected call to method reset on object of type java.security.MessageDigest. Expect a call to one of the following methods digest,update File name => utils/src/main/java/com/cloud/utils/crypt/RSAHelper.java: Line number => 81: API name => Cipher: File name => utils/src/main/java/com/cloud/utils/ssh/SSHKeysHelper.java: Line number => 67: API name => MessageDigest:First parameter (with value "MD5") should be any of {SHA-256, SHA-384, SHA-512} File name => utils/src/main/java/com/cloud/utils/EncryptionUtil.java: Line number => 63: API name => SecretKeySpec: File name => utils/src/main/java/com/cloud/utils/SwiftUtil.java: Line number => 234: API name => SecretKeySpec: File name => utils/src/main/java/com/cloud/utils/SwiftUtil.java: Line number => 234: API name => SecretKeySpec: File name => utils/src/main/java/com/cloud/utils/ssh/SSHKeysHelper.java: Line number => 75: API name => MessageDigest: File name => utils/src/main/java/com/cloud/utils/security/CertificateHelper.java: Line number => 72: API name => KeyStore:Unexpected call to method store on object of type java.security.KeyStore. Expect a call to one of the following methods getKey,getEntry File name => utils/src/main/java/com/cloud/utils/security/CertificateHelper.java: Line number => 117: API name => KeyStore:Unexpected call to method store on object of type java.security.KeyStore. Expect a call to one of the following methods getKey,getEntry File name => utils/src/main/java/com/cloud/utils/EncryptionUtil.java: Line number => 63: API name => SecretKeySpec: File name => utils/src/main/java/com/cloud/utils/crypt/RSAHelper.java: Line number => 79: API name => Cipher:First parameter (with value "RSA/None/PKCS1Padding") should be any of RSA/{Empty String, ECB} File name => utils/src/main/java/com/cloud/utils/security/CertificateHelper.java: Line number => 99: API name => KeyStore:Second parameter should never be of type java.lang.String. File name => utils/src/main/java/com/cloud/utils/crypt/RSAHelper.java: Line number => 81: API name => Cipher:

@mhp0rtal can you give expoits for any of those isses?
Can you also please give a version on which these apply, as the first three do not show code matching the message;
1: File name => utils/src/main/java/com/cloud/utils/ssh/SSHKeysHelper.java: Line number => 75: API name => MessageDigest:
line 71 is an empty line
2: File name => utils/src/main/java/com/cloud/utils/nio/Link.java: Line number => 371: API name => KeyStore:Second parameter should never be of type java.lang.String.
call on line 371 has only one parameter
3: File name => utils/src/main/java/org/apache/cloudstack/utils/security/DigestHelper.java: Line number => 30: API name => MessageDigest:Unexpected call to method <java.security.MessageDigest: byte[] digest()> on object of type java.security.MessageDigest. Expect a call to one of the following methods <java.security.MessageDigest: void update(byte[])>,<java.security.MessageDigest: void update(byte[],int,int)>,<java.security.MessageDigest: byte[] digest(byte[])>,<java.security.MessageDigest: void update(java.nio.ByteBuffer)>,<java.security.MessageDigest: void update(byte)>
line 30 is empty
I stopped checking there but I propose you debug your tool of investigation.
I'm closing this issue but if you feel it is still valid, please add needed extra info and reopen.

repo: apache/cloudstack


text:
Place VM's on the same network as the CloudStack-management server

ISSUE TYPE

Other

COMPONENT NAME

UI, IP
CLOUDSTACK VERSION

4.16.0.0
SUMMARY

I am trying to find a method of placing newly deployed VM's on the same network as my management server, but so far it only allows me to create guest VM's behind a NAT with a private IP address. I need to create multiple VM's hosting services that can be routed to an external DNS. My concern is that the NAT will make routing to these machines impossible.
I have tried changing the network offering on my guest network from "offering for isolated network with NAT service enabled" to "offering for isolated network with no NAT service enabled", but it gives me this error:
"can't upgrade from network offering edad787d-baa0-4ca8-a67b-bd288adc6d37 to 052cee24-272b-44f1-a4df-4b5da7a30744; check logs for more information"
I would choose this during network deployment, but I do not have the option to choose "offering for isolated network with no NAT service enabled" until after deployment.
I would appreciate any advice you can give to help guide me from here as I am new to cloudstack
EXPECTED RESULTS

Place all deployed VM's on same IP range as the management console itself.
ACTUAL RESULTS

Forced to deploy VM's behind NAT per default configuration

@CKrieger2020 this is certainly a new use case. After a quick read I think you might want to investigate going to IPv6.
Another possible way to go is to deploy in a shared network.

@CKrieger2020 - have you tried to create a new Network or create a L2 network (assuming your CS instance is on a separate L2 network than what is available in CS for users to choose from)
In my home lab I have been able to create instance using L2 which talks to my other computers on the home network.

It's possible to do this by deploying VMs on a L2 network whose vlan is vlan://untagged essentially you'll be on the same network as your host/mgmt network.

@CKrieger2020 can you check the above suggestion and re-open the issue to discuss more.
To discuss further questions, you can raise them on the users@ and/or dev@ ML https://cloudstack.apache.org/mailing-lists.html

authors: CKrieger2020 DaanHoogland nxsbi rohityadavcloud


text:
Issue while setting up CloudStack Advance Zone with security group

ISSUE TYPE

Bug Report

COMPONENT NAME
Advanced Zone with Security Groups setup

CLOUDSTACK VERSION
4.17.2

CONFIGURATION
Zone:
    IPV4 DNS: 8.8.8.8
    Internal DNS: 10.4.1.1
Pysical Network 1:
    Management Traffic: cloudbr0
    Guest Traffic: cloudbr1
Pod:
    Gateway: 10.4.1.1
    Netmask: 255.255.0.0
    IP Range: 10.4.2.1 to 10.4.2.255
Guest Traffic:
    Gateway: 10.6.1.1
    Netmask: 255.255.0.0
    IP Range 10.6.2.1 to 10.6.2.255
Host:
    IP: 10.4.1.20
    User: root
    Password: password
    Tag: h1

OS / ENVIRONMENT
Ubuntu 22.04 server with two network bridge cloudbr0 and cloudbr1

SUMMARY
Apache CloudStack v4.17.2

I am trying to setup CloudStack Advance Zone with security groups.

I have two network bridges cloudbr0 (10.4.1.1/16) and cloudbr1 (10.6.1.1/16). I am using cloudbr0 for Management Network and cloudbr1 for the Guest Network.

However, the zone creation keeps failing adding host with the error message - failed to add host as resource already exists as LibvirtComputingResource.

For some reason it seems like CloudStack is trying to add the same host twice.

STEPS TO REPRODUCE
Configuring CloudStack Advance Zene with security group on Ubuntu 22.04 server

EXPECTED RESULTS
Successfully create advance zone with security group.

ACTUAL RESULTS
Host setup fails with the following error:

Could not add host at [http://10.4.1.20] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Skipping host 10.4.1.20 because 2f02300b-d9bf-3229-acb8-21054c500f47 is already in the database for resource 2f02300b-d9bf-3229-acb8-21054c500f47-LibvirtComputingResource with ID 86f5dcd2-9d6e-444e-b0df-e0dcb1509699 - Skipping host 10.4.1.20 because 2f02300b-d9bf-3229-acb8-21054c500f47 is already in the database for resource 2f02300b-d9bf-3229-acb8-21054c500f47-LibvirtComputingResource with ID 86f5dcd2-9d6e-444e-b0df-e0dcb1509699].

Was there an issue faced during zone creation after the host addition step, maybe during setting up the stores? I had faced a similar issue in the past, where in if the zone creation fails at any point and we are prompted to rectify the issue, and then restart the zone creation workflow, it attempts to re-add the host. Can you check the database if an entry already exists in the host table and if it does, delete them and restart the zone creation process.

@Atiqul-Islam
Can you upload the full management server log ?

@Pearl1594 I am installing Cloud Stack on a fresh Ubuntu Sever, there was no host created before the zone creation.
@weizhouapache
Management Server Log

@Atiqul-Islam
it looks you use a server as both management server and cloudstack agent.
from the log, host was added twice and of course it failed at 2nd attempt. everything else looks good.

@weizhouapache
Why was the host added twice is it because I am using the same server as both management and agent?
I didn't do manually anything to create a host, I just started cloudstack and tried setting up the advanced zone with security group. Thats where I configured the host. During the process of creating the zone it seemed like cloudstack was trying to add the same zone twice.

@weizhouapache
Why was the host added twice is it because I am using the same server as both management and agent?
I didn't do manually anything to create a host, I just started cloudstack and tried setting up the advanced zone with security group. Thats where I configured the host. During the process of creating the zone it seemed like cloudstack was trying to add the same zone twice.

@Atiqul-Islam
I just wanted to confirm your configurations.
I will try to reproduce the issue.

@weizhouapache
Really appreciate the help.
We are testing out CloudStack as it is part of our stack for our next generation of software and systems. So far been stuck in that roadblock for a while. Any help is greatly appreciated.

@weizhouapache
Really appreciate the help.
We are testing out CloudStack as it is part of our stack for our next generation of software and systems. So far been stuck in that roadblock for a while. Any help is greatly appreciated.

@Atiqul-Islam  no problem.
it seems like a minor issue for you I think.
The zone has been created successfully, and system vms are Running when you enabed the zone, right ?

@weizhouapache
Really appreciate the help.
We are testing out CloudStack as it is part of our stack for our next generation of software and systems. So far been stuck in that roadblock for a while. Any help is greatly appreciated.

@Atiqul-Islam no problem.
it seems like a minor issue for you I think. The zone has been created successfully, and system vms are Running when you enabed the zone, right ?

@weizhouapache
It seems like there was possibly network issues during the setup process, some component of the Zone could be in a bad state, as there was no Virtual Router created for the guest network.
I am also getting the following error when I am trying to add an Ubuntu 20.04 iso.
Unable to resolve releases.ubuntu.com

@weizhouapache
Really appreciate the help.
We are testing out CloudStack as it is part of our stack for our next generation of software and systems. So far been stuck in that roadblock for a while. Any help is greatly appreciated.

@Atiqul-Islam no problem.
it seems like a minor issue for you I think. The zone has been created successfully, and system vms are Running when you enabed the zone, right ?

@weizhouapache
Systems VMs are up and running after I enabled the Zone. However, it seems like the zone network might not be properly configured. Some component of the Zone could be in a bad state, as there was no Virtual Router created for the guest network.
I am also getting the following error when I am trying to add an Ubuntu 20.04 iso.
Unable to resolve releases.ubuntu.com

I did check the bare metal system running the management server and the host can ping releases.ubuntu.com

@Atiqul-Islam I have checked your log. It seems everything went smoothly, except the extra step to add host again when all are done. I think you can ignore the error.
for the issue with DNS , you need to log into Secondary storage Vm (a.k.a SSVM) and check if the domain can be resolved. you might need to update the DNS and internal DNS in zone configuration

@weizhouapache
I am unable to get into SSVM console. When I try to get into the console using the GUI, it seems to cannot load the page. In addition, where do I find the login credentials to the SSVM.
Also shouldn't there be a virtual router created as well for the gateway of the guest network?

@weizhouapache
I am unable to get into SSVM console. When I try to get into the console using the GUI, it seems to cannot load the page. In addition, where do I find the login credentials to the SSVM.
Also shouldn't there be a virtual router created as well for the gateway of the guest network?

@Atiqul-Islam sorry for late response.
you can ssh into system vms and virtual routers from the kvm host.
ssh -p 3922 -i /root/.ssh/id_rsa.cloud 169.254.x.x
or "virsh console s-xx-VM"
the credential is root/password
The virtual router will be created when a vm is created I think.

@Atiqul-Islam I am closing this issue. please reopen or create a new one if ou think that is invalid.

authors: Atiqul-Islam DaanHoogland Pearl1594 weizhouapache


text:
[UI] Storage menu not showing even with API permissions

ISSUE TYPE

Bug Report

COMPONENT NAME
UI

CLOUDSTACK VERSION
Main

SUMMARY
As reported in https://github.com/apache/cloudstack/pull/8713#issuecomment-1969866705, the Storage submenu in the sidebar is not displayed to users when they do not have permission to the API listVolumesMetrics. However, roles can have permissions to other APIs, such as listBackups and listSnapshots, in which case the submenu should be displayed. This scenario is probably not exclusive to the Storage menu.
STEPS TO REPRODUCE
Create a role with permission to allow the APIs listBackups and listSnapshots and deny the API listVolumesMetrics. The UI dashboard will not show the Storage menu in the sidebar.
EXPECTED RESULTS
The UI should show the Storage submenu alongside with its own Backups and Snapshots submenus.
ACTUAL RESULTS
The submenu Storage is not displayed, even though the role has permission to list snapshots and backups.

@DaanHoogland @winterhazel @sureshanaparti, I am probably overthinking this scenario, however, the permission property in the JS component (storage.js) works like an AND operator. Maybe it could function like an OR operator as well; what do you guys think?

@DaanHoogland @winterhazel @sureshanaparti, I am probably overthinking this scenario, however, the permission property in the JS component (storage.js) works like an AND operator. Maybe it could function like an OR operator as well; what do you guys think?

This was exactly my thought when I learned of it @BryanMLima . Let's try.

Hey everyone,
I've implemented @BryanMLima's idea in #8978. The filtering still works as an AND operator for routes that correspond to a page; however, I have changed so that routes corresponding to sections get shown if the user has access to any of its pages.

Closing this as this was addressed in PR #8978.

authors: BryanMLima DaanHoogland winterhazel


text:
Allow CGN (RFC6598) to be used within a VPC

Add the CGN network to the list of allowed netblocks for guest networks.  Additionally convert the previous strings to a list to allow easier modification/expansion in the future if needed.

Tested in a 4.8 lab. Verified as per screen shots that all RFC 1918 and RFC 6598 ranges work. Attempting to use a public cidr produces an error as expected. Interface is correctly configured on the VR and the routing table is correct and matches the interface the private gateway was configured on (eth3).
LGTM

LGTM (did not test it, kicked a packaging build)

Packaging result: ✖centos6 ✖centos7 ✖debian repo: http://packages.shapeblue.com/cloudstack/pr/1606

Packaging result: ✔centos6 ✔centos7 ✖debian repo: http://packages.shapeblue.com/cloudstack/pr/1606

@blueorangutan package

@rhtyd a Trillian-Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian repo: http://packages.shapeblue.com/cloudstack/pr/1606

@kiwiflyer are there Marvin tests that verify this behavior?

John, I'll defer to Aaron on this, as he submitted the PR.

@leprechau following up regarding Marvin tests.  Unless they are ready, we are likely going to have to push this PR to 4.9.2.0 since I am trying to get a 4.8.2.0 RC cut ASAP (we are week late already).

@jburwell Not really sure what you would want as far as tests.  There shouldn't be any change that would alter existing functionality as demonstrated by the above screenshots.  If there is something I'm missing please let me know.

@leprechau you have expanded the type of networks blocks supported by the system.  Therefore, there should be a Marvin test case that attempts to specify a CGN network block for a guest network and verifies that the system behaves as expected.  In particular, the test should verify that the API calls are successful, and that the networks are implemented as expected when a CGN network block is specified.

@jburwell Is there already an existing test for the previous behavior that could be modified?  I haven't had a chance to do much with Marvin other than briefly browse the tests folder.

@leprechau You may add an unit test as well for NetUtils.validateGuestCidr(). Check NetUtilsTest.java, there is already an existing test for 192.168.

@leprechau I would like to get this fix into 4.8.2.0, 4.9.1.0, and 4.10.0.0.  Have you had a chance to add the unit tests requested?  Also, is there a JIRA ticket for this bug?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-209

LGTM on tests and code review.
@leprechau @kiwiflyer can we have a JIRA id for this and use that in the commit summary.

Ping @leprechau @kiwiflyer

I've added issue Jira CLOUDSTACK-9661 to track this.
https://issues.apache.org/jira/browse/CLOUDSTACK-9661
@leprechau @rhtyd

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-378

What's the current status of this PR?  Is this acceptable as is?

Is it appropriate to add a comparatively complex integration test to cover a functional predicate whose cognitive load is so small?
I will expand the associated unit test to include boundary conditions testing of the CGN address range, and that would seem to me to be sufficient. Please advise.
@karuturi

@rossor unittest would be nice to have. Can you also rebase this PR to 4.9?

@rhtyd Can you run tests on this please?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-678

@borisstoyanov @rhtyd @DaanHoogland Can you run tests on this PR? it only has package build

@leprechau @karuturi I can run tests but there are no added unit or integration tests. This has no value for this fix, only for detecting regression in other code or in the tests due to this change.
@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1045)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29262 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1606-t1045-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_outofbandmanagement.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Test completed. 47 look ok, 2 have error(s)

Test
Result
Time (s)
Test File

test_04_rvpc_privategw_static_routes
Failure
350.99
test_privategw_acl.py

test_02_list_snapshots_with_removed_data_store
Error
0.04
test_snapshots.py

test_01_vpc_site2site_vpn
Success
180.27
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
71.19
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
265.66
test_vpc_vpn.py

test_02_VPC_default_routes
Success
299.61
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
528.51
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
500.14
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1423.61
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
635.89
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Success
741.67
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1328.48
test_vpc_redundant.py

test_09_delete_detached_volume
Success
156.56
test_volumes.py

test_08_resize_volume
Success
156.50
test_volumes.py

test_07_resize_fail
Success
161.62
test_volumes.py

test_06_download_detached_volume
Success
151.32
test_volumes.py

test_05_detach_volume
Success
150.75
test_volumes.py

test_04_delete_attached_volume
Success
151.32
test_volumes.py

test_03_download_attached_volume
Success
156.47
test_volumes.py

test_02_attach_volume
Success
84.79
test_volumes.py

test_01_create_volume
Success
621.14
test_volumes.py

test_deploy_vm_multiple
Success
268.00
test_vm_life_cycle.py

test_deploy_vm
Success
0.03
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.03
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
61.86
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.22
test_vm_life_cycle.py

test_08_migrate_vm
Success
36.00
test_vm_life_cycle.py

test_07_restore_vm
Success
0.13
test_vm_life_cycle.py

test_06_destroy_vm
Success
125.82
test_vm_life_cycle.py

test_03_reboot_vm
Success
125.95
test_vm_life_cycle.py

test_02_start_vm
Success
10.23
test_vm_life_cycle.py

test_01_stop_vm
Success
30.28
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
105.88
test_templates.py

test_08_list_system_templates
Success
0.06
test_templates.py

test_07_list_public_templates
Success
0.04
test_templates.py

test_05_template_permissions
Success
0.06
test_templates.py

test_04_extract_template
Success
5.14
test_templates.py

test_03_delete_template
Success
5.12
test_templates.py

test_02_edit_template
Success
90.09
test_templates.py

test_01_create_template
Success
55.66
test_templates.py

test_10_destroy_cpvm
Success
131.70
test_ssvm.py

test_09_destroy_ssvm
Success
168.99
test_ssvm.py

test_08_reboot_cpvm
Success
131.52
test_ssvm.py

test_07_reboot_ssvm
Success
133.92
test_ssvm.py

test_06_stop_cpvm
Success
131.52
test_ssvm.py

test_05_stop_ssvm
Success
133.94
test_ssvm.py

test_04_cpvm_internals
Success
0.98
test_ssvm.py

test_03_ssvm_internals
Success
3.61
test_ssvm.py

test_02_list_cpvm_vm
Success
0.13
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.14
test_ssvm.py

test_01_snapshot_root_disk
Success
11.17
test_snapshots.py

test_04_change_offering_small
Success
209.89
test_service_offerings.py

test_03_delete_service_offering
Success
0.04
test_service_offerings.py

test_02_edit_service_offering
Success
0.08
test_service_offerings.py

test_01_create_service_offering
Success
0.11
test_service_offerings.py

test_02_sys_template_ready
Success
0.16
test_secondary_storage.py

test_01_sys_vm_start
Success
0.19
test_secondary_storage.py

test_09_reboot_router
Success
40.44
test_routers.py

test_08_start_router
Success
35.45
test_routers.py

test_07_stop_router
Success
10.18
test_routers.py

test_06_router_advanced
Success
0.07
test_routers.py

test_05_router_basic
Success
0.05
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.82
test_routers.py

test_03_restart_network_cleanup
Success
65.78
test_routers.py

test_02_router_internal_adv
Success
1.18
test_routers.py

test_01_router_internal_basic
Success
0.64
test_routers.py

test_router_dns_guestipquery
Success
74.75
test_router_dns.py

test_router_dns_externalipquery
Success
0.08
test_router_dns.py

test_router_dhcphosts
Success
236.57
test_router_dhcphosts.py

test_router_dhcp_opts
Success
21.78
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
5.16
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
130.94
test_reset_vm_on_reboot.py

test_createRegion
Success
0.04
test_regions.py

test_create_pvlan_network
Success
5.28
test_pvlan.py

test_dedicatePublicIpRange
Success
0.55
test_public_ip_range.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
546.27
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
416.44
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
97.64
test_privategw_acl.py

test_01_primary_storage_nfs
Success
35.96
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.22
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.51
test_portable_publicip.py

test_isolate_network_password_server
Success
59.26
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.13
test_over_provisioning.py

test_oobm_zchange_password
Success
25.76
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.39
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
5.23
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.38
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
15.40
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
10.47
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
15.35
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
10.34
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
67.78
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.55
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.28
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
5.29
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.21
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.14
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.15
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
23.42
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.35
test_non_contigiousvlan.py

test_01_nic
Success
454.73
test_nic.py

test_releaseIP
Success
278.18
test_network.py

test_reboot_router
Success
383.49
test_network.py

test_public_ip_user_account
Success
10.27
test_network.py

test_public_ip_admin_account
Success
40.32
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
66.92
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.85
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
121.72
test_network.py

test_delete_account
Success
303.00
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.80
test_network.py

test_01_port_fwd_on_src_nat
Success
111.79
test_network.py

test_nic_secondaryip_add_remove
Success
227.76
test_multipleips_per_nic.py

test_list_zones_metrics
Success
0.27
test_metrics_api.py

test_list_volumes_metrics
Success
5.54
test_metrics_api.py

test_list_vms_metrics
Success
222.04
test_metrics_api.py

test_list_pstorage_metrics
Success
0.42
test_metrics_api.py

test_list_infrastructure_metrics
Success
0.64
test_metrics_api.py

test_list_hosts_metrics
Success
0.48
test_metrics_api.py

test_list_clusters_metrics
Success
0.47
test_metrics_api.py

login_test_saml_user
Success
19.90
test_login.py

test_assign_and_removal_lb
Success
133.15
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
187.31
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
207.95
test_loadbalance.py

test_03_list_snapshots
Success
0.08
test_list_ids_parameter.py

test_02_list_templates
Success
0.05
test_list_ids_parameter.py

test_01_list_volumes
Success
0.03
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.07
test_iso.py

test_05_iso_permissions
Success
0.07
test_iso.py

test_04_extract_Iso
Success
5.19
test_iso.py

test_03_delete_iso
Success
95.15
test_iso.py

test_02_edit_iso
Success
0.06
test_iso.py

test_01_create_iso
Success
21.04
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
184.00
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
153.19
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
520.03
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
430.26
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.25
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.13
test_global_settings.py

test_rolepermission_lifecycle_update
Success
6.15
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
6.00
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
5.85
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.93
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
6.00
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.92
test_dynamicroles.py

test_role_lifecycle_update
Success
10.96
test_dynamicroles.py

test_role_lifecycle_list
Success
5.89
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.93
test_dynamicroles.py

test_role_lifecycle_create
Success
5.89
test_dynamicroles.py

test_role_inuse_deletion
Success
5.89
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
8.12
test_dynamicroles.py

test_role_account_acls
Success
8.59
test_dynamicroles.py

test_default_role_deletion
Success
6.02
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.07
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.04
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.05
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.07
test_disk_offerings.py

test_01_create_disk_offering
Success
0.11
test_disk_offerings.py

test_deployvm_userdispersing
Success
20.57
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
20.68
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
116.08
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
10.40
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
45.68
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
6.03
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
5.99
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
247.71
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
202.41
test_deploy_vm_iso.py

test_DeployVmAntiAffinityGroup
Success
55.94
test_affinity_groups.py

test_03_delete_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_02_revert_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_01_create_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.02
test_staticroles.py

test_01_scale_vm
Skipped
0.00
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.05
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.01
test_deploy_vgpu_enabled_vm.py

@rossor Can you check on the travis and jenkins failures?
looks like there are checkstyle issues in the new code.

@rossor @nathanejohnson Can we get the code formatted correctly so the tests pass?

@kiwiflyer @karuturi Tests passed!

authors: DaanHoogland blueorangutan jburwell karuturi kiwiflyer koushik-das leprechau rhtyd rossor


text:
CLOUDSTACK-7958: Add configuration for limit to CIDRs for Admin API calls

The global setting 'management.admin.cidr' is set to 0.0.0.0/0,::/0
by default preserve the current behavior and thus allow API calls
for Admin accounts from all IPv4 and IPv6 subnets.
Users can set it to a comma-separated list of IPv4/IPv6 subnets to
restrict API calls for Admin accounts to certain parts of their network(s).
This is to improve Security. Should a attacker steal the Access/Secret key
of a Admin account he/she still needs to be in a subnet from where Admin accounts
are allowed to perform API calls.
This is a good security measure for APIs which are connected to the public internet.
This PR also includes a commit to cleanup and improve NetUtils.
No existing methods have been altered. That has been verified by adding additional Unit Tests for this.

@DaanHoogland: I improved the logging as you suggested/requested.
A TRACE for every request and WARN when a request is denied. Tried this locally:

2017-04-14 15:45:58,901 WARN  [c.c.a.ApiServlet] (catalina-exec-17:ctx-5955fcab ctx-c572b42e) (logid:7b251506) Request by accountId 2 was denied since 192.168.122.1 does not match 127.0.0.1/8,::1/128

In this case only localhost (IPv4/IPv6) is allowed to perform requests.

@PaulAngus This is what we talked about in Prague. Mind taking a look?

Nice @wido, will give it a go soon!

Thanks @remibergsma. Added the return statement.
Thinking about it. Does a 401 sound good? Or should we maybe use a 403 Forbidden?

@wido I'm playing a bit with it, also because currently CloudMonkey displays a clear message but the UI will simply freeze and do nothing. Only when you look at the underlying API calls you'll see why it isn't working. I'm testing to see the http status code makes any difference of that we need to handle it in the UI somewhere.
We should also think about the order: first alert on the CIDR check and then check user/pass (as it is now) or the other way around.
Also noticed it doesn't work with spaces before/after comma's so we might want to add a .replaceAll("\\s","") or similar.

@wido very nice feature.
It would also be nice if this can be added to domain or account setting.

@remibergsma: Yes, I am aware of that UI problem. Not sure how to fix it.
After thinking about it, I went for '403 Forbidden' and also stripped whitespace from the config key.
I think the order is OK right now unless other opinions?
@ustcweizhou: That is a lot more difficult then a global value, isn't it? Since you have to query it every time.
Or should configkey allow this very easily?

@wido I think we need to do the check on two places, also on the login() method. That makes sure we don't issue a session key when user/pass are OK but we still reject it based on the CIDR. In my testing that also fixes the UI issue. There are two ways to authenticate so that makes sense I'd say. It'll then also work with authentication plugins, such as LDAP/AD.
Switching the scope of the config is easy, but indeed you'll be querying it on every API call. That does have the benefit you don't need to restart the mgt server when you make a change, but the downside is also obvious. One way to resolve it, is to make a global config setting that switches the feature on/off (and that config is loaded at bootstrap) so you can opt-in for the more heavy checks.
I'll play a bit more with it tonight.

@remibergsma: I pulled your code, thanks! It now works per account @ustcweizhou
How does this look?

Good one @remibergsma. I reverted that piece and also the baremetal refusal of users.

There were some conflicts after changes were made in master. Fixed those.
As this one is merge ready, can it go into master now?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-839

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-844

@wido can you check/fix build failures?

@rhtyd: I went through the logs, but there is nothing that I can find that points to this PR.
What I do see:
2017-07-19 15:16:07,845 DEBUG [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:ctx-09893152) (logid:21cb76f0) Performing image store createTemplate async callback
2017-07-19 15:16:07,859 WARN  [o.a.c.alerts] (RemoteHostEndPoint-1:ctx-09893152) (logid:21cb76f0)  alertType:: 28 // dataCenterId:: 1 // podId:: null // clusterId:: null // message:: Failed to register template: d8e68e73-fa32-4309-97e5-436da98c7568 with error:  HTTP Server returned 403 (expected 200 OK)
2017-07-19 15:16:07,863 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-1:ctx-09893152) (logid:21cb76f0) Failed to register template: d8e68e73-fa32-4309-97e5-436da98c7568 with error:  HTTP Server returned 403 (expected 200 OK)
Looking at the logs even further:

2017-07-19 17:07:59,887 DEBUG [c.c.a.ApiServer] (qtp411594792-25:ctx-019cac6d ctx-c78c28dc) (logid:92eb3c37) CIDRs from which account 'Acct[b707c234-6ca4-11e7-849c-00163e174325-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2017-07-19 17:07:59,920 DEBUG [c.c.a.ApiServer] (qtp411594792-24:ctx-65cfefc0 ctx-264d5734) (logid:a06700ca) CIDRs from which account 'Acct[b707c234-6ca4-11e7-849c-00163e174325-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2017-07-19 17:08:59,944 DEBUG [c.c.a.ApiServer] (qtp411594792-29:ctx-b8e61282 ctx-e1ad15c2) (logid:54c2e421) CIDRs from which account 'Acct[b707c234-6ca4-11e7-849c-00163e174325-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2017-07-19 17:08:59,970 DEBUG [c.c.a.ApiServer] (qtp411594792-25:ctx-3901b66d ctx-5861a4b0) (logid:220067e9) CIDRs from which account 'Acct[b707c234-6ca4-11e7-849c-00163e174325-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2017-07-19 17:09:00,840 DEBUG [c.c.a.ApiServer] (qtp411594792-24:ctx-b991e50f ctx-df021a2a) (logid:f3c648cf) CIDRs from which account 'Acct[b707c234-6ca4-11e7-849c-00163e174325-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0

Seems like this PR isn't blocking any API-calls or such. Everything seems to pass.

@wido it's a build failure issue, please see travis (job#1) failure:
[INFO] Compiling 45 source files to /home/travis/build/apache/cloudstack/vmware-base/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/build/apache/cloudstack/vmware-base/src/com/cloud/hypervisor/vmware/mo/HypervisorHostHelper.java:[1456,20] error: cannot find symbol
[INFO] 1 error
...
[INFO] Apache CloudStack VMware Base ...................... FAILURE [  4.785 s]

Fixed @rhtyd

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:52 min (Wall Clock)
[INFO] Finished at: 2017-07-24T14:07:11+02:00
[INFO] Final Memory: 118M/1939M
[INFO] ------------------------------------------------------------------------

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-856

@wido still failing, please do a clean rebuild, check Travis?

ACS CI BVT Run
Sumarry:
Build Number 1006
Hypervisor xenserver
NetworkType Advanced
Passed=102
Failed=11
Skipped=12
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0
Failed tests:

test_scale_vm.py

ContextSuite context=TestScaleVm>:setup Failing since 33 runs

test_loadbalance.py

test_01_create_lb_rule_src_nat Failed

test_02_create_lb_rule_non_nat Failed

test_non_contigiousvlan.py

test_extendPhysicalNetworkVlan Failed

test_deploy_vm_iso.py

test_deploy_vm_from_iso Failing since 63 runs

test_volumes.py

test_06_download_detached_volume Failing since 3 runs

test_vm_life_cycle.py

test_10_attachAndDetach_iso Failing since 63 runs

test_routers_network_ops.py

test_01_isolate_network_FW_PF_default_routes_egress_true Failing since 96 runs

test_02_isolate_network_FW_PF_default_routes_egress_false Failing since 96 runs

test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 94 runs

test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failing since 94 runs

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_disk_offerings.py

I tried again @rhtyd

wido@wido-laptop:~/repos/cloudstack$ mvn -T2C clean install

This works! All unit tests also pass. No build failure either:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:30 min (Wall Clock)
[INFO] Finished at: 2017-07-25T09:27:50+02:00
[INFO] Final Memory: 119M/1556M
[INFO] ------------------------------------------------------------------------

@wido make sure you're not breaking noredist builds as well, see Travis and try to get it green;
[INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ cloud-plugin-hypervisor-vmware ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 51 source files to /home/travis/build/apache/cloudstack/plugins/hypervisors/vmware/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/build/apache/cloudstack/plugins/hypervisors/vmware/src/com/cloud/hypervisor/guru/VMwareGuru.java:[534,35] error: cannot find symbol
[INFO] 1 error

Aha @rhtyd!
I've fixed that

ACS CI BVT Run
Sumarry:
Build Number 1014
Hypervisor xenserver
NetworkType Advanced
Passed=102
Failed=9
Skipped=12
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0
Failed tests:

test_vm_snapshots.py

test_change_service_offering_for_vm_with_snapshots Failed

test_deploy_vm_iso.py

test_deploy_vm_from_iso Failing since 69 runs

test_list_ids_parameter.py

ContextSuite context=TestListIdsParams>:setup Failing since 45 runs

test_volumes.py

test_06_download_detached_volume Failed

test_vm_life_cycle.py

test_10_attachAndDetach_iso Failing since 69 runs

test_routers_network_ops.py

test_01_isolate_network_FW_PF_default_routes_egress_true Failing since 102 runs

test_02_isolate_network_FW_PF_default_routes_egress_false Failing since 102 runs

test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 100 runs

test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failing since 100 runs

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_disk_offerings.py

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-867

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

I've tried several times now, the environments fails to come up; perhaps @borisstoyanov  can takeover testing...
21:38:41 FAILED - RETRYING: TASK: get wait for state of system VMs to be Running (1 retries left).
21:38:47 fatal: [pr2046-t1277-kvm-centos7-mgmt1]: FAILED! => {"attempts": 200, "changed": true, "cmd": "cloudmonkey list systemvms | jq '.systemvm[]| select(.systemvmtype==\"consoleproxy\")|.state'", "delta": "0:00:00.186658", "end": "2017-07-28 20:38:19.617884", "failed": true, "rc": 0, "start": "2017-07-28 20:38:19.431226", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
21:38:47
21:38:47 cmd: cloudmonkey list systemvms | jq '.systemvm[]| select(.systemvmtype=="consoleproxy")|.state'
21:38:47
21:38:47 start: 2017-07-28 20:38:19.431226

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1018

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1038

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@wido I've tried 4 times now, the trillian environment fails to come up due to systemvms fail to come up. Do you think there are changes that might affect systemvm agents/patching? /cc @borisstoyanov

FYI, we haven't seen this in other master PRs, so could be related to these changes...

@wido can you fix the conflicts?

@rhtyd Fixed!

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1427

I will fix fhe conflicts asap

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1500

very good !

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1505

@wido can you resolve the conflicts, thanks.

I will in a few days.
Can we merge this one afterwards? I keep resolving conflicts which happen since other code is merged ;)

Sure, ping me @wido

@wido sorry to bring up the old pr but how can I configure under this account level?
I logged as in a regular user with "Domain admin" role but i dont see any settings tab under the account

@wido sorry to bring up the old pr but how can I configure under this account level? I logged as in a regular user with "Domain admin" role but i dont see any settings tab under the account

@ravening please have a look at #4339

authors: blueorangutan borisstoyanov cloudmonger ravening remibergsma rhtyd ustcweizhou weizhouapache wido


text:
COLLECTIONS-701 SetUniqueList.add() crashes due to infinite recursion…

… when it receives itself

Hi @drajakumar ,
I'm not sure this patch makes sense. Take a look at org.apache.commons.collections4.list.Collections701Test: For ArrayList and HashSet, adding a collection to itself is fine.
In this patch, the argument is not only silently ignored, but the behavior is not even documented. Whatever we do, we really need to document anything that deviates from the standard JRE List contract.
IMO, the fix should be so that a SetUniqueList behaves like a ArrayList and HashSet, it just works.

You did not have to close the PR, I was hoping you would provide a more complete solution ;-)

@garydgregory can you kindly check the new fix, thank you!

@garydgregory can you kindly check the new fix, thank you!

authors: drajakumar garydgregory

repo: apache/commons-collections


text:
LANG-1258: Add ArrayUtils#toStringArray(Object[]) method

patch supplied by IG

Coverage increased (+0.04%) to 93.581% when pulling bfc8805ac898cb4d7a8ca45ab1d5106bc2b63342 on PascalSchumacher:ArrayUtils#toStringArray into 91d6bd74fa358fdc8d7cb7681c76c509fd9a8e7d on apache:master.

Coverage increased (+0.04%) to 93.581% when pulling bfc8805ac898cb4d7a8ca45ab1d5106bc2b63342 on PascalSchumacher:ArrayUtils#toStringArray into 91d6bd74fa358fdc8d7cb7681c76c509fd9a8e7d on apache:master.

Patch looks good. I wonder if the inline if will raise a warning in checkstyle. Other than that, +1 :D

I added null elements handling in PR to this branch, because in current state NPE would be thrown in case of null in array.

I have updated the pull request with @Xaerxess changes.

Coverage increased (+0.01%) to 93.551% when pulling bf978e7ec7a1bb3d1d671331383619d81bae95ed on PascalSchumacher:ArrayUtils#toStringArray into 91d6bd74fa358fdc8d7cb7681c76c509fd9a8e7d on apache:master.

Sure, consistency in API is the key. But stringIfNull is used when array itself is null, not element, so it wouldn't be consistent with the API.

@Xaerxess Sorry for the confusion, I wasn't talking about toString, but about the toPrimitive methods e.g.
https://github.com/apache/commons-lang/blob/96c8ea2fb3719e2f6e3d7a4d7b46718f26515a86/src/main/java/org/apache/commons/lang3/ArrayUtils.java#L4497
As these also convert the type of the array I think they are the most similar existing methods (compared to the new method in this pull request).

Coverage increased (+0.008%) to 93.563% when pulling 8f3577b6919c8bbd806cd499b36babd61f9d3bb5 on PascalSchumacher:ArrayUtils#toStringArray into ff4497aff8cc9de4e0b2c6e5e23e5b6550f76f29 on apache:master.

Coverage increased (+0.03%) to 93.588% when pulling 8f3577b6919c8bbd806cd499b36babd61f9d3bb5 on PascalSchumacher:ArrayUtils#toStringArray into ff4497aff8cc9de4e0b2c6e5e23e5b6550f76f29 on apache:master.

Coverage increased (+0.03%) to 93.588% when pulling 8f3577b6919c8bbd806cd499b36babd61f9d3bb5 on PascalSchumacher:ArrayUtils#toStringArray into ff4497aff8cc9de4e0b2c6e5e23e5b6550f76f29 on apache:master.

Coverage increased (+0.008%) to 93.563% when pulling 8f3577b6919c8bbd806cd499b36babd61f9d3bb5 on PascalSchumacher:ArrayUtils#toStringArray into ff4497aff8cc9de4e0b2c6e5e23e5b6550f76f29 on apache:master.

Coverage increased (+0.04%) to 93.595% when pulling 8f3577b6919c8bbd806cd499b36babd61f9d3bb5 on PascalSchumacher:ArrayUtils#toStringArray into ff4497aff8cc9de4e0b2c6e5e23e5b6550f76f29 on apache:master.

Merged: https://github.com/apache/commons-lang/commit/8d95ae41975a2307501aa0f4a7eba296c59edce9 and https://github.com/apache/commons-lang/commit/8d601ab71228f7c3dff950540e7ee6e4043e9053
Thanks everybody!

authors: PascalSchumacher Xaerxess coveralls kinow

repo: apache/commons-lang


text:
WKWebview Won't Load

I receive the following error when I have wk webview installed:
Failed to load resource: file:///var/containers/Bundle/Application/39F29CF9-4A2F-4A20-A296-0CAE487974B3/my.app/www/plugins/cordova-plugin-wkwebview-engine/src/www/ios/ios-wkwebview.js The requested URL was not found on this server.
Failed to load resource: file:///var/containers/Bundle/Application/39F29CF9-4A2F-4A20-A296-0CAE487974B3/my.app/www/plugins/cordova-plugin-wkwebview-engine/src/www/ios/ios-wkwebview-exec.js The requested URL was not found on this server.
Error: Module cordova-plugin-wkwebview-engine.ios-wkwebview-exec does not exist.
I have the following in my config.xml
<feature name="CDVWKWebViewEngine">
        <param name="ios-package" value="CDVWKWebViewEngine" />
</feature>
<preference name="CordovaWebViewEngine" value="CDVWKWebViewEngine" />

I noticed when I run cordova build ios, the 2 js files in the plugins folder get created, then deleted. Not sure why. Every other plugin is working fine.

I had the same issue and solved it installing : cordova plugin add cordova-plugin-wkwebview-engine. Apparently this plugin needs to be installed for the fix to work.
Dont't forget to add ALL this in your config:

..........[other code]

Closing, the solution is posted above.

authors: adadgio albertleao breautek

repo: apache/cordova-plugin-wkwebview-engine


text:
Microsoft Surface hybrid touch events

Bug Report
Problem
Touch events are not fired correctly
What is expected to happen?
Moving camera in webgl using touch on screen
What does actually happen?
Not any touch events are fired, only mouse and mouse down, few mouse move then nothing, no mouse up.
If I touch without moving mouse up is fired.
Information

Same site on web work correctly with the same device.
Command or Code

Environment, Platform, Device

Microsoft Surface hybrid computer
Version information

Cordova 10.0.0
cordova-windows 8.0.0-dev
Windows 10
Checklist

[x] I searched for existing GitHub issues
[x] I updated all Cordova tooling to most recent version
[x] I included all the necessary information above

We are archiving this repository following Apache Cordova's Deprecation Policy. We will not continue to work on this repository. Therefore all issues and pull requests are being closed. Thanks for your contribution.

repo: apache/cordova-windows


text:
Call CouchDB's CSRF validation

COUCHDB-2762

+1

authors: kxepal rnewson

repo: apache/couchdb-chttpd


text:
Changes feed with selector filter doesnt show deleted docs

Description
Deleted documents are not included in the changes feed when filtering with _selector. It does work while filtering by doc_ids, which is not applicable in my scenario.
Steps to Reproduce
# Create document
curl -H 'Content-Type: application/json' -X PUT "localhost:5984/storage/123456" -d '{"name": "Document Name"}'

# Start changes feed
curl -H 'Content-Type: application/json' -X POST "localhost:5984/storage/_changes?feed=longpoll&filter=_selector&since=now" -d '{"selector": {"name": "Document Name"}}'

# Remove document (in another shell, of course)
curl -H 'Content-Type: application/json' -X DELETE "localhost:5984/storage/123456?rev=<LAST-REV>"

Expected Behaviour
The request to _changes should terminate with a results object for that particular document, including deleted=true.
Your Environment
CouchDB running from (official) docker image:
{
    "couchdb": "Welcome",
    "features": [
        "pluggable-storage-engines",
        "scheduler"
    ],
    "git_sha": "c298091a4",
    "uuid": "54b4e44520a6fc9996a7eb635783fa96",
    "vendor": {
        "name": "The Apache Software Foundation"
    },
    "version": "2.3.1"
}

In your third step you remove the 'name' property of the document, and thus it no longer matches your selector.
The DELETE method performs a PUT preserving only _id, _rev and _deleted (set to true).
Instead do a PUT, keeping the "name" field and adding "_deleted":true

Hi there,
This is not a CouchDB bug. GitHub is for actual CouchDB bugs only.
If you are looking for general support with using CouchDB, please try one of these other options:

The user mailing list. Signup instructions are here
The Slack/IRC chat room. Joining instructions are here

Well, that does make sense. I would suggest to mention this in the docs even though it is self-explanatory.
Thanks for the quick response and sorry for opening this as a bug.

authors: modul rnewson wohali

repo: apache/couchdb


text:
Remove get_details replicator job gen_server call

This was used from a test only and it wasn't reliable. Because of replicator
job delays initialization the State would be either #rep_state{} or #rep. If
replication job hasn't finished initializing, then state would be #rep{} and a
call like get_details which matches the state with #rep_state{] would fail with
the badmatch error.
As seen in issue #686
So remove get_details call and let the test rely on task polling as all other
tests do.

@nickva: While on it could you fix a compile warning on line 100?
couchdb/src/couch_replicator/test/couch_replicator_compact_tests.erl:100: Warning: variable 'RepId' is unused

authors: iilyak nickva


text:
Implement native parsing of CSV files

What is the problem the feature request solves?
We can probably accelerate reading of CSV files by continuing to use JVM Spark to read bytes from disk but then parse the CSV in native code.
Describe the potential solution
No response
Additional context
No response

Hello.
I would like to start working on this.

authors: andygrove psvri

repo: apache/datafusion-comet


text:
Using Expr::field panics

Describe the bug
After https://github.com/apache/datafusion/pull/10375 merged Expr::field now panics if you use it (as we did in influxdb) DataFusion panics when you try to execute it
To Reproduce
Try to evaluate an expression like col("props").field("a")
Here is a full reproducer in the sql_integration test:
(venv) andrewlamb@Andrews-MacBook-Pro:~/Software/datafusion$ git diff
diff --git a/datafusion/core/tests/expr_api/mod.rs b/datafusion/core/tests/expr_api/mod.rs
index d7e839824..d4141a836 100644
--- a/datafusion/core/tests/expr_api/mod.rs
+++ b/datafusion/core/tests/expr_api/mod.rs
@@ -58,6 +58,25 @@ fn test_eq_with_coercion() {
     );
 }

+
+#[test]
+fn test_expr_field() {
+    // currently panics with
+    // Internal("NamedStructField should be rewritten in OperatorToFunction")
+    evaluate_expr_test(
+        col("props").field("a"),
+        vec![
+            "+------------+",
+            "| expr       |",
+            "+------------+",
+            "| 2021-02-01 |",
+            "| 2021-02-02 |",
+            "| 2021-02-03 |",
+            "+------------+",
+        ],
+    );
+}
+

Expected behavior
Ideally the test should pass Expr::field would continue to work
We could also potentially remove Expr::field but I think that would be less user friendly
Additional context
I am pretty sure I Expr::field is widely used so I think we should continue to support it if possible

I wonder if we could have Expr::field call get_field if the core functions feature was enabled and panic otherwise 🤔
That would be easy to use for most people and backwards compatible

I can fix this along with #10374

Thank you @jayzhan211 🙏  -- I will review it now

repo: apache/datafusion


text:
[Bug] [Next-UI][V1.0.0-Beta] Local log file cache is not cleared

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
View the first component log, when viewing the second component log, it is displayed as the first component log

What you expected to happen
View the first component log, when viewing the second component log, it is displayed to see the second component log
How to reproduce
View the first component log, when viewing the second component log, it is displayed as the first component log
Anything else
No response
Version
3.0.0-alpha
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

i will fix it

authors: XuXuClassMate labbomb

repo: apache/dolphinscheduler


text:
[Feature-15260][dolphinscheduler-datasource-hana] add hana  related dependencies

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

[Feature][dolphinscheduler-datasource-hana] add hana related dependencies #15259

same with #15127

与#15127相同

same with #15127
This feature was developed by me, but not the PR I submitted. I may be clearer on how to modify it. If you need to merge # 15127, please merge it into the dev branch as soon as possible. My # 15146 requires this modification item

与#15127相同

This feature was developed by me, but not the PR I submitted. I may be clearer on how to modify it. If you need to merge #15127  , please merge it into the dev branch as soon as possible. My #15146  requires this modification item

cc @caishunfeng

test use in the wrong place

Codecov Report
Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (0c470ff) 38.19% compared to head (ad073a9) 38.16%.

:exclamation: Current head ad073a9 differs from pull request most recent head 98670be. Consider uploading reports for the commit 98670be to get more accurate results

Files
Patch %
Lines

.../plugin/datasource/hana/HanaDataSourceChannel.java
0.00%
2 Missing :warning:

...in/datasource/hana/HanaPooledDataSourceClient.java
0.00%
2 Missing :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15260      +/-   ##
============================================
- Coverage     38.19%   38.16%   -0.03%
+ Complexity     4673     4671       -2
============================================
  Files          1278     1285       +7
  Lines         44482    44463      -19
  Branches       4783     4770      -13
============================================
- Hits          16988    16968      -20
- Misses        25632    25633       +1
  Partials       1862     1862

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter davidzollo fuchanghai xujiaqiang


text:
[Feature] Mysql subtable with binlog model merge one table

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
I want to megre Mysql subtable into one table and can update by unique key
Use case
No response
Related issues
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

You can use Agg key and replace_if_not_null implement multitable merge.

repo: apache/doris


text:
Enhancement delete skiplist for duplicate table in memtable

Proposed changes
delete skiplist for dup table when data load.
Problem summary
when load data, data will be insert into skiplist in memtable, for each row skiplist need to Find and Sort, which time cost is O(log(n)), I don't think it's a good way.
There are two ways to insert data into memtable:

insert into skiplist: O(log(n)), when need flush, the data is already sorted
insert into a block(append only): O(1), when need flush, sort once.

this pr implement way2 for duplicate table: append data to a block and sort when need flush.
I don't know how good or bad it turned out
This pr needs to use the community's testing framework
So I submit first to see the result.

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 36.64 seconds
load time: 517 seconds
storage size: 17134295852 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230112052016_clickbench_pr_78264.html

authors: hello-stephen zbtzbtzbt


text:
Chore make compile option work on C objects && some refactor of cmakelists

Proposed changes
make compile option work on C objects && some refactor of cmakelists
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.6 seconds
load time: 467 seconds
storage size: 17170873384 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230206145530_clickbench_pr_90979.html

authors: BiteTheDDDDt hello-stephen


text:
Fixadd sync after insert into table for nereids_p0

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
Bug fix ScannerContext is done make query failed

Proposed changes
fix ScannerContext is done make query failed
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[Fix](inverted index) reorder ConjunctionQuery deconstruct order

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
cases Add backup & restore test case of dup table

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


text:
enhance Reduce log in tablet meta

Proposed changes
Reduce log in tablet meta
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

authors: dataroaring platoneko


text:
Fix query not hit partition in original planner

Proposed changes
Fix issue  that query not hit partition  in original planner,  which will cause  serve performance degradation.
this issue seems introduced by https://github.com/apache/doris/pull/21533

run buildall

run p0

run buildall

run p0

run buildall

run  buildall

could u submit a pr to master?

authors: GoGoWen morrySnow


text:
2.0upgrade dependencies

Proposed changes
Issue Number: close #xxx

run buildall


text:
[fix](inverted index)Add exception check when write bkd index

Proposed changes
We are not catching the exception when add values in bkd_writer, if error throws, BE will run into segment fault.
So we add the exception check here to avoid coredump.

run buildall