text:
Refactor/libvirt resource

This PR contains the refactor of the LibvirtComputingResource. The motivation behind this effort is to increase maintainability and testability of the code.
The LibvirtComputingResource went from 5.5k to 3.3k lines of code. The unit tests went from 300 to 5050 lines of code, covering 23.3% of the whole KVM plugin (instead of the previous 4%).
All the code extracted from the LibvirtComputingResource has 64.8% coverage.
This has already been rebased and built. I'm creating the PR now but will test it against a KVM hypervisor before it gets pushed towards the Master branch.
Cheers,
Wilder

Please, do not merge it before I test it with a KVM hypervisor.
Cheers,
Wilder

Yeah, I think this is a big enough change that we should give all KVM "stakeholders" an opportunity to review and test before merge. At least the committers, but anyone who wants to participate, of course.  Mike T, Wido, myself, Lucian, Andrija, just to try to get an array of use cases and storage types represented. Hopefully at least someone who uses OVS and vxlan.

Thanks, guys.
Would be really nice to test it ASAP so we avoid another rebase.
Most of the changes are purely structural, I tried my best to keep the behaviour as it is. If you look at the wrappers, they have the same content that the methods had when inside the LibvirtComputingResource. Are there changes in the behaviour? Yes, a few. For example, in few methods we had try/catch blocks with e.printStackTrace() and after that still returning true (as a successful operation). There were also methods returning always true and that return was being ignored. Those kind of things I changed.
Right now I'm busy getting my KVM environment test up and running. I will test the changes with OVS and VLAN and send a report to the list. My plan is to do it today/tomorrow
After the code reaches master, I will change the flyweight implementation in order to use the Annotations (suggested by @mlsorensen) and also increase UT coverage.
If you would have time to test it this week it would be very appreciated.
Cheers,
Wilder

looks nice :+1:

Looks good, I guess Travis is sort of green (with on failing test due to timeout). I will help test it tonight or tomorrow.

Thanks, @bhaisaab and @K0zka :)
Yep, "linbvirt" is a typo. :( When I apply the annotations (after the merge) I will rename it. :)
Cheers,
Wilder

Hi @mlsorensen @bhaisaab
Did you guys have time to test this PR?
My KVM environment is up and running and I just started testing. I will send a report soon.
Cheers,
Wilder

Okay.... but thanks a lot for start looking into it, @mlsorensen.
My env was a bit bumpy today and I did no manage to do all I had planned for, but I'm moving on with the tests just to identify performance improvements for our current test environment. It's taking about 10min to start a VM... so, running a test list test_accounts.py takes a very long time.
I will keep you up to date.
Cheers,
Wilder

Yeah, I got past tomcat and other systemd switchover issues on > 4.5 code, but then ran into other issues. The mgmt server configs got all messed up by the new RPMs. I got the server xml worked out but it couldn't connect to the db, 'access denied for cloud@localhost'. It doesn't seem to even be reading my db.properties. Mainly master teething issues around upgrade, I presume, but I've spent too much time on it today. I'll try a fresh install later.
One thing in particular I'd like to see tested is live migration. If I don't manage to test this please do at least this one.

@mlsorensen
"I got the server xml worked out but it couldn't connect to the db, 'access denied for cloud@localhost'"
I've seen this before as well when testing 4.5 upgrades. I'd check permissions on files cloudstack touches, for me it was permission related and it was not reading db.properties file like you mentioned.

@wilderrodrigues any update on this PR, if it's well tested, let's merge it?

@bhaisaab
I'm testing since yesterday. My KVM environment is a bit slow, but all the tests are going okay.
Please check the partial report below (last just wait a couple of hours then merge it):
Hi all,
I took some time to run integration/smoke tests on the LibVirt refactor I finished last week. All the test went well, as detailed in the report below.
More information about my environment and what was covered.
Environment:
Centos7 + KVM + Qemu
MySQL/MariaDB 5.5.41
Management Server running CentOS 7
Component/Smoke tests: https://github.com/apache/cloudstack (test/integration/component and smoke)
Storage type: NFS shared
Isolation method: VLAN
Test Create Account and user for that account ... === TestName: test_01_create_account | Status : SUCCESS ===
ok
Test Sub domain allowed to launch VM  when a Domain ... === TestName: test_01_add_vm_to_subdomain | Status : SUCCESS ===
ok
Test delete domain without force option ... === TestName: test_DeleteDomain | Status : SUCCESS ===
ok
Test delete domain with force option ... === TestName: test_forceDeleteDomain | Status : SUCCESS ===
ok
Test to verify Non Root admin previleges ... === TestName: test_01_non_root_admin_Privileges | Status : SUCCESS ===
ok
Test Remove one user from the account ... === TestName: test_01_user_remove_VM_running | Status : SUCCESS ===
ok
Test to verify service offerings at same level in hierarchy ... === TestName: test_01_service_offering_hierarchy | Status : SUCCESS ===
ok
Test to verify service offerings at same level in hierarchy ... === TestName: test_01_service_offering_siblings | Status : SUCCESS ===
ok
Test to verify template at same level in hierarchy ... === TestName: test_01_template_hierarchy | Status : SUCCESS ===
ok
Test update admin details ... === TestName: test_updateAdminDetails | Status : SUCCESS ===
ok
Test update domain admin details ... === TestName: test_updateDomainAdminDetails | Status : SUCCESS ===
ok
Test user update API ... === TestName: test_updateUserDetails | Status : SUCCESS ===
ok
Test login API with domain ... === TestName: test_LoginApiDomain | Status : SUCCESS ===
ok
Test if Login API does not return UUID's ... === TestName: test_LoginApiUuidResponse | Status : SUCCESS ===
ok

Ran 14 tests in 7517.316s
OK
/tmp//MarvinLogs/test_accounts_RWB0RK/results.txt
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... SKIP: At least two hosts should be present in the zone for migration
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok

Ran 10 tests in 2936.289s
OK (SKIP=1)
/tmp//MarvinLogs/test_vm_life_cycle_F2MIF6/results.txt
====Trying SSH Connection: Host:192.168.22.21 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 192.168.22.21 port : 22 SUCCESSFUL===
{Cmd: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.2.152 service dnsmasq status via Host: 192.168.22.21} {returns: [u'Checking DNS forwarder and DHCP server: dnsmasq(running).']}
====Trying SSH Connection: Host:192.168.22.21 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:192.168.22.21 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 192.168.22.21 port : 22 SUCCESSFUL===
===SSH to Host 192.168.22.21 port : 22 SUCCESSFUL===
{Cmd: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.2.152 service haproxy status via Host: 192.168.22.21} {returns: [u'haproxy is running.']}
{Cmd: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.2.152 service haproxy status via Host: 192.168.22.21} {returns: [u'haproxy is running.']}
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok

Ran 7 tests in 1995.909s
OK
/tmp//MarvinLogs/test_routers_QV22YU/results.txt

@bhaisaab @mlsorensen
I just sent the test report to your emails.
I'm fully confident with the changes and would like to suggest a merge of the PR.
Thanks in advance.
Cheers,
Wilder

Hi @bhaisaab ,
Is that okay if I merge the PR? It's been here for 7 days and has been tested.
Cheers,
Wilder

@wilderrodrigues thanks for sharing the test reports, while I've not personally tested. Looks good, please go ahead.

@wilderrodrigues build-master-jdk18 is failing in jenkins. http://jenkins.buildacloud.org/job/build-master-jdk18/38/
It fails on the below test
com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest.testGetHostStatsCommand

Test set: com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest
Tests run: 143, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.744 sec <<< FAILURE! - in com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest
testGetHostStatsCommand(com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest)  Time elapsed: 0.185 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.lang.NumberFormatException> but was<java.lang.NullPointerException>
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetHostStatsCommandWrapper.execute(LibvirtGetHostStatsCommandWrapper.java:77)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetHostStatsCommandWrapper.execute(LibvirtGetHostStatsCommandWrapper.java:34)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:182)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest.testGetHostStatsCommand(LibvirtComputingResourceTest.java:930)

@wilderrodrigues Sorry, I could not test the PR earlier. I see the following issues while trying to bring up my KVM setup:

LibvirtUtilitiesHelper NPE. This could be due to spring configuration.  LibvirtUtilitiesHelper  ws not injected.

2015-05-14 00:00:05,362 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) Execption while executing command com.cloud.agent.api.StartCommand
java.lang.NullPointerException
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:72)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:182)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1251)
at com.cloud.agent.Agent.processRequest(Agent.java:503)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
at com.cloud.utils.nio.Task.run(Task.java:84)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

executeRequest in LibvirtComputingResource is not logging any exception. It return Unsupported command answer. This makes it difficult to debug exceptions thrown during command execution,

Hi @kishankavala,
I will have a look at that, although that's pretty weird since I have tested on a KVM environment as well. Don't see why the bean would have been injected for me and not for you.
Have you done a full clean/install of the latest source?
Cheers,
Wilder

Hi @karuturi,
I had a look and it has more to do with the hard coded paths in the code (which were already present) and the setup of the machine where the Java8 builds run.
Good it failed, because that code has never been tested before.
I will make the test better and push a fix asap.
Thanks for the heads up.
Cheers,
Wilder

authors: K0zka bhaisaab karuturi kishankavala mlsorensen serverchief wilderrodrigues


text:
Cleanup deprecation issues

Changes based on my review of the main branch prior to release while writing additional documentation.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.82%. Comparing base (9f6b23b) to head (4e8e479).
Report is 189 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #272      +/-   ##
============================================
+ Coverage     91.90%   95.82%   +3.92%
- Complexity      575      657      +82
============================================
  Files            22       23       +1
  Lines          1247     1366     +119
  Branches        210      225      +15
============================================
+ Hits           1146     1309     +163
+ Misses           63       25      -38
+ Partials         38       32       -6

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Opened CLI-333 to address the Builder producer method naming issue.

source: gharchive/pull-request

authors: Claudenw codecov-commenter

repo: apache/commons-cli

license_type: permissive

license_source: bigquery


text:
IMAGING-355 - Large animated GIF takes too much heap memory in getMetadata

IMAGING-355 - Large animated GIF takes too much heap memory in getMetadata

Codecov Report

Merging #301 (a4983f3) into master (038666a) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master     #301   +/-   ##
=========================================
  Coverage     70.81%   70.81%
  Complexity     3405     3405
=========================================
  Files           333      333
  Lines         16828    16828
  Branches       2588     2588
=========================================
  Hits          11916    11916
  Misses         3897     3897
  Partials       1015     1015

Impacted Files
Coverage Δ

...he/commons/imaging/formats/gif/GifImageParser.java
77.07% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@garydgregory and @kinow I implemented the changes per your request.  However, I am not sure how much value passing stopReadinngBeforeImageData adds.  getImageInfo already has a unit test and I added a test for getMetadata.

Add stopReadingBeforeImageData to GifImageParameters
Add unit test for IMAGING-355

authors: andrewmkhoury codecov-commenter

repo: apache/commons-imaging


text:
Native File plugin not recognized on iOS 14

Bug Report
Problem
What is expected to happen?
The file plugin is installed and works as expected.
What does actually happen?
Whenever I try to run the app on my emulator or my device, the app gets stuck. When I check the safari developer. tools, the console logs a warning that the file plugin is not installed (but it is installed), and of course my app fails because I use that plugin. The plugin works correctly on iOS 13 and iOS 12, but it stopped working on iOS 14. The plugin also works correctly on Android.

Command or Code
Running the app in Xcode Emulator.
Environment, Platform, Device
iOS version 14. The plugin seems to be recognized correctly on iOS 13.
Version information
Ionic:

   Ionic CLI                     : 6.12.2 (/Users/marioacarlos/.nvm/versions/node/v12.16.2/lib/node_modules/@ionic/cli)
   Ionic Framework               : @ionic/angular 5.4.4
   @angular-devkit/build-angular : 0.901.12
   @angular-devkit/schematics    : 10.0.0
   @angular/cli                  : 9.1.12
   @ionic/angular-toolkit        : 2.3.3

Cordova:

   Cordova CLI       : 10.0.0
   Cordova Platforms : android 9.0.0, ios 6.1.1
   Cordova Plugins   : cordova-plugin-ionic-keyboard 2.2.0, cordova-plugin-ionic-webview 5.0.0, (and 19 other plugins)

Utility:

   cordova-res (update available: 0.15.2) : 0.14.0
   native-run (update available: 1.2.2)   : 1.0.0

System:

   Android SDK Tools : 26.1.1 (/Users/marioacarlos/Library/Android/sdk)
   ios-deploy        : 1.10.0
   ios-sim           : ios-sim/9.0.0 darwin-x64 node-v12.16.2
   NodeJS            : v12.16.2 (/Users/marioacarlos/.nvm/versions/node/v12.16.2/bin/node)
   npm               : 6.14.8
   OS                : macOS Big Sur
   Xcode             : Xcode 12.2 Build version 12B45b

Cordova Plugins List:
cordova-open-native-settings 1.5.2 "Native settings"
cordova-plugin-app-version 0.1.9 "AppVersion"
cordova-plugin-background-mode 0.7.3 "BackgroundMode"
cordova-plugin-device 2.0.3 "Device"
cordova-plugin-dialogs 2.0.2 "Notification"
cordova-plugin-extended-device-information 2.2.0 "ExtendedDeviceInformation"
cordova-plugin-file 6.0.2 "File"
cordova-plugin-globalization 1.11.0 "Globalization"
cordova-plugin-inappbrowser 4.0.0 "InAppBrowser"
cordova-plugin-ionic-keyboard 2.2.0 "cordova-plugin-ionic-keyboard"
cordova-plugin-ionic-webview 5.0.0 "cordova-plugin-ionic-webview"
cordova-plugin-media-capture 3.0.3 "Capture"
cordova-plugin-media 5.0.3 "Media"
cordova-plugin-nativeaudio 3.0.9 "Cordova Native Audio"
cordova-plugin-network-information 2.0.2 "Network Information"
cordova-plugin-speechrecognition 1.1.2 "Speech Recognition"
cordova-plugin-splashscreen 6.0.0 "Splashscreen"
cordova-plugin-statusbar 2.4.3 "StatusBar"
cordova-plugin-whitelist 1.3.4 "Whitelist"
cordova-sqlite-storage 5.1.0 "Cordova sqlite storage plugin - cordova-sqlite-storage plugin version"
cordova.plugins.diagnostic 6.0.2 "Diagnostic"
instabug-cordova 8.0.4 "instabug-cordova"
ionic-plugin-deeplinks 1.0.20 "Ionic Deeplink Plugin"

Checklist

[x] I searched for existing GitHub issues
[x] I updated all Cordova tooling to most recent version
[x] I included all the necessary information above

Sorry. This is not an issue with the plugin. We had a function that called the plugin before the device was ready.

source: gharchive/issue

repo: apache/cordova-plugin-file


text:
the request Json Object with the field '"new_edits": false' use in conjonction with the "/db/_bulk_docs" seems to not work

Description
I want to update docs massively whithout specify _rev informations in the bulk doc and generate conflicts
to do this i use  the field '"new_edits": false' in the request Json Object use in conjonction with the "/db/_bulk_docs", but it doesnt work.
in the documentation of the handler "/db/_bulk_docs" we can read this
Request JSON Object:
new_edits (boolean) – If false, prevents the database from assigning them new revision IDs. Default is true. Optional
Steps to Reproduce
here is what I do :
i have 3 docs in my db :
{ "_id": "pk5896",  "_rev": "1-946f6e944df9b086f03fb3e45935ab3b", "produit": "pomme", "type": "fruit"}
{ "_id": "ls5875",  "_rev": "1-4183ce21a4c67d05cb9a291b36710f7f", "produit": "banane", "type": "fruit"}
{ "_id": "it8765",  "_rev": "9-6058b0806c4b0e7d53058f22dd4b8c10", "produit": "salade", "type": "légume"}
I want to update this docs in bulk mode whitout _rev specified to generate conflicts.
in the documentation it is specified to position the field "new_edits": false in the json bulk doc like this :
{
"new_edits": false,
"docs": [
{ "_id": "pk5896", "produit": "pomme", "type": "fruit"},
{ "_id": "ls5875", "produit": "banane", "fonction": "fruit"},
{ "_id": "it8765", "produit": "salade", "fonction": "légume"},
]
}
but when i try to POST the document, couchdb return to me an error :
curl -sSX POST -H "Content-Type: application/json" 'http://user:mdp@serveur.domain:5984/db/_bulk_docs' -d @fic.json
{
"error": "unknown_error",
"reason": "function_clause",
"ref": 2857759975
}
if i change the value of new_edits : false by true, couchdb respond normaly :
[
{
"id": "pk5896"
"error": "conflict",
"reason": "Document update conflict."
},
{
"id": "ls5875",
"error": "conflict",
"reason": "Document update conflict."
},
{
"id": "it8765",
"error": "conflict",
"reason": "Document update conflict."
}
]
Expected Behaviour / question
is this option is always supported or i am doing a mistake  (syntax, comprehension of the focntion, other reason) ?
does it work only on _local documents ?
I have find this too in the documentation :
http://docs.couchdb.org/en/latest/whatsnew/2.3.html?highlight=new_edits
#1683: Attempting to create a local document with an invalid revision no longer throws a badarg exception.
Also, when setting new_edits to false and performing a bulk write operation, local documents are no longer written into the wrong btree.
Finally, it is no longer possible to create a document with an empty ID during a bulk operation with new_edits set to false.
Environment

CouchDB Version used: 2.3
Operating System and version: RHEL 7

Thanks for you'r reply.

When new_edits=false is used, you should always include _revisions key into the document. This is not correctly documented. You can see format of this field here for example:
https://docs.couchdb.org/en/stable/api/document/common.html?highlight=_revisions#getting-a-list-of-revisions

Kindly open a new issue at https://github.com/apache/couchdb-documentation for the docs bug? Thanks!

thanks both for the reply.
so, i will close this issue and open a new for the documentation bug.
before closing this issue can you tell me if there is a similar method like this one described (bulk mode) for generating massive doc conflicts ?

No, the _bulk_docs endpoint is the only bulk endpoint useful for this in CouchDB of which I know.

See  https://github.com/apache/couchdb-documentation/issues/390 .

authors: olivenvrac ondra-novak wohali

repo: apache/couchdb

license_source: github-api


text:
feat(mango): add allow_fallback for user-specified indexes on _find

This is an alternative to #4710 based on the results of the related discussion.
It is not always beneficial for the performance if the Mango query planner tries to assign an index to the selector.  User-specified indexes may save the day, but since they are only hints for the planner, fallbacks may still happen.
Introduce the allow_fallback flag which can be used to tell if falling back to other indexes is acceptable when an index is explicitly specified by the user.  When set to false, give up on planning and return an HTTP 400 response right away.  This way the user has the chance to learn about the requested but missing index, optionally create it and try again.
By default, fallbacks are allowed to maintain backwards compatibility.  It is possible to set allow_fallback to true but currently it coincides with the default behavior hence becomes a no-op in practice.
Fixes #4511
Testing recommendations
The changes could be verified by the following commands:
make eunit apps=mango
make mango-test

This is a non-breaking change, so there should be no observable difference in the behavior for the users.  When allow_fallback is set to false and use_index is set to some index, it may return HTTP 400 depending on the suitability of the given index.
Checklist

[x] Code is written and works correctly
[x] Changes are covered by tests
[x] Documentation changes were made in the src/docs folder

Yes, I agree.  @rnewson, @willholley, @ricellis please comment if you have the time.

I was under the impression that allow_fallback was going to apply to all _find requests not just those with use_index. I know @willholley disagreed with that overloading, but I thought that was only related to the cluster toggles. I didn't think there was an objection to disallowing all_docs fallback as long as applied consistently.

I do not want to respond on behalf of @willholley but in my understanding, in his respective comment he opposed to extending the allow_fallback mechanism to _all_docs: "I'm less keen on overloading it to removing the use of the _all_docs index".  He also included a historical reference to suggest that it would not give us much: "Mango did originally prevent the _all_docs fallback unless the query contained a clause that would require it. [..] lots of users just blindly added a { _id: { "$gt": 0 } } clause to [..] work around the restriction."

if allow_fallback: false will allow fallback, then it will not be merged.

@rnewson do you mean that we shall extend this change to disabling the _all_docs fallback as well?

I don't think this issue / PR has anything to do with query efficiency - only fixing the unclear semantics of use_index.
I'm open to a discussion about if we want to / how to prevent use of inefficient indexes or, but that's much broader problem space in which I don't see _all_docs as any different to user-defined indexes.

Hi,
_find was briefly discussed in the last couchdb meeting.
I've personally gone off the idea of yet more flags inside the /dbname/_find request body that intend to coerce it to make less bad decisions, I would not like to see it happen without a new developer vote.
I see a fork in the road for /dbname/_find. It takes one of two forks as I see it;

CouchDB's indexing will get richer, or mango will get smarter (capable of using multiple indexes for a query, using information from one to more efficiently skip through another). In other words, to become what it looks like it is in on the outside, but is not on the inside.

It continues to be just a naive selector between existing indexes, albeit indexes with the very significant benefit of not requiring a javascript callout.

If we take the first path, then neither the new allow_fallback parameter or the existing use_index parameter would be necessary, we'd gladly deprecate them.
If we take the second path, then we're making a bad idea worse.
I would prefer to add _find at the same url places where index-specific queries were performed prior to Mango. i.e, /dbname/_design/ddocname/_find/viewname and /dbname/_design/ddocname/_search/indexname. Requests to those endpoints will take a json request body, treat it as a selector, and either reject with a 400 Bad Request if the index is unsuitable, or proceed with the query against the index in the URL. use_index in that request body would generate a 400 Bad Request.

I was under the impression that allow_fallback is a reasonable compromise to treat these problems, especially it was the one you proposed and there seem to be an agreement around it.  That said, I do not agree that it would make things worse.  I really like that it goes well with the previously established principles of the top-level _find endpoint.
I believe that _find shall always be declarative in nature and as such, making it work like _search makes less sense.  If users need something that Search can do, they can always choose to use that instead.
Now we have a tool at hand at least that can save users from running badly planned queries, which could be useful even after the enhancements you suggested in the first point above have been implemented.  We could always deprecate it later, once we have a better view at which one of the branches has been eventually taken or it is deemed to be unnecessary.

I liked @rnewson's idea of a very simple/predictable view that uses the selector syntax instead of javascript so tried this: https://github.com/apache/couchdb/pull/5310. It's even simpler than proposed, as it just uses the _design/.../_view/... pattern that we know (and love?).
As for this particular PR, I think it's fine. We already got _find with it's good/bad parts, so adding more control over index selection seem reasonable.
As for the future of _find, it's hard to say. Will we get a super-duper query planner like Postgres? Probably not any time soon. But I could see adding incremental improvements such as being able to merge two or more indexes to get the $or working properly (we already merge index results in fabric anyway!), so wouldn't like to preemptively neuter _find just yet.
Anyway just my $0.02.

@pgj had a chat just now to go over our differences. I'm happy to see allow_fallback:true|false merged, and will do when CI finishes successfully.
I do want us to follow up with either the /db/_design/foo/_find/bar or /db/_design/foo/_view/bar that takes a selector, as I think those are easier to advocate for, and to see in logs and client libraries, to help users migrate to a safer place where they can't accidentally fall into seriously suboptimal query performance.
I also think allow_fallback should default to false in a future CouchDB major release.

Thank everyone who participated in the discussion.

authors: nickva pgj ricellis rnewson willholley


text:
Temporary exclude modules which still depend on 'javax.*' by displacing them to separate 'javax' profile

Temporary exclude modules which still depend on 'javax.*' by displacing them to separate 'javax' profile

@jimma could you please check it out? removing (temporary) modules which do not comply with jakarta.* and fail compilation right now, thanks!

@jimma could you please check it out? removing (temporary) modules which do not comply with jakarta.* and fail compilation right now, thanks!

@reta  This looks good to me.  Thanks for working this out.

authors: jimma reta

repo: apache/cxf


text:
suggest add synax select from generate_series()

Is your feature request related to a problem or challenge?
it is complex and slowly to use unnest().
Describe the solution you'd like
it can run select from generate_series() as duckdb does.
Describe alternatives you've considered
No response
Additional context
D:\duckdb>datafusion-cli
DataFusion CLI v37.0.0

> select sum(a) from unnest(generate_series(1,1000000)) as t(a);
+--------------+
| SUM(t.a)     |
+--------------+
| 500000500000 |
+--------------+
1 row(s) fetched.
Elapsed 3.648 seconds.
> select sum(a) from generate_series(1,1000000) as t(a);
Error during planning: table function 'generate_series' not found
>
D:\duckdb>duckdb011
v0.10.1 4a89d97db8
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.

D .timer on
D select sum(a) from unnest(generate_series(1,1000000)) as t(a);
┌──────────────┐
│    sum(a)    │
│    int128    │
├──────────────┤
│ 500000500000 │
└──────────────┘
Run Time (s): real 0.854 user 0.859375 sys 0.078125
D select sum(a) from generate_series(1,1000000) as t(a);
┌──────────────┐
│    sum(a)    │
│    int128    │
├──────────────┤
│ 500000500000 │
└──────────────┘
Run Time (s): real 0.033 user 0.015625 sys 0.000000

I think we could add this as a user defined table function in datafusion-cli quite easily
For example we could follow the model of parquet_metadata (which also follows duckdb):
SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size
FROM parquet_metadata('hits.parquet')
WHERE path_in_schema = '"WatchID"'
LIMIT 3;

+----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
| path_in_schema | row_group_id | row_group_num_rows | stats_min           | stats_max           | total_compressed_size |
+----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
| "WatchID"      | 0            | 450560             | 4611687214012840539 | 9223369186199968220 | 3883759               |
| "WatchID"      | 1            | 612174             | 4611689135232456464 | 9223371478009085789 | 5176803               |
| "WatchID"      | 2            | 344064             | 4611692774829951781 | 9223363791697310021 | 3031680               |
+----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
3 rows in set. Query took 0.053 seconds.

It is documented here https://arrow.apache.org/datafusion/user-guide/cli.html
The code for it is here; https://github.com/apache/arrow-datafusion/blob/637293580db0634a4efbd3f52e4700992ee3080d/datafusion-cli/src/functions.rs#L215-L442

related https://github.com/apache/arrow-datafusion/issues/9354

I think switching generate_series to a UDTF will break the API. I think in most cases it's an acceptable API change. However there is (at least for me) 1 particular use case that of the array generated from generate_series that will probably need closer attention.

Example
I need to spread the data equally over equal intervals between 2 data points. I use `generate_series` to generate an array per row. The way that I do this now is as follows:
create table manual(time timestamp, index_value DOUBLE);
insert into
  manual (time, index_value)
values
  ('2022-01-31 23:00:00.0000000', 935035.000),
  ('2022-02-28 23:00:00.0000000', 954469.000),
  ('2022-03-31 22:00:00.0000000', 905204.000),
  ('2022-04-30 22:00:00.0000000', 971415.000);

with
    index_values AS (
    select
        time AS to_time,
        lag(time, 1) over (order by time) AS from_time,
        index_value AS to_index,
        lag(index_value, 1) over (order by time) AS from_index
    from manual
),
    series AS (
        select
            unnest(generate_series(from_time, to_time, interval '1 month')) AS time,
            (case
                when to_index < from_index THEN (999999 - from_index) + to_index
                else to_index - from_index end)
                / coalesce(nullif(array_length(generate_series(from_time, to_time, interval '1 month')), 0), 1) AS delta_value
        from index_values
    ) SELECT * FROM series;

Specifically array_length(generate_series(from_time, to_time, interval '1 month')) will need to be done differently if generate_series becomes a UDTF. It will probably need to become COUNT(time) OVER (partition by to_time, from_time) :
with
    index_values AS (
    select
        time AS to_time,
        lag(time, 1) over (order by time) AS from_time,
        index_value AS to_index,
        lag(index_value, 1) over (order by time) AS from_index
    from manual
),
    series AS (
        select
            time,
            (case
                when to_index < from_index THEN (999999 - from_index) + to_index
                else to_index - from_index end)
                / coalesce(nullif(count(time) over (partition by to_time, from_time), 0), 1) AS delta_value
        from index_values, generate_series(from_time, to_time, interval '900 seconds') AS time
    ) SELECT * FROM series;

(If you have other/better suggestions for how data can be spreaded/interpolated in raw DF, all ears :) )

TL;DR: If you're using generate_series to generate a series on a row-based basis, you may need to consider OVER PARTITION BY (your_row_composite_key) if this API change takes place
Just mentioning this here in case anyone is using generate_series in a similar way as I do

FWIW I think we should consider keeping the existing generate_series function and add a new user defined table function with the same name

That might work, but I wonder how it will be inferred and whether it will cause any conflicts. Some examples that can be ambiguous on top of my head:
select generate_series(1, 100), * from generate_series(10, 100)

This can be either (postgresql behavior):

generate_series
generate_series

1
10

2
10

3
10

1
11

2
11

3
11

1
12

2
12

3
12

1
13

2
13

3
13

or
+------------------------------------+----------------------------------------------+
| generate_series(Int64(1),Int64(3)) | UNNEST(generate_series(Int64(10),Int64(13))) |
+------------------------------------+----------------------------------------------+
| [1, 2, 3]                          | 10                                           |
| [1, 2, 3]                          | 11                                           |
| [1, 2, 3]                          | 12                                           |
| [1, 2, 3]                          | 13                                           |
+------------------------------------+----------------------------------------------+
4 row(s) fetched.
Elapsed 0.001 seconds.

Also, I wonder how the results of something like this would look like 🤔:
select generate_series(1, 3), array_length(generate_series(10, 11))

That might work, but I wonder how it will be inferred and whether it will cause any conflicts. Some examples that can be ambiguous on top of my head:
select generate_series(1,3), * from generate_series(10, 13)

This can be either (postgresql behavior):
generate_series	generate_series
1	10
2	10
3	10
1	11
2	11
3	11
1	12
2	12
3	12
1	13
2	13
3	13
or
+------------------------------------+----------------------------------------------+
| generate_series(Int64(1),Int64(3)) | UNNEST(generate_series(Int64(10),Int64(13))) |
+------------------------------------+----------------------------------------------+
| [1, 2, 3]                          | 10                                           |
| [1, 2, 3]                          | 11                                           |
| [1, 2, 3]                          | 12                                           |
| [1, 2, 3]                          | 13                                           |
+------------------------------------+----------------------------------------------+
4 row(s) fetched.
Elapsed 0.001 seconds.

Also, I wonder how the results of something like this would look like 🤔:
select generate_series(1, 3), array_length(generate_series(1, 3))

Maybe we can use the 2nd interpretation (the same as DuckDB), currently, DataFusion's generate_series in expression is already different from Postgres's semantics

I think we could add this as a user defined table function in datafusion-cli quite easily

@alamb Is it possible to implement this UDTF inside datafusion crate? 🤔  I find this table function can make end-to-end tests for spilling related queries much easier

I think we could add this as a user defined table function in datafusion-cli quite easily

@alamb Is it possible to implement this UDTF inside datafusion crate? 🤔 I find this table function can make end-to-end tests for spilling related queries much easier

Makes sense to me -- we may need to create a datafusion-functions-table sub crate too, however 🤔 as a place to put it

repo: apache/datafusion


text:
Fix parquet statistics for ListingTable and Utf8View with schema_force_string_view, rename config option to schema_force_view_types

Which issue does this PR close?
Closes #12123
Rationale for this change
On write: parquet file written with utf8/large-utf & binary/large-binary schema (so is in metadata).
On read: we would like to be able to read as the more performant view types.
Previous work has already used the schema_force_string_view to read into utf8view and binaryview, by passing around a bool to the ParquetOpener.
This work is to focused on getting the parquet statistics, on read, to properly compute when reading as view types.
What changes are included in this PR?

move the schema_force_string_view up a few lines, to be with the "read" (not write) config options.
remove the passing around of bools

this was done by merging table_schema (with views) and file_schema (without views)

add tests which run with true|false for schema_force_string_view

~There are two tests marked as incomplete: an expected panic and a commented out test. Both tests will successfully pass once the next (anticipated) arrow release occurs. In order to prove this, there is a PoC over here which branches of the in-progress WIP for the arrow upgrade and then adds the commits from this PR.~. I merged in the latest main with the arrow-rs release (thank you!) and those tests are now passing.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.

Thanks again @wiedld

authors: alamb wiedld


text:
feat: add support for Substrait ExtendedExpression

Which issue does this PR close?
This partially addresses #7687 (though the issue talks about a python component which is not yet included here)
Rationale for this change
Described in issue.
What changes are included in this PR?
Adds serialization to and from Substrait's top-level ExtendedExpression message
Are these changes tested?
Yes, new unit tests are added.
Are there any user-facing changes?
New methods.  No changes to existing behavior.

@vbarua  / @Blizzara  do either of you have time to review this PR?

Just did another round, LGTM!
As a aside, I feel like we should look into splitting the consumer.rs and producer.rs into couple files, as they're growing quite large...

Thanks @Blizzara  🙏
Let's do the other suggestions as a follow on PR (I like the idea of splitting consumer/producer)
Thanks again @westonpace

As a aside, I feel like we should look into splitting the consumer.rs and producer.rs into couple files, as they're growing quite large...

@Blizzara agreed.  One simple split for now could be to split the expressions/literals/types/schema from the relations.  At some distant point (not in my current plans) I could imagine the crate itself splitting to create a lightweight expression manipulation library that depends only on datafusion-expr.

I have found the mechanical "split the code into different files" is always a good first step towards anything more substantial

authors: alamb westonpace


text:
doc: correct desc in README.md

Brief Summary of The Change

correct the docs in readme
Pull Request checklist
I confirm that the following checklist has been completed.

[x] Add/Change test cases for the changes.
[x] Add/Change the related documentation, should also change docs/source/config.rst when you change file default_config.yaml.
[ ] (Optional) Add your change to UPDATING.md when it is an incompatible change.

Codecov Report

Merging #105 (ab0af26) into main (a2fe58b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #105   +/-   ##
=======================================
  Coverage   90.75%   90.75%
=======================================
  Files          63       63
  Lines        2314     2314
=======================================
  Hits         2100     2100
  Misses        214      214

Flag
Coverage Δ

unittests
90.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter zhongjiajie

repo: apache/dolphinscheduler-sdk-python


text:
[DSIP-27] Deprecate write java code in JAVA task

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Motivation
Right now, we support two kind of usage of JAVA tasks,

The first type is JAVA, user can write java code, once  the worker receive this kind of task, it will compile the java code to class, and then execute the class file.
The second type is JAR, user can submit a fat jar, once the worker receive this kind of task, it will directly execute the fat jar.
This issue is want to remove the first type of JAVA task, since this is more like a demo, no one will submit java code, and in most of the time, the java application is not written within one file.
So we only need to support submit Jar.
Design Detail
Further more, we would better support user submit jars rather than a fat jar. The command may look like below, this should be more usefully
java $JAVA_OPTS \
  -Dxx.yy.args=1 \
  -cp "xx.jar":"xx.conf" \
  xx.Application args

So we need to change the current task mode into: Fat JAR, NORMAL JAR
Compatibility, Deprecation, and Migration Plan
This is not compatible with the current version.
Test Plan
Add unit test.
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

LGTM

The DSIP has been added to OSPP, no further development needs to be specified.

How about this

Remove the selection boxes for Run Type, and add dependency package parameters. If this parameter is empty, it means fat jar, and if it is not empty, it means normal jar.
Add Main class parameters.

The DSIP has been added to OSPP, no further development needs to be specified.

Why can this be added to OSPP so casually? OSPP has a very long end time, which will block releases.

The DSIP has been added to OSPP, no further development needs to be specified.

Why can this be added to OSPP so casually? OSPP has a very long end time, which will block releases.

I apologize for not explaining in advance. Initially, I thought it wasn't a highly important feature, just a few versions of delay.

LGTM, In addition, We should be cautious about adding new tasks in the future, and try to receive enough votes and requests to add new tasks.

Please assign to me

I am a student selected in this OSPP, and here is part of my implementation plan.
1. Delete the Java type
If you want to add a new task type, you need to first delete the related code of the Java type from the previous task types.

For example, this piece of code first determines whether it is a Java type. You can use this as a starting point to delete the JAVA type and leave a mark. Later, you can add the Normal Jar type.
By the same token, in all related places, first delete the code of the Java type task, including calls, uses, etc. By the way, record the position. After deletion is complete and other functions are normal, add the Normal jar type at these recorded places.
2. Partial implementation of the Normal Jar type
First, we need to add a new constant in the JavaConstants class to represent the NORMAL JAR task type:
// Original code
public class JavaConstants {
    public static final String RUN_TYPE_JAVA = "JAVA";
    public static final String RUN_TYPE_JAR = "JAR";
}

// New code
public class JavaConstants {
    public static final String RUN_TYPE_JAVA = "JAVA";
    public static final String RUN_TYPE_JAR = "JAR";
    public static final String RUN_TYPE_NORMAL_JAR = "NORMAL_JAR";  // New constant
}

Then, update the checkParameters method in the JavaParameters class to accept the new run type:
// Original code
public class JavaParameters extends AbstractParameters {
    // ...
    public boolean checkParameters() {
        return runType != null && (runType.equals(JavaConstants.RUN_TYPE_JAVA) || runType.equals(JavaConstants.RUN_TYPE_JAR));
    }
}

// New code
public class JavaParameters extends AbstractParameters {
    // ...
    public boolean checkParameters() {
        return runType != null && (runType.equals(JavaConstants.RUN_TYPE_JAVA) || runType.equals(JavaConstants.RUN_TYPE_JAR) || runType.equals(JavaConstants.RUN_TYPE_NORMAL_JAR));  // Updated condition
    }
}

Update the handle method in the JavaTask class to handle the new run type. This may involve adding a new method, such as buildNormalJarCommand, to build the command to execute the NORMAL JAR task:
// Original code
public class JavaTask extends AbstractTask {
    // Other code
    @Override
    public void handle(TaskCallBack taskCallBack) throws TaskException {
        // Other code
        switch (javaParameters.getRunType()) {
            case JavaConstants.RUN_TYPE_JAVA:
                command = buildJavaCommand();
                break;
            case JavaConstants.RUN_TYPE_JAR:
                command = buildJarCommand();
                break;
            default:
                throw new RunTypeNotFoundException("run type is required, but it is null now.");
        }
        // Other code
    }
}

// New code
public class JavaTask extends AbstractTask {
    // Other code
    @Override
    public void handle(TaskCallBack taskCallBack) throws TaskException {
        // ...
        switch (javaParameters.getRunType()) {
            case JavaConstants.RUN_TYPE_JAVA:
                command = buildJavaCommand();
                break;
            case JavaConstants.RUN_TYPE_JAR:
                command = buildJarCommand();
                break;
            case JavaConstants.RUN_TYPE_NORMAL_JAR:  // New case
                command = buildNormalJarCommand();  // New method
                break;
            default:
                throw new RunTypeNotFoundException("run type is required, but it is null now.");
        }
        // ...
    }

    // New method
   protected String buildNormalJarCommand() {
    // Get the absolute path of the JAR file
    ResourceContext resourceContext = taskRequest.getResourceContext();
    String mainJarAbsolutePathInLocal = resourceContext
            .getResourceItem(javaParameters.getMainJar().getResourceName())
            .getResourceAbsolutePathInLocal();

    // Build the command
    StringBuilder builder = new StringBuilder();
    builder.append(getJavaCommandPath())
           .append("java").append(" ")
           .append(buildResourcePath()).append(" ")
           .append("-cp").append(" ")
           .append(taskRequest.getExecutePath()).append(FOLDER_SEPARATOR)
           .append(mainJarAbsolutePathInLocal).append(" ")
           .append(javaParameters.getMainClass()).append(" ")  // Main class name
           .append(javaParameters.getMainArgs().trim()).append(" ")  // Main class parameters
           .append(javaParameters.getJvmArgs().trim());  // JVM parameters

    return builder.toString();
} }

This method first gets the absolute path of the JAR file, then builds a command to execute the NORMAL JAR task. This command includes the path of the Java command, the class path, the path of the JAR file, the main class name, the main class parameters, and the JVM parameters.

@ruanwenjun

authors: ailiujiarui pinkfloyds ruanwenjun xinxingi zhuxt2015

repo: apache/dolphinscheduler


text:
[Fix] fix deserialization error for QueryPlan exception

Proposed changes
Issue Number: close #65
Problem Summary:
Deserialization error on QueryPlan with exceptions.
Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No Need)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Thank you for your pr, is it also possible to add @JsonIgnoreProperties(ignoreUnknown = true) directly to the entity class of QueryPlan ?

Yes. Do you mean adding the annotation alone or also with the exception field? I think the exception msg is helpful and should be exposed to users properly.

However, the response of queryPlan has been judged in RestService.getQueryPlan, and an exception is also thrown when an error occurs, so is there no need to record the exception in queryPlan?

repo: apache/doris-flink-connector


text:
fix map literal to double in FilterSelectivityCalculator

Proposed changes

fix literal to double bug: all literal type implements getDouble() function
fix exploration rule execution order bug: rules are executed in reversed order
fix regression test nereids_syntax_p0/view bug: result is not fully ordered
Issue Number: close #xxx

Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.66 seconds
load time: 556 seconds
storage size: 17154655417 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221028163542_clickbench_pr_35520.html

Literal.getDouble() should supply a default value without throw exception.
public double getDouble() {
    try {
        return Double.parseDouble(getValue().toString());
    } catch (Throwable t) {
        return Double.NaN;
    }
}

parseDouble

Good idea.

Literal.getDouble() should supply a default value without throw exception.
for example:
public double getDouble() {
    try {
        return Double.parseDouble(getValue().toString());
    } catch (Throwable t) {
        return Double.NaN;
    }
}

org.apache.commons.lang3.math.NumberUtils is also alternatives
public double getDouble() {
    return NumberUtils.toDouble(getValue().toString(), Double.NaN);
}

Good idea! thx

authors: 924060929 englefly hello-stephen

repo: apache/doris


text:
fix: fix some bugs in DpHyper

Proposed changes

make edge according to connectivity
pruning columns and make complex projects above join

Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.99 seconds
load time: 495 seconds
storage size: 17122508972 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230131090927_clickbench_pr_87717.html

authors: XieJiann hello-stephen


text:
Enhencement enhencement regression test of java udf

Proposed changes
when run regression test of java udf many times.
if failed in some reason, the next time will meet error like: function already exist
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run build all

./run buildall


text:
chore Add config item 'exit_on_exception'

Proposed changes
Add one configuration item: 'exit_on_exception'.
If  'exit_on_exception' is set to true, the process of the BE will crash instead of throwing an exception.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall


text:
fix fix unset collected rows

Proposed changes
As title.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


text:
fix fix test_query_sys_tables schema_privileges regression case.

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run p1


text:
bugfixFix some spelling mistakes in english doc

Proposed changes
Issue Number: close #xxx
Fix spelling mistake in the English docs
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
DRILL-2653: Improve web UI experience when there is an error in a sto…

…rage plugin configuration

@sudheeshkatkam @jacques-n Can you please review?

@sudheeshkatkam Just saw that you actually had a patch for this, I was looking at DRILL-2663 originally, and found the duplicate DRILL-2653 when I went to write the commit message. I think I'll merge this version because it is a little clearer for users in the case of repeated failures, but I didn't mean to ignore your patch.

That's fine, Jason :)

@sudheeshkatkam Message fixed, also added more colors :)

+1

authors: jaltekruse sudheeshkatkam

repo: apache/drill


text:
DRILL-4862: binary_string should use another buffer as out buffer to …

…be used in more generic usage.

+1

+1

authors: chunhui-shi gparai sudheeshkatkam


text:
Validate datasource retention rules: Reject rules that fully contain subsequent rules' interval

Druid's retention rules are evaluated in the order in which they are set. Unfortunately, a mistake in the order of rules by an operator can lead to unintentional data loss, and if it goes unnoticed and auto-kill is enabled, data could be permanently deleted.
To address this issue, a guard rail is added in the rules API. This guard rail aims to identify rules that fully contains the interval for any subsequent rules A rule is considered good as long as it will run at some point (and doesn't fully hide other rules in the chain) - it's ensured by checking at the interval boundaries as well.
Some examples of bad rules that will be disallowed (see unit tests for more examples):

[loadByPeriod(P6M), loadByPeriod(P3M)]
[loadByInterval(2020-01-01/2050-01-01), loadByInterval(2021-01-01/2023-01-01)]
[dropBeforeByPeriod(P3M), dropBeforeByPeriod(P6M)]
[loadForever, loadByPeriod(P1M)]
[loadForever, dropForever]

Examples of rules that will be allowed because they will be evaluated at some point:

[loadByPeriod(P5Y), loadByInterval(2021-01-01/2023-01-01)]; the loadByInterval rule will be evaluated at some point.
[loadByPeriod(P1Y), loadByPeriod(P2Y)]

For testing purposes, if a user wants to intentionally set a rule that fully contains other rules, they can do it temporarily and then fetch the correct set of rules from the rules audit history after their testing is done.

Release note
Datasource rules API will reject the rules payload if a rule fully contains subsequent rules' interval.

Key changed/added classes in this PR

Rule.java
RulesResource.java
RulesResourceTest.java

This PR has:

[x] been self-reviewed.
[x] a release note entry in the PR description.
[x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[x] been tested in a test Druid cluster.

@abhishekrb19 , it seems like a nice idea to have these checks while updating retention rules.
If there is a cluster that already defines rules that would fail the check, would that cluster break after this change?

Thanks for the review, @kfaraz. Re:

If there is a cluster that already defines rules that would fail the check, would that cluster break after this change?

No, this is an API-only validation change, so any existing rules in a cluster will remain as-is.

To allow for the introduction of new rules and cases where the user does want to add these absurd rules that have been supported till now, WDYT of having a URL parameter ?forced which bypasses these validation checks. The new check would be the default, however, if someone wants to override it, they can set the flag and bypass them.

@LakshSingla, re:

where the user does want to add these absurd rules that have been supported till now, WDYT of having a URL parameter ?forced which bypasses these validation checks.

hmm, a parameter to bypass API validation seems a bit hacky to me. If users want to "temporarily" set something for testing or so, they can set effective rules and then always retrieve the "actual" rules from the audit history from the console/API. I think that should suffice?

authors: LakshSingla abhishekrb19 kfaraz

repo: apache/druid


text:
Migration icbc

What is the purpose of the change

跨注册中心&注册中心内迁移
应用维度下multiple
元数据高可用

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #6773 (e05844b) into master (a568518) will increase coverage by 0.67%.
The diff coverage is 31.43%.

@@             Coverage Diff              @@
##             master    #6773      +/-   ##
============================================
+ Coverage     58.88%   59.55%   +0.67%
+ Complexity      508      286     -222
============================================
  Files          1026     1001      -25
  Lines         41383    39957    -1426
  Branches       6028     5921     -107
============================================
- Hits          24369    23797     -572
+ Misses        14257    13443     -814
+ Partials       2757     2717      -40

Impacted Files
Coverage Δ
Complexity Δ

...dubbo/rpc/cluster/directory/AbstractDirectory.java
71.87% <0.00%> (-24.90%)
0.00 <0.00> (ø)

...ster/support/registry/ZoneAwareClusterInvoker.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...he/dubbo/common/config/CompositeConfiguration.java
65.21% <0.00%> (-4.56%)
0.00 <0.00> (ø)

...apache/dubbo/common/config/ConfigurationUtils.java
59.37% <0.00%> (+0.75%)
0.00 <0.00> (ø)

.../apache/dubbo/config/bootstrap/DubboBootstrap.java
41.69% <0.00%> (-1.00%)
0.00 <0.00> (ø)

...pport/zookeeper/ZookeeperDynamicConfiguration.java
62.06% <0.00%> (ø)
0.00 <0.00> (ø)

...n/java/org/apache/dubbo/metadata/MetadataInfo.java
35.32% <ø> (ø)
0.00 <0.00> (ø)

...adata/store/zookeeper/ZookeeperMetadataReport.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

.../dubbo/registry/client/DefaultServiceInstance.java
57.14% <0.00%> (-5.72%)
0.00 <0.00> (ø)

...ache/dubbo/registry/client/InstanceAddressURL.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

... and 246 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a568518...e05844b. Read the comment docs.

Codecov Report

Merging #6773 (e05844b) into master (a568518) will increase coverage by 0.67%.
The diff coverage is 31.43%.

@@             Coverage Diff              @@
##             master    #6773      +/-   ##
============================================
+ Coverage     58.88%   59.55%   +0.67%
+ Complexity      508      286     -222
============================================
  Files          1026     1001      -25
  Lines         41383    39957    -1426
  Branches       6028     5921     -107
============================================
- Hits          24369    23797     -572
+ Misses        14257    13443     -814
+ Partials       2757     2717      -40

Impacted Files
Coverage Δ
Complexity Δ

...dubbo/rpc/cluster/directory/AbstractDirectory.java
71.87% <0.00%> (-24.90%)
0.00 <0.00> (ø)

...ster/support/registry/ZoneAwareClusterInvoker.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...he/dubbo/common/config/CompositeConfiguration.java
65.21% <0.00%> (-4.56%)
0.00 <0.00> (ø)

...apache/dubbo/common/config/ConfigurationUtils.java
59.37% <0.00%> (+0.75%)
0.00 <0.00> (ø)

.../apache/dubbo/config/bootstrap/DubboBootstrap.java
41.69% <0.00%> (-1.00%)
0.00 <0.00> (ø)

...pport/zookeeper/ZookeeperDynamicConfiguration.java
62.06% <0.00%> (ø)
0.00 <0.00> (ø)

...n/java/org/apache/dubbo/metadata/MetadataInfo.java
35.32% <ø> (ø)
0.00 <0.00> (ø)

...adata/store/zookeeper/ZookeeperMetadataReport.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

.../dubbo/registry/client/DefaultServiceInstance.java
57.14% <0.00%> (-5.72%)
0.00 <0.00> (ø)

...ache/dubbo/registry/client/InstanceAddressURL.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

... and 246 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a568518...e05844b. Read the comment docs.

authors: codecov-io quhongwei

repo: apache/dubbo


text:
FINERACT-706 - Mojaloop fineract 1.0

Description
Describe the changes made and why they were made. Ignore if these details are present on the associated Jira ticket
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[x] Commit message starts with the issue number from https://issues.apache.org/jira/projects/FINERACT/. Ex: FINERACT-646 Pockets API.

[x] Coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions have been followed.

[ ] API documentation at https://github.com/apache/fineract/blob/develop/api-docs/apiLive.htm has been updated with details of any API changes.

[ ] Integration tests have been created/updated for verifying the changes made.

[x] All Integrations tests are passing with the new commits.

[x] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the list.)

Our guidelines for code reviews is at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide

@conradsp : The pr at https://github.com/apache/fineract/pull/565 has been updated with Integration tests cases and looks ready to merge (from a cursory glance). Could you please close this PR and merge https://github.com/apache/fineract/pull/565 if the changes look good to you?

The changes in this MR were merged with Pull Request 565

authors: conradsp vishwasbabu

repo: apache/fineract


text:
[FLINK-20355][docs] Add new native K8s documentation page

Rework of the existing native K8s documentation.
cc @rmetzger, @XComp, @wangyang0918

Thanks a lot for the review @zentol and @wangyang0918. I have addressed most of your comments and pushed an update.

Thanks for the review @XComp. I've addressed your comments and pushed an update.

Thanks again for the review @XComp. I've addressed your comments and squashed all fixup commits.

repo: apache/flink


text:
[FLINK-20658][connectors/jdbc] Establish connection through driver with given name

What is the purpose of the change
Establish jdbc connection through driver with given name.
Brief change log

Establish jdb connection through driver with given name.

Verifying this change
This change is already covered by existing tests:

SimpleJdbcConnectionProviderTest.testEstablishDriverConnection(ignored previously)

This change added tests and can be verified as follows:

SimpleJdbcConnectionProviderTest.testEstablishUnregisteredDriverConnection

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

e2e tests failed unrelated, I have reported it to FLINK-21025.


text:
[FLINK-24419][Table SQL/API] Casting to a CHAR() and VARCHAR() doesn't trim the string to the specified precision

What is the purpose of the change
FLINK-24413: Fix trimming when casting to CHAR and VARCHAR
Brief change log

Update casting logic in StringToBinaryCastRule.java
Update unit tests

Verifying this change
Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
(Please pick either of the following options)
This change is already covered by existing tests, such as CastRulesTest.java.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@shenzhu Could please adjust the title of your PR to:
[FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY

?

@shenzhu Could please adjust the title of your PR to:
[FLINK-24419][table-planner] Trim to precision when casting to BINARY/VARBINARY

?

Sure, just updated PR and commit message, thank you!

Hey @matriv , I updated this PR based on your review, would you mind taking a look at it when you have a moment? Thanks!

Hi @shenzhu I left a couple more comments, but overall the approach is not 100% correct, because it applies the trimming only when casting from string types. It should also apply it when casting between binary types themselves, i.e. a byte array with 10 bytes and type BINARY(10), casted to let's say a BINARY(5) (or VARBINARY(5)) should be trimmed to 5 bytes. So this logic should be applied in for all supported casts to BINARY/VARBINARY.

Hey @matriv , thanks for your review!
I checked the codebase and found currently we support casting from STRING/BINARY/VARBINARY/RAW to BINARY/VARBINARY. For these supported casts, STRING to BINARY/VARBINARY is supported by StringToBinaryCastRule, and the rest is supported via old casting rules in ScalarOperatorGens(BINARY|VARBINARY and RAW).
If I understand it correctly, we have two options for this task:

Create BinaryToBinaryCastRule class and RawToBinaryCastRule class to follow the new casting rules
Update the casting logic in ScalaOperatorGens for BINARY/VARBINARY and RAW to add truncation logic

I'm a little prefer Option 1 because seems that's the direction the community is trying to move forward(?), what do you think about this?
Thanks for your help!

@shenzhu Thx again for your time and effort here.
It would be great to go for option 1. since as you already mentioned we're trying to move all the logic in ScalarOperatorGens into the new java structure under the CastRule interface.

@shenzhu Thx again for your time and effort here. It would be great to go for option 1. since as you already mentioned we're trying to move all the logic in ScalarOperatorGens into the new java structure under the CastRule interface.

Hey @matriv , thanks for your feedback!
I updated this PR with option 1, would you mind taking a look at it when you have a moment?

Unless I'm mistaken, this PR is also fixing this issue correct? https://issues.apache.org/jira/browse/FLINK-25051 Can you double check @shenzhu?

Unless I'm mistaken, this PR is also fixing this issue https://issues.apache.org/jira/browse/FLINK-25051 correct?
Yep, true, I asked @shenzhu to do it (see previous comments). @shenzhu Could you please split your PR in 2 commits?
One commit that does the porting and references FLINK-25051 and one that applies the trimming on top.

Can you double check @shenzhu? My understanding is that the existing codebase doesn't have an inverse BINARY -> RAW casting, right?
This is not supported yet: https://issues.apache.org/jira/browse/FLINK-24577 is still open

@shenzhu Again thx for your effort!
Please rebase with master and then:

I need to ask you now to use the legacyBehaviour from the context to decide if you apply the trimming, similarly to: https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RowToStringCastRule.java#L170 It would be great to not even include the if statement at all in the generated code if the legacyBehaviour is true.

Please also adjust the config in CastFunctionITCase with the following

  @Override
   protected Configuration configuration() {
       return super.configuration()
               .set(TableConfigOptions.LOCAL_TIME_ZONE, TEST_TZ.getId())
               .set(
                       ExecutionConfigOptions.TABLE_EXEC_LEGACY_CAST_BEHAVIOUR,
                       LegacyCastBehaviour.DISABLED);
   }
``` to make sure that we always use the new behaviour for the IT tests.

@shenzhu Again thx for your effort!
Please rebase with master and then:

I need to ask you now to use the legacyBehaviour from the context to decide if you apply the trimming, similarly to: https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/RowToStringCastRule.java#L170 It would be great to not even include the if statement at all in the generated code if the legacyBehaviour is true.
Please also adjust the config in CastFunctionITCase with the following

  @Override
   protected Configuration configuration() {
       return super.configuration()
               .set(TableConfigOptions.LOCAL_TIME_ZONE, TEST_TZ.getId())
               .set(
                       ExecutionConfigOptions.TABLE_EXEC_LEGACY_CAST_BEHAVIOUR,
                       LegacyCastBehaviour.DISABLED);
   }
``` to make sure that we always use the new behaviour for the IT tests.

Sure, thanks for your feedback! I will update this PR.

Hey @AHeise , would you mind taking a look at this PR when you have a moment? Thanks!

authors: matriv shenzhu slinkydeveloper


text:
[FLINK-35941] Add CompiledPlan annotations to BatchExecLimit

What is the purpose of the change

Adds Compiled Plan annotations to BatchExecLimit.
Tests the new annotations with the existing TestPrograms.

Verifying this change
This change adds a BatchRestoreTest to cover the new annotations and show that the batch compiled plan can be restored and executed correctly.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (yes)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@flinkbot run azure


text:
[backport] [FLINK-5300] Add more gentle file deletion procedure

Backport of #2970 to the release-1.1 branch.
Before deleting a parent directory always check the directory whether it contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not instructed to
delete a non-empty directory.

I left a review on #2970 https://github.com/apache/flink/pull/2970#issuecomment-265994539
that can be applied here as well.

I updated the PR wrt the results of the discussion in #2970.

Looks good to be merged imo. I just kicked off RC2... If you would like to have this in 1.1.4, I can re-trigger the build.

Thanks for the review @uce. I think we can include this also in a possible 1.1.5 later. Who knows which other issues will still come up.

Failing test cases are unrelated. Merging this PR.

Has been merged into the release-1.1 branch.

authors: StephanEwen tillrohrmann uce


text:
[FLINK-1458] Allow Interfaces and abstract types in TypeExtractor

Kryo already supports them, so it was just a question of the
TypeExtractor allowing them.
I also added tests for this.

Looks good to me so far, modulo two comments:

Since the type extractor code is changed a lot, it would be good to have a test validating that the TypeExtractor allows abstract types and interfaces.
I would change the log level of the messages (abstract types are less performant, type is no POJO) to INFO, since WARN implies to me that something is wrong (which is not)

But I extended the test with a SerializerTest for both Interfaces (traits) and abstract classes. The other thing I will change, yes.

The tests cover that the serializer supports this. One issue was that the type extractor rejected the types, even though they were supported.

You're right.

Hi @aljoscha
Still getting this warning on compiling for accessing a of type Map[String,Any]
Type Any has no fields that are visible from Scala Type analysis. Falling back to Java Type Analysis (TypeExtractor).
Flink version 1.1.1


text:
[FLINK-7889] Enable dependency convergence in flink-streaming-java

Base on #4777

Doesn't this require fixing/enabling convergence in flink-shaded-hadoop and flink-runtime?

@pnowojski, I found to fix this PR, must first solve the flink-core, flink-runtime, and so on.  Please ignore this PR, I will close it.
Please feel free to continue working on Enable dependency convergence :)

Yes that's correct. Next in line is flink-shaded-hadop.


text:
[FLINK-9078][E2ETests] Add test that verifies correct avro behaviour

What is the purpose of the change
There was a specific kind of classloading issue where a flink job would fail with a ClassCastException when restarting after failure, which has now been fixed.
This end2end test adds automated testing to verify that this error does not occur anymore.
Brief change log

Add example module with avro based on quickstart example
Add test script
Add utility functions and improve output in common.sh

Verifying this change
This change added tests and can be verified as follows:
Run ./run-pre-commit-tests.sh
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@aljoscha @patricklucas Could you maybe have a look at this?

@twalthr How are we planning to proceed on this PR?


text:
[FLINK-11971] Fix kubernetes check in end-to-end test

@dawidwys You originally added this. Do you think this simpler check would work as well?
@hequn8128 Sorry for taking this issue, but I quickly wanted to fix this before creating a new RC because I was running the end-to-end tests on my Mac.

cc @sunjincheng121

@flinkbot approve description
@flinkbot approve consensus

Thank you @aljoscha @dawidwys I am checking the change now...

Thanks for the quick fix @aljoscha!
It works well from my local ENV(iMac)
Merging...
Best,
Jincheng

@flinkbot approve all

Thanks for merging!


text:
GEODE-3473: Move geode-protobuf files under org.apache.geode.internal.

Signed-off-by: Galen O'Sullivan gosullivan@pivotal.io
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[x] Have you written or updated unit tests to verify your changes?

[n/a] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

@kohlmu-pivotal, would you please have a look?

repo: apache/geode


text:
[GOBBLIN-1447] Fix url for linkedin jfrog, add pentaho dependency link

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1447

Description

[x] Here are some details about my PR, including screenshots (if applicable):

We have a dependency on Calcite, which has a dependency on pentaho. Due to either a moved repository or broken link it can no longer be resolved without adding a repository that points to the dependency.
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Was this pentaho dependency present in the old "linkedin.jfrog.com" repository? Maybe we can add it to the new one as well.

Codecov Report

Merging #3284 (fcb95cb) into master (e4d6c1b) will increase coverage by 3.51%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3284      +/-   ##
============================================
+ Coverage     43.03%   46.54%   +3.51%
- Complexity     9302    10031     +729
============================================
  Files          2037     2037
  Lines         79273    79273
  Branches       8841     8841
============================================
+ Hits          34114    36900    +2786
+ Misses        41928    38950    -2978
- Partials       3231     3423     +192

Impacted Files
Coverage Δ
Complexity Δ

...he/gobblin/source/PartitionAwareFileRetriever.java
48.14% <0.00%> (-7.41%)
0.00% <0.00%> (ø%)

...src/main/java/org/apache/gobblin/runtime/Task.java
69.61% <0.00%> (+0.22%)
86.00% <0.00%> (+1.00%)

.../org/apache/gobblin/runtime/SafeDatasetCommit.java
50.00% <0.00%> (+0.51%)
29.00% <0.00%> (ø%)

...ache/gobblin/cluster/GobblinHelixMultiManager.java
53.64% <0.00%> (+0.52%)
17.00% <0.00%> (+1.00%)

...main/java/org/apache/gobblin/util/ConfigUtils.java
58.86% <0.00%> (+0.63%)
41.00% <0.00%> (+1.00%)

...ompaction/mapreduce/CompactionJobConfigurator.java
69.17% <0.00%> (+0.75%)
26.00% <0.00%> (+1.00%)

...ain/java/org/apache/gobblin/runtime/fork/Fork.java
76.30% <0.00%> (+0.80%)
73.00% <0.00%> (ø%)

.../modules/scheduler/GobblinServiceJobScheduler.java
58.88% <0.00%> (+1.01%)
25.00% <0.00%> (ø%)

...e/gobblin/util/filesystem/FileSystemDecorator.java
13.07% <0.00%> (+1.96%)
13.00% <0.00%> (+2.00%)

...rg/apache/gobblin/runtime/AbstractJobLauncher.java
61.67% <0.00%> (+2.03%)
39.00% <0.00%> (+2.00%)

... and 139 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4d6c1b...fcb95cb. Read the comment docs.

+1.

@aplex it was not present in the old repository, I think it was hosted somewhere that was referenced by our (very old) Hive version, and got moved somewhere that is no longer available.

authors: Will-Lo aplex codecov-commenter sv2000

repo: apache/gobblin


text:
[Bug report] Create fileset schema, it won't use schema location, but use catalog location and miss the file separator

Version
0.5.1
Describe what's wrong
Firstly, create a hadoop catalog, for example:
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{   "name": "schema2",   "comment": "comment",   "properties": {     "location": "hdfs://hive:9000"   } }' http://localhost:8090/api/metalakes/metalake_demo/catalogs/hadoop2/schemas
Then create a schema with location:
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{   "name": "schema2",   "comment": "comment",   "properties": {     "location": "hdfs://hive:9000/user2"   } }' http://localhost:8090/api/metalakes/metalake_demo/catalogs/hadoop2/schemas
It reports an error:
{"code":1001,"type":"IllegalArgumentException","message":"Failed to operate schema(s) [schema2] operation [CREATE] under catalog [hadoop2], reason [Relative path in absolute URI: hdfs://hive:9000schema2]","stack":["java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hdfs://hive:9000schema2","\tat org.apache.hadoop.fs.Path.initialize(Path.java:259)","\tat org.apache.hadoop.fs.Path.<init>(Path.java:157)","\tat org.apache.hadoop.fs.Path.<init>(Path.java:125)","\tat com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.lambda$getSchemaPath$8(HadoopCatalogOperations.java:619)","\tat java.util.Optional.map(Optional.java:215)","\tat com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.getSchemaPath(HadoopCatalogOperations.java:619)","\tat com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.createSchema(HadoopCatalogOperations.java:384)","\tat com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$createSchema$4(SchemaOperationDispatcher.java:100)","\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:103)","\tat com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:72)","\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:98)","\tat com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$createSchema$5(SchemaOperationDispatcher.java:100)","\tat com.datastrato.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:107)","\tat com.datastrato.gravitino.catalog.SchemaOperationDispatcher.createSc
Please note here: " [Relative path in absolute URI: hdfs://hive:9000schema2]", which misses a "/" separator.
Here has two issues:

The system should automatically add the separator if missing;
As the location was specified the schema creation request, seems it doesn't use this property, but use its catalog's storage location, and then contact the schema name as the location.

Error message and/or stacktrace
{"code":1001,"type":"IllegalArgumentException","message":"Failed to operate schema(s) [schema2] operation [CREATE] under catalog [hadoop2], reason [Relative path in absolute URI: hdfs://hive:9000schema2]","stack":["java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hdfs://hive:9000schema2","\tat org.apache.hadoop.fs.Path.initialize(Path.java:259)","\tat org.apache.hadoop.fs.Path.(Path.java:157)","\tat org.apache.hadoop.fs.Path.(Path.java:125)","\tat com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.lambda$getSchemaPath$8(HadoopCatalogOperations.java:619)","\tat java.util.Optional.map(Optional.java:215)","\tat com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.getSchemaPath(HadoopCatalogOperations.java:619)","\tat com.datastrato.gravitino.catalog.hadoop.HadoopCatalogOperations.createSchema(HadoopCatalogOperations.java:384)","\tat com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$createSchema$4(SchemaOperationDispatcher.java:100)","\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:103)","\tat com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:72)","\tat com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:98)","\tat com.datastrato.gravitino.catalog.SchemaOperationDispatcher.lambda$createSchema$5(SchemaOperationDispatcher.java:100)","\tat com.datastrato.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:107)","\tat com.datastrato.gravitino.catalog.SchemaOperationDispatcher.createSchema(SchemaOperationDispatcher.java:98)","\tat com.datastrato.gravitino.catalog.SchemaNormalizeDispatcher.createSchema(SchemaNormalizeDispatcher.java:47)","\tat com.datastrato.gravitino.listener.SchemaEventDispatcher.createSchema(SchemaEventDispatcher.java:76)","\tat com.datastrato.gravitino.server.web.rest.SchemaOperations.lambda$createSchema$2(SchemaOperations.java:105)","\tat com.datastrato.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:35)","\tat com.datastrato.gravitino.server.web.rest.SchemaOperations.lambda$createSchema$3(SchemaOperations.java:101)","\tat java.security.AccessController.doPrivileged(Native Method)","\tat javax.security.auth.Subject.doAs(Subject.java:422)","\tat com.datastrato.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:25)","\tat com.datastrato.gravitino.server.web.Utils.doAs(Utils.java:121)","\tat com.datastrato.gravitino.server.web.rest.SchemaOperations.createSchema(SchemaOperations.java:95)","\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)","\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)","\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)","\tat java.lang.reflect.Method.invoke(Method.java:498)","\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)","\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)","\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)","\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)","\tat org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)","\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:292)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:274)","\tat org.glassfish.jersey.internal.Errors.process(Errors.java:244)","\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)","\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)","\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)","\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)","\tat org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)","\tat org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)","\tat org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)","\tat org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)","\tat com.datastrato.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:59)","\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)","\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)","\tat com.datastrato.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:97)","\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)","\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)","\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)","\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)","\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)","\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)","\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)","\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)","\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)","\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)","\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)","\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)","\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)","\tat org.eclipse.jetty.server.Server.handle(Server.java:516)","\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)","\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)","\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)","\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)","\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)","\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)","\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)","\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)","\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)","\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)","\tat java.lang.Thread.run(Thread.java:750)","Caused by: java.net.URISyntaxException: Relative path in absolute URI: hdfs://hive:9000schema2","\tat java.net.URI.checkPath(URI.java:1823)","\tat java.net.URI.(URI.java:745)","\tat org.apache.hadoop.fs.Path.initialize(Path.java:256)","\t... 89 more"]}%
How to reproduce
Use gravitino playground, v0.5.1:
Firstly, create a hadoop catalog, for example:
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{   "name": "schema2",   "comment": "comment",   "properties": {     "location": "hdfs://hive:9000"   } }' http://localhost:8090/api/metalakes/metalake_demo/catalogs/hadoop2/schemas
Then create a schema with location:
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{   "name": "schema2",   "comment": "comment",   "properties": {     "location": "hdfs://hive:9000/user2"   } }' http://localhost:8090/api/metalakes/metalake_demo/catalogs/hadoop2/schemas
Additional context
No response

The problem here is that "hdfs://hive:9000" is not a full qualified path, so using this to create a sub-path will lead to en error. To handle this scenario, we need to add "/" to the end of the path.

authors: jerryshao shaofengshi

repo: apache/gravitino


text:
solved npe isssue #4605

Hi @developer-rohith, we use spotless to ensure the code is consistently formatted. You'll need to run ./gradlew :catalogs:catalog-jdbc-doris:spotlessApply and push the changes to correct the above build error. It's best if you try and build yourself before pushing your changes.

Looks like you still need to run ./gradlew :catalogs:catalog-jdbc-doris:spotlessApply to reformat and commit the changes.

authors: developer-rohith justinmclean