text:
Android API 32-33 Open NON Media Files

Problem
Hi everyone, i recently noticed that since the update to Android 12-13 / API 32-33, the readAsDataURL and readAsDataText methods are no longer working (return NULL) in case of reading PDF/TXT files, while they work no problem for pictures or videos. The folder I'm reading from is the Downloads folder of my Internal Storage.
What is expected to happen?
The result should return the Text or the Base64 Version (as Images)
What does actually happen?
The result is always null
Command or Code
`
Hi, this is my actual code for reading NON MEDIA File in MyFileService in Ionic 6; this code works fine when targetSDK is 29-30, but the Google Playstore no longer accepts this SDK. When i changed to target 32-33 it didn't work anymore:
`//PROPERTY
import { Injectable } from '@angular/core';
import { File } from '@ionic-native/file/ngx';
import { Device } from '@awesome-cordova-plugins/device/ngx';
import { FileChooser } from '@ionic-native/file-chooser/ngx';
import { FilePath } from '@awesome-cordova-plugins/file-path/ngx';
import { Platform } from '@ionic/angular';
import { HttpService } from './http.service';
import { Chooser } from '@awesome-cordova-plugins/chooser/ngx';
//CONSTRUCTOR
public file: File,
public device: Device,
private platform: Platform,
public fileChooser: FileChooser,
private chooser: Chooser,
public http: HttpService,
private filePath: FilePath
....
//SNIPPET WITH ERROR
this.fileChooser
.open()
.then((uri) => {
this.filePath
.resolveNativePath(uri)
.then((url) => {
this.file
.resolveLocalFilesystemUrl(url)
.then((fileEntry: any) => {
this.platform.ready().then(() => {
debugger;
this.file.checkFile(fileEntry['nativeURL'].replace(fileEntry['name'], ''), fileEntry['name']).then(response => {
debugger;
if (response === true) {
this.file
.readAsText(
fileEntry['nativeURL'].replace(fileEntry['name'], ''),
fileEntry['name']
)
.then((result) => {
if (result) {
---->     //RESULT NULL FOR PDF and TXT <---------
debugger;
resolve(result);
} else {
reject('Errore');
}
})
.catch((err) => {
console.log('err-->' + JSON.stringify(err));
});
}
}).catch(err => {
debugger;
});
});
});
})
.catch((err_3) => {
reject(err_3);
});
})
.catch((err_2) => {
reject(err_2);
});``
Environment, Platform, Device
Ionic Cordova Android Platform 11
Version information
Ionic:
Ionic CLI                     : 6.20.3 (/usr/local/lib/node_modules/@ionic/cli)
Ionic Framework               : @ionic/angular 6.4.1
@angular-devkit/build-angular : 12.0.5
@angular-devkit/schematics    : 12.2.18
@angular/cli                  : 12.0.5
@ionic/angular-toolkit        : 4.0.0
Cordova:
Cordova CLI       : 11.0.0
Cordova Platforms : android 11.0.0, ios 6.2.0
Cordova Plugins   : cordova-plugin-ionic-keyboard 2.2.0, cordova-plugin-ionic-webview 4.2.1, (and 27 other plugins)
Utility:
cordova-res                          : not installed globally
native-run (update available: 1.7.2) : 1.7.1
System:
Android SDK Tools : 26.1.1 (/Users/giuseppetaormina/Library/Android/sdk)
ios-sim           : 8.0.2
NodeJS            : v18.12.0 (/usr/local/bin/node)
npm               : 8.19.2
OS                : macOS
Xcode             : Xcode 14.3 Build version 14E222b

The folder I'm reading from is the Downloads folder of my Internal Storage.

Are you sure you don't mean external storage?
Android, Internal Storage refers to to the storage medium guaranteed to be on the device, it usually contains the app install, and the data partition is private to the app only.
External Storage on the other hand may or may not be on the device. If the device has a physical removable storage, like an sdcard, then external storage will be that storage medium, but android also emulates external storage for devices that either don't have an attached storage medium or just plainly doesn't support one. The Download/ directory is found on the External Storage medium. It may have a path such as /sdcard/Download/or /storage/emulated/0/Download.
If the latter, then it's bit of a known issue unfortunately. As of API 30, Scoped Storage rules are fully enforced and there is a caveat on API 29 devices which I'll explain a bit later.
What is Scoped Storage
Scoped Storage is a privacy-focused storage system introduced by Android. It applies to External Storage medium and apps no longer have broad access to the external storage. WRITE_EXTERNAL_STORAGE no longer gives any permission, instead apps may freely read and write to the external storage. However they can only read and write to files that the app has created. If a file already exists but was created from from App A, then App B cannot read or write to that file using the Filesystem API.
Natively to gain read & write access to these files, there is a non-filesystem API called the MediaStore API. The owner of the file must implement a file provider service so it can grant permission to a third-party app trying to read or write to it. I want to emphasize that this API isn't filesystem-like at all so it will be difficult to treat it as such which is why it hasn't really been addressed or resolved in this plugin. For example, you cannot progrommatically list or view the contents of a directory (you can but it will be filtered to only files owned/created by your app) and instead the MediaStore will open the system's file picker and the user may choose the file they want to open.
Additionally, specificially on API 29 devices which is when Scoped Storage was first introduced (but while targeting API 29, it could be opted out, which is not possible today if you intend to deploy to Google Play store). API 29 devices does not have a File System bridge API into the scoped storage module. So using the Native File APIs to read/write files into external storage simply does not work and the only API available to do those actions is the previously mentioned MediaStore API.
As a workaround for now, you'll need to use a plugin that implements/exposes the MediaStore API.

Hi Norman,
thanks a lot for the answer. Hi Norman, thank you so much for your quick reply. I had already seen similar Issues but I wanted to be sure about the presence of valid alternatives. I confirm that the path I access is file:///storage/emulated/0/Download/new.txt , so the problem is the one you explained. I just have a question about the implementation. You said that if the file is created by my app it is likely that it can be read, but if it is external and it is not MEDIA no. Can you confirm it? I'll explain the cases of my app:

In my app it should be possible to export user profile file (with all the data that I have generated in the app) as a base64  with a format of my choice (*.vu) which was read directly with the readAsText function. This file would be used to carry my app's user data from one device to another.
In my app it should be possible to read text files (txt, doc, pdf) to include as document attachments. Here I think you need to access totally external files coming from spaces like storage/emulated or google drive.

Do you have any suggestions for me based on the above points? Many thanks in advance
Giuseppe

@gtaormina i think that latest code will fix this issue

I'm going to shed some more light now that I've understand more about the Android scoped filesystem since the last post in May.
Short answer is unfortunately I don't think the issue is resolvable by the plugin, and you'll likely need to use another plugin that interfaces MediaStore API rather than the File API.
Long answer with history...
History
In API 29, Android introduces a system called Scoped Storage, however on API 29 devices for apps already published to the app store had the ability to opt out to the legacy system via requestLegacyExternalStorage flag. The legacy system gave apps free reign over external storage for as long as WRITE_EXTERNAL_STORAGE was granted.
Scoped Storage (API 29-32)
Scoped Storage makes the WRITE_EXTERNAL_STORAGE permission obsolete as apps have the ability to write to external storage without permission now, but they cannot write to a file that already exists if the file is owned by another app. Apps can also read from external storage without permission but visibility is limited to files only owned by your own app. READ_EXTERNAL_STORAGE will grant you to read media files, but not documents.
Special Notes for API 29
On API 29 specifically, android does not have a File API bridge to Scoped Storage framework. This means if requestLegacyExternalStorage is not enabled, or if you have a brand new app that isn't published to the app store, all file APIs will fail to access external storage. In API 30 and onwards, Scoped Storage framework is forcefully enabled.
Only in API 30 and onward, Android has implemented a File Bridge to allow File APIs to access content in external storage, allowing File APIs (and this file plugin) to somewhat work again. It also means this plugin will not work at all on API 29. It's important to note that even with these file based APIs enabled once again, they only have access to media files.

To help your app work more smoothly with third-party media libraries, Android 11 allows you to use APIs other than the MediaStore API to access media files from shared storage using direct file paths.

Paths like Documents or Downloads are not readable except for media files (images, videos and/or audio). In order to read non-media files, MediaStore API must be used.
What is the MediaStore vs File APIs
To the non-android devs that might be reading this, I'll write a quick explanation on the MediaStore API vs File API.
The File API, is kinda what it sounds. It's a pretty standard filesystem oriented API where you operate on directories and files and you're able to stream data in and out of it. The API offers full programmatic control over the filesystem.
MediaStore API is not a filesystem-oriented API. it's more like a database with a query system. The filesystem details is abstracted and you don't have programmatic control over the filesystem structure, and in some cases you have limited discoverability of files. It's a privacy-focused API so it comes with many restrictions. For example, you generally cannot programmatically read the list of files that may be present on the device. Rather you'll need to open an Intent (e.g. a file picker) where the user selects the file then, and only then your app can become aware that file exists.
Attempting to use the MediaStore in a "file-like" API fashion as this plugin implements I don't think is feasible. Which is why I think a different plugin is required to properly handle accessing external storage on Android.
Notes on API 33+
READ_EXTERNAL_STORAGE is now obsolete. It's still required to support API 32 and earlier, but on API 33, we have 3 new permissions:

READ_MEDIA_AUDIO
READ_MEDIA_IMAGES
READ_MEDIA_VIDEO

You'll notice we still don't have a permission for docs, so we cannot "discover" documentation. We can only open docs via a system file picker intent, like in previous android versions and only if the app that owns the document has a content provider implemented to grant third-party apps access to their document files.
Notes on MANAGE_EXTERNAL_STORAGE Permission
With the introduction of scoped storage, Android introduced MANAGE_EXTERNAL_STORAGE that I believe effectively gives you free reign over external storage similar to the legacy system. However this permission is protected and requires justification. Google will not allow any app with this permission be published to the app store unless if they have a very good reason to use it. An example of that the primary focus of your app is a file manager app, or an anti virus app. Therefore majority of users cannot take advantage of this permission.
So with all that being said, I kinda foresee external storage support being stripped out in favour for a media store oriented plugin, however that is yet to been discussed at the Apache development level... https://www.npmjs.com/package/cordova-plugin-saf-mediastore is one plugin I've heard people had success with, but this isn't an endorsement and I'm not familiar with the NOPL license. So do your own research.

I'm going to shed some more light now that I've understand more about the Android scoped filesystem since the last post in May.
Short answer is unfortunately I don't think the issue is resolvable by the plugin, and you'll likely need to use another plugin that interfaces MediaStore API rather than the File API.
Long answer with history...
History
In API 29, Android introduces a system called Scoped Storage, however on API 29 devices for apps already published to the app store had the ability to opt out to the legacy system via requestLegacyExternalStorage flag. The legacy system gave apps free reign over external storage for as long as WRITE_EXTERNAL_STORAGE was granted.
Scoped Storage (API 29-32)
Scoped Storage makes the WRITE_EXTERNAL_STORAGE permission obsolete as apps have the ability to write to external storage without permission now, but they cannot write to a file that already exists if the file is owned by another app. Apps can also read from external storage without permission but visibility is limited to files only owned by your own app. READ_EXTERNAL_STORAGE will grant you to read media files, but not documents.
Special Notes for API 29
On API 29 specifically, android does not have a File API bridge to Scoped Storage framework. This means if requestLegacyExternalStorage is not enabled, or if you have a brand new app that isn't published to the app store, all file APIs will fail to access external storage. In API 30 and onwards, Scoped Storage framework is forcefully enabled.
Only in API 30 and onward, Android has implemented a File Bridge to allow File APIs to access content in external storage, allowing File APIs (and this file plugin) to somewhat work again. It also means this plugin will not work at all on API 29. It's important to note that even with these file based APIs enabled once again, they only have access to media files.

To help your app work more smoothly with third-party media libraries, Android 11 allows you to use APIs other than the MediaStore API to access media files from shared storage using direct file paths.

Paths like Documents or Downloads are not readable except for media files (images, videos and/or audio). In order to read non-media files, MediaStore API must be used.
What is the MediaStore vs File APIs
To the non-android devs that might be reading this, I'll write a quick explanation on the MediaStore API vs File API.
The File API, is kinda what it sounds. It's a pretty standard filesystem oriented API where you operate on directories and files and you're able to stream data in and out of it. The API offers full programmatic control over the filesystem.
MediaStore API is not a filesystem-oriented API. it's more like a database with a query system. The filesystem details is abstracted and you don't have programmatic control over the filesystem structure, and in some cases you have limited discoverability of files. It's a privacy-focused API so it comes with many restrictions. For example, you generally cannot programmatically read the list of files that may be present on the device. Rather you'll need to open an Intent (e.g. a file picker) where the user selects the file then, and only then your app can become aware that file exists.
Attempting to use the MediaStore in a "file-like" API fashion as this plugin implements I don't think is feasible. Which is why I think a different plugin is required to properly handle accessing external storage on Android.
Notes on API 33+
READ_EXTERNAL_STORAGE is now obsolete. It's still required to support API 32 and earlier, but on API 33, we have 3 new permissions:

READ_MEDIA_AUDIO
READ_MEDIA_IMAGES
READ_MEDIA_VIDEO

You'll notice we still don't have a permission for docs, so we cannot "discover" documentation. We can only open docs via a system file picker intent, like in previous android versions and only if the app that owns the document has a content provider implemented to grant third-party apps access to their document files.
Notes on MANAGE_EXTERNAL_STORAGE Permission
With the introduction of scoped storage, Android introduced MANAGE_EXTERNAL_STORAGE that I believe effectively gives you free reign over external storage similar to the legacy system. However this permission is protected and requires justification. Google will not allow any app with this permission be published to the app store unless if they have a very good reason to use it. An example of that the primary focus of your app is a file manager app, or an anti virus app. Therefore majority of users cannot take advantage of this permission.
So with all that being said, I kinda foresee external storage support being stripped out in favour for a media store oriented plugin, however that is yet to been discussed at the Apache development level... https://www.npmjs.com/package/cordova-plugin-saf-mediastore is one plugin I've heard people had success with, but this isn't an endorsement and I'm not familiar with the NOPL license. So do your own research.

Has anyone a hint on how to use the cordova-plugin-saf-mediastore properly?
I cannot get the function "getURI()" to work when I try passing "Pictures" as folder. I also tried the root-directory path and other combinations.
My file-path I have is: "file:///storage/emulated/0/Pictures/cpcp_capture_65e7de6f.jpg"
Goal: delete this file
I appreciate any help! Thank you 💯

Closing because this is not actionable by Cordova in terms of file plugin API.
Android simply doesn't allow the file plugin to read non-media files owned by other apps using the file API.
MediaStore API is required.
This is now noted at https://github.com/apache/cordova-plugin-file#androids-external-storage-quirks

authors: EYALIN MauriceFrank breautek gtaormina

repo: apache/cordova-plugin-file


text:
Fix reply filter for open_revs

Current filter removing all not_found's from the final reply in presence of any {ok, Doc} in it, which is not always the correct behaviour, since we can have missing doc for one of the passed revisions. In general, in absence of latest attribute, we are expecting one reply per revision which could be either #doc or not found.
This fix's making sure that we are removing not_found's only for the revisions that also have {ok, Doc} reply.
COUCHDB-3097

+1 after adding a comment or tweaking that bit of logic to be a bit less subtle.

@davisp I did both, does it look better now?

100% better. I kept staring at the {not_found, missing} tuple in both clauses of the fold and my brain just couldn't process what was going on for awhile. This is much better.
+1 and good find on that.

source: gharchive/pull-request

authors: davisp eiri

repo: apache/couchdb-fabric

license_type: permissive

license_source: bigquery


text:
Add official support of Couchdb  for Ubuntu 18.04

Hello,
Please add this task in your 2.2.0 milestone.
https://github.com/apache/couchdb/issues/1314 (Duplicate of this but it's closed and task is still open)

@codeyash The convenience binaries for CouchDB are only semi-official.
Whether that ticket is open or not, the work is ongoing.

Any expected date please?

Hello,
I cannot install the couchdb 2.1.1 in Ubuntu 18.04. I use this url:
https://apache.bintray.com/couchdb-deb/dists/bionic/main/
But it returns an error. Any suggestions?  Thank you.

Please see https://github.com/apache/couchdb/issues/1314

source: gharchive/issue

authors: FgfdCBVCbnsebtgf codeyash wohali

repo: apache/couchdb

license_source: github-api


text:
Broken unicode in CouchDB 3.0.0

Description
After upgrading to CouchDB 3 we had a big problem with JS query server that now incorrectly works with unicode characters.
Steps to Reproduce
Create view with following map function:
function (doc) { log(doc.string_field.length); }
Put document with field string_field with any unicode characters. We're have text fields with Hebrew and Russian. Now you can see in console incorrect string length. Thus substring and slice also don't work.
For example:
{"string_field": "мама мыла раму"}
Expected Behaviour
Correct behaviour of length, substring and slice.
Your Environment
CouchDB 3.0 in single node mode or CouchDB 3.0 in cluster mode with 3 nodes. Installed from official docker image on 64bit host system.

CouchDB version used: CouchDB 3.0, git_sha: 03a77db6c
Operating system and version: Official docker image - couchdb:3

Duplicate of #2756 . Thanks for the additional background.

authors: kimaiavd wohali


text:
Add Credo to Elixir test suite

Overview
This PR adds Credo as the static code analysis tool of choice for the Elixir test suite.
Testing recommendations
Just issue make elixir-credo first, then make elixir
Related Issues or Pull Requests
#1756
Checklist

[X] Code is written and works correctly;
[X] Changes are covered by tests;
[X] Documentation reflects the changes;

Here is a rationale of this PR.
I added Credo to the Mix project, then I disabled some "not so useful" checks. They are:

Credo.Check.Refactor.CyclomaticComplexity
Credo.Check.Refactor.LongQuoteBlocks
Credo.Check.Refactor.Nesting

Then, I added Credo to the Makefile and Makefile.win with a proper goal, and I added that goal to make elixir. This way, make elixir formats the code according to #1767, then executes the elixir-credo goal erroring on basically everything.
The only check I maintained but I changed its exit code to 0 is Credo.Check.Design.TagTODO, because I want to be warned about TODOs and monitor them, but I still want to be able to use them as reminders.
@wohali I think your eyes could be useful here 🙂

I have a general question about max line length. This commit explicitly sets the max line length to 120, up from its default of 96. For Erlang code, the current consensus (and also @davisp's emelio) appears to be 80, for which there is ample precedent. Because it forces you to write more concise code, and it’s great being able to have two code windows open side by side (on a 15" laptop), I hope we also adopt 80 for Elixir code. But I'm curious to what others think it should be.

@jaydoane also, as to your comment on "two code windows open side by side (on a 15" laptop)" is highly variable. I use a 15" laptop and often get 3 or 4 code windows open side by side, with >80 characters each. Resolution and font size (and your own visual acuity!) enter into it a lot :)

@jaydoane Hrm, I don't mind it...

Some comments on the above discussion:

Wow, you were great guys, that was such a beautiful debate 😬
I enforced max line length to 120 because our formatter enforces the coding style to 120-chars lines. I'm gonna take it to 90 in Credo, and of course I'm updating the .formatter.exs as well.
Also, I love the 120 limit, and the community does. But if we want to enforce a 90 columns limit, that's the beautiful point of having a tool that you can configure 😹
About disabling Credo.Check.Consistency.ParameterPatternMatching and the long quotes one: Consistency.ParameterPatternMatching is ok, but I think: 1) that we need a long quote block for our test helper 2) that long quote blocks are smells and we should limit that smell to a known place.

Enforcing a 90 columns format will cause almost all of our tests to be reformatted. @wohali et al. do you want me to go further or do we want to do that in another subsequent PR?

Reformatting all the lines that require it to pass a make check invocation is totally fine. Just keep it to an isolated commit and then make sure that make check fails after the fact so that we're not re-introducing breaking things after the fact.

@davisp thanks! I'm going further then

Hi all!
Not long ago we also added checking elixir code format with mix format before running elixir test suite. Since line length is more code formatting issue rather than some logic error, like not used local var or something like that, I believe it's better to address this to mix format.
Here is formatter options, :line_length exactly what we need. We just need to add this in our .formatter.exs wuth reasonable value. In that case make elixir if new changes have incorrect line length. And developer just need to run mix format to reformat his changes.
In that case mix credo have to address only logic code errors.

@dottorblaster sorry, looks like you are going to add line_length in formatter.

@van-mronov yeah line_length is already inside the .formatter.exs and I'm gonna enforce that to 90 as I said before, no problem

@wohali @davisp @jaydoane @van-mronov I added two more commits for that, gonna edit the Makefile soon, can you provide me feedbacks?
@van-mronov I still prefer to maintain max length checks in Credo too because it's more developer friendly. I enforced the value formatter-side as well.

+1 LGTM

authors: davisp dottorblaster jaydoane van-mronov wohali


text:
Optimize ddoc cache

Overview
The previous version of ddoc_cache was written to rely on evicting
entries after a maximum TTL. This leads to issues on clusters that have
a large amount of load on databases with a large Q. What ends up
happening is that when a design document is evicted we suddenly have a
thundering herd scenario as every client attempts to reinsert it into
the cache.
This change instead relies on a monitor process for each cache entry
that periodically attempts to refresh the cache. This way normal clients
accessing a popular design document will never hit a point where it
doesn't exist in cache. And we'll have at most one reader trying to
write the value.
Testing recommendations
make eunit apps=ddoc_cache
You'll find its got 100% test coverage if you enable cover.
One note on the tests that's a bit different than other suites. I stumbled across a fun pattern of adding a function call to various locations in the code so that I can meck:wait on it. This is a compile time switch so its only affects the eunit runs. Production code has the calls compiled out and replaced by an atom which does nothing.
Checklist

[x] Code is written and works correctly;
[x] Changes are covered by tests;

Refreshing my memory on the mem3 PR I can definitely apply a lot of that to this as well so I'll update things for that.
For the accessed messages there's not a lot I think we can do there with the existing structure. Although I could see changing things around a bit to make it a LFU cache instead of an LRU cache. It'd involve a table scan to evict but would remove a lot of the message passing to ddoc_cache_lru and friends. I'll make the mem3 inspired changes first and then will look at this approach.

@chewbranca You managed to nerd snipe me for like four days trying to figure out the mem3 and concurrent LRU stuff.
For the mem3 bits I think my conclusion is that its not appropriate here. For mem3_shards we wanted to make sure that there would always be progress so we load things outside of the gen_server and send them to cache. Given that its a local db read this isn't the most terrible thing in the world. Although, for ddoc_cache its a clustered fabric call, so if we followed the mem3_shards pattern we'd actually just be putting extra load on other parts of the system like rexi, and the couch_db_updaters at 3x the rate of client requests. So having them funnel through the opener actually makes more sense here.
As to the distributed LRU approach for writes I've been playing with a couple different standalone tests to try and get a feel for our maximum throughput with that. So far (with some extremely synthetic tests) I think we're looking at topping out at around 200K updates a second (at least on my laptop). However, this test doesn't actually take into account actually evicting things. My plan for today is to write a basho_bench driver to try and gauge the relative throughput between what I've got written now and a second approach based on your direct ets write idea. I'll post results here when I get them.

@chewbranca Epic nerd snipe. I've sprinkled more parallel on the ddoc_cache. Care to take another look. Currently measuring 1M+ ops/sec against a 1,000 item cache on my MBP. I have to do some work on the test suite tomorrow to finish up the rewritten rewrite work but I'm currently pretty happy with it.
New style is one pid per entry which does all of the ets writes for the entry. the LRU process only exists to do evictions. I soaked 20,000 clients against a 500 item cache on my laptop for 30-45m and it sustained about 1.5M ops/sec without error and without blowing RAM up beyond 500M. Those numbers are for a custom module that does no loading but I think it shows that the cache itself won't be bound by the cache and instead it'll be fabric calls and or some other bottle neck before the cache.

@davisp hehehehe happy to help with the sniping ;-)

@davisp overall looks solid to me.

@chewbranca I managed to figure out a decent way to avoid the unnecessary second fabric calls in the two cases you were worried about. Now the first client that opens an entry will kick off a process that will insert the corresponding cache entry if no entry existed before. I.e., if we call ddoc_cache:open(DbName, DDocId) and there's no entry for it, we'll take the result and insert it for the revid specific version as well (assuming there's not already an existing entry). And vice versa when opening a revid specific version.
https://github.com/apache/couchdb/pull/610/commits/85b53855255be0a4166bbb14697c010ff4468d20
Also, for your comment on the revid specific entries I spent some time trying to figure out a decent way to be clever using latest=true or something but in the end I went entirely different. Now if a cache entry is not accessed within its refresh period it will voluntarily drop out of the cache. So this will apply for all entries which I think is good to help when someone wants to scan a bunch of views periodically. We'll end up dropping all of the unused entries which could help with memory pressure if someone has big design documents without much activity on more than a few.
https://github.com/apache/couchdb/pull/610/commits/32cd254d7e7c678a10a9302e48e9c6f064701701
For the configurability, I added a refresh timeout config option and documented it in the default.ini as per usual. I also added notes for max_size as well.
https://github.com/apache/couchdb/pull/610/commits/9151fb59a8f8553a95af23fa58ef67a76f069b80
And thanks for the good eye on clearing the waiters list.
https://github.com/apache/couchdb/pull/610/commits/7c376bd6efc372cd7f107989fd14e40e8ab7a7e6
The only other thing you really noted was the comment update but I think that was just due to a misunderstanding so I'm assuming you're good my comment there.

I ran the ddoc_cache_speed benchmark with the following diff:
 spawn_workers(WorkerCount) ->
     Self = self(),
-    WorkerDb = list_to_binary(integer_to_list(WorkerCount)),
-    spawn(fun() ->
+    WorkerId = WorkerCount, % rem ?RANGE,
+    WorkerDb = integer_to_binary(WorkerId),
+    spawn_link(fun() ->
         do_work(Self, WorkerDb, 0)
     end),
     spawn_workers(WorkerCount - 1).

-do_work(Parent, WorkerDb, Count) when Count >= 25 ->
+do_work(Parent, WorkerDb, Count) when Count >= 1000 ->
     Parent ! {done, Count},
     do_work(Parent, WorkerDb, 0);

     case timer:now_diff(Now, Start) of
         N when N > 1000000 ->
             {_, MQL} = process_info(whereis(ddoc_cache_lru), message_queue_len),
-            io:format("~p ~p~n", [Count, MQL]),
+            CacheSize = ets:info(ddoc_cache_lru, size),
+            io:format("~p ~p ~p~n", [Count, MQL, CacheSize]),
             report(Now, 0);
         _ ->
             receive

ddoc_cache_speed:go(2000).
1403000 1000 997
1441000 999 997
1416000 1001 998
1387000 1000 997
1403000 1001 996
1418000 1001 997

Left it running for 30 min or so and was seeing about 1M+ opens per second. Cache max size was left as the default at 1000. VM memory and number of processes stayed stable under 100MB and 3500 respectively.
Cache ets size (the 3rd column) also stayed under 1000. Reducing the max_cache to 500 also reduced the table sizes accordingly:
1336000 1500 497
1252000 1499 498
1307000 1500 497
1240000 1500 497

The interesting observation is that the LRU message queue size is in a steady steady and is almost exactly equal to NumberOfWorkers-MaxSize.  NumberOfWorkers=2000 and when MaxSize=1000, then message queue was 2000-1000=1000. Then when MaxSize was reduced to 500, queue size became 2000-500=1500.
+1 (but first see a few questions about ets delete and other minor nits).
Very nice work!

Performance Benchmarking Update: New one is faster!
And now for some graphs. For each of these graphs the max_dbs_open was set to 5,000. That's important to remember as I go through the parameter sets whether we're hitting that limit which will be an artificial limit on performance of the cache. Each of these tests is also just querying an empty view which leads to a ddoc_cache lookup for the coordinator, and then Q lookups for each RPC worker. So the requests per second numbers have to be multiplied out to get actual cache performance. Granted we really don't care given that its a constant factor.
These first two runs are 1,000 workers hitting 1,000 different design documents in 1,000 different databases with two different Q values (i.e., 1,000 workers using their own ddoc and db). For the most part their ops/sec is basically identical while latencies for the new ddoc_cache are slightly better. This suggests that something else was bottlenecking performance. Possibly the basho_bench driver and possibly something else in CouchDB.
Old ddoc_cache Q=4:

New ddoc_cache Q=4:

Old ddoc_cache Q=8

New ddoc_cache Q=8

So basically, trying to hit a bunch of different databases at once now has slightly better latencies but both are fast enough to not be the bottleneck.
Next up was a shoot the moon test to try and test the limits of what sort of performance we could get trying to crush a single entry. These two graphs show 1,000 workers hitting a single design document in a single database. The Q for both of these is 128 which means that the ddoc_cache is going to have to try and maintain 129,000 lookups/sec.
Old ddoc_cache:

New ddoc_cache:

The immediate thing to note here is that old ddoc_cache flat lines half way into its test. This was because couch_log_server spiked its message queue. The reason for this is because of the huge flood of fabric_rpc_worker timeout messages being logged. If you look at the graph for the old ddoc_cache you can see that it has two cliffs, one at 60s, and one at 120s when it then flat lines cause it pushed couch_log_server off a cliff. These drops are precisely what motivated this work in the first place when the old ddoc_cache would evict entries every 60s and the thundering herd would knock things off their rocker.
For the new ddoc_cache you'll see that its still kinda crap performance even though it doesn't totally lose its mind. In the background this was because rexi was unable to keep up with the work load and spiked pretty bad. I'm gonna be investigating that area for more optimizations once this work is wrapped up.
And finally, this last graph is a comparison between the old and new ddoc_caches sweeping through Q=8, 16, and 32 with the same 1,000 workers against a single db and ddoc. Red is old ddoc_cache, green is new ddoc_cache. Yes I know Tufte would kill me but that's the colors that get picked and I don't care enough to go fiddle with it.

As you can see the new ddoc_cache is consistently faster as well as much less variable. Looking into the variance on the best Q=64 run for the old ddoc_cache vs the worst Q=64 run for new ddoc_cache shows a good example of the variance of the old approach. For these two runs I've also included what the rexi server message queue is doing so we can see its effect on performance.
Old ddoc_cache:

New ddoc_cache:

Again we can see how badly that 60s eviction policy is when we have sustained load against a single design document. This leads to some fairly massive spikes in the system. For the old ddoc_cache on the third eviction at 180s we see it flat line again which is why those runs are so variable.
For the new ddoc_cache we can see that db3's rex has sustained elevated message counts which are holding back the benchmark back from meeting some of the old ddoc_cache spikes when rexi had a chance to clear out. Now that I can duplicate that rexi issue easily enough though I'll be working on trying to figure out why its being slow and try and optimize around it.
Hopefully this data is as convincing to everyone else as it is to me. If anyone wants me to check into anything else I certainly have the data and/or can design runs to try and run in some other configuration if requested. However poking both ends of the spectrum (lots of clients against separate design docs and lots of clients against a single design doc) I'm fairly confident that we're winning across the spectrum though most specifically for the single design doc case (which was the motivation for this work).

authors: chewbranca davisp nickva


text:
implemented AutoCloseable Client

Implemented AutoCloseable interface which simple calls destroy() method

This code does not seem to build. I got somee checkstyle errors.
I also think setting the classloader on the bus is not correct as the bus may be for a complete
bundle. So setting it when a DynamicClient is created and removing it when it is removed does not seem to be valid for all cases.
Pelase also try to not reformat the code in commits that also change the logic as this makes it a lot harder to review.
Ideally you should create one branch / pull request per jira issue and make sure the code builds on your machine using mvn clean install.

Yes I've reformatted the code because I didn't knew about your formatter (then I found it). So probably you can fix the issue better if you know how. Sorry for bundling all commits to one PR, I'm not using git too long.

No problem. I had also had a lot of issue with git when I started. I just updated the getting involved guide to show how you ideally structure your work in git. https://cxf.apache.org/getting-involved.html
It does not show the git commands you need but how it should look like.

Build triggered. sha1 is merged.

Build started sha1 is merged.

Build finished. 0 tests run, 0 skipped, 0 failed.

authors: bilak cschneider tomitribe-dev

repo: apache/cxf


text:
[Fix-#6783] switchVersion error (#6783)

Purpose of the pull request
https://github.com/apache/dolphinscheduler/issues/6783
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

@JinyLeeChina

Hi @zwZjut , to target specific issue from PR, we recommend use keyword fix: #issue_id or close: #issue_id or closes: #issue_id in your PR describe. I would not only connect issue to PR but also close issue automatically when PR is be closed.

Codecov Report

Merging #6784 (4bbe66d) into dev (975131e) will decrease coverage by 0.04%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                dev    #6784      +/-   ##
============================================
- Coverage     41.84%   41.80%   -0.05%
+ Complexity     3616     3613       -3
============================================
  Files           641      641
  Lines         25890    25891       +1
  Branches       2795     2795
============================================
- Hits          10833    10823      -10
- Misses        14073    14088      +15
+ Partials        984      980       -4

Impacted Files
Coverage Δ

...er/api/service/impl/TaskDefinitionServiceImpl.java
33.49% <100.00%> (+0.32%)
:arrow_up:

...r/plugin/registry/zookeeper/ZookeeperRegistry.java
47.27% <0.00%> (-7.28%)
:arrow_down:

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...org/apache/dolphinscheduler/remote/utils/Host.java
37.77% <0.00%> (-2.23%)
:arrow_down:

...e/dolphinscheduler/remote/NettyRemotingClient.java
52.11% <0.00%> (-0.71%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 975131e...4bbe66d. Read the comment docs.

authors: codecov-commenter zhongjiajie zwZjut

repo: apache/dolphinscheduler


text:
[Feature][Task] add seatunnel task plugin

Purpose of the pull request

add seatunnel task plugin
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Codecov Report

Merging #7131 (62f3e18) into dev (12b46df) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                dev    #7131      +/-   ##
============================================
- Coverage     41.28%   41.26%   -0.02%
+ Complexity     3665     3661       -4
============================================
  Files           634      634
  Lines         26373    26373
  Branches       2953     2953
============================================
- Hits          10889    10884       -5
- Misses        14455    14460       +5
  Partials       1029     1029

Impacted Files
Coverage Δ

...hinscheduler/common/utils/TaskParametersUtils.java
57.14% <ø> (ø)

...apache/dolphinscheduler/common/enums/TaskType.java
95.65% <100.00%> (ø)

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...org/apache/dolphinscheduler/remote/utils/Host.java
37.77% <0.00%> (-2.23%)
:arrow_down:

...dolphinscheduler/remote/future/ResponseFuture.java
81.96% <0.00%> (-1.64%)
:arrow_down:

...e/dolphinscheduler/remote/NettyRemotingClient.java
52.11% <0.00%> (-1.41%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 12b46df...62f3e18. Read the comment docs.

I restart the failing CI


text:
[Bug] (jdbc catalog) When the field value of some tables is null, the query will report an error.

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
master
What's Wrong?
CAUSED BY: ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.Long;
at org.apache.doris.qe.Coordinator.getNext(Coordinator.java:1278) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.sendResult(StmtExecutor.java:1538) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1473) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:653) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:586) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:444) ~[classes/:?]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:431) ~[classes/:?]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:240) ~[classes/:?]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:160) ~[classes/:?]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:187) ~[classes/:?]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:240) ~[classes/:?]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[classes/:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_151]
What You Expected?
Results can be obtained.
How to Reproduce?
CREATE TABLE numeric_table (
id bigint(32),
tinyint_col TINYINT,
smallint_col SMALLINT,
int_col INT,
bigint_col BIGINT,
decimal_col DECIMAL(10,2),
float_col FLOAT(8,4),
double_col DOUBLE(15,10),
boolean_col BOOLEAN,
date_col DATE,
time_col TIME,
datetime_col DATETIME,
timestamp_col TIMESTAMP
);
INSERT INTO numeric_table (id, tinyint_col, smallint_col, int_col, bigint_col, decimal_col, float_col, double_col, boolean_col, date_col, time_col, datetime_col, timestamp_col) VALUES
(1, 1, 100, 1000, NULL, 10.50, 3.1415, 3.1415926535, TRUE, '2023-01-01', '12:34:56', '2023-01-01 12:34:56', CURRENT_TIMESTAMP),
(2, 0, -50, -500, -NULL, -5.25, -2.7182, -2.7182818284, FALSE, '2023-02-02', '23:59:59', '2023-02-02 23:59:59', CURRENT_TIMESTAMP),
(3, -1, 0, 0,NULL, 0.00, 0.0000, 0.0000000000, TRUE, '2023-03-03', '00:00:00', '2023-03-03 00:00:00', CURRENT_TIMESTAMP);
Anything Else?
Because the type of a certain column is null and the ColumnValueConverter is null. But the type is forced to change.

Since the column values are all null, the type cannot be recognized, so null is returned directly.
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hello @liugddx I would like to work on this issue.

repo: apache/doris


text:
testadd test join case4

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

LGTM

authors: dataalive zy-kkk


text:
regression Add regression tests for datev2

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

I will use a 1MB dataset to replace cases for DateV2.


text:
enhance: When requiredSlots is empty, prune child previously in ColumnPruning

Proposed changes
Issue Number: close #xxx
Problem summary
When requiredSlots is empty, prune child previously in ColumnPruning.
A example:
Agg count(*)
- Filter age > 3
-- Scan id, age

We should get
Agg count(*)
- Project age
-- Filter age > 3
--- Scan id, age

instead of
Agg count(*)
- Project age, id
-- Filter age > 3
--- Scan id, age

Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

column pruning has lots of special cases, and can not image all of the cases, so I can not review this pr.
but in my experience, this code more likely make the regression test, because I remember that I've write the code like you and failed.
when you run into the bug, you can unify abstraction and modification based on all known cases.


text:
Fix Fix bugs in tombstone

Bug:

table ref has not decreased when replay by tombstone.
table sync GC tombstone information can cause errors.

Bug fix:

fixed.
fixed, db sync can be smoothly upgraded, but table sync requires GC of all binlogs

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.02 seconds
stream load tsv:          509 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161222749 Bytes

run buildall

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv:          507 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17163005615 Bytes

authors: deadlinefen hello-stephen


text:
[branch-2.0] enable hive view by default

Proposed changes
Enable hive view support by default
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fixpush more than one runtime filters into cte

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run p0

run pipelinex_p0

run pipelinex_p0

run p0

run buildall

run buildall

run p0


text:
[opt](ES catalog) Increase to 3 connect attempts per node (#30957)

Proposed changes
pick #30957

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog

Proposed changes
Issue Number: close #xxx
Enhancements
This PR introduces enhanced security measures for managing and verifying driver packages when creating Catalogs in Doris. Specifically, it adds path management and checksum verification to ensure the security of driver jars specified by the driver_url. The key highlights include:

For driver packages specified by filename within the jdbc_drivers/ directory (configured via fe.conf and be.conf), Doris assumes these are secure and does not perform additional path checks.
For driver packages specified through absolute local paths or HTTP URLs, Doris enforces checks against allowed paths configured via the jdbc_driver_secure_path FE configuration item. This setting supports multiple paths separated by semicolons. If the driver_url's path does not match any prefix in jdbc_driver_secure_path, the creation of the Catalog is denied.
A checksum parameter can be specified when creating a Catalog, allowing Doris to verify the driver package's integrity post-load. Catalog creation is aborted if verification fails.

Upgrade Impact

The default configuration for jdbc_driver_secure_path is *, indicating that all driver package paths are allowed. This ensures backward compatibility and does not affect existing Catalogs upon upgrade.
Since the verification is only performed during Catalog creation, existing Catalogs remain unaffected.

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


text:
fixFix hive p2 case (#31149)

backport https://github.com/apache/doris/pull/31149

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
opt handle oom exception in spill tasks

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run p0

run buildall

run p0

run buildall


text:
test find large memory allocation not catch exception

Proposed changes
Issue Number: close #xxx

run buildall


text:
test

based on https://github.com/apache/doris/pull/41784, resolve conflicts with the latest branch-2.1 to run branch-2.1 CI

run buildall

run buildall

run buildall

run buildall


text:
[Improvement](LDAP Auth)Enhance LDAP authentication with a configurable group filter

Proposed changes

This PR enhances LDAP authentication by adding an optional configurable filter for retrieving user groups, primarily to support Open Directory LDAP implementations. If the configurable property is left empty, the existing workflow will remain unchanged.

run buildall

run buildall

@morningman @zddr Please could you assist in merging this PR


text:
[test](index compaction)Add exception cases for index compaction

What problem does this PR solve?
Problem Summary:

Add more exception cases for index compaction
Refactor index compaction test code, add a util class for all cases
Merge index_compaction_delete case to index compaction test

Release note
None
Check List (For Author)

Test

[ ] Regression test
[x] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[x] No.
[ ] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall

run buildall

run buildall


text:
DRILL-8443: Upgrade netty due to CVE

Description
https://github.com/apache/drill/security/dependabot/45
Testing
CI build

Some tests failing with this error - IO Running in secure mode, but config doesn't have a keytab
Seems like a Hadoop issue. We may need to upgrade Hadoop or at least review the Hadoop config used in the tests.

@pjfanning I think we may have a CI issue.  @vvysotskyi @jnturton Any ideas here?  Could this be related to that issue we encountered before with the CI and connections?

Now tests for this PR fail for other reason (likely due to changes in the PR):
Error:  Failures:
Error:    TestResultSetLoaderOmittedValues.testOmittedValuesAtEndWithOverflow:264 Row 0 col d should be null
Error:    TestResultSetLoaderOverflow.testBatchSizeLimit:164 expected:<16385> but was:<8193>
Error:    TestOutputBatchSize.testSizerRepeatedRepeatedList:2922 expected:<1048576> but was:<1048560>
Error:  Errors:
Error:    TestResultSetLoaderOmittedValues>SubOperatorTest.classTeardown:39 » IllegalState Allocator[ROOT] closed with outstanding buffers allocated (9).

@pjfanning @cgivre, @rymarm has made progress on this in #2857. Shall we close this one and try to get that one over the line?

sure - I haven't been looking at this

authors: cgivre jnturton pjfanning vvysotskyi

repo: apache/drill


text:
Problem of dubbo.xsd file in dubbo2.5.6.jar

Environment

Dubbo version: 2.5.6
Operating System version: Windows10
Java version: 1.8.0_221

Steps to reproduce this issue

Document description 2.5.6 supports netty4.

I can't find the parameter client from dubbo.xsd in dubbo2.5.6.jar.

I can find the parameter client from dubbo.xsd in dobbo2.5.7.jar.

I hope:

Please update the document, Describe it in more detail.
Modify the dubbo.xsd file in dubbo2.5.6.jar, add parameter "client".

Looking forward to official reply, Thank you very much.

2.5.x version is no longer supported, it is recommended to upgrade to the new version.

authors: CrazyHZM alexander-wong-tech

repo: apache/dubbo


text:
RpcContext proxy

What is the purpose of the change
Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #12766 (b383d21) into 3.2 (848a577) will increase coverage by 0.70%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##                3.2   #12766       +/-   ##
=============================================
+ Coverage     68.66%   69.36%    +0.70%
+ Complexity      116        2      -114
=============================================
  Files          3653     1650     -2003
  Lines        170431    68384   -102047
  Branches      28021     9999    -18022
=============================================
- Hits         117021    47437    -69584
+ Misses        43284    16366    -26918
+ Partials      10126     4581     -5545

see 2036 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@CrazyHZM PTAL

authors: AlbumenJ codecov-commenter


text:
[Feature][3.3] Triple rest useTrailingSlashMatch useSuffixPatternMatch support #14036

What is the purpose of the change
support useTrailingSlashMatch useSuffixPatternMatch #14036
Brief changelog
Added constructor method and added support for useTrailingSlashMatch useSuffixPatternMatch
Verifying this change
No

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

changelog:

support useTrailingSlashMatch useSuffixPatternMatch
fix dependencies
delete a test because it always failed even I do nothing.This test case has no relation to my modifications(I clone the resource code and run test ,it logged: Wanted but not invoked:errorTypeAwareLogger.info();-> at org.apache.dubbo.config.utils.ConfigValidationUtilsTest.testCheckQosInApplicationConfig(ConfigValidationUtilsTest.java:118)
Actually, there were zero interactions with this mock.
)


text:
Update dubbo-provider.xml

What is the purpose of the change
XXXXX
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

could you please provide more detail for this PR?

could you please provide more detail for this PR?
不修改该地址zookeeper://127.0.0.1:2181?registry-type=service会报错，

introduced by https://github.com/apache/dubbo/commit/13868b920d647d032d4c6bb70ddac069934fa47e#diff-1fceec52425b4ef669f6f4ef7a316637b20cb66a2782a800042f0aadd7858244R28

thanks for your contribution

authors: baymaxxjf htynkn


text:
[Feature] 折线图series上的label支持和x轴一样的根据密度分布显示

What problem does this feature solve?
大数据量的时候想显示label，就会出现所有label都显示的请客。如果自己进行密度分布可能和x轴的数据显示不同步

What does the proposed API look like?
增加一个类似xAxis均匀显示的api   xAxis: {         data: xData,         axisLabel: {           interval: 'auto'         }       },

labelLayout { hideOverlap: true} ?

authors: MatthiasMert SONASIDANTE

repo: apache/echarts


text:
Creditbureau configuration module

This PR contains code for creditbureau configuration

@nikpawar89 Please follow coding style defined by https://google.github.io/styleguide/javaguide.html

@nikpawar89 Please let me know whether I understood correctly. With this implementation, I can define credit bureau, credit bureau products and map them to a loan product. So where is actual credit bureau high level implementation so that anybody can attach specific provider implementation?

@nazeer1100126  This module is intended for configuring the credit bureau. Once the credit bureau is mapped to a specific loan product, i'll be using this configuration to map the credit check request with specific credit bureau.The code for mapping credit check with credit bureau is not included in this module, it will be included with each specific implementation of the credit bureau in CreditBureauConfigurationAPI.java

reponening

repo: apache/fineract


text:
[hotfix] Run schema coordinator logic asynchronously to avoid blocking the main thread

This tweaks SchemaRegistry coordinator to let it handle requests / trigger checkpoints in background threads to avoid blocking the main thread.
This also adds E2e test case running on various parallelisms.

@loserwang1024 Would you like to review this PR when you have time ?

authors: leonardBang yuxiqian

repo: apache/flink-cdc


text:
[FLINK-36165][source-connector/mysql] Support capturing snapshot data with conditions

Hey, this is an implementation designed to capture snapshot data with filtering conditions.
For example, by specifying scan.snapshot.filters: db.user_table:id > 200;, we can synchronize only the user data where the id is greater than 200.
issue link: https://issues.apache.org/jira/browse/FLINK-36165

Thanks for @uicosp's great work! It's indeed a long awaited feature.
Seems Debezium has a similar option called snapshot.select.statement.overrides, which allows users to project out unwanted columns and filter rows based on custom predicates.
As these options aren't available in incremental framework, it would be nice if we could support both row and column level pushdown with similar syntax, since they're both related to tweaking snapshot querying SQL. WDYT?

Thanks for @uicosp's great work! It's indeed a long awaited feature.
Seems Debezium has a similar option called snapshot.select.statement.overrides, which allows users to project out unwanted columns and filter rows based on custom predicates.
As these options aren't available in incremental framework, it would be nice if we could support both row and column level pushdown with similar syntax, since they're both related to tweaking snapshot querying SQL. WDYT?

it is also support table api and datastream api  with  incremental framework

authors: Thorne-coder uicosp yuxiqian


text:
[hotfix] Change email/repository notifications to match with Flink Core settings

Making sure that notification schema is setup like it's currently for Flink core. Details about this file can be found in https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features

Dank.

Dank.

You're welcome :)

authors: MartijnVisser mbalassi

repo: apache/flink-kubernetes-operator


text:
[FLINK-16048][avro] Support read/write confluent schema registry avro…

… data from Kafka
What is the purpose of the change
Supports read/write with SQL using schema registry avro format.
The format details
The factory identifier (or format id)
There are 2 candidates now ~
avro-sr: the pattern borrowed from KSQL JSON_SR format [1]
avro-confluent: the pattern borrowed from Clickhouse AvroConfluent [2]
Personally i would prefer avro-sr because it is more concise and the confluent is a company name which i think is not that suitable for a format name.
The format attributes

Options
required
Remark

schema-registry.url
true
URL to connect to schema registry service

schema-registry.subject
false
Subject name to write to the Schema Registry service, required for sink

Note: the avro schema string is always inferred from the DDL schema, so user should keep the nullability correct (DDL type default is nullable but avro default is non-nullable).
Brief change log

Add avro-sr read/write row data format
Add tests

Verifying this change
Added tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: (yes / no / don't know) no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented

+1 for SR and please tag me when you open a documentation PR for this feature

+1 for SR and please tag me when you open a documentation PR for this feature

Sure， thanks for taking care the document.

After a quick glimpse. Could we unify the AvroRowDataDeserializationSchema with ConfluentRegistryAvroRowDataDeserializationSchema and RegistryAvroRowDataSerializationSchema?
I really believe we need just a single AvroRowDeserializationSchema for avro for table API.
I am quite sure sth like this would work:
RowDataDeserializationSchema extends ResultTypeQueryable {
  	private final DeserializationSchema<GenericRecord> nestedSchema;
	private final DeserializationRuntimeConverter runtimeConverter;
	private final TypeInformation<RowData> resultType;

	public AvroRowDataDeserializationSchema2(
			DeserializationSchema<GenericRecord> nestedSchema,
			DeserializationRuntimeConverter runtimeConverter,
			TypeInformation<RowData> resultType) {
		this.nestedSchema = nestedSchema;
		this.runtimeConverter = runtimeConverter;
		this.resultType = resultType;
	}

	@Override
	public void open(InitializationContext context) throws Exception {
		nestedSchema.open(context);
	}

	@Override
	public RowData deserialize(byte[] message) throws IOException {
		try {
			GenericRecord deserialize = nestedSchema.deserialize(message);
			return (RowData) runtimeConverter.convert(deserialize);
		} catch (Exception e) {
			throw new IOException("Failed to deserialize Avro record.", e);
		}
	}

       	@Override
	public boolean isEndOfStream(RowData nextElement) {
		return false;
	}

	@Override
	public TypeInformation<RowData> getProducedType() {
		return resultType;
	}
}

and then you would use it like this:
in AvroFormatFactory:
new RowDataDeserializationSchema(
	AvroDeserializationSchema.forGeneric(AvroSchemaConverter.convertToSchema(rowType)),
        createRowConverter(rowType), // we would need to move this method to some utils class or to a common abstract class for factories
	rowDataTypeInfo
);

in RegistryAvroFormatFactory:
new RowDataDeserializationSchema(
	ConfluentRegistryAvroDeserializationSchema.forGeneric(
		AvroSchemaConverter.convertToSchema(rowType),
		schemaRegistryURL
	),
        createRowConverter(rowType),
	rowDataTypeInfo
);

Thanks for the nice review, i have addressed your comments.

Would flink-avro-confluent be a better module name than flink-avro-confluent-registry? IMO registry has nothing to do with the format itself.

Would flink-avro-confluent be a better module name than flink-avro-confluent-registry? IMO registry has nothing to do with the format itself.

It depends on how we understand it, the confluent avro format is mainly designed for schema registry.

Schema registry is a terminology and people always calls "schema registry url"[1], the same for "schema registry subject".
[1] https://docs.confluent.io/current/schema-registry/index.html#high-availability-for-single-primary-setup

@danny0405
Any reasons all the fields read and written by this format has prefix 'record_' ?

@danny0405 @dawidwys
Any reasons all the fields read and written by this format has prefix 'record_' ? (I'm using flink sql for this client)
I found responsible code probably here but still have problem with this solution:
https://github.com/apache/flink/blob/de87a2debde8546e6741390a81f43c032521c3c0/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSchemaConverter.java#L365

It's because of the current strategy to infer the Avro schema is convert from the CREATE TABLE DDL, and there is no way to get the record name here. So we put a constant record as a prefix. The record write out all have explicit field name, but the type should be compatible.

@danny0405
Problem is that with this strategy I'm unable to read anything from Kafka using Confluent Registry. Example:
I have data in Kafka with following value schema:
{
  "type": "record",
  "name": "myrecord",
  "fields": [
    {
      "name": "f1",
      "type": "string"
    }
  ]
}

I'm creating table using this avro-confluent format:
create table `test` (
	`f1` STRING
) WITH (
  'connector' = 'kafka',
  'topic' = 'test',
  'properties.bootstrap.servers' = 'localhost:9092',
  'properties.group.id' = 'test1234',
   'scan.startup.mode' = 'earliest-offset',
  'format' = 'avro-confluent'
  'avro-confluent.schema-registry.url' = 'http://localhost:8081'
);

When trying to select data I'm getting error:
SELECT * FROM test;
[ERROR] Could not execute SQL statement. Reason:
org.apache.avro.AvroTypeException: Found myrecord, expecting record, missing required field record_f1

Thanks, i think this is a bug, i have logged an issue there. See https://issues.apache.org/jira/browse/FLINK-19779

I have fired a fix https://github.com/apache/flink/pull/13763/files, can you help check if possible @maver1ck :)

I will check... mvn is running

OK. It's working. I'm able to read data.

@danny0405
I think we have one more problem.
When Flink is creating schema in registry nullability is not properly set for logical types.
Examples. Table:
create table `test_logical_null` (
	`string_field` STRING,
	`timestamp_field` TIMESTAMP(3)
) WITH (
  'connector' = 'kafka',
  'topic' = 'test-logical-null',
  'properties.bootstrap.servers' = 'localhost:9092',
  'properties.group.id' = 'test12345',
   'scan.startup.mode' = 'earliest-offset',
  'format' = 'avro-confluent', -- Must be set to 'avro-confluent' to configure this format.
  'avro-confluent.schema-registry.url' = 'http://localhost:8081', -- URL to connect to Confluent Schema Registry
  'avro-confluent.schema-registry.subject' = 'test-logical-null' -- Subject name to write to the Schema Registry service; required for sinks
)

Schema:
{
  "type": "record",
  "name": "record",
  "fields": [
    {
      "name": "string_field",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "timestamp_field",
      "type": {
        "type": "long",
        "logicalType": "timestamp-millis"
      }
    }
  ]
}

For not null fields:
create table `test_logical_notnull` (
	`string_field` STRING NOT NULL,
	`timestamp_field` TIMESTAMP(3) NOT NULL
) WITH (
  'connector' = 'kafka',
  'topic' = 'test-logical-notnull',
  'properties.bootstrap.servers' = 'localhost:9092',
  'properties.group.id' = 'test12345',
   'scan.startup.mode' = 'earliest-offset',
  'format' = 'avro-confluent', -- Must be set to 'avro-confluent' to configure this format.
  'avro-confluent.schema-registry.url' = 'http://localhost:8081', -- URL to connect to Confluent Schema Registry
  'avro-confluent.schema-registry.subject' = 'test-logical-notnull-value' -- Subject name to write to the Schema Registry service; required for sinks
);

Schema
{
  "type": "record",
  "name": "record",
  "fields": [
    {
      "name": "string_field",
      "type": "string"
    },
    {
      "name": "timestamp_field",
      "type": {
        "type": "long",
        "logicalType": "timestamp-millis"
      }
    }
  ]
}

As you can see for string_field we have proper union with null (for nullable field). For timestamp_field in both examples union is missing.

@maver1ck , you are right, we ignore the nullability of TIMESTAMP_WITHOUT_TIME_ZONE, DATE, and TIME_WITHOUT_TIME_ZONE and Decimal, would fix it altogether in this PR.

Thanks @danny0405

I have updated the fix, @maver1ck , please check if you have time.

@danny0405 I see code review is still in progress.
Could you please let me know when this PR would be polished a little ?

repo: apache/flink


text:
[FLINK-22002][tests] Let taskmanager.slot.timeout fall back to akka.ask.timeout

This commit lets taskmanager.slot.timeout fall to akka.ask.timeout so that the MiniCluster
will run by default with a 5 min taskmanager.slot.timeout. This should harden against CI
pauses.

Thanks for the review @zentol. Merging this PR now.


text:
[FLINK-30629][Client/Job Submission] Increase clientHeartbeatTimeout to 1 second

What is the purpose of the change
Increase clientHeartbeatTimeout to 1 second to avoid shutting down the job in ClientHeartbeatTest.
Brief change log

Increase clientHeartbeatTimeout to 1 second

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

thanks @Myracle

authors: Myracle snuyanzin


text:
[FLINK-32073][checkpoint] Implement file merging in snapshot

What is the purpose of the change
As one part of FLIP-306, this PR enables file merging in snapshot by implementing the corresponding output&input streams for FileMergingSnapshotManager
Brief change log

Add createCheckpointStateOutputStream in FileMergingSnapshotManager.
Add output&input streams for file merging checkpoints
Add SegmentFileStateHandle to represent a state that is written into a file segment.

Verifying this change
Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
This change added several UT tests in:

FileMergingSnapshotManagerTest
FileMergingCheckpointStateOutputStreamTest
FsSegmentDataInputStreamTest.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Thanks @fredia and @Zakelly for your comments. Sorry for the late reply. I have addressed your comments. Would you please take another look?

Thanks @fredia and @Zakelly for your comments. Sorry for the late reply. I have addressed your comments. Would you please take another look?

Thanks for your effort. I have no comments but a minor one, PTAL, thanks.

CI seems Failed with "99d7b97"

@flinkbot  run azure

@curcur The CI failure has been addressed. (The file-merging checkpoint was unexpectedly enabled.) Would you please take another look?

The PR looks GTM.

I left a few comments.
squash the commits.
There are conflicts, resolve them and after turn green, ping me, and I will merge them.

@curcur I have addressed your comments by removing the unused codes and squashing the commits. The PR is now free of conflicts and has passed the CI tests. PTAL.

authors: AlexYinHan Zakelly curcur


text:
[FLINK-33410] Implement type inference for OVER function

What is the purpose of the change
It implements type inference for OVER, STREAM_RECORD and THROW_EXCEPTION functions.
Verifying this change

all available tests pass
added tests for OVER type inference

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

LGTM, only left one minor comments

authors: dawidwys liuyongvs


text:
[FLINK-31472][Connectors/Base] Update AsyncSink Throttling test to use concurrent mailbox

What is the purpose of the change
This pull request updates AsyncSinkThrottlingTest to use concurrent TestSinkInitContextAnyThreadMailbox due the concurrent nature of the test and to address CI stability issues.
Brief change log

Updated  AsyncSinkThrottlingTest to use concurrent TestSinkInitContextAnyThreadMailbox

Verifying this change
This change is a trivial rework for test suite, changes are verified by rerunning tests in debug mode.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive):no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@vahmed-hamdy would you mind creating backports ?

authors: snuyanzin vahmed-hamdy


text:
[FLINK-1528][Gelly] Added Local Clustering Coefficient Example

Hi!
I closed https://github.com/apache/flink/pull/400 and created this instead, because the commit history was messed up.
I have the following exception when running LocalClusteringCoefficientExample
Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException:
Type of TypeVariable 'K' in 'class flink.graphs.library.LocalClusteringCoefficient
$NeighborhoodEdgesFunction' could not be determined.
This is most likely a type erasure problem. The type extraction currently supports types
with generic variables only in cases where all variables in the return type can be deduced
from the input type(s).

I have no idea how to fix this though. This is @vasia's old reply:

The problem is here:
DataSet<Tuple2<K, HashSet<K>>> neighborhoods = input.reduceOnEdges(new NeighborhoodEdgesFunction<K>(), EdgeDirection.OUT);

and we try to get the return type Tuple2<K, HashSet<K>> like this:
public TypeInformation<T> getProducedType() {
     return TypeExtractor.createTypeInfo(EdgesFunction.class, function.getClass(), 2, null, null);
}

Anyone have an idea? Is it because of the nested type parameter in Tuple2<K, HashSet<K>> ?

Also, I tried propagating an additional parameter Q extends HashSet<K>, but it didn't help.
Does anybody know what's wrong here?
Thanks in advance!

Does anyone have an idea about this? Is there a way to pass the HashSet<K> type?
In any case, even if not, I think it doesn't matter in this case.
I believe we shouldn't add this as a library method, as it is a quite naive implementation of local clustering coefficient. I'd prefer if we try to keep the library methods as efficient as possible.
However, I would definitely add this as an example, since it very nicely demonstrates how to use neighborhood methods and joinWithVertices (which are missing from the other examples).
So, I would suggest we change this to an example that uses a sample dataset, with e.g. Long ids and also allows file input. What do you think @balidani?

Hi!
I agree. I will change the example then :)

I made the changes @vasia suggested

Hi @balidani! Thanks for the example :))
I'm a bit confused about the directed / undirected case. I tried testing with both a directed and an undirected input and both my tests failed.. See here are the test cases I tried.
I suppose the directed case doesn't work because you only consider the out-neighbors, when you should count all. And for the undirected case, I think that the division by 2 you're making is giving a wrong result, because you're not counting edges, just out-neighbors neighbors.

Hi @vasia! I fixed the algorithm, now it will convert all edges to a pair of edges and call distinct on the edge set. This gives the correct results now. Thanks!

Hi @balidani! I think it's still a bit confusing how this example works.
As far as I understand, you expect a directed graph as input, but then you convert it to an undirected one and compute the clustering coefficient of the undirected graph, right?
We should either document this behavior clearly in the example description or compute the result for the input we expect. I personally prefer the second :-)
Also, take a look in the other examples and write a short usage description in the beginning of the example, including the input format that you expect when args are provided.
Let me know if you have questions! Thanks!

Hey @balidani!
Would you like to finish this up?
It's not really urgent, but it's almost finished and it'd be a pity to abandon :)
Someone else could also take over of course. Just let us know!

Yeah, I should definitely finish this! I'll take a look tonight, sorry about that :)

Any progress on this pull request, or should it be closed?

@balidani,
I think it'd be better if you close this PR. I don't think we'll add another example after #1000 is merged. I can take over and probably reuse some of your code to add a local clustering coefficient library method. Would that be OK? Thanks!

@vasia yes, I'm sorry about not finishing it, but I just did not have the time lately.
Cheers!

Thats' fine :) Thanks for the fast response!

authors: StephanEwen balidani vasia


text:
[FLINK-7810] Switch from custom Flakka to Akka 2.4.x

What is the purpose of the change
Drop support for Scala 2.10 and then update to a newer Akka version. Before, we were forced to stay on our custom Flakka 2.3.x version with back ports because newer Akka does not support Scala 2.10.

wonderful! +1

Thanks for doing this, we need Akka 2.4 badly for improved SSL support.

I think this is good now, +1

authors: EronWright StephanEwen aljoscha bowenli86


text:
[Java] Move StringSerializer#isLatin to StringUtils

Is your feature request related to a problem? Please describe.

Currently, the isLatin(char[]) judgment is coupled in org.apache.fury.serializer.StringSerializer, but it is not strongly related to org.apache.fury.serializer.StringSerializer. org.apache.fury.serializer.StringSerializer is only responsible for serializing related content.
Describe the solution you'd like

isLatin(char[]) should be moved to org.apache.fury.util.StringUtils
Additional context

Maybe we can add a good first issue label to the current task.

repo: apache/fury


text:
GEODE-4403: Remove ProtobufErrorCode

as it's a duplicate of BasicTypes.ErrorCode
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[not really applicable] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

Rebased on develop, will merge if tests pass.

repo: apache/geode


text:
GEODE-6743: Remove GFJsonObject and GFJsonArray classes

Authored-by: Jens Deppe jdeppe@pivotal.io
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[ ] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[ ] Is your initial contribution a single, squashed commit?

[ ] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

@Petahhh

Failing test is unrelated to the code changes


text:
GEODE-4826: Use spotlessCheck, not spotlessApply, as input to srcDistTar

Using spotlessApply as an input to srcDistTar allowed build and other
CI tasks to pass while still having style errors on the repo.
Authored-by: Robert Houghton rhoughton@pivotal.io
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[X] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[X] Is your initial contribution a single, squashed commit?

[X] Does gradlew build run cleanly?

[n/a] Have you written or updated unit tests to verify your changes?

[n/a ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

good catch, looks like this problem was only introduced about a week ago

since this is a fix to GEODE-9284 perhaps that would be a more appropriate ticket number than GEODE-4826?

authors: onichols-pivotal rhoughton-pivot


text:
Fix the type of default values to match what Schema.parse() does.

When calling field.defaultVal() (or equivalent compat helper methods)
under Avro 1.9.2+, we get back a type that's not consistent with what
Schema.parse() creates internally. This causes two Schemas constructed
in these two different ways (Schema.parse vs Schema.createRecord) to
be considered unequal, even though their toString() representations
are identical.
Fix this situation by calling parseJsonToObject(), which results in
the default value being interpreted similar to Schema.parse().
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-XXX

Description

[ ] Here are some details about my PR, including screenshots (if applicable):

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3611 (4240794) into master (c6d6c1b) will decrease coverage by 3.13%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #3611      +/-   ##
============================================
- Coverage     46.87%   43.74%   -3.14%
+ Complexity    10687     2060    -8627
============================================
  Files          2125      408    -1717
  Lines         83157    17627   -65530
  Branches       9266     2154    -7112
============================================
- Hits          38983     7711   -31272
+ Misses        40598     9057   -31541
+ Partials       3576      859    -2717

Impacted Files
Coverage Δ

...c/main/java/org/apache/gobblin/util/AvroUtils.java
56.50% <100.00%> (ø)

...a/org/apache/gobblin/cluster/GobblinHelixTask.java
62.36% <0.00%> (-2.16%)
:arrow_down:

...ion/mapreduce/orc/OrcKeyCompactorOutputFormat.java

...ement/copy/TimeAwareCopyableGlobDatasetFinder.java

...bblin/metrics/event/EntityMissingEventBuilder.java

...n/runtime/job_exec/JobLauncherExecutionDriver.java

...n/yarn/event/GetApplicationReportFailureEvent.java

...t/version/FileStatusTimestampedDatasetVersion.java

...rg/apache/gobblin/source/jdbc/OracleExtractor.java

...in/instrumented/fork/InstrumentedForkOperator.java

... and 1710 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter srramach

repo: apache/gobblin


text:
feat: Add @Deprecated annotations and explanations across GravitinoClientBase

I added @Deprecated annotations to relevant methods in the GravitinoClientBase class to improve code clarity and guide developers towards using updated methods.   - Deprecated getVersion() method in favor of serverVersion(). - Provided explanations within the annotations to clarify why the deprecation was applied and what should be used instead.  This change ensures that outdated methods are clearly marked and helps maintain codebase consistency.  Closes #3084
As part of an effort to keep the Gravitino Java client codebase clean and up-to-date, deprecated annotations were added to certain methods in the GravitinoClientBase class. The getVersion() method was marked as deprecated since it duplicates the functionality provided by the serverVersion() method. An inline comment was provided to guide developers towards using the recommended method.
This is my first contribution to the project, and I look forward to contributing further. Please review the changes and provide feedback if needed.
What changes were proposed in this pull request?
This pull request adds @Deprecated annotations to specific methods within the GravitinoClientBase class. The getVersion() method was deprecated in favor of serverVersion() to reduce redundancy and steer developers towards the preferred method. Each deprecated method includes a detailed comment explaining why the deprecation was applied and suggesting the appropriate alternative.
Why are the changes needed?
These changes are necessary to maintain a clean and up-to-date codebase. By marking certain methods as deprecated, we help developers avoid using outdated or redundant methods and encourage best practices within the Gravitino Java client. Specifically:
The getVersion() method was deprecated as it duplicates the functionality of serverVersion().
Developers are now guided to use the recommended methods via inline comments.
Fix: #3084
Does this PR introduce any  user-facing change?
There are no direct user-facing changes. However, developers using the GravitinoClientBase class will now see deprecated warnings for certain methods, guiding them towards the recommended alternatives.
How was this patch tested?
The patch was tested by running the existing unit tests to ensure that the addition of @Deprecated annotations did not introduce any regressions or break existing functionality. No new tests were required, as this change is purely annotative and does not alter the underlying logic.

@shaofengshi can you please help to review.

repo: apache/gravitino