text:
Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37

Is your feature request related to a problem or challenge?
In 37.0.0 many of the built in functions have been migrated to UDFs as described on #8045 . The migration is completed in 38.0.0
One part of this change is that now certain Exprs must be rewritten into the appropriate functions. Most notably get_field that extracts a field from a Struct
Among other things this allows people to customize how Expr behaves: https://github.com/apache/datafusion/issues/7845#issuecomment-2066290235 or in slack to return NULLs for rows that don't pass in maps
The rewrite happens automatically as part of the logical planner (in the Analyzer pass)
However if you bypass those passes it will not happen
Yeah you need to use the FunctionRewriter here (with the relevant rewriter registered) https://github.com/apache/arrow-datafusion/blob/0573f78c7e7a4d94c3204cee464b3860479e0afb/datafusion/optimizer/src/analyzer/function_rewrite.rs#L33
Example
An example from discord: link is:
  let schema = Schema::new(vec![
        Field::new("id", DataType::Utf8, true),
        Field::new(
            "props",
            DataType::Struct(Fields::from(vec![Field::new("a", DataType::Utf8, true)])),
            true,
        ),
    ]);

    println!("schema {:?}", schema);

    let df_schema = DFSchema::try_from(schema.clone()).unwrap();

    let plan = table_scan(Some("props_test"), &schema, None)?
        .filter(col("props").field("a").eq(lit("2021-02-02")))?
        .build()?;
    println!("logical plan {:?}", plan);
    let phys = DefaultPhysicalPlanner::default().create_physical_expr(&plan.expressions()[0], &df_schema, &SessionContext::new().state())?;
    println!("phys {:?}", phys);
    Ok(())

This returns an error "NamedStructField should be rewritten in OperatorToFunction"
Describe the solution you'd like
No response
Describe alternatives you've considered
One potential workaround is to call get_field directly rather than Expr::field
So instead of
    let plan = table_scan(Some("props_test"), &schema, None)?
        .filter(col("props").field("a").eq(lit("2021-02-02")))?
        .build()?;

call like
  let plan = table_scan(Some("props_test"), &schema, None)?
        .filter(get_field(col("props", "a")).eq(lit("2021-02-02")))?
        .build()?;

Additional context
@ion-elgreco  is seeing the same issue in Delta-rs: https://github.com/apache/datafusion/issues/9904#issuecomment-2069359171

I tried it with 37.1.0 in delta-rs, but we still get this error:  internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction, wasn't this regression fixed?

@westonpace  brings it up in discord link
Another report in discord: link

Example from @ion-elgreco
@alamb this is the code:
  let (table, _metrics) = DeltaOps(table)
            .delete()
            .with_predicate("props['a'] = '2021-02-02'")
            .await
            .unwrap();

Which comes from here: https://github.com/delta-io/delta-rs/blob/main/crates%2Fcore%2Fsrc%2Foperations%2Fdelete.rs#L770-L774

I see a few options:

We can put get_field back into the core crate (and move it out of datafusion-functions) - might be the least surprising but would not allow people to customize how field access worked (e.g. for implementing JSON support)
We can make a better API / more examples how to get the core rewrites working in https://github.com/apache/datafusion/blob/07804384cbdcdd2861ec8a279632da32245e28f7/datafusion-examples/examples/expr_api.rs#L84-L110

I would be happy to work on 2 if someone could point me at how people are creating Physical exprs today

I think I'd be happy with 2.  The example you linked is how we are using datafusion.  Here is an updated example that fails with the error:
    // For example, let's say you have some integers in an array
    let b = Arc::new(Int32Array::from(vec![4, 5, 6, 7, 8, 7, 4]));
    let a = Arc::new(StructArray::new(
        vec![Field::new("b", DataType::Int32, false)].into(),
        vec![b],
        None,
    ));
    let batch = RecordBatch::try_from_iter([("a", a as _)])?;

    // If you want to find all rows where the expression `a < 5 OR a = 8` is true
    let expr = col("a")
        .field("b")
        .lt(lit(5))
        .or(col("a").field("b").eq(lit(8)));

    // First, you make a "physical expression" from the logical `Expr`
    let physical_expr = physical_expr(&batch.schema(), expr)?;

@alamb this is how we create expressions:
/// Parse a string predicate into an `Expr`
pub(crate) fn parse_predicate_expression(
    schema: &DFSchema,
    expr: impl AsRef<str>,
    df_state: &SessionState,
) -> DeltaResult<Expr> {
    let dialect = &GenericDialect {};
    let mut tokenizer = Tokenizer::new(dialect, expr.as_ref());
    let tokens = tokenizer
        .tokenize()
        .map_err(|err| DeltaTableError::GenericError {
            source: Box::new(err),
        })?;
    let sql = Parser::new(dialect)
        .with_tokens(tokens)
        .parse_expr()
        .map_err(|err| DeltaTableError::GenericError {
            source: Box::new(err),
        })?;

    let context_provider = DeltaContextProvider { state: df_state };
    let sql_to_rel =
        SqlToRel::new_with_options(&context_provider, DeltaParserOptions::default().into());

    Ok(sql_to_rel.sql_to_expr(sql, schema, &mut Default::default())?)
}

I'll work on creating an example shortly

This also leads to a sort of "variant problem" for any code we write that handles Expr.  For example, we have code that walks through an Expr and calculates which columns are referenced.  The "Expr walking" code now has to be aware of both the GetStructField and the ScalarUDF variants of field access.
It would be nicer I think if there was a single canonical way to represent a nested field access in Expr.  For example, maybe the translation from GetStructField to ScalarUDF happens as part of the translation from Expr to PhysicalExpr?  This way Expr only has GetStructField but there is still the flexibility to customize field access?

Here is an example of how to make Expr::struct work in 37.1.0: https://github.com/apache/datafusion/pull/10183
I think we need a better API to do this for real (in 38.0.0 and going forward). I will think about this -- maybe @jayzhan211  has some thoughts

The "Expr walking" code now has to be aware of both the GetStructField and the ScalarUDF variants of field access.

I think this can be controlled by the consumer  -- for example if you are walking Exprs in lancedb, you can control when you transform Expr::GetStructField into ScalarUDF and depending on where you do your analysis you only have to check for one
I think the idea is people might want to override Expr::GetStructFields semantics and they way they would do so is to rewrite it into a different function. I think this is especially compelling for supporting JSON/JSONB for example

I think this can be controlled by the consumer -- for example if you are walking Exprs in lancedb, you can control when you transform Expr::GetStructField into ScalarUDF and depending on where you do your analysis you only have to check for one

I agree (and thanks for the example).  This works for us.  All of our expr from the user start as &str.  However, if we were receiving Expr directly then it wouldn't work because we wouldn't know which approach the user used.  This is not a problem for us, we aren't expecting the user to provide direct DF structs in any of our roadmap features, I'm just thinking through possibilities.

I think the idea is people might want to override Expr::GetStructFields semantics and they way they would do so is to rewrite it into a different function. I think this is especially compelling for supporting JSON/JSONB for example

What's the motivation for doing this at the logical level instead of doing this as part of the conversion from logical to physical?

What's the motivation for doing this at the logical level instead of doing this as part of the conversion from logical to physical?

I don't think there is any particular motivation (or any reason that the conversion needs to be done at either spot) 🤔

I don't think there is any particular motivation (or any reason that the conversion needs to be done at either spot) 🤔

I think, for me, it's just a general unease with having multiple ways of expressing the same thing.  I feel like this can lead to "implicit layers" of the plan.  For example, there is already some notion of "parse plan", "unoptimized logical plan" and "optimized logical plan", and "physical plan".  The middle two are both represented by Expr which can be subtle.  Do we now add "rewritten logical plan" to the list?  Or maybe "rewritten" and "simplified" are just very transient states between "unoptimized" and "optimized"  and I am blowing things out of proportion.
Another way to tackle it could be to leave the concept of a GetIndexedField node at the parsing layer and pull it out of Expr (or deprecate).  This would force the conversion to be done between the parse plan and the logical plan.
That being said, my needs are met (thanks again for your help), and perfect is the enemy of the good, so I'm happy to leave well enough alone.

done between the parse plan and the logical plan

I had also thought about deprecate Expr and use functions directly in parsing phase. I think it might be a good idea. The downside of the current impl is that one needs to register function rewrite rule to convert Expr to Function. I think Register is better not a neccessary step for default behavior. If we have functions in parsing phase, no additional step (register rewrite) is needed.

What's the motivation for doing this at the logical level instead of doing this as part of the conversion from logical to physical?

One good reason is that we don't need physical expr if we converted it to functions in logical level. I think the early we optimize, the less duplicated things we leave.

Another way to tackle it could be to leave the concept of a GetIndexedField node at the parsing layer and pull it out of Expr (or deprecate). This would force the conversion to be done between the parse plan and the logical plan.

I agree this would be clearest (basically remove Expr::GetIndexedField and always use a function.

I think we are talking about better API design of get_field right?

I was actually thinking about an API for the usecase of "I created an Expr programatically and I want to convert it to a PhyscalExpr I can execute. We do this in InfluxDB IOx. I didn't realize that both LanceDB and Delta did the same thing
Currently there is an example of how to do this in expr_api.rs:
https://github.com/apache/datafusion/blob/711239fb13998fa5cfb1b1c865ac060613bad2c7/datafusion-examples/examples/expr_api.rs#L251-L252
Maybe it is time to "promote" that function to a real API in datafusion-core somewhere where it can be tested and better documented.

I think we can take reference from others.
Duckdb has struct_extract and map_extract

Those are interesting functions -- now that we have the notion of function packages, adding better support for the Map datatype would be sweet.

I am thinking I'll try and make a PR with such an API over the next day or two to see how it might look

@alamb just checking in, is there something we need to refactor? Or are you simplifying the API and automatically handle this within datafusion on all code paths?

@alamb just checking in, is there something we need to refactor? Or are you simplifying the API and automatically handling this within datafusion on all code paths?

I suggest for the time being you use the pattern here (is this possible)?

Here is an example of how to make Expr::struct work in 37.1.0: #10183

For the next release (38.0.0) I was thinking would move the create_physical_expr function into the core (rather than an example).
Thoughts?

I am not entirely sure, in delta-rs we just parse SQL strings into logical exprs and then convert it to physical exprs without doing any adjustments there.
I think it makes sense that these expr conversion are automatically done in core. Because logically between v36, 37 nothing changed in the intent of the operation. That the physical impl is different should be abstracted.

I am not entirely sure, in delta-rs we just parse SQL strings into logical exprs and then convert it to physical exprs without doing any adjustments there.

I think in 37.0.0 you'll need to update the code that converts to a physical expr to call the code in https://github.com/apache/datafusion/pull/10183

I think it makes sense that these expr conversion are automatically done in core.

Yes I agree. I will make a PR over the next few days with a proposed API

Here is a PR with a proposed new API: https://github.com/apache/datafusion/pull/10330

authors: alamb ion-elgreco jayzhan211 westonpace

repo: apache/datafusion


text:
Create memory table with target partitions

Describe the bug
When we create table inside slt, it is created with MemoryExec (Memory table). But it seems like the partitions is always one. If we want to create table with multiple partitions, we need to create multiple table and union with them.
To Reproduce

statement ok
create table t1(a int, b varchar) as values
(1, 'a'),
(1, 'a'),
(1, 'a');

statement ok
create table t2(a int, b varchar) as values
(2, 'b'),
(2, 'b'),
(2, 'b');

query TI
With T as
 (select * from t1 UNION ALL select * from t2)
select b, sum(DISTINCT a) from T group by b;
----
a 1
b 2

query TT
explain With T as
 (select * from t1 UNION ALL select * from t2)
select b, sum(DISTINCT a) from T group by b;
----
logical_plan
01)Projection: t.b, sum(alias1) AS sum(DISTINCT t.a)
02)--Aggregate: groupBy=[[t.b]], aggr=[[sum(alias1)]]
03)----Aggregate: groupBy=[[t.b, CAST(t.a AS Int64) AS alias1]], aggr=[[]]
04)------SubqueryAlias: t
05)--------Union
06)----------TableScan: t1 projection=[a, b]
07)----------TableScan: t2 projection=[a, b]
physical_plan
01)ProjectionExec: expr=[b@0 as b, sum(alias1)@1 as sum(DISTINCT t.a)]
02)--AggregateExec: mode=FinalPartitioned, gby=[b@0 as b], aggr=[sum(alias1)]
03)----CoalesceBatchesExec: target_batch_size=8192
04)------RepartitionExec: partitioning=Hash([b@0], 4), input_partitions=4
05)--------AggregateExec: mode=Partial, gby=[b@0 as b], aggr=[sum(alias1)]
06)----------AggregateExec: mode=FinalPartitioned, gby=[b@0 as b, alias1@1 as alias1], aggr=[]
07)------------CoalesceBatchesExec: target_batch_size=8192
08)--------------RepartitionExec: partitioning=Hash([b@0, alias1@1], 4), input_partitions=4
09)----------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2
10)------------------AggregateExec: mode=Partial, gby=[b@1 as b, CAST(a@0 AS Int64) as alias1], aggr=[]
11)--------------------UnionExec
12)----------------------MemoryExec: partitions=1, partition_sizes=[1]
13)----------------------MemoryExec: partitions=1, partition_sizes=[1]

statement ok
set datafusion.execution.target_partitions = 2

statement ok
create table t(a int, b varchar) as values
(1, 'a'),
(1, 'a'),
(1, 'a'),
(2, 'b'),
(2, 'b'),
(2, 'b');

query TI
select b, sum(DISTINCT a) from T group by b;
----
b 2
a 1

query TT
explain select b, sum(DISTINCT a) from T group by b;
----
logical_plan
01)Projection: t.b, sum(alias1) AS sum(DISTINCT t.a)
02)--Aggregate: groupBy=[[t.b]], aggr=[[sum(alias1)]]
03)----Aggregate: groupBy=[[t.b, CAST(t.a AS Int64) AS alias1]], aggr=[[]]
04)------TableScan: t projection=[a, b]
physical_plan
01)ProjectionExec: expr=[b@0 as b, sum(alias1)@1 as sum(DISTINCT t.a)]
02)--AggregateExec: mode=FinalPartitioned, gby=[b@0 as b], aggr=[sum(alias1)]
03)----CoalesceBatchesExec: target_batch_size=8192
04)------RepartitionExec: partitioning=Hash([b@0], 2), input_partitions=2
05)--------AggregateExec: mode=Partial, gby=[b@0 as b], aggr=[sum(alias1)]
06)----------AggregateExec: mode=FinalPartitioned, gby=[b@0 as b, alias1@1 as alias1], aggr=[]
07)------------CoalesceBatchesExec: target_batch_size=8192
08)--------------RepartitionExec: partitioning=Hash([b@0, alias1@1], 2), input_partitions=2
09)----------------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
10)------------------AggregateExec: mode=Partial, gby=[b@1 as b, CAST(a@0 AS Int64) as alias1], aggr=[]
11)--------------------MemoryExec: partitions=1, partition_sizes=[1]   <- It is still 1

Expected behavior
I hope we can create arbitrary partitions with setting instead of creating the test with union.
Additional context
I think batch size doesn't not change the MemoryExec too, but I'm not sure about the intended behaviour

take

I tried to implement this in a naive way by just adding the following code:
let target_partitions = self.state().config_options().execution.target_partitions;

let physical = DataFrame::new(self.state(), input);
let batches: Vec<_> = physical
    .repartition(Partitioning::RoundRobinBatch(target_partitions))?
    .collect_partitioned().await?;

to https://github.com/apache/datafusion/blob/fc703238b1d7794bd132a7fb6b97cad9ba4c7446/datafusion/core/src/execution/context/mod.rs#L795
But I found that doing Repartition on top of a simple Projection gets overridden by physical optimization: https://github.com/apache/datafusion/blob/fc703238b1d7794bd132a7fb6b97cad9ba4c7446/datafusion/core/src/physical_optimizer/enforce_distribution.rs#L975
PS: When I skip remove_dist_changing_operators, the behavior still looks weird. The number of partitions matches target_partitions, but the data isn’t distributed properly. There’s one partition with all the data and others are empty.
@jayzhan211 Would appreciate any ideas.

I'm thinking about make ValuesExec partitioned

Unassigning, no longer working on this.

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Migrate Regex Functions from static docs

Which issue does this PR close?

Closes #12868 .
Rationale for this change
Migrated regex functions.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Beyond fixing the copy/paste error for the get_regexp_replace_doc fn name lgtm.

source: gharchive/pull-request


text:
[Feature][Task-plugin] Add OpenMLDB task-plugin

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
I want to add the OpenMLDB task-plugin to support run cmds on OpenMLDB.
There're some ways to impl it.

If we impl it like zeppelin, we'll add openmldb jdbc dependency. openmldb jdbc will use openmldb-native jar which has c++ shared lib in it. The released package will contain it. openmldb-native jar is 156.9MB, a little big.
If we impl it like datax or python task, we can let users to install openmldb python sdk later, won't add it in releases. And python wheel is smaller. 23MB

Plz help me decide on which way to impl the task.
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi @vagetablechicken , it is welcome to add any task to DolphinScheduler, but this case it is a little different because the java SDK for OpenMLDB is a little big IMO. After we release version 3.0.0-alpha, we find out some of our users complain about the size of our tarball. and we have created some issues and do some effort to slim our binary package which you can see in #9813 and #9843.
So If those are the only two ways to add OpenMLDB task to DolphinScheduler, I prefer the second way, because it will not make our tarball too large. But it have to ask users to do more prepare job before they user OpenMLDB(which not out-of-box). WDYT @caishunfeng @lenboo @zhuangchong @SbloodyS

Hi @vagetablechicken , it is welcome to add any task to DolphinScheduler, but this case it is a little different because the java SDK for OpenMLDB is a little big IMO. After we release version 3.0.0-alpha, we find out some of our users complain about the size of our tarball. and we have created some issues and do some effort to slim our binary package which you can see in #9813 and #9843.
So If those are the only two ways to add OpenMLDB task to DolphinScheduler, I prefer the second way, because it will not make our tarball too large. But it have to ask users to do more prepare job before they user OpenMLDB(which not out-of-box). WDYT @caishunfeng @lenboo @zhuangchong @SbloodyS

Agreed.

authors: SbloodyS vagetablechicken zhongjiajie

repo: apache/dolphinscheduler


text:
[WIP][DSIP-69] Fix master dispatch task timeout might cause task dupl…

…icate running in worker

Purpose of the pull request
fix: https://github.com/apache/dolphinscheduler/issues/16481
Brief change log

Verify this pull request


text:
[python] Add example about how to bulk create workflow

Purpose of the pull request
Add example to show how to bulk create workflow
Verify this pull request

This pull request is code cleanup without any test coverage. I just an example

PTAL, @lenboo @CalvinKirs . I add example about how to create workflow in batch mode

Codecov Report

Merging #6659 (46d7ea7) into dev (3a8b809) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #6659      +/-   ##
============================================
- Coverage     38.49%   38.48%   -0.01%
- Complexity     3223     3226       +3
============================================
  Files           646      646
  Lines         25874    25886      +12
  Branches       2805     2808       +3
============================================
+ Hits           9959     9963       +4
- Misses        15002    15008       +6
- Partials        913      915       +2

Impacted Files
Coverage Δ

...org/apache/dolphinscheduler/remote/utils/Host.java
40.00% <0.00%> (-2.11%)
:arrow_down:

...pache/dolphinscheduler/dao/upgrade/UpgradeDao.java
4.21% <0.00%> (-0.05%)
:arrow_down:

...scheduler/dao/upgrade/DolphinSchedulerManager.java
0.00% <0.00%> (ø)

...e/dolphinscheduler/remote/NettyRemotingClient.java
53.52% <0.00%> (+1.40%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a8b809...46d7ea7. Read the comment docs.

+1


text:
Release Note 2.0.8

Previous Release Note 2.0.7
Thanks to our community users and developers, about 65 improvements and bug fixes have been made in Doris 2.0.8 version.
1 Behavior change

The ADMIN SHOW statement can not be executed with high version of mysql 8.x jdbc driver. So rename these statement, remove the ADMIN keywords.

https://github.com/apache/doris/pull/29492

ADMIN SHOW CONFIG -> SHOW CONFIG
ADMIN SHOW REPLICA -> SHOW REPLICA
ADMIN DIAGNOSE TABLET -> SHOW TABLET DIAGNOSIS
ADMIN SHOW TABLET -> SHOW TABLET

2 New features
N/A
3 Improvement and optimizations

make inverted index work with topn opt in Nereids
Limit the max string length to 1024 while collecting column stats to control BE memory usage
jdbc catalog close when jdbc client is not empty
Accept all iceberg database and do not check the name format of database
Refresh external table's rowcount async to avoid cache miss and unstable query plan
Simplify the isSplitable method of hive external table to avoid too many hadoop metrics

See the complete list of improvements and bug fixes on github .
Big Thanks
Thanks all who contribute to this release:
@924060929
@AcKing-Sam
@amorynan
@AshinGau
@BePPPower
@BiteTheDDDDt
@ByteYue
@cambyzju
@dongsilun
@eldenmoon
@feiniaofeiafei
@gnehil
@Jibing-Li
@liaoxin01
@luwei16
@morningman
@morrySnow
@mrhhsg
@Mryange
@nextdreamblue
@platoneko
@starocean999
@SWJTU-ZhangLei
@wuwenchi
@xiaokang
@xinyiZzz
@Yukang-Lian
@Yulei-Yang
@zclllyybb
@zddr
@zhangstar333
@zhiqiang-hhhh
@ziyanTOP
@zy-kkk
@zzzxl1993

中文版Release Note如下：
1 行为变更
由于ADMIN SHOW xx语句在mysql 8.x jdbc driver不能执行，所以将名字改成SHOW xx

https://github.com/apache/doris/pull/29492

ADMIN SHOW CONFIG -> SHOW CONFIG
ADMIN SHOW REPLICA -> SHOW REPLICA
ADMIN DIAGNOSE TABLET -> SHOW TABLET DIAGNOSIS
ADMIN SHOW TABLET -> SHOW TABLET

2 新功能
N/A
3 改进和优化
新优化器支持topn优化中使用倒排索引
限制统计信息string长度为1024以控制be内存消耗
修复未创建 jdbc client 时意外关闭的情况
接受所有iceberg database，不再做额外的名字检查
异步更新外表行数统计，避免同步更新带来的cache miss和plan不稳定
简化hive外表的isSplitable方法，避免过多的hadoop metric

repo: apache/doris


text:
[Proposal] Memory performance optimization

Motivation

High concurrent query performance is poor, among which Spin Lock in tcmalloc has a large CPU ratio;
The cache utilization in PageCache, MemPool, and Vec Allocator needs to be further improved;
Multiple memory allocators from Impala, ClickHouse, and Doris coexist, which leads to optimization difficulties;
(Similar problems exist while turning on or off the vectorization engine)

Expected Return

Unified memory allocator;
More memory reuse;
Improve cache hit rate and reduce cache size;
Try to introduce other memory frameworks such as JeMalloc;
Reduce additional costs such as mem tracker;

Optimization direction

Stress testing on more test sets to find memory hotspots;

#9581

Test the performance of multiple memory allocators currently existing in Doris in different scenarios, unify them;
Try to introduce tests such as JeMalloc;

Unify memory allocator
Row execution engine memory allocator
From Impala:

More previous research: https://shimo.im/docs/DXvQPcqPH3TyR6pY
Vectorized execution engine memory allocator
From ClickHouse:

In the future

Things to do

Arena is equivalent to MemPool, unify them;
SystemAllocator is equivalent to CK Allocator, unify them, and consider memory alignment, consider the use of MMap and HugePage;
Compare the performance of Malloc, ChunkAllocator, and MMap allocating memory with different lengths at different frequencies. Combine their strengths.
When removing the line storage execution engine, remove BufferAllocator, BufferPool, and SubAllocator at the same time;
Consider the implementation of spill disk.

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Has been completed


text:
bugfix fix segment writer finalize footer twice

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.32 seconds
load time: 565 seconds
storage size: 17121413068 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230128044859_clickbench_pr_85347.html

authors: hello-stephen yixiutt


text:
opt turnoff pipeline when dml temporary

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.25 seconds
stream load tsv:          517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162254830 Bytes

authors: hello-stephen morrySnow


text:
Revert "ChoreTemporarily cancel the mandatory restrictions of shellCheck"

Reverts apache/doris#24765
It's done

run buildall


text:
fix disable forbid_unknown_col_stats

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix fix performance problem caused by too many virtual function call

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run clickbench

run buildall

run clickbench

run buildall

run buildall

run clickbench

run buildall

authors: jacktengg yiguolei


text:
fix Fix regression run in cloud use non-root user #34340

cherry pick from #34340

run buildall

run buildall

run buildall


text:
[Feature](inverted index) add no need read data optimize config #36686

cherry pick from #36686

run buildall

run buildall


text:
[Opt] (multi-catalog) opt max scanner thread number in batch split mode.

What problem does this PR solve?
Problem Summary:
There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by query_parallel_instance_num, resulting in poor performance of batch split mode.
Release note
There's only one scan range for each backend in batch split mode. Each backend only starts up one ScanNode instance. However, when calculating the concurrency of scanners in the scan operator in batch split mode, it is not divided by 1, but by query_parallel_instance_num, resulting in poor performance of batch split mode.
Fix calculation of maximum number of scanner threads in batch split mode.
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[x] Manual test (add detailed scripts or steps below)

Create a hive partition table which contains many hive partitions(> 1024);
Execute select * from par_table in doris.
Open the profile to view the number of file scan operators and the number of MaxScannerThreadNum.
You should be able to see that each be node of the scan operator is 1, and there will be more MaxScannerThreadNum in the scan operator.

[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall


text:
DRILL-6561: Lateral excluding the columns from output container provided by projection push into rules

This PR is dependent upon DRILL-6545 and has 2 commits from it for compilation. Once that is merged in this PR has to be rebased on master and then checked-in.
@parthchandra - Please review it.

+1 .
I also took care of the rebase and merge

authors: parthchandra sohami

repo: apache/drill


text:
Pull-deps extremely slow and not working in docker build

Affected Version
0.22.1
Description
I was trying to add the prometheus-emitter via the pull-deps command:
java -classpath "/opt/druid/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:prometheus-emitter:${DRUID_RELEASE}
$DRUID_RELEASE was set to 0.22.1 via Dockerfile ARG.
Also tried this in a separate docker container apache/druid:0.22.1 and executed java -classpath "/opt/druid/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:prometheus-emitter:0.22.1.
The execution takes ~5 minutes each time and in the docker build process it does not even seem to install the extension.
time java -classpath "/opt/druid/lib/*" org.apache.druid.cli.Main tools pull-deps -c org.apache.druid.extensions.contrib:prometheus-emitter:0.22.1
real    4m 45.06s
user    0m 11.18s
sys     0m 0.89s

As i don't know much about pull-deps, i can't debug this and the documentation does not seem to include a debug or verbosity flag which could help me track down what is taking so long.
I couldn't find a documentation on how to install the specific plugin either, so i assembled the path module/coordinate org.apache.druid.extensions.contrib:prometheus-emitter myself by browsing the maven repo.
Would be great to know if the slowlyness is a mistake by me, if it is the tool or a temporary issue (though i could download the jar of the extensions really fast as it is less 100kb).

I guess this problem is due to slow response from mvn repository. Have you tried the following options of pull-deps tool to use a mirror mvn repository?
https://druid.apache.org/docs/latest/operations/pull-deps.html

-r or --remoteRepository
Add a remote repository. Unless --no-default-remote-repositories is provided, these will be used after https://repo1.maven.org/maven2/ and http://metamx.artifactoryonline.com/metamx/pub-libs-releases-local
--no-default-remote-repositories
Don't use the default remote repositories, only use the repositories provided directly via --remoteRepository.

Meanwhile, you can add the jvm argument -Dlog4j2.level=INFO to the command line to see the logs from the console, so that you can check which step is slow.

Thank you for this great hint.
Now it only takes 20 seconds (which in tech is still a lot), but a lot faster than ~5 minutes!
By adding the suggested log  level parameter and adjusting it to debug, i could see that the pull-deps command is complainig about resources not existing.
i am now running java -Dlog4j2.level=DEBUG -classpath "/opt/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --no-default-hadoop -c org.apache.druid.extensions.contrib:prometheus-emitter:0.22.1 and when grepping this for error, i get:
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/org/apache/apache/21/apache-21.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/com/fasterxml/jackson/jackson-bom/2.10.5.20201202/jackson-bom-2.10.5.20201202.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/com/fasterxml/jackson/jackson-parent/2.10/jackson-parent-2.10.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/com/fasterxml/oss-parent/38/oss-parent-38.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/netty/netty-bom/4.1.68.Final/netty-bom-4.1.68.Final.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/org/sonatype/oss/oss-parent/7/oss-parent-7.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.pom. Error code 404
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.pom. Error code 501, HTTPS Required
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/parent/0.7.0/parent-0.7.0.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/parent/0.7.0/parent-0.7.0.pom. Error code 404
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/parent/0.7.0/parent-0.7.0.pom. Error code 501, HTTPS Required
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.pom. Error code 404
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.pom. Error code 501, HTTPS Required
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.pom. Error code 404
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.pom. Error code 501, HTTPS Required
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.pom. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.pom. Error code 404
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.pom. Error code 501, HTTPS Required
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.apache.org/snapshots/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.jar. Error code 404
io.tesla.aether.connector.ResourceDoesNotExistException: Unable to locate resource https://repository.jboss.org/nexus/content/repositories/thirdparty-uploads/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.jar. Error code 404
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.jar. Error code 501, HTTPS Required
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.jar. Error code 501, HTTPS Required
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.jar. Error code 501, HTTPS Required
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.jar. Error code 501, HTTPS Required

I'm confused about multiple things: How did nobody ever notice that resources seem to have got lost and why would it try http in times of https?
I don't know if any of those questions really relate for pull-deps or if that is to be asked somewhere else to be honest.

Use the -r option I mentioned above to download artifacts from https://repo1.maven.org/maven2/

Even when doing that, i still see these via debug logs:
Downloading artifact io.prometheus:simpleclient:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
15:30:04.524 [main] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector AetherRepositoryConnector with priority 3.4028235E38 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.pom
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.pom. Error code 501, HTTPS Required
Downloaded artifact io.prometheus:simpleclient:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:parent:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
15:30:05.687 [main] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector AetherRepositoryConnector with priority 3.4028235E38 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/parent/0.7.0/parent-0.7.0.pom
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/parent/0.7.0/parent-0.7.0.pom. Error code 501, HTTPS Required
Downloaded artifact io.prometheus:parent:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient_httpserver:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
15:30:06.753 [main] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector AetherRepositoryConnector with priority 3.4028235E38 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.pom
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.pom. Error code 501, HTTPS Required
Downloaded artifact io.prometheus:simpleclient_httpserver:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient_common:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
15:30:07.814 [main] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector AetherRepositoryConnector with priority 3.4028235E38 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.pom
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.pom. Error code 501, HTTPS Required
Downloaded artifact io.prometheus:simpleclient_common:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient_pushgateway:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
15:30:08.991 [main] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector AetherRepositoryConnector with priority 3.4028235E38 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.pom
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.pom. Error code 501, HTTPS Required
Downloaded artifact io.prometheus:simpleclient_pushgateway:pom:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient_httpserver:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient_common:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloading artifact io.prometheus:simpleclient_pushgateway:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
15:30:12.988 [main] DEBUG org.eclipse.aether.internal.impl.DefaultRepositoryConnectorProvider - Using connector AetherRepositoryConnector with priority 3.4028235E38 for http://repo.maven.apache.org/maven2
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.jar
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient/0.7.0/simpleclient-0.7.0.jar. Error code 501, HTTPS Required
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.jar
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_httpserver/0.7.0/simpleclient_httpserver-0.7.0.jar. Error code 501, HTTPS Required
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.jar
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_common/0.7.0/simpleclient_common-0.7.0.jar. Error code 501, HTTPS Required
Downloading: http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.jar
io.tesla.aether.connector.TransferException: Failed to transfer http://repo.maven.apache.org/maven2/io/prometheus/simpleclient_pushgateway/0.7.0/simpleclient_pushgateway-0.7.0.jar. Error code 501, HTTPS Required
Downloaded artifact io.prometheus:simpleclient:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloaded artifact io.prometheus:simpleclient_httpserver:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloaded artifact io.prometheus:simpleclient_common:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)
Downloaded artifact io.prometheus:simpleclient_pushgateway:jar:0.7.0 from central (http://repo.maven.apache.org/maven2, releases)

My command executed was:
java -Dlog4j2.level=DEBUG -classpath "/opt/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --no-default-remote-repositories -r https://repo1.maven.org/maven2/ --no-default-hadoop -c org.apache.druid.extensions.contrib:prometheus-emitter:0.22.1

The pull-deps tool uses the io.tesla.aether to pull the artifacts from repostory. It seems that this lib will try to pull artifacts from http://repo.maven.apache.org/maven2. I don't know where this repo is from, maybe it is coded inside the lib.
I don't have much more energy to check that. You can check the code at src/main/java/org/apache/druid/cli/PullDependencies.java to see how the tool uses io.tesla.aether.

Ok, since i am not a java developer (and not into developing druid), i would leave this TBD for someone else.

I am seeing the same issue, when we try to download org.apache.druid.extensions.contrib:statsd-emitter:0.22.1,  org.apache.hadoop:hadoop-aws:2.8.5 and com.amazonaws:aws-java-sdk-s3:1.11.479
For now, we have manually downloaded the required jars and moving ahead. It would be nice, to get this resolved sometime soon.

The seems to resolve itself, once I upgrade to latest jdk-1.8 -- jdk-8u311-linux-x64.rpm. I was previously using, jdk-8u211-linux-x64.rpm

i can confirm that the docker image of 0.22.1 uses this version which is below your newly used version: 1.8.0_275-8u275-b01-1~deb9u1-b01 and that matches with the current distroless java version. Guess we'll have to wait until google updates it to have faster downloads or use a multistage build and copy the jars over.

repo: apache/druid


text:
Exception when running the dump-segment tool

Affected Version
25.0.0
Description
Hi,
I'm trying to run the 'dump-segment' tool, and getting the following exception -
# java --add-opens java.base/java.lang=ALL-UNNAMED -classpath "/opt/apache-druid-25.0.0/lib/*"  -Ddruid.extensions.loadList="[]" org.apache.druid.cli.Main tools dump-segment -d /tmp/segment/
Exception in thread "main" java.lang.IllegalAccessError: class org.apache.datasketches.memory.internal.AccessByteBuffer (in unnamed module @0x1dde4cb2) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x1dde4cb2
        at org.apache.datasketches.memory.internal.AccessByteBuffer.<init>(AccessByteBuffer.java:66)
        at org.apache.datasketches.memory.internal.BaseWritableMemoryImpl.wrapByteBuffer(BaseWritableMemoryImpl.java:112)
        at org.apache.datasketches.memory.Memory.wrap(Memory.java:66)
        at org.apache.druid.segment.data.LongsLongEncodingReader.<init>(LongsLongEncodingReader.java:33)
        at org.apache.druid.segment.data.CompressionFactory$LongEncodingFormat$3.getReader(CompressionFactory.java:186)
        at org.apache.druid.segment.data.CompressionFactory.getLongSupplier(CompressionFactory.java:315)
        at org.apache.druid.segment.data.CompressedColumnarLongsSupplier.fromByteBuffer(CompressedColumnarLongsSupplier.java:117)
        at org.apache.druid.segment.serde.LongNumericColumnPartSerde.lambda$getDeserializer$0(LongNumericColumnPartSerde.java:100)
        at org.apache.druid.segment.column.ColumnDescriptor.read(ColumnDescriptor.java:111)
        at org.apache.druid.segment.IndexIO$V9IndexLoader.deserializeColumn(IndexIO.java:796)
        at org.apache.druid.segment.IndexIO$V9IndexLoader.registerColumnHolder(IndexIO.java:771)
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:632)
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:202)
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:192)
        at org.apache.druid.cli.DumpSegment.run(DumpSegment.java:199)
        at org.apache.druid.cli.Main.main(Main.java:112)

My Java version is  -
# java --version
openjdk 17.0.5 2022-10-18
OpenJDK Runtime Environment (build 17.0.5+8-Ubuntu-2ubuntu120.04)
OpenJDK 64-Bit Server VM (build 17.0.5+8-Ubuntu-2ubuntu120.04, mixed mode, sharing)

(even though I tried also Java 19 on Windows, and got the same exception...)
Am I missing something? Or maybe some change broke the dump-segment tool?
Thanks!
Eran

Tested with an older version now - 0.16.1 (from https://archive.apache.org/dist/incubator/druid/), and it works fine.


text:
Replace EOL javaee/jaxb-api with jarkartaee/jaxb-api

Description

Druid is using deprecated dependency - https://github.com/javaee/jaxb-spec
The dependency is used at following places:
services/pom.xml
pom.xml
extensions-core/orc-extensions/pom.xml
core/pom.xml

Proposed Fix

Move to jarkartaee/jaxb-api 2.3.3. to avoid any breaking changes

Working on it.


text:
type设置为time时，interval设置时间戳无效

希望能写详细些，interval要如何设置才能准确分割x轴。
我设置min是new Date(2016, 0, 1, 0, 0, 0)，max是new Date(2016, 0, 5, 0, 0, 0)时，希望能按照天分割x轴，但是设置splitNumber为4时不能平均分割x轴。
然后使用interval，无论是精确到秒的时间戳，还是精确到毫秒的时间戳，都无效。

new Date('2016-7-3 8:40:52')

最新版本：type设置为time时，interval设置时间戳无效
我想设置根据每个月进行分割x轴，但是无法实现，请问那里出现了问题
xAxis: { type: 'time', interval: 2592000000, min: new Date('2016-7-3 8:40:52'), max: new Date('2021-7-3 8:40:52'), axisLabel: { show: true, formatter: '{yyyy}.{MM}' }, splitNumber: false, splitLine: { show: true }, axisTick: { show: true }},

option = {
"animation": false,
"grid": {
"bottom": 0,
"left": 0,
"right": 0,
"top": 0
},
"series": [
{
"S": 12,
"animation": false,
"color": "#E7086E",
"colorDict": "#E7086E",
"connectNulls": false,
"data": [],
"id": "白细胞计数__WN_ICIS_INSPECT_1011001__",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "line",
"max": 30,
"maxValue": 1,
"min": 8,
"minValue": 0,
"moniitemCode": "WN_ICIS_INSPECT_1011001",
"moniitemDesc": "",
"moniitemName": "白细胞计数",
"name": "白细胞计数",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M10.3,1.5c-4.7,0-8.5,3.8-8.5,8.5c0,4.7,3.8,8.5,8.5,8.5s8.5-3.8,8.5-8.5C18.8,5.3,14.9,1.5,10.3,1.5z M10.3,17.4 c-4.1,0-7.5-3.3-7.5-7.5s3.3-7.5,7.5-7.5c4.2,0,7.4,3.4,7.4,7.5S14.4,17.4,10.3,17.4z M12.1,6.1H8.4c-1.1,0-2,0.9-2,2v3.7 c0,1.1,0.9,2,2,2h3.7c1.1,0,2-0.9,2-2V8C14.1,7,13.2,6.1,12.1,6.1z",
"symbolDict": "M10.3,1.5c-4.7,0-8.5,3.8-8.5,8.5c0,4.7,3.8,8.5,8.5,8.5s8.5-3.8,8.5-8.5C18.8,5.3,14.9,1.5,10.3,1.5z M10.3,17.4 c-4.1,0-7.5-3.3-7.5-7.5s3.3-7.5,7.5-7.5c4.2,0,7.4,3.4,7.4,7.5S14.4,17.4,10.3,17.4z M12.1,6.1H8.4c-1.1,0-2,0.9-2,2v3.7 c0,1.1,0.9,2,2,2h3.7c1.1,0,2-0.9,2-2V8C14.1,7,13.2,6.1,12.1,6.1z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 1,
"yaxis": "1",
"z": 3
},
{
"S": 12,
"animation": false,
"color": "#E7086E",
"colorDict": "#E7086E",
"connectNulls": false,
"data": [],
"id": "体温__WN_ICIS_MONI_10010101__T",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "line",
"max": 30,
"maxValue": 40,
"min": 8,
"minValue": 35,
"moniitemCode": "WN_ICIS_MONI_10010101",
"moniitemDesc": "T",
"moniitemName": "体温",
"name": "体温",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M10.3,1.5c-4.7,0-8.5,3.8-8.5,8.5c0,4.7,3.8,8.5,8.5,8.5s8.5-3.8,8.5-8.5C18.8,5.3,14.9,1.5,10.3,1.5z M10.3,17.4 c-4.1,0-7.5-3.3-7.5-7.5s3.3-7.5,7.5-7.5c4.2,0,7.4,3.4,7.4,7.5S14.4,17.4,10.3,17.4z M12.1,6.1H8.4c-1.1,0-2,0.9-2,2v3.7 c0,1.1,0.9,2,2,2h3.7c1.1,0,2-0.9,2-2V8C14.1,7,13.2,6.1,12.1,6.1z",
"symbolDict": "M10.3,1.5c-4.7,0-8.5,3.8-8.5,8.5c0,4.7,3.8,8.5,8.5,8.5s8.5-3.8,8.5-8.5C18.8,5.3,14.9,1.5,10.3,1.5z M10.3,17.4 c-4.1,0-7.5-3.3-7.5-7.5s3.3-7.5,7.5-7.5c4.2,0,7.4,3.4,7.4,7.5S14.4,17.4,10.3,17.4z M12.1,6.1H8.4c-1.1,0-2,0.9-2,2v3.7 c0,1.1,0.9,2,2,2h3.7c1.1,0,2-0.9,2-2V8C14.1,7,13.2,6.1,12.1,6.1z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 2,
"yaxis": "1",
"z": 3
},
{
"S": 12,
"animation": false,
"color": "#FF8C00",
"colorDict": "#FF8C00",
"connectNulls": false,
"data": [],
"id": "收缩压__WN_ICIS_MONI_10010203__无创收缩压",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "vLine",
"max": 30,
"maxValue": 200,
"min": 8,
"minValue": 70,
"moniitemCode": "WN_ICIS_MONI_10010203",
"moniitemDesc": "无创收缩压",
"moniitemName": "收缩压",
"name": "收缩压",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M16.8,9.27v1.05L3.24,17.43V16.1L15.39,9.85V9.77L3.24,3.49V2.16Z",
"symbolDict": "M16.8,9.27v1.05L3.24,17.43V16.1L15.39,9.85V9.77L3.24,3.49V2.16Z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 3,
"yaxis": "1",
"z": 3
},
{
"S": 12,
"animation": false,
"color": "#EB7405",
"colorDict": "#EB7405",
"connectNulls": false,
"data": [],
"id": "舒张压__WN_ICIS_MONI_10010204__无创舒张压",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "vLine",
"max": 30,
"maxValue": 120,
"min": 8,
"minValue": 40,
"moniitemCode": "WN_ICIS_MONI_10010204",
"moniitemDesc": "无创舒张压",
"moniitemName": "舒张压",
"name": "舒张压",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M16.8,2.16V3.49L4.65,9.77v.08L16.8,16.1v1.33L3.24,10.32V9.27Z",
"symbolDict": "M16.8,2.16V3.49L4.65,9.77v.08L16.8,16.1v1.33L3.24,10.32V9.27Z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 4,
"yaxis": "1",
"z": 3
},
{
"S": 12,
"animation": false,
"color": "#7700B2",
"colorDict": "#7700B2",
"connectNulls": false,
"data": [],
"id": "呼吸__WN_ICIS_MONI_10010207__呼吸频率",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "line",
"max": 30,
"maxValue": 25,
"min": 8,
"minValue": 10,
"moniitemCode": "WN_ICIS_MONI_10010207",
"moniitemDesc": "呼吸频率",
"moniitemName": "呼吸",
"name": "呼吸",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M10.7,10.3l6.6,6.6c0.2,0.2,0.2,0.5,0,0.7c-0.2,0.2-0.5,0.2-0.7,0L10,11l-6.6,6.6c-0.2,0.2-0.5,0.2-0.7,0 c-0.2-0.2-0.2-0.5,0-0.7l6.6-6.6L2.7,3.7C2.5,3.5,2.5,3.2,2.7,3c0.2-0.2,0.5-0.2,0.7,0L10,9.6L16.6,3c0.2-0.2,0.5-0.2,0.7,0 c0.2,0.2,0.2,0.5,0,0.7L10.7,10.3z",
"symbolDict": "M10.7,10.3l6.6,6.6c0.2,0.2,0.2,0.5,0,0.7c-0.2,0.2-0.5,0.2-0.7,0L10,11l-6.6,6.6c-0.2,0.2-0.5,0.2-0.7,0 c-0.2-0.2-0.2-0.5,0-0.7l6.6-6.6L2.7,3.7C2.5,3.5,2.5,3.2,2.7,3c0.2-0.2,0.5-0.2,0.7,0L10,9.6L16.6,3c0.2-0.2,0.5-0.2,0.7,0 c0.2,0.2,0.2,0.5,0,0.7L10.7,10.3z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 5,
"yaxis": "1",
"z": 3
},
{
"S": 12,
"animation": false,
"color": "#E7086E",
"colorDict": "#E7086E",
"connectNulls": false,
"data": [],
"id": "血氧饱和度__WN_ICIS_MONI_10040110__SO2",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "line",
"max": 30,
"maxValue": 1,
"min": 8,
"minValue": 0,
"moniitemCode": "WN_ICIS_MONI_10040110",
"moniitemDesc": "SO2",
"moniitemName": "血氧饱和度",
"name": "血氧饱和度",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M10.4,1.5c-4.8,0-8.7,3.9-8.7,8.7s3.9,8.7,8.7,8.7c4.8,0,8.7-4,8.7-8.7C19.1,5.4,15.2,1.5,10.4,1.5z M10.5,2.5 c1.9,0,3.7,0.7,5,1.8l-5.1,5.1l-5-5C6.7,3.2,8.5,2.5,10.5,2.5z M2.8,10.2c0-1.9,0.7-3.7,1.9-5.1l5,5l-5,5 C3.5,13.8,2.8,12.1,2.8,10.2z M5.4,15.9l5-5l5.1,5.1c-1.4,1.1-3.1,1.8-5,1.8C8.6,17.8,6.8,17.1,5.4,15.9z M16.2,15.3l-5.1-5.1 L16.2,5c1.2,1.4,2,3.2,2,5.2C18.2,12.2,17.4,14,16.2,15.3z",
"symbolDict": "M10.4,1.5c-4.8,0-8.7,3.9-8.7,8.7s3.9,8.7,8.7,8.7c4.8,0,8.7-4,8.7-8.7C19.1,5.4,15.2,1.5,10.4,1.5z M10.5,2.5 c1.9,0,3.7,0.7,5,1.8l-5.1,5.1l-5-5C6.7,3.2,8.5,2.5,10.5,2.5z M2.8,10.2c0-1.9,0.7-3.7,1.9-5.1l5,5l-5,5 C3.5,13.8,2.8,12.1,2.8,10.2z M5.4,15.9l5-5l5.1,5.1c-1.4,1.1-3.1,1.8-5,1.8C8.6,17.8,6.8,17.1,5.4,15.9z M16.2,15.3l-5.1-5.1 L16.2,5c1.2,1.4,2,3.2,2,5.2C18.2,12.2,17.4,14,16.2,15.3z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 6,
"yaxis": "1",
"z": 3
},
{
"S": 12,
"animation": false,
"color": "#E7086E",
"colorDict": "#E7086E",
"connectNulls": false,
"data": [],
"id": "空腹血糖__WN_ICIS_MONI_10040202__GLU",
"itemStyle": {
"normal": {}
},
"lineStyle": {
"normal": {
"width": 1
}
},
"lineTypeDict": "point",
"max": 30,
"maxValue": 1,
"min": 8,
"minValue": 0,
"moniitemCode": "WN_ICIS_MONI_10040202",
"moniitemDesc": "GLU",
"moniitemName": "空腹血糖",
"name": "空腹血糖",
"observeId": "100001",
"smooth": 0,
"statusDict": "1",
"symbol": "path://M10.5,9.5L3.2,2.1L4,1.3l6.5,6.5L17,1.3l0.8,0.8L10.5,9.5z M3.2,17.9L4,18.8l6.5-6.5l6.5,6.5l0.8-0.8l-7.4-7.4L3.2,17.9z",
"symbolDict": "M10.5,9.5L3.2,2.1L4,1.3l6.5,6.5L17,1.3l0.8,0.8L10.5,9.5z M3.2,17.9L4,18.8l6.5-6.5l6.5,6.5l0.8-0.8l-7.4-7.4L3.2,17.9z",
"symbolSize": 8,
"type": "line",
"units": "bpm",
"wardCode": "1876391",
"yAxisIndex": 7,
"yaxis": "1",
"z": 3
}
],
"tooltip": {
"trigger": "item"
},
"xAxis": [
{
"axisLabel": {
"interval": 0,
"rotate": 15,
"show": false
},
"axisLine": {
"onZero": false,
"show": false
},
"axisTick": {
"show": false
},
"boundaryGap": false,
"interval": 60,
"max": 1665252420,
"min": 1665251580,
"splitLine": {
"lineStyle": {
"color": "#000000",
"opacity": 1,
"type": "dotted"
},
"show": true
},
"type": "time"
}
],
"yAxis": [
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 21.818181818181817,
"max": 240,
"min": 0,
"splitLine": {
"lineStyle": {
"color": "#000000",
"opacity": 1,
"type": "dotted"
},
"show": true
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 1,
"min": 0,
"moniitemCode": "WN_ICIS_INSPECT_1011001",
"moniitemName": "白细胞计数",
"splitLine": {
"show": false
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 40,
"min": 35,
"moniitemCode": "WN_ICIS_MONI_10010101",
"moniitemName": "体温",
"splitLine": {
"show": false
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 200,
"min": 70,
"moniitemCode": "WN_ICIS_MONI_10010203",
"moniitemName": "收缩压",
"splitLine": {
"show": false
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 120,
"min": 40,
"moniitemCode": "WN_ICIS_MONI_10010204",
"moniitemName": "舒张压",
"splitLine": {
"show": false
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 25,
"min": 10,
"moniitemCode": "WN_ICIS_MONI_10010207",
"moniitemName": "呼吸",
"splitLine": {
"show": false
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 1,
"min": 0,
"moniitemCode": "WN_ICIS_MONI_10040110",
"moniitemName": "血氧饱和度",
"splitLine": {
"show": false
},
"type": "value"
},
{
"axisLabel": {
"show": false
},
"axisLine": {
"show": false
},
"axisTick": {
"show": false
},
"interval": 1,
"max": 1,
"min": 0,
"moniitemCode": "WN_ICIS_MONI_10040202",
"moniitemName": "空腹血糖",
"splitLine": {
"show": false
},
"type": "value"
}
]
}
interval:60无效，v4没问题 v5.4.2不行

authors: JIANGQIAN3333 daoyuanjiao ghost yjxhome

repo: apache/echarts


text:
x轴和y轴处不应该画分隔线splitLine

One-line summary [问题简述]
设置yAxis.splitLine.lineStyle.type=dotted明显见到底部有条虚线,
靠近x轴和y轴这里不应该画分隔线splitLine才正确
Version & Environment [版本及环境]

ECharts version [ECharts 版本]:
Browser version [浏览器类型和版本]:
OS Version [操作系统类型和版本]:

Expected behaviour [期望结果]
ECharts option [ECharts配置项]

option = {

}

Other comments [其他信息]

不太理解底部有虚线是什么意思，可以截图说明吗？

option = {
xAxis: {
type: 'category',
data: ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'],
splitLine: {
show: true,
lineStyle: {
width: 5,
color: ['green']
}
}
},
yAxis: {
type: 'value',
splitLine: {
show: true,
lineStyle: {
width: 5,
color: ['green']
}
}
},
series: [{
data: [820, 932, 901, 934, 1290, 1330, 1320],
type: 'line',
smooth: true
}]
};

x 轴和y轴都看不到了,这里是否不应该画分隔线?

@yangdan8 在上面图片的这里例子中，如果 x 轴 y 轴处的 splitLine 不绘制，只有轴的细黑线，看起来并不是很协调。
如果 splitLine 是个”很强“ 的样式的话，是不是要设置成，xAxis yAxis 不显示呢？

设计图：
我写的页面

x轴看起来是是虚线 没法还原设计图

设计图： 我写的页面  x轴看起来是是虚线 没法还原设计图

@Ovilia

七年了，这个问题依旧存在吗？


text:
ignore StorageTest [FINERACT-828]

FINERACT-828

@awasum @xurror merge? Unless you're motivated to fix StorageTest... I'm giving up on it! :smile:

I just don't get the thing with the storageTest. Seems to be some inconsistences only funny part is these inconsistences does not happen all the time.

/rebase

I'm merging this

authors: vorburger xurror

repo: apache/fineract


text:
[FLINK-31820] Support data source sub-database and sub-table

related to FLINK-31820
Background & Motivation
Hi, community. As I develop full-database synchronization syntax in the company to reduce the increase in the number of connections when multiple data sources are synchronized, I find that traditional databases (Mysql, Postgres, Oracle) need to support the function of sub-database and sub-table. It supports data source sub-database sub-table on top of existing projects
Goals

Support data source sub-database sub-table
When the sub-database and table functions are realized, the original data fragments allocated according to the partition are guaranteed to be available
Prioritize to ensure that Mysql, Postgres, Oracle support sub-database sub-table

Change

Every time JdbcRowDataInputFormat switches fragments, it will call open(InputSplit inputSplit) once (corresponding to closing the current data fragmentation method: close()), the value of inputSplit corresponds to the value of x in Serializable[x][y] is incremented, and each parallel The instance will not be executed repeatedly. For example, if there are 1024 split tables and each table has 2 data splits, then the value range of inputSplit.getSplitNumber() is: [0~2047]. Therefore, when the database is divided into tables, it should be ensured that the query logic of each table after the database and table is consistent with the query logic of the original single database and single table.
JdbcMultiTableProvider needs to split the Serializable[][] twice based on the results of the original data fragmentation, according to the sub-database and sub-table, to ensure that JdbcRowDataInputFormat can obtain the split table information

Design
According to the configured url and table_name expressions, the basic encoding steps are as follows:

Query all schemas in the database
Match the schema by regular
Query matches the table under the schema
By regular matching table
Return the corresponding relationship between the database url and the table: List

@complone Please follow https://flink.apache.org/how-to-contribute/overview/ for any code contribution

@complone请按照https://flink.apache.org/how-to-contribute/overview/进行任何代码贡献

I have explained the changes according to the community contribution manual. Currently, the single test has not been fully completed. Please wait for a while.

I have explained the changes according to the community contribution manual.

Please check the commit naming convention. Also, see my remark in the original Jira ticket.

根据社区贡献手册解释了变化

community contribution manual

Thank you, please let me apologize for not understanding the community contribution manual. Since I need to develop more functions for my company recently, I haven't had time to read your content in JIRA, I will read the manual recently and explain clearly Related concepts.

authors: MartijnVisser complone

repo: apache/flink-connector-jdbc


text:
[FLINK-16854][legal] Correct dependency versions in the NOTICE file of module statefun-ridesharing-example-simulator

There are some dependencies with a wrong version in the NOTICE file:
com.google.code.findbugs:jsr305:3.0.2:compile (Remove compile)
org.hibernate.validator:hibernate-validator:6.0.17 (Version should be 6.0.17.Final)
org.jboss.logging:jboss-logging:3.3.2  (Version should be 3.3.2.Final)

CC @tzulitai

Thank you @hequn8128! LGTM, merging.

repo: apache/flink-statefun


text:
[FLINK-21789][network] Make FileChannelManagerImpl#getNextPathNum select data directories fairly

What is the purpose of the change
FileChannelManagerImpl#getNextPathNum is not atomic which may cause unfairness of data directory selection. This patch makes the selection of data directories fair which is good for load balance and performance especially when there are multiple disks.
Brief change log

Make FileChannelManagerImpl#getNextPathNum select data directories fairly.

Verifying this change
This change added tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Dropped.

repo: apache/flink


text:
[FLINK-22298][table-planner-blink] The ExecNode's id should always start from 1 in json plan test

What is the purpose of the change
The ExecNode's id should always start from 1 in json plan test, which could make the test more stable
Brief change log

Reset the id counter before generating json in in test

Verifying this change
This change is already covered by existing tests*.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

The e2e test in My private azure is successful, https://dev.azure.com/godfreyhe/godfreyhe-flink/_build/results?buildId=401&view=results. I will merge this pr


text:
[BP-1.14][FLINK-24631][k8s] Use a stable subset of labels to select jobManager and taskManager pods

Backport #17685 to release-1.14.

@flinkbot run azure


text:
[FLINK-27972][coordination] Wait until savepoint operation is complete

The StopWithSavepoint state of the adaptive scheduler has to account for a number of failure scenarios.
One of these failure cases is a task failure with the savepoint going through. This for example happens when an exception is thrown in notifyCheckpointComplete(). In FLINK-26923 the code was changed to not trigger a restart in this case, and instead hard-fail the job, on the grounds that continuing a job despite a savepoint being created could lead to duplicate data.
This change was done incorrectly, because it assumed the savepoint future to always complete first. There is however no such guarantee that this is actually the case.
Primarily because the checkpoint coordinator does not run in the main thread it can happen that any failure happening around the time the CC completes the savepoint can end up calling into StopWithSavepoint#onFailure before the state is notified about the completed checkpoint.
This PR remedies this by forcing the handling of task failures or terminal states to wait for the savepoint to complete.
In addition the PR refactors the state to be a little bit messy. Both the general and savepoint failure code paths handling were setting variables that influenced each other in certain scenarios, making the code exceptionally difficult to reason about.
This now only happens in one direction, with the savepoint failure handling potentially being skipped if another failure happened.
Finally the PR adds a lot of documentation for what failure scenarios exist and why things are done the way they are.

@flinkbot run azure

Overall this is challenging for me to evaluate, as I'm unfamiliar with this part of Flink.
I agree that the new design is more straightforward, and the comments are quite helpful.
What I was looking for, but found it difficult to be sure about, is thorough test coverage.

The tests for the savepoint operations are scattered around quite a bit.
We unfortunately can't fully cover it in the StopWithSavepointTest because that requires an actual execution graph. Creating that ourselves isn't really an option (because there are barely any contracts; everything just relies on existing behavior of the scheduler), and we also lack good test utils. Moving away from the ExecutionGraph, while technically possible, can't be done quickly because so many re-used components expect an execution graph.
The waiting for the savepoint completion in onFailure/onGloballyTerminalState is covered by the newly added cases in StopWithSavepointTest.
Not accidentally triggering 2 state transitions from the state is now enforced by 71f72cf57d820ed62560f07f62259408e3a18b52; this on it's own would've failed tests in StopWithSavepointTest, like testJobFailedAndSavepointOperationFails. we likely would've noticed the issue sooner if we had this earlier.
As for other pre-existing tests:
The AdaptiveSchedulerTest contains tests for the proper archiving of errors that occurred during StopWithSavepoint. These make sure we don't accidentally drop task failures.
The AdaptiveSchedulerITCase contains high-level tests for the happy path and certain errors on the TM side. These make sure the savepoint operation does complete if a task failed.

@flinkbot run azure

authors: alpinegizmo zentol


text:
[FLINK-22316][table] Support MODIFY column/constraint/watermark for ALTER TABLE statement

What is the purpose of the change
This pull request is a subtask of FLINK-21634 and is based on FLINK-22315 to support
ALTER TABLE table_name MODIFY { <schema_component> | (<schema_component> [, ...]) }

Brief change log

Introduce ModifySchemaConverter to convert the MODIFY statement

Verifying this change

SqlToOperationConverterTest
TableEnvironmentTest
table.q

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduces a new feature? yes
If yes, how is the feature documented? FLINK-22320 will document this new feature

BTW, I think we should also take OperationConverterUtils.convertChangeColumn into consideration. Will you help to clean the code about this?

BTW, I think we should also take OperationConverterUtils.convertChangeColumn into consideration. Will you help to clean the code about this?

Sure, I'd like to. However, it is prudent for us to put it in a separate task to complete. The reason is that the current impl uses the deprecated CatalogTableImpl while SchemaConverter uses CatalogTable.of. I'm not sure if there any compatibility issues exist. WDYT?

+1 to move to another task.


text:
[FLINK-34612][table] Migrate StreamPhysicalValuesRule to java.

What is the purpose of the change
A bit reasoning for the change
The reason is that since 1.28.0 ( CALCITE-4787 - Move core to use Immutables instead of ImmutableBeans ) Calcite started to use Immutables (https://immutables.github.io/) and since 1.29.0 removed ImmutableBeans ( CALCITE-4839 - Remove remnants of ImmutableBeans post 1.28 release ). All rule configuration related api which is not Immutables based is marked as deprecated. Since Immutables implies code generation while java compilation it is seems impossible to use for rules in Scala code.
The PR migrates StreamPhysicalValuesRule to java
Verifying this change
This change is already covered by existing tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

I'm going to cloase all PRs trying to migrate ConverterRule since except https://github.com/apache/flink/pull/24457
since

there is still no response on my questsion and they keep appearing
We don't need to migrate ConverterRule's

authors: liuyongvs snuyanzin


text:
[FLINK-8120] [flip6] Register Yarn application with correct tracking URL

What is the purpose of the change
The cluster entrypoints start the ResourceManager with the web interface URL.
This URL is used to set the correct tracking URL in Yarn when registering the
Yarn application.
Verifying this change

Tested manually

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

Thanks for the review @shuai-xu. Merging this PR.


text:
[FLINK-9458] Ignore SharedSlot in CoLocationConstraint when not using legacy mode

What is the purpose of the change
The SharedSlot in CoLocationConstraint is only set when using the legacy mode. Thus,
CoLocationConstraint#isAssignedAlive should only check the SharedSlot if it was
previously set. This fixes the NPE when restarting a job with a co-location constraint
when using the new mode.
Verifying this change

Added ExecutionGraphCoLocationRestartTest#testConstraintsAfterRestart which verifies that an ExecutionGraph with co-location constraints can be restarted with the legacy as well as the new mode.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

LGTM 👍

Thanks for the review @StefanRRichter. Merging this PR.

authors: StefanRRichter tillrohrmann


text:
[FLINK-10164] Add fromSavepoint and allowNonRestoredState CLI options to StandaloneJobClusterEntrypoint

What is the purpose of the change
This PR address the short coming that the StandaloneJobClusterEntrypoint cannot resume from a savepoint. It adds the respective CLI options to it which are used to initialize the JobGraph with the correct SavepointRestoreSettings.
Additionally, this PR updates the documentation for docker compose and Kubernetes on how to start a containerized job cluster which resumes from a savepoint.
This PR is based on #6561.
Brief change log

Added SavepointRestoreSettings to StandaloneJobClusterEntrypoint
Added savepoint options --fromSavepoint and --allowNonRestoredState to StandaloneJobClusterEntrypoint
Improved CLI help
Updated Docker template to allow specifying savepoint options
Updated Kubernetes documentation on how to resume from a savepoint

Verifying this change

Added StandaloneJobClusterConfigurationParserFactoryTest#testSavepointRestoreSettingsParsing and StandaloneJobClusterEntryPointTest#testSavepointRestoreSettings

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

Travis passed. Merging this PR.


text:
[FLINK-11368][tests] Port TaskManagerStartupTest to new code base

What is the purpose of the change
Check and port TaskManagerStartupTest to new code base.
Brief change log

Port tests of testIODirectoryNotWritable, testMemoryConfigWrong and testStartupWhenNetworkStackFailsToInitialize to new code base.
Previous test testStartupWhenTaskmanagerActorPortIsUsed could be covered by TaskManagerRunnerConfigurationTest in FLINK-11363-PR

Verifying this change
This change added tests and can be verified as follows:

port TaskManagerStartupTest to new code base.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

Failed test should have no relationship with this PR


text:
[FLINK-14396][network] Implement rudimentary non-blocking network output

What is the purpose of the change
Considering the mailbox model and unaligned checkpoints requirements in future, task network output should be non-blocking. In other words, as long as output is available, it should never block for a subsequent/future single record write.
In the first version, we only implement the non-blocking output for the most regular case, and do not solve the following cases which still keep the previous behavior.
1. Big record which might span multiple buffers
2. Flatmap-like operators which might emit multiple records in every process
3. Broadcast watermark which might request multiple buffers at a time

The solution is providing the RecordWriter#isAvailable method and respective LocalBufferPool#isAvailable for judging the output beforehand. As long as there is at-least one available buffer in LocalBufferPool, the RecordWriter is available for network output in most cases.  This doesn’t include runtime handling of this non-blocking and availability behavior in StreamInputProcessor.
Note: It requires the minimum number of buffers in output LocalBufferPool adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of backpressure future.
Brief change log

Refactor to extract AbstractAvailabilityProvider class from InputGate logics.
Provide the #isAvailable method for RecordWriter, ResultPartitionWriter and BufferPool.
Refactor the relevant implementations in LocalBufferPool for supporting updating the available state.

Verifying this change
Add new RecordWriterTest#testIsAvailableOrNot for verifying the changes.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

FYI: This PR is based on #9903 for reducing conflicts, so only review the last two commits for this PR change.
For the possible concern of removing availableMemorySegments.wait(2000) in LocalBufferPool, we could do that in a separate ticket which might effect the exiting backpressure monitor.

Thanks for the review @pnowojski .
I have updated the codes for addressing above concerns. Only left one issue of providing non-blocking request from global NetworkBufferPool, and I think it is better to be done separately together with adjusting backpressure monitor later. Because when we provide NetworkBufferPool#isAvailable to avoid availableMemorySegments.wait(2000); in LocalBufferPool, it would destroy the exiting monitor of backpressure. And now this PR could still keep the previous behavior of backpressure monitor.

Thanks for the further review and I updated the codes for new comments!
Some further thoughts for the following work:

I would create a separate ticket for refactoring the logic between local and global pool. It might not be feasible to make the global request non-blocking completely. In other words, we can not return to the mailbox directly when fails to get available buffer from global pool, because we do not want to store the already emitted record in serialization stack. But we could improve the current way a bit. It is no need to wait 2 seconds in LocalBufferPool before next try to request global buffer. Instead it can be blocked there to monitor isAvailable from global and be waked up immediately when the future is completed.

Refactor the backpressure monitor in FLINK-14472. There are two cases for causing backpressure. One is via BufferProvider#isAvailable and another is via judging whether it is waiting for global buffer available.  We could provide a transparent interface method for giving the result of backpressure. In the LocalBufferPool implementation it could consider above two cases together and the backpressure monitor does not need to understand the details of LocalBufferPool. So it is no need to analyze the task thread stack, and also solve the previous problem that backpressure is invalid if requestBuffer is called by other threads.


text:
GEODE-3788: add utility methods to get the async event queues in the …

…running cluster
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[ ] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[ ] Is your initial contribution a single, squashed commit?

[ ] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

precheckin green

repo: apache/geode


text:
HDDS-2565. Handle InterruptedException in VolumeSet

What changes were proposed in this pull request?
Re-Interrupt when catching the InterruptedException.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2565
How was this patch tested?
UT Updating to validate the behavior.

/retest

FYI: /retest is not working (yet) with github actions. Just push an empty commit (git commit --allow-empty -m "...") and push it.

Thanks @elek for the information.

The acceptance test failure does not seem to be related to this patch.

Thanks @elek for the review and commit.

authors: cxorm elek

repo: apache/hadoop-ozone


text:
HADOOP-17850. Upgrade ZooKeeper to 3.4.14 in branch-3.2.

https://issues.apache.org/jira/browse/HADOOP-17850
Upgrade ZooKeeper 3.4.14 to fix CVE-2019-0201 (https://zookeeper.apache.org/security.html).

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 59s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ branch-3.2 Compile Tests _

+1 :green_heart:
mvninstall
26m  0s

branch-3.2 passed

+1 :green_heart:
compile
0m 19s

branch-3.2 passed

+1 :green_heart:
mvnsite
0m 23s

branch-3.2 passed

+1 :green_heart:
javadoc
0m 21s

branch-3.2 passed

+1 :green_heart:
shadedclient
39m 32s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 15s

the patch passed

+1 :green_heart:
compile
0m 12s

the patch passed

+1 :green_heart:
javac
0m 12s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 15s

the patch passed

+1 :green_heart:
xml
0m  2s

The patch has no ill-formed XML file.

+1 :green_heart:
javadoc
0m 13s

the patch passed

-1 :x:
shadedclient
14m  6s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 16s

hadoop-project in the patch passed.

+1 :green_heart:
asflicense
0m 30s

The patch does not generate ASF License warnings.

58m  3s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3308/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3308

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml

uname
Linux e3272e6a5a09 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.2 / 7e2b1328de5ddf32799d69d281768531081ab95e

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3308/1/testReport/

Max. process+thread count
412 (vs. ulimit of 5500)

modules
C: hadoop-project U: hadoop-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3308/1/console

versions
git=2.17.1 maven=3.6.0

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

expecting the error of shadedclinet is fixed after merging #3305.

authors: hadoop-yetus iwasakims

repo: apache/hadoop


text:
[Bug]: Microsoft Excel Writer action: Auto Size Columns function is not working

Apache Hop version?
2.9.0
Java version?
11
Operating system
Windows
What happened?
Using Microsoft Excel Writer action, I have noticed that the Auto Size Columns function is not working.
I've checked the option Auto Size Columns in the Content tab (first image), but the excel output doesn't autosize columns (second one).
The same job in Pentaho produces an autosized excel sheet.

Issue Priority
Priority: 3
Issue Component
Component: Actions, Component: Transforms

We can try to upgrade Apache POI 5.2.4 to 5.3.0 because of the bug below:
https://bz.apache.org/bugzilla/show_bug.cgi?id=68094

.take-issue

I cannot reproduce the issue, might be because I am not on Windows.
After reading https://github.com/apache/poi/pull/541 the update should fix the widths in most cases but there are still some edge cases.

authors: hansva mmoz77 nadment

repo: apache/hop


text:
Extendable ClosableHttpClient

Allow extension of ClosableHttpClient.
Closes https://issues.apache.org/jira/browse/HTTPCLIENT-2154

@cdalexndr I am not sure I understand the rationale for removal of #doExecute method. How does this method prevent ClosableHttpClient extension?
Besides, we cannot just remove protected method without breaking API compatibility with previous 5.x releases.

The HttpClientBuilder.build() returns CloseableHttpClient instance, and I need to wrap it to provide custom functionality.
HTTPCLIENT-2154 contains example code: using a wrapper (decorator pattern) doesn't work (see error comment):
    class CustomCloseableHttpClient extends CloseableHttpClient {
        private CloseableHttpClient base;

        public CustomCloseableHttpClient( CloseableHttpClient httpClient ) {
            this.base= httpClient;
        }

        @Override
        protected CloseableHttpResponse doExecute( HttpHost httpHost, HttpRequest httpRequest, HttpContext httpContext ) throws IOException, ClientProtocolException {
            return base.doExecute( httpHost, httpRequest, httpContext ); //<-- error, cannot call protected method
        }
        ....

@cdalexndr
public class DecoratedCloseableHttpClient extends CloseableHttpClient {

    private final CloseableHttpClient closeableHttpClient;

    public DecoratedCloseableHttpClient(CloseableHttpClient closeableHttpClient) {
        this.closeableHttpClient = closeableHttpClient;
    }

    @Override
    protected CloseableHttpResponse doExecute(HttpHost target, ClassicHttpRequest request, HttpContext context) throws IOException {
        return closeableHttpClient.execute(target, request, context);
    }

    @Override
    public void close(CloseMode closeMode) {
        closeableHttpClient.close(closeMode);
    }

    @Override
    public void close() throws IOException {
        closeableHttpClient.close();
    }
}

I still do not understand the problem.

@cdalexndr What is stopping you from using the public #execute method?

@cdalexndr What is stopping you from using the public #execute method?

This can lead to side effects, but for my use case I guess it will work.
Ideally, HttpClientBuilder.build() should return an interface that doesn't have this issue.

repo: apache/httpcomponents-client