text:
ci: add qemu run support

Signed-off-by: nietingting nietingting@xiaomi.com
Summary
ci: add qemu run support
Impact
none
Testing
test on local

@ttnie please rebase to the last master


text:
Use pureconfig for CouchDbRestStore

PR for #3355 Now the CouchDB config is read from application.conf with following structure
whisk {
    couchdb {
        protocol = "http"
        host     = "localhost"
        port     = 5984
        username = "user"
        password = "password"
        provider = "CouchDB"
        databases {
          WhiskAuth       = whisk_local_subjects
          WhiskEntity     = whisk_local_whisks
          WhiskActivation = whisk_local_activations
        }
    }
  }

The config can be provided via following environment variables
CONFIG_whisk_couchdb_protocol
CONFIG_whisk_couchdb_host
CONFIG_whisk_couchdb_port
CONFIG_whisk_couchdb_username
CONFIG_whisk_couchdb_password
CONFIG_whisk_couchdb_provider
CONFIG_whisk_couchdb_databases_WhiskAuth
CONFIG_whisk_couchdb_databases_WhiskEntity
CONFIG_whisk_couchdb_databases_WhiskActivation

This would be a backward incompatible change for existing deployments where they would need to switch to new environment variables instead of existing DB_PORT etc
~For now ArtifactStoreProvider#makeStore still takes WhiskConfig for determining DB name. That would be addressed in a separate PR~ now included in this PR itself

Travis failure in Kafka test. Does not look related to this change and possibly timing related. Would trigger a rebuild
org.scalatest.exceptions.TestFailedException: 1 was not equal to 2Open stacktrace
Send message to topic.
Successfully sent message to topic: KafkaConnectorTestTopic-0@5
Receiving message from topic.
Received 1. Took 15 msec: List(Wed Feb 28 10:59:19 UTC 2018)
Send message to topic.
Successfully sent message to topic: KafkaConnectorTestTopic-0@6
Receiving message from topic.
Received 1. Took 16 msec: List(Wed Feb 28 10:59:19 UTC 2018)
Send message to topic.
Successfully sent message to topic: KafkaConnectorTestTopic-0@7
Receiving message from topic.
Received 1. Took 30 msec: List(Wed Feb 28 10:59:19 UTC 2018)
Send message to topic.
Successfully sent message to topic: KafkaConnectorTestTopic-0@8
Receiving message from topic.
Received 1. Took 5 msec: List(Wed Feb 28 10:59:30 UTC 2018)

Updated the PR to now remove usage of WhiskConfig from the ArtifactStoreProvider api completely. Now database names are also captured via the application.conf and config keys are derived on the basis of entity class name
@markusthoemmes  Let me know if any other aspect needs to be addressed as part of this PR

Last build failed due to low memory. Triggered a rebuild
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 139984896 bytes for committing reserved memory.
# An error report file with more information is saved as:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d0b80000, 139984896, 0) failed; error='Cannot allocate memory' (errno=12)
# /home/travis/build/apache/incubator-openwhisk/tests/hs_err_pid19764.log

PG1 2677 running.

@chetanmeh I'll fix the conflicts.

Thanks @markusthoemmes . Let me know if it needs any further change

source: gharchive/pull-request

authors: chetanmeh markusthoemmes

repo: apache/incubator-openwhisk

license_type: permissive

license_source: github-api


text:
[Feature] Do not report statistics for append only table or predications without keys

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Some olap queries in tpc-h will run faster when we perform them without statistics for append-only table or predications without keys. We can add an option for this to improve the olap latency. subtask of #1945
Solution
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

@FangYongs Please assign it to me, I'll take it, thanks~

Hi @FangYongs , do you have some performance benchmark?

@JingsongLi Yes. We are currently using TPC-H for performance benchmark of Flink and Paimon, and we found statistics on the append-only table can cause some query performance to regress. When we turn off statistics separately for these queries, the query latencies are lower. So we would like to add an option in Paimon to uniformly support this behavior.

@FangYongs I see, Is there some benchmark result? How large is the performance regression.

source: gharchive/issue

authors: FangYongs JingsongLi schnappi17

repo: apache/incubator-paimon


text:
Feature: restrict the replica_count while creating app

https://github.com/XiaoMi/rdsn/pull/963

XiaoMi/rdsn#963 merged

authors: Shuo-Jia hycdong

repo: apache/incubator-pegasus


text:
Support Service principal based ADLSGen2PinotFS authentication

Linkedin plans to use ADLSGen2PinotFS but requires service principal based authentication to do that for better ACL control.
The current accesskey based auth would be harder to maintain with multiple applications.
Reference docs
https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-java#connect-by-using-azure-active-directory-azure-ad

https://github.com/apache/incubator-pinot/pull/6531

repo: apache/incubator-pinot


text:
fix integration tests for kafka ports

Description
Fix Kafka port for integration tests
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

Codecov Report

Merging #6889 (0d94f6b) into master (fe596b6) will decrease coverage by 8.04%.
The diff coverage is 91.66%.

:exclamation: Current head 0d94f6b differs from pull request most recent head fb8e5ec. Consider uploading reports for the commit fb8e5ec to get more accurate results

@@             Coverage Diff              @@
##             master    #6889      +/-   ##
============================================
- Coverage     73.57%   65.52%   -8.05%
  Complexity       12       12
============================================
  Files          1421     1423       +2
  Lines         70006    70029      +23
  Branches      10115    10119       +4
============================================
- Hits          51504    45885    -5619
- Misses        15091    20866    +5775
+ Partials       3411     3278     -133

Flag
Coverage Δ
Complexity Δ

integration
?
?

unittests
65.52% <91.66%> (+0.02%)
12.00 <0.00> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

...he/pinot/spi/data/readers/BaseRecordExtractor.java
66.07% <0.00%> (-3.58%)
0.00 <0.00> (ø)

.../inputformat/protobuf/ProtoBufRecordExtractor.java
77.52% <77.77%> (-0.50%)
0.00 <0.00> (ø)

...t/local/io/compression/ChunkCompressorFactory.java
50.00% <100.00%> (+10.00%)
0.00 <0.00> (ø)

...ment/local/io/compression/ZstandardCompressor.java
100.00% <100.00%> (ø)
0.00 <0.00> (?)

...nt/local/io/compression/ZstandardDecompressor.java
100.00% <100.00%> (ø)
0.00 <0.00> (?)

...he/pinot/segment/local/utils/TableConfigUtils.java
78.47% <100.00%> (-3.35%)
0.00 <0.00> (ø)

.../segment/spi/compression/ChunkCompressionType.java
100.00% <100.00%> (ø)
0.00 <0.00> (ø)

...ot/segment/spi/creator/SegmentGeneratorConfig.java
81.00% <100.00%> (+0.65%)
0.00 <0.00> (ø)

...org/apache/pinot/spi/config/table/FieldConfig.java
96.66% <100.00%> (+0.51%)
0.00 <0.00> (ø)

...a/org/apache/pinot/minion/metrics/MinionMeter.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

... and 348 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe596b6...fb8e5ec. Read the comment docs.


text:
Hide namespace cli commands that are no longer relevant

Motivation
Explain here the context, and why you're making that change.
What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Result
After your change, what will change.

@merlimat

what is the reason for hiding this command?

In 2.0 the concept of cluster in a topic name goes away. Plus I believe we(and the rest of the industry) have standardized on the keyword topic. The hiding is to make sure that legacy users still have access to the old way.

what is the reason for hiding this command?

pulsar-admin namespaces destinations $MY_NAMESPACE
was converted into pulsar-admin namespaces topics $MY_NAMESPACE.
Initially, I used an alias in JCommander, but that doesn't really work (and also leaves topics(destinations) in the help text which is confusing.
Documentation will refer only to the preferred way, but old command will still work.

authors: merlimat rdhabalia srkukarni

repo: apache/incubator-pulsar


text:
QUICKSTEP-95: Fixed the exception due to zero tuple estimation for th…

…e empty project expression.
The problem is that tuple_slot_bytes_ is ZERO if the project expression is empty, and later results in a divide-by-zero exception.
It is easy to reproduce:
$ create table r (a int, b char(20));
$ insert into r values(1, 'madison');
$ create table s (a int, c float);
$ select s.a from r, s where r.a > 0;

I0619 18:26:45.097215 1944100864 PhysicalGenerator.cpp:196] Optimized physical plan:
TopLevelPlan
+-plan=NestedLoopsJoin
| +-left=Selection
| | +-input=TableReference[relation=r]
| | | +-AttributeReference[id=0,name=a,relation=r,type=Int]
| | | +-AttributeReference[id=1,name=b,relation=r,type=Char(20)]
| | +-filter_predicate=Greater
| | | +-AttributeReference[id=0,name=a,relation=r,type=Int]
| | | +-Literal[value=0,type=Int]
| | +-project_expressions=
| |   +-[]
| +-right=TableReference[relation=s]
| | +-AttributeReference[id=2,name=a,relation=s,type=Int]
| | +-AttributeReference[id=3,name=c,relation=s,type=Float]
| +-join_predicate=Literal[value=true]
| +-project_expressions=
|   +-AttributeReference[id=2,name=a,relation=s,type=Int]
+-output_attributes=
  +-AttributeReference[id=2,name=a,relation=s,type=Int]

Floating point exception: 8

LGTM. Merging.

authors: jianqiao zuyu

repo: apache/incubator-quickstep


text:
[Bug] [transforms] Abnormal data causes the spark job to exit

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
When I write the data from kafka into clickhouse, there may be some message data with abnormal format, so an error was reported during the execution, which directly led to the job exit . I want to skip this error data when the consumption encounters an error line, and then continue the job. I will solve it and sumit a PR later .
SeaTunnel Version
dev
SeaTunnel Config
env {
  # You can set spark configuration here
  # see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
  spark.app.name = "kafkaTestConsumer"
  spark.executor.instances = 2
  spark.executor.cores = 1
  spark.executor.memory = "1g"
  spark.streaming.batchDuration = 2
}

source {
  # This is a example input plugin **only for test and demonstrate the feature input plugin**
    KafkaStream {
        topics = "test"
        consumer.bootstrap.servers = "hadoop104:9092,hadoop105:9092,hadoop106:9092"
        consumer.group.id = "test"
    }

  # You can also use other input plugins, such as file
  # file {
  #   result_table_name = "accesslog"
  #   path = "hdfs://hadoop-cluster-01/nginx/accesslog"
  #   format = "json"
  # }

  # If you would like to get more information about how to configure seatunnel and see full list of input plugins,
  # please go to https://seatunnel.apache.org/docs/spark/configuration/source-plugins/FakeStream
}

transform {

    json {
        source_field = "raw_message",
        result_table_name = "raw_row"
    },
    sql {
        sql = "select url,user_id,u_a,uri,datetime,substring(datetime,1,10) day  from raw_row",
	#skip_error_lines = "true",
    }

  # you can also use other filter plugins, such as sql
  # Sql {
  #   sql = "select * from accesslog where request_time > 1000"
  # }

  # If you would like to get more information about how to configure seatunnel and see full list of filter plugins,
  # please go to https://seatunnel.apache.org/docs/spark/configuration/transform-plugins/Split
}

sink {
  # choose stdout output plugin to output data to console
    Clickhouse {
        host = "10.0.8.9:8123"
        clickhouse.socket_timeout = 50000
        database = "test"
        table = "authlog"
        fields = ["url", "user_id", "u_a", "uri","datetime","day"]
        username = "username"
        password = "password"
        bulk_size = 20
    }
}

Running Command
./bin/start-seatunnel-spark.sh --master local --deploy-mode client --config /data/software/apache-seatunnel-incubating-2.1.1/config/kafkatest.conf

Error Exception
2022-05-18 12:06:35 INFO  KafkaRDD:54 - Beginning offset 62441869 is the same as ending offset skipping test 2
2022-05-18 12:06:35 INFO  Executor:54 - Finished task 5.0 in stage 5169.0 (TID 22325). 1455 bytes result sent to driver
2022-05-18 12:06:35 INFO  TaskSetManager:54 - Finished task 5.0 in stage 5169.0 (TID 22325) in 1 ms on localhost (executor driver) (6/6)
2022-05-18 12:06:35 INFO  TaskSchedulerImpl:54 - Removed TaskSet 5169.0, whose tasks have all completed, from pool
2022-05-18 12:06:35 INFO  DAGScheduler:54 - ResultStage 5169 (json at Json.scala:62) finished in 0.011 s
2022-05-18 12:06:35 INFO  DAGScheduler:54 - Job 5169 finished: json at Json.scala:62, took 0.012577 s
2022-05-18 12:06:35 INFO  JobScheduler:54 - Finished job streaming job 1652846795000 ms.0 from job set of time 1652846795000 ms
2022-05-18 12:06:35 ERROR JobScheduler:91 - Error running job streaming job 1652846795000 ms.0
org.apache.spark.sql.AnalysisException: cannot resolve '`url`' given input columns: []; line 1 pos 7;
'Project ['url, 'user_id, 'u_a, 'uri, 'datetime, 'substring('datetime, 1, 10) AS day#161715]
+- SubqueryAlias `raw_row`
   +- LogicalRDD false

	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:110)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:107)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:296)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:107)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
	at org.apache.seatunnel.spark.transform.Sql.process(Sql.scala:28)
	at org.apache.seatunnel.spark.SparkEnvironment.transformProcess(SparkEnvironment.java:159)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2$$anonfun$apply$1.apply(SparkStreamingExecution.scala:52)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2$$anonfun$apply$1.apply(SparkStreamingExecution.scala:51)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2.apply(SparkStreamingExecution.scala:51)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2.apply(SparkStreamingExecution.scala:43)
	at org.apache.seatunnel.spark.stream.SparkStreamingSource$$anonfun$start$1.apply(SparkStreamingSource.scala:39)
	at org.apache.seatunnel.spark.stream.SparkStreamingSource$$anonfun$start$1.apply(SparkStreamingSource.scala:37)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2022-05-18 12:06:35 ERROR Seatunnel:60 -

===============================================================================

2022-05-18 12:06:35 ERROR Seatunnel:63 - Fatal Error,

2022-05-18 12:06:35 ERROR Seatunnel:65 - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues

2022-05-18 12:06:35 ERROR Seatunnel:67 - Reason:Execute Spark task error

2022-05-18 12:06:35 ERROR Seatunnel:68 - Exception StackTrace:java.lang.RuntimeException: Execute Spark task error
	at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
	at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
	at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.sql.AnalysisException: cannot resolve '`url`' given input columns: []; line 1 pos 7;
'Project ['url, 'user_id, 'u_a, 'uri, 'datetime, 'substring('datetime, 1, 10) AS day#161715]
+- SubqueryAlias `raw_row`
   +- LogicalRDD false

	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:110)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:107)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:296)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:107)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
	at org.apache.seatunnel.spark.transform.Sql.process(Sql.scala:28)
	at org.apache.seatunnel.spark.SparkEnvironment.transformProcess(SparkEnvironment.java:159)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2$$anonfun$apply$1.apply(SparkStreamingExecution.scala:52)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2$$anonfun$apply$1.apply(SparkStreamingExecution.scala:51)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2.apply(SparkStreamingExecution.scala:51)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2.apply(SparkStreamingExecution.scala:43)
	at org.apache.seatunnel.spark.stream.SparkStreamingSource$$anonfun$start$1.apply(SparkStreamingSource.scala:39)
	at org.apache.seatunnel.spark.stream.SparkStreamingSource$$anonfun$start$1.apply(SparkStreamingSource.scala:37)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

2022-05-18 12:06:35 ERROR Seatunnel:69 -
===============================================================================

Exception in thread "main" java.lang.RuntimeException: Execute Spark task error
	at org.apache.seatunnel.core.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:79)
	at org.apache.seatunnel.core.base.Seatunnel.run(Seatunnel.java:39)
	at org.apache.seatunnel.core.spark.SeatunnelSpark.main(SeatunnelSpark.java:32)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.sql.AnalysisException: cannot resolve '`url`' given input columns: []; line 1 pos 7;
'Project ['url, 'user_id, 'u_a, 'uri, 'datetime, 'substring('datetime, 1, 10) AS day#161715]
+- SubqueryAlias `raw_row`
   +- LogicalRDD false

	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:110)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:107)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:296)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)
	at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)
	at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:93)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:107)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
	at org.apache.seatunnel.spark.transform.Sql.process(Sql.scala:28)
	at org.apache.seatunnel.spark.SparkEnvironment.transformProcess(SparkEnvironment.java:159)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2$$anonfun$apply$1.apply(SparkStreamingExecution.scala:52)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2$$anonfun$apply$1.apply(SparkStreamingExecution.scala:51)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2.apply(SparkStreamingExecution.scala:51)
	at org.apache.seatunnel.spark.stream.SparkStreamingExecution$$anonfun$start$2.apply(SparkStreamingExecution.scala:43)
	at org.apache.seatunnel.spark.stream.SparkStreamingSource$$anonfun$start$1.apply(SparkStreamingSource.scala:39)
	at org.apache.seatunnel.spark.stream.SparkStreamingSource$$anonfun$start$1.apply(SparkStreamingSource.scala:37)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
	at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
	at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2022-05-18 12:06:35 INFO  StreamingContext:54 - Invoking stop(stopGracefully=false) from shutdown hook
2022-05-18 12:06:35 INFO  ReceiverTracker:54 - ReceiverTracker stopped
2022-05-18 12:06:35 INFO  JobGenerator:54 - Stopping JobGenerator immediately
2022-05-18 12:06:35 INFO  RecurringTimer:54 - Stopped timer for JobGenerator after time 1652846795000

Flink or Spark Version
Spark 2.4.0
Java or Scala Version
java version "1.8.0_101"
Screenshots
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thanks for your feedback, make sense to me. Do you have a plan?

Thanks for your feedback, make sense to me. Do you have a plan?

I have submit a PR to resolve it .

authors: ruanwenjun wangjunbo

repo: apache/incubator-seatunnel


text:
[Hotfix] [Connector-file-ftp] Fix the problem of writing ftp failure

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

When FTPClient creates a directory, it cannot set permissions directly. As a result, the created directory cannot be recursive, and the whole directory is all messed up.

Solution: You can send a SITE CHMOD command to change the permissions of the directory immediately after creating the directory.

Problem 2: Recursive calls in the mkdirs method may cause the working directory not to return to its original state after each recursive call. This may cause client.changeWorkingDirectory(parentDir) to return false in some cases.
Solution: The working directory can be restored after each recursive call

After the code repair is completed, execute:

How to reproduce this issue locally:
Pull this branch to your local machine:https://github.com/apache/incubator-seatunnel/pull/4647
First step: Disable some test cases (reduce debugging execution waiting time, it is recommended to only execute lines 126 to 128)

Second step: Use breakpoint debugging to prevent the container from being released
Third step: Log into the FTP container to view the content that has been written

SeaTunnel Version
v2.3.1
SeaTunnel Config
null

Running Command
null

Error Exception
null

Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@hailin0 @TyrantLucifer @EricJoy2048 PTAL, thanks.

Possible problem: The SITE CHMOD command in the commons-net library is used to set file permissions on an FTP server. However, the concept of UNIX/Linux file permissions is not present by default on the Windows operating system. Therefore, running the SITE CHMOD command on Windows is not feasible.


text:
[Fix][Spark] fixed first start shell can't run.

Purpose of this pull request
This close #1744

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

Hi, thanks for  fix this bug.And i find a error in your code.
in seatunnel-core/seatunnel-core-spark/src/main/bin/start-seatunnel-spark.sh
should be  echo "Execute SeaTunnel Spark Job: ${CMD}"，it's not flink

Hi, thanks for fix this bug.And i find a error in your code. in seatunnel-core/seatunnel-core-spark/src/main/bin/start-seatunnel-spark.sh should be echo "Execute SeaTunnel Spark Job: ${CMD}"，it's not flink

My fault. Thanks for review.

Is there a better way to solve this, if we start with debug, there will still have this bug.

Is there a better way to solve this, if we start with debug, there will still have this bug. How about execute the command at SparkStarter.

Split CMD which Starter return.
Remove call Log method.
This one.

Maybe 1 is better.

You mean we still use the SparkStarter to generate the command, but parse the command in the shell?

You mean we still use the SparkStarter to generate the command, but parse the command in the shell?

yeap

authors: BenJFan ruanwenjun whb-bigdata


text:
[SCB-99] added compensation logic on tx aborted event

Signed-off-by: seanyinx sean.yin@huawei.com
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.3%) to 94.869% when pulling 3257c07913d9a2c16db8b94ac4816714f141a2ef on SCB-99_compensation_logic into 43ab34ccc51f537587d0403586dd43cd45b333fe on master.

authors: coveralls seanyinx

repo: apache/incubator-servicecomb-saga


text:
when insert set primary key is null  will show not support complex expression value

Which version of ShardingSphere did you use?
3.1.0
Which project did you use? Sharding-JDBC or Sharding-Proxy?
Sharding-Proxy
Expected behavior
insert success
Actual behavior
insert failure
Unknown exception: INSERT INTO can not support complex expression value on sharding column 'id'.
Reason analyze (If you can)
when insert if the primary key is null it's will show not support complex expression value.
Example codes for reproduce this issue (such as a github link).
CREATE TABLE `user` (
  `id` bigint(100) NOT NULL AUTO_INCREMENT,
  `username` varchar(255) DEFAULT NULL,
  `password` varchar(255) DEFAULT NULL,
  `expiration_time` int(11) DEFAULT NULL,
  `point` varchar(255) DEFAULT NULL,
  `created_time` int(11) DEFAULT NULL,
  `pro_id` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8;

INSERT INTO user  ( id, username,password,expiration_time,point,created_time,pro_id ) VALUES( null, 'dsff','dsfdsf',2,2,1,2 )

@Yohox  Hi, I used the 4.0.0-RC1 version, success, you can try it.

i use the 4.0.0RC2 show
10002 - 2Unknown exception: [No signature of method: java.lang.String.mod() is applicable for argument types: (java.lang.Integer) values: [2]
Possible solutions: drop(int), find(), any(), use([Ljava.lang.Object;), any(groovy.lang.Closure), find(java.util.regex.Pattern)], Time: 0.038000s

No matter Groovy or Java can not support use class type String to call mod() method.
Inline expression is Groovy expression, you need to follow Groovy's grammar.
.

authors: Yohox dongzl terrymanu

repo: apache/incubator-shardingsphere

license_source: bigquery


text:
[Bug] streampark-console-webapp is building an error on MacOS

Search before asking

[X] I had searched in the issues and found no similar issues.

Java Version
openjdk version "1.8.0_412"
Scala Version
2.12.x
StreamPark Version
release-2.1.4
Flink Version
all
deploy mode
None
What happened
./build error
streampark-console-webapp
I used the environment given by the official website
Maven 3.6+
npm 7.11.2 ( https://nodejs.org/en/ )
pnpm (npm install -g pnpm)
JDK 1.8+
main npm version :

Error Exception
[INFO] StreamPark : Flink Client 2.1.4 .................... SUCCESS [  0.171 s]
[INFO] StreamPark : Flink Client Api 2.1.4 ................ SUCCESS [ 18.486 s]
[INFO] StreamPark : Flink Client Core 2.1.4 ............... SUCCESS [ 29.955 s]
[INFO] StreamPark : Console Parent 2.1.4 .................. SUCCESS [  0.262 s]
[INFO] StreamPark : Console Service 2.1.4 ................. FAILURE [04:50 min]
[INFO] StreamPark : Spark Parent 2.1.4 .................... SKIPPED
[INFO] StreamPark : Spark Core 2.1.4 ...................... SKIPPED
[INFO] StreamPark : Spark Connector 2.1.4 ................. SKIPPED
[INFO] StreamPark : Spark Connector Base 2.1.4 ............ SKIPPED
[INFO] StreamPark : Spark Connector Kafka 2.1.4 ........... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  16:12 min
[INFO] Finished at: 2024-06-13T11:14:10+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:pnpm (build) on project streampark-console-service: Failed to run task: 'pnpm run build:no-cache' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :streampark-console-service

╰─ pnpm run build:no-cache

> streampark-webapp@2.1.4 build:no-cache /Users/lzq/Desktop/Projects/Flink/incubator-streampark/streampark-console/streampark-console-webapp
> npm run clean:cache && npm run build

> streampark-webapp@2.1.4 clean:cache
> rimraf node_modules/.cache/ && rimraf node_modules/.vite

> streampark-webapp@2.1.4 build
> cross-env NODE_ENV=production vite build && esno ./build/script/postBuild.ts

vite v3.2.10 building for production...                                                                                11:18:46

 WARN  <script src="/_app.config.js?v=2.1.4-1718248725884"> in "/index.html" can't be bundled without type="module" attribute                                                                                11:18:47

✓ 226 modules transformed.                                                                                11:20:47
✓ 242 modules transformed.                                                                                11:20:56
✓ 263 modules transformed.                                                                                11:20:58
✓ 267 modules transformed.                                                                                11:21:00
✓ 329 modules transformed.                                                                                11:21:23
transforming (5086) src/components/SimpleMenu/src/components/useMenu.ts
 WARN  Use of eval is strongly discouraged, as it poses security risks and may cause issues with minification                                                                                11:22:35

✓ 5086 modules transformed.                                                                                11:22:58
rendering chunks (298)...
 ERROR  [vite:esbuild-transpile] Transform failed with 1 error:                                                                                11:23:10
assets/index.2e479bd0.js:53580:18: ERROR: Transforming async generator functions to the configured target environment ("es2015" + 2 overrides) is not supported yet

Transforming async generator functions to the configured target environment ("es2015" + 2 overrides) is not supported yet
53578|  };
53579|
53580|  const readBytes = async function* (iterable, chunkSize, encode) {
   |                    ^
53581|    for await (const chunk of iterable) {
53582|      yield* streamChunk(ArrayBuffer.isView(chunk) ? chunk : (await encode(String(chunk))), chunkSize);

                                                                                11:23:10
✨ [vite-plugin-theme:antd-dark] - extract antd dark css code file is successfully:
                                                                                11:23:10
✨ [vite-plugin-theme] - extract css code file is successfully:
                                                                                11:23:10
✨ [vite-plugin-imagemin]- compressed image resource successfully:
                                                                                11:23:10

                                                                                11:23:10
✨ [vite-plugin-compression]:algorithm=gzip - compressed file successfully:
                                                                                11:23:10

 ERROR  error during build:                                                                                11:23:10
Error: Transform failed with 1 error:
assets/index.2e479bd0.js:53580:18: ERROR: Transforming async generator functions to the configured target environment ("es2015" + 2 overrides) is not supported yet
    at failureErrorWithLog (/Users/lzq/Desktop/Projects/Flink/incubator-streampark/streampark-console/streampark-console-webapp/node_modules/.pnpm/esbuild@0.15.18/node_modules/esbuild/lib/main.js:1575:15)
    at /Users/lzq/Desktop/Projects/Flink/incubator-streampark/streampark-console/streampark-console-webapp/node_modules/.pnpm/esbuild@0.15.18/node_modules/esbuild/lib/main.js:814:29
    at responseCallbacks.<computed> (/Users/lzq/Desktop/Projects/Flink/incubator-streampark/streampark-console/streampark-console-webapp/node_modules/.pnpm/esbuild@0.15.18/node_modules/esbuild/lib/main.js:680:9)
    at handleIncomingPacket (/Users/lzq/Desktop/Projects/Flink/incubator-streampark/streampark-console/streampark-console-webapp/node_modules/.pnpm/esbuild@0.15.18/node_modules/esbuild/lib/main.js:735:9)
    at Socket.readFromStdout (/Users/lzq/Desktop/Projects/Flink/incubator-streampark/streampark-console/streampark-console-webapp/node_modules/.pnpm/esbuild@0.15.18/node_modules/esbuild/lib/main.js:656:7)
    at Socket.emit (node:events:365:28)
    at addChunk (node:internal/streams/readable:314:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at Pipe.onStreamRead (node:internal/stream_base_commons:190:23)

 ELIFECYCLE  Command failed with exit code 1.

### Screenshots

<img width="1737" alt="image" src="https://github.com/apache/incubator-streampark/assets/16716138/d3826f22-93f6-4d36-a5e6-e2cb7f9ee317">

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!(您是否要贡献这个PR?)

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

You can try the following version
https://github.com/apache/incubator-streampark/blob/56d8724128dd624447368b3a4191842cfebcd655/streampark-console/streampark-console-service/pom.xml#L707-L708

answered

https://github.com/apache/incubator-streampark/pull/3726

authors: SbloodyS chestnutqiang lordk911 wolfboys

repo: apache/incubator-streampark


text:
Add support for deck.gl "polygon" layer

Add support for Polygon layer in Superset
https://uber.github.io/deck.gl/#/documentation/layer-catalog/polygon-layer

I'd like to create a choropleth map using polygon data in units of municipalities.
What type of data should I use?
If there is a table containing geojson in the column, can I visualize it?
I would like to know sample data and how to set up queries.

authors: hughhhh moriryota

repo: apache/incubator-superset


text:
Pie Chart from Plotly

Make sure these boxes are checked before submitting your issue - thank you!

[Y] I have checked the superset logs for python stacktraces and included it here as text if any
[Y] I have reproduced the issue with at least the latest released version of superset
[Y] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version
Latest one at the moment. 2018/01/16
Steps to reproduce
I used some guides in previous issues to create a plotly pie chart.
first I create a p_pie.js files in superset/assets/visualizations/
p_pie.txt
then I edit superset/assets/javascripts/explore/stores/controls.jsx by adding:
  pie_label_type_plotly: {
    type: 'SelectControl',
    label: t('Label Type'),
    default: 'key',
    choices: [
      ['key', 'Category Name'],
      ['value', 'Value'],
      ['percent', 'Percentage'],
      ['key_value', 'Category and Value'],
      ['key_percent', 'Category and Percentage'],
    ],
    description: t('What should be shown on the label?'),
  },

and superset/assets/javascripts/explore/stores/visTypes.js by adding:
  p_pie: {
    label: t('Pie Chart - Plotly'),
    showOnExplore: true,
    controlPanelSections: [
      {
        label: t('Query'),
        expanded: true,
        controlSetRows: [
          ['metrics', 'groupby'],
          ['limit'],
        ],
      },
      {
        label: t('Chart Options'),
        controlSetRows: [
          ['pie_label_type'],
          ['donut', 'show_legend'],
          ['labels_outside'],
          ['color_scheme'],
        ],
      },
    ],
  },

add to superset/data/__init__.py:
        Slice(
            slice_name="GendersPlotly",
            viz_type='p_pie',
            datasource_type='table',
            datasource_id=tbl.id,
            params=get_slice_json(
                defaults,
                viz_type="p_pie", groupby=['gender'])),

add to superset/viz.py:
class PiePlotlyViz(NVD3Viz):

    """Annoy visualization snobs with this controversial pie chart"""

    viz_type = 'p_pie'
    verbose_name = _('Distribution - Plotly - Pie Chart')
    is_timeseries = False

    def get_data(self, df):
        df = df.pivot_table(
            index=self.groupby,
            values=[self.metrics[0]])
        df.sort_values(by=self.metrics[0], ascending=False, inplace=True)
        df = df.reset_index()
        df.columns = ['x', 'y']
        return df.to_dict(orient='records')

add to superset/assets/visualizations/main.js:
  p_pie: 'p_pie',

and
  [VIZ_TYPES.p_pie]: require('./p_pie.js'),

Now when I am trying to execute yarn run build
I get two similar errors
ERROR in dashboard.be55626a4dddad6eed81.entry.js from UglifyJs
Unexpected token: name (ARRAY_TYPE) [dashboard.be55626a4dddad6eed81.entry.js:76869,4]

ERROR in explore.332aa2868b37b6962d8d.entry.js from UglifyJs
Unexpected token: name (ARRAY_TYPE) [explore.332aa2868b37b6962d8d.entry.js:76764,4]

beside the error could you please tell me if there are other files that I should change?

You may want to look at the input parameters required by plotly and ensure it maps to the correct classes
For a Plotly "Time Series"  you can use the NVD3TimeSeries which has the relevant payload formats.
For a Plotly Pie Chart, you may want to use the BaseViz and format the inputs correctly
I was able to get a base Plotly Pie/Bar as well as a TimeSeries plot working.  However, this was more for understanding how to customize Superset.  Given a lot of the functionality exists in the existing charts, it would seem redundant to add this to Superset. I do like the zooming and export options from Plotly though.

We're looking into making visualizations work as plugins eventually and hoping people would create their own and share them. Closing since this has been stale for a while.

authors: ahsanshah mistercrunch rahimramezanian


text:
build(deps-dev): update cypress to 5.5.0, improvements for running locally

SUMMARY
Updates cypress to latest version (5.5.0) due to patched performance issues in 5.2.0.
Improvements for running Cypress locally, including:

Fix OOM crashing of headed chrome test runs (via numTestsKeptInMemory)
Update docs and build script to use chrome as cypress browser
Correct documentation on manual cypress runs
tox: add missing cypress testenv
tox: cleanup orphaned background flask processes

Also reduces the number of retries globally from 2 to 1 to reduce time spent on failed runs.
TEST PLAN
See updated CONTRIBUTING.md documentation.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #11603 (b9e3ba3) into master (bd79bd2) will decrease coverage by 12.09%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #11603       +/-   ##
===========================================
- Coverage   66.41%   54.31%   -12.10%
===========================================
  Files         878      412      -466
  Lines       42272    14315    -27957
  Branches     3949     3674      -275
===========================================
- Hits        28074     7775    -20299
+ Misses      14095     6363     -7732
- Partials      103      177       +74

Flag
Coverage Δ

cypress
54.31% <ø> (-0.62%)
:arrow_down:

javascript
?

python
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...uperset-frontend/src/dashboard/util/dnd-reorder.js
0.00% <0.00%> (-100.00%)
:arrow_down:

...rset-frontend/src/dashboard/util/getEmptyLayout.js
0.00% <0.00%> (-100.00%)
:arrow_down:

...dashboard/components/resizable/ResizableHandle.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/dashboard/util/getFilterScopeFromNodesTree.js
0.00% <0.00%> (-93.48%)
:arrow_down:

...src/dashboard/components/gridComponents/Header.jsx
10.52% <0.00%> (-86.85%)
:arrow_down:

...rc/dashboard/components/gridComponents/Divider.jsx
13.33% <0.00%> (-86.67%)
:arrow_down:

...ontend/src/dashboard/util/getDashboardFilterKey.ts
14.28% <0.00%> (-85.72%)
:arrow_down:

...set-frontend/src/views/CRUD/welcome/EmptyState.tsx
2.94% <0.00%> (-84.56%)
:arrow_down:

...src/components/FilterableTable/FilterableTable.tsx
2.64% <0.00%> (-83.43%)
:arrow_down:

...et-frontend/src/components/ListView/ActionsBar.tsx
11.11% <0.00%> (-81.20%)
:arrow_down:

... and 703 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd79bd2...f8274a1. Read the comment docs.

cc @adam-stasiak @ktmud

authors: codecov-io robdiciuccio


text:
Add yarn.lock for reproducible build (#4274)

See https://github.com/apache/incubator-superset/issues/4274.
I also add some words in contribution docs.

Superseded by https://github.com/apache/incubator-superset/pull/4679 and https://github.com/apache/incubator-superset/pull/4674

authors: timifasubaa xiaohanyu


text:
Druid subqueries support

CATEGORY

[x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
Druid SQL supports subqueries so superset should allow users to explore druid queries.
AFTER SCREENSHOTS

ADDITIONAL INFORMATION

[x] Has associated issue: #7849
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Wait does Druid support subqueries now!?

Looking at the docs, it looks like it does support subqueries, @john-bodley @betodealmeida does your version of Druid in production support them? Should we merge this?

Oh, is this new? What version?

@betodealmeida I've tested on druid 0.12.3

This is awesome! Thanks, @michalmisiewicz!

@john-bodley, I know AirBnB was running an older version of Druid. How do you think we should handle cases like this? Should we set druid version in config.py and add something like this:
# config.py
DRUID_VERSION = (0, 12, 3)

# superset/db_engine_specs/druid.py
allows_subquery = config['DRUID_VERSION'] >= (0, 12, 3)

?

@betodealmeida I just checked the version of our Druid clusters. A couple of them are running on versions pre 0.12.3 and thus we would need to have a mechanism for enabling this feature on a per cluster basis.
Rather than having this defined globally in the config it seems we'll need to add a field (and thus database column) to the druidclustermodelview.

We have a method to find out which version is the cluster running here https://github.com/apache/incubator-superset/blob/master/superset/connectors/druid/models.py#L165
Now it's a matter of making allows_subquery a method that receives the cluster/database, or something like that.

The subqueries are supported since druid 0.10 - the first version with sql support.
https://github.com/apache/incubator-druid/blob/master/docs/content/querying/sql.md

Hi, I am totally new Superset user - is this option "Superset should enable user to create visualization for druid in SQL lab " will be then soon available?

authors: KreteJalakasPiP betodealmeida john-bodley michalmisiewicz mistercrunch


text:
[docs] Fix, UPDATING had two changes that were not included on 0.35

CATEGORY
Choose one

[ ] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[X] Documentation

SUMMARY
UPDATING had 2 changes wrongly identified has shipped with 0.35.X
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS

Codecov Report

Merging #8754 into master will increase coverage by 0.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #8754      +/-   ##
==========================================
+ Coverage   65.74%   65.87%   +0.13%
==========================================
  Files         483      483
  Lines       24123    24123
  Branches     2767     2767
==========================================
+ Hits        15860    15892      +32
+ Misses       8087     8055      -32
  Partials      176      176

Impacted Files
Coverage Δ

superset/utils/core.py
88.08% <0%> (+0.16%)
:arrow_up:

superset/db_engine_specs/base.py
81.12% <0%> (+0.33%)
:arrow_up:

superset/connectors/sqla/models.py
84.97% <0%> (+0.64%)
:arrow_up:

superset/db_engine_specs/sqlite.py
66.66% <0%> (+9.99%)
:arrow_up:

superset/dataframe.py
94.48% <0%> (+10.23%)
:arrow_up:

superset/db_engine_specs/postgres.py
83.33% <0%> (+27.77%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 385484e...cb3a21f. Read the comment docs.

authors: codecov-io dpgaspar


text:
[TUBEMQ-126] Increase the unflushed data bytes control

Implement the unflushSize as a factor to trigger data flush, and the changes consist of the following 3 parts:
MsgFileStore: Make use of curUnflushSize and unflushDataHold settings to trigger;
Metadata: Reuse unflushDataHold to store the flush strategy;
Details: TUBE-126

Thanks for your contribution @hystericalhell , LGTM

authors: aloyszhang hystericalhell

repo: apache/incubator-tubemq


text:
[FEATURE][RUST] Require memory from unified memory pool when reading data

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature
Before writing data into server, the memory allocation is needed.
So require memory from unified memory pool when reading data should be supported for limited memory used.
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Hi @zuston I am interested in this task, can you assign it to me?

authors: wenlongbrother zuston

repo: apache/incubator-uniffle


text:
[Android] Inconsistent websocket handling

Bug description
Following works for web, but not mobile:
const ws = weex.requireModule('webSocket')
ws.WebSocket('ws://echo.websocket.org','');
ws.onopen = function(event) {
  console.log('onopen', event);
}

If we pass function as argument instead, if works:
ws.onopen(function(event) {
  console.log('onopen', event);
})

I found about it, after hours of research, here. And documentation does not say anything about it.
It is critical, if you want to use some websocket JS library, which uses Websocket under the hood.
Expected behavior
I expect API to be consistent across platforms.

Thanks for your reporting, we will look into that.

As there any update?

@umlink I am appreicated if you can fix it, thanks.

authors: Cheese-Yu YorkShen azamat-sharapov

repo: apache/incubator-weex


text:
[Android] rows property doesn't work in textarea

Describe the bug

rows property doesn't work in textarea when type property exists.
return-key-type proterty  works in textarea only if the type property exists.
so i can't have both rows and return-key-type in the same time.

To Reproduce

http://dotwe.org/vue/503e96850b51ce01bfbf056a2e95d666
http://dotwe.org/vue/c9e15259ce2a1e52b385dd8e69201694

Environment
Android
SDK version 0.28

#3176


text:
[ZEPPELIN-559] Cassandra interpreter v2

What is this PR for?

Update the Cassandra interpreter to V2 to support new schema commands to describe Cassandra 3.x features:

User Defined Functions
User Defined Aggregates
Materialized Views

Add support for single line comment using double slashes (//)
Add DESCRIBE TYPES
The Java driver version is bumpted to 3.0.0-rc1
The contextual HELP menu is also updated and a ChangeLog section is added.

What type of PR is it?
[Improvement]
Todos

[x] - Add new PR to update official Cassandra interpreter with new features
[ ] - Code review by the community (warning, Scala inside)

Is there a relevant Jira issue?
ZEPPELIN-559
ZEPPELIN-575 for doc
How should this be tested?

Download and install locally Cassandra 3.1.1
Start Cassandra
Clone this pull request locally with:

git fetch origin pull/600/head:CassandraInterpreterV2
git checkout CassandraInterpreterV2

Build this version of Zeppelin with mvn clean package -DskipTests
Start Zeppelin and update the property cassandra.hosts of the Cassandra interpreter (set it to localhost or 127.0.0.1 depending on your configuration)
In a paragraph, execute all the CQL statements given in this CassandraInterpreterV2TestData.cql by copy-pasting them
Start a new %cassandra paragraph and execute DESCRIBE FUNCTION test.maxOf;. You should be able to see:

Start a new %cassandra paragraph and execute DESCRIBE AGGREGATE test.group_by;. You should be able to see:

Start a new %cassandra paragraph and execute DESCRIBE MATERIALIZED VIEW test.user_by_country;. You should be able to see:

Start a new %cassandra paragraph and execute SELECT id,val1,val2,maxOf(val1,val2) FROM test.test_max;. You should be able to see:

Start a new %cassandra paragraph and execute SELECT group_by(category,sales_count) FROM test.items;. You should be able to see:

Start a new %cassandra paragraph and execute SELECT * FROM test.user_by_country WHERE country='FR';. You should be able to see:

Questions:

Does the licenses files need update? --> No
Is there breaking changes for older versions? --> No
Does this needs documentation? --> Yes

@doanduyhai thanks for the contribution, I know you gave some details in the JIRA issue, but could you update this PR description with our PR template? (https://github.com/apache/incubator-zeppelin/blob/master/CONTRIBUTING.md#creating-a-pull-request)

Just checked the full process, Cassandra interpreter v2 and new CQL scripts, all is ok

@doanduyhai Great improvement! Somehow last commit didn't trigger CI. Could you trigger CI for last commit?

@corneadoug @Leemoonsoo
Code review has been done by @brianwawok and I pushed some refactoring.
The build succeeded but Travis marked it as failed because : The log length has exceeded the limit of 4 Megabytes (this usually means that test suite is raising the same exception over and over)
Looks like this is a very common issue with Travis, it is reported here: https://github.com/travis-ci/travis-ci/issues/3865

@Leemoonsoo @corneadoug Finally successful build

Those headers :)
I don't usually review interpreters, but I will test it out :)

Thanks @doanduyhai  for big improvement!
LGTM

LGTM. Any more comments or discussion on this PR?

Merge if there're no more discussions

Tested, very nice interpreter thank you @doanduyhai !
A small amendment: to test this, make sure UDF sandbox is enabled in conf/cassandra.yaml
enable_user_defined_functions: true
Let's merge it

authors: Leemoonsoo corneadoug doanduyhai felixcheung victorcouste

repo: apache/incubator-zeppelin


text:
KAFKA-12867: Fix ConsumeBenchWorker exit behavior for maxMessages config

Bug:
The trogdor ConsumeBenchWorker has a bug. If one of the consumption tasks completes executing successfully due to maxMessages being consumed, then, the consumption task notifies the doneFuture causing the entire  ConsumeBenchWorker to halt. This becomes a problem when more than 1 consumption task is running in parallel, because the successful completion of 1 of the tasks shuts down the entire worker while the other tasks are still running. When the worker is shut down, it kills all the active consumption tasks, which is not the desired behavior.
How to reproduce?:
The bug is easily reproducible by running a ConsumeBenchSpec task configured with maxMessages value and threadsPerWorker > 1. When a trogdor workload is started with such a spec, and when one of the threads (i.e. consumption task) has consumed maxMessages, you can notice that it prematurely shuts down the worker although the other threads have not consumed at least maxMessages.
Fix:
The fix is to not notify the doneFuture when 1 of the consumption tasks complete without error. Instead, we should defer the notification to the CloseStatusUpdater thread.

cc @junrao @apovzner @rajinisivaram for review.
@apovzner It appears this behavior has been around since ConsumeBenchWorker was first implemented: https://github.com/apache/kafka/pull/4775. Please let me know your thoughts.

repo: apache/kafka


text:
KAFKA-14795: Provide message formatter for RemoteLogMetadata

This commit introduces a formatter for RemoteLogMetadata.
Example usage:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic __remote_log_metadata --from-beginning --formatter 'org.apache.kafka.server.log.remote.metadata.storage.serialization.RemoteLogMetadataSerde$RemoteLogMetadataFormatter'

RemoteLogSegmentMetadata{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=M1z1YtfhQ5i7oqLNve_0UQ:topic1-0, id=iWtc1Z6xQu2_DJXTklzKxQ}, startOffset=97990, endOffset=98467, brokerId=0, maxTimestampMs=1678292889855, eventTimestampMs=1678292938280, segmentLeaderEpochs={0=97990}, segmentSizeInBytes=511460, state=COPY_SEGMENT_STARTED}

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@satishd perhaps you would like to review this since it's associated with KIP405?

I'll take a look this week.

LGTM, the only comment is should we make the formatter as a separate file? @satishd , thoughts?

@showuon I am fine with formatter inside the serde class.

Failed tests are unrelated
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testConnectorReconfiguration
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsWithCustomForwardingAdminIntegrationTest.testReplicationIsCreatingTopicsUsingProvidedForwardingAdmin()
    Build / JDK 11 and Scala 2.13 / kafka.api.PlaintextAdminIntegrationTest.testReplicaCanFetchFromLogStartOffsetAfterDeleteRecords(String).quorum=kraft
    Build / JDK 8 and Scala 2.12 / kafka.zk.ZkMigrationIntegrationTest.[1] Type=ZK, Name=testDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT
    Build / JDK 8 and Scala 2.12 / kafka.zk.ZkMigrationIntegrationTest.[1] Type=ZK, Name=testDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT
    Build / JDK 17 and Scala 2.13 / kafka.zk.ZkMigrationIntegrationTest.[1] Type=ZK, Name=testDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT
    Build / JDK 17 and Scala 2.13 / kafka.zk.ZkMigrationIntegrationTest.[1] Type=ZK, Name=testDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT
    Build / JDK 11 and Scala 2.13 / kafka.zk.ZkMigrationIntegrationTest.[1] Type=ZK, Name=testDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT
    Build / JDK 11 and Scala 2.13 / kafka.zk.ZkMigrationIntegrationTest.[1] Type=ZK, Name=testDualWrite, MetadataVersion=3.4-IV0, Security=PLAINTEXT

authors: divijvaidya ivanyu satishd showuon


text:
KAFKA-15351: Ensure log-start-offset not updated to local-log-start-offset when remote storage enabled

Part-1: When tiered storage is enabled on the topic, and the last-standing-replica is restarted, then the log-start-offset should not reset its offset to first-local-log-segment-base-offset.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@showuon @divijvaidya @clolov @satishd
In this patch, we have covered the case where the log-start-offset gets reseted to local-log-start-offset when the last standing replica is restarted and enabled the OffloadAndConsumeFromLeaderTest with broker bounce. In the next patch, we can cover the case where the log-start-offset should be updated by the remote-log-segment deleter threads on becoming the leader. Let me know if we want to merge both the cases in the same PR.
Please take a look when you get chance!

Test Report
[SUCCESS] (1) create topic: Topic[name=topicA partition-count=1 replication-factor=1 segment-size=1 assignment=null properties={remote.storage.enable=true, local.retention.bytes=1, index.interval.bytes=1, segment.index.bytes=12}]

[SUCCESS] (2) produce-records: topicA-0
    ProducerRecord(topic=topicA, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k1, value=v1, timestamp=null)
    ProducerRecord(topic=topicA, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k2, value=v2, timestamp=null)
    ProducerRecord(topic=topicA, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k3, value=v3, timestamp=null)
    Segment[partition=topicA-0 offloaded-by-broker-id=0 base-offset=0 record-count=1]
    Segment[partition=topicA-0 offloaded-by-broker-id=0 base-offset=1 record-count=1]

[SUCCESS] (3) create topic: Topic[name=topicB partition-count=1 replication-factor=1 segment-size=2 assignment=null properties={remote.storage.enable=true, local.retention.bytes=1, index.interval.bytes=1, segment.index.bytes=24}]

[SUCCESS] (4) produce-records: topicB-0
    ProducerRecord(topic=topicB, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k1, value=v1, timestamp=null)
    ProducerRecord(topic=topicB, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k2, value=v2, timestamp=null)
    ProducerRecord(topic=topicB, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k3, value=v3, timestamp=null)
    ProducerRecord(topic=topicB, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k4, value=v4, timestamp=null)
    ProducerRecord(topic=topicB, partition=0, headers=RecordHeaders(headers = [], isReadOnly = true), key=k5, value=v5, timestamp=null)
    Segment[partition=topicB-0 offloaded-by-broker-id=0 base-offset=0 record-count=2]
    Segment[partition=topicB-0 offloaded-by-broker-id=0 base-offset=2 record-count=2]

[SUCCESS] (5) bounce-broker: 0

[SUCCESS] (6) consume-action:
  topic-partition = topicA-0
  fetch-offset = 1
  expected-record-count = 2
  expected-record-from-tiered-storage = 1

[SUCCESS] (7) consume-action:
  topic-partition = topicB-0
  fetch-offset = 1
  expected-record-count = 4
  expected-record-from-tiered-storage = 3

Content of local tiered storage:

Broker IDFile                                       | Offsets |      Records
----------------------------------------------------------------------------------------
topicB-0                                            |         |
    00000000000000000000-ZYHNDiMkQAuKx_Hm7yettA.log |       0 |     (k1, v1)
                                                    |       1 |     (k2, v2)
                                                    |         |
    00000000000000000002-404zVl_XTuCyOEBrnEVxvQ.log |       2 |     (k3, v3)
                                                    |       3 |     (k4, v4)
                                                    |         |
topicA-0                                            |         |
    00000000000000000001-dwVJFEg8SmqzcKuJrD0C8w.log |       1 |     (k2, v2)
                                                    |         |
    00000000000000000000-dYoeCs5FSKSEekm7aBq3aQ.log |       0 |     (k1, v1)
                                                    |         |

Overall LGTM! Thanks for the PR. Questions:

So, the solution we're going to adopt to fix the problem is only update log start offset in remote-log-segment deleter threads on becoming the leader. Is that right?
If so, I guess in the next patch, we'll add logics to handle the 2 cases raised in JIRA, is that right?

Thank you.

yes, case-1 will be handled by this approach. For case-2, we may have to check for end-to-end consistency and comeback on this.

Still cannot open this page: https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14301

Can you pause merging this until I finish my review? I have a few questions about this change which I would like to clear up before it makes it in trunk?

Okay, thank you for the pull request, but can you go into a bit more detail as to what it addresses? As far as I can tell it fixes a test case, but it doesn't address either of the two cases specified in https://issues.apache.org/jira/browse/KAFKA-15351? In other words, I do not understand why we are ensuring that the log-start-offset is not updated to the local-log-start-offset when remote storage is enabled and why that is the correct approach? As far as I understood from the description of case 1 in the JIRA ticket we want to reconcile a stale log start offset from a replica with the latest global log start offset which I believe is only available via inspecting tiered storage?

Added the case-3 in the KAFKA-15351 ticket. This patch addresses only the case-3.

Okay, but I still do not understand, what is preventing the value in the log-start-checkpoint-offset from being stale as well? Or is the idea here that we will start with the log-start-*-offset which we knew was correct at some point in time and then we will contact tiered storage to try and figure out what later value it might have moved to? If so, does it matter which offset we start from as long as we have the logic for checking what is in tiered storage?

Okay, but I still do not understand, what is preventing the value in the log-start-checkpoint-offset from being stale as well? Or is the idea here that we will start with the log-start-*-offset which we knew was correct at some point in time and then we will contact tiered storage to try and figure out what later value it might have moved to? If so, does it matter which offset we start from as long as we have the logic for checking what is in tiered storage?

yes, correct. The log-start-offset will be updated to the correct value eventually when the rlm-task updates the log-start-offset after becoming leader (patch needs to be opened).

If so, does it matter which offset we start from as long as we have the logic for checking what is in tiered storage?

If the log-start-offset moves ahead, then currently remote-log-segments are deleted due to breach by log-start-offset. Once the rlm-task updates the log-start-offset after becoming leader, then those remote-log-segments won't be moved. But, the log-start-offset should increase monotonically, it should not go back and forth.

Got it, okay, then this makes sense to me, thank you for the explanation!

@kamalcph , looks like there are some end to end authorization tests failed due to this change. Could you help fix them? Thanks.
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14301/7/#showFailuresLink

@kamalcph , looks like there are some end to end authorization tests failed due to this change. Could you help fix them? Thanks. https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14301/7/#showFailuresLink

checking the test failures.

Thanks @kamalcph for fixing the test failures. The test failures with the latest job are unrelated to this change. Merging to trunk and 3.6.

authors: clolov kamalcph satishd showuon


text:
KAFKA-2669; Fix LogCleanerIntegrationTest

LogCleanerIntegrationTest calls LogCleaner.awaitCleaned() to wait until cleaner has processed up to given offset. However, existing awaitCleaned() implementation doesn't wait for this. This patch fix the problem.

In addition, MockScheduler should execute all remaining tasks when it is shutdown. This is useful, e.g., to execute scheduled file deletion task in LogCleanerIntegrationTest so that we avoid "too many open file" exception.

@auradkar @ijuma Thanks for your suggestion. I have updated patch accordingly.

Not sure why test failed.. It passed on my machine. Will make slight change and test again.

@ijuma @auradkar Thanks for taking time to review the patch. Can you see if the updated patch address your concern?

LGTM

Thanks @ijuma @lindong28 for the thorough discussion, LGTM.

Hi, would you expect this patch to work on Windows?
This assert keeps failing on Windows:
assertTrue("log cleaner should have processed up to offset " + firstDirty2, lastCleaned2 >= firstDirty2);
Errors are:
cleanerTest[0]: "log cleaner should have processed up to offset 597"
cleanerTest[1]: "log cleaner should have processed up to offset 599"
cleanerTest[2]: "log cleaner should have processed up to offset 599"
cleanerTest[3]: "log cleaner should have processed up to offset 599"

authors: MaximGurschi guozhangwang ijuma lindong28


text:
KAFKA-10030 allow fetching a key from a single partition

StreamThreadStateStoreProvider#stores throws exception whenever taskId is not found, which is not correct behaviour in multi-threaded env where state store partitions are distributed among several StreamTasks.
final Task task = tasks.get(keyTaskId);
if (task == null) {
throw new InvalidStateStoreException(
String.format("The specified partition %d for store %s does not exist.",
storeQueryParams.partition(),
storeName));
}
Reproducible with KStream number of threads more then 1
StoreQueryIntegrationTest#streamsConfiguration
config.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
Suggested solution is to not throw exception if at least one state store is found, which is always true when using StoreQueryParameters.withPartition
https://issues.apache.org/jira/browse/KAFKA-10030?jql=project %3D KAFKA AND component %3D streams
@mjsax @guozhangwang

Retest this please.

@vinothchandar @brary for review

Retest this please.

Retest this please.

Retest this please.

Retest this please.

Retest this please.

Seems Jenkins is too busy... Will try again later

Retest this please.

Retest this please.

15:01:33 [ant:checkstyle] [ERROR] /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.13/streams/src/test/java/org/apache/kafka/streams/integration/StoreQueryIntegrationTest.java:25: Using the '.*' form of import should be avoided - org.apache.kafka.streams.*. [AvoidStarImport]
15:01:33 [ant:checkstyle] [ERROR] /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.13/streams/src/test/java/org/apache/kafka/streams/integration/StoreQueryIntegrationTest.java:56: Using the '.*' form of import should be avoided - org.hamcrest.Matchers.*. [AvoidStarImport]

cc @dima5rr

Retest this please.

Retest this please.

Retest this please.

Retest this please.

Retest this please.

Retest this please.

Jenkins does not cooperator right now. Will try again later.

Test this please

Thanks for the PR @dima5rr.
Merged to trunk and cherry-picked to 2.6 and 2.5 branches.

authors: abbccdda dima5rr mjsax vvcephei


text:
[metrics] Add a new metric 'last_consult_timestamp' for tablet

Add a new metric 'last_consult_timestamp' for tablet to indicate the
last write or scan timestamp of the tablet. We can judge whether a
tablet is 'hot' or 'cold' by this metric.
Change-Id: I90738ba90eaa8f78c8d721dde2eed2b723d5572d

Thank you for the patch. Can you please submit this patch to Gerrit instead following the contribution guide here:
https://kudu.apache.org/docs/contributing.html#_contributing_patches_using_gerrit

repo: apache/kudu


text:
(ci) Update MacOS runner to macos12

Due to https://github.blog/changelog/2024-05-20-actions-upcoming-changes-to-github-hosted-macos-runners/   we are change macos11 up to next, macos12

Need to confirm why there are go test failure on macos 12.

realpath util not included at macos-12 (see: https://apple.stackexchange.com/questions/450035/is-the-unix-realpath-command-distributed-with-macos-ventura)

realpath are fixed, some go integration test broken by i/o timeout

authors: PragmaTwice aleksraiden

repo: apache/kvrocks


text:
[KYUUBI #5417] should not check in-subquery in permanent view

Why are the changes needed?
Fix #5417
If there is is a view with in-subquery, authz will still request this in-subquery's interval privilege, it's not we want.
For view
CREATE VIEW db.view1
AS
WITH temp AS (
    SELECT max(scope) max_scope
    FROM db1.table1)
SELECT id as new_id FROM db1.table2
WHERE scope in (SELECT max_scope FROM temp)

When we query the view
SEELCT * FROM db.view1

Before this pr, since spark will first execute subquery, it will first request [default/table1/scope] then request [default/view1/new_id]
after this pr, it only request  [default/view1/new_id]
How was this patch tested?

[x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Was this patch authored or co-authored using generative AI tooling?
no

Thanks @AngersZhuuuu, merged to master

A late LGTM. Thanks for fixing this. Is that possible to backport to 1.8?

authors: AngersZhuuuu bowenliang123 yaooqinn

repo: apache/kyuubi


text:
[Feature][Module Name] Log Analysis Tool

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Problem Description
Business pain points, cumbersome log queries, time-consuming and repetitive work
Description
Automate manual queries, relying solely on task IDs, automatically analyze logs and pull exceptions locally
Use case
No response
Solutions
Implementing automated query analysis using shell scripts
Anything else
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

提供最新版DataSphereStudio全家桶安装服务。
包括（DataSphereStudio, Linkis, Exchangeis(Datax, Sqoop), Scriptis, Streamis(Flink), Visualis, DolphinScheduler)

另外包含 元数据，数据安全，数据质量，数据血缘。（这些都是DataSphereStudio不自带的服务）
包含hadoop集群搭建（hive, spark, flink, kafak, kylin）
提供上述所有系统二次定制开发服务
提供系统搭建运维疑难问题解决

添加备注，DSS大数据
持续为DataSphereStudio开源贡献力量

authors: YinDonglai ahaoyao

repo: apache/linkis


text:
BUGFIX: Remove type onp.bool.

I have to built mxnet locally on my M1 machine and then installed. But when I import the lib it threw AttributeError error with message module 'numpy' has no attribute 'bool'. This PR is a fix for the deprecation issue.
Description
Remove type onp.bool as np.bool was a deprecated alias for the builtin bool with numpy version 1.20.
Checklist
Essentials

[x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[x] Changes are complete (i.e. I finished coding on this PR)
[] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@mxnet-bot run ci

repo: apache/mxnet


text:
NIFI-1421 Update SplitXML to support namespace declarations

I've done a fairly minimal update to SplitXML to correctly support namespace declarations.
Let me know if I've missed anything.

@rpmiskin all looks good - thanks for contributing this back! I have merged to master, so it will be included in the 0.5.0 release. Thanks!

repo: apache/nifi


text:
NIFI-7681 - Add update-bucket-policy command, add option to specify timeout and fix documentation to include previously implemented commands

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-7681
Apart from the description present in the JIRA ticket, a few other miscellaneous changes have also been made:-

Both NiFiClient and NiFiRegistryClient have configs to specify readTimeout and connectionTimeout parameters, however there was no way to specify those parameters in toolkit. I have added a couple of flags, -rto and -cto to specify the read and connection timeouts respectively. They can also be specified in the properties file by their longnames readTimeout and connectionTimeout.

Toolkit documentation had a few commands missing, so I updated it to reflect all previously implemented commands

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

@bbende Thanks for your quick review! I have incorporated the changes you have suggested. Could you please review gain? Thanks once again!

Looks good, thanks! Will merge

authors: bbende jayaaditya


text:
NIFI-8498: Optional removal of fields with UpdateRecord

Field removal from record
Field removal from recursive record structures (record within record)
Field removal from schema only if there is no data in the record for a field
Unit tests

https://issues.apache.org/jira/browse/NIFI-8498

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables removal of fields from records with the UpdateRecord processor as described in
https://issues.apache.org/jira/browse/NIFI-8498
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

I'd like to recommend a little refactor about optionals. I think there's a lot of branching among the private methods based on optional values. My idea is to let the methods provide optionals up through the call chain and handle it in the first caller method. I'll show you a sketch, I didn't test the code.
`
private Record getRecordByPath(String recordPath) {
Optional fieldValueOption = RecordPath.compile(recordPath).evaluate(originalRecord).getSelectedFields().findAny();
return fieldValueOption.flatMap(FieldValue::getParentRecord).orElseGet(() -> getRecordByPathNameThisBetter(recordPath));
}
    private Record getRecordByPathNameThisBetter(String recordPath) {
        Path newPath = new Path(recordPath); // Cutting the "/*" from the end.
        if (!newPath.isEmpty()) {
            return getTargetAsRecord(newPath.toString()).orElse(originalRecord);
        }
        return originalRecord;
    }

    private Optional<Record> getTargetAsRecord(String path) {
        Optional<FieldValue> targetFieldOption = RecordPath.compile(path).evaluate(originalRecord).getSelectedFields().findAny();
        //TODO: is it possible that there are two elements in a record with the same path ?
        return targetFieldOption.flatMap(this::getFieldValueAsRecord);
    }

    private Optional<Record> getFieldValueAsRecord(FieldValue fieldValue) {
        return Optional.ofNullable((Record) fieldValue.getValue());
    }

`

@pgyori - should it be closed in favor of #5381 ?

@pvillard31 , it absolutely should, thanks! I completely abandoned this one, sorry. I'll close it now.

Closing this PR as I eventually implemented this functionality in https://github.com/apache/nifi/pull/5381