text:
Standby meta server exits abnormally with core dump after receiving http request "/meta/cluster"

It is found that after sending http request "/meta/cluster" to any standby meta server (for example, to sending request to a local meta server with port 34601, the command would be curl -v 127.0.0.1:34601/meta/cluster), it would exits abnormally with core dump. Running gdb over the core dump file would output following messages:
root@d04a6b37202f:/data/code/incubator-pegasus/onebox/meta1# gdb /data/code/incubator-pegasus/onebox/meta1/pegasus_server 'core.1703160155. meta.default.4.149'
Core was generated by `/data/code/incubator-pegasus/onebox/meta1/pegasus_server config.ini -app_list m'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f686b5fb653 in dsn::replication::meta_http_service::get_cluster_info_handler (this=0x55da218de340, req=..., resp=...) at /data/code/incubator-pegasus/src/meta/meta_http_service.cpp:487
487	        _service->_balancer->get_balance_operation_count(balance_operation_type));
[Current thread is 1 (Thread 0x7f685f73a640 (LWP 423))]
Installing openjdk unwinder
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 52, in <module>
    class Types(object):
  File "/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py", line 66, in Types
    nmethodp_t = gdb.lookup_type('nmethod').pointer()
gdb.error: No type named nmethod.
(gdb) bt
#0  0x00007f686b5fb653 in dsn::replication::meta_http_service::get_cluster_info_handler (this=0x55da218de340, req=..., resp=...) at /data/code/incubator-pegasus/src/meta/meta_http_service.cpp:487
#1  0x00007f686c1a89c7 in std::function<void (dsn::http_request const&, dsn::http_response&)>::operator()(dsn::http_request const&, dsn::http_response&) const (__args#1=..., __args#0=..., this=0x55da21f73660) at /usr/include/c++/11/bits/std_function.h:590
#2  dsn::http_server::serve (this=<optimized out>, msg=0x55da220648f0) at /data/code/incubator-pegasus/src/http/http_server.cpp:152
#3  0x00007f686c205d6b in std::function<void (dsn::message_ex*)>::operator()(dsn::message_ex*) const (__args#0=<optimized out>, this=0x55da219387d0) at /usr/include/c++/11/bits/std_function.h:586
#4  dsn::rpc_request_task::exec (this=0x55da21938700) at /data/code/incubator-pegasus/src/runtime/task/task.h:435
#5  0x00007f686c2083cf in dsn::task::exec_internal (this=0x55da21938700) at /data/code/incubator-pegasus/src/runtime/task/task.cpp:173
#6  0x00007f686c22285a in dsn::task_worker::loop (this=0x55da21f44f00) at /data/code/incubator-pegasus/src/runtime/task/task_worker.cpp:245
#7  0x00007f686a031253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f6869cbaac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#9  0x00007f6869d4bbf4 in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) f 1
#1  0x00007f686c1a89c7 in std::function<void (dsn::http_request const&, dsn::http_response&)>::operator()(dsn::http_request const&, dsn::http_response&) const (__args#1=..., __args#0=..., this=0x55da21f73660) at /usr/include/c++/11/bits/std_function.h:590
590		return _M_invoker(_M_functor, std::forward<_ArgTypes>(__args)...);
(gdb) f 0
#0  0x00007f686b5fb653 in dsn::replication::meta_http_service::get_cluster_info_handler (this=0x55da218de340, req=..., resp=...) at /data/code/incubator-pegasus/src/meta/meta_http_service.cpp:487
487	        _service->_balancer->get_balance_operation_count(balance_operation_type));
(gdb) list
482	        _meta_function_level_VALUES_TO_NAMES.find(_service->get_function_level())->second + 3);
483	    std::vector<std::string> balance_operation_type;
484	    balance_operation_type.emplace_back("detail");
485	    tp.add_row_name_and_data(
486	        "balance_operation_count",
487	        _service->_balancer->get_balance_operation_count(balance_operation_type));
488	    double primary_stddev, total_stddev;
489	    _service->_state->get_cluster_balance_score(primary_stddev, total_stddev);
490	    tp.add_row_name_and_data("primary_replica_count_stddev", primary_stddev);
491	    tp.add_row_name_and_data("total_replica_count_stddev", total_stddev);
(gdb) p _service
$1 = (dsn::replication::meta_service *) 0x55da21aaa000
(gdb) p _service->_balancer
$2 = std::shared_ptr<dsn::replication::server_load_balancer> (empty) = {get() = 0x0}

It is obvious that the direct reason for core dump is _service->_balancer is null. The root cause for this problem is that since meta server is built with DSN_MOCK_TEST, besides primary meta server, redirect_if_not_primary would also return true for any standby meta server,  whose _service->_balancer is certainly null.
bool meta_http_service::redirect_if_not_primary(const http_request &req, http_response &resp)
{
#ifdef DSN_MOCK_TEST
    return true;
#endif
    rpc_address leader;
    if (_service->_failure_detector->get_leader(&leader))
        return true;
    // set redirect response
    resp.location = "http://" + leader.to_std_string() + '/' + req.path;
    if (!req.query_args.empty()) {
        resp.location += '?';
        for (const auto &i : req.query_args) {
            resp.location += i.first + '=' + i.second + '&';
        }
        resp.location.pop_back(); // remove final '&'
    }
    resp.location.erase(std::remove(resp.location.begin(), resp.location.end(), '\0'),
                        resp.location.end()); // remove final '\0'
    resp.status_code = http_status_code::temporary_redirect;
    return false;
}

This issue has been fixed by https://github.com/apache/incubator-pegasus/pull/1816.

repo: apache/incubator-pegasus


text:
Fix the issue that PinotFs doesn't init before the first get in SegmentGenerationJobRunner

Description
Fix the issue that PinotFs doesn't init before the first get in SegmentGenerationJobRunner
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document

I think PinotFS should always be initialized first before launching the job. Is there a scenario where we don't get a chance to init the PinotFS?

Codecov Report

Merging #6759 (77ab7aa) into master (e9170aa) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #6759      +/-   ##
============================================
- Coverage     43.35%   43.34%   -0.01%
  Complexity        7        7
============================================
  Files          1411     1411
  Lines         68666    68666
  Branches       9918     9918
============================================
- Hits          29768    29765       -3
- Misses        36373    36386      +13
+ Partials       2525     2515      -10

Flag
Coverage Δ
Complexity Δ

integration
43.34% <ø> (-0.01%)
0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

...ion/groupby/AggregationGroupByTrimmingService.java
66.66% <0.00%> (-27.54%)
0.00% <0.00%> (ø%)

...e/pinot/core/segment/index/loader/LoaderUtils.java
57.77% <0.00%> (-13.34%)
0.00% <0.00%> (ø%)

.../filter/predicate/InPredicateEvaluatorFactory.java
55.17% <0.00%> (-11.50%)
0.00% <0.00%> (ø%)

...t/core/segment/index/readers/DoubleDictionary.java
60.00% <0.00%> (-10.00%)
0.00% <0.00%> (ø%)

.../controller/helix/core/SegmentDeletionManager.java
53.27% <0.00%> (-9.84%)
0.00% <0.00%> (ø%)

...che/pinot/core/segment/memory/PinotDataBuffer.java
54.16% <0.00%> (-7.50%)
0.00% <0.00%> (ø%)

...er/starter/helix/SegmentMessageHandlerFactory.java
69.33% <0.00%> (-6.67%)
0.00% <0.00%> (ø%)

...operator/filter/RangeIndexBasedFilterOperator.java
40.42% <0.00%> (-4.26%)
0.00% <0.00%> (ø%)

...rg/apache/pinot/core/query/utils/idset/IdSets.java
25.00% <0.00%> (-2.78%)
0.00% <0.00%> (ø%)

...server/starter/helix/HelixInstanceDataManager.java
78.84% <0.00%> (-2.57%)
0.00% <0.00%> (ø%)

... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9170aa...77ab7aa. Read the comment docs.

source: gharchive/pull-request

authors: Jackie-Jiang codecov-io fx19880617

repo: apache/incubator-pinot

license_type: permissive

license_source: github-api


text:
[API-Draft][DOC] Add jdbc connector doc

Purpose of this pull request
Add jdbc connector doc

Check list

[x] Code changed are covered with tests, or it does not need tests for reason:
[x] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

Also the file name should be Upper Camel Case

repo: apache/incubator-seatunnel


text:
SCB-317 Merge frontend & service-center release

Signed-off-by: asifdxtreme mohammad.asif.siddiqui1@huawei.com
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run go build go test go fmt go vet to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.4%) to 74.01% when pulling fcc57d8180bf87420929b4e33d33d230251f240e on asifdxtreme:master into 802ae41fe48d0c731c9a6d38813989d161963795 on apache:master.

Codecov Report

Merging #299 into master will increase coverage by 0.52%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #299      +/-   ##
==========================================
+ Coverage   69.84%   70.36%   +0.52%
==========================================
  Files          17       17
  Lines        3455     3455
==========================================
+ Hits         2413     2431      +18
+ Misses        872      853      -19
- Partials      170      171       +1

Impacted Files
Coverage Δ

server/service/tag.go
75.43% <0%> (-0.59%)
:arrow_down:

server/service/util/dependency.go
57.78% <0%> (+2.95%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 802ae41...fcc57d8. Read the comment docs.

@WillemJiang for source release we download it from github using tag, and then we sign it
So while pushing it to apache svn we will create a new folder named 'src' and we will put the source release there.

authors: asifdxtreme codecov-io coveralls


text:
Decide SQLs that need a default schema

So that these SQLs can be correctly executed before use db.

Duplicate with #2673

source: gharchive/issue

repo: apache/incubator-shardingsphere

license_source: bigquery


text:
Can not find a way to set KeyGeneratorConfiguration for broadcast table

I can  not find a way to set KeyGeneratorConfiguration for broadcast table.
for example:

't_country' is a broadcast table.(id is the PRIMARY key and AUTO_INCREMENT)
there are two  data-node 'db1' and 'db2'.
in 'db1',the AUTO_INCREMENT of t_country is begin with 100,
in 'db2',the AUTO_INCREMENT of t_country is begin with 200.
insert a record to 't_country'.
the id of the new record from 'db1'  is 101.
the id of the new record  from 'db1'  is 201.

is this a problem?

I think it's ok, broadcast table does't need a global unique primary key, usually we fetch some dictionary info using left-join on country_code as above.

ok. That is reasonable

authors: betterjava cherrylzhao


text:
#3861, PostgreSQL Proxy read parameters error in all types except String

Fixes #3861.

Pull Request Test Coverage Report for Build 1259

0 of 2 (0.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.008%) to 65.982%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

sharding-proxy/sharding-proxy-transport/sharding-proxy-transport-postgresql/src/main/java/org/apache/shardingsphere/shardingproxy/transport/postgresql/packet/command/query/binary/bind/PostgreSQLComBindPacket.java
0
1
0.0%

sharding-proxy/sharding-proxy-transport/sharding-proxy-transport-postgresql/src/main/java/org/apache/shardingsphere/shardingproxy/transport/postgresql/packet/command/query/binary/bind/protocol/PostgreSQLStringBinaryProtocolValue.java
0
1
0.0%

Totals

Change from base Build 643:
-0.008%

Covered Lines:
11434

Relevant Lines:
17329

💛  - Coveralls


text:
[Improve] Improve streampark-test-utils module based on [3.10 Pre-Conditions Checking]

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
[Improve] Improve streampark-test-utils module based on 3.10 Pre-Conditions Checking
Usage Scenario
No response
Related issues
#2986
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

N.A

repo: apache/incubator-streampark


text:
streamx console integrated Dingding group robot alarm function

Feature Request
Is your feature request related to a problem? Please describe:
streamx console  integrated Dingding group robot alarm function
Describe the feature you'd like:
After the task error is reported, the nail group robot alarm function is added on the basis of the original mail alarm
Teachability, Documentation, Adoption, Migration Strategy:
streamx/streamx-console/streamx-console-service/src/main/resources/application.yml file add the follow config
streamx:
dingding:
enabled: false
secret:
accessToken:
url: https://oapi.dingtalk.com/robot/send?access_token=%s&timestamp=%s&sign=%s
Add Dingding alarm code in AlertServiceImpl.java alarm logic, alarm according to streamx.dingding.enabled = true, the default is false

supported

authors: haoziaigemao wolfboys


text:
[STREAMPIPES-<Jira issue #447>] Processing Element: Detect Value Change

Hi,
This is a draft pull request to push my updates incrementally and to seek help. I have created the folder structure and will be implementing the required methods. Any changes in the folder structure can be made.
Hrushikesh

Sure. I'll have a look at it.

repo: apache/incubator-streampipes-extensions


text:
@hos_144

A clear and concise description of what the bug is.
Expected results
what you expected to happen.
Actual results
what actually happens.
Screenshots
If applicable, add screenshots to help explain your problem.
How to reproduce the bug

Go to '...'
Click on '....'
Scroll down to '....'
See error

Environment
(please complete the following information):

superset version: superset version
python version: python --version
node.js version: node -v

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[ ] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.

@Giphyn-nehboh-5qonhu

@mistercrunch this is the third one today. not sure what's going on......

repo: apache/incubator-superset


text:
fix: enforce mandatory chart name on save and edit

SUMMARY
Currently it is possible to save charts without names, causing confusion in the React CRUD list view. This PR changes the following:

Enforce the required slice name field when editing a chart on the React CRUD view.
Migrate the ControlLabel to FormLabel on SaveModal.tsx and add `required props to chart and dashboard and enforce required chart name on both "Save" and "Save & Go To Dashboard".

BEFORE

AFTER

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

@Steejay got it; I switched the order of the buttons. I think the colors are off in the animation, here's what they actually look like:

@villebro great! thanks Ville.

authors: Steejay villebro


text:
[Contrib] Fix error message at callback_get_section_size()

Hi @kevinthesun
This PR improves the error message for raising RuntimeError when calling  callback_get_section_size, I would appreciate if you can help to review/manage it.
many thanks,

cc @weberlo can you take a look?

LGTM. Thanks for the fix!

Thanks @cchung100m

authors: cchung100m tqchen weberlo

repo: apache/incubator-tvm


text:
[Bugfix] fskip of EliminateCommonSubexpr cannot always return false

Originally, fskip returns false at the end of PackedFunc, discards return true in 'cast' case

A little fix for fskip in EliminateCommonSubexpr, it will not return false in 'cast' case.

could you take a look at the ci error?

also can we have a test case?

authors: vinx13 yuliujq yzhliu


text:
[#720][FOLLOW-UP] Correct the shuffle server id

What changes were proposed in this pull request?
Assign the correct shuffle server id
Why are the changes needed?
#720 follow-up pr
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing UTs

I have an idea for ShuffleServerId, and new ShuffleFlushManager with Future<String> shuffleServerId
public class ShuffleServerIdCreator {

  private String ip;
  private Integer grpcPortConf;
  private Integer nettyPortConf;

  private final ExecutorService executor = Executors.newSingleThreadExecutor();
  private final SynchronousQueue<Integer> nettyPortFuture = new SynchronousQueue();

  public ShuffleServerIdCreator(String ip, Integer grpcPortConf, Integer nettyPortConf) {
    this.ip = ip;
    this.grpcPortConf = grpcPortConf;
    this.nettyPortConf = nettyPortConf;
  }

  public Future<String> create() {
    return executor.submit(() -> {
      if (nettyPortConf > 0) {
        return ip + "-" + grpcPortConf + "-" + nettyPortConf;
      } else if (nettyPortConf == 0) {
        return ip + "-" + grpcPortConf + "-" + nettyPortFuture.take();
      } else {
        return ip + "-" + grpcPortConf;
      }
    });
  }

  public void setNettyActualPort(int actualNettyPort) {
    nettyPortFuture.offer(actualNettyPort);
  }
}

I have an idea for ShuffleServerId, and new ShuffleFlushManager with Future<String> shuffleServerId
public class ShuffleServerIdCreator {

  private String ip;
  private Integer grpcPortConf;
  private Integer nettyPortConf;

  private final ExecutorService executor = Executors.newSingleThreadExecutor();
  private final SynchronousQueue<Integer> nettyPortFuture = new SynchronousQueue();

  public ShuffleServerIdCreator(String ip, Integer grpcPortConf, Integer nettyPortConf) {
    this.ip = ip;
    this.grpcPortConf = grpcPortConf;
    this.nettyPortConf = nettyPortConf;
  }

  public Future<String> create() {
    return executor.submit(() -> {
      if (nettyPortConf > 0) {
        return ip + "-" + grpcPortConf + "-" + nettyPortConf;
      } else if (nettyPortConf == 0) {
        return ip + "-" + grpcPortConf + "-" + nettyPortFuture.take();
      } else {
        return ip + "-" + grpcPortConf;
      }
    });
  }

  public void setNettyActualPort(int actualNettyPort) {
    nettyPortFuture.offer(actualNettyPort);
  }
}

A little complex. I fix this issue in another way. You can take a look.

authors: jerqi xumanbu

repo: apache/incubator-uniffle


text:
[Bug][Manager] Stream and source status cannot be updated

What happened
When the enable_create_resource parameter of inlongGroup is set to 0, the stream and source status cannot be updated after the group configuration is completed.
What you expected to happen
When the enable_create_resource parameter of inlongGroup is set to 0, the stream and source status can be updated after the group configuration is completed.
How to reproduce
Create a new data access and set it as unable to create resources.
Environment
No response
InLong version
master
InLong Component
InLong Manager
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I will deal with this problem！

repo: apache/inlong


text:
KAFKA-12789: Remove Stale comments for meta response handling logic

According to my understanding, the following paragraph looks like a stale comments.

public void handleSuccessfulResponse(RequestHeader requestHeader, long now, MetadataResponse response) {
...
// Don't update the cluster if there are no valid nodes...the topic we want may still be in the process of being
// created which means we will get errors and no nodes until it exists
if (response.brokers().isEmpty()) {
log.trace("Ignoring empty metadata response with correlation id {}.", requestHeader.correlationId());
this.metadata.failedUpdate(now);
} else {
this.metadata.update(inProgress.requestVersion, response, inProgress.isPartialUpdate, now);
}
...

The comments above mean we will may get errors and no nodes if the topic we want may still be in the process of being created.
However, every meta request will return all brokers from the logic of the server side, just as followed

def handleTopicMetadataRequest(request: RequestChannel.Request): Unit = {
...
val brokers = metadataCache.getAliveBrokers
...
}

I studied the related git commit history and figured out why.

This comments was first introduced in KAFKA-642 (e11447650a). which means meta request only need brokers related to the topics we want.
KAFKA-1535 (commitId: 4ebcdfd51f) changed the server side logic. which has the metadata response contain all alive brokers rather than just the ones needed for the given topics.
However, this comments are retained till now.
So According to my understanding, this comments looks like a stale one and can be removed.

Thanks for the PR. My question is since this processing logic is retained, what would be the actual reason for returning an empty broker list, other than topic in creation? IMHO, we should update the comment to mention that, instead of just removing the whole comment.

@abbccdda Thanks for your advice. The new comment has been added

repo: apache/kafka


text:
KAFKA-14601: Improve exception handling in KafkaEventQueue

If KafkaEventQueue gets an InterruptedException while waiting for a condition variable, it
currently exits immediately. Instead, it should complete the remaining events exceptionally and
then execute the cleanup event. This will allow us to finish any necessary cleanup steps.
In order to do this, we require the cleanup event to be provided when the queue is contructed,
rather than when it's being shut down.
Also, handle cases where Event#handleException itself throws an exception.
Remove timed shutdown from the event queue code since nobody was using it, and it adds complexity.
Add server-common/src/test/resources/test/log4j.properties since this gradle module somehow avoided
having a test log4j.properties up to this point.

One question I had is, do we see interrupts in production environments, or just on Jenkins? I would expect KafkaEventQueue to normally get shutdown cleanly via the normal shutdown routines, which (AFAIK) don't use interrupts.

Good question. In general our normal shutdown process doesn't use InterruptedException. (At least for the classes that use KafkaEventQueue -- maybe there is someone somewhere using it....) So this is mainly for making Jenkins work smoothly.


text:
[DO NOT MERGE] KAFKA-14533: re-enable the 'false' parameter of SmokeTestDriverIntegrationTest

Doing the PR to re-enable the test build that I temporarily disabled in https://github.com/apache/kafka/pull/13147 in order to help with debugging and to stabilize the 3.4 release branch.
I'm still debugging this so we should not merge this PR just yet, but I wanted to at least open it now so it's not forgotten

cc @lucasbru @mjsax

re-enabled already in https://github.com/apache/kafka/commit/beb0be5fe45d08ff45d6cdddfd86feac8500a7d7#diff-26010bb30ef947a13d4d6d35866c892258dc59dad4d5e2d40c9bb6059fb35381R99


text:
KAFKA-14586: Adding redirection for StreamsResetter

Adding Redirection for StreamsResetter. The erstwhile StreamsResetter resided in core/src/main/scala/kafka/tools package and that's where I added this class.
I tried to run the main method but it fails with
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.kafka.tools.StreamsResetter
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at kafka.tools.StreamsResetter.main(StreamsResetter.java:28)

but I created another class and invoked the class in this PR using the same way and it seemed to invoke this.
cc @mjsax , @fvaleri , @mimaison

Not merging immediately to let @mjsax a chance to review.
Feel free to merge to trunk and 3.5, otherwise I'll do it when I come online tomorrow.

Thanks @fvaleri . I did it but initially had thought I would do it after the merge.

Backported to 3.5 as well.

Thanks all of you!

authors: mimaison mjsax vamossagar12


text:
KAFKA-17217: Batched acknowledge requests per node in ShareConsumeRequestManager

In ShareConsumeRequestManager, currently every time we perform a commitSync/commitAsync/acknowledgeOnClose  we create one AcknowledgeRequestState for each call. But this can be optimised further as we can batch up the acknowledgements to be sent to the same node before the next poll() is invoked.
This will ensure that between 2 polls, the acknowledgements are accumulated in one request per node and then sent during poll, resulting in lesser RPC calls.
To achieve this, we are storing a pair of acknowledge request states for every node, the first value denotes the requestState for commitAsync() and the second value denotes the requestState for commitSync() and acknowledgeOnClose(). All the acknowledgements to be sent to a particular node are stored in the corresponding acknowledgeRequestState based on whether it was synchronous or asynchronous.

Hi @AndrewJSchofield, Thanks for the review, the formatting changes had come in due to a setting by the IDE, I have changed it now. I have addressed the remaining comments and updated the PR.


text:
KNOX-2689 - Redeploying a topology only if it was actually changed

What changes were proposed in this pull request?
From now on, a topology is redeployed only, and only if, there is a change within the topology itself. Not directly in the topology file, because Knox does not count adding/removing comments in the topology as an update. The change has to be a meaningful change such as a change in any of the services, providers, applications, or topology-level attributes.
With this change in place, the possibility to redeploy a topology, by simply touching it, is removed. Instead, clients should use KnoxCLI like this:
knoxcli.sh redeploy --cluster topologyName

E.g. knoxcli.sh redeploy --cluster sandbox

How was this patch tested?
Updated and executed JUnit tests:
$ mvn clean -Dshellcheck=true verify -Prelease,package
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  26:06 min
[INFO] Finished at: 2021-11-15T15:37:32+01:00
[INFO] ------------------------------------------------------------------------

In addition to unit testing, I ran the following E2E test steps.

Cc. @zeroflag

repo: apache/knox


text:
hive job use overrided MR job configuration by cube properties

This patch allows hive mr job use overrided MR job configuration by cube properties.
The test case which uses hadoop-2.6.0,hbase-1.1.3 and hive-1.1.0 passes as expected.

@liyang-gmt8 @shaofengshi ,Can you help me review it?

repo: apache/kylin


text:
[Question] [linkis-computation-engineconn] Flow task，ec log Failed to submit task1 scala.MatchError: null

Before asking

[X] I have read the How to ask for help (我已阅读如何寻求帮助指引文档).

Your environment
版本信息：linkis1.4.0, dss1.1.2, qualitis0.9.2, hadoop3.1.1, hive3.1.0, spark2.4.5, scala:2.12.17
Describe your questions
问题描述：在工作流中执行组件，ec的stdout日志中报Failed to submit task1 scala.MatchError: null
Eureka service list

eg:
Some logs info or acctch file
ec的stdout日志中报Failed to submit task1 scala.MatchError: null

解决办法：修改linkis-computation-engineconn模块下的ComputationExecutor.scala文件，在def execute(engineConnTask: EngineConnTask)方法下添加如下代码：
case _ =>
  logger.warn(s"task get response is $executeResponse")

close by #4972

authors: peacewong zslali

repo: apache/linkis


text:
define current build outputTimestamp

see https://maven.apache.org/guides/mini/guide-reproducible-builds.html
having a value defined here make it follow your releases, instead of having a value fixed by inheritance from parent

I'll apply the same to release-2.x.

authors: garydgregory hboutemy


text:
Create a simple JMH benchmark to measure FST compilation / traversal times

Description
Over in #12543 we are struggling to measure the performance cost of different ways of creating an on-heap reader/writer.  We have been using the "rough" numbers coming out of Test2BFST runs but this is non-ideal -- it is test code, running with assertions, perhaps doing sub-optimal FST usage (not matching what, say, block tree would do to the terms index).
Let's create a simple micro-benchmark to more readily benchmark FST changes?

We can use https://github.com/apache/lucene/pull/12879 as a benchmark candidate to compare against the current baseline.

I can look into this. Is this place https://github.com/apache/lucene/tree/main/lucene/benchmark/src/java/org/apache/lucene/benchmark the correct path to add the benchmark, or is https://github.com/mikemccand/luceneutil a better place?
I think we can try to create a FST with term from some corpus, like wikipedia, then repeatedly call Util.get(), and measure the op/sec, etc. How does that sound?

I can look into this. Is this place https://github.com/apache/lucene/tree/main/lucene/benchmark/src/java/org/apache/lucene/benchmark the correct path to add the benchmark, or is https://github.com/mikemccand/luceneutil a better place?

I think maybe luceneutil?  We would want to feed it a fairly large set of terms, ideally just extracted from a Lucene index using something like what IndexToFST (in luceneutil) does?

I think we can try to create a FST with term from some corpus, like wikipedia, then repeatedly call Util.get(), and measure the op/sec, etc (JMH should give us that). How does that sound?

+1, sounds great!  I can help fold this into the nightly charts...

Thanks @dungba88.

repo: apache/lucene


text:
METRON-1801 Allow Customization of Elasticsearch Document ID

Currently, the Metron GUID is always used as the Elasticsearch document ID. As documented in METRON-1677, using a randomized UUID like Java's UUID.randomUUID() can negatively impact Elasticsearch performance.  This change allows a user to customize the identifier that is used by Elasticsearch when indexing documents.
A property es.document.id was added that sets the message field that is used to define the document ID when a message is indexed by Elasticsearch.

To allow Elasticsearch to define its own document id, this property should be set to a blank or empty string. The client will not set the document ID and Elasticsearch will define its own.

In most cases allowing Elasticsearch to define the document ID is the most performant option. This is the default behavior.

Metron versions 0.6.0 and earlier defined the document ID using the Metron GUID, which is a randomized UUID using Java's UUID.randomUUID(). Using a randomized UUID can negatively impact Elasticsearch indexing performance. To maintain backwards compatibility with legacy versions of Metron use the following setting.
es.document.id = guid

To use a custom document ID, create an enrichment that defines a new message field; for example one called my_document_id. Then use this field to set the document ID as follows. This will set the document ID to the value of the message field my_document_id.
es.document.id = my_document_id

If a message does not contain the es.document.id field, a warning is issued and no document ID is set by the client.

Changes

The ElasticsearchWriter was updated to allow the document ID to be configurable.

A 'search by GUID' in the REST layer was implicitly using the document ID, whereas it should be using the Metron GUID.

Search results should use the Metron GUID as the ID returned to the UI.  All IDs visible to the user should always be the Metron GUID, not the document ID.

The MPack was updated to allow the user to define the es.document.id on the Metron > Config > Index Settings tab.

The default behavior was changed to allow Elasticsearch to set the document ID.  This is the most performant option in most cases.  I updated the Upgrading.md doc to describe how to revert to the legacy behavior.

Testing

Spin-up a development environment.  You may need to stop the PCAP and/or Profiler topology to free-up slots to allow indexing to occur.
cd metron-deployment/development/centos6
vagrant up

Open the Alerts UI and ensure that alerts are visible.  Notice that the ID listed in the table has not changed.  This will always display the Metron GUID, no matter what ID used for the document.

Click on a GUID in the table to search for a single alert.

Create a meta-alert and ensure that alerts tied to the meta-alert are still discoverable by GUID.

Open Ambari and go to Metron > Configs > Index Settings.  Notice that the default setting of es.document.id is blank, thus allowing Elasticsearch to define its own document ID.

Open Kibana and verify that indeed Elasticsearch is generated its own document IDs.  You will notice an _id field which has been generated by Elasticsearch.  This will be different than the UUID generated by Metron and stored as part of the document as guid.

Stop the indexing topologies using Ambari.

Login to the VM.
vagrant ssh
sudo su -

Delete the existing indices in Elasticsearch.
curl -XDELETE http://node1:9200/bro*
curl -XDELETE http://node1:9200/snort*

Open Ambari, and go to Metron > Configs > Index Settings.  Edit the "Elasticsearch Document ID Source Field" and set it to 'guid'.  This will restore the legacy behavior where the document ID is to the Metron GUID.

Restart the Indexing Topology in Ambari.

Open the Alerts UI and ensure that alerts are visible.  Notice that the ID listed in the table has not changed.  This will always display the Metron GUID, no matter what ID used for the document.

Click on a GUID in the table to search for a single alert.

Create a meta-alert and ensure that alerts tied to the meta-alert are still discoverable by GUID.

Open Kibana and verify that the document ID matches the Metron GUID.

I would also advise running the UI e2e test suite with this change.

Pull Request Checklist

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?
[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
[x] Have you included steps or a guide to how the change may be verified and tested manually?
[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
[x] Have you written or updated unit tests and or integration tests to verify your changes?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

Thanks, Nick. So if es.document.id is not provided, as the default, doc id won't be send to ES indexing, right? I guess it would be also nice to provide some guidance on how document ID should be defined (in the case of custom ID). Otherwise, users may create some serious issues with the indexing and search throughput.

@mraliagha I updated the README to (hopefully) better explain your options in using es.document.id.  I sensed by your question that what I had originally was not very clear.

@nickwallen in the case of event logs and the fact that retrieval segmentation would be mostly based on timestamp, it is recommended to use timestamp as a prefix of the id. For example, something like timestamp+hash(original_string).

@mraliagha, that's a good suggestion.  I believe we can functionally achieve that be creating a custom id field in the format you suggest (with a Stellar field transform) and set that field to be the ES id with the Ambari property exposed in this PR.  Do you feel it's worth documenting as an optimization?
I spun this up in full dev and ran through all the testing instructions.  Everything worked as advertised.  I think there are just a couple open questions but this is pretty close in my opinion.

Looks good to me.  +1

@mraliagha, that's a good suggestion. I believe we can functionally achieve that be creating a custom id field in the format you suggest (with a Stellar field transform) and set that field to be the ES id with the Ambari property exposed in this PR. Do you feel it's worth documenting as an optimization?

Yes, I think it is worth documenting as people can easily create serious issues with Lucene based indexers by messing with ID. It can give users an understanding of where it is safe to play with the ID and what the recommendations are. I see if I can find any articles to share it as a part of the manual.

This change was reverted here.  A new pull request will be opened with the functionality. See also this mailing list thread.

authors: merrimanr mraliagha nickwallen

repo: apache/metron


text:
METRON-1986 Batch Profiler Fails to Resolve Stats Stellar Functions

The Batch Profiler does not seem to see the STATS_* functions. When I attempt to run a profile using any of those functions, it fails. It is able to 'see' a limited subset of functions, but not all those that I would expect.
Root Cause
The ClassIndex library is used to create an index of all Stellar functions at compile-time. This index is packaged in the JAR as META-INF/annotations/org.apache.metron.stellar.dsl.Stellar.
When creating a shaded jar using the Maven Shade Plugin, the index files created by ClassIndex are overwritten by default. ClassIndex provides a special transformer for Maven which merges the index files instead of overwriting them.
Without using this transformer, the index files are overwritten and Stellar function resolution will not work. This is why Stellar function resolution is not working.
https://github.com/atteo/classindex#making-shaded-jar
Testing

Spin-up this profile in the Batch Profiler.  It should write successfully to HBase.

{
  "profiles": [
    {
      "profile": "profile-using-stats",
      "foreach": "ip_src_addr",
      "update": { "s": "STATS_ADD(s, length)" },
      "result": "STATS_SUM(s)"
    }
  ],
  "timestampField":"timestamp"
}

Pull Request Checklist

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?
[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
[x] Have you included steps or a guide to how the change may be verified and tested manually?
[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
[x] Have you written or updated unit tests and or integration tests to verify your changes?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

FYI - I added an integration test for this just to try and help avoid future issues like this.  Unfortunately though, the integration test will not catch this issue.  Our integration tests do not run from the shaded jars that we build.  That is a gap in our testing capabilities currently.

As a side note, I found that this plugin has issues if not placed last in the plugins list while working through the ES REST client migration.
https://github.com/apache/metron/pull/1242/files#diff-1f250467ce2f882bfb71cc0928813df8R301

This looks good, thanks for the fix @nickwallen. +1 by inspection.

authors: mmiklavc nickwallen


text:
MINIFICPP-883: Add windows build script

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file?
[ ] If applicable, have you updated the NOTICE file?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@am-c-p-p Can you take a look at this and verify it?

Closing and combining this with #595

repo: apache/nifi-minifi-cpp


text:
NIFI-13759 Fix if statement, so it not throws error

…fy filter parameter

Summary
NIFI-13759
Fix if checks in code, to check the real variable

@exceptionfactory I fixed it https://github.com/apache/nifi-python-extensions/pull/13

repo: apache/nifi-python-extensions


text:
NIFI-3167: Delete button in cluster table wrapping to next line

NIFI-3167:

Ensuring the DELETE button is visible for cluster nodes.
Removing unnecessary wrapper element and nonbreaking spaces.

@mcgilman reviewing

Tested with 3 node cluster on Firefox, Safari and Chrome.  Confirmed existing in Firefox was resolved with patch.
+1
Will merge into master shortly.

authors: YolandaMDavis mcgilman

repo: apache/nifi


text:
NIFI-3393 - replicates cluster/nodes/{id} to cluster coordinator

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit?

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Will review...

Looks good @pvillard31! This has been merged to master.


text:
NIFI-4612: Allow AvroSchemaRegistry to disable name validation

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Code change LGTM, full build OK, merging to master, thanks @mattyb149

Hello Team I am currently facing the hyphen ( dash ) issue in Apache Avro schema that we use along with Schema Registry for deserialization.   Few field names in our Avro Schema contain dashes. I need help to locate the above proposed solution i.e. how to use the AvroSchemaRegistry and the boolean value to disable the validation. I assume its not a property in the Confluent Schema Registry. Can someone share any resource which shows how to bring in AvroSchemaRegistry in picture with the Avro Schema so that I can set the property to disable the field name validation somwhere.
We Specify the .avsc files in aspring boot microservice which after compiled generated the corresponging POJOS that we use in other micro-services (spring boot apps). We do not have any other Java implementation to manipulate the avro schemas.

How do the other apps work with an invalid schema? If you need to "normalize" the names so the schema is valid (and the data is not already in Avro format), I wrote a blog post showing how to do that.

Hi @mattyb149 , other apps dont work with invalid schema . What I mean in until now we did not have any case where avro schema field name had dash ( hiphen) in it.   But now in some use cases we do have field names with dash ( hiphen ) which does not get compiled by Avro.  So can you please help me to understand where and how can I implement / incorporate the given solution in this ticket to disable name validation.


text:
NIFI-9878 fix for hanging client thread with handshake does not complete

Summary
NIFI-9878
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 8
[x] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Very nice work @jlshoem
Tested flow as in jira ticket before and after code changes were done.  Set up ListenHTTP (with random port number) and used that same port for server port for DistributedMapCacheClientService to mimic issue.
Issue was seen as described before code fixes and issue has been resolved with code fix.
After handshake timeout, flowfiles will go now to failure relationship with handshake timeout error in logs as opposed to accumulating on input queue with no message.  Also after its timeout period, PutDistributedMapCache can now be turned off normally without need to force terminate
Also, additional/similar issue seen with shutting down DistributedMapCacheClientService before code fix (as it never truly shutdown - it went into disabling state - nifi restart was needed to fully disable it).  After code fix DistributedMapCacheClientService will fully shutdown after the handshake timeout even when pointing to the "wrong" port number.  So this is a really nice side effect of this effort.
It would be nice if DistributedMapCacheClientService would never even start if it was pointing to a port that it shouldn't be, as that would be useful in some of the use cases of this ticket, but that is pushing ticket scope.  That can be another ticket

Thank You @exceptionfactory!

authors: NissimShiman jlshoem


text:
Dynamic loading app update

Summary
This PR replaces the dynamically loaded module_initialize function with the attribute((destructor)) keyword, and optimizes the compilation method for dynamic modules. Now, by simply adding the DYNLIB flag in the Makefile, the apps can be compiled as dynamic modules.
Impact
sotest/modprint
Testing
qemu with arm an547

[Experimental Bot, please feedback here]
No, this PR does not fully meet the NuttX requirements.
Missing Information:

Summary:

A clear explanation of why this change is necessary is missing. What specific problems does it solve? Is it a performance improvement, a bug fix, or a new feature?
What is the "functional part of the code" being changed? Be specific – mention the files/modules affected.

Impact:

Many fields are incomplete. Address all impact categories. For example:

Is new feature added? (If so, describe the feature)
Impact on build:  You mention changes in the Makefile. This requires a "YES" and a description of the changes.
Impact on documentation: Does this change necessitate documentation updates?

Testing:

Insufficient detail:

Provide specific OS and version information for your build host(s) (e.g., "Linux Ubuntu 20.04").
"qemu with arm an547" is too vague.  Specify the QEMU version and the exact ARM architecture (e.g., ARMv7-M, ARMv8-A).
No testing logs: You must include actual logs from your testing, both before and after the change, to demonstrate the issue and the fix.

Recommendations:

Expand the Summary:  Provide a clear and concise rationale for the change.
Complete the Impact Section: Address all impact categories with specific details.
Provide Detailed Testing Information: Include OS versions, QEMU details, and actual testing logs.

By addressing these points, your PR will be more informative and easier for reviewers to evaluate.

this pr depends on 13819

depends on  13852

look like ci build break

I restarted the job, let's see if it works.

@anjiahao1 please fix:
====================================================================================
Configuration/Tool: canmv230/remote
2024-10-12 10:36:39
------------------------------------------------------------------------------------
  Cleaning...
  Configuring...
  Building NuttX...
/github/workspace/sources/apps /github/workspace/sources/nuttx
riscv-none-elf-ld: cannot open map file /github/workspace/sources/apps/binmap/init.map: No such file or directory
make[3]: *** [/github/workspace/sources/apps/Application.mk:304: /github/workspace/sources/apps/bin/sh] Error 1
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:53: /github/workspace/sources/apps/system/nsh_install] Error 2
make[1]: *** [Makefile:99: .import] Error 2
make: *** [Makefile:106: import] Error 2
/github/workspace/sources/nuttx

repo: apache/nuttx-apps


text:
How to correctly configure the concurrency of a single function?

We hardware is 12vCPU and 32GB RAM K8s cluster, 3 nodes.
My problem is that whenever I try to configure the container concurrency, such as setting the -c parameter to 2, 5, 10, 50, etc., but now it seems that OpenWhisk can only work correctly when the concurrency is 1.
The specific performance is that the error of preheating the container directly occurs in the function call, as follows：

And the test program also shows a large number of timeouts, and this test runs normally when -c is configured as 1, I don’t know how to set up concurrency：

I have configured value.yaml according to the suggestion you gave in the previous issue, as follows：

At the same time, I also found an obvious problem, that is, when the number of preheated containers reaches 200, it is difficult to configure upwards, and there will be an error that the api gateway resource cannot be found. This is in the previous issue.
My current difficulty is that it is difficult to use OpenWhisk to cope with a large amount of load. The data I currently get is only 200TPS, which is a very bad value. My function is very simple, just remotely calling a python3 runtime function of a redis database.
Could you give me some advice?

this is my helm chart value.yaml file:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# This file defines the default values for all variables
# used in the OpenWhisk Helm Chart.  The variables are grouped
# by component into subtrees.
#
# You _MUST_ override the default value of some of the whisk.ingress variables
# to reflect your specific Kubernetes cluster.  For details, see the appropriate
# one of these files:
#   docs/k8s-docker-for-mac.md
#   docs/k8s-aws.md
#   docs/k8s-ibm-public.md
#   docs/k8s-google.md
#   docs/k8s-diy.md (for do-it-yourself clusters).
#
# Production deployments _MUST_ override the default credentials
# that are used in whisk.auth and db.auth.
#
# The file docs/configurationChoices.md discusses other common
# configuration options for OpenWhisk and which variables to override
# to enable them.
#
# The file values-metadata.yaml contains a description of each
# of these variables and must also be updated when any changes are
# made to this file.

# Overall configuration of OpenWhisk deployment
whisk:
  # Ingress defines how to access OpenWhisk from outside the Kubernetes cluster.
  # Only a subset of the values are actually used on any specific type of cluster.
  # See the "Configuring OpenWhisk section" of the docs/k8s-*.md that matches
  # your cluster type for details on what values to provide and how to get them.
  ingress:
    apiHostName: 192.168.1.21
    apiHostPort: 31001
    apiHostProto: "https"
    type: NodePort
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    domain: "domain"
    awsSSL: "false"
    useInternally: false
    tls:
      enabled: true
      createsecret: true
      secretname: "ow-ingress-tls-secret"
      secrettype: "type"
      crt: "server.crt"
      key: "server.key"
  # Production deployments _MUST_ override these default auth values
  auth:
    system: "789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP"
    guest: "23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP"
  systemNameSpace: "/whisk.system"
  limits:
    actionsInvokesPerminute: "50000000"
    actionsInvokesConcurrent: "50000000"
    triggersFiresPerminute: "50000000"
    actionsSequenceMaxlength: "50000000"
    actions:
      time:
        min: "100ms"
        max: "5m"
        std: "1m"
      memory:
        min: "16m"
        max: "4096m"
        std: "64m"
      concurrency:
        min: 1
        max: 2000
        std: 1
      log:
        min: "0m"
        max: "10m"
        std: "10m"
    activation:
      payload:
        max: "1048576"
  loadbalancer:
    blackboxFraction: "100%"
    timeoutFactor: 2
  # Kafka configuration. For all sub-fields a value of "" means use the default from application.conf
  kafka:
    replicationFactor: ""
    topics:
      prefix: ""
      cacheInvalidation:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
      completed:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
      events:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
      health:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
      invoker:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
      scheduler:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
      creationAck:
        segmentBytes: ""
        retentionBytes: ""
        retentionMs: ""
  containerPool:
    userMemory: "65536m"
  runtimes: "runtimes1.json"
  durationChecker:
    timeWindow: "1 d"
  testing:
    includeTests: true
    includeSystemTests: false
  versions:
    openwhisk:
      buildDate: "2022-10-14-13:44:50Z"
      buildNo: "20221014"
      gitTag: "ef725a653ab112391f79c274d8e3dcfb915d59a3"
    openwhiskCli:
      tag: "1.1.0"
    openwhiskCatalog:
      gitTag: "1.0.0"
    openwhiskPackageAlarms:
      gitTag: "2.3.0"
    openwhiskPackageKafka:
      gitTag: "2.1.0"

k8s:
  domain: cluster.local
  dns: kube-dns.kube-system
  persistence:
    enabled: true
    hasDefaultStorageClass: true
    explicitStorageClass: openwhisk-nfs

# Images used to run auxillary tasks/jobs
utility:
  imageName: "openwhisk/ow-utils"
  imageTag: "ef725a6"
  imagePullPolicy: "IfNotPresent"

# Docker registry
docker:
  registry:
    name: ""
    username: ""
    password: ""
  timezone: "UTC"

# zookeeper configurations
zookeeper:
  external: false
  imageName: "zookeeper"
  imageTag: "3.4"
  imagePullPolicy: "IfNotPresent"
  # Note: Zookeeper's quorum protocol is designed to have an odd number of replicas.
  replicaCount: 1
  restartPolicy: "Always"
  connect_string: null
  host: null
  port: 2181
  serverPort: 2888
  leaderElectionPort: 3888
  persistence:
    size: 256Mi
  # Default values for entries in zoo.cfg (see Apache Zookeeper documentation for semantics)
  config:
    tickTime: 2000
    initLimit: 5
    syncLimit: 2
    dataDir: "/data"
    dataLogDir: "/datalog"

# kafka configurations
kafka:
  external: false
  imageName: "wurstmeister/kafka"
  imageTag: "2.12-2.3.1"
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  connect_string: null
  port: 9092
  persistence:
    size: 2Gi

# Database configuration
db:
  external: false
  # Should we run a Job to wipe and re-initialize the database when the chart is deployed?
  # This should always be true if external is false.
  wipeAndInit: true
  imageName: "apache/couchdb"
  imageTag: "2.3"
  imagePullPolicy: "IfNotPresent"
  # NOTE: must be 1 (because initdb.sh enables single node mode)
  replicaCount: 1
  restartPolicy: "Always"
  host: null
  port: 5984
  provider: "CouchDB"
  protocol: "http"
  # Production deployments _MUST_ override the default user/password values
  auth:
    username: "whisk_admin"
    password: "some_passw0rd"
  dbPrefix: "test_"
  activationsTable: "test_activations"
  actionsTable: "test_whisks"
  authsTable: "test_subjects"
  persistence:
    size: 5Gi

# CouchDB, ElasticSearch
activationStoreBackend: "CouchDB"

# Nginx configurations
nginx:
  imageName: "nginx"
  imageTag: "1.21.1"
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  httpPort: 80
  httpsPort: 443
  httpsNodePort: 31001
  httpNodePort: 31005
  workerProcesses: "auto"
  certificate:
    external: false
    cert_file: ""
    key_file: ""
    sslPassword: ""

# Controller configurations
controller:
  imageName: "openwhisk/controller"
  imageTag: "ef725a6"
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  port: 8080
  options: ""
  jvmHeapMB: "4096"
  jvmOptions: ""
  loglevel: "INFO"

# Scheduler configurations
scheduler:
  enabled: false
  imageName: "openwhisk/scheduler"
  imageTag: "ef725a6"
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  endpoints:
    akkaPort: 25520
    port: 8080
    rpcPort: 13001
  options: ""
  jvmHeapMB: "4096"
  jvmOptions: ""
  loglevel: "INFO"
  protocol: "http"
  maxPeek: 10000
  # Sometimes the kubernetes client takes a long time for pod creation
  inProgressJobRetention: "600 seconds"
  blackboxMultiple: 100
  dataManagementService:
    retryInterval: "1 second"
  queueManager:
    maxSchedulingTime: "600 seconds"
    maxRetriesToGetQueue: "60"
  queue:
    idleGrace: "60 seconds"
    stopGrace: "60 seconds"
    flushGrace: "120 seconds"
    gracefulShutdownTimeout: "15 seconds"
    maxRetentionSize: 1000000
    maxRetentionMs: 6000000
    maxBlackboxRetentionMs: 3000000
    throttlingFraction: 1.0
    durationBufferSize: 100
  scheduling:
    staleThreshold: "100ms"
    checkInterval: "100ms"
    dropInterval: "10 minutes"

# etcd (used by scheduler and controller if scheduler is enabled)
etcd:
  # NOTE: external etcd is not supported yet
  external: false
  clusterName: ""
  imageName: "quay.io/coreos/etcd"
  imageTag: "v3.4.0"
  imagePullPolicy: "IfNotPresent"
  # NOTE: setting replicaCount > 1 will not work; need to add etcd cluster configuration
  replicaCount: 1
  restartPolicy: "Always"
  port: 2379
  leaseTimeout: 1
  poolThreads: 15
  persistence:
    size: 5Gi

# Invoker configurations
invoker:
  imageName: "openwhisk/invoker"
  imageTag: "ef725a6"
  imagePullPolicy: "IfNotPresent"
  restartPolicy: "Always"
  runtimeDeleteTimeout: "30 seconds"
  port: 8080
  options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider"
  jvmHeapMB: "4096"
  jvmOptions: ""
  loglevel: "INFO"
  containerFactory:
    useRunc: false
    impl: "kubernetes"
    enableConcurrency: true
    networkConfig:
      name: "bridge"
      dns:
        inheritInvokerConfig: true
        overrides:          # NOTE: if inheritInvokerConfig is true, all overrides are ignored
          # Nameservers, search, and options are space-separated lists
          # eg nameservers: "1.2.3.4 1.2.3.5 1.2.3.6" is a list of 3 nameservers
          nameservers: ""
          search: ""
          options: ""
    kubernetes:
      isolateUserActions: true
      replicaCount: 1

# API Gateway configurations
apigw:
  imageName: "openwhisk/apigateway"
  imageTag: "1.0.0"
  imagePullPolicy: "IfNotPresent"
  # NOTE: setting replicaCount > 1 is not tested and may not work
  replicaCount: 1
  restartPolicy: "Always"
  apiPort: 9000
  mgmtPort: 8080

# Redis (used by apigateway)
redis:
  external: false
  imageName: "redis"
  imageTag: "4.0"
  imagePullPolicy: "IfNotPresent"
  # NOTE: setting replicaCount > 1 will not work; need to add redis cluster configuration
  replicaCount: 1
  restartPolicy: "Always"
  host: null
  port: 6379
  persistence:
    size: 2Gi

# User-events configuration
user_events:
  imageName: "openwhisk/user-events"
  imageTag: "ef725a6"
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  port: 9095

# Prometheus configuration
prometheus:
  imageName: "prom/prometheus"
  imageTag: v2.14.0
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  port: 9090
  persistence:
    size: 1Gi
  persistentVolume:
    mountPath: /prometheus/

# Grafana configuration
grafana:
  imageName: "grafana/grafana"
  imageTag: "6.3.0"
  imagePullPolicy: "IfNotPresent"
  replicaCount: 1
  restartPolicy: "Always"
  port: 3000
  adminPassword: "admin"
  dashboards:
  - https://raw.githubusercontent.com/apache/openwhisk/master/core/monitoring/user-events/compose/grafana/dashboards/openwhisk_events.json
  - https://raw.githubusercontent.com/apache/openwhisk/master/core/monitoring/user-events/compose/grafana/dashboards/global-metrics.json
  - https://raw.githubusercontent.com/apache/openwhisk/master/core/monitoring/user-events/compose/grafana/dashboards/top-namespaces.json

# Metrics
metrics:
  # set true to enable prometheus exporter
  prometheusEnabled: false
  # passing prometheus-enabled by a config file, required by openwhisk
  whiskconfigFile: "whiskconfig.conf"
  # set true to enable Kamon
  kamonEnabled: false
  # set true to enable Kamon tags
  kamonTags: false
  # set true to enable user metrics
  userMetricsEnabled: false

# Configuration of OpenWhisk event providers
providers:
  # CouchDB instance used by all enabled providers to store event/configuration data.
  db:
    external: false
    # Define the rest of these values if you are using external couchdb instance
    host: "10.10.10.10"
    port: 5984
    protocol: "http"
    username: "admin"
    password: "secret"
  # Alarm provider configurations
  alarm:
    enabled: true
    imageName: "openwhisk/alarmprovider"
    imageTag: "2.3.0"
    imagePullPolicy: "IfNotPresent"
    # NOTE: replicaCount > 1 doesn't work because of the PVC
    replicaCount: 1
    restartPolicy: "Always"
    apiPort: 8080
    dbPrefix: "alm"
    persistence:
      size: 1Gi
  # Kafka provider configurations
  kafka:
    enabled: true
    imageName: "openwhisk/kafkaprovider"
    imageTag: "2.1.0"
    imagePullPolicy: "IfNotPresent"
    # NOTE: setting replicaCount > 1 has not been tested and may not work
    replicaCount: 1
    restartPolicy: "Always"
    apiPort: 8080
    dbPrefix: "kp"

busybox:
  imageName: "busybox"
  imageTag: "latest"

# Used to define pod affinity and anti-affinity for the Kubernetes scheduler.
# If affinity.enabled is true, then all of the deployments for the OpenWhisk
# microservices will use node and pod affinity directives to inform the
# scheduler how to best distribute the pods on the available nodes in the cluster.
affinity:
  enabled: true
  coreNodeLabel: core
  edgeNodeLabel: edge
  invokerNodeLabel: invoker
  providerNodeLabel: provider

# Used to define toleration for the Kubernetes scheduler.
# If tolerations.enabled is true, then all of the deployments for the OpenWhisk
# microservices will add tolerations for key openwhisk-role with specified value and effect NoSchedule.
toleration:
  enabled: true
  coreValue: core
  edgeValue: edge
  invokerValue: invoker

# Used to define the probes timing settings so that you can more precisely control the
# liveness and readiness checks.
# initialDelaySeconds - Initial wait time to start probes after container has started
# periodSeconds - Frequency to perform the probe, defaults to 10, minimum value is 1
# timeoutSeconds - Probe will timeouts after defined seconds, defaults to 1 second,
#                  minimum value is 1
# for more information please refer - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
# Note - for now added probes settings for zookeeper, kafka, and controller only.
#        in future all components probes timing settings should be configured here.

probes:
  zookeeper:
    livenessProbe:
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 1
    readinessProbe:
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 1
  kafka:
    livenessProbe:
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 1
    readinessProbe:
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 5
  controller:
    livenessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1
  scheduler:
    livenessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1
    readinessProbe:
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 1

# Pod Disruption Budget allows Pods to survive Voluntary and Involuntary Disruptions.
# for more information refer - https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
# Pod Disruptions budgets currently supported for the pods which are managed
# by one of Kubernetes built-in controllers (Deployment, ReplicationController,
# ReplicaSet, StatefulSet).
# Caveats -
# - You can specify numbers of maxUnavailable Pods for now as integer. % values are not
# supported.
# - minAvailable is not supported
# - PDB only applicable when replicaCount is greater than 1.
# - Only zookeeper, kafka, invoker and controller pods are supported for PDB for now.
# - Invoker PDB only applicable if containerFactory implementation is of type "kubernetes"

pdb:
  enable: false
  zookeeper:
    maxUnavailable: 1
  kafka:
    maxUnavailable: 1
  controller:
    maxUnavailable: 1
  invoker:
    maxUnavailable: 1
  elasticsearch:
    maxUnavailable: 1

# ElasticSearch configuration
elasticsearch:
  external: false
  clusterName: "elasticsearch"
  nodeGroup: "master"
  # The service that non master groups will try to connect to when joining the cluster
  # This should be set to clusterName + "-" + nodeGroup for your master group
  masterServiceValue: ""
  # Elasticsearch roles that will be applied to this nodeGroup
  # These will be set as environment variables. E.g. node.master=true
  roles:
    master: "true"
    ingest: "true"
    data: "true"
  replicaCount: 1
  minimumMasterNodes: 1
  esMajorVersionValue: ""
  # Allows you to add any config files in /usr/share/elasticsearch/config/
  # such as elasticsearch.yml and log4j2.properties, e.g.
  #  elasticsearch.yml: |
  #    key:
  #      nestedkey: value
  #  log4j2.properties: |
  #    key = value
  esConfig: {}
  # Extra environment variables to append to this nodeGroup
  # This will be appended to the current 'env:' key. You can use any of the kubernetes env
  # syntax here
  #  - name: MY_ENVIRONMENT_VAR
  #    value: the_value_goes_here
  extraEnvs: []
  # Allows you to load environment variables from kubernetes secret or config map
  # - secretRef:
  #     name: env-secret
  # - configMapRef:
  #     name: config-map
  envFrom: []
  # A list of secrets and their paths to mount inside the pod
  # This is useful for mounting certificates for security and for mounting
  # the X-Pack license
  #  - name: elastic-certificates
  #    secretName: elastic-certificates
  #    path: /usr/share/elasticsearch/config/certs
  #    defaultMode: 0755
  secretMounts: []
  image: "elasticsearch"
  imageTag: "latest"
  imagePullPolicy: "IfNotPresent"
  podAnnotations: {}
  labels: {}
  esJavaOpts: "-Xmx1g -Xms1g"
  resources:
    requests:
      cpu: "1500m"
      memory: "4Gi"
    limits:
      cpu: "3000m"
      memory: "8Gi"
  initResources: {}
  sidecarResources: {}
  networkHost: "0.0.0.0"
  volumeClaimTemplate:
    accessModes: [ "ReadWriteOnce" ]
    resources:
      requests:
        storage: 30Gi
  rbac:
    create: false
    serviceAccountName: ""
  podSecurityPolicy:
    create: false
    name: ""
    spec:
      privileged: true
      fsGroup:
        rule: RunAsAny
      runAsUser:
        rule: RunAsAny
      seLinux:
        rule: RunAsAny
      supplementalGroups:
        rule: RunAsAny
      volumes:
        - secret
        - configMap
        - persistentVolumeClaim
  persistence:
    annotations: {}
  extraVolumes: []
  # - name: extras
  #   mountPath: /usr/share/extras
  #   readOnly: true
  extraVolumeMounts: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']
  extraContainers: []
  # - name: do-something
  #   image: busybox
  #   command: ['do', 'something']
  extraInitContainers: []
  # The default is to deploy all pods serially. By setting this to parallel all pods are started at
  # the same time when bootstrapping the cluster
  podManagementPolicy: "Parallel"
  # The environment variables injected by service links are not used, but can lead to slow Elasticsearch boot times when
  # there are many services in the current namespace.
  # If you experience slow pod startups you probably want to set this to `false`.
  enableServiceLinks: true
  protocol: http
  connect_string: null
  host: null
  httpPort: 9200
  transportPort: 9300
  service:
    labels: {}
    labelsHeadless: {}
    type: ClusterIP
    nodePort: ""
    annotations: {}
    httpPortName: http
    transportPortName: transport
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
  updateStrategy: RollingUpdate
  podSecurityContext:
    fsGroup: 1000
    runAsUser: 1000
  securityContext:
    capabilities:
      drop:
      - ALL
    # readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 1000
  # How long to wait for elasticsearch to stop gracefully
  terminationGracePeriod: 120
  sysctlVmMaxMapCount: 262144
  readinessProbe:
    failureThreshold: 3
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 3
    timeoutSeconds: 5
  # https://www.elastic.co/guide/en/elasticsearch/reference/7.8/cluster-health.html#request-params wait_for_status
  clusterHealthCheckParams: "wait_for_status=green&timeout=1s"
  ## Use an alternate scheduler.
  ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
  ##
  schedulerName: ""
  imagePullSecrets: []
  nodeSelector: {}
  tolerations: []
  nameOverride: ""
  fullnameOverride: ""
  # https://github.com/elastic/helm-charts/issues/63
  masterTerminationFix: false
  lifecycle: {}
    # preStop:
    #   exec:
    #     command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
    # postStart:
    #   exec:
    #     command:
    #       - bash
    #       - -c
    #       - |
    #         #!/bin/bash
    #         # Add a template to adjust number of shards/replicas
    #         TEMPLATE_NAME=my_template
    #         INDEX_PATTERN="logstash-*"
    #         SHARD_COUNT=8
    #         REPLICA_COUNT=1
    #         ES_URL=http://localhost:9200
    #         while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done
    #         curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}'
  sysctlInitContainer:
    enabled: true
  keystore: []
  # Deprecated
  # please use the above podSecurityContext.fsGroup instead
  fsGroup: ""
  indexPattern: "openwhisk-%s"
  username: "admin"
  password: "admin"

akka:
  actorSystemTerminateTimeout: "30 s"

@style95 Boss, I would like to ask you how to configure concurrency correctly. At present, as long as we use the -c parameter to configure an Action concurrency greater than 1, a large number of errors will occur.

The performance is highly related to the duration of your action.
Let's say, your action takes 1 second to finish and you have 200 containers.
Then your TPS will be just 200 TPS.
If your action completes in 100ms, your TPS will be 2000 TPS.
And you said, your machines have 32Gb of memory but you configured around 65GB of memory for runtime containers.
It will exceed the limit of your machine and might cause OOM.
Since core components like invokers should be running as well, you need to configure less than 32GB for runtime containers.
And take a look at the duration of your activations.

The performance is highly related to the duration of your action. Let's say, your action takes 1 second to finish and you have 200 containers. Then your TPS will be just 200 TPS. If your action completes in 100ms, your TPS will be 2000 TPS.
And you said, your machines have 32Gb of memory but you configured around 65GB of memory for runtime containers. It will exceed the limit of your machine and might cause OOM. Since core components like invokers should be running as well, you need to configure less than 32GB for runtime containers. And take a look at the duration of your activations.
You can refer to this guide for intra-concurrency. There are still some known issues with it though.

Sorry, I have 3 nodes, so actually I have 32*3GB=96GB memory available, in K8s 2 work nodes have 64GB, and we also config it pool as 48GB, But it look like not good, Although the execution time of Action is very short, only 100ms,etc:

But the result is very confusing, My K6 test very unstable and poor performance, this is 300uvs and get 200TPS:

We also find some internal error:

But, I appreciate your comments, I will study the guide in the link carefully, I think I should turn to nodejs functions to support concurrent access, thanks!
In the future, I will pay more attention to how to improve the TPS of OpenWhisk and solve some existing problems, which will be very helpful to academia and industry.

ok, so you are running only one invoker.
Since the invoker communicates with runtime containers, it's generally better to have more than one invoker in terms of performance. And the userMemory config applies per invoker, so if you add more invokers, you need to decrease the value.
And I can see the "internal error" in activations, do you have any logs regarding that?
And it seems the error has changed from "developer error" to "internal error".

On the whole, most hot starts are successful, but a small number of continuous internal errors:

It seems that the container failed to start or the scheduling failed. I seem to have disabled the activation log:

I use kubectl to check the pod's log, and it turns out that the node is overwhelmed, but in fact, I set the kubelet to have 400 pods per node, but the scheduling still cannot be completed.

errors message is Pod event for failed pod 'wskowdev-invoker-00-1481-prewarm-nodejs16' null: 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 Too many pods.、Pod event for failed pod 'wskowdev-invoker-00-1484-prewarm-nodejs16' null: Successfully assigned openwhisk/wskowdev-invoker-00-1484-prewarm-nodejs16 to work2 and failed starting prewarm, removed

So this is not an OpenWhisk issue.
Your K8S cluster does not have enough resources.

So this is not an OpenWhisk issue. Your K8S cluster does not have enough resources.

Thank you, I need update my cluster!

repo: apache/openwhisk


text:
HDDS-10383. Introduce a Provider for client-side thread resources passing

What changes were proposed in this pull request?
Introduce a Provider for client-side thread resources passing.
Detail:  HDDS-9912

The changes here are split from HDDS-9912, which just introduces BlockOutputStreamResourceProvider and implements its parameter passing.
The changes doesn't actually use BlockOutputStreamResourceProvider's thread resources. So the logic of creating and using threads in BlockOutputStream remains unchanged.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10383
How was this patch tested?
Existing Test.

@xichen01 , thanks for the update! Please see the comment inlined and also https://issues.apache.org/jira/secure/attachment/13066945/6222_review.patch

Thanks for your comment. I have applied it.

repo: apache/ozone


text:
HDDS-10541. Replace GSON with Jackson in debug commands

What changes were proposed in this pull request?
This update addresses compatibility concerns with Java 17 by transitioning from GSON to Jackson in the Debug-Shell module. This change simplifies dependencies and ensures smooth operation across different Java versions.
Parent Jira Issue: HDDS-10538
This patch streamlines our Debug-Shell Implementation.
Affected Classes:
- ChunkKeyHandler.java
- ReadReplicas.java
- ContainerChunkInfo.java

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10541
How was this patch tested?
To test out the changes we have to run :- TestOzoneDebugShell Integration test.
The forked CI also ran successfully.

@dombizita @tanvipenumudy please review

Thanks @ArafatKhan2198 for the patch, @tanvipenumudy for the review.
Tested this with Java 17, works well.


text:
[Question] What are the risks associated with the Java API?

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
https://paimon.apache.org/docs/0.8/program-api/java-api/ comes with a warning at the top

We do not recommend using the Paimon API naked, unless you are a professional downstream ecosystem developer, and even if you do, there will be significant difficulties.

If you are only using Paimon, we strongly recommend using computing engines such as Flink SQL or Spark SQL.

The following documents are not detailed and are for reference only.

Can you elaborate on the difficulties that will be encountered?
Solution
No response
Anything else?
No response
Are you willing to submit a PR?

[x] I'm willing to submit a PR!

The main difficulty is to decide where you should use each class and call each method.
For example, consider a distributed system with one master node and several workers node. TableScan should only be used in master, while TableRead and TableWrite should only be used in workers. Also you need to design how to distribute Splits generated from TableScan to the workers. You also need to be careful on TableCommit because it can only run with 1 parallelism (otherwise the consistency guarantee is broken).
All in all, these things are exactly what you need to concern when designing a distributed system.

repo: apache/paimon


text:
[Bug] Read orc timestamp incorrectly after change the time zone

Search before asking

[X] I searched in the issues and found nothing similar.

Paimon version
0.9-snapshot
Compute Engine
spark & flink
Minimal reproduce step
write orc with timestamp_ltz & timestamp_ntz, then change the timezone, then read them
What doesn't meet your expectations?
Result is uncorrect
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

After reading the ORC code and testing it myself, I found that ORC itself will perform timezone conversion on the returned data.
e.g.
insert into tbl values (timestamp'2024-01-01 00:00:00', timestamp_ntz'2024-01-01 00:00:00')

here is the result from orc timestamp read vector

op
ts_ltz ms
ts_ntz

shanghai write, shanghai read
1704009600000
1704038400000

shanghai write, UTC read
1704009600000
1704067200000

ts_ltz is good because orc reads it in UTC both, paimon can handle them without timezone conversion,
but for ts_ntz, orc has adjusted the result based on the write timezone (read is UTC both for it is ntz), this makes it difficult to integrate with Paimon reader, because we don't know its write timezone(although we can get it from orc schema meta)

Spark found this problem too, see https://github.com/apache/spark/pull/34741#issuecomment-983660633

It seems like the ORC lib (the default behavior) is designed for people who want to deal with java.sql.Timestamp directly, not an engine like Spark that only treats ORC as a storage layer.

Spark's solution:

Write TIMESTAMP_NTZ as ORC int64, with a column property to indicate it's TIMESTAMP_NTZ (writing TIMESTAMP_LTZ should add the column property as well)
Set useUTCTimestamp to true in the reader if the ORC file was written by the latest Spark version


text:
Please delete old releases from mirroring system

To reduce the load on the ASF mirrors, projects are required to delete old releases [1]
Please can you remove all non-current releases?
i.e. the ones listed as affected.
Thanks!
Also, if you have a release guide, perhaps you could add a cleanup stage to it?
[1] http://www.apache.org/dev/release.html#when-to-archive
Reporter: Sebb
Assignee: Ryan Blue / @rdblue
Note: This issue was originally created as PARQUET-506. Please see the migration documentation for further details.

Ryan Blue / @rdblue:
I removed old release artifacts when I added Parquet MR 1.9.0. I'll update the docs to recommend removing old versions when adding new ones.

repo: apache/parquet-java


text:
Enable rendering of Indian languages, by reading and utilizing the GSUB table

Implemented proper rendering of Indian languages, which need extensive Glyph substitution. The GSUB table has been read and used effectively to replace some compound words with their respective Glyphs. All tests are passing. I have tested this for the Bengali font. Please review these changes and let me know if it makes sense to incorporate these.
Thanks,
Palash.

Hello, yes, thanks, there has been a need for this for years... however this is a read only repository so you'll have to open an issue in JIRA ( https://issues.apache.org/jira/browse/PDFBOX ). We'll also need an ICLA, please read https://www.apache.org/licenses/icla.pdf , sign and submit it as explained in https://www.apache.org/licenses/#clas .
I only had a quick look (I just woke up), I noticed that there are a few tables and a font. Where do these come from? I see this is only for test so in the worst case we could download it. (Make a text search for "PDFBOX-4038" to see how this is done). We also need the apache header for any new non binary files.
If not in your proposed patch, it would be nice to have an example with a simple text that would not look OK without your patch.
Does your patch change any API ina "breaking" way? If yes it will be used for the trunk only, if not then it could also be used for 2.0.

The font license is OK, we'll only have to include the license text:
https://www.apache.org/legal/resolved.html#category-b

I entered "The quick brown fox jumps over the lazy dog" into Google translate, and the result was "দ্রুত গাঢ় শেয়াল অলস কুকুর জুড়ে জাম্প". Then I created a PDF file with this. With your font, I got a line with incomplete glyphs:
example-new.pdf
Code:
                PDFont font7 = PDType0Font.load(document, new File("Lohit-Bengali.ttf"));
                stream.setFont(font7, 12);
                stream.newLine();
                stream.showText("দ্রুত গাঢ় শেয়াল অলস কুকুর জুড়ে জাম্প");

Hi Tilman,
Thanks a lot for the quick response. Please find my reply below:

ICLA: Sure, I would send you a signed/scanned copy by 16th April, Monday
apache header for any new non binary files: Done
example with a simple text: Done: /pdfbox-examples/src/main/java/org/apache/pdfbox/examples/pdmodel/BengaliPdfGenerationHelloWorld.java.
Please find attached the before/after pdf files
Bengali-text-after.pdf
Bengali-text-before.pdf
Does your patch change any API in a "breaking" way: I don't think so, the only "major" change I did is made PDFont::encode(String text) non-final, and overrode it in PDType0Font
Incomplete Bengali text with my changes: Not sure, I just ran it, able to get "almost" complete file, using jdk8.
a.pdf

Thanks,
Palash.

Closing the pull request, as the files have been checked in to the trunk by Tilman Hausherr: https://svn.apache.org/r1829710

authors: THausherr paawak

repo: apache/pdfbox


text:
PHOENIX-5140 TableNotFoundException occurs when we create local async…

originally, table.getParentTableName() will return the parent table as dot seperator such as foo.table1, but the physical table name was stored as colon seperator in hbase:meta such as foo:table1
so this fix will get parent physical table name with colon seperator

As @gjacoby126 mentioned in the JIRA looks okay but we would want a test.

authors: clevertension dbwong

repo: apache/phoenix