text:
HDDS-10875. XceiverRatisServer#getRaftPeersInPipeline should be called before XceiverRatisServer#removeGroup

What changes were proposed in this pull request?
From the https://issues.apache.org/jira/browse/HDDS-10750?focusedCommentId=17847435&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17847435 in HDDS-10750, it's found GroupMismatchException are thrown during the ClosePipelineCommandHandler.
This is because XceiverRatisServer#removeGroup is called before XceiverRatisServer#getRaftPeersInPipeline, which causes XceiverRatisServer#getRaftPeersInPipeline to throw GroupMismatchException when it's trying to get the RaftServerProxy#getDivision since the group has been removed.
Therefore, we need to first call the XceiverRatisServer#getRaftPeersInPipeline before calling XceiverRatisServer#removeGroup.
This patch also catch the GroupMismatchException in case the group has been removed by earlier ClosePipelineCommandHandler in other datanode for the same pipeline. The datanode will also try to remove the Ratis group from the other datanodes (ignoring the GroupMismatchException) before removing its own Ratis group.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10875
How was this patch tested?
Clean CI: https://github.com/ivandika3/ozone/actions/runs/9145754999

Thank you for the review @szetszwo.

How about catching GroupMismatchException in the existing try-catch?

Sure, updated. Was using a nested try-catch since GroupMismatchException is an Ratis-specific exception.

In XceiverServerRatis#removeGroup, it seems the Ratis exception during group remove (e.g. GroupMismatchException) is wrapped in IOException, not sure why.
RaftClientReply reply;
    try {
      reply = server.groupManagement(request);
    } catch (Exception e) {
      throw new IOException(e.getMessage(), e);
    }

Currently, I'm using HddsClientUtils.containsException to extract GroupMismatchException from the IOException. However, we can also remove the IOException wrapping as well.

Found RaftRetryFailureException due to org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception. Since the current Raft client policy is from RequestTypeDependentRetryPolicyCreator, it seems the StatusRuntimeException retry policy is  MultipleLinearRandomRetry (others) which might block the datanode when it's trying to remove group from other datanode due to connection timeout.
024-05-19 04:08:53,822 [f3d35daa-4aaf-4c71-973e-2392b4bdb0cc-PipelineCommandHandlerThread-0] WARN  commandhandler.ClosePipelineCommandHandler (ClosePipelineCommandHandler.java:lambda$null$1(131)) - Failed to remove group group-72E5E9088A19 of pipeline PipelineID=8643905b-c6aa-4c00-96b1-72e5e9088a19 on peer de8c5e91-909e-46c9-a24e-1aa548fa4b98
org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed GroupManagementRequest:client-363EA46DA04D->de8c5e91-909e-46c9-a24e-1aa548fa4b98@group-72E5E9088A19, cid=120, seq=null, RW, null, Remove:group-72E5E9088A19, delete-dir for 25 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@313565ca, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@313565ca}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@313565ca, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
	at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:353)
	at org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:129)
	at org.apache.ratis.client.impl.GroupManagementImpl.remove(GroupManagementImpl.java:61)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ClosePipelineCommandHandler.lambda$null$1(ClosePipelineCommandHandler.java:123)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1652)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ClosePipelineCommandHandler.lambda$handle$2(ClosePipelineCommandHandler.java:120)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:223)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupAdd(GrpcClientProtocolClient.java:170)
	at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:98)
	at org.apache.ratis.client.impl.BlockingImpl.sendRequest(BlockingImpl.java:145)
	at org.apache.ratis.client.impl.BlockingImpl.sendRequestWithRetry(BlockingImpl.java:109)
	... 16 more
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
	at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupManagement(AdminProtocolServiceGrpc.java:468)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$groupAdd$5(GrpcClientProtocolClient.java:172)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:221)
	... 20 more
Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /10.1.0.100:15049
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
	at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)
	at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131)
	at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
	at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
	at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
	at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
	at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
	at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)

@szetszwo I have made some changes on the patch. Could you help review this again? Thank you.

+1 to the latest change.

Thank you for the reviews @szetszwo , merged.


text:
HDDS-11009. Increase Surefire fork heap to 8GB

What changes were proposed in this pull request?
Fix OutOfMemoryError in TestRandomKeyGenerator with Ratis 3.1.0 release candidates.
https://issues.apache.org/jira/browse/HDDS-11009
How was this patch tested?
Tested using ci-with-ratis workflow (this run is from a different branch, to workaround HDDS-11042, a bug in the workflow):
https://github.com/adoroszlai/ozone/actions/runs/9615310805
Regular CI:
https://github.com/adoroszlai/ozone/actions/runs/9616439402

Now that Ratis 3.1.0 is released, this can be included in #6872, the PR to bump Ratis.

source: gharchive/pull-request

repo: apache/ozone

license_type: permissive

license_source: github-api


text:
Add new Parquet CLI tools

This issue tracks adding parquet-cli from rdblue/parquet-cli.
Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue
PRs and other links:

PR #384

Note: This issue was originally created as PARQUET-777. Please see the migration documentation for further details.

Ryan Blue / @rdblue:
Merged #384.

source: gharchive/issue

repo: apache/parquet-java


text:
PHOENIX-7268 Namespace mapped system catalog table not snapshotted be…

…fore upgrade

Please fix the checkstyle error, @szucsvillo .

Please fix the checkstyle error, @szucsvillo .

Yes, thanks, I forgot

authors: stoty szucsvillo

repo: apache/phoenix

license_source: bigquery


text:
FileWriter bug fixes

This PR includes 2 bug fixes

FileWriter was appending header line by default. Pushed this down to CsvWriter where its required.
Ensure that the number of rows written to the files is equal to the value provided in the spec.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (50912eb) 61.52% compared to head (414b24d) 46.41%.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12208       +/-   ##
=============================================
- Coverage     61.52%   46.41%   -15.11%
+ Complexity     1152      943      -209
=============================================
  Files          2415     1809      -606
  Lines        131157    95158    -35999
  Branches      20245    15352     -4893
=============================================
- Hits          80692    44171    -36521
- Misses        44571    47833     +3262
+ Partials       5894     3154     -2740

Flag
Coverage Δ

custom-integration1
?

integration
?

integration1
?

integration2
?

java-11
?

java-21
46.41% <ø> (-14.99%)
:arrow_down:

skip-bytebuffers-false
46.41% <ø> (-15.09%)
:arrow_down:

skip-bytebuffers-true
?

temurin
46.41% <ø> (-15.11%)
:arrow_down:

unittests
46.41% <ø> (-15.11%)
:arrow_down:

unittests1
46.41% <ø> (-0.20%)
:arrow_down:

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

repo: apache/pinot


text:
[Minor] Small refactor of raw index creator constructor to be more clear

For version before v4, only numDocsPerChunk is required
For version v4, only chunkSize is required

Codecov Report
Attention: Patch coverage is 0% with 23 lines in your changes are missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (453b99c).
Report is 409 commits behind head on master.

Files
Patch %
Lines

...r/impl/fwd/MultiValueFixedByteRawIndexCreator.java
0.00%
8 Missing :warning:

...tor/impl/fwd/MultiValueVarByteRawIndexCreator.java
0.00%
8 Missing :warning:

...or/impl/fwd/SingleValueVarByteRawIndexCreator.java
0.00%
7 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13093       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2432        -4
  Lines        133233   134148      +915
  Branches      20636    20778      +142
=============================================
- Hits          82274        0    -82274
- Misses        44911   134148    +89237
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
0.00% <0.00%> (-61.71%)
:arrow_down:

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <0.00%> (-61.75%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: Jackie-Jiang codecov-commenter


text:
Changed CompoundPinotMetricsFactory to use PluginManager instead of Class.forName

This is a small PR created to be an example of the kind of migrations we should apply in order to be able to use the improvements proposed in https://github.com/apache/pinot/pull/13930, which would let us run Pinot without shading.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (59551e4) to head (38d881a).
Report is 1293 commits behind head on master.

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (38d881a). Click for more details.
HEAD has 53 uploads less than BASE

Flag
BASE (59551e4)
HEAD (38d881a)

integration
7
1

integration2
3
1

temurin
12
1

java-21
7
0

skip-bytebuffers-true
3
0

skip-bytebuffers-false
7
1

unittests
5
0

unittests1
2
0

java-11
5
1

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #14402       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436        3     -2433
  Lines        133233        6   -133227
  Branches      20636        0    -20636
=============================================
- Hits          82274        0    -82274
+ Misses        44911        6    -44905
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <ø> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <ø> (ø)

java-11
0.00% <ø> (-61.71%)
:arrow_down:

java-21
?

skip-bytebuffers-false
0.00% <ø> (-61.75%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <ø> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

These changes are definitely necessary but not sufficient to ensure the above - we can still run into dependency conflict issues between plugin dependencies and Pinot dependencies if we don't use shading (alternative would be to introduce full classloader isolation where plugin dependencies aren't added to the main application classpath).

Yeah, sure. This is necessary but not sufficient. We still need to change the packaging and use the new classpath realm system. But we cannot turn it on until we actually apply this change everywhere we can load plugins. This include plugins loaded with ServiceLoader (which I encouraged in the past), which will need to be rewritten to use PluginManager instead of ServiceLoader

Anyway, remember we are NOT using shading right now. We are shading (paying the cost in terms of build time and artifact size) but changing all dependencies with the same prefix, so that shading is not preventing any issue between shaded plugins and Pinot

authors: codecov-commenter gortiz


text:
Rust support for Pulsar Functions

As the Rust language has became much more mature, and the ecosystem is growing, it would be great if Apache Pulsar supports Rust for writing Pulsar Functions.
The implementation should be quite straightforward as Golang is already supported and both languages produce binaries.

Nice feature, cared about too.

authors: cloorc genert

repo: apache/pulsar


text:
schemas feature request

Feature request:
schemas ( https://github.com/apache/pulsar/blob/701f3ea7e3541f61f064167043de04e8a2db5c0f/pulsar-client-schema/src/main/java/org/apache/pulsar/client/api/Schema.java  )
If I want to pass arbitrary/dynamic Json schemas that are not POJOs - is there a way to pass something dynamic there?
I would like to create a Json schema that specifies a https://json-schema.org/ , or maybe a Map<String, String> properties
All examples I’ve seen are JSONSchema.of. Or is possible to describe the new schema over JSON? It seems that yes: https://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats but I don’t see any example or they are not exposed.

After discussion, the way to do what I want is to:

1 - create a schemaless topic
2 - use the POST /admin/v2/schemas/:tenant/:namespace/:topic/schema endpoint to set a schema straight after that ( http://pulsar.apache.org/docs/en/admin-api-schemas/#pulsar-admin )
3 - use AUTO_PRODUCE_BYTES as mentioned above

@nha @merlimat
sorry for asking.
In my case, I have some raw json string data send to topic and can't hard code pojo class, I want to use the pulsar schema feature to validation the structure. I have a topic "my-topic" and associated with json schema in below, then I try to transmission some message.
var producer = client.newProducer(Schema.AUTO_PRODUCE_BYTES();
producer.send("{\"y\": 1}".getBytes()); // here! the value is 1(number) not string.

var reader = client.newReader(Schema.AUTO_CONSUME())
var message = reader.readNext();
I got {"y": 1}

my question is how pulsar schema works? The message should be rejected.
{
  "version": 1,
  "schemaInfo": {
    "name": "my-topic",
    "schema": {
      "type": "record",
      "name": "Data",
      "namespace": "com.iot.test",
      "fields": [
        {
          "name": "value",
          "type": [
            "null",
            "string"
          ]
        }
      ]
    },
    "type": "JSON",
    "properties": {
      "__alwaysAllowNull": "true"
    }
  }
}

authors: nha whisper-bye


text:
Pulsar Dashboard Displays Very Little Information

Describe the bug
The pulsar-dashboard does not display the information published by the pulsar cluster.
To Reproduce
Steps to reproduce the behavior:

Install a bare metal pulsar

on an openstack cloud
rather than a DNS server distribute identical /etc/hosts files to each of the vm's and the development machine
3 zookeeper each on their own vm
3 bookie/broker each pair on their own vm

On the development machine: Git clone the pulsar

build the docker image : docker build -t apachepulsar/pulsar-dashboard dashboard
2: export SERVICE_URL=http://ledge2:8080/
3: docker run -p 80:80   -e SERVICE_URL=$SERVICE_URL   apachepulsar/pulsar-dashboard

Construct a

single tenant
several namespaces
several topics

Expected behavior
The pulsar-dashboard should show clusters, tenants, namespaces, etc.
Screenshots
The pulsar  http server shows the presence of the cluster ...

The pulsar-dashboard apparently does not know about the cluster...

But no if you bring up one of the tabs having a CLUSTERS menu on the right there is the missing cluster...

Desktop (please complete the following information):

OS: Ubuntu 18.04 for the development machine and vm's

Additional context
Because /etc/hosts is used instead of DNS it is necessary to use the ledger2 ipaddr.
 export SERVICE_URL=http://128.60.107.171:8080/

The problem is with the name lookup. The messages returned by pulsar contain the urls for other objects. Those url contain host names and the docker image has no way of looking them up. This is easily remedied by changing the way the docker image is run.
docker run --network=host -p 80:80 -e SERVICE_URL=http://ledger2:8080 apachepulsar/pulsar-dashboard

Note the --network=host parameter.
In my case this amounts to granting access to the /etc/hosts files.
I would think that this would be the recommended way to run pulsar-dashboard in the "getting started" section of the documentation.

I cannot get it to work either:
docker run -it \
                                  -p 6650:6650 \
                                  -p 8080:8080 \
                                  -v $PWD/pulsar-data:/pulsar/data \
                                  apachepulsar/pulsar:latest bin/pulsar standalone

docker run --network=host -p 80:80 -e SERVICE_URL=http://localhost:8080 apachepulsar/pulsar-dashboard

Dashboard doesn't show any tenants and clusters.

@ruudk Does the pulsar server respond to requests generally?
That is, if you point your browser to the $SERVICE_URL/admin/v2/clusters what do you get?
The pulsar-dashboad does is wrap the https://pulsar.apache.org/docs/latest/reference/RestApi/ .
If the pulsar server does not respond the dashboard will show nothing.

@ruudk Another issue may be a docker interaction.
How does a docker process with --network=host look up 'localhost'?
You might want to try providing either the hostname or ipaddr for the machine hosting your pulsar service.

// http://localhost:8080/admin/v2/clusters
[
  "standalone"
]

So that works.
I think it would help a lot of the dashboard would just show errors when it cannot connect to the pulsar API. Right it just acts as if everything is empty and nothing is wrong.
When I run the dashboard with --network=host it says:
WARNING: Published ports are discarded when using host network mode

and it doesn't even come up.
Is there a way to somehow enable/view a debug log? To know what the error is?

FYI: I run this all on my local machine with Docker.

Even when I use docker-compose with this configuration it doesn't work:
version: '3.3'

services:
    pulsar:
        image: apachepulsar/pulsar:latest
        hostname: pulsar
        command: bin/pulsar standalone --advertised-address 127.0.0.1
        ports:
            - 6650:6650
            - 8080:8080
        volumes:
            - ./pulsar-data:/pulsar/data
    dashboard:
        image: apachepulsar/pulsar-dashboard
        environment:
            SERVICE_URL: "http://pulsar:8080"
        ports:
            - 8081:80

Did you try...
docker run -p 80:80 -e SERVICE_URL=http://localhost:8080 apachepulsar/pulsar-dashboard

I tried out this and it worked:
docker run -it \
                                                 -p 6650:6650 \
                                               -p 80:80  -p 8080:8080 \
                                                 -v $PWD/pulsar-data:/pulsar/data \
                                                 apachepulsar/pulsar-standalone:latest

Weird thing is that I have to wait a couple of minutes before data becomes visible. The UI is there, but it shows "No tenants" etc... but after a couple of minutes of starts to appear.

@ruudk in your docker compose case, you shouldn't use --advertised-address 127.0.0.1, you should either remove --advertised-address, or advertise an address that your dashboard docker can connect.

@sijie Thanks. Solved it with this config:
version: '3.3'

services:
    pulsar:
        image: apachepulsar/pulsar:latest
        command: bin/pulsar standalone
        ports:
            - 6650:6650
            - 8080:8080
        volumes:
            - ./pulsar-data:/pulsar/data
    dashboard:
        image: apachepulsar/pulsar-dashboard
        environment:
            SERVICE_URL: "http://pulsar:8080"
        ports:
            - 8081:80

Closed this issue since it was resolved

authors: phreed ruudk sijie


text:
[Doc] 'JSON Web Token Authentication' is not documented in the Security chapter of Pulsar document

Need to add the corresponded reference content of 'JSON Web Token Authentication' link in the Security chapter of Pulsar document. Its descriptive md. file is not added at present.

@sijie From the engineer part, who is familiar with "JSON Web Token Authentication"? We need some support to write this topic.

@Jennifer88huang @Monica-zy :
It is already documented: http://pulsar.apache.org/docs/en/security-token-client/ and http://pulsar.apache.org/docs/en/security-token-admin/

@Monica-zy Try to investigate the issue. If needed, you can come up with a solution with new structure.

Here is my suggested solution: Merge and rewrite the original two file security-token-client.md and security-token-admin.md into a new security-jwt.md which links to the list item "JSON Web Token Authentication" in the file security-overview.md; adjust the related content in security-overview and unify the sidebar title into a similar format as much as possible.
Changes in security-overview.md:

Changes in sidebar:

and welcome better suggestions and opinions to discuss here~

@Monica-zy In the TLS transport topic and other topics, make the heading consistent. The following is an example.

When you update this, make sure that

Update all similar cases in the Security and Deployment chapters;
Check links related to those headings;
Add related lead-in sentences.

For the overall structure, I'll sync with you later.

authors: Jennifer88huang Monica-zy sijie


text:
[pulsar-broker] Handle NPE in unblock stuck subscrption task when dis…

…patcher is not created
Motivation
It fixes #10429 NPE when subscription has not created dispatcher yet.
2021-04-29T00:11:51,161 [pulsar-stats-updater-29-1] ERROR org.apache.pulsar.broker.service.persistent.PersistentTopic - Got exception when creating consumer stats for subscription __compaction: null
java.lang.NullPointerException: null
at org.apache.pulsar.broker.service.persistent.PersistentSubscription.checkAndUnblockIfStuck(PersistentSubscription.java:1101) ~[org.apache.pulsar-pulsar-broker-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]
at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$updateRates$45(PersistentTopic.java:1642) ~[org.apache.pulsar-pulsar-broker-2.8.0-SNAPSHOT.jar:2.8.0-SNAPSHOT]

/pulsarbot run-failure-checks

@rdhabalia nice work

branch-2.7 does not have the feature of allowing the broker to discover and unblock stuck subscription so we don't need to cherry-pick this one to branch-2.7

authors: codelipenghui devinbost lhotari rdhabalia


text:
[ Issue 13479 ] Fixed internal topic effect by InactiveTopicPolicy.

Fixes #13479
Master Issue: #13479
Motivation
see #13479
Modifications

make healthcheck  internal topic as SystemTopic.

Verifying this change

[x] Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:
none.
Documentation

[x] no-need-doc

@codelipenghui  @ericsyh @eolivelli @BewareMyPower
PTAL :)
Another things:
Although this PR can fix the issue #13479,  But I have some advice as below:

healthcheck keyword name is common , maybe we can change this to some complex name.
I think we can collect all internal or system theme names into one container, because they are currently everywhere. Unclear and difficult to maintain.

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

@codelipenghui   @eolivelli @BewareMyPower  @ericsyh  @Technoboy-
PTAL :)

/pulsarbot rerun-failure-checks

@mattisonchao There are many conflicts with branch-2.8, could you please help push a new PR to fix the problem of branch-2.8?

Sure, I will fix it ASAP
On Tue, Jan 18, 2022 at 17:39, lipenghui @.***> wrote:

@mattisonchao https://github.com/mattisonchao There are many conflicts
with branch-2.8, could you please help push a new PR to fix the problem of
branch-2.8?
—
Reply to this email directly, view it on GitHub
https://github.com/apache/pulsar/pull/13611#issuecomment-1015232766, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AR2NWC2IWTN3BMOTERE7HYLUWUYMNANCNFSM5LI4UYFA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

@codelipenghui
work done.
The new PR is #13816.

authors: codelipenghui mattisonchao


text:
[improve][broker] Use HierarchyTopicPolicies to update topic-level subscribeRate

Motivation
Use HierarchyTopicPolicies to update topic-level subscribeRate
Modifications
Use updateSubscribeRateLimiter() to update topic-level subscribeRate, instead of initializeTopicSubscribeRateLimiterIfNeeded.   After this Modification,  broker-level, ns-level and topic-level subscribeRate updating all use the same method updateSubscribeRateLimiter()
Verifying this change

[x] Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as (please describe tests).

org.apache.pulsar.broker.admin.TopicPoliciesTest#testDisableSubscribeRate

Documentation

[x] no-need-doc
(Please explain why)

@Jason918  @gaozhangmin  @codelipenghui  PTAL


text:
[cleanup][client] Use TestNG instead of JUnit

Motivation
The codebase has two unit test frameworks: JUnit and TestNG, which always make us confused about which one to use. I check the git commits and found that #1032 does this thing, but still has some tests using JUnit. We need to remove the JUnit, use the TestNG as the unit test framework in the codebase.
Modifications

Use TestNG instead of JUnit

Documentation
Need to update docs?

[x] no-need-doc
(Please explain why)

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks


text:
[feature][doc] Sync language tab choices

Fixes #10282
Motivation
To sync the language tab choices on the same page to provide better experience. See also the original issue for more detail.
Modifications
Add groupId attribute to <Tabs>. This PR has been split into 3 commits for easier review, which may be squashed when merging.
Note that some pages use options like pulsar-admin, REST API and Java, while others use Admin CLI, REST API and Java Admin API. Not sure if we should make these options consistent, but I've ensured that the options on the same page is consistent so that the tab choices are synced on the same page.
Verifying this change

[x] Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: no
Anything that affects deployment: no

Documentation
Check the box below or label this PR directly.
Need to update docs?

[x] doc-required
(Your PR needs to update docs and you will update later)

[ ] doc-not-needed
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

@urfreespace review? thanks

@SignorMercurio

Can you attach your local preview screenshots in the PR description? Preview guide: https://docs.google.com/document/d/1wszdtMRo6MhKbVaggPK7_bnKaC4TewuT--GWZZxJNGg/edit#
I've re-labeled your PR. Doc label guide: https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0/edit#bookmark=id.5d66olk7l4oz

authors: Anonymitaet SignorMercurio


text:
[pulsar-admin] remove duplicate shaded ahc.properties file in client and admin

Motivation
pulsar-client and pulsar-client-admin modules use the duplicate ahc.properties file for async-http-client. so, removing file from one of the location pulsar-client-admin and adding the symlink from pulsar-client module.

rerun integration tests


text:
[Issue 9155][build]Fixing the mac build if you use a different openssl

Fixes #9155
Motivation
When I attempted to build the c library on macosx with openssl 1.1.1i it failed with linkage errors. I tracked those errors down to the fact that the OPENSLL_ROOT_DIR was incorrect outside of mac vms used by the upstream.
Modifications
I set the default value first and then reset it only if running on mac.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
No
Documentation
-None

@BewareMyPower Please help review this PR

@BewareMyPower Please help review this PR

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

authors: codelipenghui phijohns-tibco sijie


text:
consume message Hang

问题描述
一个消费者订阅多个topic，刚上线时消息消费一切正常，过一段时间之后（时间不确定多长），消费位点会出现不提交的情况，但是消息是在正常消费的，通过vkmq-admin查询消费位点的情况，消息一直在堆积。

查看是否有消费超时发生（会有重试消息）

查看是否有消费超时发生（会有重试消息）

存在重试消息，但是消费者还是不能正常消费，一直卡在产生重试消息的那个时间点

看消费线程是否一直block住了，如果有一条消息，消费没有返回，就会导致消费offset不会前进

看消费线程是否一直block住了，如果有一条消息，消费没有返回，就会导致消费offset不会前进

怎么才能避免这种情况

这个是有些问题，目前就会有。如果做的好一些，就是这个队列长时间被block住，server端得释放当前当前队列，让别的client可以消费

@xiaoyanghapi  一个订阅组只能订阅一个topic，参考。

authors: Ah39 gaoyf ssssssnake xiaoyanghapi

repo: apache/rocketmq


text:
无法发送请求到 rocketMQ 服务器

近期出现以下这个异常,我百度谷歌后没有找到问题的解决方法,我到运行环境是mac,其他同事并不会出现这种情况
org.apache.rocketmq.remoting.exception.RemotingSendRequestException: send request to <122.112.204.214:9876> failed
at org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.invokeSyncImpl(NettyRemotingAbstract.java:389) ~[rocketmq-remoting-4.2.0.jar:4.2.0]
at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeSync(NettyRemotingClient.java:369) ~[rocketmq-remoting-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.MQClientAPIImpl.getTopicRouteInfoFromNameServer(MQClientAPIImpl.java:1207) ~[rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.MQClientAPIImpl.getTopicRouteInfoFromNameServer(MQClientAPIImpl.java:1197) ~[rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.factory.MQClientInstance.updateTopicRouteInfoFromNameServer(MQClientInstance.java:605) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.factory.MQClientInstance.updateTopicRouteInfoFromNameServer(MQClientInstance.java:492) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.factory.MQClientInstance.findConsumerIdList(MQClientInstance.java:1052) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.consumer.RebalanceImpl.rebalanceByTopic(RebalanceImpl.java:262) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.consumer.RebalanceImpl.doRebalance(RebalanceImpl.java:225) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.consumer.DefaultMQPushConsumerImpl.doRebalance(DefaultMQPushConsumerImpl.java:994) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.factory.MQClientInstance.doRebalance(MQClientInstance.java:955) [rocketmq-client-4.2.0.jar:4.2.0]
at org.apache.rocketmq.client.impl.consumer.RebalanceService.run(RebalanceService.java:41) [rocketmq-client-4.2.0.jar:4.2.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: io.netty.handler.codec.EncoderException: java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:302) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.ChannelDuplexHandler.write(ChannelDuplexHandler.java:106) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
... 1 more
Caused by: java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at org.apache.rocketmq.remoting.protocol.RemotingCommand.encodeHeader(RemotingCommand.java:427) ~[rocketmq-remoting-4.2.0.jar:4.2.0]
at org.apache.rocketmq.remoting.protocol.RemotingCommand.encodeHeader(RemotingCommand.java:400) ~[rocketmq-remoting-4.2.0.jar:4.2.0]
at org.apache.rocketmq.remoting.netty.NettyEncoder.encode(NettyEncoder.java:36) ~[rocketmq-remoting-4.2.0.jar:4.2.0]
at org.apache.rocketmq.remoting.netty.NettyEncoder.encode(NettyEncoder.java:29) ~[rocketmq-remoting-4.2.0.jar:4.2.0]
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:302) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.ChannelDuplexHandler.write(ChannelDuplexHandler.java:106) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
... 1 more

Hi. I had met the same trouble. The cause is that because I compiled rocketmq with JDK11, but running rocketmq in JDK8.
In JDK11(in fact since JDK9) ByteBuffer.flip() returns a ByteBuffer. BUT It used to return a Buffer in JDK8.
You can look at the link https://github.com/plasma-umass/doppio/issues/497 to get more info.
Hope can help you.

Please describe the issue in English,thanks.

@wlliqipeng 为何如此装逼?买个奥迪也他妈的也想装高富帅!

Hi. I had met the same trouble. The cause is that because I compiled rocketmq with JDK11, but running rocketmq in JDK8.
In JDK11(in fact since JDK9) ByteBuffer.flip() returns a ByteBuffer. BUT It used to return a Buffer in JDK8.
You can look at the link plasma-umass/doppio#497 to get more info.
Hope can help you.

It`s ok ，thanks

authors: Suaron-Biliu asiaon123 hongjunxin wenkikro wlliqipeng


text:
proxy module monitoring indicators are improved

Current proxy module monitoring indicators are not perfect, need to improve the monitoring indicators for connection, time, thread several dimensions.
//todo Details of indicators will be added later

Are you interested in contributing more metrics to the proxy? First, we need to implement these metrics mentioned in #5366. And let's move to GitHub Discussion, see #6153.

Are you interested in contributing more metrics to the proxy? First, we need to implement these metrics mentioned in #5366. And let's move to GitHub Discussion, see #6153.

get

@ShadowySpirits @socutes I am interested in this and like to learn more about the issue.

authors: Abhijeetmishr ShadowySpirits socutes


text:
[ISSUE #2011] Add en README.md doc

What is the purpose of the change
fix #2011  , add en README.md doc and its relevant  #docs.
Brief changelog
add README.md, RocketMQ_Example.md, dledger/quick_start.md, dledger/deploy_guide.md, msg_trace/user_guide.md, operation.md under directory docs/en.

Coverage increased (+0.03%) to 50.85% when pulling 7e6a4065091c21d8d4d455c3c63c5cfd5e57493d on lizhiboo:master into 83c5820d20e4b13d1f2daaf61ca7d34b703fac58 on apache:develop.

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@8ef01a6). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             develop    #2016   +/-   ##
==========================================
  Coverage           ?   45.06%
  Complexity         ?     4195
==========================================
  Files              ?      545
  Lines              ?    35732
  Branches           ?     4744
==========================================
  Hits               ?    16103
  Misses             ?    17579
  Partials           ?     2050

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ef01a6...7e6a406. Read the comment docs.

authors: codecov-commenter coveralls lizhiboo


text:
[ISSUE #2783]Fix bug in ThreadLocalIndex

Fix bug in ThreadLocalIndex#getAndIncrement

Coverage increased (+0.01%) to 51.942% when pulling b78326d869a09ed9f37b85dc957b39dff7a4ef87 on Kingdon1992:fix_bug_in_ThreadLocalIndex into 7680063f89a2aa5ae003f5d19ebd3343e828b94f on apache:develop.

Coverage increased (+0.01%) to 51.942% when pulling b78326d869a09ed9f37b85dc957b39dff7a4ef87 on Kingdon1992:fix_bug_in_ThreadLocalIndex into 7680063f89a2aa5ae003f5d19ebd3343e828b94f on apache:develop.


text:
No 4. add acl

add acl
issue #403

NameSrv鉴权架构设计：

基于RPCHook实现调用拦截器。
自定义RemoteCommand的ACL鉴权Header。
客户端acl鉴权接口，提供不同鉴权方式的扩展点。
服务端acl鉴权接口，提供不同的权限管理数据源的鉴权方式，如数据库，Redis,XML,properties等方式。

基本思路：

producer鉴权

上图所示的是NameSrc收到查找topic路由请求之后的执行逻辑。先注册一个ACLHook，
如果鉴权失败，则抛出一个运行时异常，后续processRequest真实处理逻辑调用就会被阻断。
此时NameSrc就会返回给Producer一个鉴权失败信息。

consumer鉴权

整体流程类似Producer

方案2.
思路：
在该方案里，我们通过在客户端进行鉴权的方式，这里的客户端包括：producer和consumer
通过源码可以知道在客户端[producer | consumer]中，都会使用NettyRemotingClient类
的作为数据发送接口，并提供三个方法发送数据。通过在这个三个方法里添加 钩子类 hook 在数据
发送动作前提供鉴权操作。
使用：
用户在使用 rocketmq 时，如果要使用 鉴权 功能，则把提供一个 AclClientHook 并传递到 DefaultMQProducer，
AclClientHook 提供对到鉴权查询的服务功能。这里的 AclClientHook 是一个 鉴权钩子，里面描述了鉴权中心的访问接口 AclService，不规范具体实现。
提供对到鉴权查询的服务功能。这里的AclClientHook，如下：
RpcHook acl = new AclClientHook();
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name", ack);
这里我们提供了一个基于mysql的单应用的鉴权中心： ACLCenter。


text:
Fix code style in any net modules.

Make sure set the target branch to develop
What is the purpose of the change
Fix code style in any net modules.
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

maven build so slow.

Why those checks stuck?

Codecov Report

Merging #4989 (b89e9f6) into develop (b2f3027) will decrease coverage by 0.07%.
The diff coverage is 60.49%.

@@              Coverage Diff              @@
##             develop    #4989      +/-   ##
=============================================
- Coverage      43.26%   43.19%   -0.08%
- Complexity      7733     7739       +6
=============================================
  Files            995      995
  Lines          69069    69029      -40
  Branches        9143     9133      -10
=============================================
- Hits           29886    29814      -72
- Misses         35437    35471      +34
+ Partials        3746     3744       -2

Impacted Files
Coverage Δ

...e/rocketmq/broker/topic/TopicRouteInfoManager.java
18.48% <0.00%> (ø)

...che/rocketmq/common/protocol/route/BrokerData.java
30.43% <0.00%> (+1.26%)
:arrow_up:

...ache/rocketmq/common/protocol/route/QueueData.java
37.73% <0.00%> (-0.45%)
:arrow_down:

...rocketmq/common/protocol/route/TopicRouteData.java
22.31% <ø> (ø)

...org/apache/rocketmq/common/rpc/ClientMetadata.java
0.00% <0.00%> (ø)

...va/org/apache/rocketmq/namesrv/NamesrvStartup.java
15.96% <0.00%> (-0.71%)
:arrow_down:

...org/apache/rocketmq/namesrv/NamesrvController.java
64.95% <50.00%> (+1.62%)
:arrow_up:

...rocketmq/remoting/netty/NettyRemotingAbstract.java
53.26% <57.77%> (-0.70%)
:arrow_down:

...e/rocketmq/remoting/netty/NettyRemotingServer.java
60.90% <64.28%> (+1.27%)
:arrow_up:

...e/rocketmq/namesrv/routeinfo/RouteInfoManager.java
64.04% <67.03%> (-1.66%)
:arrow_down:

... and 29 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: SeaItFover codecov-commenter


text:
[ISSUE #8740] fix rocksDBConfigToJson command

Which Issue(s) This PR Fixes

Fixes #8740
Brief Description

How Did You Test This Change?

Codecov Report
Attention: Patch coverage is 0% with 39 lines in your changes missing coverage. Please review.

Project coverage is 47.34%. Comparing base (525f877) to head (49508ba).

Files with missing lines
Patch %
Lines

...s/command/metadata/RocksDBConfigToJsonCommand.java
0.00%
39 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8738      +/-   ##
=============================================
- Coverage      47.46%   47.34%   -0.12%
+ Complexity     11566    11546      -20
=============================================
  Files           1282     1282
  Lines          89813    89843      +30
  Branches       11553    11556       +3
=============================================
- Hits           42626    42538      -88
- Misses         41956    42056     +100
- Partials        5231     5249      +18

Flag
Coverage Δ

47.34% <0.00%> (-0.12%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
Missing semicolon causes compiler error !

Missing semicolon causes compiler error, should inform with descriptive message for developer, like missing semicolone.
I forgot to put semicolon at the end of the first line by mistake and got the compiler error :
The code :

Compiler error :

Hi @nihavend this seems a compiler error, maybe better log in the compiler issues instead here

It has been fixed already @carlosrovira and was created long time ago at 1 Dec 2019. I will consider for the new ones.

authors: carlosrovira nihavend

repo: apache/royale-asjs


text:
add ingress-controller integration apache dubbo test

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #5154 (216d929) into master (acf7721) will decrease coverage by 0.21%.
The diff coverage is 0.00%.

:exclamation: Current head 216d929 differs from pull request most recent head 4375a3e. Consider uploading reports for the commit 4375a3e to get more accurate results

@@             Coverage Diff              @@
##             master    #5154      +/-   ##
============================================
- Coverage     61.04%   60.83%   -0.21%
+ Complexity     8473     8464       -9
============================================
  Files          1242     1242
  Lines         37389    37425      +36
  Branches       3527     3525       -2
============================================
- Hits          22823    22767      -56
- Misses        12604    12698      +94
+ Partials       1962     1960       -2

Files Changed
Coverage Δ

...org/apache/shenyu/k8s/common/IngressConstants.java
0.00% <ø> (ø)

...g/apache/shenyu/k8s/parser/DubboIngressParser.java
0.00% <0.00%> (ø)

...che/shenyu/k8s/reconciler/EndpointsReconciler.java
0.00% <ø> (ø)

...pache/shenyu/k8s/reconciler/IngressReconciler.java
28.18% <0.00%> (-1.29%)
:arrow_down:

... and 20 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter runqi-zhao

repo: apache/shenyu


text:
[type:feature] init context path plugin shenyu ingress controller

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #5167 (deb7215) into master (75cdd03) will decrease coverage by 0.20%.
The diff coverage is 24.10%.

@@             Coverage Diff              @@
##             master    #5167      +/-   ##
============================================
- Coverage     60.87%   60.67%   -0.20%
- Complexity     8467     8473       +6
============================================
  Files          1242     1243       +1
  Lines         37433    37560     +127
  Branches       3532     3555      +23
============================================
+ Hits          22786    22791       +5
- Misses        12689    12806     +117
- Partials       1958     1963       +5

Files Changed
Coverage Δ

...org/apache/shenyu/k8s/common/IngressConstants.java
0.00% <ø> (ø)

...rg/apache/shenyu/k8s/parser/ContextPathParser.java
0.00% <0.00%> (ø)

...pache/shenyu/k8s/reconciler/IngressReconciler.java
33.67% <41.77%> (+2.90%)
:arrow_up:

...va/org/apache/shenyu/k8s/parser/IngressParser.java
53.65% <51.85%> (+18.65%)
:arrow_up:

... and 4 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


text:
Fix minor error

Reading an file named "" will raise the IsNotExist error, so the conditional statement is unnecessary in pkg/config/config.go
The result should match whether the pattern is . or ./.
Small improvement.

Reading an file named "" will raise the IsNotExist error, so the conditional statement is unnecessary in pkg/config/config.go
The result should match whether the pattern is . or ./.
Small improvement.

Thanks for the feedback, where are you going to use SkyWalking-eye tool?

Reading an file named "" will raise the IsNotExist error, so the conditional statement is unnecessary in pkg/config/config.go
The result should match whether the pattern is . or ./.
Small improvement.

Thanks for the feedback, where are you going to use SkyWalking-eye tool?

Yes, i will. And i am a student participating Summer 2021 of Open Source Promotion Plan.

authors: MoGuGuai-hzr wu-sheng

repo: apache/skywalking-eyes


text:
support jetty server work thread pool metric monitor

[x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. https://github.com/apache/skywalking/issues/10736.
[x] Update the CHANGES log.

the plugin supports jetty 11，but jetty 11 example is different with lower version, should i write new scenario test for 11

ok，i  write another test for 11, have fun

repo: apache/skywalking-java


text:
Adding support for zipkin UI

Marking the variable SW_ZIPKIN_ADDRESS explicitly for deploying the zipkin UI more easily on k8s cluster.

I think using these variables in Chart template is easier especially for those who are not familiar with Helm. But at least adding some guide about how to deploy the zipkin UI on k8s cluster is needed. It took me a lot of time to find out the variable SW_ZIPKIN_ADDRESS should be set.

Use  ui.zipkin.enabled and oap.port.zipkinui to control whether set the variable SW_ZIPKIN_ADDRESS or not.

@kezhenxu94

Conditionally add the exposed port 9412 in the OAP service: when the SW_QUERY_ZIPKIN is set to default, add the port 9412 to service OAP, otherwise, don't add the port.

I think this idea is better.

Always add the SW_ZIPKIN_ADDRESS, there is no side effect for this env var to be set no matter Zipkin plugin is enabled or disabled.

However, there's one side effect for this env var to be set without setting the port. The UI pod will throw an error complaining about the missing port when initializing. I think we should use the env var SW_QUERY_ZIPKIN to control whether the SW_ZIPKIN_ADDRESS is set or not. Is it a good idea?

However, there's one side effect for this env var to be set without setting the port. The UI pod will throw an error complaining about the missing port when initializing. I think we should use the env var SW_QUERY_ZIPKIN to control whether the SW_ZIPKIN_ADDRESS is set or not. Is it a good idea?

Right I missed that, that's better!

@kezhenxu94
It works well now. But I think directly hardcoding the port number(like 9141 and 9142) in the file is not very graceful.

@kezhenxu94
It works well now. But I think directly hardcoding the port number(like 9141 and 9142) in the file is not very graceful.

I agree, but I don't thin we have a more graceful way to do this, do you have any better idea, I'm open to discuss?

values.yaml
oap:
  ports:
    # add more ports here if you need, for example
    # zabbix: 10051
    grpc: 11800
    rest: 12800
    # zipkinReceiver: 9411
    # zipkinQuery: 9412

oap-deployment.yaml
        env:
{{- if .Values.oap.ports.zipkinReceiver }}
        - name: SW_RECEIVER_ZIPKIN
          value: "default"
{{- end }}
{{- if .Values.oap.ports.zipkinQuery }}
        - name: SW_QUERY_ZIPKIN
          value: "default"
{{- end }}

ui-deployment.yaml
        env:
{{- if .Values.oap.ports.zipkinQuery }}
        - name: SW_ZIPKIN_ADDRESS
          value: "http://{{ template "skywalking.oap.fullname" . }}:{{ .Values.oap.ports.zipkinQuery }}"
{{- end }}

I think it might be more convenient. What do you think? @kezhenxu94 @Yangfisher1

values.yaml
oap:
  ports:
    # add more ports here if you need, for example
    # zabbix: 10051
    grpc: 11800
    rest: 12800
    # zipkinReceiver: 9411
    # zipkinQuery: 9412

oap-deployment.yaml
        env:
{{- if .Values.oap.ports.zipkinReceiver }}
        - name: SW_RECEIVER_ZIPKIN
          value: "default"
{{- end }}
{{- if .Values.oap.ports.zipkinQuery }}
        - name: SW_QUERY_ZIPKIN
          value: "default"
{{- end }}

ui-deployment.yaml
        env:
{{- if .Values.oap.ports.zipkinQuery }}
        - name: SW_ZIPKIN_ADDRESS
          value: "http://{{ template "skywalking.oap.fullname" . }}:{{ .Values.oap.ports.zipkinQuery }}"
{{- end }}

I think it might be more convenient. What do you think? @kezhenxu94 @Yangfisher1

This works for me too, another good point is that we can also set SW_QUERY_ZIPKIN_REST_PORT to .Values.ports.zipkinQuery, @Yangfisher1 in that case users can set Zipin port to arbitrary port, I think

@innerpeacez @kezhenxu94 It works for me, making it more convenient and graceful. Users only need to configure these two ports to enable zipkin trace and they can choose any port they want.
I'd like to make these changes and add these ports into README.md.

@innerpeacez @kezhenxu94
I tested it on my k8s cluster and it worked well. Since helm only supports lowercase letters so I rename these ports.

Since helm only supports lowercase letters so I rename these ports.

Camel-Case is supported, let's keep it. ref to https://helm.sh/zh/docs/chart_best_practices/values/

@innerpeacez My bad. It's the restriction of Kubernetes as a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')

@innerpeacez My bad. It's the restriction of Kubernetes as a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')

Oooops! For more convenience. Named zkreceiver zkquery, to avoid using gotemplate index function. You think? @Yangfisher1

I vote for not to use index function, using . is also friendly for completion in IDEs。

@innerpeacez
I'm ok with this. But I'm concerned that using zk instead of zipkin might be confusing. Using 'index' would ensure the readability of these ports.

We can use zipkinreceiver or zipkinquery?

@innerpeacez
I'm ok with this. But I'm concerned that using zk instead of zipkin might be confusing. Using 'index' would ensure the readability of these ports.

We can use zipkinreceiver or zipkinquery?

It's better. I think.

@kezhenxu94 @innerpeacez
Renaming is done.

Congrats @Yangfisher1 to be 800th GitHub contributor of SkyWalking.
Welcome.

@kezhenxu94 @innerpeacez @wu-sheng Thank you for your help! This is a wonderful community and everyone is kind🤩

repo: apache/skywalking-kubernetes


text:
Dynamic configuration key bug of endpoint-name-grouping

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?

Which company or project?

What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.

EndpointNameGroupingRuleWatcher class provides an incorrect module name, which makes the configuration key wrong. The wrong name includes the unexpected package name. Referring to the following codes,
https://github.com/apache/skywalking/blob/master/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/group/EndpointNameGroupingRuleWatcher.java#L38

The problem has been modified and submitted to PR;
https://github.com/apache/skywalking/pull/5833

authors: tangshan-brs wu-sheng

repo: apache/skywalking


text:
[Bug] skywalking-python not work with uwsgi + flask in master workers mode and threads mode

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
Python Agent (apache/skywalking-python)
What happened
I can't use skywalking under uwsgi and flask in master workers mode and threads mode.
I have try solution in https://github.com/apache/skywalking/issues/5263 and https://github.com/apache/skywalking/issues/6324 , but not work for me.
When I use uwsgi in master workers mode and threads mode, the skywalking python sdk does not report the metrics while the program is running, it reports metrics only after I close uwsgi...

I think this problem is caused by mixing gevent and uwsgi together.
What you expected to happen
sdk should be able to report metrics in real time...
How to reproduce
Ways
clear Docker env
docker run -t -d python:3.7.12-buster

pip install flask
pip install uwsgi
pip install gevent
pip install "apache-skywalking"
pip install "apache-skywalking[http]"

root@142fcc3e5aeb:~/flask-traceing# pip list
Package            Version
------------------ ---------
apache-skywalking  0.7.0
certifi            2021.10.8
charset-normalizer 2.0.12
click              8.0.4
Flask              2.0.3
gevent             21.12.0
greenlet           1.1.2
grpcio             1.44.0
grpcio-tools       1.44.0
idna               3.3
importlib-metadata 4.11.1
itsdangerous       2.1.0
Jinja2             3.0.3
MarkupSafe         2.1.0
packaging          21.3
pip                21.2.4
protobuf           3.19.4
pyparsing          3.0.7
requests           2.27.1
setuptools         57.5.0
six                1.16.0
typing_extensions  4.1.1
urllib3            1.26.8
uWSGI              2.0.20
Werkzeug           2.0.3
wheel              0.37.0
wrapt              1.13.3
zipp               3.7.0
zope.event         4.5.0
zope.interface     5.4.0

code 1
from gevent import monkey
monkey.patch_all()

import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

from skywalking import agent, config

# grpc version
config.protocol = 'grpc'
config.init(collector_address='xxxxx', service_name='test-1')
config.flask_collect_http_params = True
config.logging_level = 'DEBUG'
agent.start()

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()

code 2
from skywalking import agent, config

# grpc version
config.protocol = 'grpc'
config.init(collector_address='xxx', service_name='test-1')
config.flask_collect_http_params = True
config.logging_level = 'DEBUG'
agent.start()

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()

uwsgi run mode 1
uwsgi --die-on-term \
    --http 0.0.0.0:5000 \
    --http-manage-expect \
    --master --workers 3 \
    --enable-threads \
    --threads 3 \
    --manage-script-name \
    --mount /=main:app

uwsgi run mode 2
uwsgi --die-on-term \
    --http 0.0.0.0:5000 \
    --http-manage-expect \
    --manage-script-name \
    --mount /=main:app

Result
Not use uwsgi Work OK
python main.py
....
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [b31a30ec931111ecb19c0242ac110004]
 * Serving Flask app 'main' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [21/Feb/2022 12:28:02] "GET / HTTP/1.1" 200 -
skywalking [Thread-11] [DEBUG] reporting segment Segment@140325469202192[segment_id=b628b966931111ecbaf00242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7fa011bea3d0>], timestamp=1645446482149, related_traces=[<skywalking.trace.segment._NewID object at 0x7fa011bea850>]]
127.0.0.1 - - [21/Feb/2022 12:28:03] "GET / HTTP/1.1" 200 -
skywalking [Thread-14] [DEBUG] reporting segment Segment@140325443643856[segment_id=b73bda68931111ecaf140242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7fa01038a710>], timestamp=1645446483952, related_traces=[<skywalking.trace.segment._NewID object at 0x7fa01038a8d0>]]

code 1 + uwsgi run mode 1 Not OK
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 7200)
E0221 12:19:43.643491357    7200 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI worker 1 (pid: 7211, cores: 3)
E0221 12:19:43.646077330    7200 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI worker 2 (pid: 7214, cores: 3)
E0221 12:19:43.648015448    7200 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI worker 3 (pid: 7217, cores: 3)
E0221 12:19:43.649466988    7200 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI http 1 (pid: 7220)
[pid: 7214|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 293 bytes} [Mon Feb 21 12:19:49 2022] GET / => generated 12 bytes in 7 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)
[pid: 7217|app: 0|req: 1/2] 127.0.0.1 () {28 vars in 294 bytes} [Mon Feb 21 12:19:50 2022] GET / => generated 12 bytes in 7 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)
^CSIGINT/SIGTERM received...killing workers...
gateway "uWSGI http 1" has been buried (pid: 7220)
skywalking [Thread-3] [DEBUG] reporting segment Segment@140155123891344[segment_id=90aab898931011ec89130242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7f78685f7d90>], timestamp=1645445989751, related_traces=[<skywalking.trace.segment._NewID object at 0x7f78685f7bd0>]]
skywalking [Thread-3] [DEBUG] reporting segment Segment@140155123891344[segment_id=91560ef0931011ec89130242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7f78685f7d90>], timestamp=1645445990874, related_traces=[<skywalking.trace.segment._NewID object at 0x7f78685f7bd0>]]
skywalking [Thread-8] [DEBUG] grpc channel connectivity changed, [ChannelConnectivity.CONNECTING -> ChannelConnectivity.READY]
skywalking [Thread-8] [DEBUG] grpc channel connectivity changed, [ChannelConnectivity.CONNECTING -> ChannelConnectivity.READY]
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [8ce135b6931011ec89130242ac110004]
skywalking [Thread-8] [DEBUG] grpc channel connectivity changed, [ChannelConnectivity.CONNECTING -> ChannelConnectivity.READY]
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [8ce135b6931011ec89130242ac110004]
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [8ce135b6931011ec89130242ac110004]
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
goodbye to uWSGI.

code 1+ uwsgi run mode 2 NOT OK
WSGI app 0 (mountpoint='/') ready in 0 seconds on interpreter 0x5594e7fc3ce0 pid: 8531 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 8531, cores: 1)
[pid: 8531|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 293 bytes} [Mon Feb 21 12:22:07 2022] GET / => generated 12 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)
[pid: 8531|app: 0|req: 2/2] 127.0.0.1 () {28 vars in 293 bytes} [Mon Feb 21 12:22:08 2022] GET / => generated 12 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)
^Cskywalking [Thread-3] [DEBUG] reporting segment Segment@139745731000592[segment_id=e2ff80ec931011ecb2ce0242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7f1916a7bb10>], timestamp=1645446127879, related_traces=[<skywalking.trace.segment._NewID object at 0x7f1916a7bad0>]]
skywalking [Thread-5] [DEBUG] reporting segment Segment@139745731000848[segment_id=e3990ca8931011ecb2ce0242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7f1916a7bf10>], timestamp=1645446128886, related_traces=[<skywalking.trace.segment._NewID object at 0x7f1916a7bed0>]]
skywalking [Thread-6] [DEBUG] grpc channel connectivity changed, [ChannelConnectivity.CONNECTING -> ChannelConnectivity.READY]
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [e122606e931011ecb2ce0242ac110004]

code 2+ uwsgi run mode 1 NOT OK
hang when request ...
root@sssss:~/flask-traceing# curl -v 127.0.0.1:5000
* Expire in 0 ms for 6 (transfer 0x55f6bd4f1fb0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55f6bd4f1fb0)
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.64.0
> Accept: */*
>

*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 9801)
E0221 12:23:54.949025574    9801 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI worker 1 (pid: 9817, cores: 3)
E0221 12:23:54.951940621    9801 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI worker 2 (pid: 9819, cores: 3)
E0221 12:23:54.954821063    9801 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI worker 3 (pid: 9820, cores: 3)
E0221 12:23:54.957141004    9801 fork_posix.cc:70]           Fork support is only compatible with the epoll1 and poll polling strategies
spawned uWSGI http 1 (pid: 9821)
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [2299ccf8931111ecb7390242ac110004]
skywalking [QueryProfileCommandThread] [DEBUG] query profile commands
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [2299ccf8931111ecb7390242ac110004]
skywalking [QueryProfileCommandThread] [DEBUG] query profile commands
skywalking [QueryProfileCommandThread] [DEBUG] query profile commands
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [2299ccf8931111ecb7390242ac110004]
^CSIGINT/SIGTERM received...killing workers...
gateway "uWSGI http 1" has been buried (pid: 9821)
skywalking [QueryProfileCommandThread] [DEBUG] query profile commands
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [2299ccf8931111ecb7390242ac110004]
skywalking [QueryProfileCommandThread] [DEBUG] query profile commands
^C

code 2+ uwsgi run mode 2 NOT OK
WSGI app 0 (mountpoint='/') ready in 1 seconds on interpreter 0x55d66f587ce0 pid: 9274 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 9274, cores: 1)
[pid: 9274|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 294 bytes} [Mon Feb 21 12:23:12 2022] GET / => generated 12 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)
skywalking [HeartbeatThread] [DEBUG] service heart beats, [test-1], [0761d84a931111eca1990242ac110004]
[pid: 9274|app: 0|req: 2/2] 127.0.0.1 () {28 vars in 294 bytes} [Mon Feb 21 12:23:12 2022] GET / => generated 12 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 79 bytes (1 switches on core 0)
^Cskywalking [Thread-7] [DEBUG] reporting segment Segment@139889892138320[segment_id=093abc54931111eca1990242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7f3aa7541c50>], timestamp=1645446192021, related_traces=[<skywalking.trace.segment._NewID object at 0x7f3aa7541cd0>]]
skywalking [Thread-8] [DEBUG] reporting segment Segment@139889841328976[segment_id=09cb0a02931111eca1990242ac110004, spans=[<skywalking.trace.span.EntrySpan object at 0x7f3aa44cd550>], timestamp=1645446192967, related_traces=[<skywalking.trace.segment._NewID object at 0x7f3aa44cd510>]]

Anything else
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hello, this issue seems still deeply related to the two previous issues that you linked, I appreciate it if you could allow some time for further investigation. And thanks for the detailed case trial.  @arcosx

Hello, this issue seems still deeply related to the two previous issues that you linked, I appreciate it if you could allow some time for further investigation. And thanks for the detailed case trial. @arcosx

OK, waiting for good news😁. If there is something I need to cooperate with you can reply.

Hi @kezhenxu94 @Superskyyy , after some research, I don't think skywalking-python has any problem, it's uWSGI that needs special treatment, this solution below can solve my problem, including issues https://github.com/apache/skywalking/issues/6324.
from uwsgidecorators import postfork
from skywalking import agent, config

@postfork
def init_tracing():
    config.protocol = 'grpc'
    config.init(collector_address='xxxxx', service_name='test-1')
    config.flask_collect_http_params = True
    config.logging_level = 'DEBUG'
    agent.start()

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello World!'

if __name__ == '__main__':
    app.run()

From uWSGI doc we can know

uWSGI is a preforking (or “fork-abusing”) server, so you might need to execute a fixup task after each fork(). The postfork decorator is just the ticket. You can declare multiple postfork tasks. Each decorated function will be executed in sequence after each fork().

Do you have a better solution? Maybe we can note this in skywalking's documentation, after all uWsgi is a popular library.

@arcosx Hello, great to see you found a solution, would you mind opening a PR to document this? I believe it is valuable since both packages are widely used, and from our observation, the issue occurs rather frequently. Probably this should go to the FAQ folder, with a doc name of your choice. https://github.com/apache/skywalking-python/tree/master/docs/en/setup/faq

@arcosx thanks for the sharing, I'll modify the label to documentation if you'd like to open a PR to add that into our doc.

@arcosx Hello, great to see you found a solution, would you mind opening a PR to document this? I believe it is valuable since both packages are widely used, and from our observation, the issue occurs rather frequently. Probably this should go to the FAQ folder, with a doc name of your choice. https://github.com/apache/skywalking-python/tree/master/docs/en/setup/faq

I think adding to the FAQ is a good way 😃 and I will submit a PR.

Notice, if FAQ menu doesn't exist, you need to update menu yaml file to make this available on the website.

@arcosx Any update?

@arcosx Any update?

Sorry for the late submit, the PR here https://github.com/apache/skywalking-python/pull/188

Hi @Superskyyy
I'm trying to run a Django application with gunicorn. It is unable to start the agent. I followed this thread and https://opentelemetry-python.readthedocs.io/en/latest/examples/fork-process-model/README.html
Here is my wsgi.py
import os

from django.core.wsgi import get_wsgi_application

from uwsgidecorators import postfork
from skywalking import agent, config

def post_fork(server, worker):
    print("abc")
    config.init(collector_address='xxxxx', service_name='test-1')
    agent.start()

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

application = get_wsgi_application()

this is the command to start the application
gunicorn myproject.wsgi --workers 2
Any suggestions.

@Superskyyy
Thanks for your response. Tried with config file and it worked.

I have a similar problem.
wsgi.py:
`import os
from django.core.wsgi import get_wsgi_application
from skywalking import agent, config
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'self-healing.settings_prod')
config.init(collector_address='xxxx:12800', service_name='self-healing', protocol='http')
agent.start()
`
start command:
gunicorn self-healing.wsgi:application -b0.0.0.0:8000 -w1 --access-logfile=/tmp/access.log
skywalking  web:
`Traceback:
Traceback (most recent call last):
File "/root/miniconda3/envs/self-healing/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
return self._session_cache
AttributeError: 'SessionStore' object has no attribute '_session_cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/miniconda3/envs/self-healing/lib/python3.7/site-packages/skywalking/plugins/sw_mysqlclient.py", line 70, in execute
return self._self_cur.execute(query, args)
File "/root/miniconda3/envs/self-healing/lib/python3.7/site-packages/skywalking/plugins/sw_pymysql.py", line 49, in _sw_execute
span.tag(TagDbInstance((this.connection.db or b'').decode('utf-8')))
AttributeError: 'str' object has no attribute 'decode'`
Thanks for your help

Hello this is not the same problem related to forking. It seems like its not a problem with skywalking agent but with Django?

Hello this is not the same problem related to forking. It seems like its not a problem with skywalking agent but with Django?

I can access it normally through python manage.py run without reporting an error, but there will be a problem through gunicorn. Finally, I refer to https://skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi/, and the same problem occurs through uwsgi.
Is there any way to find out what the problem is?

Hello this is not the same problem related to forking. It seems like its not a problem with skywalking agent but with Django?

I can access it normally through python manage.py run without reporting an error, but there will be a problem through gunicorn. Finally, I refer to skywalking.apache.org/docs/skywalking-python/next/en/setup/faq/how-to-use-with-uwsgi, and the same problem occurs through uwsgi.
Is there any way to find out what the problem is?

Is everything fine when you use uwsgi/gunicorn Django without enabling skywalking agent? Thanks

@Superskyyy
It is normal without enabling the agent.
When I start with gunicorn, the debugging code and execution results are as follows:
           span.tag(TagDbType('mysql'))             print(this.connection.db)             print(type(this.connection.db))             #span.tag(TagDbInstance((this.connection.db or b'').decode('utf-8')))             span.tag(TagDbStatement(query))
b'self-healing' <class 'bytes'> self-healing <class 'str'> self-healing <class 'str'> self-healing <class 'str'> self-healing <class 'str'> self-healing <class 'str'> self-healing <class 'str'>

@Superskyyy
It is normal without enabling the agent.
When I start with gunicorn, the debugging code and execution results are as follows:
        span.tag(TagDbType('mysql'))
        print(this.connection.db)
        print(type(this.connection.db))
        #span.tag(TagDbInstance((this.connection.db or b'').decode('utf-8')))
        span.tag(TagDbStatement(query))

result:
b'self-healing'
<class 'bytes'>
self-healing
<class 'str'>
self-healing
<class 'str'>
self-healing
<class 'str'>
self-healing
<class 'str'>
self-healing
<class 'str'>
self-healing
<class 'str'>

I finally referred to the solution of # 7457, specified post_fork in gunicorn, and encountered a similar # 7457 problem.

@Beancurd-vv @pacific28 @WinterWaterWarm FYI 1.0.0 is voting soon, prefork server support is added and can be automatically injected without postfork hooks.


text:
Feature/slingbucks2

Rewrite of the Slingbucks sample with HTL, service user and JCR Content Package using the new WCMIO Content Package Plugin (1.5.0).
For the time being the WCMIO Content Package must be download from https://github.com/wcm-io/wcm-io-tooling/releases/tag/wcmio-content-package-maven-plugin-1.5.0 and installed locally until the plugin is available on Maven Central.

With the upcoming migration to git this repository will be deleted. If the pull request contains data that should be re-submitted I suggest that you save the changes locally and resubmit them after the migration, which will likely happen on Monday the 9th of October.
Note that we'll likely revisit the package situation once the filevault-package-maven-plugin has its first release in Jackrabbit.

authors: rombert schaefa

repo: apache/sling


text:
exclude eclipse settings from license check

Description
Gradle build fails because of eclipse internal licenses.
Solution
Exclude eclipse settings folder from checks
Tests
Gradle build runs again.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[ ] I have created a Jira issue and added the issue ID to my pull request title. ( it's minor, not worth mentioning in the release notes)
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

I got lost in git. ;-)
Will open a new PR.

@madrob
Hi Mike, replaced by #2082 ...

repo: apache/solr