text:
[standalone] remove noisy log on pulsar standalone startup

Motivation
Running bin/pulsar standalone you see a useless/noisy log as the global cluster does not exist yet
org.apache.pulsar.broker.web.RestException: Cluster does not exist
	at org.apache.pulsar.broker.admin.impl.ClustersBase.lambda$getCluster$1(ClustersBase.java:122) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_292]
	at org.apache.pulsar.broker.admin.impl.ClustersBase.getCluster(ClustersBase.java:122) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_292]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_292]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) ~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) ~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:65) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [org.eclipse.jetty-jetty-io-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [org.eclipse.jetty-jetty-io-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [org.eclipse.jetty-jetty-io-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) [org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.63.Final.jar:4.1.63.Final]

Modifications
Check if global cluster exists before trying to call get method (which logs an exception on the broker side)
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.

/pulsarbot rerun-failure-checks

@codelipenghui @lhotari PTAL

Committed to master branch.
Thank you @nicoloboschi

authors: eolivelli nicoloboschi


text:
fix wrong result for looking up a non-exist topic by rest api

Fixes #13028
Motivation
For now, the result of the lookup command for a topic not existing by the rest API will return a brokerUrl.
This pull request resolves this problem.
Modifications
Add topic exist check before get the brokerUrl

if not exist, return a ResetException with status code NOT_FOUND
if exist, get the brokerUrl

Verifying this change
This change added tests and can be verified as follows:
HttpTopicLookupv2Test.testLookupTopicNotExist
Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[ ] doc-required
(If you need help on updating docs, create a doc issue)

[x] no-need-doc
(Please explain why)

[ ] doc
(If this PR contains doc changes)

@codelipenghui yes, I found this problem in the unit test too.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@codelipenghui PTAL, thanks !

@aloyszhang It will be a breaking change if users use HTTP lookup of the client and enable topic auto-creation

I don't think this will actually break an application, as eventually user will first produce to then consume from a topic to make use of it, and if topic auto creation is enabled topic will always be created anyway, so that should be fine.

cc @congbobo184 @BewareMyPower

cc@eolivelli

@eolivelli if there's no objection, I think we can merge this to the master branch.

source: gharchive/pull-request

authors: MarvinCai aloyszhang

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
[fix][test] Fix flaky test PersistentTopicsTest#testTriggerCompactionTopic

Fixes #15894
Motivation
testTriggerCompactionTopic is flaky, because createNonPartitionedTopic  and compact use the same response:
https://github.com/apache/pulsar/blob/ecd275dc21f33483a649e5b872990771257b1d45/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/PersistentTopicsTest.java#L902-L908
they are both return Status.NO_CONTENT, so the test fails sporadically with below error:
But was 2 times:
-> at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$internalTriggerCompactionNonPartitionedTopic$309(PersistentTopicsBase.java:3804)
-> at org.apache.pulsar.broker.admin.v2.PersistentTopics.lambda$createNonPartitionedTopic$2(PersistentTopics.java:352)

Documentation

[x] doc-not-needed

Add release/xxx labels because I see it also fail in branch-2.8: https://github.com/apache/pulsar/runs/7552588163?check_suite_focus=true

Remove these labels since this flakiness is related to the code changes from https://github.com/apache/pulsar/pull/14030.

authors: BewareMyPower Technoboy-


text:
The consumption performance index cannot be obtained, and the obtaine…

Fixes:#8023
Motivation
The consumption performance index cannot be obtained, and the obtained value is all 0
Modifications
Add code to collect the indicators of stats.rateOut and stats.throughputOut in the getTopicStats method of the NamespaceStatsAggregator class
Verifying this change
Make sure that the change passes the CI checks.
This change added tests
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes
•Dependencies (does it add or upgrade a dependency): (no)
•The public API: (no)
•The schema: (no)
•The default values of configurations: (no)
•The wire protocol: (no)
•The rest endpoints: (no)
•The admin cli options: (no)
•Anything that affects deployment: (don't know)
Documentation
•Does this pull request introduce a new feature? (no)

/pulsarbot run-failure-checks

move to milestone 2.8.0

Closed as stale and conflict. Please rebase and resubmit the patch if it's still relevant.

authors: codelipenghui g0715158 tisonkun


text:
Issue 9130: Pulsar-admin sinks create: bad error message "java.lang.NullPointerException: path is 'null'." in case of missing "--name" parameter

Fixes #9130
Motivation
Return a better error to the user
Modifications
Add an explicit validation and do not let WebTarget#path throw an internal error.
The validation covers tenant,namespace and sinkname in all of the functions regarding sinks.
We must do the validation here because we have to prevent code to pass bad values to JAX-RS client
Verifying this change
This change added unit tests

/pulsarbot rerun-failure-tests

/pulsarbot rerun-failure-tests

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@eolivelli Can you rebase to the latest master?

@eolivelli Can you rebase to the latest master?

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

authors: eolivelli sijie


text:
[ISSUE #461]Purge filterserv related code in client

What is the purpose of the change
fix issue[461]
Brief changelog
1 add  @Deprecated in Class MQClientInstance
2 add @Deprecated in Class ClientConfig
Verifying this change
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.03%) to 42.27% when pulling 55a05b55cab9515ba9769fa3cfd479c2adafd817 on wlliqipeng:develop into 7a6164d4db0dbed88d5ce90201c7f4af88470219 on apache:develop.

LGTM

authors: coveralls wlliqipeng

repo: apache/rocketmq


text:
DataGrid/DataGridColumn problems in Scroller/Group/Panel

There are several problems:

Horizontal (may be vertical) scroll bar does not function.

All fields are displayed in the fixed Panel width, consequently, the width of each field is ignored.

Compared with the Flash output, rows are not drawn gray color by every two rows.

textAlign attribute does not function.

headerStykeName attribute does not function.

Would you please check the attachment?  Thanks.
DGC2.txt
DGC2.xlsx

I switched your example to AdvancedDataGrid (instead of DataGrid) and I think it works ok.  If you don't want to use ADG, you can try moving some of the functionality down into DataGrid classes.

Alex, thanks.  I checked with the latest build but no output.  I used VS Code with the latest ActionScript/MXML extension.  I will recheck tomorrow.

It might have required some changes I had pending.  I've pushed the latest changes, so try again and if still no output, check the console for errors.

Try giving the Application tag a width and height.  In Flex, the SWFs had a default width/height.  We haven't implemented that for JS yet partly because I'm not sure whether that is worth emulating.  I assumed that most folks do set some sort of width/height (often just 100%) but maybe not.

Alex,
mx:DataGrudColumn does not support height attribute.  Would you please check it?

I don't see a height attribute for mx:DataGridColumn in the Flex documentation.  Where is it documented and/or what is it supposed to do?

Alex, I’ve just tried your suggestion of specifying height.  So far, I do not know how to fix the no output. With your first fix, there was output.

This is what worked for me:
<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009"
			   xmlns:mx="library://ns.apache.org/royale/mx"
			   xmlns:s="library://ns.apache.org/royale/spark" width="1400" height="400">
	<fx:Script>
		<![CDATA[

			import mx.collections.ArrayCollection;

			[Bindable]
			private var myCp:ArrayCollection = new ArrayCollection([
				{
					no: "1",
					contactPointId: "00000001",
					contactPoint: "No-Paper",
					zipCode: "123-4567",
					prefecture: "Kagawa",
					city: "Takamatsu",
					address: "",
					phoneNumber: "",
					faxNumber: "123456789",
					templateId1: "ABCD0001",
					templateId2: "ABCD0002",
					templateId3: "ABCD0003",
					templateId4: "ABCD0004",
					templateId5: "ABCD0005",
					multiFormFlagLabel: "Multi-form (AB, CD, both AB and CD)",
					kanjiFlagLabel: "Not Required",
					furiganaFlagLabel: "Not Required",
					loanAmountFlagLabel: "Not required",
					insuranceAmountSumFlagLabel: "Not Required",
					insurancePlanId: "00112233",
					partnerIdLabel: "Partner A",
					userName: "",
					updatedDate: "",
					deleteFlagLabel: ""
				}
			]);
		]]>

	</fx:Script>

	<s:Scroller width="100%" height="100%">
		<s:Group>
			<s:Panel x="19" y="235" width="1324" height="394" title="Search Result">
				<mx:AdvancedDataGrid id="workTable" tabFocusEnabled="false" x="-1" y="0" width="1324" height="100%"
							 horizontalScrollPolicy="on" dataProvider="{myCp}">
					<mx:columns>
						<mx:AdvancedDataGridColumn width="65" dataField="no" headerText="No." />
						<mx:AdvancedDataGridColumn dataField="contactPointId" headerText="お客さま連絡先ID" width="300" />
						<mx:AdvancedDataGridColumn dataField="contactPoint" headerText="お客さま連絡先（宛名）" width="350"/>
						<mx:AdvancedDataGridColumn dataField="zipCode" headerText="郵便番号" width="200"/>
						<mx:AdvancedDataGridColumn dataField="prefecture" headerText="住所（都道府県）" width="300"/>
						<mx:AdvancedDataGridColumn dataField="city" headerText="住所（区市町村）" width="300"/>
						<mx:AdvancedDataGridColumn dataField="address" headerText="住所（残り）" width="200"/>
						<mx:AdvancedDataGridColumn dataField="phoneNumber" headerText="電話番号" width="200"/>
						<mx:AdvancedDataGridColumn dataField="faxNumber" headerText="FAX番号" width="200"/>
						<mx:AdvancedDataGridColumn dataField="templateId1" headerText="連絡票テンプレート（漢字）ID" width="300"/>
						<mx:AdvancedDataGridColumn dataField="templateId2" headerText="連絡票テンプレート（ひらがな）ID" width="300"/>
						<mx:AdvancedDataGridColumn dataField="templateId3" headerText="連絡票テンプレート（カタカナ）ID" width="300"/>
						<mx:AdvancedDataGridColumn dataField="templateId4" headerText="連絡票テンプレート（英語）ID" width="300"/>
						<mx:AdvancedDataGridColumn dataField="templateId5" headerText="連絡票テンプレート（混合）ID" width="300"/>
						<mx:AdvancedDataGridColumn dataField="multiFormFlagLabel" headerText="複票フラグ" width="200"/>
						<mx:AdvancedDataGridColumn dataField="kanjiFlagLabel" headerText="連絡要漢字フラグ" width="200"/>
						<mx:AdvancedDataGridColumn dataField="furiganaFlagLabel" headerText="連絡要フリガナフラグ" width="200"/>
						<mx:AdvancedDataGridColumn dataField="loanAmountFlagLabel" headerText="要ローン金額フラグ（連絡票）" width="300"/>
						<mx:AdvancedDataGridColumn dataField="insuranceAmountSumFlagLabel" headerText="要合計保険金額フラグ（連絡票）" width="300"/>
						<mx:AdvancedDataGridColumn dataField="insurancePlanId" headerText="プランID" width="200"/>
						<mx:AdvancedDataGridColumn dataField="partnerIdLabel" headerText="パートナー" width="200"/>
						<mx:AdvancedDataGridColumn dataField="userName" headerText="更新ユーザー" width="200"/>
						<mx:AdvancedDataGridColumn dataField="updatedDate" headerText="更新日時" width="200"/>
						<mx:AdvancedDataGridColumn dataField="deleteFlagLabel" headerText="削除" width="100" />
					</mx:columns>
				</mx:AdvancedDataGrid>
			</s:Panel>
			<s:Label y="93" width="1900" height="45" fontSize="30" textAlign = "center" text="Contact Master" verticalAlign="top"/>
		</s:Group>
	</s:Scroller>
</s:Application>

Alex, thanks.  I've confirmed the fix.  I will check why my mxml does not work.

source: gharchive/issue

repo: apache/royale-asjs


text:
Using HTTPService fails to compile when adding Network dependancy to working program

With a working pom / basic example adding a HTTPService call and the Network library  as a dependancy fails to compile.
Example code:
<?xml version="1.0" encoding="utf-8"?>
<js:Application xmlns:fx="http://ns.adobe.com/mxml/2009"
                xmlns:js="library://ns.apache.org/royale/basic"
                applicationComplete="init()">

    <fx:Script><![CDATA[
        import org.apache.royale.net.HTTPService;

        public var http:HTTPService = new HTTPService();

        public function init():void {
            load.url = "test.JSON";
            load.send();
        }
        ]]></fx:Script>

    <js:valuesImpl>
        <js:SimpleCSSValuesImpl/>
    </js:valuesImpl>

    <js:initialView>
        <js:View>
            <js:Label text="Hello World" />
        </js:View>
    </js:initialView>

</js:Application>

Add to pom.xml
    <dependency>
      <groupId>org.apache.royale.framework</groupId>
      <artifactId>Network</artifactId>
      <version>0.9.0-SNAPSHOT</version>
      <type>swc</type>
    </dependency>

Fails to compile with several errors as it cannot find HTTPService:
/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(7): col: 16 Warning: Definition org.apache.royale.net.HTTPService could not be found.

        import org.apache.royale.net.HTTPService;
               ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(9): col: 25 Error: Type was not found or was not a compile-time constant: HTTPService.

        public var http:HTTPService = new HTTPService();
                        ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(9): col: 43 Error: Call to a possibly undefined method HTTPService.

        public var http:HTTPService = new HTTPService();
                                          ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(13): col: 13 Error: Access of possibly undefined property load.

            load.url = "test.JSON";
            ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(14): col: 13 Error: Access of possibly undefined property load.

            load.send();
            ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(7): col: 16 Definition org.apache.royale.net.HTTPService could not be found.

        import org.apache.royale.net.HTTPService;
               ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(9): col: 25 Type was not found or was not a compile-time constant: HTTPService.

        public var http:HTTPService = new HTTPService();
                        ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(9): col: 43 Call to a possibly undefined method HTTPService.

        public var http:HTTPService = new HTTPService();
                                          ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(13): col: 13 Access of possibly undefined property load.

            load.url = "test.JSON";
            ^

/Users/justinmclean/IdeaProjects/FlexJSTest/src/Example.mxml(14): col: 13 Access of possibly undefined property load.

            load.send();

Working pom.xml had Basic, HTML, Language and playerglobal as dependancies.

Debug which may help:
[DEBUG] org.example.helloworld:test:swf:1.0-SNAPSHOT
[DEBUG]    org.apache.royale.framework:Basic:swc:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Core:swc:swf:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Core:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Binding:swc:swf:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Binding:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Graphics:swc:swf:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Graphics:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Collections:swc:swf:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Collections:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.typedefs:royale-typedefs-gcl:swc:typedefs:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.typedefs:royale-typedefs-js:swc:typedefs:0.9.0-SNAPSHOT:runtime
[DEBUG]    org.apache.royale.framework:HTML:swc:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Basic:swc:swf:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Basic:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]    org.apache.royale.framework:Network:swc:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Reflection:swc:swf:0.9.0-SNAPSHOT:compile
[DEBUG]       org.apache.royale.framework:Reflection:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]    org.apache.royale.framework:Language:swc:js:0.9.0-SNAPSHOT:compile
[DEBUG]    com.adobe.flash.framework:playerglobal:swc:20.0:provided

I'd guess you're missing the NetworkJS dependency.

It used to be enough to just do this to add both:
    <dependency>
      <groupId>org.apache.royale.framework</groupId>
      <artifactId>Network</artifactId>
      <version>0.9.0-SNAPSHOT</version>
      <type>swc</type>
    </dependency>

Looks like something has changed?

Are you missing js ?

Turns out you need both swf and js classifiers are needed like so:
    <dependency>
      <groupId>org.apache.royale.framework</groupId>
      <artifactId>Network</artifactId>
      <version>0.9.0-SNAPSHOT</version>
      <type>swc</type>
      <classifier>swf</classifier>
    </dependency>
    <dependency>
      <groupId>org.apache.royale.framework</groupId>
      <artifactId>Network</artifactId>
      <version>0.9.0-SNAPSHOT</version>
      <type>swc</type>
      <classifier>js</classifier>
    </dependency>

But you don't need to do this for Basic dependancy.

authors: Harbs justinmclean piotrzarzycki21


text:
sink jdbc connect hive Error

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
ERROR org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter - close error
java.lang.LinkageError: loader constraint violation: when resolving method 'org.apache.thrift.transport.TTransport org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSocketTransport(java.lang.String, int, int)' the class loader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @a92104c of the current class, org/apache/hive/jdbc/HiveConnection, and the class loader 'app' for the method's defining class, org/apache/hadoop/hive/common/auth/HiveAuthUtils, have different Class objects for the type org/apache/thrift/transport/TTransport used in the signature (org.apache.hive.jdbc.HiveConnection is in unnamed module of loader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @a92104c, parent loader 'app'; org.apache.hadoop.hive.common.auth.HiveAuthUtils is in unnamed module of loader 'app')
SeaTunnel Version
2.3.4
SeaTunnel Config
env {
  execution.parallelism = 1
  job.mode = "STREAMING"
  checkpoint.interval = 5000
  #read_limit.bytes_per_second=70000
  read_limit.rows_per_second=400
}
source {
  Postgres-CDC {
    result_table_name ="r_name"
    table-names = ["xx_1","xxs_2","xxx_3"]
    username = "postgres"
    password = "postgres"
    database-names = ["test"]
    schema-names = ["public"]
    base-url = "jdbc:postgresql://ip/test?loggerLevel=OFF"
    startup.mode = latest
  }
}
sink {
    jdbc {
      source_table_name ="r_name"
      url = "jdbc:hive2://127.0.0.1:10000/hive_test"
      driver = "org.apache.hive.jdbc.HiveDriver"
      user = "root"
      password = "root"
      generate_sink_sql = true
      database = hive_test
    }
}

Running Command
seatunnel --config ./myConfig/postgresql-cdc-hive2.conf

Error Exception
ERROR org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSinkWriter - close error
java.lang.LinkageError: loader constraint violation: when resolving method 'org.apache.thrift.transport.TTransport org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSocketTransport(java.lang.String, int, int)' the class loader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @a92104c of the current class, org/apache/hive/jdbc/HiveConnection, and the class loader 'app' for the method's defining class, org/apache/hadoop/hive/common/auth/HiveAuthUtils, have different Class objects for the type org/apache/thrift/transport/TTransport used in the signature (org.apache.hive.jdbc.HiveConnection is in unnamed module of loader org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader @a92104c, parent loader 'app'; org.apache.hadoop.hive.common.auth.HiveAuthUtils is in unnamed module of loader 'app')

Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

As far as I know, hive jdbc does not support upsert operation at the moment and I am curious how you do it?
cc @hailin0 @ic4y

repo: apache/seatunnel


text:
URIRegisterDTO add eventType

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #4748 (f9a8b0b) into master (b50a1e9) will increase coverage by 0.06%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #4748      +/-   ##
============================================
+ Coverage     66.91%   66.97%   +0.06%
- Complexity     8079     8094      +15
============================================
  Files          1106     1106
  Lines         32256    32245      -11
  Branches       2923     2919       -4
============================================
+ Hits          21583    21597      +14
+ Misses         8956     8925      -31
- Partials       1717     1723       +6

Impacted Files
Coverage Δ

...ore/register/ClientInfoRefreshedEventListener.java
0.00% <0.00%> (ø)

... and 56 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter tian-pengfei

repo: apache/shenyu


text:
[ISSUE #4687] shenyu-sdk-spring test case.

#4538
Make sure that:

[x] You have read the contribution guidelines.
[x] You submit test cases (unit or integration tests) that back your changes.
[x] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #4913 (4ac138d) into master (3ab7df0) will decrease coverage by 0.05%.
Report is 1 commits behind head on master.
The diff coverage is 80.95%.

@@             Coverage Diff              @@
##             master    #4913      +/-   ##
============================================
- Coverage     64.04%   64.00%   -0.05%
- Complexity     8312     8433     +121
============================================
  Files          1180     1195      +15
  Lines         34867    35368     +501
  Branches       3115     3203      +88
============================================
+ Hits          22331    22636     +305
- Misses        10763    10891     +128
- Partials       1773     1841      +68

Files Changed
Coverage Δ

...ng/annotation/RequestHeaderParameterProcessor.java
73.07% <42.85%> (ø)

...ing/annotation/RequestParamParameterProcessor.java
86.36% <80.00%> (ø)

...ing/annotation/PathVariableParameterProcessor.java
88.00% <88.88%> (ø)

...ache/shenyu/sdk/spring/ShenyuClientsRegistrar.java
66.04% <100.00%> (ø)

...ring/annotation/CookieValueParameterProcessor.java
94.11% <100.00%> (ø)

... and 16 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

pls review. @moremind

authors: codecov-commenter damonxue


text:
[Bug]

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
Java Agent (apache/skywalking-java)
What happened
I use the taobao SDK, which contains the jar package for HTTP client and is a custom class to load all the classes in the jar package. An exception occurs。
the SDK customer class load code:
package com.taobao.pac.sdk.cp.sar;
import com.taobao.pac.sdk.cp.Logger;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Enumeration;
import java.util.Properties;
public class IClassLoader extends URLClassLoader {
private static final String version = "1.0.0.5";
private static IClassLoader instance;
private OpenClassLoader parent;
private static final boolean isForceUpdate = false;
private static final String standardClasspathConfigFile = "/sar/sar.properties";
private static final String standardClasspathDir = "/sar/jars/";
private static String workDir = System.getProperty("user.home");
private IClassLoader(URL[] urls) {
    this(urls, IClassLoader.class.getClassLoader());
}

private IClassLoader(URL[] urls, ClassLoader parent) {
    super(urls, (ClassLoader)null);
    this.parent = new OpenClassLoader(parent);
}

public static synchronized IClassLoader getInstance() throws IOException {
    if (instance != null) {
        return instance;
    } else {
        instance = getInstance("/sar/sar.properties");
        return instance;
    }
}

private static IClassLoader getInstance(String classpathConfigFile) throws IOException {
    Class clazz = IClassLoader.class;
    InputStream in = clazz.getResourceAsStream(classpathConfigFile);
    Properties config = new Properties();
    String sarDir = workDir + "/sar/jars/".replace("/", File.separator) + "1.0.0.5" + File.separator;
    File sarDirFile = new File(sarDir);
    if (!sarDirFile.exists() && !sarDirFile.mkdirs()) {
        throw new RuntimeException("create sar dir fail:" + sarDir);
    } else {
        Logger.error("初始化sar目录：" + sarDirFile);

        IClassLoader var34;
        try {
            config.load(in);
            String content = config.getProperty("jars");
            Logger.error("sar中jar包列表如下：" + content);
            String[] jars = content.split(";");
            URL[] urls = new URL[jars.length];

            for(int i = 0; i < jars.length; ++i) {
                String jarName = jars[i];
                jarName = jarName.trim();
                String resource = "/sar/jars/" + jarName;
                URL url = clazz.getResource(resource);
                if (url == null) {
                    throw new RuntimeException("can't found resources:" + resource);
                }

                String urlStr = url.toString();
                String jarFileName;
                if (urlStr.startsWith("wsjar:")) {
                    jarFileName = urlStr.substring(2);
                    url = new URL(jarFileName);
                }

                if (url.getProtocol().equals("jar")) {
                    jarFileName = sarDir + jarName;
                    File jarFile = new File(jarFileName);
                    if (jarFile.exists()) {
                        Logger.error("jar包已经存在：" + jarFileName);
                    } else {
                        jarFile.createNewFile();
                        InputStream jarIn = null;
                        OutputStream jarOut = null;

                        try {
                            jarIn = clazz.getResourceAsStream(resource);
                            jarOut = new FileOutputStream(jarFile);
                            transfer(jarIn, jarOut);
                        } catch (Exception var29) {
                            Exception e = var29;
                            throw new RuntimeException(e);
                        } finally {
                            if (jarIn != null) {
                                jarIn.close();
                            }

                            if (jarOut != null) {
                                jarOut.flush();
                                jarOut.close();
                            }

                        }

                        Logger.error("jar包初始化成功：" + jarFileName);
                    }

                    url = jarFile.toURI().toURL();
                } else {
                    Logger.error("加载文件协议资源：" + resource);
                }

                Logger.error("加载url：" + url);
                urls[i] = url;
            }

            var34 = new IClassLoader(urls);
        } catch (IOException var31) {
            IOException e = var31;
            Logger.error("加载文件协议资源异常", e);
            throw e;
        } finally {
            if (in != null) {
                in.close();
            }

        }

        return var34;
    }
}

private static void transfer(InputStream is, OutputStream os) throws IOException {
    byte[] buf = new byte[4096];

    while(true) {
        int count = is.read(buf);
        if (count < 0) {
            return;
        }

        os.write(buf, 0, count);
    }
}

public Class<?> loadClass(String name) throws ClassNotFoundException {
    try {
        return super.loadClass(name);
    } catch (ClassNotFoundException var3) {
        return this.parent.loadClass(name);
    }
}

public Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
    try {
        return super.loadClass(name, resolve);
    } catch (ClassNotFoundException var4) {
        return this.parent.loadClass(name, resolve);
    }
}

public URL getResource(String name) {
    URL url = super.getResource(name);
    if (url == null) {
        url = this.parent.getResource(name);
    }

    return url;
}

public Enumeration<URL> getResources(String name) throws IOException {
    Enumeration[] tmp = new Enumeration[2];
    Enumeration<URL> resources = super.getResources(name);
    tmp[0] = resources;
    if (this.parent != null) {
        Enumeration<URL> parentResources = this.parent.getResources(name);
        tmp[1] = parentResources;
    }

    return new CompoundEnumeration(tmp);
}

public InputStream getResourceAsStream(String name) {
    InputStream in = super.getResourceAsStream(name);
    return in != null ? in : this.parent.getResourceAsStream(name);
}

public static void setWorkDir(String workDir) {
    if (workDir != null && workDir.trim().length() >= 1) {
        if (workDir.endsWith(File.separator)) {
            workDir = workDir.substring(0, workDir.length() - 1);
        }

        IClassLoader.workDir = workDir;
    } else {
        throw new RuntimeException("非法sar包工作目录");
    }
}

interface ConfigKey {
    String jars = "jars";
}

}
`
An exception occurs, main" java.lang.IllegalAccessError: tried to access class org.apache.http.impl.client.InternalHttpClient$$sw$auxiliary$8shium3 from class org.apache.http.impl.client.InternalHttpClient.
If I prioritize using this SDK and there are no system exceptions,
If I prioritize using the client of the application's built-in Apache, an exception occur
What you expected to happen
If I prioritize using this SDK and there are no system exceptions,
If I prioritize using the client of the application's built-in Apache, an exception occur
How to reproduce
If I prioritize using this SDK and there are no system exceptions,
If I prioritize using the client of the application's built-in Apache, an exception occur
Anything else
If I prioritize using this SDK and there are no system exceptions,
If I prioritize using the client of the application's built-in Apache, an exception occur
Are you willing to submit a pull request to fix on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Ask taobao. It is not our codes, not our responsibility.

i delete code:         agentBuilder = agentBuilder.with(new AgentBuilder.InjectionStrategy.UsingUnsafe.OfFactory(factory));
it works will

repo: apache/skywalking

license_source: bigquery


text:
Support rocketmq monitoring

[x] Update the documentation to include this new feature.
[x] Tests(including UT, IT, E2E) are added to verify the new feature.
[x] If it's UI related, attach the screenshots below.
[x] Update the CHANGES log.

According to the latest requirement of the community about feature change, you should submit another pull request to finish the SWIP process.
There are two SWIPs existed for your references.

According to the latest requirement of the community about feature change, you should submit another pull request to finish the SWIP process.
There are two SWIPs that exist for your reference.

Ok, i have started the disscusion

FYI, RocketMQ upstream is working on the release.
Thread, https://lists.apache.org/thread/wp8nl4tmd3t15dlc8jb7kq07z1cv4qh0

FYI, RocketMQ upstream is working on the release.
Thread, https://lists.apache.org/thread/wp8nl4tmd3t15dlc8jb7kq07z1cv4qh0

The official image has been  pulished, I changed the repo in e2e

Do we need a doc update to refer to new RocketMQ release?

Do we need a doc update to refer to new RocketMQ release?

No, the personal repo was only used in the e2e. The docs do not mention the repo.

The Card Widget requires SINGLE_VALUE. Please check others UI templates.

Done

The Card Widget requires SINGLE_VALUE. Please check others UI templates.

May I ask a question? Why should meter_rocketmq_topic_consumer_latency be as a single value even though it is a line graph?

The Card Widget requires SINGLE_VALUE. Please check others UI templates.

May I ask a question? Why should meter_rocketmq_topic_consumer_latency be as a single value even though it is a line graph?

Are you use a Card widget? If so, we should use MQE to control which value should be picked, rather than UI. Because UI will pick the first value, which may be days ago, or minutes away, depending on the UI query condition.

The Card Widget requires SINGLE_VALUE. Please check others UI templates.

May I ask a question? Why should meter_rocketmq_topic_consumer_latency be as a single value even though it is a line graph?

Are you use a Card widget? If so, we should use MQE to control which value should be picked, rather than UI. Because UI will pick the first value, which may be days ago, or minutes away, depending on the UI query condition.

Okay, I got it, But it is a line widget

@wankai123
Should I restore to Time series ?

Should I restore to Time series ?

Oops, sorry. you are right, then should restore both expression and type.

Should I restore to Time series ?

Oops, sorry. you are right, then should restore both expression and typesOfMQE for this Widget.

Should I restore to Time series ?

Oops, sorry. you are right, then should restore both expression and typesOfMQE for this Widget.

Okay

Should I restore to Time series ?

Oops, sorry. you are right, then should restore both expression and typesOfMQE for this Widget.

That's ok

OK. We are still waiting for RocketMQ exporter released. It seems not that quick.

@duhenglucky Is there an estimated release date?

@duhenglucky Is there an estimated release date?

I got the reply from the rocketmq community they will release it after addressing a little problem

OK, I will be on vacation from tomorrow, so I will be off line for most of time. @wankai123 will review this later when you are ready.

Is RocketMQ community ready for image?

Is RocketMQ community ready for image?

They will release in Feb ,But it would be after the Spring Festival

OK, thanks for the update.

What happened?

@wankai123
Hi, I found MQE seems work

  - name: producer_offset
    exp: rocketmq_producer_offset.sum(['cluster','topic']).downsampling(MAX)

  - name: consumer_group_offset
    exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).downsampling(MAX)

@wankai123 Hi, I found MQE seems work

yes, in your case it works, because these expressions are added suffix in rocketmq-topic.yaml: .endpoint(['cluster'], ['topic'], Layer.ROCKETMQ). MAL will remove metric-level relevant labels.
So actually the result metric producer_offset without labels and consumer_group_offset has a label group.

@wankai123 Hi, I found MQE seems work

yes, in your case it works, because these expressions are added suffix in rocketmq-topic.yaml: .endpoint(['cluster'], ['topic'], Layer.ROCKETMQ). MAL will remove metric-level relevant labels.
So actually the result metric producer_offset without labels and consumer_group_offset has a label group.

Ok, I got it

They tagged during Chinese new year?

They tagged during Chinese new year?

Yes, 5 days ago

@wankai123 When you are back, please take a look.
@peachisai How is your showcase side change?

@wankai123 When you are back, please take a look.
@peachisai How is your showcase side change?

Is the K8S config requested?
I'm trying to write k8s config file

Is the K8S config required?

Yes, it's required.

Could you provide screenshots for latest changes?

Could you provide screenshots for latest changes?

Producer or Produce? I think it should be producer. Could you cross check the existing configurations?

producer

Fixed

Please add hierarchy node metrics for rocketMQ service in rocketmq-cluster.json

What are these metrics for?

Please add hierarchy node metrics for rocketMQ service in rocketmq-cluster.json

What are these metrics for?

show the metrics on hierarchy topology when mouse over the node.

After the showcase is updated, I think you could write a blog in CN to introduce this. I could try to do an x-post on both RocketMQ and SkyWalking sides. FYI @duhenglucky

@peachisai Are you working on the showcase or the blog? @wankai123 Should we consider this as a part of showcase? Or the showcase is already too complex?

OK. But I think we should be careful about whether activate this on the demo.skywalking.a.o. We have a heavy resource requirement of CPU and Memory already.

@peachisai Are you working on the showcase or the blog? @wankai123 Should we consider this as a part of showcase? Or the showcase is already too complex?

Yes, I have tried to test the showcase locally, and now I am trying to establish the local environment. These days, I'm a little busy, but it is in process.

@peachisai This PR seems missing doc update about opentelemetry-receiver.md. Could you open another PR to update the table in the bottom?

@peachisai This PR seems missing doc update about opentelemetry-receiver.md. Could you open another PR to update the table in the bottom?
Ok, I will update it later

authors: peachisai wankai123 wu-sheng


text:
JTS and Polygons jar file

Hi,
Is there a way to install/download the library jts-core in the folder SOLR_INSTALL/server/solr-webapp/webapp/WEB-INF/lib/? As specified in the official guide: https://solr.apache.org/guide/8_10/spatial-search.html#jts-and-polygons-flat
Actually, I tried to put in the middle an initContainer that downloads such jar, but I get obviously a Permission denied from the Solr container since only root can write on the final solr container.
I tried also to set a securityContext only for my initContainer, in order to run as root, but that configuration has no effect in the initContainer, I think it is not seen by the Solr CRD.
podOptions:
  initContainers:
  - name: "install-jts-core"
    image: solr:8.9.0
    command: ['sh', '-c', 'wget -O /opt/solr-8.9.0/server/solr-webapp/webapp/WEB-INF/lib/jts-core-1.15.1.jar https://repo1.maven.org/maven2/org/locationtech/jts/jts-core/1.15.1/jts-core-1.15.1.jar']
    securityContext:   <--- this has no effect on SolrCloud CRD
      runAsUser: 0

Another disperate attempt was to set a podSecurityContext.runAsUser: 0, so for all containers in the pod, but Solr does not run as root, I discarded that option by the way.
Any hint/idea/solution please?
Thank you very much in advance.

Hello @fernaspiazu, this is a good question.
Starting with Solr Operator v0.5.0, you should be able to do the following:

Use the init container, and save it in an empty-dir volume. Mount this volume into your init-container at any location. 2. Mount this emtpy-dir jar volume in your Solr container, somewhere under the data path (/var/solr/data/libs).
Then use SolrCloud.spec.additionalLibs=['/var/solr/data/libs']. (Note this will only work for more recent solr versions, but that is fine since you are running 8.9.0.

I recognize that the Solr documentation says the Jar has to be placed in WEB_INF/lib, but I really don't think that's possible given the Solr docker image.
Give this a try and let me know if it works.

Hi @HoustonPutman,
Thank you very much for your answer.
I have recently found a solution that may not be elegant, but works well in any version of Solr image, below a configuration example:
podOptions:
  initContainers:
  - name: "install-jts-lib"
    image: solr:8.9.0
    command:
    - 'sh'
    - '-c'
    - |
      wget -O /tmp-webinf-lib/jts-core-1.15.1.jar https://repo1.maven.org/maven2/org/locationtech/jts/jts-core/1.15.1/jts-core-1.15.1.jar
      cp -R /opt/solr/server/solr-webapp/webapp/WEB-INF/lib/* /tmp-webinf-lib
    volumeMounts:
    - mountPath: /tmp-webinf-lib
      name: web-inf-lib
  volumes:
  - name: web-inf-lib
    source:
      emptyDir: {}
    defaultContainerMount:
      name: web-inf-lib
      mountPath: /opt/solr/server/solr-webapp/webapp/WEB-INF/lib

In this example, I create an emptyDir volume, attach it in any directory of the initContainer, but in the target directory of the final Solr image, this will empty the WEB-INF/lib directory, but since I'm using the same Solr image, I can copy the content of WEB-INF/lib (jars and folders) at the end.
The effect is that the final container will have all the content it should have had plus the jts-core-1.15.1.jar jar.
This works also with other files or libraries you want to bring in the Solr container.
Let me know what do you thing of this workaround 👍
Thank you again.

That's really clever!
Glad it's working for you!

I close this issue, till another similar use case appears.
Thanks again.

For those who might be interested in an alternative to @fernaspiazu's trick, here's how I did it:
lifecycleHooks:
  postStart:
    exec:
      command:
        - /bin/sh
        - -ceux
        - >
          jts_version=1.18.2 &&
          dir=/opt/bitnami/solr/server/solr-webapp/webapp/WEB-INF/lib &&
          test -f ${dir}/jts-core-${jts_version}.jar ||
          curl -o ${dir}/jts-core-${jts_version}.jar https://repo1.maven.org/maven2/org/locationtech/jts/jts-core/${jts_version}/jts-core-${jts_version}.jar

Although it works fine for me in practice, this solution probably relies on the postStart being called before the app starts up.
And no such guarantee in terms of timing exists as far as I can read on https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/.

This hook is executed immediately after a container is created. However, there is no guarantee that the hook will execute before the container ENTRYPOINT.

So your mileage may vary.

Seeing two people independently try to do this (and its awkward) makes me wonder if Solr should finally ship with JTS.  There used to be a license issue many years ago but no longer an issue.  https://issues.apache.org/jira/browse/SOLR-11953
Annoyingly, due to the way JTS is referenced via Spatial4j via Lucene, it has to go in WEB-INF/lib, not in a solr/lib

I'm actually editing in values file.
You want me to send the solrCloud file?
I'm refering to these charts  : https://solr.apache.org/operator/resources.html#helm-charts

Ahh I'm sorry I was wrong about the extraVolumes part, you can remove that. Otherwise it looks like you have the wrong indentation for volumes. it should be under podOptions, not initContainers.

authors: HoustonPutman dalbani dsmiley fernaspiazu sandeepg123456

repo: apache/solr-operator


text:
[SPARK-15858][ML]: Fix calculating error by tree stack over flow prob…

What changes were proposed in this pull request?
What changes were proposed in this pull request?
Improving evaluateEachIteration function in mllib as it fails when trying to calculate error by tree for a model that has more than 500 trees
How was this patch tested?
the batch tested on productions data set (2K rows x 2K features) training a gradient boosted model without validation with 1000 maxIteration settings, then trying to produce the error by tree, the new patch was able to perform the calculation within 30 seconds, while previously it was take hours then fail.
PS: It would be better if this PR can be cherry picked into release branches 1.6.1 and 2.0

Can one of the admins verify this patch?

Jenkins add to whitelist

Jenkins test this please

Test build #60366 has started for PR 13624 at commit 7e9e383.

Test build #60366 has finished for PR 13624 at commit 7e9e383.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60366/
Test PASSed.

Merged build finished. Test PASSed.

I think this is correct and can see why it's faster.

Applied the required changes on styles. (not changes on the logic)

Test build #61177 has started for PR 13624 at commit 3f3792f.

Test build #61177 has finished for PR 13624 at commit 3f3792f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61177/
Test PASSed.

Test build #61257 has started for PR 13624 at commit 5d047dd.

Test build #61257 has finished for PR 13624 at commit 5d047dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61257/
Test PASSed.

Test build #61277 has started for PR 13624 at commit 9506288.

Test build #61277 has finished for PR 13624 at commit 9506288.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61277/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61329 has started for PR 13624 at commit 2050459.

Test build #61329 has finished for PR 13624 at commit 2050459.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61329/
Test PASSed.

Merged to master

@mhmoudr  There were a few follow-up comments on your PR commit, visible here: [https://github.com/apache/spark/commit/393db655c3c43155305fbba1b2f8c48a95f18d93]
Could you please send a quick follow-up PR?  Thanks!

Linking JIRA for those fixes: [https://issues.apache.org/jira/browse/SPARK-16920]

authors: AmplabJenkins SparkQA jkbradley mhmoudr srowen

repo: apache/spark


text:
[Minor][PySpark] Improve error message when running PySpark with different minor versions

What changes were proposed in this pull request?
Currently the error message is correct but doesn't provide additional hint to new users. It would be better to hint related configuration to users in the message.
How was this patch tested?
N/A because it only changes error message.
Please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request.

cc @holdenk

No jira is opened since I think it is a trivial change. If needed, please let me know.

Test build #68382 has started for PR 15822 at commit ccdac28.

Test build #68382 has finished for PR 15822 at commit ccdac28.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68382/
Test PASSed.

Merged build finished. Test PASSed.

cc @davies

Thanks - did this come from a discussion somewhere?

yeah, as per the discussion at https://github.com/apache/spark/pull/15659#issuecomment-259157516.

One minor nit but otherwise looks good to me, thanks for taking the time to improve the error messaging around this :)

lgtm

Merged to master/2.1

authors: AmplabJenkins SparkQA davies holdenk rxin srowen viirya


text:
[SPARK-18578][SQL] Full outer join in correlated subquery returns incorrect results

What changes were proposed in this pull request?

Raise Analysis exception when correlated predicates exist in the descendant operators of either operand of a Full outer join in a subquery as well as in a FOJ operator itself
Raise Analysis exception when correlated predicates exists in a Window operator (a side effect inadvertently introduced by SPARK-17348)

How was this patch tested?
Run sql/test catalyst/test and new test cases, added to SubquerySuite, showing the reported incorrect results.

Test build #69135 has started for PR 16005 at commit 099f75d.

@hvanhovell I want to get your opinion on this. The more I read the code in this block of pullOutCorrelatedPredicates
  // Simplify the predicates before pulling them out.
  val transformed = BooleanSimplification(sub) transformUp {
    case f @ Filter(cond, child) => ...
    case p @ Project(expressions, child) => ...
    case a @ Aggregate(grouping, expressions, child) => ...
    case w : Window => ...
    case j @ Join(left, _, RightOuter, _) => ...
    case j @ Join(left, right, FullOuter, _) => ...
    case j @ Join(_, right, jt, _) if !jt.isInstanceOf[InnerLike] => ...
    case u: Union => ...
    case s: SetOperation => ...
    case e: Expand => ...
    case l : LocalLimit => ...
    case g : GlobalLimit => ...
    case s : Sample => ...
    case p =>
      failOnOuterReference(p)
      ...
  }

The code disallows operators in a sub plan of an operator hosting correlation on a case by case basis. As it is today, it only blocks Union/Intersect/Except/Expand/LocalLimit/GlobalLimit/Sample/FOJ and right table of LOJ (and left table of ROJ). That means any LogicalPlan operators that are not in the list above are permitted to be under a correlation point. Is this risky? There are many (30+ at least from browsing the LogicalPlan type hierarchy) operators derived from LogicalPlan class. Should we whitelist what operators allowed? For the case of ScalarSubquery, it explicitly checks that only SubqueryAlias/Project/Filter/Aggregate are allowed (CheckAnalysis.scala around line 126-165 in and after def cleanQuery). If we go this route, we should allow, in addition to the ones allowed in ScalarSubquery: Join, Distinct, Sort, OneRowRelation. I am debating about including Window though.

Test build #69137 has started for PR 16005 at commit 5d6cc8e.

@nsyca +1000 on whitelisting operators. That is what we should have done from the start.
Let's break it down:

LeafNodes should not be a problem. We don't need to explicitly handle them
We should allow the following UnaryNode: Project, Filter, Aggregate, SubqueryAlias,  Distinct, Generate (only when join=true), BroadcastHint, Sort, Repartition & RedistributeData (parent of SortPartitions and RepartitionByExpression). We need to find out what other systems allow for Window.
The only BinaryNode we should allow is Join with special cases for Left/Right/Full. We should also make sure that the LeftAnti and LeftSemi are handled properly.

Test build #69135 has finished for PR 16005 at commit 099f75d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69135/
Test PASSed.

@hvanhovell I will work on the whitelist in a new JIRA under SPARK-18455. It will be my top priority task and I hope we can merge it in the next minor release of 2.0.x. Let's have this PR scoped for the FOJ and Window cases. Shall we?

Test build #69137 has finished for PR 16005 at commit 5d6cc8e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69137/
Test PASSed.

LGTM. Merging to master/2.1/2.0. Thanks!

Hmmm. I cannot merge to 2.0 :(... Can you open a backport against 2.0?

authors: AmplabJenkins SparkQA hvanhovell nsyca


text:
[MINOR][PYTHON] Fix typo docstring: 'top' -> 'topic'

What changes were proposed in this pull request?
Fix typo in docstring.

Can one of the admins verify this patch?

Sure, it's worth a search for similar instances because sometimes typos spread via copy and paste. Could you make a pass over related code?

I've (rip)grep'ed  over all .py and .md files in the repository searching for topi? (case insensitive regex) and haven't seen other case of this typo (writing top instead of topic).

Test build #3577 has started for PR 16967 at commit fe1ff4f.

Test build #3577 has finished for PR 16967 at commit fe1ff4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged to master

authors: AmplabJenkins SparkQA rolando srowen


text:
wip: [SPARK-22409] Introduce function type argument in pandas_udf

What changes were proposed in this pull request?

Add a "function type" argument to pandas_udf.
Add a new public enum class PandasUdfType in pyspark.sql.functions
Refactor udf related code from pyspark.sql.functions to pyspark.sql.udf
Merge "PythonUdfType" and "PythonEvalType" into a single enum class "PythonEvalType"

Example:
from pyspark.sql.functions import pandas_udf, PandasUdfType

@pandas_udf('double', PandasUdfType.SCALAR):
def plus_one(v):
    return v + 1

Design doc
https://docs.google.com/document/d/1KlLaa-xJ3oz28xlEJqXyCAHU3dwFYkFs_ixcUXrJNTc/edit
How was this patch tested?
Added PandasUDFTests
TODO:

[x] Implement proper enum type for PandasUdfType
[ ] Update documentation
[ ] Add more tests in PandasUDFTests

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83296/
Test FAILed.

cc @cloud-fan @ueshin @HyukjinKwon
I still have some left items to do but I want to get feedback to make sure we are happy with the design. Please take a look, thanks!

Ping @cloud-fan @ueshin @HyukjinKwon any thoughts on this?

overall LGTM, but let's not add udf types that are not implemented yet, like the AGGREGATE

Thanks @cloud-fan. I will continue to finish the TODO items. Decide the best naming probably needs a bit more careful thinking.
I can remove unimplemented types. However, I think it'd be good if we can include those when thinking about overall naming scheme  of function types.

Test build #83698 has started for PR 19630 at commit 88a6a9a.

Test build #83698 has finished for PR 19630 at commit 88a6a9a.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class PandasUDFType(enum.Enum):

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83698/
Test FAILed.

Test build #83699 has started for PR 19630 at commit ee8009d.

Test build #83699 has finished for PR 19630 at commit ee8009d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83699/
Test FAILed.

Test build #83701 has started for PR 19630 at commit 75d31b4.

Test build #83701 has finished for PR 19630 at commit 75d31b4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83701/
Test FAILed.

Test build #83709 has started for PR 19630 at commit 94cded6.

Test build #83709 has finished for PR 19630 at commit 94cded6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83709/
Test FAILed.

Merged build finished. Test FAILed.

generally LGTM, but need more discussion on the naming.

Thanks @cloud-fan. I will do some clean up / add more tests today.
I might need some help to get the build to pass because I added enum34 as a dependency. cc @HyukjinKwon I suppose we need to update the python environment in Jenkins?

Test build #83794 has started for PR 19630 at commit 464a4e8.

Test build #83794 has finished for PR 19630 at commit 464a4e8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83794/
Test FAILed.

Test build #83853 has started for PR 19630 at commit a741a05.

Test build #83853 has finished for PR 19630 at commit a741a05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83853/
Test PASSed.

Test build #83862 has started for PR 19630 at commit bdd3e7a.

Test build #83862 has finished for PR 19630 at commit bdd3e7a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83862/
Test FAILed.

Merged build finished. Test FAILed.

Test build #83898 has started for PR 19630 at commit 658f41b.

Test build #83899 has started for PR 19630 at commit 3af57d5.

Test build #83898 has finished for PR 19630 at commit 658f41b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83898/
Test PASSed.

Test build #83906 has started for PR 19630 at commit 565ba23.

Test build #83899 has finished for PR 19630 at commit 3af57d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83899/
Test PASSed.

Test build #83906 has finished for PR 19630 at commit 565ba23.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83906/
Test PASSed.

Looks clean and pretty solid in general. Let me take another look to double check, probably, within this weekend and maybe I will leave it to @ueshin if I can take the look ahead.

Test build #83942 has started for PR 19630 at commit 1f2c47b.

LGTM

Test build #83942 has finished for PR 19630 at commit 1f2c47b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83942/
Test PASSed.

LGTM

Test build #83956 has started for PR 19630 at commit cf1d1ca.

As a side question, is there an easy way to run coverage reports with pyspark tests?

To be honest, I know a super hacky funny workaround I use sometimes but .. want to know if there is any easy way or cleaner way ..

which, for example, shows the coverage like this:

The difficulty is to enable it for workers too ... if anyone does not know the easy way let me share mine .. I am ashamed to share it actually ..

Test build #83956 has finished for PR 19630 at commit cf1d1ca.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83956/
Test FAILed.

retest this please

Test build #83959 has started for PR 19630 at commit cf1d1ca.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83961/
Test FAILed.

retest this please

Test build #83962 has started for PR 19630 at commit cf1d1ca.

Test build #83959 has finished for PR 19630 at commit cf1d1ca.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83959/
Test FAILed.

retest this please

Test build #83965 has started for PR 19630 at commit cf1d1ca.

Test build #83965 has finished for PR 19630 at commit cf1d1ca.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83965/
Test FAILed.

Test build #83962 has finished for PR 19630 at commit cf1d1ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83962/
Test PASSed.

@HyukjinKwon Thanks for the reply on coverage. It'd be great to have an easy way to run coverage :)

Everyone, I don't have more changes to the PR. I think all comments are addressed at this point. Please let me know if I missed anything or there are more comments. Thank you!

thanks, merging to master, cheers!

OK, mine was, with this diff:
--- a/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala
@@ -38,7 +38,7 @@ private[spark] class PythonWorkerFactory(pythonExec: String, envVars: Map[String
   // (pyspark/daemon.py) and tell it to fork new workers for our tasks. This daemon currently
   // only works on UNIX-based systems now because it uses signals for child management, so we can
   // also fall back to launching workers (pyspark/worker.py) directly.
-  val useDaemon = !System.getProperty("os.name").startsWith("Windows")
+  val useDaemon = false

   var daemon: Process = null
   val daemonHost = InetAddress.getByAddress(Array(127, 0, 0, 1))

pip install coverage
# Build Spark (http://spark.apache.org/docs/latest/building-spark.html)
rm python/lib/pyspark.zip
rm -fr .coverage
rm -fr coverage_html
echo "
#!/usr/bin/env bash
coverage run -p \$@
" > coverage_python
chmod 755 coverage_python
PATH=`pwd`:$PATH PYSPARK_PYTHON=coverage_python SPARK_TESTING=1 bin/pyspark pyspark.sql.tests VectorizedUDFTests
coverage combine
coverage html -d coverage_html -i
open coverage_html
# Open up index.html in your browser.

Actually, R has a flag for useDaemon:
https://github.com/apache/spark/blob/478fbc866fbfdb4439788583281863ecea14e8af/core/src/main/scala/org/apache/spark/api/r/RRunner.scala#L362
It'd be great if we have this flag too. It makes easier to test Windows specific issue too ..

Thanks @HyukjinKwon!

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan icexelloss viirya


text:
[SPARK-28332][SQL] SQLMetric wrong initValue

What changes were proposed in this pull request?
Currently SQLMetrics.createSizeMetric create a SQLMetric with initValue set to -1.
If there is a ShuffleMapStage with lots of Tasks which read 0 bytes data, these tasks will send the metric(the metric value still be the initValue with -1) to Driver,  then Driver do metric merge for this Stage in DAGScheduler.updateAccumulators, this will cause the merged metric value of this Stage set to be a negative value.
This is incorrect， we should set the initValue to 0 .
Another same case in SQLMetrics.createTimingMetric/createNanoTimingMetric
How was this patch tested?
No more tests

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12588/
Test PASSed.

Test build #107453 has started for PR 25095 at commit 5cc78fb.

Test build #107453 has finished for PR 25095 at commit 5cc78fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107453/
Test PASSed.

Hm, looks like a good point. @cloud-fan do you know why this is initialized to -1?

Probably, this work around is related to the -1 value. But, this change looks reasonable to me, too.

looking at SPARK-11013, we do have a reason to use -1 as initial value, and we will filter out -1s when doing metrics merging. @windpiger can you give an example about when this can go wrong?

I noticed that we filter out -1s in SQLMetrics.stringValue. Is there any other place Spark do metrics merging?

Probably, this work around is related to the -1 value. But, this change looks reasonable to me, too.

This workaround change cause a incorrect dataSize metric, If we use dataSize to do some other logical
such as used to get the statistic of QueryStageExec in AdaptiveExecutionMode, this will cause the statistic of QueryStageExec set to be a  negative value, and then will miss the re-optimize (SMJ->BHJ) because  the plan.stats.sizeInBytes < 0 in the this canBroadcast method

shall we change SQLMetric#merge to ignore -1?

shall we change SQLMetric#merge to ignore -1?

related affected method which use _value initialized to -1
add method
merge method
+=
Maybe we should re-think this original case SPARK-11013, and avoid a -1 initvalue to be set.

I think there are 2 kinds of sql metrics:

executor side just set a value, and driver side merges the values by calculating some statistics
executor side accumulates values (by calling add), and driver side merges the values by sum

Only the first case need to set the initial value as -1. Maybe we can refactor SQLMetrics a little to enforce it.
BTW it's hard to avoid invalid accumulators. Currently Spark just sends the physical plan tree to the executor side to do execution, and accumulators are carried in the physical plan node. Ideally we should have something that is separated from the physical plan, so that we can precisely control which parts need to go to executor side, and avoid sending unrelated accumulators. This is a lot of work.

I think there are 2 kinds of sql metrics:

executor side just set a value, and driver side merges the values by calculating some statistics
executor side accumulates values (by calling add), and driver side merges the values by sum

Only the first case need to set the initial value as -1. Maybe we can refactor SQLMetrics a little to enforce it.
BTW it's hard to avoid invalid accumulators. Currently Spark just sends the physical plan tree to the executor side to do execution, and accumulators are carried in the physical plan node. Ideally we should have something that is separated from the physical plan, so that we can precisely control which parts need to go to executor side, and avoid sending unrelated accumulators. This is a lot of work.

Thanks, I will be consider how to refactor SQLMetrics

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15667/
Test PASSed.

@windpiger what about updating this to ignore negative updates instead?

Ping @windpiger ?

Fixed in #26899

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan maropu srowen windpiger


text:
[SPARK-31045][SQL] Add config for AQE logging level

What changes were proposed in this pull request?
This PR adds an internal config for changing the logging level of adaptive execution query plan evolvement.
Why are the changes needed?
To make AQE debugging easier.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added UT.

cc @gatorsmile @cloud-fan

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24077/
Test PASSed.

Test build #119340 has finished for PR 27798 at commit e93825d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119340/
Test PASSed.

Test build #119362 has started for PR 27798 at commit 68dd879.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24099/
Test PASSed.

Test build #119362 has finished for PR 27798 at commit 68dd879.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119362/
Test FAILed.

Test build #119403 has started for PR 27798 at commit 7e999fa.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24140/
Test PASSed.

Test build #119403 has finished for PR 27798 at commit 7e999fa.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119403/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24151/
Test PASSed.

Test build #119416 has started for PR 27798 at commit 3151b01.

Test build #119416 has finished for PR 27798 at commit 3151b01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119416/
Test PASSed.

thanks, merging to master/3.0!

Could you make a quick hot fix?

I just fixed it via #27826 coincidentally - please take a look.

authors: AmplabJenkins HeartSaVioR SparkQA cloud-fan dongjoon-hyun maryannxue


text:
[SPARK-31365][SQL][FOLLOWUP] Refine config document for nested predicate pushdown

What changes were proposed in this pull request?

This is a followup to address the https://github.com/apache/spark/pull/28366#discussion_r420611872 by refining the SQL config document.
Why are the changes needed?

Make developers less confusing.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Only doc change.

cc @HyukjinKwon

Test build #122380 has started for PR 28468 at commit 196d4c9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27045/
Test PASSed.

Thanks @viirya.

Merged to master/3.0.

Thanks @HyukjinKwon @maropu

Test build #122380 has finished for PR 28468 at commit 196d4c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122380/
Test PASSed.

authors: AmplabJenkins HyukjinKwon SparkQA maropu viirya


text:
[SPARK-33789][SQL][TESTS] Refactor unified V1 and V2 datasource tests

What changes were proposed in this pull request?

Move common utility functions such as test(), withNsTable() and checkPartitions() to DDLCommandTestUtils.
Place common settings such as version, catalog, defaultUsing, sparkConf to CommandSuiteBase.

Why are the changes needed?
To improve code maintenance of the unified tests.
Does this PR introduce any user-facing change?
No
How was this patch tested?
By running the affected test suites:
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowPartitionsSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowTablesSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *AlterTableAddPartitionSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *AlterTableDropPartitionSuite"

Test build #132826 has started for PR 30779 at commit 89e3898.

Test build #132826 has finished for PR 30779 at commit 89e3898.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132826/

Test build #132828 has started for PR 30779 at commit 5836e54.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37428/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37430/

Test build #132834 has started for PR 30779 at commit 04fd90b.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37430/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37430/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37436/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37436/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37436/

Test build #132828 has finished for PR 30779 at commit 5836e54.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait AlterTableDropPartitionSuiteBase extends QueryTest with DDLCommandTestUtils

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132834/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132828/

@MaxGekk can you fix the conflicts?

Test build #132860 has started for PR 30779 at commit 35dc058.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37462/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37462/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37462/

Test build #132860 has finished for PR 30779 at commit 35dc058.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SubqueryExec(name: String, child: SparkPlan, maxNumRows: Option[Int] = None)

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132860/

thanks, merging to master!

authors: AmplabJenkins MaxGekk SparkQA cloud-fan


text:
[SPARK-5007] [CORE] Try random port when startServiceOnPort to reduce the chance of port collision

When multiple Spark programs are submitted at the same node (called springboard machine). The ports (default 4040) of these SparkUIs are from 4040 to 4056. Then the Spark programs submitted later would fail because of SparkUI port collision.
The chance of port collision could be reduced by setting spark.ui.port or spark.port.maxRetries.
However, I think it's better to try random port when startServiceOnPort to reduce the chance of port collision.

@YanTangZhai I don't think we want to retry on a random port necessarily. If your SparkUI fails to bind on 4040, it is easy to find it on 4041 and configure your firewall to open say the port range 4040 - 4050 to allow 10 concurrent SparkUIs. With this change it is impossible to predict the appropriate range. I would recommend that we close this issue instead.

@andrewor14 @rxin Oh, I see. Thank you very much.


text:
[SPARK-41311][SQL][TESTS] Rewrite test RENAME_SRC_PATH_NOT_FOUND to trigger the error from user space

What changes were proposed in this pull request?
This updates the test for error class RENAME_SRC_PATH_NOT_FOUND in QueryExecutionErrorsSuite to trigger the error with user-facing APIs.
Why are the changes needed?
To test the code more directly.
Does this PR introduce any user-facing change?
No
How was this patch tested?
The only change is the modification of the RENAME_SRC_PATH_NOT_FOUND test.
License
This contribution is my original work. I license the work to the project under the project’s open source license.

+1, LGTM. Merging to master.
Thank you, @ibuder.

authors: MaxGekk ibuder


text:
[SPARK-43248][SQL] Unnecessary serialize/deserialize of Path on parallel gather partition stats

What changes were proposed in this pull request?

Remove unnecessary serialize/deserialize of Path on parallel gather partition stats.
Why are the changes needed?

Simplify code, since Path is serializable in Hadoop3.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Pass GA.

cc @sunchao

kindly ping @sunchao

Oops almost forgot. Merged to master, thanks!


text:
[SPARK-44969][SQL] Reuse ArrayInsert in ArrayAppend

What changes were proposed in this pull request?
In the PR, I propose to replace the current implementation of the ArrayAppend expression by a runtime replaceable one to ArrayInsert with posExpr = -1.
Why are the changes needed?
To improve code maintenance.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
By running the modified test suite:
$ build/sbt "test:testOnly *CollectionExpressionsSuite"

Was this patch authored or co-authored using generative AI tooling?
No.

late LGTM

authors: MaxGekk cloud-fan


text:
[SPARK-44662] Perf improvement in BroadcastHashJoin queries with stream side join key on non partition columns

What changes were proposed in this pull request?
On the lines of DPP which helps DataSourceV2 relations when the joining key is a partition column, the same concept can be extended over to the case where joining key is not a partition column.
In this PR, the keys available in the BroadcastHashJoinExec are pushed down to the DataSourceV2 scans in form of a SortedSet structure.
For non partition columns, the DataSources like iceberg have max/min stats on columns available at manifest level, and for formats like parquet , they have max/min stats at various data structure levels. The passed SortedSet can be used to prune using ranges at both driver level ( manifests files) as well as executor level ( while actually going through chunks , row groups etc at parquet level)
If the data is stored as Columnar Batch format , then it would not be possible to filter out individual row at DataSource level, even though we have keys.
But at the scan level, ( ColumnToRowExec) it is still possible to filter out as many rows as possible , if the query involves nested joins. Thus reducing the number of rows to join at the higher join levels.
Attaching link to a presentation which outlines the idea:
Broadcast Keys pushdown
SPIP : SPIP-44662
Why are the changes needed?
There is scope of improvement in the performance of Inner and Left Semi join queries when using BroadcastHashJoin
Does this PR introduce any user-facing change?
No
How was this patch tested?
Ran TPCDS suite using iceberg as DataSource.
Converted many of the existing Spark Query tests to also run using iceberg as data source.
Will be adding more unit tests.

Will be closing this PR and creating a new one as I have renamed the branch on which this PR has been created


text:
[SPARK-45523][Python] Refactor the null-checking to have shortcuts

What changes were proposed in this pull request?
This is a follow-up of apache/spark#43356.
Refactor the null-checking to have shortcuts.
Why are the changes needed?
The null-check can have shortcuts for some cases.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
The existing tests.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @dtenedor

The failed tests seem not related to this PR. Let me merge this now.

Thanks. merging to master.


text:
[SPARK-45875][CORE] Remove MissingStageTableRowData from core module

What changes were proposed in this pull request?
SPARK-15591(https://github.com/apache/spark/pull/13708) introduced the MissingStageTableRowData, but it is no longer used after SPARK-20648(https://github.com/apache/spark/pull/19698), so this PR removes it.
Why are the changes needed?
Clean up unused code.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions
Was this patch authored or co-authored using generative AI tooling?
No

Thanks @dongjoon-hyun ~