text:
NIFI-3739: Added ConsumeKafkaRecord_0_10 and PublishKafkaRecord_0_10 …

…processors
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@joewitt agreed on all fronts. Will push a new commit.

can we update this message "ConsumeKafkaRecord_0_10[id=b28c1bea-015b-1000-db37-08c3c0beaeee] Failed to parse message from Kafka using the configured Record Reader. Will route message as its own FlowFile to the 'parse.failure' relationship: org.apache.nifi.schema.access.SchemaNotFoundException: Schema Encoding appears to be of an incompatible version. The latest known Protocol is Version 1 but the data was encoded with version 80
"
to indicate that either the encoded version is a given value or the data is not encoded in this form at all.

one more observation and cannot recall if I mentioned this yet.  But here is some original JSON we'd get from provenance.  This is before turning into avro and then turning back into json.
{code}
{
"eventId":"3ddbed75-c436-4112-a2bd-bbdd05320b23",
"eventOrdinal":13720400,
"eventType":"CREATE",
"timestampMillis":1493439571667,
"timestamp":"2017-04-29T04:19:31.667Z",
"durationMillis":-1,
"lineageStart":1493439571667,
"componentId":"b259f36c-015b-1000-3ca6-c664e156cdf7",
"componentType":"GenerateFlowFile",
"componentName":"GenerateFlowFile",
"entityId":"2ab3c2d8-cb45-44ae-bd45-72080aff3e78",
"entityType":"org.apache.nifi.flowfile.FlowFile",
"entitySize":0,
"updatedAttributes":
{
"path":"./",
"uuid":"2ab3c2d8-cb45-44ae-bd45-72080aff3e78",
"filename":"464576540850588"
},
"previousAttributes":{},
"actorHostname":"10.0.0.15",
"contentURI":"http://10.0.0.15:8080/nifi-api/provenance-events/13720400/content/output",
"previousContentURI":"http://10.0.0.15:8080/nifi-api/provenance-events/13720400/content/input",
"parentIds":[],
"childIds":[],
"platform":"nifi",
"application":"NiFi Flow"
}
{code}
Here is the after avro and then back to json view
{code}
{
"eventId" : "edf16cc9-b50d-4cbb-9356-58eb4cf8fd3e",
"eventOrdinal" : 13585491,
"eventType" : "CREATE",
"timestampMillis" : "2017-04-29 04:18:50",
"durationMillis" : -1,
"lineageStart" : "2017-04-29 04:18:50",
"details" : null,
"componentId" : "b259f36c-015b-1000-3ca6-c664e156cdf7",
"componentType" : "GenerateFlowFile",
"componentName" : "GenerateFlowFile",
"entityId" : "ef12f52c-4ae4-46be-bc45-698200f08e54",
"entityType" : "org.apache.nifi.flowfile.FlowFile",
"entitySize" : 0,
"previousEntitySize" : null,
"updatedAttributes" : {
"path" : "./",
"filename" : "464535310815392",
"uuid" : "ef12f52c-4ae4-46be-bc45-698200f08e54"
},
"previousAttributes" : { },
"actorHostname" : "10.0.0.15",
"contentURI" : "http://10.0.0.15:8080/nifi-api/provenance-events/13585491/content/output",
"previousContentURI" : "http://10.0.0.15:8080/nifi-api/provenance-events/13585491/content/input",
"parentIds" : [ ],
"childIds" : [ ],
"platform" : "nifi",
"application" : "NiFi Flow",
"transitUri" : null
}
{code}
Notice timestamp millis.  Seems like we should let json and csv writing use the timestamp like it would be in an avro 'timestamp-millis' sort of way rather than a formatted date/time string.  We'd want to do this consistently on reading and writing.  The current default changes its representation i think and makes it a bit tougher to do math with.

@joewitt agreed. Pushed a new commit.

@markap14 can you please squash at this point and rebase with master (now conflicts).  Thanks

ok i think we're there now.  Will merge shortly.  +1

@joewitt excellent! Thanks for such a diligent review & testing - you're certainly keeping me on my toes! ;-)

repo: apache/nifi


text:
NIFI-8326: Send records as individual messages in Kafka RecordSinks

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
The KafkaRecordSink implementations used to send all records in the RecordSet as one Kafka message. Because of Kafka message size constraints, this can make the sinks unusable for things like Provenance (when all provenance events are in a single RecordSet). Instead this PR sends each record as a separate Kafka message.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Hey @mattyb149 - is this ready for final review?
(also, while playing more with this, I realized some authN methods are missing https://issues.apache.org/jira/browse/NIFI-8399)

It's not ready yet, I need to change the producer.send() to not call get() directly after, or else each message will be ack'ed one at a time, which impacts performance.

@pvillard31 @markap14 Should be ready now, thanks!

Hi there - is it something we can get through the finish line? @mattyb149 did you get a chance to look at Mark's comments? This would definitely help a lot to fix this issue.

@pvillard31 This one is hopefully ready for final review, and I can look at the Kerberos user/pass after this is merged

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
Support to save crash dump in the flash

Currently there is not a simple way to save crash dump in the flash.
This information (crash dump) could be lost on systems running in the field with watchdog enabled.
Related issue: https://github.com/apache/nuttx/issues/3417
Ref: https://cwiki.apache.org/confluence/display/NUTTX/Debug+Output+(SYSLOG)+Issues

You can try:
https://github.com/apache/nuttx/blob/master/binfmt/binfmt_coredump.c#L46
with:
https://github.com/apache/nuttx/blob/master/libs/libc/stream/lib_mtdoutstream.c
to save the core dump to flash and follow https://github.com/apache/nuttx/pull/4871
or with:
https://github.com/apache/nuttx/blob/master/sched/misc/assert.c#L642-L646
to sieral and follow the https://github.com/apache/nuttx/pull/9369

Thank you very much @xiaoxiang781216 I'll try it and write a Guide in our documentation

BTW, you can even get the local variables with the default panic log by minidump server: https://github.com/apache/nuttx/pull/5836

source: gharchive/issue

repo: apache/nuttx


text:
Documentation: turn warnings into errors

Summary
Documentation: turn warnings into errors
depends on https://github.com/apache/nuttx/pull/11277
fixes https://github.com/apache/nuttx/issues/11084
Impact
Testing

For some reason warnings_filter Sphinx extension works on my local machine but doesn't want to work with our CI.

Ok, I found the problem. It seems that this extension doesn't work with our parallel build.


text:
gdbstub:fix get thread rsp command crash

gdb/thread.c:1309: internal-error: void switch_to_thread(thread_info*): Assertion `thr != NULL' failed.
Summary
fix gdbustub send thread command crash
Impact
gdbstub
Testing
stm32f429zi with gdbstub

look like ci build break

@acassis please take a look.

@xiaoxiang781216 I think the issue is because on esp32s3 the _start is called __esp32s3_start ?
@tmedicci may I rename __esp32s3_start  to _start?


text:
ORC-885: Update bench README.md and allow user env shell

What changes were proposed in this pull request?
This PR aims to update READM.md in bench submodule and allow user environment shell in fetch-data.sh.
Why are the changes needed?

ORC-821 added mvnw in the parent directory.
Since ORC-354, -Pbenchmark is recommended.
In some OS environments, /bin/bash is too old to run the script.

How was this patch tested?
Pass the CIs.

Could you review this, @pgaref and @williamhyun ?

Thank you so much, @omalley !

Merged to main/1.7.

repo: apache/orc

license_source: bigquery


text:
Multistage engine and multivalue column semantics incompatibilities

Multistage engine and multivalue column semantics incompatibilities
In Pinot filtering on a multivalue column with = operator actually a contains operator, aggregation to a  multivalue column is actually an unnest operation.
It works well in 1.0 engine even with filtering the result (multivalue column) in the HAVING clause.
However, in 2.0 engine, this irregular behavior is not compatible with relation algebra semantics. That's why one should use ArrayToMv function, which tries to break the border between "normal" field semantics and multivalue field semantics.
In most cases it works well, but there are some cases when it doesn't work as expected.
Example:
select id, mvc from (select id, arraytomv(mv) as mvc from mytable group by id, mvc) embd where mvc = 'a'

(assuming that mvc is a multivalue column)
If there is a row in mytable with mvc containing a and b, the result will contain two rows with the same id and mvc = a and b respectively (even though the where clause is applied).
The main cause of this behavior is that relation algebra transformation pushes down the where clause to the inner query like this: select id, mvc from (select id, arraytomv(mv) as mvc from mytable where arraytomv(mv) = 'a' group by id, mvc)
In normal field semantics, this query will return only one row with mvc = a. However, in multivalue field semantics, it will return two rows with mvc = a and b respectively.
So a complete rework of multivalue operators is needed. Multivalue is actually a set, not a value, so for example an UNNEST(...) operator is suggested (Calcite supports it), and for filter mv = SOME(...) is suggested (Calcie supports it too).
For fast fix of this issue, it should be prevented to push down filter on the multivalue-column group by query to the inner query.

could you attach the explain plan result for this query.
This is actually a problem b/c the mv column is not exactly semantically "relational algebra" it is a short-handed representation of saying "it should behave as an array (with some extra op) when xxx and should behave as a set (with some extra op) when yyy"
so there's really no convention or compatibility fix.
for this specific query
select id, mvc from (
  select id, arraytomv(mv) as mvc from mytable group by id, mvc) embd
where mvc = 'a'

the where clause is not because it is not suppose to be pushed into the inner query, but the question on whether we should applied it as a filter before/after the group-by.
agree we definitely need some redesign to make sure this behavior is clear. even if it means some of the existing syntax wont make sense. for example: in this case, UNNEST must be explicitly specify and mvc should be treated as normal ARRAY type. then the confusion should go away

authors: gyorfimi walterddr

repo: apache/pinot


text:
Possible storage optimization for MV forward index

We recently observed a huge increase on table size upon adding a MV column. Table size went from 2TB to 12TB.
Column was dictionary encoded (back then raw MV column support wasn't added or was in the middle of being added). Column is needed in WHERE clause so it is dictionary encoded along with inverted index. Both forward and inverted index have resulted in size increase. Some stats from a sample segment:
mvCol.cardinality = 131483
mvCol.totalDocs = 287714
mvCol.dictionary.size = 525940
mvCol.forward_index.size = 800860697
mvCol.inverted_index.size = 553252156
mvCol.maxNumberOfMultiValues = 16649
mvCol.totalNumberOfEntries = 336962215
The numDocs in segment is fairly low 287k. Given that totalNumberOfEntries is around 336million, cardinality is super low around 131k.
The fwd index size of 800MB is majorly coming from rawDataSize section computed as :
rawDataSize = ((long) totalNumValues * numBitsPerValue + 7) / 8;
numBitsPerValue is 18 given that cardinality is 131k
So dictId for each of the 336million values is encoded with 18 bits in the rawData section.

Haven't thought much about a solution yet but given that there is so much duplicate data (and I have checked that there are repetitive runs) in the above sample, one potential way could be to use RLE along with bit packing where run length itself is bit-packed and/or a hybrid combination of RLE and bit-packing that switches between the two depending on data (something like what Parquet does).

Another solution would be to consider variable length bit encoding where instead of current way of using fixed number of bits (max number of bits essentially) for each dictId, we use based on the value. But in this case, a fixed 5 bits per dictId are going to be needed to indicate how many bits are used to encode the dictId

Another way could be to have sort of dictionary on top of dictionary -- this can work if entire array is duplicated. So for example, if dictId array [1, 2, 3, 4, 5, 10] is appearing multiple times across rows/docs, we create a dictId for this dictId array and use the former dictId to encode the array values instead of using array of dictIds.

We can also have a way of encoding the fwd index of the column using general purpose compression schemes (like LZ4) and still keep a dictionary structure. Currently, if we enable LZ4 / SNAPPY / ZSTD codecs on a column, it must be marked as noDictionary

Even though the number of docs is low, the total number of entries is quite high - 336 million. There is not much we can do to reduce the size of forward index without sacrificing the access speed. One option would be to eliminate the forward index if it's not needed post-filtering and just keep the inverted index similar to what we do for some text index columns.
One thing that stands out is that the Inverted index size (500 MB) seems quite high for 336 million. Can you verify if we do the runCompress on the bitmap. cc @richardstartin added something here recently.
I like the 4th idea, will be great to apply compression on any column with or without dictionary encoding

Daniel Lemire has done a lot of work on Integer compression techniques. In fact, we did use PFOR and PFORDelta in the early days of Pinot and removed them because of latency requirements of site facing. It might be worth revisiting given the broader use cases for Pinot.
https://github.com/lemire/FastPFor - there are a lot of references

+1 for the using LZ4 compression for the dictionary encoded fwd index.

BTW - compressing fwd index for dictionary encoded columns can/should be done for SV columns also.

Working on adding support for compressing raw index with dictionary

Working on adding support for compressing raw index with dictionary

@siddharthteotia For clarification, are you suggesting compressing bit-compressed forward index when a column is dictionary encoded?

@siddharthteotia how fast have you got with this? Do you have the frequency distribution of the column? I have some ideas for variable length dictionary encoding which will pay off if the frequency distribution has skew and we sort the dictionary by descending frequency.

In addition to the above, given that there are runs, LZ4 will do a better job if the values are byte aligned - runs of 18 bit dictionary codes looks like a random sequence of bytes to LZ4 because the 2 bit offsets scramble the bytes (I have a demo of this effect with base64 encoding). It’s worth experimenting with rounding up to 24 prior to LZ4 compression, iff LZ4 compression is enabled.

Hey @Jackie-Jiang @walterddr
As discussed over slack, we got the compression results for the actual table that ran into this forward index size bloat issue. I've updated the document in this section. The TL;DR is that for the actual table the compression savings are very minimal. I updated the recommendations to indicate that for now it does not make sense to try to solve this by compressing the data using any of the approaches.
@siddharthteotia and I would like to keep this issue open to explore further ideas in the future or perhaps revisit compression with dictionary in case we find users who have sufficient repeatability in their data to benefit from compression.
Also, as discussed over our call, there may be some use of implementing Approach 2 from the proposed approaches for the sake of speeding up the query rather than saving on storage costs (i.e. have a dictionary and store the forward index in raw format -> which can help avoid an additional dictionary lookup). I had started some work on Approach 2 and have an initial PR before we ran these compression experiments. My PR stores the data as raw + compressed in the forward index but creates a dictionary. I need to spend some time on seeing how best to divide up the PRs before submitting this to OSS. Just wanted to give a heads up.
cc @siddharthteotia

Hey @Jackie-Jiang @walterddr
As discussed over slack, we got the compression results for the actual table that ran into this forward index size bloat issue. I've updated the document in this section. The TL;DR is that for the actual table the compression savings are very minimal. I updated the recommendations to indicate that for now it does not make sense to try to solve this by compressing the data using any of the approaches.
@siddharthteotia and I would like to keep this issue open to explore further ideas in the future or perhaps revisit compression with dictionary in case we find users who have sufficient repeatability in their data to benefit from compression.
Also, as discussed over our call, there may be some use of implementing Approach 2 from the proposed approaches for the sake of speeding up the query rather than saving on storage costs (i.e. have a dictionary and store the forward index in raw format -> which can help avoid an additional dictionary lookup). I had started some work on Approach 2 and have an initial PR before we ran these compression experiments. My PR stores the data as raw + compressed in the forward index but creates a dictionary (passthrough compression can be enabled to avoid decompression overhead). I need to spend some time on seeing how best to divide up the PRs before submitting this to OSS. Just wanted to give a heads up.
cc @siddharthteotia

I can’t access the document but was byte alignment (rounding the dictionary’s bits up to the next multiple of 8, so padding each dictionarized value with leading zeros) prior to LZ4 compression attempted? If the dictionary codes aren’t byte aligned, byte-oriented compression schemes won’t work well. I explained this on a call with @siddharthteotia several months ago.

I can’t access the document but was byte alignment (rounding the dictionary’s bits up to the next multiple of 8, so padding each dictionarized value with leading zeros) prior to LZ4 compression attempted? If the dictionary codes aren’t byte aligned, byte-oriented compression schemes won’t work well. I explained this on a call with @siddharthteotia several months ago.

@richardstartin can you provide a gmail ID I can share the document with so that you can access the results? We didn't try the bit alignment to the next multiple of 8. Today morning I ran a few experiments to try that out. We haven't implemented a full-fledged index creator for the approach to compress the bit packed dictionary IDs in the forward index, but use an approximate approach where we extract the dictionary encoded forward index from the segment file and apply lz4 / zstd compression for the whole extracted file using command line compression tools (which is why it's approximate).
When I tried it this morning, the size of the forward index increased after rounding up from 18 to 24 bits as expected. LZ4 compression couldn't bring the size down to the original bit encoded format using 18 bits. ZSTD performed better and did bring down the size a bit (compression ratio compared to the original dict encoded forward index using 18 bits: 1.15).
I've updated these findings in the document as well.

authors: Jackie-Jiang ashishkf kishoreg richardstartin siddharthteotia somandal


text:
wip: add support for mv cols with lucene index

wip PR for adding support for MV cols with lucene index
addresses: https://github.com/apache/pinot/issues/10975

Codecov Report

Merging #11000 (0205485) into master (3c92b32) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##           master   #11000     +/-   ##
=========================================
  Coverage    0.11%    0.11%
=========================================
  Files        2189     2137     -52
  Lines      117997   115502   -2495
  Branches    17852    17566    -286
=========================================
  Hits          137      137
+ Misses     117840   115345   -2495
  Partials       20       20

Flag
Coverage Δ

integration1temurin11
?

integration1temurin17
?

integration1temurin20
?

integration2temurin11
?

integration2temurin17
?

integration2temurin20
?

unittests1temurin11
?

unittests1temurin17
?

unittests1temurin20
?

unittests2temurin11
?

unittests2temurin17
?

unittests2temurin20
0.11% <0.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ime/impl/invertedindex/NativeMutableTextIndex.java
0.00% <0.00%> (ø)

...me/impl/invertedindex/RealtimeLuceneTextIndex.java
0.00% <0.00%> (ø)

...ot/segment/spi/index/mutable/MutableTextIndex.java
0.00% <0.00%> (ø)

... and 131 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter itschrispeck


text:
Fix the potential access to upsert metadata manager after it is closed

Even though very unlikely, there is still a code path which can potentially access the upsert metadata after the manager is closed:

Thread 1 finishes the _stopped flag check but not invoking startOperation() yet
Thread 2 stops and closes the metadata manager
Thread 1 calls startOperation() and start accessing the metadata

In this PR, make startOperation() return false if the pending operations (reference count) already drops to 0, and also merge stopped flag check into startOperation() to simplify the logic. This is not super performance sensitive, so using very tight synchronized block should be fine

Codecov Report

Merging #11692 (c7a767d) into master (1748be4) will decrease coverage by 48.60%.
Report is 2 commits behind head on master.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #11692       +/-   ##
=============================================
- Coverage     63.02%   14.43%   -48.60%
+ Complexity     1120      201      -919
=============================================
  Files          2343     2343
  Lines        125672   125670        -2
  Branches      19314    19317        +3
=============================================
- Hits          79209    18142    -61067
- Misses        40808   105992    +65184
+ Partials       5655     1536     -4119

Flag
Coverage Δ

integration
?

integration1
?

integration2
?

java-11
?

java-17
?

java-20
14.43% <0.00%> (-48.46%)
:arrow_down:

temurin
14.43% <0.00%> (-48.60%)
:arrow_down:

unittests
14.43% <0.00%> (-48.59%)
:arrow_down:

unittests1
?

unittests2
14.43% <0.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

...cal/upsert/BasePartitionUpsertMetadataManager.java
0.00% <0.00%> (-39.62%)
:arrow_down:

... and 1510 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: Jackie-Jiang codecov-commenter


text:
Cleanup of the method hasAccess.

Labels

cleanup

Description
As per the discussion comment on the PR, This PR is just a followup change and making sure the endpointUrl should be passed in all the calls. I have looked at all the method calls, and we are passing endpointUrl.  Very minor cleanup changes.
cc:  @soumitra-st , @zhtaoxiang

Codecov Report
Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (a001e05).
Report is 772 commits behind head on master.

Files
Patch %
Lines

...t/controller/api/resources/PinotQueryResource.java
0.00%
1 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (a001e05). Click for more details.
HEAD has 53 uploads less than BASE

Flag
BASE (59551e4)
HEAD (a001e05)

integration
7
1

integration2
3
1

temurin
12
1

java-21
7
1

skip-bytebuffers-true
3
1

skip-bytebuffers-false
7
0

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13633       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2480       +44
  Lines        133233   137038     +3805
  Branches      20636    21306      +670
=============================================
- Hits          82274        0    -82274
- Misses        44911   137038    +92127
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
?

skip-bytebuffers-true
0.00% <0.00%> (-27.73%)
:arrow_down:

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: abhioncbr codecov-commenter


text:
Update created-at property in createTableDirectWithWriteDelegation + enable ErrorProne ModifiedButNotUsed

Description
Update created-at property in createTableDirectWithWriteDelegation + enable ErrorProne ModifiedButNotUsed
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Test Configuration:

Firmware version:
Hardware:
Toolchain:
SDK:

Checklist:
Please delete options that are not relevant.

[x] I have performed a self-review of my code

Rebased on main branch :)

repo: apache/polaris


text:
OpenID: introducing support for OpenID configuration

BREAKING CHANGE: provider configuration changed from auth.authentication.provider to auth.authentication.jwt.enabled
Motivation
Allow the configuration of the OpenID authentication provider along side with JWT.
Modifications
Provider configuration auth.authentication.provider (only 'jwt' value was accepted) was decommissioned in favor of auth.authentication.jwt.enabled:
From:
auth:
  authentication:
    enabled: false
    provider: "jwt"

To:
# Enable or disable broker authentication and authorization.
auth:
  authentication:
    enabled: false
    jwt:
      enabled: false
      # Enable JWT authentication
      # If the token is generated by a secret key, set the usingSecretKey as true.
      # If the token is generated by a private key, set the usingSecretKey as false.
      usingSecretKey: false
    openid:
      enabled: false
#     # https://pulsar.apache.org/docs/next/security-openid-connect/#enable-openid-connect-authentication-in-the-broker-and-proxy
      openIDAllowedTokenIssuers: []
      openIDAllowedAudiences: []
      openIDTokenIssuerTrustCertsFilePath:
      openIDRoleClaim:
      openIDAcceptedTimeLeewaySeconds: "0"
      openIDCacheSize: "5"
      openIDCacheRefreshAfterWriteSeconds: "64800"
      openIDCacheExpirationSeconds: "86400"
      openIDHttpConnectionTimeoutMillis: "10000"
      openIDHttpReadTimeoutMillis: "10000"
      openIDKeyIdCacheMissRefreshSeconds: "300"
      openIDRequireIssuersUseHttps: "true"
      openIDFallbackDiscoveryMode: "DISABLED"

Verifying this change

[ ] Make sure that the change passes the CI checks.

Very cool - is there any presumption of what happens when the JWT token expires? Or if the client id / secret are rotated?

Very cool - is there any presumption of what happens when the JWT token expires? Or if the client id / secret are rotated?

According with the doc, the client will handle the refresh:

When the token expires, the Pulsar Server challenges the client to re-authenticate with the Identity Provider and provide a new access token. If the client fails to re-authenticate, the Pulsar Server closes the connection.

I'm performing some tests. I let you know ;)

authors: brunodomenici frankjkelly

repo: apache/pulsar-helm-chart


text:
Pulsar release 2.8.2 documentation package is missing

Describe the bug
The 2.8.2 release isn't listed on this page:
https://pulsar.apache.org/en/versions/
The documentation is missing for 2.8.2:
https://pulsar.apache.org/docs/en/2.8.2/
(compare to 2.8.1: https://pulsar.apache.org/docs/en/2.8.1/ )
Pulsar 2.8.2 can be found on the download page and release notes. These are fine:
https://pulsar.apache.org/download
http://pulsar.apache.org/release-notes

@315157973 Could you please take a look? I believe that the step Update the site for minor releases is missing for 2.8.2 .

OK, I will take a look

Hi @315157973
For a minor release, we need to modify the following types of files:

doc set (includes more than 100+ .md files)
version-x.x.x-sidebars.json
release.json
versions.json

I've checked the 2.8.2 PR history, the versions.json has not been updated. I guess you need to update it.

authors: 315157973 Anonymitaet lhotari

repo: apache/pulsar


text:
Provide commands to delete or reformat cluster metadata

Problem
Sometimes a tool to delete or reformat cluster metadata would be very helpful for testing purposes.
This is related to #5825

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.

authors: sijie tisonkun


text:
[improve][doc] added instructions about how to run ci from fork

Motivation
As commented here, moving this instruction, how to run Pulsar CI from fork, to the contribution page in order to make it more accessible for new contributors.
Modifications
Added instructions about how to run Pulsar CI from forked github repositories in the contribution page.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[ ] no-need-doc
(Please explain why)

[x] doc
(Your PR contains doc changes)

[ ] doc-added
(Docs have been already added)

Thank you for checking this PR. I resolved the above comments.

@heesung-sn @Anonymitaet It looks like these instructions have been dropped and aren't on the new website. Would it be possible to get the content to the new website too?

@heesung-sn @Anonymitaet It looks like these instructions have been dropped and aren't on the new website. Would it be possible to get the content to the new website too?

Raised a PR to the new website repo, https://github.com/apache/pulsar-site/pull/111/
@Anonymitaet could you check this by any chance?

@heesung-sn @Anonymitaet It looks like these instructions have been dropped and aren't on the new website. Would it be possible to get the content to the new website too?

Raised a PR to the new website repo, apache/pulsar-site#111 @Anonymitaet could you check this by any chance?

sure, commented in your new PR.

authors: Anonymitaet heesung-sn lhotari


text:
[fix][test] Fix flaky test org.apache.pulsar.broker.service.persistent.PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest#setup

Motivation
In PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest, the setup method is executed every time the method is executed, resulting in an error: java.lang.RuntimeException: broker already started

Successfully executed after being modified to BeforeClass
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/lordcheng10/pulsar/pull/34

@JipeiWang @codelipenghui @AnonHxy @Jason918 @eolivelli PTAL,thanks!

/pulsarbot run-failure-checks

Codecov Report

Merging #18115 (51f1412) into master (6c65ca0) will decrease coverage by 11.16%.
The diff coverage is 25.00%.

@@              Coverage Diff              @@
##             master   #18115       +/-   ##
=============================================
- Coverage     34.91%   23.75%   -11.17%
+ Complexity     5707     3008     -2699
=============================================
  Files           607      367      -240
  Lines         53396    41182    -12214
  Branches       5712     4221     -1491
=============================================
- Hits          18644     9782     -8862
+ Misses        32119    29821     -2298
+ Partials       2633     1579     -1054

Flag
Coverage Δ

unittests
23.75% <25.00%> (-11.17%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../org/apache/pulsar/broker/admin/AdminResource.java
32.08% <0.00%> (-0.40%)
:arrow_down:

.../pulsar/broker/admin/impl/SchemasResourceBase.java
38.38% <0.00%> (-7.08%)
:arrow_down:

.../org/apache/pulsar/broker/admin/v2/Namespaces.java
6.19% <0.00%> (-1.84%)
:arrow_down:

...broker/delayed/InMemoryDelayedDeliveryTracker.java
0.00% <0.00%> (ø)

.../pulsar/broker/service/AbstractBaseDispatcher.java
35.89% <ø> (-9.97%)
:arrow_down:

...che/pulsar/broker/service/BacklogQuotaManager.java
9.09% <0.00%> (-0.40%)
:arrow_down:

.../service/SystemTopicBasedTopicPoliciesService.java
48.10% <0.00%> (-3.50%)
:arrow_down:

.../pulsar/broker/stats/BrokerOperabilityMetrics.java
98.21% <ø> (+5.56%)
:arrow_up:

...g/apache/pulsar/compaction/CompactedTopicImpl.java
11.42% <0.00%> (+0.71%)
:arrow_up:

...sistent/PersistentDispatcherMultipleConsumers.java
39.07% <28.57%> (-12.63%)
:arrow_down:

... and 333 more

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


text:
[feat][io] Upgarde ClickHouse driver to support loadbalance policy

This closes #12998.
Documentation
May or may not we add some docs on this feature. I'm not sure.

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

Codecov Report

Merging #18774 (cf7bb1a) into master (b36e012) will decrease coverage by 16.56%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master   #18774       +/-   ##
=============================================
- Coverage     47.74%   31.17%   -16.57%
+ Complexity    10623     6928     -3695
=============================================
  Files           703      737       +34
  Lines         68828    71340     +2512
  Branches       7381     7669      +288
=============================================
- Hits          32861    22242    -10619
- Misses        32307    46067    +13760
+ Partials       3660     3031      -629

Flag
Coverage Δ

unittests
31.17% <ø> (-16.57%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ava/org/apache/pulsar/broker/admin/v1/Brokers.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pulsar/broker/admin/v1/Clusters.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../org/apache/pulsar/broker/admin/v1/Properties.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pulsar/broker/admin/v2/ResourceGroups.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...sar/broker/stats/metrics/ManagedLedgerMetrics.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ar/common/naming/PartitionedManagedLedgerInfo.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ker/service/schema/exceptions/SchemaException.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...naming/RangeEquallyDivideBundleSplitAlgorithm.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...e/pulsar/broker/admin/impl/ResourceQuotasBase.java
0.00% <0.00%> (-96.43%)
:arrow_down:

...he/pulsar/broker/service/AnalyzeBacklogResult.java
0.00% <0.00%> (-92.31%)
:arrow_down:

... and 256 more

@zhicwu Thanks for your explanation! I think most of the usages of ClickHouse JDBC Driver are managed within the Pulsar IO framework, so the impact should also be managed.
Still, it doesn't tell why we should use a shaded dependency here - what's the benefit? The NAR file size is a downsize, BTW.

@nlu90 Agree. Test coverage is lack for this connector. I'm trying to add some.

Hi @zhicwu! Let me ask some questions in details:

Data format between client and server was changed from TabSeparated to RowBinary, meaning we no longer can pass null to a non-nullable column

What's the behavior if we pass null to a non-nullable column previously?

Type mapping and timezone handling were changed as well, meaning same query may give us different results but I guess it should be mainly about timestamp with timezone

Is there a CHANGELOG for the details how the mapping and handling change?

@zhicwu Thanks for your explanation! I think it's good to merge now.
Merging...
@codelipenghui @eolivelli @Technoboy- FYI release note required on the next release.

authors: codecov-commenter tisonkun


text:
Bump maven-enforcer-plugin to 3.2.1

Fixes #360

Codecov Report

Merging #361 (146fe7d) into master (67ea9d6) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #361   +/-   ##
=========================================
  Coverage     33.81%   33.81%
  Complexity      645      645
=========================================
  Files           220      220
  Lines         11448    11448
  Branches        279      279
=========================================
  Hits           3871     3871
  Misses         7326     7326
  Partials        251      251

Flag
Coverage Δ

java
60.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: aaron-ai codecov-commenter

repo: apache/rocketmq-clients


text:
When I Maven installed this project, an error was reported。an error is that response must be non-null but is marked as nullable

[ERROR] High: response must be non-null but is marked as nullable [org.apache.rocketmq.remoting.netty.NettyRemotingAbstract] At NettyRemotingAbstract.java:[lines 219-231] NP_PARAMETER_MUST_BE_NONNULL_BUT_MARKED_AS_NULLABLE
[INFO]
To see bug detail using the Spotbugs GUI, use the following command "mvn spotbugs:gui"
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache RocketMQ 5.1.1-SNAPSHOT ..................... SUCCESS [  8.889 s]
[INFO] rocketmq-common 5.1.1-SNAPSHOT ..................... SUCCESS [01:01 min]
[INFO] rocketmq-remoting 5.1.1-SNAPSHOT ................... FAILURE [ 28.522 s]
[INFO] rocketmq-client 5.1.1-SNAPSHOT ..................... SKIPPED
[INFO] rocketmq-store 5.1.1-SNAPSHOT ...................... SKIPPED
[INFO] rocketmq-tiered-store 5.1.1-SNAPSHOT ............... SKIPPED
[INFO] rocketmq-srvutil 5.1.1-SNAPSHOT .................... SKIPPED
[INFO] rocketmq-filter 5.1.1-SNAPSHOT ..................... SKIPPED
[INFO] rocketmq-acl 5.1.1-SNAPSHOT ........................ SKIPPED
[INFO] rocketmq-broker 5.1.1-SNAPSHOT ..................... SKIPPED
[INFO] rocketmq-tools 5.1.1-SNAPSHOT ...................... SKIPPED
[INFO] rocketmq-controller 5.1.1-SNAPSHOT ................. SKIPPED
[INFO] rocketmq-namesrv 5.1.1-SNAPSHOT .................... SKIPPED
[INFO] rocketmq-proxy 5.1.1-SNAPSHOT ...................... SKIPPED
[INFO] rocketmq-container 5.1.1-SNAPSHOT .................. SKIPPED
[INFO] rocketmq-test 5.1.1-SNAPSHOT ....................... SKIPPED
[INFO] rocketmq-distribution 5.1.1-SNAPSHOT ............... SKIPPED
[INFO] rocketmq-openmessaging 5.1.1-SNAPSHOT .............. SKIPPED
[INFO] rocketmq-example 5.1.1-SNAPSHOT .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:39 min
[INFO] Finished at: 2023-04-23T23:13:18+08:00
[INFO] Final Memory: 77M/844M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:4.2.2:check (check) on project rocketmq-remoting: failed with 1 bugs and 0 errors  -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :rocketmq-remoting

Could you please provide your command? It's mvn clean install?

I run mvn spotbugs:gui but do not reproduce your problem. Additionally, the param response of writeResponse function indeed could be null, it would be appropriate to add @Nullable.

it happened when I used idea for Maven clean。yes，I think we need to delete @Nullable。For developers, this is unfriendly encoding。

i run maven install。 this error happens。

I can not reproduce your error. Maybe you can try 'mvn clean' before install command.

it happened when I used idea for Maven clean。yes，I think we need to delete @nullable。For developers, this is unfriendly encoding。

I have the same problem as you. I deleted @nullable and it worked. But I don't know why.

authors: Seven-Hc ShadowySpirits drpmma wwx412775

repo: apache/rocketmq


text:
[ISSUE #3371]Part D: Add ak and ck stats to broker stats manager

What is the purpose of the change
issue #3371
Brief changelog
Add stats about ak and ck message.
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.04%) to 53.289% when pulling 17f282231e4ff5ea5b70e016e601beb5b53f7189 on maixiaohai:add_ak_ck_write_stats_to_schedule_topic into dd09ba1ec869acb1538d2c73f98d44066bbbb00f on apache:5.0.0-alpha.

Codecov Report

Merging #3373 (17f2822) into 5.0.0-alpha (48d3c7e) will increase coverage by 0.05%.
The diff coverage is 66.66%.

@@                Coverage Diff                @@
##             5.0.0-alpha    #3373      +/-   ##
=================================================
+ Coverage          46.81%   46.87%   +0.05%
- Complexity          4965     4977      +12
=================================================
  Files                610      612       +2
  Lines              41966    42024      +58
  Branches            5757     5739      -18
=================================================
+ Hits               19647    19698      +51
- Misses             19671    19682      +11
+ Partials            2648     2644       -4

Impacted Files
Coverage Δ

...cketmq/broker/processor/PopBufferMergeService.java
45.47% <50.00%> (+0.34%)
:arrow_up:

...rocketmq/broker/processor/AckMessageProcessor.java
47.00% <100.00%> (+2.20%)
:arrow_up:

...apache/rocketmq/broker/longpolling/PopRequest.java
31.03% <0.00%> (-13.80%)
:arrow_down:

...org/apache/rocketmq/store/ha/WaitNotifyObject.java
66.07% <0.00%> (-12.78%)
:arrow_down:

...rocketmq/broker/pagecache/ManyMessageTransfer.java
36.36% <0.00%> (-9.80%)
:arrow_down:

.../rocketmq/broker/pagecache/OneMessageTransfer.java
24.00% <0.00%> (-9.34%)
:arrow_down:

...org/apache/rocketmq/common/stats/StatsItemSet.java
41.79% <0.00%> (-8.96%)
:arrow_down:

...ocketmq/broker/processor/SendMessageProcessor.java
39.89% <0.00%> (-7.29%)
:arrow_down:

.../java/org/apache/rocketmq/acl/common/AclUtils.java
79.85% <0.00%> (-3.27%)
:arrow_down:

...ketmq/client/impl/consumer/PullMessageService.java
52.30% <0.00%> (-3.08%)
:arrow_down:

... and 57 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48d3c7e...17f2822. Read the comment docs.

authors: codecov-commenter coveralls maixiaohai


text:
RemoteObject Data Retrieval is Very Slow

In our application we are using mx:RemoteObject and in resultHandler we are recieving data as
e.result as Arraycollection but Grid is not populating because Java is sending much data (8000 fields)
when we decrease the data to (300 fields) the Grid will be populated

When populated rows are more than 300 and we assign e.result to adg.dataprovider. it takes alot time to show data in grid. Debugger stops  more than a minute at line adg.dataprovider = e.result; if data is much more like 8000 rows debugger stops at line adg.dataprovider = e.result and never excute it

@aharui Thank u so much Alex
Before these commits
Virtual layout in ADG. Should fix #468
some refactoring and exposing more protected members to allow override
our CompanySetup looks like

After these commits
Virtual layout in ADG. Should fix #468
some refactoring and exposing more protected members to allow override
our CompanySetup looks like

Probably some exception is being thrown.  Check the console and/or debug the code.

Thanks a lot @aharui @piotrzarzycki21 . RemoteObject Retrieval issue has been resolved.

authors: aharui alinakazi pashminakazi

repo: apache/royale-asjs


text:
the history data in service_instance_inventory will not be deleted while ttl is set in application.yml

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ *] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
the history data in service_instance_inventory will not be deleted while ttl is set in application.yml.so when i select a long time duration in  service tab,it will display too many expired service name choices with different pid.is it a bug or how can i delete the history service_instance_inventory?

Bug

Which version of SkyWalking, OS and JRE?
6.0GA  centos 7  jdk1.8

Which company or project?

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.

Requirement or improvement

Please describe about your requirements or improvement suggestions.

We don't delete inventory metadata.

We don't delete inventory metadata.

can i delete inventory metadata in mannul?

can i delete inventory metadata in mannul?

This is an open source project. You can do anything you want.

repo: apache/skywalking


text:
Selected collector grpc service running, reconnect:false

Agent has been printing the following log:
 Selected collector grpc service running, reconnect:false
Everything seems normal.

I restarted the node in the cluster and everything resumed.

This is a normal debug log, nothing more.


text:
skywalking print error in biz log

Please answer these questions before submitting your issue.

Why do you submit this issue?
[x] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?
Version 6.6.0  in jdk 1.8 and centos 7.0

Which company or project?
koolearn

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
I found some no class found some error in my application biz log(not in skywalking agent log), then I download the agent jar in product machine and decompile the jar found the class exists.

Jun 01, 2020 4:51:36 PM org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls cancelThrow
SEVERE: RuntimeException encountered while closing call
java.lang.NoClassDefFoundError: org/apache/skywalking/apm/dependencies/io/grpc/internal/DelayedStream$8
	at org.apache.skywalking.apm.dependencies.io.grpc.internal.DelayedStream.cancel(DelayedStream.java:267)
	at org.apache.skywalking.apm.dependencies.io.grpc.internal.DelayedClientTransport$PendingStream.cancel(DelayedClientTransport.java:367)
	at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.cancel(ClientCallImpl.java:395)
	at org.apache.skywalking.apm.dependencies.io.grpc.PartialForwardingClientCall.cancel(PartialForwardingClientCall.java:39)
	at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCall.cancel(ForwardingClientCall.java:22)
	at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(ForwardingClientCall.java:44)
	at org.apache.skywalking.apm.dependencies.io.grpc.PartialForwardingClientCall.cancel(PartialForwardingClientCall.java:39)
	at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCall.cancel(ForwardingClientCall.java:22)
	at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(ForwardingClientCall.java:44)
	at org.apache.skywalking.apm.dependencies.io.grpc.PartialForwardingClientCall.cancel(PartialForwardingClientCall.java:39)
	at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCall.cancel(ForwardingClientCall.java:22)
	at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCall$SimpleForwardingClientCall.cancel(ForwardingClientCall.java:44)
	at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.cancelThrow(ClientCalls.java:237)
	at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:134)
	at org.apache.skywalking.apm.network.register.v2.ServiceInstancePingGrpc$ServiceInstancePingBlockingStub.doPing(ServiceInstancePingGrpc.java:158)
	at org.apache.skywalking.apm.agent.core.remote.ServiceAndEndpointRegisterClient.run(ServiceAndEndpointRegisterClient.java:179)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Requirement or improvement

Please describe about your requirements or improvement suggestions.

Sorry, this class exists, for sure. Don't know what happens at your local env.

authors: darcydai wu-sheng


text:
Support RedisTemplate via Jedis Plugin

Please answer these questions before submitting pull request

Why submit this pull request?

[ ] Bug fix

[x] New feature provided

[ ] Improve performance

Related issues

Bug fix

Bug description.

How to fix?

New feature or improvement

Describe the details and related test reports.

Coverage decreased (-0.0007%) to 16.247% when pulling e096fc6b60043945819a637e6ae1c9162afb9722 on liuhaoyang:feature/redistemplate_plugin into ac2cc7339de628feb9ae168219d8e0deff8d6194 on apache:master.

authors: coveralls liuhaoyang


text:
fix cannot use 'System.out' through configuration.

Please answer these questions before submitting pull request

Why submit this pull request?

[x] Bug fix

[ ] New feature provided

[ ] Improve performance

Related issues

Bug fix

Bug description.
According documents, we can use System.out to output logs through set logging.dir as null or blank string.

I think logging to Console is still useful, especially when testing under Windows.
So I think we need it. And set the default is FileWriter.
This is just my opinion.


text:
Revert IoTDB as storage option due to negative perf testing result.

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[x] Update the CHANGES log.

Revert #7766. I am so sorry @LIU-WEI-git @jixuan1989 I have to initialize this revert due to our recent performance testing result.
Here are the graphs of SkyWalking OAP server so11y(self-obsevability) result.

Before 9:45, the OAP includes the pull request of IoTDB storage implementation, you can see that, it provides much lower performance than it should be(after 9:45). The analysis capability drops from 30+k/s to < 10k/s, which is hard to believe.
Both of these testing are running in new deployed environment, most importantly, IoTDB storage option is even NEVER ACTIVATED, which could only provide one possibility, the packages(jars) we introduced have clear conflicts and performance impact to SkyWalking server.
This is the first time I see this happened, which surprised me too. But tests verified this repeatedly. So I am submitting this pull request, and will ask @mrproliu to verify again according this branch.

We have the 2nd time verification rechecked. Before 10:48, it was this PR's codebase testing, and after than we reset back to master branch(including IoTDB implementation for now).
Same drop happened.
__

According to this dependencies change, we are going to do one more attempting, which is, removing logback-*, even I think this has very little chance to affect this, due to we don't output logs many things during testing. But, this is only possible guess, others have to be rechecked by IoTDB team.

We are keeping eye on this :D.

OK, after removing the logback, the perf seems back to normal. It seems log4j co-existing with logback causing huge performance impact. We are still checking.

@mrproliu If you could confirm this removing works, please submit a pull request to remove logback dependency.
@kezhenxu94 Is there a chance we could add a blacklist in the dependency check script to avoid this case in the future?


text:
[SPARK-12000] Fix API doc generation issues

This pull request fixes multiple issues with API doc generation.

Modify the Jekyll plugin so that the entire doc build fails if API docs cannot be generated. This will make it easy to detect when the doc build breaks, since this will now trigger Jenkins failures.
Change how we handle the -target compiler option flag in order to fix javadoc generation.
Incorporate doc changes from @thunterdb (in #10048).

Closes #10048.

Test build #46907 has started for PR 10049 at commit 38c1390.

Chatted with @thunterdb and we figured out that the Scala compiler error reported in SPARK-12000 does not occur under Java 8. Tim's Javadoc changes in his PR are required for Javadoc compilation to work under JDK 8, but do not address the issue where Scaladoc compilation fails under Java 7. I'm going to see if I can fix the Java 7 doc build, since we use Java 7 for release and snapshot builds.
In principle, we could use Java 8 to do the snapshot builds while targeting Java 7, but then we'd have to configure out build to use a Java 7 class library in oder to avoid the introduction of subtle incompatibilities caused by Java 8 standard library classes being used.

It looks like Jenkins is able to successfully generate the Scaladocs using Java 7, but my Mac laptop hits that compiler error. I'm going to try upgrading to a newer Java 7 JDK to see if that resolves things.
For posterity, I'm currently encountering the compiler issues while using the following JDK:
[joshrosen ~]$ java -version
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Yep, looks like upgrading to a newer JDK fixed things. Here's what worked for me:
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

Given that using a newer JDK seems to be how we've been able to successfully publish Scaladoc for 1.5.x, I'm going to go ahead and recommend a JDK upgrade as the solution to the Scaladoc issue. Therefore, I think that we're good to merge this and resolve the JIRA issue.

LGTM

Going to merge this now into master and branch-1.6.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46913/
Test FAILed.

Merged build finished. Test FAILed.

Test build #46907 has finished for PR 10049 at commit 38c1390.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46907/
Test PASSed.

Hmm, now I seem to have started having problems even in spite of the JDK update. Will investigate.

Now I'm very confused: it looks like Jenkins is actually using a different Java 7 JDK:
Using /home/jenkins/tools/hudson.model.JDK/JDK_7u60 as default JAVA_HOME.

authors: AmplabJenkins JoshRosen SparkQA marmbrus

repo: apache/spark


text:
[SPARK-12002][Streaming][PySpark] Fix python direct stream checkpoint recovery issue

Fixed a minor race condition in #10017
Closes #10017

Test build #46987 has started for PR 10074 at commit 82b1a07.

Test build #46987 has finished for PR 10074 at commit 82b1a07.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46987/
Test PASSed.

merging to master and 1.6

Thanks @jerryshao

Thanks a lot @zsxwing for this fix.

authors: AmplabJenkins SparkQA jerryshao zsxwing


text:
[Style] Minor style change on CoarseGrainedExecutorBackend

Removed  unnecessary brackets
Idiomatic usage of argv.nonEmpty instead of !argv.isEmpty

Can one of the admins verify this patch?

Thanks for looking at this, but I'm not sure that this minor style cleanup adds a whole lot of value so I'd prefer not to do this type of local cleanup for cleanup's sake.

sgtm, was looking at low hanging fruit while I get acclimated with spark.

authors: AmplabJenkins JoshRosen financeCoding


text:
[SPARK-16731][SQL] use StructType in CatalogTable and remove CatalogColumn

What changes were proposed in this pull request?
StructField has very similar semantic with CatalogColumn, except that CatalogColumn use string to express data type. I think it's reasonable to use StructType as the CatalogTable.schema and remove CatalogColumn.
How was this patch tested?
existing tests.

cc @yhuai @liancheng

Test build #62873 has started for PR 14363 at commit 776b267.

Test build #62873 has finished for PR 14363 at commit 776b267.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62873/
Test FAILed.

Test build #62879 has started for PR 14363 at commit 3f5480c.

Test build #62879 has finished for PR 14363 at commit 3f5480c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62879/
Test FAILed.

Test build #62889 has started for PR 14363 at commit addc585.

Test build #62889 has finished for PR 14363 at commit addc585.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62889/
Test FAILed.

Test build #62965 has started for PR 14363 at commit 97b0492.

Test build #62965 has finished for PR 14363 at commit 97b0492.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 implicit class SchemaAttribute(f: StructField)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62965/
Test FAILed.

Test build #62969 has started for PR 14363 at commit b781ef8.

Test build #62969 has finished for PR 14363 at commit b781ef8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 implicit class SchemaAttribute(f: StructField)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62969/
Test PASSed.

Merged build finished. Test PASSed.

Test build #63045 has started for PR 14363 at commit 80d2f50.

Test build #63045 has finished for PR 14363 at commit 80d2f50.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63045/
Test PASSed.

Do we know which hive type strings cannot be parsed by spark?

LGTM. Thanks. Merging to master.

Do we know which hive type strings cannot be parsed by spark?

varchar(length) and char(length). see https://github.com/apache/spark/pull/14363#discussion_r72735867 for what we break.

Thanks. But, what are specific cases are not supported? If there is any case, we should make change to support that, right?

for a hive table(created by hive) with varchar(length) column, we can describe it but can't read data from it before this PR. Now we can't describe it. Do you think we should fix it? BTW there is no test for this case.

TestHive.sessionState.metadataHive.runSqlHive("CREATE TABLE test (id varchar(50))")
TestHive.sessionState.metadataHive.runSqlHive("INSERT INTO TABLE test VALUES ('4')")
spark.sql("select * from test").show()
spark.sql("describe test").show()

Are you saying this case? I tried. It works.

Oh sorry I misread our parser rules. varchar(length) is supported but the length is ignored. I checked with hive again, looks like the only unsupported data type is UNIONTYPE

Your concern is valid. We are missing the test cases for verifying these scenarios.
I saw a discussion in a wechat group about the issue in integration between Hive and Spark. They are complaining Spark is unable to read the data wrote by Hive. In Hive refactoring, I am wondering if we also need to build the test cases to cover these cases?

@cloud-fan There is a case that i met.  The varchar(length)/char(length) type is not a String Type. But now SparkSQL consider them a string type. So there are different result with the following example:
TestHive.sessionState.metadataHive.runSqlHive("CREATE TABLE test (id varchar(50))")
TestHive.sessionState.metadataHive.runSqlHive("INSERT INTO TABLE test VALUES ('abcdef')")
TestHive.sessionState.metadataHive.runSqlHive("CREATE TABLE test_parquet (id varchar(2) stored as  parquet)")
TestHive.sessionState.metadataHive.runSqlHive("insert overwrite table varchar_parquet1 select * from test")
the result of varchar_parquet1 are 'ab'.
spark.sql("insert overwrite table varchar_parquet1 select * from test").show()
the result of varchar_parquet1 are 'abcdef'.

Well, Spark SQL is not announced to be fully compatible with hive, I think it's reasonable to have some issues. cc @rxin @yhuai should we fix this?

@lianhuiwang Parquet is a little bit different here. We are converting it to data source tables when inserting rows into Parquet. To get the expected results, you just need to set spark.sql.hive.convertMetastoreParquet to false.
If you are choosing textfile, it works as expected.

authors: AmplabJenkins SparkQA cloud-fan gatorsmile lianhuiwang yhuai


text:
[SPARK-17715] [Scheduler] Make task launch logs DEBUG

What changes were proposed in this pull request?
Ramp down the task launch logs from INFO to DEBUG. Task launches can happen orders of magnitude more than executor registration so it makes the logs easier to handle if they are different log levels. For larger jobs, there can be 100,000s of task launches which makes the driver log huge.
How was this patch tested?
No tests, as this is a trivial change.

Can one of the admins verify this patch?

Jenkins, add to whitelist.

Jenkins, test this please.

LGTM.

ok to test

Merging into master

Test build #66120 has started for PR 15290 at commit 300439e.

Test build #66120 has finished for PR 15290 at commit 300439e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66120/
Test PASSed.

authors: AmplabJenkins SparkQA andrewor14 dafrista rxin


text:
[MINOR][DOCS][PYTHON] Adding missing boolean type for replacement value in fillna

What changes were proposed in this pull request?
Currently pyspark Dataframe.fillna API supports boolean type when we pass dict, but it is missing in documentation.
How was this patch tested?

spark.createDataFrame([Row(a=True),Row(a=None)]).fillna({"a" : True}).show()
+----+
|   a|
+----+
|true|
|true|
+----+

Please review http://spark.apache.org/contributing.html before opening a pull request.

Can you fix up the PR title - it seems to get truncated
Is there any existing test to add to?

Hi @felixcheung Thanks for the review. I have added a small testcase.

Test build #3669 has started for PR 17688 at commit acae068.

Test build #3669 has finished for PR 17688 at commit acae068.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

My apologies for style failures, i have fixed them.

Jenkins, retest this please

Test build #76042 has started for PR 17688 at commit 1ec1c0d.

Test build #76042 has finished for PR 17688 at commit 1ec1c0d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76042/
Test PASSed.

We should also update the list of types a few lines up while we are fixing this. thanks a lot for catching this @vundela

@holdenk Thanks for the review. Can you please let me know the line number where you are expecting list of types missing. Is this for fillna or other API?

LGTM too. I just quickly checked if there are similar instances but I could not find and I checked R's one and Scala one.

@vundela L1237

good catch - instead of duplicating it, perhaps just say supported data types or supported data types above

LGTM, thanks @HyukjinKwon for noticing the lack of bool in the scala code.

@holdenk, Thanks for reviewing it. I guess L1237 can't be changed until we support boolean type. L1240 specifically talks about the types of values supported in dict. Please let me know if you think otherwise.

Yup, it's fine hence the LGTM :)

Thanks @holdenk

LGTM too.

merged to master/2.2

authors: AmplabJenkins HyukjinKwon SparkQA felixcheung holdenk vundela


text:
[MINOR][DOC] Update deprecation notes on Python/Hadoop/Scala.

What changes were proposed in this pull request?
We had better update the deprecation notes about Python 2.6, Hadoop (before 2.6.5) and Scala 2.10 in 2.2.0-RC4 documentation. Since this is a doc only update, I think we can update the doc during publishing.
BEFORE (2.2.0-RC4)

AFTER

How was this patch tested?
Manual.
SKIP_API=1 jekyll build

Hi, @marmbrus .
Could you review this PR about 2.2.0?

Test build #77754 has started for PR 18207 at commit 69774c1.

I wasn't clear whether we really truly don't support 2.6 in this 2.2.0 release, given https://issues.apache.org/jira/browse/SPARK-12661 ?

Oh, I see. Then, may I ask this at mailing list to be more clearer?

Test build #77754 has finished for PR 18207 at commit 69774c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77754/
Test PASSed.

/cc @joshrosen

As of #17355 we no longer test against Python 2.6. That doesn't mean that 2.6 won't work today, but there's nothing stopping 2.6 support from breaking in a future 2.2.x release because we are no longer testing against that release.
#17355 replaced our Python 2.6 testing environment with a Python 2.7 release, so we can now begin to use language features and libraries which are only available from 2.7 onwards (such as set and dictionary comprehensions).
Therefore, this documentation change looks correct to me.

Thank you for confirming, @JoshRosen .

I believe we still support Python 2.6, given Jenkins runs 2.6... There seems to be no point in removing that support this late in the release cycle.

Jenkins runs with ['python2.7', 'python3.4', 'pypy'] only, doesn't it? Also, this is a major release cycle with the other big changes. For me, removing Python 2.6 is not proper with subsequent minor version release cycles like 2.2.1 and 2.2.2.

@rxin, as of #17355 Jenkins is using Python 2.7 sourced from a virtualenv instead of Python 2.6. That patch was merged into master before branch-2.2 was cut.

OK great then we have officially deprecated it, haven't we?

Yep. Right. Then, could you officially resolve SPARK-12661, too?

Even if Python 2.6 "works by accident" on 2.2.0 that should not impact this
wording: as long as we marked Python 2.6 as deprecated in an earlier
release then it should be fine to claim non support in 2.2.0.
On Tue, Jun 6, 2017, 8:25 AM Dongjoon Hyun notifications@github.com wrote:

@dongjoon-hyun commented on this pull request.
In docs/index.md
https://github.com/apache/spark/pull/18207#discussion_r120393293:

uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scala version
({{site.SCALA_BINARY_VERSION}}.x).

-Note that support for Java 7 was removed as of Spark 2.2.0.
+Note that support for Java 7, Python 2.6 and old Hadoop versions before 2.6.5 were removed as of Spark 2.2.0.
Did you test on Python 2.6 at RC4 testing? And it happens to be supported
at 2.2.0?
Actually, Jenkins does not run the tests over two months (since 29th Mar).
As @JoshRosen https://github.com/joshrosen said, there is no guard
anymore in order to guarantee still more supported on 2.2.1 or later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18207#discussion_r120393293, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADGPJtuOTgL3_uYB4wzbbBpEBmlb1o2ks5sBW99gaJpZM4NweCt
.

Put another way: if we found a bug only impacting Python 2.6 in 2.2.0 would
we fix that bug in 2.2.1? If not then the right thing to do is to say 2.6
isn't supported.
On Tue, Jun 6, 2017, 8:44 AM Josh Rosen rosenville@gmail.com wrote:

Even if Python 2.6 "works by accident" on 2.2.0 that should not impact
this wording: as long as we marked Python 2.6 as deprecated in an earlier
release then it should be fine to claim non support in 2.2.0.
On Tue, Jun 6, 2017, 8:25 AM Dongjoon Hyun notifications@github.com
wrote:

@dongjoon-hyun commented on this pull request.
In docs/index.md
https://github.com/apache/spark/pull/18207#discussion_r120393293:

uses Scala {{site.SCALA_BINARY_VERSION}}. You will need to use a compatible Scala version
({{site.SCALA_BINARY_VERSION}}.x).

-Note that support for Java 7 was removed as of Spark 2.2.0.
+Note that support for Java 7, Python 2.6 and old Hadoop versions before 2.6.5 were removed as of Spark 2.2.0.
Did you test on Python 2.6 at RC4 testing? And it happens to be supported
at 2.2.0?
Actually, Jenkins does not run the tests over two months (since 29th
Mar). As @JoshRosen https://github.com/joshrosen said, there is no
guard anymore in order to guarantee still more supported on 2.2.1 or
later.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18207#discussion_r120393293, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADGPJtuOTgL3_uYB4wzbbBpEBmlb1o2ks5sBW99gaJpZM4NweCt
.

I think the question is whether we can get this change into the 2.2 release. It'd be ideal, but not sure it's worth another RC. Of course, this runs into the discussion we're currently having about when to update docs.

I hope it goes to Release Note instead of blocking the current voting.
Something like 2.1.0 Release Note.

Merged to master/2.2

Thank you all!

authors: AmplabJenkins JoshRosen SparkQA dongjoon-hyun marmbrus rxin srowen


text:
[SPARK-11334][CORE] clear idle executors in executorIdToTaskIds keySet

What changes were proposed in this pull request?
quote from #11205

Executors may never be idle. Currently we use the executor to tasks mapping relation to identify the status of executors, in maximum task failure situation, some TaskEnd events may never be delivered, which makes the related executor always be busy.

#19580 fix the incorrect number of running tasks.
This PR solves the problem in a similar way on the fact that TaskEnd events may never be delivered but StageCompleted events would be delivered eventually.
How was this patch tested?
Existing tests

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can you please check again with latest master code, I doubt the issue is not valid any more in the latest code.

+1 seems you didn't test against the latest master code or 2.3?

I have checked the latest master code.
        // If the executor is no longer running any scheduled tasks, mark it as idle
        if (executorIdToTaskIds.contains(executorId)) {
          executorIdToTaskIds(executorId) -= taskId
          if (executorIdToTaskIds(executorId).isEmpty) {
            executorIdToTaskIds -= executorId
            allocationManager.onExecutorIdle(executorId)
          }
        }

To remove a executor, we have to mark it idle. To mark a executor idle, we have to invoke allocationManager.onExecutorIdle. In the latest master code, the two places that invokes the method are onExecutorAdded and onTaskEnd. For a running executor, the only way to mark it idle is the code snippet above.
#19580 makes sense on the basis that the event TaskEnd for some reason may be lost.
On the same basis, the Set executorIdToTaskIds(executorId) may never be empty.

the issue is not valid any more in the latest code

Please tell me why, thanks! The TaskEnd event will never be lost or other improves ? Which PR?
We are using Spark 2.1.1, and the issue is valid.
quote from #11205 by @jerryshao

Verified again, looks like the 2nd bullet is not valid anymore, I cannot reproduce it in latest master branch, this might have already been fixed in SPARK-13054.

Spark 2.1.1 is shipped with the fixes for SPARK-13054, but still we encountered the issue.

We improve the DAGScheduler to always send TaskEnd message. So the issue I found before may not be valid.
We refactored the LiveListenerQueue to make it more robust for internal listener. We cannot guarantee that event will never be lost, but the chance is quite small (SPARK-18838).

IMHO you (as a PR submitter) should validate this issue with latest code and make sure you can reproduce it with latest code.

authors: AmplabJenkins jerryshao jiangxb1987 sadhen


text:
[SPARK-30155][SQL] Rename parse() to parseString() to avoid conflict in Scala 2.13

What changes were proposed in this pull request?
Rename internal method LegacyTypeStringParser.parse() to parseString().
Why are the changes needed?
In Scala 2.13, the parse() definition clashes with supertype declarations.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests.

Test build #114962 has started for PR 26784 at commit f0d6635.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19783/
Test PASSed.

Test build #114962 has finished for PR 26784 at commit f0d6635.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114962/
Test PASSed.

Thank you, @srowen . Merged to master.

authors: AmplabJenkins SparkQA dongjoon-hyun srowen


text:
[SPARK-18409][ML][FOLLOWUP] LSH approxNearestNeighbors optimization

What changes were proposed in this pull request?
compute count and quantile on one pass
Why are the changes needed?
to avoid extra pass
Does this PR introduce any user-facing change?
No
How was this patch tested?
existing testsuites

Test build #115665 has started for PR 26990 at commit 586d947.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20462/
Test PASSed.

Test build #115665 has finished for PR 26990 at commit 586d947.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115665/
Test FAILed.

retest this please

Test build #115723 has started for PR 26990 at commit 586d947.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20517/
Test PASSed.

Test build #115723 has finished for PR 26990 at commit 586d947.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115723/
Test PASSed.

friendly ping @huaxingao @srowen

The change looks good to me. Seems that approxQuantile does this under the hood.

@srowen

approxQuantile basically do this already under the hood?

Yes, approxQuantile also use QuantileSummaries like this.

I'm OK with it. It sounds like an optimization without much if any downside.

Merged to master, thanks @srowen @huaxingao for reviewing!

authors: AmplabJenkins SparkQA huaxingao srowen zhengruifeng


text:
[SPARK-27946][SQL][FOLLOW-UP] Change doc and error message for SHOW CREATE TABLE

What changes were proposed in this pull request?

This is a follow-up for #24938 to tweak error message and migration doc.
Why are the changes needed?

Making user know workaround if SHOW CREATE TABLE doesn't work for some Hive tables.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Existing unit tests.

cc @gatorsmile

Test build #118089 has started for PR 27505 at commit e5f496e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22854/
Test PASSed.

Test build #118089 has finished for PR 27505 at commit e5f496e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118089/
Test FAILed.

retest this please

Test build #118092 has started for PR 27505 at commit e5f496e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22857/
Test PASSed.

Test build #118092 has finished for PR 27505 at commit e5f496e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118092/
Test PASSed.

I'm going to merge this to master and branch-3.0.

authors: AmplabJenkins SparkQA viirya


text:
[SPARK-31330][INFRA][FOLLOW-UP] Move sbin and some files into appropriate categories in autolabeller

What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/commit/1b8701504457ec291e6221dd63eec6e6be999150. Now, we automatically label PRs, and seems working fine.
This PR proposes to correct some minor list and categories.
1. Move sbin from CORE into DEPLOY components.
$ ls sbin

decommission-slave.sh          start-all.sh                   start-slave.sh                 stop-master.sh                 stop-thriftserver.sh
slaves.sh                      start-history-server.sh        start-slaves.sh                stop-mesos-dispatcher.sh
spark-config.sh                start-master.sh                start-thriftserver.sh          stop-mesos-shuffle-service.sh
spark-daemon.sh                start-mesos-dispatcher.sh      stop-all.sh                    stop-slave.sh
spark-daemons.sh               start-mesos-shuffle-service.sh stop-history-server.sh         stop-slaves.sh

2.
/sbin/*mesos*.sh -> MESOS
/bin/spark-shell* -> SPARK SHELL.
Why are the changes needed?
To label correctly and dev can take an advantage of it such as checking the PRs of a specific component.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
It was not tested yet. It can be tested after it was merged.

cc @Ngone51, @nchammas, @HeartSaVioR, @dongjoon-hyun. Also @srowen.

Not related to this PR but just realized that we've missed TEST label. Or we ignored it intentionally?

Test build #121189 has started for PR 28201 at commit 4c44877.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25876/
Test PASSed.

I wanted to add but could not come up with how to do because it should only contain test files.

Thanks @Ngone51. Let me merge this since there's nothing related to ci builds here :-).

Test build #121189 has finished for PR 28201 at commit 4c44877.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121189/
Test PASSed.

authors: AmplabJenkins HyukjinKwon Ngone51 SparkQA


text:
[SPARK-33927][BUILD] Fix Dockerfile for Spark release to work

What changes were proposed in this pull request?
This PR proposes to fix the Dockerfile for Spark release.

Port https://github.com/apache/spark/commit/b135db3b1a5c0b2170e98b97f6160bcf55903799 to Dockerfile
Upgrade Ubuntu 18.04 -> 20.04 (because of porting b135db3)
Remove Python 2 (because of Ubuntu upgrade)
Uses built-in Python 3.8.5 (because of Ubuntu upgrade)
Node.js 11 -> 12 (because of Ubuntu upgrade)
Ruby 2.5 -> 2.7 (because of Ubuntu upgrade)
Python dependencies upgrade to the latest as it's used in GitHub Actions build (unrelated to the issue itself)

Why are the changes needed?
To make a Spark release :-).
Does this PR introduce any user-facing change?
No, dev-only.
How was this patch tested?
Manually tested via:
cd dev/create-release/spark-rm
docker build -t spark-rm --build-arg UID=$UID .

...
Successfully built 516d7943634f
Successfully tagged spark-rm:latest

Thank you, @HyukjinKwon !

cc @dongjoon-hyun, @sarutak, @cloud-fan FYI

Test build #133522 has started for PR 30971 at commit 7bce81e.

Thank you @dongjoon-hyun for double checking!

Feel free to merge, @HyukjinKwon . Thank you always for this preparation!

Merged to master and branch-3.1.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38110/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38110/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38110/

Test build #133522 has finished for PR 30971 at commit 7bce81e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133522/

Sorry for the late reply. It LGTM!

authors: AmplabJenkins HyukjinKwon SparkQA dongjoon-hyun sarutak


text:
[SPARK-45737][SQL] Remove unnecessary .toArray[InternalRow] in SparkPlan#executeTake

What changes were proposed in this pull request?
https://github.com/apache/spark/blob/8dd3ec87e26969df6fe08f5fddc3f8d6efc2420d/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L535-L559
In the above code, the input parameters of mutable.Buffer#prependAll and mutable.Growable#++= functions are IterableOnce

mutable.Buffer#prependAll

  def prependAll(elems: IterableOnce[A]): this.type = { insertAll(0, elems); this }

mutable.Growable#++=

  @`inline` final def ++= (xs: IterableOnce[A]): this.type = addAll(xs)

and the type of rows is Iterator[InternalRow], which inherits from IterableOnce
val rows = decodeUnsafeRows(res(i)._2)
private def decodeUnsafeRows(bytes: ChunkedByteBuffer): Iterator[InternalRow]

So there is no need to cast to an Array of InternalRow anymore.
Why are the changes needed?
Remove unnecessary .toArray[InternalRow]
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GitHub Actions
Was this patch authored or co-authored using generative AI tooling?
No

Thanks @dongjoon-hyun


text:
[SPARK-46392][SQL] Push down Cast expression in DSv2

What changes were proposed in this pull request?

Push down Cast expression.
Why are the changes needed?

Push down Cast expression to accelerate computation.
Does this PR introduce any user-facing change?

Considering table blow:
CREATE TABLE `tableA`(
  s string)
PARTITIONED BY (
  `dt` string COMMENT '日期, yyyyMMdd',
  `hour` string COMMENT '小时, HH')

if we execute sql:
select * from tableA where dt=20231212 and hour=22

After V2ScanRelationPushDown Rule been applyed,
Only IsNotNull(dt), IsNotNull(hour) be pushed to datasource, log like this:
Pushing operators to tableA
Pushed Filters: IsNotNull(dt), IsNotNull(hour)
Post-Scan Filters: (cast(dt#520 as int) = 20231212),(cast(hour#521 as int) = 10)

But what we need is all expression: IsNotNull(dt), IsNotNull(hour), EqualTo(dt,20231212), EqualTo(hour,10) should been pushed to dataSource, log like blow:
Pushing operators to tableA
Pushed Filters: IsNotNull(dt), IsNotNull(hour), EqualTo(dt,20231212), EqualTo(hour,10)
Post-Scan Filters:

the root cause is: translateLeafNodeFilter in PushDownUtils  does not handle  Cast expression case
How was this patch tested?

NEW UT
Was this patch authored or co-authored using generative AI tooling?

NO

CREATE TABLE tableA(
  s string)
PARTITIONED BY (
  dt string COMMENT '日期, yyyyMMdd',
  hour string COMMENT '小时, HH')

Could you update your example with a working reproduction? This create table statement is invalid. It partitions on two columns dt and hour that do not exist in the column declaration. And the subsequent select is also invalid.; it's missing a table name.

Could you update your example with a working reproduction? This create table statement is invalid. It partitions on two columns dt and hour that do not exist in the column declaration. And the subsequent select is also invalid.; it's missing a table name.

create table statement:
```CREATE TABLE tableA(
user_id string COMMENT 'from deserializer')
PARTITIONED BY (
dt string COMMENT '日期, yyyyMMdd',
hour string COMMENT '小时, HH')
ROW FORMAT SERDE
'org.apache.paimon.hive.PaimonSerDe'
STORED BY
'org.apache.paimon.hive.PaimonStorageHandler'
WITH SERDEPROPERTIES (
'serialization.format'='1')
LOCATION
'xxxx'
TBLPROPERTIES (
'bucket'='50',
'bucketing_version'='2',
'bukect-key'='user_id',
'file.format'='parquet',
'merge-engine'='partial-update',
'partial-update.ignore-delete'='true',
'primary-key'='user_id',
'transient_lastDdlTime'='1701679855',
'write-only'='false')
It is a Paimon table, but it does not matter.

My apologies. It looks like I made some mistake trying to run a similar CREATE TABLE statement. Your original statement was fine.
By the way, if I want to reproduce this log you shared:

Pushing operators to tableA
Pushed Filters: IsNotNull(dt), IsNotNull(hour)
Post-Scan Filters: (cast(dt#520 as int) = 20231212),(cast(hour#521 as int) = 10)

How do I do that? It's not from the EXPLAIN output, it seems.

How do I do that? It's not from the EXPLAIN output, it seems.

Just run the sql and you will find the log in driver log.

Hmm, I am seeing a different plan:
spark-sql (default)> CREATE TABLE test (
                   >     s string
                   > )
                   > PARTITIONED BY (
                   >   dt string,
                   >   hour string
                   > );
Time taken: 0.222 seconds
spark-sql (default)> explain select * from test where dt=20231212 and hour=22;
== Physical Plan ==
Scan hive spark_catalog.default.test (1)

(1) Scan hive spark_catalog.default.test
Output [3]: [s#0, dt#1, hour#2]
Arguments: [s#0, dt#1, hour#2],
  HiveTableRelation [
    `spark_catalog`.`default`.`test`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
    Data Cols: [s#0], Partition Cols: [dt#1, hour#2], Pruned Partitions: []
  ], [isnotnull(dt#1), isnotnull(hour#2), (cast(dt#1 as int) = 20231212), (cast(hour#2 as int) = 22)]

And if I try another table, but this time USING parquet, I see PartitionFilters accepts the cast without issue:
spark-sql (default)> CREATE TABLE test (
                   >     s string
                   > )
                   > USING parquet
                   > PARTITIONED BY (
                   >   dt string,
                   >   hour string
                   > );
Time taken: 0.1 seconds
spark-sql (default)> explain select * from test where dt=20231212 and hour=22;
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet spark_catalog.default.test[s#19,dt#20,hour#21] Batched: true, DataFilters: [], Format: Parquet,
  Location: InMemoryFileIndex(0 paths)[],
  PartitionFilters: [isnotnull(dt#20), isnotnull(hour#21), (cast(dt#20 as int) = 20231212), (cast(hour#21 as int) = 22)],
  PushedFilters: [], ReadSchema: struct<s:string>

Anyway, I don't mean to waste your time. I was just trying to reproduce the issue, but it seems there are more details involved that I don't follow.

Anyway, I don't mean to waste your time. I was just trying to reproduce the issue, but it seems there are more details involved that I don't follow.

Thanks for your review, i will task times to describe this issue more details, maybe it take some days.

And if I try another table, but this time USING parquet, I see PartitionFilters accepts the cast without issue:

BTW, What is your Final Physical Plan? is FileSourceScanExec?,  the PR described here is BatchScanExec

I see this if I enable additional logging:
23/12/15 13:09:47 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.

I guess I need to change the table format to force a BatchScanExec.

I will close this as it is unreasonable.

I will close this as it is unreasonable.


text:
[MINOR][SQL] Remove out-of-dated comment in CollectLimitExec

What changes were proposed in this pull request?

This patch removes an out-of-dated code comment that was added by #35975.
Why are the changes needed?

Removed out-of-dated comment to reduce misunderstanding and confusion.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Existing test
Was this patch authored or co-authored using generative AI tooling?

No

Merged into master for Spark 4.0. Thanks @viirya @HyukjinKwon

authors: LuciferYang viirya


text:
[SPARK-50632][BUILD] Upgrade tink to 1.16.0

What changes were proposed in this pull request?
The pr aims to upgrade tink from 1.15.0 to 1.16.0.
Why are the changes needed?
The full release notes as follows: https://github.com/tink-crypto/tink-java/releases/tag/v1.16.0
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.

Thanks for the review, merging to master!