text:
validateEdge 返回 false 后，会将错误的连接和删除该连接的操作放入撤销栈

Expected Behavior

validateEdge 返回 false 后，不会在撤销栈产生操作
Current Behavior
validateEdge 返回 false 后，会将错误的连接和删除该连接的操作放入撤销栈，撤销一次则校验失败的连接会重现。

Possible Solution

https://github.com/antvis/X6/blob/master/packages/x6/src/view/edge.ts#L2634-L2668
将 stopArrowheadDragging 方法改成异步返回，返回后再调用 stopBatch 结束操作。（但是异步会向外传播，影响范围有点大）
现在我的解决方案是把校验改成强制同步的，对我目前的应用够了
EdgeView.prototype.stopArrowheadDragging = function stopArrowheadDragging(
    this: EdgeView,
    e: JQuery.MouseUpEvent,
    x: number,
    y: number,
) {
    const graph = this.graph;
    const data = this.getEventData<ArrowheadDragging>(e);
    if (graph.options.connecting.snap) {
        this.snapArrowheadEnd(data);
    } else {
        this.arrowheadDragged(data, x, y);
    }

    const valid = FunctionExt.toAsyncBoolean(
        graph.hook.validateEdge(this.cell, data.terminalType, data.initialTerminal),
    );
    const continuation = (valid: boolean): boolean => {
        if (valid) {
            this.finishEmbedding(data);
            this.notifyConnectionEvent(data, e);
        } else {
            // If the changed edge is not allowed, revert to its previous state.
            this.fallbackConnection(data);
        }

        this.afterArrowheadDragging(data);
        return valid;
    };
    if (FunctionExt.isAsyncLike(valid)) {
        valid.then(continuation).catch(() => {
            //
        });
    } else {
        try {
            continuation(valid);
        } catch {
            //
        }
    }
};

Steps To Reproduce
https://codesandbox.io/s/amazing-dew-8nl6t?file=/src/app.tsx
试图连接两个节点，然后撤销和重做
<!-- Provide a log message if relevant -->

Additional Context

Your Environment

x6: 1.18.5
OS: [e.g. macOS Sierra 10.12.3]
Browser: [e.g. chrome 78.0.3904.108]

说起来，这里是同步的
https://github.com/antvis/X6/blob/master/packages/x6/src/graph/hook.ts#L409-L429
这里也是同步的
https://github.com/antvis/X6/blob/master/packages/x6/src/graph/options.ts#L380-L387
把 stopArrowheadDragging  也改成同步的也不是不行吧


text:
Events spanning multiple days are not (well) supported

If an event spans multiple days it only shows on the day it starts. It would be more intuitive if the event is repeated for all days it is in effect.

As far as I remember, this was the limitation of ics library (assuming you mean events from ics files), it did not parse multi days events. Hopefully with icalendar we can bypass that.

Latest changes to timezone conversions introduced an issue:
datetime.date(2023, 1, 1).astimezone  # AttributeError: 'datetime.date' object has no attribute 'astimezone'

Furthermore multiday event does not mean the time difference is larger than 1 day.
start = datetime.datetime(2023, 1, 1, 23, 0)
end = datetime.datetime(2023, 1, 2, 5, 0)
is_multiday = True if (end - start).days > 0 else False
# is_multiday == False but that's wrong

Latest changes to timezone conversions introduced an issue:
datetime.date(2023, 1, 1).astimezone  # AttributeError: 'datetime.date' object has no attribute 'astimezone'

Does this happen when some events in .ics have no timezone or it's something else?

Furthermore multiday event does not mean the time difference is larger than 1 day.

I see, I thought datetime implemented the difference operator in a smarter way, then will need to remake it...

Latest changes to timezone conversions introduced an issue:
datetime.date(2023, 1, 1).astimezone  # AttributeError: 'datetime.date' object has no attribute 'astimezone'

The commit d74fcc1ff0ad8d99ac33ef32197a9381540116e9 seems to fix the issue with events without timezone.

Almost woks except for events set as all day. It looks like when start/end time is written in "all day" format as:
DTSTART;VALUE=DATE:20231107

something breaks as compared to the format when time is set:
DTSTART;TZID=Europe/Istanbul:20231114T100000

Which is weird since date is actually parsed correctly...

p.s. on the side note, I noticed a strange thing in Nextcloud calendar: all day multiday events export as one day more than they should. Maybe we should report it if it's not my mistake...

I believe the answer is in rfc5545 - Internet Calendaring and Scheduling Core Object Specification (iCalendar):
The "DTSTART" property for a "VEVENT" specifies the inclusive
start of the event.  For recurring events, it also specifies the
very first instance in the recurrence set.  The "DTEND" property
for a "VEVENT" calendar component specifies the non-inclusive end
of the event.
..
..
The following is an example of the "VEVENT" calendar component
used to represent a multi-day event scheduled from June 28th, 2007
to July 8th, 2007 inclusively.  Note that the "DTEND" property is
set to July 9th, 2007, since the "DTEND" property specifies the
non-inclusive end of the event.

  BEGIN:VEVENT
  UID:20070423T123432Z-541111@example.com
  DTSTAMP:20070423T123432Z
  DTSTART;VALUE=DATE:20070628
  DTEND;VALUE=DATE:20070709
  SUMMARY:Festival International de Jazz de Montreal
  TRANSP:TRANSPARENT
  END:VEVENT

I believe the answer is in rfc5545 - Internet Calendaring and Scheduling Core Object Specification (iCalendar):

Indeed, there is this feature that all day events are written as +1 day sort of, I think I fixed it in e3787ed3cdb8ce558f4bc1b5dcd157f79d808a1d Please try the main branch if you have time.

I'll assume that it works well for everybody and close this issue. If any troubles, you can mention it in #4

source: gharchive/issue

repo: anufrievroman/calcure

license: MIT

license_type: permissive

license_source: github-api


text:
added functionality to add my blog section

Related Issue
issue number: #4609
Fixes:  #4609
Description
added some functionality to add your blog section
Type of PR

[ ] Bug fix
[X] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos

Checklist:

[X] I have made this change from my own.
[X] I have taken help from some online resources.
[X] My code follows the style guidelines of this project.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly in hard-to-understand areas.
[X] I have made corresponding changes to the documentation.
[X] My changes generate no new warnings.
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers and screenshots after making the changes.

@abhi03ruchi pls check my pr

blog title  bhi add karo .
and when we click on publish blog , confirm Ok , then we must be redirected to the blog page , jaha hume apna blog deekhe .

also add [Home button] on top left corner , jaisa https://swapreadsconnect.netlify.app/assets/html/blog , mein hai .

source: gharchive/pull-request


text:
[Bug] getOffsetFromDirectParent values rounding to nearest pixel

repro: load https://wayfinderanimationtool.com/trialgrounds, notice some test cases show a very small sliver of red
so it turns out that HTMLElement.offsetLeft and offsetTop are integer values in the specs. not the precise values :|
getting the location of an element compared to its parent is surprisingly difficult, even though it shouldn't be. not sure how i'm gonna fix this issue yet. as far as i know, the precise values are not accessible anywhere in the specs, and i'd rather not use jquery or some other lib to fix this
medium priority--perfection is preferred, but the inaccuracy is very minor, so somewhat "acceptable" for now

Did some more research...
getBoundingClientRectangle() does not look like a viable solution for multiple reasons. it does provide subpixel precision, but it's viewport-based, and it doesn't work if scale, rotate, skew, or other properties are involved anywhere in the dom tree
will continue researching

Will write a new library to support this feature, and may eventually use it in Wayfinder

repo: anxpara/wayfinder-animation-tool


text:
Why isn't @hotwired/turbo a turbo-stream package dependency?

@hotwired/turbo is imported here:
https://github.com/anycable/anycable-client/blob/master/packages/turbo-stream/stream_source_element.js#L1

Oh, it should be there. As a peer dependency, I think. Will add.

Fixed and released as @anycable/turbo-stream 0.2.1

Great, thanks

authors: palkan yenshirak

repo: anycable/anycable-client


text:
MSW: Unstable mock generated for object with unspecified property name

In case object has unspecified (dynamic) property names like export type PetSomeField = {[key: string]: string[]}; mock generator function creates unique property name on every api generation call. This causes useless changes in generated files.
mock generator function:
export const getCreatePetsMock = () => ({someField: faker.helpers.randomize([{
  'cl2fwufng0000ky8xa7qhg7u0': [...Array(faker.datatype.number({min: 1, max: 10}))].map(() => (faker.random.word()))
}, undefined])})

openapi: '3.0.0'
info:
  version: 1.0.0
  title: Swagger Petstore
  license:
    name: MIT
servers:
  - url: http://petstore.swagger.io/v1
paths:
  /pets:
    post:
      summary: Create a pet
      operationId: createPets
      responses:
        '200':
          description: Created Pet
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Pet'
components:
  schemas:
    Pet:
      type: object
      properties:
        someField:
          type: object
          additionalProperties:
            type: array
            items:
              type: string

Hello @yekver, what would be a good solution for you? Always the same static object?

Hello @yekver, what would be a good solution for you? Always the same static object?

Yeah I think that in this case generator should return stable object.

Maybe a solution here is to seed the cuid generator with the operationId + path into the return type

CUID was removed https://github.com/anymaniax/orval/issues/974 and now MSW/Faker is reliable on every run.

authors: anymaniax arlyon melloware yekver

repo: anymaniax/orval


text:
It can't process sysl file generated by XSD to produce model diagram

Expected behavior
It can produce model digram with sysl file generated by XSD
Actual behavior
It can't produce model digram with sysl file generated by XSD
Steps to reproduce the behavior
There is xsd file:
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">

    <xs:element name="account">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="id" type="xs:string"/>
                <xs:element name="name" type="xs:string"/>
                <xs:element name="age" type="xs:int"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>

Then process xsd file to generate sysl file with command import:
##########################################
##                                      ##
##  AUTOGENERATED CODE -- DO NOT EDIT!  ##
##                                      ##
##########################################

Account:
    @description =:
        | No description.

    #---------------------------------------------------------------------------
    # definitions

    !type account:
        id <: string:
            @json_tag = "id"
        name <: string:
            @json_tag = "name"
        age <: int:
            @json_tag = "age"

And then it can't produce diagram with this generated sysl file.

Discussion logs:
There was some confusion initially over whether the original XSD importer was broken. As I recall, it wasn’t broken, it just didn’t generate project files for the diagram generator to pick up. But that was as intended. Project files capture a user’s intent to generate some view(s) of the system and should generally be hand-written and live separately from the imported schema.
By my understanding, the idea behind the new command was to allow diagram generation without a project file, but also, and more to the point, entirely without the need for a diagram specification expressed in Sysl syntax.

As latest discussion, the solution is merging this feature to existing command datamodel.

Use http://github.com/anz-bank/sysl/projects to track items on this issue.

repo: anz-bank/sysl


text:
re-iterator version of SIMD-170 impl

Problem
SIMD-170
Summary of Changes
Fixes #

Thank you for the list. tbh I am not confident the re-iteration per call will be OK, it kinda negates the purpose of static meta; the lookup at AhashMap when calling BUILTIN_INSTRUCTION_COSTS.contains_key(program_id) can add up quickly. This approach might be slightly better for backporting than https://github.com/tao-stones/solana/commit/86fcaebcf9f885b9e2a93123736692f9273a09d9; But I still lean to doing heavy lifting in try_from() cause it's called once per transaction, leaving sanitize_and_convert_to_compute_budget_limits() as lightweight as possible (just few if...else and some math).

repo: anza-xyz/agave


text:
tr_tr.json

Turkish Language File

Added to version 1.2.0


text:
Improve BulkSplitOptimizationIT

Partially addresses #5123
The most important change in this PR is just to bump the timeout from 2 to 5 minutes since the test often times out
I also...

added logging to make it easier to tell how far the test got and where it times out at in the future
moved the creation of the rfiles to the before block
created the test table using NewTableConfiguration and made the initial availability HOSTED to speed things up slightly

With the new logging you added do you know where the most time is being spent in the test?

I added some timing logging to the test to find out. Here is what the logs show:
Took 31778 ms to stop all tservers
Took 3 ms to start all tservers
Generated 100 files in 403 ms
Created table in 23334 ms
Imported into table in 601 ms
Took 3031 ms for split count to reach expected range

repo: apache/accumulo


text:
ARTEMIS-1214 Improve performance of read on Paging

For NIO and MAPPED SequentialFile based Pages uses a READ_ONLY memory mapping to perform the bulk read.

I've built an "enhanced" solution that brings mapping views of native File as a public SequentialFile feature: https://github.com/franz1981/activemq-artemis/tree/mapped_view_seq_file.
I could consider to push it only if others beside paging could benefit from having such feature implemented on SequentialFile.
@clebertsuconic @mtaylor @michaelandrepearce Makes sense?Any of you know of other points in the code that could need it?

I'm a bit concerned with isolation of concern on Page.. Why don't you bring your change to SequentialFile and use a method on SequentialFile to read it...
if reading files through mmap is that faster than NIO, lets always read using mmap... even the load on libaio would speed up, as it's using (as far as I remember).
is this the same on any env? lets just use that way always?

About the environments,yes it is.
But with libAIO I'm not sure, because libAIO doesn't rely on the page cache,so it won't benefits by exploiting it.

libaio should read a whole file on load... it could use the same mmap mechanism to read the file..
we only need writes to go through libaio, on the cases where users want to sync.

In the other branch I've linked I've a proposal about having it as a default method on SequentialFile too, but it needs some unit tests in that case IMHO....but if right now is needed only on paging it could be pushed later when other parts of Artemis will needed...

@franz1981  there are only two parts...
paging/reading.. journal/loading.. journal/printdata (which is using the load.. so I'm not counting this one).. only two

I wasn't aware we were reading from LibAIO SequenceFile..where we are using it?

journal.load... but it could use any mechanism we think it's fast at that point... (for reading that is).

In that case I can close this PR and make another one using the other branch I've posted...but I'm concerned on the use case here...Mapped reads are the fastest just because NIO/MAPPED share the same OS page cache while writing ,with ASYNCIO I need to do some testz before...
My supposition is that It will rise major page faults on each OS page read because the OS page cache is cold with ASYNCIO that doesn't use it for the writes...so the only benefit will be to avoid to instantiate the direct ByteBuffer ...is not a big improvement...
But If the read was performed with a huge direct ByteBuffer like the paging case..it could worth...wdyt?

@franz1981 @clebertsuconic  been going through this the last two days, one bit I'm unclear on is where or what logic flow are we intending this to improve things for?
Journal
Paging
I ask as if its for the latter (paging), and we're doing it to use page caching as another layer of keeping as much as possible in memory (though not on heap), then i don't think AIO is an option there, as such should probably just scope the work to NIO and MAPPED.
If its for Journal, i thought the journal was write heavy, as the read is only on broker restart, where do we really care, for the extra complexity it will cause if we had to deal with AIO? (i may have miss understood the read path for Journal btw)
Cheers
Mike

@michaelandrepearce
This PR aim to improve the performances (and memory footprint too) while performing reads while Paging when NIO (not JDBC) Paging is in charge.
In addition, the proposal of @clebertsuconic is to use this enhancement as a feature of all the SequentialFile implementations that could support it, similar to what I've done in this branch: https://github.com/franz1981/activemq-artemis/tree/mapped_view_seq_file.
The reason is to speed up the load phase of the journal, but as I've pointed out (and you too), with AIO is needed some additional tests to be sure it's worth.
Right now I'll prefer to stick to support it only with Paging.
Just thinking loud...
A different case would be if we start to use the Journal to be read (LMDB do that and AFAIK Kafka too) and using a map (a BloomFilter would be better) to store where the the records are stored in the journal.
In that case having AIO and exploiting the OS Page Cache could be a very nice idea to avoid the cache to be filled of new OS pages that doesn't have the locality required by the read load patterns.

@franz1981 agreed, I think it's sensible to do just the initial scope of paging and later revisit the other / possible future use cases. It avoids muddying the waters/risks. IMO

@michaelandrepearce @clebertsuconic I just need to push another little improvement and I think it is ready to be merged.
FYI the improvement is to recycle the ActiveMQBuffers.wrappedBuffer used to read/write the page.
I've found that the JVM is not able to allocate it on the stack if not after a huge number of iterations: with pooling it wil make the writes near to garbage free.

repo: apache/activemq-artemis

license_source: bigquery


text:
ARTEMIS-1816 OpenWire should avoid ByteArrayOutputStream lazy allocation

OpenWireMessageConverter::toAMQMessage on bytes messages is lazy
allocating a write buffer with a default size of 1024 even when
it won't be used to write anything.
It avoid an useless allocation by reducing it to new byte[0].

@clebertsuconic I need to run all the CI tests on it to be safe :+1:

@franz1981 So I think that you could avoid subclassing the ActiveMQBytesMessage if you built the properties yourself instead of calling setObjectProperty in setAMQMsgObjectProperties in the converter as that is where the trigger point is for allocating the bytes stream (see initializeWriting and its uses in ActiveMQBytesMessage).  In a manner similar to how the converter puts the body into the message using setContent you could create and marshal the properties and store them using setMarshalledProperties in the OpenWire base Message class.  The properties are marshalled using the utility class MarshallingSupport#marshalPrimitiveMap so it is possible to roll your own and then bypass that bit of the ActiveMQMessage's normal handling.

for optimizing OpenWire.. I would prefer implementing OpenWireMessage properly. There's a PR open for that already.

That would be the more ideal thing to do yes.

@franz1981 sorry.. but I -1 on this.. you ok with closing this?

I agree but given the short window until a new release i have preferred to push this (that's quite stable and safe) and taking my time to improve the other optimal way. That pr currently broke (last time I have checked) too many tests and I think it worth to make it stable without rushing.

The change as it is seems acceptable as it does reduce unnecessary allocations when ByteMessages types cross to OpenWire.  I've reviewed the changes as it relates to the OpenWire objects and the way the converter uses it and I don't see anything adverse happening from this.  As always a run through the tests would be good for some validation.

@tabish121 @franz1981 fair enough.. lets run the whole testsuite then.. then we can merge this.

@tabish121  no regressions on the testsuites.. I will let you decide about merging this or not.

Ok, I'll be "the decider".  Will check with franz to see how much it's worth to him


text:
AMBARI-23165. Install Wizard/Customise Services - Clear Filters is no…

…t working (alexantonenko)
What changes were proposed in this pull request?
Install Wizard/Customise Services - Clear Filters is not working
How was this patch tested?
manually
Please review Ambari Contributing Guide before opening a pull request.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1023/
Test PASSed.

authors: asfgit hiveww

repo: apache/ambari


text:
fix: make regex for hosts consistent with APISIX

Type of change:
Fixes #2064
Currently there is a mismatch between the regex that APISIX allowed[1] and what ApisixRoute CRD allowed. This PR makes sure that both are consistent.
More specifically - Both 1.2.3.4 and 1.2.3.4:443 should be considered valid hosts field

https://github.com/apache/apisix-ingress-controller/blob/ea6fbc95d774d9517aded455b59ce238c06c1edf/conf/apisix-schema.json#L961

[ ] Bugfix
[ ] New feature provided
[ ] Improve performance
[ ] Backport patches
[ ] Documentation
[ ] Refactor
[ ] Chore
[ ] CI/CD or Tests

What this PR does / why we need it:

Pre-submission checklist:

[ ] Did you explain what problem does this PR solve? Or what new features have been added?
[ ] Have you added corresponding test cases?
[ ] Have you modified the corresponding document?
[ ] Is this PR backward compatible? If it is not backward compatible, please discuss on the mailing list first

Is there a problem with the admin-api mode?
We need test cases to cover it.

Is there a problem with the admin-api mode? We need test cases to cover it.

The admin api is fine. The regex of the same field that admin api expects and what ingress controller expects is different. So this PR makes sure that regex is same for the same field.
Okay I'll add a test case.

Codecov Report
Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (c111c12) 36.81% compared to head (9bad35b) 36.80%.
Report is 10 commits behind head on master.

:exclamation: Current head 9bad35b differs from pull request most recent head 48b04ca. Consider uploading reports for the commit 48b04ca to get more accurate results

Files
Patch %
Lines

pkg/providers/ingress/ingress.go
0.00%
3 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2065      +/-   ##
==========================================
- Coverage   36.81%   36.80%   -0.01%
==========================================
  Files          93       93
  Lines        7863     7865       +2
==========================================
  Hits         2895     2895
- Misses       4580     4582       +2
  Partials      388      388

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

This issue needs to be fixed first for the CI to pass here https://github.com/api7/lua-resty-radixtree/issues/138

authors: AlinsRan Revolyssup codecov-commenter

repo: apache/apisix-ingress-controller


text:
How to submit posts in multiple languages?

We support content in English and Chinese now, and we need a guide on how to submit content in English when using English mode, and how to submit content in Chinese when using Chinese mode.

Done in https://apisix.apache.org/docs/general/blog/.

repo: apache/apisix-website


text:
[Docs]: SEO for blogs URI, title and content

Current State
The title and content of the blog article do not consider SEO, for example, there are no keywords such as API gateway and ingress, and the content is mainly technical.
That way, it's hard to find these if users don't search for Apache APISIX.
For example, https://apisix.apache.org/blog/2022/02/21/nacos, from URI, title to the content, there is no keyword for API gateway. Other articles have similar issues.
Desired State
We need to go through all blogs and other content completely, SEO optimization from URI, title and content
Would you like to fix this issue?
No response

Attempted to solve in https://github.com/apache/apisix-website/pull/945, finally completed in https://github.com/apache/apisix-website/pull/1141 and https://github.com/apache/apisix-website/pull/1158.


text:
fix(go/adbc/driver/snowflake): handle quotes properly

fixes #1721

I'm fixing CI so we can provide a wheel for testing.

I'm not sure this is the way to go. One, we're basically opening things up to SQL injection attacks if we don't escape/quote input. Two, it breaks the idea that what you pass in is the actual table name, not the concrete syntax.

@lidavidm Ultimately this comes down to the question of who is responsible for ensuring we avoid sql injection vs ease of use.
Because we provide this whole bulk ingestion API via options to define the table name etc, is ADBC itself (and therefore every driver) responsible for ensuring there's no sql injection? Or is the application that is using ADBC responsible for ensuring there's no injection in the value they pass for the option of the table name? Since ADBC itself allows arbitrary SQL strings it feels like it should be the application's responsibility for ensuring there's no sql injection rather than ADBC (and therefore every driver).
There's also the case that blindly adding quotes like we were doing isn't sufficient to prevent sql injection anyways and only served to create failure cases for the ingestion rather than potentially reasonable results (see here).
One possible solution could be for us to try to use the snowflake IDENTIFIER function https://docs.snowflake.com/en/sql-reference/identifier-literal which might solve the issue while preventing injection? it's still not perfect though unless we also check for un-escaped single quotes in the passed in string and forcibly escape them etc.

I feel like the injection thing is a red herring. Mainly, I think this depends on whether or not the API in question is defined as accepting a table name or a table identifier. If the former, then I would expect it to take the desired table name and do whatever is necessary to ensure a table with that name is created, including quoting if necessary. If the latter, then I would expect the identifier to already be quoted properly by the user of the API.
This distinction matters, because if the API allows me to create a table for a non-default schema (or in another database) then I have to be able to supply an identifier. If the separator is the canonical period, then I can't supply a table name containing a period without quoting it myself because anything else would be ambiguous.

if the API allows me to create a table for a non-default schema (or in another database) then I have to be able to supply an identifier

There are separate parameters for schema/catalog, so we're already down the path of name, not syntactical identifier

This is the sqlglot reference.. There is a default dialect and 20 other flavors including snowflake, postgres, etc..
https://github.com/tobymao/sqlglot/blob/main/sqlglot/dialects/dialect.py
def normalize_identifier(self, expression: E) -> E:
"""
Transforms an identifier in a way that resembles how it'd be resolved by this dialect.
    For example, an identifier like `FoO` would be resolved as `foo` in Postgres, because it
    lowercases all unquoted identifiers. On the other hand, Snowflake uppercases them, so
    it would resolve it as `FOO`. If it was quoted, it'd need to be treated as case-sensitive,
    and so any normalization would be prohibited in order to avoid "breaking" the identifier.

    There are also dialects like Spark, which are case-insensitive even when quotes are
    present, and dialects like MySQL, whose resolution rules match those employed by the
    underlying operating system, for example they may always be case-sensitive in Linux.

    Finally, the normalization behavior of some engines can even be controlled through flags,
    like in Redshift's case, where users can explicitly set enable_case_sensitive_identifier.

    SQLGlot aims to understand and handle all of these different behaviors gracefully, so
    that it can analyze queries in the optimizer and successfully capture their semantics.
    """

After talking with @lidavidm and based on the responses from everyone here, I'm gonna update this PR to have the following behavior:

We will still wrap ingesting table names in quotes, following the example of JDBC
I'll fix the behavior so it will properly escape quotes for Snowflake's dialect. We'll first do a pass and replace " with "" before wrapping the input with quotes, this should prevent things from erroring like in the examples in the linked issue.

If an option is desired to allow turning off this auto-quoting to allow a user to provide a literal, we can add that in a follow-up PR.
Does this work for everyone?

@davlee1972 we're not aiming to be a SQL dialect with ADBC, so I don't think that sqlglot is a good comparison here. we're more similar to ODBC and JDBC, which have behavior as quoted by @lidavidm's comments.

I'm not asking you to be a SQL dialect, but you might want to incorporate the dialects into the CI builds so they find the right table name..
Here's some samples..
from sqlglot import exp
from sqlglot.optimizer import optimize

table_names = {
    'lower_case | unquoted': exp.table_("dummy", quoted=False),
    'lower_case | quoted': exp.table_("dummy", quoted=True),
    'upper_case | unquoted': exp.table_("DUMMY", quoted=False),
    'upper_case | quoted': exp.table_("DUMMY", quoted=True),
    'mixed_case | unquoted': exp.table_("duMMy", quoted=False),
    'mixed_case | quoted': exp.table_("duMMy", quoted=True)
}

dialects = ['snowflake', 'postgres', 'mysql', 'tsql', 'oracle']

for dialect in dialects:
    for k, v in table_names.items():
        print("dialect: " + dialect + ", case + quoted: " + k)
        print("in: " + str(v.this))
        print("out: " + optimize(v, dialect=dialect).sql())
        print("")

Dialect results..
dialect: snowflake, case + quoted: lower_case | unquoted
in: dummy
out: "DUMMY"

dialect: snowflake, case + quoted: lower_case | quoted
in: "dummy"
out: "dummy"

dialect: snowflake, case + quoted: upper_case | unquoted
in: DUMMY
out: "DUMMY"

dialect: snowflake, case + quoted: upper_case | quoted
in: "DUMMY"
out: "DUMMY"

dialect: snowflake, case + quoted: mixed_case | unquoted
in: duMMy
out: "DUMMY"

dialect: snowflake, case + quoted: mixed_case | quoted
in: "duMMy"
out: "duMMy"

dialect: postgres, case + quoted: lower_case | unquoted
in: dummy
out: "dummy"

dialect: postgres, case + quoted: lower_case | quoted
in: "dummy"
out: "dummy"

dialect: postgres, case + quoted: upper_case | unquoted
in: DUMMY
out: "dummy"

dialect: postgres, case + quoted: upper_case | quoted
in: "DUMMY"
out: "DUMMY"

dialect: postgres, case + quoted: mixed_case | unquoted
in: duMMy
out: "dummy"

dialect: postgres, case + quoted: mixed_case | quoted
in: "duMMy"
out: "duMMy"

dialect: mysql, case + quoted: lower_case | unquoted
in: dummy
out: "dummy"

dialect: mysql, case + quoted: lower_case | quoted
in: "dummy"
out: "dummy"

dialect: mysql, case + quoted: upper_case | unquoted
in: DUMMY
out: "DUMMY"

dialect: mysql, case + quoted: upper_case | quoted
in: "DUMMY"
out: "DUMMY"

dialect: mysql, case + quoted: mixed_case | unquoted
in: duMMy
out: "duMMy"

dialect: mysql, case + quoted: mixed_case | quoted
in: "duMMy"
out: "duMMy"

dialect: tsql, case + quoted: lower_case | unquoted
in: dummy
out: "dummy"

dialect: tsql, case + quoted: lower_case | quoted
in: "dummy"
out: "dummy"

dialect: tsql, case + quoted: upper_case | unquoted
in: DUMMY
out: "dummy"

dialect: tsql, case + quoted: upper_case | quoted
in: "DUMMY"
out: "dummy"

dialect: tsql, case + quoted: mixed_case | unquoted
in: duMMy
out: "dummy"

dialect: tsql, case + quoted: mixed_case | quoted
in: "duMMy"
out: "dummy"

dialect: oracle, case + quoted: lower_case | unquoted
in: dummy
out: "DUMMY"

dialect: oracle, case + quoted: lower_case | quoted
in: "dummy"
out: "dummy"

dialect: oracle, case + quoted: upper_case | unquoted
in: DUMMY
out: "DUMMY"

dialect: oracle, case + quoted: upper_case | quoted
in: "DUMMY"
out: "DUMMY"

dialect: oracle, case + quoted: mixed_case | unquoted
in: duMMy
out: "DUMMY"

dialect: oracle, case + quoted: mixed_case | quoted
in: "duMMy"
out: "duMMy"

We expect a catalog, schema, and table name, and as others have chimed in above this is typical of database APIs. If Snowflake's quoting rules mean that the SQL identifier dummy results in the name DUMMY, then we expect the latter.
It happened to work before because we were just concatenating SQL strings internally, but this does not work for all drivers and all users - for instance, Flight SQL now has an ingestion API that works on names and not SQL identifiers. And, not all users are using sqlglot to escape names for them, nor would we expect or require them to do so; hence why these APIs all work on names. I'd be supportive of a Snowflake driver-specific option to disable this at the user's own risk, but for the purposes of this PR let's fix the quoting first.

I filed #1766 for an option to disable the quoting behavior

Can we change 1766 from disable quoting to something like allow identifiers.. This would cover both quoting and case sensitivity

@davlee1972 do you mean just changing what it is called? Or are you saying that it should functionally do more than just disable the quoting?

Just the name..

Sure that makes perfect sense to me. I'd be fine with naming it as such.

Sorry for the back and forth.. I think it should called "quoted_identifier" = True by default..
This is a common term used across database products..
A quoted identifier begins and ends with double quotation marks (").

let's shift the discussion to #1766 so that we don't lose track of it

authors: CurtHagenlocher davlee1972 lidavidm zeroshade

repo: apache/arrow-adbc


text:
Remove input schema from PhysicalExpr, move the validation logic to phyiscal expression planner

Describe the bug
https://github.com/apache/arrow-datafusion/pull/6114
The input_schema was added to the InListExpr by mistake, it cause issues during transform() rewrite process, should remove it.
To Reproduce
No response
Expected behavior
No response
Additional context
No response

I may be interested in helping with this one as it affected us upstream in IOx and I am looking for excuses to help clean up DataFUsion code

authors: alamb mingmwang

repo: apache/arrow-datafusion


text:
Update ClickBench benchmarks with DataFusion 36

Like https://github.com/apache/arrow-datafusion/issues/8789
Is your feature request related to a problem or challenge?
DataFusion 36 has been released. https://crates.io/crates/datafusion/36.0.0
It would be great to update ClickBench https://benchmark.clickhouse.com/ with runs from the latest version.
It appears we still are at 34: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/results/partitioned.json
Describe the solution you'd like
Perhaps we can follow the model of https://github.com/ClickHouse/ClickBench/pull/159(thanks @kmitchener )
Describe alternatives you've considered
No response
Additional context
No response

@kmitchener  any chance you are willing to run the benchmark numbers again for DF 36?

@alamb  @kmitchener  I went ahead and ran the benchmarks against 36.0.0 - the results are available on this branch
I've added them to the handy html from here which includes the results from releases 33 and 34 for comparison
I accidentally created a PR against the ClickBench repo.  I was hoping to get your feedback on these results first before doing that....

I have a basic question about the time being reported for the queries.  If you take a look at the shell script
https://github.com/ClickHouse/ClickBench/blob/main/datafusion/run.sh#L34
the variable RES holds the result, it's taking the text from the 7th column in the 2nd line
RES=`datafusion-cli -f $CREATE_SQL_FILE /tmp/query.sql 2>&1 | grep "Query took" | sed -n 2p | awk '{print $7}'`

If I run the cli there are two query times being reported (assuming one for creating the table and one for the query as per the note in the run.sh.
/arrow-datafusion/datafusion-cli/target/release/datafusion-cli -f create_partitioned.sql /tmp/query.sql 2>&1
DataFusion CLI v36.0.0
0 rows in set. Query took 0.034 seconds.

+---------------------+-----------+
| m                   | pageviews |
+---------------------+-----------+
| 2013-07-15T12:40:00 | 513       |
| 2013-07-15T12:41:00 | 457       |
| 2013-07-15T12:42:00 | 470       |
| 2013-07-15T12:43:00 | 468       |
| 2013-07-15T12:44:00 | 453       |
| 2013-07-15T12:45:00 | 462       |
| 2013-07-15T12:46:00 | 481       |
| 2013-07-15T12:47:00 | 458       |
| 2013-07-15T12:48:00 | 466       |
| 2013-07-15T12:49:00 | 467       |
+---------------------+-----------+
10 rows in set. Query took 0.154 seconds.

I just wanted to double-check that using the time from the 2nd line of the output for the result is correct?

@pmcgleenon second time looks correct to me

@Dandandan thanks for the confirmation, yep the result will be 0.154 in the example above

For reference, here's the clickbench PR with the update https://github.com/ClickHouse/ClickBench/pull/178

@alamb @kmitchener I went ahead and ran the benchmarks against 36.0.0 - the results are available on this branch

Thank you very much @pmcgleenon

I accidentally created a PR against the ClickBench repo. I was hoping to get your feedback on these results first before doing that....

I took a quick look at the results and in general they looked reasonable to me
One thing I noticed is that there didn't seem to be a DataFusion 36 run for single file parquet (only partitioned).
Also, I wonder if we should remove the older datafusion versions 🤔
Initial performance observation
The only query that seems to have gotten substantially faster is Q9: I think the improvement is due to https://github.com/apache/arrow-datafusion/pull/8721 from @korowa
It was hoping to see a bigger improvement due to https://github.com/apache/arrow-datafusion/pull/8827 but it seems that has not yet been released. So we'll have to try with DataFusion 37 again,
There are some queries that show slight degregation in speed -- I am not sure if that is realted to measurement variance or if we have increased our per partition file overheard or something. It would be nice to see the numbers for single file and see if they showed a similar pattern

Thanks @alamb.   I've fixed the file so that the Datafusion single file is visible for release 36.     I've included 33, 34, 36 in this file so the results can be compared easily with the previous results - the previous versions will not be visible on the Clickbench overall results.
The published clickbench file will only include the latest datafusion release (36.0.0 tag)  - this is available here
For a quick comparison here's the single file results for Datafusion 36

Yep Q9 is showing a decent improvement.  There are some queries where performance regressed from 33 to 34, and this is also still visible in 36 (e.g. Q36-Q39)

I think the numbers are good to publish.
In terms of the slowdown I agree it is likely something real (and we are tracking something similar in https://github.com/apache/arrow-datafusion/issues/8836)
Perhaps there is per-file or per-partition overhead that has increased in 34.0.0 that we haven't gotten to the bottom of. @Ted-Jiang  I wonder if this could be related to caching the file metadata -- for example I wonder if using something like  https://docs.rs/datafusion/latest/datafusion/execution/cache/cache_manager/struct.CacheManager.html would improve the performance as it would avoid some small IOs at the start of the query
It sort of feels like cheating however to cache the metadata, though I suppose it would be allowed 🤔

@alamb Sorry for the late response.

@Ted-Jiang is there some way to test via configuration setting if caching the per-file metadata would help these queries?

There are two kinds of cache in datafusion ListFilesCache and FileStatisticsCache

ListFilesCache: Before I change this into session level, i think it is already on in global level (so no config setting in df 🤣 ). it will reuse the result under the same path of list files.
FileStatisticsCache: this cache is use for CBO usage, it keeps off all the time. I add this cache in our internal system to cache FileStatistics for join selection usage.

If it turns out to make a difference, maybe we could provide a simple LRU type cache by default in DataFusion
This sounds like great,  like linked_hash_map

@alamb the Clickbench PR with the datafusion 36 updates has been merged https://github.com/ClickHouse/ClickBench/pull/178
So I think we are good here!

Thank you very much @pmcgleenon  🙏

Filed https://github.com/apache/datafusion/issues/11567 to track udating for DataFusion 40

authors: Dandandan Ted-Jiang alamb pmcgleenon


text:
MINOR: fix concat_ws corner bug

Which issue does this PR close?

Closes #2100.
Rationale for this change

Function concat_ws is not ignoring NULL in last argument and seperator text is added at the end of the result.
To Reproduce
SELECT concat_ws('|', 'a', NULL);
-- Result: a|

Expected behavior
SELECT concat_ws('|', 'a', NULL);
-- Result: a

What changes are included in this PR?

skip adding separator when NULL in last argument.
Are there any user-facing changes?

No.

Now it's failing with two NULLs at last arguments.
To Reproduce
SELECT concat_ws('|','a',NULL, NULL);
-- Result: a|

Expected behavior
SELECT concat_ws('|','a',NULL, NULL);
-- Result: a

How about
  let mut owned_string = Vec::with_capacity(args.len());
  for arg_index in 1..args.len() {
      let arg = &args[arg_index];
      if !arg.is_null(index) {
          owned_string.push(arg.value(index));
      }
  }
  owned_string.join(sep)

How about
  let mut owned_string = Vec::with_capacity(args.len()-1);
  for arg_index in 1..args.len() {
      let arg = &args[arg_index];
      if !arg.is_null(index) {
          owned_string.push(arg.value(index));
      }
  }
  owned_string.join(sep)

Could also use an iterator + flat_map + collect (into Vec<String>) for a rusty version.

@doki23 @Dandandan thanks for the advice, I'd try it

@mateuszkj @doki23 @Dandandan PTAL, thanks

@Dandandan  @houqp @alamb PTAL

authors: Dandandan WinkerDu doki23 mateuszkj


text:
fix: Sort with a lot of repetition values

Which issue does this PR close?

Closes #.
Rationale for this change
While sorting by a column with a great many repetition values, we may get non-continuous indices from a batch to output. This fails the current assumption that indices are always continuous, and make extend output wrong slices.
For example, in our test on a 1TB TPC-DS dataset, sort by inventory.inv_date_sk. we are witnessing
CompositeIndex { batch_idx: 68, row_idx: 4164 }
CompositeIndex { batch_idx: 68, row_idx: 4293 }
CompositeIndex { batch_idx: 68, row_idx: 4294 }
CompositeIndex { batch_idx: 68, row_idx: 4295 }
CompositeIndex { batch_idx: 68, row_idx: 4296 }
CompositeIndex { batch_idx: 68, row_idx: 4297 }

using start_row_idx, 4164 with length 6 is wrong, since rows with indices from 4165 to 4169 should be output elsewhere, and it's likely 4296 or 4297 are pointing to the next sort key.
What changes are included in this PR?

Bugfix along with the minimum parquet test file that could reproduce the bug.
Are there any user-facing changes?
No.

cc @richox

Good to have this fixed!
I wonder if we could better switch to use stable sort in the arrow rs lexsort implementation instead (offering both stable/unstable sort), for performance.

I wonder if we could better switch to use stable sort in the arrow rs lexsort implementation instead (offering both stable/unstable sort), for performance.

@Jimexist  added unstable sorting here: https://github.com/apache/arrow-rs/pull/552
Maybe we could add the choice of which to use

authors: Dandandan alamb yjshen


text:
fix(flightsql): remove Any encoding of DoPutPreparedStatementResult

Which issue does this PR close?

Closes #5731.
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

LGTM for the prepared statement result (i think the title just needs to be updated). Im also not sure if there are other payloads that rust wraps in any which other languages dont - I havent had the time to compare those.

LGTM for the prepared statement result (i think the title just needs to be updated). Im also not sure if there are other payloads that rust wraps in any which other languages dont - I havent had the time to compare those.

Thanks @matthewmturner  -- I updated the title of the PR
Given this PR is clearly more compatible than the current code, I think it is good to go.
If/when we evaluate the other methods and find additional issues, let's file additional tickets

authors: alamb erratic-pattern matthewmturner

repo: apache/arrow-rs


text:
[C#] Improve Field interoperability with C# Type

Describe the enhancement requested
Make builders from native c# types to arrow types
int = new Int32Type()
string = new StringType()
Component(s)
C#

Need to split the need

repo: apache/arrow


text:
[C++] Threads::Threads dependency is missing for static linking

Describe the bug, including details regarding any error messages, version, and platform.
This is caused by #39824.
example-cpp-minimal-build-static:
https://github.com/ursacomputing/crossbow/actions/runs/8210651831/job/22458477613#step:3:1523
[100%] Linking CXX executable arrow-example
/usr/bin/ld: /usr/local/lib/libarrow.a(file.cc.o): in function `void std::lock<std::unique_lock<std::mutex>, std::unique_lock<std::mutex>>(std::unique_lock<std::mutex>&, std::unique_lock<std::mutex>&)':
file.cc:(.text._ZSt4lockISt11unique_lockISt5mutexES2_JEEvRT_RT0_DpRT1_[_ZSt4lockISt11unique_lockISt5mutexES2_JEEvRT_RT0_DpRT1_]+0x21): undefined reference to `pthread_mutex_trylock'
/usr/bin/ld: /usr/local/lib/libarrow.a(cancel.cc.o): in function `arrow::(anonymous namespace)::SignalStopState::RegisterHandlers(std::vector<int, std::allocator<int> > const&)':
cancel.cc:(.text+0x133e): undefined reference to `pthread_create'
...

Component(s)
C++

Issue resolved by pull request 40433
https://github.com/apache/arrow/pull/40433


text:
[C++][Parquet] Strip debug symbols from libparquet libraries in release builds by default

This will reduce the size of shipped binaries – debug symbols can always be restored using $PARQUET_CXXFLAGS
Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wesm) / @wesm
Note: This issue was originally created as PARQUET-961. Please see the migration documentation for further details.

Wes McKinney / @wesm:
PR: https://github.com/apache/parquet-cpp/pull/305

Wes McKinney / @wesm:
Issue resolved by pull request 305
https://github.com/apache/parquet-cpp/pull/305


text:
ARROW-14153: [C++][Dataset] Add support for batch_size in the ORC Scanner

This PR includes the changesets from #11763 and adds some minor changes.

Closing now that #11763 is merged.


text:
ARROW-15523: [Python] Support for Datasets as inputs of Joins

This supports providing Datasets as inputs of the join operation.
Until https://issues.apache.org/jira/browse/ARROW-15526 is completed, the join will temporary return a Table in case of Datasets too.

@jorisvandenbossche do you have a chance to review this one? I already have an open branch depending on it, so I hope to be able to address feedbacks and merge it as quickly as possible.

Hmm, apparently you get strange errors in the merge script if two people try to merge at the same time ;)

authors: amol- jorisvandenbossche


text:
ARROW-497: Integration harness for streaming file format

These tests pass locally for me. Thanks @nongli for this!

LGTM

thanks

authors: nongli wesm


text:
GH-33610: [Dev] Remove ARROW from title check to only allow GH issues

Now that all issues from JIRA are migrated to GitHub we should always use the GH corresponding issue on the PR title

Closes: #33610

The PR template also reference the ARROW- syntax

Example of output for merge_script using an ARROW- related issue:
$ ./dev/merge_arrow_pr.sh
ARROW_HOME = /home/raulcd/code/arrow/dev
ORG_NAME = apache
PROJECT_NAME = arrow
Which pull request would you like to merge? (e.g. 34): 33603
Traceback (most recent call last):
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 749, in <module>
    cli()
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 716, in cli
    pr = PullRequest(cmd, github_api, ORG_NAME, jira_con, pr_num)
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 487, in __init__
    self.issue = self._get_issue()
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 524, in _get_issue
    self.cmd.fail("PR titles with ARROW- prefixed tickets on JIRA "
  File "/home/raulcd/code/arrow/dev/merge_arrow_pr.py", line 439, in fail
    raise Exception(msg)
Exception: PR titles with ARROW- prefixed tickets on JIRA are unsupported, update the PR title from ARROW-16921. Possible GitHub id could be: GH-32240

authors: assignUser raulcd


text:
MINOR: [Docs] Add C++/Run-End Encoded to the implementation status table

C++ passes the integration tests and has types for REE arrays.  I believe it is fair to say that C++ supports REE.

CC @felipecrv


text:
ARROW-7103: [R] Various minor cleanups

Remove covr from Suggests so that it is not installed in CI; add a note to the README about how to use it (following ARROW-7067)
Add a note to NEWS about the Datasets work that was just merged (following ARROW-6340)
Remove suggestion to install_github() because it's unlikely to work (based on user reports)
Move setting of ARROW_R_CXXFLAGS to before where a debug message prints the flags being used
Remove boost_filesystem from lib flags for macOS and Windows build scripts (following ARROW-6743)

@ursabot crossbow submit macos-r-autobrew


text:
ARROW-10830 [Rust] avoid hard crash in json reader

return parsing error to caller instead

LGTM

authors: Dandandan houqp


text:
AVRO-2985: upgrade spotless-maven-plugin

Upgrade spotless-maven-plugin from v1.3.0 to v2.6.1 in the attempt to solve the current travis-ci build.
Needs work
Jira

[x] My PR addresses AVRO-2985 issue:

https://issues.apache.org/jira/browse/AVRO-2985

Tests

[x] My PR does not need testing as it just upgrades an internal dependency.

At least Apache Avro completes :-) , and now it fails on:
[INFO] Apache Avro Java ................................... SUCCESS [  8.959 s]
[INFO] Apache Avro ........................................ SUCCESS [01:10 min]
[INFO] Apache Avro Compiler ............................... FAILURE [  0.675 s]

with:
[ERROR] Failed to execute goal on project avro-compiler: Could not resolve dependencies for project org.apache.avro:avro-compiler:bundle:1.11.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.velocity:velocity-engine-core:jar:2.2, org.apache.ant:ant-launcher:jar:1.10.9: Could not transfer artifact org.apache.velocity:velocity-engine-core:jar:2.2 from/to central (https://repo.maven.apache.org/maven2): Connection reset -> [Help 1]

velocity-engine-core is already at the latest 2.2:
https://mvnrepository.com/artifact/org.apache.velocity/velocity-engine-core
Wondering if another option is to introduce repository in the .m2 profile ?!?

Another way is to have in build.sh a mvn bash function (as alternative to an alias), that would use travis_retry:
if [ -z "$var" ]; then
  function mvn() {
    travis_retry mvn
  }
fi

Hello!  I just noticed this PR after creating and merging AVRO-2994 :/  I have no idea how I missed your work, since I was searching for the fix as well!
I solved the problem by upgrading maven to 3.6.3 in our build tool.  If I understand correctly, your fix would allow it to build correctly even with older versions of maven?  I like the sound of that!
travis_retry (as well as the TRAVIS_* environment variables) are not available in the docker image.  We could pass them, for example, or roll our own retry function.  I created AVRO-2992 to track that problem (which is slightly different than the spotless update issue.)

Closing as obsolete!
Avro already uses spotless-m-p:2.11.1 and there are no issues at TravisCI!

authors: RyanSkraba azbarcea martin-g

repo: apache/avro


text:
with_metadata not working in Python SDK ReadFromKafka

Here's the error generated when with_metadata is set as True in ReadFromKafka. It seems the SchemaCoder is not matching with KafkaConsumer record which has an extra timestamp_type at position 4 which is missing in the below SchemaCoder. Please check.
generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Unable to encode element 'org.apache.beam.sdk.io.kafka.KafkaIO$ByteArrayKafkaRecord@45b2487a' with coder 'SchemaCoder<Schema: Fields: Field{name=topic, description=, type=STRING NOT NULL, options={{}}} Field{name=partition, description=, type=INT32 NOT NULL, options={{}}} Field{name=offset, description=, type=INT64 NOT NULL, options={{}}} Field{name=timestamp, description=, type=INT64 NOT NULL, options={{}}} Field{name=key, description=, type=BYTES, options={{}}} Field{name=value, description=, type=BYTES, options={{}}} Field{name=headers, description=, type=ARRAY<ROW<key STRING NOT NULL, value BYTES NOT NULL> NOT NULL>, options={{}}} Field{name=timestampTypeId, description=, type=INT32 NOT NULL, options={{}}} Field{name=timestampTypeName, description=, type=STRING NOT NULL, options={{}}} Encoding positions: {headers=6, timestampTypeName=8, partition=1, offset=2, topic=0, value=5, key=4, timestamp=3, timestampTypeId=7}

Is this part of 2.51.0 Release? The reason I'm asking because I don't see this issue in the list.
https://github.com/apache/beam/milestone/15

repo: apache/beam


text:
[BEAM-4371] (do-not-merge-testing) Namespace shares the same name between Jenkins Performance test jobs

A new name for namespace now is created in a triggered Jenkins job instead of in common_job_properties. This will eliminate possibility to use same namespace between same type of performance test when seed job wasn't triggered yet.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ x  ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run seed job

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run seed job

Run seed job

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run seed job

Run Java JdbcIO Performance Test

Run seed job

Run Java JdbcIO Performance Test

Run seed job

Run seed job

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java JdbcIO Performance Test

Run Java MongoDBIO Performance Test

Run Java MongoDBIO Performance Test

Run Java HadoopInputFormatIO Performance Test

Run Java JdbcIO Performance Test

I triggered all tests this affects - looks ok.
@chamikaramj @jasonkuster  Would you be interested in reviewing this? Thanks!

LGTM

authors: chamikaramj kkucharc


text:
[BEAM-3573] Move GCPIO test utilities to main jar

These test utilities are used by clients to test their code. They are not isolated to the test suite jar.

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

There is some hope that eliminating bad test jar deps might mitigate https://issues.apache.org/jira/browse/BEAM-5035, https://issues.apache.org/jira/browse/BEAM-5207, https://issues.apache.org/jira/browse/BEAM-5116. In any case it is the right direction so no harm in moving to find out how much it helps.

I did little to no digging beyond disabling parallel tests to fix flakes. This looks like it might fix the root cause. LGTM.

run java postcommit

The failure in postcommit is the same from #6864. We have tests in the Dataflow worker codebase that are not thread safe. I will drop that commit, then this should be g2g.

run java postcommit

authors: apilloud kennknowles


text:
Work around issue in dill for wordcount example save_main_session

This change works around a bug in the dill library (https://github.com/uqfoundation/dill/issues/300), where classes fail to unpickle in the main session if its super() is referenced.  This change modifies the wordcount example to not use such a reference.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Run Python PostCommit

Run Python PostCommit

@robertwb Thanks, I incorporated your suggestion.

Thanks, I removed it.


text:
Add empty file in order to create PR

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Python ReleaseCandidate


text:
[DO NOT MERGE]Start snapshot build for release process

Run Gradle Publish

Run Gradle Publish

Run Seed Job


text:
BIGTOP-3346: Introduce Ruby to Bigtop Toolchain

As discussion on PR #620 for BIGTOP-3319,
this PR is to introduce Ruby to Bigtop Toolchain for Logstash support.

I prefer to have ruby from distro or rvm.

@oflebbe really thanks for your comments.
We do need the ruby-related facilities such as ruby rumtime, rake, gem and bundle to build logstash.
And for the Arm64 dependency issues, the ruby-2.4 is the latest version that works with logstash.
But ruby version is 2.0 in Centos 7 and 2.3 in ubuntu16.04.
When leveraging rvm to install ruby, rvm would pull the ruby binaries and spend less time than  compiling ruby directly on x86_64.
But on Arm64, no ruby binaries are available and rvm would also download source code and compile ruby.
IMHO, our Bigtop should support different kind of distros and platforms(x86_64/arm64).
If ruby-2.4 were downloaded and compiled directly , it could guarantee that ruby-related facilities would be properly installed on the various distros and platforms though it may cost extra time just on x86 when building bigtop_toolchain. WDYT?
Thanks!

Regarding centos7 we should use https://www.softwarecollections.org/en/scls/rhscl/rh-ruby26/ . This is a redhat provided service for RHEL and Centos.
Ubuntu 16.04 should be removed, since it is outdated. We should support 18.04 instead.

@guyuqi @oflebbe I think we can leverage Ruby on those platforms which already have ruby that meet the requirement. I'm guessing that most of the user just choose one arch X platform. If we compile our own dependencies, then user much do so in there env, which is not that ideal, does this make sense?

Just my two cents, how about we only support Logstash on the distros that provides Ruby 2.4+ natively and doesn't on relatively old ones, i.e., CentOS 7, Debian 9 (which provides Ruby 2.3.3) and Ubuntu 16.04?

I agree @sekikn . All the platforms you are mentioning should be deprecated and therefore it should be ok to not provide new functionality.

I agree @sekikn . All the platforms you are mentioning should be deprecated and therefore it should be ok to not provide new functionality.

Yup, it makes sense. Thanks, @oflebbe @sekikn @evans-ye

From the disscussion above, this PR would be abandoned.
For logstash, the ruby native package would not be installed in bigtop_toolchain.
Just support Logstash on the distros that provides Ruby 2.4+ for runtime. (distros: Ubuntu-18.04, Debian-10, Centos-8 and Fedora-31)

authors: evans-ye guyuqi oflebbe sekikn

repo: apache/bigtop


text:
如何通过pprof工具生成火焰图？

在这篇文章（https://brpc.apache.org/zh/docs/builtin-services/cpu_profiler/），介绍了如何进行cpu profile。
但是我折腾了2天时间，没搞定如何用pprof工具生成火焰图。
按照文章里的指示，我得到了类似红框中的文件。

但是如何把这个文件转变成火焰图呢？网上找了很多资料，都没搞定。
我希望是能在linux服务器上直接生成一个svg文件，然后下载到本地可以用浏览器查看的那种形式。

可以考虑用golang带的pprof，可以有个交互式的web，直接就能看到flamegraph
本地安装golang，之后执行 go tool pprof -http 127.0.0.1:web_port http://brpc_ip:internal_port/pprof/profile?seconds=30

linux服务器上并没有web浏览器

可以考虑用golang带的pprof，可以有个交互式的web，直接就能看到flamegraph
本地安装golang，之后执行 go tool pprof -http 127.0.0.1:web_port http://brpc_ip:internal_port/pprof/profile?seconds=30

linux服务器上没有web浏览器，看不了网页。

@617076674 这个只是启动一个web服务，可以用本地机器访问，和--web参数不一样，可以看看pprof的帮助文档

@617076674 这个只是启动一个web服务，可以用本地机器访问，和--web参数不一样，可以看看pprof的帮助文档

有没有能直接生成svg文件的方式呢？

https://brpc.apache.org/zh/docs/builtin-services/cpu_profiler/#火焰图
设置好FlameGraph路径后，在cpu内置服务可以选择火焰图展示profile结果。

authors: 617076674 chenBright lorinlee

repo: apache/brpc


text:
Overlap whitelist broken with multiple projects

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Jun 6, 2020, 09:34
The overlap whitelist feature is not properly functional when used in conjunction with multiple projects, as the algorithm is based on simple element names and not element full paths.
This will break down in scenarios where multiple junctioned projects bear the same element names.
Consider the current approximate algorithm in Element.stage_dependency_artifacts(), observe the XXX comments I added in the fragment below:

    for dep in self.dependencies(scope):
        result = dep.stage_artifact(sandbox, path=path, include=include, exclude=exclude, orphans=orphans)
        if result.overwritten:
            for overwrite in result.overwritten:
                # Completely new overwrite
                if overwrite not in overlaps:
                    # Find the overwritten element by checking where we've
                    # written the element before
                    for elm, contents in files_written.items():
                        if overwrite in contents:
                            #
                            # XXX Here we take note of the Element's name, local to it's project
                            #
                            overlaps[overwrite] = [elm, dep.name]
                else:
                    overlaps[overwrite].append(dep.name)
        files_written[dep.name] = result.files_written

        if result.ignored:
            ignored[dep.name] = result.ignored

    if overlaps:
        overlap_warning = False
        warning_detail = "Staged files overwrite existing files in staging area:\n"
        for f, elements in overlaps.items():
            overlap_warning_elements = []
            # The bottom item overlaps nothing
            overlapping_elements = elements[1:]
            for elm in overlapping_elements:
                #
                # XXX Here we use `Element.search()` with a project-local element name,
                #     which may not be unique in the dependency graph.
                #
                element = cast(Element, self.search(scope, elm))
                if not element.__file_is_whitelisted(f):
                    overlap_warning_elements.append(elm)
                    overlap_warning = True

            warning_detail += _overlap_error_detail(f, overlap_warning_elements, elements)

        if overlap_warning:
            self.warn(
                "Non-whitelisted overlaps detected", detail=warning_detail, warning_token=CoreWarnings.OVERLAPS
            )

In GitLab by [Gitlab user @tristanvb] on Jun 6, 2020, 09:34
marked this issue as related to #931

In GitLab by [Gitlab user @tristanvb] on Jun 6, 2020, 09:34
marked this issue as related to #931

In GitLab by [Gitlab user @tristanvb] on Jun 6, 2020, 09:35
changed the description

In GitLab by [Gitlab user @tristanvb] on Jun 6, 2020, 09:35
changed the description

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:16
mentioned in commit fbfa12c10578c58baa73f1d0fb5a988f039b7fdf

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:16
mentioned in commit fbfa12c10578c58baa73f1d0fb5a988f039b7fdf

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:17
mentioned in commit 14e3fb72eed094ba52cec3dfd6cb428cb8c18d94

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:17
mentioned in commit 14e3fb72eed094ba52cec3dfd6cb428cb8c18d94

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:18
mentioned in merge request !2060

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:18
mentioned in merge request !2060

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:44
mentioned in commit 399087e30278f2653d1a8f6d0f49ab5a9d290a9d

In GitLab by [Gitlab user @tristanvb] on Sep 7, 2020, 07:44
mentioned in commit 399087e30278f2653d1a8f6d0f49ab5a9d290a9d

In GitLab by [Gitlab user @marge-bot123] on Sep 7, 2020, 08:50
mentioned in commit 27daa281a6d4ed8b5659e56a12452d28175f9b4b

In GitLab by [Gitlab user @marge-bot123] on Sep 7, 2020, 08:50
mentioned in commit 27daa281a6d4ed8b5659e56a12452d28175f9b4b

repo: apache/buildstream


text:
Builds can fail depending on host resources, some control is desireable

See original issue on GitLab
In GitLab by [Gitlab user @jjardon] on Jan 10, 2018, 15:14
In some scenarios, builds can fail depending on how much resources they have available. This is worsened when coupled with fault intolerant tooling, such as compilers which abort at the first sight of an OOM condition.
A recently observed case is when building LLVM (which manages to link many large binaries in parallel). In this case the build was working fine with a reasonable amount of available RAM, but begins to fail when given more CPUs, triggering more parallel linking and reaching the OOM condition.
Various approaches are possible to address the load here. A declarative approach might include some additional user configuration to describe the capabilities of the host, and / or some extension to the .bst format to allow one to describe the resource consumption characteristics of a given build (for instance, one might be able to express the nature of the LLVM build in a llvm.bst file so as to inform BuildStream about the per-job memory requirements of the element).
None of the approaches I've come up with so far are in any way attractive, though; open to suggestions.

In GitLab by [Gitlab user @tristanvb] on Jan 15, 2018, 06:09
changed title from {-max-jobs is not part of the cache key: Think in a way on how to handle builds that can be build in 6 cores but not in 8, because memory exhaustion or other reasons-} to {+Builds can fail depending on host resources, some control is desireable+}

In GitLab by [Gitlab user @tristanvb] on Jan 15, 2018, 06:09
changed the description

In GitLab by [Gitlab user @tristanvb] on Mar 9, 2019, 09:03
marked this issue as related to #942

In GitLab by [Gitlab user @abderrahimk] on May 24, 2019, 07:59
marked this issue as related to #1033

In GitLab by [Gitlab user @cs-shadow] on Oct 10, 2019, 13:21
[Gitlab user @jjardon] ping :)
What do you think needs to happen here? Did !1473 help with this?

In GitLab by [Gitlab user @tristanvb] on Jun 1, 2020, 08:43
marked this issue as related to #1333


text:
Allow to remote debug the Operator

Since the code refactoring for Camel-K 2, it is much harder to debug the operator, especially on MacOS. The goal of this task is to provide a way to remote debug the operator in order to be closer to the target environment and to be local OS agnostic.

I don't see any doc about how to debug the operator with Minikube. I only see that https://camel.apache.org/camel-k/next/contributing/developers.html#debugging but it seems to be incomplete as it doesn't explain how to install camel-k without the operator like we had before in doc page that has been removed.
When I try to launch the operator like I did with v1, I meet many problems due to the fact that it assumes the location of different programs like mvnw.
For example:
{"level":"info","ts":1687796092.647079,"logger":"camel-k.maven.build","msg":"Executed command: /bin/cp --recursive /usr/share/maven/mvnw/. ."}
{"level":"info","ts":1687796092.6537018,"logger":"camel-k.maven.build","msg":"cp: illegal option -- -"}
{"level":"info","ts":1687796092.653837,"logger":"camel-k.maven.build","msg":"usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file target_file"}
{"level":"info","ts":1687796092.653853,"logger":"camel-k.maven.build","msg":"       cp [-R [-H | -L | -P]] [-fi | -n] [-aclpsvXx] source_file ... target_directory"}

Let's say that someone re-adapts the code to make it work as before, TBH, in practice who cares if the operator works properly on the local machine? What is important is that the operator works properly on its target environment which is at the end a docker container in a pod.
So the idea of my proposal is to be able to remote debug directly from the cluster, WDYT? I have already something working, I can push branch to show you the idea if you want

Yeah, that would be perfect. The idea of removing the "support" of local running operator it's exactly that one, ie, the fact that it's difficult to support the operator running both in the cluster and in a local machine. If you manage to have the cluster debugging in the cluster it would be the perfect solution IMO. I'll have a look at the PR shortly, thanks a lot!

FWIW, I would be very cautious about removing support for local running operator as it would make the whole development process a lot slower and it would be yet another barrier for newcomers as opposed of may other operators I'm aware of.
Briefly looking at the root issue it seems that the major barrier is the process of building the builder image which as previously discussed should happen only in rare cases ad should be provided out of the box so my recommendation would be to add support for remote debugging but also trying to fix the root issue instead of removing local development support

This PR doesn't remove anything, it only adds a new way to debug the operator.

I was mainly looking at the discussion in this thread where I read:

Let's say that someone re-adapts the code to make it work as before, TBH, in practice who cares if the operator works properly on the local machine?

The idea of removing the "support" of local running operator it's exactly that one, ie, the fact that it's difficult to support the operator running both in the cluster and in a local machine

If the plan is only to add remote debugging, that's fine but the discussion was IMHO heading also to other discussions. It would be nice to open some issues about what does not work when running the operator locally and explore options to address them.

I agree that it would be nice to have full local support as it shows the business logic is very well encapsulated and easy to debug. However it's not easy to maintain full compatibility with Kubernetes (not to talk about the various K8S flavors) and a local execution as during the development of the operator we are making always the assumption we are in a K8S environment. This dynamic builder stuff is just one (and at this stage the most visible) limitation, but we have much more.
To me the point is that the operator needs anyhow a Kubernetes cluster to run (watch a namespace, CRDs, etc...). Running a Camel K Operator pod is fairly simple and limited in resource consumption. In my experience, I've found much easier to debug in a cluster (I use a local one) than when I used to run the operator binary in my machine.
Said that, I understand it's a matter of habits and tastes, so, nothing to argue against the possibility to support the local running.

to be clear, local run to me means running the operator outside the operator cluster, ideally as the operator talks to the k8s client, there should not be much difference in running inside or outside the cluster
I am missing something?

I think there are a couple of things to take into account as well. The CRDs needs to be installed. It happens to me several time that while debugging locally I forgot to refresh the CRDs. With an installation this is not happening. The access to registry or to maven repos sometimes is flaky. Then, talking about the needs to have a local set of runtime dependencies. The installation procedure takes care to bundle and refresh, so you always get the latest one available. If you run locally you need to remind to point to your local repository. I think they are little details, but in my experience enough to decide to always debug with the installer.

FWIW, I would be very cautious about removing support for local running operator as it would make the whole development process a lot slower and it would be yet another barrier for newcomers as opposed of may other operators I'm aware of.

+1000

@claudio4j What do you use as cluster? Minikube?

This is the launch.json
        {
          "name": "Camel K Operator - cluster",
          "type": "go",
          "debugAdapter": "dlv-dap", // `legacy` by default
          "request": "attach",
          "mode": "remote",
          "port": 4040,
          "host": "127.0.0.1", // can skip for localhost
          "trace": "verbose",
          "showLog": true,
          "logOutput": "dap",
          "substitutePath": [
            { "from": "${workspaceFolder}/cmd/kamel/main.go", "to": "/usr/local/bin/kamel" },
          ]
      }

I didn't spend much time investigating.

Try with this configuration
{
            "name": "Remote Debug",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "port": 4040,
            "host": "localhost",
            "cwd": "${workspaceFolder}",
            "trace": "verbose"
        }

It worked, thanks Nicolas.

The documentation supporting the local operator execution will be back as we have now removed the dynamic builder container generation.

authors: claudio4j essobedo johnpoth lburgazzoli squakez

repo: apache/camel-k


text:
Upgrade Google Libraries BOM to version 26.28.0

Description

Target

[x] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[x] I checked that each commit in the pull request has a meaningful subject line and body.

[x] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

/component-test google

repo: apache/camel


text:
Splunk added validateCertificates parameter

fixes https://issues.apache.org/jira/browse/CAMEL-20393
It is not possible to override ssl configuration of splunk client (the ssl context is created by static method in the client without a way of overriding or customizing).
On the other hand, the client contains a parameter validateCertificates, which is usable for the testing with the real instance,
This PR adds parameter validateCertificates for the component.
Description

Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

Do I need to run any special profile to get generated the missing change?
diff --git a/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt b/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt
index fa134a41..fc8b3a93 100644
--- a/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt
+++ b/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt

Do I need to run any special profile to get generated the missing change?
diff --git a/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt b/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt
index fa134a41..fc8b3a93 100644
--- a/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt
+++ b/dsl/camel-kotlin-api/src/generated/kotlin/org/apache/camel/kotlin/components/SplunkUriDsl.kt

Hm ... you shouldn't need anything else. It should be generated as part of the normal build.

I build whole project several times (also tried formatting , ...) but no success.

authors: JiriOndrusek orpiske


text:
[HOTFIX] Support both listfile() and listfile(maxCount) in InsertStag…

…e flow
Why is this PR needed?
Flink writes files to obs with setting the filename in order of time, when loading stages files to carbondata, files are list and loaded with the parameter "maxcount" which will return "maxcount" files with smaller filename, in the other words, "maxcount" files with earliest generation time. but in same senario, it is possible that the filename of stage files is not in order of time, a switch is used here to judge whether to list files with specify batch size, or just list all files in the stage dir.
What changes were proposed in this PR?
A swith "CARBON_STAGE_FILENAME_IS_IN_ORDER_OF_TIME" is added.
Does this PR introduce any user interface change?

No

Is any new testcase added?

No

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/296/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2000/

@marchpure Please rebase and fix the comments

repo: apache/carbondata


text:
CASSNDRA-18780 Fixed ConnectionBurnTest fails with NullPointerException

Thanks for sending a pull request! Here are some tips if you're new here:

Ensure you have added or run the appropriate tests for your PR.
Be sure to keep the PR description updated to reflect all changes.
Write your PR title to summarize what this PR proposes.
If possible, provide a concise example to reproduce the issue for a faster review.
Read our contributor guidelines
If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:
<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

@belliottsmith I've addressed your comments, please, take a look again :-)

@belliottsmith Hi Benedict, I think we are really close to fixing it! :-) Would you mind taking one last look?

repo: apache/cassandra


text:
Add UEFI support

This adds a new boolean parameter to the cloudstack_instance resource, called "uefi". When set, the instance will boot in UEFI/Legacy mode.

@vdombrovski whaouuuuuu : awesome, thanks for this; it's very usefull.

authors: bragonznx vdombrovski

repo: apache/cloudstack-terraform-provider


text:
failure in test_affinity_groups_projects

test_affinity_groups_projects started failing on several PRs:
+-----------------------+--------------------+---------+-----------------------+
|         Test          |       Result       |  Time   |       Test file       |
+=======================+====================+=========+=======================+
| ContextSuite context= | builtins.Exception | 203.734 | test_affinity_groups_ |
| TestDeployVMAffinityG |                    |         | projects              |
| roups>:teardown       |                    |         |                       |
+-----------------------+--------------------+---------+-----------------------+

ISSUE TYPE

Bug Report

COMPONENT NAME

Marvin

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Looks like is this fixed @DaanHoogland ?

authors: DaanHoogland rohityadavcloud

repo: apache/cloudstack


text:
api: rename ListUsageRecords file name to ListUsageRecordsCmd

Description

Refactor file name to be consistent with the API name.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[x] Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):
How Has This Been Tested?

@rhtyd @rafaelweingartner @GabrielBrascher any thoughts on this while we are here?

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2599

@blueorangutan test

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan test

@dhlaluku a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-3395)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30881 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3151-t3395-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 66 look OK, 4 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

ContextSuite context=TestTemplateHierarchy>:setup
Error
1519.98
test_accounts.py

test_04_extract_Iso
Failure
1.08
test_iso.py

test_04_extract_template
Failure
1.08
test_templates.py

test_06_download_detached_volume
Failure
10.42
test_volumes.py

All OK, merging this since master is unfrozen now.

authors: blueorangutan borisstoyanov dhlaluku rhtyd


text:
kvm: upgrade libvirt-java to v0.5.3

Reference: https://gitlab.com/libvirt/libvirt-java/-/issues/6
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Minor nit - jar is at https://libvirt.org/maven2/org/libvirt/libvirt/ but not sync'd yet on https://mvnrepository.com/artifact/org.libvirt/libvirt

@blueorangutan package

UI build: :heavy_check_mark:
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6472 (SL-JID-1780)

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 3600

@GabrielBrascher not necessary, if the upstream project publishes their own repo, we can add it in pom.xml.

@blueorangutan test

Trillian test result (tid-4344)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37111 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6472-t4344-kvm-centos7.zip
Smoke tests completed. 98 look OK, 0 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File

authors: blueorangutan rohityadavcloud


text:
Add GenericArrayType support to TypeUtils.containsTypeVariables

Currently, org.apache.commons.lang3.reflect.TypeUtils#containsTypeVariables doesn't contain a special case for GenericArrayType like E[], so if the passed argument is a GenericArrayType instance containing type variables, this method will return false.

Coverage decreased (-0.02%) to 95.323% when pulling 138878b3609c65fe01c57e96f41754b806016404 on dimitrovchi:master into 1d037da2a85c920af7ab8aa96fa7529d5b4584b7 on apache:master.

Please look at the CI build. There was an error during artifact fetching:
[ERROR] Failed to execute goal org.codehaus.mojo:clirr-maven-plugin:2.8:check (default-cli) on project commons-lang3: Execution default-cli of goal org.codehaus.mojo:clirr-maven-plugin:2.8:check failed: Plugin org.codehaus.mojo:clirr-maven-plugin:2.8 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.bcel:bcel:jar:6.0 from/to google-maven-central (https://maven-central.storage-download.googleapis.com/repos/central/data/): GET request of: org/apache/bcel/bcel/6.0/bcel-6.0.jar from google-maven-central failed:
resulting in overall build failure. Is there a possibility to rerun checks without committing anything else?

resulting in overall build failure. Is there a possibility to rerun checks without committing anything else?

Restarted in Travis interface the job that failed (JDK: openjdk11 Java).

I've added the test: https://github.com/apache/commons-lang/pull/437/files
Please look at CI logs again: the logs show checkstyle rules violations in Functions.java but I didn't change that file:
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[148] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[148,59] (javadoc) JavadocMethod: Expected @param tag for 'pRunnable'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[149,5] (whitespace) FileTabCharacter: File contains tab characters (this is the first instance).
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[161] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[161,20] (javadoc) JavadocMethod: Expected @param tag for '<I>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[161,64] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[161,68] (javadoc) JavadocMethod: Expected @param tag for 'pConsumer'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[174] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[174,20] (javadoc) JavadocMethod: Expected @param tag for '<O>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[174,64] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[174,68] (javadoc) JavadocMethod: Expected @param tag for 'pCallable'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,20] (javadoc) JavadocMethod: Expected @param tag for '<I1>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,22] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,23] (javadoc) JavadocMethod: Expected @param tag for '<I2>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,40] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,79] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,82] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[187,86] (javadoc) JavadocMethod: Expected @param tag for 'pConsumer'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,20] (javadoc) JavadocMethod: Expected @param tag for '<I>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,21] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,22] (javadoc) JavadocMethod: Expected @param tag for '<O>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,35] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,68] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,70] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[200,74] (javadoc) JavadocMethod: Expected @param tag for 'pFunction'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,20] (javadoc) JavadocMethod: Expected @param tag for '<I1>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,22] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,23] (javadoc) JavadocMethod: Expected @param tag for '<I2>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,25] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,26] (javadoc) JavadocMethod: Expected @param tag for '<O>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,42] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,45] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,83] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,86] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,88] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[213,92] (javadoc) JavadocMethod: Expected @param tag for 'pFunction'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[226] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[226,20] (javadoc) JavadocMethod: Expected @param tag for '<I>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[226,67] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[226,71] (javadoc) JavadocMethod: Expected @param tag for 'pPredicate'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,20] (javadoc) JavadocMethod: Expected @param tag for '<I1>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,22] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,23] (javadoc) JavadocMethod: Expected @param tag for '<I2>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,41] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,82] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,85] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[239,89] (javadoc) JavadocMethod: Expected @param tag for 'pPredicate'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[252] (javadoc) JavadocMethod: Expected @return tag.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[252,20] (javadoc) JavadocMethod: Expected @param tag for '<O>'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[252,64] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[252,68] (javadoc) JavadocMethod: Expected @param tag for 'pSupplier'.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[399,21] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/main/java/org/apache/commons/lang3/Functions.java:[399,67] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[42,15] (imports) UnusedImports: Unused import - org.junit.jupiter.api.Assertions.assertFalse.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[171,55] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[171,96] (blocks) LeftCurly: '{' at column 96 should have line break after.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[180,9] (whitespace) FileTabCharacter: File contains tab characters (this is the first instance).
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[214] (regexp) RegexpSingleline: Line has trailing spaces.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[260] (regexp) RegexpSingleline: Line has trailing spaces.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[260,98] (blocks) LeftCurly: '{' at column 98 should have line break after.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[308,41] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[308,49] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[312,33] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[357,42] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[357,52] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[357,60] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[361,34] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[361,44] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[395,55] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
[ERROR] src/test/java/org/apache/commons/lang3/FunctionsTest.java:[395,92] (blocks) LeftCurly: '{' at column 92 should have line break after.

The same false-positive build failure happened to https://travis-ci.org/apache/commons-lang/jobs/575873214

Closing: This already works in git master. I added the test only to git master. TY.

authors: coveralls dimitrovchi garydgregory kinow

repo: apache/commons-lang