text:
Adding and design the Footer of About Us page also add function to subscribe the newsletter.

Related Issue
Footer of about us page is not looking good.
Fixes:  #3320
Description
As you can see below video demo the fotter section of About us page is not alsign correctly also not looking good and also the newsletter form is not working so i redesign the footer of about us page as you can see also add newsletter subscribe function in it.
Type of PR

[ ] Bug fix
[ ] Feature enhancement

Screenshots / videos (if applicable)
Before Footer of About us page
https://github.com/user-attachments/assets/fcf70264-c037-4c12-b1f9-74b01160bc76
After modification and redesigning the footer of about us page:
https://github.com/user-attachments/assets/e7db447c-065a-4d37-812c-58928c8edec4
Checklist:

[ ] I have made this change from my own.
[ ] My code follows the style guidelines of this project.
[ ] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] My changes generate no new warnings.
[ ] I have tested the changes thoroughly before submitting this pull request.
[ ] I have provided relevant issue numbers and screenshots after making the changes.

@sailaja-adapa Now, please check i have solve the conflict issue #3346

@sailaja-adapa i request you to please add level 2 in it.


text:
Modified Labels of Contact Us Form

Related Issue
Modified Labels of Contact Us Form and changed the color of Address
Fixes:  #4086
Description
Modified the color of Labels of Contact Us Form and changed the color of Address so that they are visible in dark mode

Type of PR

[x] Bug fix
[x] Feature enhancement

Screenshots / videos (if applicable)
Before

After

Checklist:

[x] I have made this change from my own.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

highlight the email box also with the color in dark mode .

highlight the email box also with the color in dark mode .

Completed the task

why have u made changes in .vscode/settings.json file ?

Sorry for the inconvenience but i haven't made that change , it might be due to my vs code
Sorry for the inconvinience caused

source: gharchive/pull-request

license: MIT

license_type: permissive

license_source: github-api


text:
added basic structure with react

added basic structure with react, added the pages (Home, About, Contact)

Nice work @Shruti3004 !
@anushbhatia can you please setup a default build in netlify since we are shifting to React

Thank You @smaranjitghose :)

repo: anushbhatia/Social-Scheduler


text:
Pasting new content into already named tab, resets the tab name

Describe the bug
When pasting new JSON content into an already named OK JSON window, the previously given name is removed, and is replaced with "Today at {TIME}"
(this may be related to #17)
To Reproduce
Steps to reproduce the behavior:

Open the OK JSON app
Click the "Rename tab" button to name the initial tab (e.g. "My First Tab")
Paste JSON content into the window
Note that the Tab name changed from "My First Tab" to "Today at {TIME}"

This occurs on any open tab, whether you have one tab open, or many. When you paste new content, the name is changed from what you named it to.
Expected behavior
The tab name that was given to the tab by the user should be preserved
Desktop (please complete the following information):

OS: macOS 13.2.1 (22D68)
Browser: N/A - using the OK JSON macOS app
Version 1.9.0 (26)

It’s fixed in 2.0.

source: gharchive/issue

repo: anyboxhq/everything-okjson


text:
How do I remove some object from display?

I can display an object on Render Window using a ShowShape call.
            var box = ShapeBuilder.MakeBox(GP.XOY(), 10, 20, 30);
            render.ShowShape(box, Vector3.Green);

How do I remove something from display?

You can use the Scene API to remove the node:
var id = box.GetUuid(); render.GetScene().RemoveNode(id);

authors: anycad uzername


text:
fix(msw): generated mocks for polymorphic type

Status

READY
(Sorry, didn't mean to pollute your repo, I've never really played around with the fork/pr flow and I managed to break things)
Description
Fixes #555 - When a response contains a polymorphic type, the generated mock object code was missing curly braces. This error was introduced in https://github.com/anymaniax/orval/commit/899a8cbd4c93190bd01ebd5e36be81d750b7608f.
Previously, when calling resolveMockValue within a oneOf, the combine option was undefined. The change resulted in the option being defined, resulting in a failing check for object boundaries.
This fix adds a check for (combine.separator === 'oneOf' && separator === 'allOf') when determining object boundaries.
I've also run all the tests, and to be sure I also diffed the tests before and after the fix and can confirm that the only generated output being changed is the new test sample, so this doesn't affect anything else.
Todos

[x] Tests
[ ] Documentation - N/A
[ ] Changelog Entry (unreleased)  - N/A

Steps to Test or Reproduce

Added a sample YAML swagger definition to tests (polymorphic.yaml)
Added the sample case to the test react-query.config.ts

Thanks @DominicBach!

authors: DominicBach anymaniax

repo: anymaniax/orval


text:
02_bubble Lab2 commit for review

Fixes #237
Review of colleague's PR #212
Changes proposed in this PR:
02_bubble code committed for Bubble and Insertion sort as part of second assignment for go-course.

Codecov Report

Merging #239 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #239   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           7      8    +1
  Lines         102    123   +21
=====================================
+ Hits          102    123   +21

Impacted Files
Coverage Δ

02_bubble/saurabhvyas9/main.go
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01df2de...aab556e. Read the comment docs.

You probably just need to use rebase to reduce your commits... just refer to this checklist for Lab2 and you'll be all good.  If u need help with rebase for your commits I should be able to help because I'm becoming an expert :o - I had to do lots of squashing and fixups myself for Lab1 & Lab2 :)

Forgot to paste the link with the checklist - https://github.com/anz-bank/go-course/wiki/Lab-2:-Collected-Code-Review-Comments

Still needs commit message update, see wiki.

authors: codecov-io juliaogris pfannerj saurabhvyas9

repo: anz-bank/go-course


text:
Lab04 and Lab05 - Create programs to print numeronyms and IP Address as dotted quad

Fixes #67
Changes proposed in this PR:

Added programs to find and print numeronyms and to print IP Address as dotted quad.
Added unit tests for the same.

Codecov Report

Merging #79 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master    #79   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          11     12    +1
  Lines         170    191   +21
=====================================
+ Hits          170    191   +21

Impacted Files
Coverage Δ

04_numeronym/selva/main.go
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 290e700...6f04231. Read the comment docs.

Please fix broken build.

Hi,
There has been  no activity in the last 20 days so I will close this PR.
Feel free to open it again if you have something to add.
Thanks!
Julia

authors: codecov-io delvein juliaogris

repo: anz-bank/go-training


text:
Include RestRequestOption interface and OnResult implementation

add RestRequestOption in restlib
add unit tests on rest request result
update arrai transformation

Closes #105 #159

Failed on compare Sysl and arr.ai transforms output, still need to work on the sysl transformation.

repo: anz-bank/sysl-go


text:
Shreds incur extra copies in several places

Problem

The shred object currently contains owned memory via a Vec<u8>. In several places, shreds are very temporarily created as an intermediate. Given the owned memory, we have extra copies:

Shred insertion into Blockstore - copy buffers from a PacketBatch to individual Shreds:
https://github.com/anza-xyz/agave/blob/12d009ececd7190a9f9f940715832001c03be2f4/core/src/window_service.rs#L333

Shred fetch from Blockstore via Blockstore::get_slot_entries_in_block()
https://github.com/anza-xyz/agave/blob/12d009ececd7190a9f9f940715832001c03be2f4/ledger/src/blockstore_db.rs#L1587
https://github.com/anza-xyz/agave/blob/12d009ececd7190a9f9f940715832001c03be2f4/ledger/src/shredder.rs#L404-L405
https://github.com/anza-xyz/agave/blob/12d009ececd7190a9f9f940715832001c03be2f4/ledger/src/blockstore.rs#L3579-L3584

For 1., the shreds have a short, common lifetime, that ends shortly after this copy. One minor exception is for any duplicate shreds that may be found (the duplicates have some further processing).
For 2., the copy into common buffer (2nd snippet) and bincode deserialize (3rd snippet) are necessary. However, the initial copy to owned memory (1st snippet) is seemingly not necessary.
Looking at MNB metrics:

I see about ~2700 packets / second coming through (8k on recv-window-insert-shreds.num_packets and datapoint reported every 3 seconds)
I see about ~1500 shreds / second (600 for shred_insert_is_full.last_index * 2.5 blocks / second).

So, in total, this is about ~4200 allocations and ~5 MB of memory copy that could be avoided, per second
Proposed Solution
Rework Shred to allow owned or borrowed memory, such as what Cow provides

At some point in the near future, I'll gather some data with perf so we can get a better idea of how much the previously mentioned numbers contribute to overall process numbers. Doing so might help us attach some sense of priority (ie if these are 0.01% of all allocations, we might decide we have bigger fish to fry at the moment).

repo: anza-xyz/agave


text:
Use try_read() in get_environments_for_epoch() to prevent recursive read

Problem
Recursive read locks on program cache can cause deadlocks in multithreaded environment, where write locks are taken intermittently.
Summary of Changes
Replaced read() with try_read().
Fixes #

Hm, won't this just cause panic when e.g. another thread has write-locked the lock?

Hm, won't this just cause panic when e.g. another thread has write-locked the lock?

Panic at what line of code?

You are right, I mistook test code for production. @LucasSte can you test if nested read locking but with try_read() is still caught by shuttle?
Because if it isn't that would make this PR counter productive, as it would conceal the underlying issue. But that is a general discussion that also revolves around saturating arithmetics and replacing unwraps with soft errors, even if we expect them to never happen.

If we call try_read with an existing read lock, it will return None, in which case, an unwrap will panic. I tested it with the code before fix #1616, but I replaced the read lock in TransactionBatchProcessor::get_environments_for_epoch by a try_read.
I can also run this PR through the test setting if necessary.

You are right, I mistook test code for production. @LucasSte can you test if nested read locking but with try_read() is still caught by shuttle?
Because if it isn't that would make this PR counter productive, as it would conceal the underlying issue. But that is a general discussion that also revolves around saturating arithmetics and replacing unwraps with soft errors, even if we expect them to never happen.

If we call try_read with an existing read lock, it will return None, in which case, an unwrap will panic. I tested it with the code before fix #1616, but I replaced the read lock in TransactionBatchProcessor::get_environments_for_epoch by a try_read.
I can also run this PR through the test setting if necessary.

try_read() returning None is the right outcome in this case. The most important thing is that it doesn't result in a deadlock.
This PR by itself should not change the behavior. The modified API is only used by ledger-tool. So it's unlikely that it'll generate any different results.

The modified API is only used by ledger-tool. So it's unlikely that it'll generate any different results.

You're talking about Bank::load_program() which calls TransactionBatchProcess::get_environments_for_epoch() right ? If so, should we annotate both functions with dev-context-only-utils? This would allow it to be pub for the ledger-tool use-case, but only pub(crate) otherwise. Or, would this function be needed by someone who is eventually depending on SVM crate in isolation too ?

The modified API is only used by ledger-tool. So it's unlikely that it'll generate any different results.

You're talking about Bank::load_program() which calls TransactionBatchProcess::get_environments_for_epoch() right ? If so, should we annotate both functions with dev-context-only-utils? This would allow it to be pub for the ledger-tool use-case, but only pub(crate) otherwise. Or, would this function be needed by someone who is eventually depending on SVM crate in isolation too ?

Yeah, that's a good idea. I'll push an update to the PR.

The modified API is only used by ledger-tool. So it's unlikely that it'll generate any different results.

You're talking about Bank::load_program() which calls TransactionBatchProcess::get_environments_for_epoch() right ? If so, should we annotate both functions with dev-context-only-utils? This would allow it to be pub for the ledger-tool use-case, but only pub(crate) otherwise. Or, would this function be needed by someone who is eventually depending on SVM crate in isolation too ?

Yeah, that's a good idea. I'll push an update to the PR.

@steviez running into issue with this approach. Likely because it's not a dev only dependency for solana-runtime crate. The ledger-tool crate needs access of the symbol via solana-runtime. Or, maybe I am doing something wrong.

Looks like you have a CI failure from clippy too

authors: Lichtso LucasSte pgarg66 steviez


text:
Restrict nil media characteristic option selection

Nil AVMediaSelectionOption should not be selected because .loaded state for item.asset.statusOfValue(forKey: "availableMediaCharacteristicsWithMediaSelectionOptions") fired before item.asset.loadValuesAsynchronously(forKeys:completionHandler:) completion handler
@aol-public/mobile-sdk-team: Ready for review.
JIRA Issue

Fixed in https://github.com/aol-public/OneMobileSDK-videorenderer-ios/pull/59

license: mit

license_source: bigquery


text:
Add an instanceOf Predicate to Predicates

The type Predicate in Predicates is primarily for deconstructing an Object by guards, it's more verbose than neccessary if all you want to do is check the type.

Added

repo: aol/cyclops-react


text:
error 4

Elasticsearch version:
6.0.0
Moloch version:
1.6.1
OS name and version:
centos7
How was Moloch built/installed: (rpm, deb, easybutton, ...)
rpm
Provide logs, stack traces and steps to reproduce:

Running moloch-captrue, generating a failure, reporting an error message:
moloch-pkt0[26409]:segfault at 110001 ip 000000000049ab7 sp 00007fe4358e9750 error 4 in moloch-capture[400000+23700]

Please try 1.8.0.  If you still have issues see https://molo.ch/faq#moloch-capture-crashes on how to get a useful stack trace.

authors: awick tomrin

repo: aol/moloch


text:
How to configure Elasticsearch auth?

Elasticsearch version:
2.4.4
Moloch version:
latest
OS name and version:
CentOS 7
How to configure Elasticsearch with Auth?

What type of auth?  Currently moloch doesn't support any ES auth.

Xpack https://www.elastic.co/guide/en/x-pack/current/security-getting-started.html

So you want basic auth support?

Yes,ACL is very important

ACL is on the ES side, moloch doesn't nor will never do anything for that.

Could you add a option for http auth for ES?ACL is used for ES cluster in production.

This is why I was trying understand what your request was.

Fixed in 2abdfa4669d98f60443ac570938735451cf6a5bd  Just set the user/pass in the url

Hi I'm attempting to use an Elastic Cloud instance to store the data captured by Moloch.  I have X-Pack security installed on this cluster and https.  I'm attempting to pass use: https://username:password@cluser_ip:9243 but I keep getting errors that my cluster is unreachable.  I have confirmed that my cluster is actually up and running by CURLing the url. Is xpack security currently supported in Moloch?

Please always provide versions number in bug reports, in this case for both moloch and es.  Your future bug report recipients will appreciate it.
Moloch just uses libcurl, so the exact same url should work with both.  Double check you are using the exact same url.

Sorry about that @awick.
Moloch Version: https://files.molo.ch/builds/centos-6/moloch-1.1.1-1.x86_64.rpm
Elasticsearch: 6.2.4

I tried 6.2.4 with xpack both locally and with found.io and it worked.  The found.io version required a newer version to perl to get the right CA certs.
You can add --debug --debug --debug to your capture command and look at the logs for the elasticsearch variable to double check it matches what you are using with curl and to see the output.

Do you happen to know which version of Perl that was?

Using MacPorts I went to v5.26.2 and the latest LWP

Alright great.  Updating perl on my machine now.

@awick - First, thanks for your work on Arkime. It's a great tool and I'm grateful to everyone who helped build it.
On the issue of ES auth it'd be great if instead of/as well as basic auth the use of an API key was supported. Basic auth, particularly where the username and password are embedded in the URL has multiple well-known drawbacks, and in this case the password is logged in the clear in log files:  e.g. /data/moloch/logs/capture.log. It'd be awesome if we could use API keys to authenticate with ES instances in Elastic Cloud instead and, I'd hope (but am absolutely no expert on this codebase) that it wouldn't be too difficult to achieve?
Thanks.


text:
てすと

てすとてすと

てすてすと

てすてすと

repo: aomizuanikeda/github-slideshow


text:
Import Table command does not provide proper volumes to the PreferredVolumeChooser

Originally reported by @drewfarris on https://issues.apache.org/jira/browse/ACCUMULO-4862:
The PopulateMetadataTable phase of the Import Table command obtains table directories (e.g: hdfs://volume-a/accumulo/tables, hdfs://volume-b/accumulo/tables) instead of volumes (e.g: hdfs://volume-a/accumulo,hdfs://volume-b/accumulo). These are passed into the volume chooser in getClonedTabletDir(..)
The PreferredVolumeChooser compares these table directories with proper volumes configured in the table.custom.preferredVolumes property. The result is an empty set. An AccumuloException is thrown with the message "After filtering preferred volumes, there are no valid instance volumes" and this terminates the Import Table process.

@ctubbsii - thanks.

authors: ctubbsii drewfarris

repo: apache/accumulo


text:
Possible fix and unit test. Fixes #961

One possible fix for this issue.  This just ignores the failed file.  AFAIK we aren't actually doing anything with this failed file so it could just be removed.  But maybe that be done for 2.0

Should we at least log a debug message?

It already logs the original error.  I am not sure how much value another log message adds...


text:
IteratorEnvironment fixes

closes #4810

Fixed some inconsistencies and bugs in (non-test) implementations of IteratorEnvironment

Fixed RFileScanners unexpected UnsupportedOperationException when calling env.getServiceEnv() or env.getConfig() (both deprecated) or env.getPluginEnv()
Some impls returned a new ServiceEnvironment for each call to getServiceEnv()/getPluginEnv(). This did not align as well with the IteratorEnvIT which would assert that these two calls return the same object. Memoize a supplier to the ServiceEnvironment to avoid creating the object unless needed and to return the same object each time.
Seems like isUserCompaction() and isFullMajorCompaction() are supposed to be throwing an IllegalStateException if it's not a major compaction. Added missing javadoc to isUserCompaction() and changed impls that were not adhering to these docs.

Added tests for these environments to IteratorEnvIT

Questions:

Need confirmation, but it was my interpretation that a TableId doesn't make sense in the context of RFileScanner. Is this correct? ANSWERED
I have getConfiguration() throw an error in RFileScanners IteratorEnvironment since I'm not sure how we would return the system config in this context and I don't think it would even make sense here. Does this seem right? ANSWERED
Made the IteratorEnvironment for RFileScanner return null and not throw an error on env.getTableId() so something like: env.getPluginEnv().getConfiguration(env.getTableId()) can be done. Thoughts on this? ANSWERED

After a conversation with @kevinrr888, I added commit 5924b02c065297886ede479fbcf9750e699e4362 to incorporate some of the changes we discussed, that he intends to build on and polish up. The main points included were:

Push a reasonable getConfig() implementation up into IteratorEnvironment's default method (depends on the ability to have getTableId() return a null value)
Include concrete implementations of getPluginEnv() instead of just getServiceEnv() to make it easier to just drop the deprecated one when merging to newer branches
Consider using a Proxy instance of TableId instead of a real instance, so it isn't possible to be confused with any similar user-constructed TableId (may not work)
Update javadocs in RFile to make it clearer how properties set on the scanner builder (such as iterators) will be available to client-side iterators from the plugin environment

The main thing left unresolved after our conversation was how to fix the mismatch between the implementations and the javadoc for all the methods that inconsistently return null or throw IllegalStateException or UnsupportedOperationException, when calling a method that doesn't apply to a given situation (like asking for a table ID or table name when operating directly on RFiles or asking if it's a user-initiated compaction when it's a scan). Nearly every caller seems to have a unique set a requirements/implementation, and it's a bit hard to home in on the best implementation, especially for the default methods, and make sure the javadoc matches the behavior.

Comment about IteratorEnvironment.getConfig():
It might be strange to have a default impl for this method but not others... If we do want a default impl for this that isnt just throwing a UOE, we should probably change the others too... But that should be done in a follow on if that is desired. Keeping the default as throwing a UOE keeps it consistent with the other default impls/more inline with the original approach (throw UOE by default, concrete impls should have an actual impl). The only IteratorEnvironment which would be using the default getConfig() would be ClientSideIteratorScanner. But we can just add the impl there (done in latest commit). It is also hard to know what would be a good default impl since its not exactly clear how getConfig() was originally working.
Comment about RFileScanner.IterEnv.getTableId()
Proxy can only be used to create proxies for interfaces, not concrete classes, so can't be used for TableId. Opting to just return null with added javadoc.
Background: we want to support getConfiguration(TableId) since this is the method defined as returning the props set on the table and getConfiguration() should not be returning table props, so we cant substitute that method. So, something needs to be passed, but it doesn't matter what since it wont be used. The way getConfiguration(TableId) will be called is env.getPluginEnv().getConfiguration(env.getTableId()) (from javadocs for IteratorEnvironment.getPluginEnv()) So, we should have env.getTableId() returning something.
No solutions seem great, but Proxy is a no-go, a real dummy id may be confusing (e.g., a user may think that it is a real table id), null may lead to NPE. I figure null makes the most sense here. A NPE would only occur if the user is trying to do something with the table id which they shouldnt be doing anyways. It is also now noted in the javadocs that null will be returned so if a NPE does occur, it should be easy to see why.
Comment about UOE/ISE
Did not change anything further than I had already changed in my earlier commits relating to when/how UOE or ISE are thrown, how its documented etc. Some future considerations would be

IteratorEnvironment.getAuthorizations() currently documented as throwing UnsupportedOperationException if scope != scan (should probably be ISE if anything).
IteratorEnvironment.isFullMajorCompaction() and isUserCompaction() should maybe throw ISE in the default if the scope is majc

Maybe these shouldn't be throwing exceptions at all and only returning false... This would probably be the cleanest if possible. Would have to look more into why these are throwing exceptions in the first place.

Comment about RFileScanners createServiceEnv()
Christopher suggested having this be an inner class instead of anon class. Leaving this comment as a reminder. Will do once the other more important changes are looked at; want to avoid confusing diff between commits for now.


text:
Artemis upgrade version notes for 2.27.0

An attempt to clarify the use / scope of artemis upgrade in the release notes.
This is a good PR to bike shed wording in, or add any critical notes.
If there was a help page for artemis upgrade I would link it.

Error:  Failures:
Error:    AcknowledgementTest.testDupsOKAcknowledgeQueue:555->ActiveMQServerTestCase.assertRemainingMessages:364 expected:<20> but was:<21>

Seems like an unstable test. I've seen that in other PRs, but I don't see a ticket. Create one?

Ahh...I'm afraid I have just pushed a bunch of changes in this same area a little while ago. I knew Clebert wouldnt be working on this today, and didnt think to check for a PR with other changes first, sorry. You can give them a look over and see what you think.
Regarding the test, feel free to create a JIRA.

Cool, no I like your wording better than mine. I'll close this and open a jira about the test. Thanks!

authors: TheCycoONE gemmellr

repo: apache/activemq-artemis


text:
AMQNET-718 Cancel inflight messages after send timeout

[x] It requires the release of https://github.com/Azure/amqpnetlite/pull/466.

@michaelandrepearce How should I configure broker to check this?
<address-settings>
   <address-setting match="exampleQueue">
      <max-size-bytes>100000</max-size-bytes>
      <address-full-policy>BLOCK</address-full-policy>
   </address-setting>
</address-settings>

Does max-size-bytes represents the number of messages for AMQP and the number of bytes for core? Docs are pretty opaque in that matter.

address, max size bytes, the size in bytes that the address is at, the particular bit of code that calculating is the isFull method and in PagingStoreImpl and then isRejectingMessages  informs the address code if it should reject. How that maps to AMQP protocol manager is within the serverSend in AMQPSessionCallback which the ProtonReceiverContext uses.

authors: Havret michaelpearce-gain

repo: apache/activemq-nms-amqp


text:
AMBARI-24266 Error in Validating And Submitting Workflow With Node as Hive

We have disabled hive action in WFM as its not supported in HDP now.
Manually tested the changes.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3027/
Test PASSed.

authors: asfgit venkatasairamlanka

repo: apache/ambari


text:
[AMBARI-24865] Build error at Findbugs with Maven 3.6.

What changes were proposed in this pull request?
Same as #2581, for branch-2.7.
How was this patch tested?
Build with Maven 3.6:
$ mvn -version
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T20:41:47+02:00)
...
$ mvn -am -pl ambari-server -DskipTests clean verify
...
[INFO] Ambari Server 2.7.3.0.0 ............................ SUCCESS [02:08 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Build with Maven 3.5:
$ mvn -version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T20:33:14+02:00)
...
$ mvn -am -pl ambari-server -DskipTests clean verify
...
[INFO] Ambari Server 2.7.3.0.0 ............................ SUCCESS [02:06 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4471/
Test PASSed.

authors: adoroszlai asfgit


text:
Fix various DDL errors

What changes were proposed in this pull request?

Forward reference to topology_request table in all create scripts.

psql:/var/lib/ambari-server/resources/Ambari-DDL-Postgres-CREATE.sql:639: ERROR:  relation "topology_request" does not exist

Forward reference to hosts table in most scripts.

ERROR 1215 (HY000) at line 66: Cannot add foreign key constraint

Identifier dependent_service_group_cluster_id  too long for Oracle.  Replaced in SQL and the entity.

  dependent_service_group_cluster_id NUMBER(19) NOT NULL,
  *
ERROR at line 6:
ORA-00972: identifier is too long

Identifier host_component_desired_state_id too long for Oracle.  Replaced in SQL and the entity.

  host_component_desired_state_id NUMBER(19) NOT NULL,
  *
ERROR at line 3:
ORA-00972: identifier is too long

Reference to non-existent service_name column (was changed to service_id in some DDL scripts, leftover as service_name in some others).

CREATE INDEX idx_host_component_state on hostcomponentstate(host_id, component_name, service_name, cluster_id)
                                                                                *
ERROR at line 1:
ORA-00904: "SERVICE_NAME": invalid identifier

Wrong order of NOT NULL vs DEFAULT (part of #657 for trunk).

  kkp_id BIGINT NOT NULL DEFAULT 0,
                         *
ERROR at line 2:
ORA-00907: missing right parenthesis

How was this patch tested?
Tested schema creation on Postgres, MySQL and Oracle.
Tested cluster deployment on Postgres.
MySQL script still has a problem that was fixed in #695, but I'm not sure about cherry-picking it to the feature branch, because the fix turned out to have a problem on versions prior to MySQL 5.6.5.

Wrong base branch.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/1276/
Test FAILed.
Test FAILured.


text:
test: remove custom apisix-default.yaml

Please answer these questions before submitting a pull request

Why submit this pull request?

[ ] Bugfix

[ ] New feature provided

[ ] Improve performance

[ ] Backport patches

Related issues

Bugfix

Description

How to fix?

New feature or improvement

Describe the details and related test reports.

Backport patches

Why need to backport?

Source branch

Related commits and pull requests

Target branch

Codecov Report

Merging #494 (d24e7b2) into master (e8eddcc) will decrease coverage by 0.39%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #494      +/-   ##
==========================================
- Coverage   37.24%   36.84%   -0.40%
==========================================
  Files          49       49
  Lines        4030     4030
==========================================
- Hits         1501     1485      -16
- Misses       2328     2342      +14
- Partials      201      203       +2

Impacted Files
Coverage Δ

pkg/apisix/cluster.go
25.91% <0.00%> (-5.27%)
:arrow_down:

pkg/apisix/route.go
33.08% <0.00%> (-2.21%)
:arrow_down:

pkg/ingress/ingress.go
8.58% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8eddcc...d24e7b2. Read the comment docs.

authors: codecov-commenter tokers

repo: apache/apisix-ingress-controller


text:
Also run yarn build to catch JavaScript errors in CI

Which issue does this PR close?
Closes #270.
Rationale for this change
Treat the yarn build as a first-class citizen.
What changes are included in this PR?
A CI job to build yarn.
Are there any user-facing changes?
TypeScript code should always at least compile if not work.

@andygrove would love to get this running in CI so other PRs don't silently break things.

repo: apache/arrow-ballista


text:
Remove redundant rust dir

Which issue does this PR close?

N/A
Rationale for this change

Simplify the project structure by removing the redundant rust folder. In a future PR, I plan on moving the ui/scheduler files into the scheduler module.
What changes are included in this PR?

Remove rust folder and update build and docs

Are there any user-facing changes?

Noi

cc @mingmwang @Ted-Jiang @avantgardnerio


text:
[WIP] Add spark 35

Which issue does this PR close?
Adds Spark 3.5
Rationale for this change
Spark 3.5 is the latest Spark release.
What changes are included in this PR?
Adds Spark 3.5 support, this introduces a new per-spark version of shims as part of adding interfacing to the 3.5 APIs.
This has the advantage of static compile checks over the existing shim approach which does reflection for everything, but I can switch to the runtime reflection model for consistency if that's preferred.
How are these changes tested?
WIP

Some class and function has changed in spark 3.5, maybe we need some more shim classes to support both spark 3.5 and the versions before.
I has a similar commit to do this:
https://github.com/apache/arrow-datafusion-comet/commit/b038267eb5e6ddafccf84b0147760f6992f36077

Yup this adds some shims (there are existing shims too for 3.3 etc but they use reflection heavily).

FWIW I created https://github.com/apache/arrow-datafusion-comet/issues/140 to improve the current Spark shim handling.

I love it :)
On Thu, Feb 29, 2024 at 10:45 AM Chao Sun @.***> wrote:

FWIW I created #140
https://github.com/apache/arrow-datafusion-comet/issues/140 to improve
the current Spark shim handling.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow-datafusion-comet/pull/127#issuecomment-1971741645,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAOT5NRC355APXEYYRFIADYV53MJAVCNFSM6AAAAABD5DERESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRG42DCNRUGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

FYI https://github.com/apache/datafusion-comet/pull/318 is just merged

Thanks for getting this started @holdenk. I have now created https://github.com/apache/datafusion-comet/pull/587 to add experimental support for Spark 3.5.1 (there will be follow up PRs to fix some test failures), so will close this one.

authors: andygrove holdenk kazuyukitanimura sunchao wankunde

repo: apache/arrow-datafusion-comet


text:
Implement array_has function

Is your feature request related to a problem or challenge?
Summary

Characteristic
Description

Function name:
array_has

Aliases:
list_has, array_contains, list_contains

Original function?:
No

Function Description:
Azure DataBricks: Returns true if array contains value.  Clickhouse: Checks whether the ‘arr’ array has the ‘elem’ element. Returns 0 if the element is not in the array, or 1 if it is.  DuckDB: Returns true if the list contains the element.

Sources:
Concept Azure ClickHouse DuckDB

Examples:
select array_has([1, 2, 3, 4], 3);
----
true

select array_has([1, 2, 3, 4], 5);
----
false

Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response

The original function is No in the description, but I think array_has is the same as the current function array_contains right?

repo: apache/arrow-datafusion


text:
add documentation to Datafusion cli's new commands

Which issue does this PR close?

Add the document for DDL sql and CLI command.
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

LGTM assuming we fix the linting error

PTAL


text:
feat: support month and year interval for date_bin on constant data

Which issue does this PR close?
This is fir first part of https://github.com/apache/arrow-datafusion/issues/5689 to support month & year interval for constant input.
The month interval for array input was converted into ScalarValue::IntervalDayTime (days) somewhere. My next PR is to find that place and convert month interval into ScalarValue::IntervalMonthDayNano instead.  I think if I do that, they will automatically work with this code

Closes #.
Rationale for this change
In IOx, we see users use data_bin with month interval to group data by month for their monthly report. We think it is a very useful feature to support.

What changes are included in this PR?
Since each month does not include the same number of nanoseconds and for date_bin, it also depends on the origin input. This PR mainly add a function to handle month interval for date bin and some related refactoring work

Are these changes tested?
Yes

Are there any user-facing changes?
Yes, date_bin with month and year interval is now working on constant input

@alamb

The month interval for array input was converted into ScalarValue::IntervalDayTime (days) somewhere. My next PR is to find that place and convert month interval into ScalarValue::IntervalMonthDayNano instead. I think if I do that, they will automatically work with this code

I suspect the code that is do this is the coercion to the type signature defined here:
https://github.com/apache/arrow-datafusion/blob/1646bf6aae1e568e38c5a43e9a44c34e26222341/datafusion/expr/src/function.rs#L474-L494
I think you could move the Interval(MonthDayNano) signatures above the Interval(DayTime) ones

@alamb

I think you could move the Interval(MonthDayNano) signatures above the Interval(DayTime) ones

It works like a charm. This PR now has dateline works for everything (if I do not miss other cases).
To avoid many duplicate tests with and without INTERVAL keyword, I added INTERVAL into tests without it before and remove it for the ones with it. These make the tests more concise and I know all of them work correctly

@alamb The PR is now ready for re-reviewing with all questions answered

@alamb thank you for pining me. @NGA-TRAN  thank you for the great works. the codes looks good to me.
i tested this in your branch, i wonder whether this corner cases make sense
❯ select date_bin('1 month', timestamp '2000-02-29T00:00:00', timestamp '2000-01-31T00:00:00');
+----------------------------------------------------------------------------------+
| datebin(Utf8("1 month"),Utf8("2000-02-29T00:00:00"),Utf8("2000-01-31T00:00:00")) |
+----------------------------------------------------------------------------------+
| 2000-02-29T00:00:00                                                              |
+----------------------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

❯ select date_bin('1 month', timestamp '2000-01-31T00:00:00', timestamp '2000-02-29T00:00:00');
+----------------------------------------------------------------------------------+
| datebin(Utf8("1 month"),Utf8("2000-01-31T00:00:00"),Utf8("2000-02-29T00:00:00")) |
+----------------------------------------------------------------------------------+
| 2000-01-29T00:00:00                                                              |
+----------------------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

i feel like the mechanism behind isn't consistent. i thought that the second's is 2000-01-31T00:00:00

@waitingkuo
You have great tests. I agree they look inconsistent. I think we may want to define the behavior here and see whether we want to keep it or not:
I think this one is easy to understand: the origin is Jan 31 so any other month will always be the last day of that month (28, 29, 30, 31 depending on  the month and year)
❯ select date_bin('1 month', timestamp '2000-02-29T00:00:00', timestamp '2000-01-31T00:00:00');
+----------------------------------------------------------------------------------+
| datebin(Utf8("1 month"),Utf8("2000-02-29T00:00:00"),Utf8("2000-01-31T00:00:00")) |
+----------------------------------------------------------------------------------+
| 2000-02-29T00:00:00                                                              |
+----------------------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

This is is tricky and we need to identify the behavior.
❯ select date_bin('1 month', timestamp '2000-01-31T00:00:00', timestamp '2000-02-29T00:00:00');
+----------------------------------------------------------------------------------+
| datebin(Utf8("1 month"),Utf8("2000-01-31T00:00:00"),Utf8("2000-02-29T00:00:00")) |
+----------------------------------------------------------------------------------+
| 2000-01-29T00:00:00                                                              |
+----------------------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

For February, 29 (or 28) is the last day of the month, but for other month is it not the last day of the month. So chromo's Months that we are using in this PR do not consider 29 as the last day of the month and just simple add/minus 30 days which I think makes sense.
❯ select date_bin('1 month', timestamp '2000-01-31T00:00:00', timestamp '2000-03-29T00:00:00');
+----------------------------------------------------------------------------------+
| datebin(Utf8("1 month"),Utf8("2000-01-31T00:00:00"),Utf8("2000-03-29T00:00:00")) |
+----------------------------------------------------------------------------------+
| 2000-01-29T00:00:00                                                              |
+----------------------------------------------------------------------------------+
1 row in set. Query took 0.005 seconds.

So I think this is the behavior is acceptable. What do you, @alamb and @stuartcarnie think?

I am adding @stuartcarnie 's suggestion and benchmark into this PR

@alamb : I have added @stuartcarnie's the suggestion without the benchmark because I got Audit error if the benchmark is included. I am not sure what is the policy for adding benchmark

That's great, thanks @NGA-TRAN! date_bin is used extensively in IOx, so downstream will definitely appreciate it!

I have added the tests with origin on 29th of Feb and Mar and interval is a month. I have added explanation for the tests on why the bins are always on the 29th

authors: NGA-TRAN alamb stuartcarnie waitingkuo


text:
#723 limit pruning rule to simple expression

Which issue does this PR close?
Closes #723.
Rationale for this change

see in issue https://github.com/apache/arrow-datafusion/issues/723 step 2
What changes are included in this PR?

Are there any user-facing changes?

I've committed some test cases in parquet_pruing.rs:
for int32 column:

SELECT * FROM t where i < 1
SELECT * FROM t where -i > -1
SELECT * FROM t where i = 1
SELECT * FROM t where abs(i) = 1 and i = 1 : use i=1 as prune predicate
SELECT * FROM t where abs(i) = 1 : not supported
SELECT * FROM t where i+1 = 1 : not supported
SELECT * FROM t where 1-i > 1: not supported

for float64 column

SELECT * FROM t where f < 1
SELECT * FROM t where -f > -1
SELECT * FROM t where abs(f - 1) <= 0.000001  and f >= 0.1 : use f>=0.1 as prune predicate
SELECT * FROM t where abs(f-1) <= 0.000001 : not supported
SELECT * FROM t where f+1 > 1.1 : not supported
SELECT * FROM t where 1-f > 1 : not supported

nice work indeed @lvheyang , thank you so much for making these changes; you've done a much better job to validate which predicate expressions are compatible with parquet predicate push-down compared to the naiive implementation in the PR where I introduced it; hopefully this can soon be extended for even more expressions.

Thanks for your kind help @yordan-pavlov @alamb , I will try to extend more expressions in future works. 😄

authors: lvheyang yordan-pavlov


text:
Use new DecimalArray creation API in parquet crate

~Builds on https://github.com/apache/arrow-rs/pull/1223 so draft until that is done~
Rationale:
https://github.com/apache/arrow-rs/pull/1223 introduces a more performant and idiomatic API for creating DecimalArrays than DecimalBuilder so update the code to use that.

Codecov Report

Merging #1247 (acbb5cc) into master (aac1844) will increase coverage by 0.02%.
The diff coverage is 93.12%.

@@            Coverage Diff             @@
##           master    #1247      +/-   ##
==========================================
+ Coverage   82.96%   82.99%   +0.02%
==========================================
  Files         178      178
  Lines       51547    51644      +97
==========================================
+ Hits        42765    42860      +95
- Misses       8782     8784       +2

Impacted Files
Coverage Δ

arrow/src/datatypes/datatype.rs
66.80% <ø> (ø)

parquet/src/arrow/array_reader.rs
78.31% <85.71%> (+0.10%)
:arrow_up:

arrow/src/array/array_binary.rs
93.33% <92.79%> (-0.15%)
:arrow_down:

arrow/src/array/data.rs
83.13% <100.00%> (+0.10%)
:arrow_up:

parquet/src/arrow/arrow_writer.rs
97.96% <100.00%> (-0.02%)
:arrow_down:

parquet/src/arrow/converter.rs
63.96% <100.00%> (-0.39%)
:arrow_down:

parquet_derive/src/parquet_field.rs
65.98% <0.00%> (+0.22%)
:arrow_up:

arrow/src/datatypes/field.rs
54.10% <0.00%> (+0.30%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aac1844...acbb5cc. Read the comment docs.

FYI @liukun4515

authors: alamb codecov-commenter

repo: apache/arrow-rs


text:
Fix bitmask creation also for simd comparisons with scalar

Which issue does this PR close?

Closes #1285 (now also for comparisons with a scalar).
Rationale for this change
Followup to #1286, which only fixed the bitmask creation for comparing two arrays, but missed the same bug when comparing with a scalar.

Noticed by @JasonLi-cn and @alamb in https://github.com/apache/arrow-rs/commit/411171ac7ad3efe9391b2a0f8709f7fe97672a20. Sorry again for missing this earlier.

Thanks @jhorstmann  -- I'll get this in and spin up a new RC 👍

For reference, before the fix tests with the larger inputs tests would fail like this:
---- compute::kernels::comparison::tests::test_primitive_array_eq_scalar stdout ----
thread 'compute::kernels::comparison::tests::test_primitive_array_eq_scalar' panicked at 'assertion failed: `(left == right)`
  left: `BooleanArray
[
  false,
  false,
  true,
  false,
  false,
  false,
  false,
  true,
  false,
  false,
  ...80 elements...,
  false,
  false,
  true,
  false,
  false,
  false,
  false,
  true,
  false,
  false,
]`,
 right: `BooleanArray
[
  false,
  false,
  true,
  true,
  true,
  true,
  true,
  true,
  true,
  true,
  ...80 elements...,
  false,
  false,
  true,
  false,
  false,
  false,
  false,
  true,
  false,
  false,
]`', arrow/src/compute/kernels/comparison.rs:2518:9

authors: alamb jhorstmann


text:
feat: add method for async read bloom filter

Which issue does this PR close?
Impl #3851
We want to filter row_groups in Datafusion but there is no async API for reading bloom filter.
What changes are included in this PR?
Implemented a function get_row_group_column_bloom_filter for ParquetRecordBatchStreamBuilder to support reading bloom filter outside arrow.
Are there any user-facing changes?
Add an function get_row_group_column_bloom_filter in ParquetRecordBatchStreamBuilder

@tustvold Sure, I will try to add two test cases:

for the parquet file has bloom_filter_length header
for the parquet file has no bloom_filter_length header

@tustvold Can i create two test parquet files and commit to https://github.com/apache/parquet-testing/ ?

You could, but I don't have merge rights there so it may take some time.
A quicker option might be to use an existing file for 1., and to write a file to a Vec for 2.

@tustvold There is no test parquet files with bloom_filter in parquet-testing, I will generate two parquet files in memory.

Would you mind take a look at data_index_bloom_encoding_stats.parquet?

authors: hengfeiyang mapleFU tustvold


text:
[Python[Docs] fill_null not showing in pyarrow API reference

Describe the bug, including details regarding any error messages, version, and platform.
Can't sign up for Jira, hence I opened the issue here.
fill_null was removed in 909379516a3e575168ba6488621c364e60376390 Seems to be accidential. It is not visible in the docs anymore.
https://arrow.apache.org/docs/python/api/compute.html#structural-transforms
Component(s)
Python

Can't sign up for Jira, hence I opened the issue here.

No problem. We migrated to here from Jira.

authors: kou phofl

repo: apache/arrow


text:
[C++][Parquet] Remove boost dependency

At a glance, parquet-cpp slightly uses boost dependency. It seems to be possible to remove boost dependency if we use C++11 feature.
If we remove boost dependency, parquet-cpp can be more portable and lightweight. Also, C+<u>11 would allow us to modernize C+ codes.
Reporter: Hyunsik Choi
Assignee: Wes McKinney](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wesm) / @wesm
Note: This issue was originally created as PARQUET-277. Please see the migration documentation for further details.

Wes McKinney / @wesm:
@nongli I don't think it's possible to remove boost as a build dependency for libparquet because of Thrift, but we should absolutely prevent boost from being included in the public API headers. Are you comfortable reducing the scope of this issue to that? I can take care of both this and PARQUET-446 in a single patch. Not having to find and include Thrift and Boost in downstream projects will help me a lot.

Julien Le Dem / @julienledem:
@nongli comments?

Nong Li / @nongli:
Yea, I think this is a good idea. We should use standard libraries over external libraries where possible. I think we have some patches that are going towards this direction.

Wes McKinney / @wesm:
With PARQUET-416 we are now on C++11 and the only boost dependency are Thrift APIs that use boost::shared_ptr

Kalon Mills / @kalaxy:
I don't think we can remove the boost dependency entirely as we depend on thrift.  Thrift still requires boost even in the most recent release.  It looks like they have a issue tracking c++11 conversion but that it isn't a top priority for core developers,  THRIFT-2242.
However, I agree that the parquet-cpp library proper would do well to replace as much boost with c++11 as possible.  I too think it would make the library much nicer.

Wes McKinney / @wesm:
Resolved by https://github.com/apache/parquet-cpp/pull/49


text:
[C++][Parquet] usage of THRIFT_HOME in build?

Hi,
I posted this issue a while back on the old parquet-cpp github page and forgot about it...until just now.
First of all, to briefly describe my use case: I am using parquet-cpp as a dependency in a project which already has a dependency on thrift. As such, I'd like to be able to build parquet-cpp against the version of thrift which we are already using in the project.
When I first spotted the notes regarding THRIFT_HOME in the readme, I thought this looked like exactly what I wanted – that is, a way to configure the parquet-cpp build process to use external/pre-built thrift artifacts. So, for starters: am I simply misinterpreting the meaning of THRIFT_HOME? (e.g. does that specify a destination directory for thrift artifacts built during the parquet-cpp 3p build step?) And if so, is there any good way (or bad way) to accomplish what I'm trying to do here?
Otherwise, when I try to use my own thrift installation (via THRIFT_HOME), the build fails like so:

$ make parquetcpp
cd parquet-cpp && source ./thirdparty/set_thirdparty_env.sh && make
make[1]: Entering directory parquet-cpp' make[2]: Entering directory parquet-cpp'
make[3]: Entering directory parquet-cpp' Scanning dependencies of target parquet_thrift make[3]: Leaving directory parquet-cpp'
make[3]: Entering directory `parquet-cpp'
[  1%] Building CXX object src/parquet/thrift/CMakeFiles/parquet_thrift.dir/parquet_types.cpp.o
parquet-cpp/src/parquet/thrift/parquet_types.cpp: In member function ‘uint32_t parquet::format::Statistics::read(apache::thrift::protocol::TProtocol*)’:
parquet-cpp/src/parquet/thrift/parquet_types.cpp:179:3: error: ‘TInputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TInputRecursionTracker tracker(*iprot);
^
parquet-cpp/src/parquet/thrift/parquet_types.cpp: In member function ‘uint32_t parquet::format::Statistics::write(apache::thrift::protocol::TProtocol*) const’:
parquet-cpp/src/parquet/thrift/parquet_types.cpp:244:3: error: ‘TOutputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TOutputRecursionTracker tracker(*oprot);
^
parquet-cpp/src/parquet/thrift/parquet_types.cpp: In member function ‘uint32_t parquet::format::SchemaElement::read(apache::thrift::protocol::TProtocol*)’:
parquet-cpp/src/parquet/thrift/parquet_types.cpp:357:3: error: ‘TInputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TInputRecursionTracker tracker(*iprot);
^
parquet-cpp/src/parquet/thrift/parquet_types.cpp: In member function ‘uint32_t parquet::format::SchemaElement::write(apache::thrift::protocol::TProtocol*) const’:
parquet-cpp/src/parquet/thrift/parquet_types.cpp:471:3: error: ‘TOutputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TOutputRecursionTracker tracker(*oprot);
^
<SNIP ... you get the idea>
^
parquet-cpp/src/parquet/thrift/parquet_types.cpp: In member function ‘uint32_t parquet::format::FileMetaData::read(apache::thrift::protocol::TProtocol*)’:
parquet-cpp/src/parquet/thrift/parquet_types.cpp:2646:3: error: ‘TInputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TInputRecursionTracker tracker(*iprot);
^
parquet-cpp/src/parquet/thrift/parquet_types.cpp: In member function ‘uint32_t parquet::format::FileMetaData::write(apache::thrift::protocol::TProtocol*) const’:
parquet-cpp/src/parquet/thrift/parquet_types.cpp:2775:3: error: ‘TOutputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TOutputRecursionTracker tracker(*oprot);
^
make[3]: *** [src/parquet/thrift/CMakeFiles/parquet_thrift.dir/parquet_types.cpp.o] Error 1
make[3]: Leaving directory parquet-cpp' make[2]: \*\*\* [src/parquet/thrift/CMakeFiles/parquet_thrift.dir/all] Error 2 make[2]: Leaving directory parquet-cpp'
make[1]: *** [all] Error 2
make[1]: Leaving directory `parquet-cpp'
make: *** [<...>/libparquet.a] Error 2

This kinda smells like a circular dependency (as all of the types/headers mentioned above look fine), but I haven't dug through the internals of the parquet-cpp build logic enough to diagnose it in more detail. Before doing so, I wanted to post here for a sanity check on my use case, and the intended semantics/usage of THRIFT_HOME.
Any help/clarification will be much appreciated!
Reporter: William Forson / @wdforson
Original Issue Attachments:

cmake_output_original.txt

Note: This issue was originally created as PARQUET-798. Please see the migration documentation for further details.

Uwe Korn / @xhochy:
THRIFT_HOME should tell the CMake where to find a built THRIFT, e.g. so that the thrift header should reside at ${THRIFT_HOME}/include/thrift/Thrift.h.
To better understand your problem:

What version of Thrift do you want to use? We currently use 0.9.1 in parquet-cpp but 0.9.3 seems to work too.
Did you explicitly exclude Thrift from being build in the third-party step?
Can you do a clean build and post the output of the cmake run. This should give us some insight on the location where it found Thrift.
Are you on Linux or Mac?

William Forson / @wdforson:
Hi Wes,
Yeah, that's a fair point. Sorry if my current output isn't as helpful as it could be – I'm trying to do a couple things at once right now. A few weeks back, I got to the point (through manually editing cmake files) where all of my thrift artifacts appeared to be getting picked up during the cmake step, after which I got the compilation error shown above in my original comment...but I totally understand that no one here can provide specific guidance on that error until I get back to that point.
For the moment, I'm curious about the possibility of building thrift 0.9.3 via the parquet-cpp 3p dependency build process. Uwe L. Korn mentioned that "0.9.3 seems to work too", and I see a comment about "Thrift 0.9.3" in thirdparty/build_thirdparty.sh. Is there a flag, environment variable, etc that can be used to build thrift 0.9.3? (if so, this might solve my problems completely, at least for the moment)
Thanks!

William Forson / @wdforson:
UPDATE: almost there! Only issue I'm now having is that make install didn't copy libparquet_thrift.a to the installation target.
Obviously it's easy enough to augment my build logic to do that copy, but out of curiosity: is there any reason that lib should not be needed? Without it, my builds fail at link-time due to undefined references (e.g. parquet::format::SchemaElement::__set_scale(int)).

William Forson / @wdforson:
(I mean, I guess I can just munge thirdparty/versions.sh)

William Forson / @wdforson:

I'm really sorry for posting my last couple comments more or less in realtime, rather than waiting until I'd exhausted my own attempts to get things working – I blame this on slack!
Anyway, I have now come full circle, and I'm back to getting the compilation error shown in my original post. excerpt:

parquet-cpp/src/parquet/thrift/parquet_types.cpp:2646:3: error: ‘TInputRecursionTracker’ is not a member of ‘apache::thrift::protocol’
apache::thrift::protocol::TInputRecursionTracker tracker(*iprot);
^

At any rate, I'll hold off posting anything else here until I'm more confident that I'm not doing anything obviously wrong.

William Forson / @wdforson:
Thanks for the response!
It's a shame I didn't post my original issue to the correct page, as I spent some time playing around with this last month but have forgotten everything by now.
Anyway, I'm attaching the cmake output with THRIFT_HOME set, but without any other tweaks, as cmake_output_original.txt. At this point, the thrift-finding cmake logic picks up my system libraries rather than the ones under $THRIFT_HOME/lib. (IIRC it was reasonably easy to get around this by tweaking the cmake logic, btw) For reference:

$ ls -R thrift_home/
thrift_home/:
bin  include  lib
thrift_home/bin:
thrift
thrift_home/include:
thrift
thrift_home/include/thrift:
async        cxxfunctional.h  qt                       TDispatchProcessor.h  TLogging.h    transport
concurrency  processor        server                   thrift-config.h       TOutput.h     TToString.h
config.h     protocol         TApplicationException.h  Thrift.h              TProcessor.h
thrift_home/include/thrift/async:
TAsyncBufferProcessor.h  TAsyncDispatchProcessor.h  TAsyncProtocolProcessor.h    TEvhttpClientChannel.h
TAsyncChannel.h          TAsyncProcessor.h          TConcurrentClientSyncInfo.h  TEvhttpServer.h
thrift_home/include/thrift/concurrency:
BoostThreadFactory.h  Monitor.h                PosixThreadFactory.h  StdThreadFactory.cpp  ThreadManager.h
Exception.h           Mutex.h                  StdMonitor.cpp        StdThreadFactory.h    TimerManager.h
FunctionRunner.h      PlatformThreadFactory.h  StdMutex.cpp          Thread.h              Util.h
thrift_home/include/thrift/processor:
PeekProcessor.h  StatsProcessor.h  TMultiplexedProcessor.h
thrift_home/include/thrift/protocol:
TBase64Utils.h       TCompactProtocol.h    TJSONProtocol.h         TProtocolException.h  TVirtualProtocol.h
TBinaryProtocol.h    TCompactProtocol.tcc  TMultiplexedProtocol.h  TProtocol.h
TBinaryProtocol.tcc  TDebugProtocol.h      TProtocolDecorator.h    TProtocolTap.h
thrift_home/include/thrift/qt:
TQIODeviceTransport.h  TQTcpServer.h
thrift_home/include/thrift/server:
TConnectedClient.h    TServerFramework.h  TSimpleServer.h    TThreadPoolServer.h
TNonblockingServer.h  TServer.h           TThreadedServer.h
thrift_home/include/thrift/transport:
PlatformSocket.h     THttpServer.h     TServerTransport.h      TSSLServerSocket.h     TVirtualTransport.h
TBufferTransports.h  THttpTransport.h  TShortReadTransport.h   TSSLSocket.h           TZlibTransport.h
TFDTransport.h       TPipe.h           TSimpleFileTransport.h  TTransportException.h
TFileTransport.h     TPipeServer.h     TSocket.h               TTransport.h
THttpClient.h        TServerSocket.h   TSocketPool.h           TTransportUtils.h
thrift_home/lib:
libthrift-0.9.3.so libthrift.la          libthriftqt.a   libthriftqt.so libthriftz-0.9.3.so libthriftz.la
libthrift.a         libthriftqt-0.9.3.so libthriftqt.la libthrift.so    libthriftz.a         libthriftz.so

Wes McKinney / @wesm:
There are still problems with your environment – TInputRecursionTracker is in Thrift 0.9.3 but not Thrift 0.9.0 or earlier versions. This suggests that while parquet.thrift is being compiled with Thrift 0.9.3, the wrong Thrift headers are being included in the gcc invocation. Please attach a complete cmake log starting from a clean out-of-source build or in-source build from a clean git checkout. Can you also run with make VERBOSE=1 so that the log shows the gcc commands?

William Forson / @wdforson:
Thanks – I'll check this out and give THRIFT_HOME another try.
re. in-source vs out-of-source builds: good point, but my project's 3p build logic rebuilds the world from scratch every time. (however, it seems possible that doing in-source builds affected my earlier efforts to get THRIFT_HOME working. although I was diligently purging CMakeCache.txt, stale artifacts may very well have been biting me elsewhere)

Wes McKinney / @wesm:
@wdforson I've been setting custom $THRIFT_HOME for a long time with no issues. Perhaps the .. in your path is the problem? Have you tried putting the absolute path to the Thrift installation instead? Inside cmake_modules/FindThrift.cmake, we have:
find_path(THRIFT_INCLUDE_DIR thrift/Thrift.h HINTS
  $ENV{THRIFT_HOME}/include/
  /usr/local/include/
  /opt/local/include/
)

So, it seems that find_path doesn't like your environment variable. Removing your /usr/local Thrift may help with debugging. Also, be sure to delete CMakeCache.txt – I recommend doing out-of-source builds for better consistency (https://github.com/apache/parquet-cpp#out-of-source-builds)

Wes McKinney / @wesm:
Here is an example of an out of source build with all dependencies specified as environment variables with the absolute path to the installation location:
https://github.com/conda-forge/parquet-cpp-feedstock/blob/master/recipe/build.sh
This picks up the THRIFT_HOME set in the Circle CI environemtn. So I recommend:

Make sure you're using an up to date version of cmake
Specify absolute paths (no .. in the environment variable)
Do out-of-source builds (you're doing in-source builds currently)
Make sure to delete CMakeCache.txt after changing any aspect of your environment

William Forson / @wdforson:
Wes: now that is exactly the answer I needed, to the question I wasn't smart enough to ask (or rather, which I had asked early on, but hastily mis-answered...without ever looking back).
As to the proximal cause of this most recent iteration of the error: when I previously deleted my system thrift libraries, I forgot about the headers, and evidently the parquet_thrift build was picking those up. (somewhat interestingly, I had also overlooked the system's thrift compiler executable, but apparently the build still picked up the 0.9.3 compiler under my customized THRIFT_HOME).
Thank you so much for putting up with my rambling back-and-forth and helping me in spite of myself!


text:
[C++][FlightRPC] Reduce code repetition in flight/types.cc with templates

Describe the enhancement requested
Local (to the .cc) templates can be used to reduce repetition in serde code.
Component(s)
C++, FlightRPC

Issue resolved by pull request 43237
https://github.com/apache/arrow/pull/43237


text:
[Python] Failed to compile ARM64 PyArrow for Windows ARM

May I get your support on compiling ARM64 PyArrow.
I’m trying to compile it through the below steps on Windows on ARM device. When I run the last line command ‘python setup.py build_ext –inplace’ in step 5 to compile the PyArrow(Python extension), I get many error messages like below:
  python_to_arrow.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: __cdecl arrow::LargeBinaryBuilder::LargeBinaryBuilder(class std::shared_ptr<class arrow::DataType> const &,class arrow::MemoryPool *)" (__imp_??0LargeBinaryBuilder@arrow@@QEAA@AEBV?$shared_ptr@VDataType@arrow@@@std@@PEAVMemoryPool@1@@Z) referenced in function "protected: virtual class arrow::Status __cdecl arrow::internal::PrimitiveConverter<class arrow::LargeBinaryType,class arrow::py::`anonymous namespace'::PyConverter>::Init(class arrow::MemoryPool *)" (?Init@?$PrimitiveConverter@VLargeBinaryType@arrow@@VPyConverter@
?A0xc3172271@py@2@@internal@arrow@@MEAA?AVStatus@3@PEAVMemoryPool@3@@Z) [C:\source\arrow\python\build\temp.win-arm64-cpython-312\arrow_python.vcxproj]

By comparing the ‘DUMPBIN’ result of ‘arrow.lib’ between x64 & arm64, the ARM64 lib missed the function with the parameter 'class std::shared_ptr const &,class arrow::MemoryPool *' in 'arrow.lib':
LargeBinaryBuilder is a template class. Do you have idea why the function missed while compiling it to ARM64? And how to fix it? Thanks in advance!
/// \class LargeBinaryBuilder
/// \brief Builder class for large variable-length binary data
class ARROW_EXPORT LargeBinaryBuilder : public BaseBinaryBuilder<LargeBinaryType> {
public:
  using BaseBinaryBuilder::BaseBinaryBuilder;

  /// \cond FALSE
  using ArrayBuilder::Finish;
  /// \endcond

  Status Finish(std::shared_ptr<LargeBinaryArray>* out) { return FinishTyped(out); }

  std::shared_ptr<DataType> type() const override { return large_binary(); }
};

Due to the reason mentioned in below link, I compiled the Cpp library with clang-cl for ARM64, but compiled the x64 version with MSVC cl.
https://arrow.apache.org/docs/developers/cpp/windows.html#building-on-windows-arm64-using-ninja-and-clang
The detailed steps:
1. Install ARM64 Python 3.12.6 and necessary Python extension.

2. Open "ARM64 Native Tools Command Prompt for VS 2022" command line and run below commands:

cd C:\source
git clone https://github.com/apache/arrow.git

3. Add below patch:
Patch 1:
diff --git a/cpp/src/arrow/CMakeLists.txt b/cpp/src/arrow/CMakeLists.txt
index c911f0f4e..ddd4dc0bb 100644
--- a/cpp/src/arrow/CMakeLists.txt
+++ b/cpp/src/arrow/CMakeLists.txt
@@ -955,7 +955,7 @@ if(CXX_LINKER_SUPPORTS_VERSION_SCRIPT)
endif()

if(ARROW_BUILD_STATIC AND ARROW_BUNDLED_STATIC_LIBS)
-  set(ARROW_BUILD_BUNDLED_DEPENDENCIES TRUE)
+  set(ARROW_BUILD_BUNDLED_DEPENDENCIES FALSE)
else()
   set(ARROW_BUILD_BUNDLED_DEPENDENCIES FALSE)
endif()

Patch 2:
diff --git a/python/setup.py b/python/setup.py
index 60b9a696d..6cde02919 100755
--- a/python/setup.py
+++ b/python/setup.py
@@ -165,7 +165,7 @@ class build_ext(_build_ext):
         _build_ext.initialize_options(self)
         self.cmake_generator = os.environ.get('PYARROW_CMAKE_GENERATOR')
         if not self.cmake_generator and sys.platform == 'win32':
-            self.cmake_generator = 'Visual Studio 15 2017 Win64'
+            self.cmake_generator = 'Visual Studio 17 2022'
         self.extra_cmake_args = os.environ.get('PYARROW_CMAKE_OPTIONS', '')
         self.build_type = os.environ.get('PYARROW_BUILD_TYPE',
                                          'release').lower()

4. Run commands below to compile arrow Cpp code:

set ARROW_HOME=C:\source\arrow\Install
set CMAKE_PREFIX_PATH=C:\source\arrow\Install

mkdir arrow\cpp\build
pushd arrow\cpp\build

set CC=clang-cl
set CXX=clang-cl

cmake -G "Ninja" -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% -DCMAKE_UNITY_BUILD=ON -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_DATASET=ON -DARROW_FILESYSTEM=ON -DARROW_HDFS=ON -DARROW_JSON=ON -DARROW_PARQUET=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON ..
cmake --build . --target install --config Release

popd

5. Run commands below to compile Pybind code:

pushd arrow\python
set PYARROW_BUNDLE_ARROW_CPP=1
python setup.py build_ext --inplace

Component(s)
C++, Python

Could you show the full build log for PyArrow?
FYI: You don't need the second patch by set PYARROW_CMAKE_GENERATOR=Visual Studio 17 2022.

PyArrow.compile.fail.zip
Thanks @kou ! I've attached the PyArrow compiling log together with the x64 & arm64 'arrow.lib' dump log.
It seems many functions haven't been compiled into arm64 version 'arrow.lib'.

It seems that Arrow C++ uses clang-cl but PyArrow doesn't use clang-cl.
Can we use clang-cl for PyArrow too?

Thanks for your suggestion.
After modify the code as below and compile again:
diff --git a/python/setup.py b/python/setup.py
index 60b9a696d..b75adb0fa 100755
--- a/python/setup.py
+++ b/python/setup.py
@@ -165,7 +165,7 @@ class build_ext(_build_ext):
         _build_ext.initialize_options(self)
         self.cmake_generator = os.environ.get('PYARROW_CMAKE_GENERATOR')
         if not self.cmake_generator and sys.platform == 'win32':
-            self.cmake_generator = 'Visual Studio 15 2017 Win64'
+            self.cmake_generator = 'Ninja'

I got the link error below. After remove the content of function 'arrow::gdb::TestSession', I can compile the PyArrow successfully.
I'll test if the basic functions work.
[53/72] Linking CXX shared library arrow_python.dll
FAILED: arrow_python.dll arrow_python.lib
C:\windows\system32\cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=CMakeFiles\arrow_python.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\arm64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\arm64\mt.exe --manifests  -- C:\PROGRA~1\LLVM\bin\lld-link.exe /nologo CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\arrow_to_pandas.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\benchmark.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\common.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\datetime.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\decimal.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\deserialize.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\extension_type.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\gdb.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\helpers.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\inference.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\io.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\ipc.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\numpy_convert.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\numpy_init.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\numpy_to_arrow.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\python_test.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\python_to_arrow.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\pyarrow.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\serialize.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\udf.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\csv.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\filesystem.cc.obj  /out:arrow_python.dll /implib:arrow_python.lib /pdb:arrow_python.pdb /dll /version:0.0 /machine:ARM64  /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO  C:\source\arrow\Install\lib\arrow_dataset.lib  C:\source\arrow\Install\lib\arrow_acero.lib  C:\source\arrow\Install\lib\parquet.lib  C:\source\arrow\Install\lib\arrow.lib  ws2_32.lib  C:\Programs\Python\Python312-arm64\libs\python312.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
LINK: command "C:\PROGRA~1\LLVM\bin\lld-link.exe /nologo CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\arrow_to_pandas.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\benchmark.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\common.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\datetime.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\decimal.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\deserialize.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\extension_type.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\gdb.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\helpers.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\inference.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\io.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\ipc.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\numpy_convert.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\numpy_init.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\numpy_to_arrow.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\python_test.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\python_to_arrow.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\pyarrow.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\serialize.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\udf.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\csv.cc.obj CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\filesystem.cc.obj /out:arrow_python.dll /implib:arrow_python.lib /pdb:arrow_python.pdb /dll /version:0.0 /machine:ARM64 /NODEFAULTLIB:LIBCMT /INCREMENTAL:NO C:\source\arrow\Install\lib\arrow_dataset.lib C:\source\arrow\Install\lib\arrow_acero.lib C:\source\arrow\Install\lib\parquet.lib C:\source\arrow\Install\lib\arrow.lib ws2_32.lib C:\Programs\Python\Python312-arm64\libs\python312.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST:EMBED,ID=2" failed (exit code 1) with the following output:
lld-link: error: undefined symbol: __declspec(dllimport) public: __cdecl arrow::TimeScalar<class arrow::Time32Type>::TimeScalar<class arrow::Time32Type>(int, enum arrow::TimeUnit::type)
>>> referenced by CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\gdb.cc.obj:(void __cdecl arrow::gdb::TestSession(void))
>>> referenced by CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\gdb.cc.obj:(void __cdecl arrow::gdb::TestSession(void))

lld-link: error: undefined symbol: __declspec(dllimport) public: __cdecl arrow::TimeScalar<class arrow::Time64Type>::TimeScalar<class arrow::Time64Type>(__int64, enum arrow::TimeUnit::type)
>>> referenced by CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\gdb.cc.obj:(void __cdecl arrow::gdb::TestSession(void))
>>> referenced by CMakeFiles\arrow_python.dir\pyarrow\src\arrow\python\gdb.cc.obj:(void __cdecl arrow::gdb::TestSession(void))

OK. Could you open a PR that updates https://arrow.apache.org/docs/developers/python.html ? We need to add https://arrow.apache.org/docs/developers/cpp/windows.html#building-on-windows-arm64-using-ninja-and-clang like document to there.
For the cmake_generator change: We can use set PYARROW_CMAKE_GENERATOR=Ninja as I mentioned at https://github.com/apache/arrow/issues/44310#issuecomment-2395321324 .
For the link error: Could you try the following?
diff --git a/cpp/src/arrow/scalar.h b/cpp/src/arrow/scalar.h
index 7a273c46c1..a4fd9453ef 100644
--- a/cpp/src/arrow/scalar.h
+++ b/cpp/src/arrow/scalar.h
@@ -464,7 +464,7 @@ struct ARROW_EXPORT Date64Scalar : public DateScalar<Date64Type> {
 };

 template <typename T>
-struct ARROW_EXPORT TimeScalar : public TemporalScalar<T> {
+struct TimeScalar : public TemporalScalar<T> {
   using TemporalScalar<T>::TemporalScalar;

   TimeScalar(typename TemporalScalar<T>::ValueType value, TimeUnit::type unit)

@kou I can compile it successfully by below steps. Need modify the code to disable 'ARROW_BUILD_BUNDLED_DEPENDENCIES':
If not disable this, will get this error('arrow_bundled_dependencies.lib' can't be found, it hasn't be compiled.):
[190/191] Install the project...-- Install configuration: "RELEASE"
-- Installing: C:/zhanweiw/source/Python/Src/arrow/Install/include/arrow/util/config.h
CMake Error at src/arrow/cmake_install.cmake:40 (file):
  file INSTALL cannot find
  "C:/zhanweiw/source/Python/Src/arrow/cpp/build/release/arrow_bundled_dependencies.lib":
  File exists.
Call Stack (most recent call first):
  cmake_install.cmake:37 (include)

Install ARM64 Python 3.12.6 and necessary Python extension.
Install LLVM.
Open "ARM64 Native Tools Command Prompt for VS 2022" command line and run below commands:

cd C:\source
git clone https://github.com/apache/arrow.git

Add below patch:
diff --git a/cpp/src/arrow/CMakeLists.txt b/cpp/src/arrow/CMakeLists.txt
index c911f0f4e..ddd4dc0bb 100644
--- a/cpp/src/arrow/CMakeLists.txt
+++ b/cpp/src/arrow/CMakeLists.txt
@@ -955,7 +955,7 @@ if(CXX_LINKER_SUPPORTS_VERSION_SCRIPT)
 endif()

 if(ARROW_BUILD_STATIC AND ARROW_BUNDLED_STATIC_LIBS)
-  set(ARROW_BUILD_BUNDLED_DEPENDENCIES TRUE)
+  set(ARROW_BUILD_BUNDLED_DEPENDENCIES FALSE)
 else()
   set(ARROW_BUILD_BUNDLED_DEPENDENCIES FALSE)
 endif()

diff --git a/cpp/src/arrow/scalar.h b/cpp/src/arrow/scalar.h
index 7a273c46c..a4fd9453e 100644
--- a/cpp/src/arrow/scalar.h
+++ b/cpp/src/arrow/scalar.h
@@ -464,7 +464,7 @@ struct ARROW_EXPORT Date64Scalar : public DateScalar<Date64Type> {
 };

 template <typename T>
-struct ARROW_EXPORT TimeScalar : public TemporalScalar<T> {
+struct TimeScalar : public TemporalScalar<T> {
   using TemporalScalar<T>::TemporalScalar;

   TimeScalar(typename TemporalScalar<T>::ValueType value, TimeUnit::type unit)

diff --git a/python/setup.py b/python/setup.py
index 60b9a696d..36c3afa9f 100755
--- a/python/setup.py
+++ b/python/setup.py
@@ -400,6 +400,7 @@ setup(
     distclass=BinaryDistribution,
     # Dummy extension to trigger build_ext
     ext_modules=[Extension('__dummy__', sources=[])],
+    version="17.0.0",
     cmdclass={
         'build_ext': build_ext
     },

Run commands below to compile arrow Cpp code:

set ARROW_HOME=C:\source\arrow\Install
set CMAKE_PREFIX_PATH=C:\source\arrow\Install
set PYARROW_CMAKE_GENERATOR=Ninja

mkdir arrow\cpp\build
pushd arrow\cpp\build

set CC=clang-cl
set CXX=clang-cl

cmake -G "Ninja" -DCMAKE_INSTALL_PREFIX=%ARROW_HOME% -DCMAKE_UNITY_BUILD=ON -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_DATASET=ON -DARROW_FILESYSTEM=ON -DARROW_HDFS=ON -DARROW_JSON=ON -DARROW_PARQUET=ON -DARROW_WITH_LZ4=ON -DARROW_WITH_SNAPPY=ON -DARROW_WITH_ZLIB=ON -DARROW_WITH_ZSTD=ON ..

cmake --build . --target install --config Release

popd

Run commands below to compile PyArrow:

pushd arrow\python
set PYARROW_BUNDLE_ARROW_CPP=1
python setup.py build_ext --inplace
python setup.py bdist_wheel
popd

We can work on the second diff (cpp/src/arrow/scalar.h) in #44364.
Could you open a new issue for the first diff (cpp/src/arrow/CMakeLists.txt)? Let's work on it as a separated task.
Why do we need the third diff (python/setup.py)?

Without the third diff, I'll get 'pyarrow-0-cp312-cp312-win_arm64.whl'.
With the change, I'll get 'pyarrow-17.0.0-cp312-cp312-win_arm64.whl'.

I've created new ticket for building 'arrow_bundled_dependencies.lib' issue:
https://github.com/apache/arrow/issues/44368

Without the third diff, I'll get 'pyarrow-0-cp312-cp312-win_arm64.whl'. With the change, I'll get 'pyarrow-17.0.0-cp312-cp312-win_arm64.whl'.

Hmm. It's strange.
https://github.com/apache/arrow/blob/dcc1ee5b1d4851870724ab5e4cf475bcac007b56/python/pyproject.toml#L79-L84
should be used for version information.
@jorisvandenbossche Do you have any idea why this was happen? ("the third diff" is a diff in https://github.com/apache/arrow/issues/44310#issuecomment-2402288125 .)

@kou
I can't find file 'pyarrow/_generated_version.py' from build:
version_file = 'pyarrow/_generated_version.py'

And it seems no code is using "fallback_version" in ARM64 windows:
https://github.com/search?q=repo%3Aapache%2Farrow+fallback_version&type=code

I think that pyarrow/_generated_version.py is generated automatically.
I think that fallback_version is used by "setuptools" not PyArrow.
Could you try python -m pip install . instead of  python setup.py build_ext --inplace and python setup.py bdist_wheel?

I think that pyarrow/_generated_version.py is generated automatically.
I think that fallback_version is used by "setuptools" not PyArrow.
Could you try python -m pip install . instead of python setup.py build_ext --inplace and python setup.py bdist_wheel?

Got below issue with above command:
           [114/334] Compiling C object numpy/_core/_multiarray_tests.cp312-win_arm64.pyd.p/meson-generated__multiarray_tests.c.obj
            FAILED: numpy/_core/_multiarray_tests.cp312-win_arm64.pyd.p/meson-generated__multiarray_tests.c.obj
            "clang-cl" "-Inumpy\_core\_multiarray_tests.cp312-win_arm64.pyd.p" "-Inumpy\_core" "-I..\numpy\_core" "-I..\numpy\_core\src\multiarray" "-I..\numpy\_core\src\npymath" "-Inumpy\_core\include" "-I..\numpy\_core\include" "-I..\numpy\_core\src\common" "-IC:\Programs\Python\Python312-arm64\Include" "-IC:\Users\zhanw\AppData\Local\Temp\pip-install-l3cp_q1s\numpy_e774c5d2837a4788b968dbaa52f05202\.mesonpy-8gh30u70\meson_cpu" "-DNDEBUG" "/MD" "/nologo" "/showIncludes" "/utf-8" "/W2" "/clang:-std=c11" "/O2" "/Gw" "-fno-strict-aliasing" "/clang:-ftrapping-math" "-DNPY_HAVE_CLANG_FPSTRICT" "-DNPY_HAVE_NEON_VFPV4" "-DNPY_HAVE_NEON_FP16" "-DNPY_HAVE_NEON" "-DNPY_HAVE_ASIMD" "-DNPY_INTERNAL_BUILD" "-DHAVE_NPY_CONFIG_H" "-D_FILE_OFFSET_BITS=64" "-D_LARGEFILE_SOURCE=1" "-D_LARGEFILE64_SOURCE=1" "/Fdnumpy\_core\_multiarray_tests.cp312-win_arm64.pyd.p\meson-generated__multiarray_tests.c.pdb" /Fonumpy/_core/_multiarray_tests.cp312-win_arm64.pyd.p/meson-generated__multiarray_tests.c.obj "/c" numpy/_core/_multiarray_tests.cp312-win_arm64.pyd.p/_multiarray_tests.c
            ..\numpy\_core\src\multiarray\_multiarray_tests.c.src(1883,17): error: invalid operand in inline asm: 'fstcw ${0:w}'
             1883 |         __asm__("fstcw %w0" : "=m" (cw));
                  |                 ^
            ..\numpy\_core\src\multiarray\_multiarray_tests.c.src(1883,17): error: unrecognized instruction mnemonic
            <inline asm>(1,2): note: instantiated into assembly here
                1 |         fstcw
                  |         ^
            2 errors generated.

authors: kou zhanweiw


text:
[C#][Integration] Set up integration testing of C# Flight

Describe the enhancement requested
Archery includes Flight integration tests but not for C#. These would be useful for catching issues like #44360, although it appears that DoExchange is not currently included in the integration tests.
Component(s)
C#, Integration

take

Issue resolved by pull request 44377
https://github.com/apache/arrow/pull/44377

authors: CurtHagenlocher adamreeve


text:
ARROW-2400: [C++] Fix Status destructor performance

On certain compilers, splitting off the slow path improves performance significantly.

Codecov Report

Merging #2154 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2154      +/-   ##
==========================================
+ Coverage   86.39%   86.39%   +<.01%
==========================================
  Files         230      230
  Lines       40701    40706       +5
==========================================
+ Hits        35162    35167       +5
  Misses       5539     5539

Impacted Files
Coverage Δ

cpp/src/arrow/status.h
95.29% <100%> (+0.29%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 884474c...2b7ab66. Read the comment docs.

authors: codecov-io pitrou


text:
PARQUET-1402: [C++]: parquet-mr writes zero dictionary_page_offset

when it is (supposed?) equal to data_page_offset

Could you provide more colour on this change?

Closing in favor of #4359 which includes a test case. Thanks for reporting @rip-nsk!

authors: fsaintjacques rip-nsk wesm


text:
GH-35404: Temporarily pin sphinx to 5.x to fix the doc build

See https://github.com/apache/arrow/issues/35403 about requiring some fixes on our side to support sphinx 6. Until then, let's pin sphinx to <6.

Closes: #35404

@github-actions crossbow submit preview-docs

@github-actions crossbow submit preview-docs

This seems to have worked: http://crossbow.voltrondata.com/pr_docs/35405/


text:
AVRO-3404: idl syntax for avsc equivalent

Make sure you have checked all steps below.
Jira

[X] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-3404
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[X] My PR extends the IdlReader ~adds the following~ unit tests ~OR does not need testing for this extremely good reason~:

Commits

[X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[X] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

This PR requires #1588 to be merged first.

Thanks for the rebase!  Finally we're done 1.11.1, let's get this in for 1.12 !

authors: RyanSkraba opwvhk

repo: apache/avro


text:
AVRO-2514:Making Avro serialization error message more actionable with...

field name
Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Avro Jira issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

https://issues.apache.org/jira/browse/AVRO-2514
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

Sorry for the slow response, thank you for your review, I use generics just to adapt to the three exceptions, but I neglected that they are all runtime exceptions, so I updated it.

Sorry for the slow response, thank you for your review. I am using generics just to adapt to three exceptions, but I can't catch runtime exceptions here, which will make them backward compatible, because writeWithoutConversion will recursively handle null pointer exceptions and supplement the exception information.
Now I updated it.

Thanks for the update!  I think I see the problem with catching all runtime exceptions (some runtime exceptions should not be decorated).
What do you think about listing the types of errors that can be decorated in a unit test?  It would be awesome if there was an exhaustive list, but even the common cases would be helpful!


text:
[Feature Request]: Update the default behaviour in BigQueryIO.Write.Method.DEFAULT to STORAGE_API_AT_LEAST_ONCE

What would you like to happen?
The default behaviour of  BigQueryIO.Write.Method for unbounded collections is to use STREAMING_INSERTS, which is now categorised as legacy .
Two new methods STORAGE_API_AT_LEAST_ONCE and STORAGE_WRITE_API are available, being STORAGE_API_AT_LEAST_ONCE the closest in the underlying semantics to STREAMING_INSERTS (best effort deduplication but no guarantees of only once). Using the storage API is also cheaper than the legacy streaming inserts by 50%, with the first 2TB free..
Should the default method point to STORAGE_API_AT_LEAST_ONCE instead of keep using STREAMING_INSERTS?
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[X] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

We usually do not change the default. For this case, the recommended way is to use Managed IO once we onboard BigQuery IO to it.

authors: borjavb liferoad

repo: apache/beam


text:
Remove testSelectFromTableWithMap from ZetaSQLDialectSpecTest

Although table with a map field is supported by SchemaBaseBeamTable, ZetaSQL does not have a map type.
r: @apilloud
cc: @kennknowles @amaliujia
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

This test is still useful as it is testing what will happen when reading from a table that has unsupported type (aka. MAP). Current expected result is the map column will be skipped.

Interesting point. It may add convenience to Beam SQL to be able to process rows that include unsupported types. Because Beam schemas are not limited to a particular SQL dialect this will apply for SqlTransform for sure. I'm not sure what the right answer is. This is a good question for the dev@ list. Options seem to be (1) error (2) error only if the user tries to "touch" the column otherwise it doesn't matter. The test should be named and commented to be about this, not about map.

Created JIRA to track this: https://issues.apache.org/jira/browse/BEAM-8593

The test should be named and commented to be about this, not about map.

Agree.

authors: amaliujia kennknowles robinyqiu


text:
[BEAM-14473] Throw error if using globally windowed, unbounded side input

Throws an error at pipeline construction time if a user tries to use a globally windowed, unbounded side input. This avoids users accidentally writing a streaming DoFn that would block until the end of the global window and prompts them to window the side input. Also adds a unit test that checks the side input error conditions in TryParDo.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

authors: asf-ci jrmccluskey


text:
#20812 handle @RequiresStableInput in portable flink

Fixes #20812
Fixes #21333
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

assign set of reviewers

Run Java Flink PortableValidatesRunner Streaming

Run Flink ValidatesRunner

Run Java Flink PortableValidatesRunner Streaming

Run Flink ValidatesRunner

Run Java Flink PortableValidatesRunner Streaming

Run Flink ValidatesRunner

Run Java Flink PortableValidatesRunner Streaming

R: @lukecwik

@aaltay @robertwb @kennknowles Looks I'm unable to find a reviewer for this one. It fixes correctness bug in Flink, cau you please help find one? Thanks. :+1:

@kileys - Would you be able to review this? It is a medium sized change and does not require too much flink runner context.

Also @johnjcasey is interested in this feature working more broadly

@lukecwik thanks for the review! :+1:
I'll look into it in a few days.

@lukecwik I was looking into using the RequiresStableInputIT, unfortunately, it looks not easily reusable. It is somewhat used internally in some tests inside FlinkRunner (namely FlinkRequiresStableInputTest), but it is reused only partially. I could copy&paste the code and try to run it using Flink portable runner, but a better approach seems to be to turn it into @ValidatesRunner test, that could be turned on/off automatically. That looks like something orthogonal to this PR and thus I'd prefer to do it in different issue, so that we don't block this one. WDYT?

@lukecwik I was looking into using the RequiresStableInputIT, unfortunately, it looks not easily reusable. It is somewhat used internally in some tests inside FlinkRunner (namely FlinkRequiresStableInputTest), but it is reused only partially. I could copy&paste the code and try to run it using Flink portable runner, but a better approach seems to be to turn it into @ValidatesRunner test, that could be turned on/off automatically. That looks like something orthogonal to this PR and thus I'd prefer to do it in different issue, so that we don't block this one. WDYT?

I wouldn't say it is orthogonal. It is hard to have confidence that something works and isn't regressed without there being good test coverage. Your call if you still want to forge ahead without the tests.

I wouldn't say it is orthogonal. It is hard to have confidence that something works and isn't regressed without there being good test coverage. Your call if you still want to forge ahead without the tests.

100% agree with the test part. The current code has good coverage, but it builds upon the current implementation detail and it might break. Got your point, I will have a look if I can meaningfully extend the integration test and run in against both classic and portable runners. I'd definitely want to keep the original tests as well, because these tests should be more conclusive (under current implementation).

I managed to get the same test to run for both classic and portable runner. But ... there is an issue with Pipeline fusion, it seems it must respect the boundary given by @RequiresStableInput. Started discussion about that on the mailing list

@lukecwik is there any way to get logs of the failed pre commit?
https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/25421/
The tests pass locally, so without the log I'm not sure how to fix the problem. :-/

@lukecwik I finally managed to make the tests pass, so this is ready for final review. I will squash the commits after.

@lukecwik would you have time for review, or could you recommend someone please?

I'm glad that you were able to work through the problems and get the tests fixed and passing.

authors: aaltay je-ik kennknowles lukecwik