text:
[BEAM-11483] Ignore windowed GBK tests in Spark portable streaming.

see also BEAM-10784
R: @tysonjh
cc: @fernando-wizeline

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go

---
---

---

---

Java

Python

---

---

---

XLang

---

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Java Spark PortableValidatesRunner Streaming

Run Java PreCommit

Run Java PreCommit

I'm just going to merge this since I can't get Java PreCommit to pass and I'm reasonably sure this does not affect it.


text:
Added pandas to the INTERACTIVE_BEAM requirements.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @rohdesamuel
PTAL

facets-overview>=1.0.0,<2 includes pandas transitively. Closing this PR.

source: gharchive/pull-request

repo: apache/beam

license_type: permissive

license_source: github-api


text:
[BEAM-1220] Remove incubation references (master branch only)

R: @jbonofre, @dhalperi

LGTM

authors: davorbonaci dhalperi


text:
[BEAM-1409] Check that Elements, Timers have permitted Timestamps

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Add a checkArgument to ImmutableListBundleFactory that elements must
be timestamped earlier than the maximum representable timestamp.
Add a checkArgument to TimerData that Timers must fire before the
maximum representable timestamp.

R: @dhalperi @bjchambers (either)

LGTM ,please self-merge.

Coverage increased (+0.01%) to 69.595% when pulling ebc973254a8e85eb4e9476ed1778a068dd7436f8 on tgroh:checkArgument_not_past_infinity into a26fd1ff34bb5a51d6cc82ab9eb3c7e40cef49d3 on apache:master.

retest this please

Coverage increased (+0.003%) to 69.588% when pulling ebc973254a8e85eb4e9476ed1778a068dd7436f8 on tgroh:checkArgument_not_past_infinity into a26fd1ff34bb5a51d6cc82ab9eb3c7e40cef49d3 on apache:master.

authors: coveralls dhalperi tgroh


text:
Fix a small wrong url link in notebook

I'm new to this project and glad to contribute. 😊
This PR could correct the URL link in notebooks for "Tour of Beam".

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[x] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Thanks, however I see both https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/reading-and-writing-data.ipynb and https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/windowing.ipynb are existing links from my side. What kind of issue in your side for loading https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/reading-and-writing-data.ipynb ?

Thanks, however I see both https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/reading-and-writing-data.ipynb and https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/windowing.ipynb are existing links from my side. What kind of issue in your side for loading https://colab.sandbox.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/reading-and-writing-data.ipynb ?

Actually both links work well and there's no issue with loading that link.
The issue is: the heading of that line is "windowing", but when you click it, it will direct you to "reading-and-writing-data". I think this is a mismatch. :)


text:
Use java 11 for java_tests workflow

Fix #31100
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Successful run: https://github.com/apache/beam/actions/runs/8834529792

R: @stankiewicz @ahmedabu98


text:
[BEAM-4393]: Copy root repo's LICENSE & NOTICE into shadowJar

r: @kennknowles
cc: @apilloud

LGTM! I verified that this fixes BEAM-4393.
Only one question: should this go in jar instead of shadowJar?

Hey @apilloud , my understanding so far is, we use shadowJar to rewrite Jar. How do you feel like? @lukecwik

I believe we run the jar rule to build a jar with only our code, then we run the shadowJar rule to bundle that jar with third party code. This only needs to be in shadowJar, but that leaves our code in jar without a LICENSE file.

That seems wrong. The shadowJar is the real thing and the "normal" jar is just an intermediate calculation.

I didn't find any Jar{} closure so far in https://github.com/apache/beam/blob/master/build_rules.gradle, which means, we possibly use the default configuration. I'm not sure whether we need LICENSE in Jar.

Yea, OK. I didn't mean this PR is wrong. It seems this PR does put the LICENSE in the actual thing we want to ship to users. I guess the rest has to do with the release process.

Merging in the interest of moving things slightly better. If our current release process is mostly right, then this PR fixes it.

authors: apilloud boyuanzz kennknowles


text:
[BEAM-588] Implement profiling for Python DirectRunner and SDK Harness.

These are enabled with the existing --profile_xxx flags.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

R: @tweise

@robertwb thanks! This looks good to me, although I have not tried to use it myself. I hope that @rakeshcusat and @akshaybalwally will do so and offer some feedback.

Addressed comments and rebased. Though additional feedback would be good, I think this is at least a good start and we can continue to iterate with more experience.
The tools to install are printed along with the profile if profiling is enabled and they are not available.

Would be nice to mention this in the docs as follow-up.

Run Python PreCommit

Running precommits again before merging just to be sure due to issues with  #6903.

authors: robertwb tweise


text:
[BEAM-6115] SyntheticSource bundle size parameter sometimes should be casted to int

The bundle size parameter used in Python Synthetic Sources was becoming float instead of int. It caused failure while running load tests on Dataflow (load tests details here).

Follow this checklist to help us incorporate your contribution quickly and easily:

[ x ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

@lgajowy Could you take look at this small fix, please? cc: @pabloem (We discussed it - I explained the issue to Łukasz.)

Thanks y'all.

authors: kkucharc pabloem


text:
BIGTOP-4212: Add language prompt of AI Chat

https://issues.apache.org/jira/projects/BIGTOP/issues/BIGTOP-4212

Thanks @kevinw66

repo: apache/bigtop-manager


text:
BuildStream backgrounded after exiting a temporary shell

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Jun 25, 2017, 12:09
After shelling into a build directory as an option for dealing with a build failure in interactive mode, when exiting the shell; the master BuildStream process receives SIGSTOP somehow.
Need to track this down, the reason for this happening is unclear.
This does not happen with bst shell, only when running a pipeline where an element has failed.

In GitLab by [Gitlab user @juergbi] on Jul 21, 2017, 08:07
assigned to [Gitlab user @juergbi]

In GitLab by [Gitlab user @juergbi] on Jul 21, 2017, 08:08
mentioned in commit 1431e545adf73a5b35a2b28bcc4c542fd29001dc

In GitLab by [Gitlab user @juergbi] on Jul 21, 2017, 08:08
mentioned in merge request !62

In GitLab by [Gitlab user @juergbi] on Jul 25, 2017, 18:38
mentioned in commit 8f401127c450ea1c2a1e2878d2ac28a7166d355c

In GitLab by [Gitlab user @juergbi] on Jul 25, 2017, 18:56
closed via merge request !62

In GitLab by [Gitlab user @juergbi] on Jul 25, 2017, 18:56
closed via commit 8f401127c450ea1c2a1e2878d2ac28a7166d355c

source: gharchive/issue

repo: apache/buildstream


text:
CTRL+C doesn't properly kill a non-interactive shell

See original issue on GitLab
In GitLab by [Gitlab user @bochecha] on Sep 30, 2017, 16:47
During development, I'm running a just-built web app in a bst shell. I can test the app in a browser as I work on it.
This gives me the following processes:
  PID TTY      STAT   TIME COMMAND
 1075 ?        Ss     0:01 /usr/lib/systemd/systemd --user
 2120 ?        Rsl   26:43  \_ /usr/libexec/gnome-terminal-server
32328 pts/4    Ss     0:00  |   \_ bash
14880 pts/4    Sl+    0:01  |   |   \_ /usr/bin/python3 /home/mathieu/.local/bin/bst shell app.bst ideascube
14890 ?        Ss     0:00  |   |       \_ /usr/bin/python3 /home/mathieu/.local/bin/bst shell app.bst ideascube
14893 pts/4    S+     0:00  |   |       \_ /usr/bin/bwrap ...
14894 pts/4    S+     0:00  |   |           \_ /usr/bin/bwrap ...
14895 pts/4    Sl+    0:00  |   |               \_ /usr/bin/python3.6 /usr/bin/ideascube

The systemd one is just my user session, the gnome-terminal-server is because I use GNOME Terminal, and the bash one is the interactive shell in which I'm running bst shell .... I included them to show the process hierarchy, as it shows something further down.
When I'm done, I hit ctrl+c to kill the app, which printed Aborted! in the console, terminating the app.
However, this does not give me back my bash prompt, bst is just hung there:
  PID TTY      STAT   TIME COMMAND
 1075 ?        Ss     0:01 /usr/lib/systemd/systemd --user
 2120 ?        Ssl   26:49  \_ /usr/libexec/gnome-terminal-server
32328 pts/4    Ss     0:00  |   \_ bash
14880 pts/4    Sl+    0:01  |   |   \_ /usr/bin/python3 /home/mathieu/.local/bin/bst shell app.bst ideascube
14890 ?        Ss     0:00  |   |       \_ /usr/bin/python3 /home/mathieu/.local/bin/bst shell app.bst ideascube

If I press ctrl+c a second time, then the parent bst process gets stopped, and I get back to a prompt.
However, something is left lingering:
  PID TTY      STAT   TIME COMMAND
 1075 ?        Ss     0:01 /usr/lib/systemd/systemd --user
 2120 ?        Ssl   26:49  \_ /usr/libexec/gnome-terminal-server
32328 pts/4    Ss     0:00  |   \_ bash
14890 ?        Ss     0:00  \_ /usr/bin/python3 /home/mathieu/.local/bin/bst shell app.bst ideascube

One of the BuildStream processes never exited, and it got reparented to the session systemd process.

I found about #40, but that was for interactive shells, whereas here I'm running a non-interactive command.
I would pretty much expect BuildStream to just pass the ctrl+c to the child command. In most cases, the child command would exit.

In GitLab by [Gitlab user @bochecha] on Sep 30, 2017, 16:50
changed the description

In GitLab by [Gitlab user @tristanvb] on Oct 1, 2017, 06:27
You are certainly running an interactive shell if you are running bst shell.
Interestingly you are running into a scenario I did not test well but recently ensured was possible; i.e. you are running your program in the sandbox without a shell; so job control cannot happen there, and either BuildStream or the calling shell will be handling cntl-c
In the mean time, I'm not sure this will work well, but you might try a workaround of launching ideascube from a shell in the sandbox, e.g.:
bst shell app.bst sh -c ideascube

This should cause the shell in your sandbox (assuming you have one) to do the job control and exit in a way that BuildStream expects...

In GitLab by [Gitlab user @bochecha] on Oct 3, 2017, 15:03
So I tried the command you suggested, except it really was:
bst shell app.bst sh -- -c ideascube

(as you suggested it, the -c option was passed to BuildStream which complained it wasn't a valid option)
I got the exact same behaviour as before.

In GitLab by [Gitlab user @tlater] on Oct 11, 2017, 11:58
created branch 98-ctrl-c-doesn-t-properly-kill-a-non-interactive-shell

In GitLab by [Gitlab user @tlater] on Oct 11, 2017, 12:28
mentioned in merge request !106

In GitLab by [Gitlab user @tristanvb] on Oct 23, 2017, 05:28
This was fixed by !106

In GitLab by [Gitlab user @tristanvb] on Oct 23, 2017, 05:28
closed


text:
The interface SpringApplicationConfiguration is deprecated

From @SpringApplicationConfiguration(classes=MyConfig.class) to @SpringBootTest(classes=MyConfig.class)
Note: https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-1.4.0-M2-Release-Notes

Thanks for the PR it has been merged. Do you mind closing this?

Sure. Thank you!

You may have reopened this by mistake. Can you close?

authors: Pogorelov davsclaus

repo: apache/camel


text:
DO NOT MERGE - test for pr comment

Description

Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally from root folder and I have committed all auto-generated changes.

/component-test aws

/component-test aws

/component-test aws

/component-test aws

/component-test aws

/component-test aws


text:
CAMEL-21398: migrate the Smooks Data Format from the Camel cartridge to the Apache Camel code base

Description
Migrate the Smooks Data Format from the Smooks Camel Cartridge to the Apache Camel code base. See https://github.com/smooks/smooks-camel-cartridge/issues/211
Target

[x] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[x] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[x] I checked that each commit in the pull request has a meaningful subject line and body.

[x] I have run mvn clean install -DskipTests locally from root folder and I have committed all auto-generated changes.

Can you add this file to the PR so the job can build
(use "git add ..." to include in what will be committed)
docs/components/modules/dataformats/pages/smooks-dataformat.adoc

authors: cjmamo davsclaus


text:
[CARBONDATA-3332] Blocked concurrent compaction and update

Problem:
When update and compaction are executed concurrently then update is trying to update the contents of segment 0 whereas compaction has already marked the segment as COMPACTED. This compacted segment is of no use to the update command and therefore when trying to access the segment info from map it throws key not found.
Solution:
Block update and compaction concurrent operations.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2818/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2819/

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/11079/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3051/

Does "delete command" have this issue?
Shall we unify the load data lock ， such as update, compaction, delete and so on

@qiuchenjian Yes delete should have this problem but i was not able to reproduce. I will handle for delete as well.

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2821/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3053/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2822/

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/11082/

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3054/

@ravipesala Please review. Build passed

LGTM

authors: CarbonDataQA kunal642 qiuchenjian ravipesala

repo: apache/carbondata


text:
[WIP] Support creating MV datamap without giving filter columns in projection and bug fixes[

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1204/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1223/

Build Failed with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1213/

Build Failed  with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1205/

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1206/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1225/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1215/

@Indhumathi27 can you once remove the ignored tests from mvcreatetestcase class and check

@akashrn5 already changed ignored tests to test

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1232/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1242/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1251/

LGTM


text:
CLOUDSTACK-9554 Juniper Contrail plug-in is publishing events to wron…

Juniper Contrail plug-in is publishing events to message bus instead of event bus

LGTM. @yvsubhash can you rebase the branch and edit the PR to base branch 4.9?

@rhtyd How abt rebasing it against branch 4.8 and edit the PR to base it to the same?

okay

@rhtyd PR pointed to 4.8 branch

ACS CI BVT Run
Sumarry:
Build Number 130
Hypervisor xenserver
NetworkType Advanced
Passed=101
Failed=4
Skipped=6
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:

test_non_contigiousvlan.py

test_extendPhysicalNetworkVlan Failed

test_deploy_vm_iso.py

test_deploy_vm_from_iso Failing since 15 runs

test_nic.py

test_01_nic Failed

test_vm_life_cycle.py

test_10_attachAndDetach_iso Failing since 16 runs

Skipped tests:
test_01_test_vm_volume_snapshot
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_snapshots.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_routers_network_ops.py
test_disk_offerings.py

I don't have the specific hardware to trigger and run regression tests on the changes. Can we get a second code review and then we can merge this. /cc @karuturi @jburwell and others?

Merging based on code review and Travis, regression tests.

authors: cloudmonger rhtyd yvsubhash

repo: apache/cloudstack


text:
CLOUDSTACK-8886: Limitations is listUsageRecords output, listUsageRec…

As @kansal  is inactive created new branch and raised the PR. This is continuation of PR#858
https://github.com/apache/cloudstack/pull/858
Problem: Only domainid is returned by usageReports API call. In cloudstack documention it mentions "domain" as being in the usage response. The API should really be returning the domain as account information has both account and accountid.
Fix: Missing setDomainName at the time of creating response.

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-482

[INFO] Apache CloudStack Plugin - F5 ..................... SUCCESS [2.778s]
[INFO] Apache CloudStack Plugin - Juniper SRX ............ FAILURE [0.107s]
[INFO] Apache CloudStack VMware Base ..................... SKIPPED
....
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project cloud-plugin-network-srx: Could not resolve dependencies for project org.apache.cloudstack:cloud-plugin-network-srx:jar:4.10.0.0-SNAPSHOT: Could not find artifact com.cloud.com.f5:icontrol:jar:1.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]

@jayapalu looks like some dependency is missing

code LGTM

LGTM

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-483

@blueorangutan test

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

LGTM
API response:

admin
8c91c04e-f282-11e6-8a09-d4ae52cb9a54
8c91ab6a-f282-11e6-8a09-d4ae52cb9a54
ROOT
a6401f5b-b090-4a64-9d73-c04369d15ca8

VM-5e458101-b6a7-477e-9086-ee659ce0a700 running time (ServiceOffering: 1) (Template: 111)

0.12778 Hrs
1
0.12778
5e458101-b6a7-477e-9086-ee659ce0a700
VM-5e458101-b6a7-477e-9086-ee659ce0a700
2b10fa98-c167-444f-b9c0-752611a9f267
02567d08-f283-11e6-8a09-d4ae52cb9a54
5e458101-b6a7-477e-9086-ee659ce0a700
Simulator
2017-02-14'T'07:07:15+00:00
2017-02-14'T'07:15:00+00:00

Trillian test result (tid-818)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33181 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1939-t818-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Test completed. 47 look ok, 2 have error(s)

Test
Result
Time (s)
Test File

test_02_redundant_VPC_default_routes
Failure
858.87
test_vpc_redundant.py

test_04_rvpc_privategw_static_routes
Failure
350.74
test_privategw_acl.py

test_01_vpc_site2site_vpn
Success
165.29
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
66.22
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
255.96
test_vpc_vpn.py

test_02_VPC_default_routes
Success
277.32
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
537.76
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
511.00
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1454.05
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
543.71
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1276.54
test_vpc_redundant.py

test_09_delete_detached_volume
Success
151.44
test_volumes.py

test_08_resize_volume
Success
156.39
test_volumes.py

test_07_resize_fail
Success
161.44
test_volumes.py

test_06_download_detached_volume
Success
156.31
test_volumes.py

test_05_detach_volume
Success
155.78
test_volumes.py

test_04_delete_attached_volume
Success
151.19
test_volumes.py

test_03_download_attached_volume
Success
156.20
test_volumes.py

test_02_attach_volume
Success
95.37
test_volumes.py

test_01_create_volume
Success
711.56
test_volumes.py

test_03_delete_vm_snapshots
Success
275.19
test_vm_snapshots.py

test_02_revert_vm_snapshots
Success
95.73
test_vm_snapshots.py

test_01_create_vm_snapshots
Success
163.73
test_vm_snapshots.py

test_deploy_vm_multiple
Success
267.62
test_vm_life_cycle.py

test_deploy_vm
Success
0.03
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.02
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.56
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.25
test_vm_life_cycle.py

test_08_migrate_vm
Success
40.89
test_vm_life_cycle.py

test_07_restore_vm
Success
0.13
test_vm_life_cycle.py

test_06_destroy_vm
Success
130.79
test_vm_life_cycle.py

test_03_reboot_vm
Success
125.88
test_vm_life_cycle.py

test_02_start_vm
Success
10.16
test_vm_life_cycle.py

test_01_stop_vm
Success
40.31
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
40.55
test_templates.py

test_08_list_system_templates
Success
0.03
test_templates.py

test_07_list_public_templates
Success
0.04
test_templates.py

test_05_template_permissions
Success
0.05
test_templates.py

test_04_extract_template
Success
5.16
test_templates.py

test_03_delete_template
Success
5.10
test_templates.py

test_02_edit_template
Success
90.12
test_templates.py

test_01_create_template
Success
45.46
test_templates.py

test_10_destroy_cpvm
Success
161.57
test_ssvm.py

test_09_destroy_ssvm
Success
163.59
test_ssvm.py

test_08_reboot_cpvm
Success
101.46
test_ssvm.py

test_07_reboot_ssvm
Success
133.56
test_ssvm.py

test_06_stop_cpvm
Success
161.70
test_ssvm.py

test_05_stop_ssvm
Success
138.81
test_ssvm.py

test_04_cpvm_internals
Success
1.13
test_ssvm.py

test_03_ssvm_internals
Success
3.23
test_ssvm.py

test_02_list_cpvm_vm
Success
0.12
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.13
test_ssvm.py

test_01_snapshot_root_disk
Success
11.26
test_snapshots.py

test_04_change_offering_small
Success
240.79
test_service_offerings.py

test_03_delete_service_offering
Success
0.04
test_service_offerings.py

test_02_edit_service_offering
Success
0.08
test_service_offerings.py

test_01_create_service_offering
Success
0.17
test_service_offerings.py

test_02_sys_template_ready
Success
0.12
test_secondary_storage.py

test_01_sys_vm_start
Success
0.17
test_secondary_storage.py

test_09_reboot_router
Success
35.29
test_routers.py

test_08_start_router
Success
30.28
test_routers.py

test_07_stop_router
Success
10.16
test_routers.py

test_06_router_advanced
Success
0.06
test_routers.py

test_05_router_basic
Success
0.04
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.66
test_routers.py

test_03_restart_network_cleanup
Success
65.81
test_routers.py

test_02_router_internal_adv
Success
1.01
test_routers.py

test_01_router_internal_basic
Success
0.55
test_routers.py

test_router_dns_guestipquery
Success
76.75
test_router_dns.py

test_router_dns_externalipquery
Success
0.08
test_router_dns.py

test_router_dhcphosts
Success
276.87
test_router_dhcphosts.py

test_router_dhcp_opts
Success
22.04
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
0.09
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
130.90
test_reset_vm_on_reboot.py

test_createRegion
Success
0.04
test_regions.py

test_create_pvlan_network
Success
5.22
test_pvlan.py

test_dedicatePublicIpRange
Success
0.42
test_public_ip_range.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
479.77
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
365.52
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
92.24
test_privategw_acl.py

test_01_primary_storage_nfs
Success
35.78
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.18
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.43
test_portable_publicip.py

test_isolate_network_password_server
Success
89.29
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.13
test_over_provisioning.py

test_oobm_zchange_password
Success
30.63
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.31
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
10.24
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.32
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
10.30
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
15.31
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
15.31
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
15.32
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
87.50
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.17
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.09
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
5.17
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.10
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.09
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.08
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
23.40
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.30
test_non_contigiousvlan.py

test_01_nic
Success
429.19
test_nic.py

test_releaseIP
Success
253.18
test_network.py

test_reboot_router
Success
408.75
test_network.py

test_public_ip_user_account
Success
10.26
test_network.py

test_public_ip_admin_account
Success
40.28
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
67.07
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.74
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.14
test_network.py

test_delete_account
Success
272.80
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.74
test_network.py

test_01_port_fwd_on_src_nat
Success
111.92
test_network.py

test_nic_secondaryip_add_remove
Success
217.71
test_multipleips_per_nic.py

login_test_saml_user
Success
19.27
test_login.py

test_assign_and_removal_lb
Success
133.10
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
187.46
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
207.69
test_loadbalance.py

test_03_list_snapshots
Success
0.06
test_list_ids_parameter.py

test_02_list_templates
Success
0.04
test_list_ids_parameter.py

test_01_list_volumes
Success
0.03
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.06
test_iso.py

test_05_iso_permissions
Success
0.06
test_iso.py

test_04_extract_Iso
Success
5.18
test_iso.py

test_03_delete_iso
Success
95.17
test_iso.py

test_02_edit_iso
Success
0.06
test_iso.py

test_01_create_iso
Success
21.00
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
193.16
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
147.95
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
525.68
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
431.04
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.27
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.13
test_global_settings.py

test_rolepermission_lifecycle_update
Success
6.23
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
6.00
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
5.87
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.90
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
6.11
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.93
test_dynamicroles.py

test_role_lifecycle_update
Success
10.98
test_dynamicroles.py

test_role_lifecycle_list
Success
5.92
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.94
test_dynamicroles.py

test_role_lifecycle_create
Success
5.93
test_dynamicroles.py

test_role_inuse_deletion
Success
5.88
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
8.30
test_dynamicroles.py

test_role_account_acls
Success
8.29
test_dynamicroles.py

test_default_role_deletion
Success
6.00
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.07
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.04
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.05
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.07
test_disk_offerings.py

test_01_create_disk_offering
Success
0.10
test_disk_offerings.py

test_deployvm_userdispersing
Success
20.58
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
45.74
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
50.63
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
10.38
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
66.23
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
5.99
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
5.99
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
202.48
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
207.59
test_deploy_vm_iso.py

test_DeployVmAntiAffinityGroup
Success
66.14
test_affinity_groups.py

test_01_test_vm_volume_snapshot
Skipped
0.00
test_vm_snapshots.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.02
test_staticroles.py

test_11_ss_nfs_version_on_ssvm
Skipped
0.02
test_ssvm.py

test_01_scale_vm
Skipped
0.00
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.03
test_primary_storage.py

test_nested_virtualization_vmware
Skipped
0.00
test_nested_virtualization.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.03
test_deploy_vgpu_enabled_vm.py

test_3d_gpu_support
Skipped
0.03
test_deploy_vgpu_enabled_vm.py

merging

authors: blueorangutan borisstoyanov jayapalu karuturi kishankavala ustcweizhou


text:
Add limit configuration for number of projects

Description
The following PR creates a new feature of adding a limit to the number of projects of accounts within a singular domain. This helps administrators organize resources.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI

Feature/Enhancement Scale

[ ] Major
[x] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[x] Minor
[ ] Trivial

Screenshots (if appropriate):

How Has This Been Tested?
The following issue has been tested with several mocked test cases that specify domain max projects and account max projects with valid and invalid inputs. Finally, the UI has been verified using a local linux environment; see above image in screenshots.

@blueorangutan package

@borisstoyanov a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖ el7 ✖ el8 ✖ el9 ✖ debian ✖ suse15. SL-JID 9835

@my-code-AL can you look at the unit tests @borisstoyanov pointed at?

@DaanHoogland Will do! I've been busy at work but will fix soon!

@blueorangutan package

@my-code-AL a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 10330

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 10337

@blueorangutan LLtest alma9 kvm-alma9 keepEnv

@DaanHoogland a [LL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[LL]Trillian test result (tid-6953)
Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9
Total time taken: 50617 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9172-t6953-kvm-alma9.zip
Smoke tests completed. 134 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

ContextSuite context=TestClusterDRS>:setup
Error
0.00
test_cluster_drs.py

test_01_invalid_upgrade_kubernetes_cluster
Failure
0.01
test_kubernetes_clusters.py

test_02_upgrade_kubernetes_cluster
Failure
0.00
test_kubernetes_clusters.py

test_03_deploy_and_scale_kubernetes_cluster
Failure
0.00
test_kubernetes_clusters.py

test_04_autoscale_kubernetes_cluster
Failure
0.00
test_kubernetes_clusters.py

test_05_basic_lifecycle_kubernetes_cluster
Failure
0.00
test_kubernetes_clusters.py

test_06_delete_kubernetes_cluster
Failure
0.00
test_kubernetes_clusters.py

test_08_upgrade_kubernetes_ha_cluster
Failure
0.00
test_kubernetes_clusters.py

test_10_vpc_tier_kubernetes_cluster
Failure
0.00
test_kubernetes_clusters.py

test_11_test_unmanaged_cluster_lifecycle
Error
0.00
test_kubernetes_clusters.py

test_06_purge_expunged_vm_background_task
Failure
377.98
test_purge_expunged_vms.py

setting the limit to 1 an error occurs on creating the second:

after deleting the first prj the second gets created.
setting for a domain to two

a second user can create their own prj:

good work @my-code-AL , totally out of scope but I was wondering if there is any reason this setting is not dynamic (i.e. can be changed without restart) work for another day maybe ;)

setting the limit to 1 an error occurs on creating the second:
after deleting the first prj the second gets created.
setting for a domain to two
a second user can create their own prj:
good work @my-code-AL , totally out of scope but I was wondering if there is any reason this setting is not dynamic (i.e. can be changed without restart) work for another day maybe ;)

Will do! Thank you for your help and patience!

authors: DaanHoogland blueorangutan borisstoyanov my-code-AL


text:
UI: Add UI support for forvpc and guestiptype parameters of the listNetworkOfferings API

Description
Currently, the listNetworkOfferings API supports the forvpc and guestiptype parameters (see API docs). The first is used to filter network offerings that can only be used for creating VPC tiers; the second allows filtering by guest network type (available options are: Isolated, Shared and L2).
This PR adds support to these parameters in the UI. The forvpc filter has been added as a normal filter (declared in the filters property of the network offerings section's object) and the guestiptype as a search filter (declared in the searchFilters property).
Furthermore, this PR also fixes a bug that caused duplicate options to appear in search filter lists.

Fixes #9924
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[X] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI
[ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[X] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[X] Trivial

Screenshots (if appropriate):

How Has This Been Tested?
forvpc filter tests

I checked that when selecting VPC on the section's filter, a request to listNetworkOfferings was sent, with the forvpc parameter set to true.
I checked that when selecting Guest Network on the section's filter, a request to listNetworkOfferings was sent, with the forvpc parameter set to false.
I checked that when selecting All on the section's filter, a request to listNetworkOfferings was sent, without the forvpc parameter.

guestiptype filter tests

I checked that when selecting the Isolated option on the Guest Type search filter, a request to listNetworkOfferings was sent, with the guestiptype parameter set to Isolated.
I checked that when selecting the Shared option on the Guest Type search filter, a request to listNetworkOfferings was sent, with the guestiptype parameter set to Shared.
I checked that when selecting the L2 option on the Guest Type search filter, a request to listNetworkOfferings was sent, with the guestiptype parameter set to L2.

Duplicate options in search filter lists

Before

After

@blueorangutan ui

@bernardodemarco a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/9947 (QA-JID-481)

@blueorangutan ui

@JoaoJandre a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/9947 (QA-JID-486)

@blueorangutan ui

@kiranchavala a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/9947 (QA-JID-489)

authors: JoaoJandre bernardodemarco blueorangutan kiranchavala


text:
IMAGING-138  Provide some Ant tasks to perform image conversions

Adds a single Ant task of image conversion, tested on a small PNG file converted into ICNS.

If we want to support Ant, we should split Commons Imaging into a Maven multi-module project and put this into a new module to create jar called commons-imaging-ant IMO.

authors: garydgregory ghost

repo: apache/commons-imaging

license_source: bigquery


text:
Camera Not Working in Android 5.1/ Lollipop

Bug Report
Problem
What is expected to happen?
Camera wont open in android 5.1/lollipop
What does actually happen?
When opening camera error occurs illegal argument exception
Information

Command or Code

function capturePhoto() {
navigator.camera.getPicture(onPhotoURISuccess, onFail, {
quality: 90,
destinationType: destinationType.DATA_URL,
correctOrientation: true,
});
}
Environment, Platform, Device

Android 5.1 Emulator
Version information

Cordova 9.0
"cordova-plugin-camera": "^4.1.0",
Checklist

[ X] I searched for existing GitHub issues
[ X] I updated all Cordova tooling to most recent version
[ X] I included all the necessary information above

Are you using cordova-android@9? If so, can you retry with setting the preference android-targetSdkVersion to 28? Curious if this is an API level introduced issue.

Was there a fix for this? I'm also experiencing this problem.

@betagole I think this is an emulator issue. just edit your emulator camera settings to emulated

I wasn't using an emulator. I downgraded to @4.1.0 and it worked on my phone now.

hello please make sure you updated your webview, maybe some js not running correctly on 5.1

authors: betagole breautek cyrilxx

repo: apache/cordova-plugin-camera


text:
Add back elixir tests to make check

Overview
Add back the elixir tests to make check. I want to use this branch to try and fix any flaky tests.
Testing recommendations
Elixir tests should pass more consistently
Related Issues or Pull Requests
https://github.com/apache/couchdb/issues/1902
Checklist

[ ] Code is written and works correctly;
[ ] Changes are covered by tests;
[ ] Documentation reflects the changes;

I will try after travis is happy with this PR.

@garrensmith you broke Jenkins.
Good place to start: https://builds.apache.org/blue/organizations/jenkins/CouchDB/detail/master/499/pipeline
Some of those are being killed because test runs are taking > 1h. Ycan relax that delay in Jenkinsfile but that's not great that our tests are taking 1h

authors: garrensmith jiangphcn wohali

repo: apache/couchdb


text:
support window functions

Which issue does this PR close?

Closes #542 .
Rationale for this change
As stated in #542, datafusion adds SortExec and RepartitionExec based on the window clauses partition by and order by (See enforce_distribution.rs in datafusion and WindowAggExec::required_input_distribution .
From what I can see, this restriction on window functions in ballista was put way back when datafusion was in the same repo as ballista and window function support was still in its infancy. Now that datafusion enforces the right hash distribution by placing repartition nodes, I believe ballista currently already supports window functions.
I have tested partitioned, unpartitioned and sorted window expressions with ballista with this PR and compared them with datafusion, it seems to be working as intended. Parititioned window functions get repartitioned by their partition expressions and are parallelised.
Not sure what is the best way to test this given distribution handling is done by datafusion here.

What changes are included in this PR?
Allow queries with window expressions

Are there any user-facing changes?

@milenkovicm should be good to go. I also enabled a previously ignored test, as the fix it mentions is merged for a while now

repo: apache/datafusion-ballista


text:
Support date_bin on timestamps with timezone, properly accounting for Daylight Savings Time

Is your feature request related to a problem or challenge?
Summary
Given this data in UTC timestamps
> select * from t_utc;
+----------------------+
| column1              |
+----------------------+
| 2024-04-30T21:30:00Z |
| 2024-04-30T22:30:00Z |
| 2024-04-30T23:30:00Z |
| 2024-05-01T00:00:00Z |
| 2024-05-01T00:30:00Z |
| 2024-05-01T10:30:00Z |
| 2024-05-01T20:30:00Z |
+----------------------+
7 row(s) fetched.
Elapsed 0.002 seconds.

How to create the data

-- Dates right around the first of May
create or replace table t AS
VALUES
  ('2024-04-30T21:30:00'),
  ('2024-04-30T22:30:00'),
  ('2024-04-30T23:30:00'),
  ('2024-05-01T00:00:00'),
  ('2024-05-01T00:30:00'),
  ('2024-05-01T10:30:00'),
  ('2024-05-01T20:30:00')
;

create or replace view t_utc as
select column1::timestamp  AT TIME ZONE 'UTC' as "column1"
from t;

I would like to be able to get this output (bin the data by their day in Brussels, properly accounting for daylight savings time).
+---------------------------+---------------------+
| time in Brussels          | date_bin            |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T00:00:00 | <-- in 2024-04-01
| 2024-05-01T00:30:00+02:00 | 2024-05-01T00:00:00 | <-- note this is in 2024-05-01
| 2024-05-01T01:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T00:00:00 |
+---------------------------+---------------------+
7 row(s) fetched.
Elapsed 0.004 seconds.

We can't figure out how to do this at the moment
Details
Broken out from @Abdullahsab3's great ticket https://github.com/apache/datafusion/issues/10368
We would like to apply date_bin and have it bin the dates in local time, properly including daylight savings time calculations
For example, given the following data at 12:01AM in the morning on the first day of each month,I would like to bin it into the first day of each month.
create or replace table t AS
VALUES
  ('2024-01-01T00:00:01'),
  ('2024-02-01T00:00:01'),
  ('2024-03-01T00:00:01'),
  ('2024-04-01T00:00:01'),
  ('2024-05-01T00:00:01'),
  ('2024-06-01T00:00:01'),
  ('2024-07-01T00:00:01'),
  ('2024-08-01T00:00:01'),
  ('2024-09-01T00:00:01'),
  ('2024-10-01T00:00:01'),
  ('2024-11-01T00:00:01'),
  ('2024-12-01T00:00:01')
;

This works as expected when there is no timestamp (all timestamps end up in the correct day)
> select arrow_typeof(column1::timestamp) as arrow_type, column1::timestamp, date_bin(interval '1 day', column1::timestamp) as "date_bin" from t;
+-----------------------------+---------------------+---------------------+
| arrow_type                  | t.column1           | date_bin            |
+-----------------------------+---------------------+---------------------+
| Timestamp(Nanosecond, None) | 2024-01-01T23:59:00 | 2024-01-01T00:00:00 | <-- binned correctly
| Timestamp(Nanosecond, None) | 2024-02-01T23:59:00 | 2024-02-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-03-01T23:59:00 | 2024-03-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-04-01T23:59:00 | 2024-04-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-05-01T23:59:00 | 2024-05-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-06-01T23:59:00 | 2024-06-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-07-01T23:59:00 | 2024-07-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-08-01T23:59:00 | 2024-08-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-09-01T23:59:00 | 2024-09-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-10-01T23:59:00 | 2024-10-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-11-01T23:59:00 | 2024-11-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-12-01T23:59:00 | 2024-12-01T00:00:00 |
+-----------------------------+---------------------+---------------------+
12 row(s) fetched.
Elapsed 0.004 seconds.

However, once the timestamp is in a timezone it has problems:
> create or replace view t_timezone
as
select column1::timestamp  AT TIME ZONE 'Europe/Brussels' as "column1"
from t;
0 row(s) fetched.
Elapsed 0.001 seconds.

The timestamps now binned at incorrectly (the are binned based on UTC midnight, not midnight in Brussels) so for example a time in 2024-01-01 goes into a 2023-12-31 bucket:
> select
   arrow_typeof(column1) as arrow_type,
   column1,
   date_bin(interval '1 day', column1) as "date_bin"
  from t_timezone;
+------------------------------------------------+---------------------------+---------------------------+
| arrow_type                                     | column1                   | date_bin                  |
+------------------------------------------------+---------------------------+---------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-01-01T00:00:01+01:00 | 2023-12-31T01:00:00+01:00 | <-- binned in previous day
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-02-01T00:00:01+01:00 | 2024-01-31T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-03-01T00:00:01+01:00 | 2024-02-29T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-04-01T00:00:01+02:00 | 2024-03-31T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-05-01T00:00:01+02:00 | 2024-04-30T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-06-01T00:00:01+02:00 | 2024-05-31T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-07-01T00:00:01+02:00 | 2024-06-30T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-08-01T00:00:01+02:00 | 2024-07-31T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-09-01T00:00:01+02:00 | 2024-08-31T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-10-01T00:00:01+02:00 | 2024-09-30T02:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-11-01T00:00:01+01:00 | 2024-10-31T01:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-12-01T00:00:01+01:00 | 2024-11-30T01:00:00+01:00 |
+------------------------------------------------+---------------------------+---------------------------+

The third argument to date_bin allows  changing the starting offset for a day, so let's set it to midnight in Europe/Brussels time. This almost works but does not take into account daylight savings time so the timestamps are binned incorrectly during DST:
> select
   arrow_typeof(column1) as arrow_type,
   column1,
   date_bin(interval '1 day', column1, '2020-01-01T00:00:00Z'::timestamp AT TIME ZONE 'Europe/Brussels') as "date_bin"
  from t_timezone;
+------------------------------------------------+---------------------------+---------------------------+
| arrow_type                                     | column1                   | date_bin                  |
+------------------------------------------------+---------------------------+---------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-01-01T00:00:01+01:00 | 2024-01-01T00:00:00+01:00 | <-- correct!
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-02-01T00:00:01+01:00 | 2024-02-01T00:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-03-01T00:00:01+01:00 | 2024-03-01T00:00:00+01:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-04-01T00:00:01+02:00 | 2024-03-31T00:00:00+01:00 | <-- DST! Binned into previous day
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-05-01T00:00:01+02:00 | 2024-04-30T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-06-01T00:00:01+02:00 | 2024-05-31T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-07-01T00:00:01+02:00 | 2024-06-30T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-08-01T00:00:01+02:00 | 2024-07-31T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-09-01T00:00:01+02:00 | 2024-08-31T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-10-01T00:00:01+02:00 | 2024-09-30T01:00:00+02:00 |
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-11-01T00:00:01+01:00 | 2024-11-01T00:00:00+01:00 | <-- DST over, now correct again
| Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-12-01T00:00:01+01:00 | 2024-12-01T00:00:00+01:00 |
+------------------------------------------------+---------------------------+---------------------------+
12 row(s) fetched.
Elapsed 0.003 seconds.

Describe the solution you'd like
Describe alternatives you've considered
No response
Additional context
No response

The way you can perform this binning  in postgres is somewhat paradoxically to convert a timestamp with a timezone back to a timestamp without timezone and then apply date_bin.
The syntax to convert a timestamp to something without timezone is 🤯 : you apply AT TIME ZONE to a timestamp with a timezone and then it has no timezone) as @mhilton  notes in https://github.com/apache/datafusion/issues/10368#issuecomment-2111684373

postgres=# select pg_typeof('2024-05-21T12:00:00'::timestamp AT TIME ZONE 'America/New_York');
        pg_typeof
--------------------------
 timestamp with time zone
(1 row)

postgres=# select pg_typeof('2024-05-21T12:00:00'::timestamp AT TIME ZONE 'America/New_York' AT TIME ZONE 'Europe/Brussels'
);
          pg_typeof
-----------------------------
 timestamp without time zone. <-- WTF no timezone as the result of AT TIME ZONE
(1 row)

If we cast using arrow_cast back to Timestamp(Nanosecond, None) the binning does appear to work correctly
> create or replace view t_roundtrip as select arrow_cast(column1, 'Timestamp(Nanosecond, None)') as "column1" from t;
0 row(s) fetched.
Elapsed 0.002 seconds.

> select arrow_typeof(column1) as arrow_type, column1, date_bin(interval '1 day', column1) as "date_bin" from t_roundtrip;
+-----------------------------+---------------------+---------------------+
| arrow_type                  | column1             | date_bin            |
+-----------------------------+---------------------+---------------------+
| Timestamp(Nanosecond, None) | 2024-01-01T00:00:01 | 2024-01-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-02-01T00:00:01 | 2024-02-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-03-01T00:00:01 | 2024-03-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-04-01T00:00:01 | 2024-04-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-05-01T00:00:01 | 2024-05-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-06-01T00:00:01 | 2024-06-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-07-01T00:00:01 | 2024-07-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-08-01T00:00:01 | 2024-08-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-09-01T00:00:01 | 2024-09-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-10-01T00:00:01 | 2024-10-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-11-01T00:00:01 | 2024-11-01T00:00:00 |
| Timestamp(Nanosecond, None) | 2024-12-01T00:00:01 | 2024-12-01T00:00:00 |
+-----------------------------+---------------------+---------------------+
12 row(s) fetched.
Elapsed 0.003 seconds.

Given the statement in the description, here is the best I can come up with using arrow_cast
-- Times in brussels
WITH t_brussels
AS (
  SELECT
    column1 AT TIME ZONE 'Europe/Brussels' as ts -- timestamp in specified timezone
  FROM t_utc
)
SELECT
  ts as "time in Brussels",
  date_bin(
    interval '1 day',
    arrow_cast(ts, 'Timestamp(Nanosecond, None)'),
    '2020-01-01T00:00:00Z'::timestamp
  ) as date_bin
FROM
  t_brussels;

+---------------------------+---------------------+
| time in Brussels          | date_bin            |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T00:00:00 |
| 2024-05-01T00:30:00+02:00 | 2024-04-30T00:00:00 |
| 2024-05-01T01:30:00+02:00 | 2024-04-30T00:00:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T00:00:00 |
+---------------------------+---------------------+
7 row(s) fetched.
Elapsed 0.004 seconds.

@mhilton  and I agree that if we had the functionality suggested by @Abdullahsab3's on https://github.com/apache/datafusion/issues/10368#issue-2277903243

given a UTC timestamp, I would like to have that timestamp in local time of a given timezone

We think we could get the right value out of date_bin.
For example,  If we had a function like remove_timezone that behaved like
select remove_timezone("time in Brussels")

+---------------------------+---------------------+
| time in Brussels          | remove_timezone            |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T23:30:00 |
| 2024-05-01T00:30:00+02:00 | 2024-05-01T00:30:00 | <-- timezone removed (no +02:00, but value not adjusted)
| 2024-05-01T01:30:00+02:00 | 2024-05-01T01:30:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T02:30:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T12:30:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T22:30:00 |
+---------------------------+---------------------+

Note that this is different than arrow_cast("time in Brussels", 'Timestamp(Nanosecond, None)') because arrow_cast will adjust the timestamp values
-- Times in brussels
WITH t_brussels
AS (
  SELECT
    column1 AT TIME ZONE 'Europe/Brussels' as ts -- timestamp in specified timezone
  FROM t_utc
)
SELECT
  ts as "time in Brussels",
  arrow_cast(ts, 'Timestamp(Nanosecond, None)') as arrow_cast
FROM
  t_brussels;

+---------------------------+---------------------+
| time in Brussels          | arrow_cast          |
+---------------------------+---------------------+
| 2024-04-30T23:30:00+02:00 | 2024-04-30T21:30:00 |
| 2024-05-01T00:30:00+02:00 | 2024-04-30T22:30:00 | <-- note this is now in 2024-04-30 not 2024-05-01
| 2024-05-01T01:30:00+02:00 | 2024-04-30T23:30:00 |
| 2024-05-01T02:00:00+02:00 | 2024-05-01T00:00:00 |
| 2024-05-01T02:30:00+02:00 | 2024-05-01T00:30:00 |
| 2024-05-01T12:30:00+02:00 | 2024-05-01T10:30:00 |
| 2024-05-01T22:30:00+02:00 | 2024-05-01T20:30:00 |
+---------------------------+---------------------+
7 row(s) fetched.
Elapsed 0.003 seconds.

My suggested next steps for this ticket:

Someone prototype the "strip_timezone" function as a ScalarUDF and verify that we can in fact we can achieve the expected result from date_bin
If we can achieve the expected results, then file another ticket / proposal with how to integrate this into DataFusion (e.g. strip_timezone function, and/or update the arrow_cast kernel, and/or add syntax like WITHOUT TIME ZONE)

The challenge with an approach that relies on stripping the timezone, is you end up back with the ambiguous timestamp issue. The UX of relying on users to strip the timezone is rather unfortunate. I'm curious as to why we wouldn't want to just make date_bin timezone aware? This would appear to have a lot less potential edge cases

I'm curious as to why we wouldn't want to just make date_bin timezone aware? This would appear to have a lot less potential edge cases

I believe @mhilton had thought about this and maybe has some thoughts

I'm curious as to why we wouldn't want to just make date_bin timezone aware?

I'm not sure that it's obvious how a time-zone aware date_bin would behave. For example On the day in october when many time zones repeat an hour would the two hours count into a single bin? What about if you had hour long bins that start 30 minutes into the hour? Given that the two hours are easily distinguishable the answer is probably not, and if you really want that behviour then stripping time zone information would be a reasonable way to get it.
If we restricted timezone awareness intervals of unit days then that would be much more reasonable. We could make it so that date_bin understands the number of hours in any given day in a time zone and bins a time accordingly. Anyone needing a strict 24 hour binning could specify the interval in hour units.

To make date_bin timezone aware, there are some edge cases we need to consider when design it:

Daylight Saving Time (DST) Transitions:

Spring Forward: When the clocks move forward, there is a "missing" hour.
Fall Back: When the clocks move backward, there is an "extra" hour. For example, in US central time zone, when DST ends at 2:00 AM, the clocks are set back to 1:00 AM. This means that there are two 1:20 AM in a day. If a user do a date_bin with 10 min interval, how to handle the returned data? Aggregate them? Return two sets of data?

Crossing Midnight Boundaries:

Ensure that timestamps are correctly binned at the right local day and hour boundaries, especially when converting from UTC to a local timezone. This edge case is what this issue is trying to solve.

Timezone Offsets:

Different timezones have different offsets from UTC, and these can change over time (e.g., due to DST).
For certain timezones, ensure the offset works correctly at the 30-min mark when DST happens.  e.g.:

Lord Howe Island Time Zone (LHST) in Australia shifts time by 30 min during DST
Nepal (NPT) shifts by 45 min
Chatham Islands, New Zealand (CHAST) shifts by 12 hr and 45 min.

Ensure the function correctly applies the current offset for the given timestamp, considering historical changes in timezone rules. e.g.:

America/Sao_Paulo stoped doing DST in 2019
Russia set the clocks ahead permanently in 2011, then was reversed in 2014
Iceland, Turkey, Egypt, India, and China

Leap Years and Leap Seconds:

Leap Years: February 29th is handled correctly in leap years.
Leap Seconds: this is less common. e.g. the most recent one was on Dec. 31, 2016, a leap second was introduced at 23:59:60 UTC

Handling Null and Invalid Timestamps:

Ensure that null or invalid timestamps are handled gracefully, either by ignoring them or providing a default bin. e.g. '2021-03-28T02:30:00' AT TIME ZONE 'Europe/Brussels' does not exist

Time Precision:

Ensure that the function handles different precisions of timestamps correctly (e.g., seconds, milliseconds, nanoseconds).

Fall Back: When the clocks move backward, there is an "extra" hour. For example, in US central time zone, when DST ends at 2:00 AM, the clocks are set back to 1:00 AM. This means that there are two 1:20 AM in a day. If a user do a date_bin with 10 min interval, how to handle the returned data? Aggregate them? Return two sets of data?

I want to highlight this open-ended question and hear how others think

Thanks for filing the ticket and for all the detailed explanations! very enriching
I wonder whether the Postgres behavior is actually that bad. Though it looks weird, it still is generic enough to make it widely applicable. The issue with making date_bin specifically timezone-aware in my opinion is the fact that future or similar grouping functions (or rolling windows functions) will also have to be implemented in a timezone-aware way, accounting for similar pitfalls and edge cases. The same problem will also occur in other functionalities; for example if you would like to return local time. I personally think that the problem might be better tackled as an additional time functionality, which may be the same way that Postgres does it.
The postgres way of converting UTC to local time of a given timezone is:
select '2024-05-21T12:00:00Z'::timestamp AT TIME ZONE 'UTC' AT TIME ZONE 'Europe/Brussels';

timezone
--
2024-05-21 14:00:00

As already mentioned by Andrew, the AT TIME ZONE operator in postgres converts a timezone-aware timestamp to local time (with no offset or timezone information), and local time to a timezone-aware timestamp. Though the overloaded functionalities of the AT TIME ZONE operator in Postgres are weird, they are definitely usable in my opinion. I also like the idea of having a separate to_local_time/strip_timezone function, though this would break Datafusions compatibility with Postgres.

Thank you @tustvold and @Abdullahsab3 and @mhilton and @appletreeisyellow for the thoughts.
From my perspective, the current (non timezone aware) date_bin function has the benefit of being (relatively) simple to implement and fast (as it is some arithmetic calculation on the underlying integer value without having to do DST conversions per row)
Given the differences in underlying timestamp representation between arrow and postgres I do think some difference is likely inevitable and thus likely not a deal breaker.
Here are my suggested next steps  @appletreeisyellow tries to prototype one or both proposals and see if we can get it to produce the desired results:

create a to_local_time function
Modify to the date_bin function to make it timezone aware

I think the to_local_time might be the simplest approach.

A ScalarUDF
a Signature that takes Timestamp(.., *)
produces a Timestamp(.., None).
The invoke function would do the oppositte of whatever cast(Timestamp(.., None) --> Timestamp(.., TZ)) does

I was looking into this issue again. I am still validating this, but I found that using to_char(time AT TIME ZONE 'UTC' AT TIME ZONE 'Europe/Brussels', '%F %X') also results in correct local time. (see https://docs.rs/chrono/latest/chrono/format/strftime/index.html)

Very nice! Thanks for verifying @appletreeisyellow. I was able to do some minimal validations today on our systems and it seems indeed to be binning correctly. Happy to see this as an alternative to what I was doing as a fix for this issue
My main concerns with the use of to_char are:

Performance: Not super familiar with Datafusion internals, but I assume to_char will parse the given timestamp in the provided format, and return a string, which will be coerced to a (localtime) timestamp.
Probably not a problem and I am just overthinking it, but I wonder whether %F %X will always produce the correct localtime. My concern is mostly the %F part

@appletreeisyellow  and I just had a chat about what to do about this ticket. It is great that we seem to have found a workaround there are at least two potential issues we see with the workaround:

It is non obvious and likely would not be obvious to other future users
It may have performance issues, as @Abdullahsab3 notes in https://github.com/apache/datafusion/issues/10602#issuecomment-2140461400

We think it would be great to improve DataFusion so future users (both of InfluxDB and other systems) who use date_bin have fast queries and are not confused.
Ideal Behavior
We think the ideal user experience would be as @tustvold  suggests for date_bin to be timezone aware and just "do the right thing and bin dates respecting the timezone of the timestamp".
While this might be different than what postgres does, the underlying treatment if timestamps is already different, so I don't think we need to be quirk for quirk compatible.
Proposal
Thus we propose @appletreeisyellow will try and complete her prototype work to make date_bin do the intuitive thing with timestamps with timezones
It is likely we can't devote a large amount of time to this project, so if the implementation is taking too long we may have to abandon this effort or go with a less optimal route (e.g. a function to strip off timezones like Postgres)

I filed https://github.com/apache/arrow-rs/pull/5826 to document more precisely what is going on with casting

I wrote up a high-level doc on how to implement the date_bin with timezone awareness. If you cannot access the document, feel free to request access and I'll give you the permission to view/comment.

Another thing that needs to be taken into consideration: I think the returned timestamp from date_bin needs to be in the timezone that was provided as an argument. For example:
The returned timestamp is in correct local time, but the underlying Arrow kernel has no timezone information about it (which I think means that the timestamp will be interpreted as being in UTC)
❯ select to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X')::timestamp;
+----------------------------------------------------+
| to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X")) |
+----------------------------------------------------+
| 2024-04-30T04:00:00                                |
+----------------------------------------------------+
1 row in set. Query took 0.002 seconds.

❯ select arrow_typeof(to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X')::timestamp);
+------------------------------------------------------------------+
| arrow_typeof(to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X"))) |
+------------------------------------------------------------------+
| Timestamp(Nanosecond, None)                                      |
+------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

When we add the timezone information:
❯ select to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X') at time zone 'Europe/Brussels';
+----------------------------------------------------+
| to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X")) |
+----------------------------------------------------+
| 2024-04-30T04:00:00+02:00                          |
+----------------------------------------------------+
1 row in set. Query took 0.001 seconds.

❯ select arrow_typeof(to_char('2024-04-30T02:00:00' at time zone 'UTC' at time zone 'Europe/Brussels', '%F %X') at time zone 'Europe/Brussels');
+------------------------------------------------------------------+
| arrow_typeof(to_char(Utf8("2024-04-30T02:00:00"),Utf8("%F %X"))) |
+------------------------------------------------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels"))                   |
+------------------------------------------------------------------+
1 row in set. Query took 0.001 seconds.

The timezone information there is important, as the timestamp will be otherwise interpreted by external integrations (such as Grafana) as being in UTC.
Note that applying at time zone 'Europe/Brussels' will never fail during daylight savings hours, as the returned value from to_char is always in localtime, meaning that the 'missing/additional hour' should not be an issue for the at time zone operator.

I think the returned timestamp from date_bin needs to be in the timezone that was provided as an argument (in the internal Arrow represenation)

Thanks for highlighting it. Yes, this is the expected behavior. The timezone representation of the returned timestamp will be the same as the input timestamp. For example:

if the input doesn't have a timezone, then the date_bin returns timezone as None:

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20Z'));
+---------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20Z"))) |
+---------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, None)                                                                       |
+---------------------------------------------------------------------------------------------------+

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20'));
+--------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20"))) |
+--------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, None)                                                                      |
+--------------------------------------------------------------------------------------------------+

if the input has a timezone in UTC, then the date_bin returns timezone as Some('UTC')

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20Z' AT TIME ZONE 'UTC'));
+---------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20Z"))) |
+---------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, Some("UTC"))                                                                |
+---------------------------------------------------------------------------------------------------+

if the input has a timezone in Europe/Brussels, then the date_bin returns timezone as Some('Europe/Brussels')

select arrow_typeof(date_bin(interval '1 day', '2024-03-30T00:00:20Z' AT TIME ZONE 'Europe/Brussels'));
+---------------------------------------------------------------------------------------------------+
| arrow_typeof(date_bin(IntervalMonthDayNano("18446744073709551616"),Utf8("2024-03-30T00:00:20Z"))) |
+---------------------------------------------------------------------------------------------------+
| Timestamp(Nanosecond, Some("Europe/Brussels"))                                                    |
+---------------------------------------------------------------------------------------------------+

I probably missed some prior discussion. Do we want date_bin syntax?
Can we support more standard date_trunc('day', timestamp_tz)?

I think we already have date_trunc
docs: https://datafusion.apache.org/user-guide/sql/scalar_functions.html#date-trunc
code: https://github.com/apache/datafusion/blob/dce77db316beb5bf5dc2326c350b2d642edbfed8/datafusion/functions/src/datetime/date_trunc.rs#L361
I think the difference between date_trunc and date_bin is that date_bin can bin to dates that are not aligned with the start/stop of the interval (e.g. you could bin days at noon rather than at midnight)
However it is a good question if date_trunc would be a better way to get the desired behavior

Update: @appletreeisyellow merged an workaround -- to_local_time in https://github.com/apache/datafusion/pull/11401
She has no further current concrete plans to work on this ticket at this time

FYI - I came across the Jiff crate the other day that looks to have some nice support for tz aware calendar handling. From the benchmarks I've run it's not quite as quick as chrono for some things and about equivalent to others. I would recommend anyone who wants to tackle this issue to have a close look at it to see if it solves some of the intricacies around tz conversion and DST arithmetic and rounding.

Author of Jiff here. I don't have a ton of context on the specific problem in this issue, but I'd be happy to field questions about DST safe arithmetic. Jiff in particular is heavily inspired by Temporal, and Temporal (and Jiff) both intend to match RFC 5545 for duration handling.

authors: Abdullahsab3 BurntSushi Omega359 alamb appletreeisyellow findepi mhilton tustvold

repo: apache/datafusion


text:
octet_length() function not working for StringView columns (SQLancer)

Describe the bug
Native StringView support for octet_length() has been added https://github.com/apache/datafusion/issues/11858
However it's not working for StringView column inside table
See reproducer in datafusion-cli
(Compiled from latest main using cargo run, commit a58416c2e)
The last query should work since this function should already have StringView support
DataFusion CLI v41.0.0
> create table t1(v1 text);
0 row(s) fetched.
Elapsed 0.058 seconds.

> insert into t1 values ('DataFusion'), ('datafusion');
+-------+
| count |
+-------+
| 2     |
+-------+
1 row(s) fetched.
Elapsed 0.047 seconds.

> create table t1_stringview as
select arrow_cast(v1, 'Utf8View') as v1
from t1;
0 row(s) fetched.
Elapsed 0.011 seconds.

# Now we have two equivalent tables `t1` and `t1_stringview`
# The difference is physical representation for string column (StringArray and StringViewArray)

> select octet_length(v1) from t1;
+---------------------+
| octet_length(t1.v1) |
+---------------------+
| 10                  |
| 10                  |
+---------------------+
2 row(s) fetched.
Elapsed 0.006 seconds.

> select octet_length(v1) from t1_stringview;
Arrow error: Compute error: length not supported for Utf8View

To Reproduce
No response
Expected behavior
No response
Additional context
Found by SQLancer https://github.com/apache/datafusion/issues/11030

take

This issue will be fixed once the arrow_string dependency is updated.

Array-string dependency was updated to 53.1.0 which includes the update from  apache/arrow-rs#6305. I'll work on a PR to verify the fix in octet_length()


text:
Improved performance for streaming group by

Is your feature request related to a problem or challenge?
During review of https://github.com/apache/arrow-datafusion/pull/6932 there were several suggstions made about how we could improve the performance of streaming / bounded grouping (when the data is sorted by some/all of the group keys)
This ticket tracks those improvements
Describe the solution you'd like
Suggestion 1
@tustvold  suggests on https://github.com/apache/arrow-datafusion/pull/6932#r1267305736

It occurs to me that a potentially faster way to detect the group boundaries would be to use the inequality kernel offset by one w.r.t to each other, ORing the results together, and then iterating over the set bits.
There would be some subtlety to handle nulls correctly, but it would likely be significantly faster and would not require converting to the row format
We could likely do something similar for window functions if we aren't already

Note that IOx does something similar to compute partition boundaries here: https://github.com/influxdata/influxdb_iox/blob/d3b3805e5fa87214ea934427df3967a57a14669c/iox_query/src/provider/deduplicate/algo.rs#L101-L207
Suggestion 2
From https://github.com/apache/arrow-datafusion/pull/6932#r1267344573

In the case of GroupOrderingFull it seems unnecessary to be computing hashes at all, we can just group based on whenever the sort key changes?

This could likely go quite a bit faster as hashing is a significant part of the time
Describe alternatives you've considered
No response
Additional context
No response

@alamb I think this issue can be closed. Suggestion 1 is done, suggestion 2 is no longer relevant as there are no hashes computed for GroupOrderingFull

Thanks @smiklos  -- done!

authors: alamb smiklos


text:
[Bug-FE][UI Next][V1.0.0-Alpha]Data source types and data source examples  Showing errors

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
Data source types and data source examples Showing errors

What you expected to happen
Show the last time I saved database information
How to reproduce
I edit and save the database for sqlserver, click save workflow, edit again when the component shows the default mysql database, not show the database type and database I set last time
Anything else
No response
Version
dev
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I will fix it.

repo: apache/dolphinscheduler


text:
[2.0.1-prepare] revert resource change

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Codecov Report

:exclamation: No coverage uploaded for pull request base (2.0.1-prepare@1a13934). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             2.0.1-prepare    #7145   +/-   ##
================================================
  Coverage                 ?   32.11%
  Complexity               ?     1572
================================================
  Files                    ?      433
  Lines                    ?    14496
  Branches                 ?     1441
================================================
  Hits                     ?     4655
  Misses                   ?     9383
  Partials                 ?      458

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a13934...347ef5e. Read the comment docs.

authors: CalvinKirs codecov-commenter


text:
[Feature-8497][UI] Support dependent task re-run automatically

Purpose of the pull request

Dependent tasks can re-run automatically in the case of complement

Brief change log

Related issues
close #8497

PTAL @SbloodyS @songjianet

Codecov Report

Merging #8739 (3ffa1df) into dev (63e85f3) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #8739      +/-   ##
============================================
- Coverage     45.29%   45.28%   -0.01%
+ Complexity     4075     4074       -1
============================================
  Files           702      702
  Lines         27112    27112
  Branches       2897     2897
============================================
- Hits          12280    12279       -1
- Misses        13678    13679       +1
  Partials       1154     1154

Impacted Files
Coverage Δ

...lphinscheduler/rpc/common/AbstractRpcCallBack.java
0.00% <0.00%> (-100.00%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e34f6fc...3ffa1df. Read the comment docs.

authors: codecov-commenter devosend


text:
K8S 部署官方提供的BE镜像拉取后4G多，可以简化吗

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
K8S 部署官方提供的BE镜像拉取后4G多，可以简化吗
docker pull apache/doris:1.2.2-be-arm
Solution
是否可以给出具体部署的流程，官方文档里面的很多没法直接部署
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

估计简化不了，看了下 镜像  selectdb/doris.be-ubuntu:2.1.1
apache-doris-2.1.1-bin-amd64/be    就有4.2G

repo: apache/doris


text:
fix fix some logic about broker load with parquet with new file scanner

Proposed changes
Issue Number: close #xxx
Problem summary
Fix some logic about broker load using new file scanner, with parquet format:

If columns are specified in load stmt, but none of them are in parquet file,
error will be thrown like err: No columns found in file. See parquet_s3_case4

If the first column of table are not in table, the result number of rows is wrong.
See parquet_s3_case8

If column specified in columns in load stmt does not exist in file and table,
error will be thrown like: failed to find default value expr for slot: x1. See parquet_s3_case2

Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[x] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Which part is the code change for case 3 ? If column specified in columns in load stmt does not exist in file and table, error will be thrown like

Line 608 of be/src/vec/exec/scan/vfile_scanner.cpp


text:
regression-test add vertical base compaction test

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

authors: dataroaring yixiutt


text:
fix insert default value should not change return type of function object in function set

Proposed changes
Issue Number: close #xxx
Problem summary
function now's return type changed to datetimev2 by mistake.
It can be reproduced in the following way
CREATE TABLE `testdt` (
  `c1` int(11) NULL,
  `c2` datetimev2 NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=OLAP
DUPLICATE KEY(`c1`, `c2`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`c1`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2",
"light_schema_change" = "true",
"disable_auto_compaction" = "false"
);

 insert into testdt2(c1) values(1);

select now();

Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

./run buildall

authors: morrySnow yiguolei


text:
fix fix invalid value error when casting null date value to string then casting to date value

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.51 seconds
stream load tsv:          505 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17167096805 Bytes

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.17 seconds
stream load tsv:          545 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.7 seconds inserted 10000000 Rows, about 336K ops/s
storage size: 17162138226 Bytes

authors: hello-stephen jacktengg


text:
feature support array type in fold constant framework

Proposed changes
Issue Number: close #xxx

use legacy planner way to process constant folding result from be
support signature with complex type for constant folding on fe

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.85 seconds
stream load tsv:          540 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17161859929 Bytes

authors: hello-stephen morrySnow


text:
Bug Fix potential incorrect query result caused by concurrent NewOlapScanNode initialization and Compaction

Proposed changes

Optimize fetching delete predicates code:

Avoid traversing rowsets of tablet to  get specified version schema.
Get delete predicates directly from captured consistent rowsets instead of tablet to get rid of holding meta lock.

Fix incorrect query result when compaction eliminate delete predicates between NewOlapScanNode::_init_scanners and NewOlapScanner::init.

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall


text:
(enhencement)(RowsetWriter) Don't delete files when beta rowset writer destructed

Proposed changes
Issue Number: close #xxx
The files of corresponding remote file system shouldn't be deleted when the rowset writers's dtor is called, instead they should be deleted asynchronously by the remote file system. Otherwise the dtor would do S3 io which might cost a lot of time.

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

authors: ByteYue dataroaring


text:
doc add auto_analyze_table_width_threshold description. (#27818)

backport https://github.com/apache/doris/pull/27818
Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
feature: optimize logical group expression in dphyp

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


text:
debug brpc debug

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run p0 10

run pipelinex_p0 10

run pipelinex_p0 10

run pipelinex_p0 10

run p0 10

run buildall

run p0 10

run pipelinex_p0 10

run pipelinex_p0 10

run p0 10

run pipelinex_p0 10

run p0 10

run p0 10

authors: dataroaring kaijchen


text:
fix Add retry when delete bitmap lock expired on insert into select

Proposed changes
this pr do the same things with https://github.com/apache/doris/pull/32829
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


text:
branch-2.1 fix some tests in external p0

Also move the analysis exception of "Not support insert with partition spec in hive catalog."
from create sink phase to bind sink phase.
So that when set enable_fallback_to_original_planner=false;, the return error will be correct.

run buildall

run buildall


text:
Enhancement Add more error msgs for wrong data types in orc and parquet reader.

Proposed changes
Enhancement Add more error msgs for wrong data types in orc and parquet reader.

run buildall


text:
Fix TopN should forbid two-phase topN if child is DistributionSpecGather

Proposed changes
TopN should forbid two-phase topN if child is DistributionSpecGather.

run buildall

run buildall

run buildall

run buildall

run p0

run buildall

run buildall

run buildall

run p0

run buildall

run buildall


text:
regression fix typo

Proposed changes
Issue Number: close #xxx

run buildall

run p0

run p0