text:
[BEAM-12739] Add AWS services as a runtime dependency to Flink Job Server to support S3

Declare the AWS IO SDK as a runtime dependency for the Flink Job Server
This is needed to allow the job server to use S3 for staging
artifacts.

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

@angoenka @mxm PTAL

There is currently a persistent failure causing tests to fail tracked in BEAM-12676, waiting for resolution before kicking off any failing tests again.

R: @lukecwik

authors: jlewi lukecwik


text:
Updated the base images to use debian:bullseye

Python, Go, Java dockerfiles are updated for Beam Playground to use debian bullseye
Python, Gom Java dockerfiles are updated for Beam SDK to use debian bullseye
Java dockerfile uses openjdk. openjdk updates the base image on their end.
Even though the default images uses latest distribution(eg: bullseye) of the OS,  it's good to have control over the distribution that we use by specifying the distribution name while pulling the image.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R : @tvalentyn , @emilymye

cc: @snkalinin FYI re: playground changes.

I wasn't sure if this was just a missed file, but did you want to update the Java image as well? https://github.com/apache/beam/blob/da11b84ba29f75a5764826f465b5bca9fad72a0e/sdks/java/container/Dockerfile#L19
I think it might be better to pin it to a specific version just for consistency go/python, e.g. by doing
FROM openjdk:${java_version}-bullseye

If not, can you change description (non-Java in title maybe) saying specifically why we aren't updating Java?
I believe the Java image updated here is for Playground, not the SDK itself

I wasn't sure if this was just a missed file, but did you want to update the Java image as well?
https://github.com/apache/beam/blob/da11b84ba29f75a5764826f465b5bca9fad72a0e/sdks/java/container/Dockerfile#L19
I think it might be better to pin it to a specific version just for consistency go/python, e.g. by doing
FROM openjdk:${java_version}-bullseye

If not, can you change description (non-Java in title maybe) saying specifically why we aren't updating Java?
I believe the Java image updated here is for Playground, not the SDK itself

I agree to have control over the distribution of the OS we use(Like how we do it with Python and Java). I pinned the OS distribution to openjdk.

retest this please

FYI @fernando-wizeline who is working on Java 17 (17-bullseye should exist, but possibly touching same Dockerfile)

Failures look unrelated.

source: gharchive/pull-request

authors: AnandInguva emilymye tvalentyn

repo: apache/beam

license_type: permissive

license_source: github-api


text:
Upgrade checker framework to 3.14.0

This upgrade has some changes to types in the JDK.
This PR includes #24385 which upgrades and fixes things to 3.13.0 so that PR comes first.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[x] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

unclear about https://ci-beam.apache.org/job/beam_PreCommit_Go_Commit/13082

Run Go PreCommit

Run Java PreCommit

Run Java_Debezium_IO_Direct PreCommit


text:
[Python] Skip fastavro 1.8.4

Skip fastavro 1.8.4 because of a regression caught in #28811

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Run Python 3.8 PostCommit

Run Python 3.9 PostCommit

Run Python 3.10 PostCommit

Run Python 3.11 PostCommit

Run Python_Dataframes PreCommit

Run Python_PVR_Flink PreCommit

Run Python_Coverage PreCommit

R: @tvalentyn

it's not sure if this will get fixed in 1.8.5 onward, should we set upper bound version instead?

Yes, I'm actually thinking about the same thing after looking at trace in https://github.com/fastavro/fastavro/issues/720

I raised an issue with them. Let's see if that is included in next release. If not, we can set upper bound I think

if it is a submission-runtime compatibility , rebuilding Dataflow containers to include latest fastavro version will fix it.

and we don't need to change versions.

but that doesn't explain why some external users got affected. i am not sure I completely understand the failure mode here.

Also note that it only fails on direct and flink runner test. Dataflow running same test having no issue

Scott mentioned about catching EOFError as well along with StopIteration https://github.com/fastavro/fastavro/issues/720#issuecomment-1751589076. Closing this one in favor of https://github.com/apache/beam/pull/28896

authors: Abacn riteshghorse tvalentyn


text:
add remaining metrics

Plumbing remaining autoscaling metrics: OutstandingBytes and MaximumOutstandingBytes

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

ptal @scwhittle


text:
Clean up integration test manual run parameters

Reduce a friction running integration tests locally, and update the documentation accordingly
Previously, gcpProject is used in some projects, while dataflowProject is used in some others
This change dedup the parameters used to run integration tests manually and align these additional parameters

Change dataflowProject -> gcpProject

Change dataflowRegion -> gcpRegion

Change dataflowTempRoot -> gcpTempRoot

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Python Coverage PreCommit failed test
test_batch_size_with_auto_sharding_0 (apache_beam.io.gcp.bigquery_test.PipelineBasedStreamingInsertTest) failed

not related to this change. Same test also exercised on Python PreCommit and passed.


text:
[BEAM-6082] Fix enum for SQL query 5 and 7

When I switched to an enum, I mapped the SQL queries to the wrong enums. This fixes it. Please do free review the query numbers associated with the enums to make sure I got it right this time around.

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

LGTM
The question here is why wrong mapping does not throw an exception. We might can add some checking later.

I guess the issue is that the launcher itself doesn't have a lot of testing. We have tests that SqlQuery5 matches the output of Query5Model but they instantiate directly. We don't have tests that createJavaQueries().get(HOT_ITEMS) matches the output of createJavaQueryModels().get(HOT_ITEMS). And reading the code, createSqlQueryModels() returns an empty map, presumably due to impedance mismatches between Java and SQL types for things.

Run Java PreCommit

Not one, two, or three flakes, but four!

authors: amaliujia chamikaramj kennknowles


text:
[WIP][BEAM-7909] customized containers for python

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Portable_Python PreCommit


text:
Traceback when tracking an element

See original issue on GitLab
In GitLab by [Gitlab user @bochecha] on Apr 26, 2019, 20:57
Summary
With Buildstream 1.2.5, as well as the tip of the bst-1.2 branch, I get a traceback when tracking an element.
Steps to reproduce

Use Buildstream 1.2 (I have not tried reproducing with the master branch)
git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk
cd freedesktop-sdk
git checkout master
bst track components/libglvnd.bst

What is the current bug behavior?
The tracking seems to have worked fine, however I get a traceback.
It seems to be in the codewhich prints messages to the console.
What is the expected correct behavior?
Tracking works, no traceback.
Relevant logs and/or screenshots
$ bst track components/libglvnd.bst
[--:--:--][][] START   Track
[--:--:--][][] START   Loading elements
[00:00:00][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:00][][] SUCCESS Resolving elements
[--:--:--][][] START   Resolving cached state
[00:00:00][][] SUCCESS Resolving cached state

BuildStream Version 1.2.5+62.g162c4166
  Session Start: Friday, 26-04-2019 at 21:54:57
  Project:       freedesktop-sdk (/home/mathieu/Projects/desktop/freedesktop-sdk)
  Targets:
  Cache Usage:   11.7G / infinity (6%)

User Configuration
  Configuration File:      Default Configuration
  Log Files:               /home/mathieu/.cache/buildstream/logs
  Source Mirrors:          /home/mathieu/.cache/buildstream/sources
  Build Area:              /home/mathieu/.cache/buildstream/build
  Artifact Cache:          /home/mathieu/.cache/buildstream/artifacts
  Strict Build Plan:       Yes
  Maximum Fetch Tasks:     10
  Maximum Build Tasks:     4
  Maximum Push Tasks:      4
  Maximum Network Retries: 2

Project Options
  bootstrap_build_arch: x86_64
  snap_grade:           devel
  target_arch:          x86_64

Loaded Plugins
  Source Plugins
    - git_tag

Pipeline

===============================================================================
[--:--:--][????????][track:components/libglvnd.bst       ] START   freedesktop-sdk/components-libglvnd/????????-track.15689.log
Exception in callback Job._parent_recv()
handle: <Handle Job._parent_recv()>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 642, in _parent_recv
    self._parent_process_queue()
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 634, in _parent_process_queue
    self._parent_process_envelope(envelope)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 606, in _parent_process_envelope
    self._scheduler.context.message(envelope.message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_context.py", line 376, in message
    self._message_handler(message, context=self)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/app.py", line 701, in _message_handler
    text = self.logger.render(message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/widget.py", line 593, in render
    return self._render(message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/widget.py", line 628, in _render
    text += widget.render(message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/widget.py", line 227, in render
    plugin = Plugin._lookup(element_id)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/plugin.py", line 659, in _lookup
    assert unique_id in cls.__TABLE, "Could not find plugin with ID {}".format(unique_id)
AssertionError: Could not find plugin with ID 215
Exception in callback Job._parent_recv()
handle: <Handle Job._parent_recv()>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 642, in _parent_recv
    self._parent_process_queue()
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 634, in _parent_process_queue
    self._parent_process_envelope(envelope)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_scheduler/jobs/job.py", line 606, in _parent_process_envelope
    self._scheduler.context.message(envelope.message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_context.py", line 376, in message
    self._message_handler(message, context=self)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/app.py", line 701, in _message_handler
    text = self.logger.render(message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/widget.py", line 593, in render
    return self._render(message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/widget.py", line 628, in _render
    text += widget.render(message)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/_frontend/widget.py", line 227, in render
    plugin = Plugin._lookup(element_id)
  File "/home/mathieu/.local/lib/python3.7/site-packages/buildstream/plugin.py", line 659, in _lookup
    assert unique_id in cls.__TABLE, "Could not find plugin with ID {}".format(unique_id)
AssertionError: Could not find plugin with ID 215
[--:--:--][        ][track:components/libglvnd.bst-0     ] INFO    Found new revision: v1.1.1-0-gbc9990b7bede44899627354d0c356f48950f4ad1
[00:00:00][????????][track:components/libglvnd.bst       ] SUCCESS freedesktop-sdk/components-libglvnd/????????-track.15689.log
[00:00:02][][] SUCCESS Track

Resolved key Summary
     waiting 3d8725eac964652be6b2238c9ab2ad5aed5d1d087d4a3becd62095e35794dcf9 components/libglvnd.bst

Pipeline Summary
  Total:       0
  Session:     1
  Track Queue: processed 1, skipped 0, failed 0

BuildStream version affected: /milestone %BuildStream_v1.2

In GitLab by [Gitlab user @bochecha] on Apr 26, 2019, 20:58
changed the description

In GitLab by [Gitlab user @tristanvb] on Apr 29, 2019, 09:02
This is a regression introduced by 9e85e36e and e0ca1ce9, reverting those in bst-1.2 fixes the issue.
In master it is introduced by bd7e75f9 and 90af8990.
Let me see if I can wrangle up a regression test for this...

In GitLab by [Gitlab user @bochecha] on Apr 29, 2019, 09:05

This is a regression introduced by 9e85e36e and 9e85e36e, reverting those in bst-1.2 fixes the issue.

Those are the same two commits, did you typo one of them?

In GitLab by [Gitlab user @tristanvb] on Apr 29, 2019, 10:19

Those are the same two commits, did you typo one of them?

Sorry, fixed that in the above comment.
Also, it took me a while but I've figured out the root causes of the change:

The code has always been invalid around the cloning of Sources, such that Source's which are cloned for the purpose of accessing mirrors do not inherit the plugin ID

This has mostly gone unnoticed because we prefer using the overall task ID (which is the owning element) for logging purposes

[Gitlab user @BenjaminSchubert]'s change of plugin ID handling has made it possible for a plugin to have ID 0, which was never valid (but easy to miss and not obviously documented). The plugin IDs intentionally start at ID 1 so that code can safely make the assumption that a falsy value for an ID holding variable represents an unset plugin ID.

All of these combined together, have caused line 227 of _frontend/widget.py to think that the task_id is not available for the message in question, which in turn caused the code to hit the unknown, cloned plugin ID which was created only in the child process.
I will try to put together a regression test, but the ultimate fixes for this are simple:

Change plugin.py to initialize the monotonic counter at 1, by calling itertools.count(1) instead of itertools.count().

Allow explicit setting of unique_id in Plugin constructor, so that Source can create the cloned sources which inherit their actual ID, which is important for any case that this ID needs to be reported to the main process.

In GitLab by [Gitlab user @tristanvb] on Apr 29, 2019, 11:41
mentioned in commit 0312f051fe78d620a56371c7c07fce40c52a510e

In GitLab by [Gitlab user @tristanvb] on Apr 29, 2019, 11:43
mentioned in merge request !1315

In GitLab by [Gitlab user @tristanvb] on Apr 29, 2019, 11:47
mentioned in commit 202559b3a791742d2f7c8a943ca3bf496c75b0a6

In GitLab by [Gitlab user @tristanvb] on Apr 29, 2019, 11:48
mentioned in merge request !1316

In GitLab by [Gitlab user @marge-bot123] on Apr 30, 2019, 06:54
mentioned in commit e435ca5255c77c6184b1cffe568646ae6045a77c

In GitLab by [Gitlab user @tristanvb] on Apr 30, 2019, 12:53
mentioned in commit 60bbe8daa1995b471aa69e234a4d39d61525adbd

In GitLab by [Gitlab user @marge-bot123] on Apr 30, 2019, 14:50
closed via merge request !1316

In GitLab by [Gitlab user @marge-bot123] on Apr 30, 2019, 14:50
mentioned in commit 377f20fe5f2a20d8d432e9a5a09e666cdba5afb3

source: gharchive/issue

repo: apache/buildstream


text:
Move to a threading scheduler

See original issue on GitLab
In GitLab by [Gitlab user @BenjaminSchubert] on Aug 29, 2020, 14:51
Summary
This is a meta issue to track the work to move towards a threading scheduler.
Rationale
Historically, we had a threaded scheduler. We moved to a multiprocessed one, in order to speed up BuildStream, which was, at the time, doing a lot of operations in the main python process.
BuildStream has evolved a lot, and a lot of the long running tasks have been delegated to other tools (buildbox-casd, buildbox-run) and we are thus running less long running operations in BuildStream core.
It is therefore less important now to be able to run the core on multiple CPUs at the same time.
We are at a point, where, the multi processing is more of a burden than a help, and we should move back to a threaded scheduler.
We should get the following benefits out of it:

Easier to debug plugins and scheduler code, since everything will be in the same process, pdb and such tools should have an easier time.
Easier error handling and reporting. Since we will have a more visible trace if a thread goes awry
Less fork() overhead, since forking in python is expensive (noticeable in benchmarks even)
Easier Windows/MacOS support (for BuildStream only, buildbox-* would still need other things), since we would not need to rely on fork(), as Windows doesn't support it, and MacOS's default is to not use it as it can cause problems
We will stop being in a unsupported usage of Python, since using asyncio together with forking is explicitely not supported and has lots of various problems.

This will however bring some disadvantages:

More pressure on the ioloop
We need to keep long running, blocking operations to a minimum in the scheduler
Potential concurrency bugs, now that we share the same address space
Single CPU usage, no more easy multi-cpu work.

I believe the benefits outweight the problems, and we should therefore move.
Work to do

[ ] Move BuildStream to a threaded scheduler, without too may invasive changes (!1982)
[ ] Remove the parent-child 'Job' separation, and stop sending messages across when we can avoid
[ ] Rewrite the messenger, to allow specifying a 'job' messenger, to allow a better integration with the ioloop
[ ] Integrate the CasUsageMonitor into the IOLoop instead of having it as a separate thread

Potential optimisations / areas to look into

[ ] Using a per-queue pool of threads for finer handling
[ ] Remove need for child watchers, and try alternative loop implementation (uvloop?)
[ ] Move more of the pre-public API code to be async, for better integration

In GitLab by [Gitlab user @BenjaminSchubert] on Aug 29, 2020, 17:17
mentioned in merge request !1982

In GitLab by [Gitlab user @BenjaminSchubert] on Aug 29, 2020, 17:17
mentioned in merge request !1982

In GitLab by [Gitlab user @BenjaminSchubert] on Aug 30, 2020, 11:00
assigned to [Gitlab user @BenjaminSchubert]

In GitLab by [Gitlab user @BenjaminSchubert] on Aug 30, 2020, 11:00
assigned to [Gitlab user @BenjaminSchubert]


text:
AaaP: Rework artifact class to use artifact proto

See original issue on GitLab
In GitLab by [Gitlab user @raoul].hidalgocharman on Mar 26, 2019, 17:59
Background
As part of #909, we want to change the internals of buildstream to use the artifact proto instead of dealing with looking in different directories for files, buildtrees, logs etc. This is dependent on both !1252 and !1259.
Task description
A few different places will need to be updated for this to work. Predominantly the Artifact class as we have moved most of the logic in there. This is probably an incomplete list that will be updated as more parts are found.

[x] Change Artifact class methods to use the artifact proto instead of directories. Pretty much every method bar _get_cache_key will need to change, as this predominantly deals with current directory structure of artifacts.

[x] Change appropriate ArtifactCache methods.

[x] The pull and push methods will need to change as it has to upload the different components, using CASCache.push_directory and then push the artifact proto. The subdir and exclude_subdir options aren't necessary now.
contains will need to deal with checking that files are actually present
[x] list_artifacts
[x] remove will need to follow files/logs/buildtree etc. and remove them
[x] get_artifact_directory/get_artifact_logs
[x] commit will need to deal with creating the artifact proto - This isn't actually used anyway

[x] Element

[x] __pull_directories method can be removed
[x] __pull_strong/__pull_weak don't need to specify subdir and exclude_subdirs`
[x] _pull_pending needs to be updated to more sensibly check for buildtree presence.

[x] CASCache

[x] _reachable_refs_dir will need to change as it can't follow refs that point to artifact protos. This is artifact specific, so this might need more thought as we don't want to put artifact specific logic here.
[x] CASQuotas clean will also need to change as it has to be able to remove unrequired source refs and artifact protos. This will probably require a change to how callbacks are registered with CASQuota

[x] TestArtifact methods that deal with the artifact directory structure will need to change

[x] remove_artifact_from_cache
[x] is_cache
[x] get_digest
[x] extract_buildtree

[x] Any tests that are affected by these changes

Acceptance Criteria
Buildstream should behave identically, so switch over to the artifact proto with a passing test suite.

In GitLab by [Gitlab user @raoul].hidalgocharman on Mar 28, 2019, 12:32
mentioned in commit 984a5d2d19539ceadc2a40612c0530df57a52e60

In GitLab by [Gitlab user @raoul].hidalgocharman on Mar 29, 2019, 10:22
mentioned in commit 176a05e5d743d904829e712b8ee859bf67ab5f15

In GitLab by [Gitlab user @raoul].hidalgocharman on Mar 29, 2019, 10:22
mentioned in commit 3f110e9bf97a010ea8e556e8d964c3aac87c593e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 2, 2019, 11:34
changed the description

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 2, 2019, 15:37
mentioned in commit 901e3e4a835dd526f646b69d18e8d121b8d8f2ff

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 2, 2019, 15:37
mentioned in commit f81a8f3bc20de9ae413ebb5f19fc44efced7e548

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 2, 2019, 15:37
mentioned in commit 540299223b8c3291cfd3450611ffa0f2eab372be

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 4, 2019, 17:18
mentioned in merge request !1259

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit 82e2809a9d200850af6e3b794bea093b0d082b88

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit dbceea7f35927c0ab3801b794a72c57522e6af50

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit 5bece872be86b6ce338ef0fede218cbac578194e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit bf508a9ed42063ecd4b2d834942de19cb7ed63aa

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit e946944031db8cfd9b22f3f5f6207cdde1bf4d7c

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit e35d3e59e47b7bb4262405f669310b27feb09db6

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 8, 2019, 18:03
mentioned in commit bb5c80ca1c646c10e655558ca46bef5bd1673e83

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 9, 2019, 14:21
marked the task Change Artifact class methods to use the artifact proto instead of directories. Pretty much every method bar _get_cache_key will need to change, as this predominantly deals with current directory structure of artifacts. as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 9, 2019, 14:22
changed the description

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 10, 2019, 17:45
mentioned in commit 154d5e1f5cc05cb4e9f430896a8949eed16db05b

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 10, 2019, 17:45
mentioned in commit 6ca7dfbaefe6137cab4845d12e19d5e3224805c2

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 10, 2019, 17:45
mentioned in commit 8aaab6f6ab961f9a37680652487afd20c3a6ee06

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 11, 2019, 10:59
mentioned in commit a6435354641bac3d3e9da7367030243a0622ca92

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 11, 2019, 11:08
mentioned in commit 9ada4525448970bc7a8999c2eb6f7380db93cbfa

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 11, 2019, 11:57
mentioned in commit 0441c8a61f16dba13f3db0d4a32001950b87b1e1

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 11, 2019, 13:44
mentioned in commit 42a8159d480b7f6d3307b8e41ae63582c02b2261

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 11, 2019, 15:05
mentioned in commit b2018ca54f5bb4d8578e8c54bbc58b2f63cdddd8

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 10:39
mentioned in commit f186fe1f23f0f18b4c987b57ab09396c6e15e2b0

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 10:39
mentioned in commit b0db4418acd87fab0f895cd3fb6cc9301083f1d1

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 10:39
mentioned in commit a5a9b064c465eacdbb4ab2e75449140f45dcfa60

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 10:39
mentioned in commit 0326b9aed2c24f8af08faa3c2ae7c18fb78708b4

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task Change appropriate ArtifactCache methods. as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task commit will need to deal with creating the artifact proto - This isn't actually used anyway as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task Element as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task __pull_directories method can be removed as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task __pull_strong/__pull_weak don't need to specify subdir and exclude_subdirs` as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task _pull_pending needs to be updated to more sensibly check for buildtree presence. as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task CASCache as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task TestArtifact methods that deal with the artifact directory structure will need to change as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task extract_buildtree as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:18
marked the task Any tests that are affected by these changes as completed

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:33
mentioned in commit 604b70db0f949e80aa63e34effcbbd0ccbaf8a21

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 12:33
mentioned in commit d4af73d386e4e376758dec173e9ea8a48c55eb43

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 13:55
mentioned in commit 2707091aec1b610de55f9321c8750dfcabd69bdf

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit 5a17be49f9cb128a180c2c2fb8b73d7c45ce82ee

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit 7dca701b2730da4f473888757cf05521facdfb27

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit 64b783a5d7815ef43b0a1178494417c552da7772

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit 4163205c9334147375afe8ca3b4cce05335d317e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit d8a427c3998a0a248e1e4e35a6bc70859421c621

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit d014609447b544405b9868f9f26a9382169ff4a6

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit ec4934fbf5f59322a0271f2a0ba38a39ac64644c

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 16:40
mentioned in commit 01d22aabaca509663a83c54bb876d03f6a5113b9

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 17:55
mentioned in commit 7b73021c6c48ed6c02a76c4f126a31627f8bbfd9

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 12, 2019, 17:56
mentioned in commit 24ac056f9b7c4a37271c89fe22e8700f2f0728fa

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 11:24
mentioned in commit f72126745dd5b3d2d1d68a78fa4bb697345449e8

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 11:24
mentioned in commit 71ba6ff1e34d5e9ce18839bc2b868d1e2b2d8874

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 12:52
mentioned in commit ddd48d87488d39334833ef83c7dc6a4e90a43ab6

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 12:52
mentioned in commit 3233ab893134358734a2c950b85749ac66def756

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 13:17
mentioned in commit a17a94bd8750c0422924856e0a416a7cde2507db

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 13:18
mentioned in merge request !1292

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit 9c977f93c14f592edad663378481804757fbb586

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit 99b75d9aaaa5d192054b3e2fe97d2e9769bccca0

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit 05796be4ca427785fa54570c6f67595ce42a5e31

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit f1c4000dd719dce484c09bd45b62e101c678f66f

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit e3ecfd098ad22589b26249965edc2a410c8ebcec

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit f7db4cb12643200aa5144fcd395c689f7dcbf463

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:18
mentioned in commit d48bb749eb0d94bbea50c9e454759fb50f3a7ce8

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit edf96fe559b9a32ef8025ce37c3febdca867b56d

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit 28a148f023812f70b5b74090545c1622d758b94e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit 30857be65b5d92a155c853af451ea1dcc142ff2a

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit 373aa3e48a1c59227021be8a8f658c1a4a3d8a3b

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit cfe6f4e1c830d397e37f07ec19bb65b71702b843

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit 358d07ed7de649df37dc8f2d599992a505a4eee0

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit bb9d464d8ea30414490905b3a208690d330517ba

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 16:19
mentioned in commit 8c4f842720902eab54b26fbf2dc2a7d68bd3549a

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit 73cd3d0daeca72423d785a10ebf1f654e46321ef

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit 3ceae38077732d74c1a23d994346508fedde4326

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit bcbe88abffec65a8252658ecc8f9dd08220c2f31

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit 5bd56a52ba4f14638cba461bcc9e3efeff024167

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit 2c3ad65d01bd528489e4b25b7fd292e462e1f9a9

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit 7d2c914d7f718c31c1d11ccc5e7084a5b2de5c34

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 15, 2019, 18:12
mentioned in commit 3b25f1a2d9e22933b81ca29b2eb7a2193fad37a8

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit c25382f82d01d1f393da0e681248e4226e553fa8

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit 01eab74c0f10b7a64f251e7a1280bdd4a81c278d

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit c3dd9f0b4f69ad1a8065a9c7af1a49b402f4ef4c

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit e2285e58472633b7cf836c9c950e32507231190a

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit 29401d18c7d97d57fd56110fb67dde02daa662f1

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit b93f25e3283f7997e5708b4ae4379868e666ae4e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit d2d7c12a2a98b0ed6fb70152da580410b9c9cb55

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 23, 2019, 14:04
mentioned in commit 70a8b142c4984255483580481d51356444ab96b3

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit 8839d0828029da1dadad3293a2fd1c8cf25aa076

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit 9d907e76960b57008f3081319bcbdf029cfb8d0e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit ee6e0d8f21def8ab7d2ef6c4bab899a9528a0808

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit 6dd0bfc8625e213d1fbc6019bbdd16b1f0e271e6

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit ee4fc0a050f3c40aae70abe1dcb268e311559c28

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit 5c3237e3bed1ace558e5f42ac014199b8044d2d9

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit 9aeb8bd299beb303cf7eeb26e70418f623617625

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 24, 2019, 15:36
mentioned in commit 424c79523e700624e6379de9344c928e429fdfc5

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit 861d0385db28b6ed08adcc5c6408b9bdd55791eb

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit f18f27b354e45e33917d0133700ceec92278826c

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit 6077e2fda9b0fb885b444633c5fd76b6e7d057a5

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit ffe86486ef4889bd655b68e6429ba1e2374cffdb

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit 1748949d6aa82d9496a8148d60bb469e7096fd62

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit 1261b74bbe6e988a077c50bfecf14b48238380dc

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit e15087ee2127afb9dc3efcc4f2005e443f59c5dd

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 25, 2019, 12:17
mentioned in commit c5712e123318379f04c3ba3cef0957b4e8fabc23

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit d19d31cf8600a73e2b63ece4284288e745c852fb

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit 080edf8967c85262ec713cc2e9a5ff340ca0b24f

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit c65ba16fb630b1f184b49aa1d720865133c7fef6

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit 97479a69d681710202ced5a81089c20e60e5b4da

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit 8046145bc6f21895a3e59c33348de79a69898f03

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit e53a64fb47f7b558aaaf0ebfb2d8bd63e0a53c9b

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit 46d6dc9b1d329472656980c022bfcdbcca44b9db

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 26, 2019, 14:09
mentioned in commit d8b1ff90339e31daa16cf5c9251f11c14d22d8d6

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit 9cf5954309e0fef9ab0bb5b4fd69b974322e986e

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit 7eb853ee547019b44cd30fd8fd4fa7cf90a6daa3

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit ce63f44a8b14b79f26c57fef59d4380488a6067b

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit b1f3fc9d439bfc464c1f6a08e315227150255640

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit d5bb8fb31a9c2bae2e5b7ec87b2c79ef21f34e7d

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit cd14f0663b24b6d55b0386dd554e6105c7b67ea9

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit e36e171098a06cbd505a5e8f7230c534bd5a5751

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 15:41
mentioned in commit f66a56e0db8123618f83aea191038d0e863c90a8

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 18:11
mentioned in commit 262af53e414997b99af462c7595c4167a6b8752c

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 18:11
mentioned in commit 4c4e254f6664b1f7637f72245b3b35c8f9afd6eb

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 18:11
mentioned in commit 5f48f0bb2844045a578638ea0f7c2494698ddf00

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 18:11
mentioned in commit 7172ae0dbb45db45a02bcf1d43be11daad99370b

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 18:11
mentioned in commit d7a54f7205d5fd5a03266ac8885cc6fff579ae54

In GitLab by [Gitlab user @raoul].hidalgocharman on Apr 30, 2019, 18:11
mentioned in commit a042ab3f50d3db4fcfd5911605f1b12d4b1de0d0

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit fa84bb18f4d68ce99cc074fff5d872b3e508ff5f

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit 52561d24170ecbfe59631f1920b9d57b3005ab5f

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit 300fe76404a164ccca73dda06441d5762361ef11

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit f8cb98501ef2b42f8e5910d6aa08dc9b67b215de

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit 7bf401210ac8770057e4e3aa79bfc6723216e96e

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit 6cdb6b2e8f474fba84d9a614a2f10cb7e1d0f92a

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 13:05
mentioned in commit 968640464f34db21bd757f8df61194a2dbf79358

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 18:11
mentioned in commit 237e4c14523e22b308f395b1b1243a27badb2ace

In GitLab by [Gitlab user @raoul].hidalgocharman on May 1, 2019, 18:11
mentioned in commit 84e79e60c1b6ee0faae625070ce127bb57459110

In GitLab by [Gitlab user @raoul].hidalgocharman on May 2, 2019, 16:45
mentioned in commit fbdd8f7303a46dd3dcd7412702f77c718576e336

In GitLab by [Gitlab user @raoul].hidalgocharman on May 2, 2019, 16:45
mentioned in commit fd758f945c67415f31355ac9145d4562e8083be2

In GitLab by [Gitlab user @raoul].hidalgocharman on May 2, 2019, 16:45
mentioned in commit 206f8fbd68f498f87039ed7725c51b89ea194bdc

In GitLab by [Gitlab user @raoul].hidalgocharman on May 2, 2019, 16:45
mentioned in commit 45d22ee08542098abcd00eb3f8603ed31a524722

In GitLab by [Gitlab user @raoul].hidalgocharman on May 2, 2019, 16:45
mentioned in commit 928cf5ea6dc8af678217d4a5780a43bbab139994

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 12:17
mentioned in commit 83be5825e347221a67dcf1247298a49bf9ddbce6

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 12:17
mentioned in commit dd6019fdbd541f9f59eae242ad13f74b8fe74cf8

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 12:17
mentioned in commit 6a075e780f2efdc1217ac7550a2306b138801db8

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 12:17
mentioned in commit 13538fe3ced8cb68430de6f5494145938ef0c664

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 12:17
mentioned in commit 6990d2a494c0340c191da729ab95bbc22fef5611

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 16:49
mentioned in commit 3a56b1bacbdffc5ee802621127946ac5e6c82d1a

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 16:49
mentioned in commit 7d4c2d4e85b00d783b63c488572d3e9f11c10082

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 16:49
mentioned in commit ecc14131f8adfa84143467a376df0441cd72effa

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 16:49
mentioned in commit b0c1d591f57661a782568fd755097f239b274b55

In GitLab by [Gitlab user @raoul].hidalgocharman on May 7, 2019, 16:49
mentioned in commit 2aaa91776f22816c2c2c1404adbf054822863a07

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 12:00
mentioned in commit cafd5f8457f1ffd3e3cfd164e0081eab0814b88b

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 12:00
mentioned in commit cf512db62b2c4061534b899cbf83ba840ac0441c

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 12:00
mentioned in commit 01afb50fa046b2fb2e55c0a9515beb6ffbcad544

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 12:00
mentioned in commit 41735ec1118b9ca4bf2a0b25a6c0baba14861a5e

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 12:00
mentioned in commit 21ded798c87dc4e147f8daaa3b506cc0775c423e

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 13:44
mentioned in commit 57a1d756f1046b0d3cc2f48c7af51e805219d9b5

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 13:44
mentioned in commit a5a8ce4813f0397ac35325e5ad5c43f4a547cb9b

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 13:44
mentioned in commit 91e49886127555e8625aefddbaa32326d98b3293

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 13:44
mentioned in commit 35e3c8c2b1350cf424e084312ff798f0dcb7e26d

In GitLab by [Gitlab user @raoul].hidalgocharman on May 8, 2019, 13:44
mentioned in commit 83ae884725b3ce31dcd695265d3ec3d6be8dea14

In GitLab by [Gitlab user @raoul].hidalgocharman on May 9, 2019, 09:54
mentioned in commit fb8438dee9476e2f572947dc63179213d9a39179

In GitLab by [Gitlab user @raoul].hidalgocharman on May 9, 2019, 09:54
mentioned in commit 3d9f947bf248ae53900259a509400874bd1306fe

In GitLab by [Gitlab user @raoul].hidalgocharman on May 9, 2019, 09:54
mentioned in commit 5858b563b2ac7420360c699e4170110ce3390cff

In GitLab by [Gitlab user @raoul].hidalgocharman on May 9, 2019, 09:54
mentioned in commit f4fd000e9662550e9be569223eed97e0d04480a4

In GitLab by [Gitlab user @raoul].hidalgocharman on May 9, 2019, 09:54
mentioned in commit b2f7fd64c4d5cd6372b61279bb7dc149b8acdc8c

In GitLab by [Gitlab user @tpollard] on May 9, 2019, 12:29
We should look to complete a more realistic RE build before this is merged, to ensure compatibility beyond buildstreams included tests

In GitLab by [Gitlab user @tpollard] on May 9, 2019, 16:46
I've built freedesktop-sdk's minimal image against this MR without an issue

In GitLab by [Gitlab user @raoul].hidalgocharman on May 13, 2019, 15:07
mentioned in commit 9ab7c44481e22f1c626fcca60d4591be25b884a4

In GitLab by [Gitlab user @raoul].hidalgocharman on May 13, 2019, 15:07
mentioned in commit 65528a16810564851628396790fbd95c01e9c1e7

In GitLab by [Gitlab user @raoul].hidalgocharman on May 13, 2019, 15:07
mentioned in commit f4a3e562d8660c9529131e3bf7680345c612c633

In GitLab by [Gitlab user @raoul].hidalgocharman on May 13, 2019, 15:07
mentioned in commit 25a041fbe5e1cda75eb3ace7bb4a066b48ee9968

In GitLab by [Gitlab user @raoul].hidalgocharman on May 13, 2019, 15:07
mentioned in commit 33b6021d0fd7ea6f119ee55c4296f5d77577b315

In GitLab by [Gitlab user @raoul].hidalgocharman on May 14, 2019, 16:59
mentioned in commit aba2c1cb26fe2c69c0ef7771e088a4edbf4cb853

In GitLab by [Gitlab user @raoul].hidalgocharman on May 14, 2019, 16:59
mentioned in commit 49d35ab5cf105a77d12117a5b8c392f7f0d82698

In GitLab by [Gitlab user @raoul].hidalgocharman on May 14, 2019, 16:59
mentioned in commit af9cb8735e5526649fdb195f1171e64f847229eb

In GitLab by [Gitlab user @raoul].hidalgocharman on May 14, 2019, 16:59
mentioned in commit 9d626aae7dad619371084977acb27ce32630c049

In GitLab by [Gitlab user @raoul].hidalgocharman on May 14, 2019, 16:59
mentioned in commit f5cbc05679b40c3a370e8f8f8cc63b23464a5b21

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 10:23
mentioned in commit d9bd12e42bfde1bf1df3e03accd6039b1c05403d

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 10:23
mentioned in commit 84b4c0b8a4e6ba60cf61f9bde0844f1399e0c007

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 10:23
mentioned in commit bb67b3606794b199a91d4543c054992833b7f2ca

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 10:23
mentioned in commit 8bdb767bf5d96885ba25e28a5bb7f36b434f3c78

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 10:23
mentioned in commit 40b83903a19434b03e972c89aede6da285dea69b

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 11:18
mentioned in commit 108e30a44aded323c9b11ddf623bdcc6802d8193

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 11:18
mentioned in commit 1d3f336641c08f1b047ac5d20c3577b94fc57a18

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 11:18
mentioned in commit 30e89a6a3cdcea83e0d83e373e50b88c02df3667

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 11:18
mentioned in commit 96dce3ebad4ff454f0ba2cba273e849f9370f9aa

In GitLab by [Gitlab user @raoul].hidalgocharman on May 15, 2019, 11:18
mentioned in commit 14dc7e498f769c71b119036a5fa9dd0b2af4c089

In GitLab by [Gitlab user @marge-bot123] on May 15, 2019, 12:44
closed via merge request !1292

In GitLab by [Gitlab user @marge-bot123] on May 15, 2019, 12:44
mentioned in commit 7fb538647ccb9ed7b42a60f4b663b10c40dcc772


text:
[CARBONDATA-1263]Single pass load does not take default value false for blank or inval…

Issue :- When Single pass value is given wrong (other than true/false) along with External Dictionary, Error is logged but Load is continue with Single pass=false . Which is not correct as for external  dictionary single pass must be true.
Solution :- Load is interrupted and  Error is thrown with proper error message.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2913/

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/327/

Can one of the admins verify this patch?

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/38/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1216/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/585/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3083/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4327/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4005/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4369/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5793/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5926/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/78/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8826/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/759/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/581/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/888/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9155/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1089/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9158/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9159/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1090/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9160/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9161/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9163/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1102/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1104/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1105/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9184/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9198/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2029/

Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10281/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3417/

Build Failed  with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/103/

Build Failed  with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/102/

Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/99/

Build Failed  with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/8/

authors: BJangir CarbonDataQA CarbonDataQA1 CarbonDataQA2 asfgit ravipesala

repo: apache/carbondata


text:
[WIP][CARBONDATA-1617] Merging carbonindex files within segment

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[X ] Make sure the PR title is formatted like:
[CARBONDATA-<Jira issue #>] Description of pull request

[ X] Make sure to add PR description including
Problem :

The first-time query of carbon becomes very slow. It is because of reading many small carbonindex files and cache to the driver at the first time.
Many carbonindex files are created in below case
Loading data in large cluster
For example, if the cluster size is 100 nodes then for each load 100 index files are created per segment. So after 100 loads, the number of carbonindex files becomes 10000. .
It will be slower to read all the files from the driver since a lot of namenode calls and IO operations.
Solution :
Merge the carbonindex files in two levels.so that we can reduce the IO calls to namenode and improves the read performance.
Merge within a segment.
Merge the carbonindex files to single file immediately after load completes within the segment. It would be named as a .carbonindexmerge file. It is actually not a true data merging but a simple file merge. So that the current structure of carbonindex files does not change. While reading we just read one file instead of many carbonindex files within the segment.

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
 Please provide details on
 - Whether new unit test cases have been added or why no new tests are required?
 - How it is tested? Please attach test report.
 - Is it a performance related change? Please attach the performance test report.
 - Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/679/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1316/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/683/

restest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/684/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1320/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1321/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/687/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/691/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1325/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/712/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/714/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1348/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1349/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/717/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1353/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/748/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/750/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1385/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/751/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1387/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1388/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/759/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1396/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/764/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1402/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/774/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1407/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/776/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1409/

LGTM

authors: CarbonDataQA jackylk ravipesala


text:
[CARBONDATA-3982] Use Partition instead of Span to split legacy and non-legacy segments for executor distribution in indexserver

Why is this PR needed?
When we use segments.span to split legacy segments and non-legacy segments where legacy segment index size is zero, segments list is empty.
What changes were proposed in this PR?
Use segments.partition instead of span to split legacy segments and non-legacy segments
Does this PR introduce any user interface change?

No

Is any new testcase added?

yes
Yes

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2295/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4034/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2300/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4039/

retest this please

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4052/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2314/

retest this please

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/4053/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2315/

LGTM


text:
CASSANDRA-16186 (trunk): Remove the SEPExecutor blocking behavior

If the number of tasks added to a SEPExecutor exceed the max queue size, the threads adding those task will be block until enough space become available for all the blocked tasks. At this point all the blocked threads will be released at once.
As the maxTasksQueued was always set to INTEGER.MAX_VALUE, the code was in practice never using. By consequence, removing the code was a better option than trying to fix it.

Merged manually

repo: apache/cassandra


text:
In Centos-8，Build Fail

ISSUE TYPE

Bug Report

COMPONENT NAME

packaging

CLOUDSTACK VERSION

4.16

CONFIGURATION

OS / ENVIRONMENT

CentOS-8.2
SUMMARY

In CentOS-8, python should be used instead of python2
STEPS TO REPRODUCE

EXPECTED RESULTS

[INFO]
[INFO] --- exec-maven-plugin:1.2.1:exec (compile) @ cloud-apidoc ---
Scanned and found 624 APIs
./build-apidoc.sh: line 83: python: command not found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CloudStack Developer Tools - Checkstyle Configuration SUCCESS [  0.656 s]
[INFO] Apache CloudStack 4.16.0.0-SNAPSHOT ................ SUCCESS [01:12 min]
[INFO] Apache CloudStack Framework - Managed Context ...... SUCCESS [ 38.273 s]
[INFO] Apache CloudStack Framework ........................ SUCCESS [  0.032 s]
[INFO] Apache CloudStack Framework - Certificate Authority  SUCCESS [  0.727 s]
[INFO] Apache CloudStack Utils ............................ SUCCESS [ 56.689 s]
[INFO] Apache CloudStack Framework - Database ............. SUCCESS [  8.167 s]
[INFO] Apache CloudStack Framework - Configuration ........ SUCCESS [  4.165 s]
[INFO] Apache CloudStack Framework - Direct Download to Primary Storage SUCCESS [  0.606 s]
[INFO] Apache CloudStack API .............................. SUCCESS [ 14.381 s]
[INFO] Apache CloudStack Framework - REST ................. SUCCESS [  9.835 s]
[INFO] Apache CloudStack Framework - IPC .................. SUCCESS [  3.446 s]
[INFO] Apache CloudStack Cloud Engine ..................... SUCCESS [  0.012 s]
[INFO] Apache CloudStack Cloud Engine API ................. SUCCESS [  4.078 s]
[INFO] Apache CloudStack Cloud Engine Schema Component .... SUCCESS [ 19.056 s]
[INFO] Apache CloudStack Framework - Security ............. SUCCESS [  1.720 s]
[INFO] Apache CloudStack Core ............................. SUCCESS [ 10.795 s]
[INFO] Apache CloudStack Agents ........................... SUCCESS [ 12.225 s]
[INFO] Apache CloudStack Framework - Clustering ........... SUCCESS [  4.436 s]
[INFO] Apache CloudStack Framework - Event Notification ... SUCCESS [  1.293 s]
[INFO] Apache CloudStack Framework - Jobs ................. SUCCESS [  5.083 s]
[INFO] Apache CloudStack Cloud Engine Internal Components API SUCCESS [  2.428 s]
[INFO] Apache CloudStack Agent Management Servers Load Balancer SUCCESS [  0.610 s]
[INFO] Apache CloudStack Framework - Storage Config Drive Component SUCCESS [ 15.878 s]
[INFO] Apache CloudStack Server ........................... SUCCESS [01:29 min]
[INFO] Apache CloudStack Framework - Quota ................ SUCCESS [  6.012 s]
[INFO] Apache CloudStack Usage Server ..................... SUCCESS [  8.661 s]
[INFO] Apache CloudStack Cloud Engine API ................. SUCCESS [  1.593 s]
[INFO] Apache CloudStack Cloud Engine Orchestration Component SUCCESS [  6.834 s]
[INFO] Apache CloudStack Cloud Services ................... SUCCESS [  0.009 s]
[INFO] Apache CloudStack Secondary Storage ................ SUCCESS [  0.008 s]
[INFO] Apache CloudStack Secondary Storage Service ........ SUCCESS [ 10.557 s]
[INFO] Apache CloudStack Engine Storage Component ......... SUCCESS [  6.229 s]
[INFO] Apache CloudStack Cloud Engine Service ............. SUCCESS [  4.404 s]
[INFO] Apache CloudStack Engine Storage Cache Component ... SUCCESS [  1.923 s]
[INFO] Apache CloudStack Engine Storage Image Component ... SUCCESS [  2.445 s]
[INFO] Apache CloudStack Engine Storage Volume Component .. SUCCESS [  3.783 s]
[INFO] Apache CloudStack Engine Storage Data Motion Component SUCCESS [  7.764 s]
[INFO] Apache CloudStack Engine Storage Snapshot Component  SUCCESS [ 10.811 s]
[INFO] Apache CloudStack Plugin POM ....................... SUCCESS [  0.521 s]
[INFO] Apache CloudStack Plugin - ACL Dynamic Role Based .. SUCCESS [  4.282 s]
[INFO] Apache CloudStack Plugin - ACL Static Role Based ... SUCCESS [  1.731 s]
[INFO] Apache CloudStack Plugin - ACL Project Role Based .. SUCCESS [  1.656 s]
[INFO] Apache CloudStack Plugin - Explicit Dedication Processor SUCCESS [  1.867 s]
[INFO] Apache CloudStack Plugin - Host Affinity Processor . SUCCESS [  4.775 s]
[INFO] Apache CloudStack Plugin - Host Anti-Affinity Processor SUCCESS [  1.713 s]
[INFO] Apache CloudStack Plugin - SNMP Alerts ............. SUCCESS [  9.543 s]
[INFO] Apache CloudStack Plugin - Syslog Alerts ........... SUCCESS [  4.579 s]
[INFO] Apache CloudStack Plugin - API Discovery ........... SUCCESS [  4.590 s]
[INFO] Apache CloudStack Plugin - API Rate Limit .......... SUCCESS [  7.260 s]
[INFO] Apache CloudStack Plugin - Storage Volume default provider SUCCESS [  3.262 s]
[INFO] Apache CloudStack Plugin - Storage Volume SolidFire Provider SUCCESS [  7.011 s]
[INFO] Apache CloudStack Plugin - API SolidFire Integration Testing SUCCESS [  2.035 s]
[INFO] Apache CloudStack Plugin - Dummy Backup and Recovery Plugin SUCCESS [  1.720 s]
[INFO] Apache CloudStack Plugin - Inbuilt Root Certificate Authority SUCCESS [  5.877 s]
[INFO] Apache CloudStack Plugin - Quota Service ........... SUCCESS [  5.842 s]
[INFO] Apache CloudStack Plugin - Dedicated Resources ..... SUCCESS [  5.746 s]
[INFO] Apache CloudStack Plugin - Implicit Dedication Planner SUCCESS [  6.606 s]
[INFO] Apache CloudStack Plugin - User Concentrated Pod Deployment Planner SUCCESS [  1.615 s]
[INFO] Apache CloudStack Plugin - User Dispersing Deployment Planner SUCCESS [  1.714 s]
[INFO] Apache CloudStack Plugin - In Memory Event Bus ..... SUCCESS [  3.994 s]
[INFO] Apache CloudStack Plugin - Kafka Event Bus ......... SUCCESS [  2.955 s]
[INFO] Apache CloudStack Plugin - RabbitMQ Event Bus ...... SUCCESS [  2.175 s]
[INFO] Apache CloudStack Plugin - Skip Heurestics Planner . SUCCESS [  1.587 s]
[INFO] Apache CloudStack Plugin - Host Allocator Random ... SUCCESS [  1.546 s]
[INFO] Apache CloudStack Plugin - Hypervisor Baremetal .... SUCCESS [  2.869 s]
[INFO] Apache CloudStack Plugin - Hypervisor Hyper-V ...... SUCCESS [  3.893 s]
[INFO] Apache CloudStack Plugin - Open vSwitch ............ SUCCESS [  2.368 s]
[INFO] Apache CloudStack Plugin - Hypervisor KVM .......... SUCCESS [ 24.010 s]
[INFO] Apache CloudStack Plugin - Hypervisor OracleVM3 .... SUCCESS [  8.331 s]
[INFO] Apache CloudStack Plugin - Hypervisor OracleVM ..... SUCCESS [  2.783 s]
[INFO] Apache CloudStack Plugin - Hypervisor UCS .......... SUCCESS [  2.193 s]
[INFO] Apache CloudStack Plugin - Hypervisor XenServer .... SUCCESS [ 54.181 s]
[INFO] Apache CloudStack Plugin - Cloudian Connector ...... SUCCESS [  9.223 s]
[INFO] Apache CloudStack Plugin - Prometheus Exporter ..... SUCCESS [  1.997 s]
[INFO] Apache CloudStack Plugin - Kubernetes Service ...... SUCCESS [  8.778 s]
[INFO] Apache CloudStack Plugin - Metrics ................. SUCCESS [  2.202 s]
[INFO] Apache CloudStack Plugin - BigSwitch Virtual Network Segment SUCCESS [  6.989 s]
[INFO] Apache CloudStack Plugin - Dns Notifier Example .... SUCCESS [  1.604 s]
[INFO] Apache CloudStack Plugin - Network Internal Load Balancer SUCCESS [  7.467 s]
[INFO] Apache CloudStack Framework - Spring Life Cycle .... SUCCESS [  1.554 s]
[INFO] Apache CloudStack Plugin - Network Juniper Contrail  SUCCESS [ 17.186 s]
[INFO] Apache CloudStack Plugin - Network Elastic Load Balancer SUCCESS [  5.377 s]
[INFO] Apache CloudStack Plugin - GloboDNS ................ SUCCESS [  9.155 s]
[INFO] Apache CloudStack Plugin - Network Netscaler ....... SUCCESS [  4.555 s]
[INFO] Apache CloudStack Plugin - Network Nicira NVP ...... SUCCESS [  9.932 s]
[INFO] Apache CloudStack Plugin - Network Opendaylight .... SUCCESS [  4.694 s]
[INFO] Apache CloudStack Plugin - Palo Alto ............... SUCCESS [  5.572 s]
[INFO] Apache CloudStack Plugin - Stratosphere SSP ........ SUCCESS [  5.511 s]
[INFO] Apache CloudStack Plugin - Network Brocade VCS ..... SUCCESS [ 15.814 s]
[INFO] Apache CloudStack Plugin - Network VXLAN ........... SUCCESS [  5.402 s]
[INFO] Apache CloudStack Plugin - Power Management Driver ipmitool SUCCESS [  4.315 s]
[INFO] Apache CloudStack Plugin - Power Management Driver nested-cloudstack SUCCESS [  5.071 s]
[INFO] Apache CloudStack Plugin - Power Management Driver Redfish SUCCESS [  1.815 s]
[INFO] Apache CloudStack Plugin - Storage Image default provider SUCCESS [  2.054 s]
[INFO] Apache CloudStack Plugin - Storage Image S3 provider SUCCESS [  1.958 s]
[INFO] Apache CloudStack Plugin - Storage Image sample provider SUCCESS [  1.784 s]
[INFO] Apache CloudStack Plugin - Storage Image Swift provider SUCCESS [  1.951 s]
[INFO] Apache CloudStack Plugin - Storage Volume CloudByte Provider SUCCESS [  3.642 s]
[INFO] Apache CloudStack Plugin - Storage Volume Datera Provider SUCCESS [  3.351 s]
[INFO] Apache CloudStack Plugin - Storage Volume Nexenta Provider SUCCESS [  3.504 s]
[INFO] Apache CloudStack Plugin - Storage Volume sample provider SUCCESS [  1.879 s]
[INFO] Apache CloudStack Plugin - Storage Allocator Random  SUCCESS [  1.744 s]
[INFO] Apache CloudStack Plugin - User Authenticator LDAP . SUCCESS [ 59.052 s]
[INFO] Apache CloudStack Plugin - User Authenticator MD5 .. SUCCESS [  4.161 s]
[INFO] Apache CloudStack Plugin - User Authenticator PBKDF2-SHA-256 SUCCESS [  4.551 s]
[INFO] Apache CloudStack Plugin - User Authenticator Plain Text SUCCESS [  1.633 s]
[INFO] Apache CloudStack Plugin - User Authenticator SAML2  SUCCESS [ 14.750 s]
[INFO] Apache CloudStack Plugin - User Authenticator SHA256 Salted SUCCESS [  4.373 s]
[INFO] Apache CloudStack Framework - Spring Module ........ SUCCESS [  4.661 s]
[INFO] Apache CloudStack Secondary Storage Controller ..... SUCCESS [  5.270 s]
[INFO] Apache CloudStack Console Proxy .................... SUCCESS [  0.009 s]
[INFO] Apache CloudStack Console Proxy - RDP Client ....... SUCCESS [  8.067 s]
[INFO] Apache CloudStack Console Proxy - Server ........... SUCCESS [  7.436 s]
[INFO] Apache CloudStack System VM ........................ SUCCESS [ 26.290 s]
[INFO] Apache CloudStack Client UI ........................ SUCCESS [ 32.441 s]
[INFO] Apache CloudStack Framework - QuickCloud ........... SUCCESS [  0.074 s]
[INFO] Apache CloudStack Test ............................. SUCCESS [  5.673 s]
[INFO] Apache CloudStack Developer Mode ................... SUCCESS [  1.255 s]
[INFO] Apache CloudStack Developer Tools .................. SUCCESS [  0.009 s]
[INFO] Apache CloudStack apidocs .......................... FAILURE [  8.653 s]
[INFO] Apache CloudStack marvin ........................... SKIPPED
[INFO] Apache CloudStack DevCloud4 ........................ SKIPPED
[INFO] Apache CloudStack DevCloud-KVM 4.16.0.0-SNAPSHOT ... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

ACTUAL RESULTS

tools/apidoc/build-apidoc.sh: line 83: python: command not found,must be python2

@saliven1970 were you able to build after replacing that line to python2?
I guess it would still fail with other components especially marvin.
As a workaround you can add a symlink for python2 while building,
ln -sf /usr/bin/python2 /usr/bin/python

Well, that's a good idea. Anyway, it will be upgraded to python3 in the future

@saliven1970 were you able to build after replacing that line to python2?
I guess it would still fail with other components especially marvin.
As a workaround you can add a symlink for python2 while building,
ln -sf /usr/bin/python2 /usr/bin/python

I tried this work around and it is still failing. Any alternative solution?

authors: gokula-krishna-dev saliven1970 shwstppr

repo: apache/cloudstack