text:
[BEAM-9482] Disable Kafka perf tests.

Disable a perma-red suite https://ci-beam.apache.org/job/beam_PerformanceTests_Kafka_IO.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @chamikaramj or @ibzib

Run Seed Job

Run Seed Job


text:
[DO NOT MERGE] Test failures

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Seed Job

Run Jpms Dataflow Java 11 PostCommit

Run Jpms Dataflow Java 17 PostCommit

source: gharchive/pull-request

repo: apache/beam

license_type: permissive

license_source: github-api


text:
DONT MERGE

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Inference Benchmarks


text:
Add support to use side inputs with Combine.PerKeyWithHotKeyFanout

Addresses #20637
This PR fixes the bug that you cannot use side inputs with Combine.PerKeyWithHotKeyFanout by passing down the side inputs from Combine.PerKey

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

assign set of reviewers

Run Java PreCommit

@damondouglas what are next steps here?

@damccorm How do we rerun the two failing checks?

Run Java PreCommit

@marc7806 Thank you again for submitting this PR. I'm running ./gradlew :runners:google-cloud-dataflow-java:validatesRunner to check this PR as the regular ./gradlew :sdks:java:core:check does not trigger relevant tests.

authors: damccorm damondouglas marc7806


text:
Fix javadoc in FnDataReceiver

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

retest this please

This LGTM.

authors: jbonofre lukecwik


text:
[BEAM-4610] Add SSL support for RedisIO

Hi all,
Here is the PR for SSL connection support to RedisIO. The idea is to just add a SSL flag to allow the Redis client to open a SSL connection with its default SSL parameters. Thanks in advance for the review. (R: @iemejia )
Regards,
Guobao
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Thanks @iemejia for the code review! FYI, the modification has been done and it is ready to get merged.

Run Java PreCommit

Run Java PreCommit

authors: EdgarLGB iemejia


text:
Check whether progress reporting could be incorrect in certain cases (technical debt)

See original issue on GitLab
In GitLab by [Gitlab user @jonathanmaw] on Jul 26, 2019, 14:31
Background
!1482 adds a way to report progress when loading elements, but there are points where an element might be loaded or resolved that happen outside the scope of a SimpleTask.
Tests have been added to check that the tallies are correct when loading across a junction in a simple case, but the full spread of possibilities where elements might be loaded or resolved has not been investigated.
Task description

[x] Trace through loader code to see which points Loader._collect_element are called to see which ones will and won't have a Task passed in.
[x] Assess whether a Task should be passed in for those cases.
[x] Write tests to cover any edge cases that are not already covered

Acceptance Criteria

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jul 30, 2019, 16:03
mentioned in merge request !1358

In GitLab by [Gitlab user @LaurenceUrhegyi] on Jul 30, 2019, 16:03
mentioned in merge request !1358

In GitLab by [Gitlab user @tlater] on Sep 10, 2019, 11:31
assigned to [Gitlab user @tlater]

In GitLab by [Gitlab user @tlater] on Sep 10, 2019, 11:31
assigned to [Gitlab user @tlater]

In GitLab by [Gitlab user @tlater] on Sep 13, 2019, 17:42
This seems to only occur here: https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/_loader/loader.py#L575
Which, given that this only loads a junction element and creates a Loader from it, which in turn is always given a Task when it loads elements, should be fine.

In GitLab by [Gitlab user @tlater] on Sep 13, 2019, 17:42
This seems to only occur here: https://gitlab.com/BuildStream/buildstream/blob/master/src/buildstream/_loader/loader.py#L575
Which, given that this only loads a junction element and creates a Loader from it, which in turn is always given a Task when it loads elements, should be fine.

In GitLab by [Gitlab user @tlater] on Sep 18, 2019, 18:33
mentioned in merge request !1608

In GitLab by [Gitlab user @tlater] on Sep 18, 2019, 18:33
mentioned in merge request !1608

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Write tests to cover any edge cases that are not already covered as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Write tests to cover any edge cases that are not already covered as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Write tests to cover any edge cases that are not already covered as incomplete

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Write tests to cover any edge cases that are not already covered as incomplete

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Trace through loader code to see which points Loader._collect_element are called to see which ones will and won't have a Task passed in. as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Trace through loader code to see which points Loader._collect_element are called to see which ones will and won't have a Task passed in. as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Assess whether a Task should be passed in for those cases. as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:26
marked the task Assess whether a Task should be passed in for those cases. as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:27
marked the task Write tests to cover any edge cases that are not already covered as completed

In GitLab by [Gitlab user @tlater] on Oct 8, 2019, 14:27
marked the task Write tests to cover any edge cases that are not already covered as completed

In GitLab by [Gitlab user @marge-bot123] on Oct 10, 2019, 13:45
closed via merge request !1608

In GitLab by [Gitlab user @marge-bot123] on Oct 10, 2019, 13:45
closed via merge request !1608

In GitLab by [Gitlab user @marge-bot123] on Oct 10, 2019, 13:45
mentioned in commit 6c6c581162ed7d87bad680330610e29d828d0f25

In GitLab by [Gitlab user @marge-bot123] on Oct 10, 2019, 13:45
mentioned in commit 6c6c581162ed7d87bad680330610e29d828d0f25

source: gharchive/issue

repo: apache/buildstream


text:
"Too many open files" when running with remote execution

When building with remote execution, I get the following error. I don't know if it's consistent, but retrying once gives the same error.
[16:35:19][00:11:56][e8a2758a][   build:bootstrap/build/gcc-stage1.bst] BUG     Build

    An unhandled exception occured:

    Traceback (most recent call last):
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_scheduler/jobs/job.py", line 438, in child_action
        result = self.child_process()  # pylint: disable=assignment-from-no-return
                 ^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 91, in child_process
        return self._action_cb(self._element)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_scheduler/queues/buildqueue.py", line 55, in _assemble_element
        element._assemble()
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/element.py", line 1689, in _assemble
        collect = self.assemble(sandbox)  # pylint: disable=assignment-from-no-return
                  ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/buildelement.py", line 315, in assemble
        with sandbox.batch(root_read_only=True, label="Running commands"):
      File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
        next(self.gen)
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/sandbox.py", line 265, in batch
        batch.execute()
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/_sandboxreapi.py", line 234, in execute
        self.sandbox._run_with_flags(
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/sandbox.py", line 374, in _run_with_flags
        return self._run(command, flags=flags, cwd=cwd, env=env)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/_sandboxreapi.py", line 101, in _run
        action_result = self._execute_action(action, flags)  # pylint: disable=assignment-from-no-return
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/sandbox/_sandboxremote.py", line 221, in _execute_action
        cascache.pull_tree(casremote, tree_digest)
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 278, in pull_tree
        digest = self._fetch_tree(remote, digest)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 644, in _fetch_tree
        dirdigests = self.add_objects(buffers=dirbuffers)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 370, in add_objects
        tmp = stack.enter_context(self._temporary_object())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/contextlib.py", line 505, in enter_context
        result = _enter(cm)
                 ^^^^^^^^^^
      File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
        return next(self.gen)
               ^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/_cas/cascache.py", line 595, in _temporary_object
        with utils._tempnamedfile(dir=self.tmpdir) as f:
      File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
        return next(self.gen)
               ^^^^^^^^^^^^^^
      File "/home/abderrahimkitouni/.local/pipx/venvs/buildstream/lib/python3.11/site-packages/buildstream/utils.py", line 1268, in _tempnamedfile
        with _signals.terminator(close_tempfile), tempfile.NamedTemporaryFile(
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/tempfile.py", line 702, in NamedTemporaryFile
        file = _io.open(dir, mode, buffering=buffering,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/tempfile.py", line 699, in opener
        fd, name = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.11/tempfile.py", line 395, in _mkstemp_inner
        fd = _os.open(file, flags, 0o600)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    OSError: [Errno 24] Too many open files: '/home/abderrahimkitouni/.cache/buildstream/tmp/tmpr9a6fdwb'

@abderrahim did you ever figure out what this was about? Is there some fd leak?

Nope, I just increased the limit with ulimit and got my build going.

authors: abderrahim nanonyme


text:
Support for incremental builds for workspaces

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Jan 16, 2018, 08:09
Currently BuildStream never caches any build trees after a build, as noted in #21. However, when a workspace is active we have the opportunity to reuse the same workspace directory to build inside of - and workspace builds are where incremental builds will benefit the developer a lot by reducing the edit/compile/test cycles.
This is partly discussed in this mail thread in October 2017, since then a lot of work and discussion has taken place in MR !126.

In GitLab by [Gitlab user @tristanvb] on Jan 16, 2018, 08:16
mentioned in merge request !126

In GitLab by [Gitlab user @cs-shadow] on Jan 25, 2018, 11:58
mentioned in commit 5f1be604603843bc266252cf5a3a91939d2c09f5

In GitLab by [Gitlab user @cs-shadow] on Jan 25, 2018, 13:16
closed via commit 5f1be604603843bc266252cf5a3a91939d2c09f5


text:
WIP: bst fmt: Add basic functionality

See original merge request on GitLab
In GitLab by [Gitlab user @coldtom] on Nov 15, 2018, 16:55
Description
Adds a bst fmt command, which modifies the format of a projects element files into a canonical yaml format. This has the main benefit of automating files into the format bst track will dump them into when run.
Currently the functionality is somewhat stinted by #767, and there are a couple of other features I would like to add to this - namely an option which doesn't modify the yaml, but creates a diff, and a way for a user to define the order of top-level nodes in a .bst file, into which bst fmt sorts the nodes.
Changes proposed in this merge request:

Add a bst fmt command which formats elements into a canonical format

This merge request,  when approved, will close #485

In GitLab by [Gitlab user @jjardon] on Nov 15, 2018, 17:14
changed the description

In GitLab by [Gitlab user @coldtom] on Nov 15, 2018, 17:59
added 1 commit
1d2d0de4 - bst fmt: Add basic functionality
Compare with previous version

In GitLab by [Gitlab user @knownexus] on Nov 19, 2018, 14:21
Commented on buildstream/_frontend/cli.py line 374
Shouldn't there be a requirement for --all to be enabled before we allow --except?
I suppose it doesn't break anything if you do it without, but it seems to have no value in other cases from what i can see

In GitLab by [Gitlab user @knownexus] on Nov 19, 2018, 14:25
Commented on buildstream/_scheduler/queues/formatqueue.py line 12
Is this called anywhere? I can't see it but i may be missing it

In GitLab by [Gitlab user @coldtom] on Nov 21, 2018, 12:56
Commented on buildstream/_scheduler/queues/formatqueue.py line 12
As I understand it this is where one implements the process performed by the queue, and is called by the scheduler. It's an abstract method in the Queue class. However I was wrong to give a return value for the feature implemented here, I'll fix that presently.

In GitLab by [Gitlab user @coldtom] on Nov 21, 2018, 13:47
Commented on buildstream/_frontend/cli.py line 374
Other elements with similar functionality use the same method of implementing --except, personally I'm of the opinion that adding a guard against it would make the code more awkward for little benefit, but I'm not opposed to it.

In GitLab by [Gitlab user @tlater] on Nov 21, 2018, 16:16
Commented on buildstream/_scheduler/queues/formatqueue.py line 12
[Gitlab user @knownexus] it's part of the Queue API here. It's what keeps us from creating new job types for every queue type.

In GitLab by [Gitlab user @tlater] on Nov 21, 2018, 16:19
Commented on buildstream/_stream.py line 297
While I understand that you're just reusing code here, I'd prefer if you changed the name of the kwarg to do so. Perhaps modify_selection or somesuch?

In GitLab by [Gitlab user @tlater] on Nov 21, 2018, 16:26
Commented on buildstream/_frontend/cli.py line 374
I think that the help string should at least mention that it has no effect unelss --all is specified. Otherwise this is probably fine.

In GitLab by [Gitlab user @tlater] on Nov 21, 2018, 16:27
Commented on buildstream/_scheduler/queues/formatqueue.py line 15
I'd be pretty impressed if you could do this. Any suggestions on how this would be implemented? Surely you'd run into the halting problem here.

In GitLab by [Gitlab user @tlater] on Nov 21, 2018, 16:38
So, the ML was pretty happy with this, but as much as I see it being useful, I worry about feature creep.
We have been adding a lot of new commands to BuildStream in this cycle, but they have mostly been there to expose missing bits of API and clean up inconsistencies. This adds an entirely new feature (tracking partially does this, but that's by accident, not design). I would be unhappy landing this without having [Gitlab user @tristan] or [Gitlab user @juergbi] at least approve the feature (neither of them seems to have replied to the ML topic).
Before we land this we should probably also define what "canonical" means, document it, and ensure that our implementation here actually applies that format through tests. I doubt that it currently is particularly stable, or matches what the larger yaml community thinks should be "canonical". In my mind this aims to be an opinionated formatter, and if we're writing one of those, we should do it right. Hence I also think that this entire feature probably should be a separate project. bst-lint is something that has been discussed before, and probably more suitable.
Also, on that note, please add some tests beyond ensuring that completion works.

In GitLab by [Gitlab user @coldtom] on Nov 22, 2018, 09:48
I understand concerns about feature creep, I don't think BuildStream would need this feature if not for bst track changing the formatting on being run. As this is due to ruamel.yaml, there isn't much we can do but mitigate the damage.
Currently canonical format is "whatever the ruamel.yaml dumper spits out by default" but this could be configured to be something defined and consistent. The format we consider "canonical" will have to be consistent with what the dumper does when run, otherwise we will have the same issue when running bst track. This may limit how opinionated we can be.
I'm not opposed to the formatter being a separate tool, but if it is we need to document that bst track will modify your yaml format and we strongly recommend using the tool.

In GitLab by [Gitlab user @coldtom] on Nov 22, 2018, 09:54
Commented on buildstream/_scheduler/queues/formatqueue.py line 15
While considering the implementation details I realised that this would offer little to no optimisation given the feature as it stands. The quickest way I can think of to check if an element is formatted correctly is to dump the node and compare the files, but obviously this is slower than just dumping the node.

In GitLab by [Gitlab user @coldtom] on Nov 22, 2018, 16:44
Commented on buildstream/_stream.py line 297
changed this line in version 3 of the diff

In GitLab by [Gitlab user @coldtom] on Nov 22, 2018, 16:44
Commented on buildstream/_scheduler/queues/formatqueue.py line 15
changed this line in version 3 of the diff

In GitLab by [Gitlab user @coldtom] on Nov 22, 2018, 16:44
added 74 commits
1d2d0de4...abef70fe - 72 commits from branch master3bf4ed98 - bst fmt: Add basic functionality8d77677e - bst fmt: Add tests for core functionality
Compare with previous version

In GitLab by [Gitlab user @tristanvb] on Dec 3, 2018, 09:45
Commented on buildstream/element.py line 1342
This probably needs to have a lot more control over the ordering of keys in the output.
I.e. see https://gitlab.com/BuildStream/buildstream/issues/485#note_121782763

In GitLab by [Gitlab user @coldtom] on Dec 13, 2018, 11:31
marked as a Work In Progress

In GitLab by [Gitlab user @coldtom] on Dec 13, 2018, 13:46
added 157 commits
8d77677e...224aa4c2 - 154 commits from branch master4d3c032a - bst fmt: Add basic functionality6b0a2e1a - bst fmt: Add tests for core functionality778f1db9 - bst-fmt: Allow greater control over order
Compare with previous version

In GitLab by [Gitlab user @coldtom] on Dec 17, 2018, 12:37
added 15 commits
778f1db9...b23bec55 - 12 commits from branch master95307bb4 - bst fmt: Add basic functionality574e5a78 - bst fmt: Add tests for core functionalityc1cef0cc - bst-fmt: Allow greater control over order
Compare with previous version

In GitLab by [Gitlab user @coldtom] on Dec 20, 2018, 11:41
added 12 commits
c1cef0cc...aae5e4b3 - 9 commits from branch master67d8d0a1 - bst fmt: Add basic functionality9b5e2133 - bst fmt: Add tests for core functionalityecaf433f - bst-fmt: Allow greater control over node order
Compare with previous version

In GitLab by [Gitlab user @coldtom] on Dec 20, 2018, 11:43
Commented on buildstream/element.py line 1342
[Gitlab user @tristanvb] I have reworked this to have greater control over the order of dumping, but it has introduced an issue in stripping comments in some situations. I still need to add tests but want feedback on the approach used first.

In GitLab by [Gitlab user @LaurenceUrhegyi] on Mar 26, 2019, 18:15
Hi, thanks for the contribution! Unfortunately this WIP MR is over a month without an update from the author, so as per our policy I'm going to close this to keep our queue tidy. Please do re-open the MR if you come back to work on it. Cheers!

In GitLab by [Gitlab user @LaurenceUrhegyi] on Mar 26, 2019, 18:15
closed

In GitLab by [Gitlab user @jjardon] on Mar 26, 2019, 19:32
This MR is waiting for feedback from the maintainers; reopening

In GitLab by [Gitlab user @jjardon] on Mar 26, 2019, 19:32
reopened

In GitLab by [Gitlab user @jjardon] on May 4, 2019, 03:54
[Gitlab user @coldtom] can you please rebase this? For the comments seems It's (almost) ready to merge

In GitLab by [Gitlab user @danielsilverstone-ct] on Jul 4, 2019, 09:44
Hi, thanks for the contribution! Unfortunately this WIP MR is over a month without an update from the author, so as per our policy I'm going to close this to keep our queue tidy. [Gitlab user @coldtom] please re-open the MR if you come back to work on it. Cheers!

In GitLab by [Gitlab user @danielsilverstone-ct] on Jul 4, 2019, 09:44
closed

In GitLab by [Gitlab user @jjardon] on Dec 9, 2019, 09:15
Again, This MR was waiting for feedback from the maintainers; reopening

In GitLab by [Gitlab user @jjardon] on Dec 9, 2019, 09:15
reopened

In GitLab by [Gitlab user @BenjaminSchubert] on Dec 9, 2019, 10:13
I'm not sure what we would gain with that command:

The ruamel formatter in theory should not change the format of the file at all. In practice, if it does, that's a bug that should be reported upstream and fixed.
Other tools (like yamllint) are very good at linting your yaml, and I believe that having them as a precommit or part of the pipeline might be easier and less work to maintain in BuildStream itself.

In GitLab by [Gitlab user @jjardon] on Dec 12, 2019, 04:34
[Gitlab user @BenjaminSchubert] does yamllint convert to a canonical format or only errors/warns if something is not following the format? This is about the former

In GitLab by [Gitlab user @BenjaminSchubert] on May 11, 2020, 20:46
[Gitlab user @jjardon] woops seems I missed that one. No, yamllint doesn't convert to a canonical format. Seems like we'd still need that one then.


text:
kamel run directory (self contained)

It would also in the future be nice to be able to run by just specifying a directory, then the CLI figures out what files are in the directory and what kind they are
myrepo/foo
- MyRoute.java
- application.properties
- myawesomelib.jar

kamel run github:user/myrepo/foo

And it gets translated into that example above from Luca in #1522 (incl adding the -d for the JAR).
In fact this is "anti modeline", as you dont use the //came-k: comments but I love when end users can just keep all their stuff in the same folder, and also seperated a bit. properties goes into real properties files. And extra JARs are just downloaded JARs (no funky long maven dependencies).

I think this can be made compatible with modeline.
Thinking to a local directory, the CLI can look for all compatible files with a // camel-k: language=x (or another flag that marks it as runnable) and run each runnable file using existing modeline options.
It's common e.g. to write different .properties files that should be picked by some integrations, but not all. Modeline now also contain dependencies that should be read by the CLI.
The only problem that I see in a remote (github.com/user/repo/) is that the CLI would require to list the files or check them out, which is not a lightweight operation.

Ah yeah the github is not as important (just cool for demos and new users to Camel K).
Good point about shared properties files etc or allowing to specify which properties file are to be used by which runnable file.
So you could have
myrepo/foo
- MyRoute.java
- application.properties
- Funky.groovy
- func.properties
- myawesomelib.jar

Then in MyRoute.java and Funky.groovy you can use the modeline to specify which properties files and JARs to include.
And then you can run it as 2 integrations by
kamel run foo

authors: davsclaus nicolaferraro

repo: apache/camel-k


text:
1.9.x TestHealthTrait/Readiness_condition_with_stopped_route failure

Right now, only this test is failing:
    --- FAIL: TestHealthTrait/Readiness_condition_with_stopped_route (416.44s)

Originally posted by @squakez in https://github.com/apache/camel-k/issues/3256#issuecomment-1119494943

Should be fixed already.

authors: squakez tadayosi


text:
Bitcoin Source fails with

It seems like there is a dependency missing for the bitcoin-source.

Note: All the following resources were neated and stripped of namespace for ease of use during debugging.

When running:
apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
  annotations:
    camel.apache.org/operator.id: camel-k
  name: bitcoin-rate
spec:
  sink:
    ref:
      apiVersion: camel.apache.org/v1
      kind: Kamelet
      name: log-sink
  source:
    ref:
      apiVersion: camel.apache.org/v1
      kind: Kamelet
      name: bitcoin-source

it is built correctly, but throws a NoClassDefFoundError during startup for javax/ws/rs/Path.
Operator and client version are both 2.2.0.
The integration platform is configured as follows:
apiVersion: camel.apache.org/v1
kind: IntegrationPlatform
metadata:
  annotations:
    camel.apache.org/operator.id: camel-k
  labels:
    app: camel-k
  name: cs
spec:
  build:
    maven:
      settings:
        secretKeyRef:
          key: settings.xml
          name: maven-settings
      settingsSecurity:
        secretKeyRef:
          key: settings-security.xml
          name: maven-settings
    registry:
      address: harbor.192-168-178-58.sslip.io
      insecure: true
      organization: myorg
      secret: harbor
    runtimeVersion: 3.6.0
  cluster: Kubernetes
  traits:
    camel:
      runtimeVersion: 3.6.0

Full stacktrace as logged:
2024-07-15 17:31:12,191 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus
        at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
        at io.quarkus.runtime.Application.start(Application.java:101)
        at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
        at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
        at io.quarkus.runner.GeneratedMain.main(Unknown Source)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:61)
        at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:32)
Caused by: java.lang.NoClassDefFoundError: javax/ws/rs/Path
        at si.mazi.rescu.RestInvocationHandler.<init>(RestInvocationHandler.java:59)
        at si.mazi.rescu.RestProxyFactory.createProxy(RestProxyFactory.java:47)
        at si.mazi.rescu.RestProxyFactoryImpl.createProxy(RestProxyFactoryImpl.java:9)
        at org.knowm.xchange.client.ExchangeRestProxyBuilder.build(ExchangeRestProxyBuilder.java:80)
        at org.knowm.xchange.binance.service.BinanceBaseService.<init>(BinanceBaseService.java:37)
        at org.knowm.xchange.binance.service.BinanceMarketDataServiceRaw.<init>(BinanceMarketDataServiceRaw.java:23)
        at org.knowm.xchange.binance.service.BinanceMarketDataService.<init>(BinanceMarketDataService.java:28)
        at org.knowm.xchange.binance.BinanceExchange.initServices(BinanceExchange.java:29)
        at org.knowm.xchange.BaseExchange.applySpecification(BaseExchange.java:108)
        at org.knowm.xchange.binance.BinanceExchange.applySpecification(BinanceExchange.java:72)
        at org.knowm.xchange.ExchangeFactory.createExchange(ExchangeFactory.java:110)
        at org.knowm.xchange.ExchangeFactory.createExchange(ExchangeFactory.java:53)
        at org.apache.camel.component.xchange.XChangeComponent.createExchange(XChangeComponent.java:64)
        at org.apache.camel.component.xchange.XChangeComponent.getOrCreateXChange(XChangeComponent.java:72)
        at org.apache.camel.component.xchange.XChangeComponent.createEndpoint(XChangeComponent.java:43)
        at org.apache.camel.support.DefaultComponent.createEndpoint(DefaultComponent.java:170)
        at org.apache.camel.impl.engine.AbstractCamelContext.doGetEndpoint(AbstractCamelContext.java:804)
        at org.apache.camel.impl.engine.AbstractCamelContext.getEndpoint(AbstractCamelContext.java:738)
        at org.apache.camel.support.CamelContextHelper.resolveEndpoint(CamelContextHelper.java:123)
        at org.apache.camel.reifier.SendReifier.resolveEndpoint(SendReifier.java:45)
        at org.apache.camel.reifier.SendReifier.createProcessor(SendReifier.java:37)
        at org.apache.camel.reifier.ProcessorReifier.makeProcessor(ProcessorReifier.java:864)
        at org.apache.camel.reifier.ProcessorReifier.addRoutes(ProcessorReifier.java:604)
        at org.apache.camel.reifier.RouteReifier.doCreateRoute(RouteReifier.java:213)
        at org.apache.camel.reifier.RouteReifier.createRoute(RouteReifier.java:76)
        at org.apache.camel.impl.DefaultModelReifierFactory.createRoute(DefaultModelReifierFactory.java:49)
        at org.apache.camel.impl.DefaultCamelContext.startRouteDefinitions(DefaultCamelContext.java:724)
        at org.apache.camel.component.kamelet.KameletComponent$LifecycleHandler.createRouteForEndpoint(KameletComponent.java:416)
        at org.apache.camel.component.kamelet.KameletComponent$LifecycleHandler.onContextInitialized(KameletComponent.java:430)
        at org.apache.camel.impl.engine.AbstractCamelContext.doInit(AbstractCamelContext.java:2360)
        at org.apache.camel.quarkus.core.FastCamelContext.doInit(FastCamelContext.java:176)
        at org.apache.camel.support.service.BaseService.init(BaseService.java:78)
        at org.apache.camel.impl.engine.AbstractCamelContext.init(AbstractCamelContext.java:1983)
        at org.apache.camel.support.service.BaseService.start(BaseService.java:105)
        at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2002)
        at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:208)
        at org.apache.camel.quarkus.main.CamelMain.doStart(CamelMain.java:94)
        at org.apache.camel.support.service.BaseService.start(BaseService.java:113)
        at org.apache.camel.quarkus.main.CamelMain.startEngine(CamelMain.java:140)
        at org.apache.camel.quarkus.main.CamelMainRuntime.start(CamelMainRuntime.java:49)
        at org.apache.camel.quarkus.core.CamelBootstrapRecorder.start(CamelBootstrapRecorder.java:45)
        at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot173480958.deploy_0(Unknown Source)
        at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot173480958.deploy(Unknown Source)
        ... 13 more
Caused by: java.lang.ClassNotFoundException: javax.ws.rs.Path
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
        at io.quarkus.bootstrap.runner.RunnerClassLoader.loadClass(RunnerClassLoader.java:115)
        at io.quarkus.bootstrap.runner.RunnerClassLoader.loadClass(RunnerClassLoader.java:65)
        ... 56 more

Please let me know in case you need additional information.

This is a problem in the Camel K runtime. If you run the Kamelet in pure Camel with Jbang it will work out of the box. Eventually this should go in Camel K/Camel Quarkus issues. Thanks for reporting.

authors: mwmahlberg oscerd

repo: apache/camel-kamelets


text:
Extend the pod specs for the dev mode and build

Is there a way we could extend / customize the pod / deployment yaml file of the run in devmode and build? Basically, we need to push some traits / tolerances to the pods running, and could not find a way to update it.
Thanks!

Add deployment.jkube.yaml to your project with deployment spec fragment, ex:
spec:
  replicas: 1
  template:
    spec:
      containers:
        - volumeMounts:
            - name: xxx
              mountPath: /xxxxx
      serviceAccount: karavan
      imagePullSecrets:
        - name: karavan-registry

works for build, not for devmode

Traits are high level named features of Camel K Karavan does not use Camel-K but Camel+Jib+JKube

authors: arheom mgubaidullin

repo: apache/camel-karavan


text:
Upgrade Netty to version 4.1.100.Final (#11690)

Description

Target

[x] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[x] I checked that each commit in the pull request has a meaningful subject line and body.

[x] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

/component-test camel-amqp camel-aws/camel-aws2-kinesis camel-coap camel-grpc camel-hl7 camel-iec60870 camel-lumberjack camel-netty-http camel-netty camel-opentelemetry camel-platform-http-main camel-rest-openapi camel-salesforce/camel-salesforce-component camel-stitch camel-syslog camel-telegram camel-undertow camel-webhook camel-whatsapp

repo: apache/camel


text:
CAMEL-12030: Add CoAP response code header

https://issues.apache.org/jira/browse/CAMEL-12030

Thanks the PR has been merged.

authors: jamesnetherton oscerd


text:
[UI] UI Changes to support new storage plugin for PowerFlex/ScaleIO storage pool

Generate URL from the UI inputs when adding "PowerFlex" Primary Storage
Allow VM Snapshot for stopped VM on KVM hypervisor and PowerFlex/ScaleIO storage pool

@blueorangutan package

@shwstppr a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/863 (JID-3673)

@sureshanaparti is this ready to review/test?

@sureshanaparti is this ready to review/test?

not yet @GabrielBrascher

@blueorangutan package

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

@borisstoyanov a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/863 (JID-3816)

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/863 (JID-3816)

authors: GabrielBrascher blueorangutan borisstoyanov shwstppr sureshanaparti

repo: apache/cloudstack-primate