text:
[Failing Test]: Beam ML unit tests fail with Keras 3.3.3 and TF 2.16.1

What happened?
Sample failure:
$ pytest apache_beam/ml/inference/tensorflow_inference_test.py::TFRunInferenceTest::test_predict_numpy_with_batch_size
================================================================================================================================== test session starts ===================================================================================================================================
platform linux -- Python 3.9.18, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/valentyn/projects/beam/beam/beam/sdks/python
configfile: pytest.ini
plugins: xdist-3.6.1, requests-mock-1.12.1, timeout-2.3.1, hypothesis-6.102.1
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 1 item

apache_beam/ml/inference/tensorflow_inference_test.py F                                                                                [100%]

======================================================================================================================================== FAILURES ========================================================================================================================================
_________________________________________________________________________________________________________________ TFRunInferenceTest.test_predict_numpy_with_batch_size __________________________________________________________________________________________________________________

self = <apache_beam.ml.inference.tensorflow_inference_test.TFRunInferenceTest testMethod=test_predict_numpy_with_batch_size>

    def test_predict_numpy_with_batch_size(self):
>     model = _create_mult2_model()

apache_beam/ml/inference/tensorflow_inference_test.py:220:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/ml/inference/tensorflow_inference_test.py:68: in _create_mult2_model
    inputs = tf.keras.Input(shape=(3))
test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-ml/py39-ml/lib/python3.9/site-packages/keras/src/layers/core/input_layer.py:143: in Input
    layer = InputLayer(
test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-ml/py39-ml/lib/python3.9/site-packages/keras/src/layers/core/input_layer.py:46: in __init__
    shape = backend.standardize_shape(shape)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

shape = 3

    def standardize_shape(shape):
        if not isinstance(shape, tuple):
            if shape is None:
                raise ValueError("Undefined shapes are not supported.")
            if not hasattr(shape, "__iter__"):
>               raise ValueError(f"Cannot convert '{shape}' to a shape.")
E               ValueError: Cannot convert '3' to a shape.

test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-ml/py39-ml/lib/python3.9/site-packages/keras/src/backend/common/variables.py:530: ValueError

Issue Failure
Failure: Test is continually failing
Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

cc: @damccorm - we might want to prioritize https://github.com/apache/beam/issues/30908.

to repro:
gradlew :sdks:python:test-suites:tox:py39:testPy39ML -Pposargs=apache_beam/ml/

repo: apache/beam


text:
[BEAM-11707] Change WindmillStateCache cache invalidation to be based…

… upon
reference invalidation instead of expensive set management. Reduce
operations of shared cache by caching per-key object sets locally and
flushing as groups to shared cache. Remove byte tracking which could
be racy based upon background evictions in favor of just iterating
for rendering the status page. Since we render only once every 10 minutes
by default this seems fine.
This also lets us capture more stats.
This previously showed up as 6% cpu on benchmarks and dropped to 1.25% with this change with throughput increase as well.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @dpmills @reuvenlax

friendly ping

@reuvenlax PTAL thanks!

run dataflow valiatesrunner

Run Dataflow ValidatesRunner

Run Java PostCommit

This appears to break https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/361/testReport/junit/org.apache.beam.sdk.transforms/ParDoTest$StateTests/testOrderedListStateRangeDeleteUnbounded/

I'm not exactly sure why this isn't triggering already but I believe the issue can occur with the IdTracker possibly writing empty maps here:
https://github.com/apache/beam/blob/3bb232fb098700de408f574585dfe74bbaff7230/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L767
Then if a read is performed and it is not cached, there is a non-null encoded value (mapcoder encodes size) and then an EmptyMap is coming from MapCoder:
https://github.com/apache/beam/blob/7c43ab6a8df9b23caa7321fddff9a032a71908f6/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/MapCoder.java#L111
Since the creation of the maps only looks if null is returned, the EmptyMap is possibly inserted into
https://github.com/apache/beam/blob/3bb232fb098700de408f574585dfe74bbaff7230/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillStateInternals.java#L702
I think the IdTracker should delete the values if the maps are empty instead of caching empty maps.

Run Dataflow ValidatesRunner

Run Java PostCommit

source: gharchive/pull-request

authors: reuvenlax scwhittle

license_type: permissive

license_source: github-api


text:
Improvements to build_release_candidate.sh to build from tag with shallow clone

With these changes, I have successfully gotten the source release and wheels onto dist.apache.org. Currently blocked on unrelated problems building docker images.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

---

XLang

---

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

You can see from each separate commit what it is doing.

I think Cham is not available so I am going back further in release manager timeline to someone who might have these fresh in their minds.

This doesn't touch anything that even has tests so I won't want Jenkins resources.

R: @pabloem

sorry about the delay! looking....

authors: kennknowles pabloem


text:
Fix multi_process_shared.py release with AutoProxy

Right now, releasing a multi_process_shared object that exists in a different process fails. There were actually 2 issues I ran into debugging this:

The first fix you'll see is more specific to some older python environments. Some environments have a version of AutoProxy that doesn't know how to handle the manager_owned kwarg which is passed in by the multiprocessing library as part of this flow. To work around this, I added a patched version that can handle that arg.

The releasing logic was wrong. Specifically, we weren't properly registering the singletonProxy_release so that it was discoverable across processes, and we were passing in the wrapped proxy object to call this instead of the actual real proxy object. I updated both of these.

Fixes #27602

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Python_Runners Precommit

Run Python_Runners PreCommit


text:
End to end BoundedTrie metrics through worker and runner APIs.

This should finish the Python implementation.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @rohitsinha54

The prism and flink failures look unrelated.


text:
[BEAM-4649] Wait on connection channel to be ready

A fix similar to https://github.com/apache/beam/pull/5780/files
R: @alanmyrvold @robertwb
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---
---

retest this please


text:
Add helper task to print pipeline options for Dataflow portability

Currently it is difficult to manually run a portabiltity pipeline as a
Beam user. This change adds a new Gradle task to conveniently generate
necessary artifacts and print the necessary commandline options.
Usage:
./gradlew -p runners/google-cloud-dataflow-java printFnApiPipelineOptions

Sample output:
> Task :beam-runners-google-cloud-dataflow-java:printFnApiPipelineOptions
To run a Dataflow job with portability, add the following pipeline options to your command-line:
--dataflowWorkerJar=/usr/local/google/home/swegner/beam/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.9.0-SNAPSHOT.jar --workerHarnessContainerImage=gcr.io/dataflow-build/swegner/beam/java:20181107141351 --experiments=beam_fn_api

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

R: @boyuanzz
CC: @lukecwik @rohdesamuel @ajamato @Ardagan @HuangLED @huygaa11

Run Dataflow PortabilityApi ValidatesRunner

We also launch portable pipelines in javaPrecommit: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/examples/build.gradle#L79. Do you think it's necessary adding this help into the precommit? @swegner

The reason I found a need for this: I'm trying to run a simple prototype pipeline from outside of the Gradle build (as a user, following the Java Quickstart). So the commandline I need to run is something like:
mvn compile exec:java -Dexec.mainClass=example.MyPrototype -Dexec.args=" \
  --runner=DataflowRunner \
  --project=<..> --gcpTempLocation=<..> \
  [extra portability args]" \
  -Pdataflow-runner

I don't believe we publish maven artifacts to fulfill [extra portability args], so we need to run the Gradle tasks and use their results.
It seems there's a lot of redundant logic in the examples project you link above (i.e. buildAndPushDockerContainer. @boyuanzz do you know if the redundancy is intentional or should I unify them?

Run Dataflow PortabilityApi ValidatesRunner

For end users, they are supposed to use the worker jar cached by Dataflow service.

@boyuanzz do you know if the service caches worker jars for portable pipelines (in addition to legacy?) Also, are we still publishing a 'latest' image for running from a -SNAPSHOT build? /cc @lukecwik

For end users, they are supposed to use the worker jar cached by Dataflow service.

@boyuanzz do you know if the service caches worker jars for portable pipelines (in addition to legacy?)
I believe the service also caches portable worker jar. If the dataflowWorkerJar option is missing, then it will pick the caches worker jar. Here is the design doc, maybe more illustrative: https://docs.google.com/document/d/1-m-GzkYWIODKOEl1ZSUNXYbcGRvRr3QkasfHsJxbuoA/edit#heading=h.gh88g5y0rekp

Thanks, good to know. I think this helper is still useful in order to use the worker image from HEAD since the cached version in the service may be updated less frequently.
I believe this is ready for review. @boyuanzz PTAL

Yes, it would be very helpful if dev is trying to test their dataflow worker changes. Btw, the ValidatesRunner test takes ~3 hr to complete.

Run Java PortabilityApi PostCommit

FYI, this appears to be breaking post-commits: https://issues.apache.org/jira/browse/BEAM-6028

authors: boyuanzz swegner


text:
[BEAM-7766] Default to UNRECOGNIZED state when a state cannot be accurately interpreted by the SDK.

Add an UNRECOGNIZED pipeline state to accommodate cases where a runner (e.g. v1beta3api) introduces a pipeline state, and (older) version of Beam SDK cannot interpret it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

retest this please

cc: @kennknowles

Run Python_PVR_Flink PreCommit

R: @kennknowles (once you are back).


text:
BIGTOP-3580. Fix deployment failure on Debian 10 caused by the Debian 11 release.

https://issues.apache.org/jira/browse/BIGTOP-3580

Tested on Debian 9, 10 and Ubuntu 18.04, 20.04.

repo: apache/bigtop

license_source: bigquery


text:
Add docs for Scheduled and PeriodicEffectorPolicy

Docs for brooklyn-server#822

LGTM @drigodwin

Merging as brooklyn-server#822 is merged

authors: drigodwin grkvlt

repo: apache/brooklyn-docs


text:
Fixed BlobStoreContextFactoryImpl. It can now have dsl in its config

Fixes a bug introduced in this PR https://github.com/apache/brooklyn-server/commit/013b38bf0f146114d3a67423c0dcc3661b50f88c#diff-30cb47bc0f055776b4a246abeddd4ad7L99
Before that change we would ask the JCloudsLocation directly for config. JCloudsLocation uses a ResolvingConfigBag which will resolve dsl (e.g. "$brooklyn:external").
After the above change, we get the ConfigBag from the location first and ask the ConfigBag for the result. When we get the ConfigBag from the location we create a new ConfigBag to return, but we do not create a resolving config bag.
This PR changes back to asking the location directly for config

LGTM; I also ran the live tests in org.apache.brooklyn.core.mgmt.persist.jclouds: BlobStoreTest and BlobStoreExpiryTest.
It's a shame about the signature BlobStoreContextFactory.newBlobStoreContext(Location) - it taking a ConfigBag felt nicer, but I see how that doesn't work (without resorting to the ResolvingConfigBag, which feels overly complicated). It also means there is less symmetry with ComputeServiceRegistry.findComputeService(ConfigBag conf, boolean allowReuse), but we can live with that as well.
I think we can treat org.apache.brooklyn.api.location.Location.BlobStoreContextFactory as being like an internal interface so don't need to worry about backwards compatibility.
A shame we don't have any non-live tests for this stuff. Longer term, we could add something akin to the StubbedComputeServiceRegistry but it's certainly not the place of this PR to add that.
Merging now.

@aledsage Why should BlobStoreContextFactory be regarded as an internal interface? It's already being used in a downstream project that this PR has broken. Should the project use something else?

@sjcorbett oops, I didn't realise it was in use. In hindsight, it's understandable that power-users in downstream projects would implement this so they can get the exact (configuration of) BlobStoreContext that they need.
Other than the hassle of updating your impl, do the changes here make things particularly difficult for you?

I previously had in my head that we'd really only have two impls: the BlobStoreContextFactoryImpl and a stub test impl for unit testing (which isn't written yet).
That's how ComputeServiceRegistry is used, I believe (but who knows what folk have done in their downstream projects!).
I hope that BlobStoreContextFactoryImpl can become configurable enough that folk don't need to create their own implementations, but maybe that's unrealistic.

There's not a good parallel in the downstream project at the moment. It only has the config bag to use, not a location. I think I'll just take a copy of the previous version of the class.

authors: Graeme-Miller aledsage sjcorbett

repo: apache/brooklyn-server


text:
junctions don't work with project.refs references

See original issue on GitLab
In GitLab by [Gitlab user @abderrahimk] on Apr 6, 2018, 17:10
With ref-storage: project.refs, I get the following error, even though I used bst track and the ref is present in project.refs.
[00:00:02][][] FAILURE Fetching https://gitlab.com/freedesktop-sdk/freedesktop-sdk/
[00:00:02][][] FAILURE Loading pipeline
Error loading pipeline: git source at base/freedesktop-sdk-junction.bst [line 3 column 2]: expected ref 'None' was not found in git repository: 'https://gitlab.com/freedesktop-sdk/freedesktop-sdk/'

A workaround is to manually add the ref in the .bst file for the junction, but I really don't like that.
I've already mentioned it in https://gitlab.com/BuildStream/buildstream/issues/292#note_64566339 but I think this more important than the rest.

In GitLab by [Gitlab user @abderrahimk] on Apr 6, 2018, 17:17
changed the description

In GitLab by [Gitlab user @tristanvb] on Apr 6, 2018, 17:46
Commented on wrong issue from handphone at 2am in bar it seems: please verify that this is fixed in master, i fixed this yesterday.

In GitLab by [Gitlab user @abderrahimk] on Apr 6, 2018, 19:09
Indeed, this was fixed in 857751fb6479afa0ee83d4759891f7c4837b87bb. Is a new release supposed to happen soon?

In GitLab by [Gitlab user @abderrahimk] on Apr 6, 2018, 19:09
closed

source: gharchive/issue

repo: apache/buildstream


text:
No warning produced for import element with configure-commands set

See original issue on GitLab
In GitLab by [Gitlab user @Qinusty] on Sep 19, 2018, 15:21
Summary
When building and element with kind: import and...
config:
  configure-commands:
  - commandA.sh

BuildStream goes about the build without warning or failing. This is perhaps confusing for the user expecting their commands to be executed.
Steps to reproduce
Build this element within a project
kind: import

sources:
- kind: git
  url: https://gitlab.com/BuildStream/buildstream.git
  track: master

config:
  configure-commands:
  - ls

What is the current bug behavior?
BuildStream performs the build with no warnings or indication that anything is wrong.
What is the expected correct behavior?
BuildStream should at least warn the user that they're probably not going to see the behaviour they're expecting.
Relevant logs and/or screenshots
Possible fixes
Other relevant information

BuildStream version affected: master

In GitLab by [Gitlab user @Qinusty] on Sep 20, 2018, 15:19
This will end up being an enhancement and addition to the Plugin API.
What I propose is that we add BST_VALID_CONFIGURATION_ITEMS which when set to None (default) will not perform any validation on the configuration of an element.
When an element plugin provides a BST_VALID_CONFIGURATION_ITEMS, Element.__extract_config() will verify that the keys used in the .bst file are valid.
Perhaps this could make use of the Configurable Warnings and raise warnings by default.

In GitLab by [Gitlab user @Qinusty] on Sep 20, 2018, 16:46
Rough working implemented in https://gitlab.com/BuildStream/buildstream/tree/Qinusty/663-missing-cache-key-workspace-open, nested dict checking is broken.
Feel free to use this for the base of the future work.

In GitLab by [Gitlab user @toscalix] on Oct 1, 2018, 08:56
assigned to [Gitlab user @Qinusty]

In GitLab by [Gitlab user @juergbi] on Feb 10, 2019, 20:27
mentioned in commit 354c563d47f5b2e76d4ba9d545104191277eb8ce

In GitLab by [Gitlab user @juergbi] on Feb 10, 2019, 20:27
mentioned in merge request !1141

In GitLab by [Gitlab user @juergbi] on Feb 11, 2019, 05:10
closed via merge request !1141

In GitLab by [Gitlab user @juergbi] on Feb 11, 2019, 05:10
mentioned in commit 5e1be71f1816e4782eec50ad36ede227402be9b2


text:
MDC not logging overriden breadcrumbId or custom property

Configured log to print MDC fields camel.exchangeId, camel.breadcrumbId and custom property 'traceId'. Only camel.exchangeId is printed in log, however twice as it is also the breadcrumbId. Code and log output:
quarkus.log.level=INFO
quarkus.log.console.enable=true
quarkus.log.console.color=true
quarkus.log.console.format=  (%t) %X{camel.exchangeId} %X{camel.breadcrumbId} %X{traceId} %s%e%n

@ApplicationScoped
public class SimpleRoute  extends EndpointRouteBuilder {

    @Override
    public void configure() throws Exception {
        getContext().setUseMDCLogging(true);
        getContext().setUseBreadcrumb(true);
        from("timer://first?period=5000")
                .setHeader(Exchange.BREADCRUMB_ID, constant("bar"))
                .setProperty(Exchange.BREADCRUMB_ID, constant("bar"))
                .setProperty("traceId", constant("bar"))
                .setHeader("traceId", constant("bar"))
                .log("processing done");
    }
}

(Camel (camel-26) thread #42 - timer://first) 0FC1A7169D4FB9B-0000000000000001 0FC1A7169D4FB9B-0000000000000001  processing done

Does your example work in plain Camel without Quarkus?
MDCUnitOfWork handles adding fields to the MDC and it probably runs before you attempt to override camel.breadcrumbId.
If you want to override the default behaviour, you can extend MDCUnitOfWork. There's an old stackoverflow.com thread that talks about it (might need some adapting for Camel 3.x):
https://stackoverflow.com/questions/29045419/apache-camel-mdc-add-field-from-body

Does your example work in plain Camel without Quarkus?
MDCUnitOfWork handles adding fields to the MDC and it probably runs before you attempt to override camel.breadcrumbId.
If you want to override the default behaviour, you can extend MDCUnitOfWork. There's an old stackoverflow.com thread that talks about it (might need some adapting for Camel 3.x):
https://stackoverflow.com/questions/29045419/apache-camel-mdc-add-field-from-body

@jamesnetherton yes, it worked without quarkus. Would be nice if it would work also with quarkus out of the box

@jamesnetherton yes, it worked without quarkus. Would be nice if it would work also with quarkus out of the box

@jamesnetherton another observation, gelf logger is logging correctly:
quarkus.log.handler.gelf.include-full-mdc=true
and then, camel_breadcrumbId is logged with value set

@jamesnetherton another observation, gelf logger is logging correctly:
quarkus.log.handler.gelf.include-full-mdc=true
and then, camel_breadcrumbId is logged with value set

I've implemented CustomUnitOfWorkFactory but it is not picked up by camel
@ApplicationScoped
public class CustomUnitOfWorkFactory implements UnitOfWorkFactory {

I've implemented CustomUnitOfWorkFactory but it is not picked up by camel
@ApplicationScoped
public class CustomUnitOfWorkFactory implements UnitOfWorkFactory {

I've implemented CustomUnitOfWorkFactory but it is not picked up by camel

Add the @Unremovable annotation, otherwise Quarkus will clean up unused beans.

I've implemented CustomUnitOfWorkFactory but it is not picked up by camel

Add the @Unremovable annotation, otherwise Quarkus will clean up unused beans.

@jamesnetherton  @Unremovable doesn't help. Factory still not called. I've also injected it into the route to make sure it is not removed

@jamesnetherton  @Unremovable doesn't help. Factory still not called. I've also injected it into the route to make sure it is not removed

You need camel-quarkus-main on the classpath in order for the auto discovery of the UnitOfWorkFactory to happen.
Otherwise you could do it manually in the RouteBuilder:
public void configure() {
            getContext().getExtension(ExtendedCamelContext.class).setUnitOfWorkFactory(new CustomUnitOfWorkFactory());
}

You need camel-quarkus-main on the classpath in order for the auto discovery of the UnitOfWorkFactory to happen.
Otherwise you could do it manually in the RouteBuilder:
public void configure() {
            getContext().getExtension(ExtendedCamelContext.class).setUnitOfWorkFactory(new CustomUnitOfWorkFactory());
}

@jamesnetherton Thank you! that worked.
In case someone need it. You have to override beforeProcess method in your CustomUnitOfWork implementation
    @Override
    public AsyncCallback beforeProcess(Processor processor, Exchange exchange, AsyncCallback callback) {
        String value = exchange.getIn().getHeader(Exchange.BREADCRUMB_ID, String.class);
        if (value != null) {
            MDC.put("camel." + Exchange.BREADCRUMB_ID, value);
        }
        return super.beforeProcess(processor, exchange, callback);
    }

then in config file use format %X{camel.breadcrumbId}

@jamesnetherton Thank you! that worked.
In case someone need it. You have to override beforeProcess method in your CustomUnitOfWork implementation
    @Override
    public AsyncCallback beforeProcess(Processor processor, Exchange exchange, AsyncCallback callback) {
        String value = exchange.getIn().getHeader(Exchange.BREADCRUMB_ID, String.class);
        if (value != null) {
            MDC.put("camel." + Exchange.BREADCRUMB_ID, value);
        }
        return super.beforeProcess(processor, exchange, callback);
    }

then in config file use format %X{camel.breadcrumbId}

authors: ennishol jamesnetherton

repo: apache/camel-quarkus


text:
CAMEL-11373: Add camel-twilio Karaf feature and update doc

https://issues.apache.org/jira/browse/CAMEL-11373

@davsclaus Done. I hope it's fine now.

Thanks for the PR it has been merged. Do you mind closing this?

Thank you!

authors: davsclaus tadayosi

repo: apache/camel


text:
fixed typo

Signed-off-by: eric-lee-ltk dagang.li@huawei.com

Thanks for the PR.

authors: eric-lee-ltk oscerd


text:
Resolves CAMEL-15033: fix xrefs

This fixes the xref problems (locally) and there are no link check errors, but I see
✖ 26017 problems (26017 errors, 0 warnings)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
ERROR: "check:html" exited with 1.

These don't immediately appear to relate to the antora-build pages.

Thank you!

authors: djencks oscerd


text:
[CARBONDATA-1490] Fixed memory allocation for carbon row

Analysis: While creating carbon row the space for measures is assigned based on outputLength and not on measureCount which will tend to allocate more space than is actually needed.
outputLength = measureCount + ((noDictCount + complexCount) > 0 ? 1 : 0) + 1
When the table has 1 complex column then measureCount=0, noDictCount=0, complexCount=1
therefore, outputLength = 2
Even if there are no measure columns the measure object is created with 2 null values.
Solution: Create measure object based on the measureCount.

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/239/

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/115/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/870/

LGTM

repo: apache/carbondata


text:
[CARBONDATA-1894] Add compactionType Parameter to compaction event

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[x] Testing done
Manual Testing

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1935/

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/707/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1938/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/712/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2264/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2265/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/717/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1944/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2274/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/728/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1956/

LGTM

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2277/

authors: CarbonDataQA dhatchayani manishgupta88 ravipesala


text:
[CARBONDATA-3620][CARBONDATA-3622]: Update does not load cache in memory, behavior inconsistent with scenario when index server is not running

Problems
1:  Segments for prepriming were never being sent to the index server  after update success.
2 While dropping a table the table is dropped first and then clear datamap is called, If the table is being accessed somewhere while clearing datamaps in the index server then it will fail and throw exception, thus the cache value will never be cleared.
Solutions
1: Segments for prepriming  to the index server added after Update success.
2: While dropping the table, to clear cache first and then drop the table.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1187/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1197/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1207/

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1189/

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1198/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1206/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1216/

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1209/

@vikramahuja1001 in description, in Problem section, please list problem of both jira's and then in solution section, write corresponding solution points.

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1218/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1228/

LGTM

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1225/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1235/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1244/

LGTM

retest this please

Build Success with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/1282/

Build Success with Spark 2.2.1, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.2/1291/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1302/

@vikramahuja1001 please resolve conflicts

@vikramahuja1001 please rebase

retest this please

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1499/

@akashrn5 , done. Please check

retest this please

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1570/

LGTM

LGTM


text:
[CARBONDATA-4288][CARBONDATA-4289] Fix various issues with Index Server caching mechanism.

Why is this PR needed?
There are 2 issues in the Index Server flow:

In case when there is a main table with a SI table with prepriming disabled and index serve enabled, new load to main table and SI table put the cache for the main table in the index server. Cache is also getting again when a select query is fired. This issue happens because during load to SI table, getSplits is called on the main table segment which is in Insert In Progress state. Index server considers this segment as a legacy segment because it's index size = 0 and does not put it's entry in the tableToExecutor mapping. In the getsplits method isRefreshneeded is false the first time getSplits is called. During the select query, in getSplits method isRefreshNeeded is true and the previous loaded entry is removed from the driver but since there is no entry for that table in tableToExecutor mapping, the previous cache value becomes dead cache and always stays in the index server. The newly loaded cache is loaded to a new executor and 2 copies of cache for the same segment is being mantained.
Concurrent select queries to the index server shows wrong cache values in the Index server.

What changes were proposed in this PR?
The following changes are proposed to the index server code:

Removing cache object from the index server in case the segment is INSERT IN PROGRESS and in the case of legacy segment adding the value in tabeToExecutor mappping so that the cache is also removed from the executor side.
Concurrent queries were able adding duplicate cache values to other executors. Changed logic of assign executors method so that concurrent queries are not able to add cache for same segment in other executors

Does this PR introduce any user interface change?

No

Is any new testcase added?

Yes. Test case added to check assignExecutors method using multiple threads.

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4190/

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/337/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5939/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4193/

LGTM

LGTM

LGTM

Build Failed  with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/341/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5940/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4194/


text:
CASSANDRA-16360 CRC32 is inefficient on x86 and replaced with CRC32C

Thanks for sending a pull request! Here are some tips if you're new here:

Ensure you have added or run the appropriate tests for your PR.
Be sure to keep the PR description updated to reflect all changes.
Write your PR title to summarize what this PR proposes.
If possible, provide a concise example to reproduce the issue for a faster review.
Read our contributor guidelines
If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:
<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

I may be missing something here but during handshake initiation you have to deal with 3 possible communication configurations with the new CRC.

initiator
peer

old-CRC
new-CRC

new-CRC
old-CRC

new-CRC
new-CRC

My reading of the Handshake.java file indicates that the initial handshake has a CRC digest attached.
If the old-CRC is initiating then the new-CRC has to validate against the old-CRC, so it must have the old-CRC code and be ready to used it.
if the new CRC is initiating then it must use the old CRC to communicate with the old-CRC.
So the old-CRC code still has to exist in the codebase and it has to be used for session initiation.
Is there something in the new code that guarantees that the initiation sequencde uses the old-CRC?
My thought is that we could use another flag in the init header the on initiation being set means (I accept new-CRC) and the a new-CRC supporting client could respond with a message that also has the bit set and uses the new-CRC, so that we can jump into the new protocol faster.

Can you, please, resolve conflicts with trunk?

authors: Claudenw Mmuzaf nizhikov

repo: apache/cassandra


text:
CASSANDRA-19577 Make queries visible to the "system_views.queries" virtual table at the coordinator level

patch by Caleb Rackliffe; reviewed by David Capwell for CASSANDRA-19577

Committed as https://github.com/apache/cassandra/commit/bbcf92b2c45e04e06554fb634166c0e37af38891


text:
CLIMATE-739 - Add img/globe.png to correct file not found.

Pulled img/globe.png from older branch to correct file not found in main.html.

Can one of the admins verify this patch?

Thanks @MichaelArthurAnderson can you please close this issue off?

By any chance is the jira down this morning?   Couldn't log in and still can't in spite of 3 password resets from two different devices.
Sent from my iPhone

On Feb 2, 2016, at 13:15, Lewis John McGibbney notifications@github.com wrote:
Thanks @MichaelArthurAnderson can you please close this issue off?
—
Reply to this email directly or view it on GitHub.

@MichaelArthurAnderson no Jira is not down. All of these messages are being shadowed over there :|
I meant just this issue. Can you close it? It has been closed over on Jira. Thanks

repo: apache/climate


text:
Check and fix all UI issue from the archived primate repository

Triage and address open issues from https://github.com/apache/cloudstack-primate/issues
And ask authors to port their PRs (if not done already) https://github.com/apache/cloudstack-primate/pulls
ISSUE TYPE

Task

COMPONENT NAME

UI

https://github.com/apache/cloudstack-primate/issues/158

[x] https://github.com/apache/cloudstack/pull/4762 - When opening a dialog, the first element should be active (with the cursor ready for typing). This avoids extra clicks.
[x] https://github.com/apache/cloudstack/pull/4763 - Some dialogs do not have 'Cancel' buttons - for example 'Assign Instance to Another Account'. The 'x' icon on the top right corner is always an option but I think we should be consistent here.

https://github.com/apache/cloudstack-primate/issues/462

[x] https://github.com/apache/cloudstack/pull/4764 - Create zone form is not aligned in mobile view
[x] https://github.com/apache/cloudstack/pull/4765 - Page size changer doesn't show up on mobile mode
[x] https://github.com/apache/cloudstack/pull/4766 - Enter to submit a form
[x] https://github.com/apache/cloudstack/pull/4769 - While listing domains, on full page reload, it goes back to the root domain
[x] https://github.com/apache/cloudstack/pull/4767 - Account in user info card doesn't directly go to account but to the account listview

https://github.com/apache/cloudstack-primate/issues/465 & https://github.com/apache/cloudstack-primate/issues/615

[x] https://github.com/apache/cloudstack/pull/4782 - Refactor async job polling codebase-wide and show multiple async jobs

https://github.com/apache/cloudstack-primate/issues/719

[x] https://github.com/apache/cloudstack/pull/4811 - Move fetch data to created()

https://github.com/apache/cloudstack-primate/issues/743

[x] https://github.com/apache/cloudstack/pull/4820 - Edit instance - offer existing Groups

Thanks for the update @utchoang

@utchoang are all the issues fixed/raised, can this ticket be closed?

Currently, there is still the PR (https://github.com/apache/cloudstack-primate/pull/898) to consider. I will update it now.

[x] https://github.com/apache/cloudstack/pull/4885 - Multiple Management Server support

@utchoang thanks, the PRs that introduce new features can be raised against master; PRs fixing any bug or introducing (minor) enhanacement can be considered for 4.15 branch. If all the PRs have been raised, can I close this issue?

@rhtyd Yes, it is possible to close it

authors: rhtyd utchoang

repo: apache/cloudstack


text:
Can't add a host in Advanced Zone - Unable to add the host

ISSUE TYPE

Bug Report

COMPONENT NAME

Zone, management

CLOUDSTACK VERSION

4.15.1

CONFIGURATION

KVM
Advanced Zone
Single server installation (mgmt, host, db, storage)
LocalStorage enabled
OS / ENVIRONMENT

Ubuntu 20.04.2
SUMMARY

I'm able to add a host X to the Basic Zone
However, I'm getting an error when I try to add the second (Advanced) Zone and add the same host X to it.
I've also tried removing the host X from the Basic Zone first, expunging all VMs and SystemVMs and restarting cloudstack-management and cloudstack-agent services. Still getting the same exception
log
STEPS TO REPRODUCE

1. Setup CloudStack on a single server (mgmt, host, db, storage)
2. Add a Basic Zone
3. Add the host X to the Basic Zone
4. Add Advanced Zone
5. Add  host X to the Advanced Zone
6. Notice the exception - Unable to add the host
7. Shut down the Basic Zone
8. Remove host X from the Basic Zone
9. Add host X to the Advanced Zone
10. Notice the exception - Unable to add the host

EXPECTED RESULTS

1. Add a host to the Advanced Zone
2. Add the same host X to both Basic and Advanced zones

ACTUAL RESULTS

1. Can add a host to the Basic Zone
2. Fail to add a host to the Advanced Zone

@tampler was the username 'root' when you added the host ?

@tampler you can not add same host to multiple zones. it is not an issue.

@tampler
by the way, could you subscribe to cloudstack users mailing list and ask question in mailing list ?
https://cloudstack.apache.org/mailing-lists.html

@weizhouapache
Hello, that's not a question, that's a BUG report. Have you looked into a log attached? There's an exception adding a host in Ubuntu for an Advanced Zone, please re-open my issue

@tampler
by the way, could you subscribe to cloudstack users mailing list and ask question in mailing list ?
https://cloudstack.apache.org/mailing-lists.html
No, sorry, I can't. Not using mailing lists

@tampler, it is failing because of this
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper and I guess you have to allow sudo to be used without a password

@slavkap Wow thanks a lot for your hint. You are my CloudStack Failry now :grinning: :grinning: :grinning: :innocent:
Sure, adding my non-root user to sudoers, who don't need password made the trick
Thanks a lot for your patience and a wise help. I really appreciate
Closing the issue

authors: slavkap tampler weizhouapache


text:
VR won't start with Linstor cluster having non-hypervisor nodes

ISSUE TYPE

Bug Report

COMPONENT NAME

Linstor, KVM, VR

CLOUDSTACK VERSION

4.18.1

CONFIGURATION

3 ubuntu 22.0.4 KVM nodes in the Linstor cluster.
1 Node configured as Combined Linstor node type and the only one providing the storage.
OS / ENVIRONMENT

Ubuntu 22.0.4 KVM
Linstor
SUMMARY

On deploying Virtual router onto a KVM cluster configured as Linstor cluster well where there is one Linstor node outside the KVM cluster then the VR won't start. The patching step times out.
024-02-02 10:38:25,968 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) (logid:277cdb3e) Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh -n r-4-VM -c  template=domP name=r-4-VM
---truncated-------
2024-02-02 10:38:25,970 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) (logid:277cdb3e) Executing while with timeout : 300000
STEPS TO REPRODUCE

1. Create a Linstor cluster with 3 nodes
2. Add a resource group with storagepool only from the node 1 i.e in combined type.
3. Add the 2 satellites as KVM hosts in the cloudstack
4. Deploy a vm with a new guest network with VR.

EXPECTED RESULTS

VR should be created and configured on Linstor.

ACTUAL RESULTS

VR fails to get configured on Linstor ( only when using the above mentioned design) .

@rp- can you have a look?

Update: this issue appears even after adding the storage provider node to the cloudstack as KVM host, the VR volume doesn't seem to be created properly?
root@ref-trl-5315-k-Mu22-jithin-raju-kvm1:~# linstor volume list -r cs-2cebae04-4f67-4b98-99c4-7338be759d71
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node                                 ┊ Resource                                ┊ StoragePool ┊ VolNr ┊ MinorNr ┊ DeviceName    ┊ Allocated ┊ InUse ┊    State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ ref-trl-5315-k-Mu22-jithin-raju-kvm1 ┊ cs-2cebae04-4f67-4b98-99c4-7338be759d71 ┊ cs-pool     ┊     0 ┊    1004 ┊ /dev/drbd1004 ┊     1 MiB ┊ InUse ┊ UpToDate ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

The template cached in primary storage seems to be usable in all the hosts?

2024-02-05 06:18:30,863 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) Executing: qemu-img convert -O raw -U --image-opts driver=qcow2,file.filename=/mnt/20420ecb-1fed-3417-a6e8-43314da1d9f0/7b1d0679-89c3-4c80-8dd0-14045e2faada.qcow2 /dev/drbd1000
2024-02-05 06:18:30,863 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) Executing while with timeout : 10800000
2024-02-05 06:18:34,632 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) Execution is successful.
2024-02-05 06:18:34,633 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) Executing: qemu-img resize /dev/drbd1000 5242880000
2024-02-05 06:18:34,633 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) Executing while with timeout : 3600000
2024-02-05 06:18:34,636 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) Execution is successful.
2024-02-05 06:18:34,636 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) (logid:2ce412cf) WARNING: Image format was not specified for '/dev/drbd1000' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
Image resized.

root@ref-trl-5315-k-Mu22-jithin-raju-kvm1:~# fdisk -l /dev/drbd1000
Disk /dev/drbd1000: 4.89 GiB, 5245915136 bytes, 10245928 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes

root@ref-trl-5315-k-Mu22-jithin-raju-kvm3:~# fdisk -l /dev/drbd1000
Disk /dev/drbd1000: 4.88 GiB, 5242880000 bytes, 10240000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x082ce66c

Device          Boot   Start      End Sectors  Size Id Type
/dev/drbd1000p1 *       2048   780287  778240  380M 83 Linux
/dev/drbd1000p2       782334 10237951 9455618  4.5G  5 Extended
/dev/drbd1000p5       782336  1279999  497664  243M 82 Linux swap / Solaris
/dev/drbd1000p6      1282048 10237951 8955904  4.3G 83 Linux

This should be fixed by:
4a86a0d2339bbd809d87e7179f510e018030cbbc and
67cb9b9e40b33a165d1846eb80dea21a7d623a1b
Not yet in any release version

That would be #8082 and #8114

That would be #8082 and #8114

that would close this

authors: DaanHoogland rajujith rp-


text:
Veeam integration

ISSUE TYPE

Bug Report

COMPONENT NAME

Veeam integration plugin

CLOUDSTACK VERSION

4.19.1

CONFIGURATION

Master node on Ubuntu 22, advanced networking, veeam version 12
OS / ENVIRONMENT

SUMMARY

After configuring the Veeam plugin, Veeam B&R to support the plugin and VeeamEM, when I try to create a Backup Offering, I receive the following error only in the zone where I have a vCenter configured and properly registered with Veeam.
"
Request failed. (530)
Veeam B&R API call unauthorized, please ask your administrator to fix integration issues.
"
STEPS TO REPRODUCE

Register the Veeam plugin in CS

EXPECTED RESULTS

I should be able to select the backup job with tag associated with my vCenter.

ACTUAL RESULTS

We receive error:

"
Request failed. (530)
Veeam B&R API call unauthorized, please ask your administrator to fix integration issues.
"

and we can observe the following log in the master node:

"
2024-07-19 13:51:31,354 DEBUG [o.a.c.b.BackupManagerImpl] (qtp364604394-15:ctx-88652954 ctx-70dd5d41) (logid:db53cbb4) Listing external backup offerings for the backup provider configured for zone ID 2
2024-07-19 13:51:31,392 INFO  [c.c.a.ApiServer] (qtp364604394-15:ctx-88652954 ctx-70dd5d41) (logid:db53cbb4) Veeam B&R API call unauthorized, please ask your administrator to fix integration issues.
2024-07-19 13:51:31,393 DEBUG [c.c.a.ApiServlet] (qtp364604394-15:ctx-88652954 ctx-70dd5d41) (logid:db53cbb4) ===END===  172.16.2.79 -- GET  zoneid=14a81d5c-8fda-4de6-a4f3-1932c562e559&command=listBackupProviderOfferings&response=json
"

If I try to import the backup offering in a different zone I can see all offerings.
Here attached the configuration CS side
![image](https://github.com/user-attachments/assets/f1e30e55-d424-49f4-a833-4f191c70347a)

Hi @DaanHoogland ,
I thought the integration was ok with CS version 4.19.1 after this issue https://github.com/apache/cloudstack/issues/8461
Am I getting something wrong
Ty!

yes @cdfgallo , it should be. I have no veeam though so I can not check.

I'm using Shapeblue packages. Idk if it's related to that

I don't think there is a difference between shapeblue packages and upstream for this release, so that is not an issue.

@cdfgallo
I think this issue has been fixed in 4.19.1.0 by #8241
can you share

the cloudstack version
the veeam version
the global and zone settings for veeam ?

Hi @weizhouapache ,
I'm indeed using version 4.19.1.0 and Veeam B&R version 12 (do you need the minor also?).
Here are the settings for global and the specific zone which is not working (I didn't make any modification in the zone settings from the defaults)

Ty!

I suspect this is a configuration issue.
I cannot tell what is wrong, maybe wrong username or password or api url ?
@omurozlu  has reported another veeam issue in #9440, but it looks like the backup offering has been successfully imported.

@cdfgallo did you install the veeam Enterprise manager ?
https://docs.cloudstack.apache.org/en/4.19.1.0/adminguide/veeam_plugin.html#installing-veeam-backup-and-recovery-for-use-with-cloudstack
https://helpcenter.veeam.com/docs/backup/em/introduction.html?ver=120

@weizhouapache indeed it was. Looks like that if you specify the password in the global settings and also in the zone, the two values are added together (it was too strange that I had no problem importing backup in the other two zones..).
In the Veeam EM I also have to install Veeam Console in order to have the Veeam.Backup.PowerShell module, correct?

@kiranchavala yes, we have a Veeam B&R server and a Veeam EM server.

Hi @weizhouapache
I am also encountering a similar issue with CloudStack version 4.19.0.1

@steven3579
Could you share the exception message  or management server log when you try to create a backup offering

Hi @kiranchavala
the following log in the master node
2024-08-05 02:37:09,172 DEBUG [c.c.a.t.Request] (RedundantRouterStatusMonitor-3:ctx-f566fac2) (logid:9808fbc4) Seq 1-2170172070439207155: Received:  { Ans: , MgmtId: 204998180945383, via: 1(esxi01.defensive.cloud), Ver: v1, Flags: 10, { CheckRouterAnswer } }
2024-08-05 02:37:09,172 DEBUG [c.c.a.m.AgentManagerImpl] (RedundantRouterStatusMonitor-3:ctx-f566fac2) (logid:9808fbc4) Details from executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP

2024-08-05 02:37:09,358 INFO  [c.c.a.ApiServer] (qtp239372207-181794:ctx-1229870c ctx-377e6018) (logid:afaed4b5) Veeam B&R API call unauthorized, please ask your administrator to fix integration issues.
2024-08-05 02:37:09,359 DEBUG [c.c.a.ApiServlet] (qtp239372207-181794:ctx-1229870c ctx-377e6018) (logid:afaed4b5) ===END===  x.x.x.x -- GET  zoneid=780c89df-02ad-447e-9ce9-874fd8295ddf&command=listBackupProviderOfferings&response=json
2024-08-05 02:37:09,638 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Ping from 4(v-226-VM)
2024-08-05 02:37:09,795 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-3:ctx-a638e145) (logid:736a6ab7) HA health check task is running...

@steven3579
can you check by the following comands ?
curl -k https://xxx:9398/api/
ssh username@xxx hostname

https://xxx:9398/api/ : value of  backup.plugin.veeam.url
username:  value of backup.plugin.veeam.username

Hi @weizhouapache
I have tried running the two commands above, and here are the results. Thank you
curl -k https://xxx:9398/api/
<?xml version="1.0" encoding="utf-8"?><EnterpriseManager xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.veeam.com/ent/v1.0"><Links><Link Href="https://xxx:9398/api/logonSessions" Type="LogonSessionList" Rel="Down" /><Link Href="https://xxx:9398/api/sessionMngr/?v=latest" Type="LogonSession" Rel="Create" /></Links><SupportedVersions><SupportedVersion Name="v1_5"><Links><Link Href="https://xxx:9398/api/sessionMngr/?v=v1_5" Type="LogonSession" Rel="Create" /></Links></SupportedVersion><SupportedVersion Name="v1_6"><Links><Link Href="https://xxx:9398/api/sessionMngr/?v=v1_6" Type="LogonSession" Rel="Create" /></Links></SupportedVersion><SupportedVersion Name="v1_7"><Links><Link Href="https://xxx:9398/api/sessionMngr/?v=v1_7" Type="LogonSession" Rel="Create" /></Links></SupportedVersion></SupportedVersions></EnterpriseManager>

ssh username@xxx
Microsoft Windows [Version 10.0.20348.1129]
(c) Microsoft Corporation. All rights reserved.

administrator@xxx  C:\Users\Administrator>

Hi @weizhouapache
I have upgraded to CloudStack version 4.19.1.1 and successfully connected. I have an issue. If my backed-up VM is deleted in CloudStack, how can I restore it from the backup on Veeam?
And i have tried deleting a VM in CloudStack, but I cannot proceed with the restoration.
Thanks Team!

if users have issue with veeam , it might be caused by #9891
if so, please try options
(1) option 1: update the data_center_details table with password in plaintext
(2) option 2: reset zone settings, and set veeam password in global setting

authors: DaanHoogland cdfgallo kiranchavala steven3579 weizhouapache


text:
CLOUDSTACK-10242: Properly parse incoming rules to Sec Group

With merge of PR #2028 the separator for lines to the Security Group
Python script changed from : to ; to support IPv6 addresses.
This broke certain situations where rules were parsed improperly.
This commit fixes that
Signed-off-by: Wido den Hollander wido@widodh.nl

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2612

authors: blueorangutan wido


text:
Avoid inefficient usage of arraylist

Hi,
We find that there are three ArrayList objects which are not manipulated by random access. Due to the memory reallocation triggered in the successive insertions, the time complexity of add method of ArrayList is amortized O(1). We notice that these objects are only used for traversal and the retrieval for the first or the last element.
This functionality can be implemented by LinkedList. Moreover, the insertion of LinkedList is strictly O(1) time complexity because no memory reallocation occurs.
We discovered this inefficient usage of containers by our tool Ditto. The patch is submitted. Could you please check and accept it? We have tested the patch on our PC. The patched program works well.
Bests
Ditto

Besides, we also found several TreeMap objects, which are only used for key-value pair retrieval. It would be better to replace it with HashMap.

Coverage remained the same at 94.682% when pulling c9d521b20256f15c4fc68beb57757f976a2a3110 on DittoTool:master into fd44e6b491c9d606d0b91ac0029269d265024cd9 on apache:master.

Are there any sources, that you can quote with regards to the complexity of ArrayList vs. LinkedList? Thanks, Jochen

Besides: What is the tool Ditto, that you ae referring to?

I do not think we should change internals until we have an example that demonstrates whatever problem this tool thinks it found. This feels backward to me because accessing the first and/last element of an array is fast and an array list grows in chunks.

Tree maps are also handy for reproducing the same result over and over, also needs a demonstration of what the problem is in pragmatic terms.

I do not think we should change internals until we have an example that demonstrates whatever problem this tool thinks it found. This feels backward to me because accessing the first and/last element of an array is fast and an array list grows in chunks.

I think exactly the same, which is why was asking for more infos.

Closing, no reply from OP.

authors: DittoTool FastAtlas coveralls garydgregory jochenw

repo: apache/commons-codec


text:
IO-458: add SequenceReader

https://issues.apache.org/jira/browse/IO-458

Coverage increased (+0.02%) to 89.508% when pulling de6b684aac7563b290b130864a002f101c89173c on RobberPhex:IO-458-SequenceReader into e46146c67026956d3f48d44be5c4df0fb78d5a42 on apache:master.

authors: RobberPhex coveralls

repo: apache/commons-io


text:
Merge pull request #1 from apache/master

Pull recent changes - May 15, 2016

Pull recent changes - June 3, 2016

Pull recent changes - June 3, 2016

Coverage increased (+0.01%) to 93.419% when pulling a7298622e112280fe8ef46e5844155eb474d3a98 on Force66:master into 03a2ab275b94f19003de12b0009639bf8f831a13 on apache:master.

authors: Derek-Ashmore coveralls

repo: apache/commons-lang


text:
CB-11661: (ios) Add mandatory iOS 10 privacy description

Platforms affected
iOS
What does this PR do?
Adds the mandatory NSCameraUsageDescription key to the info.plist, allowing to pass the value as a variable.
Documented the usage of the variable on plugin install as iOS quirk
What testing has been done on this change?
Checklist

[x] ICLA has been signed and submitted to secretary@apache.org.
[x] Reported an issue in the JIRA database
[x] Commit message follows the format: "CB-3232: (android) Fix bug with resolving file paths", where CB-xxxx is the JIRA ID & "android" is the platform affected.
[ ] Added automated test coverage as appropriate for this change.

Is there a way to localise such texts in plist?

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

Merged

This works great, but it seems you also need to add the plist item NSPhotoLibraryUsageDescription for accessing the photo library...

@possan - I will try and a PR for this soon because it's currently crashing my app at the moment, if I get it to work that is :)

authors: cordova-qa jacquesdev mhartington pke possan stevengill

repo: apache/cordova-plugin-camera


text:
CB-11541 iOS: Add extra labels for phone, ims

Platforms affected
iOS
What does this PR do?
Refactor converting methods. Add posibility to use default labels: 'home fax', 'work fax'
What testing has been done on this change?
Auto test
Checklist

[x] ICLA has been signed and submitted to secretary@apache.org.
[x] Reported an issue in the JIRA database
[x] Commit message follows the format: "CB-3232: (android) Fix bug with resolving file paths", where CB-xxxx is the JIRA ID & "android" is the platform affected.
[x] Added automated test coverage as appropriate for this change.

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

LGTM 👍
Please squash the commits before I merge it

@alsorokin, done.

Cordova CI Build has one or more failures.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android
Link
Link
Link

authors: alsorokin cordova-qa matrosov-nikita

repo: apache/cordova-plugin-contacts


text:
Fix crypto deprecations

COUCHDB-2825

this isn't a 'fix' :)

What it have to be then? I'm confused.

You've just suppressed the warning, when the function is removed, couch_epi will break, which is not the case elsewhere.

ooh, never mind. Ilya took care of this already. fine...
+1

authors: kxepal rnewson

repo: apache/couchdb-couch-epi


text:
index with $or operator not working

Description
I am trying to create an index for a query with "$or" operator but the query plan does not recognize the index.
my question is, how to create a working index for the following query ?

{
   "selector": {
      "type": "project",
      "$or": [
         {
            "pmId": "8b1cee44bf16ccced9163adee20008a8"
         },
         {
            "sponsor._id": "8b1cee44bf16ccced9163adee20008a8"
         },
         {
            "AccessMembers": {
               "$elemMatch": {
                  "$and": [
                     {
                        "id": "8b1cee44bf16ccced9163adee20008a8"
                     },
                     {
                        "ACL": {
                           "$in": [
                              "View",
                              "Edit"
                           ]
                        }
                     }
                  ]
               }
            }
         }
      ]
   }
}

Steps to Reproduce
Expected Behaviour
Your Environment

CouchDB Version used:
v. 2.1.1
Browser name and version:
Operating System and version:
Windows Server 2016 Datacenter

Additional context

What is your Mango index definition?
What does the `/{db}/_explain' endpoint say when you use it?
I would think in this situation you'd need every single field you're referencing in your index: pmId, sponsor._id, and ACL.

thank you @wohali for the reply.
the index is as following:

{
	"index": {
		"fields": [
			"type",
			"pmId",
			"sponsor._id",
			"AccessMembers.[].id",
			"AccessMembers.[].ACL"
		]
	},
	"name": "pm-projects-index",
	"type": "json"
}

the "explain" result as following:

{
 "dbname": "pm",
 "index": {
  "ddoc": null,
  "name": "_all_docs",
  "type": "special",
  "def": {
   "fields": [
    {
     "_id": "asc"
    }
   ]
  }
 },
 "selector": {
  "$and": [
   {
    "type": {
     "$eq": "project"
    }
   },
   {
    "$or": [
     {
      "pmId": {
       "$eq": "8b1cee44bf16ccced9163adee20008a8"
      }
     },
     {
      "sponsor._id": {
       "$eq": "8b1cee44bf16ccced9163adee20008a8"
      }
     },
     {
      "AccessMembers": {
       "$elemMatch": {
        "$and": [
         {
          "id": {
           "$eq": "8b1cee44bf16ccced9163adee20008a8"
          }
         },
         {
          "ACL": {
           "$in": [
            "View",
            "Edit"
           ]
          }
         }
        ]
       }
      }
     }
    ]
   }
  ]
 },
 "opts": {
  "use_index": [],
  "bookmark": "nil",
  "limit": 25,
  "skip": 0,
  "sort": {},
  "fields": "all_fields",
  "r": [
   49
  ],
  "conflicts": false,
  "stale": false,
  "update": true,
  "stable": false,
  "execution_stats": false
 },
 "limit": 25,
 "skip": 0,
 "fields": "all_fields",
 "mrargs": {
  "include_docs": true,
  "view_type": "map",
  "reduce": false,
  "start_key": null,
  "end_key": "",
  "direction": "fwd",
  "stable": false,
  "update": true,
  "conflicts": "undefined"
 }
}

The simplest is just creating an index on type. The add in the execution stats to see how many docs are ignored or not.

From: brifai notifications@github.com
Sent: Saturday, June 29, 2019 12:19 PM
To: apache/couchdb
Cc: Subscribed
Subject: Re: [apache/couchdb] index with $or operator not working (#2063)
thank you @wohalihttps://github.com/wohali for the reply.
the index is as following:
{
"index": {
"fields": [
"type",
"pmId",
"sponsor._id",
"AccessMembers.[].id",
"AccessMembers.[].ACL"
]
},
"name": "pm-projects-index",
"type": "json"
}
the "explain" result as following:
{
"dbname": "pm",
"index": {
"ddoc": null,
"name": "_all_docs",
"type": "special",
"def": {
"fields": [
{
"_id": "asc"
}
]
}
},
"selector": {
"$and": [
{
"type": {
"$eq": "project"
}
},
{
"$or": [
{
"pmId": {
"$eq": "8b1cee44bf16ccced9163adee20008a8"
}
},
{
"sponsor._id": {
"$eq": "8b1cee44bf16ccced9163adee20008a8"
}
},
{
"AccessMembers": {
"$elemMatch": {
"$and": [
{
"id": {
"$eq": "8b1cee44bf16ccced9163adee20008a8"
}
},
{
"ACL": {
"$in": [
"View",
"Edit"
]
}
}
]
}
}
}
]
}
]
},
"opts": {
"use_index": [],
"bookmark": "nil",
"limit": 25,
"skip": 0,
"sort": {},
"fields": "all_fields",
"r": [
49
],
"conflicts": false,
"stale": false,
"update": true,
"stable": false,
"execution_stats": false
},
"limit": 25,
"skip": 0,
"fields": "all_fields",
"mrargs": {
"include_docs": true,
"view_type": "map",
"reduce": false,
"start_key": null,
"end_key": "",
"direction": "fwd",
"stable": false,
"update": true,
"conflicts": "undefined"
}
}
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/apache/couchdb/issues/2063?email_source=notifications&email_token=AABL2AUWVGULUJD6MCB3U3DP44ZKPA5CNFSM4H3Z4I5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3WGBY#issuecomment-506946311, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AABL2ASF4FIGXFGCE3SVY6DP44ZKPANCNFSM4H3Z4I5A.

Thank you @garrensmith for your input, but can this query be indexed ?

If you use an $or operator then all the fields in the index need to be on both sides of the or otherwise it won’t work. That’s why I think keep the index fields to a minimum or use a partial_filter_selector when building the index.

From: brifai notifications@github.com
Sent: Saturday, June 29, 2019 2:44 PM
To: apache/couchdb
Cc: garren smith; Mention
Subject: Re: [apache/couchdb] index with $or operator not working (#2063)
Thank you @garrensmithhttps://github.com/garrensmith for your input, but can this query be indexed ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/apache/couchdb/issues/2063?email_source=notifications&email_token=AABL2ATFHY6I7F7SHCGPQTTP45KKNA5CNFSM4H3Z4I5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3YENY#issuecomment-506954295, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AABL2AWY4INWQQWLLX7XTBDP45KKNANCNFSM4H3Z4I5A.

Thank you @garrensmith , @wohali do you have any input about it ?

no, garren is the expert here

authors: brifai garrensmith wohali

repo: apache/couchdb


text:
Enable tail call optimization in couch_replicator_docs

update_docs/4 function was recursively called from catch part of a
try ... catch block. That construct prevented tail call optimization to
take place.
Jira: COUCHDB-3324

@davisp pointed out that current code already has TCO and this is not needed
Used this script from him (with a modification to just print the memory of the process) to verify:
-module(foo).
-compile(export_all).
a() ->
    a(0).
a(I) ->
    if (I+1) rem 1000 /= 0 -> ok; true ->
        io:format("~p~n", [process_info(self(), memory)])
    end,
    try
        throw(bar)
    catch throw:bar ->
        c(a, I+1)
    end.
b() ->
    b(0).
b(I) ->
    if (I+1) rem 1000 /= 0 -> ok; true ->
        io:format("~p~n", [process_info(self(), memory)])
    end,
    try
        c(b, I+1)
    catch throw:bar ->
        foo
    end.
c(F, I) ->
    ?MODULE:F(I).

foo:b() ends up producing
...
{memory,102684144}
{memory,102684144}
{memory,102684144}
{memory,102684144}

foo:a() is stable
...
{memory,16672}
{memory,24576}
{memory,11784}
{memory,16672}

Thank you Paul and Nick for pointing out that I was wrong. The code would work as is. @nickva could you close the PR?

@iilyak ah no worries, I assumed it needed TCO as well, so I think it was a useful exercise to understand how it works.

authors: iilyak nickva


text:
Update to Spring Boot 2.4.0 / Spring Framework 5.3.1 / Spring Security 5.4.1 / Reactor 3.4.0 / Micrometer 1.6.1

Update to Spring Boot 2.4.0
Update to Spring Framework 5.3.1
Update to Spring Security 5.4.1
Update to Project Reactor 3.4.0
Update to Micrometer 1.6.1

@deki @coheigea @amarkevich the update looks good now, one thing I intended to incorporate is switch to Maven Surefire 3.0.0-M5 (so we could run JUnit 4 & JUnit 5 test cases side by side) but I run into an issue with tooling tests org.apache.cxf.tools.common.ProcessorTestBase::getClassPath() which seems to be relying on some Surefire intrinsic details, probably better to tackle as separate effort. If no objections, I would like to merge it into master.

repo: apache/cxf


text:
Implement native support StringView for REPEAT

Which issue does this PR close?
Closes #11914 and part of #11790.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

I took the liberty of merging up from main and running cargo fmt on this PR

repo: apache/datafusion


text:
When accessing the homepage on port 12345, a blank page is displayed

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
Deployment was successful, and there were no errors in the backend logs. When accessing the homepage, I can see requests to the page's JavaScript and other assets in the browser's console. However, the interface isn't displaying anything
What you expected to happen

I'm not sure what went wrong in this case
How to reproduce
The problem arises right after a successful deployment
Anything else
No response
Version
3.1.x
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

it must be centos7

which version do you use?

authors: xuweiyuaaaa zhongjiajie

repo: apache/dolphinscheduler


text:
[Feature] Show sub-process task info in workflow definition page

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

At present, I have not achieved the location well, I wonder if there is any good way.

I have a few questions.

do we recursively show all sub-process tasks (there may be multiple sub-process tasks)
how do we relate tasks under subtasks of the current task
How to determine location

I am not familiar with the frontend. At present, I reserve the showSubTask param for the frontend, and the frontend needs to add a button, which will trigger the process-definition interface of showSubTask=true when clicked and format the dag.

BTW. There are some unit-test failed. @stalary

BTW. There are some unit-test failed. @stalary

Ok, I'll check it

Codecov Report

Merging #10553 (787ed70) into dev (0aba68f) will increase coverage by 0.02%.
The diff coverage is 16.66%.

@@             Coverage Diff              @@
##                dev   #10553      +/-   ##
============================================
+ Coverage     40.87%   40.90%   +0.02%
- Complexity     4866     4877      +11
============================================
  Files           893      895       +2
  Lines         36118    36249     +131
  Branches       3992     3994       +2
============================================
+ Hits          14764    14826      +62
- Misses        19891    19955      +64
- Partials       1463     1468       +5

Impacted Files
Coverage Δ

...nscheduler/service/process/ProcessServiceImpl.java
29.91% <8.57%> (-0.67%)
:arrow_down:

...he/dolphinscheduler/dao/entity/TaskDefinition.java
70.76% <25.00%> (-0.89%)
:arrow_down:

...api/service/impl/ProcessDefinitionServiceImpl.java
31.06% <37.50%> (-0.08%)
:arrow_down:

...er/api/controller/ProcessDefinitionController.java
45.34% <100.00%> (ø)

...hinscheduler/service/alert/AlertClientService.java
60.00% <0.00%> (-6.67%)
:arrow_down:

...dolphinscheduler/remote/future/ResponseFuture.java
81.96% <0.00%> (-1.64%)
:arrow_down:

...erver/master/runner/WorkflowExecuteThreadPool.java
1.31% <0.00%> (-0.12%)
:arrow_down:

...org/apache/dolphinscheduler/alert/AlertServer.java
51.28% <0.00%> (-0.07%)
:arrow_down:

.../server/master/runner/WorkflowExecuteRunnable.java
7.69% <0.00%> (-0.04%)
:arrow_down:

.../server/master/runner/StateWheelExecuteThread.java
0.47% <0.00%> (-0.01%)
:arrow_down:

... and 34 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0aba68f...787ed70. Read the comment docs.

So far I have implemented frontend click to show sub-process task, but not automatic format, and the current implementation is not so good.

@caishunfeng @songjianet @Amy0104 I have finished the development of the feature, PTAL.

authors: SbloodyS codecov-commenter stalary


text:
[Fix-16279] Support Java 8 date/time type java.time.OffsetDateTime

Purpose of the pull request

fix #16279

Support Java 8 date/time type java.time.OffsetDateTime

refer  https://github.com/FasterXML/jackson-modules-java8/issues/219
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

When adding support for the java.time.OffsetDateTime type, it is better to add the entire JavaTimeModule or just the serializers and deserializers related to OffsetDateTime ?
@ruanwenjun

JavaTimeModule

Add JavaTimeModule is better.

authors: ruanwenjun xxzuo


text:
[Enhancement] (nereids)implement showProcessListCommand in nereids

showProcessList command is supported in legacy planner but not in nereids(new doris fe planner). you need implement the following showProcessList command in nereids:
unsupportedshowProcessListStatement
SHOW FULL? PROCESSLIST

@starocean999 @LiBinfeng-01 Please assign this issue to me

repo: apache/doris


text:
enhancement speed up decommission process (#13579)

Proposed changes
Issue Number: close #13579
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.58 seconds
load time: 446 seconds
storage size: 17179747628 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107060320_clickbench_pr_40853.html

authors: dutyu hello-stephen


text:
fix: fix filter.

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.75 seconds
load time: 492 seconds
storage size: 17123346213 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230207192642_clickbench_pr_91712.html

authors: hello-stephen jackwener


text:
fix change boost thirdparty url to offical archives.boost.io

Proposed changes
Issue Number: close #xxx
The old url https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.gz is no longer available.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[improvement](tablet report) be tablet size include index size

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

both index size and data size include invert index size,  so their sum will bigger than real disk usage


text:
improvement fix jdbc mysql catalog to_date fun pushdown

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


text:
Bug forbid create mv with value column before key column

Proposed changes
forbid create mv with value column before key column
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix Fix some regression case due to #33347

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
Fix Fix that tvf reading empty files in compressed formats.

Proposed changes
Issue Number: close #xxx
bp: #34926

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run feut

run buildall

authors: BePPPower morningman


text:
refineAdjust the value of STREAMING_HT_MIN_REDUCTION

Proposed changes
A size of 16MB is too large in many cases.

run buildall


text:
fix fix priority queue order in memtable memory limiter

Proposed changes
#41018 used priority queue when selecting memtables to flush.
But the cmp function is wrong and the order is reversed.
This PR fixes the cmp function to make larger memtables come first.

run buildall


text:
improvementSupport partition row count return -1 when it is not fully reported.

For a single partition row count, return -1 to planner when its index is not fully reported.

run buildall

run cloud_p0

run cloud_p0

run external

run cloud_p0

run cloud_p0

run buildall

run cloud_p0


text:
fix should not use selectHint in any place (#41260)

pick from master #41260

run buildall

run buildall


text:
Improvement support single column nullable join hash table context

Proposed changes
support single column nullable join hash table context

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


text:
[fix](ngram bloomfilter) fix narrow conversion for ngram bf_size

What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall


text:
DRILL-4387: GroupScan or ScanBatchCreator should not use star column …

…in case of skipAll query.
The skipAll query should be handled in RecordReader.

LGTM +1.

@jinfengni looks like this was merged, can you close the PR?

authors: amansinha100 jaltekruse jinfengni

repo: apache/drill


text:
Druid Docker config property override does not work when using custom Kubernetes config maps

Druid Docker config property override does not work when using custom Kubernetes config maps
Affected Version
0.18.1
Description
There are two related issues:

the documentation states the following: the script which launches Druid in the container will also attempt to use any environment variable starting with the druid_ prefix as a command-line configuration. (
https://druid.apache.org/docs/latest/tutorials/docker.html#configuration) However, the druid.sh script doesn't actually use the environment variable as a command-line configuration. Instead, it updates the actual property files (common and service-specific) directly. (https://github.com/apache/druid/blob/master/distribution/docker/druid.sh#L69-L80 and https://github.com/apache/druid/blob/master/distribution/docker/druid.sh#L104-L110) The documentation should be updated to reflect this.

The main issue is that while the script does update the configuration files when a druid_ prefixed environment variable is passed in, the script also takes in a few reserved environment variables, namely DRUID_CONFIG_COMMON. DRUID_CONFIG_<druid_service>. These allow for kubernetes config maps to be used in place of the default druid configurations properties files. The script replaces the default property files with the config map ones, but it does this after the environment variable overrides are made, rendering said environment variable overrides unusable.
https://github.com/apache/druid/blob/master/distribution/docker/druid.sh#L119-L132

Presumably, many who use config maps don't need an env variable override. Yet, there are valid cases when both a configmap and an environment variable override may be needed. One case is when setting a property that maps to a password/secret, such as druid.azure.key. Using an environment override would allow the property to not be hardcoded in the file.
Steps to reproduce the problem

Setup a kubernetes manifest specifying config maps for druid properties. For example, for historical:
containers:

name: druid
image: "druid"
args: [ "historical" ]
env:

name: DRUID_CONFIG_COMMON
value: "/tmp/my-conf/_common.runtime.properties"
name: DRUID_CONFIG_historical
value: "/tmp/my-conf/runtime.properties"
ports:
containerPort: 8083
name:
volumeMounts:

name: data
mountPath: /var/druid/
name: service-config-volume
mountPath: /tmp/my-conf
name: common-config-volume
mountPath: /tmp/my-conf
volumes:

name: service-config-volume
configMap:
name: historical
name: common-config-volume
configMap:
name: common

Set the following environment variables to point to the custom config map files:
name: DRUID_CONFIG_COMMON
name: DRUID_CONFIG_historical

Per the documentation, specify a druid property to override via an environment variable. (e.g. druid.azure.account)

Deploy the manifest, which launches the druid container

Actual Behavior:
Logs show that (e.g. druid.azure.account) was updated by the entrypoint script druid.sh, (e.g. Setting druid.azure.account=myaccount in /tmp/conf/druid/cluster/data/historical/runtime.properties)  but the property doesn't appear in the start up logs when printing out all of the set properties. Also, the properties files updated by the startup script and the default properties files both do not contain the overridden property.
Expected Behavior:
The overridden property appears both in the properties files as well as in the list of properties in the application logs after the druid process starts, thereby successfully overriding the desired druid configuration
Having the druid.sh startup script update replace the property files with the config map files before the environment variable overrides would seem to address this issue.

This is fixed in #11364 but its not merged yet.

#11364 was merged, this can be closed.

authors: josephglanville nate-mar

repo: apache/druid


text:
fix #413

修复 #413 的问题。修复当axios 请求参数size为-1时，PageRequest对象序列化的size成员变量为20 。

Codecov Report

Merging #426 into develop will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             develop     #426   +/-   ##
==========================================
  Coverage      22.83%   22.83%
  Complexity       291      291
==========================================
  Files             79       79
  Lines           3144     3144
  Branches         507      507
==========================================
  Hits             718      718
  Misses          2350     2350
  Partials          76       76

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 419c7d3...d0ba615. Read the comment docs.

authors: aworker codecov-io

repo: apache/dubbo-admin


text:
k8s集群外访问dubbo服务

k8s集群外访问dubbo服务怎么办，ip不通

可以试一试：https://github.com/monkeyWie/dubbo-ingress-controller

repo: apache/dubbo


text:
register MetadataService as internal service

related issue https://github.com/apache/dubbo/issues/10120

Codecov Report

Merging #10148 (2f246b7) into 3.0 (89863e0) will decrease coverage by 0.29%.
The diff coverage is 93.84%.

:exclamation: Current head 2f246b7 differs from pull request most recent head 147a29b. Consider uploading reports for the commit 147a29b to get more accurate results

@@             Coverage Diff              @@
##                3.0   #10148      +/-   ##
============================================
- Coverage     65.68%   65.39%   -0.30%
+ Complexity      320      319       -1
============================================
  Files          1224     1225       +1
  Lines         53413    53422       +9
  Branches       8077     8048      -29
============================================
- Hits          35086    34933     -153
- Misses        14508    14649     +141
- Partials       3819     3840      +21

Impacted Files
Coverage Δ

...apache/dubbo/common/constants/CommonConstants.java
100.00% <ø> (ø)

...che/dubbo/metadata/AbstractServiceNameMapping.java
83.62% <ø> (ø)

.../dubbo/registry/client/metadata/MetadataUtils.java
65.34% <84.00%> (+1.06%)
:arrow_up:

.../threadpool/manager/DefaultExecutorRepository.java
70.05% <100.00%> (+0.92%)
:arrow_up:

...n/java/org/apache/dubbo/rpc/model/ModuleModel.java
93.58% <100.00%> (+1.16%)
:arrow_up:

...pache/dubbo/rpc/model/ModuleServiceRepository.java
68.04% <100.00%> (ø)

.../metadata/ConfigurableMetadataServiceExporter.java
77.52% <100.00%> (+1.33%)
:arrow_up:

...apache/dubbo/metadata/MetadataServiceDetector.java
100.00% <100.00%> (ø)

...nt/metadata/StandardMetadataServiceURLBuilder.java
88.67% <100.00%> (+0.67%)
:arrow_up:

.../apache/dubbo/rpc/protocol/tri/TripleProtocol.java
85.36% <100.00%> (ø)

... and 77 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 89863e0...147a29b. Read the comment docs.

related https://github.com/apache/dubbo/pull/10110

authors: chickenlj codecov-commenter


text:
Revert 1 2.7.2 release

What is the purpose of the change
XXXXX
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.3 out of 6 committers have signed the CLA.:white_check_mark: chickenlj:white_check_mark: ralf0131:white_check_mark: beiwei30:x: wuwenhui:x: uglycow:x: cvictorywuwenhui seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

please provide more detail for your PR. I'm going to close this, feel free to reopen it when you think it's ready

authors: CLAassistant htynkn wuwenhui0917


text:
[Feature] Could "emphasis.focus = 'series'" support DIY other series' fade out style ?

What problem does this feature solve?
when emphasis.focus is 'series', other series fading out. If we can custom the style, such as 'opacity', we can complete more visual effect.
What does the proposed API look like?
such as the configuration:
options = {
    series: [
        ...
    ],
    emphasis: {
        focus: 'series',
        fadeOut: { // work when focus is 'series'
            lineStyle: {
                opacity: 0.5,
            }
            ...
        }
};

You can specify the blur style for each line series. Please refer to this example.

Thx, I hava come true my design by setting blur.

repo: apache/echarts


text:
[Feature] 能不能支持折线图根据数据的区间做不同的渐变颜色呢？

What problem does this feature solve?
能不能支持折线图根据数据的区间做不同的渐变颜色呢？
What does the proposed API look like?
能不能支持折线图根据数据的区间做不同的渐变颜色呢？

Can the line chart support different gradient colors based on the data range?

// areaStyle: {
//   color: function(params) {
//       // const value = params.value[1]; // 假设您正在处理的是折线图中的第二个数据点（索引为1）
//       const value = params[params.length-1]; // 假设您正在处理的是折线图中的第二个数据点（索引为1）
//       if (value > 26) {
//           // 对于大于26的值，使用从红色到深红色的渐变
//           return new echarts.graphic.LinearGradient(0, 0, 0, 1, [
//               {offset: 0, color: 'rgb(255, 0, 0)'}, // 红色
//               {offset: 1, color: 'rgb(128, 0, 0)'}  // 深红色
//           ]);
//       } else if (value >= 22) {
//           // 对于大于等于22且小于等于26的值，使用从绿色到深绿色的渐变
//           return new echarts.graphic.LinearGradient(0, 0, 0, 1, [
//               {offset: 0, color: 'rgb(0, 255, 0)'}, // 绿色
//               {offset: 1, color: 'rgb(0, 128, 0)'}  // 深绿色
//           ]);
//       } else {
//           // 对于小于22的值，使用从蓝色到深蓝色的渐变
//           return new echarts.graphic.LinearGradient(0, 0, 0, 1, [
//               {offset: 0, color: 'rgb(0, 0, 255)'}, // 蓝色
//               {offset: 1, color: 'rgb(0, 0, 128)'}  // 深蓝色
//           ]);
//       }
//   }
// }为什么不生效呢？