text:
[Feature Proposal] Snapshot Interoperability with Remote Store

Problem Description:
Today, Snapshot Feature in OpenSearch allows user to create an on-demand snapshot of the entire cluster data and metadata in a configured repository. Snapshot mechanism is used for recovering from red indices or even for cluster level recoveries in the event of failures.
Now, with Remote Store Feature, we are giving user an option to create indices which will store the translog data in remote translog store for each request and segment to remote segment store at refresh intervals. This means that the data is already present in remote store repository for such indices.
As the data for remote store backed indices is already stored in a repository, creating a snapshot of these indices will duplicate the segment data. For cost reasons, users may not want to store data in multiple repositories: a) One as part of snapshot and b) Other as part of remote store backed indices. For these use cases it would be beneficial to still provide the out of box snapshot experience but only keep 1 copy of the data. There can be other use cases as well where copying data is required as part of snapshot so we should also provide flexibility to users to choose between both the options.
Proposed Solution:
The Idea is to keep the reference to remote store metadata file in snapshot shard metadata.
So, remote store feature uses a metadata file to store the name of segment files which are live during refresh. On high level, we want to keep those metadata files for remote store backed indices in snapshot metadata and during snapshot restore operation, we will call remote store feature to restore segment files using those metadata files.
Some Key Points:

We still need to support renaming of indices during restore operation to restore an existing index with a different name.
Users can have different repositories for snapshot and remote store.
Restoring a remote store backed index from snapshot as a searchable snapshot index will not be supported in phase 1. However, for cases where complete data copy is taken in snapshot repository it should still work out of the box. For other case, we will provide the support in phase 2.
As we are just referring the data stored in remote segment store, Snapshot Status api will not have any incremental file size and file count details for remote store backed indices.

Next Steps:

Any general comments about the feature are helpful. Also let us know if this feature would help any of your current use-case.
We will share design and POCs for the proposed solution going forward.
We will create and work on different issues needed to support this feature.

Do we need to consider both option, label in place and/or create a copy?

Thanks @harishbhakuni21, this looks like a great enhancement to remote-backed storage and snapshots!
Other than the callout about searchable snapshots, are there any other limitations for the snapshot restore API? Some of the potential cases I'm thinking about:

restoring a remote-backed index snapshot to a non-remote-backed index
restoring a remote-backed index snapshot to a different remote store repository
restoring a traditional snapshot to a remote-backed index

Since the snapshot restore API allows for changing index settings, these cases (and likely many more) seem possible via the API and it would be great to keep the functionality as transparent as possible so the user doesn't need to be concerned with where the data is being stored under the hood.

Thanks @harishbhakuni21 for your suggested enhancements.

For these use cases it would be beneficial to still provide the out of box snapshot experience but only keep 1 copy of the data. There can be other use cases as well where copying data is required as part of snapshot so we should also provide flexibility to users to choose between both the options.

Since remote backend index is considered as a continuous snapshot and it  will gives the user a point on time restore capability, then why we need to have the on-demand snapshot capability regardless if you copy the data or not. Could you provide some of the use-case that still could benefit from the on-demand snapshot?
In my opinion, I would improve our snapshot restore user experience, to give the user the flexibility to restore a point on time remote backed index easily and deprecate on-demand snapshot or at least disable it if the index has the remote backed index feature enabled

Since the snapshot restore API allows for changing index settings, these cases (and likely many more) seem possible via the API and it would be great to keep the functionality as transparent as possible so the user doesn't need to be concerned with where the data is being stored under the hood.

Good Point, Agree with the functionality transparency. logically all the above usecases should be possible, but we will be performing some POCs to verify if all these cases will be supported if snapshot interop feature is enabled for a snapshot.

Since remote backend index is considered as a continuous snapshot and it will gives the user a point on time restore capability, then why we need to have the on-demand snapshot capability regardless if you copy the data or not. Could you provide some of the use-case that still could benefit from the on-demand snapshot?

@anasalkouz remote store feature today just stores the live segments of indices so it do not support point in time restore. to restore indices back to the cluster, customer still need to use snapshots.

@harishbhakuni21 Thats why I am suggested to improve our snapshot restore user experience, to give the user the flexibility to restore a point on time remote backed index easily and deprecate on-demand snapshot or at least disable it if the index has the remote backed index feature enabled.

@anasalkouz even for remote backed storage, we cannot support Point In Time Restore as mentioned above we only store live segments in remote store right now. supporting PITR is totally a different discussion. as part of this doc, we are proposing if customer do not want data duplication for remote store backed indices, how can we use the segment data in remote store and provide the same snapshot experience as today.

supporting PITR is totally a different discussion

@harishbhakuni21 It is a different discussion, but the question is whether on-demand snapshots will have any use case once PITR is supported. If the answer is "no", then this proposal is talking about introducing functionality that will presumably be deprecated once PITR is implemented, and the obvious follow up to that is whether PITR should be prioritized in lieu of implementing on-demand snapshots.
I honestly don't know the answer. There's certainly an argument for meeting users where they are today, and users do have systems and workflows built around the existing snapshot functionality. That being said, a fully implemented PITR feature does seem like it would obviate the need for on-demand snapshots.

For a snapshot to be consumable by users, it needs to copy the data to repository and also have a reference marker to that data (which is snapshot metadata). With remote store, the continuous upload of index data is being done however there is no reference to that data copy which can be used to restore. The current snapshot mechanism can be utilized to create that reference (metadata) to the already uploaded index data (with other existing functionalities like snapshot of entire cluster state and snapshot of one or multiple indices, etc). One can also use the existing snapshot mechanism as well to provide PITR sort of capability (not saying this is the only way). e.g. Using the snapshot management plugin or some custom workflows one can continue to use existing snapshot functionality over remote store backed index to create these markers into their snapshot repository (with desired frequency). Snapshot on remote store will mostly be metadata only operation which this RFC is talking about. Each of these snapshots thus generated can be referenced as point in time reference to restore an index.
The RFC also talks about supporting option to copy the data from one repository to other in snapshot format. One of the use case I can think of for this is as a service provider I may want to keep the data repository not exposed or directly accessible to the users. But the users still need to take copies of the data into their own repository for compliance purposes or use cases like powering different OS cluster with same data without incurring the cost of indexing again. For such use cases using snapshot mechanism to actually make copies of data across repositories will be useful. Since users already have systems using the existing snapshot mechanism it will work out of the box for them.

Copying data from one repository to another definitely makes sense, and the existing snapshot API will work well for that use case. Perhaps similarly, I also would expect any PITR feature to have a retention policy and not keep all data indefinitely, but there will probably be use cases for manually "pinning" a particular point in time snapshot even within the same repository to keep indefinitely, and the existing snapshot API should work for that as well.
@anasalkouz What do you think?

authors: anasalkouz andrross harishbhakuni21 nandi-github sohami


text:
[Concurrent Segment Search] SourceLookup is not thread safe across segment slices

This is a subtask for https://github.com/opensearch-project/OpenSearch/issues/8509#issuecomment-1636306061
Reproduction:
./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.aggregations.bucket.SignificantTermsSignificanceScoreIT.testScoresEqualForPositiveAndNegative" -Dtests.seed=6D7B107FC7821E62 -Dtests.locale=mk -Dtests.timezone=America/Atka

This is a race condition though so the same seed does not necessarily guarantee replication
The above IT sometimes fails with an assert codec error:
cause = {AssertionError@15455} "java.lang.AssertionError: StoredFieldsReader are only supposed to be consumed in the thread in which they have been acquired. But was acquired in Thread[opensearch[node_s0][index_searcher][T#2],5,TGRP-SignificantTermsSignificanceScoreIT] and consumed in Thread[opensearch[node_s0][index_searcher][T#1],5,TGRP-SignificantTermsSignificanceScoreIT]."
stackTrace = {StackTraceElement[29]@15492}
 0 = {StackTraceElement@15494} "org.apache.lucene.tests.codecs.asserting.AssertingCodec.assertThread(AssertingCodec.java:44)"
 1 = {StackTraceElement@15495} "org.apache.lucene.tests.codecs.asserting.AssertingStoredFieldsFormat$AssertingStoredFieldsReader.document(AssertingStoredFieldsFormat.java:74)"
 2 = {StackTraceElement@15496} "org.opensearch.search.lookup.SourceLookup.loadSourceIfNeeded(SourceLookup.java:102)"
 3 = {StackTraceElement@15497} "org.opensearch.search.lookup.SourceLookup.extractRawValues(SourceLookup.java:178)"
 4 = {StackTraceElement@15498} "org.opensearch.search.aggregations.bucket.terms.SignificantTextAggregatorFactory$SignificantTextCollectorSource$1.collectFromSource(SignificantTextAggregatorFactory.java:240)"
 5 = {StackTraceElement@15499} "org.opensearch.search.aggregations.bucket.terms.SignificantTextAggregatorFactory$SignificantTextCollectorSource$1.collect(SignificantTextAggregatorFactory.java:221)"
 6 = {StackTraceElement@15500} "org.opensearch.search.aggregations.LeafBucketCollectorBase.collect(LeafBucketCollectorBase.java:75)"
 7 = {StackTraceElement@15501} "org.opensearch.search.aggregations.bucket.terms.MapStringTermsAggregator$SignificantTermsResults$1.collect(MapStringTermsAggregator.java:508)"

This is because SourceLookup is not thread safe and it's fields are being updated by multiple threads at the same time. This is happening here:
https://github.com/opensearch-project/OpenSearch/blob/8eea7b986453271cd227a7b98ee09e70d5e28634/server/src/main/java/org/opensearch/search/aggregations/bucket/terms/SignificantTextAggregatorFactory.java#L234-L248
There are a few possible ways to make this thread safe, currently exploring the following:

Create a new SourceLookup object per slice
Make the fields of SourceLookup ThreadLocal

sourceLookup is final in SearchLookup. On the Aggregator side SignificantTextAggregatorFactory is the only place where SearchLookup::source is used, so in the aggregator side it doesn’t look like we need to access SourceLookup through SearchLookup again.
PR #8807 contains an easy fix for this wrt SignificantText, however it looks like it's possible there is a similar problem for SearchLookup in the other scripting use cases. Since this focus of this issue and parent issue is to resolve failing SignificantTerms tests, I will open a separate issue to track investigation into LeafSearchLookup use cases.

source: gharchive/issue

repo: opensearch-project/OpenSearch

license_type: permissive

license_source: github-api


text:
Start MockLogAppender before adding to static context

I observed a test failure with the message
'Attempted to append to non-started appender mock' from an assertion in
OpenSearchTestCase::after. I believe this indicates that a
MockLogAppender (which is named "mock") was added as an appender to the
static logging context and some other test in the same JVM happened to
cause a logging statement to hit that appender and cause an error, which
then caused an unrelated test to fail (because they share static state
with the logger). Almost all usages of MockLogAppender start it
immediately after creation. I found a few that did not and fixed those.
I also made a static helper in MockLogAppender to start it upon
creation.
Signed-off-by: Andrew Ross andrross@amazon.com
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

start gradle check

Let's backport to 1.x

Does this still need to be backported?

Backport PR here: https://github.com/opensearch-project/OpenSearch/pull/1659

source: gharchive/pull-request

authors: andrross dblock


text:
Update README.md

Description
Update setting names and rest api endpoints in README

Nit: let's try to write better commit messages? Having Update setting names and rest api endpoints in README would go a long way in the commit itself!

Do we want to fix the broken code cov link too?

authors: Bukhtawar dblock eirsep

repo: opensearch-project/asynchronous-search


text:
Backport bug fixes

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #200 (3e35304) into aos-1.0.0.0-ga (076e353) will increase coverage by 14.38%.
The diff coverage is 91.66%.

@@                  Coverage Diff                  @@
##             aos-1.0.0.0-ga     #200       +/-   ##
=====================================================
+ Coverage             63.39%   77.77%   +14.38%
=====================================================
  Files                   101       33       -68
  Lines                  4398     1926     -2472
  Branches                669      381      -288
=====================================================
- Hits                   2788     1498     -1290
+ Misses                 1442      425     -1017
+ Partials                168        3      -165

Flag
Coverage Δ

dashboards-reports
77.77% <91.66%> (+1.80%)
:arrow_up:

reports-scheduler
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ards-reports/public/components/main/main_utils.tsx
76.53% <ø> (ø)

...eports/public/components/utils/settings_service.ts
76.92% <76.92%> (ø)

...s-reports/server/routes/utils/dataReportHelpers.ts
92.70% <93.75%> (+30.20%)
:arrow_up:

.../components/main/report_details/report_details.tsx
73.73% <100.00%> (ø)

...ashboards-reports/server/routes/utils/constants.ts
100.00% <100.00%> (ø)

...rts/server/routes/utils/savedSearchReportHelper.ts
98.75% <100.00%> (-0.06%)
:arrow_down:

...r/routes/utils/visual_report/visualReportHelper.ts
88.75% <100.00%> (+4.22%)
:arrow_up:

...tsscheduler/action/CreateReportDefinitionAction.kt

...sscheduler/action/GetAllReportDefinitionsAction.kt

...rtsscheduler/action/GetAllReportInstancesAction.kt

... and 66 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 076e353...3e35304. Read the comment docs.

authors: codecov-commenter zhongnansu

repo: opensearch-project/dashboards-reports


text:
adding dlqObject and dlqWriter interface to

Description
The PR creates a few building blocks to begin building a generic S3 DLQ as part of #2298
Issues Resolved
None
Check List

[x] New functionality includes testing.
[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2392 (cc876ce) into main (b3142e0) will increase coverage by 0.01%.
The diff coverage is 89.79%.

:exclamation: Current head cc876ce differs from pull request most recent head ea7f231. Consider uploading reports for the commit ea7f231 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #2392      +/-   ##
============================================
+ Coverage     93.67%   93.69%   +0.01%
- Complexity     1998     2018      +20
============================================
  Files           234      235       +1
  Lines          5546     5595      +49
  Branches        443      454      +11
============================================
+ Hits           5195     5242      +47
+ Misses          243      236       -7
- Partials        108      117       +9

Impacted Files
Coverage Δ

...ensearch/dataprepper/model/failures/DlqObject.java
89.79% <89.79%> (ø)

... and 1 file with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

repo: opensearch-project/data-prepper


text:
Adding more checks around schedule and metadata in transforms

Signed-off-by: Ravi Thaluru ravi1092@gmail.com
Issue #, if available:
Description of changes:

Adding additional check when creating transform to check period is set to be greater than 0
Not allowing setting metadata id for new transform

CheckList:
[ X] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #22 (1f41089) into main (7fdd1d1) will decrease coverage by 0.09%.
The diff coverage is 83.33%.

@@             Coverage Diff              @@
##               main      #22      +/-   ##
============================================
- Coverage     77.86%   77.76%   -0.10%
+ Complexity     1861     1857       -4
============================================
  Files           255      255
  Lines          9983     9988       +5
  Branches       1512     1515       +3
============================================
- Hits           7773     7767       -6
- Misses         1396     1402       +6
- Partials        814      819       +5

Impacted Files
Coverage Δ

...earch/indexmanagement/transform/model/Transform.kt
90.10% <80.00%> (+1.33%)
:arrow_up:

.../action/explain/TransportExplainTransformAction.kt
58.82% <100.00%> (ø)

...nt/indexstatemanagement/model/destination/Slack.kt
40.90% <0.00%> (-13.64%)
:arrow_down:

.../rollup/action/start/TransportStartRollupAction.kt
58.00% <0.00%> (-6.00%)
:arrow_down:

...arch/indexmanagement/rollup/RollupSearchService.kt
62.00% <0.00%> (-4.00%)
:arrow_down:

...nt/rollup/action/stop/TransportStopRollupAction.kt
65.30% <0.00%> (-2.05%)
:arrow_down:

...ensearch/indexmanagement/rollup/model/ISMRollup.kt
93.04% <0.00%> (-1.74%)
:arrow_down:

...ndexstatemanagement/IndexStateManagementHistory.kt
78.26% <0.00%> (-1.74%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7fdd1d1...1f41089. Read the comment docs.

authors: codecov-commenter thalurur

repo: opensearch-project/index-management


text:
Add MacOS Support

Coming from https://github.com/opensearch-project/opensearch-plugins/issues/96, add MacOS (Darwin) support.

[x] Passing CI on MacOS
[x] Documentation

The mac os support is present in K-NN. Resolving this issue

Hi @navneet1v @jmazanec15 @navneet1v ,
Could you let me know if k-NN now fully support build and run on both x64 and arm64 macOS?
Thanks.

@peterzhuamazon We have CI working for mac https://github.com/opensearch-project/k-NN/blob/main/.github/workflows/CI.yml#L95-L134. I believe this will run with arm64. Locally, k-NN is able to build on x86.

Thanks @jmazanec15 seems like it is ready to go 👍

Sorry for this naive question, but what will be the next steps to make kNN work for MacOS users? I just tried it with OpenSearch 2.17.1 installed on arm64 MacOS and it still throws me an error:
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.knn] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "unknown setting [index.knn] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",
    "suppressed": [
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.knn.algo_param.ef_construction] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.knn.algo_param.ef_search] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.knn.algo_param.m] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      },
      {
        "type": "illegal_argument_exception",
        "reason": "unknown setting [index.knn.space_type] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"
      }
    ]
  },
  "status": 400
}

repo: opensearch-project/k-NN


text:
Integration test for opendistro

During CI for opendistro, while trying to disable security plugin instead of removal, observed following exception.
integration_1  | [2021-07-01T21:55:02,641][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [067ef78be3a9] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively
integration_1  | uncaught exception in thread [main]
integration_1  | [2021-07-01T21:55:02,741][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [067ef78be3a9] uncaught exception in thread [main]
integration_1  | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  | Caused by: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:722) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        ... 6 more
integration_1  | Caused by: java.lang.reflect.InvocationTargetException
integration_1  |        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
integration_1  |        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
integration_1  |        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
integration_1  |        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
integration_1  |        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:713) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        ... 6 more
integration_1  | Caused by: org.elasticsearch.ElasticsearchException: opendistro_security.ssl.transport.keystore_filepath or opendistro_security.ssl.transport.server.pemcert_filepath and opendistro_security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
integration_1  |        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initTransportSSLConfig(DefaultOpenDistroSecurityKeyStore.java:412) ~[?:?]
integration_1  |        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:248) ~[?:?]
integration_1  |        at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.<init>(DefaultOpenDistroSecurityKeyStore.java:169) ~[?:?]
integration_1  |        at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.<init>(OpenDistroSecuritySSLPlugin.java:217) ~[?:?]
integration_1  |        at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.<init>(OpenDistroSecurityPlugin.java:246) ~[?:?]
integration_1  |        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
integration_1  |        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
integration_1  |        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
integration_1  |        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
integration_1  |        at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:713) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:658) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:479) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:168) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.node.Node.<init>(Node.java:346) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
integration_1  |        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
integ

@VijayanB Can you please re-tag this issue with the appropriate version?

Closing, old/fixed.

authors: VijayanB bbarani dblock


text:
[FEATURE] Improve test coverage for MLHttpClientFactory.java

We need to improve test coverage in our repo.
This issue will target only to improve test coverage for MLHttpClientFactory.java class
From this url we can see where test coverage is missing for this class: https://app.codecov.io/gh/opensearch-project/ml-commons/blob/2.x/ml-algorithms%2Fsrc%2Fmain%2Fjava%2Forg%2Fopensearch%2Fml%2Fengine%2Fhttpclient%2FMLHttpClientFactory.java

@dhrubo-os Hello Dhrubo! I would like to take up this feature as my first issue

Sure.

@Arnav-Gr0ver any update on this?

repo: opensearch-project/ml-commons


text:
Add Initial Codebase for Integrations CLI

Description
Per issue #1451, this PR implements the basic initial setup for the integrations CLI. From the help page:
Usage: integrations-cli.py [OPTIONS] COMMAND [ARGS]...

  Create and maintain Integrations for the OpenSearch Integrations Plugin.

Options:
  --help  Show this message and exit.

Commands:
  check    Analyze the current Integration and report errors.
  create   Create a new Integration from a specified template.
  package  Package the current integration for use in OpenSearch.

It has some robustness to be added after the API is ready, particularly with regard to generating and validating sub-components such as saved objects. The current implementation only generates a configuration and references the local schema files.
The PR also creates a CI workflow that tests the new Python code.
TODO before converting from draft:

[x] Documentation
[ ] More robust error handling
[ ] General code cleanup

Issues Resolved
Part of progress on #1451. This issue can safely be archived and superseded by smaller-scope issues.
Conflicts with #1454.
Check List

[X] New functionality includes testing.

[X] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Updated TODO list:

[ ] General Code Clean-up
[ ] Error handling in clear non-happy-path cases
[ ] Fix the CI pipeline
[ ] Add more testing
[ ] Switch to Pytest for consistency with other OS CLI tools
[ ] Convert to a proper Python package
[ ] Actual dependency management and validation

[ ] Detect when components contain other components and add them
[ ] Validate if subcomponents are missing

[ ] Improve catalog searching functionality
[ ] Refactor usage of returns (failed experiment)
[ ] Proper automatic integration uploading

repo: opensearch-project/observability


text:
CVE fix:json-schema, gson & glob-parent

Description
Adding CVE fixes to opensearch-observability and observability-dashboards.
Issues Resolved
Upgraded following libraries:
#367 #366
1. gson -> 2.8.9
2. json-schema -> 0.4.0
3. glob-parent -> 6.0.1

Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Can you add the yarn.lock file to the PR too?

Can you add the yarn.lock file to the PR too?

on it 👍🏽

Codecov Report

Merging #368 (29e861d) into main (e3f1e66) will increase coverage by 14.66%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #368       +/-   ##
=============================================
+ Coverage     60.24%   74.91%   +14.66%
  Complexity      269      269
=============================================
  Files           197       37      -160
  Lines          6588     2029     -4559
  Branches       1247      231     -1016
=============================================
- Hits           3969     1520     -2449
+ Misses         2426      347     -2079
+ Partials        193      162       -31

Flag
Coverage Δ

dashboards-observability
?

opensearch-observability
74.91% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...-observability/common/constants/trace_analytics.ts

...vability/public/components/explorer/no_results.tsx

...nents/custom_panels/helpers/custom_input_modal.tsx

...omponents/explorer/visualizations/frame_layout.tsx

...alytics/components/common/plots/error_rate_plt.tsx

...omponents/common/helpers/ppl_docs/commands/eval.ts

...ytics/components/common/filters/filter_helpers.tsx

...ds-observability/common/constants/custom_panels.ts

...public/components/common/field_icon/field_icon.tsx

...tom_panels/panel_modules/panel_grid/panel_grid.tsx

... and 150 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3f1e66...29e861d. Read the comment docs.


text:
Fix missing pluralization on Role

Description
Both cluster_permission and index_permission should be plural as per the docs.

Sure thing @Xtansia, I hope I did everything correctly :smile:


text:
[Bug]: OpenSearch Snapshot failing checksum validation

Describe the bug
This is similar to a bug we faced earlier https://github.com/opensearch-project/opensearch-build/issues/2114 where the snapshot for the latest main build fails checksum validation. Related error reported on Dashboards https://github.com/opensearch-project/OpenSearch-Dashboards/issues/3787. This also breaks our CI workflows.
To reproduce
Run yarn opensearch snapshot
Expected behavior
No response
Screenshots

Host / Environment
OSD main
Additional context
No response
Relevant log output
info Installing from snapshot
   │ info version: 3.0.0
   │ info install path: /__w/OpenSearch-Dashboards/OpenSearch-Dashboards/.opensearch/3.0.0
   │ info license: oss
   │ info Verifying snapshot URL at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz
   │ info downloading artifact from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz
   │ info downloading artifact checksum from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz.sha512
   │ERROR artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz does not match expected checksum
   │        expected: 16e9c818034f17dcac24017a614300952de9c1d4c99c2708e85ef7d02d646a6c452b5e4e766a115a794ea28350cca5ea746e5bf033aa23d504eddb9b06464580
   │        received: f1008401e42b524768a86ecef288cd746638b205df00c2e4743f6f51475072581730aeaa96951d06c2b9c274501cc148cbe8cdae01a024479d512895f1c7b876
   │        headers:
   │          accept-ranges: bytes
   │          age: 143
   │          connection: close
   │          content-length: 240730749
   │          content-type: application/x-gzip
   │          date: Wed, 05 Apr 2023 17:35:22 GMT
   │          etag: "eac155b9d05cbf94b122eaadd3cdf728-46"
   │          last-modified: Wed, 05 Apr 2023 01:41:18 GMT
   │          server: AmazonS3
   │          via: 1.1 38a1b6b88186539189dcd472aa234808.cloudfront.net (CloudFront)
   │          x-amz-cf-id: g6rMVBFUy_TUNyC6iN5kgH_7n9gIx-CUjN6pxl1P4qB0Uv9QVOlYdg==
   │          x-amz-cf-pop: LAX53-P3
   │          x-amz-server-side-encryption: AES256
   │          x-amz-version-id: 8wlhswK4CcPrNJ39UrYyW0spN8IcwY2g
   │          x-cache: Hit from cloudfront
   │        content[500 / 240730749 base64]: H4sIAAAAAAAA/+y6c5RlzZYvmuZO27Zt27Ztm5VmpZ2Vtm3bdlbaViUreavv6X73nq+7z+h+94z7z3tz7LV2RKw5Y0SsGRO/OZaDo5m9i5mRs4klHQs9Iz0jnYq8kKKKpIIqA9A/jRgZWRk52NgIGf9Gf/3/n20mVlYmFiYWNhYWdkJGJhZ2FhYgQrZ/3hL+c3JzcTVy/rPE/9N5/rq5f8LS/m+Qw3+qf1sr43/SGfjv65+VlZHp/9f//w36x/r/T5/SW/95af9VYmRiZGRnZf2P9M/OysLBxMjO/Pf6Z2ZiYuEAIvw/1sl/hf4/rn9FGVAwVCAgKKg/baH/OQLyr0+g/1xyYqpCdFLy4gygf2HM3+ajSQQGApr+w436vzPKCclLiYupqNLLibslyjksMSKG3ST8IiB2AiayMOExwJG/QgcuNkOChnJySVwfoqwn+aYaajBbZ9apJBbnf5KR4WHuS5PMG+B2iINJ01xBjE1h5HhBQTGOkS6oWUZswZtX5iMAvI7DJQkmcYKuY8ce7FrocCA/i9Md0EAoy5E4fJo71z6NX/xKOQwV1m+Pat2XiHditfGkmNo0SXXopdNklZDW95pcquWepzce5Y8DsTMuPxv9vvG7fmWdPi6Jk+ixNHr28c2vscTBFxJm5pB9JqFJmtY=

Hi, @ashwin-pc , it is possible the same issue as the PR you refered to and the comment at https://github.com/opensearch-project/opensearch-build/issues/2114#issuecomment-1138962798
Can you try it again a few hours later and if you still see the same issue?
Or , if you can use https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-3.0.0-linux-x64.tar.gz instead ?
thanks,

@jordarlu can we get clarification about the difference between the 2 domains? We were hesitant to change the domain for a few reasons.

For our CI we need both the manifest and snapshot to run the CI correctly. Is the CI endpoint manifest url the same as artifact? based on what i see they seem to be different.
We also run quite a lot of CI workflows using these snapshots, a build failure in the snapshot process would bring our CI workflows to a halt. based on what i read, the bake time for artifact.opensearch.org gives us the buffer to make sure that the builds break less often. This is my assumption, correct me if im worng here.

Based on some of the older conversations, it appears that the reason this is happening is because we cache the checksum but not the snapshot. Cant we simply not cache the checksum and generate it for every build? seems a little weird to use a cached checksum for a build.

Hello, @ashwin-pc .. unfortunately the cache mechanism is on CloudFront where we don't have management there as I learned, and let me add @peterzhuamazon and @gaiksaya here to keep me honest..
thanks,

Hi @ashwin-pc can you give us the exact step to reproduce this?
Can you show me the code of your repo that actually does the check?
I want to understand the root cause of this as this has been going on before in another issue. Thanks!

Sure here is the code: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/packages/osd-opensearch/src/artifact.js

Hello, there ... trying to see how we can move forward on this issue :
@ashwin-pc , have you got a chance to look into using ci.opensearch.org instead of artifacts.opensearch.org to download the artifact?
@rishabh6788 , I bellieve we were unable to reproduce the issue, did we?
cc : @peterzhuamazon

Hey, sorry i have not had the time to look into that unfortunately. @kavilla has more context here, looping him in.

thanks, @ashwin-pc ,,
Hi, @kavilla , wondering if you ever got a time to take a look on this issue, and if any comment on using ci.opensearch.org instead of artifacts.opensearch.org to download the artifact? thanks !

Closing this issue, @ashwin-pc @kavilla please re-open if required. Thanks

I'm facing the same issue:
C:\opensearch\OpenSearch-Dashboards>yarn run opensearch snapshot --network-timeout 1000000
yarn run v1.22.19
$ scripts/use_node scripts/opensearch snapshot --network-timeout 1000000
 info Installing from snapshot
   │ info version: 3.0.0
   │ info install path: C:\opensearch\OpenSearch-Dashboards\.opensearch\3.0.0
   │ info license: oss
   │ info Verifying snapshot URL at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip
   │ info downloading artifact from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip
   │ info downloading artifact checksum from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip.sha512
   │ERROR artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip does not match expected checksum
   │        expected: 565d4792a00913f094e77ea8d0e0047dc628fcb27d7e2b870126213aa90dd66c12ab76f8ff61d57cede6294ee7018afb7ea2e86b6e1ff91abd526350dc3fc890
   │        received: 1f7dabac5c98117078b8289eb410dacec71513a765c94017b00329cab81388e1781b0afce4b759ae5e07b03c9ab6be1010080b9ebf0b9640c4316aa18542fd9f
   │        headers:
   │          accept-ranges: bytes
   │          connection: close
   │          content-length: 246071451
   │          content-type: application/zip
   │          date: Thu, 14 Dec 2023 09:49:04 GMT
   │          etag: "4134a72eb78c6832804653e566e2393d-47"
   │          last-modified: Thu, 14 Dec 2023 07:16:08 GMT
   │          server: AmazonS3
   │          via: 1.1 b42eb3d87622a28f769121f374a92fda.cloudfront.net (CloudFront)
   │          x-amz-cf-id: ZGjEmtcIpLselNIp-9g10JO2el1qrH_9hPrXf73cpLIfaFatfyKCfg==
   │          x-amz-cf-pop: WAW51-P4
   │          x-amz-replication-status: COMPLETED
   │          x-amz-server-side-encryption: AES256
   │          x-amz-version-id: CIuO5c5BuhitcP0vpMsLvBEuQppPP68j
   │          x-cache: Hit from cloudfront
   │        content[500 / 246071451 base64]: UEsDBBQAAAgIAAAAQQAAAAAAAgAAAAAAAAAaAAAAb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC8DAFBLAwQUAAAICAAAAEEAAAAAAAIAAAAAAAAAHgAAAG9wZW5zZWFyY2gtMy4wLjAtU05BUFNIT1QvbGliLwMAUEsDBBQAAAgIAAAAQQAa+HlGSsvKAJcY3AA7AAAAb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC9saWIvb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC5qYXKsugOUJVGyLlzmKdu2bdu2bZtdZpdtV5dtddl2ddm22X/PP/euN3gz7973bq6TefbOjIiz4uzQF7kVZUDBUIGAoKCAgICEgP5ygAD99YD+c8qJqQrRScmLM4ACKf4d4Zd2mUohMBDQ5h9q1L8llBOSlxIXU1GllxN3S9RxWGREDN3W+PUbMRIDscHkuEXK85c9TT4GeoIjkoM8r2kMMk9DDvHLo6wKXWTF3QnXdjaXybu85SNoNoVBL7eZK5bjeEtWZ/lzlG8Qfkgw8407CC0216NgrLobTF/ScxHNQJ3KFytbE5uIpfgsfIidgmnTEUUTP69xeaYMat/lkSNyh8hE6Zh712hUaXw8O/gMYZznwLqkuXIdqYNp5pDXd5OfCcFY4o8=
Done in 8.95s.

authors: ashwin-pc ciesiolka jordarlu peterzhuamazon prudhvigodithi

repo: opensearch-project/opensearch-build


text:
Add few components to 2.4.1 and update observability, dashboards-reports GH repo

Signed-off-by: Sayali Gaikawad gaiksaya@amazon.com
Description
Add remaining components to 2.4.1
Issues Resolved
#2925
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@gaiksaya  you missed reports 2.4.1 changes on url.

@gaiksaya you missed reports 2.4.1 changes on url.

Can you point where? I see it updated here https://github.com/opensearch-project/opensearch-build/pull/2971/files#diff-4b1fcaaacfd496281202cb1a11b31b1f397d83f281492b38f60b820a49dd51ffR21

@gaiksaya you missed reports 2.4.1 changes on url.

Can you point where? I see it updated here https://github.com/opensearch-project/opensearch-build/pull/2971/files#diff-4b1fcaaacfd496281202cb1a11b31b1f397d83f281492b38f60b820a49dd51ffR21

OS side, since reports including reports scheduler backend plugin.

authors: gaiksaya peterzhuamazon


text:
UTF Reporting Prototype

Signed-off-by: Mikayla Thompson thomika@amazon.com
This PR is a branch off of #59 , which means that this PR currently displays the superset of both changes. Once that PR is merged, this one will show only the unique changes. In the meantime, it can be browsed by looking at specific commits.
Description

Adds a first pass at the Reporting functionality that shows users which expectations had tests that succeeded & failed and which were not run at all.
For failed & skipped Expectations, it prints the expectation description as well.
It calls out which "phases" (do we have better terminology for this?) the tests ran in--for instance whether a given expectation had a test that ran in the "pre-upgrade" phase vs. "post-upgrade".
I also added a bit of code to slightly improve how we handle temporary data storage during the robot tests. By default, it's now a json file in the /tmp/utf directory, but this can be overriden as a variable passed to the run command for the robot framework (in the same spot as the host and port are specified).

Issues Resolved
#62
Testing
In the process of adding more -- this is very minimally tested right now because all of the code lives in a FrameworkStep.
Check List

[ ] New functionality includes testing

[X] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #63 (b8737a2) into main (5f728a0) will decrease coverage by 3.19%.
The diff coverage is 55.34%.

@@            Coverage Diff             @@
##             main      #63      +/-   ##
==========================================
- Coverage   81.56%   78.36%   -3.20%
==========================================
  Files          35       37       +2
  Lines        1085     1239     +154
  Branches       89      130      +41
==========================================
+ Hits          885      971      +86
- Misses        189      257      +68
  Partials       11       11

Impacted Files
Coverage Δ

...de_testing_framework/core/test_config_wrangling.py
94.04% <ø> (ø)

...e_testing_framework/steps/step_load_test_config.py
33.33% <0.00%> (ø)

...es/upgrade_testing_framework/workflows/__init__.py
100.00% <ø> (ø)

...ade_testing_framework/steps/step_report_results.py
16.66% <16.66%> (ø)

...esting_framework/steps/step_select_expectations.py
35.71% <18.18%> (-24.29%)
:arrow_down:

...ng_framework/cluster_management/cluster_objects.py
85.71% <66.66%> (-14.29%)
:arrow_down:

...ades/upgrade_testing_framework/core/expectation.py
100.00% <100.00%> (ø)

.../upgrade_testing_framework/core/framework_state.py
100.00% <100.00%> (ø)

.../upgrade_testing_framework/core/versions_engine.py
100.00% <100.00%> (ø)

...grades/upgrade_testing_framework/steps/__init__.py
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter mikaylathompson


text:
[PROPOSAL] Lower number of maintainers required to merge from 2 to 1

What/Why
What are you proposing?
Lower the number of maintainers required to merge a PR from 2 to 1.
What problems are you trying to solve?
There are too few maintainers active and a lot of PRs waiting for weeks.

What is the developer experience going to be?
PRs approved by 1 maintainer can be merged.
Other
I will wait for at least 2 other maintainers to agree to this before making it fact.

[ ] @VijayanB
[ ] @jayeshathila
[ ] @robsears
[ ] @vamshin
[x] @VachaShah
[x] @dblock
[x] @nhtruong
[x] @harshavamsi

With 4 maintainers that have been recently most active in favor I've made the change.

repo: opensearch-project/opensearch-ruby


text:
Fix build to succeed with opensearch_version provided

Signed-off-by: Marc Handalian handalm@amazon.com
Is your feature request related to a problem? Please provide an existing Issue # , or describe.
The build is failing when opensearch version specified is not the same as the default value.
Describe the solution you are proposing
The cause of this problem is updateShas is invoked, which is a task in this project, during the doLast step of the buildRCA task.  This task is not invoked with any properties passed in during the initial build command.
This change will execute this directly with finalizedBy instead of using doLast and executing from the command line.
This change also bumps the default up to 1.1 on main.
Describe alternatives you've considered
passing the version string down to doLast.  I would rather directly invoke the task without executing from shell.
closes #51
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #52 (eb9dfc8) into main (83aafb1) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head eb9dfc8 differs from pull request most recent head cd5077c. Consider uploading reports for the commit cd5077c to get more accurate results

@@            Coverage Diff            @@
##               main      #52   +/-   ##
=========================================
  Coverage     72.18%   72.18%
  Complexity      356      356
=========================================
  Files            44       44
  Lines          2509     2509
  Branches        160      160
=========================================
  Hits           1811     1811
  Misses          592      592
  Partials        106      106

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83aafb1...cd5077c. Read the comment docs.

repo: opensearch-project/performance-analyzer


text:
OpenSearch talks to Community Contributors in HayStack and Berlin Buzzwords to Create a Persona Framework for Search

Describe the blog post your would like to write
The topic of the blog are OpenSearch personas and how we talked to the community to help establish a framework for Search Personas
What is the title of the blog post?
OpenSearch talks to Community Contributors in HayStack and Berlin Buzzwords to Create a Persona Framework for Search
Who are the authors?
Aparna Sundar
What is the proposed posting date?
July 21st

Closing since this blog is published.

authors: apasun hdhalter

repo: opensearch-project/project-website


text:
Include link checker in CI

Description
Similar to https://github.com/opensearch-project/documentation-website/pull/3584.

Run link checker in CI, fail PRs on broken links.
Improve the homegrown link-checker a bit, make HEAD requests, etc.
Fix many broken links.
Closes https://github.com/opensearch-project/documentation-website/issues/3583

Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Thanks for the review @krisfreedain!

Yes, the build will fail if a link is broken. I will move this into a cron, but in the past I've seen that cause us to stop paying attention to broken links. Are you sure? Any contributor that runs into a broken link will tend to fix it if they run into that obstacle, which I think accomplishes the best outcome of links being rarely broken.
I think that partner no longer exists, the website is MIA. Let me know what you want me to do with it: I could instead remove the link, or something else?

Yeah, I'd love to have a weekly / monthly report on broken links if you build it out as a cron job. That would be helpful (as demonstrated by the updates you found) - but I don't want to force-break the build. When a contributor files a PR, one of us that are active maintainers should be reviewing them prior to publish. Broken links are less common at that time - but a report to catch them if they break later is actionable.
Yeah - let's keep it, but I'm OK with removing the link - myself and @pajuric can follow up with them from the partner side and see what is going on. (this partner was added via PR: https://github.com/opensearch-project/project-website/pull/1381 )

thanks @dblock !

authors: dblock krisfreedain


text:
Update retrospective date for 2.13.0

Description
Update retrospective date for 2.13.0
Issues Resolved
https://github.com/opensearch-project/opensearch-build/issues/4602
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Its too late to merge now as the retro is scheduled for tomorrow. Lets schedule the retro for Monday.

@peterzhuamazon Can you please review again? I'll merge the PR then.
Thanks!

authors: bbarani gaiksaya


text:
[BUG] Critical | Submitting the rule without condition breaks the app.

What is the bug?
Submitting the rule without condition brakes the app.
How can one reproduce the bug?
Steps to reproduce the behavior:

Go to the detection rules
Click on Create detection rule
Fill all the fields except the condition field
Submit the form
Notice the toaster "No living connection"

What is the expected behavior?
The user should not be able to submit the form without the condition field filled in.
Notes
The similar happens if tag fields are not in the correct format, different message in screenshot two.
Do you have any screenshots?

Do you have any additional context?
Request:
{"id":"25b9c01c-350d-4b95-bed1-836d04a4f324","category":"dns","title":"My Custom Rule","description":"Add some text here","status":"experimental","author":"jovan cvetkovic, pera peric","references":[{"value":""}],"tags":[{"value":""}],"log_source":"","detection":"condition: ''\nSelection_1:\n  Field_1|contains:\n    - Tester\n","level":"critical","false_positives":[{"value":""}]}

Fixed

authors: amsiglan jovancacvetkovic


text:
Close hanging tasks in correlateFindings Action workflow

Description
Close hanging tasks in correlateFindings Action workflow
Issues Resolved
#879
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Moving this commit to #880

repo: opensearch-project/security-analytics


text:
Fix wrong indentation in CD

opensearch-security pull request intake form
Please provide as much details as possible to get feedback/acceptance on your PR quickly

Category: (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)
Maintenance

Github Issue # or road-map entry, if available:

Description of changes:

Why these changes are required?

What is the old behavior before changes and new behavior after changes? (Please add any example/logs/screen-shot if available)

Testing done: (Please provide details of testing done: Unit testing, integration testing and manual testing)

TO-DOs, if any: (Please describe pending items and provide Github issues# for each of them)

Is it backport from main branch? (If yes, please add backport PR # and commits #)

By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1363 (1e5fbdf) into opendistro-1.1 (80e3670) will increase coverage by 0.02%.
The diff coverage is n/a.

@@                 Coverage Diff                  @@
##             opendistro-1.1    #1363      +/-   ##
====================================================
+ Coverage             63.53%   63.56%   +0.02%
  Complexity             3023     3023
====================================================
  Files                   237      237
  Lines                 16567    16567
  Branches               2965     2965
====================================================
+ Hits                  10526    10530       +4
+ Misses                 4520     4515       -5
- Partials               1521     1522       +1

Impacted Files
Coverage Δ

...arch/security/filter/OpenDistroSecurityFilter.java
67.07% <0.00%> (-0.61%)
:arrow_down:

...search/security/tools/OpenDistroSecurityAdmin.java
47.04% <0.00%> (-0.14%)
:arrow_down:

.../dlic/auth/ldap2/LDAPConnectionFactoryFactory.java
57.25% <0.00%> (+0.76%)
:arrow_up:

...transport/OpenDistroSecuritySSLRequestHandler.java
59.03% <0.00%> (+6.02%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 80e3670...1e5fbdf. Read the comment docs.

authors: codecov-commenter vrozov

repo: opensearch-project/security


text:
Hide ldap roles

Description

Category: Enhancement, New feature
Why these changes are required?
To allow admin to have more control on what backend roles user sees.
What is the old behavior before changes and new behavior after changes?
Old behavior is that all user's roles are fetched from ldap/ad. Even if role is not used in opensearch use can view all assigned roles.
New behavior is that admin can configure what roles will be visible to the user. If flag is not configured cluster will behave the same. With new config admin is able to hide part of roles from users using simple regex.
Example config:

    authz:
      ldap_roles:
        description: "Authorize using LDAP"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          type: ldap
          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: false
            hosts:
            - 127.0.1.1:390
            bind_dn: cn=readonly,dc=example,dc=org
            password: changethistoo
            userbase: ou=People,dc=example,dc=org
            usersearch: (cn={0})
            username_attribute: cn
            skip_users:
              - admin
              - kibanaserver
            rolebase: ou=Groups,dc=example,dc=org
            rolesearch: (uniqueMember={0})
            userroleattribute: null
            userrolename: disabled
            rolename: cn
            resolve_nested_roles: true
  new ->    exclude_roles: "(this_role_will_be_hidden.*)"

Issues Resolved
https://github.com/opensearch-project/security/issues/2189
Testing
Unit testing and manual testing performed
Check List

[ ] New functionality includes testing
[ ] New functionality has been documented
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@MaciejMierzwa Can exclude_roles support a list as well?

@cwperks
Current implementation accepts regex. In case of multiple roles to filter out user can do it as following:
exclude_roles: "(excluded_role1|excluded_role2|excluded_role1|another_one.*)"

to exclude roles:

excluded_role1
excluded_role2
excluded_role1
another_one

Hey @MaciejMierzwa just checking in to see if you need help driving this home? Thanks for all your help :)

Hi @MaciejMierzwa, you will want to file an issue over on the documentation website repo explaining the change and then someone there should be able to help you.

@cwperks I guess we could do either of those options, or even circle back to your suggestion from https://github.com/opensearch-project/security/pull/3809#issuecomment-1847451658. Let me know what do you think

Hi @MaciejMierzwa and @cwperks to keep this moving forward lets just keep things as is for now and add some documentation we can always refactor this if need be but I don't want this to stall.

IMO I think supporting this as a list would be more consistent with other LDAP settings like skip_users.

Hi @MaciejMierzwa , IMO we should support wildcard-entries. Please let us know if you would need anything from us to keep this moving forward.

Hey @MaciejMierzwa , I modified this PR on my fork take in a list of globs for exclude_roles (https://github.com/cwperks/security/pull/20). A list of globs would be consistent with other configuration settings for the LDAP Authorization backend like skip_users. Let me know what you think.

@cwperks should this PR be closed in lieu of #4025

authors: DarshitChanpura MaciejMierzwa cwperks scrawfor99


text:
Version Bump: spring-beans-5.2.19 -> spring-beans-5.2.20

Signed-off-by: penghuo penghuo@gmail.com
Issues Resolved

https://github.com/opensearch-project/sql/issues/525
Related to https://github.com/opensearch-project/OpenSearch/issues/2699

Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Thanks for the fix! Do we need to bump 'spring-test' just in case?

Sure.

repo: opensearch-project/sql


text:
Error message to be more appropriate when deleting a DA without deleting the SFC

Description
Error message to be more appropriate when deleting a DA without deleting the SFC
Expected Behavior
Error message should be more suggestive - e.g. DA cannot be deleted unless the SFC is deleted
Actual Behavior
Error message when deleting DA when SFC is not deleted -  Distributed Appliance is currently protecting a workload
Steps to Reproduce

Create a VC (using SDN - Neutron-sfc), MC, DA, DS, SG.
Create SFC.
Bind the SG to SFC.
Now delete the SG.
Delete DA
Get an error message - Distributed Appliance is currently protecting a workload
This suggests that probably there is a SG being protected, whereas, in reality we need to delete the SFC to delete the DA.

Additional Information
WORKAROUND - delete the SFC and then DA delete will be successful
Environment

OSC Version/Build: Version: 1.0.0-SNAPSHOT (Build:112-g42e1905, 2017-10-11T18:46:02Z)
Cloud Provider Info:
Plugin Info:
Other:

Status

[ ] Reproduced
[ ] In Progress
[ ] Validated     =>    Verified in Version/Build:

please Handle #544 as part of this change

repo: opensecuritycontroller/osc-core

license_source: bigquery


text:
fix(*) : Fixing existing lint issues in the repo

Description:
This PR fixes all the existing linter issues.
According to the issue #1324 there were a bunch of lint issues on the repo, not all the errors mentioned in the issue appeared.  I have fixed all the errors that are appearing currently. Further the PR enables the linters check for the entire repo going forward.
Fixes : #1324 , #1578
Affected area:

New Functionality      [ ]
Documentation          [ ]
Install                [ ]
Control Plane          [ ]
CLI Tool               [ ]
Certificate Management [ ]
Networking             [ ]
Metrics                [ ]
SMI Policy             [ ]
Security               [ ]
Tests                  [ ]
CI System              [X]
Performance            [ ]
Other                  [ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution? no

Codecov Report

Merging #2023 (b7a6352) into main (3c30e60) will increase coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2023      +/-   ##
==========================================
+ Coverage   56.31%   56.33%   +0.01%
==========================================
  Files         142      142
  Lines        5775     5773       -2
==========================================
  Hits         3252     3252
+ Misses       2520     2518       -2
  Partials        3        3

Impacted Files
Coverage Δ

cmd/cli/dashboard.go
0.00% <0.00%> (ø)

cmd/cli/install.go
65.13% <ø> (ø)

cmd/cli/mesh_list.go
36.84% <0.00%> (ø)

cmd/cli/namespace_list.go
35.55% <0.00%> (ø)

cmd/cli/osm.go
0.00% <0.00%> (ø)

pkg/certificate/providers/tresor/fake.go
37.50% <ø> (+2.20%)
:arrow_up:

pkg/debugger/envoy.go
0.00% <0.00%> (ø)

pkg/envoy/lds/listener.go
38.84% <ø> (ø)

pkg/envoy/xdsutil.go
64.08% <ø> (ø)

pkg/strings/all.go
100.00% <ø> (ø)

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c30e60...b7a6352. Read the comment docs.

authors: codecov-io snehachhabria

repo: openservicemesh/osm


text:
chore(release): update version to v0.10.0-rc.1

Signed-off-by: nshankar13 nshankar@microsoft.com

[ ] I have cherry-picked any changes labelled needs-cherry-pick-vX.Y.Z

[X] I have made all of the following version and patch updates:

Updated the container image tag in charts/osm/values.yaml
Updated the chart and app version in charts/osm/Chart.yaml
Updated the image tags used in the demo manifests
Regenerated the Helm chart README.md

[X] I have checked that the base branch for this PR is correct as defined by the release guide

Is this a release candidate? Yes

Description:

Testing done:

Affected area:

Functional Area

New Functionality
[ ]

CI System
[ ]

CLI Tool
[ ]

Certificate Management
[ ]

Control Plane
[ ]

Demo
[ ]

Documentation
[ ]

Egress
[ ]

Ingress
[ ]

Install
[ ]

Networking
[ ]

Observability
[ ]

Performance
[ ]

SMI Policy
[ ]

Security
[ ]

Sidecar Injection
[ ]

Tests
[ ]

Upgrade
[ ]

Other
[ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project?

Did you notify the maintainers and provide attribution?

Is this a breaking change?

Codecov Report

Merging #4030 (1b0b897) into release-v0.10 (c05c35a) will decrease coverage by 0.00%.
The diff coverage is n/a.

:exclamation: Current head 1b0b897 differs from pull request most recent head 9a1169a. Consider uploading reports for the commit 9a1169a to get more accurate results

@@                Coverage Diff                @@
##           release-v0.10    #4030      +/-   ##
=================================================
- Coverage          67.84%   67.83%   -0.01%
=================================================
  Files                205      205
  Lines              11569    11569
=================================================
- Hits                7849     7848       -1
- Misses              3669     3670       +1
  Partials              51       51

Flag
Coverage Δ

unittests
67.83% <ø> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

cmd/cli/mesh_upgrade.go
73.58% <ø> (ø)

pkg/k8s/announcement_handlers.go
72.97% <0.00%> (-5.41%)
:arrow_down:

pkg/certificate/rotor/rotor.go
87.50% <0.00%> (+3.12%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c05c35a...9a1169a. Read the comment docs.


text:
Feature pyroscope

Description:
This PR adds scripts to install Pyroscope as an online/dynamic profiling tool in OSM namespace.
Pyroscope is installed in Golang pull mode. It requires the Golang service to have pprof debug web endpoint setup. Therefore right now osm-controller can be profiled. Other components such as osm-bootstrap and osm-injector do not support online profiling.
Resolves https://github.com/openservicemesh/osm/issues/4694

Testing done:
Tested the script in development cluster. Pyroscope is properly installed.

Affected area:

Functional Area

New Functionality
[ ]

CI System
[ ]

CLI Tool
[ ]

Certificate Management
[ ]

Control Plane
[ ]

Demo
[ ]

Documentation
[ ]

Egress
[ ]

Ingress
[ ]

Install
[ ]

Networking
[ ]

Observability
[X]

Performance
[X]

SMI Policy
[ ]

Security
[ ]

Sidecar Injection
[ ]

Tests
[ ]

Upgrade
[ ]

Other
[ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project?

Did you notify the maintainers and provide attribution?

Yes. The script is modified from Pyroscope official example. We didn't inform the maintainer.

Is this a breaking change?

No

Has documentation corresponding to this change been updated in the osm-docs repo (if applicable)?

Documentation task is tracked by https://github.com/openservicemesh/osm-docs/issues/368

Codecov Report

Merging #4744 (d559100) into main (0ad92c9) will increase coverage by 0.05%.
The diff coverage is n/a.

:exclamation: Current head d559100 differs from pull request most recent head 40eaaf1. Consider uploading reports for the commit 40eaaf1 to get more accurate results

@@            Coverage Diff             @@
##             main    #4744      +/-   ##
==========================================
+ Coverage   69.47%   69.53%   +0.05%
==========================================
  Files         217      225       +8
  Lines       15524    15983     +459
==========================================
+ Hits        10785    11113     +328
- Misses       4689     4819     +130
- Partials       50       51       +1

Flag
Coverage Δ

unittests
69.53% <ø> (+0.05%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/certificate/providers/certmanager/helpers.go
0.00% <0.00%> (-20.41%)
:arrow_down:

...certificate/providers/vault/certificate_manager.go
58.57% <0.00%> (-14.65%)
:arrow_down:

cmd/cli/verify_connectivity.go
26.31% <0.00%> (-10.53%)
:arrow_down:

pkg/certificate/providers/tresor/ca.go
61.40% <0.00%> (-8.46%)
:arrow_down:

cmd/osm-bootstrap/osm-bootstrap.go
39.91% <0.00%> (-3.49%)
:arrow_down:

...ertificate/providers/tresor/certificate_manager.go
63.63% <0.00%> (-1.32%)
:arrow_down:

pkg/configurator/client.go
88.97% <0.00%> (-1.14%)
:arrow_down:

cmd/osm-controller/osm-controller.go
13.87% <0.00%> (-0.97%)
:arrow_down:

pkg/certificate/manager.go
81.61% <0.00%> (-0.70%)
:arrow_down:

...icate/providers/certmanager/certificate_manager.go
29.56% <0.00%> (-0.53%)
:arrow_down:

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ad92c9...40eaaf1. Read the comment docs.

@jaellio Well I feel both approaches are ok. I prefer leaving it here. The files in osm-docs manifests/ are for OSM demo. I feel Pyroscope feature is not a pure demo, although we can create a feature demo in OSM documentation.

@nojnhuh do you mean moving the script and configuration both to the osm-docs repo? I do agree with the ease of discovering point.

@nojnhuh do you mean moving the script and configuration both to the osm-docs repo? I do agree with the ease of discovering point.

Yeah, I think keeping all the Pyroscope stuff as close together as we can is easiest to manage.

@nojnhuh That make sense. I will drop this PR and move it to osm-docs as an instructional guide.
cc @jaellio

authors: allenlsy codecov-commenter nojnhuh


text:
Match more possible controller log file names

Before this some controller logs were getting missed because they were named just assisted-installer-controller.logs without any -.
This meant that signatures like the one for UMN and LB-related operator failures were not being run.
cc @omertuc @paul-maidment @rccrdpccl

/lgtm

/approve

For the record the regression was due to https://github.com/openshift/assisted-installer/commit/7a9e909bc

authors: carbonin omertuc rccrdpccl


text:
add annotations to fix scorecard test failures

Signed-off-by: Nishant Parekh nparekh@redhat.com

/retest

/hold

/unhold

authors: donpenney nishant-parekh


text:
improve verify-generated.sh error message

When the verify-generated return with error, it tells the user to run 'make generate' and commit results.
There are times when it's not enough and the user need to run other make targets.
This new message aims to resolve that.
Signed-off-by: Talor Itzhak titzhak@redhat.com

/lgtm
/approve
thanks, this is something I wanted to fix myself.

authors: Tal-or fromanirh

repo: openshift-kni/numaresources-operator


text:
Use upstream cmd/tool code to reduce duplicate code

Signed-off-by: Vincent Demeester vdemeest@redhat.com

/lgtm

authors: jkandasa vdemeester

repo: openshift-pipelines/operator


text:
Correct behavior for unexpected parameters on update requests?

There are a couple branches of parameter validation where we are ignoring erroneous inputs:

Users passing updated parameters that don't exist on a plan.
Users passing enum values that aren't part of the acceptable set of values.

In both cases, we strip them from the request parameters. I think they should be Bad Request 400s, mainly because I think these represent a user's explicit intent, and they will be surprised when that intent has silently been ignored because it is fundamentally invalid. Looking for some thoughts on this before I submit my changes.

Solved with #734, they're errors and should be 400 responses.

repo: openshift/ansible-service-broker


text:
Updates to ControlPlaneMachineSet validation with CEL

/hold this needs the Kube 1.25 rebase to have landed before it will work
We have these validations already but they are currently maintained as a webhook. In an effort to reduce our code complexity and demonstrate the CEL usage for OpenShift, I've tried to replicate the validations here.
Eg. Today we validate the unions with https://github.com/openshift/cluster-control-plane-machine-set-operator/blob/1fa6ff25480e5eb4221a1f5d059dd7b416381e12/pkg/webhooks/controlplanemachineset/webhooks.go#L511-L570, which is complicated, using CEL would simplify this for us and others who are writing unions.
Q: Can users override this validation and persist objects that don't match the schema? Does this change allow us to delete the webhook code?

+1 on the proposed .testsuite.yaml approach 👍

rendering of failures could use some help, but they failed as expected.

I'll have a think about how to improve the rendering of the errors

lgtm, makes sense to me.
I can not say I like this approach to testing, but there is seems a broad agreement about how it should look.

I'll leave the test failure rendering issues for a separate PR as that will need some time to investigate. I've tested these validations manually and am pretty happy that the integration tests are doing as they should be as well, so I think this is good to merge
/label px-approved
/label docs-approved
/label qe-approved

/hold cancel
Tests mean we don't need to wait for the rebase

authors: JoelSpeed damdo deads2k lobziik

repo: openshift/api


text:
Updates ownersfile to be current

Updates Ownersfile to be more current.

/lgtm

Codecov Report

Merging #731 (0781ba0) into master (e1e8331) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #731   +/-   ##
=======================================
  Coverage   38.48%   38.48%
=======================================
  Files          41       41
  Lines        5561     5561
=======================================
  Hits         2140     2140
  Misses       3242     3242
  Partials      179      179

repo: openshift/aws-account-operator


text:
Add flag to the 'assume' command to generate a console url

What type of PR is this?
feature
What this PR does / Why we need it?
Adds a flag --console to cloud assume command to mimic the functionality of cloud console but for the new backplane flow.
Which Jira/Github issue(s) does this PR fix?
Resolves OSD-18258
Pre-checks (if applicable)

[x] Ran unit tests locally
[x] Validated the changes in a cluster
[ ] Included documentation changes with PR

Codecov Report

Merging #197 (e653476) into main (265ba04) will increase coverage by 0.13%.
The diff coverage is 66.10%.

:exclamation: Current head e653476 differs from pull request most recent head 7c82ce9. Consider uploading reports for the commit 7c82ce9 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #197      +/-   ##
==========================================
+ Coverage   51.11%   51.24%   +0.13%
==========================================
  Files          51       51
  Lines        3365     3415      +50
==========================================
+ Hits         1720     1750      +30
- Misses       1364     1377      +13
- Partials      281      288       +7

Files Changed
Coverage Δ

pkg/awsutil/credentials.go
77.77% <ø> (ø)

cmd/ocm-backplane/cloud/assume.go
75.64% <47.36%> (-11.13%)
:arrow_down:

pkg/awsutil/sts.go
54.28% <75.00%> (+12.74%)
:arrow_up:

/test all

/lgtm

/approve

authors: AlexVulaj bmeng codecov-commenter samanthajayasinghe

repo: openshift/backplane-cli


text:
Update AppStudio Component build annotations

Fixes
Related issue: PLNSRVCE-959
Description
Update build related annotations used on AppStudio Component.
Related PR into Build Service is here.
Type of change

Build related changes

/test test

Rebased

/retest

/retest

authors: Michkov mmorhun subhashkhileri

repo: openshift/hac-dev


text:
OCPBUGS-10504: Deletion of the VPCEnpoint on conflicting service names

What this PR does / why we need it:

Bug in the AWSPrivateLink controller, now we handle the conflicting VPCEndpointServiceNames
This is needed for the DR HostedCluster Migration

Which issue(s) this PR fixes:
Fixes #OCPBUGS-10504
Checklist

[x] Subject and description added to both, commit and PR.
[x] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/cherry-pick release-4.13
/cherry-pick release-4.12

/approve

/lgtm

authors: csrwng enxebre jparrill

repo: openshift/hypershift


text:
Pin version for third party release images

The hypershift operator uses thirdparty i.e upstream tooling for autoscaling (cluster autoscaler) and cluster/machine lifecycle (CAPI).
A particular version of these images must be tied to a particular Hypershift release version.
This PR ensure all third party image references are pinned to a particular tag.
Eventually this images be mirrored will be pulled from an internal registry.

/test e2e-aws
/hold

/test e2e-aws

/hold cancel

/hold
let's wait for the aws image to be built

/test e2e-aws

/hold cancel

/lgtm

authors: enxebre ironcladlou


text:
NO-JIRA: Flaky cert validation test

Fixing a flaky test validating the TLS Certificates. Samples:

https://prow.ci.openshift.org/pr-history/?org=openshift&repo=hypershift&pr=4447
https://prow.ci.openshift.org/pr-history/?org=openshift&repo=hypershift&pr=4361

Slack Thread: https://redhat-internal.slack.com/archives/G01QS0P2F6W/p1722350441545099

/area testing

/lgtm

/hold
per @muraee comment

/retest-required

We need to backport this to 4.14 as the regressing PR was backported that far
Observed 4.15 unit flake in https://github.com/openshift/hypershift/pull/4613
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_hypershift/4613/pull-ci-openshift-hypershift-release-4.15-unit/1828142569767309312
https://issues.redhat.com//browse/HOSTEDCP-1795
4.16: https://github.com/openshift/hypershift/pull/4371
4.15: https://github.com/openshift/hypershift/pull/4444
4.14: https://github.com/openshift/hypershift/pull/4473
/cherry-pick release-4.16 release-4.15 release-4.14

authors: bryan-cox jparrill sjenning


text:
OCPBUGS-39035: Ensure AzureCluster.Status.FailureDomains is populated when any NodePool specifies an Availability Zone

Ensure AzureCluster.Status.FailureDomains is populated when any NodePool specifies an Availability Zone.
Otherwise CAPZ will create Azure VMs with both Availability Zone and Availability Set configured, which results in an error:
{
        "error": {
          "code": "BadRequest",
          "message": "Virtual Machine cannot be created because both Availability Zone and Availability Set were specified. Deploying an Availability Set to an Availability Zone isn’t supported."
        }
}

/test all

/retest

/test e2e-aws

/hold

Fixed upstream: https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/5098
/close

authors: enxebre fxierh


text:
pull the version-mapping from the correct namespace

fixes the mapping bug. Testing in
https://github.com/openshift/cluster-kube-controller-manager-operator/pull/164
/hold

/retest

repo: openshift/library-go


text:
Use the service name constant for the service name

Description
this change is a no-op since both constants have the same value, but avoids a potential future issue if one of them is changed.
Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

Related Tickets & Documents

Related Issue #
Closes #

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] PR has passed all pre-merge test jobs.
[ ] If it is a core feature, I have added thorough tests.

Testing

Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

/assign @xrajesh
ptal

/lgtm

/approve

authors: bparees xrajesh

repo: openshift/lightspeed-operator


text:
Make mongo work with docker exec -ti bash -c mongo

@mfojtik @hhorak @jhadvig

LGTM

good job @mnagy !

authors: mfojtik mnagy

repo: openshift/mongodb


text:
Make replicaset.spec.revisionHistoryLimit configurable

As documented, ReplicaSets can become left out in cluster
https://github.com/openshift/oadp-operator/blob/master/docs/install_olm.md#verify-install
This can reproduced when testing upgrades
NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/openshift-adp-controller-manager-686685cc8f   1         1         1       4m33s
replicaset.apps/openshift-adp-controller-manager-77c584bf8b   0         0         0       6m33s
replicaset.apps/velero-5fdf7d65df                             1         1         1       118s
replicaset.apps/velero-6bf5469f9c                             0         0         0       5m40s

These are by design by kube to be left over. You can set following if you don't like seeing "historical" replicasets.

Clean up Policy
You can set .spec.revisionHistoryLimit field in a Deployment to specify how many old ReplicaSets for this Deployment you want to retain. The rest will be garbage-collected in the background. By default, it is 10.

/remove-lifecycle stale

repo: openshift/oadp-operator


text:
OCPBUGS-7465: fix: changes oc version dependency

Description
Currently oc-mirror is hitting 401 status code (OCPBUGS-7465)  because there was no dynamic credential store in oc (parent dependency) to get a refresh token.
The code on this PR is based on oc PR #1540. Since there is no way in golang to refer to a code in a pull request the code from the PR was downloaded locally and then vendored in oc-mirror using replaces in go.mod. So it is expected that the tidy/sanity is going to fail in the bot tests on this PR (this PR is a working in progress and depends on the merge of PR #1540.
@zhouying7780 could you please check if the 401 is not happening anymore based on the binary generated from this PR? In order to build the binary please run only make build and do not use make tidy (otherwise the oc will be replaced by the one without the fix). In order to make sure the oc-mirror version you are using contains the oc fix I added the string Code based on PR #1540 https://github.com/openshift/oc/pull/1540, so please look for this in the logs.
After the confirmation from @zhouying7780 that the bug is not happening anymore, a request to merge PR #1540 will be sent for oc team.
When the PR #1540 is merged a regression test will be required in oc-mirror since some modules were updated in oc side which oc-mirror depends on.
cc:
ardaguclu
grokspawn
Fixes OCPBUGS-7465
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
The tests performed on this PR were:
make clean
make build
make test
make test-e2e
A basic mirroring of aws-load-balancer mirrorToDisk and diskToMirror was performed but a more robust one is needed
Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

/jira refresh

It is not required anymore since the issue was happening because of the clock out of sync.
For more information see: https://access.redhat.com/solutions/6966257

repo: openshift/oc-mirror


text:
OCPBUGS-23327: Fix MirrorToDisk of oci catalogs in hidden folders

Description
When constructing the mapping (and mapping.txt) for OCI catalogs, and calculating the destination for mirroring of related images of the catalog,  we don't add (nor later remove) the catalog path from the destination. This way, the destination obtained for a relatedImage is (supposed to be) same as for MirrorToDisk workflow with registry based catalogs. This fixes the issue in OCPBUGS-23327, since the . character is no longer part of the reference to the destination related image.
Fixes #OCPBUGS-23327
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
E2E and unit tests should pass.
Mirror to disk should pass for:

./bin/oc-mirror -c isc_23327.yaml file:///tmp/23327
with the following copied catalog
 skopeo copy docker://icr.io/cpopen/ibm-zcon-zosconnect-catalog@sha256:8d28189637b53feb648baa6d7e3dd71935656a41fd8673292163dd750ef91eec oci:///home/skhoury/.ibm-pak/data/publish/latest/catalog-oci/manifest-list --all --format v2s2
and the following image set config

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
mirror:
  operators:
  - catalog: oci:///home/skhoury/.ibm-pak/data/publish/latest/catalog-oci/manifest-list
    packages:
    - name: ibm-zcon-zosconnect
      channels:
      - name: v1.0
    full: true
    targetTag: 27ba8e
    targetCatalog: ibm-catalog

Expected Outcome
tar file should be correctly generated

/jira refresh

/label qe-approved

/lgtm

/cherry-pick release-4.15

authors: lmzuccarelli sherine-k zhouying7780


text:
add advanced upgrade service (AUS) CLI

the AUS CLI utility is an OCM CLI plugin to manage AUS upgrade policies on clusters and organizations in OCM
https://source.redhat.com/groups/public/sre/wiki/advanced_upgrade_service_aus
adding this to ocm-container is beneficial for AUS users, since they get the AUS plugin bundled with an up to date OCM utility. this way ocm-container can be used in CI systems where they validate their AUS policies and apply them after merge.
part of https://issues.redhat.com/browse/APPSRE-8538

Hi @geoberle, thanks for the PR!
Who is the intended audience for the AUS utility?  They majority of users of ocm-container are likely SREP, who don't really have anything to do with scheduling upgrades.  However, I think it's fair to add utilities others might find helpful as well.  We just want to balance how much may need to be in the "core" ocm-container, and what might be added by other teams using ocm-container as a base image to customize their own.
What are your thoughts on this?

authors: clcollins geoberle

repo: openshift/ocm-container


text:
show error when using odo watch on git components

What is the purpose of this change? What does it change?
Currently due to the check of source path existence using os.Stat being present before sourceType validation - odo watch showed directory doesn't exist error instead of watch not supported for git ... kind.
So resolved that and added a test.
Was the change discussed in an issue?
fixes https://github.com/openshift/odo/issues/2138

How to test changes?
odo create nodejs --git https://github.com/openshift/nodejs-ex.git
odo watch # should show an error for kind "Watch is supported by binary and local components only...."

/aprove

3rd time the charm?

/approve

/lgtm with respect to test

/lgtm

authors: amitkrout dharmit girishramnani

repo: openshift/odo


text:
Trello CLI stg

Signed-off-by: Aaron Weitekamp aweiteka@redhat.com

[test]

[test]

repo: openshift/openshift-tools


text:
Update stage console URL

Why?: The stage URL changed from https://qaprodauth.cloud.redhat.com/ to https://qaprodauth.console.redhat.com/

/lgtm

authors: typeid vkareh

repo: openshift/rosa


text:
关于@Cache 注解在 @Component 注解类下使用咨询

如下， @Component 注解类下使用咨询，MyService 中的 方法1 调用  方法2，方法2有注解 @Cache，实际上，在方法1调用 方法2 时候，注解没有生效，请问是设计如此的吗？
@Controller
public class MyController {
    @Inject
    private MyService myService;
  @Mapping("/路由1")
    @Post
    public Ret 路由方法1 () {
        return myService.方法1()
    }

}

@Component
public class MyService {

@Cache(key="方法1",second="100)
方法2 {
  return "缓存";
}

方法1 {
  return 方法2();
}

}

@Cache 及其它 Aop 的注解使用。会涉及一个动态代理的知识：
动态代理后，类实际会有两层：外层为代理类（处理Aop注解）；内层为真实类。。。。
平常调用是：使用者->(代理类->真实类)，此时Aop的注解会生效。。。
如果在真实类里，this 调用自己，就会是：真实类-》真实类。。。跳过了代理类。。。所以会失效。

所有的类代理，都差不多。。。如果想自己调用自己，仍估 AOP 有效。。。要使用：自己注入自己
@Component
public class MyService {
    @Inject
    MyService self;
}

使用注入的自己，来调用有 aop 需求的方法。

官网有详解：
https://solon.noear.org/article/442

authors: TreasureLand noear

repo: opensolon/solon


text:
Validate npm package ownership

What is it?
How I'd do it is:

User registers their package in oss-directory
Our GitHub validation app queries the npm server for the package manifest
We verify that the GitHub repo listed is also in the oss-directory project.yaml file
We piggyback on any validation we do for GitHub repo ownership

Duplicate of #2459

repo: opensource-observer/oso


text:
code style

code style

let’s try it!

authors: bigbeats withjiang