text:
Override toString for usable logs

Signed-off-by: Kunal Kotwani kkotwani@amazon.com
Description

The logs for the flakey test have the generic Java object hash representation
Adding in toString overrides for corresponding classes for usable logs

Issues Resolved

https://github.com/opensearch-project/OpenSearch/issues/1703

Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Right on time 🙂
  2> REPRODUCE WITH: ./gradlew ':qa:remote-clusters:integTest' --tests "org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks" -Dtests.seed=D94CE182D2D61A07 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ja-JP -Dtests.timezone=Asia/Aqtau -Druntime.java=17
  2> java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([D94CE182D2D61A07:DE3A3449E9506C5A]:0)
        at org.junit.Assert.fail(Assert.java:87)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.junit.Assert.assertTrue(Assert.java:53)
        at org.opensearch.cluster.remote.test.RemoteClustersIT.testHAProxyModeConnectionWorks(RemoteClustersIT.java:125)
  1> [2022-04-14T05:45:13,170][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] before test
  1> [2022-04-14T05:45:13,788][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] Configuring remote cluster [opensearch-2:9300]
  1> [2022-04-14T05:45:13,921][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] Connection info: RemoteConnectionInfo{modeInfo=SniffModeInfo{seedNodes=[opensearch-2:9300], maxConnectionsPerCluster=3, numNodesConnected=0}, initialConnectionTimeoutString='30s', clusterAlias='cluster2alt', skipUnavailable=false}
  1> [2022-04-14T05:45:14,107][INFO ][o.o.c.r.t.RemoteClustersIT] [testSniffModeConnectionFails] after test
  1> [2022-04-14T05:45:14,158][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] before test
  1> [2022-04-14T05:45:14,859][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] Configuring remote cluster [opensearch-2:9300]
  1> [2022-04-14T05:45:14,924][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] Connection info: RemoteConnectionInfo{modeInfo=ProxyModeInfo{address='opensearch-2:9300', serverName='', maxSocketConnections=18, numSocketsConnected=18}, initialConnectionTimeoutString='30s', clusterAlias='cluster2', skipUnavailable=false}
  1> [2022-04-14T05:45:15,178][INFO ][o.o.c.r.t.RemoteClustersIT] [testProxyModeConnectionWorks] after test
  2> NOTE: leaving temporary files on disk at: /var/CITOOL/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/search/qa/remote-clusters/build/testrun/integTest/temp/org.opensearch.cluster.remote.test.RemoteClustersIT_D94CE182D2D61A07-001
  2> NOTE: test params are: codec=Asserting(Lucene91): {}, docValues:{}, maxPointsInLeafNode=2010, maxMBSortInHeap=6.235365583756655, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=ja-JP, timezone=Asia/Aqtau
  2> NOTE: Linux 5.13.0-1014-aws amd64/Eclipse Adoptium 17.0.2 (64-bit)/cpus=72,threads=1,free=417673864,total=536870912
  2> NOTE: All tests run in this JVM: [RemoteClustersIT]
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/var/CITOOL/workflow/OpenSearch_CI/PR_Checks/Gradle_Check/search/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release

@kotwanikunal this looks like https://github.com/opensearch-project/OpenSearch/issues/1703. Should we start gradle check gain?

@kotwanikunal this looks like https://github.com/opensearch-project/OpenSearch/issues/1703. Should we start gradle check gain?

Ah, it got started by my comment. 😁


text:
[Extensions] Handle Deprecated and Replaced Routes in Extension Rest Handlers

Companion PR on SDK: https://github.com/opensearch-project/opensearch-sdk-java/pull/666
Description
Adds the ability to handle Deprecated and Replaced Routes to the ExtensionRestHandler.  This will permit (nearly) identical syntax for plugin migration, and provide backwards compatibility to extension users.

Routes were already registered.
Deprecated routes are just a wrapper around Routes, which add a log message.
Replaced routes are a fancy user interface to simultaneously register Routes and Deprecated Routes.

The OpenSearch integration adds a new list to the existing registration request.  In addition to the method and path sent for routes, deprecated routes have a deprecationMessage included.  This is included in a separate list where the message immediately follows the method/path.
Issues Resolved
Fixes SDK #665
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #7064 (94623f9) into main (0db20d9) will decrease coverage by 0.14%.
The diff coverage is 81.48%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #7064      +/-   ##
============================================
- Coverage     70.74%   70.61%   -0.14%
+ Complexity    59329    59281      -48
============================================
  Files          4825     4825
  Lines        284369   284390      +21
  Branches      41021    41024       +3
============================================
- Hits         201177   200808     -369
- Misses        66665    67059     +394
+ Partials      16527    16523       -4

Impacted Files
Coverage Δ

...ch/extensions/rest/RegisterRestActionsRequest.java
69.23% <54.54%> (-0.77%)
:arrow_down:

...rch/extensions/rest/RestSendToExtensionAction.java
41.50% <100.00%> (+9.64%)
:arrow_up:

... and 503 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

LGTM

source: gharchive/pull-request

authors: codecov-commenter dbwiddis vibrantvarun

repo: opensearch-project/OpenSearch

license_type: permissive

license_source: github-api


text:
2.x build fixes

Description
Describe what this change achieves.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@gaiksaya  can you please take a look at the maven action?

Codecov Report

:exclamation: No coverage uploaded for pull request base (2.x@6ff8e55). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head f8e09dd differs from pull request most recent head b75362f. Consider uploading reports for the commit b75362f to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@          Coverage Diff           @@
##             2.x     #266   +/-   ##
======================================
  Coverage       ?   83.36%
  Complexity     ?      554
======================================
  Files          ?       61
  Lines          ?     2050
  Branches       ?      146
======================================
  Hits           ?     1709
  Misses         ?      234
  Partials       ?      107

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: Arpit-Bandejiya codecov-commenter

repo: opensearch-project/asynchronous-search


text:
[DOC] Create documentation for _index APIs

What do you want to do?

[ ] Request a change to existing documentation
[x ] Add new documentation
[ ] Report a technical problem with the documentation
[ ] Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
 PUT /<index>/_create/
POST /<index>/_create/
HEAD /<index>/_doc/<id>
HEAD /<index>/_source/<id>
GET /<index>/_source/<id> // deprecated (per Nate)
PUT /<index>/_block/<block>
HEAD /<index>

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

PUT /<index>/_create/, POST /<index>/_create/,HEAD /<index>/_doc/<id> are documented here :https://opensearch.org/docs/latest/api-reference/document-apis/index-document/
HEAD /<index>/_source/<id>, GET /<index>/_source/<id> is documented but missing important description information: https://opensearch.org/docs/latest/api-reference/document-apis/get-documents/
PUT /<index>/_block/<block> is missing
HEAD /<index>: Exists https://opensearch.org/docs/latest/api-reference/index-apis/exists/

source: gharchive/issue

authors: Naarcha-AWS kolchfa-aws


text:
Modified the Readme.md for multiple problems

git commit -s -m “UpdatesToReadme
Signed-off-by: Heather Halter hdhalter@amazon.com
Related to https://github.com/opensearch-project/documentation-website/issues/531
Fixes:

Added link to style guide
Added link to terms
Removed the style text (since we point them to the style guide and terms now)
Removed 'trivial changes' section, as we don't want to allow that (it doesn't pass DCO check)
Added link to the contributions file
Other word tweaking

Things that still need work:

Everything from New Releases down needs to be reviewed. I'm not sure what that section is describing, as there's no context to it.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I mean everything between New Releases and Code of Conduct.

I mean everything between New Releases and Code of Conduct.

I think that's important content. "New releases" is internal information and we can take it out, but I think it's okay to keep it in the README. The other sections should stay in the README, though, because that's important information for anybody out there who wants to contribute to the repo.


text:
Add new rename_alias parameters for restore-snapshot

Description
Documents new parameters for restore-snapshot to be introduced in OpenSearch PR 16292.
Issues Resolved
No issue
Version
2.18
Frontend features
None
Checklist

[x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

The implementing PR for this change is not yet merged.

@mispencer: Is this update going into 2.18?

That depends on the reviewers of the main OpenSearch project, but I assume this will be in the next minor release, yes.

@Naarcha-AWS - I created the documentation of the new rename_alias_pattern and rename_alias_replacement parameters by copying and modifying the existing documentation of the rename_pattern and rename_replacement parameters. As such, many of your suggestions would also apply to that existing documentation. Do you want me to update the existing documentation with your suggestions as well?

@mispencer: That would be great!

Thanks @mispencer!

@natebower: This is ready for your review.

@natebower: Vale errors addressed.

authors: Naarcha-AWS mispencer


text:
[BUG] Multi-node integ tests are flaky

What is the bug?
Multi-node integ tests are failing randomly.   The common symptom is that the cluster does not appear to be started up when the first test(s) run, which leads to 1 in 3 chance of failure if the REST request hits a node that does not contain the primary shard for a queried index.
How can one reproduce the bug?
Occurs randomly on GitHub Actions. Latest failures :

https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911274115
https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911275308

What is the expected behavior?
Passing tests
What is your host/environment?
GitHub actions. This has occurred on all OS's.  So far we've only done multi-node tests on JDK 21, will continue to evaluate other JDKs.
Do you have any additional context?
The common line tends to be this exception, always on ::integTest-0 node.  Other nodes have failures ~20s later, associated with cluster manager (which was probably this node).
Note the sequence and timestamps involved here: we do an HTTP request here
1> [2024-01-26T16:24:31,158][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionLocalModelWorkflow] before test
  2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:37145], URI [/_plugins/_flow_framework/workflow/S90ER40BNKhClHx_ZtQu/_provision], status line [HTTP/1.1 500 Internal Server Error]
    {"error":"Cannot invoke \"String.getBytes(java.nio.charset.Charset)\" because \"src\" is null"}
        at __randomizedtesting.SeedInfo.seed([3366CB2459279EBB:C09AA671C3648979]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.flowframework.TestHelpers.makeRequest(TestHelpers.java:116)
        at app//org.opensearch.flowframework.TestHelpers.makeRequest(TestHelpers.java:89)
        at app//org.opensearch.flowframework.TestHelpers.makeRequest(TestHelpers.java:78)
        at app//org.opensearch.flowframework.FlowFrameworkRestTestCase.provisionWorkflow(FlowFrameworkRestTestCase.java:381)
        at app//org.opensearch.flowframework.rest.FlowFrameworkRestApiIT.testCreateAndProvisionRemoteModelWorkflow(FlowFrameworkRestApiIT.java:224)
  1> [2024-01-26T16:24:31,180][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionLocalModelWorkflow] initializing REST clients against [http://[::1]:38359, http://127.0.0.1:36629, http://[::1]:38133, http://127.0.0.1:37145/, http://[::1]:39713, http://127.0.0.1:40[83](https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911274115#step:4:84)5]
  1> [2024-01-26T16:24:41,246][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionLocalModelWorkflow] after test

We start the test at 16:24:31,158
The exception occurs at an unknown time (the log entry is prior to the initialization line)
We start initializing the REST clients at 16:24:31,180
We finish the test at 16:24:41.
Other tests seem to complete successfully.
Another example, where the exception appears after the "after test" log but was clearly thrown in the test:
  1> [2024-01-26T15:25:32,304][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testFailedUpdateWorkflow] before test
  1> [2024-01-26T15:25:32,304][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testFailedUpdateWorkflow] initializing REST clients against [http://[::1]:64663, http://127.0.0.1:64664,/ http://[::1]:64668, http://127.0.0.1:64669,/ http://[::1]:64675, http://127.0.0.1:64676]/
  1> [2024-01-26T15:25:52,299][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testFailedUpdateWorkflow] There are still tasks running after this test that might break subsequent tests [internal:index/shard/recovery/start_recovery].
  1> [2024-01-26T15:25:52,299][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testFailedUpdateWorkflow] after test
  2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:64669], URI [/_plugins/_flow_framework/workflow/el0FR40BN7x2uhV1Ihk7/_provision], status line [HTTP/1.1 500 Internal Server Error]
    {"error":"Cannot invoke \"String.getBytes(java.nio.charset.Charset)\" because \"src\" is null"}
        at __randomizedtesting.SeedInfo.seed([4FF4E0B23ACF7636:8C5D06D[74](https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911275308#step:4:75)3C59EC4]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:3[76](https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911275308#step:4:77))
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.flowframework.TestHelpers.makeRequest(TestHelpers.java:116)
        at app//org.opensearch.flowframework.TestHelpers.makeRequest(TestHelpers.java:89)
        at app//org.opensearch.flowframework.TestHelpers.makeRequest(TestHelpers.java:[78](https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911275308#step:4:79))
        at app//org.opensearch.flowframework.FlowFrameworkRestTestCase.provisionWorkflow(FlowFrameworkRestTestCase.java:3[81](https://github.com/opensearch-project/flow-framework/actions/runs/7671893205/job/20911275308#step:4:82))
        at app//org.opensearch.flowframework.rest.FlowFrameworkRestApiIT.testFailedUpdateWorkflow(FlowFrameworkRestApiIT.java:92)

Again, later tests complete successfully.
Occasionally we've seen multiple test failures but it's still the "first N" failures followed by success.
These are different internal tests, indicating some other commonality.
I'll be investigating these client connections and means of confirming they're connected before running tests.

Set up a test to run 15 times (all JDKs from 17 to 21 against 3 OS's).  Failures on JDKs 17, 18, 19, and 21 on macOS, and 21 on Windows.
Consistently failing some of the tests.  So far 5 out of 6 match the above log style with the null src.  One other type:
org.opensearch.flowframework.rest.FlowFrameworkRestApiIT > testCreateAndProvisionRemoteModelWorkflow FAILED
    org.junit.ComparisonFailure: expected:<[COMPLET]ED> but was:<[FAIL]ED>
        at __randomizedtesting.SeedInfo.seed([AFD09CC584B20298:5C2CF1901EF1155A]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.opensearch.flowframework.FlowFrameworkRestTestCase.getAndAssertWorkflowStatus(FlowFrameworkRestTestCase.java:[57](https://github.com/dbwiddis/flow-framework/actions/runs/7673708105/job/20917202324#step:4:58)5)
        at org.opensearch.flowframework.FlowFrameworkRestTestCase.lambda$getResourcesCreated$7(FlowFrameworkRestTestCase.java:602)
        at org.opensearch.test.OpenSearchTestCase.assertBusy(OpenSearchTestCase.java:1089)
        at org.opensearch.flowframework.FlowFrameworkRestTestCase.getResourcesCreated(FlowFrameworkRestTestCase.java:601)
        at org.opensearch.flowframework.rest.FlowFrameworkRestApiIT.testCreateAndProvisionRemoteModelWorkflow(FlowFrameworkRestApiIT.java:231)

  1> [2024-01-26T22:[59](https://github.com/dbwiddis/flow-framework/actions/runs/7673708105/job/20917202324#step:4:60):32,181][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionRemoteModelWorkflow] before test

  1> [2024-01-26T22:59:32,184][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionRemoteModelWorkflow] initializing REST clients against [http://[::1]:49243, http://127.0.0.1:49244,/ http://[::1]:49247, http://127.0.0.1:49248,/ http://[::1]:49251, http://127.0.0.1:49252]/
Suite: Test class org.opensearch.flowframework.rest.FlowFrameworkRestApiIT
  1> [2024-01-26T23:00:24,111][INFO ][o.o.f.r.FlowFrameworkRestApiIT] [testCreateAndProvisionRemoteModelWorkflow] after test

Commonality is still that this is the first test executed and the one that initializes the REST clients.

The client initialization log message comes from initClient() in OpenSearchRestTestCase which has @Before JUnit annotation.
However, in the multi-node case, there's no guarantee that the other node(s) have finished their initialization before we try to call them.
I still think this points to the build.gradle not waiting for the correct conditions before starting these tests.

repo: opensearch-project/flow-framework


text:
Incorrect indentation for extraVolumeMounts, extraEnvs, envFrom in statefulset.yaml.

Signed-off-by: Michael Primeaux michael.primeaux@mac.com
Description
Describe the bug
The Stateful Set template specifies an incorrect indentation of 10 for extraEnvs, extraVolumeMounts, and envFrom. The correct value is 8.

Error

Error: INSTALLATION FAILED: YAML parse error on logging/charts/opensearch/templates/statefulset.yaml: error converting YAML to JSON: yaml: line 122: did not find expected key
make: *** [install] Error 1

To Reproduce
Steps to reproduce the behavior:
Specify values for the extraEnvs stanza in the values.yaml file for the OpenSearch chart. For example,

extraEnvs Example

extraEnvs:
  - name: NODE_TLS_REJECT_UNAUTHORIZED
    value: "0"

Expected behavior
The expected behavior is for the chart to install.
Chart Name
Specify the Chart which is affected? OpenSearch
Screenshots
None
Host/Environment (please complete the following information):

Helm Version: v3.7.0
Kubernetes Version: 1.21 (Amazon EKS)

Additional context
None.
Issues Resolved
This will resolve issues #64 and #65
Check List

[✔] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@TheAlgo This PR replaces PR #63.

@smlx Do you have a few moments to review this PR given we've got quite a few folks waiting for it? Appreciate your support.

@DandyDeveloper @TheAlgo @smlx @peterzhuamazon Checks have passed for 1.0.4. Thanks for your support. We'll use this version for our next production release.

Thanks all. I have merged the PR.

authors: TheAlgo mprimeaux

repo: opensearch-project/helm-charts


text:
Security enabled test

Issue #, if available:
Description of changes:
Make sure we have the function of running tests against a security enabled cluster.
Commands:
./gradlew run -Dsecurity=true // bring up a security enabled cluster with ISM installed

./gradlew integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername=elasticsearch -Dhttps=true  // running test against remote security enabled domain

CheckList:

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This seems to not allow us to run security enabled integ tests in our own repo in our own workflow which is something we want. Create a new issue to get that working if it's broken.

Check if this allows the infra security enabled tests to run successfully

Codecov Report

Merging #93 (3fe69e1) into main (d415a03) will decrease coverage by 0.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main      #93      +/-   ##
============================================
- Coverage     77.95%   77.88%   -0.07%
- Complexity     1870     1876       +6
============================================
  Files           255      255
  Lines         10378    10378
  Branches       1534     1534
============================================
- Hits           8090     8083       -7
+ Misses         1464     1461       -3
- Partials        824      834      +10

Impacted Files
Coverage Δ

...statemanagement/model/destination/CustomWebhook.kt
65.21% <0.00%> (-28.99%)
:arrow_down:

...anagement/indexstatemanagement/model/Transition.kt
64.61% <0.00%> (-3.08%)
:arrow_down:

...ndexstatemanagement/IndexStateManagementHistory.kt
80.32% <0.00%> (-1.64%)
:arrow_down:

...nt/indexstatemanagement/ManagedIndexCoordinator.kt
73.72% <0.00%> (-0.43%)
:arrow_down:

...agement/indexstatemanagement/ManagedIndexRunner.kt
54.82% <0.00%> (+0.50%)
:arrow_up:

...ent/indexstatemanagement/util/ManagedIndexUtils.kt
78.69% <0.00%> (+1.30%)
:arrow_up:

...dexmanagement/transform/model/TransformMetadata.kt
90.36% <0.00%> (+3.61%)
:arrow_up:

...ent/rollup/action/explain/ExplainRollupResponse.kt
85.00% <0.00%> (+10.00%)
:arrow_up:

...ansform/action/explain/ExplainTransformResponse.kt
77.77% <0.00%> (+11.11%)
:arrow_up:

...nt/indexstatemanagement/model/destination/Chime.kt
54.54% <0.00%> (+13.63%)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d415a03...3fe69e1. Read the comment docs.

repo: opensearch-project/index-management


text:
Release version 2.1.0

This is a component issue for 2.1.0.
Coming from opensearch-build#1818. Please follow the following checklist.
Please refer to the DATES in that post.
How to use this issue

This Component Release Issue
This issue captures the state of the OpenSearch release, on component/plugin level; its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help.
Any release related work can be linked to this issue or added as comments to create visiblity into the release status.
Release Steps
There are several steps to the release process; these steps are completed as the whole component release and components that are behind present risk to the release. The component owner resolves the tasks in this issue and communicate with the overall release owner to make sure each component are moving along as expected.
Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release. The most current set of dates is on the overall release issue linked at the top of this issue.
The Overall Release Issue
Linked at the top of this issue, the overall release issue captures the state of the entire OpenSearch release including references to this issue, the release owner which is the assignee is responsible for communicating the release status broadly.  Please contact them or @mention them on that issue for help.
What should I do if my plugin isn't making any changes?
If including changes in this release, increment the version on 2.0 branch to 2.1.0 for Min/Core, and 2.1.0.0 for components. Otherwise, keep the version number unchanged for both.

Preparation

[x] Assign this issue to a release owner.
[x] Finalize scope and feature set and update the Public Roadmap.
[ ] All the tasks in this issue have been reviewed by the release owner.
[ ] Create, update, triage and label all features and issues targeted for this release with v2.1.0.

CI/CD

[ ] All code changes for 2.1.0 are complete.
[ ] Ensure working and passing CI.
[ ] Check that this repo is included in the distribution manifest.

Pre-Release

[ ] Update to the 2.1.0 release branch in the distribution manifest.
[ ] Increment the version on the parent branch to the next development iteration.
[ ] Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
[ ] Confirm that all changes for 2.1.0 have been merged.
[ ] Add this repo to the manifest for the next developer iteration.

Release Testing

[ ] Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
[ ] Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
[ ] Sanity Testing: Sanity testing and fixing of critical issues found.
[ ] File issues for all intermittent test failures.

Release

[ ] Complete documentation.
[ ] Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

[ ] Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
[ ] Add this repo to the manifest of the next eventual security patch version.
[ ] Suggest improvements to this template.
[ ] Conduct a retrospective, and publish its results.

Hi @qreshi ,
We found below issues tagged with v2.1.0 that were not closed or resolved in this release. Can you take a look and close the ones that are resolved or retag them with next version you see fit?
Thanks!
#301  Add backwards compatibility tests for automation  compatibility, v2.1.0  about 1 month ago
#258  Add security integration tests                    v2.1.0                 about 1 month ago

repo: opensearch-project/notifications


text:
[BUG] Unable to install OpenSearch from repo since SHA-1 for signing is deprecated in RHEL9

Describe the bug
Unable to install OpenSearch using yum/dnf in RHEL9 distros due to SHA-1 for signing being deprecated
https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9
To Reproduce
Steps to reproduce the behavior:

Following docs from https://opensearch.org/docs/latest/install-and-configure/install-opensearch/rpm/#install-opensearch-from-a-yum-repository
run sudo curl -SL https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/opensearch-2.x.repo -o /etc/yum.repos.d/opensearch-2.x.repo
dnf clean all
dnf update
dnf install opensearch-2.0.1

Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <opensearch@amazon.com>"
 Fingerprint: C5B7 4989 65EF D1C2 924B A9D5 39D3 1987 9310 D3FC
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
warning: Signature not supported. Hash algorithm SHA1 not available.
Key import failed (code 2). Failing package is: opensearch-2.0.1-1.x86_64
 GPG Keys are configured as: https://artifacts.opensearch.org/publickeys/opensearch.pgp
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

Expected behavior
To be able to install OpenSearch from repo using a package manager on RHEL9 base distros
Plugins
N/A
Screenshots
N/A
Host/Environment (please complete the following information):

OS: Rocky Linux 9
Version 9.1 (Blue Onyx)

Additional context
GPG key and packages should be signed using SHA256 or SHA521.

This issue has been tracked at https://github.com/opensearch-project/opensearch-build/issues/3124

Closing this issue as duplicate of https://github.com/opensearch-project/opensearch-build/issues/3124

authors: ObjectivelyAj bbarani jordarlu

repo: opensearch-project/opensearch-build


text:
OSD Version Increment

Description
Add workflow for OpenSearch Dashboard Plugins Version Increment.
This Workflow uses OSD plugin-helpers version.
Issues Resolved
https://github.com/opensearch-project/opensearch-build/issues/3333
Sample PR: https://github.com/prudhvigodithi/alerting-dashboards-plugin/pull/1/files
Sample Workflow RUN: https://github.com/prudhvigodithi/opensearch-build/actions/runs/4868420202/jobs/8681889881
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #3466 (9b6ba2d) into main (1c6da01) will decrease coverage by 0.27%.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main    #3466      +/-   ##
==========================================
- Coverage   91.81%   91.55%   -0.27%
==========================================
  Files         181      181
  Lines        5268     5268
==========================================
- Hits         4837     4823      -14
- Misses        431      445      +14

see 2 files with indirect coverage changes

authors: codecov-commenter prudhvigodithi


text:
Parameterize SettingsUpdateConsumer to avoid need to cast

Signed-off-by: Daniel Widdis widdis@gmail.com
Description
Improves on the ClusterService class committed in PR #344 by adding a type parameter indicating that the setting and consumer are the same type.
This avoids the need to cast the consumer type when there is only one pair of arguments.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #352 (1a2861c) into main (4e9416a) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #352   +/-   ##
=========================================
  Coverage     69.35%   69.35%
  Complexity      126      126
=========================================
  Files            27       27
  Lines           558      558
  Branches         15       15
=========================================
  Hits            387      387
  Misses          162      162
  Partials          9        9

Impacted Files
Coverage Δ

...ain/java/org/opensearch/sdk/SDKClusterService.java
100.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter dbwiddis

repo: opensearch-project/opensearch-sdk-java


text:
Cms/events/2024 02 07 kubecon cloudnativecon europe 2024

Description
KubeCon / CloudNativecon 2024 Community calendar entry
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Sorry about the weird branch name. The filename used for the event is correct.

repo: opensearch-project/project-website


text:
fix integtests after add latest sigma rules

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[X] New functionality includes testing.

[X] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This just deleted failing tests? Why?

hi @dblock , the integration tests are not deleted. the old sigma rules are deleted. because we updated our repo with latest sigma rules & some had only name changes.

repo: opensearch-project/security-analytics


text:
Dropdown list to set Backend role for Internal User not available

Opendistro-for-elasticsearch version 0.9.0
Ubuntu 18.04.02
While setting up backend role for internal users, dropdown list will ensure that there is no spelling mistakes and make it easier for admin to setup roles.

Apologies for the very late response here. If this is still an issue in the newer versions, please re-open pointing to the spelling mistake that needs fixing. Thanks!

authors: davidlago matsyavatar


text:
Add retry and enable UTs for Windows

Description
Added a retry around the the bootstrap that previously failed due to network issues.  Ran the new version of this workflow 100 iterations on linux and windows, both passed at 100% [1].
Without the retry during a run of 100 iterations [2], the windows test failed once during the bootstrap.  If we see future issues, we could attempt to add retries to them following this same pattern.
[1] https://github.com/peternied/security-dashboards-plugin/actions/runs/3449518073
[2] https://github.com/peternied/security-dashboards-plugin/actions/runs/3449040181
Issues Resolved

Resolves #1211
Resolves #1195

Check List

[ ] New functionality includes testing
[ ] New functionality has been documented
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1215 (48abaa3) into main (257325c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1215   +/-   ##
=======================================
  Coverage   71.78%   71.78%
=======================================
  Files          88       88
  Lines        2027     2027
  Branches      269      269
=======================================
  Hits         1455     1455
  Misses        509      509
  Partials       63       63

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter peternied


text:
feat(*): adds feature flag for retry policy

Description:
Adds a feature flag for enabling retry policy
Signed-off-by: Shalier Xia shalierxia@microsoft.com

Testing done:

Affected area:

Functional Area

New Functionality
[x ]

CI System
[ ]

CLI Tool
[ ]

Certificate Management
[ ]

Control Plane
[ ]

Demo
[ ]

Documentation
[ ]

Egress
[ ]

Ingress
[ ]

Install
[ ]

Networking
[ ]

Observability
[ ]

Performance
[ ]

SMI Policy
[ ]

Security
[ ]

Sidecar Injection
[ ]

Tests
[ ]

Upgrade
[ ]

Other
[ ]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project?no

Did you notify the maintainers and provide attribution?

Is this a breaking change?no

Codecov Report

Merging #4029 (e4e1308) into main (fc10617) will decrease coverage by 0.04%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #4029      +/-   ##
==========================================
- Coverage   67.85%   67.81%   -0.05%
==========================================
  Files         205      205
  Lines       11570    11570
==========================================
- Hits         7851     7846       -5
- Misses       3668     3673       +5
  Partials       51       51

Flag
Coverage Δ

unittests
67.81% <ø> (-0.05%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/crdconversion/crdconversion.go
69.56% <0.00%> (-5.44%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc10617...e4e1308. Read the comment docs.

authors: codecov-commenter shalier

repo: openservicemesh/osm


text:
use the CODEGEN_PKG variable

I read the https://blog.openshift.com/kubernetes-deep-dive-code-generation-customresources/ blog post and noticed that the script only works using vendoring. The CODEGEN_PKG variable is set but not used to call the generator script.

/lgtm
/approve

@sttts I think on this repo you have to merge it manually :)

authors: sttts unly


text:
[release-v1.11] Allow configuring whether to allow cross namespaces Brokers configuration references (#7455)

Cherry-pick of https://github.com/knative/eventing/commit/32d7dd83e76417f75daaa1e68c7aa2b89b14f218

/cherry-pick release-v1.12

/cc @creydr @Cali0707 @matzew

/retest-required

/test 413-test-reconciler-aws-ocp-413

/test 413-test-reconciler-aws-ocp-413

/cc @matzew @Cali0707
Can you PTAL ?

/lgtm

authors: Cali0707 matzew pierDipi

repo: openshift-knative/eventing


text:
[KNI][release-4.12][manual] add support for feature gates

enablement and reporting for kni-specific feature gates.

backport of #147

repo: openshift-kni/scheduler-plugins


text:
Set readOnlyRootFilesystem as true for PAC resources

Setting Controller, Webhook and Watcher Deployments
securityContext readOnlyRootFilesystem to true
to increase the security and to avoid being flagged
by the security scanner.
Signed-off-by: Savita Ashture sashture@redhat.com
Changes

Submitter Checklist

[ ] 📝 Please ensure your commit message is clear and informative. For guidance on crafting effective commit messages, refer to the How to write a git commit message guide. We prefer the commit message to be included in the PR body itself rather than a link to an external website (ie: Jira ticket).

[ ] ♽ Before submitting a PR, run make test lint to avoid unnecessary CI processing. For an even more efficient workflow, consider installing pre-commit and running pre-commit install in the root of this repository.

[ ] ✨ We use linters to maintain clean and consistent code. Please ensure you've run make lint before submitting a PR. Some linters offer a --fix mode, which can be executed with the command make fix-linters (ensure markdownlint and golangci-lint tools are installed first).

[ ] 📖 If you're introducing a user-facing feature or changing existing behavior, please ensure it's properly documented.

[ ] 🧪 While 100% coverage isn't a requirement, we encourage unit tests for any code changes where possible.

[ ] 🎁 If feasible, please check if an end-to-end test can be added. See README for more details.

[ ] 🔎 If there's any flakiness in the CI tests, don't necessarily ignore it. It's better to address the issue before merging, or provide a valid reason to bypass it if fixing isn't possible (e.g., token rate limitations).

/lgtm

/test

authors: savitaashture zakisk

repo: openshift-pipelines/pipelines-as-code


text:
[resolver] Skip trying to resolve Pipeline bundle

Don't error out if we have a bundle in a pipelineRef.
cc @vdemeester @sbose78
Closes #419
Signed-off-by: Chmouel Boudjnah chmouel@redhat.com

Changes

Submitter Checklist

[X] ♽  Run make test lint before submitting a PR (ie: via pre-push github hook no need to waste CPU cycle on CI
[X] 📖 If you are adding a user facing feature or make a change of the behavior, please make sure to document it
[X] 🧪 100% coverage is not a target but most of the time we would rather have a unit test if you make a code change.
[X] 🎁 If that's something that is possible to do please make sure to check if we can add a e2e test.
[X] 🔎 If there is a flakiness in the CI tests then make sure to get the flakyness fixed before merging or if that's not possible there is a good reason to bypass it.

See the developer guide for a bit more details.

Codecov Report

Merging #427 (1c89283) into main (86c6059) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #427      +/-   ##
==========================================
+ Coverage   70.56%   70.63%   +0.06%
==========================================
  Files          50       50
  Lines        2633     2629       -4
==========================================
- Hits         1858     1857       -1
+ Misses        589      587       -2
+ Partials      186      185       -1

Impacted Files
Coverage Δ

pkg/resolve/resolve.go
88.88% <100.00%> (ø)

pkg/pipelineascode/pipelineascode.go
61.68% <0.00%> (+1.32%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86c6059...1c89283. Read the comment docs.

authors: chmouel codecov-commenter


text:
Possible discrepancy in path of files of gitops/ inside ./ckcp/run.sh

Getting a file doesn't exist error while running ./ckcp/run.sh.

To recreate the error:
Do a ./ckckp/run.sh

Possible fixes

I can see this was due to this from last commit only.
Just changing that will do it.
@bnallapeta, or am I missing something while running run.sh.

Getting a file doesn't exist error while running ./ckcp/run.sh.

To recreate the error:
Do a ./ckckp/run.sh
Possible fixes

I can see this was due to this from last commit only.
Just changing that will do it.
@bnallapeta, or am I missing something while running run.sh.

@RAMESSESII2 Yes, I had actually commented this on the PR. Guess it was missed. Will fix this in the next PR.

Fixed with https://github.com/openshift-pipelines/pipelines-service/pull/30

repo: openshift-pipelines/pipelines-service


text:
WIP: Test RHODS 1.16

/env ODS_QE_CATALOG_IMAGE=quay.io/modh/rhods-operator-live-catalog
/env ODS_QE_CATALOG_IMAGE_TAG=1.16.0-rhods-4760

/test ods-nb-ux-e2e-on-ocp
/env ENABLE_AUTOSCALER=1

/test ods-nb-ux-on-ocp
/env ENABLE_AUTOSCALER=1

/test ods-plot-nb-ux-on-ocp

/test ods-nb-ux-e2e-on-ocp
/env ENABLE_AUTOSCALER=1
/env ODS_CI_NB_USERS=40

/test ods-nb-ux-on-ocp
/env ENABLE_AUTOSCALER=1
/env ODS_CI_NB_USERS=40

/test ods-plot-nb-ux-on-ocp

/test ods-plot-nb-ux-on-ocp
/env MATBENCH_DATA_URL
/test ods-nb-ux-on-ocp
/env ENABLE_AUTOSCALER=1
/env ODS_CI_NB_USERS=40

/test ods-plot-nb-ux-on-ocp
/var MATBENCH_DATA_URL
/test ods-nb-ux-on-ocp
/var ENABLE_AUTOSCALER=1
/var ODS_CI_NB_USERS=40

/test ods-plot-nb-ux-on-ocp
/var MATBENCH_DATA_URL https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/497/pull-ci-openshift-psap-ci-artifacts-master-ods-nb-ux-on-ocp/1567543422959489024/artifacts/nb-ux-on-ocp/test/artifacts/
/test ods-nb-ux-on-ocp
/var ENABLE_AUTOSCALER=1
/var ODS_CI_NB_USERS=40

repo: openshift-psap/ci-artifacts


text:
import yaml issue

when i try to run command on regular one i see the issue
[root@ip-10-201-36-93 cerberus]# python3 start_cerberus.py --config=config/config.yaml
Traceback (most recent call last):
File "start_cerberus.py", line 5, in
import yaml
ImportError: No module named 'yaml'
[root@ip-10-201-36-93 cerberus]#
[root@ip-10-201-36-93 cerberus]#
[root@ip-10-201-36-93 cerberus]#
[root@ip-10-201-36-93 cerberus]#
[root@ip-10-201-36-93 cerberus]#

@satishchennu1 Thanks for reporting the issue, https://github.com/openshift-scale/kraken/pull/52 fixes it. Thanks.

repo: openshift-scale/kraken


text:
Update generated XML

Signed-off-by: Steve Kuznetsov skuznets@redhat.com
Fixes from https://github.com/openshift/aos-cd-jobs/pull/924
/cc @sdodson @ashcrow

@stevekuznetsov sorry about that

authors: sdodson stevekuznetsov

repo: openshift/aos-cd-jobs


text:
Removed usages of deprecated io/ioutil

SA1019: "io/ioutil" has been deprecated since Go 1.19: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details.

/cc @nmagnezi

repo: openshift/appliance


text:
fix: update cad dashboard title

updating the title and grafana directory so the dashboard can be found

/lgtm
/approve

authors: ninabauer shibumi


text:
Add option to tag image using image reference

Hi, this PR is to add the possibility to tag an image with the OpenShiftImageTagger in ImageStreamImage mode using the reference of an image and not a tag.

Hey @EtienneCoutaud
So you'll need to rebase a bit.
First, on Thursday PM USA time, I checked in a change to split out the image stream name and tag name into separate fields / instance variables.  Your change looks like it is based on the old form.  So you'll need to resolve those conflicts.
Then after you opened this PR, based on input from the openshift-restclient-java folks the decision was made to go with this PR instead of yours for the openshift-restclient-java change.  So you'll need to adjust this PR to consume the similar but not exactly the same API.
Lastly, I'm wondering if we can discern between ImageStreamImage and ImageStreamTag without having the user specify with a new config panel.  Here's what I'm thinking (though have not coded it up to confirm it is correct):

Call IImageStream.getImageId(String tagName) (on source image stream) with the parameter provided.
If it returns something, the tagName provided must be an ImageStreamTag
Otherwise, we can assume it is a ImageStreamImage
Can you try that out and see if it works?  If it does not work, we'll go with the new UI field you added.

thanks

Hi @gabemontero
Okay I will try this using your change and the PR for the openshift-restclient-java to adjust with the new API.
Discerning between ImageStreamImage and ImageStreamTag without having the user specifiy is a good idea. I will implement that and see if it works

Hey @EtienneCoutaud - if you don't think you'll get back to this, and are still interested, I can convert this to an issue for us to track and close out this PR.  We have reworked the tagging code a bit since you originally created this pull.   At the moment, you specify the tag name, and we translate it to a ImageStreamImage ref.  Longer term, I'm thinking we add a toggle to choose between ImageStreamImage and ImageStreamTag ref (i.e. the equivalent the --alias option).

Hey @EtienneCoutaud - I've gone ahead and commited this enhancement via https://github.com/openshift/jenkins-plugin/commit/486f69bd22bfdab4ef72e6ba20ed0b682db61b2e, and as such, will close out this PR.  If you are at all still interested in this, please pull down the latest version and give it a try.
Thanks.

authors: EtienneCoutaud gabemontero

repo: openshift/jenkins-plugin

license_source: bigquery


text:
Add guard controller

This commit adds a guard controller which ensures
that the PDB is available and there is a guard
deployment associated with it.
Things which I deviated from original design:

I expect PDBs to be created via static resources, no updates to them
The guard controller should be smart enough to figure out the workload controller(deployment controller for now) that it should act upon based on selectors.
The guard controller should take an argument createGuardingWorkload which tells it if it needs to create a guarding controller in case of no workload controller. This solves the static pod scenarios where it will create static pods.
If the PDB has been applied on the workload which runs as a deployment, the guard controller should identify it based on the selector
Finally, guard controller should be able to update replicas of deployment based on number of master nodes.

cc @deads2k

/close

repo: openshift/library-go


text:
OCPBUGS-29548: IPI install fails on Nutanix when using DHCP

OCPBUGS-29548: IPI install fails with etcdInsufficientMembers on Nutanix when using DHCP network
The back-port of the PR https://github.com/openshift/machine-api-provider-nutanix/pull/66 to branch release-4.15.

/retest

/jira refresh

/label cherry-pick-approved

@JoelSpeed This back-port requires the "staff-eng-approved" label to merge. Can you add this label?

@sdodson Any idea why we need staff eng approval on backports at the moment?

/tide refresh

repo: openshift/machine-api-provider-nutanix


text:
Enable CEE to check etcd health

Close OSD-8444

/lgtm

repo: openshift/managed-scripts


text:
Fixes https://github.com/openshift/mysql/issues/136 add more binl…

…og settings

[test]

@christian-posta can you update the two README files w/ the new env vars you're adding?

@bparees done. thanks!

lgtm

authors: bparees christian-posta

repo: openshift/mysql


text:
fix: OADP-2094 Always check restore resource priorities

Address Tiger's comment https://github.com/openshift/oadp-operator/pull/1132#discussion_r1303365262

/ok-to-test

/override ci/prow/4.11-operator-e2e-azure

/retest

/lgtm

/override ci/prow/4.11-operator-e2e-azure

authors: kaovilai mateusoliveira43 weshayutin

repo: openshift/oadp-operator


text:
Fixes debug info test for windows

What type of PR is this?
/kind flake
What does does this PR do / why we need it:
It uses a free port from the system for the debug info unit test
How to test changes / Special notes to the reviewer:

the debug info unit test should pass on windows

/retest
/approve

/lgtm

Installing from release registry.svc.ci.openshift.org/ci-op-n19y3xw2/release@sha256:0650daa5ce4b9dc1789b3ee87ef24169d0e8ff2d192522161107f6a9e9252c07
AWS region: us-east-1 (zones: us-east-1b us-east-1c)
level=fatal msg="failed to fetch Terraform Variables: failed to fetch dependency of \"Terraform Variables\": failed to fetch dependency of \"Bootstrap Ignition Config\": failed to fetch dependency of \"Common Manifests\": failed to generate asset \"DNS Config\": getting public zone for \"origin-ci-int-aws.dev.rhcloud.com\": listing hosted zones: Throttling: Rate exceeded\n\tstatus code: 400, request id: 2a4cc475-37e4-42da-abfb-a2a210d8d834"

/retest

/retest

authors: kadel mik-dass prietyc123

repo: openshift/odo


text:
Uses the deployment while listing storage from the cluster

What type of PR is this?
/kind bug
/kind design
What does this PR do / why we need it:
It uses the deployment while listing and storage from the cluster.
Which issue(s) this PR fixes:
Fixes #4917
PR acceptance criteria:

[x] Unit test

[ ] Integration test

[ ] Documentation

[ ] Update changelog

[ ] I have read the test guidelines

How to test changes / Special notes to the reviewer:

All tests should pass.

@mik-dass: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name 	Commit 	Details 	Rerun command
ci/prow/v4.8-integration-e2e 	9781f79 link /test v4.8-integration-e2e
Full PR test history. Your PR dashboard.

• Failure [0.980 seconds]
odo project command tests
/go/src/github.com/openshift/odo/tests/integration/project/cmd_project_test.go:12
  should list current empty project in json format [It]
  /go/src/github.com/openshift/odo/tests/integration/project/cmd_project_test.go:51

#4960

  should list current empty project in json format [It]
  /go/src/github.com/openshift/odo/tests/integration/project/cmd_project_test.go:51
  Expected

      <string>: {"apiVersion":"odo.dev/v1alpha1","items":[{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test26ozz","namespace":"cmd-project-test26ozz"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test41hlw","namespace":"cmd-project-test41hlw"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test46esq","namespace":"cmd-project-test46esq"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test60gaw","namespace":"cmd-project-test60gaw"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test74uzs","namespace":"cmd-project-test74uzs"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test88ava","namespace":"cmd-project-test88ava"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"gcrgzf","namespace":"gcrgzf"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"myproject","namespace":"myproject"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"nodejs-12","namespace":"nodejs-12"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"nodejs-12-rhel7","namespace":"nodejs-12-rhel7"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"nodejs-14","namespace":"nodejs-14"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"odo-operator-test","namespace":"odo-operator-test"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"openjdk-11","namespace":"openjdk-11"},"spec":{},"status":{"active":false}},{"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"openjdk-11-rhel8","namespace":"openjdk-11-rhel8"},"spec":{},"status":{"active":false}}],"kind":"List","metadata":{}}
  to contain substring
      <string>: {"apiVersion":"odo.dev/v1alpha1","kind":"Project","metadata":{"creationTimestamp":null,"name":"cmd-project-test51vha","namespace":"cmd-project-test51vha"},"spec":{},"status":{"active":true}}
  /go/src/github.com/openshift/odo/tests/integration/project/cmd_project_test.go:57

/retest

authors: dharmit mik-dass


text:
Additional changes for SSL enabling the api and console ports

Builds upon https://github.com/openshift/openshift-online-ansible/pull/69 by adding KUBECONFIG to .bash_profile for the root user.

roles/base_os: Without this, the root user would need to manually configure this variable before attempting to run any osc commands
roles/base_os: Cleanup the firewall service definition and only pause when the service state changes.
roles/openshift_master: use Akram's suggestion of simplifying the firewall config
roles/openshift_master: explicitly disable previously exposed ports that are no longer exposed (8080/tcp I'm looking at you).

@jwhonce

repo: openshift/openshift-online-ansible


text:
[prod] add zabbix key and trigger for new check

prod PR for https://github.com/openshift/openshift-tools/pull/4240

merge this as #4240 approved

repo: openshift/openshift-tools


text:
[WIP] Pull secret hardcoded

add support for using hardcoded global pull secret (openshift-config/pull-secret) for pulling catalog image

/retest

/retest

/retest

/retest

/approve

/approve
/label backport-risk-assessed

authors: ankitathomas joelanford tmshort

repo: openshift/operator-framework-catalogd


text:
image source: avoid shadowing /bin with emptyDir mount

Bringing the fix from @joelanford downstream
https://github.com/operator-framework/rukpak/pull/565.

This blocks the payload introduction work which will follow the no-ff process.
/label px-approved
/label qe-approved
/label docs-approved

authors: timflannagan tylerslaton

repo: openshift/operator-framework-rukpak


text:
Generate Generic and GitHub webhooks for builds

cc @ncameronbritt @rhamilto

@jeff-phillips-18 PTAL

Thanks @jeff-phillips-18

repo: openshift/origin-web-catalog


text:
Adding Apache 2.0 license

fixes #104

/assign @smarterclayton

/lgtm

authors: bradmwilliams smarterclayton

repo: openshift/release-controller


text:
OCPQE-18902 will not fail the cmd when cli cannot access embargoed jira issue

if the jira issue is secured/embargoed, jira token used by cli does not have permission to access it. won't fail the cmd, continue updating the other jira issues in report.

/assign @ming1013

/lgtm

/approve

repo: openshift/release-tests


text:
OCM-5088 | fix: if no subnets available refer to doc

Related issue: OCM-5088

lgtm

/lgtm

FYI @qiujian16

authors: andreadecorte gdbranco mikeshng thomasmckay

repo: openshift/rosa


text:
Add initial runbooks for all critical CMO alerts

This adds at least a basic outline of a runbook for every critical alert shipped
by the cluster-monitoring-operator as of the OpenShift 4.9 release.  Most of
these are very incomplete, but this is a starting point.
Context: https://issues.redhat.com/browse/MON-1645

/assign @RiRa12621
/cc @openshift/sre-alert-sme

Hey @RiRa12621, let me know if you have a chance to look at this. If we want the critical alerts to link to runbooks in the 4.9 release, we have to get it all merged this week unfortunately.

Hey @bison i just got back from two weeks of PTO and will make sure to get this reviewed latest EOD tomorrow

Hey @bison i just got back from two weeks of PTO and will make sure to get this reviewed latest EOD tomorrow

Cool, no worries. Just making sure you saw.

Made some small updates, and I've left the two runbooks that might be downgraded to warning level in. Let me know if I missed anything.

/lgtm
Thanks again!

repo: openshift/runbooks


text:
Added che credentials to che-starter PR check.

Signed-off-by: Radim Hopp rhopp@redhat.com

note that this is going to create a new job, and not remove the older one. So some manual tweak will be needed on the jenkins side to remove the older job before this is merged in

@kbsingh yeah, but it looks like the only clean way to add credentials without adding it to others that use{ci_project}-{git_repo} template. Who should I bother with manual cleanup on jenkins side ?

@bstinsonmhk (bstinson@redhat.com) would be the person to ask for deletion:)

There are about 6 people worth the account, anyone should be able to help

The older job has been deleted.This should be ready to merge from our end.

@bstinsonmhk thatnks! @vpavlin @kbsingh could you please merge ?

authors: bstinsonmhk ibuziuk kbsingh rhopp vpavlin

repo: openshiftio/openshiftio-cico-jobs


text:
Change personal website to birthdate and "projects" to "about me"

Hello,
The title should be self-explanatory :).
I am currently writing my resume with your nice template but I was wondering if it was possible to :

Remove the "personal website" in the sidebar and put a birthdate instead (since I don't have any personal website !) ?
Replace the "projects" section in the sidebar by another one, for instance an "about me" section or another skills overview diagram ?

Thank you.

@Mataivic Of course! Feel free to edit the twentysecondscv.cls file as necessary.

Ok cool ! I have never tried to modify such a file, can I have a few hints to know where to start ?

I recommend the official LaTeX tutorial. Once you get the hang of basic LaTeX concepts, you can get your hands dirty with changes to the .cls file 😃

authors: Mataivic opensorceror

repo: opensorceror/Data-Engineer-Resume-LaTeX


text:
🛑 Demo is down

In fb4d8fa, Demo (https://demo.opensourcepos.org) was down:

HTTP code: 500
Response time: 16937 ms

Resolved: Demo is back up in b3894f1 after 11 minutes.

repo: opensourcepos/upptime

license: MIT


text:
ksponspeech 데이터 관련해서 질문드립니다.

❓ Questions & Help
터미널에서 dataset.test_manifest_dir 을 작성해야하는데 제가 kspon manifest 파일을 공유 받았을 때 manifest 파일에 train, val, test 파일 오디오경로들이 전부 한 파일안에 있는 것을 확인하였습니다. 그래서 저 인자값에는 어떤 것을 줘야하는지 궁금합니다.
또한 제가 봤을 때는 보이지 않았는데 train val 데이터가 셔플이 이루어지나요? 아니면 순서대로 split이 될까요
또한 이전의 질문에서 먼저 manifest 파일을 포맷을 작성한 뒤에
KsponSpeech - LightningDataModule. 이를 구현하셔야한다고 하셧는데
이코드 안에 manifest 파일을 generate하는 코드가 들어가있는데 그럼 미리 만들필요가
있는지 여쭤보고 싶습니다...
제가 커맨드를 잘 못넘기고 있는것같습니다..ㅠ
Details

안녕하세요. @MunJeongHyeon
dataset.test_manifest_dir 인자 값으로는 eval_clean.trn, eval_other.trn 가 있는 디렉토리를 지정해주시면 됩니다.  [link]
train, valid를 앞에서부터 개수대로 나누고, 각각 데이터 셔플을 진행해줍니다. 링크 참고하시면 좋을 것 같아요!  [link1][link2]

또한 이전의 질문에서 먼저 manifest 파일을 포맷을 작성한 뒤에
KsponSpeech - LightningDataModule. 이를 구현하셔야한다고 하셧는데
이코드 안에 manifest 파일을 generate하는 코드가 들어가있는데 그럼 미리 만들필요가
있는지 여쭤보고 싶습니다...

파일을 generate하는 코드가 들어가있어서 경로 지정만 잘 해주시면 manifest 파일을 generate 하고 학습이 시작됩니다.

궁금한 부분이 manifest 파일을 미리 만들어놓고 generate 하는 코드 부분을 제거해도 괜찮을까요? kspon 데이터를 사용하는 것이 아닌 제가 가진 음성데이터를 사용해보려고 합니다.

또한
dataset.test_manifest_dir 인자 값으로는 eval_clean.trn, eval_other.trn 가 있는 디렉토리를 지정해주시면 됩니다. [link]
이렇게 답장을 해주셨는데, kspon manifest 파일 예시에서는 마지막부분엔 test 데이터의 경로 /t 스크립트 /t 토큰 이렇게 들어가 있는데 test 데이터에 대해서도 들어가야하는지 궁금합니다.

궁금한 부분이 manifest 파일을 미리 만들어놓고 generate 하는 코드 부분을 제거해도 괜찮을까요? kspon 데이터를 사용하는 것이 아닌 제가 가진 음성데이터를 사용해보려고 합니다.

네, 미리 만들어놓으시면 제거해도 됩니다.

이렇게 답장을 해주셨는데, kspon manifest 파일 예시에서는 마지막부분엔 test 데이터의 경로 /t 스크립트 /t 토큰 이렇게 들어가 있는데 test 데이터에 대해서도 들어가야하는지 궁금합니다.

제가 올바르게 이해한지는 모르겠지만, generate 하는 코드를 제거 하고 manifest 파일을 미리 만들어놓으시는거라면   test 데이터도 동일하게 데이터의 경로 /t 스크립트 /t 토큰 으로 해주시면 됩니다.

제가 만약 다른 데이터셋으로 돌릴려면 test set에 대한 trn 파일을 만들어야할까요?
또한 manifest 파일에 train+val+test 데이터셋 세개가 다들어가야하는지 아니면 학습용과 테스트용을 분리해서
train_manifest.txt, test_manifest.txt로 분류를해야하는지 궁금합니다.!

다른 데이터셋으로 돌리기 위해서는 코드를 추가해주셔야 합니다.
현재 librispeech, ksponspeech, aishell 데이터셋을 지원하고 있습니다만, 다른 데이터셋 사용을 위해서는 lightning data module을 새로 만들어주셔야 합니다. KsponSpeech 같은 경우는 특이하게 train/valid 데이터와 test 데이터가 따로 공개되어 어쩔수 없이 분리하여 옵션을 받습니다만, 다른 데이터셋의 경우 dataset_path만 지정하면 알아서 처리되도록 짜여있습니다.

네 감사합니다.!!! 혹시 코드문에서 manifest 파일에는 \n이 존재하지 않는다고 되어있는데 제가 승인요청해서 받은 ksponspeech 데이터의 manifest파일에는 \n이 존재해서 질문남깁니다.

authors: MunJeongHyeon hasangchun sooftware

repo: openspeech-team/openspeech


text:
terminal hangs when train starts

❓ Questions & Help
I want to know whether it needs more configuration,
or just patience ....
I'm beginner for torch and speech to text, please share your warm advice.

environments : mac m1
command I used

HYDRA_FULL_ERROR=1 python3 ./openspeech_cli/hydra_train.py \
dataset=foreignkorean \
dataset.dataset_path={datasetpath} \
dataset.manifest_file_path={manifestpath} \
dataset.test_dataset_path={test_datasetpath} \
dataset.test_manifest_dir={manifestdir} \
tokenizer=foreignkorean_character \
tokenizer.vocab_path={vocabpath} \
model=listen_attend_spell \
audio=melspectrogram \
lr_scheduler=warmup_reduce_lr_on_plateau \
trainer=cpu \
criterion=cross_entropy

Details

in my terminal

in wnadb

terminal is printing like below....
Epoch 0:   0%| | 2/1080 [39:58<239:22:18, 799.39s/it, loss=7.04, v_num=8rcc, train_loss=7

@tae0y  I think it's very slow because you're training with cpu. I recommend training in a GPU environment.

@upskyy
감사합니다아아아!! 혹시나 구글코랩 TPU에서 돌릴분들을 위해 환경정보를 공유드립니다.
여전히 느리긴 하지만.. 이제 진행은 되네요!
thankssssss!!! I share environment for those who run this project on colab tpu.
It's still slow but it is ongoing.
#프로젝트 경로로 이동해서 의존성을 설치한다
%cd /content/drive/MyDrive/openspeech-clone
!pip install -e .

#다음 이슈를 참고해서 tpu 관련 패키지를 설치한다 https://github.com/googlecolab/colabtools/issues/3646
!pip install pytorch-lightning==1.8
!pip install cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl
!pip uninstall -y torch torchvision torchaudio torchdata torchtext
!pip install torch torchvision torchaudio torchdata torchtext --extra-index-url https://download.pytorch.org/whl/cu117


text:
[Quest] Load Form configuration from Binary resource

Description
Dynamically load and configure the app Form configuration using the Binary resource that will hold the information about the different Form configurations to be used in the configurable Quest application. This will define the Forms & Title to be presented to the user at each workflow user interaction point
Feature
Load & Sync the Binary resource from HAPI FHIR Server and not locally loaded from assets Folder
Sample Form Configuration
[
  {
    "appId": "anc",
    "form": "anc-patient-registration",
    "title": "Enroll as ANC Patient",
    "identifier": "1924"
  },
  {
    "appId": "anc",
    "form": "family-member-registration",
    "title": "Add Family Member",
    "identifier": "32821"
  },
  {
    "appId": "anc",
    "form": "non-anc-patient-vital-signs_metric",
    "title": "Add Vital Signs ",
    "identifier": "1921"
  },
  {
    "appId": "anc",
    "form": "non-anc-patient-vital-signs_standard",
    "title": "Add Vital Signs ",
    "identifier": "2007"
  },
  {
    "appId": "anc",
    "form": "family-registration",
    "title": "Add Family",
    "identifier": "32822"
  },
  {
    "appId": "anc",
    "form": "family-patient_bmi_compute",
    "title": "Compute BMI",
    "identifier": "1999"
  },
  {
    "appId": "ecbis",
    "form": "anc-patient-registration",
    "title": "Enroll as ANC Patient",
    "identifier": "1924"
  },
  {
    "appId": "ecbis",
    "form": "family-member-registration",
    "title": "Add Family Member",
    "identifier": "82953"
  },
  {
    "appId": "ecbis",
    "form": "non-anc-patient-vital-signs_metric",
    "title": "Add Vital Signs ",
    "identifier": "1921"
  },
  {
    "appId": "ecbis",
    "form": "non-anc-patient-vital-signs_standard",
    "title": "Add Vital Signs ",
    "identifier": "2007"
  },
  {
    "appId": "ecbis",
    "form": "family-registration",
    "title": "Add Family",
    "identifier": "82952"
  },
  {
    "appId": "ecbis",
    "form": "family-patient_bmi_compute",
    "title": "Compute BMI",
    "identifier": "1999"
  }
]

NOTE: The current configs are available in the assets directory. They were missed during the refactor to use downloaded Binary resource configurations. Move the sample form configuration to the resource directory where we store other sample view configurations.

I think this issue has been properly scoped. Will add some comment if anything new came up.
Working on this

Is this using a binary resource so that we can use arbitrary JSON?

Yes this should use a binary resource

Marking this ticket as DONE

Needed for ECBIS app

Functionality merged through another branch. Closing this.

authors: FikriMilano ellykits f-odhiambo pld

repo: opensrp/fhircore


text:
[Quest/eCBIS] - Client profile - top right overflow menu items are missing

Describe the bug
When you navigate to a client profile, the top right-hand side menu has only 1 option. Thus, user:

[ ] Cannot edit the client registration details.
[ ] Cannot mark a client as pregnant (pregnant outcome)
[ ] Cannot mark as the deceased client.

To Reproduce
Steps to reproduce the behavior:

Log into the app.
Register a female client over 15years.
Navigate to their profile, and tap on the top right hand corner menu. The menu will open up. Observe the options available.

Expected behavior
Once a user is registered, there should be the following options on their profile (top right menu)

Edit info - to edit registration details for the client. Tapping on the "edit info" on the top right-hand corner menu will open up the client registration form. User should be able to edit any editable parts of the form and save them once completed. All changes updated should reflect on the client's profile.
Mark as deceased - this is accessed from the top-right menu. This shows on any client's profile. Tapping on this opens the person removal form. Filling this form and confirming removal will remove the person from that household's register. The name will still show on the register but marked as deceased.
Record pregnancy - If the client is a female over 15 years old (female of reproductive age) - the 'record pregnancy option should be displayed on the top-right menu. Tapping on this option opens the pregnancy recording form.

Screenshots
If applicable, add screenshots to help explain your problem.
Smartphone (please complete the following information):

Device: [e.g. Samsung Galaxy A2 emulator]
Android version: [e.g. Settings -> About phone -> Android version]
Build number: [e.g. Settings -> About phone -> Build number]
Version [e.g. 24]

Additional context
Add any other context about the problem here.

@dubdabasoduba @f-odhiambo
I'm picking this up.

@dubdabasoduba
I have questions:

What is the indicator that a family member is deceased? Attributes or coding system perhaps?

Do you have the Questionnaire form for marking a family member as deceased?

What is the indicator that a family member is pregnant? Attributes or coding system perhaps?

Do you have the Questionnaire form for marking a family member as pregnant?

@FikriMilano

What is the indicator that a family member is pregnant? Attributes or coding system perhaps?
Do you have the Questionnaire form for marking a family member as pregnant?

This will be handled in the following issue https://github.com/opensrp/fhircore/issues/1345

Got it

v.0.0.7
Marking this as closed as the items are now available on the top-right menu.
cc @dubdabasoduba

authors: AnnieMungai FikriMilano dubdabasoduba


text:
seg fault on executing 'openssl spkac'

currently on master, running 'openssl spkac' results in a seg fault.
(gdb) run spkac
Program received signal SIGSEGV, Segmentation fault.
0x0000000000478a75 in app_load_config_internal ()
(gdb) bt
#0 0x0000000000478a75 in app_load_config_internal ()
#1 0x000000000046cc59 in spkac_main ()
#2 0x0000000000444c09 in do_cmd ()
#3 0x0000000000422457 in main ()
OpenSSL version used:
OpenSSL 3.0.0-alpha17-dev

This happens when no input file is specified.
Definitely a bug since it is documented as working (using stdin).

repo: openssl/openssl


text:
Why does OpenSSL needs Mono CA certs in its search path?

When it comes to auditing CA certificates, one needs to know WhAT and WHERE are the OpenSSL is picking up their CA certificates.
Executing the update-ca-certificates --fresh using strace -f has enabled me to compile a list of files read and written.
The list of files that are opened and read-only by OpenSSL are in the following order:
/etc/ca-certificates.conf file
/usr/share/ca-certificates/* directory
/usr/share/ca-certificates/mozilla/* directory
/usr/share/ca-certificates/*/* directory
/usr/local/share/ca-certificates/* directory
/usr/local/share/ca-certificates/*/* directory
/etc/ssl/certs directory
$CWD/<all-certs-read-before> files
/usr/lib/ssl/openssl.cnf file
/etc/ca-certificates/update.d directory
/etc/ca-certificates/update.d/jks-keystore directory
/etc/default/cacerts directory
/etc/java-11-openjdk/security/nss.cfg file
/usr/share/ca-certificates-java directory
/usr/lib/jvm/java-11-openjdk-amd64/lib/jvm.cfg file
/usr/share/ca-certificates-java/ca-certificates-java.jar file
/etc/ca-certificates/update.d/mono-keystore directory
/etc/mono/4.5/machine.config file
/etc/mono/assemblies/cert-sync/cert-sync.config file
/etc/mono/assemblies/Mono.Security/Mono.Security.config file
/etc/mono/assemblies/mscorlib/mscorlib.config file
/etc/mono/assemblies/System/System.config file
/etc/mono/config file
/etc/ssl/certs/ca-certificates.crt file
$HOME/.mono/config file
/usr/lib/mono/4.5/cert-sync.exe.config file
/usr/lib/mono/4.5/cert-sync.exe.config file
/usr/lib/mono/4.5/mscorlib.dll.config file
/usr/lib/mono/gac/Mono.Security/4.0.0.0_0738eb9f132ed765/Mono.Security.dll.config file
/usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll.config file
/usr/share/.mono/certs/Trust/ski-*.cer file
/usr/share/.mono/certs/new-certs/XXXXXXXX.0 file
$CWD/openssl EXECUTABLE!!! (why look in $CWD?)
/usr/bin/openssl
/usr/local/bin/openssl
/usr/local/sbin/openssl
/usr/sbin/openssl (VERY STRANGE ordering of /usr/[local/][s]bin/

OpenSSl also write writes to the following text files:
$CWD/ca-certificates.txt
/etc/ssl/certs/java/cacerts

so, why is Mono engaged in Trusted CA input?

misordered lookup sequence of
Observation of update-ca-certificates.
I noticed a very strange ordering of looking for the openssl binary.
Probably should have been something in the (re)order of:
$CWD/openssl (probably should NOT have this entry)
/usr/local/sbin/openssl
/usr/local/bin/openssl
/usr/sbin/openssl
/usr/bin/openssl

Auditable Impact Toward CA Certificates
Probably should OUTPUT various ‘modules’ being touched up during the rebuilding of CA certificates:

MONO
OpenJDK Java 11
Mozilla

then list the following

OS System

Auditable Output of CA Certificates
Probably should OUTPUT what various CREATION of files:
$CWD/ca-certificates.txt
/etc/ssl/certs/java/cacerts

Better Summarization
Probably should indicate those summarization AT THE END of its output, broken down by CA-CERTIFICATE MODULES. Like:
      OS System:
        Added:   0
        Deleted: 0
        Used:    0
      Mozilla package:
        Added:   0
        Deleted: 0
        Used:  129
      OpenJDK package:
        Added:   0
        Deleted: 0
        Used:   86
      MONO package:
        Added:   0
        Deleted: 0
        Used:   44
      Total Merge:   129

  Master File:  /usr/share/ca-certificates  (depends on distro)
  Master File:  /etc/ca-certificates
  Master File:  /etc/ssl/certs
  Master File:  /etc/pki/tls/certs

update-ca-certificates isn't an OpenSSL program.  As far as I understand, it ensures that all programs that may need to verify a certificate can do so against one unified store of CA certificates.  update-ca-certificates comes from Debian's package ca-certificates
Manual: https://manpages.debian.org/bullseye/ca-certificates/update-ca-certificates.8.en.html
Debian's package page: https://packages.debian.org/sid/ca-certificates

authors: egberts levitte


text:
getting public key from EVP_PKEY generated with private key

Before 3.0, I was able to get the public key of an EC private key using basic EC maths even if EC_KEY was generated only with private key information with the following function:
std::vector<uint8_t> ECDSA::GetPubKeyFromPrivKey(EC_KEY* ec_key, const EC_GROUP* ec_group)
{
	auto* bn_ctx = BN_CTX_new();

	const auto* pub_key = EC_KEY_get0_public_key(ec_key);
	if (pub_key == nullptr)
	{
		auto* pub_key2 = EC_POINT_new(ec_group);
		const auto* priv_key = EC_KEY_get0_private_key(ec_key);
		if (!EC_POINT_mul(ec_group, pub_key2, priv_key, nullptr, nullptr, bn_ctx))
		{
			BN_CTX_free(bn_ctx);

			return {};
		}
		if (!EC_KEY_set_public_key(ec_key, pub_key2))
		{
			BN_CTX_free(bn_ctx);

			EC_POINT_free(pub_key2);

			return {};
		}
		if (!EC_KEY_check_key(ec_key))
		{
			BN_CTX_free(bn_ctx);

			EC_POINT_free(pub_key2);

			return {};
		}
		pub_key = pub_key2;
	}

	auto* pub_key_bn = BN_new();
	if (EC_POINT_point2bn(ec_group, pub_key, POINT_CONVERSION_COMPRESSED, pub_key_bn, bn_ctx) == nullptr)
	{
		BN_free(pub_key_bn);
		BN_CTX_free(bn_ctx);

		return {};
	}
	std::vector<uint8_t> pub_key_buffer(BN_num_bytes(pub_key_bn));
	if (!BN_bn2bin(pub_key_bn, pub_key_buffer.data()))
	{
		BN_free(pub_key_bn);
		BN_CTX_free(bn_ctx);

		return {};
	}
	BN_free(pub_key_bn);
	BN_CTX_free(bn_ctx);

	return pub_key_buffer;
}

As most of EC_ functions are now deprecated, I can not use this function anymore. I was able to successfully import the private key to an EVP_PKEY structure using params, however, after creating the EVP_PKEY structure, I can not get the public key via params(it is empty as it was the case with this function, thats why I use EC maths to calculate public key). I searched the docs for quite some time, however I can not find useful functions that can get me to calculate the public key from only private key.
I really want to migrate to 3.0. Are there any useful examples regarding this issue I can check?
Extra:
The following does not work as pub_key is just nullptr. The same thing happened on OpenSSL 1 with EC_KEY_get0_public_key. That is why "if (pub_key == nullptr)" exists as I mathematically calculate it from private key using now deprecated EC_ functions.
std::vector<uint8_t> ECDSA::GetPubKeyFromPrivKey(const std::vector<uint8_t>& privKey)
{
	EVP_PKEY* ec_key = nullptr;

	EVP_PKEY_CTX* pCtx = EVP_PKEY_CTX_new_from_name(nullptr, "EC", nullptr);
	if (pCtx == nullptr)
	{
		return {};
	}
	EVP_PKEY_fromdata_init(pCtx);

	OSSL_PARAM params[3];
	params[0] = OSSL_PARAM_construct_utf8_string(OSSL_PKEY_PARAM_GROUP_NAME, const_cast<char*>(SN_secp256k1), 0);
	params[1] = OSSL_PARAM_construct_BN(OSSL_PKEY_PARAM_PRIV_KEY, const_cast<uint8_t*>(privKey.data()), privKey.size());
	params[2] = OSSL_PARAM_construct_end();

	EVP_PKEY_fromdata(pCtx, &ec_key, EVP_PKEY_KEYPAIR, params);
	if (ec_key == nullptr)
	{
		EVP_PKEY_CTX_free(pCtx);

		return {};
	}

	auto pub_key_buffer = GetPubKeyFromPrivKey(ec_key);
	if (pub_key_buffer.empty())
	{
		EVP_PKEY_CTX_free(pCtx);
		EVP_PKEY_free(ec_key);

		return {};
	}

	EVP_PKEY_CTX_free(pCtx);
	EVP_PKEY_free(ec_key);

	return pub_key_buffer;
}

std::vector<uint8_t> ECDSA::GetPubKeyFromPrivKey(EVP_PKEY* ec_key)
{
	BIGNUM* pub_key = nullptr;
	EVP_PKEY_get_bn_param(ec_key, OSSL_PKEY_PARAM_PUB_KEY, &pub_key);
	if (pub_key == nullptr)
	{
		return {};
	}

	std::vector<uint8_t> pub_key_buffer(BN_num_bytes(pub_key));
	if (!BN_bn2bin(pub_key, pub_key_buffer.data()))
	{
		return {};
	}

	return pub_key_buffer;
}

As most of EC_ functions are now deprecated, I can not use this function anymore. I was able to successfully import the private key to an EVP_PKEY structure using params, however, after creating the EVP_PKEY structure, I can not get the public key via params(it is empty as it was the case with this function, thats why I use EC maths to calculate public key). I searched the docs for quite some time, however I can not find useful functions that can get me to calculate the public key from only private key.

Only the EC_KEY_* functions are deprecated. Most of the remaining ones were deliberately left undeprecated - so EC_POINT_new, EC_POINT_mul, etc should all still work just fine. With that in mind your previous approach of mathematically calculating the key should still work. You will still an EC_GROUP object which you can create using one of the EC_GROUP_new_* functions.
We probably should add a feature at some point to automatically calculate the Public key from the private one.

As most of EC_ functions are now deprecated, I can not use this function anymore. I was able to successfully import the private key to an EVP_PKEY structure using params, however, after creating the EVP_PKEY structure, I can not get the public key via params(it is empty as it was the case with this function, thats why I use EC maths to calculate public key). I searched the docs for quite some time, however I can not find useful functions that can get me to calculate the public key from only private key.

Only the EC_KEY_* functions are deprecated. Most of the remaining ones were deliberately left undeprecated - so EC_POINT_new, EC_POINT_mul, etc should all still work just fine. With that in mind your previous approach of mathematically calculating the key should still work. You will still need an EC_GROUP object which you can create using one of the EC_GROUP_new_* functions.
We probably should add a feature at some point to automatically calculate the Public key from the private one.

Thanks for the reply, I was able to successfully convert my code. It is written below if anybody is interested:
std::vector<uint8_t> ECDSA::GetPubKeyFromPrivKey(EVP_PKEY* ec_key)
{
	size_t pub_key_size = 0;
	EVP_PKEY_get_octet_string_param(ec_key, OSSL_PKEY_PARAM_PUB_KEY, nullptr, 0, &pub_key_size);
	std::vector<uint8_t> pub_key_buffer(pub_key_size);
	if (!EVP_PKEY_get_octet_string_param(ec_key, OSSL_PKEY_PARAM_PUB_KEY, pub_key_buffer.data(), pub_key_buffer.size(),
	                                     &pub_key_size))
	{
		size_t group_name_size = 0;
		EVP_PKEY_get_utf8_string_param(ec_key, OSSL_PKEY_PARAM_GROUP_NAME, nullptr, 0, &group_name_size);
		std::vector<char> group_name(group_name_size + 1);
		if (!EVP_PKEY_get_utf8_string_param(ec_key, OSSL_PKEY_PARAM_GROUP_NAME, group_name.data(), group_name.size(),
		                                    &group_name_size))
		{
			return {};
		}

		int group_nid = OBJ_sn2nid(group_name.data());
		if (group_nid == NID_undef)
		{
			return {};
		}

		auto* ec_group = EC_GROUP_new_by_curve_name(group_nid);
		if (ec_group == nullptr)
		{
			return {};
		}

		auto* pub_key = EC_POINT_new(ec_group);
		if (pub_key == nullptr)
		{
			EC_GROUP_free(ec_group);

			return {};
		}
		BIGNUM* priv_key = nullptr;
		if (!EVP_PKEY_get_bn_param(ec_key, OSSL_PKEY_PARAM_PRIV_KEY, &priv_key))
		{
			EC_POINT_free(pub_key);
			EC_GROUP_free(ec_group);

			return {};
		}

		if (!EC_POINT_mul(ec_group, pub_key, priv_key, nullptr, nullptr, nullptr))
		{
			EC_POINT_free(pub_key);
			EC_GROUP_free(ec_group);

			return {};
		}

		pub_key_size = EC_POINT_point2oct(ec_group, pub_key, POINT_CONVERSION_COMPRESSED, nullptr, 0, nullptr);
		if (pub_key_size == 0)
		{
			EC_POINT_free(pub_key);
			EC_GROUP_free(ec_group);

			return {};
		}
		pub_key_buffer.resize(pub_key_size);
		if (!EC_POINT_point2oct(ec_group, pub_key, POINT_CONVERSION_COMPRESSED, pub_key_buffer.data(),
		                        pub_key_buffer.size(), nullptr))
		{
			EC_POINT_free(pub_key);
			EC_GROUP_free(ec_group);

			return {};
		}

		EC_POINT_free(pub_key);
		EC_GROUP_free(ec_group);

		if (!EVP_PKEY_set_octet_string_param(ec_key, OSSL_PKEY_PARAM_ENCODED_PUBLIC_KEY, pub_key_buffer.data(),
		                                     pub_key_buffer.size()))
		{
			return {};
		}

		auto* pCtx = EVP_PKEY_CTX_new_from_pkey(nullptr, ec_key, nullptr);
		if (pCtx == nullptr)
		{
			return {};
		}

		if (!EVP_PKEY_public_check_quick(pCtx))
		{
			EVP_PKEY_CTX_free(pCtx);

			return {};
		}

		EVP_PKEY_CTX_free(pCtx);
	}

	return pub_key_buffer;
}

authors: aizuon mattcaswell


text:
[rt.openssl.org #4154] [PATCH] BIO mem read optimization

Migrated from rt.openssl.org#4154 (status was 'new')
Requestors:

k.marinushkin@gmail.com

Attachments:

openssl_bio_s_mem.patch

From k.marinushkin@gmail.com on 2015-11-23 07:49:26:
2 BIO_s_mem issues solved:
- BIO mem read without reallocation - reading by parts becomes faster;
- flag added to rewind read write BIO mem on reset.

OpenSSL self-test report:

OpenSSL version:  1.1.0-dev
Last change:      State machine rewrite. The state machine code has been ...
Options:          -march=pentium -Wa,--noexecstack no-deprecated
no-ec_nistp_64_gcc_128 no-gmp no-jpake no-md2 no-rc5 no-sctp no-shared
no-ssl-trace no-store no-unit-test no-zlib no-zlib-dynamic static-engine
OS (uname):       Linux kirill-Lenovo-B570e 3.13.0-68-generic #111-Ubuntu
SMP Fri Nov 6 18:18:09 UTC 2015 i686 i686 i686 GNU/Linux
OS (config):      i686-whatever-linux2
Target (default): linux-elf
Target:           linux-elf
Compiler:         Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/i686-linux-gnu/4.8/lto-wrapper
Target: i686-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu
4.8.4-2ubuntu1~14.04'
--with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs
--enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.8 --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext --enable-threads=posix
--with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls
--with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap
--enable-plugin --with-system-zlib --disable-browser-plugin
--enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-i386/jre
--enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-i386
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-i386
--with-arch-directory=i386 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
--enable-objc-gc --enable-targets=all --enable-multiarch --disable-werror
--with-arch-32=i686 --with-multilib-list=m32,m64,mx32 --with-tune=generic
--enable-checking=release --build=i686-linux-gnu --host=i686-linux-gnu
--target=i686-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)

Test passed.

--
Best Regards,
Kirill Marinushkin

A.k.a #859

authors: FdaSilvaYY richsalz


text:
Problem in Building openssl_1.0.2p in Visual Studio 2015

Am building new release of openssl_1.0.2p using Visual Studio 2015 Command Prompts. I have installed perl and sed. I have written some batch files to build x86 and x64 libraries. When I built same openssl version in VS 2013 and VS 2017 am not facing any issue.
But while building in VS 2015 am unable to build. When I build x86 library in “VS2015 x86 Native Tools Command Prompt” then its building fine and resulting in generating include and lib folder. But when I try to build DLL library then am getting fatal error which says
rc /fo"tmp32dll\libeay32.res" /d CRYPTO ms\version32.rc
‘rc’ is not recognized as an internal or external command, operable program or batch file.
NMAKE : fatal error V1077: ‘rc’ : return code ‘0x1’ Stop. 1 dir(s) moved. 1 dir(s) moved.”
I have attached screenshot too

It should be noted that this project's constant integration with Visual Studio (by courtesy of appveyor.com) does use version 2015. And there are no problems observed. There is no other conclusion to draw but that this is your local environment problem. And these are not something we can't help with. While we do stand for code and build system, it's still user's responsibility to make toolchain work.

@SiddalingaSwamy12 the error message is very clear: you need to find out why rc.exe is not found in the PATH.
'rc' is not recognized as an internal or external command.

A correctly installed VS2015 does include the Windows Kits location in the path in the x86 native command tool.

Problem is Solved. Windows and Web Development package was not installed in VS 2015. I installed it and now openssl is building fine.
If anyone wants to install these packages then goto control panel-> programs and features -> VS 2015
Right Click on VS 2015 -> change -> Modify -> select Windows and Web Development - > install

Posting here because I seem to run into this occasionally, since I only build open ssl at most once a year and invariably open the wrong command prompt like the one in Visual Studio C++. Using the wrong command prompts will typically throw a build error like "'rc' is not recognized as an internal or external command." because they  were created for other purposes and so the required binaries are not available.
My experience has been that the command prompt in Visual Studio 2017, specifically "Developer Command Prompt for VS 2017" or "x64 Native Tools Command Prompt for VS 2017" work every time.

authors: SiddalingaSwamy12 dot-asm gregbown mspncp t-j-h


text:
Ignore vendor name in Clang version number.

For example, FreeBSD prepends "FreeBSD" to version string, e.g.,
FreeBSD clang version 11.0.0 (git@github.com:llvm/llvm-project.git llvmorg-11.0.0-rc2-0-g414f32a9e86)
Target: x86_64-unknown-freebsd13.0
Thread model: posix
InstalledDir: /usr/bin
This prevented us from properly detecting AVX support, etc.

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

Formality: I agree this is trivial

Note that it is not just FreeBSD, we use Clang's built-in vendor ID functionality, as does Apple. Here's a recent Clang/LLVM change that prompted investigation https://reviews.llvm.org/D69925

This pull request is ready to merge

Pushed to master and 1.1.1.
Note that util/perl/OpenSSL/config.pm doesn't exist on 1.1.1 (and there is no analogous clang check), so there was a little touch-up in the cherry-pick but it was basically clean.
Thanks for the submission!  Closing the PR accordingly.

authors: emaste juikim kaduk levitte openssl-machine


text:
Pkcs11 prov

Checklist

[ ] documentation is added or updated
[ ] tests are added or updated

You will need to rebase..
@wbeck10 any chance you could coordinate with @simo5 to work towards a common goal?

FYI @beldmit

@wbeck10 I kindly ask you to look at this project: https://github.com/latchset/pkcs11-provider and probably participate in it

This PR has the label 'hold: cla required' and is stale: it has not been updated in 30 days. Note that this PR may be automatically closed in the future if no CLA is provided.  For CLA help see https://www.openssl.org/policies/cla.html

If I may wish for something, it's a bit more story-telling in the commits.  Just titles makes it hard to review and to understand the reasoning behind some of the changes.

authors: beldmit levitte openssl-machine slontis wbeck10


text:
Fix struct timeval include guards.

Move struct timeval includes into e_os.h (where the Windows ones were).
Ensure that the include is guarded canonically.
Refer #4271

[x] tests are added or updated

Thanks.  Merged.

I like how you finally solved it

@levitte thanks, much appreciated.
It seemed more natural this way and a reduction in code is usually a good thing.

authors: levitte paulidale