text:
Add logging to 4xx responses from API at reduced rate.

Is your feature request related to a problem? Please describe.
Today, a few errors returned by opensearch are not logged if they are customer side errors. While we may provide the message as part of the response, it is not always possible to determine the cause of the errors. We can explore to determine if it may benefit to have the logging for such errors at a reduced rate (e.g. how it is done today for connection errors in nodes). This will allow to better debug issues without the need to determine which debug/trace logs should be enabled.
Describe the solution you'd like
Sample 4xx errors and log at reduced rate
Describe alternatives you've considered
Enable debug/trace logs at multiple places. This requires users/operators to still identify the classes for which logs should be enabled and may not be straight forward.

Rekindling the discussion by adding to the above points.
@mgodwan @sarthakaggarwal97 please take a look.
Is your feature request related to a problem? Please describe.
At the moment, there is no way to rate limit the logging of exceptions at a particular log level. Currently if we enable a lesser specific logging level for a class to gain more insights into the exceptions occurring, we might run into bloating of log files with excessive trivial logs. This will further enable us to add more logging, with the ability to rate limit, and debug root cause of the issues quickly.
Describe the solution you'd like
We will develop a central framework, which will enable a custom log4j filter. In this filter, we will introduce our own filtering logic, and on the basis of our algorithm, we will return the three states (ACCEPT, DENY, NEUTRAL) under different circumstances.
This will also require us to enable dynamic cluster settings, where we can tune the frequency of the threshold at which we want to limit the exceptions, the level at which we want to enable the rate limiting in the class, and the packages/classes where we are enabling the exceptions using their package name like we do for configuring dynamic logging.
{
    "transient": {
        "log_limiting.filter.org.opensearch.index.engine.InternalEngine":"true",
        "log_limiting._settings.threshold":5,
        "log_limiting._settings.level": "DEBUG"
    }
}

This is how we can dynamically toggle the settings to achieve the logging at the reduced rate.
In the cases where we have enabled different settings for the parent-child classes/packages, we can give the priority to the more specific package setting.
This approach is considered is good option for below reasons

It would enable us to log the root cause of the issues with more flexibility and without bloating the log files.
It is independent and extensible to all flows in the OpenSearch.

Describe alternatives you've considered
We considered a approach that we can enable rate limiting for few logs like in NodeConnectionsService, by placing a counter for the exception, and increasing the log level for the exception after every nth time. This approach was discarded because of the following reasons.

It will not give us flexibility of the toggling the parameters of rate limiting without a code change.
We will have to make code change to enable/disable rate limiting every time.
We will have to maintain multiple counters for each exception in the class, which is not the best design practice.

Thanks @rayshrey for adding details.

Whay criteria do we use to determine the common log within the window?
With the proposed approach, what kind of compute/JVM overhead will we incur with the log4j filter in the critical path for indexing/search requests?

In the cases where we have enabled different settings for the parent-child classes/packages, we can give the priority to the more specific package setting.

Do you mean we prioritize package over classes here? Should it not be the lowest common ancestor among a class and available settings which gets the higher priority?

"log_limiting._settings.threshold":5,
"log_limiting._settings.level": "DEBUG"

Are you proposing this as a common setting for all loggers? Do you think we can have this defined on class/package level as well if needed?

where we can tune the frequency of the threshold at which we want to limit the exceptions

How do we define the window for this threshold?

@shwetathareja Thoughts on this?

Do you mean we prioritize package over classes here? Should it not be the lowest common ancestor among a class and available settings which gets the higher priority?

@mgodwan I think the log level settings prioritizes the parent over child in applying settings (@rayshrey can verify this once), but I agree with you, we should give higher priority to the lowest common ancestor.

With the proposed approach, what kind of compute/JVM overhead will we incur with the log4j filter in the critical path for indexing/search requests?

@rayshrey three types of benchmarks will be helpful. One without the filter, one with the filter but not applied via cluster settings (since all logs will pass through the present filter even if it has no affect), and one where we have the filter applied. You can also look into microbenchmarking for this.

Thanks for the comments @mgodwan and @sarthakaggarwal97
Please find below the replies to your queries.

What criteria do we use to determine the common log within the window?

For now we have set a expiry time of 1hr for the cache. So the counter for the exception type resets after 1 hr from the last accessed time.

With the proposed approach, what kind of compute/JVM overhead will we incur with the log4j filter in the critical path for indexing/search requests?

Will benchmark the results as described by @sarthakaggarwal97 and let you know about the findings once the implementation is complete

Should it not be the lowest common ancestor among a class and available settings which gets the higher priority

@sarthakaggarwal97 I verified that the logger also follows lowest common ancestor settings. Hence we should also follow similarly. Will update the implementation accordingly.

Are you proposing this as a common setting for all loggers? Do you think we can have this defined on class/package level as well if needed?

Yes, initially though of keeping it common for all loggers. But as suggested by @mgodwan we can make this setting on a modular level. Will update the implementation accordingly.

Thanks @rayshrey for the proposal. There is already WIP for Request Tracing Framework. @Gaganjuneja has started discussion around Sampling https://github.com/opensearch-project/OpenSearch/issues/8918 and one of the option would be to enable Tail Sampling based on response status. Instead of adding specific support for sampling 4xx logs explicitly. It might be useful to onboard to Tracing framework. Thoughts @mgodwan

@shwetathareja, Thanks for calling this out. Yes, these use cases will be easily served through sampled traces.

@shwetathareja

Sampled Logging is still something most customers use today and this will benefit such customers out of the box. We already have a lot of error logs at debug/trace level, and allowing local sampling of them should provide debugging benefits without adding much cost for users interested in using only logging. Based on the infrastructure customers choose to setup, logging or tracing can help with such debugging.

I see https://github.com/opensearch-project/OpenSearch/issues/8918 suggests to use head based sampling for now. This may mean that we may miss not too frequents errors in such cases which may otherwise be caught with sampling on exceptions at a node level itself as logs are discretely written on each node.

authors: Gaganjuneja mgodwan rayshrey sarthakaggarwal97 shwetathareja


text:
[Backport -1.x]Fixed copyright to OpenSearch (#1175)

Description
[Backport -1.x]Fixed copyright to OpenSearch
Issues Resolved
#1175
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

DCO check is failing, please git commit -s --amend and force-push?

start gradle check

source: gharchive/pull-request

authors: dblock minalsha

repo: opensearch-project/OpenSearch

license_type: permissive

license_source: github-api


text:
Fixes table in Data Prepper write_json processor

Description
Fixes the table and the JSON.
Issues Resolved
Closes https://github.com/opensearch-project/documentation-website/issues/7457
Version
all
Frontend features
If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.
Checklist

[x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dlvenable: Mind taking another look at this one?

authors: Naarcha-AWS hdhalter


text:
“fix#829-PreAgg-field_doc_count”

Signed-off-by: cwillum cwmmoore@amazon.com
Description
Add documentation to Bucket Aggregation describing the use of the _doc_count field for computing documents that store pre-aggregated data.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@petardz Thanks for your input on documentation for this issue. Could you review this content for technical accuracy? Will this backported to v1.3, v.2.0, and v2.1? Thanks.

@petardz Thanks for your input on documentation for this issue. Could you review this content for technical accuracy? Will this backported to v1.3, v.2.0, and v2.1? Thanks.

@cwillum: Based on the conversation from this PR(https://github.com/opensearch-project/OpenSearch/pull/3985) it seems like this code wasn't included in OpenSearch 1.0. Furthermore, the PR only contains a "backport 2.x" label, meaning that the change will be backported to the latest minor version of OpenSearch. We shouldn't need to backport on this one. Once the PR is approved and gone through editorial review, add the "5- Done and waiting to merge" label to this PR.

@cwillum I'm sorry, link with example I provided earlier is not compatible with opensearch. Here is a one synthetic example:

Create index:

PUT my_index
{
"settings": {
"number_of_replicas": 0
},
"mappings" : {
"properties" : {
"str" : {
"type" : "keyword"
},
"number" : {
"type" : "integer"
}
}
}
}

Add few documents:
POST my_index/_doc
{
"_doc_count": 10,
"str": "abc",
"number" : 500
}

POST my_index/_doc
{
"_doc_count": 5,
"str": "xyz",
"number" : 100
}
POST my_index/_doc
{
"_doc_count": 7,
"str": "foo",
"number" : 100
}

Perform aggregations
POST my_index/_search
{
"size" : 0,
"aggs" : {
"num_terms" : {
"terms" : {
"field" : "number"
}
}
}
}
Response:

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"num_terms" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 100,
"doc_count" : 12
},
{
"key" : 500,
"doc_count" : 10
}
]
}
}
}

Notice how _doc_count was used when calculating doc_count of buckets

@cwillum: please use the following example:
PUT /my_index/_doc/1
{
"response_code": 404,
"date":"2022-08-05",
"_doc_count": 20
}
PUT /my_index/_doc/2
{
"response_code": 404,
"date":"2022-08-06",
"_doc_count": 10
}
PUT /my_index/_doc/3
{
"response_code": 200,
"date":"2022-08-06",
"_doc_count": 300
}
GET /my_index/_search
{
"size": 0,
"aggs": {
"response_codes": {
"terms": {
"field" : "response_code"
}
}
}
}
Response:
{
"took" : 20,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"response_codes" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : 200,
"doc_count" : 300
},
{
"key" : 404,
"doc_count" : 30
}
]
}
}
}

authors: Naarcha-AWS cwillum kolchfa-aws petardz


text:
Tarball installation guide

Signed-off-by: jeffhuss jeffhuss@amazon.com
Description
Refactor of the tarball installation guide.
Issues Resolved
Related to issue #789
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Still working on this but I wanted to get a PR up in case anyone to see the progress.

The bulk of the work here is validation of steps and inquiries as to best practices and what we should/shouldn't recommend to users.

I think the DCO check failed due to syntax issues. Rather than fix it I'm just going to open a new draft PR. Smarter mice, etc.


text:
Service port for performance analyzer

Signed-off-by: Philipp Hölscher phoelsch@outlook.de
Description
The change adds an additional port to the statefulset and service to expose the performance analyzer.
Issues Resolved
251 - [Enhancement][opensearch]add port 9600 (Performance Analyzer)
Check List

[X ] Commits are signed per the DCO using --signoff

For any changes to files within Helm chart directories:

[X ] Helm chart version bumped
[X ] Helm chart CHANGELOG.md updated to reflect change

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@Phoelsch Thanks for this PR. Can you please check the contributing guidelines and refactor this PR? Thanks

@TheAlgo I checked the guidlines even before creating the pull request. Can please give me a hint what`s missing or wrong?

@Phoelsch Your commits are not signed

@TheAlgo Thanks, commit is now signed.

I am good with this change @Phoelsch thanks for the contribution.
@prudhvigodithi please take a look on this.
Thanks.

I actually wonder if we need to explicitly label this as performanceAnalyzer rather than metric.
Many people uniformally recognize metrics ports as prometheus compliant metrics. Aka, they can be pulled in via prometheus.
Additional, many people may be using the prometheus metric plugin to generate such alerts and have probably already added a port called metrics to achieve prometheus polling.
Could we change this to be more explicitly labeled as performance analyzer?

We actually plan to remove the coupled dependency between OS and PA process:

https://github.com/opensearch-project/opensearch-build/pull/2876
So think again, probably dont need to specifically specify this port?

@peterzhuamazon Do we have the facilities in the chart to enable and include this for users by removing this?
Aka, can they independently define the plugins, ports etc. Necessary to at least not inadvertedly remove the functionality from our chart entirely?

@peterzhuamazon Do we have the facilities in the chart to enable and include this for users by removing this?
Aka, can they independently define the plugins, ports etc. Necessary to at least not inadvertedly remove the functionality from our chart entirely?

Hi @DandyDeveloper
The current change planned in that PR is that if user does not add anything then behavior is mostly same.
The PA and OS will both start, but this time killing PA process wont bring down OS process anymore.
If use define env var DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true then PA will not start during the creation of the container.
User can modify the pa config to change port from 9600 to something else if needed, before starting container.
Thanks.

One small question @Phoelsch if in a chart one does not provide the value for the port? Will it throw error? Will this break the chart?

The port is set to 9600 as default with values.yaml. If its not explicitly overwritten by the user it will still be 9600. I dont see any problem with that.

@Phoelsch But if someone is upgrading with previous values.yaml , will it not break?

One small question @Phoelsch if in a chart one does not provide the value for the port? Will it throw error? Will this break the chart?

The port is set to 9600 as default with values.yaml. If its not explicitly overwritten by the user it will still be 9600. I dont see any problem with that.

@Phoelsch But if someone is upgrading with previous values.yaml , will it not break?

@TheAlgo If I upgrade an existing deployment to this version the belonging  values.yaml with the new default value will be used.  How could it happen that the previous values.yaml will be used?

One small question @Phoelsch if in a chart one does not provide the value for the port? Will it throw error? Will this break the chart?

The port is set to 9600 as default with values.yaml. If its not explicitly overwritten by the user it will still be 9600. I dont see any problem with that.

@Phoelsch But if someone is upgrading with previous values.yaml , will it not break?

@TheAlgo If I upgrade an existing deployment to this version the belonging values.yaml with the new default value will be used. How could it happen that the previous values.yaml will be used?

@Phoelsch Someone can still use the old values.yaml and issue an upgrade?

One small question @Phoelsch if in a chart one does not provide the value for the port? Will it throw error? Will this break the chart?

The port is set to 9600 as default with values.yaml. If its not explicitly overwritten by the user it will still be 9600. I dont see any problem with that.

@Phoelsch But if someone is upgrading with previous values.yaml , will it not break?

@TheAlgo If I upgrade an existing deployment to this version the belonging values.yaml with the new default value will be used. How could it happen that the previous values.yaml will be used?

@Phoelsch Someone can still use the old values.yaml and issue an upgrade?

@Phoelsch Opinions?

@Phoelsch Can you check the comments wen free so that we can have a closure on this PR?

Hey @Phoelsch this a nice PR to get merged, can you please fix the conflicts so that we can proceed to merge?
Thank you

@TheAlgo Im sorry for the delay. Still I dont get you`re problem und could not reproduce that. We could use a default value within the statefulset template, but that would be awful. The Helm Docs itself says that the values.yaml should be used as I did.

@TheAlgo I'm sorry for the delay. Still I don't get you`re problem und could not reproduce that. We could use a default value within the statefulset template, but that would be awful. The Helm Docs itself says that the values.yaml should be used as I did.

Yes that might not be the right way. Lets have a minor version upgrade for this. What do you think? Tagging @prudhvigodithi  as well

@Phoelsch Any updates on this PR?

@Phoelsch Any updates on this PR?

Resolved

@Phoelsch Any updates on this PR?

@TheAlgo resolved

Hey @Phoelsch this is a good change for 1.x version as well, can you please backport to 1.x branch?
Thank you

authors: DandyDeveloper Phoelsch TheAlgo peterzhuamazon prudhvigodithi

repo: opensearch-project/helm-charts


text:
Optimize reduceToTopK in ResultUtil by removing pre-filling and reducing peek calls

Description
This PR optimizes the reduceToTopK method by eliminating unnecessary pre-filling of the priority queue, reducing redundant peek() calls, and adding null safety checks for better performance.
Related Issues
Closes https://github.com/opensearch-project/k-NN/issues/2145
Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[ ] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Offline synced with @navneet1v and @jmazanec15, we now focused on optimizing reduceToTopK function in ResultUtil, updated the PR.

@junqiu-lei Did we compare removing the map with this approach to see which is faster? or is it not possible? I feel like its a simpler code that way if its yielding better results

removing the map approach: See if we can preserve the order of the results for each leaf instead of converting it to map.

We might have to first check if KNNResults array is in order. but that way we can do something similar to what lucene does and just pick the top elements and stop at k in reduceToTopK

I think this is another place we can optimize, no we don't have comparing the difference so far. For this PR, we might can use micro benchmark for testing the improvement if possible.

So, I think the most optimal way to do this would be to require that each perLeafResults comes in order from top scores to worst scores. Then, a simple algorithm that basically just scans the top of each set of results and then only takes the top k. That being, that would require how results are returned in the different methods. Do you think we should do this @navneet1v ?

I can raise other PR for this part optimization.

So, I think the most optimal way to do this would be to require that each perLeafResults comes in order from top scores to worst scores. Then, a simple algorithm that basically just scans the top of each set of results and then only takes the top k. That being, that would require how results are returned in the different methods. Do you think we should do this @navneet1v ?

Yes I think we should change the return types

repo: opensearch-project/k-NN


text:
Version upgrade for compatibility

Description
Version upgrade for compatibility
Signed-off-by: @akbhatta
Issues Resolved
NA
Check List

[x] New functionality includes testing.

[x] All tests pass, including unit test, integration test and doctest

[x] New functionality has been documented.

[x] New functionality has javadoc added
[x] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I think it's related to this #179

authors: akbhatta zhongnansu

repo: opensearch-project/notifications


text:
[Bug] performance improvement

Description
Multiple re-renders in case of update chart fixed
Issues Resolved
Bug #1163
Please note
Fix is only tested for event analytics for now. This pr may require fixes from PR #1147. Once #1147 is merged and these changes are approved will update this PR

Codecov Report

Merging #1176 (a2c6858) into main (e486d06) will increase coverage by 18.16%.
The diff coverage is n/a.

:exclamation: Current head a2c6858 differs from pull request most recent head 27f9763. Consider uploading reports for the commit 27f9763 to get more accurate results

@@              Coverage Diff              @@
##               main    #1176       +/-   ##
=============================================
+ Coverage     53.70%   71.87%   +18.16%
  Complexity      291      291
=============================================
  Files           279       42      -237
  Lines          9490     2311     -7179
  Branches       2244      240     -2004
=============================================
- Hits           5097     1661     -3436
+ Misses         4223      509     -3714
+ Partials        170      141       -29

Flag
Coverage Δ

dashboards-observability
?

opensearch-observability
71.87% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...rds-observability/common/utils/settings_service.ts

...ic/components/custom_panels/custom_panel_table.tsx

.../public/components/event_analytics/utils/utils.tsx

...fig_panes/config_controls/config_switch_button.tsx

...nts/visualizations/charts/shared/shared_configs.ts

...alytics/components/common/plots/error_rate_plt.tsx

...g_panes/config_controls/config_treemap_parents.tsx

...s/config_panel/config_panes/default_vis_editor.tsx

...onents/application_analytics/components/create.tsx

...visualizations/visualization_specs/default_spec.ts

... and 227 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter ramneet-persistent

repo: opensearch-project/observability


text:
Fixes for BWC issues, grammar updating

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1235 (1a02b05) into 2.4 (04852a7) will decrease coverage by 3.24%.
The diff coverage is 90.00%.

@@             Coverage Diff              @@
##                2.4    #1235      +/-   ##
============================================
- Coverage     53.26%   50.01%   -3.25%
============================================
  Files           282      240      -42
  Lines          9690     7318    -2372
  Branches       2274     2036     -238
============================================
- Hits           5161     3660    -1501
+ Misses         4358     3628     -730
+ Partials        171       30     -141

Flag
Coverage Δ

dashboards-observability
50.01% <90.00%> (+2.35%)
:arrow_up:

opensearch-observability
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...c/components/visualizations/charts/bar/bar_type.ts
100.00% <ø> (ø)

...isualizations/charts/data_table/data_table_type.ts
100.00% <ø> (ø)

...omponents/visualizations/charts/lines/line_type.ts
77.77% <ø> (ø)

...ents/visualizations/charts/metrics/metrics_type.ts
100.00% <ø> (ø)

.../public/components/custom_panels/helpers/utils.tsx
73.22% <66.66%> (-0.16%)
:arrow_down:

.../public/components/event_analytics/utils/utils.tsx
56.28% <94.11%> (+4.28%)
:arrow_up:

...ic/components/visualizations/charts/lines/line.tsx
43.03% <0.00%> (-2.54%)
:arrow_down:

...ponents/visualizations/charts/helpers/viz_types.ts
29.56% <0.00%> (-0.87%)
:arrow_down:

...ervability/action/GetObservabilityObjectRequest.kt

...vability/action/DeleteObservabilityObjectAction.kt

... and 55 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter mengweieric


text:
Remove metrics unit of measures

Signed-off-by: Shenoy Pratik sgguruda@amazon.com
Description
Remove metrics unit of measures
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1261 (7cd41af) into 2.4 (0ea635a) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                2.4    #1261   +/-   ##
=========================================
  Coverage     70.27%   70.27%
  Complexity      292      292
=========================================
  Files            42       42
  Lines          2395     2395
  Branches        248      248
=========================================
  Hits           1683     1683
  Misses          571      571
  Partials        141      141

Flag
Coverage Δ

opensearch-observability
70.27% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


text:
Publish opensearch-dsl to pypi

Is your feature request related to a problem? Please describe
publish opensearch-dsl 1.0 to pypi
Repo: https://github.com/opensearch-project/opensearch-dsl-py
branch: 1.0
Describe the solution you'd like
Upload opensearch-dsl to pypi and test pip install opensearch-dsl works
Describe alternatives you've considered
No response
Additional context
No response

PR to fix twine issue - https://github.com/opensearch-project/opensearch-dsl-py/pull/30
Released - https://pypi.org/project/opensearch-dsl/1.0.0/

source: gharchive/issue

repo: opensearch-project/opensearch-build


text:
Registration for OpenSearchCon 2023 is now open!

Describe the blog post your would like to write
This is a blog post promoting OpenSearchCon 2023 registration and CFP.
What is the title of the blog post?
Registration for OpenSearchCon 2023 is now open!  (Full markdown of body below)
[START]

OpenSearchCon is back! The OpenSearch community will get together again September 27-29 to learn, share, and connect about all things search and analytics. The event is in-person only and there is no fee to attend.
We’re making room for more builders and more content by taking over the Sheraton Grand in Seattle. Special room rates are available for those who wish to stay at the event hotel. OpenSearchCon 2023 passes are limited, so we encourage everyone to register today in order to claim your spot at this year’s conference.
The Call for Papers is still open as well but only until the end of the day on June 16. If you have a story you’d like to share, submit your talk NOW on the Call for Presentations site - time is of the essence!
OpenSearchCon is for builders, with conference tracks designed to explore and learn more about building applications at scale as an open-source contributor.  The current schedule for OpenSearchCon:

Sept. 27 - Registration and check-in open (come get your swag!!), exhibit hall is open for exhibitors to set up, and afternoon workshops and round table discussions with partners, contributors and users. There will be an evening cocktail reception as well!
Sept. 28 - Main conference day with three session rooms/tracks, large exhibit hall, opening/closing keynote, and sessions all day. Food/drinks will be served regularly to keep the community going, and dinner in the main room at the end of the day
Sept. 29th - Half-day on Friday, focused on buttoning up any missed conversations, and a handful more of user and/or partner round table discussions.

This year, we will have three session rooms instead of a single main room.  The Search and Analytics tracks will cover a broad range of technology topics, including Open Telemetry, Lucene, Vector Databases, AI/ML, and more. The main stage Community track is where we’ll kick things off with the opening keynote, and will be all about living in the world of open-source software.
First and foremost, this is a community oriented developer meetup - and anyone who wants to become a better builder in this space ought to be there. See you in Seattle!
You can learn more and sign up for OpenSearchCon 2023 here.
Keep an eye for this blog for updates on #OpenSearchCon 2023. We’ll see you in Seattle!
[END]
Who are the authors?
Me - I have an author profile already.  how do I review and edit my author profile if I want to?
What is the proposed posting date?
Patti tells me I have to post it today.
What front-matter do you want loaded?
meta_description: "OpenSearchCon 2023 - Coming to the Sheraton Grand Hotel in Seattle September 27-29. Register today!"
excerpt: The OpenSearch Project invites all builders to Seattle this fall, to celebrate the success of the OpenSearch Project and look to the future of search, analytics, and Generative AI. Join us to learn how to build at scale!
authors:
- dimwetoth
feature_image: /assets/102261489/13cfc936-f1c7-4e4b-91c5-9927af4a05e2

Who are the authors?
Me - I have an author profile already. how do I review and edit my author profile if I want to?

Looks like your metadata is in _authors/daryllswager.markdown

authors: dimwetoth nateynateynate

repo: opensearch-project/project-website


text:
Past meetups are in wrong sort order

On http://opensearch.org/events/past.html, it appears that the past meetups are not in date order. It should be from most recent to least recent.

Closed by #278


text:
[BUG] Issues in threat intel monitor scans

Sorting by _seq_no field fails on empty index as mapping isn't created yet. This failure is expected and needs to be handled to return empty list instead of throwing exception and calling it a monitor failure
Caused by: org.opensearch.index.query.QueryShardException: No mapping found for [_seq_no] in order to sort on
        at org.opensearch.search.sort.FieldSortBuilder.resolveUnmappedType(FieldSortBuilder.java:564) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:411) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.sort.SortBuilder.buildSort(SortBuilder.java:168) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.parseSource(SearchService.java:1268) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.createContext(SearchService.java:998) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:606) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:579) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

There is an edge case scenario when there are hits in data but no iocs are actually present to scan. That case is causing creation of a grouped listener of size 0 which is throwing error
[2024-09-15T18:03:08,110][ERROR][o.o.s.t.i.s.IoCScanService] [384634db970ddbb1dc6211644bcfe4fe] Threat intel monitor oulm9JEBop9BFvxFVds2: Unexpected failure in running scan for 246 docs
java.lang.IllegalArgumentException: groupSize must be greater than 0 but was 0
        at org.opensearch.action.support.GroupedActionListener.<init>(GroupedActionListener.java:66)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.getGroupedListenerForIocScanFromAllIocTypes(SaIoCScanService.java:320)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.matchAgainstThreatIntelAndReturnMaliciousIocs(SaIoCScanService.java:254)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.IoCScanService.scanIoCs(IoCScanService.java:85)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$onGetIocTypeToIndices$7(TransportThreatIntelMonitorFanOutAction.java:186)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchDataFromShards$9(TransportThreatIntelMonitorFanOutAction.java:224)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:81)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.fetchLatestDocsFromShard(TransportThreatIntelMonitorFanOutAction.java:261)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$fetchLatestDocsFromShard$11(TransportThreatIntelMonitorFanOutAction.java:291)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.securityanalytics.threatIntel.model.monitor.TransportThreatIntelMonitorFanOutAction.lambda$searchShard$15(TransportThreatIntelMonitorFanOutAction.java:352)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:150)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Index out of bounds exception as wrong listener is used.
[2024-09-15T06:50:29,074][ERROR][o.o.s.t.i.s.SaIoCScanService] [efdc3b06f28dcf50f11616883b4a106e] Threat intel monitor oulm9JEBop9BFvxFVds2 scan with 48720 user data indicators failed for ioc Type ipv4-addr
Failed to execute phase [expand],
        at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:780)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:501)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:458)
        at org.opensearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:300)
        at org.opensearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:138)
        at org.opensearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:158)
        at org.opensearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:122)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:950)
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 1
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:63)
        at java.base/jdk.internal.util.Preconditions$2.apply(Preconditions.java:60)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
        at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
        at java.base/java.lang.invoke.VarHandleReferences$Array.compareAndSet(VarHandleReferences.java:655)
        at java.base/java.util.concurrent.atomic.AtomicReferenceArray.compareAndSet(AtomicReferenceArray.java:153)
        at org.opensearch.common.util.concurrent.AtomicArray.setOnce(AtomicArray.java:79)
        at org.opensearch.action.support.GroupedActionListener.onResponse(GroupedActionListener.java:75)
        at org.opensearch.securityanalytics.threatIntel.iocscan.service.SaIoCScanService.lambda$performScanForMaliciousIocsPerIocType$24(SaIoCScanService.java:369)
        at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90)
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115)
        at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109)
        at org.opensearch.core.action.ActionListener$5.onResponse(ActionListener.java:268)
        at org.opensearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:769)
        at org.opensearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:132)
        at org.opensearch.action.search.SearchPhase.recordAndRun(SearchPhase.java:61)
        at org.opensearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:491)
        ... 14 more

Was this fixed in #1317? Close?
[Catch All Triage - 1, 2, 3, 4]

authors: dblock eirsep

repo: opensearch-project/security-analytics


text:
[BUG] Security UI not showing for non-default admin user

Describe the bug
Referencing #578 which is closed but concerns Opendistro and not OpenSearch and #15 which is still open but concerns a very old version of Opendistro I also have problems to get the Security UI to show up. I have an admin user which has both the roles security_manager and security_rest_api_access as well as all_access assigned but I still do not see the Security UI. I'm expecting it within "OpenSearch plugins" but it is also not to be found within "Stack Management" if it could be there.
I'm unsure what role(s) a user needs to get access to the Security UI and if there is a bug or if I'm doing something wrong.
The documentation is not clear on this either. Even in #578 you find both above mentioned roles. I have not set security_rest_api_roles_enabled and I'm not sure how to set this within opensearch.yml.
To Reproduce
Steps to reproduce the behavior:

Login in with non-default admin user that has security_manager and/or security_rest_api_access roles assigned.
Click on the menu icon on the left.
Observe in the opened drawer that there is no "Security" menu entry within "OpenSearch plugins".

Expected behavior
I'm able to access the Security UI with a non-default admin user which has the right roles assigned (it is unclear which those are).
Plugins
OpenSearch Security plugin 1.0.0
and maybe others that come with a default setup
Host/Environment (please complete the following information):
unrelated to client config

After upgrade to the newest OpenSearch 1.2.0 the issue is still there. I am using OpenID Connect IdP, the admin backend role is mapped to all_access, security_manager and security_rest_api_access. No traces of the Security UI are visible.

I faced the same problem. The minimal configuration that worked for me is:

in opensearch.yml : plugins.security.restapi.roles_enabled: ["all_access"]
in roles_mapping: mapping all_access role to your admin user

Of course other combinations also work. My current setting is:

in opensearch.yml : plugins.security.roles_mapping_resolution: BOTH
in opensearch.yml : plugins.security.restapi.roles_enabled: ["admin"]
in internal_users.yml: adding the "admin" backend role to my admin user
in roles_mapping.yml: mapping the all_access role to "admin" backend role

I agree that documentation is not clear on this topic.

@jacen05 Awesome! Works like a charm, thanks a lot! 🍻


text:
Revert build breaking change from Lucene upgrade to 9.3.0

Signed-off-by: Craig Perkins cwperx@amazon.com
Description
Reverts https://github.com/opensearch-project/security/pull/1990. Core is reverting the Lucene upgrade for a future release and we will re-apply this change in tandem.

Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)

Maintenance
Check List

[ ] New functionality includes testing
[ ] New functionality has been documented
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@cwperks Is there a tracking issue on OpenSearch?

@peternied There was some back-and-forth regarding whether the Lucene 9.3.0 upgrade should be moved to a future release or included in the 2.2.0 release. This PR was drafted to revert the coordinated change in security in the event that core decided to downgrade back to Lucene 9.2.0.

@cwperks Thanks for the context

authors: cwperks peternied

repo: opensearch-project/security


text:
go-mocks: generate mocks based on rules

Description:
Currently the mocks are not generated consistently
across the code due to inconsistencies arising from:

mocks not being generated using the same mode (reflect vs source)
manually edited mock files

Both of the above present problems that are not easy to catch. For
example, manually edited mock files don't reflect the interface
verbatim and will introduce bugs.
This change addresses both these problems by:

defining mockgen rules using a file to specify the parameters
generating mocks the same way for each of the rules
integrating checks for the mocks in the CI

Signed-off-by: Shashank Ram shashr2204@gmail.com

Affected area:

New Functionality      [ ]
Documentation          [ ]
Install                [ ]
Control Plane          [ ]
CLI Tool               [ ]
Certificate Management [ ]
Networking             [ ]
Metrics                [ ]
SMI Policy             [ ]
Security               [ ]
Tests                  [ ]
CI System              [ ]
Performance            [ ]
Other                  [X]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? If so, did you notify the maintainers and provide attribution?
No

Codecov Report

Merging #2617 (eedb9c0) into main (5afa26d) will increase coverage by 0.12%.
The diff coverage is 4.08%.

@@            Coverage Diff             @@
##             main    #2617      +/-   ##
==========================================
+ Coverage   63.68%   63.81%   +0.12%
==========================================
  Files         152      152
  Lines        7110     7096      -14
==========================================
  Hits         4528     4528
+ Misses       2565     2551      -14
  Partials       17       17

Flag
Coverage Δ

unittests
63.81% <4.08%> (+0.12%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/certificate/mock_certificate_generated.go
0.00% <0.00%> (ø)

pkg/configurator/mock_client_generated.go
0.00% <ø> (ø)

pkg/endpoint/mock_provider_generated.go
0.00% <0.00%> (ø)

pkg/kubernetes/mock_controller_generated.go
0.00% <0.00%> (ø)

pkg/debugger/mock_debugger_generated.go
56.89% <100.00%> (+8.36%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5afa26d...5dd16f5. Read the comment docs.

authors: codecov-io shashankram

repo: openservicemesh/osm


text:
Sending arguments to bring the cluster up for a second time does not work

Not sure what the intention is here, but the way it is setup now sending aditional arguments when setting up a cluster for the second time does not work .
./oc-cluster up testing -e TEST=true
This is the content that is put into the run script (reformated for clarity)
oc cluster up
--public-hostname '127.0.0.1'
--host-data-dir '/Users/bjartek/.oc/profiles/testing/data'
--host-config-dir '/Users/bjartek/.oc/profiles/testing/config'
--routing-suffix ''
--use-existing-config
-e TEST=true
When i then take it down and up again with
./oc-cluster down && oc-cluster up testing -e TEST2=true
/Users/bjartek/.oc/profiles/testing/run -e TEST2=true is run
the output of that ( running with bash -x) is:
oc cluster up --public-hostname 127.0.0.1 --host-data-dir /Users/bjartek/.oc/profiles/testing/data --host-config-dir /Users/bjartek/.oc/profiles/testing/config --routing-suffix '' --use-existing-config -e TEST=true
In other words TEST2=true is not sent to the cluster.
I see two possible solutions here

this is the intended behavior
the intended behavior is that new arguments are sent to run aswell. We need to append '$@' at the end of the run script.

Either way it makes no difference to add $@ as an argument after sourcing run in line 80.

Not sure what to do here. The idea would be to have the ability to create a profile, and make only some values changeable, as some should not change, as data and volume dirs and ip, as these are set in the config when created.
There's an upstream change that will have "oc cluster up" not re-executing some commands when bringing an already existent cluster up, like creating the test project, logging in developer,
https://github.com/openshift/origin/issues/10999
I'll add some ideas I want to provide to this script, so if you're fancy to contribute will have some ideas on our intentions

Please add ideas and I will contribute.
Regards
Bjarte
Sent from my iPhone

On 8 Oct 2016, at 14:24, Jorge Morales Pou notifications@github.com wrote:
Not sure what to do here. The idea would be to have the ability to create a profile, and make only some values changeable, as some should not change, as data and volume dirs and ip, as these are set in the config when created.
There's an upstream change that will have "oc cluster up" not re-executing some commands when bringing an already existent cluster up, like creating the test project, logging in developer,
openshift/origin#10999
I'll add some ideas I want to provide to this script, so if you're fancy to contribute will have some ideas on our intentions
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@bjartek What is the use case for setting environment variables? Are you specifically debugging issues with OpenShift itself?
I can't see that these get passed through to any deployed application container. Does the OpenShift container honour certain environment variables?

I think this issue is based upon a misunderstanding from my side. The way profiles is implemented is a little bit different then what i though initially. Feel free to just discard this since i do not have any use case with the current profile implementation.

authors: GrahamDumpleton bjartek jorgemoralespou

license_source: bigquery


text:
[main] Mirror upstream branches

We used to not have this for this repository, I believe it's not time for having it

copied working file from eventing-core midstream (https://github.com/openshift-knative/eventing/blob/main/openshift/release/mirror-upstream-branches.sh)
call file in update-to-head.sh

/cc @creydr @matzew

repo: openshift-knative/eventing-kafka-broker


text:
sriov pkg: add DisablePlugins and GetTotalVFs methods

Added  :

WithDisablePlugins  and RemoveDisablePlugins funcs for the SriovOperatorConfig CR
GetTotalVFs func for the SriovNetworkNodeState CR

/lgtm

authors: achuzhoy evgenLevin

repo: openshift-kni/eco-goinfra


text:
manifests: relax node affinity for hypershift

For local deployment (not via OLM), we configure the controller manager using yaml files instead of CSV. These yamls use requiredDuringSchedulingIgnoredDuringExecution affinity but on hypershift it was agreed that controller and scheduler be scheduled on the hosted cluster which contains worker nodes only.
related to
https://github.com/openshift-kni/numaresources-operator/pull/1059.

/test ci-e2e-install-hypershift

repo: openshift-kni/numaresources-operator


text:
Fail faster

This change will fail if the cluster installation didn't start, and
won't wait for it to finish even if it didn't start. It also waits
less for files to download, and only waits long for the ISO file
download.

/lgtm

/lgtm

looks like ok

authors: achuzhoy ukalifon yobshans


text:
PLNSRVCE-1030: upgrade Openshift-Pipelines to 1.9

Locally tested ✔
and there doesn't seem to be any breaking changes.
Signed-off-by: Satyam Bhardwaj sabhardw@redhat.com

Did you also test infra-deployments to see nothing breaks with 1.9?

Did you also test infra-deployments to see nothing breaks with 1.9?

I didn't. I'm guessing I just need to do a draft PR in infra-deployments and check the CI progress.

Raised a PR in infra-deployments - https://github.com/redhat-appstudio/infra-deployments/pull/1242

@ramessesii2 I can't find the test logs. I am not sure if its the access issue or pods has been removed by now.

@ramessesii2 I can't find the test logs. I am not sure if its the access issue or pods has been removed by now.

No @prietyc123, I've not deleted any pods. I guess they're auto removed after a time interval. cc @xinredhat
FYI: I still have to resume working on this PR.

/retest

repo: openshift-pipelines/pipeline-service


text:
Upgraded kcp to release-0.4 for the local dev env

With the 0.4 release, removed the flags that are no more in use.

/lgtm
I may follow up on this PR to use go install to download & install the kcp binary. So far it looks like we only need kcp start.

@adambkaplan Yes, we only need kcp start (same goes for kcp running inside a pod too).
The kcp binary is generated from this line. (The make build command triggers a go build $params command which generates the kcp binary)

/lgtm
I may follow up on this PR to use go install to download & install the kcp binary. So far it looks like we only need kcp start.

yes that's a good point. However:

kcp uses "replaces" in go.mod, which seems to get into the way of go install github.com/kcp-dev/kcp@version
the existing approach allowed to follow a release branch (the prototype versions: 0.3 and now 0.4). Patches to release branches are not consistently tagged in kcp and with go install it is not possible to install binaries for a specific branch. Although, we may be fine without that.

authors: bnallapeta fgiloux

repo: openshift-pipelines/pipelines-service


text:
OCPBUGS-34399: Exposing chunksize variable to utilize docker registry config

Trying to work around a bug for r2 integration with s3 bucket, exposing chunksize variable by utilizing the docker registry S3 driver configuration option: https://docker-docs.uclv.cu/registry/storage-drivers/s3/
/hold for https://github.com/openshift/api/pull/1864 to merge
https://issues.redhat.com//browse/IR-471
/cc @flavianmissi

/test all

Running make test-unit locally gives me an error:
# k8s.io/apiserver/pkg/endpoints/handlers
vendor/k8s.io/apiserver/pkg/endpoints/handlers/watch.go:204:17: undefined: streaming.NewEncoderWithAllocator
# k8s.io/apiserver/pkg/server/routes
vendor/k8s.io/apiserver/pkg/server/routes/openapi.go:68:89: cannot use oa.Config (variable of type *common.Config) as *common.OpenAPIV3Config value in argument to builder3.BuildOpenAPISpecFromRoutes
# github.com/openshift/cluster-image-registry-operator/pkg/storage/pvc
pkg/storage/pvc/pvc.go:173:15: cannot use corev1.ResourceRequirements{…} (value of type "k8s.io/api/core/v1".ResourceRequirements) as "k8s.io/api/core/v1".VolumeResourceRequirements value in struct literal

can you have a look @deepsm007 ?

/retest

/test

/test all

https://github.com/openshift/cluster-image-registry-operator/pull/1036 to bump the api, client-go & library-go

/retest-required

/retest

unit test failing:
{  === RUN   TestMakePodTemplateSpecS3CloudFront
    podtemplatespec_test.go:560: expected env var REGISTRY_STORAGE_S3_CHUNKSIZE not found

since this env var is optional, should the test really be expecting it?

Looks like there might be some permissions missing from the operator as well, seeing this in the logs:
W0522 05:55:27.620106       1 reflector.go:539] github.com/openshift/client-go/config/informers/externalversions/factory.go:125: failed to list *v1.FeatureGate: featuregates.config.openshift.io is forbidden: User "system:serviceaccount:openshift-image-registry:cluster-image-registry-operator" cannot list resource "featuregates" in API group "config.openshift.io" at the cluster scope

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-image-registry-operator/1026/pull-ci-openshift-cluster-image-registry-operator-master-e2e-aws-ovn-upgrade/1793105514377252864/artifacts/e2e-aws-ovn-upgrade/gather-extra/artifacts/pods/openshift-image-registry_cluster-image-registry-operator-787b97fc95-pgxmm_cluster-image-registry-operator.log

/test all

/test e2e-aws-ovn-image-registry

/unhold

/retest

/retest

/retest

vsphere failures indicate that the feature gate detection code is interfering with graceful shutdown:
E0513 19:59:57.528191       1 starter.go:89] timed out waiting for FeatureGate detection
W0513 19:59:57.528346       1 builder.go:130] graceful termination failed, controllers failed with error: timed out waiting for FeatureGate detection

link to logs: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-image-registry-operator/1026/pull-ci-openshift-cluster-image-registry-operator-master-e2e-vsphere/1790093611942023168/artifacts/e2e-vsphere/gather-extra/artifacts/pods/openshift-image-registry_cluster-image-registry-operator-65f88ff885-mk88n_cluster-image-registry-operator.log
can you look into it @deepsm007 ? the operator needs to be able to shutdown gracefully as it previously did.

/test e2e-vsphere
/test e2e-vsphere-operator
/test e2e-azure-ovn
/test e2e-hypershift
/test e2e-hypershift-conformance

/test e2e-vsphere
/test e2e-vsphere-operator

vsphere failures indicate that the feature gate detection code is interfering with graceful shutdown:
E0513 19:59:57.528191       1 starter.go:89] timed out waiting for FeatureGate detection
W0513 19:59:57.528346       1 builder.go:130] graceful termination failed, controllers failed with error: timed out waiting for FeatureGate detection

link to logs: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-image-registry-operator/1026/pull-ci-openshift-cluster-image-registry-operator-master-e2e-vsphere/1790093611942023168/artifacts/e2e-vsphere/gather-extra/artifacts/pods/openshift-image-registry_cluster-image-registry-operator-65f88ff885-mk88n_cluster-image-registry-operator.log can you look into it @deepsm007 ? the operator needs to be able to shutdown gracefully as it previously did.

That run is from old commit, out of the last 4 runs 2 passed and 2 failed (not related to the PR).
This is the most recent run.

/test e2e-vsphere

/retest

/retest

/retest

/label qe-approved

/lgtm

hypershift failures are unrelated
/override ci/prow/e2e-hypershift
/override ci/prow/e2e-hypershift-conformance

/cherrypick release-4.16

authors: deepsm007 flavianmissi xiuwang


text:
Rebase to upstream master

There is no upstream release that would include fix for https://bugzilla.redhat.com/show_bug.cgi?id=1724998, so rebasing to upstream master.
Which issue(s) this PR fixes:
https://bugzilla.redhat.com/show_bug.cgi?id=1724998
Diff to upstream/master: https://github.com/kubernetes-csi/cluster-driver-registrar/compare/master...jsafrane:rebase-master
@openshift/storage

/lgtm

authors: bertinatto jsafrane

repo: openshift/csi-cluster-driver-registrar


text:
Add a minimum height to context switcher content area

Fixes

https://issues.redhat.com/browse/HAC-3103

Description

Added a minimum height to context switcher content area.

Type of change

[x] Bugfix

Screen shots / Gifs for design review

https://user-images.githubusercontent.com/6041994/226937482-962a1eee-2de8-4c1d-88db-52077e716d63.mov
Browser conformance:

[x] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge

/lgtm

repo: openshift/hac-dev


text:
CO-1114. Handle different worker-user-data secret name for 4.6

The worker-user-data secret name became worker-user-data-managed in 4.6.
Hive now supports alternate names for this secret.
This builds on https://github.com/openshift/hive/pull/1092. I was not able to push my changes into Michael's branch.

/lgtm

/lgtm cancel
This does not work for pre-releases of 4.6.

b98126cb...333ea957

Added a step in the workerUserData for stripping off the pre-release and build number from the OpenShift version prior to comparing it to 4.6.0 so that 4.6.0 pre-releases use the new name for the worker user data secret.

/lgtm

authors: abutcher dgoodwin staebler

repo: openshift/hive


text:
Various fixes related to hive developer experience (dx)

.gitignore: add venv so I stop accidentally adding it with git add hack/
hack/*.sh: use oc instead of kubectl
Makefile: add target for using podman to push to registry (local registry running in podman has no security setup, hence ignore TLS)
/assign @2uasimojo

/retitle HIVE-2656: Various fixes related to hive developer experience (dx)


text:
Add variant of kubefed's create-clusters.sh script.

What do we think about this? This is mostly a copy of kubefed's script I just included the wait util at the bottom directly.
First time usage is basically:
CREATE_INSECURE_REGISTRY=true CONFIGURE_INSECURE_REGISTRY_HOST=TRUE CONFIGURE_INSECURE_REGISTRY_CLUSTER=true NUM_CLUSTERS=1 hack/create-clusters.sh

After this you've got a registry you can use locally:
IMG=172.17.0.1:5000/hive:latest make docker-dev-push
DEPLOY_IMAGE=172.17.0.1:5000/hive:latest make deploy

Now if you want to redeploy your hive cluster, you need to drop a few options so you don't try to re-deploy the local registry:
CONFIGURE_INSECURE_REGISTRY_CLUSTER=true NUM_CLUSTERS=1 hack/create-clusters.sh

Do we like this approach? I'd almost like a script to setup local registry and configure the host, create clusters with kind directly, and a script to configure a kind cluster with the local registry.
Thoughts?

My view is that we can take this PR as is and iterate on it.
Without giving it too much thought, what I would like to see is a make target for pushing to the local registry and a make target for creating a kind cluster.
Pushing to the local registry:

Create the registry if it does not exist.
Pushes to the registry.

Creating a kind cluster:

Create the cluster if it is not already created.
Create the registry if it does not exist.
Configure the registry on the cluster.

Ok lets start with this then.

/lgtm

authors: dgoodwin staebler


text:
Wait for default ingresscontroller before starting ingress operator

We have a race condition between the HCCO and the ingress operator (which is created by CPO). Both of these components attempt to create a default ingresscontroller object. We need the HCCO to create the default ingresscontroller because when the ingress operator creates it then it does not have the correct publish strategy defined in the hosted cluster... The publish strategy is an immutable field so it can't be updated after creation.
This is solved by using an init container that won't pass until the HCCO creates the default ingresscontroller object which prevents the ingress operator from running until the object exists.

In parallel to this, I'm working to get an env var introduced into the ingress operator that will allow us to disable the default ingresscontroller auto creation (https://github.com/openshift/cluster-ingress-operator/pull/778)

Why not wait for the ingress operator change?

Why not wait for the ingress operator change?

I'm not confident we won't encounter this regardless of the cluster-ingress-operator changes. I can't guarantee there won't be a combination of components that lead to this issue... or that the operator change will make it into the 4.11 release.

authors: alvaroaleman davidvossel

repo: openshift/hypershift


text:
fix formating for releaseImage log

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/approve

/lgtm

/retitle NO-JIRA: fix formatting for releaseImage log

authors: Patryk-Stefanski bryan-cox enxebre muraee


text:
Add focus to e2e to filter the test cases

Helps in reducing the turn around time when debugging the test cases by individually focusing on a given test case.
e.g.
IMG=quay.io/harpatil/instaslice2-controller:2.4 IMG_DMST=quay.io/harpatil/instaslice2-daemonset:2.4 make test-e2e-ocp-emulated FOCUS="should verify that the Kubernetes node has the specified resource and matches total GPU memory"

/lgtm

authors: harche rphillips

repo: openshift/instaslice-operator


text:
ETCD-356: Changing etcd data directory to /var/lib/etcd

This change aligns the etcd data directory with that of OpenShift so that existing scripts, documentation, and customer/support knowledge can be more easily applied to MicroShift for etcd.

/hold
Waiting for 4.12 to branch so this can land in early 4.13

Doing it in 4.13 could potentially need a migration path?, for example customers upgrading from 4.12 to 4.13?, I propose we do it for 4.12 if possible. But I know we are already trying to squeeze too much for release. @ggiguash @dhellmann

Doing it in 4.13 could potentially need a migration path?, for example customers upgrading from 4.12 to 4.13?, I propose we do it for 4.12 if possible. But I know we are already trying to squeeze too much for release. @ggiguash @dhellmann

We do not have a requirement to be able to upgrade from 4.12 to 4.13, so it should be safe to do this work in 4.13.

This is planned to land after #976 the hold is still valid until that merges

This is ready for consideration @dhellmann @pmtk
/unhold

I'm a little torn on this. On the one hand, I see the value of being consistent. On the other hand, we don't want to support etcd running on its own, outside of MicroShift and this move makes that product decision a little fuzzier.
If we do want to do this, the PR needs to update hack/cleanup.sh as well.

I agree with @dhellmann 's comment. I'm not sure we'd want to handle etcd database out of MicroShift context.

/assign @dhellmann

/assign @pmtk

If we implement this change, the cleanup script also needs to be updated to delete the directory out of /var/lib/microshift

/hold
Pending a discussion and decision whether to proceed with this change

authors: dhellmann dusk125 ggiguash mangelajo

repo: openshift/microshift


text:
Start using RHEL 9.2 in the devenv and edge

Switch the development environment to use RHEL 9.2 beta
Enable osbuild configuration to use RHEL 9.2 beta channels
Use golang that comes with the OS
Fix the golang dependency in the MicroShift RPM spec file

Closes:

USHIFT-1023
OCPBUGS-10472

OCPBUGS-10472

/retest

/hold

Are we confident about committing to a move to a beta version of the OS? I'd feel more comfortable making this optional so we can still develop easily on 9.1 if we have to.

Are we confident about committing to a move to a beta version of the OS? I'd feel more comfortable making this optional so we can still develop easily on 9.1 if we have to.

I think we may want to try it and roll back in case there is a stopper issue. It's the OS used for OCP 4.13, so it should also be stable enough for MicroShift.

/cherrypick release-4.13

/unhold

The test just passed and then failed again.
/override ci/prow/e2e-reboot

authors: dhellmann ggiguash pacevedom


text:
create: enable image blocking

I want the ability to block and image, ex. alpine, and all images that depend on a blocked image's layer(s) with a warning log.

/lifecycle frozen

/close

authors: estroz lmzuccarelli

repo: openshift/oc-mirror


text:
Make blackbox-exporter launcher a clickable link

This PR makes the output of the blackbox-exporter launcher script
output a clickable link in the style of the cluster-console script.
Signed-off-by: Chris Collins collins.christopher@gmail.com

/retest

Nice! Thanks Chris!
Copied it into my container and it's working as expected.
[~ {prod} ]$ ocm backplane login osd-v4stg-aws
[~ {prod} (osd-v4stg-aws.z9a9.p1:default)]$ ./scratch/bin/blackbox-exporter
== Blackbox exporter is available at http://127.0.0.1:38501 ==
Handling connection for 9999

/lgtm

/retest

The builds look to be failing on the gpg key for the gcloud-sdk repo.
We aren't using that and we can install it from backplane-tools anyway so we will do that.
Step one is in #290.

/retest

authors: clcollins rendhalver

repo: openshift/ocm-container


text:
odo list follow up implementation

What does does this PR do / why we need it:
This PR implements

listing components without being in a context dir
json output for devfile components
--path for devfile components

Which issue(s) this PR fixes:
Fixes https://github.com/openshift/odo/issues/3521
PR acceptance criteria:

[ ] Unit test

[ ] Integration test

[ ] Documentation

[X] I have read the test guidelines

How to test changes / Special notes to the reviewer:

@dgolovin - this PR changes json output for config too. this is still WIP. I will inform once its consumeable

tested this branch merged with describe. It seems reporting the same component twice after pushing starter component from current folder to the cluster.
~/p/test-ws/nodejs-ex$ ~/p/vscode-openshift-tools/out/tools/linux/odo list -o json
{
	"kind": "List",
	"apiVersion": "odo.dev/v1alpha1",
	"metadata": {},
	"s2i_components": [],
	"devfile_components": [
		{
			"kind": "DevfileComponent",
			"apiVersion": "odo.dev/v1alpha1",
			"metadata": {
				"name": "nodejs",
				"creationTimestamp": null
			},
			"spec": {
				"namespace": "p4",
				"application": "app",
				"componentType": "nodejs"
			},
			"status": {
				"state": "Pushed"
			}
		},
		{
			"kind": "DevfileComponent",
			"apiVersion": "odo.dev/v1alpha1",
			"metadata": {
				"name": "nodejs",
				"creationTimestamp": null
			},
			"spec": {
				"namespace": "p4",
				"application": "app",
				"componentType": "nodejs"
			},
			"status": {
				"state": "Not Pushed"
			}
		}
	]
}

after calling odo delete the output is
{
	"kind": "List",
	"apiVersion": "odo.dev/v1alpha1",
	"metadata": {},
	"s2i_components": [],
	"devfile_components": [
		{
			"kind": "DevfileComponent",
			"apiVersion": "odo.dev/v1alpha1",
			"metadata": {
				"name": "nodejs",
				"creationTimestamp": null
			},
			"spec": {
				"namespace": "p4",
				"application": "app",
				"componentType": "nodejs"
			},
			"status": {
				"state": "Not Pushed"
			}
		}
	]
}

@girishramnani see my comments above.

@girishramnani from s2i component context folder it works as expected
~/p/test-ws/nodeja-ex1$ ~/p/vscode-openshift-tools/out/tools/linux/odo list -o json
{
	"kind": "List",
	"apiVersion": "odo.dev/v1alpha1",
	"metadata": {},
	"s2i_components": [
		{
			"kind": "Component",
			"apiVersion": "odo.dev/v1alpha1",
			"metadata": {
				"name": "nodejs-ex1",
				"namespace": "p4",
				"creationTimestamp": null
			},
			"spec": {
				"app": "app",
				"type": "nodejs",
				"sourceType": "local",
				"url": [
					"url2"
				],
				"env": [
					{
						"name": "DEBUG_PORT",
						"value": "5858"
					}
				]
			},
			"status": {
				"state": "Pushed"
			}
		}
	],
	"devfile_components": [
		{
			"kind": "DevfileComponent",
			"apiVersion": "odo.dev/v1alpha1",
			"metadata": {
				"name": "nodejs",
				"creationTimestamp": null
			},
			"spec": {
				"namespace": "p4",
				"application": "app",
				"componentType": "nodejs"
			},
			"status": {
				"state": "Pushed"
			}
		}
	]
}

Thank you for testing - will debug this.

@girishramnani would be good to have consistent names and structure (if possible) for devfile and s2i components.

move devfile component namespace name form spec to metadata
change devfile spec.application property to spec.app
change devfile spec.componentType to spec.type
add devfile spec.sourceType = "local" (optional)

Also you using kind as 'DevfileComponent' where 'odo describe' is still using Component kind. It should ether both using DevfileComponent or Component kind.
I would prefer 'DevfileComponent' kind so I can recognize component type based on json output only, without checking local context for devfile.yaml.

I would prefer 'DevfileComponent' kind so I can recognize component type based on json output only, without checking local context for devfile.yaml.

addressed this in https://github.com/openshift/odo/pull/3990

I didn't get a chance to add more tests hence opening a follow up for tests https://github.com/openshift/odo/issues/3995

It is not listing pushed s2i component outside context dir.

$ odo list
Devfile Components:
APP     NAME                     PROJECT     TYPE                 STATE
app     check-sp-final           default     java-spring-boot     Pushed
app     envcheck                 default     nodejs               Pushed
app     mnn                      default     java-spring-boot     Pushed
app     spring-failure-check     default     java-springboot      Pushed

from s2i context dir
[adisky@localhost nodejs-s2i]$ odo list
Devfile Components:
APP     NAME                     PROJECT     TYPE                 STATE
app     check-sp-final           default     java-spring-boot     Pushed
app     envcheck                 default     nodejs               Pushed
app     mnn                      default     java-spring-boot     Pushed
app     spring-failure-check     default     java-springboot      Pushed

Openshift Components:
APP     NAME         PROJECT        TYPE       SOURCETYPE     STATE
app     football     pgtqpdklxa     nodejs     local          Pushed

Update  odo list to use S2I Components as used in odo list --path

$ odo list --path ~/
Devfile Components:
APP     NAME         PROJECT       STATE          CONTEXT
app     envcheck     myproject     Not Pushed     /home/adisky/check-env-view

S2I Components:
APP     NAME                         PROJECT         TYPE       SOURCETYPE     STATE          CONTEXT
app     door2                        default         nodejs     local          Not Pushed     /home/adisky/check--nodejs/nodejs-ex
app     java-java-s2i-jcaz           default         java       local          Not Pushed     /home/adisky/check-s2i/java-s2i

$ odo list --path ~/ not listing pushed devfile components
Devfile Components:
APP     NAME         PROJECT       STATE          CONTEXT
app     envcheck     myproject     Not Pushed     /home/adisky/check-env-view

$ odo list
odo list
Devfile Components:
APP     NAME                     PROJECT     TYPE                 STATE
app     check-sp-final           default     java-spring-boot     Pushed
app     envcheck                 default     nodejs               Pushed

$ odo list --path ~/ not listing pushed devfile components
$ odo list -path ~/
Devfile Components:
APP     NAME         PROJECT       STATE          CONTEXT
app     envcheck     myproject     Not Pushed     /home/adisky/check-env-view

$ odo list
$ odo list
Devfile Components:
APP     NAME                     PROJECT     TYPE                 STATE
app     check-sp-final           default     java-spring-boot     Pushed
app     mnn                      default     java-spring-boot     Pushed

If the
app     check-sp-final           default     java-spring-boot     Pushed

is not present in the path then if wont be shown
Devfile Components:
APP     NAME         PROJECT     STATE          CONTEXT
app     blah         test123     Not Pushed     /Users/girish/GoProjects/src/github.com/openshift/odo/asd
app     asdfasdf     test123     Pushed         /Users/girish/GoProjects/src/github.com/openshift/odo/bb

S2I Components:
APP     NAME                    PROJECT     TYPE       SOURCETYPE     STATE      CONTEXT
app     nodejs-s2ihehe-beif     asdf        nodejs     local          Pushed     /Users/girish/GoProjects/src/github.com/openshift/odo/s2ihehe

the context of the component needs to be present

Looks like multiple test errors @girishramnani :
• Failure [77.301 seconds]
odo component command tests
/home/travis/gopath/src/github.com/openshift/odo/tests/integration/cmd_cmp_test.go:13
  Creating component
  /home/travis/gopath/src/github.com/openshift/odo/tests/integration/component.go:52
    should list out pushed components of different projects in json format along with path flag [It]
    /home/travis/gopath/src/github.com/openshift/odo/tests/integration/component.go:148
    Expected
        <*json.SyntaxError | 0xc00044a3a0>: {
            msg: "invalid character 'G' looking for beginning of value",
            Offset: 1,
        }
    to be nil
    /home/travis/gopath/src/github.com/openshift/odo/tests/integration/component.go:173

Did a code review + tested it locally!
/approve

Surprised to see that the TYPE column is not displayed if you add the --path=xx parameter, yet is displayed if you don't use --path.

Tested on windows10. Due to limited time i took two test script that has changes along with this pr and few random test script test-cmd-app, test-cmd-devfile-describe, test-cmd-devfile-test and test-cmd-devfile-url . These selected test script run on 4.5 cluster went fine.

/retest

I take 100% responsibility.
I am manually merging this as Prow is taking very long to merge in.
This PR has passed on Travis and it has passed 160/162 tests on Prow (the last two tests were timeout flakes).

authors: adisky amitkrout cdrage dgolovin girishramnani scottkurz

repo: openshift/odo


text:
WIP [release-4.7] Use rawhide stream

Use published packages built on OBS instead of Fedora's bodhi and
modularity

/retest

/retest

/retest

/retest

4.8 is using F34 content now

repo: openshift/okd-machine-os


text:
OSD-20513: Add support for the --noproxy directive in curl

What does this PR do? / Related Issues / Jira
This adds an additional Proxy config to support passing --noproxy to curl when probing urls. The values passed will not be proxied through the configured http/https proxy.
I opted to not update the legacy probe based on it being called "legacy" but am open to go down that route if we still have use cases for it.
I tested this with the CLI locally and ran the integration tests as well.
On the integration tests, I ran into an aws sdk issue that required bumping the dependencies.
Checklist

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have made corresponding change to the default configuration files
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have tested the functionality against gcp / aws, it doesn't cause any regression
[ ] I have added execution results to the PR's readme

Reviewer's Checklist

[ ] (This needs to be done after technical review) I've run the branch on my local, verified that the functionality is ok

How to test this PR locally / Special Instructions

Logs

/label feature

Codecov Report
Attention: Patch coverage is 7.14286% with 26 lines in your changes missing coverage. Please review.

Project coverage is 25.87%. Comparing base (d86f9d5) to head (a86013e).

Files with missing lines
Patch %
Lines

cmd/egress/cmd.go
0.00%
22 Missing :warning:

pkg/proxy/proxy.go
0.00%
2 Missing :warning:

pkg/verifier/aws/entry_point.go
0.00%
1 Missing :warning:

pkg/verifier/gcp/entry_point.go
0.00%
1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #280      +/-   ##
==========================================
- Coverage   25.91%   25.87%   -0.04%
==========================================
  Files          25       26       +1
  Lines        1837     1847      +10
==========================================
+ Hits          476      478       +2
- Misses       1333     1341       +8
  Partials       28       28

Files with missing lines
Coverage Δ

pkg/probes/curl/curl_json.go
83.33% <100.00%> (+0.33%)
:arrow_up:

pkg/verifier/aws/entry_point.go
0.00% <0.00%> (ø)

pkg/verifier/gcp/entry_point.go
0.00% <0.00%> (ø)

pkg/proxy/proxy.go
0.00% <0.00%> (ø)

cmd/egress/cmd.go
0.00% <0.00%> (ø)

/label tide/merge-method-squash

@abyrne55 addressed feedback and ran another set of tests. See below for the curl behavior as tested on it's own as well.
> export http_proxy=squid.corp.redhat.com:3128 https_proxy=squid.corp.redhat.com:3128 no_proxy=google.com
> curl redhat.com -w "%{json}" -Z -s -o /dev/null | jq .proxy_used
1
> curl google.com -w "%{json}" -Z -s -o /dev/null | jq .proxy_used
0

Tested this locally using a modified version of examples/aws/terraform/vpc-proxied-explicit. All successful. Good stuff @joshbranham. I'm approving, but before I LGTM, did we not want to put quotes around the NO_PROXY metavar in the userdata template, just in case a space sneaks in there?
/approve

Tested this locally using a modified version of examples/aws/terraform/vpc-proxied-explicit. All successful. Good stuff @joshbranham. I'm approving, but before I LGTM, did we not want to put quotes around the NO_PROXY metavar in the userdata template, just in case a space sneaks in there?
/approve

It shouldn't be possible based on strings.Join(slice, ",") but I added quotes on both templates just to be sure 👍

authors: abyrne55 codecov-commenter joshbranham

repo: openshift/osd-network-verifier


text:
Fix bug - improve the help message of 'rosa unlink ocm-role'

Related: SDA-5449

@zgalor @zvikorn @leshemnoa PTAL

/lgtm

/assign @zgalor

/lgtm

authors: oriAdler zgalor zvikorn

repo: openshift/rosa


text:
Fix help for --compute-nodes

The flag --compute-nodes is used to pass total number
of compute nodes, not compute nodes per zone.
Signed-off-by: Sai Sindhur Malleni smalleni@redhat.com

/lgtm
Thanks @smalleni for the update.

authors: ciaranRoche smalleni


text:
Add source info into callback

Fixes: https://github.com/openshift/source-to-image/issues/66

@csrwng @soltysh PTAL

LGTM

LGTM

[merge]

authors: csrwng mfojtik soltysh

repo: openshift/source-to-image


text:
Unable to configure controller and webhook deployments to pull images from private registry

The pipeline and triggers controller and webhook Deployments created by the openshift-pipelines-operator include image references specific to quay.io. This is a problematic for users that must host these images in a private registry or for disconnected users. Provide a mechanism to make these image references configurable.

/kind feature
This is more or less in the roadmap (to support air-gapped installation for example) :angel:

repo: openshift/tektoncd-pipeline-operator


text:
Bug 2108054: Reset metrics only after all check succeed

blockUpgradeOrDegradeCluster is called twice during sync(), therefore it's not a good place to reset overall error metrics. Consider this scenario:
1st sync():

loginToVCenter() succeeds

-> the first blockUpgradeOrDegradeCluster() gets CheckActionPass

runClusterCheck() fails

-> the second blockUpgradeOrDegradeCluster() gets say CheckActionBlockDriverInstall and sets appropriate error metric

2nd sync():

loginToVCenter() succeeds

-> the first blockUpgradeOrDegradeCluster() gets CheckActionPass -> this resets error metric!

runClusterCheck() does not run any checks because it's too early after the previous check, therefore it does not set any    error metric

As result, the error metric is reset, but it should stay set until runClusterCheck() confirms the error has been fixed.
Therefore reset the metric when both loginToVCenter and runClusterCheck succeeds.
cc @openshift/storage

/lgtm

/retest

/retest

authors: gnufied jsafrane


text:
iframe for oscu blog panel on trabian.com

This adds a page (iframe.htm) and a sass file (iframe.scss) to allow minimal-effort transclusion of new blogs into trabian.com

Also, this adds pym.js that allows for fully-responsive iframes. It's only 4kb, which is pretty cool.

license: mit


text:
🛑 Demo is down

In 66d5396, Demo (https://demo.opensourcepos.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Demo is back up in af400c9.

repo: opensourcepos/upptime

license: MIT


text:
How to set column width for certain sheet only

Hello,
Is there any options to set column width for certain sheet only ?

Currently, no: many options indeed apply to all sheets unfortunately. We should have a better separation between workbook options and worksheet ones.
Any PR is welcome though

repo: openspout/openspout


text:
genpkey -genparam fails for some EC curves

In latest 1.1.1-stable (but also in the first 1.1.1 release, and persists in current master):
; openssl version
OpenSSL 1.1.1  11 Sep 2018
; # with a common binary curve
; openssl genpkey -genparam -algorithm EC -pkeyopt 'ec_paramgen_curve:sect233r1' -text
-----BEGIN EC PARAMETERS-----
BgUrgQQAGw==
-----END EC PARAMETERS-----
ECDSA-Parameters: (233 bit)
ASN1 OID: sect233r1
NIST CURVE: B-233
; # using some "special" curves instead, next command fails unexpectedly
; openssl genpkey -genparam -algorithm EC -pkeyopt 'ec_paramgen_curve:Oakley-EC2N-3' -pkeyopt 'ec_param_enc:named_curve' -text
-----BEGIN EC PARAMETERS-----
-----END EC PARAMETERS-----
Error writing key
140234914886080:error:100BF079:elliptic curve routines:i2d_ECPKParameters:i2d ecpkparameters failure:../crypto/ec/ec_asn1.c:882:
140234914886080:error:100BF079:elliptic curve routines:i2d_ECPKParameters:i2d ecpkparameters failure:../crypto/ec/ec_asn1.c:882:
140234914886080:error:09072007:PEM routines:PEM_write_bio:BUF lib:../crypto/pem/pem_lib.c:658:
ECDSA-Parameters: (154 bit)
ASN1 OID: Oakley-EC2N-3
; # but the same command works if encoding to explicit parameters
; openssl genpkey -genparam -algorithm EC -pkeyopt 'ec_paramgen_curve:Oakley-EC2N-3' -pkeyopt 'ec_param_enc:explicit' -text
-----BEGIN EC PARAMETERS-----
MIGUAgEBMB0GByqGSM49AQIwEgICAJsGCSqGSM49AQIDAgIBPjAsBBQAAAAAAAAA
AAAAAAAAAAAAAAAAAAQUAAAAAAAAAAAAAAAAAAAAAAAHM48EKQQAAAAAAAAAAAAA
AAAAAAAAAAAAewAAAAAAAAAAAAAAAAAAAAAAAAHIAhQCqqqqqqqqqqqqx/PHiBvQ
ho+obAIBAw==
-----END EC PARAMETERS-----
ECDSA-Parameters: (154 bit)
Field Type: characteristic-two-field
Basis Type: tpBasis
Polynomial:
    08:00:00:00:00:00:00:00:00:00:00:00:40:00:00:
    00:00:00:00:01
A:    0
B:    471951 (0x7338f)
Generator (uncompressed):
    04:00:00:00:00:00:00:00:00:00:00:00:00:00:00:
    00:00:00:00:00:7b:00:00:00:00:00:00:00:00:00:
    00:00:00:00:00:00:00:00:00:01:c8
Order:
    02:aa:aa:aa:aa:aa:aa:aa:aa:aa:c7:f3:c7:88:1b:
    d0:86:8f:a8:6c
Cofactor:  3 (0x3)
; # yet generating a key without explicit parameters does not yield a failure!!
; openssl genpkey -algorithm EC -pkeyopt 'ec_paramgen_curve:Oakley-EC2N-3' -text
-----BEGIN PRIVATE KEY-----
MFkCAQAwCQYHKoZIzj0CAQRJMEcCAQEEFAHhSAciTgkK8EsygXni/NBIhdG+oSwD
KgAEBTq+bixE479sUE4xJJJm+x6y78MBFPDo9h+E1znuqY5z+Yxo/D9zaw==
-----END PRIVATE KEY-----
Private-Key: (154 bit)
priv:
    01:e1:48:07:22:4e:09:0a:f0:4b:32:81:79:e2:fc:
    d0:48:85:d1:be
pub:
    04:05:3a:be:6e:2c:44:e3:bf:6c:50:4e:31:24:92:
    66:fb:1e:b2:ef:c3:01:14:f0:e8:f6:1f:84:d7:39:
    ee:a9:8e:73:f9:8c:68:fc:3f:73:6b
ASN1 OID: Oakley-EC2N-3
; # nor does generating a key encoding with explicit parameters
; openssl genpkey -algorithm EC -pkeyopt 'ec_paramgen_curve:Oakley-EC2N-3' -pkeyopt 'ec_param_enc:explicit' -text
-----BEGIN PRIVATE KEY-----
MIHxAgEAMIGgBgcqhkjOPQIBMIGUAgEBMB0GByqGSM49AQIwEgICAJsGCSqGSM49
AQIDAgIBPjAsBBQAAAAAAAAAAAAAAAAAAAAAAAAAAAQUAAAAAAAAAAAAAAAAAAAA
AAAHM48EKQQAAAAAAAAAAAAAAAAAAAAAAAAAewAAAAAAAAAAAAAAAAAAAAAAAAHI
AhQCqqqqqqqqqqqqx/PHiBvQho+obAIBAwRJMEcCAQEEFABPnwFTo0d1svnE81dE
Pc/HMgl1oSwDKgAEAN+YcZmOVXfgdgEiT2E9tEmIM4oARhuzPbbxDbc0e+1kuxBT
oY4/IA==
-----END PRIVATE KEY-----
Private-Key: (154 bit)
priv:
    00:4f:9f:01:53:a3:47:75:b2:f9:c4:f3:57:44:3d:
    cf:c7:32:09:75
pub:
    04:00:df:98:71:99:8e:55:77:e0:76:01:22:4f:61:
    3d:b4:49:88:33:8a:00:46:1b:b3:3d:b6:f1:0d:b7:
    34:7b:ed:64:bb:10:53:a1:8e:3f:20
Field Type: characteristic-two-field
Basis Type: tpBasis
Polynomial:
    08:00:00:00:00:00:00:00:00:00:00:00:40:00:00:
    00:00:00:00:01
A:    0
B:    471951 (0x7338f)
Generator (uncompressed):
    04:00:00:00:00:00:00:00:00:00:00:00:00:00:00:
    00:00:00:00:00:7b:00:00:00:00:00:00:00:00:00:
    00:00:00:00:00:00:00:00:00:01:c8
Order:
    02:aa:aa:aa:aa:aa:aa:aa:aa:aa:c7:f3:c7:88:1b:
    d0:86:8f:a8:6c
Cofactor:  3 (0x3)

this will cause test failures in test_genec once #12305 is merged to master/backported to 1.1.1 as the error will be reflected also in the exit status.
Curves showing the problem:

Oakley-EC2N-3
Oakley-EC2N-4

These curves are "special" — as in "stay away from them, HERE BE DRAGONS is an understatement, dragons ran away in fear!" —:
https://github.com/openssl/openssl/blob/92db29e5e8c26af79f740dfd3bddfeae9b5345a8/crypto/ec/ec_curve.c#L3073-L3081
https://github.com/openssl/openssl/blob/92db29e5e8c26af79f740dfd3bddfeae9b5345a8/crypto/ec/ec_curve.c#L2157-L2226
Yet I fail to see the difference that could trigger the failure in i2d_ECPKParameters() when compared to a "normal" binary curve:
https://github.com/openssl/openssl/blob/92db29e5e8c26af79f740dfd3bddfeae9b5345a8/crypto/ec/ec_curve.c#L2881-L2882
https://github.com/openssl/openssl/blob/92db29e5e8c26af79f740dfd3bddfeae9b5345a8/crypto/ec/ec_curve.c#L1184-L1220

@bbbrumley @slontis suggestions on what might be going on?

Notice also https://github.com/openssl/openssl/pull/12291#issuecomment-650708674, that first uncovered the issue, quoted here for convenience:

7b1e105bde5414885a64f4c87d6a2ef063e031b6 also uncovers another error (unrelated to FIPS mode) that we missed before (it's the unexpected failure in test_genec):
; util/wrap.pl apps/openssl genpkey -genparam -algorithm EC -pkeyopt 'ec_paramgen_curve:Oakley-EC2N-4' -pkeyopt 'ec_param_enc:named_curve'
-----BEGIN EC PARAMETERS-----
-----END EC PARAMETERS-----
Error writing key
C0E03ABB137F0000:error::elliptic curve routines:i2d_ECPKParameters:i2d ecpkparameters failure:crypto/ec/ec_asn1.c:971:
C0E03ABB137F0000:error::elliptic curve routines:i2d_ECPKParameters:i2d ecpkparameters failure:crypto/ec/ec_asn1.c:971:
C0E03ABB137F0000:error::PEM routines:PEM_write_bio:BUF lib:crypto/pem/pem_lib.c:660:

(gdb) bt
#0  i2d_ECPKParameters (a=0x732400, out=0x0) at crypto/ec/ec_asn1.c:971
#1  0x00007ffff76e58de in PEM_ASN1_write_bio (i2d=0x7ffff75f9ab0 <i2d_ECPKParameters>, name=0x7ffff7846b4b "EC PARAMETERS", bp=0x72c910, x=0x732400, enc=0x0, kstr=0x0, klen=0, callback=0x0, u=0x0) at crypto/pem/pem_lib.c:337
#2  0x00007ffff76e3a25 in PEM_write_bio_ECPKParameters (out=0x72c910, x=0x732400) at crypto/pem/pem_all.c:158
#3  0x00007ffff77e89a7 in ec_param_pem (vctx=0x700070, eckey=0x7189a0, cout=0x7324c0, cb=0x7ffff77297c0 <serializer_passphrase_out_cb>, cbarg=0x72b8f0) at providers/implementations/serializers/serializer_ec_param.c:116
#4  0x00007ffff772970d in serializer_EVP_PKEY_to_bio (ctx=0x72b8f0, out=0x7324c0) at crypto/serializer/serializer_pkey.c:267
#5  0x00007ffff77279f2 in OSSL_SERIALIZER_to_bio (ctx=0x72b8f0, out=0x7324c0) at crypto/serializer/serializer_lib.c:16
#6  0x00007ffff76e92de in PEM_write_bio_Parameters (out=0x7324c0, x=0x718870) at crypto/pem/pem_pkey.c:136
#7  0x00000000004464ec in genpkey_main (argc=0, argv=0x7fffffffdf80) at apps/genpkey.c:180
#8  0x000000000044e5b7 in do_cmd (prog=0x6f8ab0, argc=8, argv=0x7fffffffdf80) at apps/openssl.c:397
#9  0x000000000044e230 in main (argc=8, argv=0x7fffffffdf80) at apps/openssl.c:272

So, thanks to @bbbrumley , the problem here seem to be that these two curves don't have an OID assigned.
So, actually, we should get a failure on keygen as well with ec_param_enc:named_curve and we currently aren't.
Compare:
; # generate a B-233 key, notice the `:sect233r1` OID encoded at byte 16
; util/shlib_wrap.sh apps/openssl genpkey -algorithm EC -pkeyopt 'ec_paramgen_curve:B-233' | openssl asn1parse
    0:d=0  hl=2 l= 126 cons: SEQUENCE
    2:d=1  hl=2 l=   1 prim: INTEGER           :00
    5:d=1  hl=2 l=  16 cons: SEQUENCE
    7:d=2  hl=2 l=   7 prim: OBJECT            :id-ecPublicKey
   16:d=2  hl=2 l=   5 prim: OBJECT            :sect233r1
   23:d=1  hl=2 l= 103 prim: OCTET STRING      [HEX DUMP]:3065020101041E00313C3F21E9CAAB5B5AB97BD93C4CF0D300CA5E7CF1AFD24E40BF844DA3A140033E0004007DBBFB0336E19B7E263DD8CE728AE2F002664E7280415689044F89DB2500F9EB792E2BF588FB3A48C68A269F79FEC3A6114E7EC2271C38F88C5ED9
; # generate a ipsec4 key, notice no OID is being included in the output
; util/shlib_wrap.sh apps/openssl genpkey -algorithm EC -pkeyopt 'ec_paramgen_curve:Oakley-EC2N-4' | tee key | openssl asn1parse
    0:d=0  hl=2 l= 100 cons: SEQUENCE
    2:d=1  hl=2 l=   1 prim: INTEGER           :00
    5:d=1  hl=2 l=   9 cons: SEQUENCE
    7:d=2  hl=2 l=   7 prim: OBJECT            :id-ecPublicKey
   16:d=1  hl=2 l=  84 prim: OCTET STRING      [HEX DUMP]:305202010104170AFC4B0519F7D5EF940655185F0DF83BD3AA4E09F4B571A13403320004001F45147ED4C61CE12BEBF34B0BEAB9D75DF25BBA74C662008C879475889A4CB009019AA2F357D5B8F3EB53C92AF099
; # try to read again the just generated key: no OID is included, openssl cannot know how to decode this EC key
; util/shlib_wrap.sh apps/openssl pkey -in key -text
unable to load key
140380458363840:error:100DC08E:elliptic curve routines:eckey_type2param:decode error:crypto/ec/ec_ameth.c:134:
140380458363840:error:100D5010:elliptic curve routines:eckey_priv_decode:EC lib:crypto/ec/ec_ameth.c:222:
140380458363840:error:0606F091:digital envelope routines:EVP_PKCS82PKEY:private key decode error:crypto/evp/evp_pkey.c:44:
140380458363840:error:0907B00D:PEM routines:PEM_read_bio_PrivateKey:ASN1 lib:crypto/pem/pem_pkey.c:88:

This issue  lead to several PRs that concur to fix it. As a summary:
In master:

e0137ca92b [EC][ASN1] Detect missing OID when serializing EC parameters and keys (#12313)
466d30c0d7 [apps/genpkey] exit status should not be 0 on output errors (#12305)
1c9761d0b5 [test][15-test_genec] Improve EC tests with genpkey (#12307)

In 1.1.1:

2797fea760 [EC][ASN1] Detect missing OID when serializing EC parameters and keys (#12312)
1940c092a5 [apps/genpkey] exit status should not be 0 on output errors (#12305)
e1c246bd76 [test][15-test_genec] Improve EC tests with genpkey (#12308)

repo: openssl/openssl


text:
Remove or resolve TODO 3.0 in man7/provider-cipher.pod

There is a quite impactful TODO:
=item "tlsaad" (B<OSSL_CIPHER_PARAM_AEAD_TLS1_AAD>) <octet string>

=for comment TODO(3.0): Consider changing this interface so that all ciphers
use the standard AEAD interface - rather than having this special purpose
interface for TLS

Sets TLSv1.2 AAD information for the associated cipher ctx.
TLSv1.2 AAD information is always 13 bytes in length and is as defined for the
"additional_data" field described in section 6.2.3.3 of RFC5246.

=item "tlsaadpad" (B<OSSL_CIPHER_PARAM_AEAD_TLS1_AAD_PAD>) <unsigned integer>

Gets the length of the tag that will be added to a TLS record for the AEAD
tag for the associated cipher ctx.
The length of the "tlsaadpad" parameter should not exceed that of a B<size_t>.

=item "tlsivfixed" (B<OSSL_CIPHER_PARAM_AEAD_TLS1_IV_FIXED>) <octet string>

=for comment TODO(3.0): This interface needs completely redesigning!

Sets the fixed portion of an IV for an AEAD cipher used in a TLS record
encryption/ decryption for the associated cipher ctx.

I don't think we should consider doing this for 3.0